Menu
9.33% Conversion Rate Increase
by placing Shipping information
above-the-fold on Product page
by placing Shipping information
above-the-fold on Product page
Shopify Customer: Under NDA
Shopify Customer: Can't share the name of the website because of NDA
Short Description of A/B test: It was the 10th A/B test for this client and we decided to implement it after we have found that by displaying Shipping information on Product page we can increase Conversion Rate
Previously we displayed this info below-the-fold under ATC (add-to-cart) button, so we decided to increase number of people who see this information. So we duplicated Shipping section (the first one remained below ATC button), while the new one was placed above the main image slider.
Key Results:
CRO toolkit used:
Technical Details of A/B test:
Short Description of A/B test: It was the 10th A/B test for this client and we decided to implement it after we have found that by displaying Shipping information on Product page we can increase Conversion Rate
Previously we displayed this info below-the-fold under ATC (add-to-cart) button, so we decided to increase number of people who see this information. So we duplicated Shipping section (the first one remained below ATC button), while the new one was placed above the main image slider.
Key Results:
- 9.33% Ecommerce Conversion Rate Increase
- 5.3% Revenue Increase
- 8.09% Add-to-cart Rate Increase
- 8.5% Engagement Increase (with Color/Size swatches)
CRO toolkit used:
- Google Analytics analysis
- A/B testing with Visual Website Optimizer
Technical Details of A/B test:
- Length: from 05.02.2017 to 13.02.2017
- Targeting: Mobile users that visited any product page
- A/B testing tool: Visual Website Optimizer integrated with Google Analytics
- Statistical significance of results:
- 75% (for Ecommerce Conversion Rate)
- 90% (for Add-to-Cart Rate)
- Sample Size: 7250 in Control and 7000 in Variation
- Number of Conversions: 218 in Control and 231 in Variation
Intro:
We started to work with this website (can't share exact name because of NDA) in the end of November 2017 and decided to use standard CRO framework that contains all types of analysis (User Behaviour, Heuristic, Google Analytics analysis) before A/B testing.
After initial CRO audit and first A/B tests, we have implemented Customer Polls and Surveys on key steps of Conversion funnel to collect feedback directly from customers. Based on Customer responses we have developed a few hypothesis and in A/B test #7 we have found that by displaying Shipping information on Product page we can significantly impact on Conversion Rate (We had 27% Revenue Increase in A/B test #7).
From that point we focused on more aggressive promotion of Shipping information in further A/B tests.
A/B testing hypothesis:
Based on Customer Surveys after purchases, we have found that the biggest concern of our customers was lack of Shipping information (Time, International shipping, terms, price). Also based on results of A/B test #7 we found that Shipping section on Product pages increased ECR and Revenue by 27%.
In addition to this data, we have found that only 9% of Mobile visitors reach this section, so it means that the majority of potential customers don't see information about Shipping.
We started to work with this website (can't share exact name because of NDA) in the end of November 2017 and decided to use standard CRO framework that contains all types of analysis (User Behaviour, Heuristic, Google Analytics analysis) before A/B testing.
After initial CRO audit and first A/B tests, we have implemented Customer Polls and Surveys on key steps of Conversion funnel to collect feedback directly from customers. Based on Customer responses we have developed a few hypothesis and in A/B test #7 we have found that by displaying Shipping information on Product page we can significantly impact on Conversion Rate (We had 27% Revenue Increase in A/B test #7).
From that point we focused on more aggressive promotion of Shipping information in further A/B tests.
A/B testing hypothesis:
Based on Customer Surveys after purchases, we have found that the biggest concern of our customers was lack of Shipping information (Time, International shipping, terms, price). Also based on results of A/B test #7 we found that Shipping section on Product pages increased ECR and Revenue by 27%.
In addition to this data, we have found that only 9% of Mobile visitors reach this section, so it means that the majority of potential customers don't see information about Shipping.
Scroll Tracking data from Hotjar
Based on this data, we expect that by duplicating Shipping section to above-the-fold part of Product pages, we will increase ECR and Revenue.
Preparation of testing variation:
As this Shipping section was dynamic (name of the Country and Flag were added dynamically based on GEO IP) and was inserted on the page by Shopify app with some delays, we had to use jQuery to implement this A/B test.
As a result, we have created jQuery code that checked every 0.5ms if Shipping section appeared on the page and if so, then it was duplicated and displayed on above-the-fold part of the page.
Preparation of testing variation:
As this Shipping section was dynamic (name of the Country and Flag were added dynamically based on GEO IP) and was inserted on the page by Shopify app with some delays, we had to use jQuery to implement this A/B test.
As a result, we have created jQuery code that checked every 0.5ms if Shipping section appeared on the page and if so, then it was duplicated and displayed on above-the-fold part of the page.
Control and Variation of A/B test #10 from VWO preview
Split-testing using Visual Website Optimizer
For A/B and Split testing we have chosen Visual Website Optimizer as it's one of the most powerful tool on the market and it can be easily integrated with Google Analytics for deep post-analysis.
For A/B and Split testing we have chosen Visual Website Optimizer as it's one of the most powerful tool on the market and it can be easily integrated with Google Analytics for deep post-analysis.
A/B-test was started on 05.02.2017 and was stopped on 13.02.2017.
Results:
1. Ecommerce Conversion Rate increased by 9.33% with 75% Statistical significance
2. Revenue increased by 5.3%
3. Add-to-Cart Rate increased by 8.09% with 90% Statistical signfiicance
4. Engagement with Color/Size swatches for choosing product variation increased by 8.5%
Split-testing rules:
- A/B test was active within more than 2 Business Cycles (as 95% of all transactions happens on the first visit)
- Both variations had more than 100 Conversions
Learnings for further A/B tests:
As we have already found through user feedback and previous A/B test, Shipping information plays very important role in decision-making process of our customers, so we have to promote it more aggressively.
Next steps:
Results:
1. Ecommerce Conversion Rate increased by 9.33% with 75% Statistical significance
2. Revenue increased by 5.3%
3. Add-to-Cart Rate increased by 8.09% with 90% Statistical signfiicance
4. Engagement with Color/Size swatches for choosing product variation increased by 8.5%
Split-testing rules:
- A/B test was active within more than 2 Business Cycles (as 95% of all transactions happens on the first visit)
- Both variations had more than 100 Conversions
Learnings for further A/B tests:
As we have already found through user feedback and previous A/B test, Shipping information plays very important role in decision-making process of our customers, so we have to promote it more aggressively.
Next steps:
- Redistribute 100% traffic to Variation and create task for Dev Team to implement Shipping section on Live website.
- As Shipping section appears on the page through Shopify app that works with 4-5 seconds delay, some users scroll-down the page, so they can miss shipping section. So we have to think how to ensure that 100% users will see it. Probably it can be achieved by placing this section below the main image.
- As Shipping play so big role in decision-making process, then it makes sense to think about A/B testing of FREE SHIPPING offer. I need to find how we can set up fair A/B test.
Ecommerce Data from Google Analytics report
Statistical Significance for Ecommerce Conversion Rate
Statistical Significance for Add-to-Cart rate
