Shopify Customer: Can't share the name of the website because of NDA

A/B testing hypothesis: Based on Customer Surveys after purchases, we have found that the biggest concern of our customers was lack of Shipping information (Time, International shipping, Terms, Price). So we expected that by adding Shipping information section on Product page we will increase Ecommerce Conversion Rate and Revenue.

Key Results:

• 27.5% Revenue increase
• 27.72% Ecommerce Conversion Rate Increase
• 6.02% Add-to-cart Rate Increase
• 20.9% Conversion Rate Increase from Checkout to Transaction

CRO toolkit used:

• Customer Surveys via Hotjar to collect customer feedback after purchase
• Customer Polls via Hotjar on Product and Cart pages to collect user feedback on key steps of conversion funnel

Technical Details of A/B test:

• Length: from 22.01.2018 to 28.01.2018
• Targeting: All users on all product pages
• A/B testing tool: Visual Website Optimizer integrated with Google Analytics
• Statistical significance of results - 95%
• Sample Size: 9800 visitors per variation
• Number of Conversions: 236 (Control) and 298 (Variation)

Intro
We started to work with this website (can't share exact name because of NDA) in the end of November 2017 and decided to use standard CRO framework that contains all types of analysis (User Behaviour, Heuristic, Google Analytics analysis) before A/B testing.

After initial CRO audit and first A/B tests, we have implemented Customer Polls and Surveys on key steps of Conversion funnel to collect feedback directly from customers.

Below you can find examples of Customer Polls and Surveys that we used:

Analysis of User Feedback

After collecting more than 200 responses, we analyzed it, categorizied into groups and it became clear for us that 30% potential customers have concerns regarding to Shipping and it either stops the from purchase or creates some level of friction before purchase.


A/B testing hypothesis

We expected that by adding Shipping information section on Product page we will increase Ecommerce Conversion Rate and Revenue.

Need to say, that after we came up with this hypothesis, we implemented the first A/B test with dynamic Shipping information (as you see it changes the name of the country and flag based on GEO IP) below the ATC (Add to Cart) button:

A/B testing with Visual Website Optimizer

For A/B testing we have chosen Visual Website Optimizer as it's one of the most powerful tool on the market and it can be easily integrated with Google Analytics for deep post-analysis.

In addition to it, we have used Shopify App that added dynamic Shipping section below ATC-button and it was modified with HTML/CSS in Visual Editor of VWO.

This A/B test was very unsuccessful 4 days after the launch. So we stopped it, but didn't give up this idea because based on User Feedback it was very promising.

So we learnt from this failure and decided to implement the same section, but just to remove the second line of text with "Estimated delivery time: 12-28 days" as it can be too frustrating for potential customers (especially in US who are the main customers of our shop where Amazon and other companies provide even next-day delivery)

New A/B test based on learning from previous failure

As a result, 1 week after the launch, results of the A/B test were statistically significant according to all A/B testing rules that we are using in our CRO process.

Results

Split-test was started on 22.01.2018 and was stopped on 29.01.2018 as it reached Statistical significance (see screenshot below).

As a result of this split-test, we increased:
1. Revenue by 27.5%
2. Ecommerce Conversion Rate by 27.72%
3. Add-to-Cart Rate by 6.02%
4. Conversion Rate from Checkout started to Transaction by 20.9%

Split-testing rules:
- All results were statistically significant with 95%.
- A/B test was active within more than 2 Business Cycles (as 95% of all transactions happens on the first visit)
- Both variations had more than 200 Conversions (244 and 346)

Learnings for further A/B tests:
1. As we saw on the previous A/B test #6 with the same idea, but with long delivery (12-28 days) mentioned in Shipping section, the time of delivery for our target audience is crucial. Because on this A/B test #7 without Time section we got absolutely opposite results (+27% ECR). So we have to aggressively display info about International Shipping, but remove any info about time of delivery

2. Based on results from GA and VWO, I see that ATC rate (Add-to-cart) didn't change significantly (only +6%), but there is a significant uplift in CR between Checkout started to Checkout Finished (from 40.95% to 49.54%), that's why we have so good total uplift. It means that information that we show on the product page impact not only on CR from Product page to Cart page, but also on CR on all other steps of the funnel.

3. Taking into consideration the previous point, I must admit that we SHOULD NOT make any parallel tests on different steps of the funnel as it's can significantly skew results, so we will not be able to understand which change increased or decreased CR: ONE A/B TEST AT A TIME

4. We have to double-down our CRO efforts on customer surveys/polls to get more insights directly from visitors and customers, because this A/B test was based only on this type of data."

Next steps:
1. To pause A/B test and distribute 100% of traffic to Variation while Dev Team implement this feature in Production
2. Create Trello task for Dev Team to hide Delivery Time line from Shipping section.
3. Prepare new A/B test with Shipping section (to make it more prominent on Product page) or to place it also on Cart page to enhance effectivity of this feature