Shopify Customer: Can't share the name of the website because of NDA

Short Description of A/B test: After User Behaviour and Heuristic analyses we have found a list of obvious UX problems that created friction for potential customers. Based on these observations we improved UX of the page

Key Results:

• 17% Revenue Increase
• 14.43% Ecommerce Conversion Rate Increase
• 9.9% Add-to-cart Rate Increase

CRO toolkit used:

• User behaviour patterns
• Heuristic Analysis;
• A/B testing with Visual Website Optimizer

Technical Details of A/B test:

• Length: from 04.12.2017 to 11.12.2017
• Targeting: only Mobile users
• A/B testing tool: Visual Website Optimizer integrated with Google Analytics
• Statistical significance of results - 95%
• Sample Size: 10200 Visitors per variation
• Number of Conversions: 417 (Control) and 471 (Variation

Intro

We started to work with this website (can't share exact name because of NDA) in the end of November 2017 and decided to use standard CRO framework that contains all types of analysis (User Behaviour, Heuristic, Google Analytics analysis) before A/B testing.


Figuring out areas that we should focus on:

Based on Google Analytics data, we have found that our client had:

• 80% of Mobile visitors
• 95% of them landed on one landing page (it was the most profitable and popular product that got 95% of all traffic)
• The weakest step of conversion funnel is between Product page and Cart page

Based on these data, we focused on analyzing most popular product page and Mobile UX on it.


1 Week later

A week after start of work, a partial CRO audit was finished and all ideas, recommendations and observations were summarized. Then all of them were divided into 2 groups:

• Obvious things that should be fixed / added / improved
• Hypothesis that should be A/B tested

The first group of recommendations was transformed into clear tasks for Developers. The second group was added to Google Spreadsheet for prioritization based on 2 factors (Potential Business Impact and Ease of Implementation):

After prioritization it was clear that we have to start A/B testing from improving Mobile UX of Product page as it has the highest potential in terms of Revenue and Conversion Rate.

We decided to combine a few recommendations in one test and proposed slightly radical redesign of product page to improve Mobile UX. We used Moqups.com to build a mockup and discuss all details with CEO and Development team.

List of Mobile UX optimizations that we applied to testing variation:
1. Decreased white-space above-the-fold
2. Remove Call Us button from Header
3. Placed Product Title and Star Ratings above main image
4. Added Left/Right arrows near main image
5. Added Color swatches instead of drop-down list for color selection
6. Placed Color swatches below the main image to improve process of color selection
7. Added gray lines to visually divide different sections of product page
8. Improved Price section
9. Changed product description to short bullet-lists
10. Added always visible sticky CTA-button on the bottom of the page with color selection field.

Preparation of testing variation

When new design of Mobile product page was approved and all technical details were discussed, Development team have changed everything according to mockup.

As for technical side, we copied original product page and applied new product page template to it. As a result, we got 2 similar product pages (Control and Variation) with different URLs that later will be used in Split-testing:

Split-testing using Visual Website Optimizer

For A/B and Split testing we have chosen Visual Website Optimizer as it's one of the most powerful tool on the market and it can be easily integrated with Google Analytics for deep post-analysis.

Results

Split-test was started on 04.12.2017 and was stopped on 11.12.2017 as it reached Statistical significance (see screenshot below).

As a result of this split-test, we increased:
1. Revenue by 17%
2. Ecommerce Conversion Rate by 14.43%
3. Add-to-Cart Rate by 9.99%

Split-testing rules:
- All results were statistically significant with 95%.
- A/B test was active within more than 2 Business Cycles (as 95% of all transactions happens on the first visit)
- Both variations had more than 400 Conversions.

Learnings for further A/B tests:
1. By improving UX of Product page we can influence on Ecommerce Conversion Rate, Revenue and AOV, so we have to continue working on it.
2. Also as we have ~20-25% of Desktop and Tablet traffic, then it makes sense to spend time on better UX for these user segments

Next steps:

1. Distribute 100% of traffic to testing version
   
2. Work on Desktop and Tablet version of product page applying learnings from current A/B test
   
3. Enhance new product page with better Reviews section in A/B test #3