Menu
18.57% Add-to-Cart Rate Increase
by adding sticky button
on the bottom of Mobile screen
by adding sticky button
on the bottom of Mobile screen
Shopify Customer: Under NDA
Shopify Customer: Can't share the name of the website because of NDA
Short Description of A/B test: It was the 13th A/B test for this client. As we are focusing on improving website mostly for Mobile visitors (it's 80% of website audience), we decided to add Sticky Add-to-Cart button that will be visible all the time on the bottom of the page on Mobile devices.
In spite of the fact that this function seems logical for mobile UX and everything is doing that, we decided to A/B test it to be sure that it improve performance of the website.
In addition to it, we added possibility to choose product variation (Color, Size) to this button to simplify the process of adding product to cart.
Key Results:
CRO toolkit used:
Technical Details of A/B test:
Short Description of A/B test: It was the 13th A/B test for this client. As we are focusing on improving website mostly for Mobile visitors (it's 80% of website audience), we decided to add Sticky Add-to-Cart button that will be visible all the time on the bottom of the page on Mobile devices.
In spite of the fact that this function seems logical for mobile UX and everything is doing that, we decided to A/B test it to be sure that it improve performance of the website.
In addition to it, we added possibility to choose product variation (Color, Size) to this button to simplify the process of adding product to cart.
Key Results:
- Add-to-Cart Rate increased by 18.57% with 90% Statistical significance
- Ecommerce Conversion Rate increased by 5% with 76% Statistical significance
CRO toolkit used:
- Heuristic Analysis
- A/B testing with Visual Website Optimizer
- Google Analytics
Technical Details of A/B test:
- Length: from 13.02.2017 to 21.02.2017
- Targeting: Mobile users that visited any product page
- A/B testing tool: Visual Website Optimizer integrated with Google Analytics
- Statistical significance of results - 90%
- Sample Size: 21533 in Control and 21803 in Variation
- Number of Conversions: 554 in Control and 484 in Variation
Intro:
It's common guideline for Mobile UX to have sticky CTA-button on product pages to give provide users with always visible CTA-button, so they can add a product to cart even if they scrolled down the page (which happens so often if you put any important content like product descriptions, reviews, additional information below the fold)
But as we using data-driven approach in our work, before implementation of this feature on the website, we had to A/B test it.
A/B testing hypothesis:
WE expect that by adding sticky CTA-button (that will be always visible) on the bottom of the Mobile product pages, we will increase Add-to-Cart rate.
Preparation of testing variation:
To implement this A/B test faster, we decided to use Shopify App that adds sticky CTA-button to Mobile product pages (with ability to show field with product variations). This app was very cheap ($5 per month), so it was much easier for us to buy already developed solution instead of custom development.
It's common guideline for Mobile UX to have sticky CTA-button on product pages to give provide users with always visible CTA-button, so they can add a product to cart even if they scrolled down the page (which happens so often if you put any important content like product descriptions, reviews, additional information below the fold)
But as we using data-driven approach in our work, before implementation of this feature on the website, we had to A/B test it.
A/B testing hypothesis:
WE expect that by adding sticky CTA-button (that will be always visible) on the bottom of the Mobile product pages, we will increase Add-to-Cart rate.
Preparation of testing variation:
To implement this A/B test faster, we decided to use Shopify App that adds sticky CTA-button to Mobile product pages (with ability to show field with product variations). This app was very cheap ($5 per month), so it was much easier for us to buy already developed solution instead of custom development.
Control and Variation of A/B test #13 from VWO preview
Split-testing using Visual Website Optimizer
For A/B and Split testing we have chosen Visual Website Optimizer as it's one of the most powerful tool on the market and it can be easily integrated with Google Analytics for deep post-analysis.
As you see, Control version has sticky button, while in Variation it was hidden. We did that because it was the simplest way of implementing this A/B test. At first we installed sticky Shopify app that added Sticky CTA-button on all product pages and it was visible for all Mobile visitors. And after that the button was hidden in Variation through CSS styles in visual editor of A/B testing tool.
So in fact Control version of the page is our testing variation.
For A/B and Split testing we have chosen Visual Website Optimizer as it's one of the most powerful tool on the market and it can be easily integrated with Google Analytics for deep post-analysis.
As you see, Control version has sticky button, while in Variation it was hidden. We did that because it was the simplest way of implementing this A/B test. At first we installed sticky Shopify app that added Sticky CTA-button on all product pages and it was visible for all Mobile visitors. And after that the button was hidden in Variation through CSS styles in visual editor of A/B testing tool.
So in fact Control version of the page is our testing variation.
A/B-test was started on 13.02.2017 and was finished on 21.02.2017.
Results:
1. Add-to-Cart Rate increased by 18.57% with 90% Statistical significance
Split-testing rules:
- A/B test was active within more than 2 Business Cycles (as 95% of all transactions happens on the first visit)
- Both variations had more than 400 Conversions
- A/B test was live within all days of the week (full-week cycle)
Learnings for further A/B tests:
Next steps:
Results:
1. Add-to-Cart Rate increased by 18.57% with 90% Statistical significance
Split-testing rules:
- A/B test was active within more than 2 Business Cycles (as 95% of all transactions happens on the first visit)
- Both variations had more than 400 Conversions
- A/B test was live within all days of the week (full-week cycle)
Learnings for further A/B tests:
- We can increase CTR of ATC button for Mobile users by always showing it on the bottom of the screen, so we have to implement it on all our Ecommerce websites.
- From the other side we see that uplift of CR on this step (from Product page to Cart page) doesn't mean that it will change overall CR and Revenue in the same way. It means that we are just simplifying the process of adding product to cart, so more visitors who are less motivated now can click on it and then we have some drop-off on the next step of the funnel (from Cart page to Checkout page). It's another one prove the statement that we have to measure total Conversion Rate in all A/B tests to be sure that uplift on any step of conversion funnel doesn't have higher drop-offs on another steps.
- We have to start optimizing Cart page to convince even less motivated users to proceed to Checkout process.
Next steps:
- Switch off this A/B test, so 100% of traffic will see Control version of the page that has sticky ATC-button
- Create task for Dev team to implement sticky ATC button on the Ecommerce Stores.
Analytics data from Visual Website Optimizer reports for Add-to-Cart Rate
Statistical Significance for Add-to-Cart rate
