Menu
31.2% Revenue uplift
with Urgency/Scarcity elements (Sales badge, Countdown, Low in Stock Availability) on Product page
with Urgency/Scarcity elements (Sales badge, Countdown, Low in Stock Availability) on Product page
Brief information about A/B test
Shopify Customer:
Can't share the name of the website because of NDA
Hypothesis:
We expect that by adding Scarcity and Urgency elements (Sales Badge, Low in Stock Availability notifications, Countdown timer) on Product page we will increase Revenue and Ecommerce Conversion Rate.
This hypothesis is based on a few previous successful A/B tests where we got Revenue and Conversion Rate uplifts on other websites by implementing these elements on Product and Cart pages.
Main Results:
1. Revenue increase by 32.1%
2. Revenue per visitor increase by 29%
3. Ecommerce Conversion Rate increase by 25.77%
4. Add-to-Cart Rate increase by 7.09%
5. CR from Cart page to Checkout increase by 6.5%
5. CR from Checkout to Transaction increase by 15%.
CRO Toolkit used:
1. Heuristic analysis
2. Behavioural psychology
3. A/B testing
Can't share the name of the website because of NDA
Hypothesis:
We expect that by adding Scarcity and Urgency elements (Sales Badge, Low in Stock Availability notifications, Countdown timer) on Product page we will increase Revenue and Ecommerce Conversion Rate.
This hypothesis is based on a few previous successful A/B tests where we got Revenue and Conversion Rate uplifts on other websites by implementing these elements on Product and Cart pages.
Main Results:
1. Revenue increase by 32.1%
2. Revenue per visitor increase by 29%
3. Ecommerce Conversion Rate increase by 25.77%
4. Add-to-Cart Rate increase by 7.09%
5. CR from Cart page to Checkout increase by 6.5%
5. CR from Checkout to Transaction increase by 15%.
CRO Toolkit used:
1. Heuristic analysis
2. Behavioural psychology
3. A/B testing
Analysis and Hypothesis
It was the 18th A/B test on this client's Shopify Ecommerce Stores (can't share the name of the website because of signed NDA).
At first we have worked only on one ecommerce store, but as the client started to grow traffic on other niches stores, we have started to apply CRO processes also on them.
We have already tested a lot of UX features, improved overall design of product pages, but they didn't include any Scarcity and Urgency triggers that could stimulate visitors make purchase. And as we have already A/B tested such elements on other store and got 24.1% Revenue uplift, we have decided to implement the same hypothesis here.
Why we A/B tested it again as it worked on another store with the same business model?
And here it is important to recall one important rule from A/B testing: If it works on one website, it's doesn't mean that it will work on another website.
As these stores have different niches (the first one is focused on winning products regardless of niche, while another one is focused only on baby care products and have specific target audience), we can't just blindly implement every successful feature from A/B tests on another websites.
When we implement of new features/elements through A/B testing, we certainly know if it really works and we decrease our risks.
There was also one difference comparing with similar A/B test on another website: we have created additional testing variation with Countdown timer to enhance scarcity and urgency effect.
As a result, in Variation 1 we added 2 elements:
1. "Today's Sale" badge on the top part of hero image above-the-fold.
2. "Availability: Low in Stock today" text below the thumbnail slider (also almost above-the-fold)
And in Variation 2 we additionally placed Countdown timer ("Today's Sale ends in: 4 hours 29 minutes and 50 seconds") below the text "Availability: Low in Stock Today". So the logic behind of it was the following: At first potential customers see that there is a Sale today. Then, when they scroll-down the page, they see Low In Stock notification which is accompanied by Countdown timer that starts from 4 hours and 29 minutes and amplify urgency effect.
Need to say that our client and business owner was sceptical about this and didn't want to implement Countdown timer on this website, but we persuaded him that it should be A/B test as we have already achieved a few big wins with scarcity and urgency elements.
At first we have worked only on one ecommerce store, but as the client started to grow traffic on other niches stores, we have started to apply CRO processes also on them.
We have already tested a lot of UX features, improved overall design of product pages, but they didn't include any Scarcity and Urgency triggers that could stimulate visitors make purchase. And as we have already A/B tested such elements on other store and got 24.1% Revenue uplift, we have decided to implement the same hypothesis here.
Why we A/B tested it again as it worked on another store with the same business model?
And here it is important to recall one important rule from A/B testing: If it works on one website, it's doesn't mean that it will work on another website.
As these stores have different niches (the first one is focused on winning products regardless of niche, while another one is focused only on baby care products and have specific target audience), we can't just blindly implement every successful feature from A/B tests on another websites.
When we implement of new features/elements through A/B testing, we certainly know if it really works and we decrease our risks.
There was also one difference comparing with similar A/B test on another website: we have created additional testing variation with Countdown timer to enhance scarcity and urgency effect.
As a result, in Variation 1 we added 2 elements:
1. "Today's Sale" badge on the top part of hero image above-the-fold.
2. "Availability: Low in Stock today" text below the thumbnail slider (also almost above-the-fold)
And in Variation 2 we additionally placed Countdown timer ("Today's Sale ends in: 4 hours 29 minutes and 50 seconds") below the text "Availability: Low in Stock Today". So the logic behind of it was the following: At first potential customers see that there is a Sale today. Then, when they scroll-down the page, they see Low In Stock notification which is accompanied by Countdown timer that starts from 4 hours and 29 minutes and amplify urgency effect.
Need to say that our client and business owner was sceptical about this and didn't want to implement Countdown timer on this website, but we persuaded him that it should be A/B test as we have already achieved a few big wins with scarcity and urgency elements.
We placed these elements on the top part of product pages (almost above-the-fold) consciously to be confident that the vast majority of visitors will see it. And we expected that as we are driving traffic from Facebook ads directly to product pages, these urgency and scarcity triggers will be relevant and will amplify each other in terms of psychological influence and we guessed that it will increase Revenue, Ecommerce Conversion Rate and Add-to-Cart Rate.
A/B testing
For A/B and Split testing we have used Visual Website Optimizer as it's one of the most powerful tool on the market and it can be easily integrated with Google Analytics for deep post-analysis.
In addition to it we are Certified Partners of VWO as we are actively using their platform and recommend it to all of our clients (btw, all our CRO customers have pricing discounts on VWO plans due to it)
We have used HTML/CSS and jQuery to build testing variation for this A/B test. We have done it without any development support, as we have a lot of experience in building testing variations using basic coding skills.
Below you can see the difference between Control and Testing version of the product page in this A/B test:
Below you can see the difference between Control and Testing version of the product page in this A/B test:
When testing variation was prepared, it was thoroughly tested by our team using query parameters, so we were able to target A/B test only on ourselves and to check if everything was displayed and worked as needed.
Then A/B test was integrated with Google Analytics for deep post-test analysis and was started for all Mobile visitors that come to product pages.
Then A/B test was integrated with Google Analytics for deep post-test analysis and was started for all Mobile visitors that come to product pages.
A/B testing rules that we used:
- A/B test should run at least 2 Business Cycles (for us it's usually just 1 session as we have specific business model)
- A/B test should run at least 1 full week (from Monday to Sunday)
- A/B test should have at least 100 conversions per testing version
- Minimal statistical significance to declare fair winner should be at least 95%
Results
This test was live from March 6, 2018 to March 15, 2018 (9 full days) and we got the following results:
1. Revenue increased by 31.2% with 95% Statistical significance
2. Revenue per Visitor increased by 29% with 95% Statistical significance
3. Ecommerce Conversion Rate increased by 25.77% with 95% Statistical significance
4. Add-to-Cart rate increased by 7.09% with 90% Statistical significance
5. Conversion Rate from Cart page to Checkout increased by 6.5%
6. Conversion Rate from Checkout to Transaction increased by 15%
Technical details of the A/B test:
1. Revenue increased by 31.2% with 95% Statistical significance
2. Revenue per Visitor increased by 29% with 95% Statistical significance
3. Ecommerce Conversion Rate increased by 25.77% with 95% Statistical significance
4. Add-to-Cart rate increased by 7.09% with 90% Statistical significance
5. Conversion Rate from Cart page to Checkout increased by 6.5%
6. Conversion Rate from Checkout to Transaction increased by 15%
Technical details of the A/B test:
- A/B testing tool: Visual Website Optimizer integrated with Google Analytics
- Length: 9 days
- Targeting: Only Mobile users, Only on Product pages
- Sample size: ~8000 visitors per variation
- Transactions: 140, 151, 182 (Control, Variation 1, Variation 2
Ecommerce Conversion Rate from VWO reports
Ecommerce Data from Google Analytics report
Statistical Significance for Ecommerce Conversion Rate increase
Statistical Significance for Add-to-Cart Rate increase
Learnings and Next steps
Learnings:
Next steps:
- Results of this A/B tests again proved the hypothesis that by adding Scarcity and Urgency features on Product page we can increase Revenue and CR. Even if we have different niches, it worked well on both websites and increased CR and Revenue by more than 20%.
- Again we observe the situation when changes on Product pages impact on user behaviour on the next steps of the funnel (Cart page, Checkout process), so we have to measure final macroconversions as well as microconversions to see the whole picture.
- As we have already finished 3 successful A/B tests with Urgency and Scarcity elements, we have to double-down on it to try to get maximum uplift.
Next steps:
- Redistribute 100% of traffic to Variation 2
- Create task for Dev Team in Trello to implement all features and CSS changes from this A/B test on Live website.
- Implement A/B test with Scarcity and Urgency features on Cart page (as we have already done on other website and got positive uplift) to enhance the effect.
- Implement countdown timer for another website as we see that on this A/B test it worked much better than just "Today's sale" badge and text about Low in Stock availability.
- To think how we can make these features even more prominent (by placing higher on the page, by improving CSS styles to make it more attractive, etc)
