I talked recently to a customer who ran a test that produced excellent increases in conversion rates. The customer complained that they did not see an increase in revenue as a result of deploying the wining combination. This was very unusual. And while it might look at first glance that the test did not produce results, a careful examination reveals a good explanation.
Let’s drill deeper to understand what is going on.
The test resulted in a 16.1% uplift in conversion with a 99.7% chance to beat original according to Google website optimizer. The test concluded on the first of March (about 45 days ago). So, how come there was no revenue impact?
1- This test was ran on the main homepage of the site. While the site gets most of its traffic through its main homepage, not all traffic to the site goes through that homepage. The uplift is specific to the traffic that goes through the main homepage. That means that the overall site conversion rate and revenue will not see the full 16.1% uplift in conversions. When taking that into account and based on a mathematical formula, we expect the overall revenue to see about an 8.05% impact as a result of this test.
2- When we examined the data again, we discovered that the customer did not allow the winning combination to run on the site. The customer’s team decided to run another completely new test on the page few days after the conclusion of the test. This new test had several new combinations that ran against the winning combination from our first test. To simplify the matters, let’s assume that the customer ran a new test with 3 challenging designs to the winning design. As a result of the new test and since the traffic is divided equally between the 4 designs, our winning design only gets 25% of the traffic. So, right away, the uplift impact is reduced further. And the site now, can only see an 8.05% * 25% = 2% uplift.
3- The final question is to examine the conversion rate for three challengers running against the main homepage. Are any of these challengers causing a drop in conversion rates? Let’s assume that one of the challengers is causing a 5% drop in conversion rates. You can right away see how that 5% for that one challenger will completely wipe out the 2% potential uplift in conversions.
What is the moral of the story?
Conversion optimization provides the most accurate means to measure the impact of producing a new design to your bottom line. You need however to carefully examine the data and what you are doing on the website before you jump into conclusions.