Conversion optimization is one of the greatest developments in modern day marketing. During no other time were marketers ever able to get instant feedback on a marketing initiative, adjust it accordingly and receive even more accurate feedback.  Multi variant testing has been at the heart of any conversion optimization approach. Yet I am reassured, one client after the next, that that Multi Variant testing is one of the main factors that reduces the quality of optimization work.

One of the pleasures of attending the computer science program at the University of Texas at Austin was having the occasional chance to attend lectures by some of the great modern day minds in computer science. Amongst the intellects I had the pleasure of meeting was professor Edsger Dijkstra.  Dijkstra contributed some of the most influential algorithms in modern computing.  He was also known for his low opinion of the GO TO statement in computer programming which led to his writing of an important paper in 1965, which was culminated in the 1968 article “A Case against the GO TO Statements.”

One of the lessons I learned from professor Dijkstra was that programmers relied heavily on compilers to discover bugs in their code rather than investing time upfront to verify the accuracy of their programs and ensuring the preciseness of their algorithms. This reliance of software on compilers is one of the main reasons we have ended up with lower quality programs. The compiler should play a supporting factor. It is not meant to replace the intellectual activities of humans ensuring the quality of software.

Which brings me to the topic at hand: Multi variant testing plays a major role in conversion optimization, and no one can argue with the benefits of using this method in testing. Our average client who uses any form of testing reports a 30% increase in conversion rates. However, we have found that the area where testing does a disservice to conversion optimization methods is when it becomes the main activity of optimization. Too many clients rely on multi-variant testing software to tell them which element combination on a page converts better. But in order to do so, many elements on a page must be tested with many different combinations. Let’s do some simple math: let say you have a page you want to test 6 different elements on it (headers, benefit list, hero shots, etc). For each of the elements, you will choose 4 different options. That means you will have a total of 4^6 = 4096 possible scenarios that you will have to test. Google Optimizer allows the maximum of 10,000 combinations.

But testing a scenario one time is meaningless. At a minimum level, you will need to run each scenario close to 200 times to make sure the data you are collecting is statistically accurate. That translates into 4096 * 200 = 819,200 tests or visitors.
The truth is that you will probably need to run between to 1 to 1.5 million tests or page views before you start getting some confidence in your testing. The greater the range of visitor clicks you are testing, the more successful your results, that’s the idea, right? But that’s quite a lot of testing to get accurate data in a short period of time.

1.5 million tests are strictly focused on a single page. Matters will only get more complicated by adding variation in campaigns, offers, products, keywords. 1.5 million tests start looking like a small number. Not only is data not manageable at that point but that is traffic you will have to pay for.

Running that huge of a number of test scenarios is not unheard of with many larger size clients. But why deal with this headache  in the first place? Why spend hundreds of thousands of dollars on PPC because you need to test whether the variation works or not, and you may or may not get the accurate results you are looking for. This only happens because an activity designed to take place at the very end of the optimization effort becomes the center piece and main force of the effort. Optimization that relies heavily on testing assumes that websites have endless number of resources to test all possible combinations. That is simply not the case.

The science of marketing is about studying human behavior and how to influence it. When relying heavily on multi variant testing software, I find that marketers are not doing what marketers traditionally do; instead they are taking a back seat to software programs. Yes software is important but the human power and intellect should be what guides that software.
Optimization done correctly starts with a full analysis of the target client and gaining deeper understanding of their business goals. That all should lead to developing site persona as a first real step in conversion optimization effort. These personas guide how pages are designed, copy created, site navigations laid out. After all these methods are employed, only then should multi variant testing software be used.

What do you think?