Conversion optimization is one of the greatest developments in modern day marketing. During no other time were marketers ever able to get instant feedback on a marketing initiative, adjust it accordingly and receive even more accurate feedback. Multivariate testing has been at the heart of any conversion optimization approach. Yet I am reassured, one client after the next, that that Multi Variant testing is one of the main factors that reduces the quality of optimization work.

One of the pleasures of attending the computer science program at the University of Texas at Austin was having the occasional chance to attend lectures by some of the great modern day minds in computer science. Amongst the intellects I had the pleasure of meeting was professor Edsger Dijkstra. Dijkstra contributed some of the most influential algorithms in modern computing. He was also known for his low opinion of the GO TO statement in computer programming which led to his writing of an important paper in 1965, which was culminated in the 1968 article “A Case against the GO TO Statements.”

One of the lessons I learned from professor Dijkstra was that programmers relied heavily on compilers to discover bugs in their code rather than investing time upfront to verify the accuracy of their programs and ensuring the preciseness of their algorithms. This reliance of software on compilers is one of the main reasons we have ended up with lower quality programs. The compiler should play a supporting factor. It is not meant to replace the intellectual activities of humans ensuring the quality of software.

This brings me to the topic at hand: multivariate testing plays a major role in conversion optimization, and no one can argue the benefits of using this method in testing. Our averag client who uses any form of testing reports a 65% increase in conversion rates. When multivariate testing becomes the main activity of optimization, though, it does a disservice to conversion optimization efforts. Too many clients rely on multivariate testing software to tell them which combination of elements on a page converts better. To do so, many elements on a page must be tested with different combinations.

Let’s do some simple math. Say you want to test six different elements on a page (headers, benefit list, hero shots, etc.). For each element, you will choose four different options. This means you will have a total of 4^6 = 4,096 possible scenarios that you will have to test. Google’s Website Optimizer allows a maximum of 10,000 combinations. As a general rule of thumb, you will need around 100 conversions per scenario to make sure the data you are collecting is statistically significant. This translates into 4,096 * 100= 409,600 conversions. If your website converts around 1%, you will need 409,600 * 100= 40,960,000 visitors before you start gaining some confidence in your testing results. If testing 4,096 variations sounds difficult, imagine how complicated matters will get by adding variation in campaigns, offers, products, and keywords.Running this many test scenarios is not unheard of for many larger websites.

But why deal with this headache in the first place? Why spend hundreds of thousands of dollars on payper- click (PPC) because you need to test whether the variation works? And there is no guarantee you’ll get the accurate results you are looking for. This only happens because an activity designed to take place at the very end of the optimization effort becomes the centerpiece of that effort. Optimization that relies heavily on testing assumes that websites have an endless number of resources to test all possible combinations. This is simply not the case.

The science of marketing is about studying human behavior and how to influence it. When relying heavily on multi variant testing software, I find that marketers are not doing what marketers traditionally do; instead they are taking a back seat to software programs. Yes software is important but the human power and intellect should be what guides that software.

Optimization done correctly starts with a full analysis of the target client and gaining deeper understanding of their business goals. That all should lead to developing site persona as a first real step in conversion optimization effort. These personas guide how pages are designed, copy created, site navigations laid out. After all these methods are employed, only then should multi variant testing software be used.

What do you think?