While I enjoyed networking at the SES conference in San Jose tremendously, I can’t critic the conference from an attendee perspective since I only attended a few sessions. I did notice that there were fewer than anticipated registrants, but popular sessions were packed. My session was titled (and yes, it was a popular session): Real World Multivariate Testing – and I was on a panel with Trevor Claiborne of Google, Jim McDonald of Match.com, and Dan Sprinkle of Acronym.

Multivariate testing is recommended during conversion optimization, and actually, each recommendation we give our clients is paired in some way with a few testing scenarios to maximize results. My focus during the session though, because most of the panelists were focusing on the benefits, was to shed light onto some of the challenges many listeners may not be aware of.

Factors

It’s important to set the stage: There are a number of factors that impact a multivariate test:

Internal Factors

• Technical Considerations
• Number of Page Views
• Current Conversion Rate
• Desired Improvement
• Number of Scenarios

And I’ll get into most of those later! But there are some factors that are forgotten – nobody assumes that these factors have a direct impact on the test:

External Factors

· Economy: Oh the economy! During this recession, everyday is full of surprises for the online retail world. Running a multivariate test in January and February of 09 when things were at an all time low, clearly results will be skewed. Keep that in mind and know what’s going on in the world before deploying a multivariate test.

· Competition: If your competitor is running a special simultaneous to your multivariate test, your results will be impacted.

With that in mind, what are some other challenges that may impact the creation and deployment of a multivariate test?

The 5-minute myth:

Our focus in on ecommerce, because that is our specialty. Now anybody who tells you creating and deploying a multivariate test takes 5 minutes has clearly never deployed a multivariate test on a dynamic website before.

Some concerns:

1. Complex scenarios: Ecommerce multivariate testing becomes even more time consuming when you’re focused on price optimization where the values are driven from a database, or site navigation testing which is built upon complex application logic – suddenly it’s more than 5 hours, 5 days, and sometimes even 5 weeks.

Something to bear in mind is that when you are undertaking such a complex project, you need to realize that the project is two-fold – a conversion optimization team working designing and creating optimized scenarios, and programmers that have to implement the test. Don’t go move to complex optimization until you’ve first tested some of the more low hanging fruits because of the time consumption, the complexities and the resource allocation.

1. Platform: Do you have a custom ecommerce solution, or are you on a hosted solution? These are crucial details that you need to sort out before multivariate testing.

• Hosted Solutions: If you’re on a hosted solution you will be faced with all sorts of limitations, you need to see if there are specific api’s and plug-ins to assist in integrating the platform with whatever testing solution you’ve selected. Whether it’s Yahoo Stores, Network Solutions, or any of the other 100 hosted platforms, you’re going to run into issues when integrating a multivariate test. Be aware of these issues before proceeding to minimize test deployment time.
• Custom Solutions: Custom solutions are great because they offer you more flexibility. But what you may face is some conflict between the testing solution and some hard-coded areas throughout the site. Again, delve into understanding your platform and how it will work and not work with the testing solution before proceeding with multivariate testing.

Internal Factors

Traffic to the Page

The reality is that the number one factor that will impact the speed of the test is the conversion rate on the page/site being tested. A low conversion rate and low traffic is a recipe for disaster. Your test will go on for months with no conclusion in sight. What to do?

1. The more visitors to the page, the faster we can conclude the test (and I’ll explain why faster tests are better).
2. When are you running the test? If you don’t have a traffic problem, you can conclude a test within hours. But does it make sense to run a test from 12 am – 4 am and look at the statistically significant results (and attempt to make assumptions and conclusions)? Not really. By limiting the time frame you could possibly exclude many visitors. So although you might have enough statistical data, I would recommend running the test for a few days to get more exact results and test all visitors vs. a single segment.
3. Traffic Segmentation: As a prelude to any multivariate test, understanding market segments and their reaction to the page can pave for stronger testing scenarios. Additionally, understanding results after the test has concluded, from each segment’s perspective can be meaningful.

The Conversion Page

Whenever we discuss the conversion page, we like to look at it from a micro vs. macro conversion perspective. Looking at the checkout process, if I have optimized my cart page for testing, my first inclination may be to set the conversion page as the confirmation page: MACRO CONVERSION. While this is not an incorrect strategy, I may be spending more time on the test since traffic drops from step to step in the checkout process, and I haven’t identified fully if the cart page has concluded the better results for me. What if my shipping/billing page is the culprit? Suddenly my results have been skewed and I can’t tell where the problem is in the checkout process (is it still the cart page or something else)?

Micro Conversion: But if I select the next step in the conversion process, in this case, the shipping billing page as my conversion page, I’m identifying the flow to that particular next step. We like to work our way up to the macro conversion or test simultaneously the same cart page at a micro or macro conversion. There is no right or wrong way, it depends on each test, the goals, and the strategy.

Test Creation

Before I start creating a multivariate test I need to do my homework: as much time as I spend analyzing results of a test I need to spend preparing for a test. One of the most efficient ways to create a successful multivariate test is through understanding the market. And what better way than doing that through persona creation.

Persona Development:

When I look at each page of my site from my personas lens, I have a better idea of what their concerns are, what I should change to better cater to them, etc. Once I’ve created a test, I can once more take my personas and look at each scenario to assess how it would impact the competitive, impulsive, logical, and caring users. Based on that information I may decide to remove scenarios where I felt the copy does not address the needs of each market segment.

Scenario Reduction is part science, part art. I can reduce some scenarios based on persona information, but other scenarios may be reduced based on general aesthetic and appeal from a usability, design and conversion perspective. This type of scenario reduction takes practice and time.

Scenario Reduction Example

With this specific client we were testing the continue shopping, proceed to checkout, and the shipping options.

We ended up with

Seems like a pretty normal outcome. But when we plugged these results into the Google Website Optimizer Calculator:

The test is estimated to take 212.736 days, which is way too long. Remember at the beginning of this blog how I discussed external factors, well in 212 days let me assure you that those external factors will kick in. We do not like to run tests for more than 4 weeks.

So based on persona likes, dislikes, the goals of our test, and some general design no-no’s; we were able to reduce the scenarios significantly:

That’s right around the time that we like to run a test.

I went on to give actual examples of successful tests and the results we concluded towards the end of the my talk. Most of the audience members enjoyed the examples because it gave them real-world samples of how to run a test and what you results you can expect to see. What I stressed throughout my presentation is the importance of the analysis that needs to happen after a test is concluded. Questions to ponder: Can I say that an element should be removed all together because it reduced cart abandonment? Or do I need to consider alternate design, copy, and location of this element?

Multivariate testing is a staple element in all of our optimization projects. If done correctly, MVT can increase conversion rates to 200 – 300%, but if done incorrectly can be a waste of time, money, and effort (even if you’re using free tools). Remember, analysis BEFORE and AFTER to maximize the ROI of testing. What are some challenges you’ve faced with multivariate testing?