What are the Key Differences between Multivariate Testing (MVT) and A/B Testing?
- Multivariate testing (MVT) techniques apply statistical hypothesis testing on multi-variable systems. Different from A/B testing which changes one variation at one time, MVT is a form of testing in which changes are made to multiple sections of a webpage on a site, and then variations are created for all the possible combinations of those changes.
- The following table will illustrate the difference between MVT and A/B testing.
| A/B Testing | Multivariate Testing | |
|---|---|---|
| Explanation | In A/B testing you split traffic amongst two or more completely different versions of a webpage. The variations of your original page can differ in any manner. You can either just change the headline; or you can even change the entire design, layout, offer and what not in the variations. | In a multivariate test, you identify a few key areas/sections for those sections specifically. For example, in a MVT you can choose to create different varations for 2 different sections: headline and image. A multivariate testing software will combine all these section specific variations to generate unique versions of the page to be tested and then simply split traffic amongst those versions. |
| Illustration | ||
| Traffic Required | Relatively Less | Huge |
| Best used for | Testing radically different ideas for conversion rate optimization | Optimizing and refining an existing landing page or homepage without doing significant investment in redesign |
More about MVT
-
There are two principal approaches used to achieve multivariate testing on websites:
- Page Tagging: A process where the website creator inserts JavaScript into the site to inject content variants and monitor visitor response. (technical team is involved)
- No Page Tagging: By establishing a DNS-proxy or hosting within a website's own datacenter, it is possible to intercept and process all web traffic to and from the site undergoing testing, insert variants and monitor visitor response. (no further technical involvement)
-
Statistical testing relies on design of experiments. Several methods in use for multivariate testing include:
- Full factorial the most straightforward method whereby all possible combinations of content variants are served with equal probability.
- Discrete choice and what has mutated to become choice modeling is the complex technique that achieves a great success in business. Choice modeling models how people make tradeoffs in the context of a purchase decision. By systematically varying the attributes or content elements, one can quantify their impact on outcome, such as a purchase decision.
- Optimal design involves iterations and waves of testings. Optimal design allows marketers the ability not only to test the maximum number of creative permutations in the shortest period of time but also to take into account relationships, interactions, and constraints across content elements on a website. This allows one to find the optimal solution unencumbered by limitations.
- Taguchi methods：with multiple variations of content in multiple locations on a website, a large number of combinations need to be tested, and medium/low traffic websites can take a long time to get a large enough sample to find statistically significant differnece in performance if differences really exist. Taguchi methods (namely Taguchi orthogonal arrays) can used in the design of experiments in order to reduce the variations but still give statistically valid results on individual content elements. Taguchi uses fractional factorial designs.