A/B Testing – An Essential Part of OptimizationDeborah Miller
Here at KTW we believe we should always be testing. There’s always room for improvement and there’s no way to know what may help if we don’t test our ideas. Also, market conditions change over time. What may have worked best a year ago may no longer be the best option. Political, financial and local issues can cause markets to change quickly. New technology sometimes provides new options in both layout/design and function that may give you better results then current options.
A/B testing involves creating 2 or more versions of a web page, email or ad. Each of these versions has a difference involving 1 and only 1 variable. For instance, a web page has a button that people click to subscribe to a newsletter. Currently that button is blue. You have read that red buttons command more attention so you form a hypothesis that having a red button will increase the number of people who click it.
You create a page identical to your current page using a red button instead of blue. EVERYTHING ELSE is exactly the same between the two pages. If you change more than one variable you won’t know which change causes any difference to click-through rates (CTR). You then show each of your two versions randomly to equal numbers of people over a period of time. When enough time has passed to accumulate a statistically significant number of page views, you can end the test and check your click-through rates for each version of the page. The page with the highest click-through rate is the winner and you can make it your permanent page.
You can also test for other results, such as conversion rate. It’s possible to get more clicks but fewer conversions, so be sure to check.
The term A/B testing is also frequently used when testing more than two variants of a single variable. You could test 3 or 4 different button colors at the same time. Be careful to take into consideration that this testing requires a large enough number of pageviews, for all pages involved, in order to reach statistical significance. If your website doesn’t receive much traffic, it will take a long time to complete the test. Every version you add will increase this time.
To avoid sampling errors and help insure the results you see are accurate and repeatable, watch the time of day the page is viewed. Make sure the 2 versions are not only randomly displayed but randomly displayed at random times of day and locations across your marketing area. This will help insure you reach a good cross section of your potential market. The same is true for audience differences such as sex and age. It could be that CTR for men increased 60% but CTR for women went down 20%. You may want to permanently segment the page based on audience demographics.
There may also be a difference between first time and returning visitors. For instance, say that version A of your test gets a 10% CTR for first time visitors but returning visitors show a 60% CTR. Look at both groups of visitors to avoid making the wrong overall choice.
A/B tests can be applied to page design, copy, buttons and forms. It can be equally effective for testing email subject line, copy and design elements. A/B tests can be run on both search and display ads. An added benefit to A/B tests (and multivariant tests) is that if you track demographics and behavior (new vs. returning) in addition to CTR, sometimes you’ll find surprising and actionable information about your target personas.