Four Things You Need to Know to Run a Successful A/B Test

Let’s take a break from SEO and talk a bit about web design optimization. Designing a website is treated by many as a wholly subjective experience. You chat with your artsy design staff about “best practices” and “emerging trends”, grind your teeth over different shades of blue, and argue futilely over red vs. green action buttons. I’m here to lift the burden from your shoulders and place it in the more capable hands of A/B testing software.

The Skinny

An A/B test pits two design options against each other. The traffic to the web page is split (usually 50/50) and measured against a certain activity. The page that produces the best results is deemed the winner and is shown publicly to 100% of the traffic. What you’ll need:

1). Two uniquely designed pages that serve the same purpose (e.g., a list view category page vs. a grid view category page)

2). Testing software (e.g., the free Google Website Optimizer tool)

3). A competent proctor (e.g., me)

Do it Right

Here are the key points to consider when starting an A/B test:

1). Know why you’re testing. Running a test is useless without having a defined reason for running it in the first place. Are you trying to increase click-through rates of a landing page? Is your new design intended to reduce bounce rate or maybe even improve the time on page? Whatever the end goal may be, make sure you’re tracking metrics are in place and you’re able to identify the end results.

  • Determine why you’re running the test
  • Select the metrics that indicate success
  • Measure the results following the test

2). Be sure the test page gets significant levels of traffic. Your test needs to be trafficked otherwise you won’t get the results you need in a timely fashion. Try testing pages associated with a paid search campaign so you can control that traffic to the page. Or, check out your analytics and see what pages are most popular on your site to find potential targets. This isn’t to say that lower traffic volume pages can’t be tested, it just takes longer and may not provide equally actionable results.

  • Pick a popular page on your site that gets a high volume of visits
  • Use analytics to find popular pages with metrics that fit the credentials of your test
  • Start considering variations

3). Make a radical change. Running an A/B test isn’t going to help you if your change is subtle. It needs to be something clearly noticeable that inspires visitors differently than the original design. Use completely different copy, or reverse the layout. Change the info-graphics used, or showcase different products. The more significant the change is, the more valuable the results.

Target category page variations


  • Don’t make subtle changes
  • Radically change the layout and style of the test page
  • This leads to more compelling, actionable results

4). Lather, rinse, repeat. One test isn’t enough. Now that you’ve got a feel for it, start testing other pages. Figure out how you can improve your checkout process or product details pages. There is no end to the improvements you can make, so keep testing new ideas and push the envelope to make your website more successful.

Wrapping Up

A/B testing takes the guess work out of design and can help design teams improve their process by solidifying concepts that work and ones that don’t. Use A/B testing to continually improve landing page performance for paid search campaigns, social media campaigns, or just general website performance. Have you run an A/B test lately? Give you opinions and feedback below.