What is A/B Testing and Why Your Online Shop Should Be Doing It

11.01.2018, 7m

Owning an online shop means constant change. Sometimes it’s just adding and removing products from your catalogue. Sometimes it means optimising the layout of your website. When it comes to the optimisation of your website, it might not always be very clear what the right decision is. That’s where A/B testing can really help you make the right call.


A/B testing is pretty much what it sounds like. It’s the process of testing out variations of your business (e.g. your website layout, marketing emails) to two groups of customers and seeing which one helps you do better business. The “winner” will then be implemented throughout your website. In today’s blog, we’ll talk about how you can plan an A/B test, what elements you can test, and other things to keep in mind.

Planning an A/B test

When testing what changes you can make to your business, there are two basic categories. You can test on-site elements of your business (your website) or off-site elements (adverts, marketing emails). The goals of these tests are obviously the same: improving the effectiveness of your business to sell more.

Planning ahead of time is important. You’ll want to decide which elements you’ll be testing beforehand. Though this might sound obvious, you shouldn’t go overboard with testing every little element on your website. You’ll really want to hone in on those elements that lead to clicks and, more importantly, conversions.

It’s also important to think about what kinds of results you’ll be expecting to see. Make sure you know what results you’re currently getting in order to compare them to the results of the changed elements.

Some A/B tests assign “A” to be the current process, where “B” is the new element. Others will make both “A” and “B” new elements, and compare those to what is currently in place.

When testing multiple new elements, be sure that at least one of these outperforms the current element before replacing the original. Otherwise, it would obviously be wise to keep the current elements in place.

AB test graphic.gifThe basics of an A/B test. Two versions of the same website are displayed to equal amounts of people. Whichever leads to better results, "wins".  

What to test

Wanting to optimise your website is great. Some changes can really bring dramatic improvements to your sales and that’s really what it’s all about, isn’t it? Before you make any changes (or test those changes), let’s look at some elements that you might want to consider testing.

  • Headlines: If your company uses a content marketing strategy, headlines are what draw people in and inspire them to click. From an SEO perspective, this is really important as well. See what helps you rank higher in search engines, which will basically help you find out if more people are clicking on your ads, links, and posts.

  • CTA buttons: CTA (Call-to-Action) buttons can be very significant. Usually placed after some relevant content, these buttons might also be used to add a product to the shopping cart. They might be used to collect leads. There are many different uses, but generally speaking, they’re important for conversions in your business.

    What they look like (images with texts vs simple buttons) and where they’re placed (at the top, middle, or bottom of your page) can also equate to big changes in performance.

  • Copy text: Similarly to headlines, the copy of your blogs, webpages, and emails can help you rank higher on Google. Good copy will also turn visitors into shoppers. Changing and tweaking your copy may well be worth A/B testing.

  • Emails: Speaking of emails, you can A/B test these as well. A huge factor for improving opening rates is the subject line. If you’re mailing list is long enough, consider sending out the same email with different subject lines to see which one is more effective (the same goes for newsletters).

    Similarly, you may want to test the content of the email to see if certain language works better than others with your customers. CTA buttons are often found in emails as well.

  • Website: this is, of course, a big one. Testing different elements might show different results. On your homepage, you might want to change the header image and see if your bounce rates improve. You might want to change the layout or the menu buttons.

    If you open a website in a new country, you might want to test some different variations of your website to see how the new market reacts. Obviously, there is a lot you can do here, so choose the elements you think could have the biggest impact.

5 SEO SEA whitepapers package

  • Ads: As with most of the elements mentioned above, small changes can make a big difference sometimes. You might think that a certain image works really well in your Facebook ad, but you might well be surprised.

    We once A/B tested an image of me vs. a shirtless man slumped on the couch, playing with his tablet. Well, needless to say, I was surprised (and very disappointed) that Mr. Topless outperformed me (by a lot!). Besides images, you can also change the text and times that your ads are shown.

  • Targeting: if you run an online shop, there’s a good chance you use Google or Facebook ads. They offer great targeting options to hone in on your target audience. If you’re a relatively new business owner, you might not know who your true audience is.

    Be careful, you might think you know who your audience is, but you might be wrong about that (you’d be surprised how often businesses are wrong about this). Testing out different audiences might surprise you. This info can help you build a buyer persona, which can be helpful for future marketing efforts. 

similar_me_vs_real_me.jpgA/B testing revealed that a guy who kind of looks like me (left) leads to more clicks than the real me (right)!  Not my best week. 

Things to keep in mind

Besides knowing what to test, there are some other things to keep in mind.

Give it time

Knowing how long to run your A/B test is one of the most common questions when it comes to knowing how effective they are. If you’re running a high traffic site, you might be able to get some accurate numbers quite quickly (e.g. one week), but chances are that you’ll need to run these tests for a few weeks (e.g. 2-6 weeks).

There are some tools (more on that below) that can help you figure out how many people you need to run this test on before you can consider your results accurate.

Similarly, you must be careful not to let these tests run too long. Why? Sometimes, other variables can affect the outcome of your results. For example, I wouldn’t recommend running an A/B test during Christmas because shopping habits tend to be quite different during the holiday season (the exception, of course, would be if you specifically want to test some “Christmas elements” on your page).

Other variables that might affect your results over time include:

  • Season (time of year)

  • holidays

  • ads you’re running during the test

  • press mentions of your business (positive or negative)

Beware of sample bias

When you’re running these experiments, you should have some kind of hypothesis as to the outcome. Naturally, you want to be right when it comes to predictions. However, some marketers may unknowingly stop tests early after achieving the outcome they predicted. Again, it’s important to give your test enough time to be sure that it is accurate.

blogImage-puzzle.jpgEnding an A/B test too early is like an incomplete puzzle. You might think you know what the image is, but you can't be sure unless you really complete it. 

Test multiple options

As mentioned above, some A/B tests will involve multiple variations for one element. This is fine, but only if you have enough traffic to make sure that your results are reliable. Testing more than two variations is also known as A/B/C tests or A/B/n tests. Generally speaking, we don’t recommend testing multiple elements on one page at once, though some marketers don’t have a problem with this if done correctly.


Testing A vs B might seem pretty straightforward. You might think that “A” leads to more conversions than “B”, so “A” is clearly the winner. However, it’s important to look at the details of the results. Though “A” performed better overall, it might be the case that “B” significantly outperformed “A” on mobile devices. Keep this in mind when analysing your results. Some other factors to be aware of are:

  • Weekend visits vs weekday visits

  • Source of traffic (people who landed on the page from a Facebook ad vs organic visitors)

  • Returning customers vs new customers

  • Desktop vs mobile

  • Time of day

  • Men vs women

  • Geographical locations

  • Age range

Tools for A/B testing

There are plenty of tools to help you with your tests. Here are some of the most popular tools:

A/B testing tools

A/B Testing Calculators


A/B testing can really help optimise your website and marketing efforts. Just remember to keep tests running long enough to know that your results are accurate, but not too long that they are skewed by other factors. Also, remember to store this information for future reference in case other stakeholders want to see this data.

All this information can really help your business improve its conversion rates. After all, if your website is your digital storefront, you’ll want to make sure you get as much “foot traffic” into your shop as possible and then turn those window shoppers into loyal customers.

Trusted Tips: If you lose customers during the checkout, download our free  whitepaper on checkout optimisation