How to master A/B testing for your startup.

A/B testing is a phrase you get to hear around a lot in the marketing world but what exactly does it mean and how can you get a good grasp of A/B testing as a growth marketer?

In my seventh weekly article detailing what I learn in my CXL mini degree scholarship. I explore these questions and share insights on how you can succeed at A/B testing based on what I have learned.

What exactly is A/B testing?

A/B testing also known as split-run testing simply refers to an experimentation process where two or more versions of a web page, landing page, email, etc are shown to different segments of a brand’s audience at the same time to determine which version leaves the maximum impact and drives business metrics.

Before 1995, there wasn’t really the web so of course, A/B testing on the web was non-existent. But there were already experiments of a sort in advertising eg sending coupon codes to people’s houses and sharing via a newspaper to see which is more effective.

Also upon the advent of the web, certain experiments were run where a couple of changes are made to a website to see if it makes a difference on the business’s key metrics.

However, the problem with the above measures and why they don’t count as A/B testing is that the same conditions don’t apply to the experiments.

Say, for example, you run an experiment where you make a change to your website in Week 1, document your results, and make a different change in Week 2. It doesn’t count as A/B testing because the conditions of the experiment are different so you can’t say with utmost surety what’s influencing your metrics.

It could be as a result of external influences or some other reason so for your experiment to count as an A/B test. The conditions have to be the same.

Why A/B testing is valuable:

It’s simple. The point of A/B testing is to ensure that companies are optimizing their efficiency. The big promise of A/B testing experimentation is to put effectiveness on top, to ensure that you are making the right decisions.

When to use A/B testing:

Let’s explore the use cases for A/B testing below;

1. DEPLOYMENTS

When you deploy something on your website. It could be a new feature or update. You’d want to ascertain that it is not having a negative effect on your KPI’s. If your KPI’s go down, then it’s a sign that you probably shouldn’t deploy it.

2. RESEARCH

• Conversion signal maps: If you have a specific web page, let’s say you have a product page with a picture, some lines, and a button. You can run experiments by just leaving out elements. If you see a positive impact or negative impact, it doesn’t matter.

The idea is to learn what elements are having an impact on the website. If you leave out an element and nothing happens. It tells you that the element doesn’t really matter. If you leave out an element and there is a big negative impact, then it’s a really important element so you can pick that to optimize.

• Show fly-ins: So this is basically when you want to test for the effect of social proof on your website e.g “Already, 24 people bought this item in the last 24 hours.” You can test to see if that motivation is specifically working for a group of people on your website.

What you can A/B test:

Copy, subject lines, design, layout, ads, landing pages, etc

What to consider before running an A/B test:

The primary thing you definitely want to consider before running an A/B test is checking to see if you actually have enough data to run an A/B test with.

As a rule, If you are below 1,000 conversions per month, you cannot run A/B tests. Yes, you can still deploy it, you can still run A/B tests, but it’s really hard to find a winner because you are just too low on data and even if you find a winner, chances are pretty high this is not a real winner.

So if you have below 1,000 conversions per month, don’t do A/B testing and conversion can be like transactions, leads, clicks, whatever goal you’re trying to optimize, so it doesn’t really have to be final transactions.

But conversions, below 1,000, no A/B testing. Above, yes, you can run A/B tests. And there’s a second border at 10,000 conversions per month but the rule of thumb is 1,000, 10,000.

Now if you’re going up from 1,000 to 10,000, you can do A/B testing. You can do more and more and more, the closer you are to 10,000 conversions per month.

My belief is if you have 10,000 conversions per month, you can run four or start four A/B tests per week, so have 200 A/B tests per year and then you’ll need a full team.

It’s just not enough anymore to have one person, you need a full team and if you grow bigger it becomes even a bigger game. Why 1,000, 10,000? Well, if you have 1,000 conversions, your challenger needs to beat the control, in reality with 15 percent. So this means that if your A, your control, has 100 conversions, which is quite low.100 conversions, your challenger needs to make 115 conversions.

A 15 percent uplift to be able to be recognized as a winner in A/B tests, now that’s a lot. Sometimes it happens that you are running such a big uplift, but it only happens once a year, twice a year not like every week and even if you have 10,000 conversions per month you still need to make an impact of 5 percent to be able to find a winning A/B test outcome

So, that’s a lot, this is why you cannot run A/B tests if you’re below 1000 conversions per month and you can calculate this for yourself for your own website.