Forget About AB Testing Until You Do This

Ever encountered this situation?

The conversion rates are down.

People in the marketing department are getting a bit anxious.

Then the C-level gets anxious.

Until someone has an idea: “Let’s do AB testing!”

People get excited.

The whole marketing team gets together and brainstorms for possible AB tests.

“Let’s change that headline,” says someone.

“Let’s put a video above the fold,” says another one.

“Let’s use gifs instead,” says the third one.

“Let’s lose the hamburger menu,” says the fourth one.

All those ideas are then put into a list. Each idea in the list is rated.

The ratings look too close to call, so they are broken down into impact, confidence, and effort. Now it feels like you’re getting somewhere.

Then a timeline is set.

Justifications are being made to get more resources for testing — the conversion rate, everyone agrees, must go up.

Finally, the testing begins.

You then realize that in order to say anything with confidence about the test, you’d need to run each test for two months.

You don’t have that kind of time.

So, most test results end up being inconclusive.

But that’s okay, you’ll judge the test results by looking at Bayesian statistics instead.

Some of the tests look like they actually might be improving something. Sure, there’s a 20% chance, that you’re making things worse, but you’ll figure it’s worth the risk.

So, you put the winning variations live.

Six months later you realize the conversion rate is still more or less the same, although you’ve run dozens of “successful” tests.

What happened?

We don’t usually think of marketing as part of the product (“The web is just something that the marketing guys update.”). But it is part of the product experience.

And in product management (as well as in life in general), it’s very tempting to jump straight to solutions without ever actually understanding the customer needs deeply.

Did you ever stop to observe and interview your potential customers? How did they find the website? What are they looking for? What’s unclear? What’s putting them off? And if they did become a paying customer, then what won them over?

It’s so easy to brainstorm for new solutions, especially when you are bombarded with stuff like “How I improved my conversion rate by 120% in 2 weeks” or “5 best checkout designs”.

What none of those articles will give you, however, are real insights about your customers.

Each product is different. There are virtually no identical SaaS businesses. Therefore they address a different set of desired user outcomes.

And each market is different as well even if you try to address them with the same product.

Health care professionals trust only other health care professionals. They won’t trust bankers. Bankers won’t trust health care professionals. They trust other bankers. But the US bankers won’t trust testimonials from the bankers in Lithuania and vice versa.

Insights like these come only from observing and talking to potential and existing customers. They won’t come from looking at funnels or conversion rates. You can’t just brainstorm your way out of real work.

Every successful startup I know is obsessed with talking to their customers. Unfortunately, the bigger the companies grow, the further the majority of employees get from their customers. It’s inevitable and you have to fight against it.

With growth come resources and specialization. Suddenly you’ve got people whose only job is to analyze statistics and brainstorm for solutions without ever meeting a real customer.

Looking at a bunch of statistics and funnels tells you nothing about what’s driving them. Psychologically-speaking.

Brainstorming for AB tests (i.e. solutions) only makes sense after you’ve got the insights from your customers on what to possibly test.

You might then realize that a website might not even be the best way to convert potential customers because they started their journey by searching Google Marketplace (instead of going to Google Search and finding your homepage). So, you might need to rethink your growth strategy.

You will never discover things like that by just looking at homepage statistics (or product usage statistics for that matter) and trying to brainstorm for solutions. Data != insight.