Scientific based AB testing is a skillset critical to any eCommerce Product Manager.

There is a lot that can go wrong with AB tests. A few weeks back I published an article about the novelty effect; a risk you can run into when not looking at either the length of time you’re running your test for, or the distribution of customer cohorts.

Besides the novelty effect, here are three things to keep in mind to avoid common pitfalls when running AB tests:

1. Isolate variables of change

The waterfall methodology incentivizes big bang projects where an end to end project plan is created, timelines are tacked on, and a whole wheelbarrow-full of change is dumped on the customer all at once.

Changing 5-10 different variables all at the same time doesn't work well with AB tests.

When you change so many variables at once, how will you be able to determine what drove what change to customer behavior or business metrics? You can’t. There is no way.

Agile practices push for incremental changes. By adjusting one variable at a time and running very quick iterations, you can see the resulting customer impact and directly correlate it to the user experience changes you’re making through statistically significant AB tests.

2. Keep your backlog flexible

A trap many Product Managers fall into often driven by organizational pressures, is creating massive backlogs with year long time horizons. If you plan everything out well in advance for the team, you won’t have the flexibility needed to react to test results.

By having a flexible backlog that can change, and appropriately setting the expectation that it will change continuously with stakeholders, you can set your team up for success.

The whole benefit of AB testing is getting early and often reads on how the end user will react to a new feature or change.

Unless you can clearly capitalize on those learnings and adjust future increments of work or the sequencing of the next feature accordingly, putting in the effort to iteratively AB test will be absolutely pointless.

During my time at Nordstrom the product page team has run multiple tests every single week.

No matter how confident you are in the success of your feature, you will constantly be surprised at the actual customer reaction.

Sometimes it will be positive and you will significantly boost conversion for a seemingly very small change, and other times you won’t get a statistically significant read on positive metrics despite feeling like that feature knocks it out of the park.

Flexibility is key when running AB test, and it closely ties in to my last key to success.

3. Don’t be afraid to throw something away

You win some, you lose some. But keeping something for the sake of keeping something is the Product Management version of hoarding.

Fine, MAYBE you will use that collectable set of plates you got as a happy meal toy 20 years ago, and MAYBE they’ll be worth something in 10 more years, but over time things like that can really stack up.

AB tests can very often have neutral results and the sunk cost fallacy does fall into play here. Just because the engineering or design team took the time to create the masterpiece that is this new feature you’re testing doesn’t automatically guarantee it a place in the experience.

Each additional piece of information or product complexity you add to your experience is either one more thing the customer needs to read and consider, or one more thing the team needs to maintain.

Don’t just keep features when they are neutral, really question whether or not that AB test meets your success criteria and adds true value to your end customer.

AB testing is an essential tool for any Product Manager. Make sure to always isolate variables of change as much as possible, keep your backlog in a position where it can react to change and new learnings, and don’t be afraid to throw features away.