There is a lot of hype around A/B testing these days. It is attractive because it seems so simple – run 2 versions of whatever you want to test – be it color of your “Buy Now” button, or the price of your SaaS service, and pick the one that performs better. It sounds like the easiest way to improve conversion rates. Over the last couple of years, there have been numerous reports that say A/B testing is one of the best things to happen to the online marketing community. How true is that? Should you believe all that you read?

Noah Kagan, from AppSumo fame, wrote this on his experiences with A/B testing problems:

" I wanted to share some harsh reality about our testing experiences … Only 1 out of 8 A/B tests have driven significant change."

The outcome of most of his tests were inconclusive. ConversionXL conveys a similar opinion in this interesting article about the hard truths of A/B testing. Is it time to rethink about using A/B testing for delivering promised results?

Some of the problems have got to do with the purpose of these tests – they are purported as a tool to improve conversion rates, but in reality they are just another tool for learning about your customers. Even as a tool to learn about your customers, A/B tests are so dependent on so many other factors, that I am not sure if it is effective at all.

There are a number of factors that determine your conversion rates. Each traffic source generates a different outcome every day. Your current active campaigns, prices, time of the year, branding, advertising, outside stimuli – all have an effect on your conversion rates. There is a very high chance that the changes you made for a Split Test, may in fact have nothing to do with the results you have achieved, unless you have very few variations in your traffic, the kind of people that are landing on your site, your products and numerous other things that affect your goals.  This means A/B testing can never accurately tell you that the change you have made is the factor that produced the resulting conversions.

My second problem with A/B testing is the obsession with choosing a “winner”. It is easy to get caught up in picking winners and forgetting the bigger goal of your business. Also, quite frequently, just because an option turned out to not win does not mean you dump it. In fact, the failures can often be more valuable in teaching you a thing or two about your customers.

A lot of advocates of A/B tests seem to convey A/B tests as a replacement for well thought out strategy. There is a parallel to this in the design world – almost everyone agrees that design by committee is almost always bad. A/B tests cannot be a replacement for the strategy, it can serve as a quick feedback mechanism for your plans.

The inherent nature of an A/B test doesn’t allow for it to optimize for your long term goals. The success of most businesses is decided in the long term – retaining customers over longer periods is key to them. Just because your test resulted in 100 additional signups mean nothing, if most of them ditch your business within the first few months. Most A/B tests are run for a much shorter duration and almost never consider the long term goals of the business.

If you think that you can get more of your visitors to buy, just by adding a bigger “Buy now” button, you’ll be surprised. Your customers are smarter than you think – there is a reason why they are not buying, and in a lot of cases, it is OK. They are probably on your site to form opinions about the product/need, and other times they are on your site to compare your product with your competitors. Understanding your visitors, and helping them achieve their objective should be your motivation. A/B testing goes the opposite way, where you offer 2 choices and let your visitors pick the best one, whereas the focus should be on providing the right information to the right people at the right time.