In A/B Testing We Trust

Idea Theorem- In A/B Testing We Trust

In A/B Testing We Trust

In A/B Testing we trust, human behavior surprises the best of us. That makes it really challenging to understand which product features or marketing campaigns will be accepted by the customers. This is where A/B Testing can help.


What is A/B Testing?


A/B test consists of A – the champion, B the challenger.


A – The champion – can be categorized as your current system or it can also be a version of an ad, a new feature or a new design

B – The challenger – is the modification that you think will improve the current system or can also be another version of an ad, a new feature or a new design

At the end of the day – we are looking for either A or B to perform and deliver better results on key metrics within a limited time.


A use case might be a landing page. Landing pages are designed with a single objective; users are expected to come to the landing page (after clicking on the ad) and then expected to fill in the details or make the purchase directly. These are simple and perfect candidates for A/B Testing or A/B/n testing. A/B/n Testing is an extension of A/B testing because it involves three or more variations instead of two.


The reason for trying out these variations of landing pages is to identify which variation gives better results and increase conversions. This doesn’t mean the landing page variations need to be very different, it could be minor changes too, for example, one landing page has a larger button size compared to the other. A lot of people think that A/B testing might not have a major impact, here are some studies which show the contrary:



A study by Evercore Equity Research says “Booking’s utilization of A/B testing to determine what drives higher conversion (i.e., button location, colors, etc.) can be better leveraged across a wider footprint, as it does not need to be bound by perceived constraints in local markets being run by regional heads. We understand that this alone has contributed to Booking.com enjoying conversion rates that approximate 2-3x the industry average”



Firefox team noticed an increase in conversions by just changing the download button text from “Try Firefox 3” to “Download Now – Free”


Objective of A/B Testing


  1. Let Customer Drive the product.
  2. Help your company drive your key metrics


Leverage A/B testing to let customer drive the productClick To Tweet


Let Customer Drive the Product

User research is done in order to understand the target customer. If you need some help understanding the target customer check out – My target audience is everyone.  The information collected from user research helps build a hypothesis, this hypothesis is used to build variations in your product features or marketing campaigns. These variations are then tested using A/B testing to figure out which truly sticks with your customer. Using this strategy, the customer is helping you drive the product.


Help Drive Your Key Metrics

The goal of experimenting with your marketing campaigns or product features is to be able to identify which one is helping you drive your key metrics. It can mean an increase in conversions, increase in average time a user spends on the platform or higher click-through rate of your marketing campaign. The key metrics will be dependent on the business and it can be specific to a product or a campaign.


A/B Testing to Drive Future Investments


One of the challenges with A/B testing is that it costs money, time, resources and a good infrastructure with good tools… that’s always not very easy to get in any organization. And if setting up the experimentation costs are high, then senior leadership is less likely to be excited about more experimentation.


By investing in third-party tools and services, you will be able to bring down the cost of experimentation and make the tests more reliable, thus creating a culture of testing across the organization. This way data will act as a fuel for the hypothesis, making your organization more informed and avoid blindly following highest paid person’s opinion (HIPPO).


Image from H. Jack West, MD


By having set up controlled experiments, you will get an opportunity to try out new ideas quickly. By trying these ideas in a controlled manner and leveraging the data that came out of these experiments, will help you make better decisions. Such as making this product faster by 2%, will increase the revenue by 10%. It will be an understatement, that the path of experimentation will not involve a lot of failure and risks (or you can say you will kiss a lot of frogs before you find the prince). Failure and risks are the two words that are taboo, these words do not go well in any project meeting. But failure and risk are part of the game to grow your business.



Failure and Risk are part of the game to grow your business Click To Tweet


What Next


Idea Theorem has recently launched UX Audit services for you to understand where you are missing pieces to gain more user base. You can also check out DIY UX Audit where we have provided you with our insights that will help your business grow.


Idea Theorem is Toronto based digital agency which creates simple and usable products for web and mobile. Our human-centered design approach lets us understand your customers, identify their pain points & deliver solutions that enhance their experience with your brand. Contact Us if you have any questions and we will be happy to help you.