A Guide To The A/B Testing Process

A/B testing refers to an experimental process. It can also be called split or bucket testing and is instrumental in the UX design process. You have a landing page, email, app, newsletter or some other form of marketing material. You make a change to it. The original is then called “A,” and the second version is “B.” Potentially, you can have many more variants. However, it can be best to just keep it more simple and focused.

You compare A and B (and other possible variants) to see which better helps convert your audience from readers and website visitors to people who take action, that purchase products and services or sign up for a newsletter or free trial. That one difference between the original and the variant can be hugely impactful, and you can learn from it to enhance your site, email, app, etc., and your overall marketing.

You can test any number of things: the headline, subheads, copy, call to action, call to action button, offers, pricing, design, pictures, layout, images, colors, logos and so on. It all depends on your goals, your marketing materials, how your business is doing, what your current conversion rates are and how you think you can improve.

1. Optimize Your Work via A/B Testing

You should always be looking to improve, so there’s no excuse not to do at least some A/B testing. It helps optimize your marketing efforts, so you know what’s working and what’s not. You know what to concentrate on, what to emphasize and what to jettison. This can save you in time, effort and costs as well as make your marketing more focused and effective.

A/B testing results gives you metrics that can steer you in the right direction to enhance your landing page, email, newsletter, app and so forth. The testing results validate your hypotheses or take you in a different direction that results in you gaining a better understanding of your customers, visitors, subscribers and audience.

If your conversion rate is improving due to one change, you know now that you should make it and allocate your finances and resources toward that end rather than wasting them on something that either doesn’t work or has plateaued in its effectiveness.

However, please note that the more things you test the more costs, effort and time you will have to put into the testing process. It could only take a small tweak to make all the difference, though, as one company discovered.

2. The A/B Testing Process Generally Involves Four Steps:

• Measurement.
• Planning and prioritizing.
• The actual testing.
• Test again and again.

Step One: Measuring Your Performance

Before you know how you can improve, you need to get a grasp of how you are currently doing. First, consider your business goals and that of your website, email, app, etc. Who is your audience and how specifically do you want to improve? What are your current conversion rates like and what other key performance indicators (KPI’s) do you want to take into account?

Not sure what to track? Here are four suggestions.
Data always helps set goals and indicate areas of improvement. Google Analytics can provide actionable data about how your audience is responding to your marketing, design, offers, pricing and more.

Qualitative data on customers, gathered from surveys, emails and other feedback, should also factor into your decision-making. You might even segment this customer information by source, behavior and outcomes to better understand customer behavior.

Step Two: Planning and Prioritizing

Once you have this valuable information gathered and sorted, you need to identity what areas to test. You might find multiple areas that need improvement, but based upon your needs and available resources, you will need to prioritize what gets tested first. Also, consider that you must look at your data as a whole, as it will help you gain insights that looking at it in separate segments might not allow you to realize.

For example, if you have a whole website, you don’t want to look at the metrics for one page but all the pages together. You want to examine the data to see what your top landing pages are and what your top exit pages are. Also consider that your homepage, category pages and product pages are the top part of the conversion funnel.

Transactional pages, pages that give site visitors the chance to sign up or purchase something, are at the bottom. You need to see where visitors might be dropping off in the conversion funnel and enhancing that area, whether it’s the top or bottom.

Remain mindful of the money, resources and time you have at your disposal. They will help dictate what changes you think will be made, as they will be based upon business need and what is feasible. Sometimes, you might need an entire overhaul of the copy or design, or you may just need to change the call to action.

Step Three: The Actual Testing

Now that you’ve set your goals and have gathered the proper data, you can form a hypothesis. Decide what change will make a difference between the original and the variant. The results will either validate your decision-making or steer you toward a better course of action. Any hypothesis you make must be testable, have a goal of solving conversion problems and be able to provide you insights into your customer base and the market.

When testing, you need to have significant volume in your test groups, results and the differences between them. You need to have statistical significance of over 95 percent for your testing. This way, you know that your results are not based just on simple chance and that you can have confidence in them.

Also, you are testing to see increases in revenue and not just increases in conversion rates. An increase in revenue could coincide with a decrease in conversion rates if you have raised prices. An increase in conversion rates might coincide with a decrease in revenues, depending on what factors play into the equation (price increase, demand, market forces). Thus, always look to the bottom line, as it is what matters the most.

Step Four: Test Again and Again

You should always be improving and optimizing. After running one test, run another and another. The more data you have the more you can understand your audience and how to propel them to action. Not all A/B testing works, and sometimes it results in only small gains. However, you can always run more tests.

A/B testing results are only another baseline for you work off of and to try to improve upon. If you don’t, your competition might, and you want to stay in the game. Testing helps you set up a cycle of continuous improvement that ensures you aren’t left behind and can remain competitive. If you want to become a market or thought leader, then testing plays a key role in making that happen.

See how A/B testing helped these three different businesses better understand what makes their customers tick and how that impacted their bottom lines.

Also, keep in mind these considerations when employing A/B testing.

5. Mobile

The mobile experience is increasing more and more in importance. People use their mobile devices to read the news, check their email, do some of their work, reach out to friends and purchase products and services. A successful business has to make their mobile experience engaging and effective, and A/B testing can help tremendously with that.

You might want to test something about the mobile experience, such as its navigation or forms, since they are always key factors in how well received an app or email on a mobile device is. Bear in mind that mobile users might be interested in different types of content than web users as well, so this should factor into your marketing and testing strategies.

6. Google and SEO

Though Google permits A/B testing, it encourages these best practices:
Do not run tests for longer than necessary, as it can be seen as an attempt to deceive search engines. Update your site and remote all test variations as soon as a test ends.

Do not cloak. Cloaking refers to showing search engines different content than what a typical user would see. Doing so can demote your search rank or have your site removed from search results altogether. Prevent cloaking by not segmenting visitors to display different content based upon a user-agent or IP address.

If you run tests with multiple URLs, use the rel=”canonical” attribute to point the variations back to the original version of the page. That way, Googlebot won’t be confused by multiple versions of the same page.

If you run a test that redirects the original URL to a variation URL, use a 302 (temporary) redirect rather than a 301 (permanent) redirect. This tells Google and other search engines that the redirect is temporary. Then they keep the original URL indexed rather than the test URL.

7. The Local Maximum

Keep in mind that most A/B testing involves testing single isolated variables. It gives you more confidence in results, but over time you may only be making small improvements that result in only small gains. This is called the Local Maximum, meaning that you might have plateaued and have to make more major changes to make larger and more significant gains.