Experimentation is at the heart of digital marketing.
Whether testing ad formats or performing conversion rate optimization with landing page designs, A/B testing allows you to make sure your activity resonates with your target audience, invest in large-scale changes and build your conversion funnel.
But running SEO A/B testing is a whole different ball game. Forget everything you think you know about A/B testing.
Running A/B tests for search engines, as opposed to for users, is a more complicated beast involving a different test design, controlling for external factors and a range of tools that can help you run experiments.
This is a guide that covers all of that and more along with case studies and example tests.
What is SEO A/B Testing and why does it matter?
SEO split-testing helps you see the true potential of your search engine optimization strategy and identify where you can invest more to scale up your strategy. As with all A/B tests, you’ll get concrete data that you can take to the powers that be to prove that SEO is a channel worth investing in for real revenue results.
A staggering 96% of companies that run SEO testing report an increase in traffic (SEMrush). Another 92% see uplifts in click-through rates, with more than half saying new customers come their way.
If that’s not a good enough reason to invest in SEO A/B testing for your strategy, what is?
SEO split tests give you more control over your SEO activity by letting you make small, low-risk changes to your website before rolling them out to the wider site. You can see what works and what doesn’t, without risking your whole site’s ranking.
Split testing for SEO works in a similar way to other A/B testing: you make one change on a web page, then monitor the results.
The difference is that you are monitoring the search engine’s response to the changes, not the users’ response. For example, rather than looking at how a user clicks through on the site and how much time they are spending on-page, you look at where your page ranks in organic search.
Why traditional A/B Tests don’t work for SEO
Traditional A/B tests are not the same as SEO split tests and they won't work to tell you how to scale your SEO strategy. The reason is simple: traditional A/B tests are for uses, not search engines.
With SEO split tests, the “user” we are testing for is Googlebot, not humans. So you can't split your audience as you would with traditional A/B tests. There is only one Googlebot, after all
You also can’t make two versions of the same page because it would cause negative issues with SEO, like duplicate content. That’s why when you are running experiments for SEO, you need to break a group of template pages into control and variant pages. It's not about splitting your users.
How to design an SEO split test
Start by selecting the pages you are going to test. Find a group of pages on the same template. For instance, for an eCommerce site you might select a group of category pages or product pages.
Then, split them into two groups: control and variant.
Control Group stays the same
Variant Group has the change implemented
Deciding which pages should be control or variant pages is the most important part of SEO testing. There are two main criteria to consider:
Both groups should have similar levels of traffic. If control pages have significantly more traffic than variant pages, the tests won't be as helpful.
Both groups should be statistically similar. In other words, if one trends upward or downward, so should the other. Otherwise, the results will be skewed and you won't be able to trust that any changes in traffic aren’t coming from external factors that impact only the control or variant group.
Next, decide what you want to change and make the tweak to the Variant group. Focus on one change at a time, so you know what is causing the response on search results.
Don't change anything on the control pages - these pages stay the same.
Then, monitor the difference in organic traffic between the two groups. You're looking for results of statistical significance. In other words, it's not enough to have a difference in the traffic to one version over another - you need to know that you can trust the result. That means the test needs to reach statistical significance.
You can use A/B testing tools that manage this for you, such as Search Pilot.
How to account for external factors like seasonality and Google updates
Another complication for SEO testing is this: how do you know that any change in organic traffic is from the change that you made and not external factors like seasonality, Google updates, etc?
It comes down to how you choose your control and variant pages.
If we do the grouping correctly, each set of pages will have similar levels of organic traffic and will be statistically similar to each other. So, when the test begins, traffic to both groups of pages will trend upwards and downwards based on external factors.
Types of things you can test (with case studies)
So, what exactly are you SEO A/B testing? Consider all the ranking factors you take into account when optimizing your website for search. There’s technical SEO, on-page, content, and off-page SEO, such as backlinks.
Where you can really test your SEO strategy is with technical and on-page elements. These are the factors that search engine crawlers scrutinize when indexing and ranking your pages and content.
At the end of the day, it doesn't matter how great your page content is, if your technical SEO isn’t functioning well, you’re losing precious traffic.
The factors that matter to crawlers are:
Inclusion of keywords in meta titles and descriptions
Placement of heading tags
Keywords in anchor text
Schema markup of a page.
With SEO split-tests, you can A/B test each of those elements and more without waiting weeks for search algorithms to feedback the results. You can see the test results far quicker, and scale the changes to similar pages on your site.
Take a look at these A/B testing examples:
For something only a few words long, meta titles play an important role in driving clicks to your website. They are one of the first things users see in your search listings.
Some ways to test different meta titles include:
Adding or removing offers, such as “free shipping” and "free trial"
Adding action words like “buy”, “download”, "shop"
Changing the character length
Test these simple changes on selected pages in a Variant set and see how different versions affect your click-through rates and rankings compared to the Control group.
In this example of a split test, the testers wanted to find out whether adding the price would have a positive or negative impact. The hypothesis was that because users are notoriously price sensitive, having prices advertised in the SERP that are even a little bit higher than the competition may mean bad news for click-through rate.
The test added the price after the product name in the title tag on an eCommerce site. The resulting impact of this SEO split test was a 15% drop in organic traffic for the variant pages compared to their forecast performance.
Your meta descriptions play a substantial role in whether or not someone will click on your content. They exist to add context and encourage click-throughs, persuading searchers to click on your page over those of competitors.
SEO tests you can run include adding or removing:
Offers like “free shipping” or "free trial"
Calls to action
Heading tags are on-page elements that tell search engines what a page is about. You will have one main H1 for each page, and several H2s or H3s as subheadings to break up your content.
Testing your heading tags helps you make it easier for search engine crawlers to understand your content, which means they in turn can do a better job of finding and ranking the best, most relevant results for their users.
Run tests with your Control and Variant groups to see whether any of these changes deliver results with any statistical significance:
Placement of the heading tag
Length of sentences
Use of H2s or H3s for subheadings
Including different keywords in headings
Image Alt Text
Image alt text is what tells Google what the image shows, so it can find suitable results for users in the Google Image results pages.
Making changes to your alt texts is a simple way to test whether or not the ranking of your image or page is affected.
Try these small alterations to your image alt text:
Try hyphens vs. spacing
Add or remove characters
Link Anchor Text
Anchor text tells Google about the link to which you are pointing, so crawlers can work out what the page is about, and what it should rank for.
With link anchor text tests, you can experiment with:
Character length of the text
Exact match vs. broad match text
Format of the link on your website
More advanced SEO tests
So far, the SEO experiments have been basic tests related to specific on-page elements on your website.
There are more advanced tests you can run on your website including:
Internal links direct people and search engines from one page to other relevant pages of your website.
They can affect user behavior which is measured by bounce rates, time on site, pages per session, and so on — all of which indicates to Google the quality of your site’s content and user experience.
These tests require careful planning because they will impact the Variant pages that you change, as well as the target pages that gain or lose internal links. So, you need to measure the impact on multiple groups of pages at the same time.
Some examples of internal linking tests are:
Internal anchor text tests
Add or remove links to related products and content
Increase or decrease the quantity of links to related products/pages
Shallow vs deep links i.e. linking to deeper pages in a site’s architecture
Delete, add, move elements
Test whether or not deleting, moving and adding elements has a positive or negative impact on search performance.
One thing you can test is deleting breadcrumbs.
Let’s say you have the following breadcrumb format on product pages: “Home > Camping > Tents". Your hypothesis is that Googlebot prefers it when your page has more context on the product by using breadcrumbs. So you run a test to compare two versions. On the Variant pages, you delete the breadcrumb.
The end result? Organic traffic to those Variant pages decreases by 6%.
In this case, Googlebot prefers the presence of breadcrumbs.
Now you can consider how to implement this change across your website.
Other elements you add, delete and move in an experiment include:
Calls to action
In this example, the client reconfigured their page layout to see how it would impact search rankings and performance. The test was to simply move a widget up the page.
Here is what the change looked like:
Not only did the change impact organic traffic, it caused a 7% drop in organic sessions.
If this had been rolled out to the whole website without testing, it would have translated to an estimated loss of 10,000 organic sessions a month!
Schema markup is a way of using structured data to show Google what additional data it can pull from your pages and show in its search results. This could be star ratings, price, and much more.
These tests require a greater understanding of HTML and CSS to run effectively. Use split testing to experiment with:
Testing structured data types (JSON-LD vs Microdata code)
Adding FAQ markup
Adding review markup
Whether Schema markup has an impact on one version over another
If you see statistically significant results from any Schema split-test, it's worth planning a wider roll-out across your site, with further tests if necessary to support your data-driven decisions.
SEO Split Testing FAQs
How long does it take to know whether the variant was effective?
How long you wait to measure your A/B tests depends on how often Google crawls your website and sees the changes. If you’re changing primary pages that Google crawls regularly, you can likely see whether those changes had any impact within two weeks. In some cases, it might take longer.
What tools are available to make split testing easier?
Split testing can be overwhelming to run by yourself, but there are some great SEO tools you can use to make it easier. For example, Clickflow is a tool designed to make SEO experimentation easier. It syncs with your Google Search Console account to find opportunities where you have a high impression count but low click-through rate. You can use the tool to run different tests and then find the data in the reports section.
Another tool that helps with testing is Google Optimize, which works directly with Google Analytics to run A/B tests on a specific website.
Whichever tool you use, remember to approach your SEO testing with a clear objective.
How do I know my test has statistical significance?
An SEO experiment with statistical significance means that the results aren’t biased or inaccurate. It proves that the result you’ve recorded is reliable and trustworthy. If you run the test with a statistical significance level of 98%, you can be sure that the results are accurate, and aren't being skewed by an anomaly or seasonal trend.
“The amount of time required for a reliable test will vary depending on factors like your conversion rates and how much traffic your website gets; a good testing tool should tell you when you’ve gathered enough data to draw a reliable conclusion.”
The best SEO testing tool will help you work out when you have good enough results to stop testing.
Ready, set, GROW
Done correctly, SEO A/B testing can be a powerful way to refine your SEO strategy with data-driven decisions. It can also help marketers prove the value of their work to stakeholders who need convincing of the power of SEO. Now, you need to invest time to make it happen.
You can’t know what’s making an impact in your search engine optimization strategy unless you test it. For every idea you try, test it against the Control group and track the results. Then, look at how you can roll out changes across your website to scale your success.