What are search engines?

Search engines discover, understand and organise the internet's content in order to offer the most relevant and useful results to the questions searchers are asking.

Google does that better than other search engines by drawing on data from the deepest corners of the web to determine exactly what a specific searcher is looking for. 

To understand how search engines work, you need to know their goal - which is to keep users coming back by consistently delivering useful search results. 

They do that by investing billions every year in developing algorithms that predict as accurately as possible which content users will find most useful in search results.

Everything search engines do revolves around that objective.

How do search engines find the most relevant and useful content for searchers?

Let’s look at how search engines work. 

To provide the most relevant and useful search results, search engines do three things:

1. Crawl: They send out robots (known as “spiders” or “crawlers”) to scour the internet for content. These robots look through the code and content of each URL, whether that’s a PDF, web page, blog article, image, video, or any other format. 

2. Index: The content found during the crawling process is organised into the index. The “indexed” pages can then be accessed quickly by the search engine when a user types a query into the search engine. 

3. Rank: When a searcher types a query, the search engine uses a ranking algorithm to weigh up the quality and relevance of pages according to what users are searching for. The results are then ordered from most to least relevant on the search engine results pages (SERPs). 

So, when you search on Google, the search engine scans its index of “hundreds of billions” of web pages and feeds it through an algorithm to find a set of results that give the best answer to your search query. 

What you see on the search engine results pages are the websites that Google finds to be the most relevant, trustworthy, and authoritative on the subject you’re searching.

That’s why it’s so important to make it as easy as possible for search engines to crawl your website. If they can’t crawl your website, they can’t index or rank it, which means it won’t be shown to searchers. 

Simple as that.

Here are some common mistakes that stop search engines from effectively crawling your website:

1. Poor site navigation - There are lots of navigational issues that hinder crawlers, including broken links and orphan pages (pages that aren’t linked to any other pages). Also, if your mobile navigation is different from desktop navigation, this hinders search engine crawlers. 

2. Content hidden behind login forms - If you ask users to log in or fill out forms before accessing content, search engine bots can’t see the protected pages. 

3. Search forms - Crawlers can’t use search forms. 

4. Text hidden within non-text content - Avoid using non-text formats (such as gifs or images) to display text that you want to be indexed. 

How do you make sure search engines can crawl your website?

There are a few techniques you can do right now to make sure search engines can crawl and index your web pages with ease.

Mostly, it comes down to improving the coding and structure of your website to ensure it can be understood by Google’s bots.

1. Intuitive information architecture - Organize and label content in a way that makes sense for users. 

2. Robots.txt files - Located in the root directory of websites, a robots.txt file instructs search engines on which parts of your site they should and shouldn't crawl.

3. Sitemap - Create a sitemap file that meets Google's standards and submit it through Google Search Console. This will help crawlers follow a path to your important pages.

Image credit: Magento

Once a search engine has indexed your site, how does it rank it?

Not all search engines operate in the same way. 

Google and Bing, for example, have different ways of ranking sites. 

We’ll focus mostly on Google in this SEO guide, for the obvious reason that it’s the biggest force in the search world. 

Google uses search algorithms to sort through the hundreds of billions of pages in its Search index to find the most relevant, useful results for its searcher - that’s its purpose. 

There are known as ‘organic search results” – this means they rank based 100% on merit.

Search engines rank their organic search results based on lots of different ranking factors, such as social metrics, keyword usage, brand signals, user interactions, and many more.  

We won’t list them all here – Google uses more than 200 ranking factors in its algorithm.

This is how experts weight the importance of different ranking factors in Google:

Image credit: Moz

All of these factors boil down to three key things:

1. Relevancy: Google looks for pages that are most closely related to your keyword.

2. Authority: This is Google’s way of determining if the content is accurate and trustworthy (more on this later).

3. Usefulness: Content can be both relevant and authoritative, but if it’s not deemed useful, Google won’t position it at the top of the search results.

But it’s not that simple. 

Google regularly changes its search algorithm to make sure it’s continually meeting its purpose of providing useful results. 

While most changes are minor, Google periodically rolls out a major update that significantly affects search rankings. Knowing these Google updates can help you prepare and improve your SEO efforts.

Here are the Google algorithm updates you need to know.

Google Panda Update (2011)

Panda was first released in 2011 to reduce the dominance of low-quality, thin content in search results, and reward unique, valuable content. It also cracked down on sites with high ratios of ads to content. It affected up to 12% of search results.

Google Penguin Update (2012)

Google Penguin targets manipulative link building practices and link spam. It was first launched as a separate “filter” for search results, then in September 2016, Google announced it was part of the core ranking algorithm.

Image credit: Moz

Google Hummingbird Update (2013)

Hummingbird is cited by experts as a core algorithm overhaul which demonstrates Google’s commitment to understanding the intent of searchers’ queries in order to match them to more relevant results.

Google Mobile Update (2015)

AKA Mobilegeddon, a new mobile-friendly ranking algorithm designed to boost mobile-friendly pages in mobile search results.  

Google RankBrain Update (2015)

Google announced it had been using machine learning to sort live search results to give searchers the best fit for their queries. RankBrain is the only live Artificial Intelligence (AI) used by Google in its search results. 

Google AdWords SERP Update (2016)

Google made major changes to its paid ad platform, AdWords (now Google Ads), by removing right-column ads and introducing 4-ad top blocks. This had major implications on the click-through rate (CTR) for paid and organic results, especially on competitive keywords.

Interstitial Penalty (2017)

Google rolled out a new penalty to punish aggressive interstitials and pop-ups with the potential to damage the mobile user experience. 

Mobile-first index rollout (2018)

This major update means Google will consider mobile friendly websites first when ranking content, even for desktop searches.

Florida 2 (2019)

This is the most recent and biggest update for some time, In March 2019, Google’s Danny Sullivan confirmed that Google is updating its algorithm with a broad core update. This simply means it is not targeting any particular niche or signals, like quality.

Some experts say the underlying goal of all broad core updates is to improve user satisfaction. Others suggest it is a rollback, as many publishers are commenting on how their sites are bouncing back from previous updates. 

Section 1 Recap

Congratulations! You’ve got to the end of the first chapter. It was a big one, but a lot goes into the workings of a search engine, and it’s critical that you know the basics before you learn SEO. You should now have a solid understanding of:

1. What a search engine is

2. How search engines work by crawling, indexing and ranking pages

3. Top mistakes that can prevent search engines from crawling your site

4. Steps you can take now to ensure search engines can crawl your site

5. Major Google updates and how they affect search engine rankings

Now you know what a search engine and how it operates, you’re ready to learn how to use content and SEO together in Section 2.