A robots.txt file is a file located in the so-called root of your website. You can think of a root as a folder containing all your site’s files, including the .htaccess file.
Robots.txt files operate based on the so-called Robots Exclusion Protocol. This protocol stipulates that, as a website administrator, you must be able to specify which subpages may and may not be indexed by fully automated search robots (also known as crawlers), which perform this indexing work on behalf of online search engines such as Google. With a robots.txt file, you can prevent certain subpages from being found and indexed by these robots.
Robots.txt files and SEO
The purpose of robots.txt files is therefore to prevent some of your website’s subpages from being indexed. In other words: with this file, you determine exactly which subpages should and should not be indexed.
It’s wise to pay attention to this, especially regarding your SEO performance—that is, your website’s online discoverability. After all, it’s quite possible that some subpages aren’t optimized for SEO at all or shouldn’t be indexed. It is possible that separate pages or XML sitemaps are generated from your website that are used to build functionality within your site: you do not want to send these to search engines.
If you forward those pages or sitemaps, you can lower your overall SEO performance—and thus your Google ranking. Furthermore, you naturally want visitors to your website to land on the correct pages, tailored specifically for them. You can prevent this by including clear instructions in a robots.txt file, thereby excluding pages that are not SEO-friendly from indexing.
How robots.txt works
An undeniable advantage of robots.txt files is that they are relatively easy to create. This is certainly the case when using WordPress as a CMS. As the owner of a WordPress website, you can simply download the well-known Yoast SEO plugin. With this, you can then very easily create a robots.txt file.
There are also various programs that allow you to generate a robots.txt file. These programs are also known as robots.txt generators. Good examples of these include SEOBook, SEOptimer, and Ryte. Once you’re working on a file, the focus then shifts primarily to the subpages that need to be indexed (allow) and those that should be ignored (disallow).
Robots.txt via Public HTML
Another way to add a robots.txt file to your WordPress website is through the file manager, which you can access via FTP, for example. Here is an example of a robots.txt file you can use for a WordPress website with pages and a blog (replace https://jouwwebsite.nl with your website’s URLs):
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://jouwwebsite.nl/page-sitemap.xml
Sitemap: https://jouwwebsite.nl/post-sitemap.xml
Please keep these points in mind
When you start working with robots.txt files, there are a number of key points and pitfalls you need to be aware of:
- The information contained in your robots.txt files is visible to anyone. You can usually view this by adding /robots.txt to your domain name. So make sure there is no strange or sensitive information in there.
- Keep in mind that backlinks from external websites can undermine your disallow actions. After all, a site linked to via a backlink will eventually appear in Google search results.
- Do you have a sitemap available to website visitors? If so, be sure to allow Google to index this sitemap.
- Are you aware that certain malware bots are active? If so, make sure these bots cannot index any of your pages.
- Is it truly highly undesirable for a specific webpage to be indexed and found via search results due to highly sensitive information? In that case, it is better to secure this page using, for example, a login screen combined with a password.
- Note: WordPress does create an .htaccess file by default, but not a robots.txt file: make sure to create a robots.txt file as soon as your website goes live.
