A robots.txt file may seem small and simple, but a single incorrect line can have major consequences for your discoverability. You use it to guide search engines: what they are allowed to crawl and what they are not.
For technical SEO, robots.txt is no place for guesswork. Especially on WordPress websites and online stores, you often see that a single incorrect line accidentally blocks important pages or, conversely, opens up unnecessary sections.
What exactly does robots.txt do?
robots.txt is located in the root of your domain and is the first file read by search engines. The file provides instructions to crawlers regarding which paths they are permitted or prohibited from visiting.
This is different from indexing. A page may technically be found by Google via links but not be crawled due to robots.txt. Conversely, a page may be crawled but still not appear in the index due to a noindex tag.
When should you use robots.txt and when should you not?
Use robots.txt primarily to manage crawl budget and restrict irrelevant or sensitive paths. Consider admin environments, internal search results, or filter variations that have no SEO value.
Do not use it as a solution to keep content out of the index. If you want a page not to appear in Google, noindex is usually the right choice, provided the crawler is still allowed to visit the page to read that instruction.
What does a proper robots.txt file look like?
A basic version is often straightforward and contains only what is truly necessary. For many websites, you need just a few rules for a clean crawl structure, not a long file full of exceptions.
A simple example is:
- User-agent: *
- Disallow: /wp-admin/
- Allow: /wp-admin/admin-ajax.php
This blocks the WordPress admin area but allows a technical function that is sometimes necessary for themes, plugins, and frontend interaction. For e-commerce sites and custom platforms, the setup may differ, but the principle remains the same: block selectively, not broadly.
What are the most common mistakes?
The most common mistake is blocking too much. A rule like Disallow: / can accidentally make your entire site invisible to crawlers, and you often don’t notice this until rankings or indexing drop.
Other errors are less visible but just as annoying:
- Incorrect file location, causing robots.txt not to be on the root
- A typo in a path, such as /wp-admn/
- Unintentional blocking of CSS or JavaScript
- Test rules that accidentally remain live
- Conflicting directives
The latter is particularly common on websites that have been modified by multiple parties. In such cases, old staging rules remain in the live file, even though no one knows exactly why they were ever implemented.
What exactly should you leave open?
Search engines need to be able to crawl enough of your page to understand it properly. If you block important CSS or JS files, Google may not be able to assess your layout or functionality as effectively, which can indirectly harm your SEO.
Therefore, in general, keep public content, categories, product pages, and relevant blog articles accessible. Instead, limit yourself to technical noise such as parameter URLs, internal search results, or excessive filter combinations that add little value.
How do you check if your robots.txt file is set up correctly?
Always test your file after making any changes. First, check if the URL is actually accessible at yourdomain.com/robots.txt, and then verify which rules are active.
Then follow up with a few practical checks:
- Test important URLs in Google Search Console
- Check to make sure your sitemap URL isn’t accidentally blocked
- Review server logs to see which crawlers are spending a lot of time on irrelevant paths
- Ensure that staging settings are never applied to production
For larger sites, log analysis often yields the greatest benefits. You can then see exactly where crawlers are spending their time and where crawl budget is being wasted.
How do you approach this with WordPress and e-commerce sites?
With WordPress, robots.txt often revolves around default directories, plugin paths, and admin elements. The trick is to cleverly combine generic behavior with the specific structure of your theme, plugins, and any custom functionality.
For e-commerce sites, pay special attention to filters, sorting options, search results, and pagination. These elements quickly generate many URL variations, and not every variation deserves crawl budget. A solid technical SEO approach ensures that your key category and product pages take priority.
What does this mean for your website?
A well-configured robots.txt file doesn’t automatically make your site easier to find, but it does prevent search engines from wasting time on the wrong pages. It keeps your site’s crawl tighter, more stable, and more predictable.
Think of robots.txt as a technical tool, not a magic bullet. Combine it with indexing control, internal link structure, and clear canonicals for the best results.
Frequently Asked Questions About robots.txt
1. Does robots.txt always have to be on your domain?
Yes, search engines expect the file to be at the root of your domain. If it’s missing, bots will usually just continue crawling without any additional restrictions. That isn’t necessarily wrong, but you do miss out on a key control mechanism.2. Is robots.txt the same as noindex?
No, these are two different things. Robots.txt controls crawling behavior, while noindex determines whether a page can be included in the index. If you really want to keep a page out of Google, robots.txt alone is usually not enough.3. Can a blocked page still appear in Google?
Yes, that can happen if there are external or internal links pointing to that URL. Google may sometimes still display the URL without content because the crawler is not allowed to read the page. That is precisely why robots.txt is not a complete indexing solution.4. Should you include your sitemap in robots.txt?
You don’t have to, but it’s helpful. This way, crawlers can find your sitemap faster, and you have everything in one place. It’s especially useful for larger websites or sites with a lot of new content.5. Why do you often block /wp-admin/ in WordPress?
Because that directory is primarily intended for administration and has no SEO value. This keeps crawlers away from a technical section that isn’t relevant to visitors. You often leave the rule for admin-ajax.php open because it may be functionally necessary.6. How often should you check robots.txt?
At a minimum, after every technical change, migration, or release. For websites with many plugins, filters, or customizations, periodic checks are advisable. This prevents old test rules or incorrect paths from remaining in place.7. Can you save crawl budget with robots.txt?
Yes, especially for large sites with many parameter URLs, search results, or duplicate pages. This helps search engines reach the pages that truly have value more quickly. This is particularly relevant if your site generates many URL variations.A robust robots.txt file starts with a clear technical decision, not a standard template. If you want to prevent minor errors from disrupting your crawl and indexing, a thorough technical SEO check often provides a quick solution. At Digital Wizards, we can address this together with your development and SEO structure if desired.
