Robots.txt Checker

A robots.txt checker is a tool that fetches the robots.txt file from the root of a domain and parses it into rule groups. Each group starts with a user-agent line and contains allow and disallow paths. The sitemap field can appear anywhere in the file. Google supports those fields and ignores others, such as crawl-delay.

The file must sit at the top level of the host, on the same protocol and port as the content it governs. A file in a subdirectory is not valid. Paths are case-sensitive and matched from the root. Google uses the most specific rule when several match, and the least restrictive rule when two rules conflict. The wildcards * and $ are supported.

The limits are practical rather than mysterious. Google reads UTF-8 text up to 500 KiB, ignores invalid lines, and may cache the file for up to 24 hours. A small site needs only a few lines, but a large one can accumulate hundreds of rules.

A robots.txt checker is useful whenever crawl behavior matters:

• After a launch, to confirm that the production file does not block the new section.
• Before a migration, to compare the old and new rules and catch a missing allow.
• When Search Console reports a URL as blocked, to find the rule responsible.
• When staging is copied to production, to make sure a temporary disallow does not travel with it.
• When a page renders badly in search, to check whether CSS or JavaScript is blocked.
• When the sitemap is not being read, to confirm the sitemap directive and its URL.

The tool is also a quick way to explain a crawl problem to someone who does not work with server configuration every day.

A robots.txt checker helps because the file is powerful and easy to misread. A single line can stop Google from crawling a whole directory, and the effect is not visible in a browser.

• Robots.txt controls crawling, not indexing. A disallowed page can still appear in results without a snippet if other sites link to it.
• noindex only works when the page is crawlable. If robots.txt blocks the URL, Google never sees the meta tag or header.
• A blocked page is a blind spot. Google cannot read the content, the canonical tag or the structured data.
• Blocked CSS and JavaScript harm rendering. Google needs those files to understand the page as a user sees it.
• A missing file is not a block. It means no rules, so every page is open to crawlers.
• A server error has a delay. Google may use the last good copy for up to 30 days while it retries.

The report makes each of those outcomes concrete for the URLs you care about.

The aim of a robots.txt checker is a crawl policy you can explain. Every disallow rule has a reason, every asset needed for rendering is reachable, and the sitemap is declared once. Nothing important is blocked by accident, and nothing private is left open because a rule was written in the wrong group.

That gives you:

• A production file that allows the pages you want indexed and blocks only what you intend.
• Crawlable CSS, JavaScript and images, so Google can render pages properly.
• A sitemap directive that points to the right file over the right protocol.
• Fewer surprises after a staging copy, a migration or a server change.
• A record of the rules you tested and the result, which is useful during an incident.

The file is small, but it sits in front of every crawl. Getting it right is worth the few minutes.

Why use a robots.txt checker instead of reading the file yourself? Traffsite tests the file the way a crawler would, and these are the main gains.

• It tests a URL, not just the text. A rule that looks harmless can match more paths than expected.
• It shows the winning rule. When allow and disallow overlap, the report makes the outcome clear.
• It catches syntax errors. A missing colon or a misspelled field can silently disable a line.
• It confirms the sitemap. The directive is easy to forget and easy to point at the wrong host.
• It is fast and needs no setup. One domain takes seconds, which helps during a release.
• It shows the effect. You see the rule that decides the result for a URL, not just the raw lines.

The tool does not decide what should be private. It shows what the current file actually does.

1. Enter the domain or full URL into the Traffsite box.
2. Read the rule groups. Check which user agents are named and whether a broad group applies to everyone.
3. Test the important URLs: the home page, a product or article page, a category page and any private area.
4. Confirm that CSS, JavaScript and image paths are allowed, especially if the site relies on client-side rendering.
5. Check the sitemap directive. It should be a full URL and should point to a file that returns 200.
6. Edit the file at the root, then run the robots.txt checker again. Allow for caching before you expect the change to appear in Google.

The robots.txt checker suits anyone who controls or reviews how a site is crawled.

• Developers use it when adding a staging environment, a new section or a server rule.
• SEO specialists use it during audits, migrations and index investigations.
• Site owners use it to confirm that a launch did not hide the new pages.
• Content editors use it when a page they published does not appear in search.
• Agencies use it to compare crawl rules across the sites they manage.

You do not need to memorize the specification. The report names the matching rule, and that is usually enough to decide the next step.