Indexability Checker

An indexability checker reads a page and its response headers, then compares them with the indexing rules Google publishes. The Traffsite indexability checker reports:

• The HTTP status code and whether the page returns a 2xx response.
• The robots meta tag, including noindex and nofollow values.
• The X-Robots-Tag header, which can also apply to PDFs and other non-HTML files.
• The robots.txt rules that match the URL.
• The canonical tag, and whether it points at the page itself.
• Soft 404 signals, such as a 200 response with an error page or very little content.

The tool tests only the signals it can see. It cannot read Google's index or promise that a page will be indexed. It can show whether a clear blocker exists, and that is the part you can fix.

Run a check whenever a page should be in search but is not:

• After a new page or section goes live.
• After a migration or domain change.
• After a template or CMS update that may add a global noindex.
• When a page loses traffic and you suspect an indexing change.
• When staging or password protected pages accidentally become public.
• When a product, category or landing page is missing from results.
• When a robots.txt edit may have blocked more than intended.

It also helps before launch. A quick check on a staging URL can catch a noindex tag before it reaches production.

The first benefit is separating crawling from indexing. Googlebot can fetch a page and still decide not to index it, and the report shows which stage is failing.

• A robots.txt block stops crawling. Google cannot see a noindex tag on a blocked page.
• A noindex meta tag or X-Robots-Tag tells Google to drop the page from results once it is crawled.
• A canonical pointing elsewhere can make Google treat another URL as the representative version.
• A soft 404 makes the page look like an error even though the server returned 200.
• A persistent 5xx response can remove a page from the index over time.

The checker also catches contradictions, such as a page that is noindex but canonicalises to itself, or one that robots.txt allows but an X-Robots-Tag blocks. Seeing every signal in one place makes the fix obvious.

The aim is a clear answer to one question: is anything stopping this page from being indexed? With that answer you can:

• Remove an accidental noindex before launch.
• Fix a robots.txt rule that blocks a whole section.
• Correct a canonical that points at the wrong URL.
• Replace a soft 404 with a real page or a proper 404.
• Confirm a page is eligible before you invest in links and content.
• Keep a record of checks after a migration.

The outcome is confidence: you know a configuration mistake is not holding the page back, so you can focus on content and links.

Why use a dedicated tool instead of checking the page source by hand?

• It reads headers as well as HTML. An X-Robots-Tag never appears in the page source.
• It checks robots.txt for the exact URL. A broad disallow can catch pages you never meant to block.
• It compares the canonical with the page URL. Relative or malformed canonicals are easy to miss.
• It detects soft 404 patterns. A not found message in the body can still return 200.
• It works on staging and production. Run the same check before and after a release.
• It gives a clear fix list. Each result names the signal, not the symptom.

1. Enter the full URL you want to test, including https://.
2. Run the check and read the status code first. A 4xx or 5xx needs attention before anything else.
3. Review the robots meta tag and X-Robots-Tag. Look for noindex on a page that should be indexed.
4. Check the robots.txt rules. If the page is blocked, Google cannot see the noindex tag.
5. Compare the canonical with the URL. If it points elsewhere, confirm that is intentional.
6. Look for soft 404 signals: an error message, a very short page or a login wall.
7. Fix the blocker at the source, then request a recrawl in Google Search Console or wait for Google to revisit.
8. Run the Traffsite indexability checker again to confirm the page is eligible.

• SEO specialists use it for audits, migrations and traffic drop investigations.
• Developers use it for post release checks for a global noindex or robots rules.
• Content editors use it when a new article does not show up in search.
• Site owners use it when a page is live but invisible.
• Agencies use it for client checks before and after a handover.

You do not need server access to read the result, only to change the robots rules, template or canonical.