Robots.txt Diff Tracker
Fetch the live robots.txt from any domain and compare against a baseline. The diff highlights every rule added or removed, flags breaking changes, and warns when a change could materially affect crawl.
GEO Intelligence
A one-line robots.txt change can block AI crawlers you still want on the site, or expose paths you meant to hide.
An intern pushes a robots.txt change on Friday and traffic drops Monday. This tracker lets you save a known-good baseline, pull the current version from the domain, and see exactly what changed between them. Catches new disallows, user-agent blocks, and sitemap removals.
Domain
Compare
Baseline (paste your known-good version)
Current (fetched or pasted)
Frequently asked questions
What does this tool do?
Fetches your live robots.txt and compares it to a baseline you provide. Highlights every added or removed rule and flags breaking changes.
Why track robots.txt changes?
A one-line edit can accidentally block search engines or AI crawlers. "Disallow: /" wipes a site from the index. Tracking changes catches these before they damage traffic.
How often should I run the check?
Weekly is sensible. Daily during periods of active development. Save the baseline after any intentional change.
What is a "breaking" change?
A new Disallow rule affecting an important user-agent, or a removed Allow that previously carved out a path. The tool highlights these specifically.
Where is the baseline stored?
In your browser's local storage. Not synced across devices. Use "Save current as baseline" to persist.
Can I use this for monitoring multiple sites?
One site at a time per browser tab. For portfolio tracking across many clients, a scheduled crawler handles it.
Does the tool follow sitemap references?
It flags added or removed Sitemap: lines but doesn't fetch the sitemaps.
What breaks AI engine access specifically?
A new Disallow against GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot. The tool highlights AI-related UAs as critical.
How are wildcards handled in the diff?
Literal textual comparison. If one version has
Disallow: /admin/ and the other has Disallow: /admin the tool flags that even though they match the same URLs. Normalise first.Can two robots.txt files produce the same crawl behaviour but look different?
Yes. Rule order and user-agent grouping can shuffle without changing effective policy. The tool doesn't simulate crawler behaviour, only text diff.
Is there an API?
Not currently. The diff runs client-side only.
Free · no account
SEORCE for Chrome, Edge and Firefox
Audits the page you are on: headings, meta tags, schema, indexability, and whether AI crawlers can read it. Export the result as a PDF. Free, with no account.
It audits one page at a time and does not track AI answers. About the extension
Across the whole site
One check here. Every page in SEORCE.
This tool does one job. SEORCE Technical Suite crawls your whole site with fast and secure crawlers and ranks technical issues by impact. AutoFix applies fixes for meta tags, schema, canonical tags, image alt text, headings and internal links for orphan pages on your live site, through one script tag, and the list keeps growing.
3,000+ users · Free forever plan, no card needed