Skip to content

Robots.txt & AI Crawler Checker

by robots-check.quietworklabs.workers.dev · remote · no sign-in to list tools · v1.0.0 · published by dev.workers.quietworklabs.robots-check

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Changes · 2
CompareRegistry entry

3 tools: 2 change things and 1 only read.

Scanned Oct 21, 2026

No check matched anything in the tool definitions. No finding is not a safety guarantee. How the checks work.

claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

Tools

Open a tool for its reasons and full description. Descriptions are the publisher's own words, shown as they are.

3 tools

Changes things 2Claude Code: ask
  • ai_crawler_accessAI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Exten…No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

    Annotations the server declares

    • readOnlyHint: true
    • openWorldHint: true
  • robots_checkCrawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest …No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.

    Annotations the server declares

    • readOnlyHint: true
    • openWorldHint: true
Only reads 1Claude Code: allow
  • sitemap_checkFinds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod…No findings

    Why it is in this group

    • The description's first verb is "find": a word for reading.
    Description, as the server wrote it
    Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

    Annotations the server declares

    • readOnlyHint: true
    • openWorldHint: true

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Install

Pinned to version 1.0.0. Values in angle brackets are yours to fill in.

claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json
{
  "permissions": {
    "allow": [
      "mcp__robots-check__sitemap_check"
    ],
    "ask": [
      "mcp__robots-check__robots_check",
      "mcp__robots-check__ai_crawler_access"
    ],
    "deny": []
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. It assumes the server is installed as robots-check. Review it before you use it.