Skip to content

RobotsGate

by robotsgate.mike-tusa.workers.dev · remote · no sign-in to list tools · v0.3.1 · published by dev.workers.mike-tusa.robotsgate

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Changes · 1
CompareRegistry entry

4 tools: 1 change things and 3 only read.

Scanned Oct 21, 2026

1 check matched something in the tool definitions. No finding is not a safety guarantee. How the checks work.

claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

Findings

Matches of fixed checks, with the text they matched.

1 warn
  • Warngenerate_robots

    The server marks "generate_robots" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "generate_robots" says "generate": a word for a change.

    What this check looks for

Tools

Open a tool for its reasons and full description. Descriptions are the publisher's own words, shown as they are.

4 tools

Changes things 1Claude Code: ask
  • generate_robotsBuild a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bot…1 finding

    Why it is in this group

    • The name "generate_robots" says "generate": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    Findings

    • Warn

      The server marks "generate_robots" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "generate_robots" says "generate": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.

    Annotations the server declares

    • title: Generate robots.txt
    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
Only reads 3Claude Code: allow
  • check_siteFetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report…No findings

    Why it is in this group

    • The description's first verb is "fetch": a word for reading.
    Description, as the server wrote it
    Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.

    Annotations the server declares

    • title: Check a live site
    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: true
  • list_crawlersReturn RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects…No findings

    Why it is in this group

    • The name "list_crawlers" says "list": a word for reading.
    Description, as the server wrote it
    Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.

    Annotations the server declares

    • title: List AI crawlers
    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • validate_robotsValidate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule.No findings

    Why it is in this group

    • The name "validate_robots" says "validate": a word for reading.
    Description, as the server wrote it
    Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.

    Annotations the server declares

    • title: Validate robots.txt
    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Install

Pinned to version 0.3.1. Values in angle brackets are yours to fill in.

claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json
{
  "permissions": {
    "allow": [
      "mcp__robotsgate__list_crawlers",
      "mcp__robotsgate__validate_robots",
      "mcp__robotsgate__check_site"
    ],
    "ask": [
      "mcp__robotsgate__generate_robots"
    ],
    "deny": []
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. It assumes the server is installed as robotsgate. Review it before you use it.