Skip to content

AI Crawler Access Check

by hamzawamiqhq · remote · no sign-in to list tools · v1.0.0 · published by GitHub user hamzawamiqhq

Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.

Changes · 1
CompareRegistry entry

1 tool: 1 change things.

Scanned Oct 21, 2026

No check matched anything in the tool definitions. No finding is not a safety guarantee. How the checks work.

claude mcp add --transport http ai-crawler-check https://check.citant.ai/mcp?src=registry

Tools

Open a tool for its reasons and full description. Descriptions are the publisher's own words, shown as they are.

1 tool

Changes things 1Claude Code: ask
  • check_ai_crawler_accessChecks whether AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others) can actually reach a website.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Checks whether AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others) can actually reach a website. Goes beyond robots.txt: fetches the site as each crawler and detects firewall/CDN blocks (Cloudflare, Akamai, Imperva), challenge pages, noindex and llms.txt. Returns a 0-100 AI Access Score, a per-crawler verdict and the top issues with fixes. Takes 15-60 seconds.

    Annotations the server declares

    • title: Check AI crawler access
    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: true

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Install

Pinned to version 1.0.0. Values in angle brackets are yours to fill in.

claude mcp add --transport http ai-crawler-check https://check.citant.ai/mcp?src=registry

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json
{
  "permissions": {
    "allow": [],
    "ask": [
      "mcp__ai-crawler-check__check_ai_crawler_access"
    ],
    "deny": []
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. It assumes the server is installed as ai-crawler-check. Review it before you use it.