Skip to content

Crawl Census

by taylorsmithgg · remote · no sign-in to list tools · v1.0.0 · published by GitHub user taylorsmithgg

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

Changes · 6
CompareRegistry entry

7 tools: 6 change things and 1 only read.

Scanned Oct 21, 2026

No check matched anything in the tool definitions. No finding is not a safety guarantee. How the checks work.

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp

Tools

Open a tool for its reasons and full description. Descriptions are the publisher's own words, shown as they are.

7 tools

Changes things 6Claude Code: ask
  • agent_profileWhat this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network e…No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    Description, as the server wrote it
    What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
  • census_factsEvery headline finding from the census as discrete, dated records rather than prose.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    Description, as the server wrote it
    Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
  • census_statsCorpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    Description, as the server wrote it
    Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
  • crawl_preflightDecide whether a crawler may fetch a list of domains before spending requests on them.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    Description, as the server wrote it
    Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
  • scan_siteRun a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt.No findings

    Why it is in this group

    • The description's first verb is "run": a word for a change.
    Description, as the server wrote it
    Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
  • submit_domainsQueue domains the census has not measured yet so a later crawl_preflight can answer them.No findings

    Why it is in this group

    • The name "submit_domains" says "submit": a word for a change.
    Description, as the server wrote it
    Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.
Only reads 1Claude Code: allow
  • site_reportReturn the most recent stored audit for a domain without triggering a new scan.No findings

    Why it is in this group

    • The description's first verb is "return": a word for reading.
    Description, as the server wrote it
    Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Install

Pinned to version 1.0.0. Values in angle brackets are yours to fill in.

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json
{
  "permissions": {
    "allow": [
      "mcp__crawl-census__site_report"
    ],
    "ask": [
      "mcp__crawl-census__scan_site",
      "mcp__crawl-census__census_stats",
      "mcp__crawl-census__crawl_preflight",
      "mcp__crawl-census__agent_profile",
      "mcp__crawl-census__census_facts",
      "mcp__crawl-census__submit_domains"
    ],
    "deny": []
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. It assumes the server is installed as crawl-census. Review it before you use it.