ML Referee

io.github.scarrico/mlrefereescarricostreamable-httpv1.1.0Scanned 42 minutes ago

Evaluate ML features beyond a baseline and test bounded feature repairs. Invite-only beta.

26 tools: 2 can delete or cancel things, 1 do something that can't be undone, 12 change things and 11 only read.

Reads private data · 1
Reads text others write · 0
Can send data out · 1
4 findings

Anlyon alternative

15 of these tools change something. Run those as governed actions.

Declare what an action changes, and Anlyon puts it under a shared limit, binds every approval to the exact request, and grades the outcome. It works on any HTTPS API you can describe as an action.

Anlyon does not run, proxy or govern this MCP server. A tool your own code calls directly is not routed through Anlyon and is not governed by it.

Import an OpenAPI spec →

Tools

Grouped by what each tool does, most consequential first. Open one for its reasons and full description; descriptions are the publisher's own words.

26 tools

Deletes or cancels 2Removes, cancels or revokes something.Claude Code: deny

delete_auditDelete retained inputs and results for an audit owned by this tenant.No findings
Why it is in this group
- The name "delete_audit" says "delete": a word for removing or ending something.
- The server marks it destructive (destructiveHint: true).
Description, as the server wrote it
Delete retained inputs and results for an audit owned by this tenant.
Annotations the server declares
- title: Delete audit
- readOnlyHint: false
- destructiveHint: true
- idempotentHint: true
- openWorldHint: false
delete_uploadDelete a tenant-private upload that is no longer needed.No findings
Why it is in this group
- The name "delete_upload" says "delete": a word for removing or ending something.
- The server marks it destructive (destructiveHint: true).
Description, as the server wrote it
Delete a tenant-private upload that is no longer needed.
Annotations the server declares
- title: Delete upload
- readOnlyHint: false
- destructiveHint: true
- idempotentHint: true
- openWorldHint: false

Can't be undone 1Sends, pays, publishes or merges: once done, it cannot be taken back.Claude Code: ask

redeem_beta_inviteRedeem one operator-issued invite for free beta audit credits before January 1, 2027.Can send data outNo findings
Why it is in this group
- The name "redeem_beta_invite" says "invite": a word for sending, and what is sent cannot be taken back.
What it can reach
- outboundThe name "redeem_beta_invite" says "invite": it sends or publishes something to other people.
Description, as the server wrote it
Redeem one operator-issued invite for free beta audit credits before January 1, 2027.
Annotations the server declares
- title: Redeem beta invite
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false

Changes things 12Creates or changes something that can be changed back.Claude Code: ask

get_creditsRead beta or paid audit-credit availability and usage.1 finding
Why it is in this group
- The name "get_credits" says "get": a word for reading.
- The server marks it as not read-only (readOnlyHint: false).
Findings
- Warn
  The server marks "get_credits" as not read-only (readOnlyHint: false), and its words say it reads.
  Matched text
  annotations: {"readOnlyHint":false,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "get_credits" says "get": a word for reading.
  What this check looks for
Description, as the server wrote it
Read beta or paid audit-credit availability and usage.
Annotations the server declares
- title: Read audit credits
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
preflight_datasetValidate a tenant-private upload and its baseline evidence/row identity without spending a credit.1 finding
Why it is in this group
- The description's first verb is "validate": a word for reading.
- The server marks it as not read-only (readOnlyHint: false).
Findings
- Warn
  The server marks "preflight_dataset" as not read-only (readOnlyHint: false), and its words say it reads.
  Matched text
  annotations: {"readOnlyHint":false,"destructiveHint":false,"idempotentHint":false,"openWorldHint":false}; The description's first verb is "validate": a word for reading.
  What this check looks for
Description, as the server wrote it
Validate a tenant-private upload and its baseline evidence/row identity without spending a credit.
Annotations the server declares
- title: Preflight feature audit
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: false
- openWorldHint: false
preflight_repairValidate a tenant-private feature-repair matrix without spending a credit.1 finding
Why it is in this group
- The description's first verb is "validate": a word for reading.
- The server marks it as not read-only (readOnlyHint: false).
Findings
- Warn
  The server marks "preflight_repair" as not read-only (readOnlyHint: false), and its words say it reads.
  Matched text
  annotations: {"readOnlyHint":false,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The description's first verb is "validate": a word for reading.
  What this check looks for
Description, as the server wrote it
Validate a tenant-private feature-repair matrix without spending a credit.
Annotations the server declares
- title: Preflight feature repair
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
quote_auditCreate a contract-bound quote for one audit credit; baseline mode and evidence cannot be substituted.No findings
Why it is in this group
- The description's first verb is "create": a word for a change.
Description, as the server wrote it
Create a contract-bound quote for one audit credit; baseline mode and evidence cannot be substituted.
Annotations the server declares
- title: Quote feature audit
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
quote_campaignPreflight all private feature shards, time a pilot, and quote credits and ETA for up to 1,000 candidates in one corrected search family.No findings
Why it is in this group
- Nothing in its name or description says what it does, so it is counted as a write.
Description, as the server wrote it
Preflight all private feature shards, time a pilot, and quote credits and ETA for up to 1,000 candidates in one corrected search family.
Annotations the server declares
- title: Quote feature campaign
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
quote_repairPreflight and quote label-based feature repair; large jobs return a measured ETA and credit requirement.No findings
Why it is in this group
- Nothing in its name or description says what it does, so it is counted as a write.
Description, as the server wrote it
Preflight and quote label-based feature repair; large jobs return a measured ETA and credit requirement.
Annotations the server declares
- title: Quote feature repair
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
record_audit_followupRecord whether a completed audit changed a decision or was confirmed prospectively.No findings
Why it is in this group
- Nothing in its name or description says what it does, so it is counted as a write.
Description, as the server wrote it
Record whether a completed audit changed a decision or was confirmed prospectively.
Annotations the server declares
- title: Record audit follow-up
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
request_beta_accessRequest operator-reviewed beta access without purchasing or granting credits.No findings
Why it is in this group
- The name "request_beta_access" says "request": a word for a change.
Description, as the server wrote it
Request operator-reviewed beta access without purchasing or granting credits.
Annotations the server declares
- title: Request beta access
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
submit_auditAccept a quote and atomically reserve its disclosed audit credits.No findings
Why it is in this group
- The name "submit_audit" says "submit": a word for a change.
Description, as the server wrote it
Accept a quote and atomically reserve its disclosed audit credits.
Annotations the server declares
- title: Submit audit
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
upload_campaign_shardEncrypt one of up to 64 private feature shards for a large campaign; retain for at least 4,200 seconds.No findings
Why it is in this group
- The name "upload_campaign_shard" says "upload": a word for a change.
Description, as the server wrote it
Encrypt one of up to 64 private feature shards for a large campaign; retain for at least 4,200 seconds.
Annotations the server declares
- title: Upload campaign shard
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
upload_private_datasetEncrypt a de-identified CSV/JSON payload for a short tenant-private retention window.No findings
Why it is in this group
- The name "upload_private_dataset" says "upload": a word for a change.
Description, as the server wrote it
Encrypt a de-identified CSV/JSON payload for a short tenant-private retention window.
Annotations the server declares
- title: Upload private dataset
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
upload_repair_matrixEncrypt one complete de-identified repair matrix of up to 100 MB and 1,000 diagnostic features.No findings
Why it is in this group
- The name "upload_repair_matrix" says "upload": a word for a change.
Description, as the server wrote it
Encrypt one complete de-identified repair matrix of up to 100 MB and 1,000 diagnostic features.
Annotations the server declares
- title: Upload repair matrix
- readOnlyHint: false
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false

Only reads 11Looks things up and changes nothing.Show 11 read-only tools

describe_refereeDiscover capabilities, privacy, and the approved public catalog.No findings
Why it is in this group
- The name "describe_referee" says "describe": a word for reading.
Description, as the server wrote it
Discover capabilities, privacy, and the approved public catalog.
Annotations the server declares
- title: Describe Referee
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
get_audit_receiptRead a tenant-private report fingerprint retained after report expiry or deletion.No findings
Why it is in this group
- The name "get_audit_receipt" says "get": a word for reading.
Description, as the server wrote it
Read a tenant-private report fingerprint retained after report expiry or deletion. Older reports may have no record. This does not retrieve report contents.
Annotations the server declares
- title: Get report receipt
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
get_audit_resultRead a completed tenant-private result and signed output manifest.No findings
Why it is in this group
- The name "get_audit_result" says "get": a word for reading.
Description, as the server wrote it
Read a completed tenant-private result and signed output manifest.
Annotations the server declares
- title: Get audit result
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
get_audit_statusRead the state of an audit owned by the authenticated tenant.No findings
Why it is in this group
- The name "get_audit_status" says "get": a word for reading.
Description, as the server wrote it
Read the state of an audit owned by the authenticated tenant.
Annotations the server declares
- title: Get audit status
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
get_beta_request_statusRead the authenticated tenant's beta-access request status.No findings
Why it is in this group
- The name "get_beta_request_status" says "get": a word for reading.
Description, as the server wrote it
Read the authenticated tenant's beta-access request status.
Annotations the server declares
- title: Get beta request status
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
get_input_schemaGet neutral column roles and validation defaults for a vertical.No findings
Why it is in this group
- The name "get_input_schema" says "get": a word for reading.
Description, as the server wrote it
Get neutral column roles and validation defaults for a vertical.
Annotations the server declares
- title: Get feature-audit schema
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
get_referee_readinessReport connection, credential, audit-credit, and submission readiness with the next action.Reads private data1 finding
Why it is in this group
- The name "get_referee_readiness" says "get": a word for reading.
What it can reach
- private-dataThe description's first sentence says "credential": it reads secrets.
Findings
- Info
  The description mentions "credential".
  Matched text
  description: "Report connection, credential, audit-credit, and submission readiness..."
  What this check looks for
Description, as the server wrote it
Report connection, credential, audit-credit, and submission readiness with the next action.
Annotations the server declares
- title: Check Referee readiness
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
get_referee_usageRead aggregate, privacy-preserving Referee activity and outcome evidence for the selected audience.No findings
Why it is in this group
- The name "get_referee_usage" says "get": a word for reading.
Description, as the server wrote it
Read aggregate, privacy-preserving Referee activity and outcome evidence for the selected audience.
Annotations the server declares
- title: Read aggregate usage
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
get_repair_schemaGet the feature-only contract for diagnosing degradation and testing bounded repairs.No findings
Why it is in this group
- The name "get_repair_schema" says "get": a word for reading.
Description, as the server wrote it
Get the feature-only contract for diagnosing degradation and testing bounded repairs.
Annotations the server declares
- title: Get feature-repair schema
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
list_verticalsList the three supported evaluation profiles and their discovery tags.No findings
Why it is in this group
- The name "list_verticals" says "list": a word for reading.
Description, as the server wrote it
List the three supported evaluation profiles and their discovery tags.
Annotations the server declares
- title: List evaluation profiles
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false
verify_audit_reportCompare a locally computed SHA-256 with the retained tenant-private receipt.No findings
Why it is in this group
- The description's first verb is "compare": a word for reading.
Description, as the server wrote it
Compare a locally computed SHA-256 with the retained tenant-private receipt. Use the exact downloaded result JSON bytes (not the API envelope), or UTF-8 result.report text with representation report_text. Never copy the expected hash as the candidate: compute it from the supplied artifact. A digest match does not validate a summary, factual correctness, or absence of harm. No report is uploaded.
Annotations the server declares
- title: Verify report fingerprint
- readOnlyHint: true
- destructiveHint: false
- idempotentHint: true
- openWorldHint: false

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Exfil surface

Which tools read private data, which read text someone else can write, and which can send data out. One of each is a path a prompt injection can use.

No private-data, untrusted-input, outbound path in this server alone: no tool is tagged untrusted-input.

Reads private data1: Reads files, mail, messages, databases, secrets, calendars or repositories.; get_referee_readiness
Reads text others write0: Brings in text someone else can write: web pages, inbound email, messages, issues and comments.
Can send data out1: Can carry data to people or systems: it sends, posts or publishes, or takes a recipient, channel or URL.; redeem_beta_invite

Other servers in the same agent add their own tools to these paths. Audit your whole setup to see them together.

Findings

Matches of fixed checks, with the text they matched.

3 warn1 info

Warnpreflight_dataset
The server marks "preflight_dataset" as not read-only (readOnlyHint: false), and its words say it reads.
Matched text
annotations: {"readOnlyHint":false,"destructiveHint":false,"idempotentHint":false,"openWorldHint":false}; The description's first verb is "validate": a word for reading.
What this check looks for
Warnpreflight_repair
The server marks "preflight_repair" as not read-only (readOnlyHint: false), and its words say it reads.
Matched text
annotations: {"readOnlyHint":false,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The description's first verb is "validate": a word for reading.
What this check looks for
Warnget_credits
The server marks "get_credits" as not read-only (readOnlyHint: false), and its words say it reads.
Matched text
annotations: {"readOnlyHint":false,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "get_credits" says "get": a word for reading.
What this check looks for
Infoget_referee_readiness
The description mentions "credential".
Matched text
description: "Report connection, credential, audit-credit, and submission readiness..."
What this check looks for

History

Changes to its tool definitions and to its remote endpoint, newest first.

No change seen yet. Its tools are rescanned every day, and a remote endpoint is looked at once a week.

All changes to this server

Install

Pinned to version 1.1.0. Values in angle brackets are yours to fill in.

claude mcp add --transport http mlreferee https://api.mlreferee.com/mcp

claude mcp add --transport http mlreferee https://api.mlreferee.com/mcp

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change or a way out, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json

{
  "permissions": {
    "allow": [
      "mcp__mlreferee__describe_referee",
      "mcp__mlreferee__list_verticals",
      "mcp__mlreferee__get_referee_readiness",
      "mcp__mlreferee__get_referee_usage",
      "mcp__mlreferee__get_input_schema",
      "mcp__mlreferee__get_repair_schema",
      "mcp__mlreferee__get_beta_request_status",
      "mcp__mlreferee__get_audit_status",
      "mcp__mlreferee__get_audit_result",
      "mcp__mlreferee__get_audit_receipt",
      "mcp__mlreferee__verify_audit_report"
    ],
    "ask": [
      "mcp__mlreferee__upload_private_dataset",
      "mcp__mlreferee__upload_campaign_shard",
      "mcp__mlreferee__upload_repair_matrix",
      "mcp__mlreferee__preflight_dataset",
      "mcp__mlreferee__preflight_repair",
      "mcp__mlreferee__quote_audit",
      "mcp__mlreferee__quote_repair",
      "mcp__mlreferee__quote_campaign",
      "mcp__mlreferee__get_credits",
      "mcp__mlreferee__redeem_beta_invite",
      "mcp__mlreferee__request_beta_access",
      "mcp__mlreferee__submit_audit",
      "mcp__mlreferee__record_audit_followup"
    ],
    "deny": [
      "mcp__mlreferee__delete_audit",
      "mcp__mlreferee__delete_upload"
    ]
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. A read that can send data out (it takes a URL, say) asks first too. It assumes the server is installed as mlreferee. Review it before you use it.

Capability tags behind 2 rules

mcp__mlreferee__get_referee_readiness private-data
mcp__mlreferee__redeem_beta_invite outbound

Keep the decision: fail CI when the tools change, follow every change, or show a badge.

Pin in CI

Fail a build when the tools you reviewed are not the tools the server serves today.

curl -s "https://anlyon.dev/api/v1/servers/io.github.scarrico/mlreferee/verify?hash=6ecaecbc0217009db7940cb0a945e26cd278f90f8eb0ff554dba7612a8c51d19" | jq -e '.changed == false'

Pinned hash 6ecaecbc0217, from Oct 27, 2026, 00:18 UTC.

Watch

Get told when its tools change: a new tool, a new description, a new class.

https://anlyon.dev/servers/io.github.scarrico/mlreferee/changes.xml

An Atom feed. Works in any feed reader or Slack's RSS app.

Badge

[![anlyon.dev](https://anlyon.dev/badge/io.github.scarrico/mlreferee.svg)](https://anlyon.dev/servers/io.github.scarrico/mlreferee)

Source: MCP Registry. Synced Oct 27, 2026, 00:17 UTC.