Skip to content

Nonobench

by mauricekleine · remote · no sign-in to list tools · v1.1.0 · published by GitHub user mauricekleine

An open-source benchmark of how well LLMs solve nonogram puzzles, from 5x5 to 20x20.

Changes · 1
CompareRegistry entry

11 tools: 1 change things and 10 only read.

Scanned Oct 21, 2026

No check matched anything in the tool definitions. No finding is not a safety guarantee. How the checks work.

claude mcp add --transport http nonobench https://www.nonobench.com/mcp

Tools

Open a tool for its reasons and full description. Descriptions are the publisher's own words, shown as they are.

11 tools

Changes things 1Claude Code: ask
  • check_solutionCheck a grid against a puzzle's clues, using the same rule as the benchmark grader.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Check a grid against a puzzle's clues, using the same rule as the benchmark grader. Reports which rows and columns do not match.

    Annotations the server declares

    • title: Check solution
    • readOnlyHint: true
    • openWorldHint: false
Only reads 10Claude Code: allow
  • compare_modelsSide-by-side core overall and per-size accuracy, cost, latency and token results for model or family names.No findings

    Why it is in this group

    • The name "compare_models" says "compare": a word for reading.
    Description, as the server wrote it
    Side-by-side core overall and per-size accuracy, cost, latency and token results for model or family names.

    Annotations the server declares

    • title: Compare models
    • readOnlyHint: true
    • openWorldHint: false
  • get_leaderboardModels ranked by accuracy; defaults to all effort levels for compatibility.No findings

    Why it is in this group

    • The name "get_leaderboard" says "get": a word for reading.
    Description, as the server wrote it
    Models ranked by accuracy; defaults to all effort levels for compatibility.

    Annotations the server declares

    • title: Get leaderboard
    • readOnlyHint: true
    • openWorldHint: false
  • get_model_puzzlesWhich puzzles one model solved, missed, timed out on, or has not run.No findings

    Why it is in this group

    • The name "get_model_puzzles" says "get": a word for reading.
    Description, as the server wrote it
    Which puzzles one model solved, missed, timed out on, or has not run.

    Annotations the server declares

    • title: Get model puzzles
    • readOnlyHint: true
    • openWorldHint: false
  • get_model_resultsAccuracy, cost, latency and token use for one model, broken down by grid size.No findings

    Why it is in this group

    • The name "get_model_results" says "get": a word for reading.
    Description, as the server wrote it
    Accuracy, cost, latency and token use for one model, broken down by grid size.

    Annotations the server declares

    • title: Get model results
    • readOnlyHint: true
    • openWorldHint: false
  • get_puzzleOne puzzle, including the clue text models were prompted with.No findings

    Why it is in this group

    • The name "get_puzzle" says "get": a word for reading.
    Description, as the server wrote it
    One puzzle, including the clue text models were prompted with. The reference solution is only included on request; some puzzles have several valid solutions.

    Annotations the server declares

    • title: Get puzzle
    • readOnlyHint: true
    • openWorldHint: false
  • get_puzzle_resultsPer-model outcomes for one puzzle.No findings

    Why it is in this group

    • The name "get_puzzle_results" says "get": a word for reading.
    Description, as the server wrote it
    Per-model outcomes for one puzzle. Answers are omitted unless requested.

    Annotations the server declares

    • title: Get puzzle results
    • readOnlyHint: true
    • openWorldHint: false
  • list_familiesModel families, available efforts and best variants.No findings

    Why it is in this group

    • The name "list_families" says "list": a word for reading.
    Description, as the server wrote it
    Model families, available efforts and best variants.

    Annotations the server declares

    • title: List families
    • readOnlyHint: true
    • openWorldHint: false
  • list_providersProvider ids, names, families and variant counts.No findings

    Why it is in this group

    • The name "list_providers" says "list": a word for reading.
    Description, as the server wrote it
    Provider ids, names, families and variant counts.

    Annotations the server declares

    • title: List providers
    • readOnlyHint: true
    • openWorldHint: false
  • list_puzzlesThe benchmark puzzles with their ids and row/column clues.No findings

    Why it is in this group

    • The name "list_puzzles" says "list": a word for reading.
    Description, as the server wrote it
    The benchmark puzzles with their ids and row/column clues.

    Annotations the server declares

    • title: List puzzles
    • readOnlyHint: true
    • openWorldHint: false
  • list_runsIndividual benchmark runs (one model on one puzzle), optionally with the raw prompt and model output.No findings

    Why it is in this group

    • The name "list_runs" says "list": a word for reading.
    Description, as the server wrote it
    Individual benchmark runs (one model on one puzzle), optionally with the raw prompt and model output.

    Annotations the server declares

    • title: List runs
    • readOnlyHint: true
    • openWorldHint: false

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Install

Pinned to version 1.1.0. Values in angle brackets are yours to fill in.

claude mcp add --transport http nonobench https://www.nonobench.com/mcp

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json
{
  "permissions": {
    "allow": [
      "mcp__nonobench__get_leaderboard",
      "mcp__nonobench__list_providers",
      "mcp__nonobench__list_families",
      "mcp__nonobench__compare_models",
      "mcp__nonobench__get_model_results",
      "mcp__nonobench__list_puzzles",
      "mcp__nonobench__get_puzzle",
      "mcp__nonobench__get_puzzle_results",
      "mcp__nonobench__get_model_puzzles",
      "mcp__nonobench__list_runs"
    ],
    "ask": [
      "mcp__nonobench__check_solution"
    ],
    "deny": []
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. It assumes the server is installed as nonobench. Review it before you use it.