Skip to content

OpenMark

by openmark.ai · remote · no sign-in to list tools · v1.0.0 · published by ai.openmark

Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.

Changes · 3
CompareRegistry entry

9 tools: 3 change things and 6 only read.

Scanned Oct 21, 2026

No check matched anything in the tool definitions. No finding is not a safety guarantee. How the checks work.

claude mcp add --transport http openmark https://openmark.ai/mcp --header 'Authorization: <your-authorization>'

Tools

Open a tool for its reasons and full description. Descriptions are the publisher's own words, shown as they are.

9 tools

Changes things 3Claude Code: ask
  • estimate_costCredits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    Description, as the server wrote it
    Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.
  • run_benchmarkSave a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls.No findings

    Why it is in this group

    • The name "run_benchmark" says "run": a word for a change.
    Description, as the server wrote it
    Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast).
  • watch_taskWatch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    Description, as the server wrote it
    Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.
Only reads 6Claude Code: allow
  • get_accountThe user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock.No findings

    Why it is in this group

    • The name "get_account" says "get": a word for reading.
    Description, as the server wrote it
    The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits.
  • get_resultsStatus of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the ch…No findings

    Why it is in this group

    • The name "get_results" says "get": a word for reading.
    Description, as the server wrote it
    Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.
  • get_task_formatHow to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips.No findings

    Why it is in this group

    • The name "get_task_format" says "get": a word for reading.
    Description, as the server wrote it
    How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name.
  • list_modelsCurrent OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list.No findings

    Why it is in this group

    • The name "list_models" says "list": a word for reading.
    Description, as the server wrote it
    Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.
  • prepare_attachmentsGet an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command.No findings

    Why it is in this group

    • The description's first verb is "get": a word for reading.
    Description, as the server wrote it
    Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark.
  • validate_taskCheck a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan.No findings

    Why it is in this group

    • The name "validate_task" says "validate": a word for reading.
    Description, as the server wrote it
    Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nothing. Returns the problems to fix and warnings worth a look.

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Install

Pinned to version 1.0.0. Values in angle brackets are yours to fill in.

claude mcp add --transport http openmark https://openmark.ai/mcp --header 'Authorization: <your-authorization>'

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json
{
  "permissions": {
    "allow": [
      "mcp__openmark__get_task_format",
      "mcp__openmark__validate_task",
      "mcp__openmark__list_models",
      "mcp__openmark__prepare_attachments",
      "mcp__openmark__get_results",
      "mcp__openmark__get_account"
    ],
    "ask": [
      "mcp__openmark__estimate_cost",
      "mcp__openmark__run_benchmark",
      "mcp__openmark__watch_task"
    ],
    "deny": []
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. It assumes the server is installed as openmark. Review it before you use it.