Spicrawl

Cancel a batch job: items not yet started are cancelled and will never run; items already finished keep their results; items already running finish first, so the job comes back `cancelling` and moves to `cancelled` when they drain. Cannot be undone — resubmit to run the URLs again. Returns the job object.

Permanently delete a session and its stored state, including the record itself (prefer spicrawl_session_release if you only want to stop using it). Idempotent: deleting an already-deleted session succeeds. Refused while a scrape holds the session's lease unless `force` is true.

End a session when you are done with it: frees its exit IP and browser state; it can no longer be used with spicrawl_scrape. Its stored cookies and storage are purged and cannot be recovered; start a new session and log in again to continue. The record stays visible (status `released`) so its history can still be inspected. Refused while a scrape holds the session's lease unless `force` is true. Returns the session object.

Billing-grade usage for your org over a date window, from the daily rollup (usage_daily), grouped by day, project, engine, feature or key. group_by=key returns per API key (key_id, name, prefix, project_id) counters per metric per day, from usage_key_daily; keyless (dashboard) usage is not in it, and a project-scoped key sees only its own project's keys. Use it to answer 'how many requests/credits did I use, and where'. The window defaults to the last 30 days including today; `to` is EXCLUSIVE (a single day 2026-08-01 is from=2026-08-01, to=2026-08-02) and the window may not exceed 400 days. The response carries stale_seconds (age of the newest rollup row; null when the window is empty). Requires the key to have the `read` scope.

Append URLs to a batch job that was submitted with `open: true`. Same item vocabulary as `spicrawl_batch_submit` (EITHER `urls` or `items`, not both, plus shared settings for these new items). Fails with CONFLICT if the job is closed or finished. Not idempotent: calling twice adds the URLs twice. Returns `{ job, items_added, items_dispatched }`.

Stop an open batch job accepting items, so it completes once its queued work drains. Idempotent (closing a closed job succeeds). Returns the job object.

Re-queue every FAILED item of a batch job so it runs again (succeeded items are untouched; a job with no failures is returned unchanged). Returns `{ job, items_reset, items_dispatched }`.

Submit many URLs to be scraped as one asynchronous job, with shared settings applied to every URL. Give EITHER `urls` or `items`, not both (at least one URL, unless `open` is true). Returns the job object (id, status, estimated_credits, progress). Poll with `spicrawl_batch_status`, read with `spicrawl_batch_results` or `spicrawl_batch_task_content`. Use this instead of many `spicrawl_scrape` calls when you have tens or thousands of URLs. Set `open: true` to keep the job accepting more URLs via `spicrawl_batch_add_items`; an open job never finishes until you call `spicrawl_batch_close`.

The docs' llms.txt: every Spicrawl documentation page with its title, a one-line description and its Markdown URL. Use it to find the right page when a search does not, or to get an overview of what is documented; read a page with spicrawl_docs_read. No parameters.

The full log record of one past API call by its request id. Every response carries its id in the X-Request-Id header (and error bodies carry it too), so this is the tool for debugging a failed or slow scrape: it shows status, error_code/error_detail, attempts, the engine and proxy (tier, country, provider, sticky) actually used, whether and how the target blocked it (vendor/signal/rule), per-stage latency spans and credits charged. Returns not_found both for unknown ids and for ones older than your retention window, and for another project's request unless your key has the `read` scope.

Recent API calls in your key's project, newest first, each with project_id, status, engine, http_status, error_code/error_detail, attempts, URL, proxy used, latency spans (queue/engine/upstream/total ms), block detection and credits. Use it to find failing calls (only_errors=true) or to spot patterns across a target. all_projects=true lists every project in your organization instead (needs a key with the `read` scope; 403 ERR::AUTH::INSUFFICIENT_SCOPE otherwise). Records only exist within the retention window reported in page.retention_hours / page.retained_since; an empty page means nothing in that window. Paginate by passing page.next_before and page.next_before_id back as before and before_id while page.has_more is true, keeping all_projects the same. Not billing-grade: use spicrawl_usage for totals.

Retrieve a web page and return its content as markdown (default), text, HTML, a JSON envelope, or a printed PDF. Handles the fetch vs. browser decision, charset decoding, and PDF-to-text for you. Optionally extract structured data with CSS selectors (`extract`) or `autoparse`, return discovered `links`, take a `screenshot`, or scope the DOM with `include_tags`/`exclude_tags`. Each successful call is billed in credits (a failure costs 0). Results are cached by default, and a cache hit is billed like the fetch that stored it (the cache saves time, not credits); set `cache=false` for time-sensitive pages. With the defaults it only reads the page. A `method` other than GET, HEAD or OPTIONS, and the `actions` that click, fill or run scripts on the page, can change state on the target site: set them only when the user asks for that. For markdown, text and HTML it returns `{ content }`. With `format: "json"`, or when `extract`, `autoparse`, `links`, `screenshot` or `network_capture` is set, it returns the JSON envelope: `content`, the site's `status`, `credits`, `engine` (the rendering tier: `fetch`, `js` or `browser`), `warnings` and `data` for extraction. Screenshots come back as image content blocks; each `screenshots` entry in the JSON keeps its metadata and says which block holds it. A PDF comes back as a resource content block, with its size, the rendering tier and the credits in the JSON.

Create a session: a persistent browser identity (cookie jar, localStorage, a fixed rendering tier and, by default, a sticky exit IP) that survives across scrapes. Use it for multi-step flows — log in, then fetch pages behind the login — by passing the returned `id` as `session_id` to every `spicrawl_scrape` call that should share the state. The rendering tier is fixed for the session's lifetime and every scrape on the session runs on it. Leave `engine` out and one is chosen for you. Release the session with `spicrawl_session_release` when done. Returns the session object (id, engine, status, sticky_key, proxy, expires_at).

Drift report for ONE day: re-derives usage from that day's request log and compares it with the billed daily rollup, so you can see whether the rollup missed or double-counted anything. Defaults to yesterday (UTC); today is still accumulating and would always show drift. Requires the `read` scope.

A compact current-period summary of your org's usage (totals as of now, from the daily rollup). No parameters. Use for a quick 'how much have I used so far'; use spicrawl_usage for a windowed breakdown. Requires the `read` scope.

List this API key's project's batch jobs, newest first, with status and progress. Returns `{ batches: [...], next_cursor? }`; pass `next_cursor` back as `cursor` for the next page.

Return a page of FINISHED items of a batch job (one record per item: seq, url, external_id, status, error, and content, or a `result_url` API path (GET /v1/batch/{id}/tasks/{seq}/content, same key) for large bodies). Unfinished items are not listed yet. Returns `{ results: [...], next_cursor? }`; pass `next_cursor` back as `cursor` for the next page. Results expire after the job's retention window (410 Gone).

Return one batch job with its status and progress (total/completed/succeeded/failed/remaining, credits charged). Poll this after `spicrawl_batch_submit`; once status is `completed`, `failed` or `cancelled`, read `spicrawl_batch_results`.

Return the full payload one item of a batch job produced, by its `seq` (items are numbered from 0 in submission order). Cheaper than paging results when you need one URL's output. A CONFLICT error means either the item has not finished yet (poll `spicrawl_batch_status` and ask again) or it finished as failed (it will never produce content; the message says which). NOT_FOUND means no such seq.

Fetches one Spicrawl documentation page as Markdown. `path` may be a page path (`guides/anti-bot`), a docs URL from spicrawl_docs_search (url or md_url), or either with a `#anchor` (the whole page is returned; the anchor names the section to look at). Only pages on the Spicrawl docs (https://docs.spicrawl.com/docs) can be read. Pages over 60000 characters are truncated, and the text says so. Don't know the path? Use spicrawl_docs_search, or spicrawl_docs_index (llms.txt) for the list of every page.

Full-text search over the Spicrawl documentation (guides, API reference, errors, limits). Use it to look up a parameter name, value or combination for a spicrawl_* tool, or to explain an error a spicrawl_* tool returned: search the error code itself (e.g. `ERR::UPSTREAM::CHALLENGE`) and the result also carries a direct link to that code's entry on the errors page. Returns the best-matching pages, each with its title, the matching sections (heading, a short snippet, URL) and md_url; then call spicrawl_docs_read with a page's md_url or url to read it in full.

Return the full stored state of a LIVE session — cookies, localStorage and other credentials — plus its fingerprint and domain scores. The `session_context` object can be passed to spicrawl_session_create to clone the session. The result contains secrets (login cookies): do not echo it to the user or logs unless asked.

Return one session's metadata: status, `engine` (its rendering tier), proxy/exit, usage count, expiry, lease holder, and a summary of its stored context. Never returns the credentials themselves (use spicrawl_session_context for that).

List this API key's project's sessions, newest first. Returns `{ sessions: [...], next_cursor? }`; pass `next_cursor` back as `cursor` for the next page. Never includes the stored credentials.