ketch

Ketch

Route every live-source question to one of ketch's five research surfaces — search, code, docs, scrape, crawl — over the transport the operator gave you, with a token budget on every fetch and a source URL on every claim. ketch is one stateless binary — call, result, exit — with web search, OSS code grep, curated library docs, and page/site extraction together, so a complete research pipeline needs no other tool and no daemon.

Transport: stateless CLI by default, MCP when the operator wired it

The CLI is ketch's identity: call → result → exit, --json on every call, exit codes as control flow, zero daemon. That is the default transport and the zero-infrastructure path. The MCP server is a supported alternative for operators who want it — never a prerequisite.

Decide once per session, before the first call:

1. which ketch succeeds → the CLI is your transport: --json on every call, exit codes as control flow.
2. Also check for ketch's tools in your tool list — the MCP tools search, code, docs, scrape, crawl and tag from a server named ketch (in Claude Code: mcp__ketch__search, …), or in pi the package's ketch_search, ketch_code, ketch_docs, ketch_scrape, ketch_crawl and ketch_tag. Present → the operator wired them up on purpose, and using them for research calls is correct and good: structured output, per-URL errors, the same options and error prefixes as MCP. Do not shell out around tools the operator set up. (pi's tools run the CLI for you, so the page-cache tradeoff in step 3 does not apply to them.)
3. Both live → either transport serves research calls, but know the tradeoff: a running MCP server holds the single-process page-cache lock, so concurrent CLI scrapes silently run cache-disabled.
4. Neither CLI nor MCP tools → ketch is not installed. Offer brew install ketch, npm i -g ketch-cli or go install github.com/1broseidon/ketch@latest — an operator action: propose, wait for confirmation.

The rule: use the transport the operator gave you — when both are live, either is fine for research calls, and operator actions are always CLI.

tag is the one exception to the operator-action rule below: it changes local state but the agent is both writer and reader, so it is published over MCP as well as the CLI.

Config discovery is CLI regardless of transport: ketch config prints effective settings and available backends as JSON; there is no config tool over MCP. Operator actions — config set, cache, browser install, crawl --background/status/stop, doctor — are deliberately not in MCP. They are always CLI.

Glossary

Use only these terms in ketch output.

Term	Meaning
surface	One of the five research operations: search, code, docs, scrape, crawl
tag	A label filed across surfaces: search hits, code hits, docs chunks, scraped pages, crawled pages. Not a research surface — it answers "what did I already find?", not "what is out there?"
transport	How a surface is called: the CLI binary (default), the optional MCP tools, or the pi package's ketch_* tools (which run the CLI)
backend	The provider behind a surface: auto (default; a fallback chain, not a provider)/brave/ddg/searxng/exa/firecrawl/keenable/tavily/tinyfish/parallel/serpbase/degoog/serply/youcom (search), grepapp/sourcegraph/github (code), context7 (docs)
operator action	A system-managing or diagnostic command — config set, cache, browser install, background crawls, doctor — CLI-only by design
error prefix	The stable class on every ketch error: CLI exit codes 2–6, mirrored as the bracketed prefix opening every MCP tool error — [validation], [not_found], [upstream], [precondition], [cancelled]
fan-out	How many queries are searched and URLs scraped under one plan
token budget	The per-call output bound: max_chars/trim on scrapes, tokens on docs, limit/--minimal on lists
probe	One cheap read-only call that tests whether a surface is configured and reachable

How to use this skill

• Default: answer one question with one or two routed calls. Use the surface routing table, token budgets, and error control flow below.
• ketch research <question>: deep multi-source research — search fan-out → scrape top hits → optional code/docs corroboration → synthesized, cited answer. Read references/verbs/ketch-research.md.
• ketch setup: configure backends with the operator — probe current state, propose exact commands, mutate only on confirmation. Read references/verbs/setup.md. Enter this verb whenever any call returns [precondition] / exit 5.

One question = one plan. Escalate a default run into ketch research when the first search shows the answer is contested, multi-part, or needs corroboration.

Non-negotiable disciplines

1. Use the transport the operator gave you. The CLI is the default; MCP tools in your list mean the operator opted in — use them for research rather than shelling out around them. When both are live, either serves research calls (a running MCP server holds the page-cache lock, so concurrent CLI scrapes run uncached); operator actions — config, cache, browser, background crawls, doctor — are always CLI.
2. Bound every fetch. max_chars 4000–8000 plus trim on any scrape of a page you have not seen — an unguarded page can cost ~25k tokens. Skipping the cap requires a stated one-line reason ("known ~200-word page").
3. Cite every claim. A research synthesis without source URLs is not a deliverable.
4. Error prefixes are control flow. Classify before reacting. Never retry [validation] or [not_found] unchanged.
5. Propose, then mutate. config set, browser install, docker runs, installs — only after the operator confirms the exact command. Never touch a value that is already configured and working.
6. The binary outranks this file. ketch config and --help are ground truth; where they disagree with a table here, trust the binary and flag the skill as drifted.

Gold decision trace

Request: "ketch research — do people actually use Go's iter.Seq in real projects, and what are the gotchas?"

Transport: operator wired mcp__ketch__* into this session → honor it; research calls go over MCP.
Plan: 2 queries · scrape top 3 · max_chars 6000 + trim · ≤8 calls

search {query: "Go iter.Seq real-world experience gotchas", limit: 5}
  → "[upstream] ddg rate limited" → an explicit backend failed; rotate to another
    usable provider from available_backends, retry once:
    search {query: ..., backend: "brave", limit: 5} → ok
search {query: "Go range-over-func adoption production", backend: "brave", limit: 5}
  → 10 results, 8 unique hosts → picked 3: official blog post, one experience
    report, one issue thread (primary sources over aggregators)

scrape {urls: [u1, u2, u3], max_chars: 6000, trim: true}
  → isError=false; checked results[] one by one: u1, u2 ok;
    u3.error = "[upstream] … 503" → dropped, will be named in synthesis

code {query: "iter.Seq", lang: "go", limit: 3}      # corroborate real usage
  → 3 repos with file/line URLs

Synthesis: five claims, each cited to its URL; u3 listed as unretrieved;
one conflict between u1 and u2 stated and attributed, not averaged.
Budget: 5 of 8 calls (the rate-limited attempt counts).

Surface routing

First match wins:

The question needs	Surface	Not
Current web pages, opinions, news, comparisons	search	docs — that is curated library docs only
How real projects call an API	code	search — blogs talk about code; code greps public OSS repos via grep.app
A library's own documentation, version-aware	docs	scrape of the docs site — docs is already extracted and token-budgeted
The content of a URL you already hold	scrape	search — never re-find a known URL
Many pages from one site	crawl	looped scrape — crawl dedupes, bounds, and streams
Anything you already found earlier in this project	tag (operation: show)	search again — you already paid for these once

In reverse: search finds URLs; scrape reads them; crawl reads a site; code reads public source; docs reads library docs. search with scrape: true fuses the first two when you will want full content from every hit — budget it like a scrape.

Keeping a working set with tag

On work that spans more than one session or more than a handful of sources, pass --tag <name> (CLI) or tag (MCP) on the calls you will want again, naming the project or scope of work — remote-access, not results-3.

It works on every surface, and one tag holds them all: search, code, docs, scrape, crawl. So a project tag ends up holding the vendor's documentation, the code that calls it, and the write-up that explained the undocumented flag, in one list. Later, tag show returns that as an llms.txt-shaped index — titles, URLs, descriptions — limited to the newest 50, and you re-read one source instead of re-running the searches that found them. Tag the sources you actually used, not every hit.

Use --limit N / MCP limit for a smaller view; 0 explicitly requests all. entries counts the whole tag and shown counts returned pages. Read the index before searching again, but do not request all of a large tag by default.

The index is durable and outlives the cached page bodies, so it still answers days later. cached: false means no fresh body was confirmed, not a dead link. When cache_status is unavailable, the page cache could not be checked; bookmarks still work and sources can still be fetched. Entries from code, docs and unscraped search hits start that way by nature, since those calls return snippets rather than fetched pages; scrape the URL when you want the whole thing. Read the index first, then fetch only what you need: assembling the whole tag defeats the point.

Check bookmark diagnostics separately from research success: CLI warnings go to stderr (structured under --json), and MCP returns warnings. A failed write does not discard the useful research result. Durable bookmarks use a separate file; test labs must set KETCH_TAGS_PATH as well as isolating the page cache.

Token budgets

Call	Bound with	Measured cost
search, limit 5	limit	~1.4 KB
code, limit 3	limit	~0.7 KB
docs, default budget	tokens (default 4000)	~3.3 KB
scrape, unknown page	max_chars 4000–8000 + trim	unguarded: up to ~100 KB (~25k tokens)
crawl (MCP)	max_pages + per-page max_chars	30 pages default, 100 cap, 3-min wall clock
tag show / MCP tag show	limit (default 50; 0 all)	bounded source metadata; no page bodies
Any CLI list	--minimal	roughly halves output

Error control flow

Exit (CLI)	Prefix (MCP)	Meaning	Do
2	[validation]	Bad input	Fix the call; retrying unchanged can never succeed
3	[not_found]	Nothing matched	Change the query or selector; not an outage
4	[upstream]	Backend or network failure	Explicit backend: rotate to another provider (available_backends in ketch config) or retry once. auto already fell through every usable provider: retry once, then report the outage
5	[precondition]	Operator config missing	Stop researching; enter ketch setup
6	[cancelled]	Cancelled or timed out	Rerun with smaller scope

Situations → class: unknown backend, regexp on github → [validation]. Selector matched nothing, a repo the code backend does not have → [not_found] (for repo, change the backend). ddg rate limit (it rate-limits readily under fan-out), DNS failure, grepapp's intermittent 504 → [upstream], rotate or retry once. Missing API key, docs backend local (planned, unimplemented), force_browser with no browser configured → [precondition]. One asymmetry: a CLI crawl interrupted by SIGINT exits 0 with partial results, by design.

Gotchas

Detail for each lives in references/surfaces.md.

• Scraping a bare domain auto-probes /llms.txt and may silently return that instead of the homepage — the title field reveals the swap; no_llms_txt opts out.
• docs is a two-step: resolve the name → vet the matches → fetch by library ID. Resolve never returns empty — garbage in gets confident fuzzy matches out, so check the name, not just the trust score.
• Batch scrape reports per-URL failures inside a successful call: isError=false with results[].error set. Check every entry.
• regexp works on grepapp and sourcegraph only; github rejects it with a pointer to those backends.
• Background crawls (--background, status, stop) are CLI-only; the MCP crawl is synchronous and capped.
• The page cache (bbolt, 72h default TTL) is single-process: a long-running MCP server holds the lock, so concurrent CLI scrapes silently run cache-disabled — ketch doctor reports the cache as locked by another process. Running the server degrades the CLI; prefer CLI-only when both would run long-term.

BAD/GOOD contrasts

BAD: scrape {url: "https://docs.example.com"} — no bound; you get llms.txt or ~25k tokens, whichever is worse. GOOD: scrape {url: "https://docs.example.com/quickstart", max_chars: 6000, trim: true} — plus no_llms_txt: true when you want the page itself, not the site's llms.txt.

BAD: Telling a user they must run an MCP server to use ketch with agents — the CLI plus a prompt block is the zero-infrastructure path, and a long-running server holds the page-cache lock against every CLI call. GOOD: CLI by default; MCP when the operator wired it — and when mcp__ketch__* tools are in your list, use them for research instead of shelling out around the operator's setup.

BAD: [upstream] ddg rate limited → retry the identical call three times. GOOD: Rotate — backend: "brave" (or another provider from available_backends; auto is a chain, not a rotation target) — retry once, and note the swap. When auto itself failed, it already tried every usable provider: retry once, then report the outage.

BAD: Fetch docs from resolve's first match because its trust score is high, even though its name is not the library you asked about. GOOD: Vet name + snippet count + trust; if no match names the intended library, say so instead of fetching junk docs.

Reference loading

• ketch research … → read references/verbs/ketch-research.md before starting.
• ketch setup, any [precondition]/exit 5, or an install → read references/verbs/setup.md.
• Full flag/param tables, CLI↔MCP name mapping, backend/key matrix, or a surface behaving oddly → read references/surfaces.md.

Scope

In scope: the five research surfaces over both transports, the research and setup verbs, token budgets, error-prefix control flow, backend configuration. Out of scope: local or private codebase search (use repo tools), pages behind auth or paywalls, bulk archival crawling beyond the caps, browser automation beyond ketch's headless-rendering fallback.

Bound every fetch; cite every claim.