socialcrawl

SocialCrawl API

One API key and one response envelope for social, commerce, and research data. Lists are { items, next_cursor?, total? }; add ?format=raw to skip the transform pipeline. Every file below lives in references/; open only the one the task needs.

API Key

Resolve the API key only when a live API call is needed. Documentation, capability, pricing, and code-generation questions do not need a key.

Check these sources in order without displaying their contents:

1. Environment variable: use SOCIALCRAWL_API_KEY when it starts with sc_ and is not an example placeholder.
2. Config file: if the environment variable is absent, load ~/.config/socialcrawl/api_key into SOCIALCRAWL_API_KEY in the current process. On POSIX shells, use SOCIALCRAWL_API_KEY="$(<"$HOME/.config/socialcrawl/api_key")"; export SOCIALCRAWL_API_KEY. On PowerShell, use $env:SOCIALCRAWL_API_KEY = (Get-Content -LiteralPath "$HOME/.config/socialcrawl/api_key" -Raw).Trim().
3. Missing key: tell the user to create or rotate a key at https://socialcrawl.dev/dashboard and configure it as SOCIALCRAWL_API_KEY or in ~/.config/socialcrawl/api_key. Do not ask them to paste a secret into chat. Stop before the live call until they say it is configured.

Never print, paste, log, or return the key. Never place a literal key in generated code, a command, a URL, an error message, or a saved transcript. Keep $SOCIALCRAWL_API_KEY (POSIX) or $env:SOCIALCRAWL_API_KEY (PowerShell) as a variable reference in examples. If the key appeared in chat or output, advise the user to revoke it in the dashboard and create a replacement.

First Live Call

Before the first live call in a session:

1. Resolve the API key using the steps above. If it is missing or a placeholder, stop before making the call.
2. Verify it with the free balance endpoint (0 credits):

   curl -s -H "x-api-key: $SOCIALCRAWL_API_KEY" "https://www.socialcrawl.dev/v1/credits/balance"
   

3. If successful, show the balance and continue with the user's request.
4. If it fails, report the error without including request headers or the key, then follow references/errors.md.

Freshness Check (do this once per session)

This file is a snapshot of the registry with this fingerprint:

registry_fingerprint: 61a90b6d5dadcc038c3e4d4165f85f74177c77c5c5dfc9c2b7712c2f5d4ddaa4
generated_at: 2026-10-07

Beside the balance check, make one free call:

curl -s -H "x-api-key: $SOCIALCRAWL_API_KEY" "https://www.socialcrawl.dev/v1/utility/endpoints?fingerprint=1" | head -c 400

• data.fingerprint equals registry_fingerprint above - use the bundled files and move on.
• It differs - this copy and the live API are out of sync (the skill or the server may be newer): a price, parameter or endpoint changed. Treat the live registry as the source of truth and resolve what the files cannot answer through /v1/utility/* (all free). Tell the user once, briefly: "this SocialCrawl skill does not match the live API; run npx skills update socialcrawl --yes to refresh it." Then answer their actual question. Do not repeat it or ask permission.
• No data.fingerprint (a 404, or an older server that answers with the plain list) - fall back to counts: compare data.stats.platforms and data.stats.endpoints from /v1/utility/endpoints with the totals at the top of references/index.md. Live numbers HIGHER means behind, as above.
• The call fails (no key yet, offline, non-200) - skip it silently. This check must never block the request.

Workflow

1. Find the call first (free, 0 credits, runs nothing). GET /v1/utility/find?task=<the job in plain words> ranks the endpoints for the job, each with method, path, params_filled (taken from your words), missing and estimated_credits. Call results[0] directly when uncertain is false: its method, path and params are the ready call (add any missing param, then apply the Cost rule). When uncertain is true, show the options. For a job that needs several calls (a profile, then posts, then comments; a brand watch), GET /v1/utility/plan?query=<the job> returns the calls in order with params, price, and which step feeds which. Write or billing requests return no results.
2. Fall back to the bundled files when offline, when find or plan answers 404 (an older server: fall back to the existing flow), or when uncertain is true. Route with Common jobs below, or grep references/index.md: its Job to recipe table, then one line per endpoint (grep -i transcript references/index.md). For a recipe, read only its section: grep -n '^## ' references/recipes.md gives the line numbers; do not open the whole file. For an endpoint, read the platform file's TOC (from the top until the first ## ), then only its section: grep -nE '^## (GET|POST|PUT|PATCH|DELETE) /v1/<platform>/<resource> - ' references/<platform>.md, and read from that line to the next ## . The section carries the exact cost, cache TTL, pagination, every parameter, the response fields, and the constraints checked before billing.
3. Estimate. Build the exact request and apply the cost gate below (one line for a small flat call, the full preflight in references/cost-gate.md for anything metered or large). A heading's number can be a per-row or per-page unit, not the request total.
4. Call. Resolve the key, then run the request exactly as the section shows it, keeping its HTTP method and body shape. If the user already asked to run this exact paid request, the cost disclosure is enough, except the Ask level: always ask (see the gate in the Cost rule); otherwise ask only before a paid call or wider scope they did not authorise.
5. Report. Return the data, then credits_used and credits_remaining. For an async job report the hold at submission and the settled charge on completion.

Common jobs

The first call for the asks agents meet most, generated from the registry recipes ({x} is yours to fill). find gives the same answer with your values filled in.

Job	First call	Then
Comments on a post to CSV	GET /v1/tiktok/post/comments?url={url}&sort=recent&scan_pages=3	paginate.py piped to to_csv.py --archetype Comment; any platform's <platform>/post/comments; recipe
Creators by niche, followers or country	GET /v1/search/creators?query={niche}&sources=tiktok,instagram&min_followers={min_followers}	search/creators has no country filter; recipe; one country: recipe; one country on a small budget: recipe
A product's reviews across retailers	GET /v1/prism/product-reviews?query={product}	covers Amazon, Google Shopping and Trustpilot reviews; Walmart, Target, Home Depot: <retailer>/reviews; recipe
Stock price and news	GET /v1/finance/ticker-search?keyword={company}	then finance/quote (quote.price.current) and finance/news; recipe
YouTube transcript	GET /v1/youtube/video/transcript?url={url}	recipe
Alert on new posts or changes	POST /v1/monitors (body recipe: instagram/profile/posts)	only when GET /v1/utility/estimate?id=monitors returns 200, send ?dry_run=1 first to validate and price without creating; otherwise describe the request and create nothing until the user agrees; a page: web/monitors; recipe
Stats for a list of post URLs	POST /v1/prism/post-stats (JSON body urls)	a file of URLs: batch.py --urls; recipe
Korean / Naver search	GET /v1/naver/brief?query={keyword}&corpora=news,blog,cafearticle	one corpus: naver/blog/search, naver/news/search; recipe
Profile lookup	GET /v1/tiktok/profile?handle={handle}	any <platform>/profile; many handles: prism/profiles; recipe
Scrape a web page	GET /v1/web/scrape?url={url}	structured fields: web/extract; a whole site: web/crawl
Brand mentions across the web	GET /v1/prism/brand-mentions?keyword={brand}&date_from={start_date}&include=digest	recipe
One topic on several platforms	GET /v1/search/multi?query={topic}&platforms=tiktok,instagram,youtube&relevance=filter	recipe

Helpers

Bundled scripts in scripts/ (Python 3). They read the key from SOCIALCRAWL_API_KEY or ~/.config/socialcrawl/api_key and never print it; --help on each gives the rest.

python3 scripts/sc.py tiktok/profile handle=nike          # one call; JSON to stdout, credits_used to stderr
python3 scripts/paginate.py tiktok/post/comments --items 500 --max-credits 20 url=<post_url> > rows.jsonl   # page to N rows within a budget
python3 scripts/to_csv.py --archetype Comment --in rows.jsonl --out comments.csv   # rows to CSV with stable columns
python3 scripts/batch.py --urls links.txt > stats.jsonl   # stats for a file of post URLs (prism/post-stats, or one prism/jobs job past 500)
python3 scripts/estimate.py tiktok/post/comments --items 500 --offline url=<post_url>   # quote offline from assets/endpoints.json

Code requests: read the section and cost-gate, write a snippet that reads SOCIALCRAWL_API_KEY from the environment with a comment stating the billing unit, and present it without running it.

Capability and pricing questions: answer from the index and references/pricing.md (cost, tiers, packs, refunds, the free-endpoint list). Auth, envelope, pagination, labels, caching, idempotency: references/api-overview.md. The free /v1/utility/* calls answer from the live registry: endpoints (filter ?platform=, ?search=), endpoint?id=tiktok/profile (one endpoint, with measured field fill rates), plan?query=, quickstart.

Ambiguous platform: if the user says "get @nike's profile" with no platform, ask which one.

Where to look

Need	Open
Any single platform (social, commerce, reviews and places, app stores, search/news/finance, research and dev, link-in-bio)	index.md lists every platform file with its endpoint count; open references/<platform>.md
One call over many platforms: Prism composites, search/multi, search/everywhere	prism.md, search.md
Scheduled re-runs with webhook delivery	monitors.md
"Which of MY list of accounts posted about X" (Cohorts, not listening)	cohorts.md
Cost of a request or a whole paged walk	cost-gate.md, pricing.md
A failed call	errors.md

Never answer "not supported" from this table alone. It is only as new as this file, and a missing row is far more often a stale copy than a real gap. Before telling a user a platform, endpoint or capability does not exist, check the live registry - it is free:

curl -s -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  "https://www.socialcrawl.dev/v1/utility/endpoints?search=douyin"

Each row carries path, method, credits_label, required_params, one_of, optional_params, paginated and cache_ttl_seconds: enough to make the call without this file. Only say it is unsupported when the search returns total: 0.

Comparing products across retailers: Amazon, Walmart, Target, eBay and Home Depot share the canonical Product / Review / Seller schemas, so rows line up. Resolve each retailer's product id (ASIN, item id, TCIN, ...) with its own search, then fetch detail and reviews per retailer. /v1/prism/product-reviews is one composite over Amazon, Google Shopping and Trustpilot reviews only; for Walmart, Target or Home Depot call <retailer>/reviews.

Several platforms at once: /v1/search/multi runs one query over several platforms' own search (each platform's page price, charged only for those that returned rows); /v1/search/everywhere (20 credits) plans, ranks, and clusters up to 17 sources; many profiles or post URLs go to POST /v1/prism/profiles / POST /v1/prism/post-stats, or the background job POST /v1/prism/jobs.

Making calls

Base URL https://www.socialcrawl.dev. Most endpoints are GET with query parameters (URL-encode values):

curl -s -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  "https://www.socialcrawl.dev/v1/{platform}/{resource}?{param}={value}"

Never assume the method. Each endpoint section is headed by its real verb with a runnable curl. Batch reads, Prism jobs, web jobs, monitors, and cohorts use POST, PATCH, PUT, or DELETE with content-type: application/json (web/parse is multipart); the list, plus streaming (Accept: text/event-stream) and two-step endpoints that need an id from an earlier call, is in api-overview.md. Rate limits: 600 requests per minute and 50 concurrent per key; both return 429 with Retry-After and are never billed. Some endpoints are task-polled and take 10-45s: use a 60s timeout.

Reading a response

Details are in api-overview.md. Five rules before the first call:

• Placement. pagination (next_cursor, has_more, page_size) and the optional meta.hint sit at the envelope root. dropped, _warnings and every page report (labels, relevance, label_share, hydration, walk, ...) sit inside data. Read data._warnings: an ignored parameter (count was ignored; use limit (page size)) or a handle that resolved to a different account (resolved_to_different_handle) is reported there, not as an error.
• Paging. Send pagination.next_cursor back as cursor and stop on has_more: false, not on a short page or on total. A zero-match list is 200 with items: [] at 0 credits.
• Default judgments are free. Comment, post, review and keyword-search rows carry labels (computed.labels) and search rows a relevance score (computed.relevance) at no extra cost; judgments=off removes them. Opt-in presets (label=spam|toxic|low_quality|quality|injection|mention, relevant_to=) add 1 credit per 25 newly judged rows. relevance=filter against the query is free.
• Spend controls. Creator post lists take stop_at_id / since so a poll pays only for new pages (pagination.stopped_at says why it stopped). Keyword searches take min_views, max_age_days, country / exclude_country, sort_rows=views, seen=<id> (repeats refunded) and max_pages (1-5, each page billed).
• meta.hint names one other call that does the job in fewer steps. Its message states the price when that call costs more; follow it only within what the user authorised.

include= hydration joins rows to a sibling endpoint and is charged per row on top of the page, so never quote the plain heading cost for a hydrated call; the ceilings are in cost-gate.md and the traps in api-overview.md.

Cost rule

Cache hits, idempotent replays, every /v1/utility/* endpoint, and /v1/credits/balance cost 0 credits. Failed calls are refunded, and a request rejected for bad params, a page past the last page, or a rate limit never deducts. Metered endpoints hold an upfront ceiling and refund down to the work done, so the response credits_used is the real charge. Quote the total for the exact request, not the endpoint's base or unit cost. Scale the gate to the call:

Call	Gate
Flat price ≤ 5 credits, user asked for it	One line: ≈ N credits
Metered, multi-page, batch, monitor/cohort, or > 25 credits	Full preflight via estimate.py / utility/estimate
Upfront hold (worst case) > 500 credits or over budget (the user's stated budget)	Ask before calling

Anything in between, or unsure of the price or params: quote first with the free GET /v1/utility/estimate?id=<platform/resource>&<params>. Any reply except a 404 means a current server (a 5xx or network failure is no reply: retry the estimate): use its quote, and dry_run=1 is then free on any endpoint (validates, fetches nothing). A 404 or ENDPOINT_NOT_FOUND means an older server: do not send dry_run=1 (it bills a normal call there, except the labelled dry run an endpoint documents); quote with scripts/estimate.py (offline, assets/endpoints.json) or the section's Pricing line. At every level, report credits_used after.

If the task needs a whole list, price the whole walk before the first page. Paging to completion multiplies the page cost by the page count, and a heading only shows one page. coverage=full on instagram/followers and instagram/following is the extreme case: 10 credits a page, about 50 accounts a page, so a 2,600-account list is roughly 520 credits. Quote that total and get agreement before starting (Multi-page walks).

Errors: classify before retrying with references/errors.md. Credits and balance: GET /v1/credits/balance (0 credits); the ledger is GET /v1/credits/transactions?limit=50 (0 credits).