runpod-vllm-claude-code

RunPod vLLM + Claude Code bridge

Two things are being solved together here, and it's worth being clear about which is which:

1. Getting a model running on RunPod. Straightforward — pick a GPU, start a vLLM container, wait for health.
2. Getting Claude Code to actually talk to it. Not straightforward — Claude Code speaks Anthropic's Messages API (/v1/messages, content blocks, SSE events). vLLM speaks OpenAI's /v1/chat/completions. Nothing bridges these two by default. ANTHROPIC_BASE_URL lets Claude Code point at any server that implements the Messages API shape — so a small translation proxy in the middle is what makes "point Claude Code at my RunPod box" actually work. scripts/anthropic-openai-proxy.js is that proxy, already fixed for the two protocol mismatches that will otherwise burn an hour of retries (see Gotchas below). Use it as-is; don't re-derive it from scratch.

This is exploratory/disposable infra by nature — a GPU pod costs real money per hour the moment it exists. Before creating one, tell the user which GPU and roughly $/hr, unless they already named the machine and accepted the cost.

Step 0: get real numbers on the model before guessing hardware

Don't trust generic "VRAM requirements" blog posts for a model name — the GPU-rental SEO space is full of content-farm sites (things like "willitrunai.com", "canitrun.dev", "orcarouter.ai") that auto-generate plausible-looking spec pages for any model name, including ones that barely exist yet. They will confidently disagree with each other.

Get ground truth from the HuggingFace API instead:

# Real param count / architecture (don't trust the repo name's number):
curl -s "https://huggingface.co/<org>/<model>/raw/main/config.json"

# Real total weight size, which is what actually has to fit in VRAM:
curl -s "https://huggingface.co/<org>/<model>/raw/main/model.safetensors.index.json" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['metadata']['total_size'] / 1e9, 'GB')"

bf16 weight size in GB ≈ params(B) × 2. Budget the GPU so weights leave at least ~20-30GB of headroom for KV cache + activations — more if the model is a dense transformer (full KV cache scales with context), less if it's a hybrid/linear-attention architecture (Mamba/GDN layers keep fixed-size state, so long context is much cheaper than a same-size dense model).

If the model is a known problem architecture (very new hybrid attention, a fresh release), search for "<architecture class name>" vllm github issues — e.g. Qwen3_5ForConditionalGeneration vllm issues — before committing to a GPU choice, since brand-new architectures sometimes have open correctness bugs in vLLM (garbled output, tensor-format mismatches). Flag this risk to the user rather than silently hoping it works.

Step 1: pick and check GPU availability/price (don't guess the type string)

RunPod's REST API 404s on /v1/gputypes and /v1/datacenters — those aren't real endpoints despite being an obvious guess. Use the GraphQL API instead:

curl -s -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{"query":"query { gpuTypes(input: {id: \"NVIDIA H100 80GB HBM3\"}) { id lowestPrice(input: {gpuCount: 1}) { uninterruptablePrice stockStatus } } }"}'

Valid gpuTypeIds include "NVIDIA H100 80GB HBM3" (SXM), "NVIDIA H100 PCIe", "NVIDIA H100 NVL", "NVIDIA A100 80GB PCIe", "NVIDIA A100-SXM4-80GB" — get the full list with query { gpuTypes { id displayName memoryInGb } } if unsure. Pass a short list of acceptable types to pod creation (see below) so RunPod falls back automatically if your first choice is out of stock, rather than failing the whole provision.

Where the key lives: check $RUNPOD_API_KEY in the environment first; if it is not set, export it (create one in the RunPod console under Settings -> API Keys).

Step 2: create the pod

curl -s -X POST https://rest.runpod.io/v1/pods \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "vllm-<model-slug>-test",
    "gpuTypeIds": ["NVIDIA H100 PCIe", "NVIDIA H100 80GB HBM3", "NVIDIA H100 NVL"],
    "imageName": "vllm/vllm-openai:latest",
    "containerDiskInGb": <weights_GB * 1.5, minimum 80>,
    "ports": ["8000/http"],
    "dockerStartCmd": [
      "--model", "<org>/<model>",
      "--host", "0.0.0.0", "--port", "8000",
      "--trust-remote-code",
      "--enforce-eager",
      "--gpu-memory-utilization", "0.90",
      "--max-model-len", "65536",
      "--served-model-name", "<served-name>",
      "--enable-auto-tool-choice",
      "--tool-call-parser", "hermes"
    ],
    "env": { "HF_HOME": "/root/.cache/huggingface" }
  }'

Notes on the flags:

• --max-model-len: go big, bigger than feels necessary. 65536 sounded generous but a real multi-hour coding session (git surgery, lots of tool output) reached ~127K tokens and hit the wall — and worse, /compact itself needs input + a real output budget for the summary, so if the conversation is already near the ceiling, compact fails with the same overflow error at the exact moment you need it most. Prefer the model's own native context length — check max_position_embeddings in the HF config.json from Step 0 — and use that if GPU memory allows (a 262144-token window on an 80GB H100 was comfortable for a ~28B model). Going from 65536 straight to the model's real max avoided a second round of the same problem.
• --tool-call-parser: pick the one matching the model family (hermes is a reasonable default for most fine-tunes; qwen3_coder + --reasoning-parser qwen3 for Qwen3.x-coder-style models). Wrong parser choice won't error loudly — the model will just never emit tool_calls, only text. If Claude Code isn't calling tools, check the parser first.
• --enforce-eager: keep it for the first smoke test — it avoids CUDA-graph compile time and any related instability while you're just checking the model responds sanely. Drop it once correctness is confirmed and you care about speed.
• Poll GET /v1/pods first and reuse a RUNNING pod with a matching name if one already exists, instead of creating a duplicate.

The vLLM URL is always https://<podId>-8000.proxy.runpod.net. Poll /health (expect 404 then a string of 502s while the image pulls and weights download, then 200) — budget several minutes for a 50GB+ model. Run the poll loop in the background (run_in_background: true) rather than blocking the conversation on it.

Once healthy, sanity-check with a raw curl before involving Claude Code at all:

curl -s "$VLLM_URL/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "model": "<served-name>", "messages": [{"role":"user","content":"What is 2+2? One word."}],
  "max_tokens": 20, "temperature": 0
}'

Coherent output here proves the model+vLLM combo works before adding the Claude Code layer's own complexity on top. If this step gives garbled/repeating tokens, it's a model or vLLM version issue, not a bridging issue — don't proceed to step 3 yet.

Step 3: bridge Claude Code to it

Copy scripts/anthropic-openai-proxy.js next to where you're working and run it:

VLLM_URL="https://<podId>-8000.proxy.runpod.net" PORT=<free-port> SERVED_MODEL="<served-name>" \
  node anthropic-openai-proxy.js

Pick a genuinely free local port first — ss -ltn | grep :<port> — this box runs many Claude sessions in parallel and a common port like 8787 is often already taken by someone else's process. Don't just assume your port is free.

Then run an isolated Claude Code invocation against it — isolated meaning it must not fall back to the user's real, billed Anthropic account if the override doesn't take:

env -u ANTHROPIC_AUTH_TOKEN -u CLAUDE_CODE_OAUTH_TOKEN \
  ANTHROPIC_BASE_URL="http://localhost:<port>" \
  ANTHROPIC_API_KEY="sk-local-dummy-key" \
  CLAUDE_CODE_MAX_CONTEXT_TOKENS=<max-model-len used above> \
  claude --bare -p "<test task>" \
  --dangerously-skip-permissions \
  --model <served-name> \
  --output-format text

• --bare is what makes this safe: it forces auth to be strictly ANTHROPIC_API_KEY/apiKeyHelper, never OAuth or keychain. Without it, a misconfigured override can silently fall through to the user's real logged-in account and bill it — --bare closes that off structurally rather than relying on the env override being perfect.
• CLAUDE_CODE_MAX_CONTEXT_TOKENS: without this, an unrecognized --model name makes Claude Code assume a 200k window for its own budgeting, which causes it to request far more max_tokens than the vLLM server's --max-model-len allows. Set it to the same number you passed vLLM.
• CLAUDE_CODE_MAX_OUTPUT_TOKENS: a separate, easy-to-miss ceiling. When /compact has to summarize a very large conversation, Claude Code computes how big an output budget it needs and can hit an internal hard limit ("Claude's response exceeded the 128000 output token maximum"), refusing client-side before ever contacting the backend — no proxy log entry, nothing to debug remotely. Set this to something reasonable (e.g. 8192) alongside CLAUDE_CODE_MAX_CONTEXT_TOKENS.
• Watch the proxy's own stdout log while this runs — it prints incoming message/tool counts per turn, which is the fastest way to see whether requests are actually reaching the remote model versus dying in the bridge.
• Give it 2-3+ minutes for a real coding task; a --dangerously-skip-permissions -p run that hasn't returned in 180s is normal, not stuck — let it background rather than assuming failure.

Mid-session: swapping the pod under a live conversation

If a real session outgrows the context window it was started with, you don't need to lose the conversation to fix it — the chat's history lives locally in Claude Code's own transcript, not on the pod. The pod is disposable and swappable:

1. Terminate the old pod, create a new one with a bigger --max-model-len (same image, same served-model-name).
2. Poll the new pod's /health same as Step 2.
3. Kill the running proxy process and restart it with the same PORT, pointing VLLM_URL at the new pod. Nothing on the Claude Code side needs to change — same ANTHROPIC_BASE_URL, same port.
4. Have the user resume: claude --bare --resume <session-id> ... with CLAUDE_CODE_MAX_CONTEXT_TOKENS bumped to match the new window.

Restarting the proxy while a request is in flight will error that one request out (Claude Code shows an API error and usually auto-retries) — tell the user before doing this if they're actively mid-turn, don't just do it silently.

Known gotchas (already fixed in the bundled proxy — know them if you have to debug further)

1. max_tokens overflow. Claude Code's default requested max_tokens can be ~32000 regardless of the actual --max-model-len. If prompt + max_tokens exceeds the server's context window, vLLM 400s on every single turn and Claude Code just retries forever, burning GPU time for nothing. Fix: clamp the forwarded max_tokens to something sane (the bundled proxy defaults to 4096, override with MAX_TOKENS_CAP).
2. Interleaved role:"system" messages. Claude Code sends <system-reminder> content as separate role:"system" turns mid-conversation (not just the one top-level system field). Anthropic's own API is fine with this, but vLLM's chat templates enforce "system must be the first message" and 400 on anything else. Fix: fold any non-leading system-role message into user before forwarding.
3. A pkill -f <pattern> that matches your own shell kills your current command (exit 144). When killing a stuck nested claude process, find its exact PID with ps aux | grep and kill <pid> — don't pkill -f "claude" from inside a shell whose own invocation string also contains "claude". This isn't just a claude-specific gotcha: pkill -f "node proxy.js" self-matched too, because the shell wrapper's own command line contained that same string. Always kill by exact PID, never by a pattern that could describe the command you're running it from.
4. A single long blocking backend response can exceed RunPod's edge-proxy timeout. Regular RunPod Pods have no configurable timeout setting exposed via the API (executionTimeoutMs/idleTimeout exist in RunPod's API but are Serverless-only, not Pods) — so a /compact that has to reason over 100K+ tokens before answering can silently exceed it, and the proxy gets back an HTML gateway error page instead of JSON (Unexpected token '<', "<!DOCTYPE"). Fix: always request stream: true (+ stream_options: {include_usage: true}) from vLLM and relay chunks as they arrive — continuous bytes flowing means the idle timeout never triggers, no matter how long generation takes. The bundled proxy already does this.
5. vLLM can report prompt_tokens: 0 in the final streaming usage chunk even when completion_tokens comes back correct and real generation happened (seen after heavy tool-call turns). Since Claude Code uses the reported input token count for its own auto-compact timing, a false 0 can make it think there's far more headroom than there really is — silently setting up the exact same overflow this whole skill exists to avoid. The bundled proxy falls back to a rough character-count estimate (fallbackInputTokens) whenever the backend's number is falsy; don't strip that out.

Watching a live session's context usage from outside

You can monitor how full a running --resume session's context is without touching it, by reading its own transcript for the latest assistant turn's usage — useful when the user wants periodic "what % full am I" updates while they work.

# --bare sessions land under ~/.claude/projects/<cwd-slug>/<session-id>.jsonl —
# NOT the account-specific ~/.claude-<acct>/ dirs. find it with:
find ~/.claude -iname "*<session-id>*"

python3 -c "
import json
with open('<transcript.jsonl>') as f:
    lines = f.readlines()
for line in reversed(lines):
    line = line.strip()
    if not line: continue
    try: o = json.loads(line)
    except: continue
    if o.get('type') == 'assistant':
        usage = o.get('message', {}).get('usage')
        if usage and usage.get('input_tokens'):
            print(o['timestamp'], usage['input_tokens'], usage['input_tokens']/<max_model_len>*100)
            break
"

Scan the file in reverse from the end, not a fixed tail -N window — a single turn with many tool round-trips can push the latest usage-bearing line much further back than a small tail captures, making it look like nothing happened when it did.

Recovering from a bad /compact

If /compact produces a summary so lossy the model loses track of the task (or worse, takes destructive action out of confusion), the raw pre-compact messages are still sitting in the transcript file — compact just tells future turns to use the summary instead of replaying everything before it. That means you can surgically remove the compact and resume from before it happened. This is genuinely easy to get wrong twice in a row (it was, here) — read references/transcript-surgery.md before attempting it, don't wing it from first principles.

Step 4: teardown — "sin dejar rastro" means verify, not assume

When the user says kill/delete everything, that means confirm the negative, not just issue the delete calls:

# kill exact PIDs (not pkill -f, see gotcha 3)
ps aux | grep -E "anthropic-openai-proxy|claude --bare" | grep -v grep
kill <pid> <pid>

# terminate the pod
curl -s -X DELETE "https://rest.runpod.io/v1/pods/<podId>" -H "Authorization: Bearer $RUNPOD_API_KEY" -w "\n%{http_code}\n"

# confirm nothing is left running or costing money
curl -s https://rest.runpod.io/v1/pods -H "Authorization: Bearer $RUNPOD_API_KEY"          # expect []
curl -s https://rest.runpod.io/v1/networkvolumes -H "Authorization: Bearer $RUNPOD_API_KEY" # expect [] unless one was deliberately kept
curl -s -X POST https://api.runpod.io/graphql -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" -d '{"query":"query { myself { clientBalance currentSpendPerHr } }"}'
# currentSpendPerHr should read 0

# delete scratch files (proxy copy, test project, logs) from wherever they were put —
# the session scratchpad directory, never the repo
rm -rf <scratch-dir>

A network volume (persistent disk) is a separate billable resource from a pod — terminating the pod does not delete an attached network volume. If one was created for this session (rather than pre-existing), delete it explicitly: DELETE /v1/networkvolumes/<id>.

Report the final state back with the actual numbers (balance, currentSpendPerHr, empty pod/volume lists) rather than just saying "done" — that's what "sin dejar rastro" is actually asking to see proof of.