runpod-serverless-deploy

Serverless deploy

Turn any deployable source into a queue-backed Serverless endpoint proven with one real job; the user leaves knowing what exists and what it bills. Steps: read the source, decide engine and hardware, provision, warm, smoke-test one real job, hand back the URL and a curl, state what is billing. Tear down only when the user framed the deploy as a throwaway test or asks for cleanup.

Two decisions are yours, unprompted: the engine family (the right worker image for this modality and model format) and the provisioning path (bake weights in, download-on-start, or a network volume; each costs a different cold start). Size hardware from the model's real metadata, never a guess. Among suitable workers prefer the smallest, most-proven image for a smoke-able deploy (a giant image makes the first job a many-minute cold start; the smoke test proves the journey, not maximal quality), and name the trade-off if the request implies a heavier engine. Creating an endpoint bills: state the GPU class's hourly cost before creating; the smoke job costs money too.

For code-first Python (writing endpoint functions in a project and iterating with the Runpod SDK's rp flash dev loop), this MCP-only bundle has no path. Say so instead of improvising one. This skill deploys an existing source (Hub repo, container image, ComfyUI workflow, Hub release) to an endpoint through the MCP tools.

Tool binding

Read the catalog before creating; create only what the user asked to deploy. A tool named here missing from the session: say so and use the nearest read; never invent a result.

Capability	Tool
GPU catalog, price, stock (the sizing math is runpod-discovery's); per-DC stock (dataCenters[]); CUDA versions with capacity (cudaVersions)	list-gpu-types / get-gpu-type (include:["AVAILABILITY"], product:["SERVERLESS"]); list-data-centers (include:["GPU_AVAILABILITY"]), get-data-center; get-capacity for the CUDA matrix
Hub / community catalog with the release config (hardware, env schema); deploy a release in one call; pin a release / read history	list-hub-repos (includeConfig:true); deploy-hub-repo; list-endpoint-releases
Template (image, disk, ports, env, registry, mounts); endpoint (GPU pools or CPU flavors, worker min/max, scaler, data centers, network volumes, timeout); network volume for large weights in a GPU-co-located DC	create-template; create-endpoint; create-network-volume
Multi-GPU / pin to one card (gpu.count, pool plus excludedTypes)	set-endpoint-gpus (pools, excludeGpuTypeIds, gpuCount; builds the exclusion list, preserves other settings), or update-endpoint with gpu.pools and gpu.excludedTypes together
Health / warmup; per-worker detail (a crash-looping container can still report RUNNING or INITIALIZING)	endpoint-health; list-endpoint-workers
Smoke test, blocking to terminal	run-endpoint, then get-job-status (wait:300000); runsync-endpoint only for a warm endpoint and a fast job (its result is kept for one minute)
Worker logs (model-load / crash evidence)	stream-worker-logs (source=system for the container lifecycle). Without it, endpoint-health + list-endpoint-workers narrow the cause but cannot settle a crash loop
Private-image registry credential; inspect; teardown	create-registry, get-registry; get-endpoint, get-template; delete-endpoint, delete-template, delete-network-volume

Phase 0 — identify the source and pick the engine (unprompted decision)

Read and classify the source you were given: the engine follows the modality and model format, not a default. Never serve a non-text-generation task on a vLLM or ComfyUI worker; transcription, TTS, OCR and embeddings are not chat-worker jobs.

Source / modality	Engine family → worker image
Text-generation LLM, safetensors weights	vLLM or SGLang → runpod/worker-v1-vllm (or an SGLang worker)
Text-generation LLM, GGUF quantized weights	llama.cpp / Ollama → a GGUF-capable worker, not vLLM
Image/diffusion workflow.json	ComfyUI → runpod/worker-comfyui (reference/comfyui-serverless.md)
Speech-to-text (transcription)	a Whisper-family model → a speech-to-text worker, not an LLM worker
Text-to-speech / voice cloning	a TTS model → a TTS worker
Speaker diarization	a diarization pipeline → a diarization worker from the Hub or public catalog
OCR / document extraction	a classic OCR engine → an OCR worker, not a VLM chat worker
Text embeddings	an embeddings server (e.g. text-embeddings-inference) → an embeddings worker
A custom handler you were given	your own image → a custom container image

• Format detection. Inspect the repo's files first: *.safetensors / *.bin → vLLM/SGLang; *.gguf → llama.cpp/Ollama. GGUF on worker-vllm is a common wrong-engine mistake: vLLM will not load it. State the detected format and the engine it implies in the final message.
• OCR sizing. For an OCR smoke deploy, default to a classic lightweight OCR worker (16–24 GB). A VLM-based OCR (7B+ params) is a big-model deploy: size it like one (48 GB class) or don't pick it for a smoke test.
• Community-template source. When the modality has no first-party Runpod worker, browse the Hub / public catalog for a community (non-Runpod) template. Name the repo and its owner from the listing, never from memory. Once one fits, deploy it and run one real job: don't stop at "I found X", and don't block on a credential the config makes optional (phase 2's runtime-optional-token rule); request the capability through the job input the config documents. Only if nothing fits, say "none found" and stop.
• Custom image. The user's image is the engine: deploy exactly the image named, never a substitute. Missing required env → phase 2's model-default rule, not a question back; the observed worker states are the report. After the deploy read the worker states (list-endpoint-workers or endpoint-health): for a custom image the workers' health IS the deliverable, and one completed job hides a crash-looping sibling, so report a crash-looping custom image as failing. A private registry needs a registry credential (create/reference it; see runpod-lifecycle-crud); never echo the registry password. For a load-balancer (custom-HTTP) endpoint, read reference/scenarios.md (guide id runpod-serverless-deploy/scenarios) before the create: the worker must pass the HTTP health check or the endpoint sits at zero ready workers, and the type cannot be changed later.

Phase 1 — size the hardware (read the catalog, cite runpod-discovery)

runpod-discovery owns the params×precision + KV/overhead VRAM math; apply its result to a real GPU pick.

• Read the model's real metadata (parameter count, precision; for diffusion the checkpoint/LoRA/VAE sizes), not the repo name alone when the config is readable. Big-model sizing is shown arithmetic, never a bare total: the parameter count and where you read it, the bytes per parameter at the chosen precision, the weights total, the KV-cache/overhead allowance, and the GPU decision they add up to.
• Pick the smallest GPU class whose VRAM fits, then confirm live stock: include:["AVAILABILITY"] goes with product:["SERVERLESS"] (stock differs by product), and the same read gives per-data-center stock and, in cudaVersions, which CUDA versions have capacity. Pin the CUDA floor on the create: gpu.minCudaVersion at the image's CUDA line (12.8 for a CUDA 12 image; 13.0 only for an image built on CUDA 13), or gpu.allowedCudaVersions for an exact set. They are mutually exclusive (400 if both are sent), and the floor must have stock in cudaVersions. Never create into a GPU class with no stock: a scale-to-zero endpoint that can never acquire a worker looks deployed but never runs.
• Pinning to one card (the user wants an exact GPU): select the pool that contains the card and list the pool's other GPU types in gpu.excludedTypes (gpu.pools takes pool ids, not GPU type ids; at least one type must remain), on the create or with set-endpoint-gpus afterwards. The read-back quotes pools and exclusions, and every other setting is confirmed unchanged from the read.
• Multi-GPU when one card can't hold the model (required VRAM > the largest single card with stock): GPUs-per-worker > 1 plus tensor parallelism in the worker env (e.g. vLLM's tensor-parallel setting = GPU count). Say you are going multi-GPU and why, that it costs proportionally more per hour, and quote the largest in-stock single card (VRAM, live stock, $/hr) against the multi-GPU pick before committing.
• Pin the pool on Hub deploys. deploy-hub-repo inherits the release's GPU pool list when gpuIds is omitted, which can silently acquire a top-tier card for a task a 24 GB card serves, at several times the hourly rate. Pass gpuIds from YOUR sizing on every Hub deploy unless the release config pins exactly the pool you chose.
• CPU serverless. create-endpoint takes one hardware choice: the gpu object (pools and count) or the cpu array of CPU flavors sized from list-cpu-types. For a CPU workload send cpu and no gpu (never force a GPU pool onto it), and quote the CPU flavor's hourly price.
• Set the values you mean. A REST create defaults to FlashBoot OFF, workers.max 3 and workers.idleTimeout 10 s: set flashboot and the worker bounds explicitly. Always send timeout (per-request execution limit, in ms) sized to the slowest real job: a v2 create without it comes back "timeout": 0 (live-verified 2026-10-29; why it matters). On a Hub deploy, read timeout back and set it with update-endpoint if it is 0. The sum of workers.max across the account is capped by a balance-tiered quota (5 to start), checked at create, so a second endpoint at the default of 3 can be refused.
• State the cost. Quote the chosen GPU's hourly rate (×GPU count) from the catalog read before creating. Creating the endpoint and running the smoke test both bill.

Phase 2 — provision

• Hub release. Deploy its listed release directly (one call). Read the release config first (hardware + env schema) so you pass only env keys it accepts and know its GPU/disk defaults. To reproduce a known-good build, pin the exact release, not "latest".
• Everything else. Create a template naming the engine image, disk size, exposed ports and env (model id, engine settings, tokens), then the endpoint referencing it. Start scale-to-zero (min workers 0) unless the user asked for warm capacity; never set a nonzero minimum silently: it is an always-on worker that bills around the clock. An image on registry.runpod.net is a Hub build: deploy it through deploy-hub-repo (the release carries the registry auth); a plain template with that image loops on a registry-auth failure.
• Provisioning path and cold start. Bake = fastest cold start, biggest image. Download-on-start = small image, slow first request per new worker. Network volume = weights staged once in a GPU-co-located data center and mounted by every worker (writable: coordinate concurrent writes), so new workers skip the download; a prebuilt worker loads from the mount, and your own handler can wrap its load in VolumeCache (Runpod Python SDK runpod>=1.12.0) to copy weights to local disk at each cold start (runpodctl/reference/model-caching.md). Choose a volume when weights are large (roughly > 30 GB) or shared; size it to the real asset total with headroom, create it in the endpoint's data center (runpod-lifecycle-crud), and say which path you took and its cold-start consequence.
• Weights too large to bake: stage them on a network volume in a data center that has the target GPU, keep minimum workers 0, and prove the wiring with read-backs (for a storage-setup ask those read-backs are the proof; a job on a large model waits for the user's ask); no staging pod the user did not ask for. State both facts plainly: the first worker downloads the weights to the volume once and later cold starts mount that cache, and a volume outside the GPUs' data center cannot mount. Full rule, and the VolumeCache variant for the user's own handler: reference/scenarios.md.
• Multi-region / HA (no auto-sync). Attach one network volume per data center. They do not auto-sync: replicate the data yourself (e.g. aws s3 sync between the volumes' S3 endpoints), or regions serve stale or missing weights.
• Required env, gated weights, runtime-optional tokens. Read the release/worker config (includeConfig: true) and SET every required env key with no default before deploying (e.g. an embeddings worker's MODEL_NAMES): without its model env a worker loads nothing and 400s every job. When the user named no model for a text-generation worker (a vLLM image with no MODEL_NAME), use the worker's documented default model if its config has one, else the smallest instruct model that fits the card (a 1–2B instruct model on a 16 GB class); state the choice and deploy, don't ask which model. A gated HuggingFace repo whose weights 403 without a token: pass the token in env; if the user gave none, say the download will 403 and ask; never deploy a build that cannot fetch its weights. A token needed only for an optional runtime capability (config default empty) branches on one check you make FIRST: did the user provide a token (pasted, set in the environment, or named as available)?
  • Token provided → using it is the job. Set it as the worker's documented env var at create, say you configured it (by env-var name, never the value), and expect the gated capability to work. Never deploy without wiring a credential the user handed you.
  • No token → deploy anyway. The deploy is NOT conditional on it: deploy, run the job, report what the endpoint returned, then name the token as the unlock.
• Never print or echo a token (HF, Civitai, registry); reference it as an env var.

Phase 3 — warm and smoke-test one real job

A create call that returned an id is not a working endpoint: prove it with one real job first. Block with server-side holds, not a poll loop handed to the user; report once terminal, or with the evidenced report when waiting stops being informative. Silently stopping mid-wait is not an answer.

• Submit, then wait server-side. Fire one small job (the smallest real input for the modality) with run-endpoint, then hold with get-job-status wait: 300000 (the maximum hold). A cold first job outlives runsync's hold, whose result is kept only a minute, so runsync-endpoint is for a warm endpoint and a fast job. The job proves the worker boots with the real asset set and returns the right output shape.
• A cold start of 1–5+ minutes on the first job is normal (image pull + model load), not a failure. If the job outlives a hold, hold again on the same job id; do not re-submit (that starts a new job). Between holds check list-endpoint-workers: unhealthy workers, a job queued beside ready workers, or zero capacity → stop waiting, read the logs. At most three holds per smoke job (get-job-status wait: 300000, same id); after the third, no fourth hold and no tight poll: the evidenced report (endpoint created correctly, job submitted, worker never became ready, what the worker states show) is the complete answer; an endless grind is not.
• Diagnose a stuck job from the reads you always have: endpoint-health, list-endpoint-workers, and the workerHealth block get-job-status attaches on IN_QUEUE.
  • initializing on the first job = the cold start may still be running → keep waiting, but read the worker logs (source=system) before the next hold: a container crash-looping on an image pull or a CUDA mismatch can still report INITIALIZING or RUNNING while the job sits in IN_QUEUE.
  • unhealthy workers, or repeated container starts with no handler output in the logs = a crash loop (a real failure, not slow warmup) → switch the worker image or GPU pool; don't wait it out.
  • total: 0 for more than a couple of minutes, or throttled above 0 = no capacity in the selected pools and data centers → widen them. Prefer a broad, in-stock GPU pool over a scarce high-VRAM tier (e.g. the 48 GB cards) so the endpoint can acquire a worker.
  • A job that never left IN_QUEUE was never parsed: no worker read its payload, so NEVER speculate the input was malformed from queue-stuck evidence; report the queue/cold-start fact with the worker states quoted. Never present a cause as "conclusive" that your own reads contradict or don't show: ready workers beside a queued job with nothing in progress point to a broken image; say so from the logs, not from the status alone.
• Work with the tools you were granted. In an MCP-only session don't reach for a web-fetch tool, shell or browser you don't have to read a worker's input schema or README; it can't be called and the deploy stalls. Take the input shape from list-hub-repos includeConfig: true and the endpoint's returned error, fix the payload, re-run. Without stream-worker-logs, diagnose from endpoint-health / list-endpoint-workers; never let an unfinished job hinge on a tool you may not have.
• On a failure, quote; don't guess. Read the log and quote it; never retry blindly or assert a cause you did not read. A terminal FAILED job: report its decisive detail (missing file, gated 403, OOM, wrong-format load error) as written, not what it probably says; fixes are under Error handling below. A terminal job's output/error IS worker-level evidence: quote it verbatim when explaining a failure (a gated-model rejection, a load error); never claim you can't quote evidence while holding a terminal result (no log-stream tool doesn't excuse it). Empty or missing output on COMPLETED jobs from a gated-capability worker is quotable evidence: pair it with the config fact (the documented token env was not set) and state the gate.

Phase 4 — hand back

• Read back what was asked. When the user asked for specific behavior settings (warm/minimum workers, idle timeout, FlashBoot, a GPU pin), read the endpoint back after the create/update and confirm those exact fields landed, plus timeout: an unread write is an unverified write, and the write call's own response is not a read. The REST PATCH is sparse (omitted fields are left untouched) except for the GPU selection: gpu.pools sent without gpu.excludedTypes clears the exclusions, so send both, and after a pin confirm the scaling fields too. The read-back result goes in the final answer.
• The hand-back is a complete final summary the user never scrolls back from: endpoint name and id; worker image or Hub repo, why that engine, and the read it came from; GPU pool and hourly price; smoke job id and terminal outcome with proof (output reference, quoted failure, or, if the wait ended without a terminal job, the evidenced report: job id, submitted payload, worker states observed, recommended next step); current worker state; sync and async URLs; kept (and what now bills) or deleted (read-back confirmed); a ready-to-run curl using $RUNPOD_API_KEY (never the literal key); the sizing arithmetic, format finding, and caching/placement facts.
• Cleanup. Default to KEEP what the user asked for: say exactly what is now billing (the endpoint, any min-worker, any network volume) and how to delete it later. Tear down (endpoint, its template, any network volume, each confirmed gone by a read-back) only when the user framed the journey as a throwaway test or asks for cleanup.

Hard rules

• Deploy only the source the user named. Never create an endpoint they did not ask for, and never touch or delete another endpoint, template, or volume.
• Never invent a model URL, a community template repo, or a Docker image. If the source can't be resolved from what the user gave plus the catalog, ask once or say "none found" and stop.
• Also binding: every rule in phases 0–4 above.
• Cold-start asks on a new endpoint (an existing endpoint is runpod-endpoint-ops'): read reference/scenarios.md first. Create with the levers that add no standing charge, and set a minimum worker only when asked or accepted: it bills around the clock.

Error handling

• Gated / 403 → say so and ask for the token; don't auto-recreate. OOM → up one GPU class (quote the new hourly cost); still failing after one step-up → re-examine the engine choice (switch image in-turn), not more GPU or waiting.
• Payload 400 → fix the payload or the missing env from the release config and re-run; don't stop to ask. Create 409 (name collision) → fresh short suffix, retry once, then surface it and stop. A 204 delete with no body is success; confirm with a read-back.

The full failure → action table (also a stuck queue past a hold, token-gated empty output, GGUF on vLLM, an empty catalog) is in reference/error-handling.md (guide id runpod-serverless-deploy/error-handling). Read it when a create or the smoke job fails, before you retry or report a cause.

Contract

The cross-journey answer contract (definite facts from reads, commit-don't-hedge, mutations bind to this conversation's creations, a standalone final message, honest failure, granted tools only) is defined in the runpod-mcp skill and applies to every reply from this journey. If the runpod-mcp skill has not been loaded in this conversation, load it now, before the next tool call: its rules on which resources you may change, when a request is the go, and when to stop for the user are not repeated here.