serving-llms-on-epyc

Serving LLMs on AMD EPYC™ (vLLM + zentorch, CPU)

Bring up a single vLLM OpenAI endpoint on an AMD EPYC™ host with the zentorch CPU backend, sized to the hardware. Container-first (Docker or Podman); conda/host is the fallback. An installed AMD Instinct GPU does not disqualify the host: select this skill when the endpoint itself should run on the EPYC CPU.

This is single-socket serving: one instance pinned to one socket and its memory (vLLM scales poorly across sockets, so we do not span them). On a dual-socket host it runs on a single socket; the multi-socket answer is multiple instances (one per socket), which is out of scope for this single-instance recipe.

Hard rule for this skill: on any failure, report the cause + logs and STOP. Do not retry, do not debug. (Debugging is a separate workflow.)

The agent does the serve flow itself -- pull, configure, launch, poll -- using the runtime validate.py reports. Never hand the user per-serve commands. Like serving-llms-on-instinct, an accessible container runtime is a one-time prerequisite: if validate.py finds none, report its one-time fix (make docker accessible / install podman / provide a conda env) and stop. Do not attempt sudo or privilege escalation.

Data file

Read data/epyc.json directly. It holds the container image, mandatory CPU run flags, supported precision, the model-support policy, the default model, and the verified throughput-flag gotcha. Its vllm_version and image tag are one validated default stack; keep them aligned and do not hardcode either from memory.

Step 1: Detect the CPU

python3 scripts/detect.py            # add --host user@box for a remote host

Returns cpu_model, is_amd_epyc, epyc_generation (Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005), zen_arch, is_supported_epyc, avx512, logical_cores, physical_cores, sockets, numa_nodes, memory_gb.

Route from detect.py -- decide the serving path:

• is_amd_epyc is false -> stop: this skill targets AMD EPYC. (Other x86 may work but is unsupported here.)
• avx512 is false -> zentorch cannot run on this CPU (its bf16 path needs AVX-512 BF16, avx512_bf16, which only lands on Zen4+). This is a pre-Zen4 EPYC (Naples / Rome / Milan, 7000 series). Do not dead-end -- it is still an EPYC host: offer the stock vLLM CPU path (plain vLLM, no zentorch acceleration -- slower, but verified working on EPYC 7763/Milan). Proceed only on the user's explicit OK, launching the official stock vLLM CPU image (Step 6 "Stock vLLM" variant); if the user declines, stop.
• is_supported_epyc is false but avx512 is true (e.g. Bergamo / Siena / EPYC 4004/4005) -> the zentorch path is not validated for this generation. Get explicit confirmation to try zentorch unvalidated, or take the same stock vLLM offer.
• else (9000 series -- Genoa/Turin/Venice -- with AVX-512 BF16) -> the validated zentorch path. Proceed.

validate.py (Step 2) reports zentorch_capable and sets requires_confirmation for the stock/unvalidated paths, so this routing is enforced there too.

Carry epyc_generation / avx512 through the later phases -- e.g. Venice packs up to 256 cores/socket, which the thread-binding in Step 5 sizes from.

Step 2: Validate the runtime and environment

python3 scripts/validate.py --image <image from data/epyc.json> --generation <epyc_generation from detect> --avx512 <avx512 from detect>

This also hard-enforces the AVX-512 gate: on a CPU without AVX-512, validate.py returns a blocking error (ready: false) so the flow stops here regardless of the Step 1 prose -- no image pull, no launch. (It reads the local CPU itself if --avx512 is omitted, so the gate holds even if the value was not passed through.)

Returns ready, requires_confirmation, zentorch_capable (false -> zentorch can't run; use the stock vLLM variant in Step 6), runtime (docker, podman, or null), runtime_detail, conda_path_available, stack, compatibility, hf_cache (resolved HF cache mount -- use hf_cache.mount at launch), ram_gb, and errors/warnings/advisories. If requires_confirmation is set (stock/unvalidated path), surface that and get the user's OK before launching. Pick the path:

• runtime is docker or podman -> container path (Step 6), used verbatim.
• runtime null but conda_path_available: true -> conda/host path.
• runtime null and no conda -> ready is false. Report the one-time onboarding fix (make docker accessible / install podman / conda env) and stop.

Do not proceed if ready is false.

Stack-compatibility gate. validate.py probes the selected runtime for its exact vllm/zentorch/torch versions and the active vLLM platform, then sets compatibility.status:

• proceed -> the stack is the validated default (or a validated family on a Zen platform); continue.
• blocked -> a stock CPU platform is active, so zentorch acceleration is not on (error). Report compatibility.message and stop.
• confirmation_required (requires_confirmation: true) -> Venice on a vLLM other than the pinned default. This recipe has not been validated on Venice with that version. Surface compatibility.message, recommend the pinned vllm_version image from data/epyc.json, and stop for an explicit user go/no-go before launching. On the pinned default vLLM, Venice proceeds with no warning.

The gate only runs once the image is local. If validate.py reports the image is not pulled, pull it (or let Step 6 pull it) and re-run validate.py so the gate probes the real stack rather than only the tag.

Step 3: Resolve and validate the model

If the user named no model, use default_model from data/epyc.json (Qwen/Qwen3-0.6B -- ungated, tiny, fast first success). Otherwise use theirs.

Check that vLLM actually supports the model (do not blanket-block multimodal). Pass the vLLM version the model will actually run on: use stack.vllm from validate.py when it was probed (the conda env may differ from the pin), else the vllm_version from data/epyc.json.

python3 scripts/check_model.py --model-id <model> --revision <rev or main> --vllm-version <stack.vllm from validate, else vllm_version from data/epyc.json>

• Exit 0 = vLLM serves it as a generation endpoint, or support is undeterminable (gated/offline) -- proceed; launch confirms.
• Exit 1 = stop: the architecture is not in vLLM's registry, it is a pooling/embedding/reranker (not a chat/completion endpoint), or it is a multimodal model with no usable chat template (launchable: false). Report the printed message and stop.

The result also carries the client endpoint the model supports:

• primary_endpoint: "chat_completions" -- a usable chat template is present (chat_template.status: present); serve and hand off /v1/chat/completions.
• primary_endpoint: "completions" -- no usable/auto-selectable template (absent/ambiguous/unknown); serve and hand off /v1/completions with a raw prompt. Chat can still be enabled by passing --chat-template <file> (or, for ambiguous, choosing one of chat_template.names); never invent one.
• Carry primary_endpoint, supported_endpoints, and chat_template through to verification (Step 7) and the handoff (Step 8).
• A multimodal model is allowed; a vLLM-supported multimodal arch may still hit a GPU-only kernel on CPU, which surfaces at load (the no-retry rule then applies).

Precision/dtype: native CPU dtypes are bf16 (default), fp16, fp32. Use bfloat16 unless the user asks otherwise.

For gated models (Llama, Gemma) HF_TOKEN must be set and the license accepted on HuggingFace; if not, stop and say so.

Step 4: Check it fits host RAM

RAM is the ceiling on CPU (weights + KV cache both live in RAM). Run on ONE line:

python3 scripts/estimate_memory.py --model-id <model> --revision <rev or main> --ram-gb <memory_gb from detect> --max-model-len <4096 or user value> --num-prompts <1 or desired concurrency>

Exit 0 = fits, exit 1 = does not fit. If fit.fits is false: do not launch. Tell the user required_gb vs ram_gb and the printed fit.action -- reduce --max-model-len to fit.suggested_max_model_len and retry, or use a smaller model. --max-model-len and --num-prompts are the two knobs that move KV. Extra flag: --weight-gb N overrides weights if a model has no HF metadata (rare). KV cache is bf16-only on zentorch CPU (no fp8 KV).

Step 5: Size the CPU runtime from the hardware

eval "$(python3 scripts/cpu_tune.py)"      # or --format json to inspect

A single instance runs on one socket, with its memory (vLLM scales poorly across sockets). cpu_tune.py exports VLLM_CPU_OMP_THREADS_BIND (the chosen socket's physical cores) and VLLM_CPU_KVCACHE_SPACE (sized from that socket's local RAM, not whole-system, so the KV pool stays on-socket). It does not set OMP_NUM_THREADS (vLLM derives it) or VLLM_CPU_NUM_OF_RESERVED_CPU (vLLM's own default).

Socket choice on a dual-socket host (load-aware): it samples per-socket CPU busy% (~0.5s) and prefers a free socket -- both free → socket 0; one free → that socket; both busy (≥ --busy-threshold, default 15%) → it warnings and proceeds on the least-busy socket. --socket N forces a choice. Single-socket hosts use socket 0.

For the chosen socket it also emits the memory-bound pin: container_cpuset (--cpuset-cpus=<cores> --cpuset-mems=<nodes>) for the container path, and conda_launch_prefix (numactl --cpunodebind/--membind, falling back to taskset CPU-only, or empty-with-note if neither tool exists) for conda. Surface warning to the user if set. On NPS2/NPS4 a socket spans multiple NUMA nodes; memory is bound across them and nps_note flags that finer binding could add performance.

Step 6: Confirm the plan, then launch (container-first)

Before launching, present this summary and wait for the user to confirm -- do not launch unprompted. This is the human gate before anything runs:

Field	Value
Model / kind	<model> -- text or multimodal (from check_model.py)
Backend	zentorch-accelerated (Zen4+) or stock vLLM CPU / no zentorch (zentorch_capable:false -- unaccelerated; verified on Milan)
Path	container (<runtime>, image from data/epyc.json) or conda/host
Precision	bfloat16 (or the user's choice)
Fit	required <required_gb> GB vs <ram_gb> GB RAM
CPU sizing	socket <chosen_socket> (<socket_choice_reason>), bind <VLLM_CPU_OMP_THREADS_BIND>, KV <VLLM_CPU_KVCACHE_SPACE> GB (socket-local), mem bound to nodes <numa_nodes_on_socket>
Hardware	EPYC <epyc_generation> (<zen_arch>), <physical_cores> cores, AVX-512 <avx512>
Port	<port>

If cpu_tune.py returned a warning (e.g. all sockets busy), include it here so the user sees it before confirming.

Proceed only on a clear "go". If the user declines or wants changes (model, --max-model-len, port), stop and adjust -- do not launch.

Build the launch from data/epyc.json. The CLI is vllm serve <model>. Do not pass --device cpu on vLLM >= 0.20 -- the zentorch plugin auto-selects the CPU platform and vllm serve rejects the flag. Only add it if vllm serve --help lists it (older vLLM).

Pick a free port first. With --network=host the port is bound directly on the host, so a busy port is a hard failure (no remapping). Choose one that is free -- e.g. PORT=8000; while ss -ltn "sport = :$PORT" | grep -q LISTEN; do PORT=$((PORT+1)); done -- and use $PORT in the launch, health poll, and handover.

Mount the HF cache that validate.py resolved. Use hf_cache.mount from validate.py (it follows symlinks and flags NFS/root-squash) rather than a raw ~/.cache/huggingface, or the bind-mount can fail at container start on NFS homes.

Container path (runtime from validate.py). The agent runs these itself, including the pull. RT is the resolved runtime verbatim:

RT="<runtime from validate.py: docker | podman>"
$RT rm -f vllm-epyc 2>/dev/null               # clear any leftover container from a prior run (name collision otherwise)
$RT pull <image from data/epyc.json>          # agent pulls; do not ask the user to
$RT run -d --name vllm-epyc \
  <run_flags from data/epyc.json>            # --ipc=host --network=host --cap-add=SYS_NICE (SYS_NICE = NUMA membind; NO --shm-size with --ipc=host)
  <hf_cache.mount from validate.py> \        # resolved real path, e.g. -v /scratch/you/hf:/root/.cache/huggingface
  <container_cpuset from cpu_tune>             # --cpuset-cpus=<cores> --cpuset-mems=<nodes>
  --env VLLM_CPU_OMP_THREADS_BIND="$VLLM_CPU_OMP_THREADS_BIND" \
  --env VLLM_CPU_KVCACHE_SPACE=$VLLM_CPU_KVCACHE_SPACE \
  --env HF_TOKEN=${HF_TOKEN} \
  <image from data/epyc.json> \
  vllm serve <model> --dtype bfloat16 --port <port> --max-model-len <len>

Conda/host path (no container runtime, conda_path_available true). eval-ing cpu_tune already exported the env vars; prefix the launch with conda_launch_prefix from cpu_tune so memory is bound to the chosen socket (empty → unpinned, with a note):

<conda_launch_prefix from cpu_tune> vllm serve <model> --dtype bfloat16 --port <port> --max-model-len <len> &
# e.g. numactl --cpunodebind=0 --membind=0 vllm serve ...

Stock vLLM path (no zentorch) -- only when validate.py reports zentorch_capable: false (pre-Zen4 EPYC like Milan) and the user confirmed the unaccelerated fallback. Use the official stock vLLM CPU image, not the zentorch image: vllm/vllm-openai-cpu:latest-x86_64 (its ENTRYPOINT is vllm serve, so pass <model> --dtype ... --port ... as args). Same sized env + flags as the container launch above (VLLM_CPU_OMP_THREADS_BIND, --cpuset-cpus/--cpuset-mems, --cap-add=SYS_NICE, --ipc=host --network=host, the resolved HF cache mount):

RT="<runtime>"
$RT rm -f vllm-epyc 2>/dev/null
$RT run -d --name vllm-epyc \
  --ipc=host --network=host --cap-add=SYS_NICE \
  <container_cpuset from cpu_tune> <hf_cache.mount from validate.py> \
  --env VLLM_CPU_OMP_THREADS_BIND="$VLLM_CPU_OMP_THREADS_BIND" \
  --env VLLM_CPU_KVCACHE_SPACE=$VLLM_CPU_KVCACHE_SPACE --env HF_TOKEN=${HF_TOKEN} \
  vllm/vllm-openai-cpu:latest-x86_64 \
  <model> --dtype bfloat16 --port <port> --max-model-len <len>

This path is unaccelerated (no zentorch), but verified working on EPYC 7763 (Milan/Zen3) with --dtype bfloat16 -- bf16 runs on stock vLLM CPU without AVX-512, so no fp32 is needed. If it still fails at load, apply the no-retry rule (report + stop).

Optional throughput flags are opt-in and must move together (see Gotchas): TORCHINDUCTOR_FREEZING=1 + VLLM_USE_AOT_COMPILE=0 (+ ZENTORCH_WEIGHT_PREPACK=1). The base launch sets none of them.

Step 7: Poll until up and responsive

A 503 while loading is normal. Poll /health until the server answers, confirm the served model is listed, then prove the selected endpoint works (from primary_endpoint in Step 3). CPU first-token compile can take a minute or two. Track a healthy flag so a timeout is a failure, not a fall-through.

# 1. container alive (conda: process alive) + /health, with a real timeout
healthy=""
for i in $(seq 1 120); do
  $RT inspect -f '{{.State.Running}}' vllm-epyc 2>/dev/null | grep -q true || { echo "FAILED: container exited"; $RT logs --tail 50 vllm-epyc; break; }
  curl -sf http://localhost:<port>/health >/dev/null 2>&1 && { healthy=1; echo "HEALTHY"; break; }
  sleep 3
done
[ -n "$healthy" ] || { echo "FAILED: not healthy before timeout"; $RT logs --tail 50 vllm-epyc; }

# 2. the served model is registered
curl -sf --max-time 30 http://localhost:<port>/v1/models

Then exercise the endpoint the model actually supports. Use deterministic sampling and a small output cap for the smoke check:

# primary_endpoint == chat_completions
curl -sf --max-time 180 http://localhost:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<served-model>","messages":[{"role":"user","content":"hi"}],"max_tokens":16,"temperature":0}'

# primary_endpoint == completions  (no chat template)
curl -sf --max-time 180 http://localhost:<port>/v1/completions -H 'Content-Type: application/json' \
  -d '{"model":"<served-model>","prompt":"Hello, world","max_tokens":16,"temperature":0}'

Confirm the response is JSON with a non-error choices[0] (chat: message.content; completion: text). An HTTP 200 that carries an error payload is not success. Resource sanity (your validation list): $RT stats --no-stream vllm-epyc.

If the server never becomes healthy, /v1/models omits the model, or the endpoint returns an error/empty choices: print the container/process logs, state the failing phase, and STOP. Do not retry. Do not start a debugging loop.

Step 8: On success, hand over the endpoint

Give the user everything needed to call the server. Print a connection table:

Field	Value
Base URL	http://localhost:<port>/v1 (the trailing /v1 matters)
Served model	<served-model> (the id from /v1/models)
Endpoint	/v1/chat/completions or /v1/completions (from primary_endpoint)
Why	chat = a chat template is present; completions = no template (raw prompts)
Runtime / port	<runtime> / <port>
Sizing	OMP threads, KV GB, --max-model-len, socket / NUMA pinning
Stop	$RT rm -f vllm-epyc (container) or kill <pid> (conda)

Then a ready-to-run example for the selected endpoint.

Chat model (primary_endpoint: chat_completions):

curl -s http://localhost:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<served-model>","messages":[{"role":"user","content":"Hello"}],"max_tokens":128,"temperature":0.7}'

Base/prompt model (primary_endpoint: completions):

curl -s http://localhost:<port>/v1/completions -H 'Content-Type: application/json' \
  -d '{"model":"<served-model>","prompt":"Hello, world","max_tokens":128,"temperature":0.7}'

OpenAI Python client (point base_url at the local server; the SDK requires a non-empty key, so any placeholder works when the server has no auth):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:<port>/v1", api_key="EMPTY")
model = client.models.list().data[0].id

# chat model:
r = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=128, temperature=0.7,
)
print(r.choices[0].message.content)

# base/prompt model:
r = client.completions.create(model=model, prompt="Hello, world", max_tokens=128)
print(r.choices[0].text)

Argument guidance to pass along (see reference.md for the full list):

• max_tokens caps the output; prompt_tokens + max_tokens must be <= --max-model-len.
• temperature (0 = deterministic/greedy, higher = more random); tune top_p or temperature, not both.
• stream: true streams tokens (SSE) instead of one blocking response.
• The model's generation_config.json can set sampling defaults; pass explicit values to be sure.

Offline (single-instance batch)

For a one-shot offline run instead of a server, replace Step 6-8 with a single vllm bench throughput (or an offline LLM.generate) using the same sized env, wait for completion, and report the metrics. Same no-retry / no-debug rule.

Gotchas

See reference.md for the full list. The load-bearing ones:

• --device cpu was removed from vllm serve in vLLM >= 0.20. The zentorch plugin auto-selects CPU. Passing it makes vllm serve error with "unrecognized arguments: --device cpu".
• TORCHINDUCTOR_FREEZING=1 alone crashes engine-core init on vLLM 0.23 / zentorch 2.11 (AssertionError: expected OutputCode, got function). It only works with VLLM_USE_AOT_COMPILE=0 set alongside it. Never set one without the other.
• /dev/shm — use --ipc=host, not --shm-size. vLLM needs a large /dev/shm (the 64MB container default is too small). The base recipe uses --ipc=host, which shares the host's large shared memory. Do not also pass --shm-size: podman errors with "cannot set shmsize when running in the host IPC Namespace", and it is redundant on docker. If you instead isolate IPC (drop --ipc=host), then add --shm-size=16g — one or the other, never both.
• NUMA / socket: one instance is pinned to one socket plus its memory -- CPU bind + --cpuset-mems (container) / numactl --membind (conda), with KV sized from that socket's local RAM. On a dual-socket host cpu_tune.py picks a free socket by load and warnings if both are busy. NPS2/NPS4 (multi-node socket) gets an nps_note that finer per-node binding could add more.
• Rootless podman + --cpuset-cpus/--cpuset-mems: these are cgroup limits and may be ignored or rejected on rootless podman without cpuset cgroup delegation (cgroup v1, or v2 without the controller delegated). This is not fatal: CPU thread binding still applies via VLLM_CPU_OMP_THREADS_BIND inside the container; only the container-level memory pin is lost (reduced NUMA locality). If the run errors specifically on the cpuset flags, drop them and proceed -- do not treat it as a launch failure.
• HF cache mount: the default mounts ~/.cache/huggingface. If HF_HOME points elsewhere (common on shared hosts, e.g. /proj/.../vllm), mount that path to /root/.cache/huggingface instead, or the model re-downloads inside the container.
• Container name reuse: a leftover vllm-epyc from a prior run makes run fail with "name already in use" -- Step 6 clears it first with $RT rm -f vllm-epyc.