llm-inference-scaling

LLM 推理扩缩容

在 Kubernetes 上通过 GPU 感知型自动扩缩容、请求排队和经济高效的竞价实例策略，水平扩展 LLM 推理。

何时使用此技能

在以下情况下使用此技能：

• LLM API 流量不可预测，需要自动扩缩容
• 在 Kubernetes 上管理一组 vLLM 或 TGI 推理 Pod
• 通过竞价／抢占式 GPU 实例降低推理成本
• 为批量推理任务实现基于队列的自动扩缩容
• 构建共享 GPU 资源的多模型服务平台

前提条件

• 配备 GPU 节点的 Kubernetes 集群（已安装 NVIDIA operator）
• 已安装 KEDA（Kubernetes 事件驱动自动扩缩容器）
• 配备 GPU 指标的 Prometheus（dcgm-exporter 或 gpu-operator）
• 用于部署 chart 的 Helm 3+

GPU 节点设置

# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=true \
  --set dcgm.enabled=true \
  --set devicePlugin.enabled=true

# Verify GPU nodes are recognized
kubectl get nodes -l nvidia.com/gpu.present=true
kubectl describe node <gpu-node> | grep nvidia

配置 GPU 资源的 vLLM 部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-8b
  labels:
    app: vllm
    model: llama-3.1-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
      model: llama-3.1-8b
  template:
    metadata:
      labels:
        app: vllm
        model: llama-3.1-8b
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "meta-llama/Llama-3.1-8B-Instruct"
        - "--tensor-parallel-size"
        - "1"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--max-num-seqs"
        - "128"
        resources:
          requests:
            nvidia.com/gpu: "1"
            memory: "20Gi"
            cpu: "4"
          limits:
            nvidia.com/gpu: "1"
            memory: "24Gi"
            cpu: "8"
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token

根据 Prometheus 指标使用 KEDA 自动扩缩容

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaledobject
spec:
  scaleTargetRef:
    name: vllm-llama-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300          # 5 min before scale-down
  pollingInterval: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_num_requests_waiting
      threshold: "10"           # scale up if >10 requests waiting
      query: |
        sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_gpu_cache_usage
      threshold: "0.8"          # scale up if KV cache >80% full
      query: |
        avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})

基于队列扩缩容（Redis + KEDA）

# ScaledJob for async batch inference
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: llm-batch-inference
spec:
  jobTargetRef:
    template:
      spec:
        containers:
        - name: inference-worker
          image: myapp/inference-worker:latest
          env:
          - name: REDIS_URL
            value: redis://redis:6379
          - name: QUEUE_NAME
            value: inference-jobs
        restartPolicy: OnFailure
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 5
  successfulJobsHistoryLimit: 3
  triggers:
  - type: redis
    metadata:
      address: redis:6379
      listName: inference-jobs
      listLength: "5"           # 1 worker per 5 queued jobs

竞价实例策略

# Mixed node pool: on-demand + spot GPUs
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-config
data:
  priorities: |
    10:  # low priority = prefer
    - .*spot.*
    50:
    - .*on-demand.*
---
# Node affinity for spot with on-demand fallback
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
          - key: node.kubernetes.io/lifecycle
            operator: In
            values: [spot]
      - weight: 20
        preference:
          matchExpressions:
          - key: node.kubernetes.io/lifecycle
            operator: In
            values: [on-demand]

GPU 节点的 Cluster Autoscaler

# AWS EKS — enable cluster autoscaler for GPU node group
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set autoDiscovery.clusterName=my-cluster \
  --set awsRegion=us-east-1 \
  --set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \
  --set extraArgs.skip-nodes-with-local-storage=false \
  --set extraArgs.expander=least-waste

# Annotate GPU node group for autoscaler
kubectl annotate node <node> \
  cluster-autoscaler.kubernetes.io/safe-to-evict="false"

需要监控的扩缩容指标

# Prometheus queries for scaling decisions
# Requests waiting in vLLM queue
sum(vllm:num_requests_waiting) by (model)

# GPU KV cache utilization (>80% = bottleneck)
avg(vllm:gpu_cache_usage_perc) by (pod)

# Tokens per second throughput
sum(rate(vllm:generation_tokens_total[5m])) by (model)

# P99 time-to-first-token
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

常见问题

问题	原因	解决方法
Pod 卡在 Pending 状态	没有可用的 GPU 节点	检查 cluster autoscaler 日志；确认节点组的限制
扩容过慢	Cluster autoscaler 延迟以及模型加载时间	预热副本；增大 minReplicaCount
GPU 碎片化	在大型 GPU 上运行多个小型模型	使用 MIG 分区或整合模型规格
竞价实例回收导致错误	竞价实例被回收	添加 PodDisruptionBudget；使用优雅关闭
KEDA 未扩缩容	Prometheus 查询未返回数据	先在 Prometheus UI 中测试查询

最佳实践

• 设置 minReplicaCount: 1 以避免冷启动；仅对批量任务将副本数缩减到 0。
• 使用 minAvailable: 1 的 PodDisruptionBudget，以应对竞价实例回收。
• 将模型权重预先拉取到共享 PVC，使 Pod 启动速度提升 5–10 倍。
• 在不同节点池上部署不同模型系列（7B 使用 A10G，70B 使用 A100）。
• 使用 Kubernetes VPA 调整 CPU／内存资源，同时使用 KEDA 管理副本数。

相关技能

• vllm-server - vLLM 配置和调优
• gpu-server-management - GPU 节点设置
• model-serving-kubernetes - KServe
• kubernetes-ops - Kubernetes 核心操作
• llm-cost-optimization - 成本策略