gpu-server-management

GPU 服务器管理

为 AI 推理和训练工作负载配置、设置并监控 NVIDIA GPU 服务器。

适用场景

在以下情况下使用此技能：

• 为 LLM 推理或模型训练搭建新的 GPU 服务器
• 安装或升级 NVIDIA 驱动和 CUDA toolkit
• 为 GPU 工作负载配置 Docker 和 NVIDIA Container Toolkit
• 使用 MIG 将 A100/H100 GPU 分区，以支持多租户工作负载
• 排查 GPU 错误、驱动问题或热降频

前提条件

• Ubuntu 22.04 LTS（推荐）或 RHEL 8/9
• NVIDIA GPU（推荐使用 A10G、A100、H100、RTX 4090 或 L40S）
• Root 或 sudo 访问权限
• 用于下载软件包的互联网连接

驱动安装（Ubuntu）

# Remove old drivers
sudo apt purge -y 'nvidia*' 'cuda*' 'libcuda*'
sudo apt autoremove -y

# Add NVIDIA package repository
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update

# Install latest driver (560.x as of 2025)
sudo apt install -y nvidia-driver-560 cuda-toolkit-12-6

# Install NVIDIA Container Toolkit (Docker GPU support)
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify
nvidia-smi
nvcc --version
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi

安装后配置

# Enable persistence mode (reduces driver initialization latency)
sudo nvidia-smi -pm 1

# Set power limit (reduce heat/noise on inference servers)
sudo nvidia-smi -pl 350          # watts; check TDP for your GPU model

# Disable ECC on inference servers (frees ~6% VRAM, less safe)
sudo nvidia-smi --ecc-config=0   # requires reboot

# Enable P2P for multi-GPU NVLink training
sudo nvidia-smi topo -m          # check NVLink topology

GPU 健康监控

# Real-time monitoring (like htop for GPUs)
watch -n 1 nvidia-smi

# Detailed stats
nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,\
utilization.memory,memory.used,memory.free,power.draw,clocks.current.graphics \
--format=csv --loop=1

# DCGM — production monitoring daemon (for clusters)
sudo apt install -y datacenter-gpu-manager
sudo systemctl start dcgm
dcgmi discovery -l                # list GPUs
dcgmi diag -r 1                  # quick health check
dcgmi diag -r 3                  # full diagnostic (takes ~20 min)

# Check GPU errors (XID errors — important for stability)
sudo dmesg | grep -i "NVRM\|nvidia\|XID"
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total \
  --format=csv,noheader

Prometheus GPU 指标（DCGM Exporter）

# Deploy DCGM Exporter for Prometheus scraping
docker run -d \
  --name dcgm-exporter \
  --gpus all \
  --cap-add SYS_ADMIN \
  -p 9400:9400 \
  --restart unless-stopped \
  nvcr.io/nvidia/k8s/dcgm-exporter:latest

# Key metrics exposed:
# DCGM_FI_DEV_GPU_UTIL          - GPU utilization %
# DCGM_FI_DEV_MEM_COPY_UTIL     - Memory bandwidth utilization
# DCGM_FI_DEV_FB_USED           - Framebuffer memory used (MB)
# DCGM_FI_DEV_SM_CLOCK          - SM clock speed (MHz)
# DCGM_FI_DEV_GPU_TEMP          - Temperature (°C)
# DCGM_FI_DEV_POWER_USAGE       - Power draw (W)
# DCGM_FI_DEV_XID_ERRORS        - XID error count (0 = healthy)

MIG 分区（A100/H100）

MIG（Multi-Instance GPU，多实例 GPU）可将一块 GPU 切分为多个相互隔离的小型 GPU。

# Enable MIG mode (requires reboot or restart of all processes)
sudo nvidia-smi -mig 1
sudo systemctl restart nvidia-persistenced

# List available MIG profiles (A100 80GB example)
nvidia-smi mig -lgip
# 1g.10gb   — 1 slice,  10GB (max 7 instances)
# 2g.20gb   — 2 slices, 20GB (max 3 instances)
# 3g.40gb   — 3 slices, 40GB (max 2 instances)
# 7g.80gb   — full GPU, 80GB (max 1 instance)

# Create MIG instances (e.g., 3× 2g.20gb + 1× 2g.20gb = multi-tenant)
sudo nvidia-smi mig -cgi 2g.20gb,2g.20gb,2g.20gb,2g.20gb -C

# List created instances
nvidia-smi mig -lgi
nvidia-smi mig -lcgi

# Use in Docker
docker run --gpus '"device=MIG-GPU-xxx/0/0"' ...

# Disable MIG
sudo nvidia-smi mig -i 0 -dci
sudo nvidia-smi mig -i 0 -dgi
sudo nvidia-smi -mig 0

GPU 服务器的内核和操作系统调优

# Increase file descriptor limits
echo '* soft nofile 1048576' | sudo tee -a /etc/security/limits.conf
echo '* hard nofile 1048576' | sudo tee -a /etc/security/limits.conf

# Disable transparent huge pages (reduces latency jitter)
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Persist via rc.local or systemd unit:
cat <<'EOF' | sudo tee /etc/rc.local
#!/bin/bash
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
nvidia-smi -pm 1
exit 0
EOF
sudo chmod +x /etc/rc.local

# PCIe performance mode
sudo nvidia-smi --auto-boost-default=0
sudo nvidia-smi --auto-boost-permission=0

多 GPU 拓扑检查

# Check NVLink and PCIe topology
nvidia-smi topo -m
# Output shows interconnect type:
# NV4 = NVLink 4.0 (H100 SXM)
# NV2 = NVLink 2.0 (A100 SXM)
# PHB = PCIe bus (slower; avoid for tensor parallel training)
# PIX = same PCIe switch (fast)

# Bandwidth test between GPUs
/usr/local/cuda/samples/bin/x86_64/linux/release/p2pBandwidthLatencyTest

常见问题

问题	原因	解决方法
nvidia-smi: command not found	未安装驱动	按照上方的驱动安装步骤操作
驱动版本不匹配	CUDA/驱动不兼容	在 developer.nvidia.com 查看兼容性矩阵
GPU 温度 >85°C	气流不畅或风扇故障	检查 nvidia-smi -q -d TEMPERATURE；重新安装散热器
XID 79 错误	GPU 硬件错误	运行 dcgmi diag -r 3；可能需要更换 GPU
容器中出现 failed to open device	未配置容器工具包	运行 nvidia-ctk runtime configure --runtime=docker
PCIe 带宽低	插槽错误或功率限制	检查 `nvidia-smi -q

最佳实践

• 始终启用持久模式（nvidia-smi -pm 1）——降低首次请求延迟。
• 监控 XID 错误；持续出现 XID 79/94 表示硬件故障。
• 训练时使用通过 NVLink 连接的 GPU；推理时通常使用 PCIe 即可。
• 设置 DCGM 告警，在温度超过 80°C 或功耗接近 TDP 时触发。
• 使用 MIG 进行多租户推理，为不同模型提供 GPU 隔离。

相关技能

• vllm-server - 在 GPU 上进行 LLM 推理
• llm-fine-tuning - GPU 训练设置
• linux-hardening - 保护主机操作系统
• prometheus-grafana - 指标仪表板