vllm-ascend

vLLM-Ascend - 大模型推理服务

vLLM-Ascend 是 vLLM 的插件，可在华为 Ascend AI 处理器上实现高效的大模型推理。它提供针对 Ascend 优化的内核、量化支持和分布式推理能力。

---

快速入门

离线批量推理

import os

# Required for vLLM-Ascend: set multiprocessing method before importing vLLM
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

from vllm import LLM, SamplingParams

# Load model with Ascend NPU (device auto-detected when vllm-ascend is installed)
llm = LLM(
    model="Qwen/Qwen2.5-7B-Instruct",
    max_model_len=4096
)

# Prepare prompts and sampling params
prompts = [
    "Hello, how are you?",
    "Explain quantum computing in simple terms.",
]
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512)

# Generate outputs
outputs = llm.generate(prompts, sampling_params)

# Print results
for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Output: {output.outputs[0].text}\n")

OpenAI 兼容 API 服务

# Start the API server
vllm serve Qwen/Qwen2.5-7B-Instruct \
    --max-model-len 4096 \
    --max-num-seqs 256 \
    --served-model-name "qwen2.5-7b"

# Or using Python
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --max-model-len 4096

API 客户端示例

import requests

# Completions API
response = requests.post(
    "http://localhost:8000/v1/completions",
    json={
        "model": "qwen2.5-7b",
        "prompt": "Once upon a time",
        "max_tokens": 100,
        "temperature": 0.7
    }
)
print(response.json())

# Chat Completions API
response = requests.post(
    "http://localhost:8000/v1/chat/completions",
    json={
        "model": "qwen2.5-7b",
        "messages": [
            {"role": "user", "content": "Hello!"}
        ],
        "max_tokens": 100
    }
)
print(response.json())

---

安装

先决条件

• CANN：8.0.RC1 或更高版本
• Python：3.9 或更高版本
• PyTorch Ascend：与你的 CANN 版本兼容

方法 1：Docker（推荐）

# Pull pre-built image
docker pull ascendai/vllm-ascend:latest

# Run with NPU access
docker run -it --rm \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/Ascend/add-ons:/usr/local/Ascend/add-ons \
    -e ASCEND_RT_VISIBLE_DEVICES=0 \
    ascendai/vllm-ascend:latest

方法 2：pip 安装

# Install vLLM with Ascend plugin
pip install vllm-ascend

# Or install from source
git clone https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
pip install -e .

验证安装

# Check vLLM Ascend installation
python -c "import vllm_ascend; print(vllm_ascend.__version__)"

# Check NPU availability
python -c "import torch; import torch_npu; print(torch_npu.npu.device_count())"

---

部署

服务模式

# Basic server deployment
vllm serve <model_path> \
    \
    --served-model-name <name> \
    --host 0.0.0.0 \
    --port 8000

# Production deployment with optimizations
vllm serve /path/to/model \
    \
    --served-model-name "qwen2.5-72b" \
    --max-model-len 8192 \
    --max-num-seqs 256 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.9 \
    --dtype bfloat16 \
    --api-key <your-api-key>

Python API

import os

# Required: Set spawn method before importing vLLM
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

from vllm import LLM, SamplingParams

# Single NPU
llm = LLM(
    model="Qwen/Qwen2.5-7B-Instruct",
    max_model_len=4096,
    dtype="bfloat16"
)

# Distributed inference (multi-NPU)
llm = LLM(
    model="Qwen/Qwen2.5-72B-Instruct",
    tensor_parallel_size=4,
    max_model_len=8192
)

# Generate
params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Hello world"], params)

LLM 引擎（高级）

from vllm import LLMEngine, EngineArgs, SamplingParams

engine_args = EngineArgs(
    model="Qwen/Qwen2.5-7B-Instruct",
    max_model_len=4096
)
engine = LLMEngine.from_engine_args(engine_args)

# Add requests and step through generation
request_id = "req-001"
prompt = "Hello, world!"
params = SamplingParams(max_tokens=50)
engine.add_request(request_id, prompt, params)

while engine.has_unfinished_requests():
    outputs = engine.step()
    for output in outputs:
        if output.finished:
            print(f"{output.request_id}: {output.outputs[0].text}")

---

量化

vLLM-Ascend 支持使用 msModelSlim 量化的模型。有关量化的详情，请参阅 msmodelslim。

使用量化模型

# W8A8 quantized model
vllm serve /path/to/quantized-model-w8a8 \
    \
    --quantization ascend \
    --max-model-len 4096

# W4A8 quantized model
vllm serve /path/to/quantized-model-w4a8 \
    \
    --quantization ascend \
    --max-model-len 4096

使用量化的 Python API

from vllm import LLM, SamplingParams

llm = LLM(
    model="/path/to/quantized-model",
    quantization="ascend",
    max_model_len=4096
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Hello"], params)

---

分布式推理

张量并行

将模型层分布到多个 NPU 上，以支持大型模型。

# 4-way tensor parallelism
vllm serve Qwen/Qwen2.5-72B-Instruct \
    --tensor-parallel-size 4 \
    --max-model-len 8192

import os
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

from vllm import LLM

llm = LLM(
    model="Qwen/Qwen2.5-72B-Instruct",
    tensor_parallel_size=4,
    max_model_len=8192
)

流水线并行

from vllm import LLM

llm = LLM(
    model="DeepSeek-V3",
    pipeline_parallel_size=2,
    tensor_parallel_size=4
)

多节点部署

# Node 0 (Rank 0)
vllm serve <model> \
    \
    --tensor-parallel-size 8 \
    --pipeline-parallel-size 2 \
    --distributed-init-method "tcp://192.168.1.10:29500" \
    --distributed-rank 0

# Node 1 (Rank 1)
vllm serve <model> \
    \
    --tensor-parallel-size 8 \
    --pipeline-parallel-size 2 \
    --distributed-init-method "tcp://192.168.1.10:29500" \
    --distributed-rank 1

---

性能优化

关键参数

参数	默认值	说明	调优建议
--max-model-len	模型最大值	最大序列长度	如果发生 OOM，请调低
--max-num-seqs	256	最大并发序列数	提高吞吐量时可调高
--gpu-memory-utilization	0.9	GPU 内存占比	如果预热时发生 OOM，请调低
--dtype	auto	数据类型	追求速度使用 bfloat16，追求兼容性使用 float16
--tensor-parallel-size	1	张量并行度	用于大型模型
--pipeline-parallel-size	1	流水线并行度	用于超大型模型

配置示例

# Small model (7B), single NPU
vllm serve <model> --max-model-len 4096 --max-num-seqs 256

# Medium model (32B), single NPU
vllm serve <model> --max-model-len 8192 --max-num-seqs 128

# Large model (72B), multi-NPU
vllm serve <model> --tensor-parallel-size 4 --max-model-len 8192

# Maximum throughput
vllm serve <model> --max-num-seqs 512 --gpu-memory-utilization 0.95

---

故障排除

常见问题

问：出现 AclNN_Parameter_Error 或 dtype 错误？

# Check CANN version compatibility
npu-smi info
# Ensure CANN >= 8.0.RC1

# Try different dtype
vllm serve <model> --dtype float16

问：内存不足（OOM）？

# Reduce max model length
vllm serve <model> --max-model-len 2048

# Lower memory utilization
vllm serve <model> --gpu-memory-utilization 0.8

# Reduce concurrent sequences
vllm serve <model> --max-num-seqs 128

问：模型加载失败？

# Check model path
ls /path/to/model

# Verify tokenizer
python -c "from transformers import AutoTokenizer; tok = AutoTokenizer.from_pretrained('/path/to/model'); print('OK')"

# Use trust_remote_code for custom models
vllm serve <model> --trust-remote-code

问：推理速度慢？

# Enable bfloat16 for faster compute
vllm serve <model> --dtype bfloat16

# Adjust block size
vllm serve <model> --block-size 256

# Enable prefix caching
vllm serve <model> --enable-prefix-caching

问：API 服务连接被拒绝？

# Check server is running
curl http://localhost:8000/health

# Verify port is not in use
lsof -i :8000

# Use explicit host/port
vllm serve <model> --host 0.0.0.0 --port 8000

环境变量

# Required: Set multiprocessing method for vLLM-Ascend
export VLLM_WORKER_MULTIPROC_METHOD=spawn

# Set Ascend device IDs
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3

# Debug logging
export VLLM_LOGGING_LEVEL=DEBUG

# Disable lazy initialization (for debugging)
export VLLM_ASCEND_LAZY_INIT=0

---

脚本

• scripts/benchmark_throughput.py - 吞吐量基准测试
• scripts/benchmark_latency.py - 延迟基准测试
• scripts/start_server.sh - 服务启动模板

---

参考资料

• references/deployment.md - 部署模式和最佳实践
• references/supported-models.md - 完整的模型支持矩阵
• references/api-reference.md - API 端点文档

---

相关技能

• msmodelslim - vLLM-Ascend 的模型量化
• ascend-docker - Ascend 的 Docker 容器配置
• npu-smi - NPU 设备管理
• hccl-test - 多 NPU 的 HCCL 性能测试

---

官方参考资料

• vLLM-Ascend 文档：https://docs.vllm.ai/projects/ascend/en/latest/
• vLLM 文档：https://docs.vllm.ai/
• Huawei Ascend：https://www.hiascend.com/document
• GitHub 仓库：https://github.com/vllm-project/vllm-ascend