ollama-stack

Ollama Stack

部署本地大模型堆栈，支持离线和隐私优先的工作流。

何时使用此技能

在以下情况下使用此技能：

• 为开发设置私有/本地大模型推理
• 构建隔离网络的 AI 环境
• 在个人硬件（Mac、Linux、配备 GPU 的 Windows）上运行模型
• 创建团队共享的推理端点
• 在决定使用云端大模型 API 之前进行原型设计

先决条件

• 8 GB+ 内存（建议使用 16 GB+ 内存运行 7B+ 模型）
• 如需 GPU 加速：配备 6 GB+ 显存的 NVIDIA GPU，或 Apple Silicon Mac
• Docker（用于容器化部署）
• 20 GB+ 磁盘空间用于存储模型

快速开始

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Start the server
ollama serve

# Pull and run a model
ollama pull llama3.1:8b
ollama run llama3.1:8b "Explain Kubernetes pods in one paragraph"

# List available models
ollama list

# Pull specific quantization
ollama pull llama3.1:8b-instruct-q4_K_M

模型选择指南

模型	大小	显存	最适合
llama3.1:8b	4.7 GB	6 GB	通用聊天、编码
llama3.1:70b	40 GB	48 GB	复杂推理
codellama:13b	7.4 GB	10 GB	代码生成
mistral:7b	4.1 GB	6 GB	快速处理通用任务
mixtral:8x7b	26 GB	32 GB	高质量 MoE
nomic-embed-text	274 MB	1 GB	用于 RAG 的嵌入
llava:13b	8 GB	10 GB	视觉 + 文本
deepseek-coder-v2:16b	9 GB	12 GB	代码生成
qwen2.5:14b	9 GB	12 GB	多语言、推理

Docker Compose — 完整堆栈

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_MAX_LOADED_MODELS=2
      - OLLAMA_FLASH_ATTENTION=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY:-change-me-in-production}
      - DEFAULT_MODELS=llama3.1:8b
    depends_on:
      ollama:
        condition: service_healthy

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    ports:
      - "4000:4000"
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml"]
    depends_on:
      ollama:
        condition: service_healthy

volumes:
  ollama_data:
  webui_data:

LiteLLM 代理配置

# litellm-config.yaml
model_list:
  - model_name: llama3
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://ollama:11434
  - model_name: codellama
    litellm_params:
      model: ollama/codellama:13b
      api_base: http://ollama:11434
  - model_name: embeddings
    litellm_params:
      model: ollama/nomic-embed-text
      api_base: http://ollama:11434

general_settings:
  master_key: sk-local-dev-key
  max_budget: 0  # unlimited for local

API 用法

Ollama 提供兼容 OpenAI 的 API：

# Chat completion
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": false
  }'

# Embeddings
curl http://localhost:11434/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nomic-embed-text",
    "input": "The quick brown fox"
  }'

# List models
curl http://localhost:11434/api/tags

Python 客户端

# pip install ollama
import ollama

# Chat
response = ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Explain Docker in 3 sentences"}],
)
print(response["message"]["content"])

# Streaming
for chunk in ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Write a haiku about containers"}],
    stream=True,
):
    print(chunk["message"]["content"], end="", flush=True)

# Embeddings
result = ollama.embed(model="nomic-embed-text", input="Hello world")
print(f"Embedding dimensions: {len(result['embeddings'][0])}")

OpenAI SDK 兼容性

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")

response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

自定义 Modelfile

使用自定义系统提示词和参数创建专用模型：

# Modelfile.devops-assistant
FROM llama3.1:8b

SYSTEM """You are a DevOps expert assistant. You provide concise, production-ready
advice about infrastructure, CI/CD, containers, and cloud services.
Always include relevant commands and config examples."""

PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
PARAMETER repeat_penalty 1.1

# Build and use custom model
ollama create devops-assistant -f Modelfile.devops-assistant
ollama run devops-assistant "Set up a GitHub Actions workflow for Docker builds"

GPU 配置

NVIDIA

# Verify GPU access
nvidia-smi
ollama run llama3.1:8b --verbose  # Shows GPU layers loaded

# Environment tuning
export OLLAMA_NUM_PARALLEL=4          # Concurrent requests
export OLLAMA_MAX_LOADED_MODELS=2     # Models in VRAM
export OLLAMA_FLASH_ATTENTION=1       # Faster attention
export CUDA_VISIBLE_DEVICES=0,1       # Multi-GPU

Apple Silicon

# Metal acceleration is automatic on macOS
# Verify with:
ollama run llama3.1:8b --verbose
# Look for: "metal" in the output

# Optimize for unified memory
export OLLAMA_NUM_PARALLEL=2          # Keep memory headroom
export OLLAMA_MAX_LOADED_MODELS=1     # One model at a time on 16GB

监控

# Check running models and memory usage
curl http://localhost:11434/api/ps

# Prometheus metrics (if enabled)
curl http://localhost:11434/metrics

# Quick health check script
#!/bin/bash
response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:11434/api/tags)
if [ "$response" = "200" ]; then
    echo "Ollama is healthy"
    curl -s http://localhost:11434/api/ps | python3 -m json.tool
else
    echo "Ollama is down (HTTP $response)"
    exit 1
fi

Systemd 服务

# /etc/systemd/system/ollama.service
[Unit]
Description=Ollama LLM Server
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_FLASH_ATTENTION=1"
LimitNOFILE=65535

[Install]
WantedBy=default.target

sudo useradd -r -s /bin/false -m -d /usr/share/ollama ollama
sudo systemctl daemon-reload
sudo systemctl enable --now ollama
sudo systemctl status ollama

安全

• 在生产环境中绑定到 127.0.0.1（默认值），使用反向代理进行远程访问
• 在 Open WebUI 上设置 WEBUI_AUTH=true
• 使用带 TLS 的 nginx 进行远程访问：

server {
    listen 443 ssl;
    server_name llm.internal.example.com;
    ssl_certificate /etc/ssl/certs/llm.pem;
    ssl_certificate_key /etc/ssl/private/llm.key;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host $host;
        proxy_buffering off;              # Required for streaming
        proxy_read_timeout 600s;          # Long model responses
        allow 10.0.0.0/8;
        deny all;
    }
}

故障排除

问题	解决方案
模型速度太慢	使用较小的量化版本（q4_K_M），启用 flash attention
内存不足	减少 num_ctx，使用较小的模型，设置 OLLAMA_MAX_LOADED_MODELS=1
未检测到 GPU	检查 nvidia-smi，重新安装 CUDA 驱动，验证 Docker GPU 运行时
连接被拒绝	检查 OLLAMA_HOST 设置，验证防火墙规则
模型下载失败	检查磁盘空间；对于自签名注册表，使用 ollama pull --insecure 重试

相关技能

• mac-mini-llm-lab — Apple Silicon 优化
• docker-compose — 服务编排
• vllm-server — 高吞吐量生产环境推理
• llm-gateway — 统一 API 路由