docker-ai-ml

Docker AI/ML — AI 工作负载

Guidance for running AI models and agents on Docker.

When to Use

ALWAYS use this skill when the user mentions:

• "docker ai", "docker model", "model runner"
• "本地 LLM", "local model", "offline model"
• "docker agent", "AI agent"
• "GPU docker", "nvidia docker"
• "sandbox", "沙箱"
• "LLM 部署", "模型部署"

Docker Model Runner

# List available models
docker model ls

# Pull a model
docker model pull ai/gemma3
docker model pull ai/qwen2.5

# Run inference
docker model run ai/gemma3 "Explain Docker in one sentence"

# Interactive chat
docker model run -i ai/gemma3

# API server mode
docker model serve ai/gemma3 --port 11434
# → access at http://localhost:11434/v1

Install Skills for AI Coding Assistants

# Install skills for Claude Code
docker model skills --claude

# For Codex CLI
docker model skills --codex

GPU Configuration

# Install NVIDIA Container Toolkit (once)
# Ubuntu/Debian:
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Run with GPU access
docker run --gpus all -it nvidia/cuda:12.6-base nvidia-smi

# Specific GPU
docker run --gpus device=0 -it nvidia/cuda:12.6-base

# GPU + Model Runner
docker model run --gpus all ai/llama3.2

Docker Agent

# agent.yaml — Define a multi-agent system
name: my-coding-agent
model:
  provider: openai
  model: gpt-4o
tools:
  - write_file
  - execute_command
  - web_search
sandbox:
  image: node:22-alpine
  workdir: /workspace

# Run agent with Docker
docker agent run -f agent.yaml "Fix the bug in src/app.js"

Docker Sandbox

# Isolated execution environment for AI agents
sandbox:
  image: python:3.12-slim
  network: none          # No network access
  read_only: true        # Read-only filesystem
  memory: 512m
  cpus: 1

Local LLM Stack

# Ollama (open-source) as alternative
docker run -d --name ollama \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  --gpus all \
  ollama/ollama

# Download and run model
docker exec ollama ollama pull llama3.2
docker exec ollama ollama run llama3.2 "Hello"

Workflow — 推荐使用流程

Step 1: 选择推理方案: Docker Model Runner (docker model) 优先，Ollama 备选 Step 2: 检查硬件: GPU 需 Linux + nvidia-container-toolkit；CPU 推理需 8-32GB RAM Step 3: 拉取模型: docker model pull <model> 或 ollama pull <model> Step 4: 启动服务: docker model run 或 ollama serve + 客户端 Step 5: 验证: 发送测试 prompt 确认模型正常响应

Gotchas — Common Pitfalls

• GPU not available: Docker Desktop Mac/Windows has limited GPU passthrough. → Recovery: Full GPU support requires Linux host + nvidia-container-toolkit; for Mac, use docker model which can offload inference.
• Model download size: LLMs are large (2-70 GB). → Recovery: docker model pull caches models in ~/.docker/models; check df -h before pulling; use quantized models (Q4/Q8) for smaller footprint.
• Memory limits: LLM inference needs significant RAM (8-32 GB typical). → Recovery: docker run --memory=16g model-runner; check RAM: free -h; use smaller models (7B instead of 70B) if limited.
• Model Runner vs Ollama: Docker Model Runner is Docker-native. Ollama has larger ecosystem. → Recovery: Use docker model for Docker integration; use Ollama for broader model selection; both can coexist.

Boundary — 能力边界（适用与不适用场景）

分类	场景	说明
✅ 能做	本地模型推理（Docker Model Runner / Ollama）	拉取+运行开源 LLM，无需 API key
✅ 能做	GPU 加速推理	Linux 宿主机 + nvidia-container-toolkit
✅ 能做	AI Agent 构建（Docker Agent / Sandboxes）	多 agent 协作、沙箱隔离
⚠️ 需条件	macOS GPU 推理	Docker Desktop 支持有限，优先用 docker model
⚠️ 需条件	大模型（70B+）推理	需 32GB+ RAM，考虑量化模型
❌ 超范围	训练/微调模型	需专用框架（PyTorch/TensorFlow），不在本技能范围
❌ 超范围	云端 API 调用（OpenAI/Claude）	非本地推理，使用对应 SDK
❌ 超范围	生产模型 Serving（K8s）	使用 docker-production + K8s 部署

When NOT to Use This Skill

❌ Skip	✅ Use Instead
Docker basics	docker-basics
Building custom model images	docker-build
Production model serving (K8s)	docker-production
Cloud AI APIs (no Docker needed)	Direct API access

Security & Stability

• Run model containers with --read-only when possible — models are read-only weights.
• Limit network access for model containers. Local inference doesn't need internet.
• GPU passthrough gives the container direct hardware access. Combine with --security-opt for isolation.
• Monitor GPU memory usage: nvidia-smi for utilization.

📚 官方文档参考

文档	地址
Docker AI 概述	https://docs.docker.com/ai-overview/
Docker Model Runner	https://docs.docker.com/ai/model-runner/
Docker Model Runner 入门	https://docs.docker.com/ai/model-runner/get-started/
Docker Agent	https://docs.docker.com/ai/docker-agent/
Docker Sandboxes	https://docs.docker.com/ai/sandboxes/
AI/ML 示例	https://docs.docker.com/reference/samples/ai-ml/

🧭 Docker Skills Journey

📍 You are here: docker-ai-ml — AI/ML 工作负载

← Prev: docker-testcontainers — Integration testing

FAQ

Q1: 如何快速上手此技能？ A: 参考上方的快速开始章节，按步骤操作即可。

Q2: 遇到版本不兼容问题怎么办？ A: 检查依赖版本，使用 lock 文件锁定，参考常见陷阱章节。

Q3: 如何在生产环境使用？ A: 参考最佳实践章节，确保配置正确，做好监控和日志。

Q4: 性能如何优化？ A: 参考性能优化相关文档，使用缓存、索引等手段。

Q5: 如何贡献或反馈问题？ A: 在 GitHub 仓库提交 Issue 或 Pull Request。

Q6: 是否支持中文？ A: 支持中文文档和中文注释，详见国内适配章节。