skypilot-multi-cloud-orchestration

SkyPilot 多云编排

使用 SkyPilot 跨云运行 ML 工作负载并自动优化成本的完整指南。

何时使用 SkyPilot

在以下情况使用 SkyPilot：

• 跨多个云（AWS、GCP、Azure 等）运行 ML 工作负载
• 需要通过自动选择云/区域来优化成本
• 在抢占式实例上运行长时间作业并自动恢复
• 管理分布式多节点训练
• 希望为 20+ 云服务商提供统一接口
• 需要避免供应商锁定

主要特性：

• 多云：AWS、GCP、Azure、Kubernetes、Lambda、RunPod，20+ 服务商
• 成本优化：自动选择最便宜的云/区域
• 抢占式实例：节省 3-6 倍成本并自动恢复
• 分布式训练：支持 gang scheduling 的多节点作业
• 托管作业：自动恢复、检查点、容错
• Sky Serve：支持自动扩缩的模型服务

改用以下替代方案：

• Modal：用于更简单的、Python 原生 API 的无服务器 GPU
• RunPod：用于单云持久 Pod
• Kubernetes：用于现有 K8s 基础设施
• Ray：用于纯基于 Ray 的编排

快速开始

安装

pip install "skypilot[aws,gcp,azure,kubernetes]"

# Verify cloud credentials
sky check

Hello World

创建 hello.yaml：

resources:
  accelerators: T4:1

run: |
  nvidia-smi
  echo "Hello from SkyPilot!"

启动：

sky launch -c hello hello.yaml

# SSH to cluster
ssh hello

# Terminate
sky down hello

核心概念

Task YAML 结构

# Task name (optional)
name: my-task

# Resource requirements
resources:
  cloud: aws              # Optional: auto-select if omitted
  region: us-west-2       # Optional: auto-select if omitted
  accelerators: A100:4    # GPU type and count
  cpus: 8+                # Minimum CPUs
  memory: 32+             # Minimum memory (GB)
  use_spot: true          # Use spot instances
  disk_size: 256          # Disk size (GB)

# Number of nodes for distributed training
num_nodes: 2

# Working directory (synced to ~/sky_workdir)
workdir: .

# Setup commands (run once)
setup: |
  pip install -r requirements.txt

# Run commands
run: |
  python train.py

关键命令

命令	用途
sky launch	启动集群并运行任务
sky exec	在现有集群上运行任务
sky status	显示集群状态
sky stop	停止集群（保留状态）
sky down	终止集群
sky logs	查看任务日志
sky queue	显示作业队列
sky jobs launch	启动托管作业
sky serve up	部署服务端点

GPU 配置

可用加速器

# NVIDIA GPUs
accelerators: T4:1
accelerators: L4:1
accelerators: A10G:1
accelerators: L40S:1
accelerators: A100:4
accelerators: A100-80GB:8
accelerators: H100:8

# Cloud-specific
accelerators: V100:4         # AWS/GCP
accelerators: TPU-v4-8       # GCP TPUs

GPU 回退

resources:
  accelerators:
    H100: 8
    A100-80GB: 8
    A100: 8
  any_of:
    - cloud: gcp
    - cloud: aws
    - cloud: azure

抢占式实例

resources:
  accelerators: A100:8
  use_spot: true
  spot_recovery: FAILOVER  # Auto-recover on preemption

集群管理

启动与执行

# Launch new cluster
sky launch -c mycluster task.yaml

# Run on existing cluster (skip setup)
sky exec mycluster another_task.yaml

# Interactive SSH
ssh mycluster

# Stream logs
sky logs mycluster

自动停止

resources:
  accelerators: A100:4
  autostop:
    idle_minutes: 30
    down: true  # Terminate instead of stop

# Set autostop via CLI
sky autostop mycluster -i 30 --down

集群状态

# All clusters
sky status

# Detailed view
sky status -a

分布式训练

多节点设置

resources:
  accelerators: A100:8

num_nodes: 4  # 4 nodes × 8 GPUs = 32 GPUs total

setup: |
  pip install torch torchvision

run: |
  torchrun \
    --nnodes=$SKYPILOT_NUM_NODES \
    --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \
    --node_rank=$SKYPILOT_NODE_RANK \
    --master_addr=$(echo "$SKYPILOT_NODE_IPS" | head -n1) \
    --master_port=12355 \
    train.py

环境变量

变量	描述
SKYPILOT_NODE_RANK	节点索引（0 到 num_nodes-1）
SKYPILOT_NODE_IPS	以换行分隔的 IP 地址
SKYPILOT_NUM_NODES	节点总数
SKYPILOT_NUM_GPUS_PER_NODE	每个节点的 GPU 数量

仅在头节点执行

run: |
  if [ "${SKYPILOT_NODE_RANK}" == "0" ]; then
    python orchestrate.py
  fi

托管作业

抢占式恢复

# Launch managed job with spot recovery
sky jobs launch -n my-job train.yaml

检查点

name: training-job

file_mounts:
  /checkpoints:
    name: my-checkpoints
    store: s3
    mode: MOUNT

resources:
  accelerators: A100:8
  use_spot: true

run: |
  python train.py \
    --checkpoint-dir /checkpoints \
    --resume-from-latest

作业管理

# List jobs
sky jobs queue

# View logs
sky jobs logs my-job

# Cancel job
sky jobs cancel my-job

文件挂载与存储

本地文件同步

workdir: ./my-project  # Synced to ~/sky_workdir

file_mounts:
  /data/config.yaml: ./config.yaml
  ~/.vimrc: ~/.vimrc

云存储

file_mounts:
  # Mount S3 bucket
  /datasets:
    source: s3://my-bucket/datasets
    mode: MOUNT  # Stream from S3

  # Copy GCS bucket
  /models:
    source: gs://my-bucket/models
    mode: COPY  # Pre-fetch to disk

  # Cached mount (fast writes)
  /outputs:
    name: my-outputs
    store: s3
    mode: MOUNT_CACHED

存储模式

模式	描述	最适合
MOUNT	从云端流式传输	大型数据集，读密集型
COPY	预取到磁盘	小文件，随机访问
MOUNT_CACHED	异步上传缓存	检查点，输出

Sky Serve（模型服务）

基本服务

# service.yaml
service:
  readiness_probe: /health
  replica_policy:
    min_replicas: 1
    max_replicas: 10
    target_qps_per_replica: 2.0

resources:
  accelerators: A100:1

run: |
  python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --port 8000

# Deploy
sky serve up -n my-service service.yaml

# Check status
sky serve status

# Get endpoint
sky serve status my-service

自动扩缩策略

service:
  replica_policy:
    min_replicas: 1
    max_replicas: 10
    target_qps_per_replica: 2.0
    upscale_delay_seconds: 60
    downscale_delay_seconds: 300
  load_balancing_policy: round_robin

成本优化

自动选择云

# SkyPilot finds cheapest option
resources:
  accelerators: A100:8
  # No cloud specified - auto-select cheapest

# Show optimizer decision
sky launch task.yaml --dryrun

云偏好

resources:
  accelerators: A100:8
  any_of:
    - cloud: gcp
      region: us-central1
    - cloud: aws
      region: us-east-1
    - cloud: azure

环境变量

envs:
  HF_TOKEN: $HF_TOKEN  # Inherited from local env
  WANDB_API_KEY: $WANDB_API_KEY

# Or use secrets
secrets:
  - HF_TOKEN
  - WANDB_API_KEY

常见工作流

工作流 1：带检查点的微调

name: llm-finetune

file_mounts:
  /checkpoints:
    name: finetune-checkpoints
    store: s3
    mode: MOUNT_CACHED

resources:
  accelerators: A100:8
  use_spot: true

setup: |
  pip install transformers accelerate

run: |
  python train.py \
    --checkpoint-dir /checkpoints \
    --resume

工作流 2：超参数扫描

name: hp-sweep-${RUN_ID}

envs:
  RUN_ID: 0
  LEARNING_RATE: 1e-4
  BATCH_SIZE: 32

resources:
  accelerators: A100:1
  use_spot: true

run: |
  python train.py \
    --lr $LEARNING_RATE \
    --batch-size $BATCH_SIZE \
    --run-id $RUN_ID

# Launch multiple jobs
for i in {1..10}; do
  sky jobs launch sweep.yaml \
    --env RUN_ID=$i \
    --env LEARNING_RATE=$(python -c "import random; print(10**random.uniform(-5,-3))")
done

调试

# SSH to cluster
ssh mycluster

# View logs
sky logs mycluster

# Check job queue
sky queue mycluster

# View managed job logs
sky jobs logs my-job

常见问题

问题	解决方案
配额超限	请求增加配额，尝试其他区域
抢占式实例被抢占	使用 sky jobs launch 自动恢复
文件同步缓慢	对输出使用 MOUNT_CACHED 模式
GPU 不可用	使用 any_of 进行回退云

参考资料

• 高级用法 - 多云、优化、生产模式
• 故障排查 - 常见问题与解决方案

Resources

• Documentation: https://docs.skypilot.co
• GitHub: https://github.com/skypilot-org/skypilot
• Slack: https://slack.skypilot.co
• Examples: https://github.com/skypilot-org/skypilot/tree/master/examples