rollback-workflow-builder

回滚工作流构建器

为生产部署构建安全、快速的回滚机制。

手动回滚工作流

# .github/workflows/rollback.yml
name: Rollback

on:
  workflow_dispatch:
    inputs:
      version:
        description: "Version to rollback to (e.g., v1.2.3 or previous)"
        required: true
        type: string
      environment:
        description: "Environment to rollback"
        required: true
        type: choice
        options:
          - staging
          - production
      reason:
        description: "Reason for rollback"
        required: true
        type: string

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: ${{ github.event.inputs.environment }}
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.inputs.version }}

      - name: Verify version exists
        run: |
          if ! git rev-parse ${{ github.event.inputs.version }} >/dev/null 2>&1; then
            echo "❌ Version ${{ github.event.inputs.version }} not found"
            exit 1
          fi
          echo "✅ Version ${{ github.event.inputs.version }} exists"

      - name: Get current version
        id: current
        run: |
          CURRENT=$(git describe --tags --abbrev=0)
          echo "version=$CURRENT" >> $GITHUB_OUTPUT
          echo "Current version: $CURRENT"

      - name: Confirm rollback
        run: |
          echo "🔄 Rolling back from ${{ steps.current.outputs.version }} to ${{ github.event.inputs.version }}"
          echo "Environment: ${{ github.event.inputs.environment }}"
          echo "Reason: ${{ github.event.inputs.reason }}"

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - run: npm ci
      - run: npm run build

      - name: Deploy rollback
        run: |
          ./scripts/deploy.sh ${{ github.event.inputs.environment }}
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

      - name: Verify deployment
        run: |
          ./scripts/health-check.sh ${{ github.event.inputs.environment }}

      - name: Create incident issue
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Rollback: ${context.payload.inputs.environment} to ${context.payload.inputs.version}`,
              body: `## Rollback Details

              **Environment:** ${context.payload.inputs.environment}
              **From:** ${{ steps.current.outputs.version }}
              **To:** ${context.payload.inputs.version}
              **Reason:** ${context.payload.inputs.reason}
              **Triggered by:** @${context.actor}
              **Time:** ${new Date().toISOString()}

              ## Next Steps
              - [ ] Verify rollback successful
              - [ ] Investigate root cause
              - [ ] Create fix
              - [ ] Update postmortem
              `,
              labels: ['incident', 'rollback']
            })

失败时自动回滚

deploy:
  runs-on: ubuntu-latest
  steps:
    - name: Deploy
      id: deploy
      run: ./scripts/deploy.sh production
      continue-on-error: true

    - name: Verify deployment
      id: verify
      if: steps.deploy.outcome == 'success'
      run: ./scripts/health-check.sh production
      continue-on-error: true

    - name: Auto-rollback on failure
      if: steps.deploy.outcome == 'failure' || steps.verify.outcome == 'failure'
      run: |
        echo "⚠️ Deployment failed, initiating automatic rollback"
        PREVIOUS_VERSION=$(git describe --tags --abbrev=0 HEAD^)
        ./scripts/deploy.sh production $PREVIOUS_VERSION

        # Verify rollback
        if ./scripts/health-check.sh production; then
          echo "✅ Rollback successful"
        else
          echo "❌ Rollback failed - manual intervention required"
          exit 1
        fi

Kubernetes 回滚

rollback-k8s:
  runs-on: ubuntu-latest
  steps:
    - name: Setup kubectl
      uses: azure/setup-kubectl@v3

    - name: Configure kubectl
      run: |
        echo "${{ secrets.KUBECONFIG }}" > kubeconfig
        export KUBECONFIG=kubeconfig

    - name: Rollback deployment
      run: |
        kubectl rollout undo deployment/myapp -n production
        kubectl rollout status deployment/myapp -n production --timeout=5m

    - name: Get rollback revision
      run: |
        kubectl rollout history deployment/myapp -n production

Docker 镜像回滚

- name: Rollback to previous image
  run: |
    # Get previous image tag
    PREVIOUS_TAG=$(docker inspect myapp:latest | jq -r '.[0].ContainerConfig.Labels.previous_tag')

    # Retag previous as latest
    docker pull myapp:$PREVIOUS_TAG
    docker tag myapp:$PREVIOUS_TAG myapp:latest
    docker push myapp:latest

    # Restart containers
    docker-compose pull
    docker-compose up -d

数据库迁移回滚

- name: Rollback database migrations
  run: |
    # Get migration to rollback to
    CURRENT=$(npm run migrate:current)
    TARGET=${{ github.event.inputs.migration }}

    echo "Rolling back from $CURRENT to $TARGET"
    npm run migrate:down -- --to=$TARGET

    # Verify rollback
    AFTER=$(npm run migrate:current)
    if [ "$AFTER" != "$TARGET" ]; then
      echo "❌ Migration rollback failed"
      exit 1
    fi
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}

回滚操作手册

# Production Rollback Runbook

## When to Rollback

Rollback if:

- Critical bugs affecting >10% of users
- Data integrity issues
- Security vulnerabilities
- Performance degradation >50%
- Error rate >5%

## Before Rollback

1. **Assess impact**: Check monitoring dashboards
2. **Identify version**: Determine last known good version
3. **Notify team**: Post in #incidents Slack channel
4. **Enable maintenance mode** (if possible)

## Rollback Steps

### Automated Rollback (Preferred)

1. Go to Actions → Rollback workflow
2. Select environment (staging/production)
3. Enter target version (e.g., v1.2.3 or "previous")
4. Enter reason for rollback
5. Click "Run workflow"
6. Monitor progress in Actions tab

### Manual Rollback (Emergency)

```bash
# 1. SSH to production server
ssh production

# 2. Check current version
docker ps | grep myapp

# 3. Pull previous version
docker pull myapp:v1.2.3

# 4. Update docker-compose
vim docker-compose.yml
# Change image: myapp:latest to myapp:v1.2.3

# 5. Deploy
docker-compose up -d

# 6. Verify
curl https://api.myapp.com/health

# 7. Check logs
docker logs myapp -f
```

回滚后

1. 验证：运行冒烟测试
2. 监控：观察错误率 15 分钟
3. 通知：在 #incidents 中更新状态
4. 禁用维护模式
5. 创建事件工单
6. 安排事后复盘

回滚验证

• 健康检查返回 200
• 错误率 <1%
• 响应时间 p95 <500ms
• 关键功能正常（登录、结账等）
• 数据库连接正常

沟通模板

🔄 ROLLBACK IN PROGRESS

Environment: Production
From: v1.3.0
To: v1.2.3
Reason: Critical bug in checkout flow
Status: In progress
ETA: 5 minutes

Updates: #incidents

常见问题

问题：回滚失败

症状： 部署无法启动
解决方法： 检查日志，确认版本存在，并确保密钥有效

问题：数据库不兼容

症状： 应用已启动，但无法读取数据
解决方法： 可能需要先回滚迁移

问题：流量未路由

症状： 用户仍看到新版本
解决方法： 清除 CDN 缓存，检查负载均衡器配置

## Health Check Script

```bash
#!/bin/bash
# scripts/health-check.sh

ENVIRONMENT=$1
BASE_URL="https://${ENVIRONMENT}.myapp.com"

echo "Running health checks for $ENVIRONMENT..."

# API health
if ! curl -f "$BASE_URL/api/health" > /dev/null 2>&1; then
  echo "❌ API health check failed"
  exit 1
fi

# Database connection
if ! curl -f "$BASE_URL/api/health/db" > /dev/null 2>&1; then
  echo "❌ Database health check failed"
  exit 1
fi

# Key endpoints
ENDPOINTS=("/api/users" "/api/products" "/api/orders")
for endpoint in "${ENDPOINTS[@]}"; do
  if ! curl -f "$BASE_URL$endpoint" > /dev/null 2>&1; then
    echo "❌ Endpoint $endpoint health check failed"
    exit 1
  fi
done

echo "✅ All health checks passed"
exit 0

最佳实践

1. 快速回滚：在 <5 分钟内回退到上一个版本
2. 自动化：一键回滚工作流
3. 经过验证：回滚后执行健康检查
4. 有文档记录：提供清晰的操作手册
5. 经过测试：定期演练回滚
6. 受到监控：发生故障时发出警报
7. 及时沟通：通知相关方

输出检查清单

• 手动回滚工作流
• 失败时自动回滚
• 特定平台的回滚（K8s/Docker）
• 数据库回滚流程
• 已编写回滚操作手册
• 健康检查脚本
• 沟通模板
• 事件问题自动化处理