system-design-generator

系统设计生成器

根据需求创建全面的系统架构方案。

系统设计文档模板

# System Design: [Feature/Product Name]

## Overview

Brief description of what we're building and why.

## Requirements

### Functional

- User can upload videos (max 1GB)
- System processes video within 5 minutes
- User receives notification when complete

### Non-Functional

- Handle 1000 uploads/day
- 99.9% uptime
- Process videos in <5 minutes (p95)
- Cost: <$0.50 per video

## High-Level Architecture

┌─────────┐ ┌──────────┐ ┌─────────────┐ │ 客户端 │─────▶│ API │─────▶│ 上传 │ │ │ │ 网关 │ │ 服务 │ └─────────┘ └──────────┘ └─────────────┘ │ ▼ ┌─────────────┐ │ 存储 │ │ (S3) │ └─────────────┘ │ ▼ ┌─────────────┐ │ 处理 │◀─┐ │ 队列 │ │ └─────────────┘ │ │ │ ▼ │ ┌─────────────┐ │ │ 处理器 │─┘ │ 工作进程 │ └─────────────┘ │ ▼ ┌─────────────┐ │通知 │ │ 服务 │ └─────────────┘

## Components

### 1. API Gateway
**Responsibilities:**
- Authentication
- Rate limiting
- Request routing

**Technology:** Kong/AWS API Gateway
**Scaling:** Auto-scale based on requests/sec

### 2. Upload Service
**Responsibilities:**
- Generate pre-signed S3 URLs
- Validate file metadata
- Enqueue processing jobs

**API:**

POST /uploads Request: { filename, size, content_type } Response: { upload_url, upload_id }

**Technology:** Node.js + Express
**Scaling:** Horizontal (stateless)

### 3. Storage (S3)
**Responsibilities:**
- Store raw videos
- Store processed outputs
- Serve content via CDN

**Structure:**

/uploads/{user_id}/{upload_id}/original.mp4 /processed/{user_id}/{upload_id}/output.mp4

### 4. Processing Queue
**Responsibilities:**
- Buffer processing jobs
- Ensure at-least-once delivery
- DLQ for failed jobs

**Technology:** AWS SQS
**Configuration:**
- Visibility timeout: 15 minutes
- DLQ after 3 retries

### 5. Processor Workers
**Responsibilities:**
- Transcode videos
- Generate thumbnails
- Update database

**Technology:** Python + FFmpeg
**Scaling:** Auto-scale on queue depth

## Data Flow

### Upload Flow
1. Client requests upload URL from Upload Service
2. Upload Service generates pre-signed S3 URL
3. Client uploads directly to S3
4. Client notifies Upload Service of completion
5. Upload Service enqueues processing job
6. Returns upload_id to client

### Processing Flow
1. Worker polls queue for jobs
2. Downloads video from S3
3. Processes video (transcode, thumbnail)
4. Uploads results to S3
5. Updates database status
6. Sends notification
7. Deletes message from queue

## Data Model

```typescript
interface Upload {
  id: string;
  user_id: string;
  filename: string;
  size: number;
  status: 'pending' | 'processing' | 'complete' | 'failed';
  original_url: string;
  processed_url?: string;
  created_at: Date;
  processed_at?: Date;
}

interface ProcessingJob {
  upload_id: string;
  attempts: number;
  error?: string;
}

API 契约

上传端点

POST   /uploads           - Request upload URL
GET    /uploads/:id       - Get upload status
DELETE /uploads/:id       - Cancel upload
GET    /uploads           - List user uploads

Webhook

POST {webhook_url}
{
  "event": "upload.completed",
  "upload_id": "...",
  "status": "complete",
  "processed_url": "..."
}

扩展性考量

当前容量

• 每天上传 1000 次 = 每分钟约 1 次
• 单个工作进程每 5 分钟可处理 1 个视频
• 当前负载需要 5 个工作进程

扩展 10 倍（每天 10,000 次）

• 每分钟约上传 10 次
• 需要 50 个工作进程
• 使用竞价实例以节省成本
• 为状态检查添加 Redis 缓存

扩展 100 倍（每天 100,000 次）

• 每分钟约上传 100 次
• 按区域分区
• 使用 Kafka 代替 SQS
• 按 user_id 对数据库进行分片

故障模式

S3 不可用

• 影响：上传失败
• 缓解措施：多区域 S3 复制

队列积压

• 影响：处理延迟
• 缓解措施：更快地自动扩展工作进程

工作进程在处理期间崩溃

• 影响：任务会重试
• 缓解措施：幂等处理

成本估算

每月（每天 1000 次上传）：

• S3 存储：$50
• S3 传输：$100
• SQS：$10
• 工作进程（EC2）：$300
• 数据库：$100 总计：约 $560/月

安全

• 预签名 URL 在 1 小时后过期
• 视频存储在私有 S3 存储桶中
• 使用 CloudFront 签名 URL 进行分发
• 按用户进行速率限制

监控

指标：

• 上传成功率
• 处理时间（p50、p95、p99）
• 队列深度
• 工作进程 CPU/内存
• 按类型统计的错误率

告警：

• 队列深度 >1000
• 处理时间 p95 >10 分钟
• 错误率 >5%

未决问题

• 视频保留策略？（30 天？1 年？）
• 视频时长上限？（影响处理时间）
• 区域数据驻留要求？

## Component Template

```markdown
### Component Name

**Responsibilities:**
- Primary responsibility
- Secondary responsibility

**Technology Stack:**
- Language: [Python/Node/Go]
- Framework: [Express/FastAPI/Gin]
- Database: [PostgreSQL/MongoDB]

**API/Interface:**
```typescript
interface ComponentAPI {
  method(params): ReturnType;
}

扩展策略：

• 水平扩展：无状态、负载均衡
• 垂直扩展：缓存层、连接池

依赖项：

• 服务 A（用于 X）
• 数据库 B（用于持久化）

故障处理：

• 使用指数退避重试
• 为下游服务设置断路器
• 回退到缓存数据

## Best Practices

1. **Start with requirements**: Functional + non-functional
2. **Draw diagrams first**: Visual clarity
3. **Define boundaries**: What's in scope vs out
4. **Document tradeoffs**: Every choice has costs
5. **Plan for failure**: What breaks and how to handle
6. **Consider scale**: Current, 10x, 100x
7. **Estimate costs**: Build vs buy decisions
8. **Leave open questions**: Don't pretend to know everything

## Output Checklist

- [ ] Requirements documented (functional + non-functional)
- [ ] High-level architecture diagram
- [ ] Component breakdown (3-7 components)
- [ ] Data flow documented
- [ ] Data model defined
- [ ] API contracts specified
- [ ] Scaling considerations (1x, 10x, 100x)
- [ ] Failure modes identified
- [ ] Cost estimate provided
- [ ] Security considerations
- [ ] Monitoring plan