it-operations

IT 运维专家

一套全面的技能，用于管理 IT 基础设施运维、确保服务可靠性、实施监控与告警策略、管理事件，并通过自动化和最佳实践保持卓越运维。

核心原则

1. 服务可靠性优先

• 主动监控：在事件发生前实施全面的可观测性
• 事件管理：结构化的响应流程与清晰的升级路径
• SLA/SLO 管理：定义并维护与业务需求一致的服务级别目标
• 持续改进：通过无责事后复盘从事件中学习

2. 自动化优先于手动流程

• 基础设施即代码：通过版本控制的代码管理基础设施配置
• Runbook 自动化：将手动流程转化为自动化工作流
• 自愈系统：针对常见问题实施自动修复
• 配置管理：保持各环境之间的一致性

3. ITIL 服务管理

• 服务战略：使 IT 服务与业务目标保持一致
• 服务设计：设计具有韧性、可扩展的服务
• 服务转换：以最小干扰管理变更
• 服务运营：有效交付并支持服务
• 持续服务改进：迭代提升服务质量

4. 卓越运维

• 文档：维护最新的 runbook、流程和架构图
• 知识管理：基于事件解决方案构建可搜索的知识库
• 容量规划：主动预测并配置资源
• 成本优化：在性能需求与基础设施成本之间取得平衡

核心工作流

基础设施运维工作流

1. MONITORING & OBSERVABILITY
   ├─ Define SLIs/SLOs/SLAs for critical services
   ├─ Implement metrics collection (infrastructure, application, business)
   ├─ Configure alerting with proper thresholds and escalation
   ├─ Build dashboards for different audiences (ops, devs, executives)
   └─ Establish on-call rotation and escalation procedures

2. INCIDENT MANAGEMENT
   ├─ Receive alert or user report
   ├─ Assess severity and impact (P1/P2/P3/P4)
   ├─ Engage appropriate responders
   ├─ Investigate and diagnose root cause
   ├─ Implement fix or workaround
   ├─ Communicate status to stakeholders
   ├─ Document resolution in knowledge base
   └─ Conduct post-incident review

3. CHANGE MANAGEMENT
   ├─ Submit change request with impact assessment
   ├─ Review and approve through CAB (Change Advisory Board)
   ├─ Schedule change window
   ├─ Execute change with rollback plan ready
   ├─ Validate success criteria
   ├─ Document actual vs planned results
   └─ Close change ticket

4. CAPACITY PLANNING
   ├─ Collect resource utilization trends
   ├─ Analyze growth patterns
   ├─ Forecast future requirements
   ├─ Plan procurement or provisioning
   ├─ Execute capacity additions
   └─ Monitor effectiveness

5. AUTOMATION & OPTIMIZATION
   ├─ Identify repetitive manual tasks
   ├─ Document current process
   ├─ Design automated solution
   ├─ Implement and test automation
   ├─ Deploy to production
   ├─ Measure time/cost savings
   └─ Iterate and improve

决策框架

告警配置决策矩阵

场景	告警类型	阈值	响应时间	升级
服务完全宕机	呼叫	立即	< 5 分钟	立即升级至值班人员
服务降级	呼叫	2-3 次失败	< 15 分钟	15 分钟后升级至值班人员
资源使用率过高	警告	持续 > 80%	< 1 小时	2 小时后升级至团队负责人
接近容量上限	信息	趋势 > 70%	< 24 小时	每周容量评审
配置漂移	工单	任何偏差	< 7 天	每月评审

事件严重程度分级

优先级 1（严重）

• 影响所有用户的完全服务中断
• 数据丢失或安全漏洞
• 财务影响 > $10K/小时
• 响应：立即，7×24，全员投入

优先级 2（高）

• 影响大量用户的部分服务中断
• 显著的性能下降
• 财务影响 $1K-$10K/小时
• 响应：工作时间内 < 30 分钟

优先级 3（中）

• 影响部分用户的服务降级
• 非关键功能受损
• 有可用的临时解决方案
• 响应：工作时间内 < 4 小时

优先级 4（低）

• 影响轻微的次要问题
• 界面显示问题
• 增强请求
• 响应：下一个工作日

变更管理风险评估

Risk Level = Impact × Likelihood × Complexity

Impact (1-5):
1 = Single user
2 = Team
3 = Department
4 = Company-wide
5 = Customer-facing

Likelihood of Issues (1-5):
1 = Routine, tested
2 = Familiar, documented
3 = Some uncertainty
4 = New territory
5 = Never done before

Complexity (1-5):
1 = Single component
2 = Few components
3 = Multiple systems
4 = Cross-platform
5 = Enterprise-wide

Risk Score Interpretation:
1-20: Standard change (pre-approved)
21-50: Normal change (CAB review)
51-75: High-risk change (extensive testing, senior approval)
76-125: Emergency change only (executive approval)

监控工具选型

需求	Prometheus + Grafana	Datadog	New Relic	ELK Stack	Splunk
成本	免费（可自托管）	$$$$	$$$$	免费-$$	$$$$$
指标	优秀	优秀	优秀	良好	良好
日志	通过 Loki	优秀	优秀	优秀	优秀
链路追踪	通过 Tempo	优秀	优秀	有限	良好
学习曲线	陡峭	中等	中等	陡峭	陡峭
云原生	优秀	优秀	优秀	良好	良好
本地部署	优秀	良好	良好	优秀	优秀
APM	通过 exporter	优秀	优秀	有限	良好

常见运维挑战

挑战 1：告警疲劳

问题：过多的误报告警导致团队倦怠

解决方案：

Alert Tuning Process:
1. Measure baseline alert volume and false positive rate
2. Categorize alerts by actionability:
   - Actionable + Urgent = Keep as page
   - Actionable + Not Urgent = Ticket
   - Not Actionable = Remove or convert to dashboard metric
3. Implement alert aggregation (group similar alerts)
4. Add context to alerts (runbook links, relevant metrics)
5. Regular review meetings (weekly) to tune thresholds
6. Track metrics:
   - MTTA (Mean Time to Acknowledge): < 5 min target
   - False Positive Rate: < 20% target
   - Alert Volume per Week: Trending down

挑战 2：危机期间的事件记录

问题：团队在高压力事件期间跳过记录

解决方案：

• 指定专门的记录员角色（不是事件指挥官）
• 使用事件管理工具（PagerDuty、Opsgenie）并启用自动时间线
• 基于模板的事件报告，包含必填字段
• 自动安排事后复盘（48 小时内）
• 将记录游戏化（跟踪并表彰详尽的记录）

挑战 3：知识孤岛

问题：关键知识困在个别团队成员的大脑中

解决方案：

Knowledge Transfer Strategy:
- Pair Programming/Shadowing: 20% of sprint capacity
- Runbook Requirements: Every system must have runbook
- Lunch & Learn Sessions: Weekly 30-min knowledge sharing
- Cross-Training Matrix: Track who knows what, identify gaps
- On-Call Rotation: Everyone rotates to spread knowledge
- Post-Incident Reviews: Mandatory team sharing
- Documentation Sprints: Quarterly focus on doc completion

挑战 4：平衡稳定性与创新

问题：运维团队为维持稳定性而抵制变革

解决方案：

• 实施变更窗口（计划维护期）
• 使用蓝绿或金丝雀部署以降低风险
• 设立“创新时间”（Google 20% 时间模式）
• 创建用于实验的沙箱环境
• 同时衡量并奖励稳定性与改进指标
• 将“减少琐务”作为 OKR 目标

关键指标与 KPI

服务可靠性指标

Availability:
  Formula: (Total Time - Downtime) / Total Time × 100
  Target: 99.9% (43.8 min/month downtime)
  Measurement: Per service, monthly

MTTR (Mean Time to Recovery):
  Formula: Sum of recovery times / Number of incidents
  Target: < 30 minutes for P1, < 4 hours for P2
  Measurement: Per severity level, monthly

MTBF (Mean Time Between Failures):
  Formula: Total operational time / Number of failures
  Target: > 720 hours (30 days)
  Measurement: Per service, quarterly

MTTA (Mean Time to Acknowledge):
  Formula: Sum of acknowledgment times / Number of alerts
  Target: < 5 minutes for pages
  Measurement: Per on-call engineer, weekly

Change Success Rate:
  Formula: Successful changes / Total changes × 100
  Target: > 95%
  Measurement: Monthly

Incident Recurrence Rate:
  Formula: Repeat incidents / Total incidents × 100
  Target: < 10%
  Measurement: Quarterly (same root cause within 90 days)

运营效率指标

Toil Percentage:
  Definition: Time spent on manual, repetitive tasks
  Target: < 30% of team capacity
  Measurement: Weekly time tracking

Automation Coverage:
  Formula: Automated tasks / Total repetitive tasks × 100
  Target: > 70%
  Measurement: Quarterly audit

On-Call Load:
  Formula: Alerts per on-call shift
  Target: < 5 actionable alerts per shift
  Measurement: Per engineer, weekly

Runbook Coverage:
  Formula: Services with runbooks / Total services × 100
  Target: 100%
  Measurement: Monthly audit

Knowledge Base Utilization:
  Formula: Incidents resolved via KB / Total incidents × 100
  Target: > 40%
  Measurement: Monthly

集成点

与开发团队

• 参与设计评审以提出运维需求
• 提供部署自动化和 CI/CD 流水线支持
• 共享监控和日志需求
• 在事件响应和事后复盘中协作
• 共同负责 SLO 和错误预算

与安全团队

• 实施安全监控和告警
• 管理访问控制和身份验证系统
• 协调漏洞修补和修复
• 开展安全事件响应
• 保持符合安全策略

与业务相关方

• 报告服务可用性和性能
• 沟通计划维护窗口
• 提供容量规划预测
• 将技术指标转化为业务影响
• 参与业务连续性规划

最佳实践

1. 无指责事后复盘

Post-Incident Review Template:
- Incident Summary (what happened, when, impact)
- Timeline of Events (detailed chronology)
- Root Cause Analysis (5 Whys or Fishbone)
- What Went Well (strengths during response)
- What Could Be Improved (opportunities)
- Action Items (with owners and due dates)
- Lessons Learned (shareable insights)

Rules:
- No blame or punishment
- Focus on systems and processes, not people
- Everyone can speak freely
- Action items must be tracked to completion

2. Runbook 标准

Runbook Contents:
  - Service Overview: Purpose, dependencies, architecture
  - SLIs/SLOs/SLAs: Defined thresholds and targets
  - Common Issues: Symptoms, causes, solutions
  - Troubleshooting Steps: Step-by-step procedures
  - Escalation Paths: Who to contact and when
  - Useful Commands: Copy-paste ready commands
  - Dashboard Links: Direct links to relevant dashboards
  - Recent Changes: Link to change log
  - Contact Information: Team, product owner, SMEs

Maintenance:
  - Review quarterly or after major incidents
  - Test procedures during low-traffic periods
  - Update after every significant change
  - Track usage metrics (page views, helpfulness ratings)

3. 值班最佳实践

On-Call Preparation:
  - Laptop with VPN access
  - Mobile device with notification apps
  - Contact list (escalation paths)
  - Access to all critical systems
  - Runbooks bookmarked
  - Backup on-call identified

During On-Call:
  - Acknowledge alerts within 5 minutes
  - Update incident status regularly
  - Follow escalation procedures
  - Document all actions in incident ticket
  - Handoff clearly to next on-call

Post On-Call:
  - Complete incident reports
  - Submit toil reduction tickets
  - Provide feedback on runbooks
  - Update on-call documentation

4. 变更管理纪律

Standard Change Process:
  1. Create change request (RFC)
  2. Document:
     - What: Specific changes being made
     - Why: Business justification
     - When: Proposed date/time
     - Who: Change implementer and approver
     - How: Step-by-step procedure
     - Risk: Assessment and mitigation
     - Rollback: Detailed rollback plan
     - Testing: Validation steps
  3. Submit for CAB review (7 days advance notice)
  4. Implement during approved window
  5. Validate success criteria
  6. Close change with actual results
  7. Post-implementation review if issues occurred

Emergency Change Process:
  - Executive approval required
  - Implement with heightened monitoring
  - Full team notification
  - Complete documentation within 24 hours
  - Mandatory post-change review

参考文件

有关详细技术指导，请参阅：

• reference/monitoring.md - 可观测性、指标、告警和仪表板设计
• reference/incident-management.md - 事件响应、根因分析、事后复盘
• reference/infrastructure.md - 服务器管理、网络运营、容量规划
• reference/automation.md - 脚本编写、配置管理、编排工具
• reference/backup-recovery.md - 备份策略、灾难恢复、业务连续性

快速开始

1. 对于新基础设施：从 reference/infrastructure.md 开始获取设置指导
2. 对于监控设置：查看 reference/monitoring.md 了解可观测性策略
3. 对于事件响应：参见 reference/incident-management.md 了解流程
4. 对于自动化项目：查看 reference/automation.md 了解工具建议
5. 对于灾难恢复规划：查阅 reference/backup-recovery.md 了解恢复策略