major-incident

重大事件管理

概述

此技能为在 ServiceNow 中管理优先级 1（P1）事件和重大事件提供了全面的框架。重大事件会对业务造成重大影响，因此需要立即采取协调一致的响应。

符合重大事件条件的情况：

• 服务完全中断，影响大量用户
• 关键业务功能不可用
• 发生重大安全漏洞
• 系统故障影响收入
• 高管宣布进入紧急状态

关键成果：

• 快速恢复服务
• 与利益相关者保持清晰沟通
• 协调技术响应
• 记录时间线，以供事后事件审查

前提条件

• 角色： incident_manager、major_incident_manager 或 itil
• 访问权限： 对事件、问题和任务表具有读写权限
• 权限： 能够召集技术团队
• 沟通： 能够访问沟通渠道（电子邮件、Teams、Slack）
• 相关技能： itsm/incident-lifecycle、itsm/incident-triage

流程

阶段 1：宣布重大事件

步骤 1.1：验证重大事件标准

宣布重大事件之前，请确认事件符合相关标准：

使用 MCP：

Tool: SN-Query-Table
Parameters:
  table_name: incident
  query: sys_id=[incident_sys_id]
  fields: number,short_description,priority,impact,urgency,state,business_service,cmdb_ci,assignment_group

重大事件检查清单：

• 影响：1-高（影响范围广，涉及许多用户）
• 紧急程度：1-高（对业务至关重要，时间紧迫）
• 业务服务层级：第 1 或第 2 级（关键/高）
• 没有可用的解决方法，或现有解决方法不足

步骤 1.2：宣布重大事件

使用 MCP：

Tool: SN-Update-Record
Parameters:
  table_name: incident
  sys_id: [incident_sys_id]
  data:
    priority: 1
    major_incident_state: accepted
    u_major_incident: true
    business_impact: |
      - Affected Users: [number]
      - Affected Services: [list]
      - Revenue Impact: [estimated]
      - Customer Impact: [description]

使用 REST API：

PATCH /api/now/table/incident/{sys_id}
Content-Type: application/json

{
  "priority": "1",
  "major_incident_state": "accepted",
  "u_major_incident": "true",
  "business_impact": "Affected Users: 500+\nAffected Services: CRM, Email\nRevenue Impact: $10K/hour"
}

步骤 1.3：指派事件指挥官

事件指挥官（IC）负责重大事件处理流程：

Tool: SN-Assign-Incident
Parameters:
  sys_id: [incident_sys_id]
  assigned_to: [incident_commander_username]
  assignment_group: Major Incident Management
  work_notes: |
    === MAJOR INCIDENT DECLARED ===
    Incident Commander: [Name]
    Declaration Time: [timestamp]
    Initial Assessment: [brief description]

    IC Responsibilities:
    - Own overall incident coordination
    - Manage bridge call
    - Ensure stakeholder communication
    - Drive to resolution

阶段 2：设置桥接电话会议

步骤 2.1：创建桥接电话会议任务

使用 MCP：

Tool: SN-Create-Record
Parameters:
  table_name: task
  data:
    parent: [incident_sys_id]
    short_description: "Major Incident Bridge Call - [INC#]"
    description: |
      Bridge Call Details:
      - Conference Number: [number]
      - Meeting ID: [id]
      - Passcode: [code]
      - Teams/Zoom Link: [link]

      Required Participants:
      - Incident Commander
      - Technical Lead(s)
      - Service Owner
      - Communications Lead
    assignment_group: Major Incident Management
    priority: 1

步骤 2.2：记录桥接电话会议安排

使用 MCP：

Tool: SN-Add-Work-Notes
Parameters:
  sys_id: [incident_sys_id]
  work_notes: |
    === BRIDGE CALL SCHEDULE ===
    Initial Call: [time] - Situation Assessment

    Ongoing Calls:
    - Every 30 minutes during active troubleshooting
    - Every 60 minutes during monitoring phase

    Participants Required:
    - [ ] Incident Commander: [name]
    - [ ] Technical Lead: [name]
    - [ ] Service Owner: [name]
    - [ ] Communications: [name]
    - [ ] [Application] SME: [name]
    - [ ] Network Team: [name]
    - [ ] Database Team: [name]

    Bridge Info: [conference details]

阶段 3：与利益相关者沟通

步骤 3.1：初始通知

面向客户的沟通：

Tool: SN-Add-Comment
Parameters:
  sys_id: [incident_sys_id]
  comment: |
    === MAJOR INCIDENT NOTIFICATION ===

    Status: Investigating

    IMPACT:
    [Service Name] is currently experiencing [brief description].
    Affected users: [scope]

    CURRENT ACTIONS:
    Our technical teams are actively investigating this issue.

    NEXT UPDATE:
    We will provide an update within 30 minutes or sooner if we have new information.

    WORKAROUND:
    [If available, otherwise "No workaround currently available"]

    For urgent assistance, contact: [support contact]

    Reference: [INC#]

步骤 3.2：高管通知模板

内部工作备注：

Tool: SN-Add-Work-Notes
Parameters:
  sys_id: [incident_sys_id]
  work_notes: |
    === EXECUTIVE NOTIFICATION SENT ===
    Time: [timestamp]
    Recipients: [list of executives]

    Subject: [MAJOR INCIDENT] [Service] - [Brief Issue]

    Summary:
    A major incident affecting [service/system] was declared at [time].

    Business Impact:
    - Users Affected: [number]
    - Revenue Impact: [estimate]
    - Customer Impact: [description]

    Current Status: [Investigating/Identified/Implementing Fix]

    ETA to Resolution: [estimate or "Under assessment"]

    Incident Commander: [name] ([contact])
    Next Update: [time]

步骤 3.3：定期状态更新

30 分钟更新模板：

Tool: SN-Add-Comment
Parameters:
  sys_id: [incident_sys_id]
  comment: |
    === STATUS UPDATE - [TIME] ===

    CURRENT STATUS: [Investigating/Identified/Implementing/Monitoring]

    PROGRESS SINCE LAST UPDATE:
    - [Action taken 1]
    - [Action taken 2]
    - [Finding or result]

    CURRENT FOCUS:
    [What the team is working on now]

    ETA: [Updated estimate]

    NEXT UPDATE: [time]

阶段 4：记录时间线

步骤 4.1：维护事件时间线

重要：为所有内容记录时间戳

Tool: SN-Add-Work-Notes
Parameters:
  sys_id: [incident_sys_id]
  work_notes: |
    === TIMELINE UPDATE ===

    [HH:MM] - [Event description]

    Examples:
    08:15 - First reports of service degradation received
    08:22 - Monitoring alert triggered: Database connection pool exhausted
    08:25 - Incident created and escalated to P2
    08:32 - Impact confirmed widespread - Major Incident declared
    08:35 - Bridge call initiated
    08:40 - Root cause identified: Memory leak in application server
    08:45 - Decision: Rolling restart of application cluster
    09:00 - Restart of node 1 complete
    09:15 - Restart of node 2 complete
    09:30 - Service restored - monitoring
    10:00 - Major incident closed - stable

步骤 4.2：查询时间线条目

检索时间线的所有日志条目：

Tool: SN-Query-Table
Parameters:
  table_name: sys_journal_field
  query: element_id=[incident_sys_id]^element=work_notes
  fields: value,sys_created_on,sys_created_by
  order_by: sys_created_on

使用 REST API：

GET /api/now/table/sys_journal_field?sysparm_query=element_id={sys_id}^element=work_notes&sysparm_fields=value,sys_created_on,sys_created_by&sysparm_order_by=sys_created_on

阶段 5：解决事件并恢复服务

步骤 5.1：确认服务已恢复

Tool: SN-Add-Work-Notes
Parameters:
  sys_id: [incident_sys_id]
  work_notes: |
    === SERVICE RESTORATION CONFIRMED ===
    Time: [timestamp]

    Verification Completed:
    - [ ] Core functionality verified
    - [ ] User access confirmed
    - [ ] Monitoring shows normal metrics
    - [ ] Sample transactions successful
    - [ ] Performance within acceptable range

    Verified By: [name]

    Remaining Actions:
    - Continue monitoring for [time period]
    - Schedule PIR
    - Update knowledge base

步骤 5.2：最终客户沟通

Tool: SN-Add-Comment
Parameters:
  sys_id: [incident_sys_id]
  comment: |
    === SERVICE RESTORED ===

    We are pleased to confirm that [Service Name] has been restored to normal operation.

    RESOLUTION SUMMARY:
    The issue was caused by [brief, non-technical explanation].
    Our team [brief description of fix].

    DURATION:
    Impact Start: [time]
    Service Restored: [time]
    Total Duration: [duration]

    NEXT STEPS:
    - We will conduct a Post-Incident Review
    - Improvements will be implemented to prevent recurrence

    We apologize for any inconvenience this may have caused.

    Thank you for your patience.

步骤 5.3：关闭重大事件

Tool: SN-Update-Record
Parameters:
  table_name: incident
  sys_id: [incident_sys_id]
  data:
    state: 6
    close_code: Solved (Permanently)
    resolution_code: Solved (Permanently)
    major_incident_state: closed
    close_notes: |
      RESOLUTION SUMMARY:
      Root Cause: [description]
      Resolution: [what was done]

      TIMELINE:
      Incident Start: [time]
      Major Incident Declared: [time]
      Root Cause Identified: [time]
      Service Restored: [time]

      IMPACT:
      Duration: [total time]
      Users Affected: [number]
      Business Impact: [description]

      FOLLOW-UP:
      - PIR scheduled for [date]
      - Problem ticket: [PRB#]

阶段 6：事后审查（PIR）

步骤 6.1：创建问题记录

关联到问题管理：

Tool: SN-Create-Record
Parameters:
  table_name: problem
  data:
    short_description: "PIR: [Original incident short description]"
    description: |
      Post-Incident Review for Major Incident [INC#]

      INCIDENT SUMMARY:
      [Brief description of what happened]

      IMPACT:
      - Duration: [time]
      - Users Affected: [number]
      - Services Affected: [list]
      - Business Impact: [description]

      TIMELINE:
      [Copy timeline from incident]

      INITIAL ROOT CAUSE:
      [Description]
    priority: 2
    assignment_group: [owning team]
    u_related_incident: [incident_sys_id]

步骤 6.2：PIR 会议议程

记录在工作备注中：

Tool: SN-Add-Work-Notes
Parameters:
  sys_id: [problem_sys_id]
  work_notes: |
    === POST-INCIDENT REVIEW AGENDA ===

    Meeting Scheduled: [date/time]
    Attendees: [list]

    AGENDA:

    1. TIMELINE REVIEW (10 min)
       - Walk through event sequence
       - Identify detection gaps

    2. ROOT CAUSE ANALYSIS (20 min)
       - Technical root cause
       - Contributing factors
       - 5-Whys analysis

    3. RESPONSE EVALUATION (15 min)
       - What went well?
       - What could be improved?
       - Communication effectiveness

    4. ACTION ITEMS (15 min)
       - Prevention measures
       - Detection improvements
       - Process improvements
       - Assign owners and due dates

    5. FOLLOW-UP (5 min)
       - Schedule follow-up if needed
       - Documentation requirements

步骤 6.3：记录 PIR 发现

Tool: SN-Update-Record
Parameters:
  table_name: problem
  sys_id: [problem_sys_id]
  data:
    root_cause: |
      ROOT CAUSE:
      [Technical description of what caused the incident]

      5-WHYS ANALYSIS:
      1. Why did the service fail?
         [Answer]
      2. Why did [answer 1] happen?
         [Answer]
      3. Why did [answer 2] happen?
         [Answer]
      4. Why did [answer 3] happen?
         [Answer]
      5. Why did [answer 4] happen?
         [Answer - usually organizational or process issue]

      CONTRIBUTING FACTORS:
      - [Factor 1]
      - [Factor 2]

    fix: |
      IMMEDIATE FIX:
      [What was done to restore service]

      PERMANENT FIX:
      [What changes will prevent recurrence]

      ACTION ITEMS:
      1. [Action] - Owner: [name] - Due: [date]
      2. [Action] - Owner: [name] - Due: [date]
      3. [Action] - Owner: [name] - Due: [date]

事件指挥官职责

角色定义

事件指挥官是重大事件期间唯一的责任人：

职责	说明
协调	负责整体响应流程
沟通	确保及时向利益相关者通报进展
决策	快速作出或上报关键决策
文档记录	确保持续维护时间线
资源分配	请求并分配技术资源
升级处理	向管理层上报阻碍因素

IC 检查清单

MAJOR INCIDENT IC CHECKLIST

IMMEDIATE (First 15 minutes):
- [ ] Confirm major incident declaration
- [ ] Establish bridge call
- [ ] Identify technical lead
- [ ] Send initial stakeholder notification
- [ ] Assign scribe for timeline

ONGOING (Every 30 minutes):
- [ ] Status update to bridge call
- [ ] Customer communication posted
- [ ] Executive update (if needed)
- [ ] Timeline reviewed and updated
- [ ] Resource needs assessed

RESOLUTION:
- [ ] Service restoration verified
- [ ] Final customer communication
- [ ] Close major incident record
- [ ] Schedule PIR
- [ ] Create problem record

POST-INCIDENT:
- [ ] Conduct PIR meeting
- [ ] Document findings
- [ ] Track action items
- [ ] Share lessons learned

工具使用摘要

MCP 工具

工具	用途	阶段
SN-Create-Incident	如果重大事件尚不存在，则创建重大事件	1
SN-Update-Record	更新事件状态、优先级和解决方案	1、5、6
SN-Add-Work-Notes	内部时间线和文档记录	全部
SN-Add-Comment	面向客户的沟通	3、5
SN-Query-Table	检索事件详情和时间线	全部
SN-Assign-Incident	指派事件指挥官	1
SN-Natural-Language-Search	查找相关事件	1

REST API 端点

端点	方法	用途
/api/now/table/incident	POST	创建事件
/api/now/table/incident/{sys_id}	PATCH	更新事件
/api/now/table/incident/{sys_id}	GET	获取事件详情
/api/now/table/sys_journal_field	GET	检索时间线条目
/api/now/table/problem	POST	为 PIR 创建问题
/api/now/table/task	POST	创建桥接电话任务

最佳实践

• 速度优先于完美： 发生重大事件时，快速行动胜过完美行动
• 及早沟通，频繁沟通： 利益相关者更喜欢频繁更新，即使内容只是“仍在调查”
• 明确责任人： 由一名事件指挥官负责，权责明确
• 无责文化： 专注于解决问题，而非追究责任；将分析留到 PIR 再进行
• 记录一切： 在工作备注中记录每项操作和决策，并注明时间戳
• 将技术工作与沟通分开： 让技术团队专注于修复，由 IC 负责沟通
• 预先定义升级路径： 提前了解需要联系的人员
• 与 ITIL 保持一致： 重大事件管理是 ITIL 4 的一项关键实践

故障排除

桥接电话无法启动

症状： 技术团队未加入桥接电话 原因： 沟通不清楚或会议详情有误 解决方案： 使用正确的详情重新发送通知；通过多个渠道（电子邮件、Teams、SMS）联系

利益相关者要求过于频繁地更新

症状： IC 花在更新上的时间超过协调工作 原因： 利益相关者感到焦虑，更新计划不明确 解决方案： 提前设定明确的更新频率；将沟通工作委派给专人

时间线存在缺漏

症状： PIR 发现事件经过信息缺失 原因： 没有专职记录员，或忙于处理紧急问题 解决方案： 始终安排专职记录员；尽可能使用自动时间线捕获

对解决状态存在争议

症状： 用户报告称“恢复”后服务仍受影响 原因： 验证不完整，或用户体验存在差异 解决方案： 明确定义恢复标准；从用户角度进行测试；使用监控数据

沟通模板

状态页更新模板

[SERVICE NAME] - [STATUS]

Current Status: [Investigating/Identified/Monitoring/Resolved]

[timestamp] - [Brief update]

Affected Components:
- [Component 1]
- [Component 2]

More information will be provided as it becomes available.

电子邮件模板

初始通知：

Subject: [MAJOR INCIDENT] [Service] - Service Disruption

A major incident affecting [Service] has been declared.

Impact: [Brief description]
Status: Investigating
Incident Commander: [Name]

Updates will be provided every 30 minutes.

Bridge Call: [details]

解决通知：

Subject: [RESOLVED] [Service] - Service Restored

The major incident affecting [Service] has been resolved.

Duration: [start time] - [end time]
Root Cause: [Brief, non-technical explanation]
Resolution: [What was done]

A Post-Incident Review will be conducted and findings shared.

We apologize for any inconvenience.

相关技能

• itsm/incident-lifecycle - 标准事件管理
• itsm/incident-triage - 事件分诊流程
• itsm/problem-analysis - 根本原因分析
• itsm/change-management - 重大事件期间的紧急变更
• development/notifications - 自动通知

参考资料

• ServiceNow 重大事件管理
• ITIL 4 事件管理实践
• Google SRE 手册——事件管理
• PagerDuty 事件响应指南