postmortem-writing

モード：認知／プロンプト駆動型 — 単独で実行できるユーティリティスクリプトはありません。エージェントのコンテキストで使用します。

ポストモーテムの作成

組織の学習を促進し、インシデントの再発を防ぐ、効果的で誰も責めないポストモーテムを作成するための包括的なガイドです。

このスキルを使用する場面

• インシデント発生後のレビューを実施する
• ポストモーテム文書を作成する
• 誰も責めないポストモーテム会議を進行する
• 根本原因と寄与要因を特定する
• 実行可能なフォローアップ項目を作成する
• 組織的な学習文化を育む

基本概念

1. 誰も責めない文化

責任追及型	誰も責めない文化
「誰がこれを引き起こしたのか？」	「どのような条件がこれを可能にしたのか？」
「誰かがミスをした」	「システムがこのミスを可能にした」
個人を罰する	システムを改善する
情報を隠す	学びを共有する
発言することへの恐れ	心理的安全性

2. ポストモーテムを実施する条件

• SEV1 または SEV2 のインシデント
• 顧客向けサービスの停止が 15 分を超える場合
• データ損失またはセキュリティインシデント
• 重大な事態になり得たニアミス
• 新しい障害モード
• 異例の介入を必要としたインシデント

クイックスタート

ポストモーテムのタイムライン

Day 0: Incident occurs
Day 1-2: Draft postmortem document
Day 3-5: Postmortem meeting
Day 5-7: Finalize document, create tickets
Week 2+: Action item completion
Quarterly: Review patterns across incidents

テンプレート

テンプレート 1：標準的なポストモーテム

# Postmortem: [Incident Title]

**Date**: 2024-01-15
**Authors**: @alice, @bob
**Status**: Draft | In Review | Final
**Incident Severity**: SEV2
**Incident Duration**: 47 minutes

## Executive Summary

On January 15, 2024, the payment processing service experienced a 47-minute outage affecting approximately 12,000 customers. The root cause was a database connection pool exhaustion triggered by a configuration change in deployment v2.3.4. The incident was resolved by rolling back to v2.3.3 and increasing connection pool limits.

**Impact**:

- 12,000 customers unable to complete purchases
- Estimated revenue loss: $45,000
- 847 support tickets created
- No data loss or security implications

## Timeline (All times UTC)

| Time  | Event                                           |
| ----- | ----------------------------------------------- |
| 14:23 | Deployment v2.3.4 completed to production       |
| 14:31 | First alert: `payment_error_rate > 5%`          |
| 14:33 | On-call engineer @alice acknowledges alert      |
| 14:35 | Initial investigation begins, error rate at 23% |
| 14:41 | Incident declared SEV2, @bob joins              |
| 14:45 | Database connection exhaustion identified       |
| 14:52 | Decision to rollback deployment                 |
| 14:58 | Rollback to v2.3.3 initiated                    |
| 15:10 | Rollback complete, error rate dropping          |
| 15:18 | Service fully recovered, incident resolved      |

## Root Cause Analysis

### What Happened

The v2.3.4 deployment included a change to the database query pattern that inadvertently removed connection pooling for a frequently-called endpoint. Each request opened a new database connection instead of reusing pooled connections.

### Why It Happened

1. **Proximate Cause**: Code change in `PaymentRepository.java` replaced pooled `DataSource` with direct `DriverManager.getConnection()` calls.

2. **Contributing Factors**:
   - Code review did not catch the connection handling change
   - No integration tests specifically for connection pool behavior
   - Staging environment has lower traffic, masking the issue
   - Database connection metrics alert threshold was too high (90%)

3. **5 Whys Analysis**:
   - Why did the service fail? → Database connections exhausted
   - Why were connections exhausted? → Each request opened new connection
   - Why did each request open new connection? → Code bypassed connection pool
   - Why did code bypass connection pool? → Developer unfamiliar with codebase patterns
   - Why was developer unfamiliar? → No documentation on connection management patterns

### System Diagram

[クライアント] → [ロードバランサー] → [決済サービス] → [データベース] ↓ コネクションプール（故障） ↓ 直接接続（原因）

## Detection

### What Worked
- Error rate alert fired within 8 minutes of deployment
- Grafana dashboard clearly showed connection spike
- On-call response was swift (2 minute acknowledgment)

### What Didn't Work
- Database connection metric alert threshold too high
- No deployment-correlated alerting
- Canary deployment would have caught this earlier

### Detection Gap
The deployment completed at 14:23, but the first alert didn't fire until 14:31 (8 minutes). A deployment-aware alert could have detected the issue faster.

## Response

### What Worked
- On-call engineer quickly identified database as the issue
- Rollback decision was made decisively
- Clear communication in incident channel

### What Could Be Improved
- Took 10 minutes to correlate issue with recent deployment
- Had to manually check deployment history
- Rollback took 12 minutes (could be faster)

## Impact

### Customer Impact
- 12,000 unique customers affected
- Average impact duration: 35 minutes
- 847 support tickets (23% of affected users)
- Customer satisfaction score dropped 12 points

### Business Impact
- Estimated revenue loss: $45,000
- Support cost: ~$2,500 (agent time)
- Engineering time: ~8 person-hours

### Technical Impact
- Database primary experienced elevated load
- Some replica lag during incident
- No permanent damage to systems

## Lessons Learned

### What Went Well
1. Alerting detected the issue before customer reports
2. Team collaborated effectively under pressure
3. Rollback procedure worked smoothly
4. Communication was clear and timely

### What Went Wrong
1. Code review missed critical change
2. Test coverage gap for connection pooling
3. Staging environment doesn't reflect production traffic
4. Alert thresholds were not tuned properly

### Where We Got Lucky
1. Incident occurred during business hours with full team available
2. Database handled the load without failing completely
3. No other incidents occurred simultaneously

## Action Items

| Priority | Action | Owner | Due Date | Ticket |
|----------|--------|-------|----------|--------|
| P0 | Add integration test for connection pool behavior | @alice | 2024-01-22 | ENG-1234 |
| P0 | Lower database connection alert threshold to 70% | @bob | 2024-01-17 | OPS-567 |
| P1 | Document connection management patterns | @alice | 2024-01-29 | DOC-89 |
| P1 | Implement deployment-correlated alerting | @bob | 2024-02-05 | OPS-568 |
| P2 | Evaluate canary deployment strategy | @charlie | 2024-02-15 | ENG-1235 |
| P2 | Load test staging with production-like traffic | @dave | 2024-02-28 | QA-123 |

## Appendix

### Supporting Data

#### Error Rate Graph
[Link to Grafana dashboard snapshot]

#### Database Connection Graph
[Link to metrics]

### Related Incidents
- 2023-11-02: Similar connection issue in User Service (POSTMORTEM-42)

### References
- [Connection Pool Best Practices](internal-wiki/connection-pools)
- [Deployment Runbook](internal-wiki/deployment-runbook)

テンプレート 2：5 Whys 分析

# 5 Whys Analysis: [Incident]

## Problem Statement

Payment service experienced 47-minute outage due to database connection exhaustion.

## Analysis

### Why #1: Why did the service fail?

**Answer**: Database connections were exhausted, causing all new requests to fail.

**Evidence**: Metrics showed connection count at 100/100 (max), with 500+ pending requests.

---

### Why #2: Why were database connections exhausted?

**Answer**: Each incoming request opened a new database connection instead of using the connection pool.

**Evidence**: Code diff shows direct `DriverManager.getConnection()` instead of pooled `DataSource`.

---

### Why #3: Why did the code bypass the connection pool?

**Answer**: A developer refactored the repository class and inadvertently changed the connection acquisition method.

**Evidence**: PR #1234 shows the change, made while fixing a different bug.

---

### Why #4: Why wasn't this caught in code review?

**Answer**: The reviewer focused on the functional change (the bug fix) and didn't notice the infrastructure change.

**Evidence**: Review comments only discuss business logic.

---

### Why #5: Why isn't there a safety net for this type of change?

**Answer**: We lack automated tests that verify connection pool behavior and lack documentation about our connection patterns.

**Evidence**: Test suite has no tests for connection handling; wiki has no article on database connections.

## Root Causes Identified

1. **Primary**: Missing automated tests for infrastructure behavior
2. **Secondary**: Insufficient documentation of architectural patterns
3. **Tertiary**: Code review checklist doesn't include infrastructure considerations

## Systemic Improvements

| Root Cause    | Improvement                       | Type       |
| ------------- | --------------------------------- | ---------- |
| Missing tests | Add infrastructure behavior tests | Prevention |
| Missing docs  | Document connection patterns      | Prevention |
| Review gaps   | Update review checklist           | Detection  |
| No canary     | Implement canary deployments      | Mitigation |

テンプレート 3：簡易ポストモーテム（軽微なインシデント）

# Quick Postmortem: [Brief Title]

**Date**: 2024-01-15 | **Duration**: 12 min | **Severity**: SEV3

## What Happened

API latency spiked to 5s due to cache miss storm after cache flush.

## Timeline

- 10:00 - Cache flush initiated for config update
- 10:02 - Latency alerts fire
- 10:05 - Identified as cache miss storm
- 10:08 - Enabled cache warming
- 10:12 - Latency normalized

## Root Cause

Full cache flush for minor config update caused thundering herd.

## Fix

- Immediate: Enabled cache warming
- Long-term: Implement partial cache invalidation (ENG-999)

## Lessons

Don't full-flush cache in production; use targeted invalidation.

ファシリテーションガイド

ポストモーテム会議の進行

## Meeting Structure (60 minutes)

### 1. Opening (5 min)

- Remind everyone of blameless culture
- "We're here to learn, not to blame"
- Review meeting norms

### 2. Timeline Review (15 min)

- Walk through events chronologically
- Ask clarifying questions
- Identify gaps in timeline

### 3. Analysis Discussion (20 min)

- What failed?
- Why did it fail?
- What conditions allowed this?
- What would have prevented it?

### 4. Action Items (15 min)

- Brainstorm improvements
- Prioritize by impact and effort
- Assign owners and due dates

### 5. Closing (5 min)

- Summarize key learnings
- Confirm action item owners
- Schedule follow-up if needed

## Facilitation Tips

- Keep discussion on track
- Redirect blame to systems
- Encourage quiet participants
- Document dissenting views
- Time-box tangents

絶対に守るべき原則

1. 必ずインシデントの解決から 48 時間以内にポストモーテムを作成してください。記憶は薄れ、この時間を過ぎると運用上の背景情報が永久に失われます。
2. 決してポストモーテムで個人を責めないでください。インシデントの発生を可能にしたシステムの状態、プロセスの不備、ツールの障害に焦点を当ててください。
3. 必ず結論を出す前に、「なぜ」を最低 3 段階までたどって根本原因を追究してください。直接的な原因は症状であり、修正が必要なシステム上の根本原因ではありません。
4. 必ずすべてのアクションアイテムに、具体的な担当者、優先度、期限を設定してください。担当者や期限のないアクションアイテムが完了することはありません。
5. 決して「うまくいったこと」のセクションを省略しないでください。肯定的な行動や効果的な対応を強化することは、障害を修正することと同じくらい重要です。

避けるべきアンチパターン

アンチパターン	問題	より良いアプローチ
責任のなすり合い	学習を妨げる	システムに焦点を当てる
表面的な分析	再発を防げない	「なぜ」を 5 回問いかける
アクションアイテムがない	時間の無駄	常に具体的な次のステップを設ける
非現実的なアクション	完了しない	実行可能なタスクに範囲を絞る
フォローアップがない	アクションが忘れられる	チケット管理システムで追跡する

ベストプラクティス

やるべきこと

• すぐに始める - 記憶はすぐに薄れます
• 具体的にする - 正確な時刻、正確なエラー
• グラフを含める - 視覚的な証拠
• 担当者を割り当てる - 担当者不在のアクションアイテムを作らない
• 広く共有する - 組織的な学習につなげる

やってはいけないこと

• 名指しで非難しない - 絶対にしないでください
• 軽微なインシデントを省略しない - パターンが見えてきます
• 責任追及の文書にしない - 学習が阻害されます
• 無意味な作業を生み出さない - アクションは意味のあるものにしてください
• フォローアップを省略しない - アクションが完了したことを確認してください

参考資料

• Google SRE - ポストモーテム文化
• Etsy の誰も責めないポストモーテム
• PagerDuty ポストモーテムガイド

メモリプロトコル（必須）

開始前： .claude/context/memory/learnings.md を読みます

完了後：

• 新しいパターン -> .claude/context/memory/learnings.md
• 問題の発見 -> .claude/context/memory/issues.md
• 決定事項 -> .claude/context/memory/decisions.md

中断を想定してください：メモリに記録されていなければ、起きていないのと同じです。