troubleshooting-guide

故障排除指南编写器

擅长编写结构化故障诊断与排除指南。

核心原则

以问题为中心的结构

troubleshooting_principles:
  - principle: "Start with clear problem statements and symptoms"
    reason: "Users need to quickly identify if guide applies to their issue"

  - principle: "Use If-Then logic flows for decision trees"
    reason: "Systematic elimination of possible causes"

  - principle: "Organize solutions by likelihood and impact"
    reason: "Try simple/common fixes first, escalate to complex"

  - principle: "Follow logical diagnostic sequence (simple to complex)"
    reason: "Minimize time to resolution"

  - principle: "Include verification steps after each fix"
    reason: "Confirm the issue is actually resolved"

  - principle: "Provide rollback instructions"
    reason: "Allow safe recovery if fix causes new issues"

关注用户体验

• 面向目标受众的水平进行编写
• 使用一致的格式
• 为每个步骤标注预计耗时
• 尽可能加入截图和示例

---

标准指南模板

# Troubleshooting: [Problem Title]

**Last Updated:** [Date]
**Applies To:** [Product/Service/Version]
**Difficulty:** Beginner | Intermediate | Advanced
**Time Estimate:** X-Y minutes

---

## Problem Statement

### Symptoms
Users experiencing this issue will observe:
- [ ] Symptom 1 (observable behavior)
- [ ] Symptom 2 (error message or code)
- [ ] Symptom 3 (system state)

### Error Messages

[准确的错误消息或代码]

### Affected Components
- Component A
- Component B

### Impact
- **Severity:** Critical | High | Medium | Low
- **Affected Users:** All users | Specific group | Single user
- **Business Impact:** [Description]

---

## Quick Checks (2-5 minutes)

Before diving into detailed troubleshooting, verify these common causes:

### Check 1: [Most Common Cause]
**Time:** 30 seconds

```bash
# Command to verify
[diagnostic command]

预期结果：[应看到的内容] **如果此步骤失败：**继续检查第 2 项

检查 2：[第二常见原因]

**耗时：**1 分钟

[验证步骤]

---

诊断步骤

步骤 1：收集信息

继续之前，请收集以下信息：

• 来自[位置]的错误日志
• 来自[位置]的系统配置
• 触发问题的用户操作

# Commands to gather diagnostic info
[command 1]
[command 2]

步骤 2：确定根本原因

使用此决策树确定原因：

Start
  │
  ├─ Is [condition A] true?
  │    ├─ YES → Go to Solution A
  │    └─ NO → Continue
  │
  ├─ Is [condition B] true?
  │    ├─ YES → Go to Solution B
  │    └─ NO → Continue
  │
  └─ None of the above → Escalate to Support

---

解决方案

解决方案 A：[修复名称]

**难度：**简单 **耗时：**5 分钟 **风险：**低

前提条件

• [前提条件 1]
• [前提条件 2]

步骤

1. 步骤 1 标题

   [command]
   

   预期结果：[说明]

2. 步骤 2 标题 [说明]

3. 步骤 3 标题 [说明]

验证修复

[verification command]

成功标志：[需要留意的内容]

回滚（如有需要）

[rollback command]

---

解决方案 B：[修复名称]

**难度：**中等 **耗时：**15 分钟 **风险：**中等

[与解决方案 A 相同的结构]

---

预防措施

为防止此问题再次发生：

1. **监控：**为[指标]设置警报
2. **配置：**确保正确配置[设置]
3. **流程：**进行更改时遵循[流程]
4. **培训：**向团队普及[最佳实践]

---

升级处理

如果以上解决方案无法解决问题：

需要升级处理的情况

• 尝试所有解决方案后问题仍然存在
• 发现数据丢失或安全问题
• 多名用户同时受到影响

需要提供的信息

• 问题开始的时间
• 已尝试的步骤
• 收集到的诊断日志
• 业务影响评估

联系方式

• 支持团队：[联系信息]
• 升级路径：[应联系的对象]
• SLA：[预计响应时间]

---

相关资源

• [相关指南链接]
• [文档链接]
• [常见问题链接]

---

修订历史

日期	作者	变更
[日期]	[姓名]	初始版本
[日期]	[姓名]	添加解决方案 C

---

## Diagnostic Patterns

### Layer-by-Layer Approach

```markdown
## Network Connectivity Troubleshooting

### Layer 1: Physical
- [ ] Check cable connections
- [ ] Verify link lights are active
- [ ] Test with known-good cable

### Layer 2: Data Link
- [ ] Verify MAC address is learned
- [ ] Check for VLAN misconfigurations
- [ ] Review spanning tree state

### Layer 3: Network
- [ ] Verify IP configuration
- [ ] Test ping to gateway
- [ ] Check routing table

### Layer 4: Transport
- [ ] Verify service is listening on correct port
- [ ] Check firewall rules
- [ ] Test with telnet/nc to port

### Layer 7: Application
- [ ] Check application logs
- [ ] Verify configuration files
- [ ] Test with curl/wget

二分排查法

## Identifying Faulty Component

Use binary search to isolate the issue:

### Step 1: Test Midpoint
Test the system at the midpoint of the data flow:

[客户端] → [负载均衡器] → [应用服务器] → [数据库] ↑ 首先在此处测试

**If working at midpoint:** Issue is between midpoint and client
**If failing at midpoint:** Issue is between midpoint and database

### Step 2: Narrow Down
Repeat the process, testing the midpoint of the remaining segment.

### Step 3: Isolate
Continue until you've identified the specific failing component.

基于症状的决策树

## Application Not Responding

┌─ 是否完全无法连接到服务器？ │ ├─ 否 → 网络/DNS 问题 │ └─ 请参阅：网络故障排除指南 │ └─ 是 → 继续 │ ├─ 服务端口是否有响应？ │ ├─ 否 → 服务未运行 │ └─ 请参阅：服务重启流程 │ └─ 是 → 继续 │ ├─ 应用程序日志中是否有错误？ │ ├─ 是 → 应用程序错误 │ └─ 请参阅：日志分析指南 │ └─ 否 → 资源耗尽 └─ 请参阅：性能故障排除指南

---

日志分析指南

常见日志位置

linux_logs:
  system:
    - /var/log/syslog
    - /var/log/messages
    - journalctl -xe

  application:
    - /var/log/[app-name]/
    - ~/.pm2/logs/
    - docker logs [container]

  web_server:
    nginx:
      - /var/log/nginx/error.log
      - /var/log/nginx/access.log
    apache:
      - /var/log/apache2/error.log
      - /var/log/httpd/error_log

  database:
    postgresql:
      - /var/log/postgresql/
    mysql:
      - /var/log/mysql/error.log

日志分析命令

# Find errors in last 100 lines
tail -100 /var/log/app.log | grep -i error

# Find errors with timestamp
grep -i error /var/log/app.log | tail -50

# Watch log in real-time
tail -f /var/log/app.log | grep --line-buffered -i error

# Count errors by type
grep -i error /var/log/app.log | sort | uniq -c | sort -rn | head -20

# Find entries around specific time
awk '/2024-01-15 14:3[0-5]/' /var/log/app.log

# Extract specific fields (JSON logs)
cat /var/log/app.json | jq 'select(.level == "error") | {time, message}'

# Search compressed logs
zgrep -i error /var/log/app.log.*.gz

错误模式识别

## Common Error Patterns

### Connection Errors

模式：“Connection refused” | “ECONNREFUSED” | “Connection timed out” 原因：服务未运行或防火墙阻止了连接 修复：检查服务状态，确认端口，检查防火墙规则

### Memory Errors

模式：“Out of memory” | “OOM” | “Cannot allocate memory” 原因：进程耗尽了可用 RAM 修复：增加内存，优化应用程序，添加交换空间

### Disk Errors

模式：“No space left on device” | “ENOSPC” | “Disk full” 原因：文件系统空间已用满 修复：清理旧文件，扩展磁盘，启用日志轮转

### Permission Errors

模式：“Permission denied” | “EACCES” | “Operation not permitted” 原因：文件／目录权限不足 修复：检查所有权，确认权限，检查 SELinux/AppArmor

### Database Errors

模式：“Too many connections” | “Connection pool exhausted” 原因：连接泄漏或连接池容量过小 修复：关闭未使用的连接，增大连接池，修复泄漏

---

特定问题模板

API 无响应

# Troubleshooting: API Not Responding

## Quick Diagnosis Script

```bash
#!/bin/bash
# api-health-check.sh

API_URL="${1:-http://localhost:8080}"
TIMEOUT=5

echo "=== API Health Check ==="
echo "Target: $API_URL"
echo

# 1. DNS Resolution
echo "1. DNS Resolution..."
if host=$(dig +short $(echo $API_URL | sed 's|.*://||' | cut -d'/' -f1 | cut -d':' -f1) 2>/dev/null); then
    echo "   ✅ DNS resolves to: $host"
else
    echo "   ❌ DNS resolution failed"
fi

# 2. Port Connectivity
echo "2. Port Connectivity..."
PORT=$(echo $API_URL | grep -oP ':\K[0-9]+' || echo "80")
HOST=$(echo $API_URL | sed 's|.*://||' | cut -d'/' -f1 | cut -d':' -f1)
if nc -z -w $TIMEOUT $HOST $PORT 2>/dev/null; then
    echo "   ✅ Port $PORT is open"
else
    echo "   ❌ Port $PORT is not reachable"
fi

# 3. HTTP Response
echo "3. HTTP Response..."
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout $TIMEOUT "$API_URL/health" 2>/dev/null)
if [ "$HTTP_CODE" = "200" ]; then
    echo "   ✅ Health endpoint returns 200"
elif [ -n "$HTTP_CODE" ] && [ "$HTTP_CODE" != "000" ]; then
    echo "   ⚠️  Health endpoint returns $HTTP_CODE"
else
    echo "   ❌ No HTTP response"
fi

# 4. Response Time
echo "4. Response Time..."
RESPONSE_TIME=$(curl -s -o /dev/null -w "%{time_total}" --connect-timeout $TIMEOUT "$API_URL/health" 2>/dev/null)
if (( $(echo "$RESPONSE_TIME < 1" | bc -l) )); then
    echo "   ✅ Response time: ${RESPONSE_TIME}s"
else
    echo "   ⚠️  Slow response: ${RESPONSE_TIME}s"
fi

echo
echo "=== Check Complete ==="

决策树

API Not Responding
       │
       ├─ Can you ping the server?
       │   ├─ NO → Check network/DNS
       │   └─ YES ↓
       │
       ├─ Is the service running?
       │   ├─ NO → Start/restart service
       │   └─ YES ↓
       │
       ├─ Is the port listening?
       │   ├─ NO → Check service configuration
       │   └─ YES ↓
       │
       ├─ Does health check pass?
       │   ├─ NO → Check dependencies (DB, cache)
       │   └─ YES ↓
       │
       └─ Check application logs for errors

### Database Connection Issues

```markdown
# Troubleshooting: Database Connection Failed

## Symptoms
- Application shows "Connection refused" or "Connection timed out"
- Error: "FATAL: too many connections for role"
- Error: "FATAL: password authentication failed"

## Quick Checks

### 1. Verify Database is Running
```bash
# PostgreSQL
sudo systemctl status postgresql
pg_isready -h localhost -p 5432

# MySQL
sudo systemctl status mysql
mysqladmin -u root -p ping

2. 测试连接

# PostgreSQL
psql -h localhost -U username -d database -c "SELECT 1"

# MySQL
mysql -h localhost -u username -p -e "SELECT 1"

3. 检查连接数

-- PostgreSQL
SELECT count(*) FROM pg_stat_activity;
SELECT max_connections FROM pg_settings WHERE name = 'max_connections';

-- MySQL
SHOW STATUS LIKE 'Threads_connected';
SHOW VARIABLES LIKE 'max_connections';

解决方案

解决方案 1：重启连接池

# If using PgBouncer
sudo systemctl restart pgbouncer

# Application restart
sudo systemctl restart myapp

解决方案 2：清除空闲连接

-- PostgreSQL: Kill idle connections older than 10 minutes
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
  AND state_change < NOW() - INTERVAL '10 minutes';

解决方案 3：增加最大连接数

-- PostgreSQL (requires restart)
ALTER SYSTEM SET max_connections = 200;

-- MySQL (can be done live)
SET GLOBAL max_connections = 200;

---

## Quality Assurance Checklist

### Pre-Publication Review

```markdown
## Troubleshooting Guide Quality Checklist

### Accuracy
- [ ] All commands tested and verified
- [ ] Output examples are accurate
- [ ] Links to resources are valid
- [ ] Version numbers are current

### Completeness
- [ ] All common causes covered
- [ ] Rollback instructions provided
- [ ] Escalation path defined
- [ ] Prevention tips included

### Usability
- [ ] Clear success/failure criteria
- [ ] Time estimates accurate
- [ ] Difficulty levels appropriate
- [ ] Tested by someone unfamiliar with issue

### Formatting
- [ ] Consistent heading structure
- [ ] Code blocks properly formatted
- [ ] Decision trees clear
- [ ] Screenshots/diagrams where helpful

### Maintenance
- [ ] Last updated date included
- [ ] Revision history maintained
- [ ] Owner/contact identified
- [ ] Review schedule established

---

最佳实践

1. 从症状入手——用户应能快速判断指南是否适用
2. 先提供简单的解决方案——在进行复杂诊断前，先检查明显的原因
3. 包含验证步骤——说明如何确认问题已解决
4. 记录回滚方法——如果修复无效，应提供回滚方法
5. 注明所需时间——用户应了解每个步骤需要多长时间
6. 让新手试用——指南应适用于不了解系统的用户
7. 定期更新——过时的指南比没有指南更糟
8. 提供升级处理途径——说明何时以及应联系谁