deadlock-prevention

死锁预防

概述

两个进程永远互相等待时，就会发生死锁。

死锁会导致永久冻结——没有崩溃，没有错误，只有沉默。进程持有资源，同时等待其他进程持有的资源，从而形成循环等待。

适用场景

• 获取多个锁或资源
• 跨表／服务的嵌套事务
• 存在跨服务调用的分布式系统
• “系统在高负载下冻结”
• 没有错误，但操作始终无法完成

铁律

NEVER acquire locks in inconsistent order across the codebase.

绝无例外：

• 不能因为“这种情况很少发生”
• 不能因为“我们会添加超时”
• 不能因为“服务不同，不会冲突”
• 不能因为“只会在重负载下发生”

锁顺序不一致，必然最终导致死锁。

死锁的四个必要条件

死锁必须同时满足全部四个条件。破坏其中任意一个即可预防死锁：

┌────────────────────────────────────────────────────────────┐
│ 1. MUTUAL EXCLUSION    │ Resource can only be held by one │
│ 2. HOLD AND WAIT       │ Hold one, wait for another       │
│ 3. NO PREEMPTION       │ Can't force release of resource  │
│ 4. CIRCULAR WAIT       │ A waits for B, B waits for A     │
└────────────────────────────────────────────────────────────┘

最实用的方法：通过保持顺序一致来打破循环等待。

检测：循环等待模式

// ❌ VIOLATION: Inconsistent lock order = guaranteed deadlock

// Service A: Transfer from X to Y
async function transferAtoB(fromId: string, toId: string, amount: number) {
  await lockAccount(fromId);  // Lock A first
  await lockAccount(toId);    // Then lock B
  // transfer...
  await unlockAccount(toId);
  await unlockAccount(fromId);
}

// Service B: Transfer from Y to X  
async function transferBtoA(fromId: string, toId: string, amount: number) {
  await lockAccount(fromId);  // Lock B first (OPPOSITE ORDER!)
  await lockAccount(toId);    // Then lock A
  // transfer...
}

// Deadlock scenario:
// T1: Locks account A, waits for B
// T2: Locks account B, waits for A
// Result: Both wait forever

预防策略

1. 保持锁顺序一致（主要策略）

// ✅ CORRECT: Always acquire locks in the same order
async function transfer(
  accountId1: string, 
  accountId2: string, 
  amount: number
): Promise<void> {
  // ALWAYS lock lower ID first - creates total ordering
  const [first, second] = accountId1 < accountId2 
    ? [accountId1, accountId2] 
    : [accountId2, accountId1];
  
  const lock1 = await acquireLock(`account:${first}`);
  const lock2 = await acquireLock(`account:${second}`);
  
  try {
    await doTransfer(accountId1, accountId2, amount);
  } finally {
    await releaseLock(lock2);
    await releaseLock(lock1);  // Release in reverse order
  }
}

2. 锁超时（打破不可抢占条件）

// ✅ CORRECT: Timeout prevents permanent deadlock
async function acquireWithTimeout(
  resource: string, 
  timeoutMs: number
): Promise<Lock | null> {
  const deadline = Date.now() + timeoutMs;
  
  while (Date.now() < deadline) {
    const lock = await tryAcquireLock(resource);
    if (lock) return lock;
    
    // Backoff with jitter
    await sleep(50 + Math.random() * 50);
  }
  
  return null;  // Timeout - caller must handle
}

async function transfer(from: string, to: string, amount: number) {
  const lock1 = await acquireWithTimeout(`account:${from}`, 5000);
  if (!lock1) throw new LockTimeoutError(from);
  
  try {
    const lock2 = await acquireWithTimeout(`account:${to}`, 5000);
    if (!lock2) {
      throw new LockTimeoutError(to);  // Release lock1 in finally
    }
    
    try {
      await doTransfer(from, to, amount);
    } finally {
      await releaseLock(lock2);
    }
  } finally {
    await releaseLock(lock1);
  }
}

3. Try-Lock 并退避（打破占有并等待条件）

// ✅ CORRECT: Don't hold while waiting - release and retry
async function acquireMultipleLocks(
  resources: string[], 
  maxRetries = 10
): Promise<Lock[]> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const locks: Lock[] = [];
    let success = true;
    
    // Sort for consistent ordering
    const sorted = [...resources].sort();
    
    for (const resource of sorted) {
      const lock = await tryAcquireLock(resource);
      if (lock) {
        locks.push(lock);
      } else {
        // Can't get this lock - release all and retry
        success = false;
        break;
      }
    }
    
    if (success) {
      return locks;
    }
    
    // Release all acquired locks
    await Promise.all(locks.map(releaseLock));
    
    // Exponential backoff with jitter
    await sleep(Math.pow(2, attempt) * 10 + Math.random() * 50);
  }
  
  throw new DeadlockPreventionError('Could not acquire all locks');
}

4. 为相关资源使用单个锁

// ✅ CORRECT: One lock covers all related resources
async function transfer(from: string, to: string, amount: number) {
  // Use a higher-level lock instead of per-account locks
  const transferLock = await acquireLock('transfer-system');
  
  try {
    // No deadlock possible - only one lock
    await doTransfer(from, to, amount);
  } finally {
    await releaseLock(transferLock);
  }
}

// Trade-off: Less parallelism, but no deadlock risk
// Use when: Operations are fast, correctness > throughput

5. 锁层级

// ✅ CORRECT: Hierarchical locking - always acquire parent before child
enum LockLevel {
  SYSTEM = 0,
  REGION = 1,
  ACCOUNT = 2,
  TRANSACTION = 3,
}

class HierarchicalLock {
  private heldLocks: Map<LockLevel, Set<string>> = new Map();
  
  async acquire(level: LockLevel, resource: string): Promise<void> {
    // Validate hierarchy - can only acquire lower levels
    for (const [heldLevel] of this.heldLocks) {
      if (level <= heldLevel) {
        throw new LockHierarchyViolation(
          `Cannot acquire ${level} while holding ${heldLevel}`
        );
      }
    }
    
    await this.doAcquire(level, resource);
    
    if (!this.heldLocks.has(level)) {
      this.heldLocks.set(level, new Set());
    }
    this.heldLocks.get(level)!.add(resource);
  }
}

数据库专用模式

SQL：锁顺序

-- ❌ VIOLATION: No consistent order
-- Transaction 1
SELECT * FROM accounts WHERE id = 'A' FOR UPDATE;
SELECT * FROM accounts WHERE id = 'B' FOR UPDATE;

-- Transaction 2 (opposite order)
SELECT * FROM accounts WHERE id = 'B' FOR UPDATE;
SELECT * FROM accounts WHERE id = 'A' FOR UPDATE;

-- ✅ CORRECT: Always order by ID
SELECT * FROM accounts 
WHERE id IN ('A', 'B') 
ORDER BY id 
FOR UPDATE;

PostgreSQL：锁超时

-- Set lock timeout to prevent permanent deadlock
SET lock_timeout = '5s';

BEGIN;
SELECT * FROM accounts WHERE id = 'A' FOR UPDATE;
-- If can't acquire within 5s, transaction fails with error
-- Better than hanging forever
COMMIT;

分布式系统模式

// ✅ CORRECT: Distributed lock with lease
class DistributedLock {
  async acquire(resource: string, leaseDuration: number): Promise<Lease> {
    const lease: Lease = {
      id: uuid(),
      resource,
      expiresAt: Date.now() + leaseDuration,
    };
    
    // Atomic set-if-not-exists with TTL
    const acquired = await redis.set(
      `lock:${resource}`,
      JSON.stringify(lease),
      'NX',  // Only if not exists
      'PX', leaseDuration  // Expire after duration
    );
    
    if (!acquired) {
      throw new LockNotAvailableError(resource);
    }
    
    return lease;
  }
  
  async release(lease: Lease): Promise<void> {
    // Only release if we still own it (compare lease ID)
    const script = `
      if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
      else
        return 0
      end
    `;
    await redis.eval(script, 1, `lock:${lease.resource}`, JSON.stringify(lease));
  }
}

抗压应对方案

1. “只要把超时时间调长就行”

施压说法：“增加超时时间，避免失败”

**回应：**超时时间越长，死锁持续到失败的时间就越长。死锁依然存在。

**行动：**修正锁顺序。超时是安全网，不是解决方案。

2. “只有高负载时才会发生”

施压说法：“这是罕见的边缘情况，不值得修复”

**回应：**生产环境就是高负载。“罕见”在规模扩大后就会变成“持续发生”。死锁会造成永久性故障。

**行动：**现在就修复。死锁会随着流量增加而增多。

3. “不同服务之间不会冲突”

施压说法：“服务彼此独立”

**回应：**如果它们共享资源（数据库行、Redis 键、文件），就会发生冲突。

**行动：**在各服务之间协调锁顺序。

4. “重启一下就行”

施压说法：“重启可以清除死锁”

**回应：**重启会终止正在进行的事务，带来数据损坏风险，并影响用户。根本原因依然存在。

**行动：**预防死锁。不要只处理症状。

危险信号——立即停止并重新考虑

如果你发现以下任意情况，死锁风险都很高：

• 不同代码路径以不同顺序获取锁
• 嵌套获取锁，却没有明确的层级
• “这个端点有时会挂起”
• 没有锁超时
• 在异步操作期间持有锁
• 跨服务加锁却没有协调
• 在循环中获取锁

以上任何一种情况都意味着：重新设计锁策略。

死锁检测

// Runtime deadlock detection
class DeadlockDetector {
  private waitGraph: Map<string, Set<string>> = new Map();  // who waits for whom
  
  recordWaiting(waiter: string, holder: string): void {
    if (!this.waitGraph.has(waiter)) {
      this.waitGraph.set(waiter, new Set());
    }
    this.waitGraph.get(waiter)!.add(holder);
    
    // Check for cycle
    if (this.hasCycle(waiter)) {
      throw new DeadlockDetectedError(
        `Deadlock: ${this.describeCycle(waiter)}`
      );
    }
  }
  
  private hasCycle(start: string, visited = new Set<string>()): boolean {
    if (visited.has(start)) return true;
    visited.add(start);
    
    for (const next of this.waitGraph.get(start) || []) {
      if (this.hasCycle(next, visited)) return true;
    }
    
    visited.delete(start);
    return false;
  }
}

常见借口（都不成立）

借口	事实
“超时能解决问题”	超时只能缓解症状，死锁依然会发生。
“正常负载下很少发生”	真正重要的正是生产负载。
“服务彼此隔离”	共享数据库 = 共享资源 = 可能发生死锁。
“卡住时重启就行”	会终止事务并破坏状态。
“锁顺序限制太多”	有序加锁能实现并行处理，死锁则会扼杀并行处理。
“测试时运行正常”	测试环境没有生产环境中的并发情况。

快速参考

策略	适用场景
保持顺序一致	默认选择——始终按顺序获取锁
锁超时	所有加锁场景的安全网
Try-Lock 并退避	竞争激烈且可以重试时
单个粗粒度锁	吞吐量要求不高且需要简单性时
锁层级	存在嵌套资源的复杂系统
带租约的分布式锁	跨服务协调

死锁预防检查清单

要求	检查
所有锁都按一致的顺序获取	☐
已配置锁超时	☐
在 finally 块中释放锁	☐
避免在 await 边界期间持有锁（如可行）	☐
已设置死锁检测／监控	☐
已为跨团队代码记录锁顺序	☐

总结

务必为锁设定顺序。始终如此。每条路径。每个服务。

死锁源于锁顺序不一致。选定一种顺序（按字母顺序、按 ID 或按层级），并在所有地方强制执行。添加超时作为安全保障。监控锁竞争。当有人催促你“直接加个超时”时，应改为修正锁顺序。