concurrency-debugging

并发调试

目的

指导 Agent 诊断并修复并发错误：阅读 ThreadSanitizer 竞争报告、使用 Helgrind 分析锁顺序、通过 GDB 线程检查检测死锁、识别常见的 std::atomic 误用模式，以及在 C++ 和 Rust 中运用先行发生关系进行推理。

触发条件

• “ThreadSanitizer 报告了数据竞争——我该如何阅读报告？”
• “我的程序发生死锁了——该如何调试？”
• “如何使用 Helgrind 查找线程问题？”
• “我正确使用了 std::atomic 吗？”
• “C++ 内存顺序中的先行发生关系是如何运作的？”
• “如何在 GDB 中找出哪些线程发生了死锁？”

工作流

1. ThreadSanitizer（TSan）——竞争检测

# Build with TSan
clang -fsanitize=thread -g -O1 -o prog main.c
# or GCC
gcc -fsanitize=thread -g -O1 -o prog main.c

# Run (TSan intercepts memory accesses at runtime)
./prog

# TSan-specific options
TSAN_OPTIONS="halt_on_error=1:second_deadlock_stack=1" ./prog

阅读 TSan 报告：

WARNING: ThreadSanitizer: data race (pid=12345)
  Write of size 4 at 0x7f1234 by thread T2:
    #0 increment /src/counter.c:8:5              ← access site in T2
    #1 worker_thread /src/counter.c:22:3

  Previous read of size 4 at 0x7f1234 by thread T1:
    #0 read_counter /src/counter.c:3:14          ← conflicting access in T1
    #1 main /src/counter.c:30:5

  Thread T2 created at:
    #0 pthread_create .../tsan_interceptors.cpp
    #1 main /src/counter.c:28:3

SUMMARY: ThreadSanitizer: data race /src/counter.c:8:5 in increment

阅读方法：

1. 第 1 行：访问类型（写入/读取）和地址
2. “Write of size” 下的调用栈：执行写入的线程
3. “Previous read/write” 下的调用栈：发生冲突的线程
4. “Thread T2 created at”：线程创建的位置
5. 修复方法：increment 和 read_counter 函数在未同步的情况下访问同一地址

常见竞争及修复方法：

竞争模式	修复方法
无锁读写全局变量	添加互斥锁或使用 std::atomic
未使用 atomic 的双重检查锁定	使用 std::once_flag + std::call_once
对共享整数使用 +=	使用 std::atomic<int>::fetch_add()
遍历容器时同时修改容器	锁定整个临界区
shared_ptr 引用计数竞争	引用计数本身是安全的（它是原子的）；但指向的对象可能不安全

2. Helgrind——锁顺序与竞争检测

Helgrind 使用 Valgrind 基础设施检测锁顺序违规（潜在死锁）和数据竞争：

# Run with Helgrind
valgrind --tool=helgrind --log-file=helgrind.log ./prog

# Lock order violation report
==1234== Thread #3: lock order "0x... M2" after "0x... M1"
==1234== observed (incorrect) order
==1234==    at pthread_mutex_lock (helgrind/...)
==1234==    by worker2 /src/worker.c:45           ← T3 takes M2 then M1
==1234==
==1234== required order established by acquisition of lock at address 0x... M1
==1234==    at pthread_mutex_lock
==1234==    by worker1 /src/worker.c:31            ← T1 takes M1 then M2

锁顺序违规 = 潜在死锁：

• 线程 T1 获取 M1，然后尝试获取 M2
• 线程 T2 获取 M2，然后尝试获取 M1
• 如果两者发生竞争，都可能导致死锁

修复方法：遵循一致的全局锁顺序。在所有地方都先获取 M1，再获取 M2。

3. 使用 GDB 检测死锁

# Attach GDB to a deadlocked process
gdb -p $(pgrep prog)

# Or run under GDB then trigger deadlock

(gdb) info threads          # list all threads and current state
# * 1  Thread 0x... (LWP 1234) "prog" ... in __lll_lock_wait ()
#   2  Thread 0x... (LWP 1235) "prog" ... in __lll_lock_wait ()
# Threads blocked in __lll_lock_wait = waiting for mutex

(gdb) thread 1
(gdb) bt                    # show which mutex thread 1 is waiting for

(gdb) thread 2
(gdb) bt                    # show which mutex thread 2 holds/waits

# Find the mutex owner
(gdb) p ((pthread_mutex_t*)0x601090)->__data.__owner   # Linux glibc mutex
# prints TID of owning thread

# Python script to dump all mutex owners (GDB 7+)
python
import gdb
for t in gdb.selected_inferior().threads():
    t.switch()
    print(f"Thread {t.num}: {gdb.execute('bt 3', to_string=True)}")
end

4. std::atomic 误用模式

// WRONG: atomic variable, but non-atomic compound operation
std::atomic<int> counter{0};
if (counter == 0) counter = 1;   // not atomic together! TOCTOU race

// CORRECT: use compare_exchange
int expected = 0;
counter.compare_exchange_strong(expected, 1);

// WRONG: relaxed ordering for sync flag
std::atomic<bool> ready{false};
// Producer:
data = 42;
ready.store(true, std::memory_order_relaxed);  // WRONG: no happens-before

// CORRECT: release-acquire for publishing data
// Producer:
data = 42;
ready.store(true, std::memory_order_release);   // syncs with acquire

// Consumer:
if (ready.load(std::memory_order_acquire)) {    // syncs with release
    use(data);  // safe to read data here
}

// WRONG: using data across threads without atomic/mutex
// int shared_data;  // non-atomic — UB on concurrent access

// CORRECT: protect with mutex or make atomic
std::mutex mtx;
std::unique_lock lock(mtx);
shared_data = 42;

5. 先行发生关系推理

在 C++ 中，以下情况会建立先行发生关系：

Sequenced-before (within a thread):
  Statement A comes before B in code → A happens-before B

Synchronizes-with (across threads):
  store(release) → load(acquire) on SAME atomic variable
    → store happens-before load
    → everything before store happens-before everything after load

Thread creation/join:
  spawn(T) → any action in T         (create synchronizes-with)
  any action in T → join(T)          (join synchronizes-before)

Mutex:
  unlock(M) → lock(M) (next acquirer)

// Establishing happens-before across threads
std::atomic<int> flag{0};
int data = 0;

// Thread 1:
data = 42;                        // A
flag.store(1, memory_order_release); // B: A sequenced-before B

// Thread 2:
while (flag.load(memory_order_acquire) != 1) {}  // C: synchronizes-with B
int x = data;                     // D: C sequenced-before D
// D reads 42: A happens-before B synchronizes-with C sequenced-before D
//             → A happens-before D

6. Rust 并发——编译时保证

Rust 通过所有权机制在编译时防止数据竞争：

use std::sync::{Arc, Mutex};
use std::thread;

// Shared mutable state: Arc<Mutex<T>>
let counter = Arc::new(Mutex::new(0u32));

let c = Arc::clone(&counter);
let t = thread::spawn(move || {
    let mut val = c.lock().unwrap();
    *val += 1;
});

t.join().unwrap();
println!("{}", *counter.lock().unwrap());

// Rust prevents:
// - Sharing &mut T across threads (Sync not impl for &mut T)
// - Moving non-Send types to threads (compiler error)
// Use TSAN_OPTIONS with cargo test if TSan checks are needed:
// RUSTFLAGS="-Z sanitizer=thread" cargo +nightly test

相关技能

• 使用 skills/runtimes/sanitizers 了解 TSan 构建标志和其他消毒器
• 使用 skills/profilers/valgrind 了解 Helgrind 和 Memcheck 集成
• 使用 skills/debuggers/gdb 进行高级 GDB 线程检查
• 使用 skills/low-level-programming/memory-model 了解 C++/Rust 内存顺序理论