core-dumps

Core Dumps

用途

指导 Agent 启用、收集和分析 Core dump，以便在不重新运行存在缺陷的程序的情况下进行事后崩溃调查。

触发条件

• “我的程序在生产环境中崩溃了——如何分析 Core？”
• “如何在 Linux 上启用 Core dump？”
• “我有一个 Core 文件，但没有符号／源代码”
• “如何使用 debuginfod 为 Core 获取符号？”
• “coredumpctl 给我看看崩溃情况”

工作流

1. 启用 Core dump（Linux）

# Per-session (lost on logout)
ulimit -c unlimited

# Persistent (add to /etc/security/limits.conf)
*   soft   core   unlimited
*   hard   core   unlimited

# Check current limit
ulimit -c

# Set core pattern (where and how cores are named)
# Default: 'core' in CWD — often not useful
sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t
# %e = executable, %p = PID, %t = timestamp

# Persistent (add to /etc/sysctl.d/99-core.conf)
kernel.core_pattern=/tmp/core-%e-%p-%t
kernel.core_uses_pid=1

2. systemd/coredumpctl（现代 Linux）

如果 systemd 管理 Core dump（常见于 Ubuntu 20+、Fedora、Arch）：

# List recent crashes
coredumpctl list

# Show details of the latest crash
coredumpctl info

# Load latest crash in GDB
coredumpctl gdb

# Load specific PID crash
coredumpctl gdb 12345

# Export core file
coredumpctl dump -o myapp.core PID

Core 存储位置：/var/lib/systemd/coredump/。

3. 启用 Core dump（macOS）

# macOS uses /cores by default (must be root-writable)
ulimit -c unlimited

# Check
ls /cores/

# launchd-launched services: set in plist
# <key>HardResourceLimits</key>
# <dict><key>Core</key><integer>9223372036854775807</integer></dict>

4. 使用 GDB 分析 Core

# Load binary and core
gdb ./prog core.12345

# If the binary was stripped, provide the unstripped copy
gdb ./prog-with-symbols core.12345

# Essential first commands
(gdb) bt                    # call stack
(gdb) bt full               # stack + locals
(gdb) info registers        # CPU state at crash
(gdb) frame 2               # jump to interesting frame
(gdb) info locals           # local variables in frame
(gdb) print ptr             # inspect a pointer

# All threads (multi-threaded crash)
(gdb) thread apply all bt full

5. 使用 LLDB 分析 Core

lldb ./prog -c core.12345

# Or
lldb
(lldb) target create ./prog --core core.12345

# Commands
(lldb) bt
(lldb) thread backtrace all
(lldb) frame select 2
(lldb) frame variable

6. 缺少符号：debuginfod

debuginfod 从中央服务提供调试符号，根据构建 ID 映射到 DWARF 数据。

# Install client (Debian/Ubuntu)
sudo apt install debuginfod

# Enable (add to ~/.bashrc or /etc/environment)
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"

# GDB auto-fetches symbols when DEBUGINFOD_URLS is set
gdb ./prog core

# Manually query
debuginfod-find debuginfo <build-id>
debuginfod-find source <build-id> /path/to/file.c

7. 缺少符号：手动方法

# Check if binary has a build ID
readelf -n ./prog | grep Build

# Find the correct debug package
# Debian: apt install prog-dbg or prog-dbgsym
# RPM: dnf install prog-debuginfo

# Point GDB to debug symbols directory
(gdb) set debug-file-directory /usr/lib/debug

# Or use eu-readelf to dump build ID, then find .debug file
eu-readelf -n ./prog
find /usr/lib/debug -name "*.debug" | xargs eu-readelf -n 2>/dev/null | grep <build-id>

8. 剥离二进制文件并保留符号

最佳实践：构建时包含符号，分发时剥离符号，并保留一份未剥离的副本。

# Build
gcc -g -O2 -o prog main.c

# Separate debug info
objcopy --only-keep-debug prog prog.debug
objcopy --strip-debug prog prog.stripped

# Add a debuglink so GDB finds the debug file automatically
objcopy --add-gnu-debuglink=prog.debug prog.stripped

# Deploy prog.stripped; keep prog.debug in a symbols store indexed by build-id

9. 无需完整调试会话即可快速排查 Core

# Print backtrace non-interactively
gdb -batch -ex 'bt full' -ex 'thread apply all bt full' ./prog core 2>&1 | tee crash.txt

# Print registers
gdb -batch -ex 'info registers' ./prog core

# Check signal that caused crash
gdb -batch -ex 'info signal' ./prog core

有关涵盖 Core pattern 标记、coredumpctl、GDB/LLDB 命令、debuginfod 服务以及剥离／符号工作流的完整速查表，请参阅 references/cheatsheet.md。

相关技能

• 如需了解完整的 GDB 会话详情，请使用 skills/debuggers/gdb
• 如需进行基于 LLDB 的分析，请使用 skills/debuggers/lldb
• 如需在问题进入生产环境之前捕获它，请使用 skills/runtimes/sanitizers
• 如需了解 readelf、构建 ID 和二进制文件检查，请使用 skills/binaries/elf-inspection