Linux core dumps with gdb and kdump vmcores with crash

Capture a Linux process core dump (core_pattern, systemd-coredump), get a backtrace with gdb, and read a kernel kdump vmcore with the crash utility.

Published 2026-10-20 by PerformanceInsights.ai

1. Make sure a crash leaves a core

cat /proc/sys/kernel/core_pattern   # where cores go, or which program receives them
ulimit -c                           # 0 means no core for this shell
ulimit -c unlimited                 # for this shell; services set LimitCORE= instead

• If core_pattern starts with |/usr/lib/systemd/systemd-coredump, systemd keeps the cores: coredumpctl list shows them, and coredumpctl dump <PID> -o core writes one out as a file.
• On Ubuntu, |/usr/share/apport/apport means apport takes them. Crashes of packaged programs end up in /var/crash/*.crash (unpack with apport-unpack).
• Our open-source configure-dumps.sh switches to plain ELF cores with unlimited size and keeps them; --undo puts things back.

2. Read a process core with gdb

gdb /path/to/the/binary core.12345     # or: coredumpctl gdb <PID>
(gdb) bt                               # stack of the crashing thread
(gdb) thread apply all bt              # every thread (deadlocks, aborts)
(gdb) info registers
(gdb) frame 2                          # select a frame, then:
(gdb) info locals

The binary must be the same build that crashed. Frames shown as ?? mean missing debug info. Fedora, Ubuntu and Debian run debuginfod servers; with DEBUGINFOD_URLS set (the default on recent releases) gdb fetches debug info for distribution packages on demand. For your own programs, keep the unstripped binary or the separate .debug file for every release.

The signal in gdb's first line is the first clue: SIGSEGV is a bad memory access, SIGABRT usually an abort() from an assert or the C library's heap checks (free(): invalid pointer), SIGBUS a bad mapping such as a truncated memory-mapped file.

3. Kernel crashes: kdump and crash

When the kernel panics there's no process to dump, so you need kdump: a reserved second kernel (the crashkernel= boot parameter) that boots on panic and saves memory to disk.

• Ubuntu / Debian: apt install linux-crashdump (Ubuntu) or kdump-tools, reboot, check with kdump-config show.
• RHEL / Fedora: kexec-tools, systemctl enable --now kdump.

The vmcore lands under /var/crash/. To read it you need the debug vmlinux for exactly that kernel (Ubuntu: the kernel's -dbgsym package from the ddebs repository, linux-image-unsigned-<version>-dbgsym on current releases; RHEL: kernel-debuginfo):

crash /usr/lib/debug/boot/vmlinux-<version> /var/crash/<dir>/dump.<stamp>
crash> sys      # kernel, panic string, uptime
crash> log      # the kernel ring buffer up to the panic
crash> bt       # the panicking task's stack
crash> ps       # what was running

Start with log: the last lines usually name the oops (BUG: unable to handle page fault, general protection fault, a hung task warning) and the module the faulting function belongs to. On RHEL the debug vmlinux is under /usr/lib/debug/lib/modules/<version>/vmlinux.

Common mistakes

• Debugging with a different build. A rebuilt binary has different addresses; gdb prints a plausible but wrong stack.
• Cores truncated by size limits. systemd-coredump drops or truncates cores above ProcessSizeMax / ExternalSizeMax in /etc/systemd/coredump.conf (32G on 64-bit systems with current systemd, lower on older releases), so a big process can lose its core; a truncated core gives gdb partial memory.
• Containers. core_pattern is host-wide, so a container's crash goes to the host's handler, not into the container.

On Windows instead? Analyze a Windows minidump with WinDbg. All guides: Guides.