LLDB FreeBSD kernel core dump support

Moritz Systems have been contracted by the FreeBSD Foundation to continue our work on modernizing the LLDB debugger’s support for FreeBSD.

The primary goal of our contract is to bring kernel debugging into LLDB. The complete Project Schedule is divided into six milestones, each taking approximately one month:

1. Improve LLDB compatibility with the GDB protocol: fix LLDB implementation errors, implement missing packets, except registers.

2. Improve LLDB compatibility with the GDB protocol: support gdb-style flexible register API.

3. Support for debugging via serial port.

4. libkvm-portable and support for debugging kernel core files in LLDB, on amd64 + arm64 platform. Support for other platforms as time permits.

5. Support for debugging the running kernel on amd64 + arm64 platform. Support for other platforms as time permits.

6. Extra month for kgdb work, processing patches on LLDB reviews or miscellaneous tasks – as time permits. Examples of misc tasks: access to extended system and process information, starting processes via shell, $_siginfo support.

This month we have been working on support for FreeBSD kernel core dumps (vmcores). FreeBSD uses two vmcore formats: full memory dumps that use an ELF container, and minidumps. The former format is misread by LLDB as a regular process core dump, the latter is not supported at all.

Our work consisted of two parts: firstly, forking FreeBSD’s libkvm into a cross-platform core dump reading library; secondly, integrating it into LLDB. We have considered three possible integration methods: a LLDB plugin, an external GDB Remote Protocol server or explicit conversion to LLDB-compatible ELF core dump.

In this post, we are going to describe what kernel core dumps are and how they differ from userspace process core dumps. Then we are going to shortly discuss how they are saved, what the minidump format looks like and how to read it. Afterwards, we’re going to discuss the new fbsdvmcore library and its integration into LLDB.

FreeBSD Kernel core dumps

Core dump basics

We have discussed core dumps as a tool for post-mortem debugging already in the past. However, our earlier article was focused on core dumps of user space programs. This time we will be discussing kernel core dumps, also called vmcores.

Kernel core dumps can be used to debug kernel bugs, in particular kernel panics. While regular core dumps capture the state of a specific program, vmcores capture the state of the whole system as a context to the kernel behavior.

The FreeBSD Handbook has a chapter dedicated to Kernel Debugging. We are only going to cover the basics necessary to understand our work.

FreeBSD currently supports two core dump formats intended for consumptions by debuggers:

1. The full memory dump format that captures the complete contents of physical memory and stores them in an ELF container.

2. The minidump format that captures only the memory pages actually used by the kernel and uses a custom container.

There is also a textdump format. However, it is intended for human consumption rather than debugger use, and therefore it is out of scope for this post.

Writing kernel core dumps

Normally core dumps are written when the kernel panics. This makes it important for the process to be as simple as possible.

The standard method for writing vmcores on FreeBSD involves using the swap space as a temporary storage. Naturally, this means that the swap partition must be large enough to hold the core dump in the requested format. The kernel writes the core dump into swap, then reboots. On the next boot, the RC scripts detect the presence of the core dump and copy it to ``/var/crash before reenabling swap.

A kernel core dump can be triggered manually using the following sysctl:

# sysctl debug.kdb.panic=1

The minidump format

The minidump format is the default format used for vmcores on modern FreeBSD versions. The format aims to efficiently store a subset of physical memory pages, skipping these are deemed inactive. This is particularly useful for systems whose swap partition is smaller than physical memory.

Picture above illustrates the minidump file format. The file starts with a header that contains a magic value identifying the file type and architecture, a version number and a few other fields that vary depending on architecture and file version. Example header fields for amd64, arm64 and i386 targets are provided in picture below.

You can note that there are a number of matching fields, mostly defining sizes of the common minidump elements.

The minidump header is followed by a dump of the kernel message buffer (of msgbufsize, rounded up to page size). Afterwards, dump_avail field follows (of rounded dumpavailsize) that describes which parts of bitmap are included in the minidump. Then, the actual memory bitmap is written (of rounded bitmapsize) that describes which memory pages are included in the minidump. This is followed by the package page directory (table on i386), and finally by the actual memory chunks.

At this point, it is worth emphasizing how compact the minidump format aims to be. The memory chunk block includes only active memory pages, with the memory bitmap indicating which pages were actually included. Then, the bitmap itself is compacted and dump_avail describes which parts of the bitmap are present.

Reading kernel core dumps

User space program core dumps use the ELF format. The contents of program’s virtual memory are written as PT_LOAD segments, while the additional program state information is written into PT_NOTE segments.

On the other hand, kernel core dumps contain the contents of the physical memory and a copy of kernel message buffer (dmesg). Rather than adding additional structures to the core dump format, additional state information (such as thread states and register contexts) have to be extracted from the kernel structures. Furthermore, due to use of physical memory addressing, virtual memory addresses need to be explicitly translated to physical addresses.

FreeBSD includes a kvm library derived from Solaris (and featured by other BSD derivates as well). This library provides consistent routines for working with kernel memory, both in the form of core dumps and live kernel memory via /dev/mem. In addition to the basic routines for reading (and writing, in the case of live kernel) memory and looking up symbols, it also provides a number of wrappers for accessing common kernel state information, e.g. process information, system load or swap information.

The implementation of libkvm relies heavily on the system headers of the FreeBSD system it is built on. As a result, its support for core dumps from other architectures is limited, and the library itself (in its FreeBSD version) is provided on FreeBSD only.

Here’s a very simple program that uses libkvm to read the value of hz kernel’s variable from a core dump. For simplicity, the paths as well as the variable’s memory address have been hardcoded.

#include <fcntl.h> /* for O_RDONLY */
#include <kvm.h>
#include <limits.h> /* for _POSIX2_LINE_MAX */
#include <stdio.h>

/* obtained via readelf(1) */
static uintptr_t hz_addr = 0xffffffff81cd4c0c;

int main() {
  kvm_t *kvm;
  char errbuf[_POSIX2_LINE_MAX];
  int hz;
  ssize_t rd;

  kvm = kvm_open2(/*execfile=*/ "/boot/kernel/kernel",
                  /*corefile=*/ "vmcore.minidump",
                  /*flags=*/ O_RDONLY,
                  /*errbuf=*/ errbuf,
                  /*resolver=*/ NULL);
  if (kvm == NULL) {
    printf("Failed to open kernel / core dump: %s\n", errbuf);
    return 1;
  }

  rd = kvm_read2(kvm, hz_addr, &hz, sizeof(hz));
  if (rd != sizeof(hz)) {
    printf("Failed to read hz: %s\n", kvm_geterr(kvm));
    kvm_close(kvm);
    return 1;
  }

  printf("hz = %d\n", hz);

  kvm_close(kvm);
  return 0;
}

Creating a portable vmcore library

Our first task was to create a portable alternative to libkvm. The goal was to create a library for reading vmcores that:

• was portable to different operating systems
• supported cross-architecture processing of vmcores (libkvm has limited support for that)
• featured a clean API without obsolete or deprecated functions

At the same time, we wanted to reuse existing code as much as possible, and ideally preserve the original history of the library in order to make it easier to analyze its code. For this reason, we have proceeded according to the following plan:

1. Split (via git-filter-repo) lib/libkvm from the FreeBSD src repository to our libfbsdvmcore repository.
2. Create a standalone build system for the original code.
3. Remove the support for live kernels, leaving only the code necessary for core dumps.
4. Remove deprecated and obsolete parts of the API.
5. Rename the symbols and files (replacing kvm with fvc).
6. Make the library portable to other operating systems.

As part of the changes, we have replaced the baroque libkvm API with only a few functions necessary for processing core dumps. Write support (usable only for live kernels) has been eliminated completely. The old symbol resolution code that utilized FreeBSD libc’s nlist(3) API has been replaced by a default resolver utilizing elf(3) library (that was an existing dependency anyway). The new resolver gained support for cross-architecture kernel decoding.

Here’s the libkvm example program rewritten using libfbsdvmcore:

#include <limits.h> /* for _POSIX2_LINE_MAX */
#include <stdio.h>

#include <fvc.h>

/* obtained via readelf(1) */
static uintptr_t hz_addr = 0xffffffff81cd4c0c;

int main() {
  fvc_t *fvc;
  char errbuf[_POSIX2_LINE_MAX];
  int hz;
  ssize_t rd;

  fvc = fvc_open(/*execfile=*/ "/boot/kernel/kernel",
                 /*corefile=*/ "vmcore.minidump",
                 /*errbuf=*/ errbuf,
                 /*resolver=*/ NULL,
                 /*resolver_data=*/ NULL);
  if (fvc == NULL) {
    printf("Failed to open kernel/coredump: %s\n", errbuf);
    return 1;
  }

  rd = fvc_read(fvc, hz_addr, &hz, sizeof(hz));
  if (rd != sizeof(hz)) {
    printf("Failed to read hz: %s\n", fvc_geterr(fvc));
    fvc_close(fvc);
    return 1;
  }

  printf("hz = %d\n", hz);

  fvc_close(fvc);
  return 0;
}

You can note that the API is very similar to libkvm’s recommended methods. fvc_open(3) is basically kvm_open2(3) with the flags argument removed (since writing is not supported) and resolver prototype changed to pass an opaque resolver_data through, and fvc_read(3) is equivalent to kvm_read2(3).

Integrating FreeBSD vmcore support into LLDB

For our work to directly benefit LLDB users, it needed to be integrated into LLDB itself. We have proposed three different options for integrating it:

1. Writing a new LLDB plugin dedicated to handling FreeBSD vmcores.
2. Creating an external GDB Remote Protocol server that would provide the core file support via appropriate remote packets.
3. Creating a LLDB trivial that converts vmcores into ELF-compatible core dumps on the fly.

In our opinion, the first option is preferable as it provides the new functionality without requiring the users to enable it explicitly. However, it requires intrusive changes to the current ELF core dump plugin in order to prevent it from trying to process full memory dump vmcores (that use ELF as their container format). As such, it may face difficulties in being accepted upstream.

The second option is an alternative that does not require any changes inside LLDB. Instead, a standalone program is written that uses the GDB Remote Protocol to implement the core dump reading backend. This option probably involves most work, and would require users to either manually launch the server and connect to it, or to use a wrapper doing that.

The third option is probably the easiest to implement, especially that a program to convert minidumps to ELF core dumps has already been created (and we have included a modified version of it in libfbsdvmcore). However, it is quite limited in functionality and requires twice the disk space, as the converted core dump needs to be written alongside the original file. Furthermore, minidumps with significant memory fragmentation result in ELF core dumps with a very high number of segments, and exceed the base limit in the ELF format (65535 segments).

All things considered, we have proceeded with the first option and submitted our proposed plugin for upstream review. However, we are prepared to fall back to one of the two other options if this does not work out.

Writing a new Process plugin for vmcores

LLDB uses Process plugins to implement both support for debugging live processes and core dumps. At the time of writing, it features three core support processes: elf-core (for userspace program ELF core dumps), mach-core (for Mach-O core dumps) and minidump (for Windows minidumps). We have proposed adding a fourth plugin called FreeBSDKernel, focused specifically on FreeBSD kernel core dumps.

The core of the plugin is ProcessFreeBSDKernel class that implements a single debugged process (since LLDB does not support strict multiprocess debugging, this is the base class for all LLDB operations). The code calls fvc_open(3) early during the instantiation and uses it to recognize whether a valid kernel image and core dump were provided. Based on the result, it either instantiates the class indicating that it is going to handle the file, or returns an empty pointer to let LLDB continue searching for a valid plugin.

The second class in the hierarchy is ThreadFreeBSDKernel. At this moment, the plugin creates exactly one thread that corresponds to the kernel’s crashing thread. The thread is passed the address of the dumppcb symbol containing the PCB of the crashed thread. It provides the API for reading stack frames and therefore producing backtraces.

The final classes are RegisterContextFreeBSDKernel* classes, defined for every supported architecture. These classes provide the abstraction for reading register value from the PCB structure. It should be noted that this structure provides only a subset of CPU registers.

The combination of these three classes provides LLDB with the ability to open FreeBSD vmcores, read virtual memory from them and use it to reconstruct backtraces and the values of the most relevant registers.

Changes merged upstream

• [lldb] [Process/elf-core] Disable for FreeBSD vmcores
• [lldb] Introduce a FreeBSDKernel plugin for vmcores

Summary

At the start of our work, LLDB lacked proper support for FreeBSD kernel vmcores. Developers and users wishing to debug kernel crashes had to use the KGDB fork of GDB. It was possible to open full memory dumps via LLDB but it did not read the memory correctly, nor recognized kernel threads. There existed a proposed third party tool to convert minidumps into ELF cores but it only addressed a part of the problem, fixing only direct memory reads.

The reference implementation of core dump file format reading was present in libkvm that was limited to running natively on FreeBSD. Furthermore, the library had poor support for cross-architecture processing of core dumps, relying entirely on the user to provide a resolver compatible with the architecture of the core dump.

We have created a cross-platform cross-architecture replacement for libkvm’s core dump support and called it libfbsdvmcore. We have confirmed that our library builds correctly on FreeBSD, Linux and NetBSD, and that it can open core dumps from amd64, arm64 and i386 platforms. It does not require a custom resolver (though it supports one if desirable); instead, it reuses libelf to perform the symbol resolution.

We have also created a new FreeBSDKernel Process plugin for LLDB. It uses libfbsdvmcore in order to open kernel core dumps, both in full memory dump and minidump formats. However, the former format also requires patching the existing elf-core plugin not to accept vmcores. The new plugin provides the ability to read kernel variables, as well as inspect the backtrace and registers of the crashing kernel thread.

This is the next step towards feature parity between GDB (or KGDB, in this instance) and LLDB, and therefore towards making it possible for FreeBSD developers and users to rely entirely on the permissively licensed toolchain of LLVM instead of the GPL tools such as GDB.

Future plans

The next step on our path towards feature parity between KGDB and LLDB is the support for debugging the running kernel. We are going to specifically focus on amd64 and arm64 platforms but our generic work should provide a good basis for support of other architectures.

We are going to focus on multiple possibilities of debugging running kernels, that is:

• accessing the live kernel memory via libkvm
• connecting to KGDB via network sockets
• connecting to KGDB via the serial port

We have already performed most of the prepatory work towards every one of these items. At this point, our primary goal is to provide proper testing and fix any bugs we may encounter.