Introduction of a new FreeBSD Remote Process Plugin in LLDB

Moritz Systems have been contracted by the FreeBSD Foundation to modernize the LLDB debugger’s support for FreeBSD. We are writing a new plugin utilizing the more modern client-server layout that is already used by Darwin, Linux, NetBSD and (unofficially) OpenBSD. The new plugin is going to gradually replace the legacy one.

The LLVM project provides a modern, modular, permissively licensed compiler infrastructure. A toolchain including Clang compiler, LLD linker and LLDB debugger is being developed as a part of it. FreeBSD has already replaced GCC with Clang as the primary system compiler. However, LLDB is still work-in-progress and it is being installed along with GDB.

The original FreeBSD plugin for LLDB used a legacy monolithic architecture. In this model the debugged program is being run inside the same process space as the debugger’s UI. The new plugin uses a client-server architecture that runs the debugged program as part of lldb-server, while LLDB’s UI runs remotely. This makes the LLDB code easier (or even possible) to reuse as a library with third party code as the debugger relies on monitoring signals and this functionality interferes with other code like GUI toolkits.

The Project Schedule is divided into three milestones, each taking approximately one month:

• M1 Introduce new FreeBSD Remote Process Plugin for x86_64 with basic support and upstream to LLVM.
• M2 Ensure and add the mandated features in the project (process launch, process attach (pid), process attach (name), userland core files, breakpoints, watchpoints, threads, remote debugging) for FreeBSD/amd64 and FreeBSD/i386.
• M3 Iterate over the LLDB tests. Detect and as time permits fix bugs. Ensure bug reports for each non-fixed and known problem. Add missing man pages and update the FreeBSD Handbook.

Handling transition to the new plugin

Since we want to be able to push our work upstream without causing even temporary regressions in FreeBSD support, we are developing it as a new plugin, with a technical name of FreeBSDRemote that coexists with the original FreeBSD plugin.

At the moment, toggling between the two plugins is possible via FREEBSD_REMOTE_PLUGIN environment variable. If it is unset, the legacy FreeBSD plugin is being used. If it is set, LLDB switches to using the new plugin. This is the approach suggested upstream, and it is consistent with how the new plugin for Windows platform is being currently developed.

Furthermore, the new plugin is also enabled automatically if lldb-server is started directly. This is because the old plugin simply does not support the client-server model.

Eventually, as the new plugin becomes par with the legacy one on features, we are going to perform the switch depending on the architecture. In other words, we are going to use the new plugin on all architectures it was ported to, and the legacy plugin on these that do not support the new one yet.

Basic platform support

We have decided to base the new plugin on our earlier work for NetBSD. This allowed us to reduce the amount of duplicate work, and instead focus on covering the differences between the two BSD platforms and improving its code. This made it possible to reach a reasonably functional debugger in a month’s time, and it should also make the remaining work easier.

With reasonably small changes, we were able to support controlling the process (resume, stop, single step), handle basic event signals, read and write general-purpose and debug registers, read and write traced process' address space, insert and handle software assisted breakpoints. We also have partial code for multithreaded programs, watchpoint support and 32-bit application debugging (multilib) support.

It is also important to note that the work results in improvements and bugfixes to other platforms (especially NetBSD) that either share similar code or provide a reference for our implementation. Ideally, we should be able to deduplicate more code from existing plugins (e.g. watchpoint support) as it should be suitable for different platforms.

Working on FPU register support

One of the more interesting concepts we have been working on are FPU (x87) registers. Unlike both the general-purpose registers and the registers used by MMX/SSE/AVX…, they can’t be trivially used directly by the program. The form (and completeness) of their dump differs depending on whether we’re running 32-bit or 64-bit, and what exact instruction is being used to generate it.

A great source of information on programming x87 FPU is Raymond Filiatreault’s Simply FPU tutorial.

FPU registers and general operation

The visible FPU registers can be classified in three groups:

• eight data registers of 80 bits accessible as a stack st(0)..st(7), each capable of storing an extended precision x87 floating point number, a 64-bit integer or up to 17-digit packed Binary-Coded Decimal. They overlap with MMX mm* registers.

• three control registers: control word (fctrl) used to program the FPU’s behavior, status word (fstat) reporting the current FPU state and tag word (ftag) reporting the state of all st(*) registers.

• three registers used to report exceptions when software exception handling is used: fop indicating the opcode of the instruction causing the exception, FIP (or fiseg:fioff) pointing to the actual instruction and FDP (or foseg:fooff) pointing to the memory operand of that instruction (if any).

The really confusing part is that FPU uses a static internal numbering for its registers, while the user-visible st(*) registers shift as values are pushed and popped. In other words, as you load values into the FPU they are loaded into internal registers 7, 6, 5… but from the user’s perspective the last value loaded is always in st(0) and the indices of previous values grow.

This isn’t really important during regular programming since only st(*) indexing is used. However, the ftag register is using internal register indexing, so if you wish to inspect it, you need to be able to map between the two systems. This is done using the TOP bitfield of fstat that indicates which internal register is currently at st(0).

FPU has two modes of handling exceptions. By default, exceptions are handled by hardware (i.e. the FPU itself). This means that invalid operations result in invalid or special values, such as Not-A-Number or infinity. If software exception handling is enabled, the FPU interrupts the instruction and expects the program to handle it.

ftag value in register dumps and the debugger

Previously, the ftag register was a 16-bit register that described each of the FPU data registers using one of four states: empty, zero, normalized value, special value (denormal, infinity, NaN). This full form is used by the FSAVE/FRSTOR instructions. However, the newer instructions (FXSAVE/XSAVE…) instead return an abridged 8-bit value that only indicates whether the register is empty or not.

It seems that this difference was omitted while writing LLDB, and all platform plugins return the abridged value on the modern processors, zero-padded to 16 bits. This is not technically a problem as long as it’s consistent but it could be a little confusing to the user and it is inconsistent with what GDB does.

The new FreeBSD plugin also exhibits this arguably buggy behavior for consistency with other platforms. However, ideally LLDB would be taught to convert from the abridged ftag to full version and back. This is generally done by combining the abridged value with inspection of actual data register contents.

FIP/FDP value in register dumps and the debugger

FIP and FDP registers are used to convey respectively the memory locations of the instruction that caused the FPU exception, and its operand. Similarly to other memory registers, they technically consist of a 16-bit segment register and a 32/64-bit offset register. However, the presence of these registers in FXSAVE etc. dumps and their format depends on how the (saving) instruction is executed.

If the instruction is executed from 32-bit code, or from 64-bit code without REX.W prefix (which is normally the case), the dump contains 16-bit segment register and 32-bit offset register. This means that the address is truncated in 64-bit programs. If the instruction is executed with REX.W prefix (i.e. FXSAVE64, XSAVE64… is called), the dump contains 64-bit address but no segment registers.

The current GDB and LLDB behavior on 64-bit Linux is to issue the 64-bit variant and split the resulting pointer, so that the higher 32 bits are exposed as fiseg/foseg while the lower 32 bits are exposed as fioff/fooff. We have lined up the FreeBSD plugin with this behavior. However, ideally we will introduce separate fip and fdp register names in the future, to expose the full pointer without the need to recombine.

It should be noted that both FreeBSD and NetBSD have issued the 32-bit variant of saving instructions, effectively truncating fip and fdp to 32 bits. We have reported this as a bug: ptrace() GETFPREGS/SETFPREGS uses 32-bit version of XSAVE/XRSTOR truncating FIP/FDP, and it has been fixed by Konstantin Belousov.

Accomplishment of the first project milestone

We have reached the first milestone of our work, that is delivering the initial functional version of the new plugin and pushing it upstream. To demonstrate our work, we have prepared two asciicasts.

The first one demonstrates running part of LLDB test suite using both plugins. The legacy plugin is used on the top pane (the default), while the new plugin is enabled via envvar on the bottom pane.

As you can see, all register tests time out using the legacy plugin, while most of them pass (with the exception of YMM register tests since YMM support is not yet implemented).

The second asciicast demonstrates various debugger functions during a sample debugging session. This time, lldb-server is spawned explicitly on the top pane, and the debugger connects to it on the bottom pane. The gdb-remote packet logging is enabled to demonstrate how the client-server model works.

It is worthwhile to notice that when using the client-server model, LLDB UI runs asynchronously. A functional command prompt is visible while the program is running.

Changes merged upstream

All of the forementioned changes were promptly tested, reviewed and upstreamed to the mainline LLVM repository. The list of the committed changes is as follows:

• [lldb] [Process/NetBSD] Fix operating on ftag register
• [lldb] [Process/NetBSD] Fix reading FIP/FDP registers
• [lldb] [Process/NetBSD] Fix crash on unsupported i386 regs
• [lldb] [test/Register] Add read/write tests for x87 regs
• [lldb] [test/Register] Mark new FP reg tests XFAIL on Windows
• [lldb] [test/Register] Attempt to fix x86-fp-read.test on Darwin
• [lldb] [Platform] Move common ::DebugProcess() to PlatformPOSIX
• [lldb] Initial version of FreeBSD remote process plugin

Plan for the next milestone

Our next goal is to reach the second milestone of the contracted work and implement the missing functions in the amd64 debugger plugin. Most notably, this includes proper threading support, remaining register types (using XSAVE) and watchpoints. Once the new plugin reaches feature parity with the old one, we are going to switch it to be used by default on amd64.

Afterwards, during the third month we would like to look into the state of LLDB test suite on FreeBSD. Ideally, we would like to reach as many passing tests as possible, and mark the remaining failures as expected, in order to be able to easily detect regressions in the future.