2021 in Review: Software Development

The FreeBSD Foundation added to our technical staff this year to increase our sponsored development, while continuing grant funding for individual projects. Here I’ll review and highlight the work we took in 2021.

In 2021 Foundation staff and grant recipients made 1468 sponsored commits to the src tree, 42 to the doc tree, and 163 to the ports tree. This represents a growth over 2020 of 70%, 37%, and 75% respectively.

We started the year with a technical staff of three, and finish with eight full-time technical staff members and contractors. We also had two co-operative education students from the University of Waterloo join us for the Spring 2021 term.

As usual staff supported the project with bug triage and fixes, supported security advisories and errata updates, participated in code reviews to help contributors make submissions to FreeBSD, supported ongoing operational needs of the project, and developed a number of features and improvements.

Staff focus areas in kernel development included virtual memory, x86 pmap, uma, tmpfs, nullfs, ffs and ufs, debugging, kqueue and POSIX timers, and job control improvements. User space work included changes to the libc, libcasper, and libthr libraries, the run-time linker, as well as the ldd, cmp, diff, makefs, elfctl, growfs, and bhyve utilities.

Foundation staff members Li-Wen Hsu and Ed Maste assisted with the migration of the FreeBSD source code repositories from Subversion to Git. A significant portion of this project was completed in 2020, but the ports tree conversion took place at the beginning of this year. We continued to work on some improvements including updates to the repository commit hooks, and continue to investigate and test permissively-licensed Git-compatible tools which may be suitable for inclusion in the base system.

The Foundation supports FreeBSD’s continuous integration effort through hardware donations and Li-Wen’s leadership of the CI team. Li-Wen keeps the CI infrastructure up-to-date, including adapting to the Project’s migration to Git, and performs a significant amount of test result triage, investigating reported issues and coordinating with the appropriate committer. You can read more about our continuous integration support in the standalone report.

Syzkaller is a automated code-coverage-guided system call fuzzer, which is very effective at finding inputs to provoke and identify kernel bugs. This year Mark Johnston contributed significantly to FreeBSD support in Syzkaller, and triaging and fixing many of the issues it found. You can read more details in Mark’s Syzkaller report.

Related work includes the kernel sanitizers KASAN (kernel address sanitizer) and KMSAN (kernel memory sanitizer). These features combine Clang compiler support with a runtime component to detect invalid operations at runtime. They may be used directly by developers and provide value there, but are very useful in detecting issues via CI tests, and in improving Syzkaller’s effectiveness.

Over the years Foundation staff have implemented various vulnerability mitigations. These include proactive mitigations for bugs in applications, as well as software mitigations for CPU speculative execution issues. We added a write-XOR-execute mapping policy, and extended the elfctl tool to make it easier for the ports infrastructure to use across supported FreeBSD releases. We also supported developers from Semihalf with their goal of starting to enable these features by default.

Andrew Turner joined our team this year, allowing us to increase our support for arm64 (or AArch64), the 64-bit Arm architecture. This year the FreeBSD core team promoted FreeBSD/arm64’s to Tier-1 status, and Andy’s work with the Foundation will support that.

Andy has added support for ISA-specific instructions and optimizations (such as SHA256), and extended indirect function (IFUNC) dispatch support to statically linked binaries. Work is in progress on RAS (Reliability, Availability and Serviceability) extensions. Two security enhancements, Pointer Authentication (PAC) and Branch Target Identification.

Konstantin Belousov modified the amd64 UEFI loader to start the kernel from the “staging area” without first copying it aside. Previously the kernel had to be copied to a fixed location, which was incompatible with some UEFI firmware implementations and had other limitations. This work addressed a number of outstanding issues in the FreeBSD problem report (bug) database.

The UNIX VFS API has traditionally supported two types of operations. One operates on paths and manipulates metadata – removing or renaming files, changing permissions, and so on. The other operates on data – for example, read, write, or truncate. These operations use a file descriptor (fd) as a handle to the opened file, and importantly once opened the handle is decoupled from the filename – the file may even be deleted, but data operations may continue on the fd.

UNIX historically provided no API for performing all metadata operations using a similar handle. One consequence is that certain operations could fundamentally not be made race-free; it was impossible to guarantee that the underlying file was not changed between two path-based metadata operations.

Linux extended the API with a facility to make path references possible, and Kostantin added a broadly compatible implementation to FreeBSD. This work adds the O_PATH flag to open(2) which does not actually open the file for access, but just acts as a reference to the filesystem object. The Samba SMB/CIFS server will make use of the new functionality – details are available in Samba’s Wiki. Details on use of path descriptors can be found in the open(2) man page.

FreeBSD includes a copy of OpenSSH in the base system. Toward the end of this year Ed updated base system OpenSSH, from 7.9p1 to the then-latest version, 8.7p1, followed by an update to 8.8p1. Ed also enabled support for FIDO/U2F hardware security keys. You can read more about this work in the security report.

Over the summer Foundation co-op student Yang Zhong built a prototype of an updated FreeBSD installer, building on initial work by Ryan Moeller. The experiment uses a web front-end which allows for a user friendly graphical install process. The installation may even be controlled by a browser on a different computer. This work was featured in Hackaday. Although not integrated into FreeBSD the Foundation will continue with this experimental effort.

Foundation co-op student Cyril Zhang worked on some container-and virtualization-related tasks over the summer. This included work to add resource accounting (RACCT) support to the runj OCI-compatible Jail runtime. This included a number of RACCT bug fixes and minor improvements, and the addition of subsystem tests. Phabricator reviews (for FreeBSD changes) and a GitHub pull request (for runj) are in progress for this work.

Cyril also worked on improving bhyve, with the primary effort changing the bhyve VM lifecycle to allow better user and jail integration.

Although Cyril’s work term is over, Foundation staff will continue to work on both of these projects.

To expand and improve FreeBSD wireless support, the Foundation has contracted Bjoern Zeeb (bz) to work on an Intel Wireless driver project. In the current phase of the project, the aims are to provide support for newer Intel chipsets and to update the mac80211 LinuxKPI compat code. The dual-licensed Intel driver code was ported in the past for the iwm(4) native driver and using the LinuxKPI compat framework allows porting the driver directly with only very minor modifications. Multiple updates during the past year have shown that pulling in newer versions can be done in under 1-2 hours.

The Intel wireless driver and firmware are now regularly synchronized with upstream so that we will have support for all modern cards currently supported on GNU/Linux. Some iwlwifi driver changes were also submitted back upstream. Several conflicts with the original implementation of LinuxKPI have been resolved and more LinuxKPI code was upstreamed to the FreeBSD main branch. After LinuxKPI 802.11 compat code was improved, data packets are going over 11a. Firmware crashes have been resolved and the 802.11 compat code was improved. The iwlwifi driver and firmware were updated from the iwlwifi-next git branch and the linux-firmware repository. For the latest state of development, follow the freebsd-wireless mailing list.

Foundation staff contributed many more small feature enhancements and improvements, including:

Linux binary compatibility allows running unmodified Linux binaries under FreeBSD without the use of virtual machines, but by providing Linux binaries with kernel interfaces identical to those of the Linux kernel. To update and improve the Linux compatibility layer, the Foundation awarded grants to Edward Tomasz Napierala (trasz@). The most recent work focused on ensuring Linux compatibility was ready for the 13.0 release, updating the Linux kernel interface to version 3.17.0, and allowing more unmodified Linux binaries to be executed on FreeBSD. The status of specific Linux applications is tracked at the Linux app status wiki page.

A debugger with kernel debugging capabilities is a critical tool for OS development. The FreeBSD base system includes LLDB, the debugger component of the LLVM project. To improve LLDB on FreeBSD, the Foundation contracted Moritz Systems. Some of the recent enhancements include porting LLDB support to the modern client-server model on all architectures, implementing support for tracing fork(2) and vfork(2) syscalls with follow-fork-mode compatibility from GDB, improving support for core dumps, and improvements for interfacing with other servers implementing the GDB remote protocol.

Fork tracing makes it possible to debug child processes. Upon fork, the debugger can either continuing tracing the parent and detach the child, or switch to tracing the child and detach the parent. It also prevents software breakpoints from leaking to child processes and causing them to crash. The newly introduced PT_COREDUMP ptrace(2) allows creating a core dump of a stopped program without crashing it. The remote protocol work included fixing issues when interfacing with GNU GDB and QEMU’s gdbserver implementations and adding support for debugging executables running on gdbserver architectures other than amd64. Moritz’s work also involved fixing many bugs, improving the state of the test suite on amd64 and arm64, improving the LLDB API, and reducing code duplication.

RAID-Z expansion is a long-requested ZFS feature that will allow users to add disks one at a time to a RAIDz vdev. This feature can be especially useful for small pools. For example, when there isn’t sufficient hardware to add storage capacity via a whole new RAID-Z vdev, disks can be added to a 3-disk RAIDz1, to make it a 4-disk, then 5-disk, etc. RAIDz1 vdev. The FreeBSD Foundation contracted Matt Ahrens to implement RAID-Z expansion and the work is to be integrated into the OpenZFS repository with support for FreeBSD and Linux. The current state of the project is that it is functionally complete and a pull request has been opened. Remaining work includes code cleanup, design documentation, and addressing test failures.

WireGuard is a secure tunneling protocol with both userland and kernel implementations. After an initial flurry of bug fixes, the FreeBSD kernel wireguard codebase is more complete and stable. Automated CI is now set up for each commit, compiling and running a small smoke test on wireguard-freebsd’s supported releases.

The Foundation is sponsoring John Baldwin to work on wireguard by updating the data path crypto in the upstream WireGuard driver to use the in-kernel OpenCrypto Framework for the data path. Data packets sent over a WireGuard tunnel are encrypted with the Chacha20-Poly1305 AEAD cipher. Unlike TLS and IPsec, Wireguard uses an 8 byte nonce rather than a 12 byte nonce with this cipher.

To date, most of the work has focused on updating OCF to better support multiple nonce (and tag/MAC) lengths for a given cipher. John had previously begun work aimed at supporting all of the AES-CCM NIST KAT vectors, many of which use non-default nonce and tag lengths. The approach has been refined to better fit the existing OCF model where nonce and MAC lengths are properties of a session (similar to key lengths). (An earlier branch had made the nonce length a property of individual operations instead.) This mostly entailed extending the /dev/crypto interface to permit setting these parameters for a session. Existing tests for OCF run in userland and use the /dev/crypto interface including both the cryptocheck utility and the NIST KAT vector tests.

Building upon these framework changes, John extended the existing Chacha20-Poly1305 cipher in OCF to support both 8 and 12 byte nonces including in the accelerated ossl(4) driver. A patch against the upstream WireGuard FreeBSD driver to make use of this for the dataplane has been verified to interoperate with the stock WireGuard driver.

Future work will focus on upstreaming the OCF changes as well as additional review of the upstream WireGuard driver.

— Contributed by Ed Maste