2022 in Review: Software Development

From ZFS support in makefs to experimental 16k page support on arm64 to lldb and wireguard improvements, the Foundation development staff and contractors had another busy year working to improve FreeBSD. From January 1, 2022 until mid-December 2022, 1114 of the 7575 commits to the src repository (about 15%) identify the FreeBSD Foundation as a sponsor. The raw commit data give a broad indication that the Foundation is dedicated to directly improving FreeBSD, however commit data only tell part of the story. Let’s review the Foundation’s role in FreeBSD development over the past year. 

Members of the Foundation team support development by serving in many ways. For example, two staff members serve on the FreeBSD core team, which takes on a variety of roles including setting Project policy and resolving disputes. We coordinate contracted development work, including internships and we contribute to cluster administration, source repository management, the security team, and continuous integration. We also played key roles in FreeBSD’s Google Summer of Code program. One Foundation staff member administered the program and served as a mentor, while two other staff members also served as project mentors. The program was a success in that all projects passed and code was contributed to the tree.  For example, drm debugfs support was added to the LinuxKPI in f697b943.  We also participated in a mentorship program through RISC-V International to get Syzkaller, a kernel fuzzer, working on FreeBSD/RISC-V. The project was a success and some RISC-V bug reports (e.g. PR 266108) have started coming in.

In 2022, 15 security advisories and 28 errata notices were sent out to the FreeBSD community. Two Foundation developers are active members of the FreeBSD Security Team who help to triage and address security problems and keep critical system components up to date. For example, Mark Johnson committed a fix to FreeBSD-SA-22:11.vm, which was a case of mishandled memory sharing in the virtual memory system and Konstantin Belousov committed a fix for FreeBSD-SA-22:09.elf, an issue which could cause a kernel crash when dumping core and saving process information. Ed Maste made a few dozen commits to keep our copy of OpenSSH updated and secure.  

While development staff often have to react to incoming bugs and other problems, there are also opportunities to develop new FreeBSD features. Sometimes we complete projects in-house and other times we contract out the work. Here is a summary of Foundation sponsored projects from 2022. 

LLDB Multiprocess Debugging 

FreeBSD includes the LLDB debugger in the base system. The LLDB project describes itself as the “next generation, high-performance debugger built as a set of reusable components highly leveraging existing libraries in the larger LLVM Project, such as the Clang expression parser and LLVM disassembler.” Starting in late 2020, The Foundation contracted Moritz Systems to improve LLDB and make it feature-complete with GDB, the GNU debugger.  Work done under past contracts includes replacing the old LLDB plugin model on FreeBSD with a modern plugin using a client-server layout, implementing SaveCore functionality, adding support for debugging via serial port, implementing follow-fork and follow-vfork operations, and adding support for live kernel debugging.

The most recent contract, which started in April 2022, aims to implement full support for debugging multiple processes simultaneously. At the start of the project, LLDB featured very limited support for multiprocess debugging. The client featured support for debugging multiple independent processes simultaneously via maintaining multiple connections to different server instances. Building upon past contracts with Moritz Systems, the server was able to process fork(2) and vfork(2) calls and either detach the newly forked child and continue tracing the parent process, or detach the parent and follow the child (equivalent to GDB’s follow-fork-mode setting).

The work already completed includes:

Enabling Snapshots on Filesystems Using Journaled Soft Updates 

While ZFS has become a filesystem of choice for most FreeBSD installations, there remain workloads such as in embedded systems, where UFS/FFS may be preferred. Similar to ZFS, the UFS/FFS filesystem has the ability to take snapshots. Because the taking of snapshots was added after soft updates were written, they were fully integrated with soft updates. However, when journaled soft updates were added in 2010, they were never integrated with snapshots. This means snapshots cannot be used on UFS/FFS filesystems running with journaled soft updates. To fill this gap, the Foundation contracted Kirk McKusick. 

There are two instances where UFS snapshots are particularly important. The first is that they allow reliable dumps of live filesystems which avoids possibly hours of down time. The second is that they allow the running of background fsck. Similar to the need for scrub in ZFS, fsck needs to be run periodically to find undetected disk failures. Snapshots allow fsck to be run on live filesystems rather than needing to schedule down time to run it. The project has two milestones. The first, which was completed in November, was to enable snapshots when running with journaled soft updates and ensure that they can be used for doing background dumps on a live filesystem. The second is to extend fsck_ffs to be able to do a background check using a snapshot on a filesystem running with journaled soft updates. This milestone is expected to be completed by the third quarter of 2023. 

ZFS Support in makefs(8) 

makefs(8) is a utility, originating in NetBSD, that creates file system images entirely in userspace. It is a useful component of a toolchain to build virtual machine (VM) images since it does not require any special privileges, unlike the approach of formatting a character device, mounting the fresh file system, and copying files onto it. Moreover, makefs can create reproducible images and aims to minimize resource consumption. Prior to work completed in 2022, FreeBSD’s makefs could build UFS, cd9660, and msdos (FAT) file system images. 

Recent work by Mark Johnston enables the creation of ZFS images by makefs, making it easier to build ZFS-based VM images. ZFS support includes the ability to create multiple datasets, with each mapped to a directory in the input file hierarchy.  Follow-up work enables the creation of ZFS-based VM and cloud images by the release(7) framework, using this new makefs extension.  makefs is intended to be a simple ZFS implementation sufficient to generate reproducible root pools.  Complicated installations can be configured by the in-kernel ZFS once the system is running.

Wireless Improvements 

The Foundation has contracted Bjoern Zeeb to improve FreeBSD wireless support. So far, much of the effort has involved work on mac80211 LinuxKPI compatibility, a framework, backed by native net80211 and kernel code, which allows us to incorporate dual-licensed Intel wireless drivers for Linux into FreeBSD with little or no modification. For example, at the time of writing, files shared between the FreeBSD iwlwifi driver and the upstream Intel Linux driver contain fewer than 400 lines of differences. Other work Bjoern completed in 2022 includes adding support for newer wireless standards, adding support to auto-load the LinuxKPI PCI drivers, meaning the iwlwifi(4) driver will automatically load during boot if a supported card is detected, and incorporating a few newer Realtek wireless drivers into the tree. Towards the end of the year, focus has begun to shift almost exclusively to supporting contemporary wireless speeds, which mostly involves supporting 11n and 11ac standards in the LinuxKPI 802.11 compat code and filling gaps, mostly for 11ac, in the native net80211 wireless stack. 

In November, the Foundation contracted a former Google Summer of Code student, En-Wei Wu, to also work on wireless networking. En-Wei plans to work on a variety tasks, including

For the latest updates on this wireless work, follow the freebsd-wireless@FreeBSD.org mailing list and check the relevant wiki pages at Intel iwlwifi status, Realtek RTW88 Status, and Realtek RTW89 Status. 

OpenStack on FreeBSD 

OpenStack is an open-source cloud operating system for different types of resources like virtual and bare-metal machines. Users can spawn FreeBSD instances on the open cloud platform, but it is not currently possible to run the OpenStack control plane on FreeBSD hosts. To remedy this, The FreeBSD Foundation contracted Chih-Hsin Chang to port key OpenStack components so that FreeBSD can function as an OpenStack host. Those components are: 

As of November 2022, Nova, configured with libvirt and bhyve is running on FreeBSD. Chih-Hsin’s current focus is on porting Neutron to complete the VM lifecycle operations. The highlights of this work include DHCP integration and FreeBSD bridge driver/agent. 

FreeBSD as a Tier 1 cloud-init Platform 

cloud-init is the standard way of provisioning servers in the cloud. Unfortunately, support for operating systems other than Linux is rather poor, and the lack of support on FreeBSD is a hindrance to cloud providers who want to offer FreeBSD as a Tier 1 platform. To improve cloud-init support on FreeBSD, the Foundation has contracted Mina Galić. The project deliverables include completing an extraction of certain networking classes, implementing ifconfig(8) and login.conf(5) parsers, implementing IPv6 configuration, creating devd.conf(5) rules for Azure, and FreeBSD Handbook documentation about productionizing FreeBSD. On the way there, BSD-related module bugs are being fixed. The status as of December is that the ifconfig parser is complete and the net/cloud-init-devel port is in the ports tree ready for testing. 

WireGuard Kernel Integration

Wireguard describes itself as “an extremely simple yet fast and modern VPN that utilizes state-of-the-art cryptography.” Thanks to work by John Baldwin under contract from the FreeBSD Foundation, Wireguard is well integrated in FreeBSD with a kernel module that takes advantage of our own cryptography implementations. The work was started in 2021, but completed in early 2022 when John added support for the XChaCha20-Poly1305 AEAD cipher, added a dedicated API to support [X]ChaCha20-Poly1035 on small, flat buffers and finally added an API wrapper for the curve25519 implementation from libsodium. 

Part-time Work on Bhyve and RISC-V 

Starting in May, the Foundation contracted John Baldwin to dedicate up to a few hours per week to Bhyve as issues arise, especially security issues. An example of the type of work John is doing under this contract can be seen here. In October the Foundation contracted Mitchell Horne to work part-time on FreeBSD/RISC-V. Mitchell has been busy with a variety of reviews, bug fixes, and other miscellaneous work to improve RISC-V on FreeBSD. 

– Contributed by Joe Mingrone with contributions from contractors