2020 in Review: Software Development Projects

Wow, what a year 2020’s been. Despite the challenges of the global pandemic, the Foundation’s software development efforts proceeded apace, and we’ve made significant contributions across a number of functional areas in FreeBSD.

Since the beginning of 2020 in the FreeBSD src repository we’ve had 838 commits tagged as being sponsored by the FreeBSD Foundation, across 12 different authors. This is a little over 30% of the commits that identify a sponsor. This includes work by project grant recipients, by the Foundation’s own staff, as well commits made on behalf of the 6 University of Waterloo co-operative education students who worked for the Foundation over two terms during the year. In addition, and in contrast with previous years many more contributions are now being made to third-party projects under Foundation sponsorship. This includes LLVM’s LLDB debugger, Open Broadcaster Software (OBS)-Studio, Firefox, and others.

Here I’ll recap the Foundation-sponsored work started and/or completed over the past year.

We sponsored Allan Jude to integrate Zstd compression support into OpenZFS. This is work that Allan started as a prototype a few years ago, with contributions and updates from others over that time. This project involved completing outstanding tasks to make it ready for merging, adding tests, and benchmarking. This work initially targeted the OpenZFS repository, and was brought into FreeBSD as part of the OpenZFS update.

Edward Napierała worked on Linux Emulation Layer (Linuxulator) improvements over the years, under Foundation sponsorship. In 2020 we sponsored Edward to continue improving the Linuxulator, but taking a different approach. Earlier grants focused on improving debugging and diagnostic tools, general infrastructure improvements, and similar tasks. In contrast, this time we focused on choosing a number of interesting server and client Linux applications. We identified any missing required functionality while trying to run them, and fixed many issues found as a result. When we started this work contemporary Linux binaries did not even start correctly, and now it is possible to run a substantial number of them. Others in the FreeBSD community have built upon Edward’s work, investigating issues with configuration, packaging, installation and such, and provided guides or sketches for running various programs on FreeBSD.

Emmanuel Vadot improved and updated the graphics (DRM) device drivers, bringing us to parity with device support in Linux kernel version 5.4. The project also made progress on the BSD-licensed implementation of the Linux KPI compatibility layer, with the intent of moving the graphics drivers back into the base system once the work is complete. The work also included a backlight control framework and backlight(8) control program. Some portions of this work were completed in the FreeBSD base system (src) repository, while others were part of the drm-kmod FreeBSD port.

The Foundation funded Kristof Provost to address bottlenecks in if_bridge(4), the network bridge device. The project was completed early in the year, delivering around a 5X throughput increase. The work is described more fully here.

We funded Kamil Rytarowski and Michał Górny of Moritz Systems to update LLDB’s FreeBSD support, triage test suite failures, and fix bugs. Before this work LLDB’s FreeBSD support relied on an older, same-process model, and was the only operating system to do so. This project ported the x86 FreeBSD target support to the new debugserver model. This work has been committed to the LLDB repository in advance of LLDB 12.0’s release. The work also identified issues in the FreeBSD base system (kernel and runtime components) that have now been fixed. Read more about their process here.

John-Mark Gurney is developing a management interface for remote access to hardware test labs, to allow reserving a device in the lab, controlling power, providing console access, and managing network-booted root filesystems.

Björn Zeeb continued work on WiFi infrastructure improvements. This work is focused on updating the net80211 stack for 802.11ac support, and allowing existing dual-licensed kernel drivers to be used on FreeBSD. Björn has been collaborating with upstream driver developers with a goal of sharing patches to ease our future maintenance burden. You’ll find more about this project in the upcoming Q4 2020 FreeBSD Status Report.

Our long-running RAID-Z expansion project continued this year, including a merge with the upstream OpenZFS with dRAID support (which includes some work developed as part of this project). Additional contributions came from community members, including additional tests and optimized math routines.

The Foundation hired six co-op students from the University of Waterloo during the 2020 winter and fall terms, as well as one intern, Ka Ho Ng. You can read more about Colin, Tiger, and Yang’s work (the first three students) in earlier Foundation reports. For the fall term (which just wrapped up) Tiger returned, and new students Yang and Zac joined us for the first time.

Tiger worked on improvements to the code-coverage guided kernel fuzzing tool Syzkaller, adding new system call definitions so that Syzkaller can expand the code it tests. A number of FreeBSD kernel bug fixes have already resulted from this work. Tiger also contributed a number of improvements to the ELF Tool Chain set of binary utilities, and worked on tooling to run tests from other tool suites against ELF Tool Chain.

Zac worked on an improvement to the pkg package management tool, investigating and upstreaming patches for FreeBSD support in FreePBX, and investigating compiler support for addressing the stack clash vulnerability.

Yang investigated and fixed a compilation bug with the kernel’s Skein-1024 assembly implementation (used by ZFS), and then a number of projects related to Capsicum: applying Capsicum to sort(1), implementing a Capsicum service to execute utilities, and finally working with developers of the Game of Trees (got) version control system to adapt it for Capsicum support.

Ka Ho improved audio support in Firefox for FreeBSD, adding a new OSS backend. He also proposed an nv(9) nvlist-based API for enumerating audio devices. A sparse file hole punching API is in progress, and some Virtio projects are being planned.

The five Foundation staff members continued contributions in 2020 in both ongoing operational tasks and software development for a number of projects.

Staff members responded to reported security vulnerabilities and release errata, prepared patches, and participated in the security advisory process. We also worked on proactive security vulnerability mitigations. Syzkaller also provided many reports of kernel issues that resulted in Foundation-sponsored bug fixes.

We participated in code reviews and supported community members in integrating changes into FreeBSD, and triaged incoming bug reports.

Several staff members contributed to Arm64 over the year, working to bring a number of subsystems towards parity with x86. This includes the Linuxulator and support for running FreeBSD/arm64 in virtulization. We also added support for features in newer versions of the Arm architecture, such as atomic instruction support for large core count systems and cryptographic instructions.

We contributed enhancements to many kernel and userland subsystems, including the x86 pmap layer, ELF run-time linker and kernel loader, the Capsicum sandboxing framework and Casper services, the threading library, some RISC-V changes, the build system, tool chain and freebsd-update, network stack stability improvements, machine-dependent optimizations, new kernel interfaces, DTrace bug fixes, documentation improvements, and others.

Finally, staff members also spent a significant amount of time on the Project’s successful svn to git transition.

With 2020 coming to a close, we look ahead to the new year. Many of our efforts will continue, including WiFi support, graphics, Linuxulator, Capsicum and security mitigations, and x86 and Arm support. We also plan to grow the team again in 2021 so that we can do even more to support the FreeBSD Project and community. Look for details early in the new year.

– contributed by Ed Maste