2016 FreeBSD Foundation Project Development Summary

Over the past year, the Foundation staff and grant recipients completed many projects. A check of the Subversion source repository shows over 950 commits sponsored by the FreeBSD Foundation. Here is a recap of the major development work undertaken in 2016.

FreeBSD/arm64 Port

The initial FreeBSD/arm64 porting work was completed in 2015, and provided a functional, but basic foundation. In 2016, Andrew Turner received a grant to integrate performance and stability improvements in FreeBSD’s 64-bit ARM port.

The virtual memory (VM) subsystem now supports four-level page tables, which is required by systems with large amounts of physical memory, and systems with multiple CPUs resulting in discontiguous memory regions. Building on this is transparent superpage support, which automatically aggregates multiple standard-sized 4 kilobyte hardware pages into fewer, much larger 2 megabyte pages. This can provide a substantial performance increase by reducing translation lookaside buffer (TLB) misses in large-memory applications, such as databases.

Other performance improvements include support for AES instructions, to speed up cryptographic operations, and the addition of a library of optimized assembly-language string routines.

Andrew investigated the stability and performance of the ZFS filesystem, DTrace dynamic tracing framework, and hwpmc hardware performance monitoring subsystems. Improvements were made where deficiencies were found, and these features are now production quality on FreeBSD/arm64.

The arm64 port now supports non-coherent device I/O, as required to access peripherals on certain hardware platforms. Updated infrastructure components are now enabled on arm64, including the “intrng” interrupt framework and “NEW_PCIB” PCI-PCI bridge driver.

Finally, Andrew added Advanced Configuration and Power Interface (ACPI) support to drivers used by FreeBSD/arm64, and enabled it on the platform. ACPI is the standard method used for conveying system configuration information in the X86 server world, and is also the method expected to be used in the arm64 server ecosystem.

VIMAGE Integration

The VIMAGE project is a kernel virtualization framework with roots in work started over ten years ago. A number of community-driven and Foundation-sponsored efforts over that time have moved VIMAGE forward, but it remained an optional feature with a worrisome “experimental” caveat.

This year the Foundation awarded a grant to Björn Zeeb to review, test, and integrate changes to address VIMAGE failures in network stack teardown and ordering, including memory leaks. These improvements are available in FreeBSD 11.0.

Blacklistd

Blacklistd provides a holistic, comprehensive solution for network daemon attack mitigation. It tallies unsuccessful access attempts by connecting host and manages firewall rules to prevent further connections. It replaces more ad-hoc tools like fail2ban or sshguard that rely on parsing log files with a small daemon that coordinates access information reported by individual network daemons. Blacklistd was implemented by Christos Zoulas in the NetBSD project.

Kurt Lidl received a project grant to port Blacklistd to FreeBSD, including testing, integration, and bug fixing. The grant also included patching daemons such as sshd, ftpd, and sendmail. This work was delivered in FreeBSD 11.0.

Filesystem Improvements

As a Foundation staff member, Edward Tomasz Napierała committed many improvements to filesystem-related functionality. This includes integration and bug fixes in the “reroot” functionality, used to unmount and then remount a new root filesystem, the autofs automount daemon, and iSCSI improvements.

USB Mass Storage Target

Edward also implemented cfumass, a CAM target layer (CTL)-based USB mass storage target. It allows the USB device support found on embedded boards like the Beaglebone Black to function as a USB mass storage device (that is, a flash memory stick), exporting a block device from a file on the embedded board’s own filesystem. This will simplify the process of interacting with a FreeBSD-based embedded device attached to common host platforms. For example, the Beaglebone Black could provide documentation for use, and Windows and macOS drivers for additional functionality.

Address Layout Randomization

Foundation staff member Konstantin Belousov developed an address randomization implementation, and presented it for testing and review. Randomizing the address of objects in memory provides some mitigation of potential vulnerabilities in applications by increasing the difficulty of successfully executing an exploit. This work is in review now and will arrive in FreeBSD soon.

Capsicum

Capsicum is described as “practical capabilities for UNIX,” a lightweight capability and sandbox framework and is applied to applications to prevent potential bugs from providing exploitable vulnerabilities.

The Foundation made several improvements under the capsicum umbrella this year. Capsicum is now supported in the “compat32” layer, allowing 32-bit applications running on 64-bit hosts to take advantage of capability syscalls.

Previously, Capsicum only reported an error code to an application that attempted to make use of an unpermitted system call or operation. As a debugging aid, it can now instead deliver a debugging trap to pinpoint the source of such an operation, to ease the process of applying Capsicum to new applications. Also, Capsicum can now permit certain “..” directory path lookups when in capability mode. This is intended to simplify adding Capsicum to certain classes of software – for example, archiving utilities.

Capsicum has also been applied to many more applications.

POSIX Process-Shared and Robust Mutexes

Process-shared mutexes allow synchronization primitives to be shared between multiple processes in a single application. Robust mutexes are guaranteed to be cleared by the operating system if a thread or process terminates while the mutex is held. Process-shared and robust mutexes are used by a number of third-party software packages, which may have slower or less functional fallback support for operating systems that lack these mutex types.

Lack of support for these mutex types has been a longstanding issue in FreeBSD. Konstantin Belousov added an implementation earlier this year which is now available in FreeBSD 11.0, and is being used by applications like the Samba file sharing server.

Packaged Base

FreeBSD’s pkg tool was introduced in 2012 for managing the third party binary package collection. The packaged base project aims to use pkg for managing the base system installation and upgrade as well.

The packaged base project began as a community-led initiative. More recently Foundation staff member Glen Barber has been overseeing and managing its development.

Tool Chain

We’ve made a great deal of progress on FreeBSD’s tool chain in 2016, continuing a goal of migrating to modern and permissively licensed tools. The LLVM debugger lldb is available in FreeBSD 11.0. The collection of binary object manipulation tools (such as readelf, strings, and nm) now come from the ELF Tool Chain project.

Additional projects are underway in the FreeBSD development branch, after FreeBSD 11. These include experiments with LLVM’s LLD linker and exception handling stack unwind library libunwind. We are on track to provide a permissively licensed tool chain in FreeBSD 12.

Reproducible Builds

A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts. Many developers in the FreeBSD community have worked on improving FreeBSD’s reproducibility over the years, but a small number of issues remained.

This year Ed Maste became involved in the broader Reproducible Builds project, which has an overall goal of providing verifiable paths from human-readable source code to installable packages or executable binaries. As part of this work with a couple of patches still in review it is now possible to build the FreeBSD base system 100% reproducibly.

2016 was a very productive year for the FreeBSD Foundation Project Development Team. Please consider helping us increase our project development work in 2017  by making a donation today!