July 2023 Software Development Projects Update

There are different ways to measure how work on FreeBSD is progressing such as the rate of commits, the number of developers, the number of FreeBSD ports, and so on. By one measure, the number of Foundation-funded projects, we are in a boom cycle. As of the time of writing, the FreeBSD Foundation has contracts open for 12 different projects. Some projects we have written about in past newsletters, others have only begun recently. What follows is a summary of this contracted work as well as some noteworthy development by Foundation employees. 

Contracted Work 

Documentation and Testing Internship 

Starting in July, Yan-Hao Wang began a summer internship with the Foundation to work on a variety of tasks. To start, Yan-Hao will focus on improving our documentation tooling. One deliverable is to build an online editor designed solely to make creating and editing man pages as simple as possible. A second is to develop an “expert system” for FreeBSD man pages and other documentation. This best-effort and proof-of-concept deliverable will involve importing FreeBSD documents such as man pages and the Handbooks into a vector database so that large language models like ChatGPT can “read” them in order to offer better answers when queried with FreeBSD questions. Making improvements to our testing framework is a second general goal of the summer’s work. For example, Yan-Hao will add missing tests for many of the userland tools under /bin, /sbin, /usr/bin, and /usr/sbin as well as libxo. Finally, some miscellaneous deliverables include updating the FreeBSD Jenkins tinderbox and developing a support roadmap for RPI4 support. 

Addressing OpenSSL 3 / LLVM 16 Ports Fallout 

A major obstacle to releasing FreeBSD 14.0, one that delayed the original release schedule, was the version of OpenSSL in the base system. With the end-of-life for OpenSSL 1.1.1 planned for September 2023, it was imperative to move to OpenSSL 3 prior to the release. Foundation team member Pierre Pronchery along with other members of the FreeBSD development community have mostly completed the update, however lots of code in both the base system and the ports tree that depend on OpenSSL require fixes to work with this new major version. Around the same time OpenSSL was updated in the base system, LLVM was updated to version 16, which also introduced some breaking changes, especially in the ports tree. Most of the critical issues with OpenSSL 3 and LLVM 15 have already been fixed, but with LLVM 16 there are approximately 800 additional ports that fail to build resulting in an additional 2800 dependent ports that are skipped in a full ports tree build. Muhammad Moinur (Moin) Rahman will complete the time-consuming and tedious work to fix all port issues related to the update to OpenSSL 3 and LLVM 16. 

SIMD-enhanced libc for amd64 

Modern computer architectures provide SIMD (single instruction multiple data) instruction set extensions to operate on multiple data at once. SIMD instruction set extensions such as SSE, AVX, and NEON are ubiquitous on modern computers and offer performance advantages for many applications. The goal of this project is to provide SIMD-enhanced versions of common libc functions, mostly those described in string(3), to speed up most C programs. 

Commonly used for numerical applications such as video codecs, graphics rendering, and scientific computing, use of SIMD techniques also aids in basic data processing tasks such as those implemented by libc functions. While other libc implementations already provide SIMD enhanced variants of standard libc functions, the FreeBSD libc largely does not. The objective of this project by Robert Clausecker is to provide such SIMD-enhanced versions of relevant libc library functions and thus improve the performance of software linked against it. As these libc functions are used by most software available for FreeBSD, these enhancements are projected to give a broad benefit for a wide range of programs. The primary focus of the project is amd64 with the aim to produce SIMD optimised implementations based on the architecture levels defined by the x86_64 psABI. 

For each function optimised, up to four implementations will be provided: 

Users will be able to select which level of SIMD enhancements to use by setting the AMD64_ARCHLEVEL environment variable. While the current project only concerns amd64, the work may be expanded to other architectures like arm64 in the future. 

Networking Summer Internship 

Naman Sood is a FreeBSD Foundation summer intern who has been working on networking related tasks such as (1) completing work started by Luiz Amaral to allow pfsync traffic to be carried over IPv6 and (2) updating work to implement RFC 4787 REQs 1 and 3 for pf full cone NAT. Naman has also fixed bugs in pw(8) and du(1) and has begun to explore completing tcp checkpoint and failover work started by Klaus P. Ohrhallinger. 

Security Sandboxing Using ktrace(1) 

Starting in June, Jake Freeland began an internship with the Foundation to work on Capsicum-related projects. Capsicum was built to limit the capabilities given to applications and libraries. The core idea is intuitive; once capability mode is entered, resource acquisition and external communication is limited. It is fairly easy to design a program around this principle, but problems arise when applications that were not designed to be sandboxed need to work in this environment. It can be difficult to determine which actions cause Capsicum violations and it is impossible to pre-open resources that have not been requested or named yet. Furthermore, the developer needs to be fully acquainted with the program before they can begin to implement Capsicum functionality. Due to these challenges, progress and development surrounding the framework has waned in recent years. 

The internship will involve working on a number of projects that, as a whole, aim to revitalize Capsicum. The primary goal is to enhance and ease the experience for developers who wish to Capsicumize their existing programs. The biggest enemy of Capsicum is its large learning curve. Refactoring a program to support capability mode often requires the developer to know what causes Capsicum violations and know how to restructure the given program to avoid violations. Sometimes this process is trivial, but larger programs often need resources on-demand and figuring out how to serve these needs can be difficult. Extending the number of tools available to the developer for convenient program Capsicumization will decrease the learning curve. If Capsicumization is easy, then more developers will adopt it. 

The first logical step in Capsicumization is determining where your program is raising capability violations. You could approach this issue by looking through the source and removing Capsicum-incompatible code, but this can be tedious and requires the developer to be familiar with everything that is not allowed in capability mode. An alternative to finding violations manually is to use ktrace(1), which logs kernel activity for a specified process. While ktrace can record and return extra information about your program’s violations, this can be inconvenient because ktrace only reports the first capability violation before the program terminates. 

A new ktrace extension written by Jake can record violations when a program is not in capability mode. This means capability violation tracing can be run on an unmodified program and since the program is not required to be run in capability mode, it will still acquire resources and execute normally even when capability violations occur. 

Upon completion of the ktrace extension Jake will begin to capsicumize various utilities, including syslogd(8), NFS daemons, ggatec(8)/ggated(8), tftpd(8), ntpd(8), and libarchive(8). 

Wireless Internship 

En-Wei Wu, a 2022 Google Summer of Code Contributor, began an internship with the FreeBSD Foundation in early 2023 to work on FreeBSD wireless drivers and tools. The work is divided into three components. 

Continuous Integration Enhancements 

FreeBSD’s Jenkins-based CI infrastructure currently initiates a job every time a developer pushes a commit to the FreeBSD src repository. While this is valuable for identifying newly introduced problems, it would be even better to easily identify these issues before they are pushed to the main branch. This goal can be achieved by making CI tests more accessible to developers in a pre-commit environment and also by having private FreeBSD runners that the popular git hosting services could leverage to create CI infrastructure for people pushing to private branches. To that end, Muhammad Moinur Rahman aims to take CI scripts written by Li-Wen Hsu and make them available to developers as part of the build system. Similar to how make universe or make tinderbox builds all supported architectures, make ci would achieve something similar for all the supported builds. A parallel goal is to make it possible for developers to run individual CI jobs when debugging problems. The hope is that this flexibility will also allow for others to integrate these builds / scripts into other CI tools like Cirrus CI that are run from Github. 

Improving the kinst DTrace provider 

DTrace is a framework that gives administrators and kernel developers the ability to observe kernel behavior in real time. DTrace has kernel modules called “providers”, which perform a particular instrumentation in the kernel using “probes”. kinst is a new DTrace provider created by Christos Margiolis and Mark Johnston as part of GSOC 2022 that allows for instruction-level tracing, that is, for a given kernel function, a user can trace each instruction in it. The provider is available in FreeBSD 14.0. 

kinst probes take the form of kinst::<function>:<offset>, where <function> is the kernel function to be traced, and <offset> is the specific instruction. The offsets can be obtained from the function’s disassembly using kgdb(1). For example, the following command will trace vm_fault()’s first instruction each time it executes: 

This project, which was completed in late July, implemented inline function tracing. In order to achieve this functionality, the DWARF debugging standard was used to be able to detect inline calls and handle them accordingly. DWARF’s and kinst’s functionality were leveraged to address some of the shortcomings of FBT, such as the tail-call optimization problem (chapter 20.4 of the DTrace manual) and the absence of inline tracing capabilities. 

Project deliverables included: 

FreeBSD as a Tier I cloud-init Platform 

cloud-init is the standard way of provisioning servers in the cloud. Unfortunately, cloud-init’s support for operating systems other than Linux is rather poor and the lack of cloud-init support on FreeBSD is a hindrance to cloud providers who want to offer FreeBSD as a Tier I platform. To remedy this, the FreeBSD Foundation has contracted Mina Galić to bring FreeBSD cloud-init support on par with Linux support. The project deliverables include completing an extraction of certain networking classes, implementing ifconfig(8) and login.conf(5) parsers, implementing IPv6 configuration, creating devd rules for Azure, and writing Handbook documentation about productionizing FreeBSD. 

A number of project milestones were completed in the last quarter: 

These are used by several cloud providers to initialize a temporary network before retrieving the actual configuration. 

The hope is that with the next cloud-init release, Vultr can be convinced to switch their FreeBSD images to cloud-init. 

ds-identify runs first and very quickly guesses the cloud provider the machine is running on. cloud-init then uses only that guess, instead of iterating and failing through a full list of all possible cloud providers. People building custom images can easily disable this (by removing “/usr/local/etc/rc.d/dsidentify“), and providing a specific listing themselves, shaving off a few more milliseconds from the boot time. 

The next steps will be to continue work on the network refactoring tasks and to add LXD support for FreeBSD, so it can be included in CI tests. The latter will include work on LXD, as well as work on the FreeBSD virtio subsystem. 

OpenStack on FreeBSD 

OpenStack is an open-source cloud operating system for many kinds of resources, from virtual machines, containers, to bare-metal servers. OpenStack’s control plane mainly targets Linux and FreeBSD is only unofficially supported as a guest operating system. Users can spawn FreeBSD instances on the open cloud platform, but it is not currently possible for administrators or operators to set up OpenStack deployments running on FreeBSD hosts. Given the increasingly important role of cloud-based deployments, and the popularity of OpenStack with various cloud providers, the FreeBSD Foundation has contracted Chin-Hsin Chang to port OpenStack components, so the system as a whole can be run on FreeBSD hosts. 

The Linux-based OpenStack components to be ported to FreeBSD/amd64 include, but are not limited to: 

As of the second quarter of 2023, some key achievements include: 

Work has begun to port nova-novncproxy and nova-serialproxy, which will increase the ways the instance console can be accessed. To lower the threshold for those who want to test the work, the development environment has also been migrated from a physical to a virtual instance. Anyone interested in testing should be aware that there is still a problem running bhyve VMs on top of Linux KVM. A detailed write-up of the issue can be found at https://hackmd.io/@starbops/SkdJON2un. Other work planned for the near future includes improving the console proxy services to make the overall workflow more fluent. Step-by-step documents for constructing a proof-of-concept site can be found at https://github.com/openstack-on-freebsd/docs. The patched version of each OpenStack component is under the same GitHub organization. 

Snapshots on Filesystems Using Journaled Soft Updates 

The UFS/FFS filesystem has the ability to take snapshots. Because the ability to take snapshots was added after soft updates were written, they were fully integrated with soft updates. However, when journaled soft updates were added in 2010, they were never integrated with snapshots. So, snapshots cannot be used on filesystems running with journaled soft updates. There are two instances where UFS snapshots are still important. First, they allow reliable dumps of live filesystems, which avoids possibly hours of down time. Second, they allow running background fsck. Similar to the need for scrub in ZFS, fsck needs to be run periodically to find undetected disk failures. Snapshots allow fsck to be run on live filesystems rather than needing to schedule down time. This work requires extensive changes in the UFS/FFS soft updates and snapshot kernel code and also in the fsck_ffs utility. 

The work is divided between two milestones. Milestone 1, which was completed in late 2022 enables snapshots when running with journaled soft updates. These snapshots are usable for doing background dumps on a live filesystem. Milestone 2 involves extending fsck_ffs to be able to do a background check using a snapshot on a filesystem running with journaled soft updates. As of mid-August, the changes that are needed in the kernel and fsck_ffs are complete and testing has gone smoothly. 

Related to this project, Kirk has been working on a number of filesystem panics that were recently reported as well as a few older UFS bugs. These included bugs that could trigger false corruption warnings in the UFS superblock integrity checks. All the fixes were tested in time for inclusion in FreeBSD 14 and those relevant to FreeBSD 13 were MFCed. 

WiFi 

The FreeBSD WiFi stack needs ongoing maintenance and development to keep up with new standards and devices. The Foundation is funding Bjoern Zeeb to integrate support for current-generation Intel WiFi devices by migrating to the dual-licensed upstream driver from the Linux kernel.  Under the contract, Bjoern also takes on related wireless work such as development of the 802.11 LinuxKPI and integration of other wireless drivers, such as those from Realtek.

In the latest quarter nearly all of Bjoern’s time was dedicated to LinuxKPI updates in preparation for an iwlwifi(4) update as well as updates to other drivers such as rtw88, rtw89, mt76, and ath10k, as well as preparations for ath11k and ath12k.  Most of this work is already in the tree and will soon be connected to the build for people to test.

Other Foundation Work 

As we reported in our 2023 Q2 status report entry, 339 src, 155 ports, and 20 doc tree commits identified the FreeBSD Foundation as a sponsor. A sample of this work includes: 

Members of the Foundation’s technology team presented at the Developer Summit held in Ottawa, Canada from May 17-18. This included hosting the GSoC, FreeBSD Foundation Technical Review, and Workflow working group sessions. Foundation team member Pierre Pronchery spoke about driver harmony between the BSDs and En-Wei Wu discussed wtap work completed under contract with the Foundation. 

Base OpenSSL 3 Update 

As already mentioned, the pending removal of upstream support for the 1.1 branch of OpenSSL was a major obstacle to releasing FreeBSD 14.0. New Foundation team member, Pierre Pronchery, along with Ed Maste and other members of the FreeBSD developer community got to work updating our base OpenSSL to version 3. 

OpenSSL is a library for general-purpose cryptography and secure communication. It provides an Open Source implementation of the SSL and TLS network protocols, which are widely used in applications such as e-mail, instant messaging, Voice over IP (VoIP), or more prominently the global Web (aka HTTPS). Assuming that the Apache and nginx web servers use OpenSSL, their combined market share for web traffic exceeds 50%, cementing the leadership and critical importance of OpenSSL as part of Internet’s infrastructure. 

Since its initial release in August 2016, the 1.1 branch of OpenSSL has been adopted by most Linux and BSD systems, while remaining supported by the upstream maintainers through a Long Term Support policy. However, official support is planned to end in the middle of September this year, and it became urgent and necessary to consider adopting its successor for Long Term Support, the 3.0 branch. 

OpenSSL has largely outgrown its ancestor SSLeay, now shipping over half a million single lines of code (SLOC) split in over 2.000 files. Perhaps as a consequence, its build system is relatively complex and normally requires Perl, which has been removed from FreeBSD’s base system since 5.0-RELEASE. Thankfully however, it was possible to import and set up OpenSSL’s 3.0.9 release the FreeBSD way, and it is now part of the base system as planned for FreeBSD’s 14.0 release. 

It is an understatement to mention that OpenSSL 3 is a new major release. First, its problematic licensing model has finally been solved, with a complete switch to the Apache License 2.0. Then, OpenSSL 3 introduces the concept of provider modules. While obsolete cryptographical algorithms have been isolated away into a “legacy” module, it is also possible to restrict the implementation to standards part of FIPS with the fips module. The latter can then benefit from a dedicated certification process, and be validated officially (like the 3.0.8 release when writing these lines). 

Moreover, the updated library comes with a version bump, as applications using OpenSSL 1.1 need to be recompiled to use 3.0. Many API functions have been deprecated and replaced with newer, more generic alternatives, however it is still possible to explicitly request older APIs and have OpenSSL 3 expose them accordingly. This possibility has been leveraged in FreeBSD to help with the transition, where a number of libraries and applications have simply been configured to request the OpenSSL 1.1 API. These components will be updated progressively over time in order to consume OpenSSL 3’s native API instead. 

While there is a known performance impact associated with the update when consuming small input block sizes, it was found to be marginal when working with blocks of 1 KB and above. Another challenge lies with the FIPS provider module, which currently requires some manual steps in order to have it working. We are currently looking for a solution to ship FreeBSD with a functional FIPS provider by default. 

Bump amd64 MAXCPU from 256 to 1024 

The default amd64 and arm64 FreeBSD kernel configurations currently support a maximum of 256 CPUs. A custom kernel can be built with support for larger core counts by setting the MAXCPU kernel option. However, commodity systems with more than 256 CPUs are becoming available and will be increasingly common during FreeBSD 14’s support lifecycle. We want to increase the default maximum CPU count to 1024 to support these systems out of the box on FreeBSD 14. Foundation team member Ed Maste and other developers have made a number of changes to support a larger default MAXCPU, including fixing the userland maximum for cpuset_t at 1024. Changes have also been made to avoid static MAXCPU-sized arrays, replacing them with on-demand memory allocation. Additional work is required to continue reducing static allocations sized by MAXCPU and addressing scalability bottlenecks on very high core count systems, but the goal is to release FreeBSD 14 with a stable ABI and KBI with support for large CPU counts.

FreeBSD on Microsoft HyperV and Azure 

Li-Wen Hsu, Wei Hu, and other member of the FreeBSD Azure Release Engineering Team, along with Souradeep Chakrabarti <schakrabarti@microsoft.com>, and other members of the Microsoft FreeBSD Integration Services Team <bsdic@microsoft.com> have been working to add FreeBSD ARM64 architecture support to Azure. Part of that work has involved publishing images to the Azure community gallery. There are some testing images available in the project’s testing public gallery, named `FreeBSDCGTest-d8a43fa5-745a-4910-9f71-0c9da2ac22bf`: 

To use them, when creating a virtual machine: 

Work in progress includes: 

The above tasks are sponsored by The FreeBSD Foundation with resources provided by Microsoft. 

Wei Hu and Souradeep Chakrabarti from are working on several Microsoft-sponsored tasks, including: 

Open tasks: 

Google Summer of Code 

Coding for the 2023 edition of GSoC has just passed the midway point and the progress of the seven FreeBSD projects are documented on the wiki. Foundation team members are serving as our organization administrators and three projects are being mentored by people connected to the Foundation. 

CI Test Harness For Bootloader 

FreeBSD supports multiple architectures, file systems, and disk-partitioning schemes. For this project, Sudhanshu Mohan Kashyap <smk@FreeBSD.org>, is writing a Lua script to permit testing the boot loader for all these combinations for tier 1 and tier 2 architectures. If time permits, further exploration may be done to integrate the scripts into the existing build infrastructure (either Jenkins or Github Actions) to generate a comprehensive summary of the test results. 

Currently any developer changes may inhibit the ability of the operating system to boot in some specific environment. These scripts provide assurance that changes do not cause regressions for the tested environments. The scripts are designed to be efficient and much less expensive than the full make universe required today. These attributes allow developers to routinely use the script, and allow integration into the CI pipelines without undue cost. 

Physical memory compaction for the FreeBSD kernel 

Most modern CPU architectures offer performance boosts by supporting pages that are larger than the standard page size. Unfortunately, allocating such pages can fail due to a high degree of physical memory fragmentation. This work implements physical memory compaction as a means of actively reducing fragmentation in running systems. The goal of this GSoC work is to add various physical memory anti-fragmentation measures to the virtual memory subsystem. 

Differential D40575 implements a well-known metric used for quantifying the degree of physical memory fragmentation. Differential D40772 implements physical memory compaction and adds a daemon that monitors the system and performs compaction when needed. 

Planned future work includes designing an appropriate benchmarking suite, running tests, and tweaking the code using feedback from reviews and test results. This is still a work in progress, so any testing, reviews, and feedback would be greatly appreciated. 

Integrate mfsBSD into the release building tools 

mfsBSD is a toolset created by Martin Matuška to create small-sized but full-featured mfsroot based distributions of FreeBSD. That is, it loads files from a storage device and stores them in memory via a memory file system (MFS). As part of this project, Soobin Rho <soobinrho@FreeBSD.org> will create an additional target of the weekly snapshots of -current and -stable versions of mfsBSD images in the src/release makefile. Currently, only the release versions of mfsBSD images are produced, which means they tend to get out of sync with the tools in base. This project aims to address that problem.