July 11, 2025

The FreeBSD Foundation kindly sponsored my trip to Ottawa for the BSDCan 2025 conference and FreeBSD developer summit.  We had the usual two-day developer summit on June 11th and 12th, followed by the conference proper on the 13th and 14th.  Per my usual routine, I took the train from Toronto to Ottawa to attend  BSDcan, this time with the wrinkle of bringing my ARM Morello desktop along for use in Brooks Davis’ talk on CHERI and upstreaming support for it to FreeBSD. Ed Maste kindly picked it up and drove it for me, which made my trip to Ottawa much easier; on the way back I had to lug it along on the train in an oversized luggage case.  I had never tried to transport a desktop computer that way before and I was pretty relieved that it still booted up fine when I got it back home!

The first day of the developer summit consisted mostly of talks, with generous breaks in between to give folks a chance to chat and catch up, or hack on some side project.  During conferences I usually have some small project or two that I work on during breaks and in the evenings at the hacking lounge in 90U; this time it was some GDB scripts for kernel debugging, prompted by a discussion with Kristof, our esteemed pf maintainer.  At the time of writing, I have not yet finished what I wanted to get done during the conference, but I really will finish it soon!

The main highlight for the first morning was the usual core@ update, where attending FreeBSD Core Team members presented updates on various topics.  Of particular interest to me was a draft policy on the use of LLM-based programming tools in the FreeBSD project.  To summarize quite heavily, the policy will forbid the incorporation of LLM-generated code into the project, while allowing their use in the development process in other ways, e.g., to help review patches, or to help write commit messages or other content that is not explicitly licensed.  The policy comes out of a desire not to “taint” the FreeBSD project with code of dubious provenance; it is well-known that many LLM models are trained on code with licenses incompatible with the BSD license that we strive to use everywhere in the project, and thus far there is not much legal precedent to suggest that we would certainly be safe from copyright violation claims should the project decide to incorporate their output.

 

On the face of it, this seems like a reasonable policy: it tries to balance the need to preserve the integrity of the project’s licensing (a big draw for large *BSD users) with the general desire to use these new tools to aid development. While the use of LLMs for programming does feel rather overhyped these days, I do find them useful for certain types of work[*], and during the ensuing discussion I was a bit disappointed by what I perceived as a quite negative stance towards LLMs in general from the room.  Nonetheless, core@’s approach feels even-handed and in line with other large OSS projects, and I’m interested to see how the landscape developers over the next few years.  My personal view is that–licensing considerations aside–we should encourage expert developers to leverage LLMs as much as they are willing to.  Longtime OSS developers are already quite used to scrutinizing and tweaking code that we did not write ourselves, and I don’t see why that same skepticism shouldn’t be enough to gate sub-par LLM outputs.

After lunch, we had a talk by Rick Miller from Verisign on the use of FreeBSD as part of a defense-in-depth strategy for core Internet DNS infrastructure.  He presented on the general use of OS diversity as a way to improve security, and on why FreeBSD in particular is a good candidate for one of the operating systems to use as part of that strategy.  As part of such a strategy, Verisign uses both Linux and FreeBSD in similar roles within their infrastructure, and even uses different application frameworks on each OS to further reduce their reliance on a single technology stack.  In particular, while Verisign’s applications use DPDK on Linux, they can also use FreeBSD’s Netmap framework to get similar low-level access to network hardware.  Rick also described various kernel security vulnerabilities that were present in one of Linux and FreeBSD but not the other, though I would expect that a large majority of CVE-worthy kernel bugs are highly OS-specific, given that many of them a memory safety bugs.  I found it impressive that Verisign commits so fully to such a strategy and hope to see more examples of this in the future. 

Following Rick’s talk, I and other members of the FreeBSD srcmgr team gave a presentation, similar to the morning’s core@ update, where we talked about what the srcmgr team has been up to and the problems we are working on solving.  Of particular note was a call for lurkers to join the team and participate in calls without being official members of srcmgr.  The aim there is to give interested developers a chance to participate in srcmgr discussions and bug/PR triage work without having to commit fully.  We had quite a few interested developers reach out, and now, several weeks later, the srcmgr biweekly calls have nine attendees instead of the usual four.

On the second day of the developer summit, I attended Brooks Davis’ talk on CHERI and the motivations for upstreaming support for it to FreeBSD.  The entire talk was given using the ARM Morello system that I had brought up from Toronto; the entire KDE desktop stack down to the PDF viewer used to render the slides ran in so-called pure capability mode, wherein all C pointers are represented using CHERI capabilities.  In this mode, many types of memory safety bugs cause the affected application to fail-closed with a deterministic crash rather than leaving it open to exploitation.  Brooks described several ongoing commercial efforts to create CHERI hardware, generally based on RISC-V, and the path to upstreaming from CheriBSD to FreeBSD.  CheriBSD contains quite a few large-scale changes, some of which are not viable upstream; hybrid mode, in which the kernel runs as a typical aarch64 binary while support pure-capability userspace applications, is an example of this.  The patches to enable hybrid kernels are quite invasive as they require explicit annotations for all userspace pointers as they are manipulated in the kernel.  As a part of the talk, Brooks also demoed several CHERI features, notably CheriBSD’s library-based compartmentalization (c18n).

 

I attended quite a few good talks during the main conference.  Highlights for me were:

  • ELF Nightmares, GOTs, PLTs and Relocations Oh My, by John Baldwin, wherein he gave a whirlwind tour of various data structures used by the static and runtime linkers to set up execution of ELF binaries.  Many of the topics, e.g., function call indirection through the PLT, are quite hard to grasp for the first time; in my experience, one benefits a lot by hearing such topics explained several times in several different ways–after a while, things slowly start to click.  John included a few excellent visualizations of various steps of runtime linker operation, very useful for helping these concepts sink in.
  • ABI Stability in FreeBSD, by ShengYi Hung, which described a new tool, ctfdiff, which lets one compare the CTF (Compact C Type Format) data of two files to detect differences in their type definitions and function signatures.  CTF info is derived from the rather more complex DWARF debug info format; it encodes the layout of all kernel types, as well as building an index of the types and signatures of all functions and global variables.  The idea here is that one could use to compare kernels built before and after a change and programmatically determine whether the change is likely to have changed a binary interface used by out-of-tree kernel modules, such as those used for graphics drivers or Virtualbox.  Binary interface changes are a common source of problems for FreeBSD users, especially desktop users, and this kind of work is a good step towards improving the current state of affairs.  In particular, today, we rely on developers to manually bump a version number when breaking changes are introduced, but this is quite easy to overlook.  After the talk we had a spirited discussion on how best to employ the tool.  One suggestion that I raised was to consider comparing the CTF type graphs of kernel modules with that of the kernel that they are to be loaded into, rather than comparing successive versions of the kernel, since the latter approach will generally lead to lots of false positives, i.e., changes that are not actually relevant to the de factor kernel binary interface.
  • Improvements to FreeBSD KASAN, by Zhuo Ying Jiang Li, which analyzed several shortcomings of FreeBSD’s KASAN implementation and presented approaches to solve them.  In short, KASAN is a mechanism to detect various types of memory safety bugs in the kernel, leveraging compiler instrumentation and a small runtime component (see sys/kern/subr_asan.c in the FreeBSD source tree) to promptly catch out-of-bounds accesses and use-after-free bugs.  She noted that FreeBSD’s implementation has two main problems, both related to the integration of KASAN with UMA (the Universal Memory Allocator, the FreeBSD kernel’s slab allocator):
  1.  It doesn’t take care to insert padding between successive structures in a slab.  While KASAN will opportunistically mark pad bytes as invalid if they arise naturally, it does not change UMA’s slab layout algorithm to explicitly add “red zones” between objects in a slab.  Such red zones are important for detecting buffer overruns, as without them, KASAN cannot properly identify, say, a memory access one byte beyond the end of an object, as that address will often belong to the next object in the slab.
  2. UMA does not quarantine freed objects, which, combined with the allocator’s default LIFO allocation scheme, reduces the likelihood that KASAN will detect certain use-after-free bugs.

 Zhuo Ying then described her approach to solving these problems through modifications to UMA, and during the question period afterward we discussed some strategies to leverage existing features in UMA to solve these problems.

 I’m hopeful that these patches will be upstreamed in the near future.

Many thanks to the FreeBSD Foundation for sponsoring my trip.

[*] For better or worse, this report was written without assistance from LLMs.

– Contributed by Mark Johnston