BSDCan 2019 Trip Report: Mark Johnston

Thanks to the FreeBSD Foundation, I was able to make the trip from Toronto to Ottawa to attend BSDCan 2019 and the FreeBSD developer summit. Following the conference, I also made it to a small hackathon held at the University of Waterloo. I work from home, which can create a sense of isolation despite the ability to easily communicate with colleagues over the Internet; conferences are thus an important way to recharge my enthusiasm for working on FreeBSD. This year’s BSDCan was not a disappointment: I attended a number of interesting talks, collaborated on some designs for future projects, and helped review and debug some code.

I arrived on the evening of the 14th and followed a long-standing tradition by heading to the Royal Oak to meet up and chat with some other developers. The following morning saw the beginning of the two-day developer summit. The morning was devoted to three presentations: by the Foundation, the BSD team at Intel, and an engineering infrastructure developer at NetApp. The last talk described the continuous integration (CI) system used in the development of ONTAP, the FreeBSD-based operating system used by NetApp filers. ONTAP has a large, complex, highly active code base with a long history, which is to say that its development requires a good amount of rigor and testing to maintain quality. To achieve this without slowing development to a crawl, NetApp uses a heavily automated CI process with many safeguards to protect against erroneous changes to the ONTAP code. The talk described various features of this system, and left me and many other developers salivating: the complexity and scope of ONTAP’s CI system is beyond the reach of most smaller engineering organizations, which end up making do with less automation and safeguards (or none at all). The CI system used for FreeBSD proper has advanced quite a bit in the past year but still runs only after a code change is committed to the repository, so developers are left without much support for validating their changes prior to commit. The NetApp presentation was a useful illustration of how advanced a CI system for a FreeBSD-based product can be, and will hopefully help inspire
advances in our own CI system.

Following lunch I participated in a working group, led by Cy Schubert, to discuss several features we wish to add to FreeBSD’s swap pager, in particular, online TRIM for freed blocks, and swap device prioritization. Currently we have no support for notifying the swap device when a block is freed (for example, when a block backs memory that is being unmapped because a process is exiting), and with certain types of drives this can lead to suboptimal write performance. Since SSD-backed swap devices are now prevalent and actually yield reasonable performance in many cases, this is becoming more important. Following the conference we have started designing some requisite changes to the swap pager. Swap device prioritization is useful in contexts where multiple swap devices are available but have different latencies, for example if one is using some combination of a compressed memory-backed device, an SSD and a HDD. In general, one would want to use the fastest device possible but currently FreeBSD just allocates blocks from all devices in a round-robin fashion. A basic version of this feature would be relatively straightforward to implement, but requires a new interface to replace or augment the rather limited swapon(2).

The second day of the developer summit included a presentation by the core team about various initiatives. Of particular interest was the announcement of a working group to investigate a transition to git from Subversion. Like many src developers I already use git for most of my day-to-day work that doesn’t directly involve committing, but I believe the stated claim that switching to git will encourage casual contributions. Much of the discussion centered around two issues: the lack of a monotonic revision number that we get with git, and the question of whether FreeBSD’s git repository would be hosted on GitHub. In my opinion the former is more of an inconvenience than a blocker: it is possible to use git itself to determine the ancestral relationship between two commits. The latter is more of a non-technical issue. It was pointed out that if we were to rely on GitHub, it would be prudent to ensure that we could decouple from it if that somehow becomes necessary in the future. To some extent this precludes tight integration with various GitHub features. I hope that the transition to git will be an opportunity for the project to build some official tooling for various workflows; today, developers must do many things manually or implement their own automation, for example for MFCs or creating Phabricator reviews. The transition to a new VCS is a good opportunity to improve on that situation.

In the afternoon I spent some time hacking on a few miscellaneous tasks, and in the evening John Baldwin and I discussed his implementation of breakpoints in bhyve’s GDB stub. This will allow developers to run GDB against a bhyve instance and set breakpoints in the guest kernel, which will be extremely useful for those who, like me, use bhyve extensively for kernel development.

On Friday the conference proper began. First I attended a talk by Elena Mihailescu on her work implementing live memory migration in bhyve. Prior to the conference I had discussed some of the design problems with her by mail, and was happy to see that she had been able to complete an initial implementation. After that I went to a talk about implementing a DRM driver, which helped clarify my limited understanding of how they are architected. The DRM framework provides many different abstractions but it can be hard to find information on how they are stitched together. Emmanuel’s talk helped define the taxonomy of various DRM objects and provided some useful references.

After lunch I attended a talk on NetBSD’s RISC-V port. Having done some work on FreeBSD’s RISC-V port, it was interesting hearing about some of the challenges faced by a different group, and about which pieces of FreeBSD they used as a reference. I found out during the talk that NetBSD has done some work to deduplicate the page table (pmap) code across different architectures. In FreeBSD each supported platform has its own independent pmap implementation, which makes for more involved maintenance, since the same bugs often appear in multiple pmaps. In the evening I spent some time talking to Steve Kiernan about some issues encountered by his company when doing a port to a new CPU architecture, and learned more about John’s bhyve GDB breakpoint implementation. I was also able to get some feedback on how to properly design a driver for ACPI-enumerated I2C devices that are commonly used for trackpads on newer laptops.

On the final day of the conference, I first attended a talk describing OpenBSD’s unveil() system call, which is used to restrict filesystem access in programs which might otherwise have ambient authority. OpenBSD’s approach is rather different from that taken by Capsicum in FreeBSD, but both face some common challenges: many programs need to access one or more files specified in their command-line arguments, and it is generally troublesome to pre-authorize access to those files before entering a sandbox because there may be a large number of them. FreeBSD’s approach is somewhat heavyweight: applications can initialize a separate process that implements a whitelist for filesystem namespace accesses and opens files on demand. However, since unveil() is implemented entirely in the kernel, it is more restricted in how it may be used to handle a large list of file paths, and from the talk I learned that the OpenBSD developers are still designing a solution to this problem.

Later that morning I attended Kristof Provost’s talk on the use of VIMAGE to implement tests for the networking stack. Network stack testing is traditionally somewhat difficult due to the global nature of many network configuration settings, but VIMAGE addresses this problem more or less directly, and some infrastructure now exists for tests to easily set up and tear down minimal jails in which to run. Having spent some time recently trying to debug issues in our multicast code, the talk inspired me to start extending this infrastructure to implement regression testing for the multicast code.

Following the day’s talks and Dan’s always-entertaining auction, we went to a closing party in the Byward Market. There I met a developer who works on memory management in the Linux kernel. He described several architectural challenges being worked on by the Linux folks, some of which apply to FreeBSD. He noted that Linux developers have been working hard to support the use of superpages for executable text segments, since this can provide substantial performance gains by improving utilization of the instruction TLB. This requires work on individual filesystems to transparently use superpages in the first place. Interestingly, FreeBSD has been able to perform this optimization for years due to the architecture of its transparent superpage implementation, and indeed, it’s known to significantly improve run times for CPU-bound programs such as the compiler. This example helps demonstrate how markedly different FreeBSD and Linux are under the hood and how different design decisions can significantly alter the user-visible properties of a feature’s implementation.

After the party I went with John Baldwin back to the hacking lounge, where he stoically debugged a boot-time hang that’s been plaguing users of Ryzen-based laptops. (See bugzilla PR 231760 for details.) These types of issues are tricky to debug remotely since they typically involve compiling kernels with various debug tracing enabled, and as a result progress on the PR had stalled; however, at the lounge a user helpfully provided two laptops demonstrating the problem, so with a couple of hours of early-morning debugging he was able to find the problem, which should be fixed shortly. The next morning I took a train back to Toronto.

–  Contributed by Mark Johnston