Guest Blog Post: Alexandru Elisei on Implementing bhyve for arm64

Check out the latest guest blog by former student Alexandru. Thanks to support from readers like you, we’re able to fund projects like this to help grow the FreeBSD community and further advance the Project.

At University Politehnica of Bucharest, the Bachelor’s curriculum includes two operating system courses: the first one, in the third year, teaches the fundamental principles common to all operating systems: processes, virtual memory, I/O. The second course is optional and is taught in the fourth year. Despite its name, the course, ‘Advanced Operating Systems’, is a course about writing Linux drivers.

Sitting at the boundary between the hardware and the user, operating systems have always been fascinating to me. This is why, by the end of my third year of college, I approached Mihai Carabaş, with whom I had worked in the past. I wanted to spend my summer working on a real operating system, and he had the perfect project in mind: the FreeBSD Foundation was offering grants to students to work on bhyve for the ARM architecture. bhyve (at some point in time, spelt BHyVe, short for FreeBSD HyperVisor) is a type 2 hypervisor, and Mihai had ported bhyve to the ARMv7-A architecture. Out of the two choices I was offered, I chose to work on porting bhyve to ARMv8-A, a port which I have called bhyvearm64. The other project, adding support for virtio-mmio to bhyve, was also completed at the university by another student.

When talking about Arm, the first thing that comes to mind is mobile phones. Nonetheless, Arm has been, and is, trying to branch into the server market with its latest architecture, ARMv8. Before talking to Mihai, I was aware of two separate companies that were designing Arm server CPUs: AMD (AMD Opteron A1100) and Cavium (ThunderX). In the meantime, AMD dropped the architecture to focus on x86 CPUs, but Cavium (with ThunderX2) and now Ampere are continuing the push for ARMv8 servers.

The project looked daunting at first: I had never before written code for an operating system, I had never before written code for Arm CPUs, and the architecture manual was almost 7,000 pages long. With Mihai’s help I was able to get past the initial stage of the project, and I soon found myself thoroughly enjoying the experience. Our university projects until then were mostly simple affairs, nothing on the scale of a full-blown operating system with millions of lines of code. “The Design and Implementation of the FreeBSD Operating System” was a very valuable resource all throughout the project, but especially during the early stages when I was first learning about FreeBSD. As I read more and more of the FreeBSD code, I started to become more familiar with the operating system and I hit the first important milestone: I was able to save and restore the CPU state for a virtual machine.

The next step in the project came naturally: to be able to execute instructions, the virtual machine needs access to memory. However, the hypervisor is required to control the memory operations that the virtual machine executes at all times. The performance overhead for implementing this control in software can be prohibiting. Arm designed a hardware feature to alleviate the performance hit, called stage 2 translation, which is similar to nested/extended page tables on x86. It works much like virtual memory, only that it uses the addresses generated by the virtual machine, not by user programs. Modifying the existing code to accommodate this new translation mechanism was surprisingly easy, perhaps a result of the monolithic approach chosen for the page table code.

With the CPU and the memory virtualized, the virtual machine was able to start running a FreeBSD guest. However, the guest would stop working very early in the boot process because the kernel expects certain parameters from the bootloader. I was using bhyveload to boot the kernel directly, and those parameters were missing. The easiest solution was to replicate the bootloader code and add it to bhyveload.

Booting the guest was accomplished just as the project was nearing its deadline. Five months had passed, and the virtual machine was able to start the boot process and print to standard output using bvmconsole. But bhyve on ARMv8 was far from being usable: the virtual machine wasn’t able to configure the interrupt controller, and thus wasn’t able to boot to userland.

I had enjoyed working on bhyvearm64 very much, so I decided that I wanted to continue the project to create a fully functional hypervisor for ARMv8. Being in my final year at the university, bhyvearm64 transitioned into the subject of my Bachelor’s thesis. Mihai and the FreeBSD Foundation were satisfied with the results, so they agreed to continue the sponsorship for the project and focus on the remaining features needed by the hypervisor: interrupt and timer virtualization. I would start working on the project at the beginning of spring, giving me a few months on which to concentrate on my studies.

During this time, although I didn’t write any code, I was still invested in bhyvearm64. At Mihai’s suggestion, I wrote a paper and submitted it to the AsiaBSDCon 2018 conference. To my surprise and delight, the paper was accepted and I was to present it at Tokyo in March 2018. The conference was very interesting, the people passionate and very knowledgeable, though I admit to being so smitten by Tokyo to dedicate an entire day to visiting the city.

April came and I started coding again. First thing on my agenda: the Arm interrupt controller, the Generic Interrupt Controller (GIC), version three. Judging from the size of the manual, seven times simpler than the CPU. In fact, deceivingly complex. Using hardware registers and memory mapped components, GIC virtualization required a mix of hardware virtualization features and software emulation.

With interrupt virtualization out of the way, one final piece of the virtualization puzzle was missing: timers. FreeBSD uses timers to schedule processes, to program actions to be executed at some point in the future and to count the passing of time, represented by ‘ticks’. Knowing timekeeping in a virtual machine to be a sensitive matter, the Arm architectural timer is composed of two separate timers: the physical timer and the virtual timer. The virtual timer is expected to be used by a virtual machine, and the physical timer by the host operating system. I was unsure of all the implications of letting the virtual machine have full control over the virtual timer, so instead I chose a trap-and-emulate approach for both timers. This is something that I want to revisit.

And there it was: a functional virtual machine running on an ARMv8.0-A simulator, and I was very proud of my work. Nonetheless, I have continued working on the hypervisor in my spare time. There are still certain features that need to be done before including the hypervisor into the mainline FreeBSD kernel, chief among them hardware validation and separating machine-dependant from machine-independent code in bhyve for x86 and arm64.

Working on bhyvearm64 has allowed me to get a better understanding of operating systems, virtualization and the ARMv8 architecture. This in turn made it possible for me to get accepted for an internship at Arm in Cambridge. Right now I’m an intern in the System Validation on Operating Systems team, and I have to admit, working at Arm has exceeded my expectations. As a fortunate side-effect, my being in Cambridge has allowed me to attend the BSDCam devsummit that took place at Churchill College this summer. Meeting the FreeBSD developers in person has been a humbling experience.

In closing, I would like to take this opportunity to express my thanks to Mihai Carabaş, to the FreeBSD Foundation and to everyone who made this project a reality.