February 20, 2020

Committer’s dedication, a Foundation grant, and Raptor Computing Systems support accelerates FreeBSD on POWER9

The power and promise of all open source software is freedom. Another way to express freedom is choice — choice of platforms, deployment models, stacks, configurations, etc.

The FreeBSD Foundation is dedicated to supporting and promoting the FreeBSD Project and community worldwide. But, what does this mean, exactly, you may wonder. The truth is it means many different things, but in all cases the Foundation acts to expand freedom and choice so that FreeBSD users have the power to serve their varied compute needs.

This blog tells the story of one specific way the Foundation helps a member of the community provide greater hardware choice for all FreeBSD users.

Piotr Kubaj is a FreeBSD ports Committer based in Poland whose main field of interest is improving ports on PowerPC64 architecture. In a paper submitted at Asia BSDCon 2019, Piotr explains his motivation:

My goal is to make the PowerPC64 platform equal to amd64 in terms of software support. This includes having usual desktop-class software like web browsers or desktop environments available. It also includes having server-class software like database servers.1

Piotr began working with POWER systems in the Fall of 2018 as a Contributor, submitting hundreds of patches. “Supporting development efforts that have a big impact on the Project, like the work Piotr is doing, is an important part of the Foundation’s mission,” said Foundation Executive Director Deb Goodkin.

In April of 2019, around the time he received the POWER9 system from the Foundation, Piotr’s developer status was elevated to Committer, allowing him to commit patches on his own.

Piotr works on two POWER9 machines:

  1. A remote server provided by Raptor Computing Systems
  2. A local workstation funded by the FreeBSD Foundation

He uses both systems so he can work in parallel on STABLE and CURRENT FreeBSD branches and because some ports need direct, local access during testing. STABLE and CURRENT run a different version of the Executable and Linkable Format (ELF) specification. Piotr maintains the old ELFv1 environment for currently-supported STABLE branches, and the new ELFv2 on CURRENT, with additional patches. More on these and other technical challenges a little further down.

Why FreeBSD on POWER9?

Two factors in particular got Piotr excited about this work: openness and affordability.

Openness

POWER9 has a 100% open source firmware stack, as opposed to modern amd64 hardware (both Intel and AMD), which is proprietary, as this article explains. In this article and others, the interesting irony that emerges is that while chip makers like Intel justify the inclusion of proprietary firmware for security reasons, these systems’ privileged access raises new security concerns. For instance, on May 1, 2017, Intel confirmed a Remote Elevation of Privilege bug in its Management Technology.2

Another aspect of open firmware relates to the baseboard management controller (BMC) stack. Open BMC is particularly important for cloud-scale deployments for at least the following three reasons:

  1. Reproduce-debug-fix-deployment cycle: When an issue arises with a cloud-scale deployment, it is difficult to reproduce the same issue in a controlled lab environment. This makes it tough to find and fix the problem quickly. An open BMC stack allows faster debugging.
  2. Security models: Modern, open BMC implementations allow end users to leverage their own security models rather than forcing them to use older models with known vulnerabilities.
  3. Configuration and monitoring: Traditional BMCs require the use of their own tooling, whereas open BMC can inherit host OS tooling.

Affordability

With POWER Version 9, both servers and workstations are now increasingly affordable, with entry-level rack mounted and tower configurations available for less than $3,000. “This makes it a very exciting time for people who prefer completely open source solutions over proprietary ones,” said Piotr.

Technical Details and Challenges

LinuxTM systems already work great on POWER9, both on server and desktop, but FreeBSD is playing catch-up. There are a number of reasons why this is the case, including IBM’s historical support for Linux on POWER, FreeBSD’s move to LLVM which is still only recently-supported on POWER (since v8), popular Linux distributions are little-endian on POWER, while FreeBSD is still big-endian, and a general lack of knowledge among upstream software developers that anything but Linux and AIXTM runs on POWER.

Within the FreeBSD community, there are numerous efforts in both base and ports to improve FreeBSD on PowerPC. Top focus areas include LLVM, ELFV1 and V2, KDE/Chromium. In addition to Piotr, other contributors working on POWER9 include Justin Hibbits (jhibbits), Leandro Lupori (luporl), Brendan Begren (bdragon), Alfredo Dal’Ava Júnior (adalava), Mikael Urankar (mikael), and Mark Linimon (linimon).

LLVM

Since POWER now supports LLVM, the FreeBSD project is working to switch the FreeBSD port of POWER9 to LLVM. This work is nearly finished, says Piotr, though there are still some LLVM-related problems he is tackling. “For example, Mesa used to build with LLVM8, which is present in base, and when I tried with LLVM9, I hit a clang crash. I have a workaround for this – Mesa uses LLVM8 from ports anyway, so we can use it for compilation as well,” said Piotr.

The change to LLVM is tightly coupled with the ABI, because LLVM requires ELFv2, which has inconsistent support among other FreeBSD ports.

ELFV1 and ELFV2

There has been great progress in getting ports to build on ELFv1, and it now compares very favorably with other architectures, including amd64.

ELFv2, in contrast, is a completely different environment—new ABI, new compiler, and new linker. This means that some ports that don’t build on ELFv1 will build fine on ELFv2, but others that build on ELFv1, won’t build on ELFv2.

Because of this inconsistency in port builds, the Project decided to only switch to LLVM on CURRENT, with STABLE branches being left on GCC. Although this requires additional maintenance, it results in the best experience for FreeBSD users.

His desire to fix and maintain all ports on both ABIs is what drove Piotr’s need for two POWER systems:

  1. A remote system, running FreeBSD 12.1-RELEASE made available to Pitor free of charge by Raptor Computing Systems. It runs poudriere bulk -a all the time and the results are available at https://talos.anongoth.pl. Pitor uses this system to fix ports on ELFv1 environment.
  2. A local system funded by FreeBSD Foundation that runs FreeBSD CURRENT with patches for ELFv2 and that is used to fix ports on ELFv2.

Until recently, the main focus has been on ELFv1 because LLVM was still very buggy. With recent LLVM improvements, Piotr is now starting to hit limits of what old systems with GCC 4.2 can build. This in turn means he can shift more focus onto ELFv2. The Project looks forward to FreeBSD 13.x with a modern, in-tree toolchain with broad ports support on ELFv2.

KDE

Another problem is that many KDE programs, as well as a few others, use QtWebEngine (basically Chromium wrapped into a thin Qt layer) to display websites. Despite Google’s membership in the OpenPOWER Foundation, Chromium has almost no POWER compatibility. To make things more challenging, the scarce available compatibility is all for little-endian, whereas FreeBSD on POWER is big-endian. Patches exist that support both little-endian and big-endian and with these Piotr successfully compiled WebEngine, which is currently in review at https://reviews.freebsd.org/D22184. This could also be the first step to getting Chromium working.

What’s Next?

Once the switch to LLVM is complete, Piotr is excited to tackle the following TODO list:

  • GPU acceleration (many desktop ports need it) and it’s nice to browse websites / play games / watch movies etc. without stuttering
  • Getting Rust and Node.js (needed to build Firefox) and Firefox patches committed
  • Porting Chromium (will use patches from QtWebEngine)
    Other things that will pop up (share your ideas here)

In closing, we hope that this deep dive into Piotr’s work offers a window into how the FreeBSD Foundation supports the community and the project. Part of what makes FreeBSD such an exciting technology and community is the fact that this dynamic plays out in hundreds of ways, across any number of technical domains, every day. If this sounds like the kind of community you’d like to work with, we invite you to check out all the ways to get involved.

We’d love to hear from others in the community who have similar stories to tell, and if you found this blog useful, please share it!

1. https://papers.freebsd.org/2019/asiabsdcon/kubaj-powerpc64_architecture_support_in_FreeBSD_ports.files/kubaj-powerpc64_architecture_support_in_FreeBSD_ports.pdf 1↩ 2. https://www.theregister.co.uk/2017/05/01/intel_amt_me_vulnerability/↩