2019 in Review: CI and Testing Advancements

One of the major goals for 2019 was to focus on improving the software quality of the FreeBSD codebase and extending the features of the FreeBSD project’s continuous integration. This year, we examined the test results and the test cases carefully with the goal of zero false positives. We believe this is very important to the Project because now the FreeBSD CI can serve as a trustworthy source for getting the latest build and test status of FreeBSD’s head and stable branches. A simple page for showing the building information of each architecture and branch combination is at http://tinderbox.freebsd.org

We added many new CI jobs for the different test needs and the hardware CI lab is online at https://ci.freebsd.org/hwlab

In January we began publishing the CI weekly report to the freebsd-testing mailing list. The archive is available at https://hackmd.io/@FreeBSD-CI. The report includes the build and test statistics, test case status and list of the bugs needed to be checked.

From the results of these systems, we worked with domain experts from the developers and contributors to fix various bugs in the systems or the test case and environment. The details of these efforts are in the weekly CI reports.

We are actively introducing FreeBSD to the CI of various software products, for example, jemalloc, zstd and QEMU. The completed and work in progress list is available at https://wiki.freebsd.org/HostedCI. They are good examples of adding FreeBSD to the CI of your project. And we are working with the hosted CI service interested in providing better support of FreeBSD.

For better connection and collaboration of the developer, contributor, and vendors around the world, we had Testing/CI working group at the developer summits in BSDCan  and EuroBSDCon 

Also of note this year, I was able to join the cluster administration team (clusteradm@). This can speed up updating the CI infrastructure in hardware and base system. In addition, this opportunity allows me to be able to help other FreeBSD.org services.

The FreeBSD Foundation has been sponsoring FreeBSD Project’s CI development since mid-2018 and that includes sponsoring the hardware. We have made many improvements in stability and scale to the CI infrastructure since then. In 2019, we moved the focus to improving the testing quality, and adding features and external collaboration. We are looking forward to providing more testing services to fulfill the need for the development of FreeBSD and the software running on FreeBSD in the year 2020. We expect more exciting news will be announced soon.