Written by Jay Faulkner, Open Source Software Developer.
Open source projects enable people from all over the world to collaborate and create high-quality software that benefits everyone – they have, in effect, revolutionised the way we think about software development.
However, one of the most impactful and valuable benefits of open source, is also one of its most invisible and potentially time-consuming – maintaining a continuous integration (CI) system.
OpenStack is a brilliant real-world example of a project which focuses on maintaining releases and high quality CI.
Here, I’m going to open your eyes to the quality of CI and stable maintenance that underpins OpenStack.
Project Testing and Maintenance
OpenStack is a collection of multiple projects, each with its own policies. Part of what makes it special is that those multiple projects are continually tested together.
This rigorous testing, a part of the integrated OpenStack release, means when you download and use multiple OpenStack services from the same release, you can be confident they’ll work well together.
In addition to delivering an integrated product that works effectively, OpenStack also commits to maintaining those projects for at least 18 months – and for some projects, much longer than that. This means that, when considered beside the twice yearly OpenStack release schedule, at any given time at least three integrated releases are supported.
The enormity of that task is multiplied again by the number of deliverables that are released. During the OpenStack Zed cycle, over 200 different projects were released.
More than 80 of these deliverables follow the stable branch support policy, which means that in the best case; we have at least 400 supported deliverable/release combinations actively having continuous integration tests run against them.
How does OpenStack run all these tests?
Zuul CI system was created originally as an OpenStack project designed to help deal with the massive complexity of testing the integration of many separate projects. In 2019, it spun out into its own top-level OpenInfra foundation project.
Additionally, infrastructure originally inspired by OpenStack is now a part of the OpenDev Collabory, where OpenStack and many other projects still run CI today using Zuul. Most of the CI infrastructure traditionally associated with OpenStack is now under the OpenDev umbrella.
OpenDev runs over 10,000 CI jobs per day on behalf of OpenStack and many other projects. Running this many tests requires a large amount of hardware and administration. These hardware resources come from partners who have donated servers or cloud credits for use by OpenDev.
A group of system administrators from various companies help support OpenDev CI resources – including Zuul as well as other hosted development tools such as Gerrit and Etherpad.
As a longtime OpenStack contributor, I can attest that some of the most helpful people are the folks responsible for keeping the CI system, along with the rest of the infrastructure we use to create OpenStack, running well.
There is also a significant amount of effort put into testing software and tools such as DevStack and Tempest. Not to mention, for Ironic, the hardware-emulation tools such as Sushy and VirtualBMC – whose recent security vulnerability was featured in my previous blog post, the Invisible Work of OpenStack: Security Bugs.
Maintenance is Worth the Effort
The effort required to maintain stable branches may seem daunting, but it is worth it.
We hear frequently from operators that they appreciate being able to use older versions of OpenStack without needing to constantly upgrade. To them, OpenStack’s CI and maintenance cycle are its killer features.
In fact, in the 2022 OpenStack User Survey, we discovered that less than 10 percent of OpenStack installations are running the latest version. Keeping these users secure, then, is one of our primary directives.
G-Research understands that keeping open source software stable and secure requires investing in those projects. I’m glad to be a part of its investment in OpenStack’s future, and to be able to continue exploring the invisible work that goes into making OpenStack great.
