By Florian Verdonck, Open-Source Software Engineer
This is the second blog in a two-part series from one of our open source software contributors.
With our researchers and engineers becomingly increasingly frustrated at the time taken to code and deploy – as well as the value benefits reducing the time to commit would bring – we embarked on a project to go faster.
Open source coders can have a difficult time with particularly large pull requests. Many programmers work on open source only part-time and can have trouble setting aside the hours to properly investigate, test, evaluate or even read through a particularly meaty code contribution. Sometimes their employers don’t necessarily view it as being mission critical or don’t understand the purpose of improvements designed to enhance the experience of the entire community.
What’s more, the progress on any open source project can be difficult to predict. Advances can come in fits and starts. That can happen when you’re building any software, but it’s particularly difficult to manage open source contributors who may be giving their time and expertise as part of a passion project, to enhance their developer résumé, or just to flex their coding expertise and publicly prove their worth. Every open source contributor develops code for his or her own reasons, and getting alignment on release schedules can be particularly challenging, especially if you are not in close contact with the maintainers of whatever project you are working on.
I saw this problem firsthand as I worked with the F# team. None of the people responsible for the project had looked into our PR in depth and it became problematic when we tried to get anything merged.
So, in early November 2022, we organised a closed virtual call with the F# team, walking them through everything we had done so far. It was useful to get the team up to speed and we got some insight of things we still needed to address.
In January 2023, we found ourselves in a similar situation, where the PR was large, and we wanted to get a proper review from the team. Since there were many new people in the F# team, it was challenging to get a good review from them. As they themselves were new to a lot of things, reviewing this properly was perhaps a little early for them.
Luckily, I was invited to speak at the NDC London conference near the end of January, where I spoke about my work in F#. While I was at NDC, I met again with Don Syme, where I asked him to look over my code.
It felt like a bold move. I was advocating for my code contribution directly to the person most responsible for the entire project. But Don’s gracious on-the-spot review sparked the rest of the F# team to take note and finally give our code a close look.
In February, the PR got merged behind a feature flag. This was a great achievement but it still had a flaw. We discovered that the result of the compilation was not entirely deterministic. If you work anywhere around computers, you know exactly how bad this is. It means that each time we ran the code, we could not be certain we would get the same output from the same inputs. No good.
Hear from Florian, Open-Source Software Engineer, on the challenges and breakthroughs behind Project Velocity, an internal initiative aimed at enhancing the .NET developer experience.
Read moreBecause the binary was not exactly the same in subsequent compilations, we were very anxious about using this. From experience, I can tell you that it is very frustrating to try to find out why certain bits in a binary are slightly shifted between runs. This felt like a needle in the haystack to find out what the difference was between the variants.
At the time, I couldn’t find a good way to spot the difference and all I had to go on was that the embedded resource signature data was not the same. That resource file is internally being used by the compiler and there is no public API available to serialise or deserialize the information. Because I couldn’t read the resource file, I started dumping the same data to JSON via a feature flag.
We immediately began working to figure out exactly what was causing this determinism problem. It took time and careful examination of the code.
Eventually, we found out that some generated names in the signature data could be slightly different for different compilation runs. This was the type of error that can be devilishly difficult to hunt down and eliminate. It wasn’t just one thing happening in just one place. We discovered that some mutation was happening on multiple threads and that could lead to a slightly different result.
In April, my second attempt at fixing this made it in. The merge date of this PR was unfortunately too close to May, (the release month of the next .NET SDK) so that fix didn’t make it into the 7.0.300 release.
This is another frustrating aspect of contributing to open source. If the stars don’t align, it can take months before your change gets available in a public release. Even if there are regular, routine private updates to whatever code you are using, there’s no clear way of ensuring that your work will make it into the public release and be integrated into what everyone uses. The open source software world relies on operators to update their software whenever a new release gets pushed out – some do, some don’t. Yes, there are nightly releases of the SDK but getting those into G-Research is difficult, and the teams won’t ever use those anyway as their main SDK to develop against.
The next release window for the .NET SDK would be in August 2023. We started testing more projects via the nightly SDKs and eventually still found some problems here and there. I was able to verify all the F# projects in the large solution over at G-Research and assert that the same binary was produced with and without the feature flag in subsequent runs. This doesn’t mean there are no more issues left, but as time went on, nothing major surfaced.
In August, the 7.0.400 SDK was released with our feature behind a flag. This was it, it was working nicely and we started using it in various open source projects to further experiment with it.
Things were finally shaping up and then I learned that G-Research could not use the 7.0.400 SDK due to a massive slow-down. Some users found that their builds were now as much as 10 times slower. I had designed something to make the builds faster and it had had the opposite effect. Boy, here we go again 🙃.
After a careful examination of the problem, I was able to determine exactly where the difficulty was. Because the teams at G-Research are mixing C# with F#, any change to the mix can affect the results. And the improvement I had engineered to F# was now throwing off what C# expected to get at a certain time, slowing the overall output significantly. The problem was not coming from anything I had coded, but I still wanted to make sure that the solution I had engineered would work cleanly.
I felt confident that it would come together with the .NET SDK update to version 8.0.100.
A large organisation like G-Research cannot just update its .NET SDK at any point. There are a lot of teams that need to coordinate this move, and in December 2023, they started to move to 8.0.100. At the time, there was also another technical migration going on within the business, so it was unlikely anyone would have any time to start investigating the new feature. January 2024 was also not a good time for the feature to be picked up by the teams as another larger initiative was taking place.
By February 2024, I was feeling adventurous. I raised the internal pull request to flip the switch myself. It was approved and on February 8, my PR was merged. The results were immediate and impressive. The solution build went from 143.3s to 121.1s – almost 15.5 percent faster! It makes a big difference to our users, helping the researchers operate more efficiently and without frustrating delays.
Getting this performance improvement was the product of a lot of hard work and persistence. The 15.5 percent performance improvement we achieved is something I am particularly proud of as I know that it really makes a real difference with the type of high-performance computing that G-Research does. It translates directly to better outcomes for our downstream users.
After achieving such a hard-fought-victory whose outcome was uncertain, I’ve been reflecting on the project and the lessons learned. After all, I had proposed a big change to the code, and most users would only notice a marginal benefit.
This kind of change, which requires a lot of work behind-the-scenes and doesn’t produce huge obvious benefits for all users, represents an especially challenging space within open source software. Because of how open source projects are governed by their distributed maintainer communities, there is a strong preference for small incremental changes designed to eventually add up to the paradigmatic change needed. This is an intuitive and sensible approach that often works but which sometimes prevents programmatic progress.
Because of this approach, there is often little reason for an independent contributor to advocate for major changes to a project’s code and push it through as I did. It’s much easier (and usually preferred) to get one or two minor pull requests approved than it is to successfully introduce large paradigmatic changes to the codebase. In the context of a project whose contributors are exploring, imagining, testing and running the code in their spare time, the changes must be small enough that they can be processed and approved within a reasonable amount of time and effort. If it can’t easily be approved, or if the value isn’t immediately obvious, good code can be passed over simply because there’s too much of it.
This problem can be visualised as the overlap in a Venn diagram that goes something like:
Many projects in open source software have reached a stage in their maturity where, for contributors to get them to do what they want, it requires a substantial re-architecting of the established codebase. This can take compromise and the sort of immense effort that I put in. More often, it leads to a splintering of projects, a disunity of effort and attention, with good coders duplicating functionality to secure minor improvements or improved utility for extreme corner-cases. Instead of putting in the work and sharing improvements with everyone, it’s easier to only solve smaller problems, which leads, inevitably, to the fragmented open source software universe we see today.
I’m grateful that I could invest the time and energy to explore this issue in the depth it required. Not everyone can go deep, like I did, to really get to the essence of the technology that underpins so much of what we all use. I feel confident that my efforts to forge a better open source software solution that can benefit big, early adopters like G-Research today, may, in the future, be applied to a much wider universe of users.
Explore our tools and learn more about how we contribute to the open-source ecosystem.
Hear from Florian, Open-Source Software Engineer, in the second part of this two part series, on the challenges and breakthroughs of an internal G-Research initiative aimed at enhancing the .NET developer experience at scale.
Read article
Each month, we provide up to £2,000 in grant money to early career researchers in quantitative disciplines. Hear from our December grant winners.
Read article
We were thrilled to welcome James Maynard, Fields Medallist 2022 and Professor of Number Theory, at the Mathematical Institute in Oxford, on stage for the latest Distinguished Speaker Symposium last month. James’ talk on Patterns in prime numbers hones in on unanswered questions within mathematics and the recent developments that have brought the solutions to those problems closer to reality. Hear more in his exclusive interview with us.
Read article