My wife is allergic to mustard. It’s a fairly common allergy, but it’s also a fairly common additive, so whenever we do the groceries, we check the ingredients of everything we buy. Having mustard as an ingredient means we can’t buy it, but so does “may contain mustard” or having “spices” as an ingredient: “may contain mustard” reads as “may require a trip to the nearest hospital,” and “spices” reads as “Russian roulette, but with food!”

Putting the list of ingredients on the package has been mandatory since the early twentieth century, but food “purity” laws banning certain additives from certain foods go back to the sixteenth century with the Reinheitsgebot (1516), which still governs what you can call beer in Germany. Without laws and regulations like that, we’d have a hard time finding safe food to eat.

Compared to food, software is obviously centuries behind: we’ve had prepared foods since the very start of humanity, while software appeared less than a century ago (two if you want to be generous with Ada Lovelace). It has, however, become a vital part of our daily lives: we can hardly do anything (including buying food) without software anymore.

With the ubiquity and importance of software in our daily lives and in every business, the lack of transparency about its contents is becoming a real liability. European regulations (i.e. the EU Cyber Resilience Act (CRA) and the NIS2 Directive) and US guidance on SBOMs (e.g. CISA’s guidance on the minimum contents of SBOMs) comprise global governments’ reactions to this liability, shifting it back from the customers and end-users of software to its creators.

Their approaches are different: where the EU mandates supply chain security (which includes SBOMs) and legally places the liability for vulnerabilities on the vendor, the US makes it part of their purchasing agreements, wielding the government’s purchasing power and prompting large companies (such as public utilities) to do the same. The effect is the same, though: if a software or “intelligent” equipment manufacturer wants to do business in the largest economies on the planet, they have to secure their supply chains and be transparent about the contents, the ingredients, of their software.

The trigger

One of the drivers for this was a wake-up call the entire software industry received with the SUNBURST, 2019 SolarWinds compromise, attributed to APT29, a.k.a. Cozy Bear. “APT29 used customized malware to inject malicious code into the SolarWinds Orion software build process that was later distributed through a normal software update (…). Victims of this campaign included government, consulting, technology, telecom, and other organizations in North America, Europe, Asia, and the Middle East.” (ibid.). The entire list of compromises, tools, and techniques they used makes for fascinating reading, but the end result, and the wide impact the compromise had, was the shocking part at the time: they were able to build a backdoor into the Orion software suite, which itself is a critical part of network security for many large businesses, and use it to compromise those targets.

SUNBURST wasn’t the only impetus for the singular focus that regulators and cybersecurity and DevSecOps professionals have shown on software supply chain security, however: a 2021 vulnerability found in Log4j, a popular logging library for Java, caused several sites to be “preventatively” shut down while fixes were deployed. I wrote about it at the time calling out “security over convenience”, and a few days later explained how to get rid of Java without getting rid of Java code as a mitigation to Java’s rather lax approach to security.

While my initial reaction at the time was to contain the threat through virtualisation (VMs, containers, …) and sandboxing, parts of the industry jumped on the software supply chain security aspect of the problem: the Log4j crisis demonstrated that vulnerabilities are often buried in subcomponents that organisations didn’t know they had.

Shifting responsibility for securing software from users, adopters, and customers of software vendors back to the vendors places a significant burden on the vendors’ DevSecOps practices. Individual developers will need to be sure that the changes they make to existing code don’t introduce new vulnerabilities early in the development process. Even (or especially) if they rely on AI to write part of their code, AI is trained on buggy and vulnerable code and will spit that back out when you use it. I wrote about this a few weeks ago on my other blog.

Does that mean you should start manually checking all your dependencies, though?

Automation: reduce time-to-market and reduce developer toil

Continuous Integration, Continuous Deployment, Continuous Testing, and Continuous Delivery all mean the same thing, but from slightly different angles: they mean automation. Automate the build, unit tests, integration tests, and system integration tests. White-box testing your code against its internal interface and class contract (unit testing), grey-box testing your components against their interface contracts (integration testing), and black-box testing the system as a whole against its highest-level contracts (system integration end-to-end testing) can and should all be automated. Only acceptance tests should involve humans once written, and even then, for the most part, they should be an inspection of the scenarios tested during end-to-end testing, followed by running those scenarios.

Static Analysis, Static Application Security Testing, Dynamic Application Security Testing, and Software Composition Analysis all belong in this pipeline as well. So should generating an SBOM and flagging vulnerable dependencies and their available updates.

The pipeline should tell you whether your code conforms to your own guidelines and consistently uses the style you approved. This is what basic static analysis gives you. C# has Roslyn Analyzers built into the .NET build system; the Java ecosystem has linters like Checkstyle, PMD, and SonarLint — pick one and stick with it; Python and Node/JavaScript/TypeScript (please use TypeScript) each come with linters in their respective ecosystems; C++ has clang-tidy as its obvious choice. Regardless of which ecosystem you use and what rules you want to impose on your code for maintainability, pick a popular, well-maintained tool and use it.

The pipeline should further tell you what the coverage of your unit tests is. Most unit testing frameworks can generate coverage information, and validating that coverage against a baseline minimum (which should be at least 85% for hand-written code and 95% for AI-generated code) is a simple matter of using vln-devsecops/actions-validate-coverage: Validates test coverage from Clover, Cobertura, or JaCoCo XML reports against a configurable minimum threshold, failing the workflow when coverage is below the required percentage.

The pipeline should also indicate whether your code has recognisable defects, especially those that affect your application’s security. This is what static application security testing (SAST) brings to the table. There are plenty of free tools that help you do this. One popular option is SonarQube, a freemium tool that supports multiple languages in its free version. Typically, setting up a SAST tool like SonarQube to run on every pull request requires a bit of setup: it needs a PostgreSQL database to store its configuration and previous findings, and the necessary scaffolding to run it. Nobody has time for that hassle, especially when you’re working on a zero-budget volunteer project, so I decided to create a GitHub Action for you: vln-devsecops/actions-sast-sonarqube: Perform SAST using SonarQube CE. The action will perform a baseline scan whenever you push to your default branch (which it helpfully provides a reusable workflow to do), and use that baseline to tell you what was introduced in your PR (which it also helpfully provides a reusable workflow to do). Once written, which took my AI sidekick and me a few days, it took all of thirty minutes to roll out across all my ~150 repos.

The next part of the pipeline is dynamic application security testing (DAST). The best free tool on the market seems to be the Zed Attack Proxy (ZAP), and I intend to explore it in the future, but as with anything dynamic, it needs more scaffolding than the static tools (which typically only need access to the code and build artifacts). More to come on this.

Next in line is software composition analysis (SCA) and SBOM generation. For languages that have their own package managers, like Java’s Maven or C#’s NuGet, generating an SBOM has become part and parcel of those managers: you can use the CycloneDX Gradle plugin to generate an (at least initial) SBOM straight from your build process in Java, use cyclonedx-dotnet to generate one for .NET, etc. Where things get hairier is when you’re using languages not directly supported by the CycloneDX community, but even there you can use a tool like cdxgen.

Going a bit further, I’ve made SCA and SBOM generation as easy as I could with vln-devsecops/actions-sca-syft-grype: Software composition analysis (SCA) using Syft and Grype: just use the reusable workflow from the repo and you’ll get an SBOM and a vulnerability scan in Grype.

Back to the kitchen

Just like cooking a good meal, development should be a worry-free exercise of creativity with a set goal. Whether that goal is to create the next great app or to feed your family, you don’t want to spend your day with churn and toil when you can be creative instead. As a developer, you want to develop “for fun and profit”. So automate the boring bits and have some fun! Integrate SA, SAST, and SCA in your automations, do automated end-to-end testing, and let automated workflows handle the churn.