Putting the list of ingredients on the package has been mandatory since the early twentieth century, but food “purity” laws banning certain additives from certain foods go back to the sixteenth century with the Reinheitsgebot (1516), which still governs what you can call beer in Germany. Without laws and regulations like that, we’d have a hard time finding safe food to eat.

Compared to food, software is obviously centuries behind: we’ve had prepared foods since the very start of humanity, while software appeared less than a century ago (two if you want to be generous with Ada Lovelace). It has, however, become a vital part of our daily lives: we can hardly do anything (including buying food) without software anymore.

With the ubiquity and importance of software in our daily lives and in every business, the lack of transparency about its contents is becoming a real liability. European regulations (i.e. the EU Cyber Resilience Act (CRA) and the NIS2 Directive) and US guidance on SBOMs (e.g. CISA’s guidance on the minimum contents of SBOMs) comprise global governments’ reactions to this liability, shifting it back from the customers and end-users of software to its creators.

Their approaches are different: where the EU mandates supply chain security (which includes SBOMs) and legally places the liability for vulnerabilities on the vendor, the US makes it part of their purchasing agreements, wielding the government’s purchasing power and prompting large companies (such as public utilities) to do the same. The effect is the same, though: if a software or “intelligent” equipment manufacturer wants to do business in the largest economies on the planet, they have to secure their supply chains and be transparent about the contents, the ingredients, of their software.

The trigger

One of the drivers for this was a wake-up call the entire software industry received with the SUNBURST, 2019 SolarWinds compromise, attributed to APT29, a.k.a. Cozy Bear. “APT29 used customized malware to inject malicious code into the SolarWinds Orion software build process that was later distributed through a normal software update (…). Victims of this campaign included government, consulting, technology, telecom, and other organizations in North America, Europe, Asia, and the Middle East.” (ibid.). The entire list of compromises, tools, and techniques they used makes for fascinating reading, but the end result, and the wide impact the compromise had, was the shocking part at the time: they were able to build a backdoor into the Orion software suite, which itself is a critical part of network security for many large businesses, and use it to compromise those targets.

SUNBURST wasn’t the only impetus for the singular focus that regulators and cybersecurity and DevSecOps professionals have shown on software supply chain security, however: a 2021 vulnerability found in Log4j, a popular logging library for Java, caused several sites to be “preventatively” shut down while fixes were deployed. I wrote about it at the time calling out “security over convenience”, and a few days later explained how to get rid of Java without getting rid of Java code as a mitigation to Java’s rather lax approach to security.

While my initial reaction at the time was to contain the threat through virtualisation (VMs, containers, …) and sandboxing, parts of the industry jumped on the software supply chain security aspect of the problem: the Log4j crisis demonstrated that vulnerabilities are often buried in subcomponents that organisations didn’t know they had.

Shifting responsibility for securing software from users, adopters, and customers of software vendors back to the vendors places a significant burden on the vendors’ DevSecOps practices. Individual developers will need to be sure that the changes they make to existing code don’t introduce new vulnerabilities early in the development process. Even (or especially) if they rely on AI to write part of their code, AI is trained on buggy and vulnerable code and will spit that back out when you use it. I wrote about this a few weeks ago on my other blog.

Does that mean you should start manually checking all your dependencies, though?

Automation: reduce time-to-market and reduce developer toil

Continuous Integration, Continuous Deployment, Continuous Testing, and Continuous Delivery all mean the same thing, but from slightly different angles: they mean automation. Automate the build, unit tests, integration tests, and system integration tests. White-box testing your code against its internal interface and class contract (unit testing), grey-box testing your components against their interface contracts (integration testing), and black-box testing the system as a whole against its highest-level contracts (system integration end-to-end testing) can and should all be automated. Only acceptance tests should involve humans once written, and even then, for the most part, they should be an inspection of the scenarios tested during end-to-end testing, followed by running those scenarios.

Static Analysis, Static Application Security Testing, Dynamic Application Security Testing, and Software Composition Analysis all belong in this pipeline as well. So should generating an SBOM and flagging vulnerable dependencies and their available updates.

The pipeline should tell you whether your code conforms to your own guidelines and consistently uses the style you approved. This is what basic static analysis gives you. C# has Roslyn Analyzers built into the .NET build system; the Java ecosystem has linters like Checkstyle, PMD, and SonarLint — pick one and stick with it; Python and Node/JavaScript/TypeScript (please use TypeScript) each come with linters in their respective ecosystems; C++ has clang-tidy as its obvious choice. Regardless of which ecosystem you use and what rules you want to impose on your code for maintainability, pick a popular, well-maintained tool and use it.

The pipeline should further tell you what the coverage of your unit tests is. Most unit testing frameworks can generate coverage information, and validating that coverage against a baseline minimum (which should be at least 85% for hand-written code and 95% for AI-generated code) is a simple matter of using vln-devsecops/actions-validate-coverage: Validates test coverage from Clover, Cobertura, or JaCoCo XML reports against a configurable minimum threshold, failing the workflow when coverage is below the required percentage.

The pipeline should also indicate whether your code has recognisable defects, especially those that affect your application’s security. This is what static application security testing (SAST) brings to the table. There are plenty of free tools that help you do this. One popular option is SonarQube, a freemium tool that supports multiple languages in its free version. Typically, setting up a SAST tool like SonarQube to run on every pull request requires a bit of setup: it needs a PostgreSQL database to store its configuration and previous findings, and the necessary scaffolding to run it. Nobody has time for that hassle, especially when you’re working on a zero-budget volunteer project, so I decided to create a GitHub Action for you: vln-devsecops/actions-sast-sonarqube: Perform SAST using SonarQube CE. The action will perform a baseline scan whenever you push to your default branch (which it helpfully provides a reusable workflow to do), and use that baseline to tell you what was introduced in your PR (which it also helpfully provides a reusable workflow to do). Once written, which took my AI sidekick and me a few days, it took all of thirty minutes to roll out across all my ~150 repos.

The next part of the pipeline is dynamic application security testing (DAST). The best free tool on the market seems to be the Zed Attack Proxy (ZAP), and I intend to explore it in the future, but as with anything dynamic, it needs more scaffolding than the static tools (which typically only need access to the code and build artifacts). More to come on this.

Next in line is software composition analysis (SCA) and SBOM generation. For languages that have their own package managers, like Java’s Maven or C#’s NuGet, generating an SBOM has become part and parcel of those managers: you can use the CycloneDX Gradle plugin to generate an (at least initial) SBOM straight from your build process in Java, use cyclonedx-dotnet to generate one for .NET, etc. Where things get hairier is when you’re using languages not directly supported by the CycloneDX community, but even there you can use a tool like cdxgen.

Going a bit further, I’ve made SCA and SBOM generation as easy as I could with vln-devsecops/actions-sca-syft-grype: Software composition analysis (SCA) using Syft and Grype: just use the reusable workflow from the repo and you’ll get an SBOM and a vulnerability scan in Grype.

Back to the kitchen

Just like cooking a good meal, development should be a worry-free exercise of creativity with a set goal. Whether that goal is to create the next great app or to feed your family, you don’t want to spend your day with churn and toil when you can be creative instead. As a developer, you want to develop “for fun and profit”. So automate the boring bits and have some fun! Integrate SA, SAST, and SCA in your automations, do automated end-to-end testing, and let automated workflows handle the churn.

The paper, “Are We Actually There? Assessing RPKI Maturity” by Schulmann et al., is an incisive and comprehensive review of the maturity of RPKI and its application to global internet routing.

Like most users of the internet, I have largely ignored — and frankly been ignorant of — BGP, the Border Gateway Protocol, which is used to interconnect the thousands of networks that compose the internet. It’s a protocol I typically don’t think much about, but of course I use it on a daily basis, because I use Google, email, and so on, and so do you.

Anyway, BGP’s lack of robust security and its vulnerability to both innocent typos and malicious attacks does not surprise me, but it also has not concerned me as much as it probably could or should.

The internet backbone is not secure. Back in the late 1980s when the Border Gateway Protocol was created, everything was based on optimism, a friendly smile, and a handshake — something worked out on the back of a napkin over a few beers. And honestly, some of the nicer networking protocols came about exactly like that: discussed over dinner and beers. One of the protocols I actually do think about a lot was born that way as well.

Anyway, back in 2012, RPKI was standardised as a suite of RFCs. The SIDR working group published the core specifications in a batch: RFC 6480 as the architectural overview, followed by a cluster of RFCs covering repository structure (6481), the ROA profile (6482), route origination validation (6483), certificate policy (6484), algorithm requirements (6485), manifests (6486), the X.509 resource certificate profile (6487), the signed object template (6488), and several more covering key rollover, trust anchor locators, and provisioning. The RPKI-to-Router protocol — which defines how a Relying Party validator feeds validated data to BGP routers — followed about a year later as RFC 6810.

RFC 7115 came out in 2014 and laid out the best current practice for deploying origin validation, including the fail-open posture that most networks still use today. Then in 2017, RFC 8182 introduced RRDP, which is an HTTP-based alternative to rsync for pulling down repository data. That same year, RFCs 8205 through 8209 defined BGPsec, which goes beyond just validating where a route originated and actually cryptographically authenticates the whole AS path. RFC 8210 also came out in 2017 as an update to the RTR protocol.

Standard tooling was added in 2018: RFC 8416 introduced SLURM, which lets operators locally override RPKI validity decisions when they need to work around known misconfigurations or complex multi-homing configurations.

So RPKI has been actively developed since 2012 — over a decade now, almost a decade and a half. And yet it is not that widely adopted. As the ACM paper makes clear, even after more than a decade, it is not necessarily all that mature.

Here is what I mean by that, and what I think the authors of the paper mean by it. When you try to find a route to some other IP address, the best route — the shortest route — is through a gateway that has the most specific announcement for that destination. Routers announce which address prefixes they have access to, and a more specific prefix (more bits) wins out over a less specific one. Fewer hops is better, and a more specific prefix gives you a better path.

So your routing table tells you what your best path is, but the contents of that routing table can basically come from anywhere.

To secure that, the messages that fill up your routing table should be authenticated. Authentication is the cornerstone of everything in security. The question is: how do you authenticate those messages, how do you sign them, and what do you do when you have no authenticated messages to work with? The RFCs try to answer that, but as Schulmann et al. show, interpretations of what they mean differ between implementers, and the software that implements them is buggy and vulnerable.

The core issue is how that authentication works, and whether everyone agrees on how it works. If they do not agree, some messages may be dropped that should not be, because they are actually authentic. Some messages may be accepted that should not be, because they are actually not authentic. And some messages will simply be accepted because you have no idea whether they are authentic or not — the authentication data just isn’t there.

If only half of service providers actually publish data to authenticate their routing updates, and only a quarter of the networks that consume routing messages actually validate any authentication at all, and those parties do not agree on how to authenticate or how to sign their messages — that is where things get messy.

And that is basically what this paper shows: RPKI is not mature. People do not agree on how to authenticate their messages, and people do not agree on how to validate whether those messages are authentic. It is a bit of a mess.

Now, I am not typically a big fan of any American administration, but the previous one did get something right. They are trying to force the larger internet service providers to secure the backbone and at least reduce the possibility of BGP hijacks (and, as far as I can tell, the current administration has not rolled this back).

Here is what I mean by that. Back in 2021, Facebook had a BGP routing SNAFU that removed the routes for its authoritative DNS servers. That was a mistake — a misconfiguration. The only intentional BGP hijack I am aware of is when Rostelecom, in April 2020, misoriginated routes that diverted traffic intended for Google, Facebook, AWS, and Cloudflare through Russia. I don’t know exactly what they did with that traffic, but, well, you know: Russian.

The FCC has published a notice of proposed rulemaking that requires major providers like AT&T, Comcast, Lumen, and T-Mobile to file their BGP security plans, covering how they intend to implement RPKI and handle all of that, and to update those filings on a regular basis. Those filings will be confidential, but like anything filed with the government, they will be subject to access to information requests.

Smaller providers — essentially the kind that serve residential customers like me — do not have to file those reports, but they must still have their plans. Providers that can formally attest that they have registered and maintained active ROAs covering at least 90% of originated routes for IP prefixes under their control can be exempt from filing those reports.

That creates an incentive to avoid paperwork and simply create those ROAs, which can be automated. Automating the underlying problem rather than automating the reporting is generally the better long-term approach, and appears to be the outcome they’re aiming for.

The incentive is right: if you can attest that 90% of your routes have ROAs, you don’t have to file.

You also don’t open up a potential leak of your vulnerable paths, and you don’t create a centralised repository maintained by the government — accessible through access to information requests — that essentially tells anyone who looks exactly where the vulnerabilities in the backbone are and which addresses can be hijacked.

The US government wanting to maintain a list like that, especially when it is not mandated by Congress and is not shielded from access to information requests or kept as an actual secret, is a real concern. The government is not good at keeping secrets.

The other thing that came to mind when reading this is the question of who controls the internet. There are six Regional Internet Registries that can assign numbers to Autonomous Systems which, loosely speaking, consist of network operators who route each other’s data.

The shortest path isn’t necessarily the best path though: you don’t necessarily want the Elbonian government, or any Elbonian entity — including whatever large Elbonian businesses you care to name — to have any control over the part of the internet backbone your data flows through.

So my mind went to what an authenticated but distributed system might look like, one where participants trust each other and sign off on their own data — a web of trust, basically. The route your data takes through the internet is mostly invisible to you, and you can’t choose the ASes your data will flow through, but what if you could?

When I thought about that for a while, I went to PGP’s web of trust model, where participants sign each other’s public keys and use their own private keys to sign messages. So, for example, AS1, AS2, AS3, and so on would sign off for each other and say, essentially, “Yes, we trust each other — at least enough to not poison the network.” Rather than routing data through ASes blindly, you’d only route through ASes you trust to not “harvest now and decrypt later”, for example.

That’s one way of saying: we trust each other, we can vouch for what the other party said. And therefore you as a consumer can say, “Well, I trust my own ISP and yea set of networks — and they in turn trust others, so I’ll trust them as well” with the usual web of trust mathematics: if I trust you at 100% and you trust Joe, I’ll trust Joe at 80%, and if Joe trusts Alice, I’ll trust Alice at 80% of 80%, which is 64%, and so on. At some point, if the trust score drops below, say, 50%, you start being sceptical. If I have a route through someone I trust at 80% and another through someone I trust at 40%, I’ll take the one with 80%, even if it’s an extra hop, because at least it doesn’t go through a path I don’t actually trust. This would require a level of transparency in routing we don’t currently have, but it’s a level of transparency SCION provides to Swiss banks.

The other piece, though, is how you look up that data. HTTP and rsync, the two transport mechanisms specified in the RFCs so far, aren’t necessarily all that well suited to a distributed trust model. What you really want is a distributed ledger, which is where a blockchain comes in, because that is exactly what a blockchain is. If any one access point goes down, you still have a dozen or so other access points that can serve the same data. That data is built on consensus and can contain the trust information as well.

As it turns out, there is a patent from Huawei that identifies exactly this idea: using a distributed ledger — a blockchain — as a decentralised peer-to-peer network of cryptographically immutable historical records that are signed, hashed, and so on.

Now, I don’t have a commercial interest in any of this, so I’m not going to file a patent on the web-of-trust idea. You’re welcome. 😉

Abstract:

The advent of quantum computing lends urgency to the development of secure protocols that can be built entirely with post-quantum algorithms. This excludes the use of the family of algorithms Diffie-Hellman and Elliptic Curve Diffie Hellman belong to, as there are no algorithms in that family that are likely to be selected in the NIST program for post-quantum cryptography, and previous candidates have been eliminated due to security issues. This paper presents a new protocol for a secure mutually authenticated key exchange with perfect forward secrecy, based on the use of KEM algorithms.

Read it here.

Note:

The introduction is a little out of date: ML-KEM is now the standard that CRYSTALS-Kyber was expected to become when I wrote the paper. Other than that, I believe the paper is correct. The reasons I didn’t publish it two years ago are twofold:

1. I intended to write a section on why the schemes in the original Kyber paper don’t provide perfect forward secrecy (it’s because they’re missing the two ephemenral keys and two nonces), and
2. The paper had completely slipped my mind due to personal circumstances.

I should also note that, as an alternative to using an AEAD for the c_m value in the third message, an HMAC over the handshake can also be used.

It didn’t take long for the committee to recognise some of the issues DNP3-SAv5 suffered from: by the time Sergey Bratus and Adam Crain published their case study of DNP3-SAv5, “Bolt-On Security Extensions for Industrial Control System Protocols”, the DNP TC had already stood up the Secure Authentication Task Force (SATF; now Cybersecurity Task Force, CSTF) to study what we then thought would be SAv6.

The CSTF’s mandate was to design the replacement for SAv5, invite industry, protocol, and cybersecurity experts to the table, and create a secure protocol for DNP3’s unique use case. Within the DNP-UG, the CSTF is unique in that it does not require DNP-UG membership to participate in the discussion: we are intentionally open to a wider audience and have had members from every corner of the industry, including security researchers, join to help ensure the protocol’s security. Our discussions immediately turned to moving SA to its own layer and addressing some of the design issues in SAv5, making it “as simple as possible, but no simpler”.

In the decade that followed, we have done just that: we have removed complexity from both the Application Layer and from the security protocol itself, removed the need — or even the possibility — of pre-shared secrets, and introduced post-quantum cryptography into what has become a versatile protocol layer, DNP3-SL. This layer establishes an Association with mutual authentication using post-quantum asymmetric cryptography and, within that Association, can establish any number of subsequent Sessions using only symmetric cryptography. This design intentionally targets brownfield use cases with resource-constrained devices while still providing post-quantum security.

During that same time, the CSTF has also been developing a new protocol called the Authorization Management Protocol (AMP). This protocol addresses another niche that, while not unique to brownfield SCADA networks, is most prevalent there: the management of authorisation and security policies in a hierarchical network where individual devices do not have direct access to Policy Decision Points or to Certificate Authorities.

As with any large-scale development process, this one has taken far longer than we would have liked. We have run over the time we had allocated to the project by several years, due in part to our rigorous review process and in part to the fact that this is largely a volunteer endeavour. At every step of the way, we have answered questions, taken comments, invited participation, and endeavoured to improve the protocol. The latest input we received, admittedly through a side channel, was again from Adam Crain. This time, he argued that SAv6 was no longer necessary, and that he wouldn’t implement it.

Not implementing a layer of the DNP3 protocol is, of course, his prerogative. Some of the arguments leading him to that decision are based on faulty or outdated data, and some on his particular use case. For some use cases, you might be better off with TLS or IPsec: greenfield implementations of DNP3 that use only IP connections and don’t care about application-to-application authentication or centrally managed RBAC and security policies are well-served with transport layer-only security. The extra complexity needed to implement the security deeper down (or further up) the stack may not be worth it if the risk of compromise doesn’t warrant it.

DNP3-SL addresses application-to-application security that can be authorised by a central authority and authenticated using either self-signed or PKI-based certificates. It works over serial and IP connections and could be carried over TLS or IPsec. The difference between those protocols and DNP3-SL, however, lies in the design decisions we’ve made throughout the process. For example, the cryptographic primitives used target hardware acceleration available in many embedded processors, but are used sparingly to accommodate brownfield implementations. Similarly, Associations are long-lived, reducing the need to use asymmetric cryptography.

Many SCADA networks still use serial communications somewhere in the network. That cannot be changed with a simple firmware update. The version of DNP3 used by a device, and whether it’s configured to use DNP3-SAv5 or DNP3-SL, can.

It is true that there are no current commercial implementations of DNP3-SL. It may also be true that some DNP3 implementations will never support it. It is, however, very likely to be implemented by at least some popular DNP3 stack vendors and by forward-looking members of the industry.

DNP3-SL is the product of more than a decade of careful, open, and collaborative work by a group of professionals who care deeply about the security of critical infrastructure. It is not a perfect protocol, and it may not be the right choice for every deployment. But for many of the brownfield SCADA networks that make up the backbone of the world’s critical infrastructure, it will be the most carefully considered option available. We invite you to review the standard, participate in the CSTF, and help us make it better.

The short version: asymmetric cryptography is in trouble

Symmetric cryptography is mostly safe. Asymmetric cryptography is not and the reason comes down to trapdoor functions.

A trapdoor function is one that’s easy to compute in one direction but hard to reverse. The classic human analogy is multiplication versus division: multiplying a number by ten is trivial (just append a zero), while dividing is slightly more work for your brain. Scale that up to enormous numbers and throw in operations like factorization, and you have a very solid trapdoor for classical computers.

For quantum computers, that trapdoor is no obstacle at all. Algorithms like Shor’s can factor large numbers exponentially faster than any classical machine, which means the mathematical foundations of RSA, ECDSA, and ECDH are fundamentally vulnerable.

We don’t have quantum computers big enough — yet

To be clear, no quantum computer today is large enough to break these algorithms in practice. But we expect that to change around 2040 or so. That might sound far away, but the timeline for replacing cryptographic infrastructure is long, which is exactly why we need to start now.

There’s also the “harvest now, decrypt later” threat: an adversary can record encrypted traffic today and decrypt it once a sufficiently powerful quantum computer exists. For long-lived secrets, that’s already a serious concern.

The NIST competition

The National Institute of Standards and Technology (NIST) ran a years-long competition inviting mathematicians and cryptographers to propose new algorithms that are hard for both quantum and classical computers to break. The winners were CRYSTALS-Dilithium (for digital signatures) and CRYSTALS-Kyber (for key encapsulation), both based on lattice mathematics.

Lattice-based cryptography relies on problems in high-dimensional geometric structures that quickly become intractable. The details are complex, but the key point is that these problems appear to be hard for quantum computers just as they are for classical ones.

What’s new, and what’s missing

The post-quantum toolkit gives us two main building blocks:

• Digital signature algorithms — analogous to ECDSA and RSA signatures.
• Key encapsulation mechanisms (KEMs) — analogous to RSA-based key exchange. In a KEM, you don’t choose the shared secret yourself; the encapsulation function generates both the key and an encapsulated version of it, which the other party decapsulates using their private key.

What we don’t have is a post-quantum equivalent of Diffie-Hellman. There was a candidate — SIDH — but it was broken by a classical computer, so it’s off the table. This means some protocols will need to be restructured to use KEM-based approaches instead of traditional key exchange.

The timeline

The pressure to move is real and near-term:

• By 2027, certificates based on classical algorithms will begin expiring on a 47-day cycle (roughly every six weeks) rather than annually. That demands significant automation investment.
• By 2030, all new critical infrastructure projects should be using post-quantum cryptography.

The case for hybrid approaches

Because post-quantum algorithms are new, they haven’t yet accumulated the decades of cryptanalysis that classical algorithms have. Some vulnerabilities may still be undiscovered. For that reason, many practitioners, myself included, advocate for hybrid key schemes that combine a classical algorithm and a post-quantum algorithm requiring both to agree before a secret is established or a signature accepted.

This approach makes sense for roughly the next five years. It means you’re not betting everything on post-quantum cryptography being completely sound while classical cryptography is still holding up.

The catch is that once a quantum computer powerful enough to break classical algorithms actually exists, hybrid schemes no longer help you — the classical component is compromised regardless. At that point, you need to be fully on post-quantum cryptography already. Which is yet another reason not to wait.

Also, if your secrets need to still be secret fifteen years from now, you should be using post-quantum crypto already.

This has been quite the few years, though! We’ve made a lot of progress with the DNP CSTF on the new DNP3 Security Layer (no longer called DNP3-SAv6) and on the Authorization Management Protocol, introducing post-quantum (or quantum-safe) crypto into the former and getting a lot closer to an Authority spec on the latter. I’ve also been involved in a number of interesting projects, most of which I can’t talk about on this public forum, and the world has mostly recovered from COVID and is now setting itself up for a whole new, but man-made, set of calamities.

None of that is what I want to talk about here – this is a Cybersecurity blog, not a lament-the-state-of-the-world blog.

Coppice

This morning, I had an idea. That, in itself, happens often enough, but this time, I thought I’d write about it and put it on one of my blogs (the others are in a similar state of abandon as this one, so don’t worry if you don’t follow those already).

A recurring theme with micro-services is the need for security and the need for availability. Both of those problems have been solved many times and in many different ways, but it feels like everyone has to re-invent the wheel every time they run into this issue. They all invariably come to one of a small set of solutions. Among those solutions are the use of service accounts and bearer tokens.

So, this morning, as I got out of the shower, I picked up my phone and recorded a voice memo.

One of the nice things with recent phones is that they automatically create a transcript of your voice memos, so I grabbed some coffee, went to my home office in the basement, and spun up a Visual Code to run this as a “vibe coding” project.

Here’s the transcript, edited for where Siri didn’t understand what I was saying. Before you read it (and feel free to skip it): it’s a bit of a ramble.

Okay, uhm. Schema: version, colon, key ID, colon, nonce, uhm, output from an HKDF. The way to calculate that is to take a secret, which stays only on the server side, assign a random nonce and, uhm, use that as the salt. Feed in the secret, the other fields, get the output, add that to the token.

So on validation, we look up the key ID, we make sure that the token isn’t expired, and we run the HKDF. Output validated against what’s in the token. If it’s the same, the token is valid.

We wouldn’t heat (sic.) a token that is already expired, but adding it to the token itself allows the client side to validate whether expiry has happened yet as well, which would be a nice feature to have.

Uhm, yeah, so, the token doesn’t contain anything secret because the actual secret is only stored server-side, but it is a secret in the sense that it allows the bearer access to whatever, uhm, the roles or privileges assigned to the master key are, so it should be kept securely, because it itself is a secret.

Okay, so, one question, maybe, is how is this better than JWT? And how is it better than a signed JWT that simply contains the claims signed on the server side.

It isn’t necessarily, it’s just different.

Use cases would be a bit different as well. So, one, for example, is. It’s more compact: it’s not a whole JWT. It’s not a JSON document. The same type of information, but it should take less space in a format like this.

The other is, uhm, you have complete centralized control over all the tokens generated by (the service), in the sense that you can assign the roles, rather than to the token itself, you can assign it to the master key. (…) When you rotate that key, all of the tokens associated with that key automatically expire. (…)

If you, uhm, if you delete the key, all of (the tokens generated with that key) would automatically go away as well, but if you assign new permissions to the key, you automatically assign new permissions to every token from that key. And that is something you can’t do with JWT.

Uhm.

You’ll still want to exchange it for a JWT though. So if you use it to log in on a service account, you should get a JWT back. That is much shorter lived because these tokens should live about a year. And it’s a shared secret so they, uhm, one year expiry seems reasonable. But the JWT that comes back should be, (…), perhaps refreshable access token that gives you the exact access that the master key would give you, but only lives for, like, (…), 10 minutes or an hour or whatever is appropriate for a, uhm, session timeout, in the system.

Right? It does allow you to (…). That scheme where you present a token, you get a new token back, a JWT back, that’s fairly standard. Uhm, but it also allows you to let that token live for, you know, a significant amount of time. And then, uhm, you know, so a year, for example, uhm, and then just hold it for whatever time it takes.

Exchanging it for a JWT token could be an action hidden from the client side. So, for example, if I send a message to my service that could (…) bearer token that is described in the entry point of that API, very well, validate that token service-side, get a JWT token back and associate it with the internal message that also gets the correlation ID. And, you know, have that go through the system.

Uhm, so, that JWT doesn’t need to be visible to the client side, but it would basically allow every, uhm, microservice in the system to validate it offline that, uhm, yes, the token has been, uhm, validated and privileges associated with that authorization token, are, you know, valid and internal to the system. So the rest of the system would only see the JWT token entry point, get the bearer token, and exchange it for JWT before passing it on into the system.

So that means one less handshake for, uhm, for the client. But also potential high availability for everything else because only the entry point would need access to the service that validates the token.

Vibe speccing

So, I fed this into Claude and asked it to write up a spec. I was honestly a bit surprised at how decent a job it did. I had used Claude before to generate code and have often compared it to an enthusiastic intern with good Internet access and a can-do attitude, who imbibes “the good stuff” when you let him but this time, with just the transcript and a prompt, it understood the token format, HKDF key derivation, the data mode, the token lifecycle, and the hidden-from-the-client JWT token exchange flow.

It did add a token ID to the spec that I hadn’t asked for, which added a requirement to have a server-side list of generated tokens that the roles were associated with. It used that token ID as the nonce, rather than an actual nonce.

So, I told it to fix that: remove the server-side database tracking the tokens, and replace the token ID with a nonce. It understood the instructions well, but the spec it produced still had a section describing the data in a server-side database that no longer existed.

While it was generating that response, though, my thoughts had drifted to auditing: if I remove the server-side database tracking tokens, I also lose the traces I have of those tokens. So I told it:

Audit requirements are missing from the spec. Let’s add that: there should be an audit trail for any action taken on these tokens, including time stamps and the unique ID of who took the action. The audit trail should survive the master key by at least a few years.

This triggered its first questions: How long should it retain the audit trail after key duration? (configurable, specify a recommended minimum only). Should the audit trail be accessible through the API? (No, SIEMS only). What actor identity should be captured? (Human or principal, whichever is available.)

It added a few new sections to the spec. I re-read its output, told it to clean up a few things (e.g. it had added “rounds” to the parameters, I told it to remove it, it asked if I wanted to move from HKDF to PBKDF2 so it could keep the parameter, I said no).

Finally, it had created the spec as a Word document. This blog doesn’t do Word well, and I wanted a Mermaid graph for a sequence diagram, so I asked it to re-generate the spec in Markdown and generate a README for a repo I would put it in. It did both, and I ran out of free tokens for the morning.

The spec

Here’s the specification it came up with. It’s actually fairly close to what I might have written…

Coppice — Technical Specification

Version 1.3 · Draft

Table of Contents

1. Overview
2. Token Format
3. Key Derivation (HKDF)
4. Server-Side Data Model
5. Token Lifecycle
6. Audit Trail
7. JWT Exchange Flow
8. API Definitions
9. Error Codes
10. Security Considerations

1. Overview

This document specifies a compact, server-authoritative authentication token scheme backed by HKDF key derivation. Tokens are opaque to clients, contain no secrets, and support centralized revocation, permission inheritance, and short-lived JWT exchange at API gateway boundaries.

Key design goals:

• Compactness — smaller on-wire footprint than a signed JWT containing equivalent claims.
• Stateless token issuance — tokens are issued without writing any per-token record to the database. The server stores only a master key record per logical credential, not one record per issued token.
• Server-authoritative permissions — roles are stored on the master key record, not embedded in the token. Any permission change propagates instantly to every token derived from that master key.
• Centralized revocation — revoking or deleting a master key immediately invalidates all tokens derived from it.
• No secret exposure — the system secret never appears in the token or in any database record.
• Client transparency — long-lived tokens are exchanged for short-lived JWTs at the API gateway, so downstream microservices never see the raw token.

2. Token Format

2.1 Structure

A token is a colon-delimited string, Base64url-encoded for safe transmission. The nonce provides per-token uniqueness, eliminating the need to store any per-token state server-side.

version:masterKeyId:nonce:expiry:hash

Where each segment is Base64url-encoded before joining.
The final token = Base64url( version + ":" + masterKeyId + ":" + nonce + ":" + expiry + ":" + hash )

2.2 Wire Format

The assembled token must be Base64url-encoded (RFC 4648 §5, no padding) before being transmitted in an Authorization header or equivalent:

Authorization: Bearer <base64url(version:masterKeyId:nonce:expiry:hash)>

3. Key Derivation (HKDF)

3.1 Algorithm

Token hashes are produced using HKDF (RFC 5869) with HMAC-SHA-256. The nonce serves as the HKDF salt, providing per-token randomness without any server-side per-token storage. The info string binds the derivation to all other token fields, ensuring that a hash cannot be transplanted to a token with different parameters.

HKDF-SHA256(
  IKM  = systemSecret,          // high-entropy server-side secret; never leaves server
  salt = nonce,                  // random bytes from token; per-token uniqueness
  info = version || masterKeyId || expiry,  // pipe-delimited ASCII
  L    = 32                      // 32-byte output
)

3.2 HKDF Inputs

4. Server-Side Data Model

4.1 Master Key Record

One master key record is created per logical credential (e.g. per API client, per service account, or per tenant integration). No per-token records are stored. All tokens derived from the same master key share its permissions and revocation status.

MasterKey {
  masterKeyId  string    // primary key; random, e.g. "mk_7f2a9b"
  version      int       // token schema version in use for this key
  tenantId     string    // scope to tenant / org
  permissions  []string  // roles / scopes granted to all tokens from this key
  revokedAt    int64?    // null = active; set to immediately revoke all tokens
  createdAt    int64
}

Because there are no per-token records, a single master key record controls arbitrarily many issued tokens. Revoking the master key invalidates all of them simultaneously. Individual token revocation is not supported by this scheme; if per-token revocation is required, issue one master key per token.

5. Token Lifecycle

5.1 Issuance

Token issuance is stateless with respect to per-token storage. The server reads the master key record but writes nothing.

1. Look up the MasterKey record for the requested masterKeyId (must exist and not be revoked).
2. Generate a cryptographically random nonce (16 bytes, from a CSPRNG).
3. Set expiry to now + TTL (default: 1 year). TTL may be overridden in the request.
4. Run HKDF with nonce as salt and version || masterKeyId || expiry as info.
5. Assemble and Base64url-encode the token: version:masterKeyId:nonce:expiry:hash.
6. Return the token to the caller. No database write occurs.

5.2 Validation

1. Decode the Base64url token; split on : to extract version, masterKeyId, nonce, expiry, hash.
2. Reject immediately if expiry < now (client-side pre-check is also possible using the expiry field).
3. Look up MasterKey by masterKeyId; return 401 if not found or revokedAt is set.
4. Verify the version field in the token matches the version on the master key record (prevent cross-version hash reuse).
5. Load the system secret; re-run HKDF using nonce as salt and masterKeyId with record parameters.
6. Compare derived hash to token hash using a constant-time comparison; return 401 on mismatch.
7. Return the permissions[] from the master key record to the caller.

5.3 Expiry

Expiry is encoded in the token itself, allowing client-side pre-validation without a server round-trip. It is also covered by the HKDF info string, so it cannot be altered without invalidating the hash. Tokens past their expiry are rejected at validation time; no cleanup is needed.

5.4 Revocation

Setting revokedAt on a MasterKey record immediately invalidates all tokens ever issued from that key, since every validation performs a live lookup of the master key record. Deleting the record has the same effect. Because no per-token records exist, individual token revocation is not possible — revocation is always at the master key level. To approximate per-token revocation, issue a dedicated master key for each token.

6. Audit Trail

Every action that creates, modifies, validates, revokes, or reads a master key — or that issues, validates, or exchanges a token — must produce an immutable audit event. The audit trail is the authoritative record of what happened, when, and who caused it. It must survive the deletion of any master key by a configurable retention period, with a recommended minimum of 7 years.

6.1 Audit Event Schema

Each audit event is a single immutable record written atomically with the action it describes. Events must never be modified or deleted within the retention window.

AuditEvent {
  eventId       string    // globally unique, e.g. UUID v4
  eventType     string    // see Section 6.2
  timestamp     int64     // Unix timestamp (millisecond precision)
  masterKeyId   string?   // the key involved; null for system-level events
  tenantId      string?   // tenant scope of the affected key
  actor {
    principalId  string   // always present: service account or API credential ID
    userId       string?  // present when a human operator is identified
    ipAddress    string?  // source IP of the request
    userAgent    string?  // client user-agent string, if available
  }
  outcome       string    // "success" | "failure"
  failureReason string?   // populated on failure; maps to error codes in Section 9
  metadata      object?   // event-type-specific fields; see Section 6.3
}

The actor block must always include at least one identity. If a request carries both a service credential and a human operator identity (e.g. via a forwarded session header), both are recorded. If only one is available, the other field is omitted — never defaulted to a placeholder.

6.2 Auditable Event Types

6.3 Event-Type Metadata

The metadata field carries structured data specific to each event type.

master_key.created / master_key.permissions_updated

metadata: {
  permissions:    string[]   // new permission set
  previousPerms:  string[]?  // prior set (permissions_updated only)
}

token.issued

metadata: {
  expiry:  int64   // token expiry as Unix timestamp
  ttl:     int64   // requested TTL in seconds
  // token string and nonce are never logged
}

token.validated / token.exchanged

metadata: {
  expiry:        int64    // expiry from the presented token
  failureReason: string?  // see error codes in Section 9
}

6.4 What Must Never Be Logged

The following values must never appear in any audit event, log line, or diagnostic output, regardless of log level or environment:

• Token strings — the Base64url-encoded credential returned to the client.
• Nonce values extracted from or used to construct tokens.
• System secret bytes or any value derived from them.
• JWT strings issued by the exchange endpoint.

Audit events record identifiers and outcomes only — never credentials or key material.

6.5 Retention Requirements

Audit events must be retained for the configured minimum period, measured from the later of: (a) the event timestamp, or (b) the revocation or deletion timestamp of the associated master key. This ensures the complete history of a credential remains available for investigation even after the credential itself is gone.

6.6 Delivery and Integration

Audit events are not exposed through the Token Service API. Consumers access them out-of-band through the deployment’s log or event infrastructure. The Token Service must support at least one of the following delivery mechanisms:

• Structured JSON log output to stdout — one event per line, suitable for ingestion by a log aggregator (e.g. Fluentd, Logstash, or a cloud-native log router).
• Direct publish to an append-only event stream (e.g. Apache Kafka, AWS Kinesis, or Azure Event Hubs) for real-time SIEM ingestion.
• Write to a dedicated append-only audit table in a database instance separate from the master key store, accessible to authorised compliance tooling only.

The Token Service must guarantee at-least-once delivery and must emit an operational alert if the audit sink becomes unreachable. Token operations must not silently succeed with audit events dropped. If the audit sink is unavailable, the service should either fail the operation or queue the event with bounded retries and alert.

6.7 Access Control on Audit Logs

Read access to the audit log is a distinct privilege, granted independently of the Token Service management credential. Write access belongs exclusively to the Token Service process. No other system or operator may append to, modify, or delete records within the retention window. Human operators granted read access must not have delete access during the retention period.

7. JWT Exchange Flow

Long-lived tokens are not forwarded into the internal microservice mesh. Instead, the API gateway validates the token and issues a short-lived JWT that downstream services can verify offline using the gateway’s public key.

7.1 Flow

See jwt-exchange-flow.mermaid for the sequence diagram. In summary:

1. Client sends a request with Authorization: Bearer <token>.
2. API Gateway calls POST /tokens/validate.
3. Token Service checks token expiry (no DB needed).
4. Token Service looks up MasterKey, checks revokedAt.
5. Token Service re-derives the HKDF hash and compares (constant-time).
6. Token Service returns { masterKeyId, tenantId, permissions[] }.
7. API Gateway mints a short-lived JWT (exp: ~1 hour, signed with gateway private key).
8. API Gateway forwards the request with the JWT to downstream services.
9. Downstream services verify the JWT signature offline — no Token Service call.

The JWT is invisible to the external client. It exists only within the service mesh for the duration of the request.

7.2 JWT Claims

{
  "sub":   "<masterKeyId>",
  "tid":   "<tenantId>",
  "scope": ["read:data", "write:data"],
  "iat":   1700000000,
  "exp":   1700003600,
  "jti":   "<random nonce>"
}

8. API Definitions

The Token Service exposes the following REST endpoints. All requests and responses use application/json. All management endpoints require a service-to-service credential (e.g. mTLS or an internal bearer token).

8.1 Endpoint Summary

8.2 POST /master-keys — Create Master Key

Creates a new master key record. A master key is the logical credential from which any number of tokens can be issued. No tokens are produced by this call.

Request

POST /master-keys
Content-Type: application/json

{
  "tenantId": "acme-corp",
  "permissions": ["read:reports", "write:data"]
}

Response 201 Created

{
  "masterKeyId": "mk_7f2a9b",
  "tenantId":    "acme-corp",
  "permissions": ["read:reports", "write:data"],
  "createdAt":   1700000000
}

8.3 POST /tokens/issue — Issue Token

Issues a new token from an existing master key. The server generates a nonce, computes the HKDF hash, and returns the encoded token. No per-token record is written to the database.

Request

POST /tokens/issue
Content-Type: application/json

{
  "masterKeyId": "mk_7f2a9b",
  "ttlSeconds":  31536000
}

Response 201 Created

{
  "token":       "djErbWtfN2YyYTliOn...",
  "masterKeyId": "mk_7f2a9b",
  "expiry":      1798761600
}

8.4 POST /tokens/validate — Validate Token

Validates a raw token and returns the associated permissions. The API gateway calls this on every inbound request before minting a JWT. Since no per-token state exists, this call performs only a master key lookup and HKDF re-derivation.

Request

POST /tokens/validate
Content-Type: application/json

{
  "token": "djErbWtfN2YyYTliOn..."
}

Response 200 OK

{
  "valid":        true,
  "masterKeyId":  "mk_7f2a9b",
  "tenantId":     "acme-corp",
  "permissions":  ["read:reports", "write:data"],
  "expiry":       1798761600
}

Response 401 Unauthorized

{
  "valid":  false,
  "reason": "expired" | "revoked" | "not_found" | "hash_mismatch" | "version_mismatch"
}

8.5 POST /tokens/exchange — Exchange for JWT

Accepts a long-lived token in the Authorization header and returns a short-lived, signed JWT for use within the service mesh. This is the primary entry-point for all authenticated API calls.

Request Headers

POST /tokens/exchange
Authorization: Bearer djErbWtfN2YyYTliOn...

Response 200 OK

{
  "jwt":       "eyJhbGciOiJSUzI1NiJ9...",
  "expiresIn": 3600
}

8.6 GET /master-keys/{masterKeyId} — Get Master Key

Returns metadata for a master key record. Never returns secret material.

Response 200 OK

{
  "masterKeyId": "mk_7f2a9b",
  "tenantId":    "acme-corp",
  "version":     1,
  "permissions": ["read:reports", "write:data"],
  "revokedAt":   null,
  "createdAt":   1700000000
}

8.7 PUT /master-keys/{masterKeyId}/permissions — Update Permissions

Replaces the permission set on a master key record. The change takes effect on the next validation for any token derived from this key — no token re-issuance required. For security, automated flows should only permit permission reduction; permission elevation should require explicit operator approval.

Request

PUT /master-keys/mk_7f2a9b/permissions
Content-Type: application/json

{
  "permissions": ["read:reports"]
}

Response 200 OK

{
  "masterKeyId": "mk_7f2a9b",
  "permissions": ["read:reports"],
  "updatedAt":   1700005000
}

8.8 DELETE /master-keys/{masterKeyId} — Revoke Master Key

Immediately revokes the master key by setting revokedAt. All subsequent validation calls for any token derived from this masterKeyId will return 401. This is the only revocation granularity available; there is no per-token revocation.

Response 204 No Content

Empty body on success.

Response 404 Not Found

{
  "error": "master_key_not_found"
}

9. Error Codes

10. Security Considerations

10.1 Token Secrecy

Although tokens contain no embedded secrets, possession of a token grants access to all resources the associated master key permits. Tokens must be treated as secrets by clients: stored securely (e.g. OS keychain or encrypted storage), never logged, and transmitted only over TLS.

10.2 Nonce Requirements

The nonce is the sole source of per-token uniqueness. It must be generated using a cryptographically secure pseudorandom number generator (CSPRNG) and must never be reused across tokens issued from the same master key. A minimum of 16 bytes (128 bits) is required. Because the nonce is included in the token and used as the HKDF salt, it need not be stored server-side, but it must be resistant to prediction.

10.3 Revocation Granularity

Revocation operates at the master key level. Revoking a master key invalidates all tokens ever issued from it. Individual token revocation is not possible without maintaining per-token state, which this scheme deliberately avoids. Systems requiring per-token revocation should issue one master key per token. This is a deliberate trade-off: stateless issuance at the cost of granular revocation.

10.4 vs. Signed JWT

This scheme differs from signed JWTs in several important ways. JWTs embed claims and permissions in the token itself, meaning permissions cannot be changed without re-issuing the token. This scheme stores permissions server-side on the master key record, enabling instant permission changes across all derived tokens. The JWT-exchange flow (Section 7) preserves offline validation for internal services: only the API gateway validates the long-lived token; downstream services validate the short-lived JWT offline without a database call.

10.5 Minimum Service Access

Only the token validation microservice requires access to the system secret store and the master key database. All other internal services interact exclusively with the short-lived JWT. This minimises the blast radius of a compromise.

10.6 Secret Rotation — Known Constraint

Because the master key record does not store a reference to the secret used at issuance, rotating the system secret is a breaking operation: all existing tokens will fail HKDF re-derivation and return hash_mismatch on their next validation. Secret rotation therefore requires coordinated re-issuance of all active tokens before the old secret is retired. Operators should treat a secret rotation event as equivalent to revoking all master keys and re-provisioning all clients. If non-breaking rotation is required, a secretId reference should be reintroduced to the MasterKey record.

10.7 Constant-Time Comparison

Hash comparison during validation must use a constant-time byte comparison function to prevent timing-based oracle attacks.

10.8 Tenant Scoping

All master key records are scoped to a tenantId. Validation endpoints must enforce tenant isolation: a token issued for tenant A must never be accepted for tenant B’s resources.

Coppice — HKDF Token Scheme · v1.3 Draft

more…

One of the most famous encryption mechanisms in history is the German Enygma machine, used by the Nazis during the Second World War to encrypt messages between different parts of the German army, and eventually cracked by the British under the leadership and guidance of Alan Turing, one of the fathers of modern computing. Since those days which, while less than a century ago and still part of living memory today, are ancient history for the annals of computing, encryption has made great strides with the development of various “block” and “stream” ciphers (i.e. encryption algorithms), and various “modes” which each have their advantages and drawbacks. All of these algorithms essentially do the same thing, though: using some mathematical trickery, they obfuscate the true meaning, and sometimes the true size, of a message or some data at rest, providing confidentiality to those who have the key to unlock that true meaning, against those who don’t.

But when is it worth doing that?

In information security (a.k.a. InfoSec), we learned a long time ago (again, computer years – it may only jhave been a decade or so) that the mantra “I have nothing to hide” is false on its face: you identify yourself to your bank with a four or five-digit PIN and a physical card, and expect your bank to believe you when you contest a fraudulent charge made with that same card and PIN. Thanks to Artificial Intelligence and Machine Learning, they probably will, but keeping your PIN secret is also part of that equation. Similarly, your IT department will tell you to change your password often, and keep it a secret. Passwords date back to the 1960s and are due for an overhaul, but they’re still secrets you need to keep. So, it’s probably fair to say everyone has secrets to keep. The answer to keeping secrets, or maintaining confidentiality of those secrets, when stored or when in transit (e.g. while being shared between you and the bank), has been the same since before computer science was a thing: encryption.

The IT, InfoSec, and more generally cybersecurity folks might just go ahead and encrypt everything, then. And why not? Today’s iPhone and Android smart phones have more computing power than a supercomputer from the 1970s by a factor of more than a thousand. The computational overhead of encrypting and decrypting everything is negligeable. But that does mean you should just encrypt everything?

Let’s take a look at the encryption you’re using right now: the Applied Paranoia blog is public. There are no secrets on this website: any secrets that are part of the CI/CD pipeline are not on the site itself – I’m paranoid enough to have checked. Yet, the website uses TLS for the connection between the Amazon-hosted server and your browser. TLS encrypts everything, but that’s not its purpose in this case: it’s there to make it clear to your browser that what it’s showing you really came from this blog, and not from some “man in the middle”. TLS isn’t there to encrypt, it’s there to authenticate. The encryption is there just because it’s part of the default settings.

That is likely true for most encryption today: the data on my laptop, my phone, my tablets, my S3 buckets, etc. is all encrypted. That means that to access any of it, I need the key – or at least I need access to the key. Without that, I don’t have access to any of it, and neither will you.

This fact is what the “bad guys” use when they attack you with ransomware: they encrypt your data, on your computer, so it’s no longer available to you until you pay the ransom. The fact that your laptop out-performs a vintage supercomputer helps them more than it helps you in this case. But the real pain comes from you no longer having the key to your data.

And that’s the drawback: sometimes you need to be able to access data – you need the data to be available to you, even if you don’t have the keys. You need to be able to read it, but not change it. You don’t need the confidentiality that comes from encryption, even if you do need the integrity you can validate with authentication. Sometimes, availability trumps confidentiality.

In Operational Technology, where physics meets software and cyber security takes on a meaning that is very different from Information Security, that “sometimes” becomes “almost always”. That is why DNP3 Secure Authentication version 6 encryption is still optional¹. OT aside, the non-availability argument applies to anything that is encrypted: encryption comes at the cost of not just some compute cycles, but of availability as well. Sometimes, we ought to ask ourselves if it’s really worth it.

Sometimes, it isn’t.

So, how does one go about choosing what to encrypt and what to leave in the clear? The answer to that question is a technique called “Design Failure Mode and Effect Analysis” or DFMEA. A DFMEA allows you to look at your architecture using the perspective of how it might fail, and to plan for such failures. You can do this before you’ve implemented anything, or at any time after that, and you should probably do this on a regular basis. It’s really not that complicated provided you know the architecture you’re working with.

Let’s look at an example web applicaiton and analyse it. The application, which we’ll call “Crassula”, implements a web store for a florist. It uses a static website generated with Jekyll, and an Angular app embedded in that website for the purchasing workflow. The static front-end and angular app are served out of an S3 bucket behind a CloudFront proxy, the back-end for the store uses some serverless functions (lambdas), and S3 bucket to download invoices from, and a NoSQL database. The whole thing is tied together using AWS’ Simple Queue Service and deployed using CloudFormation.

The S3 bucket that contains the front-end (static site and Angular app) is the first point of entry for any customer. The Angular app calls out to the API and implements the workflows, including downloading invoices from the second S3 bucket using SAS URIs, but unlike the static app it can handle temporary failures of the API or the S3 bucket with a modicum of grace. That means that, for the front-end to work properly, the S3 bucket containing it needs high availability, but we don’t care as much about its confidentiality: we don’t need its contents to be encrypted and we can leverage a content delivery network like CloudFront to ensure availability.

On the other hand, access to invoices is a different matter: invoices contain confidential, personally-identifiable information that should not be available to anyone except the customer themselves and the store’s personnel. That means our DocumentDB should almost certainly be encrypted, as should the S3 bucket containing the invoice PDFs for download. Failure to access either of these two resources would have one of two possible effects in this application: failure to access the S3 bucket would thwart the delivery of an invoice to a customer, possibly delaying payment. In such cases, an alert to the shop’s staff would be warranted to mitigate the issue. Failure to access the database could either result in a transaction not completing, or in an invoice not being generated. In both cases, staff should be alerted but in the former case, any charge should also not be authorized. In any case, proper controls for failures of availability can be implemented here whereas for an inability to download the front-end application, detection of such a failure as well as its mitigation may require additional infrastructure and resources that would arguably be needed anyway, but not due to the confidentiality of the deployed front-end.

This is the essential trade-off: failure of an encryption mechanism or loss of the key leads to data not being available. In our example, the front-end is essentially guaranteed high availability through the content delivery network which cares much less about confidentiality than it does about availability. The back-end, on the other hand, makes the trade the other way around, though mitigations for failure (of availability) can be identified and implemented.

The way to identify those failures, and the way to identify how to mitigate and (as needed) resolve those failures, are are a different topic: design failure more and effect analysis or DFMEA. You can read about that on my other blog.

Responsible disclosure doesn’t take the surprise away completely, but softens the blow a bit. It basically means that when you find a vulnerability, especially if you find it in a popular piece of software, you don’t immediately post it to your blog or social media: you use the “proper channels” to tell whoever maintains the software about it, so they can fix it, communicate with their stakeholders, and make sure that when the vulnerability does become public knowledge, it’s already been fixed. Responsible disclosure is not easy: it puts a burden on whoever found the vulnerability (who ends up being called a “security researcher” no matter what they do in their day-to-day life) to find the “proper channel”, but it also puts a burden on whoever wrote the software in the first place to create that channel. Especially for small vendors, small teams, or open source software volunteers, that can be quite a burden.

The sorry state of responsible disclosure in Canada

Some countries, like the US, have government agencies to take on some of the burden of the security researchers. Canada has no such government organization, as shown by the Cybersecure Policy Exchange, a cybersecurity lobbying organization funded by RBC and run from Ryerson University in Toronto. As they point out:

• Canada does not have a “distinct and clear disclosure process for vulnerabilities involving government systems”
• Canada does not “[describe] the vulnerability submission and verification process”
• Canada does not “[provide] terms and rules for disclosures”
• Canada does not “publicly dessimate information about vulnerabilities disclosed through [a] coordinated process”
• Canada does not “publicly give acknowledgement or credit after disclosure”

Our neighbour to the South, as well as the European Union, Japan, Russia, and Great Britain do all of these, and China does all of them except for public acknowledgement, according to the Cybersecure Policy Exchange.

It’s rather saddening to see that my adopted country is lagging behind, but this post is not about Canada.

A “distinct and clear disclosure process” would lighten the load on security researchers a bit: it allows them to contact a single (government) agency that can then coordinate with affected vendors, businesses and government services to ensure resolution of the issue and careful dissemination of relevant information. For example, the security researcher, or the affected vendor or open source project, does not have to try to find out who is affected by the vulnerability if there’s a government agency that is dedicated to doing that. Pro-actively reaching out to critical infrastructure providers (i.e. privately owned utilities for the power network, water and waste water, telephony, etc.) to get an inventory of potentially-vulnerable (that is: all) software used in critical systems or have them maintain such an inventory in case there’s a vulnerability to be dealt with is a job better suited for a government agency than for dozens of small teams.

So, absent government support, who does a security researcher contact?

Who to contact for responsible disclosure

Who to contact in businesses, small, medium, and large

Usually, the only point of contact an “outsider” has with a company is either through sales or support. Typically, neither sales nor support are trained to deal with cybersecurity issues, but they should at least know who to route the issue to. This is where things start getting complicated, though.

While large businesses may (and really should) have a dedicated cybersecurity team to help product teams coordinate their responses to vulnerabilities and incidents, those teams should provide guidance and support but will typically not know the ins and outs of the market a particular product team operates in: that’s Marketing’s job. That means that such dedicated teams will not necessarily know who uses the product and how it is used, and will not be able to analyse the issue to ascertain whether it is actually a cybersecurity issue in the product. It is up to the product teams to develop threat models, deployment models, and vulnerability management plans, provide their customers with guidelines on how to deploy the product, and show conformance to requirements assuming that those guidelines are followed, and ultimately to respond to any issues that may occur. I.e. dedicated cybersecurity teams need to adopt a “you can do it, we can help” attitude.

Smaller business don’t have dedicated teams for cybersecurity: such teams are expensive and, while they are worth the investment for larger businesses after some time (i.e. as cybersecurity incidents which would negatively affect the company’s reputation are avoided), smaller businesses simply don’t have the resources to make that investment. They may, from time to time, be able to engage with a consultant to get training on how to develop threat models and deployment models, vulnerability management plans, what zero-trust architectures are, and what some relevant industry standards are to conform to, but such engagements do not change where the responsibility for the product and its security ultimately lies: with the product team. Such engagements also do not typically include rapid response to cybersecurity incidents, even when such services are available.

That means that responsible disclosure should go to the product team first, at which point a response team made up of marketing, R&D, project management, and (internal if available, external otherwise) cybersecurity experts needs to come together and assess the issue, under the responsibility of the product team.

Customer support teams are often not trained for cybersecurity issues. In light of this, they need to be kept in the loop and, while they often won’t be included in the response team directly, they should at least be listened to and be allowed to voice their concerns and those of the customers they’re in contact with, and be informed on the progress of the analysis and the fix.

The support team will often be the best informed within the company as to how customers use the product, and may therefore be better informed than the development team to know how vulnerable systems really are. This can be an important input into the analysis, as the vulnerability may be one of deployment rather than the product itself.

Who to contact in open source projects

We’ll leave businesses aside for a moment and turn our attention to unfunded open source projects. There are thousands of these, maintained by unpaid volunteers who do this important work for the love of the challenge, the community, or whatever else drives them. I’ve contributed to a few open source projects myself: it’s fun and sometimes quite engaging, but I wouldn’t be able to express what drove me do to it.

From a cybersecurity perspective, though, it is hard to implement good security practices if the only resource you have is your own free time. GitHub and similar platforms provide some resources, such as automatic alerts to let you know you depend on something vulnerable, but it doesn’t “automagically” provide you with necessary project infrastructure and guidelines to build vulnerability management plans, threat models, etc. or to respond to emergent threats.

Most open source projects do not have documentation to tell you how to safely deploy them, do not register their users so they can be updated in case there’s a vulnerability, don’t distribute security bulletins, etc. The Apache Foundation provides some of that infrastructure and provides a framework to work in, but many other organizations do not, and most open source project are not part of any organization to start with.

So, for a run-of-the-mill open source project with no funding, no corporate sponsor, and no foundation to support it, who you gonna call? Usually, the developer.

A few prerequisites for responsible disclosure

Regardless of who you end up contacting, here’s a few things that need to be true for any responsible disclosure:

1. Disclosure needs to not be public, or at least not right away: the product team (or lonesome developer) needs to have time to prepare a response, figure out whether the vulnerability is legitimate, fix it if it is, update documentation, prepare a security bulletin, etc. A lot of work goes into responding to a vulnerability, so disclosing it too quickly will do more harm than good.
2. The product team (or lonesome developer) needs to be responsive: they need to quickly set up their response team, which needs to include someone who is in charge of coordinating the response, someone who can analyse and fix the underlying issue, and someone who knows who the stakeholders are and can coordinate with them. For small businesses and open source projects, this may well all be the same person (who is in for a pretty stressful few days/weeks in that case), but for larger businesses it really shouldn’t be.
3. Do not acknowledge a security issue without input from the development team: they need to analyse the issue first, and confirm that it’s a cybersecurity issue (and not just a run-of-the-mill bug). This is especially important for support teams: if someone calls support about an issue, acknowledging it as a cybersecurity issue may set of vulnerability management plans at the customer which may not have an “exit clause” in case the issue was not a vulnerability to begin with.

Difficulties with responsible disclosure in open source projects

Aside from the difficulties already mentioned above, open source projects have some extra difficulties that commercial software does not when it comes to responsible disclosure. These may be summarized as “everything is out there”.

If there is any kind of team or community to speak of for the project, any issues or changes to the software will generally be discussed on a mailing list that will often be public. Such mailing lists may be monitored by so-called “black hat hackers”, especially if the software is popular. Keeping vulnerabilities off that public forum is often more difficult than it may seem, as that requires the developers of the project to have some other, non-public means of communication.

Even if that is the case, a fix will eventually be published in source code, by being pushed to a git repo or even just published as a tarball. Most open source projects publish source code, and may or may not publish binaries. Again, it is easy to monitor such releases and black hat hackers generally will do just that. While the commit message may not refer to a CVE number directly, it will still generally describe something like “fix a race condition that could occur if …”, which may be enough to understand that a vulnerability is being addressed. Such changes may fix the issue, but will also expose it to knowledgeable onlookers, who may then exploit deployments that haven’t been updated yet.

Once the fix has been published, open source projects don’t necessarily know who uses their software, so it becomes difficult to discreetly contact stakeholders to have them update their software without making the vulnerability public. That means that stakeholders have no choice but to learn about the vulnerability at the same time (or after) the “bad guys” do as well.

Let’s start with some low-hanging fruit: don’t start any new Java projects. Java developers can be converted to other, similar languages such as C# with fairly minimal effort and contrary to common belief, C# is just as portable as Java for the vast majority of use-cases. C# also doesn’t have some of the cybersecurity issues that Java has been plagued with over the last few decades, in large part because it has had the opportunity to learn from Java, and from Windows, in that respect. If there is a choice for programming languages, Java should only be used if no other options are available for new projects.

But let’s face it: how often do you really start something new? In the vast majority of cases, the “something new” builds on code you already have, in a code base that may be years or decades old. If you have Java-based IP, you will be writing Java code for a very long time to come.

So, how do you go about securing those services and applications, despite them being written in Java?

The issues

There are two major issues to be concerned with:

1. The JVM may “reach out” and connect to third-party servers
2. The JVM may make changes to your filesystem

There are several other issues that you may be concerned about, but that are not specific to the JVM:

The JVM reaching out to third party servers

The main issue with log4j is that it could be coerced into using JDNI to download and execute code from a third-party server. Even if the code wasn’t executed, because the (unauthenticated) user had full control over the URI used to fetch the code and could use environment variables from the server as well, data from the environment could be leaked by the same occasion even if foreign code execution was turned off.

TLS is usually the go-to technology to secure connections between servers, and it does authenticate the server you’re connecting to, but all that does in this case is tell you that the server at very.evil.address really is at very.evil.address. It provides authentication and encryption, but it doesn’t tell you whether you can trust the resources you’re downloading, especially if they are code.

On the other hand, shutting off execution of downloadable code may break your application if it had a legitimate use for such a pattern. Remember Java is a pre-REST technology and many applications from the late 1990s are still used today. There’s a reason why similar technologies are still in every-day use. Should they be put behind a RESTful interface in stead? Probably, yes. But that can be a lot of work.

So to resolve the issue of the JVM reaching out to third-party servers, without breaking the the applications we’re trying to secure, can’t just be a matter of “no longer doing that”.

The JVM making changes to the filesystem

Once someone has been able to exploit a vulnerability, they need to “make it stick” if they want to be able to take control of the server (e.g. to enrol it in a botnet) later. Direct ways of doing that would include vulnerabilities that can upload arbitrary files and put them in a pre-set location (such as CVE-2016-3088), but if you can get the JVM to execute arbitrary code (which most of the recent vulnerabilities appear to be about), you can certainly have that arbitrary code install malware as well.

Changing things in the filesystem is not a bad thing per se: there are many legitimate reasons to do that that have no bearing whatsoever on cybersecurity. It is not something you want to disallow outright, nor something you usually can disallow outright without breaking applications. Of course, you could use databases (SQL or NoSQL alike) to manage your non-volatile state in stead, and in most cases, you should.

Approaches to a solution

The issues above can be addressed with a number of common tools. These tools may require a bit of extra infrastucture to set up and manage, but most of that can be done using IAAS services which can, in turn, be deployed using an Infrastructure-As-Code approach. I won’t go into to the details of setting up and deploying the required infrastructure below, but I will provide a bit of an overview of each of the tools I propose to use as countermeasures to Java’s two main issues, and how they help overcome them.

Sandboxing

One advantage of using a virtual machine is that it can be used to “sandbox” an application: the resources the application gets access to are limited to the ones available within the sandbox. Communications with a sandboxed application is always an issue, though.

The JVM does this as well: you can limit the amount of memory, CPUs, etc. the JVM can use for any particular application, and you can tell it where to find the classes it needs. This actually used to be an argument for using Java. The whole issue we’re discussing here, though, is that the JVM itself cannot be trusted. The first thing to do, therefore, is to sandbox the JVM.

In general, the easiest way to do this for a monolithic application (like most decades-old Java applications will tend to be), would be to wrap the whole thing in a VM of which the entire contents is controlled. This approach also works with more modern Java applications, which may be less monolithic in nature.

The question then becomes how to control the entire environment within a VM. The answer to that one is actually fairly simple: using an approach that allows you to write, as a single file or a small set of files, the procedure to set up your VM. These files should not end up in the VM itself, or should at least be deleted before the VM is deployed, but they can be versioned and tested.

Now, forget for a moment that I said “VM” in the last two paragraphs.

The “procedure to set up your VM” should of course be a Dockerfile. The VM itself could be either a pod in a Kubernetes cluster or just a Docker container running by itself. It’s not all that important which one it is, as long as you have a Docker container that contains only what your application needs to run.

Once you have that, you can (and should) set up a mechanism to frequently update your image (by re-running Docker and making sure your Dockerfile updates the image when run) so at least the underlying “VM” stays up to date. You could make this part of your CI/CD pipeline.

Limiting filesystem access

Even if you use either disposable VMs or Docker containers to host your Java applications in, you should still limit the JVM’s access to the filesystem as much as possible: run the JVM with the least privileges required (even inside the Docker container) and if you mount volumes into the Docker container or disposable VM, make sure you restrict the mount options as much as possible.

For example: most host systems will allow you to strip the “executable” bit from files written to the filesystem and will allow you to filter it out when read. This is a security measure that has been available for decades now, and is still useful today.

If you have applications that communicate with each other through shared files, make sure that once the file is written and published to the other application (or step in the pipeline), it can no longer be altered. It should effectively disappear from view for the publisher before the subscriber can access it. If you don’t, you’ll have a race condition that may become exploitable.

Configuration and code files should generally be read-only to the application, so make it so: if you have the code of the application, make sure it doesn’t need anything other than read-only access to those files.

The less your (untrusted) application can do, the less likely its capabilities will be exploited.

Code signing

Java has implemented optional code signing and signature verification for a while now. It requires the application to be distributed in a JAR file (which is very common) and requires that JAR file to be signed using (the private key corresponding to) a trusted certificate. Once that has been put in place, attempts to execute classes outside of appropriatey signed JARs will fail with an access error.

This is a simple, though somewhat arduous, way to make sure only code that you actually trust is executed: you need to make sure that all the JARs that contain such code are signed, that all the code you need is in those JARs, and that you require signature verification.

For this to actually work, though, you need a bit more than just signing all your JARs (which you can do as part of your CI/CD pipeline, provided you run that on a trusted infrastructure):

1. you need to set up a PKI, including expiry policies and the like
2. you need to audit all of your dependencies for trust
3. you need to re-package your application, using only those trusted dependencies, into signed JARs
4. you need to reconfigure your deployment to only run signed JARs
5. you need to version-control that deployment configuration
6. you need to test the deployment configuration (preferably regularly) for its inability to run badly signed, or unsigned, JARs
7. you need to make sure you re-deploy your entire application before the certificates you use to sign them expire (so likely at least once a year)

Some of these, like setting up your PKI, would be worth their own post, but I’ll just go through each in a single paragraph to explain what I mean and why it’s important.

Setting up a PKI is the first step: you need to be sure that what you trust is indicated as such, and to do that you need to authenticate it. Setting up a PKI does not have to be expensive, but you do need to know what you’re doing and you need to keep your root CA certificate safe. In this case, you should not use a commercial CA unless you root your trust in an intermediate CA certificate that you create yourself: using a commercial CA means that anyone who can pay the fee can get a trusted certificate – not what you want. There are quite a few caveats to consider here, but without a PKI you can’t implement code signing effectively. It’s just that simple.

Once you have a PKI in place you need to be sure about what you’re signing, which means auditing the code. Some of that can be done automatically: you can, for example, use a static analyzer to analyze all the third-party open source code. If you fix the issues you find (which you generally should) you should contribute the fixes back upstream: it’s the right thing to do as a netizen, and it means you don’t have to maintain the patches afterwards. Remember to use responsible disclosure if you find security issues in third party code.

Once you’ve audited your application, you should re-package it into signed JAR files, using only the trusted dependencies that you have now audited. That’s just how code signing works in Java.

Once you’ve done that, you need to configure your deployment to run only signed JAR files. There’s plenty of documentation on how to do this, so it shouldn’t be too hard, dependending on how your deployment works. This should be part of your CI/CD as well.

Make sure the deployment configuration is version controlled: you want to make sure you can audit any changes that were made to the configuration, and you want to make sure that, just like your code should have, you have a review and approval process in place that explicitly looks for cybersecurity issues. For any change made to the configuration, you need to be able to show why it was done, when it was done, by whom it was done, and what the change was.

Also make sure you (periodically) test that you really can’t run unsigned JARs or JARs signed using some other CA: a large part of your security depends on this, so the risk associated with failure is high (the likelihood of things going wrong is high, and so is the impact).

Finally, depending on your PKI expiry policies, you will need to re-deploy the entire application on a regular basis to refresh the certificates being used. This generally means that if you redeploy your application anyway, say on a monthly basis, you don’t have anything special to do other than monitor the age of your oldest deployment. Make sure this works, because your application will stop working if you forget!

Proxy

To prevent the JVM from unexpectedly reaching out to the internet, the application should be behind a proxy.

Proxies are what IT in big companies use to eavesdrop on their staff’s use of the Internet. Like you shouldn’t trust the JVM, IT doesn’t trust non-IT staff, so internet usage is monitored for anything illegal, inappropriate, or unsafe for company IP and infrastructure.

While we don’t expect the JVM to download videos of cute cats or to spontaneously buy illegal firearms, we do need to make sure that it doesn’t connect to any servers on the Internet that aren’t on a “white list”, and you’ll want to scan the contents of the messages being exchanged.

Being able to scan the messages being exchanged usually means the proxy will “spoof” the external site’s certificate with its own, which, again, means you need a PKI. Note that this implies that the proxy implements its own CA. It also means its CA certificates should not be signed by a commercial trusted root (because doing that would be very dangerous) and should not be trusted anywhere outside of your organization.

The proxy should basically, at the very least, check that the server being reached out to is on the white list, create a certificate to spoof the target, reach out to the server itself, validate then relay the request, validate then relay the response, and keep validating and relaying requests and responses as long as it passes muster.

Obviously, if your application has no legitimate reason to reach out to the Internet, you don’t need a proxy: you just need a firewall (which you need anyway).

API Gateways and deep firewalls

Outgoing traffic is not the only problem, though: ultimately, the issue is that someone from the outside is trying to coerce your application into doing something you don’t want it to do. It would be a pretty odd approach to lock your application up in a sandbox without even looking at what’s going into the application.

An API Gateway does just that. It’s usually used for three things: load balancing, HTTPS-to-HTTP forwarding (so your application code doesn’t have to deal with HTTPS), and deep inspection. We’re interested in the deep inspection in this case.

OWASP, the Open Web Application Security Project, has a set of “core rules” that you can find at coreruleset.org. Implement them. Use ModSecurity if you must (but not it will no longer have commercial support in 2024).

This is probably the hardest part (this and the proxy, really): it will cost the most to implement and maintain, but it is also your first line of defense.