Patch eMudhra, Then Inventory Your Trust Stores While You’re There
When eMudhra’s keys broke, the question was simple: which of our clients trust this root? Is your organization able to answer it? A second question is harder. For the third-party systems you connect to, if someone forged one of their certificates, would you know?
For most backend clients, the answer is no. A script error in eMudhra’s April 2025 key ceremony produced eight intermediate CA private keys small enough to fit in one byte. One of them was the number 4. Anyone can recover them from the public certificates and sign a certificate for any domain, including a partner’s. Nobody stole these keys. They were born broken.
Browsers were covered within days, Chrome within hours, and no attack has been reported. But the keys stay public, so a forgery can arrive at any time, and a backend client whose trust store was never fixed will still accept it.
Patching is only the first step. While you patch, inventory where each client’s trust lives, then standardize it.
Where trust lives outside the browser
A browser is one TLS client. Every other TLS connection your organization makes happens outside it: a Java service calling a payment API, a CI runner pulling packages, a Python job posting to a webhook, a monitoring agent reporting home. Each one gets its trust from somewhere, and most stacks touch several of these places:
One key, two chains
Each of the eight weak keys had two certificates. eMudhra issued one under its own emSign ECC Root CA — G3. Sectigo issued the other, a cross-signature from its AAA Certificate Services root (listed as “Comodo AAA Services root” in some stores). Eight keys, two issuers each: 16 certificates.
A client checking a forged certificate does not care which chain it finds. It builds whichever path ends at a root it trusts. Distrusting emSign closes the left path. The right one stays open for any client that still trusts AAA Certificate Services, and an unpatched JDK’s cacerts still does.
Sectigo revoked its cross-signed copies on October 19. That helps only clients that check revocation, and most clients outside the browser do not. For them the exposure has no end date of its own. It lasts until someone fixes the trust store, or until the certificates expire.
An inventory that stores trust as a list of root names can answer “is emSign in our stores?” and still miss this.
Three lists, not one
We have an inventory of certificates issued by our trusted CAs. That answers one part of the eMudhra question. Google and AWS set replacement deadlines of October 27 and 28. A team that knew which of its certificates came from eMudhra, and already had a second CA, could reissue before those dates. A team that did not was searching against a deadline.
Then come the questions that inventory cannot answer. Do we know who the TLS clients in our ecosystem are? Which CAs are in their app trust bundle? Where does the trust bundle that each app loads actually live? That is the second list: what we trust. It is spread across the OS store, cacerts, certifi, the roots built into Node.js and a Cisco appliance's root bundle, and it needs an inventory of its own, down to the roots and cross-signs each one holds.
And the last question: would we know if one of those TLS clients connected to a site whose certificate was forged? Answering it starts with a third list: the external services our applications call, and the CA behind each one. When a CA is distrusted, a partner’s certificate can stop validating in our clients as soon as our trust stores update. AWS’s advisory names two: identity providers behind a load balancer, and origins behind a CDN.
Usually we would not know. The forgery chains to a root the client trusts, so the handshake succeeds and nothing logs an error. Browsers would catch it through CT. Backend clients do not check CT, and an attacker aiming at them has no reason to log the certificate, so the partner’s own CT monitoring sees nothing either.
The defenses that would catch it have to be chosen in advance. The strongest is giving that connection a trust store that holds only the CA the partner actually uses, or a private CA you both agree on. Pin only where you control both ends. A pin breaks the moment the partner changes CA, which is exactly what happens in an incident like this one. Revocation helps only if the client checks it from cached revocation lists. Live checks either fail open or cause outages when the responder is down. The narrowed trust store is only possible if the third list exists.
Planning to switch CAs quickly means preparing all three before the next incident:
Inventorying is costly, and it may still not be complete. A new base image, a vendor update or a pinned package adds a trust store nobody listed. Inventory alone is the weaker recommendation. Pair it with standardizing: point applications at a small set of approved trust stores, and use the inventory to find drift from that standard. The inventory then does not need to be perfect. It needs to catch the clients that left the standard.
Ryan Hurst goes further and recommends narrowing each application’s trust to the CAs it actually needs. Standardize first, then narrow. Narrowing takes more effort, because you first have to analyze what each application actually uses.
One incident, a stack of advisories
This table is a snapshot from October 27. For the latest status across vendors, and test hosts to check your own clients, see the disclosure site. Each vendor’s own advisory remains the authority for that vendor.
Get Prem Sumetpong’s stories in your inbox
Join Medium for free to get updates from this writer.
Spring Boot published nothing, and it did not need to: a Spring Boot service trusts whatever JVM it runs on. That is the gap in one example. A team looking for a Spring Boot advisory finds none, while its exposure sits one layer down, in the JDK’s cacerts.
So what does it take to be protected?
To be definitively protected, every one of these has to be true at once:
- The vendor has shipped a fix. An announcement that a fix is coming is not a fix: Apple’s command-line fix will arrive “in a future software update”, and Cisco dates some fixes to December 8. Oracle’s fix covers only Java versions under Premier or Extended Support.
- Someone has tested the patch or instructions and pushed the change to production.
- They did it in every location: every host, every base image, every rebuilt binary, every appliance.
I have experienced all three: waiting for the vendor first, then for a team to incorporate the fix and test it, and then for a change to push it to production.
If any one of them is false, that client is still exposed. For a router, printer or appliance on frozen firmware, the first condition may never be true, and it will trust these certificates until they expire in 2039, as Scott Helme points out.
Each condition sits with someone else: the vendor’s release date, the team that patches, the team that owns the image. That is software supply chain risk. As Ryan Hurst puts it, a root store is “a set of security dependencies, and we should manage it like one.” Your protection depends on other people’s schedules, and you cannot see those dependencies without knowing where your trust lives.
There is also no standard for where a CA bundle lives or how it gets updated, so each row in the advisory table has its own location and its own update path. That is why changing a CA, or responding to a compromised one, costs time and resources.
The same incident also arrived as at least four CVEs, scored anywhere from medium (6.8 for Red Hat’s system store) to critical (9.1 for Oracle Java), depending on the component. A scanner reports each one separately, and the components with no CVE of their own, like a Spring Boot service or an appliance, do not show up at all.
Then test, because a version number is a claim. The disclosure site runs test hosts that serve chains through the compromised intermediates. Point each client at one. A certificate error means it rejects the chain. A successful connection means it still trusts a key whose value is 4.
The next one
Chrome’s removal of the emSign ECC Root CA — G3 from its root store was scheduled for October 27. Your users will not notice.
The same inventory pays off again when the CA in trouble is yours, though not in the same way for every CA.
A compromised private root is a valid scenario in the threat model. If it happened, every copy of that root would have to come out: cacerts imports, custom bundles in container images, mutual TLS trust stores, appliances. No browser vendor will push a blocklist for you, and no distribution will ship an advisory. The inventory is the only map of where that root lives, and it lets you add a replacement root to every bundle before you switch. The standard keeps that map short enough to stay current.
If a public CA you buy certificates from were distrusted, the first job is reissuing your certificates from a second CA, which is the first list. The trust-store inventory helps wherever you narrowed trust to that CA, through pins or per-partner bundles, because each of those has to change too.
eMudhra’s own customers are living this now. The company announced on October 22 that it is leaving the global TLS business, and roughly 30,800 certificates have to move to another CA.
Follow up with the disclosure site and check with your vendors. As each fix is applied, take the opportunity to inventory: record where that app’s CA bundle lives, so the different flavors of CA bundle locations are known. Then follow up with standardization changes.
If one of your CAs were distrusted or compromised tomorrow, which list would you open first?
If it were a public CA, I would open the certificate inventory first and reissue from the second CA, then the trust stores where we narrowed trust to it. If it were our private CA, I would open the trust-store inventory first, because the new root has to be in every bundle before a single certificate can move.
Sources
- Scott Helme: Eight bits of trust, eMudhra’s CA keys cracked in milliseconds
- Ryan Hurst: The WebPKI’s unmanaged software supply chain
- BleepingComputer: Mozilla, Google, Apple block compromised eMudhra CA certificates
- ThePrint: distrust vs. private key compromise
- Inc42: Google’s October 27 replacement deadline
- Cycode: why patching the OS is not enough
- BellSoft: distrusting the intermediates in Java
- Guillaume Laforge: the cross-sign trap in cacerts
- Disclosure site and test hosts
- Vendor advisories: Apple, Microsoft (Windows), Microsoft (Azure App Service), Red Hat, Ubuntu, Debian, AWS, OpenJDK, Oracle, Eclipse Temurin, IBM, Go, Python, Node.js, Cloudflare, Veeam, Cisco
The Hacker News: eMudhra to exit global SSL/TLS business
