Scenario Testing That Isn’t Theater
Article 5 of 8 | The Hidden Single Points of Failure Series
Tabletop exercises usually focus on scenarios the organization feels comfortable practicing, rather than the ones the dependency graph suggests are most important.
The Tabletop Problem
Most business continuity plans rotate through the same three scenarios: a ransomware attack, a natural disaster at headquarters, or a pandemic that disrupts staffing. Each year, the exercise feels almost identical, just with different details. This isn’t due to laziness — it’s what the tabletop format encourages. The scenario has to be easy for everyone to follow, simple enough to act on, and able to wrap up in a single afternoon. Ransomware and floods both fit these requirements.
The chosen scenario also needs to be one the organization can survive on paper, since that’s how the exercise is judged. Facilitators who suggest scenarios with no good outcome usually aren’t asked back. This narrows the options to dramatic events that justify the meeting but aren’t so deep that the report would say the organization failed. As a result, these exercises are realistic because events like ransomware and floods do happen, but they’re also a bit of theater, chosen for how well they work in a meeting rather than for revealing real weaknesses.
The events that make headlines are, almost by definition, the ones organizations already understand well enough to fear.
This process doesn’t ask any structural questions. No one looks at the vendor dependency graph to see which failure would impact the most critical functions. Instead, scenarios are picked based on what people remember, recent headlines, or incidents at other organizations. That might work for choosing a conference topic, but it’s not a good way to find the real risks to the business. The events that make headlines are usually the ones organizations already know to worry about.
The result is that exercises become more polished and better run each year, but they still focus on the same scenarios everyone already knows how to handle. The quality of the rehearsal improves, but the range of risks covered does not. A tabletop program could run for years without ever testing the part of the system most likely to cause real damage, just because no one thought to check the dependency graph instead of the news.
Threat-Led Penetration Testing Under DORA: Intent and Scope
Threat-led penetration testing (TLPT) exists, in part, as a correction to exactly this drift. Article 26 of the Digital Operational Resilience Act requires in-scope financial entities to run TLPT at least every three years, against live production systems, using scenarios built from real threat intelligence rather than a facilitator’s imagination. Financial entities don’t self-select into that population. Competent authorities identify who’s in scope under Article 26(8), weighing how far the entity’s activities and services affect the financial sector, financial-stability concerns including its systemic character, and its ICT risk profile and maturity. The regulatory technical standards under Article 26(11), developed jointly by the European Supervisory Authorities and the ECB, were built in accordance with the TIBER-EU framework the ECB and national central banks had already been running since 2019, which is why a TIBER-EU test conducted to the aligned standard can satisfy the DORA cycle rather than sitting alongside it as duplicate work.
This approach is different from a tabletop in important ways. An outside threat intelligence provider, separate from the red team, creates a custom report on the most likely attackers and their methods. That report shapes the attack scenario. The red team, which is internal most of the time but external at least once every three cycles, carries out the test on live systems, not in a test environment. Only a small White Team knows the test is happening. Afterward, a required purple-teaming phase connects what the red team did with what the blue team detected.
The scoping requirement ties back to earlier parts of this series. A threat-led test must cover the critical functions the organization decides need testing. Before making that decision, Article 26(2) says the organization must identify every relevant ICT system, process, and technology that supports those functions, including anything outsourced or handled by a third party. This identification is the assessment, and its results define the exact scope of TLPT, which the competent authority must validate. In practice, this assessment uses the same register described earlier: RT.02.01 for contracts and RT.06.01 for the business functions they support. When done right, TLPT is about testing severe but plausible scenarios based on a mapping exercise, not just gut feeling. Whether this approach works depends on how thoroughly the mapping is done.
Severe-but-Plausible, Sourced From the Graph
“Severe but plausible” is the standard that TIBER-EU scenario builders use. At first, it sounds like the same standard a good tabletop facilitator would use, but it’s not. The difference lies in how each part of the phrase is defined.
In a tabletop exercise, severity is judged by how much risk the group is comfortable talking about, and plausibility is based on what feels familiar. Both are decided by the people in the room, influenced by what they’ve read or seen happen elsewhere. That’s why ransomware is always considered, but a shared subcontractor outage rarely is. It’s not that the subcontractor scenario is less severe — it just doesn’t feel likely to people who don’t know the subcontractor exists, since it never showed up in their records.
Get Patrick Lefler’s stories in your inbox
Join Medium for free to get updates from this writer.
A graph-based exercise flips this approach. Severity is no longer a guess — it’s measured by the weighted betweenness score described earlier, which shows how central a node is in the system and how many critical functions depend on it. This score doesn’t care if anyone in the room knows the node; it only shows how much would break if it failed. Plausibility is also based on real threat intelligence: which attackers target this type of entity, what methods they use, and how often. For example, a subcontractor that handles file transfers for several financial institutions is a known target for credential theft and supply chain attacks, even if it’s never been in the news.
The real question isn’t what disaster the group can imagine and handle. It’s which node has the highest structural severity, and whether threat intelligence shows a realistic way it could be compromised.
When you combine these two factors, the way you choose scenarios changes completely. A subcontractor like Meridian Data Services, with a betweenness score three times higher than any top-tier vendor but no direct contract value, easily meets the severity test. Whether it’s plausible depends on whether threat intelligence has identified it as a target, which only happens if someone has actually named it.
Case Pattern: The Vendor Nobody Modeled as a Vendor
On July 19, 2024, a routine content update to CrowdStrike’s Falcon sensor caused Windows systems running it to crash on boot, worldwide, within the same window. Roughly 8.5 million Windows devices went down. Delta Air Lines canceled close to 7,000 flights over five days and later put the cost near $500 million. Wells Fargo, Charles Schwab, and TD Bank all reported disruptions; banks across Australia and New Zealand saw online banking services degrade the same day. Parametrix later estimated the financial hit to Fortune 500 companies alone at $5.4 billion, and total global losses have been put near $10 billion.
Before that date, no one imagined a tabletop scenario where the endpoint security agent would cause a system-wide failure. This wasn’t because the risk was unusual, but because the agent didn’t fit into the usual ways organizations think about dependencies. The vendor risk register only looked at CrowdStrike’s cost and the data it accessed, treating it as a secure, well-audited purchase that needed little monitoring. Tabletop exercises focused on what an attacker might do, always seeing the EDR agent as a defender, not a risk. Because it was bought to reduce risk, it was never considered as a possible source of problems.
If you analyze the same tool using the dependency graph, the exemption disappears. A kernel-level agent installed on every endpoint supports nearly every critical function that uses a computer — like banking terminals, trading desks, teller stations, claims processing, and call centers. When you calculate betweenness, this agent isn’t just near the top of the dependency graph; it’s at the very top, more connected than any tier-one vendor, because no other node is as widely used across the organization’s systems.
A scenario based on this data, not just imagination, asks a clear question: what happens if the agent meant to protect every critical function actually disables them all at once, due to a bad update instead of an attack? This isn’t a far-fetched idea — it’s the direct result of using the calculation from the fourth part of this series on a node that most registers never even list as a vendor.
Figure 1. Structural severity, the weighted betweenness score built out in the fourth piece in this series, plotted against how familiar each scenario feels to rehearse. The annual tabletop rotation clusters in the lower right. Meridian Data Services and the CrowdStrike pattern sit in the upper left, structurally indicated and never tested.
A Scenario Library, not a Scenario
The solution isn’t to run a better annual tabletop exercise. Instead, it’s to replace the single yearly scenario with a library of scenarios generated from the dependency graph and updated whenever the graph changes. The register-as-code system described earlier already recalculates centrality when there’s a new SOC 2 report or contract change. That same process should update the scenario library, ranking scenarios by severity instead of by who gets to pick the topic.
This ranking serves two purposes. The highest-scoring nodes, like Meridian Data Services and similar cases, become the main focus for the scoping assessment required by Article 26. These are ICT third parties that support critical functions, identified by their role in the system rather than by contract timing. Mid-level nodes, which are important but not critical enough for a live red team test, are used for tabletop exercises. Now, these exercises target nodes chosen by the graph, not just by facilitator preference. Low-scoring nodes are simply monitored, unless a graph update changes their ranking.
The library also creates a natural home for DORA’s pooled testing provision, which lets multiple financial entities share a single TLPT against a common provider rather than each running a redundant test. A subcontractor with Meridian’s structural signature is, by definition, shared across institutions; a scenario library built from centrality is what makes that sharing identifiable in the first place; nobody proposes pooling a test against a vendor their own register never flagged as central.
Updating the scenario library whenever the graph changes also fixes a problem that annual planning can’t. A node’s betweenness score can change at any time. For example, if a new subcontractor is added or a SOC 2 report names a shared processor, the ranking updates right away. A library that refreshes with each change shows these updates immediately, instead of waiting for next year’s planning. An annual tabletop calendar, planned months ahead, can’t keep up because it picks scenarios before the graph is even finished updating.
Conclusion
The issue with tabletop exercises isn’t a lack of rigor. Facilitators keep improving the exercises, scripts, and debriefs, but that doesn’t address the real problem: how scenarios are chosen in the first place. Experienced risk professionals tend to pick scenarios they’re already comfortable discussing. That’s what group discussions are good at, but it’s not what dependency graphs are for.
DORA’s threat-led testing approach gets part of this right: severity and plausibility should be based on evidence, not guesses, and the rules focus testing on third parties that support critical functions, not just on what the group finds interesting. But the regulation doesn’t create the input itself. If a TLPT only looks at tier-one vendors, it keeps all the blind spots discussed in this series, just with a more formal process. The dependency graph described earlier is what actually fills that gap, because it highlights critical nodes whether or not they seem familiar.
Meridian Data Services and a kernel-level security agent installed everywhere may seem unrelated as vendors, but as nodes, they share key traits: high betweenness, low visibility, and they’re missing from every written scenario. A scenario library built from the dependency graph will spot both before an outage happens. One based on imagination won’t find either, no matter how often it’s updated.
Next up in the Hidden Single Points of Failure Series: The RTO/RPO Lie. Recovery time and recovery point objectives are governance fiction until tested against the real dependency chain — not the vendor’s own SLA claims.
