A Passing Check Is Not an Enforced Control
From written requirements to automated enforcement.
In the last piece I connected control identifiers to the resources they governed, so a reviewer could easily see which security requirement applied to a particular resource instead of having to search through separate documentation or a spreadsheet.
That helped create a clear connection between the requirement and the technology it applied to. But an annotation is still documentation, and documentation refuses nothing.
I wanted to take the next step and build the version of the control that actually evaluates a proposed change and says no when the requirement is not met. That turned out to be a different discipline, and most of what it taught me was about the word “green.”
Making the control a program
The language is Rego, which is used to write policies that can automatically evaluate whether a proposed infrastructure change meets defined requirements. Instead of asking a person to review every configuration manually, the policy looks at the proposed infrastructure and checks it against specific rules.
For example, a security requirement might say that sensitive information stored in the cloud must be protected through encryption. Another requirement might say that administrative systems should not be exposed directly to the public internet. Another might require cloud resources to have specific labels so they can be identified, managed, and governed correctly.
The policy turns those requirements into automated checks. If a storage resource does not have the required encryption, the policy reports a violation. If a firewall allows unrestricted access to an administrative port, the policy reports a violation. If a required label is missing, the policy reports a violation. If nothing violates the policy, it returns no findings.
That last part sounds simple, but it is where the first important lesson appears. A policy returning no findings does not necessarily mean the environment is compliant. It only means the policy did not identify a violation. Writing the rule is half of it. Proving it works is the other half, and that takes two tests, not one.
The obvious test provides infrastructure that should be compliant and confirms that the policy accepts it. The test people skip provides infrastructure that should violate the requirement and confirms that the policy rejects it. That second test is critical. A policy that has only been tested against good infrastructure has never demonstrated that it can actually enforce anything. It has only demonstrated that it can recognize something it was already expected to recognize. It is the negative-test lesson from the last project arriving in a new language: a control that has only seen valid input is an assumption, not a control.
A control travels but its implementation does not
I wrote the first three policies against Google Cloud, then pointed them at an Amazon plan to see what would happen. They passed. All of them, green across the board. They passed because they had checked nothing. The policies had been written specifically to look for Google Cloud storage resources. When I evaluated an AWS environment, those Google Cloud resources were not present. The policies therefore found nothing to evaluate. No violation was reported. The result looked like compliance, but it was not compliance. That is the most dangerous state a compliance check can be in: enforced-looking and enforcing nothing. The color on the dashboard is identical whether a control is clean or simply absent.
This forced me to separate two things that can easily get mixed together: the control requirement and the way that requirement is implemented. For example, SC-28 is a NIST control focused on protecting information at rest. In practical terms, the requirement is concerned with protecting information while it is stored. That requirement can apply whether an organization uses Google Cloud, AWS, or another technology platform. The requirement stays the same. The way I verify that requirement changes depending on the technology.
A Google Cloud storage resource has its own configuration and way of representing encryption. An AWS storage resource has a different configuration and represents the relationship between storage and encryption differently. The control requirement is portable. The technical implementation is not. So I wrote AWS versions of the policies while keeping the same control identifiers. The library stayed organized around the control instead of around the cloud platform.
That became an important governance decision. If I organize the policies around technology, I end up thinking about Google Cloud controls and AWS controls as separate things. If I organize them around the requirement, I can ask a much better question: How is SC-28 being enforced in Google Cloud? How is SC-28 being enforced in AWS? The requirement stays consistent while the technical implementation changes. It is the same decision as the reusable module a project ago: settle what is invariant, and let only the rest vary.
Where the facts live
The AWS rules taught me another important lesson about automation. Sometimes the information needed to evaluate a requirement is not represented in the same way across platforms. On Google Cloud, encryption can be represented as part of the storage resource itself. The policy can look at the storage resource and determine whether encryption has been configured. AWS can represent encryption differently. The encryption configuration can exist as a separate resource that is connected to the storage resource.
Get Jay Young’s stories in your inbox
Join Medium for free to get updates from this writer.
That creates a problem when evaluating the infrastructure before it has actually been deployed. At that point, some values are already known. Other values do not exist yet. For example, a storage resource may eventually receive a name when the infrastructure is deployed. Before deployment, that final value may not exist. My first approach was to compare the names of the resources to determine whether the encryption configuration was associated with the correct storage resource. That approach depended on information that was not available yet.
The better approach was to evaluate the relationship between the resources. Instead of asking, “Do these two resource names match?” the policy asks, “Is the required encryption configuration connected to this storage resource?” That distinction is important because infrastructure automation does not always have every final value available before deployment.
A proposed infrastructure configuration contains information about resources that are already known as well as information about how those resources are supposed to be connected. The policy has to understand both. Knowing which information is available before deployment, and which information only becomes available later, was most of the work. And none of it is particularly visible from the governance side of the wall.
That is one of the things I have come to appreciate about policy as code. A governance requirement can be straightforward. Turning that requirement into a reliable automated control requires understanding how the technology actually represents the requirement.
The gate has to be seen to refuse
A policy that runs successfully in a test environment is not yet a control. It becomes a control when it is placed into the process used to approve and deploy infrastructure changes and has the ability to stop a change that violates a requirement. I wrapped the policy library in a single script that the deployment process could call. The process evaluates the proposed infrastructure before it is deployed. If the configuration meets the requirements, the process continues. If the configuration violates a required control, the process fails and the change cannot continue until the issue is corrected or an approved exception is provided.
I ran it against a compliant configuration and got a pass. Then I removed the encryption configuration from a copy and ran it again. This time it failed. It identified the storage resource, identified the control, and explained what needed to be corrected. That was the moment the exercise changed for me.
The refusal is the deliverable.
The point was no longer that I had written a policy capable of identifying a problem. The point was that the policy had become part of the process that prevents the problem from being deployed. That is the difference between checking a control and enforcing a control. A check can tell me that something is wrong. An enforced control can prevent the change from moving forward.
That distinction matters in GRC because evidence that a control identified an issue is not the same as evidence that the control prevented the issue from occurring. A gate you have only watched pass is the negative-test mistake in a new costume.
I made a smaller version of that mistake, and it is worth admitting. Early on, one policy returned “0 tests, 0 passed.” I read it as success. Nothing had failed. It meant the opposite. The policy file had not been loaded, so the policy engine had nothing to evaluate. Zero failures because zero checks. That is the precise error I have spent years documenting when other people make it: reading the absence of a complaint as evidence of compliance. Knowing its name does not make it stop being easy.
What I took from it
The thread through all of it is that green is not one thing. A check can pass because the infrastructure is sound. It can pass because the policy did not recognize the infrastructure. It can pass because the policy never ran. Those three outcomes can look identical on a dashboard and be entirely different underneath. The discipline is refusing to accept the color without asking which one it is.
That is also where the control identifiers became more useful to me. AC-3 is about access enforcement. At a governance level, it asks whether access to systems and resources is actually controlled according to defined requirements. SC-28 is about protecting information at rest. At a governance level, it asks whether information that is stored is protected through appropriate safeguards.
The identifiers provide a common language between governance and implementation. They allow the requirement to be traced from the control to the technical implementation and eventually to the evidence showing whether the requirement was met.
But the identifier itself does not enforce anything. The policy does. The deployment process does. The failed change does. That distinction sounds obvious when written down. It is much less obvious when the dashboard is green.
Two smaller lessons stayed. A separate scanner I run flags my deliberately broken test fixtures every time, correctly, because they are broken on purpose. The right response is to scope the scanner away from them and document why, not to pretend they are clean. Suppressing a finding is a risk acceptance, and it belongs where a reviewer can see it, the same rule I have always applied to a compensating control. If something is intentionally outside the normal requirement, that decision should be visible. There should be a reason, an owner, and enough documentation for someone else to understand why the exception exists and a policy library organized by control identifier instead of by platform is the version an audit can follow.
I still write requirements. I am much more interested now in the moment one stops being a sentence and becomes a program that reads a proposed change and refuses it, and in the gap, which I used to underestimate, between a check that passed and a control that is doing its job.
Repositories: compliance-policies, built while working through the Compliance as Code curriculum at the GRC Engineering Club.
