Why Security Questionnaire Automation Still Needs Human Review
Hey everyone, I’m Abhijai Khanna.
I have almost two years of experience working in GRC, with Security+ and ISO 27001 Lead Auditor certifications. I’ve worked both in consulting and internally at an HR SaaS company.
Over those two years, I’ve personally worked on 150–200+ security questionnaires, vendor assessments, and similar requests.
And after doing this repeatedly, I started noticing something that bothered me about questionnaire automation.
The problem isn’t that AI can’t answer security questions.
The problem is that sometimes it answers them too simply.
The “Yes” problem
Take a very common question:
“Is MFA enabled?”
The answer is usually:
Yes.
But that’s not necessarily a useful answer.
Where is MFA enabled?
For which applications?
For which users?
Are there exceptions?
Are privileged accounts treated differently?
Is MFA enforced through Okta, Microsoft Entra ID, AWS IAM, or something else?
What happens to service accounts?
These details can matter enormously depending on what the customer is actually asking.
During my previous work, we had tools that helped significantly with collecting evidence and answering questionnaires. They saved time, but I still found myself having to review and modify answers because a simple “Yes” didn’t always capture how the control actually worked inside the organization.
That’s where I started questioning how far questionnaire automation should go.
Then there are hallucinations
Another problem I encountered was surprisingly simple:
The answer could be technically plausible, but wrong.
For example, imagine a questionnaire asks:
“How frequently do you conduct disaster recovery drills?”
Suppose the actual answer is quarterly.
An automated system might pull information from previous responses or documentation and produce:
“We conduct disaster recovery drills annually.”
That’s a small difference in wording.
But in security and compliance, quarterly and annually are not the same thing.
And if you’re answering dozens of questionnaires, you don’t necessarily notice every small change.
You end up spending time reviewing the AI’s work anyway.
The problem with using old answers
This became even more interesting when I thought about how much information organizations accumulate over time.
A company might have answered a SOC 2 questionnaire in 2024.
Get Abhijai Khanna’s stories in your inbox
Join Medium for free to get updates from this writer.
Two years later, the company has:
- changed cloud infrastructure
- changed identity providers
- changed policies
- changed control frequencies
- introduced new systems
- changed exceptions
- changed processes
Yet an AI system looking at previous questionnaires can easily find that old answer and decide that it’s relevant.
And sometimes it is.
Sometimes it isn’t.
That’s where I think recency and context become extremely important.
A 2024 answer shouldn’t automatically become a 2026 answer just because the question looks identical.
So what should questionnaire automation actually do?
I don’t think the answer is to eliminate humans completely.
I’d argue the opposite.
AI should handle the repetitive work, but it should also know when to stop and ask for human review.
For example:
Question: Is MFA enforced for all privileged users?
AI: Yes.
Instead, the system could potentially look at the available information and say:
Review required.
The current evidence appears inconsistent with the previous response. The supporting policy was last updated in 2024, while the available technical evidence is from 2026.
Or:
Review required.
Previous questionnaires state that DR testing occurs annually, while the current policy states quarterly.
The AI isn’t deciding whether the company is secure.
It’s simply saying:
“Something here doesn’t line up. You should check this before sending it to the customer.”
That’s a very different proposition.
Where I think the industry could go
I’m not against security questionnaire automation.
Quite the opposite.
After answering 150–200+ questionnaires, I think automation is extremely useful.
But I wonder whether the next step shouldn’t simply be:
“AI generates the answer.”
Maybe it should be:
“AI generates the answer, shows why it believes the answer is correct, checks the age and consistency of the supporting information, and tells you when a human needs to look at it.”
Because ultimately, a security questionnaire isn’t just a collection of Yes/No questions.
It’s a representation of how an organization actually operates.
And organizations change.
Policies change.
Systems change.
Controls change.
Exceptions change.
Frequencies change.
So I’m curious:
For people who work with security questionnaires, vendor assessments, DDQs, or RFP security sections — how are you handling this today?
Do your current tools already solve this well?
Or do you still find yourself manually checking whether an answer is actually current and accurate before sending it to a customer?
I’d genuinely like to hear how other teams handle this.
