Agent Evaluation — TryHackMe

As AI agents shift from being novelty tools to core components of modern software infrastructure, they introduce a completely new attack surface. How do we ensure an autonomous agent behaves securely, follows its instructions, and doesn’t leak sensitive organizational data? Enter TryHackMe’s Agent Evaluation room. Part of the broader AI security path, this hands-on lab challenges us to step into the shoes of an engineer tasked with evaluating, benchmarking, and securing AI-driven systems. In this writeup, I’ll walk you through the core concepts of agent design, breaking down the exact steps and methodology needed to clear the lab and master the fundamentals of AI agent evaluation.

Let’s begin!

Task 1 — Introduction

In the previous rooms, Agent Discovery, Agent Design, Agent Foundations, and Agent Building, you explored where AI agents can support security workflows, designed the NorthStar Fashion Security Investigation Agent, and progressively built its investigation capabilities.

The agent can retrieve alerts, search related SIEM logs, check IP reputation, consult organisational context, maintain investigation context, and return a structured verdict.

Now it is time to evaluate it.

A convincing final response does not prove that the investigation was performed correctly. The agent could retrieve the wrong alert, search the wrong SIEM field, miss relevant evidence, or reach the expected verdict for the wrong reason.

In this room, you will evaluate the NorthStar agent against reviewed security investigations. You will establish a baseline, diagnose failed behaviour, apply focused improvements, retest the affected cases, and run a complete regression to confirm that previously working behaviour still passes.

The goal is not simply to make the agent produce the expected answer. It is to make its improvements observable, measurable, and supported by evidence.

Learning Objectives

By the end of this room, you will be able to:

• Evaluate agent behaviour using reviewed security investigation cases
• Identify evidence-retrieval and decision-making failures
• Use evaluation results to guide focused agent improvements
• Compare behaviour before and after a change
• Run regression tests to detect previously working behaviour that has been broken

Prerequisites

Before starting this room, you should understand the concepts introduced in:

• Agent Discovery
• Agent Design
• Agent Foundations
• Agent Building

You should also be familiar with AI agents, tools, security investigation workflows, SIEM evidence, and structured agent verdicts.

Q.1: I’m ready to learn!

Ans.: No answer needed!

Task 2 — Establish the Baseline

Before improving the agent, you first need to understand how it behaves without any changes. This starting point is called a baseline, and it provides a reference for measuring whether later improvements actually make the agent more reliable.

In this task, you will use the NorthStar Agent Evaluation Workspace to run the Security Investigation Agent against five reviewed security cases and record its initial performance.

Click the View Site button at the top of this task to open the workspace.

Reviewed Evaluation Cases

The evaluation set contains five security investigations:

+------------------------------+---------+
|             Case             |  Alert  |
+------------------------------+---------+
| Routine office sign-in       | ALT-001 |
| Credential attack            | ALT-007 |
| Unusual location             | ALT-014 |
| New device                   | ALT-038 |
| Approved authentication test | ALT-049 |
+------------------------------+---------+

Each case includes an investigation request and a separately reviewed expected outcome used by the evaluator, similar to an exam question and its answer key. The agent receives only the investigation request, such as Investigate alert ALT-038, while the evaluator keeps the expected outcome hidden so the agent cannot simply reproduce the correct answer.

What Are We Measuring?

A final verdict is only one part of an investigation, so the evaluator checks several behaviours independently:

• Alert Argument verifies that the agent retrieved the requested alert, such as calling the alert tool with alert_id="ALT-038".
• Query Syntax checks whether the SIEM search uses supported syntax. For example, network.source_ip="203.0.113.42" is valid, while network.source_ip:203.0.113.42 is not supported in this environment.
• Query Target checks whether the agent searched the right evidence. A query can be syntactically valid and still use the wrong field or value.
• Verdict checks whether the final controlled verdict matches the reviewed outcome for the case.

Together, these measurements help distinguish where an investigation failed rather than treat every incorrect result as the same problem.

Run the Baseline

In the Agent Evaluation Workspace, click Run Evaluation.

The baseline candidate will be evaluated against all five reviewed cases. You should see:

✓ Routine office sign-in
✓ Credential attack
✗ Unusual location
✗ New device
✓ Approved authentication test

3 / 5 passed

The New Device case is highlighted with:

QUERY TARGET    FAIL
VERDICT         FAIL

This tells us that the agent retrieved the correct alert and used valid SIEM syntax, but searched the wrong evidence and ultimately produced the wrong verdict.

In the next task, you will inspect this failed investigation to determine why the query target failed.

Q.1: How many reviewed cases passed the baseline evaluation?

Ans.: 3

Let us run the baseline by clicking the green View Site button at the top right corner of the task. You should be presented with a split screen view:

Click the Run Evaluation button.

You should see a Step 1 result:

3 out of 5 passed.

Q.2: Which search-related measurement failed for the New Device investigation?

Ans.: query_target

Ok, so for this one, multiple answers are accepted but I thought query_target was the most convincing one and it worked, it was a right answer.

Scroll down and you’ll see two behaviors that failed:

If you add an underscore, it becomes a search query

Task 3 — Diagnose the Failure

The baseline showed that the New Device investigation failed two measurements:

QUERY TARGET    FAIL
VERDICT         FAIL

A failed score tells us what went wrong, but not necessarily why. In this task, you will inspect the investigation evidence to identify the cause of the incorrect search target.

Return to the NorthStar Agent Evaluation Workspace and click Investigate Failure.

Inspect the Failed Search

The alert is:

Sign-in from New Device

The device associated with the alert is:

northstar-maria-phone-0002

During the investigation, the agent searched:

device.device_name="northstar-maria-phone-0002"

The SIEM accepted the query, but returned:

0 results

The evaluator therefore reported:

Query Syntax     PASS
Query Target     FAIL
Verdict          FAIL

At first, these results may seem contradictory: if the query syntax was valid, why did the investigation still fail?

Valid Syntax Does Not Mean Useful Search

query_syntax checks whether the SIEM can understand the query format.

For example:

device.device_name="northstar-maria-phone-0002"

This query uses valid SIEM syntax: a supported field, an equals sign, and a quoted value. However, query_target evaluates something different: whether the agent selected the appropriate field and value for the evidence it intended to retrieve.

In this case, the value is a platform device identifier, not a human-readable hostname. Compare the two fields:

+--------------------+---------------------------+---------------------------+
|       Field        |          Purpose          |          Example          |
+--------------------+---------------------------+---------------------------+
| device.device_name | Human-readable device or  | LAPTOP-042                |
|                    | hostname                  |                           |
| device.device_id   | Platform or               | northstar-maria-phone-000 |
|                    | managed-device identifier | 2                         |
+--------------------+---------------------------+---------------------------+

The query was therefore syntactically correct, but it searched the platform identifier using the wrong field.

In the workspace, answer:

The query is syntactically valid. Why did the search still fail?

Select:

The agent searched a device identifier using the device-name field.

What Do Zero Results Mean?

There is another important problem with the investigation.

The search returned:

0 results

A zero-result search does not prove that no related activity exists; it only shows that this specific query returned no matches. The agent should therefore reconsider whether it searched the correct field before drawing a conclusion. In this case, it could search for the same device using the appropriate field:

device.device_id="northstar-maria-phone-0002"

Alternatively, it could continue the investigation using related entities such as the account or source IP.

This gives us an important evaluation principle:

0 results ≠ evidence does not exist

Before changing the agent, we now have a clear diagnosis:

• The SIEM query format was valid
• The platform identifier was searched using the wrong field
• The empty result should not have ended the evidence search
• The agent needs better guidance for retrieving related evidence

In the next task, you will apply a focused improvement and measure whether the failed behaviour changes.

Q.1: Which SIEM field should be used for a platform device identifier?

Ans.: device.device_id

The table gave us everything we need:

In the site, click the Investigate Failure button and it says, for Step 2, exactly what the task said:

Let’s answer the question:

You should search for an identifier using device_id, not device_name. This is why the search gave no results.

Q.2: Does one zero-result search prove that related activity is absent? (Yea / Nay)

Ans.: Nay

The explanation from that question said that zero-results mean that the query had no reesults, not that the whole field was absent.

Click Continue in the View Site and move onto the next task.

Task 4 — Improve the Candidate

In the previous task, you diagnosed the New Device failure.

The agent used valid SIEM syntax, but searched the platform identifier using the wrong field:

device.device_name="northstar-maria-phone-0002"

The search returned no results, and the investigation stopped too early. Now that the cause is clear, you can make a focused improvement that addresses the specific failure rather than changing the agent blindly.

Get Cyber With Rahman’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Return to the NorthStar Agent Evaluation Workspace and continue to the improvement step.

Improve the Evidence Search

The workspace asks:

Which change best addresses the measured failure?

Select the option that improves the SIEM search guidance by:

• Using device.device_id for platform identifiers
• Retrying an empty search using related account or source IP evidence

Then click:

Apply Improvement

The workspace compares the behaviour before and after the change.

Before

device.device_name="northstar-maria-phone-0002"

0 results
Query Target    FAIL
Verdict         FAIL

After

device.device_id="northstar-maria-phone-0002"

4 results
Query Target    PASS
Verdict         FAIL

The comparison shows that the improvement worked: the agent now searches an appropriate source of evidence. However, the investigation still fails because correcting evidence retrieval does not automatically correct the agent’s decision-making.

Inspect the Retrieved Evidence

The improved search reveals that the device:

• Was registered previously
• Is company managed
• Is compliant
• Has previously been used with MFA

The alert, however, claims:

Sign-in from New Device

The authentication itself occurred, but the evidence contradicts the condition that the device is new.

This distinction is important when assigning a verdict.

+----------------------+----------------------------------------------------+
|       Verdict        |                      Meaning                       |
+----------------------+----------------------------------------------------+
| TruePositive         | Evidence supports the suspicious condition         |
|                      | asserted by the alert                              |
| BenignPositive       | The alert accurately describes real activity, but  |
|                      | that activity is routine or authorised             |
| FalsePositive        | Reliable evidence contradicts a condition asserted |
|                      | by the alert                                       |
| InsufficientEvidence | Available evidence cannot sufficiently support or  |
|                      | contradict the alert                               |
+----------------------+----------------------------------------------------+

For this investigation, the correct verdict is:

FalsePositive

The sign-in happened, but the condition that triggered the alert was contradicted by reliable historical evidence.

This is different from BenignPositive.

A routine office sign-in may be BenignPositive because the alert accurately describes activity that is expected.

A new-device alert for a device that is already registered is FalsePositive because part of the alert itself is incorrect.

Improve the Decision Guidance

In the workspace, select:

FalsePositive

Then click:

Apply Decision Guidance

The candidate is now updated with more general guidance for distinguishing routine activity from alert conditions contradicted by evidence.

The workspace then reruns the affected investigations:

Targeted Evaluation

✓ Credential attack
✓ Unusual location
✓ New device
3 / 3 passed

The focused changes improved the cases they were intended to fix, but the evaluation is not complete yet. A change can correct one behaviour while unintentionally affecting another case that previously worked.

In the next task, you will run a full regression to confirm that the improvement holds across all reviewed cases.

Q.1: Which verdict applies when reliable evidence contradicts a condition asserted by the alert?

Remember the table:

Ans.: FalsePositive

Q.2: Did fixing the evidence search automatically fix the verdict? (Yea / Nay)

Ans.: Nay

Select the correct improvemtn in the site:

Well:

The verdict didn’t pass, so the answer to this question is no (nay).

Let’s answer the question to this Step and move on.

We thought the device was new, but actually it is old, and this evidence contradicts our alert condition.

Click Apply Decision Guidance:

Moving on.

Task 5 — Test for Regressions

The targeted evaluation now passes:

✓ Credential attack
✓ Unusual location
✓ New device

3 / 3 passed

This confirms that the changes improved the investigations they were intended to fix, but the candidate is not ready yet. An improvement can correct one behaviour while unintentionally breaking another that previously worked; this is known as a regression.

For example, stronger FalsePositive guidance might fix the New Device case while incorrectly changing a routine office sign-in that should remain BenignPositive.

Why Regression Testing Matters

Targeted testing asks whether the changed behaviour improved, while regression testing checks whether that change affected behaviour that previously worked. A reliable evaluation workflow requires both.

Return to the NorthStar Agent Evaluation Workspace and answer:

The affected cases now pass. Are we ready to use the candidate?

Select:

Not yet. We need to check that previously working cases still pass.

Then click:

Run Full Regression

Run the Complete Evaluation Set

The evaluator runs all five reviewed investigations again using the improved candidate.

You should see:

✓ Routine office sign-in
✓ Credential attack
✓ Unusual location
✓ New device
✓ Approved authentication test

5 / 5 passed

The workspace also compares the new result with the original baseline:

BASELINE                 CANDIDATE

3 / 5                    5 / 5

The two previously failing investigations now pass, while the three investigations that already worked continue to pass.

This is stronger evidence of improvement than testing only the changed cases.

What Does 5 / 5 Mean?

The result shows that the candidate passed the current reviewed evaluation set.

It does not prove that the agent will always behave correctly, nor does it mean that every possible security investigation has been tested.

Real agent evaluation normally expands over time as teams discover:

• New failure modes
• Edge cases
• Regressions
• Changes to prompts, tools, or models
• New security requirements

Each useful failure can become another evaluation case that protects future versions of the agent.

The important workflow is therefore not simply:

Make a change
→ get a better answer

It is:

Measure
   ↓
Diagnose
   ↓
Improve
   ↓
Retest
   ↓
Regress

Once the full regression reaches 5 / 5, the workspace will reveal the room flag.

So, after clicking Apply Decision Guidance from the previous task, we met this:

Regression means that an improvement can break another that previously worked. It says so in the 2nd paragraph in this task:

Now let us run a full regression (after clicking the right answer you’ll see this):

Congrats!

Q.1: What type of testing checks whether a change broke behaviour that previously worked?

Ans.: Regression

Q.2: How many reviewed cases passed the final regression?

Ans.: 5

Q.3: What is the flag revealed after completing the full regression?

Ans.: Find it yourself!

Scroll to the very bottom of the site. There’s your prize:

Completing TryHackMe’s Agent Evaluation room highlights a fundamental shift in AI security. We can no longer treat large language models as basic text predictors; they are autonomous agents that plan, reason, and manipulate external tools. Testing them requires moving past simple static prompt-injection checks to evaluate full execution trajectories, sandbox isolation boundaries, and decision-making logic under adversarial conditions.

As AI security agents take on higher organizational privileges, establishing clear evaluation metrics and rigorous testing loops isn’t just a best practice — it is an absolute operational requirement to prevent catastrophic logic failures.

Happy Hacking!

Best regards,

BgB1T3.