Part 4: Building Evals - How Do You Know Your Agent Is Getting Smarter?

The Question Nobody Asks Early Enough

You've built an agent. It produces impressive-looking reports. But how do you know:

• It's actually getting the right answer?

• A change to the prompt didn't silently break it?

• The underlying model update didn't introduce a regression?

• Your new dead-end trigger is actually firing?

Without evals, you're flying blind. Every change to the agent is a guess — you push it, hope it still works, and find out the hard way when a developer says "the agent told me to add a waitForTimeout."

---

What Makes Agent Evals Different From Unit Tests

Unit tests verify deterministic code: given input X, expect output Y. Agent evals are different:

	Unit Tests	Agent Evals
Output	Deterministic	Non-deterministic (LLM varies)
Correctness	Binary (pass/fail)	Graded (0–2 per dimension)
Verification	Exact match	Semantic similarity + anti-pattern checks
Speed	Milliseconds	30–120 seconds per case
Frequency	Every commit	On agent changes + weekly

You can't assertEqual(output, expected) because the agent might phrase the same correct diagnosis differently each run. Instead, you check:

1. Did it identify the right root cause? (semantic match)

2. Did it point to the right file? (exact match)

3. Did it cite evidence? (pattern count)

4. Did it avoid known-bad conclusions? (anti-pattern check)

5. Did it follow the reasoning methodology? (structural check)

---

The Golden Dataset: Ground Truth You Already Have

The hardest part of agent evals is getting ground truth. For the trace-rca agent, ground truth is surprisingly easy: every trace you've already manually debugged is a potential eval case.

When you debug a test failure and find the root cause, you have:

• The trace .zip file (input)

• The root cause you discovered (expected output)

• The file you fixed (expected fix location)

• What the answer is NOT (anti-patterns)

Turn that into a YAML file:

trace: evals/traces/mock-mismatch-001.zip
category: mock-mismatch

expected_root_cause: "mock operationName doesn't match the actual GraphQL query sent by the app"
expected_fix_file: "e2e/specs/tickets/ticket-list.spec.ts"
expected_fix_type: "update mock operationName to match actual query"

must_not_conclude:
  - "timing issue"
  - "increase timeout"
  - "waitForTimeout"
  - "add a wait"
  - "race condition"

dead_end_triggers_expected:
  - "DE-2"
  - "DE-4"

The must_not_conclude field is the most valuable part. It encodes "the agent should NEVER say this for this trace." This catches the most dangerous regressions — where the agent produces a confidently wrong answer.

---

The Five Scoring Dimensions

Each eval case is scored on five axes (0–2 each, max 10 total):

Dimension	What it measures	0	1	2
Root Cause	Did it find the actual problem?	Wrong	Partially right	Exact
Fix Precision	Does the fix point to the right place?	Wrong file	Right area	Exact file:line
Evidence	Are claims backed by code references?	No citations	Some	Every claim cited
No Anti-Pattern	Did it avoid bad conclusions?	Suggests waitForTimeout	—	Clean
Phase Compliance	Did it follow the reasoning chain?	Skipped phases	Thin	All substantive

Not all dimensions need human review:

Fully automatable:     no_antipattern, fix_actionable, evidence_cited
Heuristic + human:     root_cause_correct, phase_compliance

The automated checks catch ~80% of regressions without any human effort. The remaining 20% (semantic root cause accuracy) gets flagged for review.

---

The Eval Runner

The runner is a zero-dependency TypeScript script that:

1. Loads golden cases from YAML files

2. Invokes the agent on each trace

3. Auto-scores the output

4. Compares against the baseline

5. Reports pass/fail/review for each case

node evals/run-eval.ts

Output:

═══════════════════════════════════════════════════════════════════════════
  TRACE-RCA AGENT EVALUATION
═══════════════════════════════════════════════════════════════════════════

Found 5 golden case(s) to evaluate.

Running: mock-mismatch-001 ... ✅ PASS (9/10) [47.2s]
Running: graphql-error-001 ... ✅ PASS (10/10) [38.6s]
Running: feature-flag-001 ... ⚠️  REVIEW (6/10) [52.1s]
Running: null-data-001 ... ✅ PASS (8/10) [41.3s]
Running: infra-failure-001 ... ✅ PASS (8/10) [22.7s]

────────────────────────────────────────────────────────────────────────
SUMMARY
────────────────────────────────────────────────────────────────────────
  Cases:     5
  Passed:    4
  Errors:    0
  Score:     41/50 (82%)
  Avg time:  40.4s

Cases marked ⚠️ REVIEW need a human to check whether the auto-scorer missed a correct-but-differently-phrased answer, or whether the agent genuinely got it wrong.

---

Anti-Pattern Scoring: The Most Valuable Check

The no_antipattern dimension is binary and fully automatable — it string-matches the agent's output against known-bad conclusions:

const hasAntipattern = golden.must_not_conclude.some((bad) =>
  output.toLowerCase().includes(bad.toLowerCase()),
);
scores.no_antipattern = hasAntipattern ? 0 : 2;

This single check catches the most dangerous regression: the agent producing a confidently wrong answer. A hesitant "I cannot determine the root cause" is fine. "This is a timing issue, add waitForTimeout(3000)" is catastrophic — it leads developers to make the test worse.

Anti-patterns I test for:

must_not_conclude:
  - "timing issue"          # almost never the real cause
  - "increase timeout"      # symptom treatment, not a fix
  - "waitForTimeout"        # explicitly prohibited in our codebase
  - "add a wait"            # same as above, different phrasing
  - "race condition"        # often used as a hand-wavy non-answer
  - "API succeeded"         # dangerous when GraphQL errors hide in 200s
  - "network call succeeded" # same as above

---

Regression Detection: The Baseline

After running evals and verifying the results are correct, save them as the baseline:

node evals/score-report.ts --update-baseline

This writes baseline.json:

{
  "total_score": 41,
  "max_score": 50,
  "case_count": 5,
  "updated_at": "2026-07-04T08:00:00.000Z",
  "per_case": [
    { "name": "mock-mismatch-001", "score": 9, "max": 10 },
    { "name": "graphql-error-001", "score": 10, "max": 10 },
    ...
  ]
}

On subsequent runs, if the overall score drops more than 10%, the runner exits with code 1:

────────────────────────────────────────────────────────────────────────
REGRESSION CHECK
────────────────────────────────────────────────────────────────────────
  Baseline:  82%
  Current:   68%
  Delta:     -14%
  ⚠️  REGRESSION DETECTED — score dropped >10% from baseline

This makes it CI-gate-able:

node evals/run-eval.ts || echo "⚠️ Agent regression — do not merge"

---

Failure Categories: Building a Diverse Dataset

A good eval suite covers the space of failures your agent encounters:

Category	What it tests	Example failure
mock-mismatch	Agent traces mock setup vs actual request	operationName typo in route handler
graphql-error-in-200	Agent looks beyond HTTP status codes	FORBIDDEN error buried in response body
feature-flag	Agent checks conditional rendering logic	Component returns null when flag is off
null-data	Agent distinguishes "no error" from "no data"	Query succeeds but returns { data: null }
infrastructure	Agent knows when NOT to diagnose	Browser crash, partial trace

Each category tests a different aspect of the agent's reasoning. A regression in mock-mismatch cases means something changed in how the agent reads route handlers. A regression in infrastructure means the agent lost its ability to say "I can't diagnose this."

---

The Score Report: Drill Down Into Weaknesses

node evals/score-report.ts

Produces a per-dimension breakdown:

═══════════════════════════════════════════════════════════════════════════
  TRACE-RCA EVAL SCORE REPORT
═══════════════════════════════════════════════════════════════════════════

  Overall Score: 41/50 (82%)  ████████░░

────────────────────────────────────────────────────────────────────────
  SCORES BY DIMENSION
────────────────────────────────────────────────────────────────────────

  Root Cause            8/10 ( 80%)  ████████░░
  Fix Precision         9/10 ( 90%)  █████████░
  Evidence              8/10 ( 80%)  ████████░░
  No Anti-Pattern      10/10 (100%)  ██████████
  Phase Compliance      6/10 ( 60%)  ██████░░░░

────────────────────────────────────────────────────────────────────────
  SCORES BY CATEGORY
────────────────────────────────────────────────────────────────────────

  mock-mismatch          1 cases   9/10 ( 90%)  █████████░
  graphql-error-in-200   1 cases  10/10 (100%)  ██████████
  feature-flag           1 cases   6/10 ( 60%)  ██████░░░░
  null-data              1 cases   8/10 ( 80%)  ████████░░
  infrastructure         1 cases   8/10 ( 80%)  ████████░░

This tells you exactly where to focus. In this example, "Phase Compliance" is the weakest dimension — the agent is sometimes skipping or rushing through reasoning phases. And feature-flag cases are the weakest category — the agent struggles with conditional rendering logic.

Both are actionable: strengthen the skill file's phase enforcement, and add a dead-end trigger for feature flag patterns.

---

When to Run Evals

Trigger	Why
Changed agent config, prompt, or skill file	Direct impact on agent behaviour
Changed analysis scripts	Output format affects how agent reasons
Weekly (scheduled)	Model drift — provider updates can change behaviour
Before merging agent PRs	Gate to prevent regressions

The weekly run is important. Unlike traditional code, LLM behaviour can change without any code change on your end — a model provider update, a temperature change, or a routing difference can all shift output quality.

---

The Feedback Loop

Evals aren't just measurement — they drive improvement:

Agent produces wrong diagnosis
        │
        ▼
Add trace to evals/traces/ + golden YAML
        │
        ▼
Run evals → confirms the failure is reproducible
        │
        ▼
Add dead-end trigger or update skill file
        │
        ▼
Run evals → confirms fix doesn't regress other cases
        │
        ▼
Update baseline

Every wrong answer becomes a test case. The agent gets smarter because its failure modes are permanently captured and guarded against.

---

Starting Small: You Don't Need 50 Cases

Start with 5 traces where you already know the answer. That's enough to:

• Catch "the agent suggests waitForTimeout" regressions (anti-pattern check)

• Verify the agent still finds the right file (fix precision)

• Detect if a prompt change broke phase compliance

Add one new case every time the agent gets something wrong. After a month, you'll have 10–15 cases covering your most common failure patterns. That's a robust eval suite.

# The whole workflow:
cp ~/Downloads/failed-trace.zip evals/traces/new-pattern-001.zip
# Write the golden YAML (30 seconds of work)
vim evals/golden/new-pattern-001.yaml
# Verify
node evals/run-eval.ts new-pattern-001
# If good, update baseline
node evals/score-report.ts --update-baseline

---

What's Next

This completes the series. From prototype to reasoning chain to security hardening to CI integration to evals — the agent is now a production tool with:

• Structured methodology that prevents lazy conclusions

• Defense-in-depth against its own untrusted input

• Automatic CI integration that delivers reports without human intervention

• A measurable quality bar that prevents regressions

The eval suite is what turns it from "a cool experiment" into "a reliable tool the team trusts."