# Part 4: Building Evals - How Do You Know Your Agent Is Getting Smarter?

## The Question Nobody Asks Early Enough

You've built an agent. It produces impressive-looking reports. But how do you know:

*   It's actually getting the right answer?
    
*   A change to the prompt didn't silently break it?
    
*   The underlying model update didn't introduce a regression?
    
*   Your new dead-end trigger is actually firing?
    

Without evals, you're flying blind. Every change to the agent is a guess — you push it, hope it still works, and find out the hard way when a developer says "the agent told me to add a `waitForTimeout`."

* * *

## What Makes Agent Evals Different From Unit Tests

Unit tests verify deterministic code: given input X, expect output Y. Agent evals are different:

|  | Unit Tests | Agent Evals |
| --- | --- | --- |
| Output | Deterministic | Non-deterministic (LLM varies) |
| Correctness | Binary (pass/fail) | Graded (0–2 per dimension) |
| Verification | Exact match | Semantic similarity + anti-pattern checks |
| Speed | Milliseconds | 30–120 seconds per case |
| Frequency | Every commit | On agent changes + weekly |

You can't `assertEqual(output, expected)` because the agent might phrase the same correct diagnosis differently each run. Instead, you check:

1.  Did it identify the right root cause? (semantic match)
    
2.  Did it point to the right file? (exact match)
    
3.  Did it cite evidence? (pattern count)
    
4.  Did it avoid known-bad conclusions? (anti-pattern check)
    
5.  Did it follow the reasoning methodology? (structural check)
    

* * *

## The Golden Dataset: Ground Truth You Already Have

The hardest part of agent evals is getting ground truth. For the trace-rca agent, ground truth is surprisingly easy: **every trace you've already manually debugged is a potential eval case.**

When you debug a test failure and find the root cause, you have:

*   The trace `.zip` file (input)
    
*   The root cause you discovered (expected output)
    
*   The file you fixed (expected fix location)
    
*   What the answer is NOT (anti-patterns)
    

Turn that into a YAML file:

```yaml
trace: evals/traces/mock-mismatch-001.zip
category: mock-mismatch

expected_root_cause: "mock operationName doesn't match the actual GraphQL query sent by the app"
expected_fix_file: "e2e/specs/tickets/ticket-list.spec.ts"
expected_fix_type: "update mock operationName to match actual query"

must_not_conclude:
  - "timing issue"
  - "increase timeout"
  - "waitForTimeout"
  - "add a wait"
  - "race condition"

dead_end_triggers_expected:
  - "DE-2"
  - "DE-4"
```

The `must_not_conclude` field is the most valuable part. It encodes "the agent should NEVER say this for this trace." This catches the most dangerous regressions — where the agent produces a confidently wrong answer.

* * *

## The Five Scoring Dimensions

Each eval case is scored on five axes (0–2 each, max 10 total):

| Dimension | What it measures | 0 | 1 | 2 |
| --- | --- | --- | --- | --- |
| **Root Cause** | Did it find the actual problem? | Wrong | Partially right | Exact |
| **Fix Precision** | Does the fix point to the right place? | Wrong file | Right area | Exact file:line |
| **Evidence** | Are claims backed by code references? | No citations | Some | Every claim cited |
| **No Anti-Pattern** | Did it avoid bad conclusions? | Suggests `waitForTimeout` | — | Clean |
| **Phase Compliance** | Did it follow the reasoning chain? | Skipped phases | Thin | All substantive |

Not all dimensions need human review:

```plaintext
Fully automatable:     no_antipattern, fix_actionable, evidence_cited
Heuristic + human:     root_cause_correct, phase_compliance
```

The automated checks catch ~80% of regressions without any human effort. The remaining 20% (semantic root cause accuracy) gets flagged for review.

* * *

## The Eval Runner

The runner is a zero-dependency TypeScript script that:

1.  Loads golden cases from YAML files
    
2.  Invokes the agent on each trace
    
3.  Auto-scores the output
    
4.  Compares against the baseline
    
5.  Reports pass/fail/review for each case
    

```bash
node evals/run-eval.ts
```

Output:

```plaintext
═══════════════════════════════════════════════════════════════════════════
  TRACE-RCA AGENT EVALUATION
═══════════════════════════════════════════════════════════════════════════

Found 5 golden case(s) to evaluate.

Running: mock-mismatch-001 ... ✅ PASS (9/10) [47.2s]
Running: graphql-error-001 ... ✅ PASS (10/10) [38.6s]
Running: feature-flag-001 ... ⚠️  REVIEW (6/10) [52.1s]
Running: null-data-001 ... ✅ PASS (8/10) [41.3s]
Running: infra-failure-001 ... ✅ PASS (8/10) [22.7s]

────────────────────────────────────────────────────────────────────────
SUMMARY
────────────────────────────────────────────────────────────────────────
  Cases:     5
  Passed:    4
  Errors:    0
  Score:     41/50 (82%)
  Avg time:  40.4s
```

Cases marked ⚠️ REVIEW need a human to check whether the auto-scorer missed a correct-but-differently-phrased answer, or whether the agent genuinely got it wrong.

* * *

## Anti-Pattern Scoring: The Most Valuable Check

The `no_antipattern` dimension is binary and fully automatable — it string-matches the agent's output against known-bad conclusions:

```typescript
const hasAntipattern = golden.must_not_conclude.some((bad) =>
  output.toLowerCase().includes(bad.toLowerCase()),
);
scores.no_antipattern = hasAntipattern ? 0 : 2;
```

This single check catches the most dangerous regression: the agent producing a **confidently wrong answer**. A hesitant "I cannot determine the root cause" is fine. "This is a timing issue, add `waitForTimeout(3000)`" is catastrophic — it leads developers to make the test worse.

Anti-patterns I test for:

```yaml
must_not_conclude:
  - "timing issue"          # almost never the real cause
  - "increase timeout"      # symptom treatment, not a fix
  - "waitForTimeout"        # explicitly prohibited in our codebase
  - "add a wait"            # same as above, different phrasing
  - "race condition"        # often used as a hand-wavy non-answer
  - "API succeeded"         # dangerous when GraphQL errors hide in 200s
  - "network call succeeded" # same as above
```

* * *

## Regression Detection: The Baseline

After running evals and verifying the results are correct, save them as the baseline:

```bash
node evals/score-report.ts --update-baseline
```

This writes `baseline.json`:

```json
{
  "total_score": 41,
  "max_score": 50,
  "case_count": 5,
  "updated_at": "2026-07-04T08:00:00.000Z",
  "per_case": [
    { "name": "mock-mismatch-001", "score": 9, "max": 10 },
    { "name": "graphql-error-001", "score": 10, "max": 10 },
    ...
  ]
}
```

On subsequent runs, if the overall score drops more than 10%, the runner exits with code 1:

```bash
────────────────────────────────────────────────────────────────────────
REGRESSION CHECK
────────────────────────────────────────────────────────────────────────
  Baseline:  82%
  Current:   68%
  Delta:     -14%
  ⚠️  REGRESSION DETECTED — score dropped >10% from baseline
```

This makes it CI-gate-able:

```bash
node evals/run-eval.ts || echo "⚠️ Agent regression — do not merge"
```

* * *

## Failure Categories: Building a Diverse Dataset

A good eval suite covers the space of failures your agent encounters:

| Category | What it tests | Example failure |
| --- | --- | --- |
| `mock-mismatch` | Agent traces mock setup vs actual request | operationName typo in route handler |
| `graphql-error-in-200` | Agent looks beyond HTTP status codes | FORBIDDEN error buried in response body |
| `feature-flag` | Agent checks conditional rendering logic | Component returns null when flag is off |
| `null-data` | Agent distinguishes "no error" from "no data" | Query succeeds but returns `{ data: null }` |
| `infrastructure` | Agent knows when NOT to diagnose | Browser crash, partial trace |

Each category tests a different aspect of the agent's reasoning. A regression in `mock-mismatch` cases means something changed in how the agent reads route handlers. A regression in `infrastructure` means the agent lost its ability to say "I can't diagnose this."

* * *

## The Score Report: Drill Down Into Weaknesses

```bash
node evals/score-report.ts
```

Produces a per-dimension breakdown:

```plaintext
═══════════════════════════════════════════════════════════════════════════
  TRACE-RCA EVAL SCORE REPORT
═══════════════════════════════════════════════════════════════════════════

  Overall Score: 41/50 (82%)  ████████░░

────────────────────────────────────────────────────────────────────────
  SCORES BY DIMENSION
────────────────────────────────────────────────────────────────────────

  Root Cause            8/10 ( 80%)  ████████░░
  Fix Precision         9/10 ( 90%)  █████████░
  Evidence              8/10 ( 80%)  ████████░░
  No Anti-Pattern      10/10 (100%)  ██████████
  Phase Compliance      6/10 ( 60%)  ██████░░░░

────────────────────────────────────────────────────────────────────────
  SCORES BY CATEGORY
────────────────────────────────────────────────────────────────────────

  mock-mismatch          1 cases   9/10 ( 90%)  █████████░
  graphql-error-in-200   1 cases  10/10 (100%)  ██████████
  feature-flag           1 cases   6/10 ( 60%)  ██████░░░░
  null-data              1 cases   8/10 ( 80%)  ████████░░
  infrastructure         1 cases   8/10 ( 80%)  ████████░░
```

This tells you exactly where to focus. In this example, "Phase Compliance" is the weakest dimension — the agent is sometimes skipping or rushing through reasoning phases. And `feature-flag` cases are the weakest category — the agent struggles with conditional rendering logic.

Both are actionable: strengthen the skill file's phase enforcement, and add a dead-end trigger for feature flag patterns.

* * *

## When to Run Evals

| Trigger | Why |
| --- | --- |
| Changed agent config, prompt, or skill file | Direct impact on agent behaviour |
| Changed analysis scripts | Output format affects how agent reasons |
| Weekly (scheduled) | Model drift — provider updates can change behaviour |
| Before merging agent PRs | Gate to prevent regressions |

The weekly run is important. Unlike traditional code, LLM behaviour can change without any code change on your end — a model provider update, a temperature change, or a routing difference can all shift output quality.

* * *

## The Feedback Loop

Evals aren't just measurement — they drive improvement:

```plaintext
Agent produces wrong diagnosis
        │
        ▼
Add trace to evals/traces/ + golden YAML
        │
        ▼
Run evals → confirms the failure is reproducible
        │
        ▼
Add dead-end trigger or update skill file
        │
        ▼
Run evals → confirms fix doesn't regress other cases
        │
        ▼
Update baseline
```

Every wrong answer becomes a test case. The agent gets smarter because its failure modes are permanently captured and guarded against.

* * *

## Starting Small: You Don't Need 50 Cases

Start with 5 traces where you already know the answer. That's enough to:

*   Catch "the agent suggests `waitForTimeout`" regressions (anti-pattern check)
    
*   Verify the agent still finds the right file (fix precision)
    
*   Detect if a prompt change broke phase compliance
    

Add one new case every time the agent gets something wrong. After a month, you'll have 10–15 cases covering your most common failure patterns. That's a robust eval suite.

```bash
# The whole workflow:
cp ~/Downloads/failed-trace.zip evals/traces/new-pattern-001.zip
# Write the golden YAML (30 seconds of work)
vim evals/golden/new-pattern-001.yaml
# Verify
node evals/run-eval.ts new-pattern-001
# If good, update baseline
node evals/score-report.ts --update-baseline
```

* * *

## What's Next

This completes the series. From prototype to reasoning chain to security hardening to CI integration to evals — the agent is now a production tool with:

*   Structured methodology that prevents lazy conclusions
    
*   Defense-in-depth against its own untrusted input
    
*   Automatic CI integration that delivers reports without human intervention
    
*   A measurable quality bar that prevents regressions
    

The eval suite is what turns it from "a cool experiment" into "a reliable tool the team trusts."