Part 2: Teaching an AI Agent to Reason — Skills, Phases & Dead-End Triggers

The Problem With Unconstrained LLMs

In Part 1, I built an agent that extracts trace data and reasons about test failures. It worked — sometimes. Other times it produced reports like:

"The test timed out waiting for the element to appear. This is likely a timing issue. Consider increasing the timeout or adding a waitForTimeout()."

This is the LLM equivalent of a doctor saying "you're sick because you have symptoms." It describes the failure without explaining the cause. And the suggested fix — waitForTimeout() — makes the test slower and still flaky.

The problem isn't intelligence. The LLM can find the root cause. The problem is that without constraints, it takes shortcuts. It pattern-matches "timeout → timing issue" instead of investigating further. It needs a structured methodology that forces it through the full diagnostic process.

---

The 7-Phase Reasoning Chain

I encoded our team's debugging methodology as a skill file — a structured document that defines exactly how the agent must reason, in what order, and what evidence it must gather at each step.

If you're familiar with prompt engineering, this is a form of Chain-of-Thought (CoT) — but constrained. Vanilla CoT ("let's think step by step") lets the model reason freely, which means it can skip evidence, jump to conclusions, or fixate on the wrong signal. The 7-phase chain forces a specific investigation order, requires evidence at each gate, and uses dead-end triggers to block known-bad reasoning paths. Think of it as CoT with guardrails — the model reasons freely within each phase, but can't skip ahead or conclude prematurely.

	Vanilla CoT	7-Phase Reasoning Chain
Structure	"Let's think step by step" (freeform)	Mandatory sequential phases with defined outputs
Direction	Model decides what to think about	Phases dictate what to investigate and in what order
Guardrails	None — model can jump to conclusions	Dead-End Triggers block known-bad inferences
Evidence	Optional	Mandatory — must cite [file]:[line]
Termination	Model decides when it's done	Can't produce a report until all phases complete

Phase 1: Data Acquisition

Run both analysis scripts. Catalogue what they found:

• Was there an error event? What's the message?

• What does the page snapshot show at failure time?

• Did any network requests fail? Any GraphQL errors in 200 responses?

This is mechanical — just gathering evidence before reasoning begins.

Phase 2: What Did the Test Expect?

Open the test file at the line that failed (from the stack trace). Quote the exact assertion:

await expect(page.getByRole('table', { name: 'Statements' })).toBeVisible();

This establishes the "expected" half of the equation.

Phase 3: What Was the Page Showing?

Read the accessibility snapshot captured at failure time. What elements are actually present? Is there a loading spinner? An error message? An empty state? Nothing at all?

This establishes the "actual" half.

Phase 4: Did Network Calls Succeed?

This is the critical gate. Check not just HTTP status codes, but response bodies. A 200 with { errors: [...] } is a failure. A 200 with { data: null } might be a failure depending on the component's logic.

Phase 5: What App Code Path Was Triggered?

Open the component that renders the expected element. Trace the logic: under what conditions does it render vs not render? Compare those conditions to what the trace actually showed (the API response, the component props, the feature flags).

Phase 6: Single Root Cause Statement

One sentence in the form: "The test expected X, but got Y, because Z."

The test expected getByRole('table') to be visible, but the component
renders null when data is undefined, because the getInformation query
returned a GraphQL error (permission denied) that the mock didn't intercept.

Phase 7: Minimal Fix

The smallest change that resolves the conflict. Not a refactor. Not a "consider also doing..." suggestion. One specific change with a file path and line number.

---

Why Sequential Phases Matter

The order is intentional. Each phase builds on the previous:

• You can't reason about "why the component didn't render" (Phase 5) until you know "what was the API response" (Phase 4)

• You can't assess "did the network fail" (Phase 4) until you know "what element was expected" (Phase 2) — because that tells you which API the component depends on

• You can't skip to a fix (Phase 7) without the root cause statement (Phase 6)

Forcing the LLM through this sequence prevents premature conclusions. It can't say "timing issue" at Phase 6 if Phase 4 already established that the API returned an error.

---

Dead-End Triggers: The Anti-Misdiagnosis System

Even with phases, the agent can still reach wrong conclusions if it misinterprets evidence. Dead-End Triggers are pre-programmed rules that fire when the agent is about to make a known-bad inference:

DE-1: "Timing Issue" Without Network Failure

If the agent is about to conclude "timing issue" or "race condition" but Phase 4 showed no network failures and no slow responses → STOP. The element isn't slow; it's not going to appear. Re-examine why the component's render condition isn't met.

DE-2: Status Code Tunnel Vision

If the agent concludes "API succeeded" based solely on HTTP 200 → STOP. Check the response body for GraphQL errors array. Re-run Phase 4.

DE-3: Missing Element ≠ Slow Element

If the page snapshot shows the element simply isn't in the DOM (not loading, not hidden — absent) AND no network failure occurred → STOP. The component never attempted to render this element. Check conditional rendering logic, feature flags, or permission checks.

DE-4: Mock Mismatch

If the test uses page.route() to mock an API, but the snapshot shows unexpected data (or no data) → STOP. Check if the mock's URL pattern / operationName actually matches what the app sends. A mock that doesn't match is invisible — the request falls through to the real endpoint.

DE-5: "Add a Wait" Fix

If the proposed fix involves waitForTimeout(), setTimeout, or increasing a timeout → REJECT. This is never the correct fix for a Playwright test. Find the actual cause and fix the condition, not the symptom.

---

How Dead-End Triggers Work in Practice

Here's a real example. Without DE-1, the agent would produce:

"The element didn't appear within 30 seconds. This suggests a timing issue with the component rendering. The test should wait for the API response to complete before asserting."

With DE-1, the agent hits the trigger at Phase 6:

• Phase 4 found: all network requests returned 200, no errors in bodies

• Phase 3 found: element is not in the DOM at all

• DE-1 fires: "timing issue" conclusion blocked

The agent is forced to continue to Phase 5, where it opens the component code:

if (!featureFlags.showStatements) return null;

Root cause: the feature flag showStatements is false in the test environment. The element will never appear regardless of how long you wait.

---

Separation of Concerns: Prompt vs Skill vs Steering

The reasoning system is split across multiple files, each with a distinct role:

File	Contains	Changes When
System prompt	Identity, security rules	Rarely (security updates)
Skill file	The 7-phase procedure + dead-end triggers	When methodology improves
Steering file	Hard "never do this" constraints	When new anti-patterns emerge
Environment context	CI runtime facts (what's mocked, URLs)	When infra changes

This separation means:

• Adding a new dead-end trigger doesn't touch security config

• Updating environment URLs doesn't touch the reasoning procedure

• Each file can be reviewed and evolved by the person who owns that concern

---

Evidence-Based Reasoning: The Hallucination Guard

Every conclusion the agent makes must cite [file]:[line]. This is enforced in the skill file:

"You must quote the specific file and line number for every factual claim. If you cannot find the code that supports a conclusion, the conclusion is speculative — say so."

This prevents the agent from:

• "The feature flag is probably off" → Must find the actual flag check in code

• "The component likely renders null" → Must show the return statement

• "The mock probably doesn't match" → Must show the mock's route pattern vs the actual request

If the agent can't find evidence, it says "I cannot determine the root cause from the available evidence" rather than guessing. An honest "I don't know" is more useful than a confident wrong answer.

---

Results

With structured reasoning:

• Misdiagnosis rate dropped significantly — "timing issue" conclusions went from ~40% to near-zero

• Reports became actionable — every report now points to a specific file:line to change

• New team members learn from the agent — reading RCA reports teaches the debugging methodology itself

---

What's Next

The reasoning system works, but the agent processes untrusted data. Trace files could contain anything — including text designed to trick the LLM into executing commands. In Part 3, I'll cover how I secured the agent against prompt injection from its own input.