Part 1: Why I Built an AI Agent to Debug My Playwright Tests

The Problem

If you've worked on a large-scale e2e test suite, you know the pain. A test fails in CI. The error message says:

Timed out 30000ms waiting for expect(locator).toBeVisible()

Great. What timed out is obvious. Why it timed out is the question that eats 30–60 minutes of your day.

The debugging ritual looks like this:

1. Download the trace .zip from CI artifacts

2. Open it in the Playwright trace viewer

3. Click through the action timeline to find the failure point

4. Check the page snapshot — what was actually rendered?

5. Check network requests — did the API return what the app expected?

6. Open the test code — what exactly was the assertion?

7. Open the app code — why didn't the component render?

Steps 1–4 are mechanical. Steps 5–7 require reasoning. And every failed test requires all seven.

Our suite has hundreds of Playwright tests across acceptance, functional, and non-functional categories. When multiple tests fail in a single CI run, the debugging backlog compounds fast. I needed something that could do the investigative grunt work so developers could skip straight to the fix.

---

Why an AI Agent, Not Just a Script

A script can extract data from a trace. I could write a parser that dumps "here's the error, here's the snapshot, here are the failed network requests." But that's just step 1–4 automated. The hard part — the reasoning — is what takes the time.

Root-cause analysis requires correlating multiple signals:

• "The test expected a table to be visible, but the snapshot shows a loading spinner. The network tab shows the GraphQL query returned 200... but wait, the response body has { errors: [{ message: 'unauthorized' }] }. The component renders nothing when the data query has errors. That's why the table never appeared."

That chain of reasoning — page state × network response × app rendering logic — is where an LLM shines. It can read the snapshot, read the API response, read the component code, and connect the dots.

But only if you give it the right structure and constraints. An unconstrained LLM will often conclude "it's a timing issue, add a waitForTimeout()" — which is almost always wrong and makes the test slower and still flaky.

---

The First Prototype

I built the agent using Kiro CLI, which lets you define workspace-local agents with a JSON config, a system prompt, tool access, and supporting files. The agent lives in the repo alongside the tests — committed and version-controlled like any other dev tool.

The first commit introduced three pieces:

1. An analysis script — Initially Python (for speed of prototyping), later ported to TypeScript. Parses Playwright's NDJSON trace format and extracts:

• Test title and metadata

• The terminal error with full stack trace

• Action timeline (marking the exact failure point)

• Page accessibility snapshot at failure time

• Console errors

• Failed network requests

2. An agent config — Defines which model to use, which tools the agent has access to, and crucially, a shell command allowlist restricting what it can execute.

3. A system prompt — The RCA methodology: what to look for, how to reason, and what conclusions to avoid.

---

The TypeScript Decision

The Python prototype worked fine, but our repo is TypeScript. The team shouldn't need a Python runtime for a dev tool.

I ported the analyzer to TypeScript with zero npm dependencies — only Node.js built-ins (fs, path, child_process). It shells out to unzip for extraction and parses JSON lines directly.

Why zero dependencies?

• No install step. Works immediately after git clone.

• No version conflicts. Doesn't pollute the project's dependency tree.

• Runs everywhere. CI containers, developer MacBooks, automated pipelines.

• Smaller attack surface. No supply-chain risk from transitive dependencies.

Node's stdlib is perfectly adequate for "read files, parse JSON, format text."

---

The Breakthrough: "200 ≠ Success"

This was the moment the agent went from "occasionally useful" to "consistently correct."

The agent kept concluding "the API call succeeded" because the network tab showed HTTP 200. But in GraphQL, everything returns 200. Errors come back as { "errors": [...] } inside the response body.

A test might fail because:

• The query returned { data: null, errors: [{ message: "Forbidden" }] }

• The component renders nothing when data is null

• The test expects a visible element that will never appear

But if you only check the status code, it looks like "the API worked fine, must be a timing issue."

I built a dedicated graphql_network_verify.ts script that:

1. Extracts response bodies from the trace's SHA1-named resource blobs

2. Parses JSON looking for errors arrays

3. Classifies errors into categories (auth, schema mismatch, service unavailable, rate-limit, etc.)

4. Produces a clear verdict: "✓ HEALTHY" or "✗ DIVERGENCE DETECTED"

This single script eliminated the most common misdiagnosis.

---

What the Agent Produces

Given a trace, the agent outputs a structured report:

Root Cause: The test expected getByRole('table', { name: 'Recent Tickets' })
to be visible, but the TicketList component renders null when the
GetTickets query returns errors. The query failed with
"UNAUTHORIZED: invalid session token" (HTTP 200, GraphQL error in body).
The mock in the test fixture intercepts /graphql but uses operationName
"ListTickets" — the app sends "GetTickets", so the mock never matched
and the real endpoint was hit with an expired auth token.

Fix: Update the mock operationName to match the actual query:
  e2e/specs/tickets/ticket-list.spec.ts:18
  - route.fulfill({ operationName: 'ListTickets', ... })
  + route.fulfill({ operationName: 'GetTickets', ... })

What makes this report useful:

• Specific — quotes the exact locator, component, query, and error

• Evidence-backed — cites file:line for every claim

• Actionable — tells you exactly what to change and where

---

What's Next

The prototype worked, but had a problem: the LLM sometimes jumped to wrong conclusions. It needed a structured reasoning methodology with guardrails. In the next post, I'll cover how I taught the agent how to think — with phased reasoning, dead-end triggers, and mandatory evidence citations.

---

Next: Part 2: Teaching an AI Agent to Reason — Skills, Phases & Dead-End Triggers