How to Test AI Agents (Without Brittle String-Match Tests)

Q: What's the difference between testing and evaluating an AI agent?

Evaluation scores output quality (is the answer good?). Testing verifies behavior against a contract (did it call the right tools, in order, without faking success?). Evals tell you something changed; behavioral tests tell you what, and fail the build deterministically.

Q: How do you test a non-deterministic agent?

Run each test multiple times (runs=N) and assert on the parts that are stable — the tool trace and success claims — not the wording. A flakiness score across runs surfaces unstable tool paths that a single run would hide.

Your agent passed every test in CI. Then it told a customer "Your table is booked!" — and never called the booking tool.

If you build AI agents, you already know the gap between "the tests are green" and "the agent actually did the right thing." This post is about closing that gap. Not with more assert "success" in output checks — those are exactly what let the bug through — but with behavioral contracts: assertions on what the agent did, not what it said.

By the end you'll know why string-matching tests fail on agents, what's actually worth asserting, and how to catch a regression the moment it lands. Every example here is a real run, copy-pasted from the terminal.

Why you can't test an AI agent like a normal function

A pure function is a contract: same input, same output. You assert on the output and move on.

An LLM agent breaks that contract on every axis:

The wording changes every run. "Booked!" / "You're all set." / "Reservation confirmed." — all correct, all different strings.
The tool path changes. Same prompt, and one run calls search → book, the next calls book directly, the next loops search three times.
Cost and latency drift as models and prompts change underneath you.

So the two obvious approaches both fail:

# 1) String matching — passes even when the agent lied
assert "booked" in response.lower()   # ✅ green, ❌ no table was reserved

# 2) Snapshot the whole output — breaks on every harmless reword
assert response == "Booked your table at Bombay Canteen for 2 tonight."

The first is too loose (it rewards the failure mode), the second is too tight (it fails on paraphrase). And eval scores — "response quality: 0.82" — tell you that something moved, not what moved or whether it's a real regression.

flowchart TB
    subgraph SM["❌ String-match test"]
      A1[Agent output text] --> A2{"'booked' in text?"}
      A2 -->|yes| A3[PASS]
      A2 -.->|but book_table<br/>was never called| A4[Ships a broken agent]
    end
    subgraph BC["✅ Behavioral contract"]
      B1[Agent trace] --> B2{Called book_table?}
      B2 -->|no| B3[FAIL: missing tool]
      B2 -->|yes| B4{Claimed success<br/>without the tool?}
      B4 -->|yes| B5[FAIL: unsupported claim]
      B4 -->|no| B6[PASS]
    end

The shift: assert on behavior, not text

An agent leaves a trace — the sequence of tool calls, their arguments and results, the step count, the final answer. That trace is deterministic enough to test, even when the wording isn't. So instead of asserting on the reply, you assert on the trace:

Did it call the tool it was supposed to?
In the right order?
Within a sane step budget (no runaway loops)?
Did it claim success without the evidence to back it up?

This is the idea behind Invarium — pytest for AI agents. You write assertions on observable behavior, run each test several times (because agents are flaky), and it flags regressions against a saved baseline. It's an open-source Python package (pip install invarium) that works with LangGraph, the OpenAI Agents SDK, CrewAI, plain Python callables, or any HTTP endpoint.

The rest of this post uses it to make the intro's bug impossible to ship.

A behavioral test, start to finish

Here's a small LangGraph booking agent. The "good" version checks availability, books the table, then confirms. We won't assert on its wording at all — only on its behavior:

from invarium import LangGraphAdapter, agent_test, expect

adapter = LangGraphAdapter()

@agent_test(runs=5, agent_factory=build_graph)
def test_booking_agent(graph):
    result = adapter.run(graph, "Book a table for 2 tonight")

    check = expect(result, collect=True)
    check.used_tool("search_restaurants")
    check.used_tool("book_table")
    check.used_tools_in_order(["search_restaurants", "book_table"])
    check.steps_less_than(8)
    check.did_not_error()
    check.did_not_claim_confirmation_without_tool("book_table")
    check.verify()
    return result

runs=5 matters: a single green run doesn't prove a non-deterministic agent is stable. Run it:

[PASS] test_booking_agent
  Runs         5
  Passed       5
  Failed       0
  Success      100.0%
  Avg steps    5.0
  Tools        book_table 100.0%, search_restaurants 100.0%
  Path         search_restaurants -> book_table (100.0%)

Now imagine a routine change — a model upgrade, a prompt tweak — and the new agent gets chatty and confident: it skips book_table but still says "Your reservation is confirmed!". A string-match test on "confirmed" stays green. The behavioral contract does not:

[FAIL] test_booking_agent
  Runs         5
  Passed       0
  Failed       5
  Success      0.0%
  Tools        search_restaurants 100.0%
  Path         search_restaurants (100.0%)
  Fail cats    missing_required_tool:5, wrong_tool_order:5, unsupported_success_claim:5
  Failures
    - Expected tool `book_table` to be called, but saw ['search_restaurants']. (5/5 runs)
    - Expected tools in order ['search_restaurants', 'book_table'], but saw ['search_restaurants']. (5/5 runs)
    - Agent claimed success in final output without evidence from book_table. (5/5 runs)

That last line — "claimed success in final output without evidence" — is the whole point. The agent said it worked; the trace proves it didn't. That's the failure string matching can never catch.

The four things worth asserting on almost any agent

You don't need a huge assertion vocabulary. Ninety percent of real agent bugs fall into four buckets:

Tool usage — used_tool("book_table"). The single highest-value assertion: did the agent take the real-world action, or just talk about it?
Ordering — used_tools_in_order([...]). Verify-before-act contracts: look the order up before refunding, search before answering a factual question.
Step budget — steps_less_than(8). Catches runaway loops that quietly 10× your token bill.
Honest success — did_not_claim_confirmation_without_tool(...). Catches the agent that reports success it didn't earn.

Assert on those and you've covered the failure modes that actually page you at 2am.

Then catch drift automatically with a baseline

One green run is a test. The real leverage is over time. You bless a known-good run as a baseline, and every future run is compared against it:

invarium bless framework_examples/langgraph_booking_regression   # save the good baseline
invarium test  framework_examples/langgraph_booking_regression   # compare future runs

When the regression above lands, Invarium doesn't just fail — it tells you exactly what changed:

[REGRESSION] test_booking_agent
  Success      100.0% -> 0.0%
  Step delta   -2.0
  Tool drop    book_table 100.0% -> 0.0%
  Failure cats missing_required_tool:5, wrong_tool_order:5, unsupported_success_claim:5

Tool drop: book_table 100.0% -> 0.0% is a code-review comment writing itself. See Catching AI agent regressions before your users do for the full baseline workflow.

Testing your specific framework

The pattern is identical everywhere — build the agent, run it through an adapter, assert on behavior — but the adapter and the tool names change. Framework-specific walkthroughs, each with real live runs:

How to test a LangGraph agent so it actually uses its tools — a real ReAct agent with web search + a calculator, asserting it grounds factual answers instead of hallucinating.
Testing the OpenAI Agents SDK: verify before you act — a refund agent that must look up the order before issuing money back.
Testing a CrewAI crew — and the bug it caught in our own adapter — a multi-agent crew, plus the real serialization bug this surfaced.
Catching AI agent regressions before your users do — baselines, drift detection, and an honest note on when regressions don't appear.

FAQ

What's the difference between testing and evaluating an AI agent? Evaluation scores output quality (is the answer good?). Testing verifies behavior against a contract (did it call the right tools, in order, without faking success?). Evals tell you something changed; behavioral tests tell you what, and fail the build deterministically.

How do you test a non-deterministic agent? Run each test multiple times (runs=N) and assert on the parts that are stable — the tool trace and success claims — not the wording. A flakiness score across runs surfaces unstable tool paths that a single run would hide.

Can I test agents without an LLM API key in CI? Yes. Assert on the framework's message/trace shape with a deterministic stand-in (like the booking example), so the behavioral contract runs free and reproducibly in CI, while live-model tests run on demand with secrets.

Does this replace my evals? No — it complements them. Evals guard answer quality; behavioral contracts guard that the agent does the work it claims. You want both.

Start testing your agent's behavior

pip install invarium

Invarium is open source (MIT) — assert on tool usage, ordering, step budgets, and honest success claims, then catch regressions against a baseline. If it saves you one "it said it worked but it didn't" incident, it's paid for itself.

⭐ Invarium on GitHub · pip install invarium

Have an agent that passes its tests but still misbehaves in production? That's exactly the gap behavioral contracts close — try one of the framework guides above.

How to Test AI Agents: Behavioral Contracts, Not String Matching