Testing a CrewAI Crew — and the Bug It Caught in Our Own Tool

We built a tool for testing AI agents. Then we pointed it at a real CrewAI crew — and it immediately surfaced a bug in our own adapter. This is the best kind of dogfooding: the tool doing its job caught something we'd never have noticed until it broke in a user's report.

This post is two things at once: a practical guide to testing a multi-agent CrewAI crew behaviorally, and the story of the real bug that testing surfaced (with the one-method fix). Every terminal block and code snippet is real.

If you haven't seen the behavioral-testing approach yet, start with the pillar: How to test AI agents: behavioral contracts, not string matching.

The crew under test

A classic two-agent pattern lots of teams ship: a Researcher gathers facts, a Writer turns them into a summary, run as a sequential CrewAI process backed by a real OpenAI model. No mocks — this makes live model calls.

from crewai import Agent, Crew, LLM, Process, Task

def build_research_crew():
    llm = LLM(model="gpt-4o-mini", temperature=0)

    researcher = Agent(role="Senior Researcher",
                       goal="Find accurate, concise key facts about {query}", llm=llm)
    writer = Agent(role="Technical Writer",
                   goal="Turn research notes into a short, clear summary", llm=llm)

    research_task = Task(
        description="Research the topic: {query}. List three concise, factual bullet points.",
        expected_output="Three factual bullet points.", agent=researcher)
    write_task = Task(
        description="Using the research notes, write a two-sentence summary about {query}.",
        expected_output="A two-sentence summary.", agent=writer, context=[research_task])

    return Crew(agents=[researcher, writer], tasks=[research_task, write_task],
                process=Process.sequential)

flowchart LR
    Q["Input: benefits of<br/>automated testing"] --> R[Researcher agent]
    R -->|research notes| W[Writer agent]
    W --> S["Two-sentence summary"]
    S -.->|assert| V1["used_any_tool() — each task ran"]
    S -.->|assert| V2["finished_successfully()"]
    S -.->|assert| V3["steps_less_than(6)"]
    classDef chk fill:#0b7285,color:#fff,stroke:#053b45,stroke-width:1px;
    class V1,V2,V3 chk;

The behavioral contract

Invarium's CrewAIAdapter normalizes each completed crew task into a step, so the contract is about the crew running its tasks and finishing cleanly within a step budget:

from invarium import CrewAIAdapter, agent_test, expect

adapter = CrewAIAdapter()

@agent_test(runs=2, agent_factory=build_research_crew)
def test_research_crew_completes_cleanly(crew):
    result = adapter.run(crew, "the benefits of automated testing in software")

    check = expect(result, collect=True)
    check.used_any_tool()          # each completed task shows up as a step
    check.finished_successfully()
    check.did_not_error()
    check.steps_less_than(6)
    check.verify()
    return result

The bug: caught the first time we ran it for real

The very first live run didn't produce a clean pass or a clean fail. It crashed while writing the trace — a TypeError, deep inside report serialization:

TypeError: Object of type UsageMetrics is not JSON serializable

Here's what happened. Newer CrewAI (1.15.x) returns a Pydantic UsageMetrics object for a crew's token_usage. Our CrewAIAdapter was storing that object straight into the result's metadata:

# before — stores a live Pydantic object that can't be JSON-encoded
metadata={"adapter": "crewai", "usage": usage},

Every other adapter had gotten plain dicts or numbers, so this path had never been exercised — until a real crew, with real token accounting, flowed through it. The trace writer tried to json.dumps a Pydantic object and blew up. A real bug, on a real agent, that no unit test with a mock had ever hit.

The fix: one defensive method

The fix is small and framework-defensive — coerce whatever CrewAI hands us into something JSON-safe, trying the Pydantic serializers first and degrading gracefully:

def _serialize_usage(self, usage: Any) -> Any:
    """Coerce CrewAI usage metrics into a JSON-serializable form.

    Newer CrewAI returns a Pydantic ``UsageMetrics`` object for ``token_usage``,
    which is not JSON serializable and would break trace/report writing.
    """
    if usage is None or isinstance(usage, (str, int, float, bool, dict)):
        return usage
    for method in ("model_dump", "dict"):
        if hasattr(usage, method):
            try:
                return getattr(usage, method)()
            except Exception:
                pass
    try:
        return dict(usage)
    except Exception:
        return str(usage)

And the call site becomes:

metadata={"adapter": "crewai", "usage": self._serialize_usage(usage)},

All 79 unit tests still passed after the change, and the live crew ran clean:

[PASS] test_research_crew_completes_cleanly
  Runs         2
  Passed       2
  Failed       0
  Success      100.0%
  Avg steps    2.0
  Latency      16.89s
  Tools        Research the topic: the benefits of automated testing... 100.0%,
               Using the research notes, write a two-sentence summary... 100.0%
  Path         Research the topic... -> Using the research notes... (100.0%)

Both tasks ran, in order, every time — and the whole thing finished in ~17s. The Path line even reads back the crew's pipeline: research → write.

The lesson: mocks hide integration bugs; real runs surface them

This is the argument for testing agents against real frameworks, not stand-ins. A mock CrewAI result would have returned a tidy dict and this bug would have shipped, waiting for a user to hit it in production. Running a real crew through the real adapter is what caught it — the same reason you run integration tests, not just unit tests. It's also the whole philosophy behind these examples: no mocks, real models, real frameworks, real behavior.

Testing your own crew

pip install "invarium[crewai]"

Multi-agent crews are exactly where behavior is hardest to eyeball — more agents, more tasks, more places to silently skip a step. A behavioral contract that asserts the crew ran its tasks and finished within budget gives you a green/red signal on every change. And if you're building your own adapter for a framework, run it against the real thing early — that's how you find the UsageMetrics in your own code before your users do.

Next: pin a baseline and catch drift over time in Catching AI agent regressions before your users do.

⭐ Invarium on GitHub · Back to: How to test AI agents