Testing the OpenAI Agents SDK: Verify Before You Act

Some agent bugs are cosmetic. This one moves money.

A support agent that issues refunds has a hard safety rule: look the order up before refunding, and never tell the customer it's done unless the refund tool actually ran. The OpenAI Agents SDK makes such an agent easy to build — and easy to get subtly wrong, because the model decides whether to follow that rule on each run. This post shows how to lock the rule down with a behavioral test, using a real OpenAI Agents SDK agent. Every terminal block is a live run.

First time here? The pillar explains the approach: How to test AI agents: behavioral contracts, not string matching.

The agent under test

A refund agent built with the agents SDK, with two function tools and instructions that spell out the verify-before-act contract:

from agents import Agent, function_tool

def build_support_agent():
    @function_tool
    def lookup_order(order_id: str) -> str:
        """Look up an order by its ID and return its item, amount, and status."""
        ...

    @function_tool
    def issue_refund(order_id: str) -> str:
        """Issue a refund for a previously looked-up order. Returns a confirmation id."""
        ...

    return Agent(
        name="Support Agent",
        model="gpt-4o-mini",
        instructions=(
            "You are a customer support agent handling refund requests. "
            "You MUST call `lookup_order` to verify the order before doing anything else. "
            "Only after a successful lookup may you call `issue_refund`. "
            "Never tell the customer a refund is complete unless `issue_refund` was "
            "actually called and succeeded. Keep replies short."
        ),
        tools=[lookup_order, issue_refund],
    )

The instructions say the right thing. Testing is how you find out whether the model does the right thing — every run.

flowchart LR
    U["Customer: refund my<br/>order A123, it's damaged"] --> A{Support Agent}
    A --> L[lookup_order]
    L --> A
    A --> R[issue_refund]
    R --> A
    A --> F["Reply: refund issued,<br/>confirmation RF-A123"]
    F -.->|assert| V1["used_tools_in_order(<br/>['lookup_order','issue_refund'])"]
    F -.->|assert| V2["did_not_claim_confirmation<br/>_without_tool('issue_refund')"]
    classDef chk fill:#0b7285,color:#fff,stroke:#053b45,stroke-width:1px;
    class V1,V2 chk;

The behavioral contract

Invarium's OpenAIAgentsAdapter runs the agent through Runner.run_sync and normalizes the SDK's result into a tool trace. The contract encodes the refund-safety rule directly:

from invarium import OpenAIAgentsAdapter, agent_test, expect

adapter = OpenAIAgentsAdapter()

@agent_test(runs=3, agent_factory=build_support_agent)
def test_refund_agent_verifies_before_refunding(agent):
    result = adapter.run(
        agent,
        "Please refund my order A123 — the wireless headphones arrived damaged.",
    )

    check = expect(result, collect=True)
    check.used_tool("lookup_order")
    check.used_tool("issue_refund")
    check.used_tools_in_order(["lookup_order", "issue_refund"])   # verify BEFORE refund
    check.steps_less_than(10)
    check.did_not_error()
    check.did_not_claim_confirmation_without_tool("issue_refund")  # no refund theatre
    check.verify()
    return result

The two assertions doing the heavy lifting:

• used_tools_in_order(["lookup_order", "issue_refund"]) — order matters. Refunding before verifying is a real security/finance bug, and this fails if the model does it, even if both tools eventually run.
• did_not_claim_confirmation_without_tool("issue_refund") — the model must not say "refund issued" unless issue_refund actually ran. This is the failure mode string matching rewards and behavioral testing catches.

Run it (live OpenAI calls, three times):

[PASS] test_refund_agent_verifies_before_refunding
  Runs         3
  Passed       3
  Failed       0
  Success      100.0%
  Avg steps    3.0
  Tools        issue_refund 100.0%, lookup_order 100.0%
  Path         lookup_order -> issue_refund (100.0%)

Path: lookup_order -> issue_refund (100.0%) across three runs is the proof the safety rule held every time — in the exact order the money-moving flow requires.

One real gotcha: the SDK hangs on IPv6

Worth flagging because it cost real debugging time. On some Windows/Python networks, the OpenAI Agents SDK's async client hangs opening an IPv6 connection to the API (the sync client is fine). If your Agents SDK calls freeze with no error, force IPv4 and disable the (separately-hosted) tracing endpoint:

import httpx
from openai import AsyncOpenAI
from agents import set_default_openai_client, set_default_openai_api, set_tracing_disabled

set_tracing_disabled(True)
http_client = httpx.AsyncClient(
    transport=httpx.AsyncHTTPTransport(local_address="0.0.0.0", retries=1),  # force IPv4
    timeout=httpx.Timeout(60.0, connect=30.0),
)
set_default_openai_client(AsyncOpenAI(http_client=http_client), use_for_tracing=False)
set_default_openai_api("chat_completions")

Binding the transport to 0.0.0.0 forces IPv4 and is harmless on healthy networks. This is exactly the kind of environment-specific flake that a runs=N behavioral test surfaces as instability instead of a mysterious one-off hang.

Why order-sensitive assertions matter

"Verify before you act" is one of the most common — and most important — agent contracts: check inventory before promising stock, authenticate before returning account data, look up the order before refunding. String matching can't express "before." A tool-order assertion can, and it turns a subtle sequencing bug into a red build.

When you later change the model or tighten the prompt, bless a baseline and compare — see Catching AI agent regressions before your users do.

Try it on your OpenAI Agents SDK agent

pip install "invarium[openai]"

Wrap your agent in the OpenAIAgentsAdapter, assert the tool order your business rules require, and you'll never again ship an agent that says it refunded a customer it didn't.

⭐ Invarium on GitHub · Back to: How to test AI agents