A script that calls the agent is the easy part. A mocking proxy, a judge, fault injection, attacks, a CI gate, stored runs and reports are the rest.
For engineering leads
Build vs Buy: AI Agent Testing Platform
Your team has an agent heading to production, and someone has said "we can write a test harness for that." They can. This page lists what that harness turns into and what you would still be maintaining a year later, so you can decide with your own numbers.
- action: MOCK_TOOL_RESPONSE
tool: holiday_calendar
response: { holidays: ["2027-01-01"] }
- action: PASSTHROUGH_TOOL
tool: reserve_flight
url: "{{BASE_API_URL}}/api/reserve"
method: POSTWhat you are weighing
Four questions behind "should we build it?"
Attack techniques change, tool formats change and the judge drifts. Someone owns each of those after the first version ships.
A product owner or a reviewer will not read harness code. Without a readable view, the results stay with the one person who built it.
Then building may be right. A trial shows exactly where a product falls short before you commit.
How CertiQA helps
Part by part: what you would build, and what exists.
Seven parts that most teams end up needing once agents reach production.
| Part | Building in-house | With CertiQA |
|---|---|---|
| Tool interception and mocking | A proxy between the agent and its tools that handles REST and MCP calls, validates arguments and returns controlled responses. | Intercepts REST and MCP tool calls, validates arguments and returns the responses you define. A tool with no mock is never sent anywhere; a real call needs an explicit step. |
| Fault injection | A way to make tools time out, return HTTP 429 or 500, or send malformed payloads, and assertions on how the agent recovers. | Timeouts, 429 and 500 responses and malformed payloads can be added to a tool call, with assertions on how the agent copes. |
| Judge and rubrics | An LLM judge, rubric authoring, repeat runs so a pass rate means something, and a way to check the judge itself is reliable. | An LLM judge scores conversations against rubrics you write, over repeated runs, and judge calibration is reported per rubric. |
| Attack catalog | Prompt-injection and jailbreak attacks mapped to the OWASP LLM Top 10, kept current as attack techniques change. | A built-in catalog of 20+ scenarios across seven OWASP LLM Top 10 categories, plus attacks generated from your agent's own tools. On Team Pro Plus and Enterprise Ultra. |
| CI gating | Pass and fail thresholds, pipeline integration, and a way to block a merge and say why. | Quality gates for your CI, with guides for GitLab, Jenkins and the CLI runner, so a failing agent test stops the merge. |
| Run history and reports | Storage for runs and traces, a way to compare two runs, and a report a lead or reviewer can use. | Stored runs, Run Comparison between a baseline and a current run, and Release Reports with a SHA-256 hash. Evidence for a reviewer, not a certification. |
| A UI others can read | Something your product owner or QA lead can open without reading the harness code. | A visual canvas where each test is a list of steps, so non-authors can review it. |
After version one
The parts that need an owner.
The attack catalog
A catalog written once goes stale. With CertiQA the 20+ built-in scenarios are maintained for you, and attacks are also generated from the tools your own agent exposes, so they match your agent rather than a generic list.
A catalog finds known classes of problem. It does not prove an agent is safe.
Ask in a trial
- Which attacks got through?
- Do the results change after a prompt edit?
The judge
A judge is a model, so it can be wrong. Rubrics are repeated over several runs so a pass rate means something, and the Release Report reports judge calibration per rubric.
You still write the rubric. The judge is only as good as the standard you give it.
Still your job
Deciding what a good answer is. CertiQA does not replace that judgement.
Tests other people can review
Each test is a list of steps on a canvas, so a product owner can read and approve it. Assistants such as Cursor and Claude Code can draft tests over MCP, and the team reviews the steps rather than generated code.
How assistants write testsAlso in one place
Web UI and API tests run alongside agent tests, with one gate and one report. See how other tools differ.
FAQ
Build vs buy questions
Should we build an AI agent test harness in-house or buy one?
Build when your agent has requirements no product covers and you have engineers who can own the harness long term. Buy when the goal is to gate releases on agent quality soon and spend engineering time on the agent itself. A short trial on one of your own agents shows which side you are on.
What does an in-house agent test harness involve?
Typically a proxy that intercepts and mocks tool calls over REST and MCP, a judge that scores answers against rubrics, fault injection for timeouts and error responses, a catalog of prompt-injection and jailbreak attacks you keep current, a CI gate with pass and fail logic, stored run history and reports, and a UI so people other than the author can read the results.
What does CertiQA provide out of the box?
Tool interception and mocking, fault injection, an LLM judge with rubrics and repeated runs, an OWASP LLM Top 10 red-team catalog of 20+ built-in scenarios plus attacks generated from your agent's own tools, CI quality gates, run history, Run Comparison, hash-stamped Release Reports and a visual canvas where tests read as plain steps. It also covers web UI and API testing in the same place.
AI agent testing in detailCan an agent test reach our real systems?
Only on purpose. A tool with no mock is never sent anywhere. To reach your own staging system, a test includes an explicit step that routes one tool to an address you choose. On a CertiQA-hosted runner, private addresses are refused, so reaching a system inside your network needs a customer-hosted runner.
Security overviewWhat does CertiQA cost compared with building?
CertiQA plans start at $299 a month, with a 14-day trial. Your in-house cost depends on your engineers' time, how long the harness takes and how much maintenance it needs, so we do not put a figure on it. Add those up for your own team and compare.
See pricingHow does CertiQA compare with other agent evaluation tools?
Some tools focus on offline prompt evaluation, others on browser automation or production tracing. CertiQA focuses on testing agents before release, with mocked tools, red-teaming and CI gates, alongside UI and API testing. The comparison page sets out the differences from public sources.
Compare toolsPoint it at one agent you ship.
The quickest comparison is your own agent. The 14-day trial includes Team Pro Plus features and 500 credits. Mock its tools, run the attacks and wire a gate into CI to see what is left to build. Also see the brief for managers and example scenarios.