Requirements live in documents and tickets, tests live in files, and the link between them lives in someone's head. Gaps show up when something ships broken.
For automation leads and staff SDETs
Your test suite keeps growing. Can anyone say what it covers?
Tests are written by several people and, increasingly, by assistants. They live across folders and repositories, and you are the one asked what is covered, what changed since the last build and whether anyone has read the new tests. This is how CertiQA helps you answer those without reading every script.
REQ-101 | Guest checkout with cardGuest checkout - card paid · Guest checkout - declined card |
REQ-104 | Promo code and taxNo test case yet |
REQ-110 | Refund within 30 daysRefund agent - inside window |
REQ-111 | Refund tool times outNo test case yet |
Example data. Mirrors the coverage matrix in a CertiQA project.
What this looks like day to day
Four questions that are hard to answer from a repository.
None of these are about writing more tests. They are about being able to see, review and maintain the tests you already have.
Generated suites arrive faster than a team can review. Reading hundreds of lines of test code to judge intent is the part that gets skipped.
A red build sends you into logs. You want to know which cases regressed, which recovered and which have been failing for a while.
The same steps are copied into every case, so one product change means fifty edits and one missed case.
How CertiQA helps
Each problem has a specific answer.
See coverage by requirement
Add your requirements to the project and link cases to the requirement keys they address. Connected AI assistants over MCP can draft scenarios against your requirements, and a person edits and approves the cases before they join a suite.
The project then shows every requirement with the cases that cover it, and flags the ones with none.
Coverage means a requirement is linked to a case. It does not say how thorough the case is, so use it to find gaps, not to prove completeness.
In the product
- A count of requirements covered out of the total.
- The cases linked to each requirement.
- A list of requirements with no linked cases.
Review tests as plain steps
Every case is a list of steps on a visual canvas: do this, expect that. Assistants such as Cursor and Claude Code can create cases in CertiQA over MCP, and a lead, product owner or domain expert reads and approves them as steps, without opening generated code.
Web UI tests still export to Playwright TypeScript when you want the code.
1 Go to /checkout
2 Type a test card that is declined into "Card"
3 Click "Pay now"
4 Expect message "Try another card" is visible
5 Expect order status is still PendingCompare two runs instead of reading logs
Pick a baseline run and a current run. Run Comparison lists regressions, improvements and persistent failures, with the change in duration and pass rate, and shows the error message and screenshot for each case.
You see
- Regressions, improvements and persistent failures.
- Duration and pass-rate change.
- The error and screenshot behind each case.
Fix shared steps once
Build login, data seeding or a payment mock once as a CertiBlock. Cases, suites and agent tests embed it, and when you edit the block each of them picks up the change on its next run. A case that runs in several suites is one definition, referenced from each.
How reusable blocks workOne case, three surfaces
A single case can drive a page, call an API and talk to an agent with mocked tools, then check what the customer sees. See AI agent testing.
When someone asks for proof
Hand a reviewer a Release Report.
Signing off a run in the app generates a Release Report for it, with the approver and decision recorded. A project-wide export for a date range is available through the API. It collects what CertiQA measured into a single export, with a SHA-256 hash so a reviewer can verify it was not changed afterwards.
It is evidence for a human reviewer, not a certification. The audit-log section covers the whole platform, not just the project, and the pack says so.
What managers ask to see- Run outcomes
- Pass rates with Wilson 95% intervals
- Fault injection recovery
- Per tool and failure mode
- Security posture
- Red-team results by category
- Judge calibration
- Per rubric
- Traced tool calls
- Trajectory and arguments
- Accessibility findings
- Automated axe-core checks
- Audit log excerpt
- Platform-wide, for the period
FAQ
Questions automation leads ask
How do I find out which requirements my automated tests cover?
Add your requirements to the project. When cases are linked to requirement keys, the project shows a matrix of every requirement with the cases that cover it and how many are covered, and flags requirements that have no linked case. Coverage here means a requirement is linked to at least one case. It does not tell you how thorough that case is.
How do AI assistants draft test cases?
Connected AI assistants like Cursor or Claude Code connect over MCP to draft test suites and cases directly from your PRDs, user stories, and application code. Cases are validated against CertiQA step schemas, and a person edits and approves them before they are added to active suites.
How do we review tests that an AI assistant wrote?
Assistants such as Cursor and Claude Code can create projects, suites and cases in CertiQA over MCP. They arrive as steps on the canvas, written as do-this, expect-that, so a lead can read, edit and approve them before they join a suite. You review the steps, not the generated code.
AI assistants over MCPHow do I see what changed between two test runs?
Open Run Comparison and pick a baseline run and a current run. It lists regressions, improvements and persistent failures, with the duration and pass-rate change, and shows the error message and screenshot for each case, so you do not have to diff terminal logs.
How do we stop the same steps being copied into every test?
Build the shared sequence, such as login, once as a CertiBlock. Cases, suites and agent tests embed it, and when you edit the block each of them picks up the change on its next run. A case that belongs to several suites is one definition referenced from each, not a copy.
Reusable test blocksDo we lose our tests if we stop using CertiQA?
Web UI tests export to Playwright TypeScript that you can read, edit and run yourself. Suites are plain steps, not an opaque format.
Web UI testing and Playwright exportWhat is a Release Report and how do I produce one?
It is a dated export of what CertiQA measured for a project, for a signed-off run or a date range. Signing off a run in the app generates one with the approver and decision recorded, and a project-wide export for a date range is available through the API. It covers run outcomes with pass rates and Wilson 95% intervals, fault-injection recovery, red-team results, judge calibration, traced tool calls, accessibility findings and an audit-log excerpt. A SHA-256 hash lets anyone verify it was not edited. It is evidence for a reviewer, not a certification, and the audit-log section covers the whole platform, not only the project.
Can one test cover the web UI, the API and an AI agent?
Yes. A single case can mix web UI steps, API calls and agent steps such as mocked tool responses, LLM judge checks and red-team scenarios, so one set of steps covers the journey.
AI agent testingStart with one requirements document and one suite.
Add the requirements, approve a plan and see which ones your tests cover. If you also lead QA, read how teams reskill with a smaller group.