AI Agent Reliability · Forward-Deployed
Your AI agent fails silently.I find it in two weeks, or you don't pay.
A 2-week Reliability Audit that catches corrupt successes generic evaluation tools miss — using deterministic replay, calibrated judges, and state-diff verification. No-Find, No-Fee.
↳ No-Find, No-Fee — definedAI agents fail in production.
Fiddler AI · citing CMU and MIT research
Benchmark successes are corrupt.
arXiv 2603.03116 · τ-bench Retail/Airline
Judge agreement can be statistically meaningless.
aehf calibration study · n=90
Instrumentation — every claim traceable
The 4-Layer Architecture
A reliability audit is a measurement problem before it is a security problem. Four layers sit between “the agent answered correctly” and “the agent behaved correctly.”
Layers
Verdicts
Signer
Layer 01
Signed findings
Governance
Formal audit certificates with four verdict classes, not vague pass/fail. Every certificate states what was replayed, what sat outside the sandbox boundary, and what was never tested.
Layer 02
Measured, not asserted
Statistical verification
LLM judges are not trusted until they agree with human labels. Cohen's κ is reported. Wilson intervals bound every rate. McNemar's exact paired test runs on before/after fixes. When n cannot support a claim, the finding is reported as descriptive only.
Layer 03
Replayed, then diffed
Execution sandbox
Deterministic replay intercepts MCP tool calls, restores pre-state, re-executes each call, and diffs observed database state against the effect the agent claimed. This is where corrupt successes surface.
Layer 04
Attributed, not hand-waved
Failure taxonomy
Every failure is attributed to one of ten dimensions — instruction integrity, planner, memory, tool selection, invocation, correctness, alignment, faithfulness, security, execution integrity — and only after the taxonomy judge is calibrated and its κ reported.
Scope of verification
“Our execution sandbox verifies tool calls that cross the MCP boundary. In-process built-in tools and network side effects are outside its boundary and are declared as such in every certificate. We do not claim to verify what we did not replay.”
Declared, not assumed · printed in every certificate
Evidence, not opinions.
Three artifacts do the arguing: a worked case, the certificate format, and an explicit list of what this engagement cannot claim.
The duplicate refund.
The agent processed the same refund twice. Outcome-based evaluation called it a success — the customer was refunded. Replay against the restored pre-state caught the second issuance, because the second call produced an effect nobody claimed.
outcome evaluation: PASS — the answer was right
Dossier · choose an artifact
Matter record
Two weeks. Fixed fee. Signed.
One engagement, one scope, one accountable signer. The guarantee is narrow enough to be real, and it is written down before the first replay.
The 2-Week Reliability Audit
Every MCP call replayed against restored state. Every judge calibrated before it is trusted. Every verdict signed by the person who ran the sandbox. If the audit finds no A1 invariant failures and no taxonomy findings above the calibrated κ threshold, inside the stated scope, the fee is zero.
Not offered: hourly work, retainers, AIUC-1 certification, or a guarantee covering what the sandbox cannot see.
Terms register
SOW-04The audit is fixed-fee and never a retainer. If you want me to re-run the A1 invariant suite on every model or prompt update after the audit, that is a separate, optional monitoring contract.
“No-Find” means no A1 invariant failures and no audit-grade taxonomy findings above the calibrated κ threshold, within the stated scope of verification.
Full definition →Two ways to get the same replay discipline.
The method does not change; only who runs it does. Take the kit and run the harness yourself, or hand me the traffic and take the signed verdict.
The 2-Week Reliability Audit
Founding client rate — first 3 clients / standard rate
I replay your production traffic, calibrate the judges, and sign the findings. You get a verdict and a certificate, not a repository.
- Your workflows, replayed against restored state
- Judges calibrated before they are trusted
- A named signer on every finding
NO-FIND, NO-FEE · SCOPE AGREED IN WRITING
The Secure Agent Kit
Two licences · one-time licence each
The same replay discipline, shipped as a repository you run yourself. You bring your own agent and your own invariants.
- Next.js App Router shell in strict TypeScript
- Supabase auth, Postgres schema, and row-level security policies
- Polar.sh checkout, webhook verification, and subscription state
Same definitions either way: the kit proves the invariants you write, the audit proves the ones your traffic actually exercises. Checkout opens on the vendor’s site. No account, no form, and no data collected here.
The two weeks, accounted for.
No discovery theatre. Each phase produces an artifact you keep — including the written declaration of what we did not test.
- 01⏱ Day 0
Intake review
You submit the structured intake: workflow, volume, tool surface, and the evaluation in place today. I confirm the audit is bounded and survivable, then reply in writing with a go / no-go and a fixed-fee scope. If it isn't viable, the no is the deliverable.
Go / no-go memo - 02⏱ Days 1–2
Scope freeze
We fix the scope of verification and write the A1 invariants before any testing. This includes an error-analysis pass on a sample of your real traces to cluster failures into fewer than 10 types before the invariants are written. The sandbox boundary is declared in writing, including what sits outside it and will never appear in the verdict.
Invariant register - 03⏱ Days 3–9
Replay + calibration
Belay replays MCP calls against restored pre-state. Judges are calibrated against human labels; κ is reported before any judge verdict counts. Every observed effect is diffed against the claimed effect.
Replay traces · κ - 04⏱ Days 10–14
Certificate + readout
You receive the verdict, the scope declaration, the statistical claims register, and a signed readout with your risk owner. Every finding is traceable to a replay you can rerun.
Signed certificate
Questions that survive a technical read.
The terms a risk owner will actually check. Definitions come first, marketing second — which is why there is no marketing.
Definition discipline
If a term is ambiguous on this site, it is defined in the certificate. No-Find, No-Fee always links to its operational definition and the scope it applies to.
“No-Find” means two conditions hold inside the stated scope of verification: no A1 invariant failures, and no audit-grade taxonomy findings above the calibrated κ threshold. If either condition fails, the fee is zero.
The guarantee does not extend to behavior outside the sandbox boundary. That exclusion is printed in the certificate, not discovered at invoice time.
Start the audit intake.
A structured intake, not a discovery conversation. Tell me the workflow, the agent’s tool surface, and what evaluation exists today. I reply in writing within one business day with a go / no-go and a fixed-fee scope.
Audit intake
INTAKE-01Structured intake · written reply within one business day
What happens next
I read the intake and reply in writing within one business day. Out-of-scope workflows get a no.
A written scope of verification, A1 invariants, and a fixed fee. No hourly, no retainer.
Two weeks of replay and calibration from scope freeze. One named signer on the certificate.
Within one business day · PKT (UTC+5)
Remote-first · working with teams worldwide
2-week, fixed-fee engagements. Scoped from the intake, not quoted by template.
AIUC-1-aligned evidence input. Not an accredited certification body.
