Accepting Q4 2026 audits

AI Agent Reliability · Forward-Deployed

Your AI agent fails silently.I find it in two weeks, or you don't pay.

A 2-week Reliability Audit that catches corrupt successes generic evaluation tools miss — using deterministic replay, calibrated judges, and state-diff verification. No-Find, No-Fee.

↳ No-Find, No-Fee — defined
Failure register — sourced
70-95%01

AI agents fail in production.

Fiddler AI · citing CMU and MIT research

27-78%02

Benchmark successes are corrupt.

arXiv 2603.03116 · τ-bench Retail/Airline

κ=0.00003

Judge agreement can be statistically meaningless.

aehf calibration study · n=90

Instrumentation — every claim traceable

MCP boundary
Belay replay
Cohen's κ
Wilson CI
McNemar exact
τ-bench
A1 invariants
AIUC-1-aligned
Sec. 01 — Method

The 4-Layer Architecture

A reliability audit is a measurement problem before it is a security problem. Four layers sit between “the agent answered correctly” and “the agent behaved correctly.”

Layers

04

Verdicts

04

Signer

01

Layer 01

Signed findings

Governance

Formal audit certificates with four verdict classes, not vague pass/fail. Every certificate states what was replayed, what sat outside the sandbox boundary, and what was never tested.

CleanQualifiedAdverseDisclaimer

Layer 02

Measured, not asserted

Statistical verification

LLM judges are not trusted until they agree with human labels. Cohen's κ is reported. Wilson intervals bound every rate. McNemar's exact paired test runs on before/after fixes. When n cannot support a claim, the finding is reported as descriptive only.

κ reportedWilson CIMcNemar exactn disclosed

Layer 03

Replayed, then diffed

Execution sandbox

Deterministic replay intercepts MCP tool calls, restores pre-state, re-executes each call, and diffs observed database state against the effect the agent claimed. This is where corrupt successes surface.

BelayMCP boundarystate diffrestored snapshot

Layer 04

Attributed, not hand-waved

Failure taxonomy

Every failure is attributed to one of ten dimensions — instruction integrity, planner, memory, tool selection, invocation, correctness, alignment, faithfulness, security, execution integrity — and only after the taxonomy judge is calibrated and its κ reported.

10 dimensionsκ-gateddescriptive fallback

Scope of verification

“Our execution sandbox verifies tool calls that cross the MCP boundary. In-process built-in tools and network side effects are outside its boundary and are declared as such in every certificate. We do not claim to verify what we did not replay.”

Declared, not assumed · printed in every certificate

Sec. 02 — Evidence

Evidence, not opinions.

Three artifacts do the arguing: a worked case, the certificate format, and an explicit list of what this engagement cannot claim.

Case 014Tier-1 supportRedacted

The duplicate refund.

The agent processed the same refund twice. Outcome-based evaluation called it a success — the customer was refunded. Replay against the restored pre-state caught the second issuance, because the second call produced an effect nobody claimed.

refunds.issuedFAIL
claimed
1
observed
2
refunds.amountFAIL
claimed
$149.00
observed
$298.00
order.status—
claimed
refunded
observed
refunded
A1-I2 · at-most-onceFAIL
claimed
declared
observed
violated
VERDICT — ADVERSE
A1-I2 violated · duplicate side effect
reproducible from trace 014

outcome evaluation: PASS — the answer was right

Shobi Builds · Reliability audit01 / 03

Dossier · choose an artifact

Matter record

Matter idSB-014-Q4
ScopeMCP boundary
SignerShoaib Hussain
Verdict classes04
Sec. 03 — Offer

Two weeks. Fixed fee. Signed.

One engagement, one scope, one accountable signer. The guarantee is narrow enough to be real, and it is written down before the first replay.

NO-FIND, NO-FEE · OPERATIONAL

The 2-Week Reliability Audit

Every MCP call replayed against restored state. Every judge calibrated before it is trusted. Every verdict signed by the person who ran the sandbox. If the audit finds no A1 invariant failures and no taxonomy findings above the calibrated κ threshold, inside the stated scope, the fee is zero.

Duration
2 weeks
Pricing
Fixed fee
Evidence
Certificate + traces
Accountability
One named signer

Not offered: hourly work, retainers, AIUC-1 certification, or a guarantee covering what the sandbox cannot see.

Terms register

SOW-04
Engagement2 weeks, fixed dates
WorkflowsIT helpdesk · Tier-1 · AP invoice
FeeFixed, scope agreed in writing
StructureNot hourly · not retainer

The audit is fixed-fee and never a retainer. If you want me to re-run the A1 invariant suite on every model or prompt update after the audit, that is a separate, optional monitoring contract.

GuaranteeNo-Find, No-Fee
DefinitionA1 failures + κ-threshold findings
ExclusionsOutside scope of verification
SignerOne named human
Guarantee, defined

“No-Find” means no A1 invariant failures and no audit-grade taxonomy findings above the calibrated κ threshold, within the stated scope of verification.

Full definition →
Bridge — DIY or done-for-you

Two ways to get the same replay discipline.

The method does not change; only who runs it does. Take the kit and run the harness yourself, or hand me the traffic and take the signed verdict.

Done for you01 / 02

The 2-Week Reliability Audit

$1,500 / $2,500

Founding client rate — first 3 clients / standard rate

I replay your production traffic, calibrate the judges, and sign the findings. You get a verdict and a certificate, not a repository.

  • Your workflows, replayed against restored state
  • Judges calibrated before they are trusted
  • A named signer on every finding
Start the audit intake
Try our Free Reliability Lab

NO-FIND, NO-FEE · SCOPE AGREED IN WRITING

Do it yourself02 / 02

The Secure Agent Kit

Solo $99 / Team $149

Two licences · one-time licence each

The same replay discipline, shipped as a repository you run yourself. You bring your own agent and your own invariants.

  • Next.js App Router shell in strict TypeScript
  • Supabase auth, Postgres schema, and row-level security policies
  • Polar.sh checkout, webhook verification, and subscription state

Same definitions either way: the kit proves the invariants you write, the audit proves the ones your traffic actually exercises. Checkout opens on the vendor’s site. No account, no form, and no data collected here.

Sec. 04 — Protocol

The two weeks, accounted for.

No discovery theatre. Each phase produces an artifact you keep — including the written declaration of what we did not test.

  1. 01⏱ Day 0

    Intake review

    You submit the structured intake: workflow, volume, tool surface, and the evaluation in place today. I confirm the audit is bounded and survivable, then reply in writing with a go / no-go and a fixed-fee scope. If it isn't viable, the no is the deliverable.

    Go / no-go memo
  2. 02⏱ Days 1–2

    Scope freeze

    We fix the scope of verification and write the A1 invariants before any testing. This includes an error-analysis pass on a sample of your real traces to cluster failures into fewer than 10 types before the invariants are written. The sandbox boundary is declared in writing, including what sits outside it and will never appear in the verdict.

    Invariant register
  3. 03⏱ Days 3–9

    Replay + calibration

    Belay replays MCP calls against restored pre-state. Judges are calibrated against human labels; κ is reported before any judge verdict counts. Every observed effect is diffed against the claimed effect.

    Replay traces · κ
  4. 04⏱ Days 10–14

    Certificate + readout

    You receive the verdict, the scope declaration, the statistical claims register, and a signed readout with your risk owner. Every finding is traceable to a replay you can rerun.

    Signed certificate
Sec. 05 — Scope & definitions

Questions that survive a technical read.

The terms a risk owner will actually check. Definitions come first, marketing second — which is why there is no marketing.

Definition discipline

If a term is ambiguous on this site, it is defined in the certificate. No-Find, No-Fee always links to its operational definition and the scope it applies to.

“No-Find” means two conditions hold inside the stated scope of verification: no A1 invariant failures, and no audit-grade taxonomy findings above the calibrated κ threshold. If either condition fails, the fee is zero.

The guarantee does not extend to behavior outside the sandbox boundary. That exclusion is printed in the certificate, not discovered at invoice time.

Sec. 06 — Intake

Start the audit intake.

A structured intake, not a discovery conversation. Tell me the workflow, the agent’s tool surface, and what evaluation exists today. I reply in writing within one business day with a go / no-go and a fixed-fee scope.

Audit intake

INTAKE-01

Enough to scope the audit.

Six required fields. The intake is structured because the audit is — not a message box and not a pitch meeting.

Prepares a structured email — you review and send it.
Nothing is sent until you send it.

Structured intake · written reply within one business day

What happens next

01
Review

I read the intake and reply in writing within one business day. Out-of-scope workflows get a no.

02
Scope + fee

A written scope of verification, A1 invariants, and a fixed fee. No hourly, no retainer.

03
Replay begins

Two weeks of replay and calibration from scope freeze. One named signer on the certificate.

Response

Within one business day · PKT (UTC+5)

Location

Remote-first · working with teams worldwide

ACCEPTING Q4 2026

2-week, fixed-fee engagements. Scoped from the intake, not quoted by template.

AIUC-1-aligned evidence input. Not an accredited certification body.