Inspector/deep QC/self-hostable
Your agent says it did the thing. Prove it.
Inspector audits another AI agent. It reads the agent’s real configuration, designs an exam from what the agent claims to do, holds six isolated conversations with it, diffs the data around every one, and reports which of its actions actually happened — and which it only said happened.
- Layers
- 5, from x-ray to forensics
- Rubric
- 17 dimensions in four bands
- Write verdicts
- 7, decided by evidence
- Accounts
- None. No login, no user database
The problem
A confident agent and a working agent read identically.
An agent that files your ticket says “I’ve created your ticket!”. An agent that files nothing says exactly the same sentence, in the same warm tone, just as fast. The customer believes it. The operator reading logs cannot tell. The request is simply gone, and nobody downstream ever learns it was dropped.
Asking the agent whether it worked is worthless — that is the one question a broken agent is guaranteed to answer wrongly. Inspector does not ask.
How it works
Five layers.
Each one narrows what the agent could be getting away with. The deterministic parts — the verdicts, the failure classifier, the credential scan, both rating caps — are plain functions with no model in the path.
LAYER 1
X-ray
read the target’s real config and tool inventory
LAYER 2
Exam design
generate scenarios, then repair them deterministically
LAYER 3
Live run
invoke the target, one isolated thread per scenario
LAYER 4
State diff
photograph the data before and after every scenario
LAYER 5
Forensics
read logs and traces, classify failures by owner
Why it holds up
Three things that do not depend on trust.
The canary
It proves the action happened.
A unique reference code is planted inside every write test, then requested back in a brand new conversation. An agent that really wrote something can find it. One that only said so cannot — and that is a phantom write, caught with no database access at all.
The caps
A generous grade cannot survive the evidence.
The rating caps are plain if statements applied after grading, never trusted to the grading model. A proven phantom write caps the agent at 2.0; a leaked credential caps it at 1.0. The model does not get a vote.
Null is an answer
It says what it could not see.
No logs, no config, no data store — the audit still runs, and every finding is stamped VERIFIED, CORROBORATED or CLAIMED according to what actually backed it. The report states its own blind spots and what access would close them.
How to use it
Four things to hand it.
Only the first two are required, and the second one is a text file.
Run it
Audit your own agent.
Credentials arrive in the request, are passed to the adapter, and die with it. They are never written to the store, never included in the saved report, and never logged.