It answered wrong.
It approved a refund it should have denied. Somewhere in the context you fed it, one piece flipped the decision. Which one?
Can’t you just ask a model?
Three confident answers, none falsifiable. So how do you actually know?
One failed run. Five layers of evidence.
Follow a wrong answer all the way back to its cause.
Asking the model why only gets you a confident guess.
Inspect the answer, walk back through its context, isolate the suspect source, and test the fix by rerunning without it.
View as
The run recorded itself.
Every step wrote a note as it happened, with no extra tracing work and no story reconstructed later.
Start at the answer. Walk back.
Follow the edges from the wrong decision. Only the notes behind it light up, and the rest of the run falls away.
Every piece of context has an address.
Rules, skills, memory and retrieved facts stop being one invisible prompt. Each lands in system, messages or tools, at a moment you can name.
The walk ends at a suspect.
One retrieved policy sits in the system slot, and the decision read it. That is a suspect, not proof.
Remove one source. Rerun the case.
Same question, same baseline, one source gone. If the decision flips, you have evidence of dependence in this case, not hidden reasoning.
Catch it before it answers.
The same record runs forward. A travel agent picks a tool by its description, and two tools tie. The library flags it. You own the fix.
The loop records itself.
Prompt, ask, return, answer. Every event drains into a typed log with its own cost, as the agent runs.
Stop guessing why your agent answered wrong.
Record every run. Walk it back to the cause. Test the fix by replaying it.
What you run
Rank the suspects. Then test one.
Both ship today and carry a beta marker. Scores from an embedding are proxies, so the rerun is what tests a source, and only for this case.
import { localizeContextBug, rerunWithoutSources, embeddingCache } from 'agentfootprint/observe'; // Point it at the step that went wrong. It ranks the suspects.const report = await localizeContextBug({ artifacts, // { snapshot, events } from the recorded run embedder: embeddingCache(embedder), atStep: answerStep,}); // Remove one source and run the case again.const again = await rerunWithoutSources({ report, ignore: ['refund-policy'], runner, originalAnswer, embedder: embeddingCache(embedder), answerChanged: (a, b) => a.includes('APPROVED') !== b.includes('APPROVED'),});