Observability sinks
The ObservabilityStrategy port — one required hot-path method that must be sync and must not throw. Compose several sinks into one, choose when delivery happens, and know exactly what a recording keeps and what it summarizes away.
Every run emits a typed AgentfootprintEvent stream. Where that stream goes
is a port with adapters, exactly like memory or hosting: the agent loop never
knows whether it is feeding CloudWatch, an OTel collector, a file, or nothing.
The port
interface ObservabilityStrategy extends BaseStrategy {
readonly capabilities: ObservabilityCapabilities;
exportEvent(event: AgentfootprintEvent): void;
}Two members of its own, six inherited — and only two of the eight are required:
| Member | Required | What it is |
|---|---|---|
name | ✅ | Registry key, lowercase-kebab ('datadog', 'agentcore'). Used to look it up and to de-dupe registrations |
capabilities | ✅ | { events?, logs?, traces?, metrics? } — declared, so a consumer can branch instead of guess |
exportEvent(event) | ✅ | The hot path. Translate to the vendor's wire format and ship |
flush?() | — | Drain whatever the hot path buffered. void for sync sinks, Promise<void> for async ones |
stop?() | — | Teardown: close clients, clear timers. Idempotent, and terminal — after it, events are dropped and there is no restart |
relevantEventTypes? | — | An event-type filter. Set it and the dispatcher only forwards those types |
validate?() | — | Config validator, called once at registration. Throw here rather than produce an empty dashboard |
_onError? | — | Error sink for this strategy's own failures. A property you assign after construction, not a constructor option |
Two laws govern exportEvent, and they are what make a bad sink survivable:
exportEventMUST be syncvoid. Buffer internally; drain inflush().
MUST NOT throw. Errors caught and routed to
_onErrorat the dispatch layer; one bad strategy never breaks the agent loop.
Who calls flush() and stop()
Three doors, since 8.12.0: the handle enable.* returns (handle.flush() /
handle.stop()), agent.shutdown(), and a standingAgent closing with its
default shutdown: 'flush'. agent.run() never flushes — that part is
unchanged, and it is why a batching exporter still needs one of the three at
process end.
Mounting one
import { agentcoreObservability } from 'agentfootprint/observe';
const handle = agent.enable.observability({
strategy: agentcoreObservability({ region: 'us-west-2', logGroupName: '/agentfootprint/assistant' }),
});ObservabilityEnableOptions is { strategy?, tier?, sampleRate?, detach?, flushOn? }.
No strategy means no telemetry — not a default one
enable.observability() called without a strategy returns a no-op handle and
attaches nothing. It does not silently fall back to
consoleObservability(): choosing a destination is an opinion the library does
not impose. Pass a strategy, or you get silence.
Axis 1 — which sink
All on agentfootprint/observe:
| Adapter | Ships to | Peer dep | Choose it when |
|---|---|---|---|
consoleObservability() | stdout | none | Local development; you want to see the stream |
cloudwatchObservability(opts) | CloudWatch Logs | @aws-sdk/client-cloudwatch-logs | You already read CloudWatch |
agentcoreObservability(opts) | CloudWatch Logs, in AgentCore's GenAI-Observability shape | @aws-sdk/client-cloudwatch-logs | A hosted AgentCore runtime, so agent events land beside the runtime's own telemetry |
xrayObservability(opts) | AWS X-Ray segment trees | @aws-sdk/client-xray | You trace with X-Ray. serviceName is required — a missing one is a TypeError |
otelObservability(opts) | Your OTel tracer, as spans + span events under gen_ai.* | @opentelemetry/api | Portability, or a collector you already run |
fileObservability(opts) | NDJSON on a local disk — one JSON.stringify(event) per line | none (node:fs) | There is no collector to ship to. A log shipper, jq or a person with grep reads the file already |
auditExport(opts) | A hash-chained bundle | none (node:crypto) | You need an evidence record, not a dashboard |
agentcoreObservability is a thin wrapper over the CloudWatch base — the only
difference is the strategy name. That is stated here rather than implied,
because it tells you the two rows have the same reliability, not merely similar
shapes.
otelObservability takes your tracer rather than bundling an SDK: OTel's SDK
is heavyweight and exporter-specific, and forcing one would defeat the
portability the standard exists for. When an injected tracer's spans do not
implement addEvent, the adapter falls back to flattened
${eventName}.${key} attributes — degraded (last-write-wins) but never silently
dropped.
Google Cloud — the same adapter, pointed at telemetry.googleapis.com
Google's Telemetry API accepts standard OTLP, and Google's own migration guide
recommends moving off the Cloud Trace exporter because its proprietary transform
can lose data. So there is no googleCloudTracer() factory and there is not going
to be one: it would construct a TracerProvider from packages you install either
way and attach a GoogleAuth header — about twenty lines of wiring behind four new
optional peer dependencies. otelObservability() already takes the tracer.
const auth = new GoogleAuth({ scopes: 'https://www.googleapis.com/auth/cloud-platform' });
const client = await auth.getClient();
const exporter = new OTLPTraceExporter({
url: 'https://telemetry.googleapis.com', // regional: telemetry.<REGION>.rep.googleapis.com
// TLS *and* renewable Google call credentials — not TLS plus copied headers.
credentials: credentials.combineChannelCredentials(
credentials.createSsl(),
credentials.createFromGoogleCredential(client),
),
});plus OTEL_RESOURCE_ATTRIBUTES=gcp.project_id=<PROJECT_ID> and
GOOGLE_CLOUD_QUOTA_PROJECT=<PROJECT_ID>. Google's console
renders the GenAI views from the OpenTelemetry GenAI semantic conventions —
the same gen_ai.* attributes this adapter already emits — once you opt in with
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. There is no
auto-instrumentation for @google/genai on npm, so agentfootprint's own provider
hooks are the instrumentation.
Take the pinned versions, and do not pass `tracer` on this one
The Google path has three sharp edges that a 2026-08 field trial found the hard
way — OTel 2.x moved span processors into the NodeTracerProvider constructor,
google-auth-library@11 hands @grpc/grpc-js a Headers object it cannot read
(so the exporter ships unauthenticated), and passing an explicit tracer to
otelObservability produced six disconnected traces with zero child spans. The
working shape registers the provider globally and omits tracer. Full
copy-paste recipe, with the exact versions:
Google Cloud & Gemini.
fileObservability rotates ONE generation, and nothing is bounded on the way out
maxBytes renames the file to <path>.1 — replacing any previous .1 — and
starts fresh. No .2, no compression, no schedule, no cross-process
coordination: it is a disk-safety ceiling, and retention belongs to the
log-management daemon you already run (omit maxBytes and the adapter never
renames anything, which is the right choice when logrotate owns the file).
Payloads are written verbatim, so narrow what reaches the strategy with
eventTypes / tier / sampleRate, or keep values out of every event at its
source with the agent's redact (a footprintjs RedactionPolicy, see
Keep personal data out of the record)
— a bounded record by construction is auditExport({ payloadMode: 'bounded' }).
An unwritable path is refused at construction, naming the path. Full detail
on On-premises & self-hosted.
auditExport is tamper-EVIDENT, not tamper-PROOF
Every record carries the SHA-256 of its own canonical serialization plus the hash
of the previous record, and verifyAuditBundle names the exact record that
broke. It does not prove provenance: an adversary holding the only copy can
recompute every hash from the mutation onward and present a self-consistent
forgery. Ship the bundle somewhere they do not control, and that gap closes.
Independently reproduced against a local harness — the cross-segment chain, 2026-08-13
auditExport's hash chain is contract-shaped and tested — independently
reproduced against a local harness, 2026-08-13. Two runs were captured as
separately drained segments under
payloadMode: 'bounded': 31 records each, 62 of 62 verified across the segment
boundary, segment 2's chain head equal to segment 1's final hash, an exact
prompt / tool-argument secret marker absent from all serialized output, and a
single edited event type caught by verifyAuditBundle at sequence 1 with a
hash mismatch.
The rung is stated rather than rounded up. That run was deterministic and local — two mock-provider runs in one process, verified in the same process (its own note: "local and deterministic, so they consumed no GCP credit"). No bundle has been written to, re-read from, or verified out of a live durable store from this repository, so this is not field-validated: the hashing law is proven, the shipping-it-somewhere-they-do-not-control half is still yours to prove. What the run does buy is that the chain held across a segment boundary under somebody else's hands, not the author's.
Two sentences from that run stay exactly where they were. The chain is
tamper-evident, not tamper-proof (above). And the strategy observes Agent
events, so ingress refusals — a 401 from identity.verify, a 429 from
admission — are not in the bundle, because they happen before a run exists.
standingAgent({ onIngressDecision }) is the separate
stream for those, and it deliberately does not join this chain.
Being scored by somebody else's evaluator
A trace-scoring service reads spans your agent already emits — but only if it can
CLASSIFY them, and only if the content it is grading is on them. Two settings on
otelObservability decide both, and both are opt-in:
scopeName— the instrumentation scope every span is recorded under (default'agentfootprint'). Some consumers ROUTE on this name rather than read it, and a name they do not recognise is skipped in silence — which looks exactly like an agent with nothing wrong. The default does not change, because a rename moves every existing dashboard's spans out from under it.captureContent(defaultfalse) — puts the turn's prompt ongen_ai.task.inputand its answer ongen_ai.task.output(AgentCore's names, not the current GenAI registry's — kept for the consumers that read them). This adapter's standing rule is that content does not ride spans; enabling it exports raw prompt and answer text to whatever backend your tracer points at, where it is typically retained and searchable. Enable it when a consumer's whole purpose is to read the content — a scorer cannot grade a turn it cannot see — and not to make a dashboard richer. It also opens the evidence gate's unsupported values and an evaluation's explanation (below). Per-inference message arrays are deliberately not emitted: thellm_start/llm_endevents carry model, usage and stop reason, never the messages, so there is nothing truthful to put there.captureToolContent(defaultfalse) — each executed tool call's arguments and result, asgen_ai.tool.call.arguments/gen_ai.tool.call.resulton itsexecute_toolspan. A switch of its own, because tool traffic is a different disclosure from what the two parties said: results are the run's raw records. What leaves is the model's view, never more, and only for a call that ran and returned: the arguments the model proposed, captured the moment the call starts (so a tool that writes into its arguments cannot change what leaves), with every key youronToolCallrules changed — or a tool's ownredactpolicy hides — shown asREDACTED, since a rule can add a value the model never saw; and the result the model read, after youronToolResultscrubs. AcodeRunnerTool's arguments are its generated program and never leave (another code-executing tool, an MCP interpreter say, is not recognized). The spec's two attributes hold objects; any other value (plain text, a number, the string"123") rides anagentfootprint.*attribute of its own. A value overmaxContentChars(default 65,536) is omitted with its size stated — never cut, because a cut JSON value reads as complete data and one oversized attribute can fail an export batch along with other runs' spans.
What the span tree carries for an evaluator
Without any option, a scorer that reads OTel also gets the structure this library records and a plain GenAI tracer does not have — content-free, and every string on it capped at 256 characters:
| On the trace | From | Default | With a content switch |
|---|---|---|---|
gen_ai.conversation.id on agent / chat / tool spans | meta.sessionId | the session id's SHA-256 — absent for a run with none, never the runId | conversationId: 'raw' exports the id itself: safe only behind a door that verifies who may open a conversation |
gen_ai.evaluation.result span event | agentfootprint.eval.score | name, score value and label, plus the target words, the ref, the threshold and the evaluator kind as agentfootprint.eval.* attributes | captureContent adds gen_ai.evaluation.explanation |
agentfootprint.tools.absent, agentfootprint.tools.coverage_declared | a tool's absent() / coverage() | a COUNT per list (checked / not checked / cannot cover); the suggested tool's name only when this exporter has seen that tool run — never a sentence the tool author wrote | same on the event; under captureToolContent the call's own result is the whole envelope, sentences included |
agentfootprint.agent.evidence_checked | the evidence gate | verdict, posture, candidates, unsupported COUNT | captureContent adds the unsupported values |
agentfootprint.findings.standing, and a count per standing on the agent span | .findings() | ids and the standing, at most 32 events per turn plus one summary; an id the model wrote that names no result is a flag, never the text | same |
agentfootprint.integrity.disposition span | the checker accounting every run files | only when a row is actionable — a check filed findings, or a registered check filed nothing while the run did work — with one event per such row; a healthy run exports nothing | checkAccounting: 'all' exports every row on every run |
gen_ai.evaluation.result is the spec's evaluation event, carried here as a
SPAN event: the spec's own reference emitter writes it through the Logs API,
and this adapter speaks the Tracer API only, so a consumer that reads
evaluation events from its logs pipeline will not find these.
Where a score lands. A score filed during the turn lands on the span it
names — the tool call (targetRef = the call id) or the iteration (targetRef
= its index) while that span is open, the agent span for 'run' / 'turn' —
else on a short agentfootprint.evaluation span under the turn. The run a
score NAMES (target: 'run', targetRef = a run id) wins over the run it was
emitted from.
Whose run it is. A score is parented to a run only when its own meta places
it in that run's session: emitted from inside that run, or from the same
session. Run ids are sequential names that anyone can write, so a score that
cannot show it belongs — every agent.emit('agentfootprint.eval.score', …)
after the run, which carries no session — is exported on an
agentfootprint.evaluation span of its OWN — a new trace, never a child of
whatever request span is active — naming the ref it claimed. A host that files scores from user input (a feedback button) checks
the run is that user's first.
Late is best effort. A finished turn's span can no longer take an event, so
a late score and the checker accounting (which always arrives after
turn_end) each get a short span under the turn — a span started after its
parent ended, which is valid OpenTelemetry and which most trace views draw
outside the parent's bar. That works while the run is among the strategy's
last 32 closed turns; past that, the score is exported unparented and reported
through onError, never dropped in silence. The durable home for scores filed
after the run is a score record, not a trace span. The structure rows ride the
explainability switch; the evaluation event does not.
AWS Bedrock AgentCore Evaluations is the consumer that forced both. Since
July 2026 it scores agents that do not run on AWS, and agentCoreEvaluationSpans
in agentfootprint/observe is the configuration it needs:
import { agentCoreEvaluationSpans } from 'agentfootprint/observe';
agent.enable.observability({ strategy: agentCoreEvaluationSpans({ serviceName: 'support-agent' }) });That is a configuration of otelObservability — same spans, same attributes —
with the scope name their classifier accepts
(AGENTCORE_EVALUATIONS_SCOPE_NAME) and captureContent on. The scope name is
not overridable through it: honouring an override would produce spans the
service skips, which is the failure the named function exists to prevent.
Delivering those spans to CloudWatch is separate and yours.
Worth stating plainly, because the two are easy to confuse: their evaluators say what the score is; the agentfootprint trace says why it happened. They compose, and neither replaces the other.
Axis 2 — composition
composeObservability(children) returns one ObservabilityStrategy named
'compose', whose capabilities are the OR of its children's:
import { composeObservability, agentcoreObservability, otelObservability } from 'agentfootprint/observe';
agent.enable.observability({
strategy: composeObservability([
agentcoreObservability({ region: 'us-west-2', logGroupName: '/agentfootprint/assistant' }),
otelObservability({ tracer }),
]),
});exportEvent fans out to every child; flush() and stop() reach all of them.
One child failing does not stop the others — the passive-recorder rule. Siblings
exist for the other strategy kinds: composeCost, composeLiveStatus,
composeLens.
Axis 3 — delivery timing
detach decides when the sink's work happens relative to the agent loop:
detach | Behaviour | Choose it when |
|---|---|---|
| omitted | The strategy's hot path runs inline, in the loop | Cheap sinks; you want back-pressure to be visible |
{ driver, mode: 'forget' } | Scheduled off the loop; the handle is discarded | Pure telemetry — the common case |
{ driver, mode: 'join-later', onHandle } | Scheduled off the loop; every handle is delivered to onHandle so you can await it | You must know delivery finished (a test, a shutdown barrier) |
driver is required — there is no library default, because the right one is
an environment fact. Drivers come from footprintjs/detach:
microtaskBatchDriver (cross-runtime, the usual in-process pick),
setImmediateDriver (Node), setTimeoutDriver (configurable delay),
immediateDriver, plus factories createSendBeaconDriver (browser, survives
page-unload) and createWorkerThreadDriver.
import { microtaskBatchDriver } from 'footprintjs/detach';
agent.enable.observability({
strategy: agentcoreObservability({ region: 'us-west-2', logGroupName: '/agentfootprint/assistant' }),
detach: { driver: microtaskBatchDriver, mode: 'forget' }, // keep the loop unblocked
});A detached sink receives a snapshot, never the live event. The driver runs
the sink's work later — after the tool that emitted a tool_start has run,
after a result has been handed on — so each event is copied with
structuredClone at the moment it is scheduled, the same law footprintjs's
deferred observers follow. A tool that writes a header into its own arguments,
or code that mutates a result after tool_end, cannot change what a detached
sink exports. The copy costs about 2 µs for a typical event and about 30 µs for
a 120 KB iteration_end. A value that cannot be copied (a function, a live
handle) is delivered as [not cloneable] and reported once per event type
through the strategy's onError — never as a live reference.
Axis 4 — volume
tier is 'minimal' | 'standard' | 'firehose', default 'standard', and
filters which event types reach the strategy. sampleRate (0–1) thins what is
left.
tier is not a privacy control. It reduces volume, not sensitivity —
redactContent does not apply to it. If a payload must not leave the process,
filter or redact it, do not lower the tier.
relevantEventTypes on the strategy itself is the finer instrument: the
dispatcher will not even forward types outside that set.
Which conversation an event belongs to
EventMeta.sessionId (9.4.0) rides beside runId — one session produces many
runs, and an event needs to say which of each it is. The CloudWatch and AgentCore
adapters serialize the whole envelope, so it arrives without their knowing about
it; otelObservability places its SHA-256 as gen_ai.conversation.id on
the run's agent, chat and tool spans — the digest, because at a door with no
verifier the session id itself opens the conversation.
await agent.run(message, { sessionId }); // or let standingAgent do it for youstandingAgent sets it on both run() and resume() from the request's own
session id. It is never derived, guessed, or defaulted to the runId: an
anonymous request has no session, and an absent key and an invented one are
different facts. Worked example: Exporters.
Who the run was for — the audit wire (9.11.0)
The stream has always said what happened and when. EventMeta.principal
and EventMeta.tenant are the who, and together the three are an audit
record rather than a debug log.
await agent.run(message, { identity: { tenant: 'acme', principal: 'alice@acme.test', conversationId } });Every event of that run carries meta.principal and meta.tenant.
They are stamped only from an identity a caller NAMED — run({ identity })
or run(input, { identity }), the same tuple memory and the permission gate
scope on. Three things deliberately do not produce them:
| Situation | principal / tenant |
|---|---|
run(msg, { identity: { tenant, principal, conversationId } }) | Stamped |
run(msg, { sessionId }) — the session-derived default | Absent |
run(msg) — anonymous | Absent |
A conversation id is not an actor
The run's internal identity is always populated: it defaults to
{ conversationId: '<runId>' }, or since 9.10.0 to { conversationId: sessionId }
on a session-bound run. Neither is a person. A sessionId in particular is
caller-supplied — anyone who can reach the host can send any string, including
somebody else's — so promoting it to "who did this" would produce an audit trail
that looks complete and names the wrong party. Absent is the honest answer, and
absent is what the meta says.
Which sinks carry it
Two kinds of adapter, and the difference decides the answer:
| Sink | Carries the actor | Why |
|---|---|---|
fileObservability | ✅ automatically | One JSON.stringify(event) per line — the whole envelope |
cloudwatchObservability | ✅ automatically | The log message is the serialized envelope |
agentcoreObservability | ✅ automatically | The same builder as CloudWatch |
auditExport | ✅ automatically | The meta rides the record, so the actor is inside the hash chain — editing who breaks the same verification as editing what |
otelObservability | ✅ placed deliberately | two agentfootprint.* span attributes — principal.id and tenant.id — on the invoke_agent run span |
xrayObservability | ✖ not mapped | Segment-mapping like OTel, but no attribute is placed. Said plainly rather than implied |
An envelope-serializing sink inherits any new meta field for free; a
span-mapping one does not, and a doc that assumed otherwise would have been
wrong about two of the six. OTel uses agentfootprint.* names rather than
enduser.id: the semantic convention for that attribute has moved once already,
and quietly claiming a convention we do not track is worse than a name that says
whose field it is.
What a recording keeps
recordRun(runner, options?) produces the canonical Recording —
{ snapshot, events, structure } — which is what an offline viewer replays.
| Option | Default | What it controls |
|---|---|---|
maxEvents | 10000 | Oldest events are dropped past this; the count is reported as droppedEvents |
boundaryDetail | 'full' | 'full' = every field, content included. 'lean' = boundary structure only, no captured payloads |
recordEmbeddings | false | Whether raw vectors survive the freeze |
Embeddings are summarized, not kept. By default every embedding /
embeddings field — in boundary payloads, in the snapshot's subflow results, in
tool results riding the event stream — is replaced with { dims, norm } when the
recording is frozen. dims is the dimensionality; norm is the L2 norm rounded
to four decimals — a checksum, not an operand.
The measurement behind the default: one retrieval turn's recording weighed
2.76 MB, about 1.1 MB of it embedding floats. Nothing that renders a recording
reads those floats, and the retrieval evidence a debugger actually needs — score,
passage, document, rejected candidates — carries no vectors and is untouched. Set
recordEmbeddings: true only when a consumer genuinely replays raw vectors
offline.
The same option exists on BoundaryRecorder, where it applies at capture
time rather than at freeze time.
Status
| Piece | Door | Status |
|---|---|---|
ObservabilityStrategy port, composeObservability | agentfootprint/observe | Shipped |
consoleObservability, auditExport / verifyAuditBundle | agentfootprint/observe | Shipped — no vendor SDK |
fileObservability | agentfootprint/observe | Field-validated — validated in an independent field trial, 2026-08 (Node 22: 30 asserted events written as NDJSON and parsed back, on a live agent run) |
otelObservability | agentfootprint/observe | Shipped — BYO tracer |
Cloud Trace / Cloud Logging via otelObservability + OTLP | agentfootprint/observe | A documented recipe, not a new adapter — field-validated in its corrected form, an independent field trial, 2026-08 |
cloudwatchObservability, agentcoreObservability, xrayObservability | agentfootprint/observe | Shipped; contract-mapped and injection-tested, command names pinned |
detach delivery, tier, sampleRate | — | Shipped |
recordRun + { dims, norm } embedding summary | agentfootprint/observe | Shipped |
EventMeta.sessionId | agentfootprint/events | Shipped (9.4.0) |
EventMeta.principal / EventMeta.tenant | agentfootprint/events | Shipped (9.11.0) — explicit identity only |
Actor on otelObservability run spans | agentfootprint/observe | Shipped (9.11.0) |
Actor on xrayObservability segments | — | Not mapped |
Next
- Identity & credentials — where the actor on the wire comes from
- Observability — the event taxonomy and the three workflows
- Exporters: AgentCore & OTEL — step-by-step wiring for both
- Offline replay — what a recording is for
- AWS & Bedrock AgentCore — the worked provider
- Google Cloud & Gemini — the OTLP recipe in full, and the rest of that column
Tools & gateways
The McpClient port — four members, three transports. Where a tool comes from is a transport decision (stdio, fixed HTTP headers, per-request vending, or your own signing fetch); what a tool is allowed to do is a governance decision, and they compose without knowing about each other.
AWS & Bedrock AgentCore
The first worked provider. Every AWS service agentfootprint adapts — Memory, Identity, Runtime hosting, Gateway MCP, Observability, Policy, S3 Vectors, Bedrock models and embedders — with its adapter, its door, its peer dependency, the SDK commands it dispatches, and its honest status.
