Core Concepts

Observability and traces

Debug a workflow from request to capability, evidence, proposal, external action, approval, and replay.

Overview

Synapsor stores the durable run graph. Trace inspection turns that graph into span-like records with workflow, branch, capability, evidence, proposal, external action, token, and latency fields.

The current MVP supports SHOW TRACE FOR AGENT RUN, SHOW TOKEN USAGE, SHOW SLOW AGENT CAPABILITIES, SHOW APPROVAL LATENCY, REST trace lookup, and observability sink registration. OpenTelemetry export is represented as a sink contract first; background exporting can be attached later.

Use trace output to answer: what did the agent ask for, which capability ran, what evidence/proposal lookup ids were produced, and where did the workflow stop?

Developer notes

Propagate request ids, run ids, evidence ids, and proposal lookup ids in support tickets.
Redact payload fields before exporting traces outside Synapsor.
Use trace inspection before scraping logs.
Treat the sink as a contract until an exporter worker is attached.

Trace SQL

SQL

-- run_... comes from BEGIN AGENT RUN, db.agent_runs.start(...), or db.agentRuns.start(...).
SHOW TRACE FOR AGENT RUN 'run_...';

SHOW SLOW AGENT CAPABILITIES
SINCE TIMESTAMP '2026-05-01'
ORDER BY p95_latency_ms DESC;

SHOW TOKEN USAGE
BY WORKFLOW
SINCE TIMESTAMP '2026-05-01';

SHOW APPROVAL LATENCY
BY WORKFLOW
SINCE TIMESTAMP '2026-05-01';

CREATE OBSERVABILITY SINK prod_otel
TYPE OPENTELEMETRY
ENDPOINT 'https://otel-collector.example.com/v1/traces'
EXPORT AGENT RUNS, AGENT RUN STEPS, CAPABILITY INVOCATIONS, WRITE PROPOSALS, EXTERNAL ACTIONS
REDACT SESSION KEYS ('customer_email', 'api_key')
SAMPLE RATE 1000; -- milli-rate: 1000 means 100% in the current MVP

Examples

python

trace = db.trace_agent_run("run_...")

for span in trace["spans"]:
    print(span["name"], span["status"], span["latency_ms"])

db.create_observability_sink(
    name="prod_otel",
    type="opentelemetry",
    endpoint="https://otel-collector.example.com/v1/traces",
    export=["agent_runs", "capability_invocations", "write_proposals"],
    redact={"payload_fields": ["ssn", "card_number"]},
    sample_rate_milli=1000,
)