Multi-agent systems
Decide which agents are runs and which are spans. wrapAgent is run-level only; a sub-agent inside a request is an agent-kind span.
wrapAgent / wrap_agent is run-level only. It always opens a new top-level run,
and nesting it never produces a parent/child trace — you get two independent runs.
A span whose kind is agent comes from a different method:
| Level | Method |
|---|---|
| Run (top-level) | wrapAgent / wrap_agent, startRun / start_run |
| Span, in-process | withSpan(name, fn, { kind: 'agent' }) / trodo.span(name, kind='agent') |
| Span, cross-process | joinRun / join_run (already defaults to kind: 'agent') |
| Span, HTTP handler | expressMiddleware() / fastapi_middleware() |
Which agents are runs?
This is a judgment call, and there are two ways to get it wrong:
- Splitting too finely. A supervisor and its five workers each get a
wrapAgent, so one user question becomes six disconnected runs. No row holds the request's cost or latency, and the delegation tree is gone. - Merging too aggressively. Genuinely separate agents get forced under one wrap because "one request, one run" was applied as a rule. Each loses its own name, success rate, latency and cost, buried inside a trace that belongs to something else.
Decide per agent, by what triggers it and what owns its lifecycle:
| Situation | Shape |
|---|---|
| A step the current request fans out to, with no independent existence | agent-kind child span |
| Has its own trigger — route, queue message, cron tick, retry policy, cache | Its own run, linked with parentRunId |
| A separate agent someone would ask "how is it doing?" about on its own | Its own run |
| Runs in another service or worker as part of the same request | Same run — propagate and join it |
| A turn in a multi-turn chat | One run per turn, grouped by conversationId |
Size isn't the test. A forty-step sub-agent that exists only to serve one request is
still spans; a two-line agent with its own queue is still its own run. When it's
genuinely unclear, separate runs linked with parentRunId is the more recoverable
choice — the relationship is recorded, and separate runs can still be read together,
whereas a merge can't be undone from stored data.
Sub-agents of one request
When the table says the sub-agents have no independent existence, wrap once at the entry point and give each one a span. Its model and tool calls nest underneath it, so the waterfall shows which sub-agent did what.
const { result } = await trodo.wrapAgent('research_assistant', async (run) => {
run.setInput([{ role: 'user', content: question }]);
const plan = await trodo.withSpan('planner', async (span) => {
span.setInput([{ role: 'user', content: question }]);
const p = await planner.run(question); // LLM span nests under 'planner'
span.setOutput(p);
return p;
}, { kind: 'agent' });
// Fan-out — each worker is its own agent span, running in its own async scope.
const findings = await Promise.all(
plan.subtasks.map((t) =>
trodo.withSpan(`researcher:${t.topic}`, async (span) => {
span.setInput([{ role: 'user', content: t.prompt }]);
const r = await researcher.run(t);
span.setOutput(r);
span.setAttribute('topic', t.topic);
return r;
}, { kind: 'agent' }),
),
);
const answer = await writer.run(findings);
run.setOutput({ answer });
return answer;
}, { distinctId: userId, conversationId: threadId });with trodo.wrap_agent('research_assistant', distinct_id=user_id,
conversation_id=thread_id) as run:
run.set_input([{'role': 'user', 'content': question}])
with trodo.span('planner', kind='agent') as span:
span.set_input([{'role': 'user', 'content': question}])
plan = planner.run(question) # LLM span nests under 'planner'
span.set_output(plan)
findings = []
for task in plan.subtasks:
with trodo.span(f'researcher:{task.topic}', kind='agent') as span:
span.set_input([{'role': 'user', 'content': task.prompt}])
result = researcher.run(task)
span.set_output(result)
span.set_attribute('topic', task.topic)
findings.append(result)
answer = writer.run(findings)
run.set_output({'answer': answer})The trace reads:
run: research_assistant (agent)
├─ planner (agent) → llm
├─ researcher:pricing (agent) → llm, tool: web_search
├─ researcher:competitors (agent) → llm, tool: web_search
└─ llm: gpt-4o (the writer's call, auto-captured)Naming
Name a sub-agent span for what it is (planner, sql_writer, critic), not where it
sits (step_2, child_agent) — the dashboard groups on the name. When one sub-agent
type fans out over many inputs, suffix the instance and repeat the discriminator as an
attribute: researcher:pricing plus setAttribute('topic', 'pricing'). That keeps every
researcher groupable while individual instances stay distinguishable.
Frameworks that already do this
If the framework owns the handoff, it emits the sub-agent spans itself — wrap the entry point and nothing else, or you'll get a duplicated layer in the waterfall:
| Framework | Sub-agent spans |
|---|---|
| OpenAI Agents SDK handoffs | Automatic |
| LangGraph / LangChain multi-agent graphs | Automatic |
| LlamaIndex agent workers | Automatic |
Vercel AI SDK with maxSteps / tools | Automatic (needs experimental_telemetry.isEnabled on every call) |
Your own supervisor loop, Promise.all, asyncio.gather | Add agent spans yourself |
Separate agents, linked not merged
An agent with its own trigger, lifecycle or identity keeps its own run. parentRunId /
parent_run_id records what caused what without collapsing them into one trace, so each
keeps its own name, success rate, latency and cost:
const { runId } = await trodo.wrapAgent('intake', fn, { distinctId });
// …later, in the queue consumer that picked up the job intake enqueued:
await trodo.wrapAgent('enrichment', fn, { distinctId, parentRunId: runId });with trodo.wrap_agent('intake', distinct_id=user_id) as run:
parent_run_id = run.run_id
# …later, in the worker:
with trodo.wrap_agent('enrichment', distinct_id=user_id,
parent_run_id=parent_run_id) as run:
...Next
- Add manual spans for tools and retrieval
- Conversations for multi-turn chat
- Distributed tracing when a sub-agent lives in another service