path: root/DESIGN.md

# panto-subagents design

## Product shape

`panto-subagents` is a separate rock exposing one extension named
`subagents`. Its model-facing tools use the `subagents.*` namespace. It does
not shell out to `panto -p`; every child is an ordinary in-process
libpantograph `Agent` driven by pantograph.

The primary owns delegation. Children share its workspace, but not its
dialogue, and cannot call `subagents.*`. Recursive delegation, worktree
management, and merge orchestration are outside this design.

## v0 tools

### `subagents.run`

One tool call starts or resumes one conversational child:

```lua
-- Start a new child.
{
  agent = "reviewer",
  prompt = "Review the authentication changes.",
  model = "anthropic:sonnet", -- optional; always provider:model
  reasoning = "high",        -- optional
}

-- Resume an existing child owned by this primary session.
{
  id = "0198...",
  prompt = "Now focus on the missing tests.",
  model = "openai:gpt-5.6", -- optional per-turn change
  reasoning = "xhigh",      -- optional per-turn change
}
```

`prompt` must be non-empty. Exactly one of `agent` and `id` is required:
`agent` selects the profile for a new child; `id` identifies a persisted child
conversation. An ID cannot be resumed by a different primary session, and
only one turn may be active for an ID.

Once a valid new child has been allocated, every settled result includes its
ID:

```lua
{
  id = "0198...",
  agent = "reviewer",
  status = "completed", -- completed, failed, or cancelled
  output = "...",       -- present on success
  error = nil,          -- present on failure
  resumable = true,
}
```

Validation failures that happen before a child is allocated have no ID.
Under the existing filesystem-store flush discipline, a new child that fails
before producing its first assistant message has no durable file; such a
result reports `resumable = false` rather than promising continuation.

Parallel delegation uses ordinary model tool batching. The primary emits
several `subagents.run` calls in one batch; pantograph starts them concurrently
under one session-wide bound, initially four. A failure in one call does not
discard successful siblings. There is no second `tasks` array, task ID, or
extension-specific batch report format.

### `subagents.models`

The model catalog is too large to place in `subagents.run`'s description or
JSON schema. A small query tool exposes it on demand:

```lua
subagents.models {}
-- Current inherited model/reasoning plus configured provider names and counts.

subagents.models {
  provider = "anthropic",
  query = "sonnet",
  limit = 10,
}

subagents.models { model = "anthropic:sonnet" }
subagents.models { agent = "reviewer" }
```

An exact model or agent lookup returns its effective model, default reasoning,
and the reasoning levels known to be valid for it. A search returns only the
requested bounded matches and says when more exist. The tool reuses
pantograph's model registry and provider-specific reasoning-option logic,
including effort choices reported dynamically by Lua protocols.

The `subagents.run` description tells the primary to omit overrides normally
and call `subagents.models` before choosing an unfamiliar model or reasoning
level. Runtime validation remains authoritative when catalog capability data
is incomplete.

## Agent profiles

Profiles are Markdown files with YAML frontmatter, compatible at the common
format boundary used by other agent harnesses:

```markdown
---
name: reviewer
description: Reviews changes for correctness and missing tests
model: anthropic:sonnet
reasoning: high
---

You are a focused code reviewer.

Report only concrete, evidence-backed findings.
```

They are discovered recursively from two layers, lowest precedence first:

1. `${XDG_CONFIG_HOME:-$HOME/.config}/panto/agents/**/*.md`
2. `<cwd>/.panto/agents/**/*.md`

The project definition shadows a user definition with the same resolved name.
`name` is optional and defaults to the extension-less filename.
`description` is included with the discovered names in `subagents.run`'s tool
description so the primary can choose a profile without loading its full
prompt.

The portable contract is `name`, `description`, and the Markdown body as the
profile's system prompt. Panto additionally recognizes `model` only in full
`provider:model` form and recognizes its existing reasoning labels. Unknown
frontmatter is ignored. A foreign model spelling that is not valid Panto
syntax is ignored with a warning so the prompt remains usable and inherits
the primary model.

For a new child, model and reasoning are resolved independently:

```text
model     = tool.model     ?? profile.model     ?? primary.model
reasoning = tool.reasoning ?? profile.reasoning ?? primary.reasoning
```

Pantograph resolves the selected model against the configured provider and
model registries, applies the reasoning level, and fails before inference if
the known combination is invalid. The Lua extension never receives provider
credentials.

The resolved model reference and reasoning label are recorded on each child
turn. On continuation, omitted overrides keep the child's last effective
values; an explicit `model` or `reasoning` changes that value for the new turn
and for later continuations.

## Conversation construction

A new child receives a fresh `Conversation` containing:

1. the primary's effective system and project context;
2. a fixed child-role instruction;
3. the selected profile's Markdown body as a separate system message; and
4. the tool's `prompt` as its only initial user message.

The parent dialogue is not copied. The primary must put task-specific context
in the user prompt.

The profile system message carries opaque, non-model-visible message metadata
recording the immutable subagent manifest: its owning primary session ID and
profile name. Each user message carries the full model reference and reasoning
label effective for that turn. This avoids a mutable sidecar manifest: resume
reads identity from the first system message and defaults from the latest user
message. The existing filesystem-store load path must preserve per-message
metadata when rebuilding a `Conversation`.

The stored conversation is canonical on resume. Profile edits affect new
children, not an existing child ID.

Every child sees the parent's normal registered tools except `subagents.*`.
Tools still run through the shared extension runtime. Children share the
workspace, so concurrent writers can race or conflict; the primary must
delegate independent work.

## Durable child sessions

Subagent conversations are ordinary `FileSystemJSONLStore` sessions, not an
in-memory `id -> Conversation` map. The public subagent ID is the actual Panto
session UUID and the JSONL filename stem.

Each primary session owns a separate child catalog below its already-resolved
per-cwd session directory:

```text
<session-dir>/
├── <primary-session-id>.jsonl
└── subagents/
    └── <primary-session-id>/
        ├── <subagent-id>.jsonl
        └── <subagent-id>.jsonl
```

This layout automatically follows `PANTO_SESSION_DIR` and the normal XDG
fallback. The main store's `list` and `resolve` operations examine only direct
`.jsonl` children, so nested subagent files never appear in `panto sessions`
or the `/resume` picker.

The primary session ID is the ownership boundary. After panto restarts and
the user resumes a primary conversation, extension activation derives the
same child-catalog path from that primary ID. A referenced child ID therefore
resolves again. A different primary uses a different directory and cannot
resolve it. There is no global child lookup and no cross-primary continuation.

Starting and resuming use the ordinary session lifecycle:

```text
new:
  session = child_store.create()
  agent = Agent.init(session, no prior conversation)
  seed system/profile messages
  run prompt

resume:
  session = child_store.resolve(id)
  conversation = session.load()
  agent = Agent.init(session, conversation)
  run prompt
```

The Agent appends through its `Session` handle, and the filesystem store syncs
completed entries normally. The Agent may be destroyed after a turn; a later
turn reconstructs the conversation from JSONL. Only active job handles,
progress, and cancellation state remain in memory.

## Foreground progress and cancellation

Silent foreground waiting is not acceptable. Each child job has its ID and
profile before inference starts and emits tagged progress events to the TUI.
The user can see the child's assistant stream, tool calls, and tool results
while the primary model remains blocked on the tool call. Concurrent children
remain visually distinguishable.

These nested events are presentation data, not partial tool results: they are
not inserted into the primary model's conversation. `subagents.run` returns
only the settled result, while the full child conversation is persisted under
its ID for inspection and continuation.

Escape cancels every still-running child belonging to the current foreground
tool operation. Completed siblings remain successful. Cancellation targets
the child streams and the protocol streams those turns opened, then lets
every affected tool call settle coherently as `cancelled` rather than
abandoning the parent's batch.

## Generic pantograph child-job seam

libpantograph remains unaware of Lua, extensions, and subagents. Its `Agent`
and `SessionStore` abstractions already provide the conversation and
persistence machinery, and no subagent scheduler or tree policy lives
anywhere in Zig. The seam above it has three layers: a Lua binding
(libpanto-lua) that adds generic job, metadata, and tool-control primitives
usable by any embedder; a thin surface in pantograph itself (`panto.ext`) for
the few things that need the running process or its credentials; and this
rock, which is Lua policy written entirely on those two layers. Nothing
subagent-specific exists below the rock.

The binding's async job is the generic replacement for a hand-rolled worker
thread:

```lua
local job = agent:run_async {
  prompt = "...",                 -- or blocks, mirroring agent:run
  metadata = { ... },             -- optional user-message metadata, a JSON
                                   -- object (array-shaped tables are refused)
  dispatch_tools = false,         -- optional, default true: false ends the
                                   -- turn at the first assistant response
                                   -- instead of dispatching its tool calls
  wake_fd = fd,                   -- optional: one byte is written on every
                                   -- event and on settle
}

job:next_event()      -- -> event table | nil (non-blocking; owned copy)
job:result()          -- -> settled result table | nil while running
job:request_cancel()
job:close()            -- joins the pump and frees the settled result; a
                        -- caller must read and cache job:result() first
```

A pump thread owned by the job, not by the caller, drives the blocking turn
to completion, copies each event off the stream before the next pull frees
it, and buffers it under a mutex. The `wake_fd` contract is deliberately
loop-agnostic — a luv consumer arms `uv.new_poll(fd)`, any other reactor
selects on it — so no loop dependency reaches into the binding. Tool dispatch
inside the turn is unchanged libpantograph behavior: when the job's agent
carries tools registered through pantograph's Lua tool source (the primary's
own extension tools, copied over via `agent:set_tools`), `invoke_batch` posts
to the Lua owner thread through the same multiplexed executor a primary's
batch uses (below), bound to that job's agent as its lane. `dispatch_tools =
false` is the generic knob that makes a one-shot structured worker possible
without any host involvement: the settled result carries the assistant's
tool calls (id, name, and raw input JSON) unresolved, and no second model
round runs.

Two more binding primitives complete the generic surface. Message metadata —
`conv:message_metadata(i)` / `conv:set_message_metadata(i, tbl)`, plus an
optional `metadata` argument on `add_system_message`/`add_user_message` —
reads and writes the same JSON-object metadata the disk format already
round-trips, on owned and borrowed conversations alike; the rock uses it for
both the subagent manifest and the per-turn model/reasoning stamp, with no
separate sidecar record. Tool control — `agent:tools()` returns a mutable
copy of an agent's declarations, `agent:set_tools(decls)` replaces them, and
`agent:set_config` additionally accepts `tool_choice` to force a named tool —
lets the rock copy the primary's tools onto a child and drop `subagents.*`,
or hand a child exactly one synthetic, call-refusing declaration for a
one-shot worker's output tool. Dispatch is name-keyed on the shared runtime
tool source, so a copied declaration reaches the same handler regardless of
which agent it is registered on.

pantograph adds only what requires the running process or its credentials.
`panto.ext.resolve_model { model = "provider:alias", reasoning = label,
tool_choice = ... }` returns an **opaque config userdata** accepted by
`panto.agent { config = ... }` and `agent:set_config`; resolution reuses the
existing provider/model registries and reasoning pipeline, fails before
inference, and the userdata exposes only `model`/`reasoning`/`style`/
`wire_model` — credentials never reach Lua. This is the one piece of the seam
that could not be a binding API. It, `panto.ext.session_info()`, and
`panto.ext.models` all read from one feature-neutral, read-only session
record: the provider and model registries, the primary's already-credentialed
live provider config, the current model label, registered extension
protocols, and the session's id and directory. `panto.ext.on("turn_start" |
"turn_interrupt" | "turn_end", fn)` fires around every turn on the primary
session — interrupt before the pump parks on Escape or Ctrl+C, end on every
exit path with a reason — generic and observe-only; the rock is one
subscriber among any number that could exist, cancelling every live child job
on interrupt and closing them on end. `event:set_component(tbl)` returns a
handle (`h:invalidate()`, `h:alive()`) so a component that mutates after it is
set — a progress card gaining a line — can ask for a repaint instead of
rendering once and going stale.

Everything else — the concurrency gate and queue, child store layout and
directory creation on top of `SessionStore`, manifest and resume-default
extraction from message metadata, tool filtering, one-shot worker
construction, cancellation propagation, and progress rendering — is Lua
policy in the rock, built entirely on the primitives above. A different
extension could build an unrelated concurrent-job feature on exactly the same
seam.

## Shared Lua runtime executor

All agents in one primary session share its Lua state, activated extensions,
and libuv loop. Children do not create Lua states, reload rocks, or
reactivate extensions.

The runtime multiplexes tool batches rather than tracking one global
in-flight batch — the primary's and one per in-flight child agent can be live
at once, all executing on the single Lua owner thread:

1. Parent tool handlers start child jobs and yield.
2. Child workers drive ordinary libpantograph streams.
3. Built-in providers run directly on those workers.
4. A child Lua tool call is posted to a thread-safe runtime queue, waking the
   shared loop through a pantograph-owned `uv_async_t`. A Lua protocol call
   made on behalf of a child travels the same queue.
5. The runtime executes a tool callback as a coroutine on the sole Lua owner
   thread, where async extension code may yield normally. A protocol callback
   is posted the same way but runs to completion under `pcall` on that
   thread: it cannot yield.
6. Completion returns to the waiting child worker, while unrelated batches
   keep making progress.

Tool batches carry explicit identity, so result recording always addresses
the right batch rather than one global slot. Each batch is bound to the agent
that raised it, and `panto.ext.agent` resolves to that Agent while the
batch's coroutine runs. Protocol sources are not bound this way — one
registered protocol serves every agent in the session, and its streams keep
the turns apart. Module globals remain shared intentionally. A
child's tool declarations are exactly whatever its own `agent:set_tools` call
put there — there is no host-side filtered view to keep in sync.

## Extension-provided protocol concurrency

A registered protocol is process-wide, not per-child: the primary and every
child job reach it through the one `ProtocolSource` the session installed.
What keeps concurrent turns apart is the stream, not a per-job lane — each
`open` returns its own. This remains a pantograph implementation detail
rather than a libpantograph or workflow concept.

Each call to an extension-provided protocol's `open` creates an independently
managed stream, conceptually:

```lua
panto.ext.register_protocol {
  name = "external-runtime",
  open = function(request)
    local transport = open_transport(request)
    return {
      next = function() return transport:next() end,
      cancel_turn = function() transport:cancel_turn() end,
      close = function() transport:close() end,
    }
  end,
  close = function() -- runtime-wide teardown
  end,
}
```

Mutable turn state belongs to the returned stream or its closure, never a
module-global "current session." Multiple streams from one registered
protocol may be open concurrently and interleave between calls. A protocol
body — `open`, or a stream's `next`, `cancel_turn`, or `close` — runs to
completion on the loop thread under `pcall`: it cannot yield, and it must not
block, because that thread is the one every other stream and tool batch needs
back. Awaiting belongs in tool handlers, which do run as coroutines and may
park on luv work; the shared pantograph scheduler remains the sole event-loop
owner.

Cancellation and compaction are scoped to the stream, not the registration.
Abandoning a turn calls that stream's own `cancel_turn`; a protocol that
defines none falls back to the registration-level `cancel_turn`, which is
process-wide by construction. A child compacting its own conversation does
not reset protocol sessions the primary and its siblings are streaming on;
only the session's own teardown runs a protocol's registration-level
`close`.

Every child turn, including a resumed child, receives a fresh protocol stream.
The loaded Panto JSONL conversation is canonical and is present in the open
request; extension protocols cannot depend on private transport history for
continuation. A protocol may serialize the structured messages directly. If
its transport only accepts a textual bootstrap, it can use the existing
generic un-summarized transcript serializer, taking care not to duplicate
system context supplied separately in the request.

## Lua workflow API

Workflows build on the same Agent jobs after v0. The canonical Lua API is
callback- and await-based so a workflow can branch and fan out from structured
child output:

```lua
return subagents.workflow(function(ctx, input)
  local split = ctx:agent {
    agent = "splitter",
    prompt = input,
    output = {
      description = "Return the independent work items.",
      schema = {
        type = "object",
        required = { "items" },
        properties = {
          items = {
            type = "array",
            items = { type = "string" },
          },
        },
      },
    },
  }:await()

  local jobs = {}
  for _, item in ipairs(split.output.items) do
    jobs[#jobs + 1] = ctx:agent {
      agent = "worker",
      prompt = "Process this item:\n" .. item,
    }
  end

  local results = ctx:await(jobs, "all")
  return results
end)
```

`ctx:agent` starts immediately and returns a job handle. `handle:await()`
waits for one. `ctx:await(handles, "all")` returns settled results in input
order. `ctx:await(handles, "first")` returns the first settled result and the
remaining handles. Child failures are result values so the callback can
decide whether to continue, retry differently, or return an error. All jobs
share pantograph's global concurrency bound.

When `output` is present, the child is a one-shot structured worker:

- it receives only one synthetic output tool;
- tool choice is forced to that tool;
- its first model response must call the tool exactly once;
- the tool input is validated against `output.schema`;
- no tool result is sent back and no second model round runs; and
- `result.output` is the decoded tool input, not parsed assistant prose.

One-shot structured workers are ephemeral and not resumable conversations.
They exist to make reliable reducers, classifiers, planners, and dynamic
fan-out inputs without inventing another structured-output protocol.

The module is available directly to human-authored Panto extensions, which
can register their own commands with `panto.ext.register_command`.

## Model-authored transient workflows

A later model-facing tool executes a transient Lua workflow without writing a
definition to disk:

```lua
subagents.lua {
  prompt = "Review the changed subsystems.",
  source = "...Lua returning subagents.workflow(function(ctx, input) ... end)...",
}
```

The source runs as a coroutine in Panto's existing Lua state, but under a
restricted `_ENV` containing the workflow API and safe standard libraries.
It cannot access `os`, `io`, `debug`, `package`, `require`, raw Panto runtime
objects, or extension globals. An instruction budget between yields and a
bounded number of jobs prevent generated Lua from wedging the owner thread or
creating unbounded inference work. Human-authored extensions are trusted Lua
and use the module directly without this restricted environment.

No separate workflow-authoring skill ships initially. The tool schema and
description are sufficient until actual traces show that longer guidance is
needed.

## TOML workflows and slash commands

Persistent TOML workflows are the smaller, fully pre-specified DAG surface.
They are discovered recursively from:

1. `${XDG_CONFIG_HOME:-$HOME/.config}/panto/workflows/**/*.toml`
2. `<cwd>/.panto/workflows/**/*.toml`

Project workflows shadow user workflows by resolved name. A workflow may
define independent roots, fixed dependencies, and per-step call overrides:

```toml
name = "review-chain"
description = "Inspect a change from two angles, then synthesize."

[[steps]]
id = "correctness"
agent = "reviewer"
prompt = "Review for correctness and regressions."

[[steps]]
id = "tests"
agent = "reviewer"
prompt = "Review test coverage and validation gaps."
reasoning = "high"

[[steps]]
id = "synthesis"
agent = "reviewer"
prompt = "Synthesize the findings and remove duplicates."
needs = ["correctness", "tests"]
```

The parser validates duplicate IDs, unknown dependencies, and cycles before
starting work, then lowers the definition onto the same Lua job/await
primitives. Every step receives the original workflow input. A dependent also
receives its dependencies' outputs, clearly labeled and ordered by `needs`.
All ready steps start concurrently. A failed node skips its dependents while
unrelated branches finish. Terminal-node results and failures are returned in
declaration order.

Every discovered workflow registers `/workflow:<name>`. The command tail—the
equivalent of `$ARGUMENTS` in prompt commands—becomes the user prompt supplied
to the DAG's head nodes. Human-authored Lua workflows register any desired
slash command themselves.

The model-facing `subagents.workflow` tool runs either a discovered workflow
by name or a transient static step definition. Output-dependent branches and
dynamic fan-out remain Lua features; TOML does not grow into a programming
language.

## Deferred model-facing background control

Foreground tool batching and workflow-internal awaits cover the initial
parallelism requirements. Background control can be added without changing
the job model:

```lua
subagents.run {
  agent = "researcher",
  prompt = "Investigate this in the background.",
  await = false,
}
-- { id = "0198...", status = "running" }

subagents.await {
  ids = { "0198...", "0199..." },
  mode = "first", -- or "all"
}
```

`first` means the first settled result, successful or not; `all` waits for
every named child. IDs are explicit so a primary never waits on unrelated
work. Results are not consumed by awaiting them. Adding this surface also
requires background result retention, targeted cancellation/status, and TUI
lifecycle behavior, so it is not part of v0.

## Validation

Pantograph checks must prove:

- independent child Agents make concurrent progress under the global bound;
- Lua tool batches from multiple children cannot corrupt results or enter the
  Lua state concurrently;
- built-in and Lua protocol children can run together;
- lane-local `panto.ext.agent` resolves to the correct child;
- progress events remain tagged to the correct child;
- cancellation settles every affected stream and tool batch coherently;
- one failed child does not cancel successful siblings; and
- child tool declarations omit `subagents.*` while shared extension
  activation happens exactly once.

Session checks must prove:

- successful child turns persist as normal JSONL sessions;
- the child ID is the persisted session ID;
- restarting and resuming a primary can continue its referenced child;
- another primary cannot resolve that ID;
- nested child files never appear in the primary session picker or listing;
- the subagent manifest and full conversation round-trip through persistence;
  and
- incomplete first turns do not falsely report durable resumability.

Extension checks must cover profile discovery and shadowing, portable
frontmatter parsing, independent model/reasoning precedence, catalog query
bounding, new-versus-resume validation, and tool-batch concurrency.

Workflow checks, when those phases land, must cover callback awaiting,
structured-tool validation, output-driven fan-out, DAG validation, dependency
prompt assembly, branch-local failure, deterministic terminal ordering, and
generated slash-command registration.

Extension-protocol checks must run multiple mocked streams concurrently and
prove that startup, events, tool calls, cancellation, transcript bootstrap,
and closure stay scoped to the stream that opened them — including that
cancelling one turn leaves its siblings' streams untouched.

## Delivery order

1. Add generic concurrent Agent jobs, global bounding, multiplexed Lua
   dispatch, per-stream protocol concurrency, nested progress, and foreground
   cancellation.
2. Add layered Markdown profiles, model/reasoning resolution, bounded catalog
   queries, durable tree-scoped child stores, continuation, and
   `subagents.run`/`subagents.models`.
3. Add canonical-conversation bootstrap support for extension protocols whose
   transports start without history.
4. Add the callback-based Lua workflow API and one-shot structured workers.
5. Add the restricted model-authored Lua workflow tool.
6. Add TOML DAG loading, `subagents.workflow`, and generated
   `/workflow:<name>` commands.
7. Add model-facing background start/await/status/cancellation only when the
   foreground behavior is proven.

## Not in v0

- cross-primary subagent lookup or continuation;
- recursive subagents;
- background jobs or model-facing await/status tools;
- worktree creation, merge orchestration, or write-conflict protection;
- durable workflow-run state or artifact management;
- agent/profile CRUD tools;
- a workflow-authoring skill; or
- Lua/TOML condition syntax beyond the callback and fixed-DAG surfaces above.