path: root/DESIGN.md

# panto-subagents design

## Product shape

`panto-subagents` is a separate rock exposing one extension named
`subagents`. Its model-facing tools use the `subagents.*` namespace. It does
not shell out to `panto -p`; every child is an ordinary in-process
libpantograph `Agent` driven by pantograph.

The primary owns delegation. Children share its workspace, but not its
dialogue, and cannot call `subagents.*`. Recursive delegation, worktree
management, and merge orchestration are outside this design.

## v0 tools

### `subagents.run`

One tool call starts or resumes one conversational child:

```lua
-- Start a new child.
{
  agent = "reviewer",
  prompt = "Review the authentication changes.",
  model = "anthropic:sonnet", -- optional; always provider:model
  reasoning = "high",        -- optional
}

-- Resume an existing child owned by this primary session.
{
  id = "0198...",
  prompt = "Now focus on the missing tests.",
  model = "openai:gpt-5.6", -- optional per-turn change
  reasoning = "xhigh",      -- optional per-turn change
}
```

`prompt` must be non-empty. Exactly one of `agent` and `id` is required:
`agent` selects the profile for a new child; `id` identifies a persisted child
conversation. An ID cannot be resumed by a different primary session, and
only one turn may be active for an ID.

Once a valid new child has been allocated, every settled result includes its
ID:

```lua
{
  id = "0198...",
  agent = "reviewer",
  status = "completed", -- completed, failed, or cancelled
  output = "...",       -- present on success
  error = nil,          -- present on failure
  resumable = true,
}
```

Validation failures that happen before a child is allocated have no ID.
Under the existing filesystem-store flush discipline, a new child that fails
before producing its first assistant message has no durable file; such a
result reports `resumable = false` rather than promising continuation.

Parallel delegation uses ordinary model tool batching. The primary emits
several `subagents.run` calls in one batch; pantograph starts them concurrently
under one session-wide bound, initially four. A failure in one call does not
discard successful siblings. There is no second `tasks` array, task ID, or
extension-specific batch report format.

### `subagents.models`

The model catalog is too large to place in `subagents.run`'s description or
JSON schema. A small query tool exposes it on demand:

```lua
subagents.models {}
-- Current inherited model/reasoning plus configured provider names and counts.

subagents.models {
  provider = "anthropic",
  query = "sonnet",
  limit = 10,
}

subagents.models { model = "anthropic:sonnet" }
subagents.models { agent = "reviewer" }
```

An exact model or agent lookup returns its effective model, default reasoning,
and the reasoning levels known to be valid for it. A search returns only the
requested bounded matches and says when more exist. The tool reuses
pantograph's model registry and provider-specific reasoning-option logic,
including effort choices reported dynamically by Lua protocols.

The `subagents.run` description tells the primary to omit overrides normally
and call `subagents.models` before choosing an unfamiliar model or reasoning
level. Runtime validation remains authoritative when catalog capability data
is incomplete.

## Agent profiles

Profiles are Markdown files with YAML frontmatter, compatible at the common
format boundary used by other agent harnesses:

```markdown
---
name: reviewer
description: Reviews changes for correctness and missing tests
model: anthropic:sonnet
reasoning: high
---

You are a focused code reviewer.

Report only concrete, evidence-backed findings.
```

They are discovered recursively from two layers, lowest precedence first:

1. `${XDG_CONFIG_HOME:-$HOME/.config}/panto/agents/**/*.md`
2. `<cwd>/.panto/agents/**/*.md`

The project definition shadows a user definition with the same resolved name.
`name` is optional and defaults to the extension-less filename.
`description` is included with the discovered names in `subagents.run`'s tool
description so the primary can choose a profile without loading its full
prompt.

The portable contract is `name`, `description`, and the Markdown body as the
profile's system prompt. Panto additionally recognizes `model` only in full
`provider:model` form and recognizes its existing reasoning labels. Unknown
frontmatter is ignored. A foreign model spelling that is not valid Panto
syntax is ignored with a warning so the prompt remains usable and inherits
the primary model.

For a new child, model and reasoning are resolved independently:

```text
model     = tool.model     ?? profile.model     ?? primary.model
reasoning = tool.reasoning ?? profile.reasoning ?? primary.reasoning
```

Pantograph resolves the selected model against the configured provider and
model registries, applies the reasoning level, and fails before inference if
the known combination is invalid. The Lua extension never receives provider
credentials.

The resolved model reference and reasoning label are recorded on each child
turn. On continuation, omitted overrides keep the child's last effective
values; an explicit `model` or `reasoning` changes that value for the new turn
and for later continuations.

## Conversation construction

A new child receives a fresh `Conversation` containing:

1. the primary's effective system and project context;
2. a fixed child-role instruction;
3. the selected profile's Markdown body as a separate system message; and
4. the tool's `prompt` as its only initial user message.

The parent dialogue is not copied. The primary must put task-specific context
in the user prompt.

The profile system message carries opaque, non-model-visible message metadata
recording the immutable subagent manifest: its owning primary session ID and
profile name. Each user message carries the full model reference and reasoning
label effective for that turn. This avoids a mutable sidecar manifest: resume
reads identity from the first system message and defaults from the latest user
message. The existing filesystem-store load path must preserve per-message
metadata when rebuilding a `Conversation`.

The stored conversation is canonical on resume. Profile edits affect new
children, not an existing child ID.

Every child sees the parent's normal registered tools except `subagents.*`.
Tools still run through the shared extension runtime. Children share the
workspace, so concurrent writers can race or conflict; the primary must
delegate independent work.

## Durable child sessions

Subagent conversations are ordinary `FileSystemJSONLStore` sessions, not an
in-memory `id -> Conversation` map. The public subagent ID is the actual Panto
session UUID and the JSONL filename stem.

Each primary session owns a separate child catalog below its already-resolved
per-cwd session directory:

```text
<session-dir>/
├── <primary-session-id>.jsonl
└── subagents/
    └── <primary-session-id>/
        ├── <subagent-id>.jsonl
        └── <subagent-id>.jsonl
```

This layout automatically follows `PANTO_SESSION_DIR` and the normal XDG
fallback. The main store's `list` and `resolve` operations examine only direct
`.jsonl` children, so nested subagent files never appear in `panto sessions`
or the `/resume` picker.

The primary session ID is the ownership boundary. After panto restarts and
the user resumes a primary conversation, extension activation derives the
same child-catalog path from that primary ID. A referenced child ID therefore
resolves again. A different primary uses a different directory and cannot
resolve it. There is no global child lookup and no cross-primary continuation.

Starting and resuming use the ordinary session lifecycle:

```text
new:
  session = child_store.create()
  agent = Agent.init(session, no prior conversation)
  seed system/profile messages
  run prompt

resume:
  session = child_store.resolve(id)
  conversation = session.load()
  agent = Agent.init(session, conversation)
  run prompt
```

The Agent appends through its `Session` handle, and the filesystem store syncs
completed entries normally. The Agent may be destroyed after a turn; a later
turn reconstructs the conversation from JSONL. Only active job handles,
progress, and cancellation state remain in memory.

## Foreground progress and cancellation

Silent foreground waiting is not acceptable. Each child job has its ID and
profile before inference starts and emits tagged progress events to the TUI.
The user can see the child's assistant stream, tool calls, and tool results
while the primary model remains blocked on the tool call. Concurrent children
remain visually distinguishable.

These nested events are presentation data, not partial tool results: they are
not inserted into the primary model's conversation. `subagents.run` returns
only the settled result, while the full child conversation is persisted under
its ID for inspection and continuation.

Escape cancels every still-running child belonging to the current foreground
tool operation. Completed siblings remain successful. Cancellation targets
the child streams and protocol lanes, then lets every affected tool call
settle coherently as `cancelled` rather than abandoning the parent's batch.

## Generic pantograph child-job seam

libpantograph remains unaware of Lua, extensions, and subagents. Its `Agent`
and `SessionStore` abstractions already provide the conversation and
persistence machinery. No subagent scheduler or tree policy belongs in the
library; generic metadata round-trip fixes and conversation helpers are
acceptable where required.

Pantograph supplies a generic asynchronous Agent job to Lua, conceptually:

```lua
local job = panto.ext.spawn_agent {
  prompt = "Inspect the auth flow.",
  store = child_store,
  session_id = prior_id, -- nil creates a new store session
  system_messages = initial_system_messages, -- new sessions only
  model = "anthropic:sonnet",
  reasoning = "high",
}

local result = job:await()
-- result.id, result.status, result.text, result.error, result.resumable
```

`spawn_agent` starts immediately. `await` yields its Lua coroutine. The host,
not the extension, supplies the resolved provider configuration, inherited
tool and protocol proxies, worker lifecycle, progress routing, cancellation,
and the global concurrency bound. The store userdata remains pinned until its
jobs settle.

Pantograph also exposes the current primary session ID and resolved per-cwd
session directory to the extension. It can then construct the nested child
store without duplicating XDG, `PANTO_SESSION_DIR`, or cwd-encoding logic.

## Shared Lua runtime executor

All agents in one primary session share its Lua state, activated extensions,
and luv loop. Children do not create Lua states, reload rocks, or reactivate
extensions.

The current runtime's singular in-flight `current_batch` must become a
multiplexed executor:

1. Parent tool handlers start child jobs and yield.
2. Child workers drive ordinary libpantograph streams.
3. Built-in providers run directly on those workers.
4. A child Lua tool or Lua protocol call is posted to a thread-safe runtime
   queue, waking the shared loop through a pantograph-owned `uv_async_t`.
5. The runtime executes the callback as a coroutine on the sole Lua owner
   thread; async extension code may yield normally.
6. Completion returns to the waiting child worker, while unrelated lanes keep
   making progress.

Tool batches carry explicit identity; result recording cannot consult one
global current batch. Every proxy source is lane-bound, and `panto.ext.agent`
resolves to the responsible Agent while that lane's coroutine runs. Module
globals remain shared intentionally. A filtered declaration view omits
`subagents.*` from children without reloading extensions.

## Extension-provided protocol concurrency

Every Agent job receives a protocol lane. A lane-bound `ProtocolSource` proxy
routes stream pulls, cancellation, and closure to that job. This remains a
pantograph implementation detail rather than a libpantograph or workflow
concept.

Each call to an extension-provided protocol's `open` creates an independently
managed stream, conceptually:

```lua
panto.ext.register_protocol {
  name = "external-runtime",
  open = function(request)
    local transport = open_transport(request)
    return {
      next = function() return transport:next() end,
      cancel_turn = function() transport:cancel_turn() end,
      close = function() transport:close() end,
    }
  end,
  close = function() -- runtime-wide teardown
  end,
}
```

Mutable turn state belongs to the returned stream or its closure, never a
module-global "current session." Multiple streams from one registered
protocol may be open and yield concurrently. Protocol extensions use
coroutine waits rather than nested event-loop runs; the shared pantograph
scheduler remains the sole event-loop owner.

Every child turn, including a resumed child, receives a fresh protocol stream.
The loaded Panto JSONL conversation is canonical and is present in the open
request; extension protocols cannot depend on private transport history for
continuation. A protocol may serialize the structured messages directly. If
its transport only accepts a textual bootstrap, it can use the existing
generic un-summarized transcript serializer, taking care not to duplicate
system context supplied separately in the request.

## Lua workflow API

Workflows build on the same Agent jobs after v0. The canonical Lua API is
callback- and await-based so a workflow can branch and fan out from structured
child output:

```lua
return subagents.workflow(function(ctx, input)
  local split = ctx:agent {
    agent = "splitter",
    prompt = input,
    output = {
      description = "Return the independent work items.",
      schema = {
        type = "object",
        required = { "items" },
        properties = {
          items = {
            type = "array",
            items = { type = "string" },
          },
        },
      },
    },
  }:await()

  local jobs = {}
  for _, item in ipairs(split.output.items) do
    jobs[#jobs + 1] = ctx:agent {
      agent = "worker",
      prompt = "Process this item:\n" .. item,
    }
  end

  local results = ctx:await(jobs, "all")
  return results
end)
```

`ctx:agent` starts immediately and returns a job handle. `handle:await()`
waits for one. `ctx:await(handles, "all")` returns settled results in input
order. `ctx:await(handles, "first")` returns the first settled result and the
remaining handles. Child failures are result values so the callback can
decide whether to continue, retry differently, or return an error. All jobs
share pantograph's global concurrency bound.

When `output` is present, the child is a one-shot structured worker:

- it receives only one synthetic output tool;
- tool choice is forced to that tool;
- its first model response must call the tool exactly once;
- the tool input is validated against `output.schema`;
- no tool result is sent back and no second model round runs; and
- `result.output` is the decoded tool input, not parsed assistant prose.

One-shot structured workers are ephemeral and not resumable conversations.
They exist to make reliable reducers, classifiers, planners, and dynamic
fan-out inputs without inventing another structured-output protocol.

The module is available directly to human-authored Panto extensions, which
can register their own commands with `panto.ext.register_command`.

## Model-authored transient workflows

A later model-facing tool executes a transient Lua workflow without writing a
definition to disk:

```lua
subagents.lua {
  prompt = "Review the changed subsystems.",
  source = "...Lua returning subagents.workflow(function(ctx, input) ... end)...",
}
```

The source runs as a coroutine in Panto's existing Lua state, but under a
restricted `_ENV` containing the workflow API and safe standard libraries.
It cannot access `os`, `io`, `debug`, `package`, `require`, raw Panto runtime
objects, or extension globals. An instruction budget between yields and a
bounded number of jobs prevent generated Lua from wedging the owner thread or
creating unbounded inference work. Human-authored extensions are trusted Lua
and use the module directly without this restricted environment.

No separate workflow-authoring skill ships initially. The tool schema and
description are sufficient until actual traces show that longer guidance is
needed.

## TOML workflows and slash commands

Persistent TOML workflows are the smaller, fully pre-specified DAG surface.
They are discovered recursively from:

1. `${XDG_CONFIG_HOME:-$HOME/.config}/panto/workflows/**/*.toml`
2. `<cwd>/.panto/workflows/**/*.toml`

Project workflows shadow user workflows by resolved name. A workflow may
define independent roots, fixed dependencies, and per-step call overrides:

```toml
name = "review-chain"
description = "Inspect a change from two angles, then synthesize."

[[steps]]
id = "correctness"
agent = "reviewer"
prompt = "Review for correctness and regressions."

[[steps]]
id = "tests"
agent = "reviewer"
prompt = "Review test coverage and validation gaps."
reasoning = "high"

[[steps]]
id = "synthesis"
agent = "reviewer"
prompt = "Synthesize the findings and remove duplicates."
needs = ["correctness", "tests"]
```

The parser validates duplicate IDs, unknown dependencies, and cycles before
starting work, then lowers the definition onto the same Lua job/await
primitives. Every step receives the original workflow input. A dependent also
receives its dependencies' outputs, clearly labeled and ordered by `needs`.
All ready steps start concurrently. A failed node skips its dependents while
unrelated branches finish. Terminal-node results and failures are returned in
declaration order.

Every discovered workflow registers `/workflow:<name>`. The command tail—the
equivalent of `$ARGUMENTS` in prompt commands—becomes the user prompt supplied
to the DAG's head nodes. Human-authored Lua workflows register any desired
slash command themselves.

The model-facing `subagents.workflow` tool runs either a discovered workflow
by name or a transient static step definition. Output-dependent branches and
dynamic fan-out remain Lua features; TOML does not grow into a programming
language.

## Deferred model-facing background control

Foreground tool batching and workflow-internal awaits cover the initial
parallelism requirements. Background control can be added without changing
the job model:

```lua
subagents.run {
  agent = "researcher",
  prompt = "Investigate this in the background.",
  await = false,
}
-- { id = "0198...", status = "running" }

subagents.await {
  ids = { "0198...", "0199..." },
  mode = "first", -- or "all"
}
```

`first` means the first settled result, successful or not; `all` waits for
every named child. IDs are explicit so a primary never waits on unrelated
work. Results are not consumed by awaiting them. Adding this surface also
requires background result retention, targeted cancellation/status, and TUI
lifecycle behavior, so it is not part of v0.

## Validation

Pantograph checks must prove:

- independent child Agents make concurrent progress under the global bound;
- Lua tool batches from multiple children cannot corrupt results or enter the
  Lua state concurrently;
- built-in and Lua protocol children can run together;
- lane-local `panto.ext.agent` resolves to the correct child;
- progress events remain tagged to the correct child;
- cancellation settles every affected stream and lane coherently;
- one failed child does not cancel successful siblings; and
- child tool declarations omit `subagents.*` while shared extension
  activation happens exactly once.

Session checks must prove:

- successful child turns persist as normal JSONL sessions;
- the child ID is the persisted session ID;
- restarting and resuming a primary can continue its referenced child;
- another primary cannot resolve that ID;
- nested child files never appear in the primary session picker or listing;
- the subagent manifest and full conversation round-trip through persistence;
  and
- incomplete first turns do not falsely report durable resumability.

Extension checks must cover profile discovery and shadowing, portable
frontmatter parsing, independent model/reasoning precedence, catalog query
bounding, new-versus-resume validation, and tool-batch concurrency.

Workflow checks, when those phases land, must cover callback awaiting,
structured-tool validation, output-driven fan-out, DAG validation, dependency
prompt assembly, branch-local failure, deterministic terminal ordering, and
generated slash-command registration.

Extension-protocol checks must run multiple mocked streams concurrently and
prove that startup, events, tool calls, cancellation, transcript bootstrap,
and closure remain lane-local.

## Delivery order

1. Add generic concurrent Agent jobs, global bounding, multiplexed Lua
   dispatch, protocol lanes, nested progress, and foreground cancellation.
2. Add layered Markdown profiles, model/reasoning resolution, bounded catalog
   queries, durable tree-scoped child stores, continuation, and
   `subagents.run`/`subagents.models`.
3. Add canonical-conversation bootstrap support for extension protocols whose
   transports start without history.
4. Add the callback-based Lua workflow API and one-shot structured workers.
5. Add the restricted model-authored Lua workflow tool.
6. Add TOML DAG loading, `subagents.workflow`, and generated
   `/workflow:<name>` commands.
7. Add model-facing background start/await/status/cancellation only when the
   foreground behavior is proven.

## Not in v0

- cross-primary subagent lookup or continuation;
- recursive subagents;
- background jobs or model-facing await/status tools;
- worktree creation, merge orchestration, or write-conflict protection;
- durable workflow-run state or artifact management;
- agent/profile CRUD tools;
- a workflow-authoring skill; or
- Lua/TOML condition syntax beyond the callback and fixed-DAG surfaces above.