diff options
| -rw-r--r-- | DESIGN.md | 573 | ||||
| -rw-r--r-- | README.md | 90 |
2 files changed, 663 insertions, 0 deletions
diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..3615eb2 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,573 @@ +# panto-subagents design + +## Product shape + +`panto-subagents` is a separate rock exposing one extension named +`subagents`. Its model-facing tools use the `subagents.*` namespace. It does +not shell out to `panto -p`; every child is an ordinary in-process +libpantograph `Agent` driven by pantograph. + +The primary owns delegation. Children share its workspace, but not its +dialogue, and cannot call `subagents.*`. Recursive delegation, worktree +management, and merge orchestration are outside this design. + +## v0 tools + +### `subagents.run` + +One tool call starts or resumes one conversational child: + +```lua +-- Start a new child. +{ + agent = "reviewer", + prompt = "Review the authentication changes.", + model = "anthropic:sonnet", -- optional; always provider:model + reasoning = "high", -- optional +} + +-- Resume an existing child owned by this primary session. +{ + id = "0198...", + prompt = "Now focus on the missing tests.", + model = "openai:gpt-5.6", -- optional per-turn change + reasoning = "xhigh", -- optional per-turn change +} +``` + +`prompt` must be non-empty. Exactly one of `agent` and `id` is required: +`agent` selects the profile for a new child; `id` identifies a persisted child +conversation. An ID cannot be resumed by a different primary session, and +only one turn may be active for an ID. + +Once a valid new child has been allocated, every settled result includes its +ID: + +```lua +{ + id = "0198...", + agent = "reviewer", + status = "completed", -- completed, failed, or cancelled + output = "...", -- present on success + error = nil, -- present on failure + resumable = true, +} +``` + +Validation failures that happen before a child is allocated have no ID. +Under the existing filesystem-store flush discipline, a new child that fails +before producing its first assistant message has no durable file; such a +result reports `resumable = false` rather than promising continuation. + +Parallel delegation uses ordinary model tool batching. The primary emits +several `subagents.run` calls in one batch; pantograph starts them concurrently +under one session-wide bound, initially four. A failure in one call does not +discard successful siblings. There is no second `tasks` array, task ID, or +extension-specific batch report format. + +### `subagents.models` + +The model catalog is too large to place in `subagents.run`'s description or +JSON schema. A small query tool exposes it on demand: + +```lua +subagents.models {} +-- Current inherited model/reasoning plus configured provider names and counts. + +subagents.models { + provider = "anthropic", + query = "sonnet", + limit = 10, +} + +subagents.models { model = "anthropic:sonnet" } +subagents.models { agent = "reviewer" } +``` + +An exact model or agent lookup returns its effective model, default reasoning, +and the reasoning levels known to be valid for it. A search returns only the +requested bounded matches and says when more exist. The tool reuses +pantograph's model registry and provider-specific reasoning-option logic, +including effort choices reported dynamically by Lua protocols. + +The `subagents.run` description tells the primary to omit overrides normally +and call `subagents.models` before choosing an unfamiliar model or reasoning +level. Runtime validation remains authoritative when catalog capability data +is incomplete. + +## Agent profiles + +Profiles are Markdown files with YAML frontmatter, compatible at the common +format boundary used by other agent harnesses: + +```markdown +--- +name: reviewer +description: Reviews changes for correctness and missing tests +model: anthropic:sonnet +reasoning: high +--- + +You are a focused code reviewer. + +Report only concrete, evidence-backed findings. +``` + +They are discovered recursively from two layers, lowest precedence first: + +1. `${XDG_CONFIG_HOME:-$HOME/.config}/panto/agents/**/*.md` +2. `<cwd>/.panto/agents/**/*.md` + +The project definition shadows a user definition with the same resolved name. +`name` is optional and defaults to the extension-less filename. +`description` is included with the discovered names in `subagents.run`'s tool +description so the primary can choose a profile without loading its full +prompt. + +The portable contract is `name`, `description`, and the Markdown body as the +profile's system prompt. Panto additionally recognizes `model` only in full +`provider:model` form and recognizes its existing reasoning labels. Unknown +frontmatter is ignored. A foreign model spelling that is not valid Panto +syntax is ignored with a warning so the prompt remains usable and inherits +the primary model. + +For a new child, model and reasoning are resolved independently: + +```text +model = tool.model ?? profile.model ?? primary.model +reasoning = tool.reasoning ?? profile.reasoning ?? primary.reasoning +``` + +Pantograph resolves the selected model against the configured provider and +model registries, applies the reasoning level, and fails before inference if +the known combination is invalid. The Lua extension never receives provider +credentials. + +The resolved model reference and reasoning label are recorded on each child +turn. On continuation, omitted overrides keep the child's last effective +values; an explicit `model` or `reasoning` changes that value for the new turn +and for later continuations. + +## Conversation construction + +A new child receives a fresh `Conversation` containing: + +1. the primary's effective system and project context; +2. a fixed child-role instruction; +3. the selected profile's Markdown body as a separate system message; and +4. the tool's `prompt` as its only initial user message. + +The parent dialogue is not copied. The primary must put task-specific context +in the user prompt. + +The profile system message carries opaque, non-model-visible message metadata +recording the immutable subagent manifest: its owning primary session ID and +profile name. Each user message carries the full model reference and reasoning +label effective for that turn. This avoids a mutable sidecar manifest: resume +reads identity from the first system message and defaults from the latest user +message. The existing filesystem-store load path must preserve per-message +metadata when rebuilding a `Conversation`. + +The stored conversation is canonical on resume. Profile edits affect new +children, not an existing child ID. + +Every child sees the parent's normal registered tools except `subagents.*`. +Tools still run through the shared extension runtime. Children share the +workspace, so concurrent writers can race or conflict; the primary must +delegate independent work. + +## Durable child sessions + +Subagent conversations are ordinary `FileSystemJSONLStore` sessions, not an +in-memory `id -> Conversation` map. The public subagent ID is the actual Panto +session UUID and the JSONL filename stem. + +Each primary session owns a separate child catalog below its already-resolved +per-cwd session directory: + +```text +<session-dir>/ +├── <primary-session-id>.jsonl +└── subagents/ + └── <primary-session-id>/ + ├── <subagent-id>.jsonl + └── <subagent-id>.jsonl +``` + +This layout automatically follows `PANTO_SESSION_DIR` and the normal XDG +fallback. The main store's `list` and `resolve` operations examine only direct +`.jsonl` children, so nested subagent files never appear in `panto sessions` +or the `/resume` picker. + +The primary session ID is the ownership boundary. After panto restarts and +the user resumes a primary conversation, extension activation derives the +same child-catalog path from that primary ID. A referenced child ID therefore +resolves again. A different primary uses a different directory and cannot +resolve it. There is no global child lookup and no cross-primary continuation. + +Starting and resuming use the ordinary session lifecycle: + +```text +new: + session = child_store.create() + agent = Agent.init(session, no prior conversation) + seed system/profile messages + run prompt + +resume: + session = child_store.resolve(id) + conversation = session.load() + agent = Agent.init(session, conversation) + run prompt +``` + +The Agent appends through its `Session` handle, and the filesystem store syncs +completed entries normally. The Agent may be destroyed after a turn; a later +turn reconstructs the conversation from JSONL. Only active job handles, +progress, and cancellation state remain in memory. + +## Foreground progress and cancellation + +Silent foreground waiting is not acceptable. Each child job has its ID and +profile before inference starts and emits tagged progress events to the TUI. +The user can see the child's assistant stream, tool calls, and tool results +while the primary model remains blocked on the tool call. Concurrent children +remain visually distinguishable. + +These nested events are presentation data, not partial tool results: they are +not inserted into the primary model's conversation. `subagents.run` returns +only the settled result, while the full child conversation is persisted under +its ID for inspection and continuation. + +Escape cancels every still-running child belonging to the current foreground +tool operation. Completed siblings remain successful. Cancellation targets +the child streams and protocol lanes, then lets every affected tool call +settle coherently as `cancelled` rather than abandoning the parent's batch. + +## Generic pantograph child-job seam + +libpantograph remains unaware of Lua, extensions, and subagents. Its `Agent` +and `SessionStore` abstractions already provide the conversation and +persistence machinery. No subagent scheduler or tree policy belongs in the +library; generic metadata round-trip fixes and conversation helpers are +acceptable where required. + +Pantograph supplies a generic asynchronous Agent job to Lua, conceptually: + +```lua +local job = panto.ext.spawn_agent { + prompt = "Inspect the auth flow.", + store = child_store, + session_id = prior_id, -- nil creates a new store session + system_messages = initial_system_messages, -- new sessions only + model = "anthropic:sonnet", + reasoning = "high", +} + +local result = job:await() +-- result.id, result.status, result.text, result.error, result.resumable +``` + +`spawn_agent` starts immediately. `await` yields its Lua coroutine. The host, +not the extension, supplies the resolved provider configuration, inherited +tool and protocol proxies, worker lifecycle, progress routing, cancellation, +and the global concurrency bound. The store userdata remains pinned until its +jobs settle. + +Pantograph also exposes the current primary session ID and resolved per-cwd +session directory to the extension. It can then construct the nested child +store without duplicating XDG, `PANTO_SESSION_DIR`, or cwd-encoding logic. + +## Shared Lua runtime executor + +All agents in one primary session share its Lua state, activated extensions, +and luv loop. Children do not create Lua states, reload rocks, or reactivate +extensions. + +The current runtime's singular in-flight `current_batch` must become a +multiplexed executor: + +1. Parent tool handlers start child jobs and yield. +2. Child workers drive ordinary libpantograph streams. +3. Built-in providers run directly on those workers. +4. A child Lua tool or Lua protocol call is posted to a thread-safe runtime + queue, waking the shared loop through a pantograph-owned `uv_async_t`. +5. The runtime executes the callback as a coroutine on the sole Lua owner + thread; async extension code may yield normally. +6. Completion returns to the waiting child worker, while unrelated lanes keep + making progress. + +Tool batches carry explicit identity; result recording cannot consult one +global current batch. Every proxy source is lane-bound, and `panto.ext.agent` +resolves to the responsible Agent while that lane's coroutine runs. Module +globals remain shared intentionally. A filtered declaration view omits +`subagents.*` from children without reloading extensions. + +## Extension-provided protocol concurrency + +Every Agent job receives a protocol lane. A lane-bound `ProtocolSource` proxy +routes stream pulls, cancellation, and closure to that job. This remains a +pantograph implementation detail rather than a libpantograph or workflow +concept. + +Each call to an extension-provided protocol's `open` creates an independently +managed stream, conceptually: + +```lua +panto.ext.register_protocol { + name = "external-runtime", + open = function(request) + local transport = open_transport(request) + return { + next = function() return transport:next() end, + cancel_turn = function() transport:cancel_turn() end, + close = function() transport:close() end, + } + end, + close = function() -- runtime-wide teardown + end, +} +``` + +Mutable turn state belongs to the returned stream or its closure, never a +module-global "current session." Multiple streams from one registered +protocol may be open and yield concurrently. Protocol extensions use +coroutine waits rather than nested event-loop runs; the shared pantograph +scheduler remains the sole event-loop owner. + +Every child turn, including a resumed child, receives a fresh protocol stream. +The loaded Panto JSONL conversation is canonical and is present in the open +request; extension protocols cannot depend on private transport history for +continuation. A protocol may serialize the structured messages directly. If +its transport only accepts a textual bootstrap, it can use the existing +generic un-summarized transcript serializer, taking care not to duplicate +system context supplied separately in the request. + +## Lua workflow API + +Workflows build on the same Agent jobs after v0. The canonical Lua API is +callback- and await-based so a workflow can branch and fan out from structured +child output: + +```lua +return subagents.workflow(function(ctx, input) + local split = ctx:agent { + agent = "splitter", + prompt = input, + output = { + description = "Return the independent work items.", + schema = { + type = "object", + required = { "items" }, + properties = { + items = { + type = "array", + items = { type = "string" }, + }, + }, + }, + }, + }:await() + + local jobs = {} + for _, item in ipairs(split.output.items) do + jobs[#jobs + 1] = ctx:agent { + agent = "worker", + prompt = "Process this item:\n" .. item, + } + end + + local results = ctx:await(jobs, "all") + return results +end) +``` + +`ctx:agent` starts immediately and returns a job handle. `handle:await()` +waits for one. `ctx:await(handles, "all")` returns settled results in input +order. `ctx:await(handles, "first")` returns the first settled result and the +remaining handles. Child failures are result values so the callback can +decide whether to continue, retry differently, or return an error. All jobs +share pantograph's global concurrency bound. + +When `output` is present, the child is a one-shot structured worker: + +- it receives only one synthetic output tool; +- tool choice is forced to that tool; +- its first model response must call the tool exactly once; +- the tool input is validated against `output.schema`; +- no tool result is sent back and no second model round runs; and +- `result.output` is the decoded tool input, not parsed assistant prose. + +One-shot structured workers are ephemeral and not resumable conversations. +They exist to make reliable reducers, classifiers, planners, and dynamic +fan-out inputs without inventing another structured-output protocol. + +The module is available directly to human-authored Panto extensions, which +can register their own commands with `panto.ext.register_command`. + +## Model-authored transient workflows + +A later model-facing tool executes a transient Lua workflow without writing a +definition to disk: + +```lua +subagents.lua { + prompt = "Review the changed subsystems.", + source = "...Lua returning subagents.workflow(function(ctx, input) ... end)...", +} +``` + +The source runs as a coroutine in Panto's existing Lua state, but under a +restricted `_ENV` containing the workflow API and safe standard libraries. +It cannot access `os`, `io`, `debug`, `package`, `require`, raw Panto runtime +objects, or extension globals. An instruction budget between yields and a +bounded number of jobs prevent generated Lua from wedging the owner thread or +creating unbounded inference work. Human-authored extensions are trusted Lua +and use the module directly without this restricted environment. + +No separate workflow-authoring skill ships initially. The tool schema and +description are sufficient until actual traces show that longer guidance is +needed. + +## TOML workflows and slash commands + +Persistent TOML workflows are the smaller, fully pre-specified DAG surface. +They are discovered recursively from: + +1. `${XDG_CONFIG_HOME:-$HOME/.config}/panto/workflows/**/*.toml` +2. `<cwd>/.panto/workflows/**/*.toml` + +Project workflows shadow user workflows by resolved name. A workflow may +define independent roots, fixed dependencies, and per-step call overrides: + +```toml +name = "review-chain" +description = "Inspect a change from two angles, then synthesize." + +[[steps]] +id = "correctness" +agent = "reviewer" +prompt = "Review for correctness and regressions." + +[[steps]] +id = "tests" +agent = "reviewer" +prompt = "Review test coverage and validation gaps." +reasoning = "high" + +[[steps]] +id = "synthesis" +agent = "reviewer" +prompt = "Synthesize the findings and remove duplicates." +needs = ["correctness", "tests"] +``` + +The parser validates duplicate IDs, unknown dependencies, and cycles before +starting work, then lowers the definition onto the same Lua job/await +primitives. Every step receives the original workflow input. A dependent also +receives its dependencies' outputs, clearly labeled and ordered by `needs`. +All ready steps start concurrently. A failed node skips its dependents while +unrelated branches finish. Terminal-node results and failures are returned in +declaration order. + +Every discovered workflow registers `/workflow:<name>`. The command tail—the +equivalent of `$ARGUMENTS` in prompt commands—becomes the user prompt supplied +to the DAG's head nodes. Human-authored Lua workflows register any desired +slash command themselves. + +The model-facing `subagents.workflow` tool runs either a discovered workflow +by name or a transient static step definition. Output-dependent branches and +dynamic fan-out remain Lua features; TOML does not grow into a programming +language. + +## Deferred model-facing background control + +Foreground tool batching and workflow-internal awaits cover the initial +parallelism requirements. Background control can be added without changing +the job model: + +```lua +subagents.run { + agent = "researcher", + prompt = "Investigate this in the background.", + await = false, +} +-- { id = "0198...", status = "running" } + +subagents.await { + ids = { "0198...", "0199..." }, + mode = "first", -- or "all" +} +``` + +`first` means the first settled result, successful or not; `all` waits for +every named child. IDs are explicit so a primary never waits on unrelated +work. Results are not consumed by awaiting them. Adding this surface also +requires background result retention, targeted cancellation/status, and TUI +lifecycle behavior, so it is not part of v0. + +## Validation + +Pantograph checks must prove: + +- independent child Agents make concurrent progress under the global bound; +- Lua tool batches from multiple children cannot corrupt results or enter the + Lua state concurrently; +- built-in and Lua protocol children can run together; +- lane-local `panto.ext.agent` resolves to the correct child; +- progress events remain tagged to the correct child; +- cancellation settles every affected stream and lane coherently; +- one failed child does not cancel successful siblings; and +- child tool declarations omit `subagents.*` while shared extension + activation happens exactly once. + +Session checks must prove: + +- successful child turns persist as normal JSONL sessions; +- the child ID is the persisted session ID; +- restarting and resuming a primary can continue its referenced child; +- another primary cannot resolve that ID; +- nested child files never appear in the primary session picker or listing; +- the subagent manifest and full conversation round-trip through persistence; + and +- incomplete first turns do not falsely report durable resumability. + +Extension checks must cover profile discovery and shadowing, portable +frontmatter parsing, independent model/reasoning precedence, catalog query +bounding, new-versus-resume validation, and tool-batch concurrency. + +Workflow checks, when those phases land, must cover callback awaiting, +structured-tool validation, output-driven fan-out, DAG validation, dependency +prompt assembly, branch-local failure, deterministic terminal ordering, and +generated slash-command registration. + +Extension-protocol checks must run multiple mocked streams concurrently and +prove that startup, events, tool calls, cancellation, transcript bootstrap, +and closure remain lane-local. + +## Delivery order + +1. Add generic concurrent Agent jobs, global bounding, multiplexed Lua + dispatch, protocol lanes, nested progress, and foreground cancellation. +2. Add layered Markdown profiles, model/reasoning resolution, bounded catalog + queries, durable tree-scoped child stores, continuation, and + `subagents.run`/`subagents.models`. +3. Add canonical-conversation bootstrap support for extension protocols whose + transports start without history. +4. Add the callback-based Lua workflow API and one-shot structured workers. +5. Add the restricted model-authored Lua workflow tool. +6. Add TOML DAG loading, `subagents.workflow`, and generated + `/workflow:<name>` commands. +7. Add model-facing background start/await/status/cancellation only when the + foreground behavior is proven. + +## Not in v0 + +- cross-primary subagent lookup or continuation; +- recursive subagents; +- background jobs or model-facing await/status tools; +- worktree creation, merge orchestration, or write-conflict protection; +- durable workflow-run state or artifact management; +- agent/profile CRUD tools; +- a workflow-authoring skill; or +- Lua/TOML condition syntax beyond the callback and fixed-DAG surfaces above. diff --git a/README.md b/README.md new file mode 100644 index 0000000..8c24c36 --- /dev/null +++ b/README.md @@ -0,0 +1,90 @@ +# panto-subagents + +Delegation and workflows for [Pantograph](https://github.com/travisp/pantograph). +The `subagents` extension lets a primary agent start specialized child agents, +run several at once, and continue their conversations later. + +## Install + +```toml +[extensions] +rocks = ["panto-subagents"] +``` + +For a local checkout, use `paths = ["/path/to/panto-subagents"]` instead. + +## Agent profiles + +Agents are Markdown files with YAML frontmatter: + +```markdown +--- +name: reviewer +description: Reviews changes for correctness and missing tests +model: anthropic:sonnet +reasoning: high +--- + +You are a focused code reviewer. Report only concrete findings. +``` + +Profiles are loaded recursively from: + +- `${XDG_CONFIG_HOME:-$HOME/.config}/panto/agents/` +- `.panto/agents/` in the current project + +Project profiles override user profiles with the same name. `name` defaults to +the filename. `model` and `reasoning` are optional; each value follows the same +precedence: tool call, then profile, then the primary agent. Models use full +`provider:model` names. + +## Delegation + +The primary uses `subagents.run` to start a child from a profile: + +```lua +subagents.run { + agent = "reviewer", + prompt = "Review the authentication changes.", +} +``` + +Every started child returns an ID. Passing it back resumes the same +conversation, including after restarting Panto and resuming the primary: + +```lua +subagents.run { + id = "0198...", + prompt = "Now focus on the missing tests.", +} +``` + +Children receive the primary's system and project context plus their own +profile prompt, but not the primary's conversation. They share its workspace +and tools except for `subagents.*`. + +Multiple calls in one tool batch run concurrently. Their tagged assistant and +tool activity is visible in the TUI while they work. Child conversations are +stored as normal JSONL sessions beneath their owning primary session, so they +remain resumable without appearing in the user's `/resume` picker. A child ID +cannot be resumed from another primary conversation. + +`subagents.models` provides bounded search and validation for configured +providers, models, and model-specific reasoning levels without placing the +entire model catalog in every tool prompt. + +## Workflows + +TOML workflows define fixed dependency graphs. They are loaded from +`${XDG_CONFIG_HOME:-$HOME/.config}/panto/workflows/` and +`.panto/workflows/`, then exposed as `/workflow:<name>` commands. Ready steps +run concurrently; each dependent step receives its predecessors' labeled +outputs. + +The Lua workflow API handles dynamic branching and fan-out. It can await one +job or a group, and supports one-turn workers whose validated tool input is +their structured result. Panto extensions can use this API directly, while +`subagents.lua` runs model-authored one-off workflows in a restricted Lua +environment. + +See [DESIGN.md](DESIGN.md) for the runtime and persistence design. |