diff options
Diffstat (limited to 'REFACTOR.md')
| -rw-r--r-- | REFACTOR.md | 196 |
1 files changed, 196 insertions, 0 deletions
diff --git a/REFACTOR.md b/REFACTOR.md new file mode 100644 index 0000000..3eec504 --- /dev/null +++ b/REFACTOR.md @@ -0,0 +1,196 @@ +# panto-subagents refactor: extension-API-first + +## Goal + +The v0 implementation works end to end, but pantograph absorbed the feature: +`subagent_host.zig`, `subagent_runtime.zig`, and `tui_subagents.zig` are a +subagent backend in core, and the rock is mostly validation and formatting +glue. Per the "Extension features: two responses" rubric in pantograph's +AGENTS.md, this refactor takes response 2 retroactively: the missing *general* +capabilities are added to libpanto-lua and `panto.ext`, subagent policy moves +into this rock, and the feature-shaped core surface is deleted. + +Definition of done, in one line: `rg -i subagent pantograph/src` returns +nothing, and every behavior in DESIGN.md still holds through the real binary. + +The product surface (DESIGN.md) is unchanged. This is a re-division of the +implementation, not a redesign. DESIGN.md's "Generic pantograph child-job +seam" section is superseded by this document and should be rewritten to match +when this lands. + +## What exists today (all currently uncommitted working-tree state) + +Generic and staying (invisible runtime correctness, landed with v0): + +- The multiplexed Lua executor: tool batches keyed by id instead of a single + `current_batch`, cross-thread posted jobs onto the Lua owner thread + (`lua_executor.zig`, `lua_runtime.zig`). +- Lane identity: `panto.ext.agent` resolves through `coroutine.running()`; + per-open protocol stream state and stream-level `cancel_turn` + (`lua_provider.zig`). +- libpantograph: message metadata survives the JSONL load path; + `UserMessage.metadata` exists on `Agent.run`. +- `panto.ext.models` (bounded catalog queries) and `panto.ext.session_info()`. + +Feature-shaped and going: `subagent_host.zig` (vtable), `subagent_runtime.zig` +(spawn transaction, worker threads, `Gate` concurrency bound, metadata +stamping, manifest extraction, resume defaults, one-shot workers, progress +queue), `tui_subagents.zig` (hardcoded progress panel), and the +`spawn_agent`/`await_jobs` thunks in `lua_bridge.zig`. + +## New libpanto-lua APIs + +These are library bindings, useful to any embedder of libpanto-lua; nothing +about them is subagent-specific. + +### Async agent jobs + +```lua +local job = agent:run_async { + prompt = "...", -- or blocks, mirroring agent:run + metadata = { ... }, -- optional user-message metadata (JSON-encoded) + dispatch_tools = false, -- optional: end the turn at the first + -- assistant response instead of dispatching + -- its tool calls; they are reported settled + wake_fd = fd, -- optional: a byte is written on every event + -- arrival and on settle +} + +job:next_event() -- -> event table | nil (non-blocking; owned copy) +job:result() -- -> settled result | nil while running +job:request_cancel() +job:close() -- join + release; safe after settle or cancel +``` + +- Implemented with a binding-internal pump thread per job: it drives the + blocking `stream.next()`, copies each event payload (payloads die at the + next pull), buffers under a mutex, and writes one byte to `wake_fd`. +- The fd contract is deliberately loop-agnostic: a luv consumer arms + `uv.new_poll(fd)`; a Go or C embedder selects on it. No loop dependency + enters libpanto-lua. +- Tool dispatch inside the turn is unchanged libpantograph behavior; when the + agent's tools include panto's Lua tool source, its `invoke_batch` posts to + the Lua owner thread through the existing executor — that path already + exists and does not change. +- `dispatch_tools = false` is the generic knob that makes structured one-shot + workers a pure consumer feature: the settled result carries the assistant's + tool calls (name + input JSON) unresolved, and no second model round runs. + +### Conversation message metadata + +```lua +conv:message_metadata(i) -- -> table | nil (decoded JSON) +conv:set_message_metadata(i, tbl) -- JSON-encodes; nil clears +``` + +Plus optional `metadata` on `conv:add_system_message`/`add_user_message`. +The disk format already round-trips metadata; this is access only. + +### Agent tool control + +```lua +agent:tools() -- -> array of decl tables { name, description, schema }, + -- a mutable copy +agent:set_tools(decls) -- replace this agent's tool list with these decls; + -- handlers resolve by name through the owning source +``` + +Copying between agents works because dispatch is name-keyed on the shared +runtime. `agent:set_config` additionally accepts `tool_choice` (force a named +tool). This retires the Zig-side `toolSourceExcluding`. + +## New/changed pantograph surfaces + +- `panto.ext.resolve_model { model = "provider:alias", reasoning = label }` + → an **opaque config userdata** accepted by `panto.agent { config = ... }` + and `agent:set_config`. Resolution and validation reuse the existing + registry/reasoning pipeline and fail before inference; credentials are + embedded host-side and never readable from Lua. This is the one place the + credential boundary requires a host API rather than a binding API. +- `panto.ext.models` and `panto.ext.session_info()` remain as-is (already + generic). +- `spawn_agent`, `await_jobs`, the job-handle userdata, and the + `SubagentHost` vtable are deleted once the rock is ported. Awaiting becomes + ordinary luv: the rock arms `uv.new_poll` on each job's `wake_fd` and its + tool-handler coroutine yields/resumes through the standard async-tool + contract. +- Progress rendering: `tui_subagents.zig` is deleted. The rock pulls its own + job events and surfaces activity through the existing extension event and + component machinery (`panto.ext.on`/`emit` + component swap). If that + machinery proves insufficient for a live multi-card panel, the fallback is + one small *generic* host API (a "live status card" keyed by an id — second + consumer: any long-running extension activity), to be co-designed before + building, per AGENTS.md. + +## Policy migration map (the closure check) + +Every deleted Zig behavior must have a Lua expression using only the APIs +above. This table is the completeness test for the API set: + +| Behavior (today in Zig) | New home in the rock | +|------------------------------------------|----------------------| +| Child store layout `<sd>/subagents/<id>` | string assembly (already rock-side) | +| Store create/resolve, ownership boundary | `panto.file_system_jsonl_store(dir)` + `store:resolve/load` | +| Manifest on profile system message | `conv:set_message_metadata` at seeding | +| Per-turn model/reasoning stamp | `run_async { metadata = ... }` | +| Resume defaults from last user message | `store:load` + `conv:message_metadata` scan | +| Primary system context for new children | copy system messages from `panto.ext.agent:conversation()` | +| Concurrency bound (4) + queue + cancel-while-queued | rock-side gate: start ≤4 jobs, queue the rest, drop queued on cancel (~15 lines of Lua) | +| `subagents.*` exclusion (no recursion) | `agent:set_tools(filtered)` from the primary's `agent:tools()` | +| One-shot structured workers | new agent + single synthetic tool + `tool_choice` + `dispatch_tools = false` + `panto.null_store()`; validate with the existing jsonschema path | +| `resumable` determination | fresh `store:resolve(id)` after failure | +| Result shaping (id/agent/status/output) | already rock-side; loses the host `JobResult` intermediary | +| Progress cards | rock consumes `job:next_event()`; renders via event/component machinery | +| Escape cancels all children | rock tracks its live jobs; a host cancellation event (or the turn-teardown hook that fires it) calls `job:request_cancel()` on each — verify the existing bus exposes interrupt; if not, add a generic turn-lifecycle event | + +Two rows need small verifications during implementation (marked above): +turn-lifecycle/interrupt visibility to extensions, and whether the component +machinery can host the live panel. Both resolutions must stay feature-neutral. + +## Deletions + +pantograph: `subagent_host.zig`, `subagent_runtime.zig`, `tui_subagents.zig`, +the spawn/await thunks and job userdata in `lua_bridge.zig`, the subagent +wiring in `main.zig`/`tui_app.zig`, and `subagent_integration_tests.zig` in +its current form (see validation). The executor, lane machinery, and +`lua_provider` lane calls remain as internal runtime correctness — reviewed +afterward for members only the deleted code used. + +## Validation + +- The v0 end-to-end proof must survive: a scripted extension-protocol fixture + driving the real binary with the real rock — parallel `subagents.run` + + unknown-agent error in one batch, child resume with history, workflow + diamond, sandbox fan-out, catalog round-trip, child JSONL + isolation/manifest/metadata asserted from bytes on disk. (Reconstruct the + harness from this list; treat it as the primary gate.) +- Rock suite (`mise run check`) extended to cover the migrated policy: + gating/bound behavior, resume-default extraction, one-shot capture via + unresolved tool calls, tool filtering, manifest seeding. +- pantograph integration tests rewritten against the *generic* surfaces + (async jobs driving concurrent agents, batch isolation, lane agent + resolution, cancellation) with no subagent vocabulary. +- `rg -i subagent /Users/travis/Code/pantograph/src` → empty. +- All suites green: `mise exec -- zig build test`, `mise run check`. + +## Delivery order + +1. libpanto-lua: `run_async` jobs + fd contract + `dispatch_tools` knob, with + binding-level tests (fixture provider, no panto involvement). +2. libpanto-lua: message-metadata access; agent tool get/set + `tool_choice`. +3. pantograph: `resolve_model` opaque config; verify interrupt/lifecycle + event visibility (add the generic event if missing). +4. Rock: port spawn/await/bound/policy onto the new APIs behind the same tool + surfaces; port progress to event/component rendering. +5. Delete the pantograph subagent files and thunks; rewrite integration tests + generically; run the full validation list. +6. Update DESIGN.md's seam sections to describe the final division. + +## Interop with the panto-on-libuv project (separate; order-independent) + +The `wake_fd` contract works identically in both worlds: today the rock arms +it with `uv.new_poll` on luv's loop; after the libuv unification it is the +same call on the shared process loop. If the libuv project lands first, the +executor internals this refactor relies on (posted jobs onto the Lua owner) +will already use `uv_async` instead of the pipe waker — no change to anything +specified here. Neither project should block on the other. |