diff options
| author | t <t@tjp.lol> | 2026-11-17 00:53:11 -0700 |
|---|---|---|
| committer | t <t@tjp.lol> | 2026-11-17 15:42:02 -0700 |
| commit | 413d9a4b241a0f4111e39fdc0059ac15069d034f (patch) | |
| tree | 55f416467d7d1f1c6a36e875c3dd60e14874a1b0 /PLAN.md | |
Diffstat (limited to 'PLAN.md')
| -rw-r--r-- | PLAN.md | 556 |
1 files changed, 556 insertions, 0 deletions
@@ -0,0 +1,556 @@ +# panto-mcp: plan + +An MCP client for panto, shipped as an out-of-tree Lua extension (rock +`panto-mcp`, entry name `mcp`), plus the general-purpose panto core additions +it needs. Written 2026-11-16 from a planning session; readable without that +conversation. + +Research reports referenced below were written to `/tmp` (volatile); the facts +this plan depends on are restated inline. + +- `/tmp/mcp-research.md`: spec revisions, transports, lifecycle, auth, configs. +- `/tmp/mcp-discovery-research.md`: opencode/pi discovery, provider-native + deferred tools, generic-dispatch prior art. +- `/tmp/mcp-optional-features.md`: prompts, resources, elicitation, tasks. +- `/tmp/whiteboard-mcp-optional-features.html`: whiteboard of the above. + +## Goals + +- Configure MCP servers in panto's layered `config.toml`: remote by URL, local + by command. +- Speak the latest released revision (**2026-07-28**) and fall back to + **2025-11-25** (and the 2025-06-18 / 2025-03-26 versions it negotiates down + to). Most servers in the wild still speak the handshake revisions. +- Never leave orphaned server processes under any normal or abnormal panto + exit (one accepted gap, below). +- Make large MCP catalogs cheap: deferred discovery through `mcp.search` / + `mcp.call`, and "code mode" through an `mcp` namespace in `std.lua`. +- Tools, resources and prompts. Remote OAuth. + +## Non-goals (v1) + +| Feature | Why not | Revisit when | +|---|---|---| +| Elicitation | Needs core "user input for extensions" (TODOs.md). Not advertised, so servers must not send it; GitHub hides `delete_repository`. | After that core project; noted on its TODOs.md entry. | +| Tasks (2026 extension) | No popular server needs it for ordinary calls. Travis plans background-able tool calls first. | After background-able tool calls. | +| Tasks (2025 experimental wire) | Superseded, wire-incompatible. | Never, unless a concrete server requires it. | +| Sampling, roots, logging | Deprecated in 2026-07-28. Not advertised. | — | +| Resource subscriptions / `subscriptions/listen` | Long-lived stream, ack/demux/reconnect; on-demand reads suffice. | A real server workflow needs push updates. | +| 2024-11-05 HTTP+SSE transport | Deprecated; SHOULD NOT adopt. | A user hits a server that only speaks it. | +| Native provider deferred loading (`defer_loading`) | Only Anthropic Messages and OpenAI Responses have it; both still need the catalog up front. `mcp.search`/`mcp.call` is provider-neutral and cache-stable. | — | +| Importing `.mcp.json` / Claude Desktop configs | Not needed at first. | Later convenience. | +| Persistent broker surviving `/new`/`/resume`/`/reload` | Restarting is far simpler; remote servers dominate; switches are infrequent. | Startup cost becomes a real complaint. | +| Windows | panto doesn't target it. | — | + +## Key facts that shape the design + +- **stdio is newline-delimited JSON-RPC** on the child's stdin/stdout; stderr is + free-form logging. No port handoff exists. A local server that speaks HTTP is + configured by URL. +- **2026-07-28 is stateless.** No `initialize`, no sessions, no ping, no SSE + resumability. Every request carries `_meta` protocol version, client + capabilities and client info; servers MUST implement `server/discover`. + Results carry `resultType`. HTTP is one POST per request, answered with JSON + or a request-scoped SSE stream, with required `MCP-Protocol-Version`, + `Mcp-Method` and (for `tools/call`, `resources/read`, `prompts/get`) + `Mcp-Name` headers. +- **2025-11-25 is handshake-based:** `initialize` → `notifications/initialized`, + optional `Mcp-Session-Id`, GET notification stream, DELETE to end a session. +- **MCP has no tool search.** All proposals were closed or are dormant; the + official client guidance puts progressive discovery in the host. +- **Spec shutdown order for stdio:** close the server's stdin, wait, SIGTERM, + wait, SIGKILL (current opencode: 2s / 2s, whole process group). +- **panto constraints found in code:** + - Tools and commands are harvested only during `activate()` + (`src/lua_runtime.zig` `activateEntry`); a `ToolSource`'s `tools` slice is + fixed at registration. + - `panto.ext.http_request` is GET-only, fully buffered, `keep_alive = false`, + 120s max timeout, 16 global concurrent requests, auth over HTTPS only + (`src/lua_http.zig`). Its `Service` is created per Lua runtime and + destroyed on every session switch (`src/live_session.zig`). + - The TUI's SIGINT/SIGTERM/SIGHUP handlers restore the terminal and re-raise + without running `session_end` (`src/tui_terminal.zig`), so extension + cleanup cannot be relied on for abnormal exits. + - `session_end` gets at most 16 non-blocking loop passes before teardown + (`src/main.zig`), so it cannot wait out grace periods. + - Core auth has API keys and device-code OAuth only (libpantograph + `src/auth.zig`); no authorization-code + PKCE flow. + - `agent:conversation()` on the host agent returns a borrowed, mutable view + with `add_user_blocks` / `add_assistant_message`; `agent:submit` accepts + blocks. Persistence and rendering of view edits are unverified (spike S2). + +## Architecture + +```mermaid +flowchart LR + subgraph panto["panto process (Lua runtime, rebuilt per session)"] + EXT["mcp extension<br/>config · catalog · search · validate"] + NS["std.lua `mcp` namespace"] + T["tools: mcp.search / mcp.call / mcp.read_resource<br/>+ eager mcp.<server>.<tool>"] + C["commands: /mcp:<server>:<prompt>"] + H["core http_request<br/>(process-lifetime pool)"] + A["core auth (oauth_pkce)"] + MC[("metadata cache<br/>(disk, per server)")] + end + SUP["panto lua supervisor<br/>(own process group)"] + SRV["stdio server<br/>(own process group)"] + REM["remote server"] + EXT --- NS & T & C + EXT <-->|"load at activation · save on connect"| MC + EXT -- "stdin/stdout NDJSON (pipes)" --> SRV + EXT -- "fd 3 lifeline" --> SUP + SUP -- "owns / signals group" --> SRV + EXT -- "POST · JSON or SSE" --> H --> REM + H -. "Bearer" .- A +``` + +### Lifetime + +One Lua runtime per session. Servers start **lazily**, driven by a disk +metadata cache, so a session switch normally spawns nothing. On `session_end` +the extension closes everything without waiting (the supervisor finishes local +teardown; HTTP requests are cancelled). Running servers simply restart on +their next use after `/new`, `/resume`, `/reload`. + +**Metadata cache.** Every successful connect saves the server's discovery +result: detected era, server info/instructions, capabilities, and the full +tool, prompt, resource and resource-template lists. + +- One JSON file per server, keyed by a **config fingerprint**: a hash (FNV in + Lua) of the connection identity (`url`, or `command`/`args`/`env`/`cwd`, + plus `auth`). A config change means a cold cache. Writes are atomic + (temp file + rename). +- Location: `${XDG_DATA_HOME:-$HOME/.local/share}/panto-mcp/`, resolved in + Lua with the same XDG rule panto uses (`$XDG_DATA_HOME` if set and + non-empty, else `$HOME/.local/share`). The extension owns this directory; + no core cache-dir API. +- Replaces the earlier plan to keep the detected era in `panto.ext.state`. + +**Activation:** + +- **Warm cache** (entry with a matching fingerprint): register + `/mcp:<server>:<prompt>` commands and any eager `mcp.<server>.<tool>` tools + straight from the cache. **No process starts.** Eager tools are therefore + present from the first turn, without post-activation registration. +- **Cold cache** (first run, or config changed): connect that server in the + background (non-blocking) to populate the cache, then register its prompts + and eager tools post-activation (core C1/C2). + +**What starts a server:** `mcp.call`, an eager tool call, +`mcp.read_resource`, running a prompt command, or `/mcp refresh [server]`. +`mcp.search` searches the cache and starts nothing (a cold server still +connecting is waited for). Every connect re-lists everything, rewrites the +cache, and reconciles registrations: new prompts/eager tools are registered, +removed ones unregistered (C1/C2; an eager change busts the prompt cache once, +which is acceptable for a rare server-side change). + +**Staleness:** the cache is refreshed on every connect (no `ttlMs`-based +expiry in v1). `mcp.call` validates against the **live** schema, since the +connect it triggers re-lists first. A tool missing after the refresh errors +with the closest matches; a removed prompt fails on `prompts/get` and its +command is unregistered. + +A call to a server that is still connecting waits for it (bounded by the +server's startup timeout). A failed connection is retried lazily on the next +use, with the error visible in `mcp.search` / `/mcp` status. + +### Configuration + +Extension-owned tables, read from `panto.ext.config` (merged layers; rigs and +project configs may define servers, which is no worse than shipping Lua). + +```toml +[mcp] +startup_timeout_ms = 30000 # per server: spawn/connect + discovery +call_timeout_ms = 300000 # per request; progress does not extend past this +idle_timeout_ms = 60000 # HTTP streams: max silence between bytes + +[mcp.servers.github] +url = "https://api.githubcopilot.com/mcp/" +expose = "search" # "search" (default) | "eager" +auth = "mcp.github" # optional: named panto [auth.*] session +headers = { "X-Tenant" = "acme" } # non-secret; ${env:VAR} expanded +enabled = true +enabled_tools = [] # empty = all +disabled_tools = ["delete_*"] # glob over MCP tool names; applied after enabled_tools + +[mcp.servers.fs] +command = "npx" +args = ["-y", "@modelcontextprotocol/server-filesystem", "${project_root}"] +env = { "TOKEN" = "${env:FS_TOKEN}" } +cwd = "." # relative to project root +``` + +- Exactly one of `url` / `command` per server; invalid servers are reported and + skipped, never fatal to panto. +- `panto.ext.config` leaves `${env:VAR}` literal; the extension expands it in + `args`, `env`, `headers`, `url`. `${project_root}` comes from + `panto.ext.dirs`. Missing env → empty string, matching core semantics. +- Server names use `^[A-Za-z0-9_-]+$`, but also cannot contain `__` or end in + `_`: panto decodes `__` as a provider-name separator, so those names make + server/tool boundaries ambiguous. Invalid server entries are warned about + and skipped, never fatal to panto. +- Secrets for remote servers should go through `auth = "<session>"` so they + never pass through Lua; `headers` with `${env:}` remain possible. +- For automatic OAuth, omit `auth`; optional `[mcp.servers.<name>.oauth]` accepts + `client_id`, required matching `issuer` with that ID, optional `client_secret` + (`${env:VAR}` expanded) and fallback `scopes`. Otherwise use public DCR. +- Child env: inherit panto's environment plus `env`. (Codex-style `env_vars` + allowlisting is skipped; add if needed.) + +### Tool surface and token cost + +Every registered tool costs context tokens, so nothing is unconditional. + +| Exposed to model | When | +|---|---| +| `mcp.search`, `mcp.call`, `mcp.read_resource` | At activation, iff ≥1 enabled server has `expose = "search"`. Static declarations: cache-stable for the session. | +| `mcp.<server>.<tool>` per eager tool | At activation from a warm cache; otherwise after that eager server's first connect (post-activation registration, C1), visible from the next provider request. | +| `mcp.read_resource` (eager-only configs) | Registered when the first eager server (cached or live) advertises `resources`. | +| std.lua `mcp` namespace | Iff ≥1 enabled server. No tool cost; a short hint. | +| System hint | Iff ≥1 enabled server: one line per server (name, transport, mode) built from **config only**, so it never changes mid-session; points at search tools when exposed, otherwise at eager tools and docs. | + +Eager tools can miss the first turn only on a cold cache (the user submits +before the first-ever connect finishes). Search mode has no such race +(`mcp.call` waits). + +**`mcp.search {query?, server?, limit?}`**: connects the relevant servers if +needed and searches the cached catalog (tools, resources, resource templates) +client-side. Ranking: simple term scoring over name, title, description and +parameter names/descriptions (BM25-ish; no dependency). Each hit returns +`server`, `name`, title/description and the **full input schema**, so search +doubles as describe. Server `instructions` (from discover/initialize) are +returned when `server` is given, truncated. + +**`mcp.call {server, tool, input}`**: exact `server` + `tool` only; never fuzzy +execution. Unknown names return an error listing the closest matches. +`input` is validated against the tool's live input schema before sending; +failure returns the validation errors plus the schema. Declared non-strict +(an `any`-shaped `input` cannot be expressed in OpenAI/Anthropic strict mode). + +**`mcp.read_resource {server, uri}`**: `resources/read` on the originating server. +Never fetches arbitrary URLs itself; a `resource_link` in a tool result names +its server. + +Eager tool names are `mcp.<server>.<tool>`, with the MCP tool name sanitized +to `[A-Za-z0-9_-]` and `__` collapsed (panto encodes `.` as `__` on the wire). +Every tool gets a suffix derived only from its own MCP name, so adding another +tool cannot rename it. Truncation keeps the encoded wire name within **64 +bytes**. The limit is the strictest declared provider cap checked: Anthropic +Messages tool names allow 128 characters; OpenAI Chat Completions function +names cap at 64; OpenAI Responses documents function names as strings without +a stated limit. Sources: [Anthropic Messages](https://platform.claude.com/docs/en/api/http/messages/create), +[OpenAI Chat Completions](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create), +[OpenAI Responses](https://developers.openai.com/api/reference/resources/responses/methods/create). +Eager tools are validated the same way as `mcp.call` (panto has no schema +validation of its own). + +### std.lua `mcp` namespace ("code mode") + +Registered with `require("_std").lua_namespace("mcp", factory)`; available to +child agents too. Functions run in the std.lua handler coroutine and yield on +luv: + +```lua +mcp.servers() -- name, transport, mode, era, state, error +mcp.search(query, {server=, limit=}) +mcp.call(server, tool, input) -- validated; returns {text, attachments, structured, is_error} +mcp.start(server, tool, input) -- handle; handle:await(); for parallel calls +mcp.read_resource(server, uri) +mcp.prompts(server), mcp.prompt(server, name, args) +``` + +Reference docs ship as a require-able module (`require("panto_mcp.docs")`), +like `subagents.docs` and `_std.docs`: discovery, examples, result shapes, +errors. Only the one-line hint is injected. Per-server proxy sugar +(`mcp.github.create_issue{}`) is skipped; add it if models reach for it. + +### Input validation + +A pure-Lua JSON Schema subset validator, ported from panto-subagents' +`workflow.lua` (copied, not a dependency). Covers `type`, `properties`, +`required`, `additionalProperties`, `enum`/`const`, `items`/`prefixItems`, +numeric/string/array bounds, `anyOf`/`oneOf`/`allOf`/`not`, local `$ref`/ +`$defs`. Unknown keywords (including `pattern`/`format` if unsupported) are +**ignored, not rejected**: validation exists to catch cheap mistakes, and a +false rejection would block a valid call. Errors carry JSON-pointer paths. + +### Result mapping (tool calls, resources, prompts) + +| MCP content | panto result | +|---|---| +| `text` | text | +| `image` | attachment (base64 decoded in Lua; size-capped) | +| `audio` | text placeholder with MIME type and size | +| `resource` (embedded text) | text, prefixed with its URI | +| `resource` (embedded blob) | attachment if image, else placeholder | +| `resource_link` | text line: name, URI, description, and "read with `mcp.read_resource{server=…, uri=…}`" (or `mcp.read_resource` in std.lua) | +| `structuredContent` | JSON text, only if no text block already carries it | +| `isError: true` | tool error (model-visible) with the content | +| JSON-RPC `error` | tool error: `<server>: <message> (<code>)` | +| `resultType: input_required` | tool error: server requested user input, which panto does not support yet | + +Output is capped (configurable later if needed); truncation is stated in the +result. + +### Prompts + +- Each server prompt becomes a slash command **`/mcp:<server>:<prompt>`**, + registered at activation from the metadata cache, or after the first connect + on a cold cache (core C2), and unregistered if a later connect shows it gone. Description = prompt title/description plus its argument names, so the + fuzzy command selector finds it. +- Arguments: `name=value` tokens with shell-style quoting; a prompt with exactly + one argument accepts the trimmed raw remainder when no matching `name=` token + appears. Missing required arguments → error listing them. +- On invoke, start `prompts/get` in an async coroutine from the synchronous + command callback. Refuse if a turn or non-prompt command is active, and + re-check when the result arrives. Prompt responses append real user/assistant + roles to the borrowed host conversation; a trailing user message is submitted + to start a turn. Assistant media is represented by text placeholders. +- Prompt names the core rejects (slash/whitespace) are skipped with a warning + visible in `/mcp`. Async failures mount a visible error above the input until + the next `command_start`. One-argument positional values preserve the trimmed + raw remainder when no matching `name=` argument is present. +- **Assistant-tail rule:** if the prompt does not end with a user message, + append all messages and skip `agent:submit`. Emit `agent_submission` so the + idle TUI sync renders and persists the tail without starting a turn. +- All prompt message blocks are mapped before conversation mutation. Only + `image/*` content becomes user media; assistant media and non-image blobs + become text placeholders. This avoids partial history from rejected media. + +- **Conversation sync verified in ws-1:** idle `onPaint` syncs the conversation + tail and calls `flushPersist`; idle submission draining syncs/persists before + queued `agent:submit` work starts. Assistant-tail prompts emit + `agent_submission` to request that same idle sync. No C5 core change is needed. + +### Resources + +Built in v1 because tool results can carry `resource_link`s that are only +usable with `resources/read`. + +- Catalog: `resources/list` and `resources/templates/list` (paginated, cached + per `ttlMs`), searchable through `mcp.search`. +- Reading: `mcp.read_resource` tool and namespace function; templates are filled by the + model/user into a concrete URI, then read. +- Only servers advertising `resources` are queried. Some legacy clients treat + `-32601` from list methods as fatal; we never call unadvertised methods. + +## Protocol client + +Layers (each a module; transports are interchangeable under the client): + +1. **Transport** + - *stdio:* NDJSON framing over luv pipes; partial-line buffering; stderr + captured into a bounded per-server ring for diagnostics (never treated as + an error signal). + - *HTTP (modern):* POST per request with the required headers; response is + JSON or a request-scoped SSE stream parsed in Lua. `x-mcp-header` tool + annotations are validated and mirrored as `Mcp-Param-*` headers. + Cancellation = cancel the request (close its stream). + - *HTTP (legacy 2025-11-25):* adds `Mcp-Session-Id` capture/echo, re-init on + 404, best-effort DELETE on close. The optional GET notification stream opens + for eager servers advertising `tools.listChanged`, closes with the session, + retries after an established stream ends, and is replaced after legacy re-init. +2. **JSON-RPC:** id allocation, pending-request table, per-request timeout + (`call_timeout_ms`), cancellation (`notifications/cancelled` on stdio), + late responses to cancelled ids ignored. Progress notifications are accepted + and ignored (no progress token is sent in v1). +3. **Era negotiation:** + - Modern first: `server/discover` with 2026-07-28 `_meta`. Success → modern; + `UnsupportedProtocolVersion (-32022)` → pick from `supported`. + - Any other error, a recognizable legacy rejection, an empty/unrecognized + HTTP 400, or a probe timeout → legacy `initialize` (offer `2025-11-25`, + accept the server's counter-offer among versions we implement), then + `notifications/initialized`. + - The detected era is stored in the server's metadata cache entry to skip + the probe next time; if the cached era fails, re-probe. +4. **Capabilities advertised:** none (`{}`): no elicitation, sampling, roots, + tasks. Client info `{name = "panto-mcp", version}`. +5. **Server→client traffic (legacy):** answer `ping`; reply `-32601` to any + other request. Handle `notifications/tools|resources|prompts/list_changed` + by re-listing (eager: unregister/re-register; search: invalidate catalog; + prompts: re-register commands). + +## Local server lifecycle (no orphans) + +```mermaid +sequenceDiagram + participant P as panto (extension) + participant W as supervisor (panto lua) + participant S as server + P->>W: spawn detached: fds 0,1,2 = server pipes, fd 3 = lifeline + W->>S: spawn detached (new process group), inheriting fds 0,1,2 + W-->>W: close own copies of 0/1; report server pid on stderr marker + P->>S: JSON-RPC over the pipes (no relaying through W) + alt normal session_end / quit + P->>S: close stdin write end + P->>W: close lifeline (fd 3) + else panto dies (SIGKILL, crash, SIGHUP, OOM) + Note over P,W: kernel closes panto's fds: stdin EOF + lifeline EOF + end + W->>W: wait grace (2s) for S to exit + W->>S: SIGTERM process group, wait 2s + W->>S: SIGKILL process group + W-->>W: exit once group is gone +``` + +- One supervisor per local server: `uv.spawn(uv.exepath(), {"lua", + supervisor_path, ...})`, starts in ~16ms. `supervisor_path` resolved via + `package.searchpath` so it works from a rock or a checkout. +- Both supervisor and server run in their own process groups (`detached`), so + terminal Ctrl+C / Ctrl+Z / SIGHUP do not hit them directly; panto's death is + observed only through the lifeline. The supervisor also traps + INT/TERM/HUP and runs the same teardown. +- The server's group is signalled, catching `npx`/`docker`-style wrappers that + leave grandchildren. +- panto never waits during `session_end`: it closes two fds and returns; process + handles are unref'd so they don't hold the loop. +- **Accepted gap:** if panto and the supervisor are both SIGKILLed at once, a + server that ignores stdin EOF survives. No pidfile sweep (rare enough). +- Precedent: `../panto-claudesub/panto-claudesub/_session.lua` (detached group, + bounded TERM → KILL escalation, confirmed exit). + +## Remote auth + +Order of preference per server: `auth = "<session>"` (static API key or OAuth +session in `[auth.*]`), then automatic MCP OAuth on a 401. + +**Extension (MCP policy):** +1. On 401, parse `WWW-Authenticate` for `resource_metadata`; else try the + RFC 9728 well-known PRM URIs (endpoint path, then origin root). +2. Fetch authorization-server metadata (RFC 8414, then OIDC discovery + variants); require an exact `issuer` match and PKCE (`S256`) support. +3. Client registration: configured `client_id` requires an exact `oauth.issuer` + binding (optional env-expanded `client_secret`) → public Dynamic Client Registration. + Confidential DCR is refused; no client secret is stored in `panto.ext.state`. + Public registrations are stored per issuer and retained on every failure. Any + configured issuer pins selection from authorization_servers, even without a client ID. + CIMD is deferred (`TODOs.md`). +4. Define a runtime `oauth_pkce` session `mcp.<server>.<identity-hash>` (≤128 bytes), + using 128 bits of SHA-256 over canonical resource, exact issuer and client ID. + Retire the old name with core logout on changes. Use discovered endpoints, + challenge → PRM → configured scopes, and `resource` = canonical server URI. + Warm activation restores discovery without network work; no attestation flag. +5. Login is user-initiated: `/mcp-login <server> [callback-url | --reregister]`. + Reuse a definition unless missing or core login fails; keep registrations on all + failures. Only a changed discovered issuer or explicit --reregister discards one. + The escape hatch rediscovers/registers once; errors mention it. Consent with an + existing usable token does not lock out tools, and give-up clears login_required. +6. Authenticated 401: one forced refresh, then one retry; parallel calls await the + shared refresh. Scope 403 redefines the scope union and asks for re-consent, + without disabling working calls or an artificial attempt counter. Other 403s + report permission errors; explicit non-PKCE auth errors point to core auth. + Legacy GET reconnect backs off to 60 seconds and stops with a /mcp warning on 401. +7. Root PRM fallback validates `resource` against the origin. Loopback discovery/ + endpoints require a loopback resource; reject leading-zero IPv4 octets. Cache + status reads/poll at ~1 second and back off failed discovery from 5 to 60 seconds. + +**Core (credential mechanism, C4):** generic `oauth_pkce` auth type +(authorization code + PKCE S256, `state` check, optional `iss` validation, +`resource` parameter, loopback callback on an ephemeral port, **pasted +callback URL fallback** for SSH/headless, token storage under `auth/`, +refresh, header injection) plus `panto.ext` APIs to define a named session at +runtime and start its login. Requests then use `http_request{auth = name}`; +tokens never enter Lua. + +## Core additions (pantograph / libpanto) + +New `panto.ext` surfaces are co-designed with Travis before building; each +passes the AGENTS.md tests (second consumer, no "mcp" in core names, mechanism +not policy). + +| # | Addition | Second consumer | +|---|---|---| +| C1 | `register_tool` after activation + `unregister_tool(name)`. Changes take effect at the next turn (per-turn snapshot); in-flight calls unaffected; same collision rules; child agents sharing the Lua tool source see it. Needs copy-on-write declarations in the Lua `ToolSource` (provider serialization reads them off the owner thread). | Any extension whose tools depend on async discovery: plugin loaders, per-skill tools, a tool set that changes with project state. | +| C2 | `register_command` after activation + `unregister_command(name)`; the fuzzy selector and `render_header` command list pick them up. | subagents' `/workflow:<name>` when profiles change on disk; skills exposed as commands. | +| C3 | `http_request`: `method` (GET/POST/PUT/PATCH/DELETE), `body`; streaming via `on_head(head)` / `on_data(chunk)` hooks (presence of `on_data` disables buffering; final callback still fires with status/headers or error); `idle_timeout_ms`; no total cap for streams (`timeout_ms = 0`); keep-alive with a process-lifetime connection pool (callbacks still die with the runtime); auth allowed over plain HTTP to loopback; streams counted separately from the 16 buffered slots; bounded chunk queue for backpressure. | Lua protocol providers streaming a custom wire API; webhook/REST clients; the messaging bridge's long-poll. | +| C4 | `oauth_pkce` auth type + runtime session definition and login APIs (above). Seeded `config.toml` documents the new auth fields. | Messaging bridge (Slack/Telegram OAuth); any extension calling an OAuth-protected API. | +| C5 | *Only if spike S2 fails:* host-agent method to append arbitrary-role messages that persists and renders. | Fork/import tools, transcript surgery extensions, prompt-library extensions. | + +Draft Lua shape for C3 (to be co-designed): + +```lua +local handle, err = panto.ext.http_request({ + url = url, method = "POST", body = json, auth = "mcp.github", + headers = { ["content-type"] = "application/json", + accept = "application/json, text/event-stream" }, + timeout_ms = 0, idle_timeout_ms = 60000, + on_head = function(head) end, -- {status, headers} + on_data = function(chunk) end, -- raw bytes; SSE parsing is the caller's +}, function(response, err) end) -- buffered: {status, body, headers}; streamed: no body +handle:cancel() +``` + +## Build phases + +Spikes first, then extension work proceeds in parallel with core work +(stdio + search mode needs no core change). + +- [x] **S1 supervisor spike:** luv `spawn` with a 4th stdio slot from Lua; + detached supervisor + detached server; EOF propagation when panto is + `kill -9`ed; group kill of an `sh -c 'trap "" TERM; sleep …'` tree. +- [x] **S2 conversation spike:** source-verified: asynchronous idle edits sync + through `onPaint`; `agent:submit` synchronizes before the TUI drains its + queued work. Assistant-tail prompts explicitly emit `agent_submission` + to wake that same render/persist path. C5 is unnecessary. +- [x] **S3 tool-source mutability:** read libpanto's registry/`ToolSource` + snapshot path to size C1. +- [x] **E1 core client:** rock skeleton (`panto-mcp` → `init.lua`, + `panto_mcp.*` modules), config parsing/expansion, JSON-RPC, stdio + transport + supervisor, era negotiation, catalog + metadata cache + (lazy start), `mcp.search` / + `mcp.call`, validator, result mapping, `session_end` teardown, `/mcp` + status command. +- [x] **E2 code mode:** std.lua `mcp` namespace, `panto_mcp.docs`, system hint. +- [x] **C3 HTTP** → **E3 HTTP transport:** modern POST/SSE, legacy sessions, + static headers and `auth = "<session>"`. +- [x] **E4 resources:** list/templates/read, `mcp.read_resource`, resource blocks in + results, catalog search over resources. +- [x] **C1/C2 dynamic registration** → **E5 eager mode** (including legacy HTTP + GET notifications) and **E6 prompts** (`/mcp:<server>:<prompt>`); no C5 fix. +- [x] **C4 OAuth** → **E7 MCP OAuth:** PRM/AS discovery, configured/DCR registration, + `/mcp-login` (loopback + pasted callback), refresh-on-401 and non-blocking scope step-up; + issuer-bound configured clients, identity-scoped sessions, public-only DCR, + rediscovery/backoff/status caching, warm restoration and real-core TCP-fake checks. + Real-provider pass remains below. +- [ ] **E8 release:** README, rockspec, mise tasks (copy panto-web's + `version`/`release`), luarocks upload. Files done; tag/upload left to Travis. +- [x] Note on pantograph `TODOs.md` "User input for extensions" that it enables + MCP elicitation in panto-mcp. + +## Testing + +Per pantograph AGENTS.md: never test timeouts by sleeping; inject the clock / +timer factory so timeouts fire deterministically. + +- `_selfcheck.lua` run with `panto lua` (like panto-web/panto-claudesub): + fake stdio servers written in Lua for both eras (modern discover; legacy + initialize; legacy server that errors on pre-initialize requests; server that + ignores EOF; server that spawns a grandchild), and a luv TCP fake for HTTP + (JSON response, SSE response, legacy session 404 → re-init, 401 → PRM). +- Eager tools/prompts: warm activation is idle; a cold `mcp.search` connect + reconciles eager declarations; `list_changed` reconciles add/remove/schema + changes. Tests cover multi-server prompt preservation, `/compact` command + busy state, visible async failures, image-only media and all-message preflight, + raw one-argument remainders, invalid command/server names, stable eager names, + exact cache pruning, and legacy GET reconnect/re-init. +- Validator, SSE parser, NDJSON framer, result mapping, name sanitizing, arg + parsing: table-driven unit checks in the same selfcheck. +- OAuth: table-driven challenge/URL/issuer/S256/scope checks; injected HTTP/auth/timers + cover discovery → public DCR → user login → reconnect, refresh and scope upgrades. + Regressions cover foreign-issuer credentials, identity-scoped sessions, non-locking + 403s, concurrent refresh calls, confidential-DCR refusal, rediscovery and backoff. + `PANTO_MCP_HTTP_E2E=1` adds a new-core browser/pasted-callback flow, forced refresh, + scope re-consent and warm reuse against a local TCP fake (no real credentials). +- Orphan check: spawn supervisor+server from a stand-in parent process, + `kill -9` the parent, wait on the server group's exit event (bounded), assert + nothing survives. +- Core additions get Zig tests in pantograph/libpanto alongside their code. +- One manual end-to-end pass against real servers: a filesystem stdio server + via `npx`, a remote legacy server, a remote 2026 server, and one OAuth server + (GitHub or Linear). + +## Resolved design checks + +- **Provider tool-name limits:** use 64 encoded wire bytes, the strictest declared + bound in the primary provider docs linked above. |
