summaryrefslogtreecommitdiff
path: root/PLAN.md
diff options
context:
space:
mode:
Diffstat (limited to 'PLAN.md')
-rw-r--r--PLAN.md556
1 files changed, 556 insertions, 0 deletions
diff --git a/PLAN.md b/PLAN.md
new file mode 100644
index 0000000..a0003c8
--- /dev/null
+++ b/PLAN.md
@@ -0,0 +1,556 @@
+# panto-mcp: plan
+
+An MCP client for panto, shipped as an out-of-tree Lua extension (rock
+`panto-mcp`, entry name `mcp`), plus the general-purpose panto core additions
+it needs. Written 2026-11-16 from a planning session; readable without that
+conversation.
+
+Research reports referenced below were written to `/tmp` (volatile); the facts
+this plan depends on are restated inline.
+
+- `/tmp/mcp-research.md`: spec revisions, transports, lifecycle, auth, configs.
+- `/tmp/mcp-discovery-research.md`: opencode/pi discovery, provider-native
+ deferred tools, generic-dispatch prior art.
+- `/tmp/mcp-optional-features.md`: prompts, resources, elicitation, tasks.
+- `/tmp/whiteboard-mcp-optional-features.html`: whiteboard of the above.
+
+## Goals
+
+- Configure MCP servers in panto's layered `config.toml`: remote by URL, local
+ by command.
+- Speak the latest released revision (**2026-07-28**) and fall back to
+ **2025-11-25** (and the 2025-06-18 / 2025-03-26 versions it negotiates down
+ to). Most servers in the wild still speak the handshake revisions.
+- Never leave orphaned server processes under any normal or abnormal panto
+ exit (one accepted gap, below).
+- Make large MCP catalogs cheap: deferred discovery through `mcp.search` /
+ `mcp.call`, and "code mode" through an `mcp` namespace in `std.lua`.
+- Tools, resources and prompts. Remote OAuth.
+
+## Non-goals (v1)
+
+| Feature | Why not | Revisit when |
+|---|---|---|
+| Elicitation | Needs core "user input for extensions" (TODOs.md). Not advertised, so servers must not send it; GitHub hides `delete_repository`. | After that core project; noted on its TODOs.md entry. |
+| Tasks (2026 extension) | No popular server needs it for ordinary calls. Travis plans background-able tool calls first. | After background-able tool calls. |
+| Tasks (2025 experimental wire) | Superseded, wire-incompatible. | Never, unless a concrete server requires it. |
+| Sampling, roots, logging | Deprecated in 2026-07-28. Not advertised. | — |
+| Resource subscriptions / `subscriptions/listen` | Long-lived stream, ack/demux/reconnect; on-demand reads suffice. | A real server workflow needs push updates. |
+| 2024-11-05 HTTP+SSE transport | Deprecated; SHOULD NOT adopt. | A user hits a server that only speaks it. |
+| Native provider deferred loading (`defer_loading`) | Only Anthropic Messages and OpenAI Responses have it; both still need the catalog up front. `mcp.search`/`mcp.call` is provider-neutral and cache-stable. | — |
+| Importing `.mcp.json` / Claude Desktop configs | Not needed at first. | Later convenience. |
+| Persistent broker surviving `/new`/`/resume`/`/reload` | Restarting is far simpler; remote servers dominate; switches are infrequent. | Startup cost becomes a real complaint. |
+| Windows | panto doesn't target it. | — |
+
+## Key facts that shape the design
+
+- **stdio is newline-delimited JSON-RPC** on the child's stdin/stdout; stderr is
+ free-form logging. No port handoff exists. A local server that speaks HTTP is
+ configured by URL.
+- **2026-07-28 is stateless.** No `initialize`, no sessions, no ping, no SSE
+ resumability. Every request carries `_meta` protocol version, client
+ capabilities and client info; servers MUST implement `server/discover`.
+ Results carry `resultType`. HTTP is one POST per request, answered with JSON
+ or a request-scoped SSE stream, with required `MCP-Protocol-Version`,
+ `Mcp-Method` and (for `tools/call`, `resources/read`, `prompts/get`)
+ `Mcp-Name` headers.
+- **2025-11-25 is handshake-based:** `initialize` → `notifications/initialized`,
+ optional `Mcp-Session-Id`, GET notification stream, DELETE to end a session.
+- **MCP has no tool search.** All proposals were closed or are dormant; the
+ official client guidance puts progressive discovery in the host.
+- **Spec shutdown order for stdio:** close the server's stdin, wait, SIGTERM,
+ wait, SIGKILL (current opencode: 2s / 2s, whole process group).
+- **panto constraints found in code:**
+ - Tools and commands are harvested only during `activate()`
+ (`src/lua_runtime.zig` `activateEntry`); a `ToolSource`'s `tools` slice is
+ fixed at registration.
+ - `panto.ext.http_request` is GET-only, fully buffered, `keep_alive = false`,
+ 120s max timeout, 16 global concurrent requests, auth over HTTPS only
+ (`src/lua_http.zig`). Its `Service` is created per Lua runtime and
+ destroyed on every session switch (`src/live_session.zig`).
+ - The TUI's SIGINT/SIGTERM/SIGHUP handlers restore the terminal and re-raise
+ without running `session_end` (`src/tui_terminal.zig`), so extension
+ cleanup cannot be relied on for abnormal exits.
+ - `session_end` gets at most 16 non-blocking loop passes before teardown
+ (`src/main.zig`), so it cannot wait out grace periods.
+ - Core auth has API keys and device-code OAuth only (libpantograph
+ `src/auth.zig`); no authorization-code + PKCE flow.
+ - `agent:conversation()` on the host agent returns a borrowed, mutable view
+ with `add_user_blocks` / `add_assistant_message`; `agent:submit` accepts
+ blocks. Persistence and rendering of view edits are unverified (spike S2).
+
+## Architecture
+
+```mermaid
+flowchart LR
+ subgraph panto["panto process (Lua runtime, rebuilt per session)"]
+ EXT["mcp extension<br/>config · catalog · search · validate"]
+ NS["std.lua `mcp` namespace"]
+ T["tools: mcp.search / mcp.call / mcp.read_resource<br/>+ eager mcp.&lt;server&gt;.&lt;tool&gt;"]
+ C["commands: /mcp:&lt;server&gt;:&lt;prompt&gt;"]
+ H["core http_request<br/>(process-lifetime pool)"]
+ A["core auth (oauth_pkce)"]
+ MC[("metadata cache<br/>(disk, per server)")]
+ end
+ SUP["panto lua supervisor<br/>(own process group)"]
+ SRV["stdio server<br/>(own process group)"]
+ REM["remote server"]
+ EXT --- NS & T & C
+ EXT <-->|"load at activation · save on connect"| MC
+ EXT -- "stdin/stdout NDJSON (pipes)" --> SRV
+ EXT -- "fd 3 lifeline" --> SUP
+ SUP -- "owns / signals group" --> SRV
+ EXT -- "POST · JSON or SSE" --> H --> REM
+ H -. "Bearer" .- A
+```
+
+### Lifetime
+
+One Lua runtime per session. Servers start **lazily**, driven by a disk
+metadata cache, so a session switch normally spawns nothing. On `session_end`
+the extension closes everything without waiting (the supervisor finishes local
+teardown; HTTP requests are cancelled). Running servers simply restart on
+their next use after `/new`, `/resume`, `/reload`.
+
+**Metadata cache.** Every successful connect saves the server's discovery
+result: detected era, server info/instructions, capabilities, and the full
+tool, prompt, resource and resource-template lists.
+
+- One JSON file per server, keyed by a **config fingerprint**: a hash (FNV in
+ Lua) of the connection identity (`url`, or `command`/`args`/`env`/`cwd`,
+ plus `auth`). A config change means a cold cache. Writes are atomic
+ (temp file + rename).
+- Location: `${XDG_DATA_HOME:-$HOME/.local/share}/panto-mcp/`, resolved in
+ Lua with the same XDG rule panto uses (`$XDG_DATA_HOME` if set and
+ non-empty, else `$HOME/.local/share`). The extension owns this directory;
+ no core cache-dir API.
+- Replaces the earlier plan to keep the detected era in `panto.ext.state`.
+
+**Activation:**
+
+- **Warm cache** (entry with a matching fingerprint): register
+ `/mcp:<server>:<prompt>` commands and any eager `mcp.<server>.<tool>` tools
+ straight from the cache. **No process starts.** Eager tools are therefore
+ present from the first turn, without post-activation registration.
+- **Cold cache** (first run, or config changed): connect that server in the
+ background (non-blocking) to populate the cache, then register its prompts
+ and eager tools post-activation (core C1/C2).
+
+**What starts a server:** `mcp.call`, an eager tool call,
+`mcp.read_resource`, running a prompt command, or `/mcp refresh [server]`.
+`mcp.search` searches the cache and starts nothing (a cold server still
+connecting is waited for). Every connect re-lists everything, rewrites the
+cache, and reconciles registrations: new prompts/eager tools are registered,
+removed ones unregistered (C1/C2; an eager change busts the prompt cache once,
+which is acceptable for a rare server-side change).
+
+**Staleness:** the cache is refreshed on every connect (no `ttlMs`-based
+expiry in v1). `mcp.call` validates against the **live** schema, since the
+connect it triggers re-lists first. A tool missing after the refresh errors
+with the closest matches; a removed prompt fails on `prompts/get` and its
+command is unregistered.
+
+A call to a server that is still connecting waits for it (bounded by the
+server's startup timeout). A failed connection is retried lazily on the next
+use, with the error visible in `mcp.search` / `/mcp` status.
+
+### Configuration
+
+Extension-owned tables, read from `panto.ext.config` (merged layers; rigs and
+project configs may define servers, which is no worse than shipping Lua).
+
+```toml
+[mcp]
+startup_timeout_ms = 30000 # per server: spawn/connect + discovery
+call_timeout_ms = 300000 # per request; progress does not extend past this
+idle_timeout_ms = 60000 # HTTP streams: max silence between bytes
+
+[mcp.servers.github]
+url = "https://api.githubcopilot.com/mcp/"
+expose = "search" # "search" (default) | "eager"
+auth = "mcp.github" # optional: named panto [auth.*] session
+headers = { "X-Tenant" = "acme" } # non-secret; ${env:VAR} expanded
+enabled = true
+enabled_tools = [] # empty = all
+disabled_tools = ["delete_*"] # glob over MCP tool names; applied after enabled_tools
+
+[mcp.servers.fs]
+command = "npx"
+args = ["-y", "@modelcontextprotocol/server-filesystem", "${project_root}"]
+env = { "TOKEN" = "${env:FS_TOKEN}" }
+cwd = "." # relative to project root
+```
+
+- Exactly one of `url` / `command` per server; invalid servers are reported and
+ skipped, never fatal to panto.
+- `panto.ext.config` leaves `${env:VAR}` literal; the extension expands it in
+ `args`, `env`, `headers`, `url`. `${project_root}` comes from
+ `panto.ext.dirs`. Missing env → empty string, matching core semantics.
+- Server names use `^[A-Za-z0-9_-]+$`, but also cannot contain `__` or end in
+ `_`: panto decodes `__` as a provider-name separator, so those names make
+ server/tool boundaries ambiguous. Invalid server entries are warned about
+ and skipped, never fatal to panto.
+- Secrets for remote servers should go through `auth = "<session>"` so they
+ never pass through Lua; `headers` with `${env:}` remain possible.
+- For automatic OAuth, omit `auth`; optional `[mcp.servers.<name>.oauth]` accepts
+ `client_id`, required matching `issuer` with that ID, optional `client_secret`
+ (`${env:VAR}` expanded) and fallback `scopes`. Otherwise use public DCR.
+- Child env: inherit panto's environment plus `env`. (Codex-style `env_vars`
+ allowlisting is skipped; add if needed.)
+
+### Tool surface and token cost
+
+Every registered tool costs context tokens, so nothing is unconditional.
+
+| Exposed to model | When |
+|---|---|
+| `mcp.search`, `mcp.call`, `mcp.read_resource` | At activation, iff ≥1 enabled server has `expose = "search"`. Static declarations: cache-stable for the session. |
+| `mcp.<server>.<tool>` per eager tool | At activation from a warm cache; otherwise after that eager server's first connect (post-activation registration, C1), visible from the next provider request. |
+| `mcp.read_resource` (eager-only configs) | Registered when the first eager server (cached or live) advertises `resources`. |
+| std.lua `mcp` namespace | Iff ≥1 enabled server. No tool cost; a short hint. |
+| System hint | Iff ≥1 enabled server: one line per server (name, transport, mode) built from **config only**, so it never changes mid-session; points at search tools when exposed, otherwise at eager tools and docs. |
+
+Eager tools can miss the first turn only on a cold cache (the user submits
+before the first-ever connect finishes). Search mode has no such race
+(`mcp.call` waits).
+
+**`mcp.search {query?, server?, limit?}`**: connects the relevant servers if
+needed and searches the cached catalog (tools, resources, resource templates)
+client-side. Ranking: simple term scoring over name, title, description and
+parameter names/descriptions (BM25-ish; no dependency). Each hit returns
+`server`, `name`, title/description and the **full input schema**, so search
+doubles as describe. Server `instructions` (from discover/initialize) are
+returned when `server` is given, truncated.
+
+**`mcp.call {server, tool, input}`**: exact `server` + `tool` only; never fuzzy
+execution. Unknown names return an error listing the closest matches.
+`input` is validated against the tool's live input schema before sending;
+failure returns the validation errors plus the schema. Declared non-strict
+(an `any`-shaped `input` cannot be expressed in OpenAI/Anthropic strict mode).
+
+**`mcp.read_resource {server, uri}`**: `resources/read` on the originating server.
+Never fetches arbitrary URLs itself; a `resource_link` in a tool result names
+its server.
+
+Eager tool names are `mcp.<server>.<tool>`, with the MCP tool name sanitized
+to `[A-Za-z0-9_-]` and `__` collapsed (panto encodes `.` as `__` on the wire).
+Every tool gets a suffix derived only from its own MCP name, so adding another
+tool cannot rename it. Truncation keeps the encoded wire name within **64
+bytes**. The limit is the strictest declared provider cap checked: Anthropic
+Messages tool names allow 128 characters; OpenAI Chat Completions function
+names cap at 64; OpenAI Responses documents function names as strings without
+a stated limit. Sources: [Anthropic Messages](https://platform.claude.com/docs/en/api/http/messages/create),
+[OpenAI Chat Completions](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create),
+[OpenAI Responses](https://developers.openai.com/api/reference/resources/responses/methods/create).
+Eager tools are validated the same way as `mcp.call` (panto has no schema
+validation of its own).
+
+### std.lua `mcp` namespace ("code mode")
+
+Registered with `require("_std").lua_namespace("mcp", factory)`; available to
+child agents too. Functions run in the std.lua handler coroutine and yield on
+luv:
+
+```lua
+mcp.servers() -- name, transport, mode, era, state, error
+mcp.search(query, {server=, limit=})
+mcp.call(server, tool, input) -- validated; returns {text, attachments, structured, is_error}
+mcp.start(server, tool, input) -- handle; handle:await(); for parallel calls
+mcp.read_resource(server, uri)
+mcp.prompts(server), mcp.prompt(server, name, args)
+```
+
+Reference docs ship as a require-able module (`require("panto_mcp.docs")`),
+like `subagents.docs` and `_std.docs`: discovery, examples, result shapes,
+errors. Only the one-line hint is injected. Per-server proxy sugar
+(`mcp.github.create_issue{}`) is skipped; add it if models reach for it.
+
+### Input validation
+
+A pure-Lua JSON Schema subset validator, ported from panto-subagents'
+`workflow.lua` (copied, not a dependency). Covers `type`, `properties`,
+`required`, `additionalProperties`, `enum`/`const`, `items`/`prefixItems`,
+numeric/string/array bounds, `anyOf`/`oneOf`/`allOf`/`not`, local `$ref`/
+`$defs`. Unknown keywords (including `pattern`/`format` if unsupported) are
+**ignored, not rejected**: validation exists to catch cheap mistakes, and a
+false rejection would block a valid call. Errors carry JSON-pointer paths.
+
+### Result mapping (tool calls, resources, prompts)
+
+| MCP content | panto result |
+|---|---|
+| `text` | text |
+| `image` | attachment (base64 decoded in Lua; size-capped) |
+| `audio` | text placeholder with MIME type and size |
+| `resource` (embedded text) | text, prefixed with its URI |
+| `resource` (embedded blob) | attachment if image, else placeholder |
+| `resource_link` | text line: name, URI, description, and "read with `mcp.read_resource{server=…, uri=…}`" (or `mcp.read_resource` in std.lua) |
+| `structuredContent` | JSON text, only if no text block already carries it |
+| `isError: true` | tool error (model-visible) with the content |
+| JSON-RPC `error` | tool error: `<server>: <message> (<code>)` |
+| `resultType: input_required` | tool error: server requested user input, which panto does not support yet |
+
+Output is capped (configurable later if needed); truncation is stated in the
+result.
+
+### Prompts
+
+- Each server prompt becomes a slash command **`/mcp:<server>:<prompt>`**,
+ registered at activation from the metadata cache, or after the first connect
+ on a cold cache (core C2), and unregistered if a later connect shows it gone. Description = prompt title/description plus its argument names, so the
+ fuzzy command selector finds it.
+- Arguments: `name=value` tokens with shell-style quoting; a prompt with exactly
+ one argument accepts the trimmed raw remainder when no matching `name=` token
+ appears. Missing required arguments → error listing them.
+- On invoke, start `prompts/get` in an async coroutine from the synchronous
+ command callback. Refuse if a turn or non-prompt command is active, and
+ re-check when the result arrives. Prompt responses append real user/assistant
+ roles to the borrowed host conversation; a trailing user message is submitted
+ to start a turn. Assistant media is represented by text placeholders.
+- Prompt names the core rejects (slash/whitespace) are skipped with a warning
+ visible in `/mcp`. Async failures mount a visible error above the input until
+ the next `command_start`. One-argument positional values preserve the trimmed
+ raw remainder when no matching `name=` argument is present.
+- **Assistant-tail rule:** if the prompt does not end with a user message,
+ append all messages and skip `agent:submit`. Emit `agent_submission` so the
+ idle TUI sync renders and persists the tail without starting a turn.
+- All prompt message blocks are mapped before conversation mutation. Only
+ `image/*` content becomes user media; assistant media and non-image blobs
+ become text placeholders. This avoids partial history from rejected media.
+
+- **Conversation sync verified in ws-1:** idle `onPaint` syncs the conversation
+ tail and calls `flushPersist`; idle submission draining syncs/persists before
+ queued `agent:submit` work starts. Assistant-tail prompts emit
+ `agent_submission` to request that same idle sync. No C5 core change is needed.
+
+### Resources
+
+Built in v1 because tool results can carry `resource_link`s that are only
+usable with `resources/read`.
+
+- Catalog: `resources/list` and `resources/templates/list` (paginated, cached
+ per `ttlMs`), searchable through `mcp.search`.
+- Reading: `mcp.read_resource` tool and namespace function; templates are filled by the
+ model/user into a concrete URI, then read.
+- Only servers advertising `resources` are queried. Some legacy clients treat
+ `-32601` from list methods as fatal; we never call unadvertised methods.
+
+## Protocol client
+
+Layers (each a module; transports are interchangeable under the client):
+
+1. **Transport**
+ - *stdio:* NDJSON framing over luv pipes; partial-line buffering; stderr
+ captured into a bounded per-server ring for diagnostics (never treated as
+ an error signal).
+ - *HTTP (modern):* POST per request with the required headers; response is
+ JSON or a request-scoped SSE stream parsed in Lua. `x-mcp-header` tool
+ annotations are validated and mirrored as `Mcp-Param-*` headers.
+ Cancellation = cancel the request (close its stream).
+ - *HTTP (legacy 2025-11-25):* adds `Mcp-Session-Id` capture/echo, re-init on
+ 404, best-effort DELETE on close. The optional GET notification stream opens
+ for eager servers advertising `tools.listChanged`, closes with the session,
+ retries after an established stream ends, and is replaced after legacy re-init.
+2. **JSON-RPC:** id allocation, pending-request table, per-request timeout
+ (`call_timeout_ms`), cancellation (`notifications/cancelled` on stdio),
+ late responses to cancelled ids ignored. Progress notifications are accepted
+ and ignored (no progress token is sent in v1).
+3. **Era negotiation:**
+ - Modern first: `server/discover` with 2026-07-28 `_meta`. Success → modern;
+ `UnsupportedProtocolVersion (-32022)` → pick from `supported`.
+ - Any other error, a recognizable legacy rejection, an empty/unrecognized
+ HTTP 400, or a probe timeout → legacy `initialize` (offer `2025-11-25`,
+ accept the server's counter-offer among versions we implement), then
+ `notifications/initialized`.
+ - The detected era is stored in the server's metadata cache entry to skip
+ the probe next time; if the cached era fails, re-probe.
+4. **Capabilities advertised:** none (`{}`): no elicitation, sampling, roots,
+ tasks. Client info `{name = "panto-mcp", version}`.
+5. **Server→client traffic (legacy):** answer `ping`; reply `-32601` to any
+ other request. Handle `notifications/tools|resources|prompts/list_changed`
+ by re-listing (eager: unregister/re-register; search: invalidate catalog;
+ prompts: re-register commands).
+
+## Local server lifecycle (no orphans)
+
+```mermaid
+sequenceDiagram
+ participant P as panto (extension)
+ participant W as supervisor (panto lua)
+ participant S as server
+ P->>W: spawn detached: fds 0,1,2 = server pipes, fd 3 = lifeline
+ W->>S: spawn detached (new process group), inheriting fds 0,1,2
+ W-->>W: close own copies of 0/1; report server pid on stderr marker
+ P->>S: JSON-RPC over the pipes (no relaying through W)
+ alt normal session_end / quit
+ P->>S: close stdin write end
+ P->>W: close lifeline (fd 3)
+ else panto dies (SIGKILL, crash, SIGHUP, OOM)
+ Note over P,W: kernel closes panto's fds: stdin EOF + lifeline EOF
+ end
+ W->>W: wait grace (2s) for S to exit
+ W->>S: SIGTERM process group, wait 2s
+ W->>S: SIGKILL process group
+ W-->>W: exit once group is gone
+```
+
+- One supervisor per local server: `uv.spawn(uv.exepath(), {"lua",
+ supervisor_path, ...})`, starts in ~16ms. `supervisor_path` resolved via
+ `package.searchpath` so it works from a rock or a checkout.
+- Both supervisor and server run in their own process groups (`detached`), so
+ terminal Ctrl+C / Ctrl+Z / SIGHUP do not hit them directly; panto's death is
+ observed only through the lifeline. The supervisor also traps
+ INT/TERM/HUP and runs the same teardown.
+- The server's group is signalled, catching `npx`/`docker`-style wrappers that
+ leave grandchildren.
+- panto never waits during `session_end`: it closes two fds and returns; process
+ handles are unref'd so they don't hold the loop.
+- **Accepted gap:** if panto and the supervisor are both SIGKILLed at once, a
+ server that ignores stdin EOF survives. No pidfile sweep (rare enough).
+- Precedent: `../panto-claudesub/panto-claudesub/_session.lua` (detached group,
+ bounded TERM → KILL escalation, confirmed exit).
+
+## Remote auth
+
+Order of preference per server: `auth = "<session>"` (static API key or OAuth
+session in `[auth.*]`), then automatic MCP OAuth on a 401.
+
+**Extension (MCP policy):**
+1. On 401, parse `WWW-Authenticate` for `resource_metadata`; else try the
+ RFC 9728 well-known PRM URIs (endpoint path, then origin root).
+2. Fetch authorization-server metadata (RFC 8414, then OIDC discovery
+ variants); require an exact `issuer` match and PKCE (`S256`) support.
+3. Client registration: configured `client_id` requires an exact `oauth.issuer`
+ binding (optional env-expanded `client_secret`) → public Dynamic Client Registration.
+ Confidential DCR is refused; no client secret is stored in `panto.ext.state`.
+ Public registrations are stored per issuer and retained on every failure. Any
+ configured issuer pins selection from authorization_servers, even without a client ID.
+ CIMD is deferred (`TODOs.md`).
+4. Define a runtime `oauth_pkce` session `mcp.<server>.<identity-hash>` (≤128 bytes),
+ using 128 bits of SHA-256 over canonical resource, exact issuer and client ID.
+ Retire the old name with core logout on changes. Use discovered endpoints,
+ challenge → PRM → configured scopes, and `resource` = canonical server URI.
+ Warm activation restores discovery without network work; no attestation flag.
+5. Login is user-initiated: `/mcp-login <server> [callback-url | --reregister]`.
+ Reuse a definition unless missing or core login fails; keep registrations on all
+ failures. Only a changed discovered issuer or explicit --reregister discards one.
+ The escape hatch rediscovers/registers once; errors mention it. Consent with an
+ existing usable token does not lock out tools, and give-up clears login_required.
+6. Authenticated 401: one forced refresh, then one retry; parallel calls await the
+ shared refresh. Scope 403 redefines the scope union and asks for re-consent,
+ without disabling working calls or an artificial attempt counter. Other 403s
+ report permission errors; explicit non-PKCE auth errors point to core auth.
+ Legacy GET reconnect backs off to 60 seconds and stops with a /mcp warning on 401.
+7. Root PRM fallback validates `resource` against the origin. Loopback discovery/
+ endpoints require a loopback resource; reject leading-zero IPv4 octets. Cache
+ status reads/poll at ~1 second and back off failed discovery from 5 to 60 seconds.
+
+**Core (credential mechanism, C4):** generic `oauth_pkce` auth type
+(authorization code + PKCE S256, `state` check, optional `iss` validation,
+`resource` parameter, loopback callback on an ephemeral port, **pasted
+callback URL fallback** for SSH/headless, token storage under `auth/`,
+refresh, header injection) plus `panto.ext` APIs to define a named session at
+runtime and start its login. Requests then use `http_request{auth = name}`;
+tokens never enter Lua.
+
+## Core additions (pantograph / libpanto)
+
+New `panto.ext` surfaces are co-designed with Travis before building; each
+passes the AGENTS.md tests (second consumer, no "mcp" in core names, mechanism
+not policy).
+
+| # | Addition | Second consumer |
+|---|---|---|
+| C1 | `register_tool` after activation + `unregister_tool(name)`. Changes take effect at the next turn (per-turn snapshot); in-flight calls unaffected; same collision rules; child agents sharing the Lua tool source see it. Needs copy-on-write declarations in the Lua `ToolSource` (provider serialization reads them off the owner thread). | Any extension whose tools depend on async discovery: plugin loaders, per-skill tools, a tool set that changes with project state. |
+| C2 | `register_command` after activation + `unregister_command(name)`; the fuzzy selector and `render_header` command list pick them up. | subagents' `/workflow:<name>` when profiles change on disk; skills exposed as commands. |
+| C3 | `http_request`: `method` (GET/POST/PUT/PATCH/DELETE), `body`; streaming via `on_head(head)` / `on_data(chunk)` hooks (presence of `on_data` disables buffering; final callback still fires with status/headers or error); `idle_timeout_ms`; no total cap for streams (`timeout_ms = 0`); keep-alive with a process-lifetime connection pool (callbacks still die with the runtime); auth allowed over plain HTTP to loopback; streams counted separately from the 16 buffered slots; bounded chunk queue for backpressure. | Lua protocol providers streaming a custom wire API; webhook/REST clients; the messaging bridge's long-poll. |
+| C4 | `oauth_pkce` auth type + runtime session definition and login APIs (above). Seeded `config.toml` documents the new auth fields. | Messaging bridge (Slack/Telegram OAuth); any extension calling an OAuth-protected API. |
+| C5 | *Only if spike S2 fails:* host-agent method to append arbitrary-role messages that persists and renders. | Fork/import tools, transcript surgery extensions, prompt-library extensions. |
+
+Draft Lua shape for C3 (to be co-designed):
+
+```lua
+local handle, err = panto.ext.http_request({
+ url = url, method = "POST", body = json, auth = "mcp.github",
+ headers = { ["content-type"] = "application/json",
+ accept = "application/json, text/event-stream" },
+ timeout_ms = 0, idle_timeout_ms = 60000,
+ on_head = function(head) end, -- {status, headers}
+ on_data = function(chunk) end, -- raw bytes; SSE parsing is the caller's
+}, function(response, err) end) -- buffered: {status, body, headers}; streamed: no body
+handle:cancel()
+```
+
+## Build phases
+
+Spikes first, then extension work proceeds in parallel with core work
+(stdio + search mode needs no core change).
+
+- [x] **S1 supervisor spike:** luv `spawn` with a 4th stdio slot from Lua;
+ detached supervisor + detached server; EOF propagation when panto is
+ `kill -9`ed; group kill of an `sh -c 'trap "" TERM; sleep …'` tree.
+- [x] **S2 conversation spike:** source-verified: asynchronous idle edits sync
+ through `onPaint`; `agent:submit` synchronizes before the TUI drains its
+ queued work. Assistant-tail prompts explicitly emit `agent_submission`
+ to wake that same render/persist path. C5 is unnecessary.
+- [x] **S3 tool-source mutability:** read libpanto's registry/`ToolSource`
+ snapshot path to size C1.
+- [x] **E1 core client:** rock skeleton (`panto-mcp` → `init.lua`,
+ `panto_mcp.*` modules), config parsing/expansion, JSON-RPC, stdio
+ transport + supervisor, era negotiation, catalog + metadata cache
+ (lazy start), `mcp.search` /
+ `mcp.call`, validator, result mapping, `session_end` teardown, `/mcp`
+ status command.
+- [x] **E2 code mode:** std.lua `mcp` namespace, `panto_mcp.docs`, system hint.
+- [x] **C3 HTTP** → **E3 HTTP transport:** modern POST/SSE, legacy sessions,
+ static headers and `auth = "<session>"`.
+- [x] **E4 resources:** list/templates/read, `mcp.read_resource`, resource blocks in
+ results, catalog search over resources.
+- [x] **C1/C2 dynamic registration** → **E5 eager mode** (including legacy HTTP
+ GET notifications) and **E6 prompts** (`/mcp:<server>:<prompt>`); no C5 fix.
+- [x] **C4 OAuth** → **E7 MCP OAuth:** PRM/AS discovery, configured/DCR registration,
+ `/mcp-login` (loopback + pasted callback), refresh-on-401 and non-blocking scope step-up;
+ issuer-bound configured clients, identity-scoped sessions, public-only DCR,
+ rediscovery/backoff/status caching, warm restoration and real-core TCP-fake checks.
+ Real-provider pass remains below.
+- [ ] **E8 release:** README, rockspec, mise tasks (copy panto-web's
+ `version`/`release`), luarocks upload. Files done; tag/upload left to Travis.
+- [x] Note on pantograph `TODOs.md` "User input for extensions" that it enables
+ MCP elicitation in panto-mcp.
+
+## Testing
+
+Per pantograph AGENTS.md: never test timeouts by sleeping; inject the clock /
+timer factory so timeouts fire deterministically.
+
+- `_selfcheck.lua` run with `panto lua` (like panto-web/panto-claudesub):
+ fake stdio servers written in Lua for both eras (modern discover; legacy
+ initialize; legacy server that errors on pre-initialize requests; server that
+ ignores EOF; server that spawns a grandchild), and a luv TCP fake for HTTP
+ (JSON response, SSE response, legacy session 404 → re-init, 401 → PRM).
+- Eager tools/prompts: warm activation is idle; a cold `mcp.search` connect
+ reconciles eager declarations; `list_changed` reconciles add/remove/schema
+ changes. Tests cover multi-server prompt preservation, `/compact` command
+ busy state, visible async failures, image-only media and all-message preflight,
+ raw one-argument remainders, invalid command/server names, stable eager names,
+ exact cache pruning, and legacy GET reconnect/re-init.
+- Validator, SSE parser, NDJSON framer, result mapping, name sanitizing, arg
+ parsing: table-driven unit checks in the same selfcheck.
+- OAuth: table-driven challenge/URL/issuer/S256/scope checks; injected HTTP/auth/timers
+ cover discovery → public DCR → user login → reconnect, refresh and scope upgrades.
+ Regressions cover foreign-issuer credentials, identity-scoped sessions, non-locking
+ 403s, concurrent refresh calls, confidential-DCR refusal, rediscovery and backoff.
+ `PANTO_MCP_HTTP_E2E=1` adds a new-core browser/pasted-callback flow, forced refresh,
+ scope re-consent and warm reuse against a local TCP fake (no real credentials).
+- Orphan check: spawn supervisor+server from a stand-in parent process,
+ `kill -9` the parent, wait on the server group's exit event (bounded), assert
+ nothing survives.
+- Core additions get Zig tests in pantograph/libpanto alongside their code.
+- One manual end-to-end pass against real servers: a filesystem stdio server
+ via `npx`, a remote legacy server, a remote 2026 server, and one OAuth server
+ (GitHub or Linear).
+
+## Resolved design checks
+
+- **Provider tool-name limits:** use 64 encoded wire bytes, the strictest declared
+ bound in the primary provider docs linked above.