summaryrefslogtreecommitdiff
path: root/PLAN.md
blob: a0003c831c8789970a7db0cf80791f813d5b841f (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
# panto-mcp: plan

An MCP client for panto, shipped as an out-of-tree Lua extension (rock
`panto-mcp`, entry name `mcp`), plus the general-purpose panto core additions
it needs. Written 2026-11-16 from a planning session; readable without that
conversation.

Research reports referenced below were written to `/tmp` (volatile); the facts
this plan depends on are restated inline.

- `/tmp/mcp-research.md`: spec revisions, transports, lifecycle, auth, configs.
- `/tmp/mcp-discovery-research.md`: opencode/pi discovery, provider-native
  deferred tools, generic-dispatch prior art.
- `/tmp/mcp-optional-features.md`: prompts, resources, elicitation, tasks.
- `/tmp/whiteboard-mcp-optional-features.html`: whiteboard of the above.

## Goals

- Configure MCP servers in panto's layered `config.toml`: remote by URL, local
  by command.
- Speak the latest released revision (**2026-07-28**) and fall back to
  **2025-11-25** (and the 2025-06-18 / 2025-03-26 versions it negotiates down
  to). Most servers in the wild still speak the handshake revisions.
- Never leave orphaned server processes under any normal or abnormal panto
  exit (one accepted gap, below).
- Make large MCP catalogs cheap: deferred discovery through `mcp.search` /
  `mcp.call`, and "code mode" through an `mcp` namespace in `std.lua`.
- Tools, resources and prompts. Remote OAuth.

## Non-goals (v1)

| Feature | Why not | Revisit when |
|---|---|---|
| Elicitation | Needs core "user input for extensions" (TODOs.md). Not advertised, so servers must not send it; GitHub hides `delete_repository`. | After that core project; noted on its TODOs.md entry. |
| Tasks (2026 extension) | No popular server needs it for ordinary calls. Travis plans background-able tool calls first. | After background-able tool calls. |
| Tasks (2025 experimental wire) | Superseded, wire-incompatible. | Never, unless a concrete server requires it. |
| Sampling, roots, logging | Deprecated in 2026-07-28. Not advertised. | — |
| Resource subscriptions / `subscriptions/listen` | Long-lived stream, ack/demux/reconnect; on-demand reads suffice. | A real server workflow needs push updates. |
| 2024-11-05 HTTP+SSE transport | Deprecated; SHOULD NOT adopt. | A user hits a server that only speaks it. |
| Native provider deferred loading (`defer_loading`) | Only Anthropic Messages and OpenAI Responses have it; both still need the catalog up front. `mcp.search`/`mcp.call` is provider-neutral and cache-stable. | — |
| Importing `.mcp.json` / Claude Desktop configs | Not needed at first. | Later convenience. |
| Persistent broker surviving `/new`/`/resume`/`/reload` | Restarting is far simpler; remote servers dominate; switches are infrequent. | Startup cost becomes a real complaint. |
| Windows | panto doesn't target it. | — |

## Key facts that shape the design

- **stdio is newline-delimited JSON-RPC** on the child's stdin/stdout; stderr is
  free-form logging. No port handoff exists. A local server that speaks HTTP is
  configured by URL.
- **2026-07-28 is stateless.** No `initialize`, no sessions, no ping, no SSE
  resumability. Every request carries `_meta` protocol version, client
  capabilities and client info; servers MUST implement `server/discover`.
  Results carry `resultType`. HTTP is one POST per request, answered with JSON
  or a request-scoped SSE stream, with required `MCP-Protocol-Version`,
  `Mcp-Method` and (for `tools/call`, `resources/read`, `prompts/get`)
  `Mcp-Name` headers.
- **2025-11-25 is handshake-based:** `initialize` → `notifications/initialized`,
  optional `Mcp-Session-Id`, GET notification stream, DELETE to end a session.
- **MCP has no tool search.** All proposals were closed or are dormant; the
  official client guidance puts progressive discovery in the host.
- **Spec shutdown order for stdio:** close the server's stdin, wait, SIGTERM,
  wait, SIGKILL (current opencode: 2s / 2s, whole process group).
- **panto constraints found in code:**
  - Tools and commands are harvested only during `activate()`
    (`src/lua_runtime.zig` `activateEntry`); a `ToolSource`'s `tools` slice is
    fixed at registration.
  - `panto.ext.http_request` is GET-only, fully buffered, `keep_alive = false`,
    120s max timeout, 16 global concurrent requests, auth over HTTPS only
    (`src/lua_http.zig`). Its `Service` is created per Lua runtime and
    destroyed on every session switch (`src/live_session.zig`).
  - The TUI's SIGINT/SIGTERM/SIGHUP handlers restore the terminal and re-raise
    without running `session_end` (`src/tui_terminal.zig`), so extension
    cleanup cannot be relied on for abnormal exits.
  - `session_end` gets at most 16 non-blocking loop passes before teardown
    (`src/main.zig`), so it cannot wait out grace periods.
  - Core auth has API keys and device-code OAuth only (libpantograph
    `src/auth.zig`); no authorization-code + PKCE flow.
  - `agent:conversation()` on the host agent returns a borrowed, mutable view
    with `add_user_blocks` / `add_assistant_message`; `agent:submit` accepts
    blocks. Persistence and rendering of view edits are unverified (spike S2).

## Architecture

```mermaid
flowchart LR
  subgraph panto["panto process (Lua runtime, rebuilt per session)"]
    EXT["mcp extension<br/>config · catalog · search · validate"]
    NS["std.lua `mcp` namespace"]
    T["tools: mcp.search / mcp.call / mcp.read_resource<br/>+ eager mcp.&lt;server&gt;.&lt;tool&gt;"]
    C["commands: /mcp:&lt;server&gt;:&lt;prompt&gt;"]
    H["core http_request<br/>(process-lifetime pool)"]
    A["core auth (oauth_pkce)"]
    MC[("metadata cache<br/>(disk, per server)")]
  end
  SUP["panto lua supervisor<br/>(own process group)"]
  SRV["stdio server<br/>(own process group)"]
  REM["remote server"]
  EXT --- NS & T & C
  EXT <-->|"load at activation · save on connect"| MC
  EXT -- "stdin/stdout NDJSON (pipes)" --> SRV
  EXT -- "fd 3 lifeline" --> SUP
  SUP -- "owns / signals group" --> SRV
  EXT -- "POST · JSON or SSE" --> H --> REM
  H -. "Bearer" .- A
```

### Lifetime

One Lua runtime per session. Servers start **lazily**, driven by a disk
metadata cache, so a session switch normally spawns nothing. On `session_end`
the extension closes everything without waiting (the supervisor finishes local
teardown; HTTP requests are cancelled). Running servers simply restart on
their next use after `/new`, `/resume`, `/reload`.

**Metadata cache.** Every successful connect saves the server's discovery
result: detected era, server info/instructions, capabilities, and the full
tool, prompt, resource and resource-template lists.

- One JSON file per server, keyed by a **config fingerprint**: a hash (FNV in
  Lua) of the connection identity (`url`, or `command`/`args`/`env`/`cwd`,
  plus `auth`). A config change means a cold cache. Writes are atomic
  (temp file + rename).
- Location: `${XDG_DATA_HOME:-$HOME/.local/share}/panto-mcp/`, resolved in
  Lua with the same XDG rule panto uses (`$XDG_DATA_HOME` if set and
  non-empty, else `$HOME/.local/share`). The extension owns this directory;
  no core cache-dir API.
- Replaces the earlier plan to keep the detected era in `panto.ext.state`.

**Activation:**

- **Warm cache** (entry with a matching fingerprint): register
  `/mcp:<server>:<prompt>` commands and any eager `mcp.<server>.<tool>` tools
  straight from the cache. **No process starts.** Eager tools are therefore
  present from the first turn, without post-activation registration.
- **Cold cache** (first run, or config changed): connect that server in the
  background (non-blocking) to populate the cache, then register its prompts
  and eager tools post-activation (core C1/C2).

**What starts a server:** `mcp.call`, an eager tool call,
`mcp.read_resource`, running a prompt command, or `/mcp refresh [server]`.
`mcp.search` searches the cache and starts nothing (a cold server still
connecting is waited for). Every connect re-lists everything, rewrites the
cache, and reconciles registrations: new prompts/eager tools are registered,
removed ones unregistered (C1/C2; an eager change busts the prompt cache once,
which is acceptable for a rare server-side change).

**Staleness:** the cache is refreshed on every connect (no `ttlMs`-based
expiry in v1). `mcp.call` validates against the **live** schema, since the
connect it triggers re-lists first. A tool missing after the refresh errors
with the closest matches; a removed prompt fails on `prompts/get` and its
command is unregistered.

A call to a server that is still connecting waits for it (bounded by the
server's startup timeout). A failed connection is retried lazily on the next
use, with the error visible in `mcp.search` / `/mcp` status.

### Configuration

Extension-owned tables, read from `panto.ext.config` (merged layers; rigs and
project configs may define servers, which is no worse than shipping Lua).

```toml
[mcp]
startup_timeout_ms = 30000     # per server: spawn/connect + discovery
call_timeout_ms = 300000       # per request; progress does not extend past this
idle_timeout_ms = 60000        # HTTP streams: max silence between bytes

[mcp.servers.github]
url = "https://api.githubcopilot.com/mcp/"
expose = "search"              # "search" (default) | "eager"
auth = "mcp.github"            # optional: named panto [auth.*] session
headers = { "X-Tenant" = "acme" }   # non-secret; ${env:VAR} expanded
enabled = true
enabled_tools = []             # empty = all
disabled_tools = ["delete_*"]  # glob over MCP tool names; applied after enabled_tools

[mcp.servers.fs]
command = "npx"
args = ["-y", "@modelcontextprotocol/server-filesystem", "${project_root}"]
env = { "TOKEN" = "${env:FS_TOKEN}" }
cwd = "."                      # relative to project root
```

- Exactly one of `url` / `command` per server; invalid servers are reported and
  skipped, never fatal to panto.
- `panto.ext.config` leaves `${env:VAR}` literal; the extension expands it in
  `args`, `env`, `headers`, `url`. `${project_root}` comes from
  `panto.ext.dirs`. Missing env → empty string, matching core semantics.
- Server names use `^[A-Za-z0-9_-]+$`, but also cannot contain `__` or end in
  `_`: panto decodes `__` as a provider-name separator, so those names make
  server/tool boundaries ambiguous. Invalid server entries are warned about
  and skipped, never fatal to panto.
- Secrets for remote servers should go through `auth = "<session>"` so they
  never pass through Lua; `headers` with `${env:}` remain possible.
- For automatic OAuth, omit `auth`; optional `[mcp.servers.<name>.oauth]` accepts
  `client_id`, required matching `issuer` with that ID, optional `client_secret`
  (`${env:VAR}` expanded) and fallback `scopes`. Otherwise use public DCR.
- Child env: inherit panto's environment plus `env`. (Codex-style `env_vars`
  allowlisting is skipped; add if needed.)

### Tool surface and token cost

Every registered tool costs context tokens, so nothing is unconditional.

| Exposed to model | When |
|---|---|
| `mcp.search`, `mcp.call`, `mcp.read_resource` | At activation, iff ≥1 enabled server has `expose = "search"`. Static declarations: cache-stable for the session. |
| `mcp.<server>.<tool>` per eager tool | At activation from a warm cache; otherwise after that eager server's first connect (post-activation registration, C1), visible from the next provider request. |
| `mcp.read_resource` (eager-only configs) | Registered when the first eager server (cached or live) advertises `resources`. |
| std.lua `mcp` namespace | Iff ≥1 enabled server. No tool cost; a short hint. |
| System hint | Iff ≥1 enabled server: one line per server (name, transport, mode) built from **config only**, so it never changes mid-session; points at search tools when exposed, otherwise at eager tools and docs. |

Eager tools can miss the first turn only on a cold cache (the user submits
before the first-ever connect finishes). Search mode has no such race
(`mcp.call` waits).

**`mcp.search {query?, server?, limit?}`**: connects the relevant servers if
needed and searches the cached catalog (tools, resources, resource templates)
client-side. Ranking: simple term scoring over name, title, description and
parameter names/descriptions (BM25-ish; no dependency). Each hit returns
`server`, `name`, title/description and the **full input schema**, so search
doubles as describe. Server `instructions` (from discover/initialize) are
returned when `server` is given, truncated.

**`mcp.call {server, tool, input}`**: exact `server` + `tool` only; never fuzzy
execution. Unknown names return an error listing the closest matches.
`input` is validated against the tool's live input schema before sending;
failure returns the validation errors plus the schema. Declared non-strict
(an `any`-shaped `input` cannot be expressed in OpenAI/Anthropic strict mode).

**`mcp.read_resource {server, uri}`**: `resources/read` on the originating server.
Never fetches arbitrary URLs itself; a `resource_link` in a tool result names
its server.

Eager tool names are `mcp.<server>.<tool>`, with the MCP tool name sanitized
to `[A-Za-z0-9_-]` and `__` collapsed (panto encodes `.` as `__` on the wire).
Every tool gets a suffix derived only from its own MCP name, so adding another
tool cannot rename it. Truncation keeps the encoded wire name within **64
bytes**. The limit is the strictest declared provider cap checked: Anthropic
Messages tool names allow 128 characters; OpenAI Chat Completions function
names cap at 64; OpenAI Responses documents function names as strings without
a stated limit. Sources: [Anthropic Messages](https://platform.claude.com/docs/en/api/http/messages/create),
[OpenAI Chat Completions](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create),
[OpenAI Responses](https://developers.openai.com/api/reference/resources/responses/methods/create).
Eager tools are validated the same way as `mcp.call` (panto has no schema
validation of its own).

### std.lua `mcp` namespace ("code mode")

Registered with `require("_std").lua_namespace("mcp", factory)`; available to
child agents too. Functions run in the std.lua handler coroutine and yield on
luv:

```lua
mcp.servers()                       -- name, transport, mode, era, state, error
mcp.search(query, {server=, limit=})
mcp.call(server, tool, input)       -- validated; returns {text, attachments, structured, is_error}
mcp.start(server, tool, input)      -- handle; handle:await(); for parallel calls
mcp.read_resource(server, uri)
mcp.prompts(server), mcp.prompt(server, name, args)
```

Reference docs ship as a require-able module (`require("panto_mcp.docs")`),
like `subagents.docs` and `_std.docs`: discovery, examples, result shapes,
errors. Only the one-line hint is injected. Per-server proxy sugar
(`mcp.github.create_issue{}`) is skipped; add it if models reach for it.

### Input validation

A pure-Lua JSON Schema subset validator, ported from panto-subagents'
`workflow.lua` (copied, not a dependency). Covers `type`, `properties`,
`required`, `additionalProperties`, `enum`/`const`, `items`/`prefixItems`,
numeric/string/array bounds, `anyOf`/`oneOf`/`allOf`/`not`, local `$ref`/
`$defs`. Unknown keywords (including `pattern`/`format` if unsupported) are
**ignored, not rejected**: validation exists to catch cheap mistakes, and a
false rejection would block a valid call. Errors carry JSON-pointer paths.

### Result mapping (tool calls, resources, prompts)

| MCP content | panto result |
|---|---|
| `text` | text |
| `image` | attachment (base64 decoded in Lua; size-capped) |
| `audio` | text placeholder with MIME type and size |
| `resource` (embedded text) | text, prefixed with its URI |
| `resource` (embedded blob) | attachment if image, else placeholder |
| `resource_link` | text line: name, URI, description, and "read with `mcp.read_resource{server=…, uri=…}`" (or `mcp.read_resource` in std.lua) |
| `structuredContent` | JSON text, only if no text block already carries it |
| `isError: true` | tool error (model-visible) with the content |
| JSON-RPC `error` | tool error: `<server>: <message> (<code>)` |
| `resultType: input_required` | tool error: server requested user input, which panto does not support yet |

Output is capped (configurable later if needed); truncation is stated in the
result.

### Prompts

- Each server prompt becomes a slash command **`/mcp:<server>:<prompt>`**,
  registered at activation from the metadata cache, or after the first connect
  on a cold cache (core C2), and unregistered if a later connect shows it gone. Description = prompt title/description plus its argument names, so the
  fuzzy command selector finds it.
- Arguments: `name=value` tokens with shell-style quoting; a prompt with exactly
  one argument accepts the trimmed raw remainder when no matching `name=` token
  appears. Missing required arguments → error listing them.
- On invoke, start `prompts/get` in an async coroutine from the synchronous
  command callback. Refuse if a turn or non-prompt command is active, and
  re-check when the result arrives. Prompt responses append real user/assistant
  roles to the borrowed host conversation; a trailing user message is submitted
  to start a turn. Assistant media is represented by text placeholders.
- Prompt names the core rejects (slash/whitespace) are skipped with a warning
  visible in `/mcp`. Async failures mount a visible error above the input until
  the next `command_start`. One-argument positional values preserve the trimmed
  raw remainder when no matching `name=` argument is present.
- **Assistant-tail rule:** if the prompt does not end with a user message,
  append all messages and skip `agent:submit`. Emit `agent_submission` so the
  idle TUI sync renders and persists the tail without starting a turn.
- All prompt message blocks are mapped before conversation mutation. Only
  `image/*` content becomes user media; assistant media and non-image blobs
  become text placeholders. This avoids partial history from rejected media.

- **Conversation sync verified in ws-1:** idle `onPaint` syncs the conversation
  tail and calls `flushPersist`; idle submission draining syncs/persists before
  queued `agent:submit` work starts. Assistant-tail prompts emit
  `agent_submission` to request that same idle sync. No C5 core change is needed.

### Resources

Built in v1 because tool results can carry `resource_link`s that are only
usable with `resources/read`.

- Catalog: `resources/list` and `resources/templates/list` (paginated, cached
  per `ttlMs`), searchable through `mcp.search`.
- Reading: `mcp.read_resource` tool and namespace function; templates are filled by the
  model/user into a concrete URI, then read.
- Only servers advertising `resources` are queried. Some legacy clients treat
  `-32601` from list methods as fatal; we never call unadvertised methods.

## Protocol client

Layers (each a module; transports are interchangeable under the client):

1. **Transport**
   - *stdio:* NDJSON framing over luv pipes; partial-line buffering; stderr
     captured into a bounded per-server ring for diagnostics (never treated as
     an error signal).
   - *HTTP (modern):* POST per request with the required headers; response is
     JSON or a request-scoped SSE stream parsed in Lua. `x-mcp-header` tool
     annotations are validated and mirrored as `Mcp-Param-*` headers.
     Cancellation = cancel the request (close its stream).
   - *HTTP (legacy 2025-11-25):* adds `Mcp-Session-Id` capture/echo, re-init on
     404, best-effort DELETE on close. The optional GET notification stream opens
     for eager servers advertising `tools.listChanged`, closes with the session,
     retries after an established stream ends, and is replaced after legacy re-init.
2. **JSON-RPC:** id allocation, pending-request table, per-request timeout
   (`call_timeout_ms`), cancellation (`notifications/cancelled` on stdio),
   late responses to cancelled ids ignored. Progress notifications are accepted
   and ignored (no progress token is sent in v1).
3. **Era negotiation:**
   - Modern first: `server/discover` with 2026-07-28 `_meta`. Success → modern;
     `UnsupportedProtocolVersion (-32022)` → pick from `supported`.
   - Any other error, a recognizable legacy rejection, an empty/unrecognized
     HTTP 400, or a probe timeout → legacy `initialize` (offer `2025-11-25`,
     accept the server's counter-offer among versions we implement), then
     `notifications/initialized`.
   - The detected era is stored in the server's metadata cache entry to skip
     the probe next time; if the cached era fails, re-probe.
4. **Capabilities advertised:** none (`{}`): no elicitation, sampling, roots,
   tasks. Client info `{name = "panto-mcp", version}`.
5. **Server→client traffic (legacy):** answer `ping`; reply `-32601` to any
   other request. Handle `notifications/tools|resources|prompts/list_changed`
   by re-listing (eager: unregister/re-register; search: invalidate catalog;
   prompts: re-register commands).

## Local server lifecycle (no orphans)

```mermaid
sequenceDiagram
  participant P as panto (extension)
  participant W as supervisor (panto lua)
  participant S as server
  P->>W: spawn detached: fds 0,1,2 = server pipes, fd 3 = lifeline
  W->>S: spawn detached (new process group), inheriting fds 0,1,2
  W-->>W: close own copies of 0/1; report server pid on stderr marker
  P->>S: JSON-RPC over the pipes (no relaying through W)
  alt normal session_end / quit
    P->>S: close stdin write end
    P->>W: close lifeline (fd 3)
  else panto dies (SIGKILL, crash, SIGHUP, OOM)
    Note over P,W: kernel closes panto's fds: stdin EOF + lifeline EOF
  end
  W->>W: wait grace (2s) for S to exit
  W->>S: SIGTERM process group, wait 2s
  W->>S: SIGKILL process group
  W-->>W: exit once group is gone
```

- One supervisor per local server: `uv.spawn(uv.exepath(), {"lua",
  supervisor_path, ...})`, starts in ~16ms. `supervisor_path` resolved via
  `package.searchpath` so it works from a rock or a checkout.
- Both supervisor and server run in their own process groups (`detached`), so
  terminal Ctrl+C / Ctrl+Z / SIGHUP do not hit them directly; panto's death is
  observed only through the lifeline. The supervisor also traps
  INT/TERM/HUP and runs the same teardown.
- The server's group is signalled, catching `npx`/`docker`-style wrappers that
  leave grandchildren.
- panto never waits during `session_end`: it closes two fds and returns; process
  handles are unref'd so they don't hold the loop.
- **Accepted gap:** if panto and the supervisor are both SIGKILLed at once, a
  server that ignores stdin EOF survives. No pidfile sweep (rare enough).
- Precedent: `../panto-claudesub/panto-claudesub/_session.lua` (detached group,
  bounded TERM → KILL escalation, confirmed exit).

## Remote auth

Order of preference per server: `auth = "<session>"` (static API key or OAuth
session in `[auth.*]`), then automatic MCP OAuth on a 401.

**Extension (MCP policy):**
1. On 401, parse `WWW-Authenticate` for `resource_metadata`; else try the
   RFC 9728 well-known PRM URIs (endpoint path, then origin root).
2. Fetch authorization-server metadata (RFC 8414, then OIDC discovery
   variants); require an exact `issuer` match and PKCE (`S256`) support.
3. Client registration: configured `client_id` requires an exact `oauth.issuer`
   binding (optional env-expanded `client_secret`) → public Dynamic Client Registration.
   Confidential DCR is refused; no client secret is stored in `panto.ext.state`.
   Public registrations are stored per issuer and retained on every failure. Any
   configured issuer pins selection from authorization_servers, even without a client ID.
   CIMD is deferred (`TODOs.md`).
4. Define a runtime `oauth_pkce` session `mcp.<server>.<identity-hash>` (≤128 bytes),
   using 128 bits of SHA-256 over canonical resource, exact issuer and client ID.
   Retire the old name with core logout on changes. Use discovered endpoints,
   challenge → PRM → configured scopes, and `resource` = canonical server URI.
   Warm activation restores discovery without network work; no attestation flag.
5. Login is user-initiated: `/mcp-login <server> [callback-url | --reregister]`.
   Reuse a definition unless missing or core login fails; keep registrations on all
   failures. Only a changed discovered issuer or explicit --reregister discards one.
   The escape hatch rediscovers/registers once; errors mention it. Consent with an
   existing usable token does not lock out tools, and give-up clears login_required.
6. Authenticated 401: one forced refresh, then one retry; parallel calls await the
   shared refresh. Scope 403 redefines the scope union and asks for re-consent,
   without disabling working calls or an artificial attempt counter. Other 403s
   report permission errors; explicit non-PKCE auth errors point to core auth.
   Legacy GET reconnect backs off to 60 seconds and stops with a /mcp warning on 401.
7. Root PRM fallback validates `resource` against the origin. Loopback discovery/
   endpoints require a loopback resource; reject leading-zero IPv4 octets. Cache
   status reads/poll at ~1 second and back off failed discovery from 5 to 60 seconds.

**Core (credential mechanism, C4):** generic `oauth_pkce` auth type
(authorization code + PKCE S256, `state` check, optional `iss` validation,
`resource` parameter, loopback callback on an ephemeral port, **pasted
callback URL fallback** for SSH/headless, token storage under `auth/`,
refresh, header injection) plus `panto.ext` APIs to define a named session at
runtime and start its login. Requests then use `http_request{auth = name}`;
tokens never enter Lua.

## Core additions (pantograph / libpanto)

New `panto.ext` surfaces are co-designed with Travis before building; each
passes the AGENTS.md tests (second consumer, no "mcp" in core names, mechanism
not policy).

| # | Addition | Second consumer |
|---|---|---|
| C1 | `register_tool` after activation + `unregister_tool(name)`. Changes take effect at the next turn (per-turn snapshot); in-flight calls unaffected; same collision rules; child agents sharing the Lua tool source see it. Needs copy-on-write declarations in the Lua `ToolSource` (provider serialization reads them off the owner thread). | Any extension whose tools depend on async discovery: plugin loaders, per-skill tools, a tool set that changes with project state. |
| C2 | `register_command` after activation + `unregister_command(name)`; the fuzzy selector and `render_header` command list pick them up. | subagents' `/workflow:<name>` when profiles change on disk; skills exposed as commands. |
| C3 | `http_request`: `method` (GET/POST/PUT/PATCH/DELETE), `body`; streaming via `on_head(head)` / `on_data(chunk)` hooks (presence of `on_data` disables buffering; final callback still fires with status/headers or error); `idle_timeout_ms`; no total cap for streams (`timeout_ms = 0`); keep-alive with a process-lifetime connection pool (callbacks still die with the runtime); auth allowed over plain HTTP to loopback; streams counted separately from the 16 buffered slots; bounded chunk queue for backpressure. | Lua protocol providers streaming a custom wire API; webhook/REST clients; the messaging bridge's long-poll. |
| C4 | `oauth_pkce` auth type + runtime session definition and login APIs (above). Seeded `config.toml` documents the new auth fields. | Messaging bridge (Slack/Telegram OAuth); any extension calling an OAuth-protected API. |
| C5 | *Only if spike S2 fails:* host-agent method to append arbitrary-role messages that persists and renders. | Fork/import tools, transcript surgery extensions, prompt-library extensions. |

Draft Lua shape for C3 (to be co-designed):

```lua
local handle, err = panto.ext.http_request({
  url = url, method = "POST", body = json, auth = "mcp.github",
  headers = { ["content-type"] = "application/json",
              accept = "application/json, text/event-stream" },
  timeout_ms = 0, idle_timeout_ms = 60000,
  on_head = function(head) end,      -- {status, headers}
  on_data = function(chunk) end,     -- raw bytes; SSE parsing is the caller's
}, function(response, err) end)      -- buffered: {status, body, headers}; streamed: no body
handle:cancel()
```

## Build phases

Spikes first, then extension work proceeds in parallel with core work
(stdio + search mode needs no core change).

- [x] **S1 supervisor spike:** luv `spawn` with a 4th stdio slot from Lua;
      detached supervisor + detached server; EOF propagation when panto is
      `kill -9`ed; group kill of an `sh -c 'trap "" TERM; sleep …'` tree.
- [x] **S2 conversation spike:** source-verified: asynchronous idle edits sync
      through `onPaint`; `agent:submit` synchronizes before the TUI drains its
      queued work. Assistant-tail prompts explicitly emit `agent_submission`
      to wake that same render/persist path. C5 is unnecessary.
- [x] **S3 tool-source mutability:** read libpanto's registry/`ToolSource`
      snapshot path to size C1.
- [x] **E1 core client:** rock skeleton (`panto-mcp` → `init.lua`,
      `panto_mcp.*` modules), config parsing/expansion, JSON-RPC, stdio
      transport + supervisor, era negotiation, catalog + metadata cache
      (lazy start), `mcp.search` /
      `mcp.call`, validator, result mapping, `session_end` teardown, `/mcp`
      status command.
- [x] **E2 code mode:** std.lua `mcp` namespace, `panto_mcp.docs`, system hint.
- [x] **C3 HTTP** → **E3 HTTP transport:** modern POST/SSE, legacy sessions,
      static headers and `auth = "<session>"`.
- [x] **E4 resources:** list/templates/read, `mcp.read_resource`, resource blocks in
      results, catalog search over resources.
- [x] **C1/C2 dynamic registration** → **E5 eager mode** (including legacy HTTP
      GET notifications) and **E6 prompts** (`/mcp:<server>:<prompt>`); no C5 fix.
- [x] **C4 OAuth** → **E7 MCP OAuth:** PRM/AS discovery, configured/DCR registration,
      `/mcp-login` (loopback + pasted callback), refresh-on-401 and non-blocking scope step-up;
      issuer-bound configured clients, identity-scoped sessions, public-only DCR,
      rediscovery/backoff/status caching, warm restoration and real-core TCP-fake checks.
      Real-provider pass remains below.
- [ ] **E8 release:** README, rockspec, mise tasks (copy panto-web's
      `version`/`release`), luarocks upload. Files done; tag/upload left to Travis.
- [x] Note on pantograph `TODOs.md` "User input for extensions" that it enables
      MCP elicitation in panto-mcp.

## Testing

Per pantograph AGENTS.md: never test timeouts by sleeping; inject the clock /
timer factory so timeouts fire deterministically.

- `_selfcheck.lua` run with `panto lua` (like panto-web/panto-claudesub):
  fake stdio servers written in Lua for both eras (modern discover; legacy
  initialize; legacy server that errors on pre-initialize requests; server that
  ignores EOF; server that spawns a grandchild), and a luv TCP fake for HTTP
  (JSON response, SSE response, legacy session 404 → re-init, 401 → PRM).
- Eager tools/prompts: warm activation is idle; a cold `mcp.search` connect
  reconciles eager declarations; `list_changed` reconciles add/remove/schema
  changes. Tests cover multi-server prompt preservation, `/compact` command
  busy state, visible async failures, image-only media and all-message preflight,
  raw one-argument remainders, invalid command/server names, stable eager names,
  exact cache pruning, and legacy GET reconnect/re-init.
- Validator, SSE parser, NDJSON framer, result mapping, name sanitizing, arg
  parsing: table-driven unit checks in the same selfcheck.
- OAuth: table-driven challenge/URL/issuer/S256/scope checks; injected HTTP/auth/timers
  cover discovery → public DCR → user login → reconnect, refresh and scope upgrades.
  Regressions cover foreign-issuer credentials, identity-scoped sessions, non-locking
  403s, concurrent refresh calls, confidential-DCR refusal, rediscovery and backoff.
  `PANTO_MCP_HTTP_E2E=1` adds a new-core browser/pasted-callback flow, forced refresh,
  scope re-consent and warm reuse against a local TCP fake (no real credentials).
- Orphan check: spawn supervisor+server from a stand-in parent process,
  `kill -9` the parent, wait on the server group's exit event (bounded), assert
  nothing survives.
- Core additions get Zig tests in pantograph/libpanto alongside their code.
- One manual end-to-end pass against real servers: a filesystem stdio server
  via `npx`, a remote legacy server, a remote 2026 server, and one OAuth server
  (GitHub or Linear).

## Resolved design checks

- **Provider tool-name limits:** use 64 encoded wire bytes, the strictest declared
  bound in the primary provider docs linked above.