tui plan

2 files changed, 489 insertions, 0 deletions

diff --git a/docs/libpanto-cleanup.md b/docs/archive/libpanto-cleanup.md
index 7c53e02..7c53e02 100644
--- a/docs/libpanto-cleanup.md
+++ b/docs/archive/libpanto-cleanup.md
diff --git a/docs/tui-refactor-plan.md b/docs/tui-refactor-plan.md
new file mode 100644
index 0000000..640db66
--- /dev/null
+++ b/docs/tui-refactor-plan.md
@@ -0,0 +1,489 @@
+# `panto` CLI/TUI Refactor Plan
+
+Status: **draft, in review**. A near-full rewrite of panto's terminal
+interface, heavily influenced by [pi](https://github.com/earendil-works/pi-mono)'s
+`pi-tui`. This document is the plan of record; iterate here before writing code.
+
+## 1. Goal & guiding principle
+
+Replace today's print-based `CLIRenderer` (`src/main.zig`) with a real terminal
+UI that *feels native* while supporting rich, dynamic, extension-controlled
+components.
+
+The defining principle, borrowed from pi: **a scrollback-preserving differential
+line renderer, not a full-screen TUI.**
+
+- We never switch to the terminal's alternate screen buffer. We render into the
+  normal buffer. Consequence: the user's shell scrollback before launching
+  `panto` survives, and scrolling up in tmux/Ghostty is just the terminal's own
+  native scroll — not something panto implements.
+- Each frame, components produce lines of text; the engine diffs against the
+  previously rendered lines and rewrites only what changed, moving the cursor up
+  with ANSI control codes and reprinting the minimal region. A spinner reprints
+  one line; collapsing all tool calls reprints the visible region.
+- Content that grows past the top of the viewport scrolls up into real native
+  scrollback. The engine only ever manipulates the visible terminal height.
+
+This is what produces the blend the design is chasing: native, print-style feel
+with full raw-mode dynamism layered on top.
+
+### Confidence note
+
+The mechanics below were derived by reading pi's compiled `pi-tui`
+(`tui.js`, `terminal.js`) directly — high confidence on *what pi does*. Claims
+about terminal/PTY behavior and performance ("uncapped redraw can be slower",
+"IMEs anchor to the hardware cursor") are reasoning + pi's own code comments,
+not independently benchmarked; flagged inline as **[verify]** where it matters.
+
+## 2. Architecture overview
+
+```
+                ┌───────────────────────────────────────────────┐
+                │ Streaming agent events (libpanto pull Stream) │
+                └─────────────────────────┬─────────────────────┘
+                                        │ deltas mutate component state
+                                        ▼
+   ┌───────────────┐   render(width)   ┌──────────────────┐  ANSI bytes  ┌───────────┐
+   │  Components   │ ───────────────►  │   TUI engine     │ ───────────► │ Terminal  │
+   │ (built-in &   │ ◄───────────────  │ (diff, viewport, │              │ (raw mode │
+   │  extension)   │  firstLineChanged │  cursor, write)  │ ◄─ raw input │  stdin)   │
+   └───────────────┘                   └──────────────────┘   keys/paste └───────────┘
+                                              ▲
+                                              │ input routing, focus, overlays
+                                        ┌─────┴───────┐
+                                        │ Input layer │ Kitty kbd protocol +
+                                        │             │ modifyOtherKeys fallback
+                                        └─────────────┘
+```
+
+Layers, bottom-up:
+
+1. **Terminal** — raw mode, stdin decoding, ANSI output, capability negotiation.
+2. **Input layer** — Kitty keyboard protocol negotiation with `modifyOtherKeys`
+   fallback, bracketed paste, `Key` decoding, keybinding lookup.
+3. **TUI engine** — owns the component list, runs the differential render,
+   tracks viewport/scrollback, composites overlays, positions the hardware
+   cursor for IME, routes input to the focused component.
+4. **Components** — built-in and extension-supplied. Produce lines from
+   structured state; receive input when focused.
+5. **App / chat loop** — wires libpanto's streaming events to component state
+   and `requestRender`.
+
+## 3. Rendering model (the core)
+
+### 3.1 Output discipline
+
+- **No alternate screen.** Never emit `?1049h`/`?47h`.
+- **Synchronized output.** Wrap every frame in `\x1b[?2026h` … `\x1b[?2026l`
+  so the terminal composites the whole frame atomically (no tearing/flicker).
+- **Full redraw only when forced:** first paint, terminal width change (wrapping
+  changes), terminal height change (viewport realignment). A full redraw clears
+  scrollback (`\x1b[2J\x1b[H\x1b[3J`); ordinary frames never do.
+- **Width contract.** Every line a component returns must have visible width
+  ≤ render width. The engine validates and treats overflow as a hard error
+  (caught, terminal restored, diagnostic written) — components must truncate.
+
+### 3.2 Viewport & scrollback tracking
+
+The engine tracks, across renders:
+
+- `previous_lines` — the last rendered line array (also the diff baseline and
+  the cache reused for clean-below-cut reprinting; see §3.3).
+- per-component **line offset and line count** (so a length change in one
+  component shifts everything below it, and offsets below are recomputed).
+- `viewport_top` / `max_lines_rendered` — the working area. Lines above
+  `viewport_top` have scrolled into native scrollback and are off-limits to
+  differential updates. If a change lands above `viewport_top`, the engine must
+  fall back to a full redraw.
+
+### 3.3 The dirty model (`firstLineChanged`) — primary fast path
+
+This is the component→engine contract that makes streaming cheap. It exists at
+**two layers** that optimize two different costs:
+
+- **Producing lines** (component CPU: markdown, highlight, wrap) — the expensive
+  one during streaming.
+- **Writing lines** (engine: diff + cursor + write) — cheap regardless.
+
+**Component-side signal:** each component exposes
+
+```
+firstLineChanged: ?usize   // lowest line index (component-local) whose
+                           // rendered output differs from the last render,
+                           // or null if nothing changed.
+```
+
+Defined as *lowest differing output line*, **not** "where text was appended."
+A markdown delta that restyles earlier lines (e.g. a closing code fence) makes
+those lines differ, so an honest implementation already reports the earlier
+index — no special reflow case is needed.
+
+**Engine pass (top-to-bottom, single walk):**
+
+1. Accumulate each component's global offset from its (possibly cached) line
+   count.
+2. Compute `cut = min(offset_i + firstLineChanged_i)` across **all** components
+   (not just the first non-null — a footer/spinner ticking mid-stream is the
+   common counter-example to "stop at first").
+3. Reprint from `cut` downward:
+   - components **above** the cut: untouched.
+   - the component **owning** the cut and any dirty component below it:
+     re-render from its local `firstLineChanged`.
+   - clean components **below** the cut (`firstLineChanged == null`): reuse their
+     **cached** lines verbatim — reprinted because they're below the rolled-back
+     point, but **not re-rendered** (no component CPU). The engine already holds
+     these lines for the diff baseline, so no new state.
+4. Handle length changes: if a component now returns fewer lines (e.g. ctrl+o
+   collapse), clear orphaned trailing lines and recompute offsets below.
+5. **Line-diff backstop.** From `cut` downward, still run the old-vs-new line
+   diff. `firstLineChanged` decides *where re-rendering starts*; the diff is the
+   *correctness floor* (handles length deltas, defends against an inaccurate
+   signal). The signal is the fast-path input; the diff is the guarantee.
+
+**Lifecycle (no desync):** `firstLineChanged` is *derived from the component's
+render cache*, mirroring pi's `invalidate()` model — state mutation invalidates
+the cache (dirties), a successful render re-populates it (cleans). It is not a
+separately hand-managed field that can drift.
+
+For streaming specifically, the cut is almost always deep in the last
+component, so per-frame work is: re-render one short tail + memcpy cached lines
+below + write from the cut. This is the "outperform the print CLI" path.
+
+### 3.4 Coalescing / frame rate
+
+- `requestRender()` schedules a frame; multiple requests within a window
+  coalesce into one render.
+- **Target ~120fps (≈8ms) adaptive:** render immediately when idle; coalesce
+  only under burst. Coalescing is a *ceiling* to avoid redrawing faster than the
+  terminal drains (which queues deltas and *grows* latency) — **[verify]** with
+  a P1 benchmark rather than removing the cap outright.
+
+### 3.5 Virtual cursor + hardware-cursor sync (IME) — kitchen sink
+
+The cursor is drawn by the component as part of its rendered output (a styled
+reverse-video block), not the terminal's hardware cursor. This keeps the cursor
+correct under differential rendering, lets it live inside wrapped text/overlays,
+and lets us style it.
+
+Because IMEs (CJK input, accent/emoji composition) anchor their candidate popup
+to the *hardware* cursor **[verify]**, we keep the hidden hardware cursor in sync
+with the virtual one:
+
+- A focused component emits an invisible marker (APC escape, zero-width) at the
+  virtual cursor's location.
+- After rendering, the engine scans the visible region for the marker, strips
+  it, computes its row/col, and moves the (normally hidden) hardware cursor
+  there.
+- `Focusable` is the interface a component implements to participate.
+
+Full scope from day one: composite the virtual cursor, keep hardware cursor
+synced. (If schedule slips, the safe deferral is to leave the hardware cursor at
+end-of-content and add marker-positioning later — but the interface ships now.)
+
+## 4. Core interfaces
+
+Idiomatic Zig interfaces (vtable-over-`*anyopaque` or tagged dispatch — TBD in
+§4.5). Designed to bridge cleanly to Lua now, and to a C ABI *later, elsewhere*
+(out of scope here, but the shapes stay coherent: lines-out as arrays, structured
+data in).
+
+### 4.1 `Component`
+
+```
+Component = struct {
+    // Produce lines for the given width. Each line's visible width <= width.
+    render: fn (self, width: usize, alloc) []const []const u8,
+
+    // Lowest component-local line index whose output changed since last render,
+    // or null. Derived from the render cache. See §3.3.
+    firstLineChanged: fn (self) ?usize,
+
+    // Drop cached render state (e.g. on theme change). Re-dirties.
+    invalidate: fn (self) void,
+
+    // Optional: receive input when focused.
+    handleInput: ?fn (self, data: []const u8) void,
+
+    // Optional: opt into key-release events (Kitty protocol).
+    wantsKeyRelease: bool = false,
+};
+```
+
+### 4.2 `Focusable` (virtual cursor / IME)
+
+```
+Focusable = struct {
+    focused: bool,          // set by engine on focus change
+    // The component emits CURSOR_MARKER at the cursor position in render()
+    // output when focused; the engine handles the rest (§3.5).
+};
+```
+
+Container components that embed an input must propagate `focused` to the child.
+
+### 4.3 Input types
+
+```
+Key = struct {
+    code: KeyCode,          // enum: char, enter, escape, tab, arrows, fn-keys…
+    mods: Mods,             // packed struct { ctrl, alt, shift, super, hyper… }
+    event: KeyEvent,        // press | repeat | release
+    text: ?[]const u8,      // resolved text for printable keys
+};
+```
+
+- Rich model from day one (modifiers + press/repeat/release), even though legacy
+  fallback can only populate a subset.
+- `Keybinding` storage: a lookup from `Key` pattern → action, owned by the app,
+  passed to components that need app-level bindings.
+
+### 4.4 Theme
+
+A **fresh, dedicated theme module** collecting all colors in one place (today's
+CLI just emits raw ANSI dim/cyan inline). Deep configurable theming is out of
+scope for this project — the goal is centralization, not a theming system. The
+module exposes a theme object of named foreground/background style functions
+(mirrors pi's `fg(name, text)` / `bg(name, text)`), passed to components at
+render. Markdown theming hooks reserved for the deferred markdown work (§8).
+
+### 4.5 Dispatch mechanism — decided: vtable
+
+**Vtable-of-fn-pointers over `*anyopaque`.** A tagged union is rejected: it
+requires knowing all component types up front, but extensions must be able to
+**define their own** components, not just replace built-ins (§7). The vtable is
+also the natural shape for the Lua bridge (and a future C ABI). Built-in and
+extension components are indistinguishable to the engine.
+
+## 5. Input layer
+
+- **Raw mode** via termios; restore on exit (and on crash — install handlers).
+- **Kitty keyboard protocol** negotiation at startup: request desired flags
+  (disambiguate, report-events, report-alternates), query, use if supported.
+  This is what enables unambiguous modifier chords (`Ctrl+I` ≠ `Tab`, lone
+  `Esc`, key releases).
+- **`modifyOtherKeys` fallback** (`\x1b[>4;2m`) for terminals without Kitty.
+- **Bracketed paste** (`\x1b[?2004h`): wrap pastes so the editor treats them as
+  literal text, not a stream of keypresses.
+- **Stdin buffering/splitting:** batched input must be split into individual
+  key sequences so `matchesKey`/release detection work (pi's `StdinBuffer`).
+- Disable/restore all of the above cleanly on stop and on suspend/resume.
+
+Platform note: panto targets Unix first (matches current build). Windows VT
+input is explicitly out of scope for this refactor.
+
+## 6. Built-in components
+
+All are **public** and **replaceable via events** (§7), and their render-to-lines
+logic is itself public so a handler can construct the default and wrap/filter its
+output. Each takes **structured data in** and produces **lines out**.
+
+> **Invariant — no "active component."** The engine holds a *list* of live
+> components; there is never a notion of "the current component." Multiple tool
+> calls (and future subagents) render in parallel, each bound to its own
+> instance/call-id. No code may stash `active_component` or reach for "the"
+> component of a kind — every event and delta carries its own instance identity.
+> Violating this corrupts parallel tool/subagent rendering.
+
+| Component | State in | Notes |
+|---|---|---|
+| **welcome** | version, cwd, model info | shown at session start |
+| **thinking text** | accumulating thinking deltas | dimmed; streams |
+| **assistant text** | accumulating assistant deltas | streams; markdown **deferred** (§8) but hosts it |
+| **user text** | submitted user message | |
+| **tool use** | tool name, args, status, result | **one component owns call + result**; collapsible (ctrl+o) — collapse is a length change (§3.3) |
+| **compaction summary** | compaction event data | shown when context is compacted |
+| **input box** | editor buffer, cursor | `Focusable`; single-row by default, **shift+enter inserts a newline** (enter submits), grows one row per line. **P1:** unbounded growth (no cap). **P2:** configurable cap (default 8) then scrolls within that window showing the last 8 contiguous lines. Raw-key editing |
+| **footer** | model, git branch, token/usage stats | persistent bottom line(s); during P1 also renders a **frame-timing element** (last frame's render time shown inverted as a theoretical-max "fps"), used to validate perf then removed |
+
+The chat transcript is a `Container` of these in order; the input box and footer
+are pinned below it.
+
+## 7. Extension UI surface — one event mechanism
+
+There is **one** mechanism for all extension UI: string-keyed events with
+handlers that can set/replace the component for that event. It subsumes slot
+replacement, custom tool rendering, and imperative component spawning.
+
+### 7.1 Events are open strings, not an enum
+
+Event types are **strings**. Extensions register handlers for, and **fire**,
+their own event types. Built-in events are simply event strings panto emits
+itself; extension events are mechanically identical.
+
+```
+panto.on("tool", handler)          // subscribe (built-in or custom event)
+panto.emit("my-event", data)       // fire — from a custom tool, a keybinding,
+                                   // or inside another event's handler
+```
+
+This is also the **only** way a component gets on screen: pick an event string,
+register a handler that sets your component, then emit the event. There is no
+separate `addComponent` API — component additions are *always* tied to an event
+firing. (Imperative "spawn a status panel on a timer" = define an event type and
+emit it.)
+
+### 7.2 The event object's UI API
+
+Handlers receive an `event` carrying structured data plus:
+
+```
+event.getComponent() -> ?Component   // the component currently chosen for this
+                                     // event: the built-in default at first,
+                                     // or whatever a prior handler set.
+event.setComponent(c: Component)     // set/replace it.
+// + structured data, e.g. event.tool_name, event.args, ...
+```
+
+`getComponent()` is named to make clear it returns *whatever is current*, not a
+frozen "default" — after the first handler runs it is no longer the default.
+
+**Wrapping is the documented, expected pattern:**
+
+```
+panto.on("tool", (e) => {
+  if (e.tool_name !== "skill") return;
+  const inner = e.getComponent();      // the default tool-use component
+  e.setComponent(wrapSkill(inner));    // decorate / replace
+});
+```
+
+### 7.3 Handler ordering & precedence
+
+Handlers for an event run in **registration order**. Precedence is
+**last-wins-blind**: the final `setComponent` is used. This is acceptable
+*because* the chain-friendly path (`getComponent` → wrap → `setComponent`) is the
+documented norm — a handler that clobbers without reading the current component
+is at fault, not the framework. No magic merge.
+
+### 7.4 Timing & lifecycle
+
+- An event fires **once, at its component-creation boundary**, *before* the
+  component first paints — so `setComponent` swaps it before anything renders.
+- For `tool`, that boundary is tool-use **start** (name known). The single
+  `tool` event covers the whole call: the chosen component is then **driven by
+  panto** with the streaming args/result/status as structured deltas. The
+  handler does **not** re-fire per delta; it chooses the component once.
+- All streamable components share this shape: fire once at start, hand over a
+  component, drive with deltas (§8).
+- **No "active component" (§6 invariant).** Each `tool`/subagent event yields its
+  own component instance keyed by call-id. Parallel calls each get their own.
+
+### 7.5 The skills example (worked)
+
+A skills extension:
+1. Registers the `skill` agent tool (existing tool machinery).
+2. Registers a `skill` component type.
+3. `panto.on("tool", ...)` and, when `event.tool_name === "skill"`, calls
+   `event.setComponent(skillComponent)` (optionally wrapping `getComponent()`).
+
+panto needs no concept of "skills." It emits `tool` at tool-use start; the
+handler claims the call by name and swaps the component; panto drives that
+instance with the tool's deltas. A subagents extension works identically.
+
+### 7.6 Zig → Lua bridge
+
+Implement the event system + component vtable in Zig (matching `libpanto`'s
+"shape the internal type to *be* the public API, then allowlist in
+`public.zig`" convention), then bridge to Lua via the existing
+`lua_bridge`/`lua_runtime` machinery. A Lua handler receives a bridged `event`
+object; a Lua-defined component implements the same
+`render`/`firstLineChanged`/`handleInput` vtable contract across the bridge.
+Native `.so`/C-ABI loading is **explicitly out of scope** for this project, but
+the vtable stays C-ABI-friendly so it can be added elsewhere later.
+
+## 8. Streaming integration
+
+- **Stateful component + `requestRender`** (no per-delta "append" method on the
+  interface). A delta arrives → append to the owning component's internal buffer
+  → mark dirty → `requestRender()`. The dirty model (§3.3) makes the resulting
+  frame minimal.
+- The app loop consumes libpanto's pull `Stream` (as `CLIRenderer.driveTurn`
+  does today) but, instead of writing to stdout, routes each event to component
+  state:
+  - thinking delta → thinking-text component
+  - text delta → assistant-text component
+  - tool-use start/args/result → tool-use component (one **per call**, keyed by
+    call-id; never "the" tool component — see §6 invariant)
+  - compaction/retry events → compaction component / status
+- Each component-creation boundary emits the corresponding built-in event (§7)
+  *before* first paint, so handlers can replace/wrap before anything renders.
+- **Append fast path** is handled entirely by `firstLineChanged` + cached
+  lines (§3.3): the streaming component re-renders only its dirty tail; the
+  engine reprints from the cut. No separate API surface.
+- **Markdown caveat (deferred):** when markdown lands, the assistant-text
+  component must cache finished blocks and only re-render the last open block, so
+  `firstLineChanged` stays near the tail instead of jumping to line 0 each token.
+  The component interface already accommodates this; only the markdown renderer
+  is deferred.
+
+## 9. Phasing
+
+**P1 — Engine + input + chat skeleton (the core bet).**
+- Terminal raw mode, ANSI output, synchronized output, capability detect.
+- Differential renderer: viewport/scrollback tracking, `firstLineChanged` dirty
+  model + cached-line reuse + line-diff backstop, full-redraw triggers.
+- Input layer: Kitty negotiation + `modifyOtherKeys` fallback + bracketed paste
+  + `Key` decoding.
+- Minimal components: assistant text, user text, input box (single-row +
+  shift+enter growth, **unbounded in P1** — no cap yet), footer.
+- Footer renders the frame-timing element (theoretical-max "fps" from the last
+  frame's render time) for perf validation; expected to sit well north of 120
+  always, after which the element is removed.
+- Wire libpanto streaming into component state.
+- **Benchmark streaming feel** vs. current print CLI; validate the 120fps /
+  coalescing target and the append fast path. **[verify]** the perf assumptions.
+  Methodology is left to the implementation agent's discretion — assumptions are
+  acceptable for now; we expect to refine once the streaming TUI is exercised in
+  practice.
+
+**P2 — Full built-in set + overlays + editor.**
+- Remaining built-ins: welcome, thinking, tool-use (collapsible), compaction.
+- Overlay system (modal components composited over base content; positioning,
+  focus stack).
+- Full editor: configurable line cap (default 8) with scroll-window past it
+  (show last 8 contiguous lines), word-nav, kill-ring/undo as warranted.
+- Footer with git branch + usage stats.
+
+**P3 — Extension components + cursor/IME polish.**
+- Event system (`on`/`emit`, `event.get/setComponent`); built-in events wired at
+  component-creation boundaries; public default components.
+- Zig event + component API → Lua bridge.
+- Virtual cursor compositing + hardware-cursor sync (IME marker positioning).
+
+**Deferred (post-refactor):**
+- Markdown + syntax highlighting renderer (hosting components designed for it
+  now; see §8 caveat).
+- Image rendering (Kitty/iTerm protocols).
+- Windows / native `.so` C-ABI extension loading.
+
+## 10. Open questions / decisions still pending
+
+All P1-blocking questions resolved. Remaining item (5) is non-blocking and left
+for the implementation agent to take a first swing at.
+
+5. **Built-in event taxonomy**: confirm the exact set/names of events panto
+   emits and their precise creation-boundary timing (draft in §8: `session_start`,
+   `user_message`, `thinking`, `assistant_text`, `tool`, `compaction`). The
+   implementation agent should take a first swing; we expect to correct it later
+   as the TUI streaming is put through its paces.
+
+*Resolved:*
+- Dispatch mechanism = vtable (§4.5); extension UI = one string-event
+  mechanism (§7); no "active component" (§6).
+- **(1) Editor depth**: single-row default; **shift+enter** inserts a newline
+  (enter submits); grows one row per line. **P1** ships height-1 + shift+enter
+  with *unbounded* growth (no cap — the cap and its scroll-window are one
+  feature, deferred together). **P2** adds the configurable cap (default 8) and
+  scroll-window showing the last 8 contiguous lines (§6 input box).
+- **(2) Theme source**: fresh dedicated theme module centralizing all colors;
+  no configurable theming system this project (§4.4).
+- **(3) App loop location**: new module; `main.zig` shrinks to wiring (§2/§9).
+- **(4) Benchmark methodology**: left to the implementation agent's discretion;
+  assumptions acceptable for now. Footer surfaces a frame-timing/"fps" element
+  during P1 to validate, then it's removed (§6 footer, §9 P1).
+
+## 11. Non-goals
+
+- Full-screen TUI / alternate screen.
+- Implementing our own scrollback or scroll handling (native terminal does it).
+- Windows support, image rendering, markdown — all explicitly later/deferred.