initial commit, moved libpanto over from the pantograph repo

2 files changed, 533 insertions, 0 deletions

diff --git a/docs/libpanto-bindings.md b/docs/libpanto-bindings.md
new file mode 100644
index 0000000..2872fde
--- /dev/null
+++ b/docs/libpanto-bindings.md
@@ -0,0 +1,517 @@
+# Plan: `libpanto` language bindings (Go + Python + Lua)
+
+## Goal
+
+Expose `libpanto` to other languages, targeting **Go**, **Python**, and **Lua**
+first. The bindings are organized as a family of sibling packages, named
+uniformly:
+
+| package        | language | role                                                          |
+| -------------- | -------- | ------------------------------------------------------------- |
+| `libpanto`     | Zig      | the core library (exists today)                               |
+| `libpanto-c`   | Zig      | a C-ABI shared library + header wrapping the public Zig API   |
+| `libpanto-go`  | Go       | idiomatic Go bindings over `libpanto-c` via cgo               |
+| `libpanto-py`  | Zig      | a CPython extension implemented in pure Zig (`@cImport`)      |
+| `libpanto-lua` | Zig      | a Lua C-module implemented in pure Zig (`@cImport`)           |
+
+Three consumers, two paths to the core:
+
+- **Go → `libpanto-c` → `libpanto`.** cgo is effectively the only option, and
+  cgo can only call C. So a C ABI is mandatory.
+- **Python → `libpanto`** directly. `libpanto-py` is a native CPython
+  extension written *in Zig*: it `@cImport`s `Python.h`, builds `PyObject`
+  glue against the translated C types, and `build.zig` emits the loadable
+  `.so`. It calls the Zig API directly and **does not** depend on `libpanto-c`.
+- **Lua → `libpanto`** directly. `libpanto-lua` is the same pattern as
+  `libpanto-py`, one rung over: a native Lua C-module written *in Zig* that
+  `@cImport`s `lua.h` / `lauxlib.h`, builds the `luaL_Reg` tables and userdata
+  types against the translated C types, and `build.zig` emits a loadable
+  `panto.so` (no `lib` prefix) exporting `luaopen_panto`. Discovered via
+  `package.cpath`, loaded by `require('panto')`. It calls the Zig API directly
+  and **does not** depend on `libpanto-c`. This is also the bindings package
+  that the panto CLI's *own* embedded Lua extension environment consumes — see
+  "Lua: one `require('panto')`, two surfaces" below.
+
+### Lua version targeting: no stable ABI, so pick a version
+
+Unlike CPython (where `abi3` collapses the version axis to one artifact),
+**Lua has no stable-ABI escape hatch.** Lua guarantees binary compatibility
+only across *bugfix* releases of one version (e.g. 5.4.1 ↔ 5.4.8); across
+versions there is **no ABI compatibility and the C API signatures themselves
+differ** (`lua_resume`, `lua_pcall`/`lua_pcallk`, `lua_load` gained params; the
+globals model changed from `_G` to `_ENV` upvalues in 5.2+). A module compiled
+for the wrong version fails loud at `require` time (typically
+`undefined symbol: luaL_checkversion`), not silently. So the artifact matrix is
+`{Lua version} × platform`, and supporting more than one version means
+conditional compilation guarded by `LUA_VERSION_NUM` (plus a `lua-compat`-style
+shim layer) producing **separate per-version binaries** — never one fat object.
+
+**v1 targets Lua 5.4 only.** That is the established current version (5.5.0
+shipped 22 Dec 2025 and is too new to matter yet) and, critically, it is the
+version of the panto CLI's embedded Lua, so it directly unblocks libpanto
+features for CLI extensions. **5.3, 5.2, and earlier are explicitly out of
+scope** — all EOL, with adoption draining to 5.4.
+
+Two shim efforts are **deferred** (see "Out of scope for v1"):
+
+- a **LuaJIT (Lua 5.1 ABI)** artifact, important for the large LuaJIT-based
+  ecosystem; and
+- a **Lua 5.5** artifact, at which point we'll seriously consider bumping the
+  CLI's embedded Lua to 5.5 as well.
+
+Note on the LuaJIT shim specifically: LuaJIT is ABI-compatible with Lua 5.1 but
+*selectively backports* pieces of 5.2/5.3 (`goto`, `load()`, some C API like
+`luaL_setfuncs`), and **preprocessor detection of which backports are present
+is not reliable**. So that deferred work is a *LuaJIT-target* shim validated
+against actual LuaJIT — not a clean textbook "Lua 5.1.5" build.
+
+### Why this split (decisions settled)
+
+- **`libpanto-c` is built regardless**, because Go requires it. It doubles as
+  the stable-ABI validation platform for the C surface.
+- **No cffi.** A pure-Python cffi binding was considered and rejected. Its only
+  real appeal (no native build step, trivial wheels) holds solidly *only* in
+  ABI mode, which is also where streaming callbacks and silent struct/ABI
+  drift bite hardest; API mode reintroduces the C-compile/per-platform-wheel
+  cost without the control of a real native module. Either way it would be a
+  throwaway second validator of the same C surface that `libpanto-go` already
+  exercises. Net: redundant. We'd reimplement it natively on the soon side.
+- **`libpanto-py` is pure Zig, not C-over-`libpanto-c`.** The decisive win is
+  that a Zig→`.so` compilation involves **no C translation units in the repo at
+  all** — one toolchain, header translation done at compile time by `@cImport`.
+  Going straight to the Zig API (rather than through `libpanto-c`) also skips a
+  redundant marshalling layer.
+- **`libpanto-lua` is pure Zig too, same reasoning as `libpanto-py`.** Lua C
+  modules are structurally the CPython-extension pattern: one shared object
+  exporting a fixed-name init function (`luaopen_panto`), discovered on
+  `package.cpath`, loaded by `require`. Zig `@cImport`s the Lua headers and
+  emits the `.so` directly against the Zig API — no C translation units, no
+  `libpanto-c` dependency. The extra payoff unique to Lua: the panto CLI
+  *already* embeds Lua, so this package is not just an external binding but the
+  thing the CLI's own extension VM loads (below).
+
+## The core architectural decision: ship a *pull* streaming API
+
+This is the spine of the whole project, and it requires an **internal refactor
+of `libpanto` first** (Phase 0). Everything else wraps it.
+
+### Why pull, not push
+
+`libpanto` today streams via a **push** `Receiver` vtable
+(`provider.zig`): the provider loop calls `onMessageStart`, `onContentDelta`,
+`onBlockComplete`, `onMessageComplete`, etc. Push is the wrong primitive to
+*export*, for two independent reasons that land on the same conclusion:
+
+1. **It doesn't compose into the idiomatic surfaces we want.**
+   - Go: the idiomatic streaming shape is a single goroutine driving a
+     range-over-func iterator (`for ev := range stream.Iter`) and/or a channel.
+     cgo callbacks *into* Go are slow and awkward (`//export`, pointer rules,
+     no closures). A goroutine draining a pull API is the clean build.
+   - Python: a (sync or async) generator **is** a pull construct —
+     `__next__` *is* "give me the next event." A push callback can't `yield`;
+     to turn pushed values into a `for ev in stream`, you must run the producer
+     and consumer in different stack frames concurrently (a thread + queue).
+     So push → generator *forces* a thread+queue regardless. Pull → generator
+     is a 1:1 mapping with zero adaptation.
+
+2. **Pull is the more primitive primitive.** Push composes trivially on top of
+   pull (`while (try s.next()) |ev| receiver.onEvent(ev);` — five lines).
+   Pull does **not** compose cheaply on top of push: to expose `next()` over a
+   `Receiver`-driven core you must suspend *inside* a callback and resume the
+   provider loop later, which means a thread+queue or hand-rolled coroutine
+   state — i.e. you rebuild pull anyway, the hard way.
+
+**Conclusion:** ship only the pull API. `Receiver` leaves the public surface
+entirely. Anyone who wants push writes the five-line wrapper themselves.
+
+### The Zig interface
+
+```zig
+// run() drops the receiver parameter entirely and returns a Stream.
+// Whether it also drops `conv` depends on sequencing against
+// docs/pluggable-session-store.md (see note below) — either form is fine.
+pub fn run(self: *Agent) !Stream  // or: run(self: *Agent, conv: *Conversation) !Stream
+
+// Stream is a resumable handle, closely linked to SSEParser + the provider
+// read loop. next() yields events until the turn is exhausted.
+pub fn next(self: *Stream) !?Event
+```
+
+`Stream.next()` returns `!?Event`, which gives **three orthogonal channels**:
+
+| signal             | meaning                                              |
+| ------------------ | ---------------------------------------------------- |
+| `Event` (a value)  | streaming progress, including the terminal event     |
+| `null`             | the stream is exhausted (already past the terminal)  |
+| `error.X`          | a genuine failure (network, parse, provider error)   |
+
+### The terminal-event invariant (documented contract)
+
+> Every `next()` call on a stream **after** it has yielded a `MessageComplete`
+> event returns `null`, and `null` is **never** returned before
+> `MessageComplete`.
+
+Consequences, and why this exact shape:
+
+- **`MessageComplete` is a normal `Event` variant**, not an error and not
+  signalled by `null`. It is the in-band success terminal.
+- **Intelligent consumers never observe `null`.** They stop after *consuming*
+  `MessageComplete`. `null` exists only as the defensive answer to "you called
+  `next()` one time too many," distinct from a real `error`.
+- Python: emit `MessageComplete`, then raise `StopIteration` on the *next*
+  call (which sees `null`) — or, more precisely, the generator stops right
+  after yielding `MessageComplete`, so well-behaved code raises `StopIteration`
+  without ever materializing a `None`.
+- Go: the `Iter` range-over-func stops *after* yielding `MessageComplete`, not
+  by waiting to observe a `null` sentinel.
+- Mid-stream **provider errors surface as a Zig `error` from `next()`** (the
+  `!` in `!?Event`), not as an `Event.Error` variant. This keeps the `Event`
+  union success-only and maps cleanly to a raised Python exception and a Go
+  `error`. (`error.StreamExhausted` is *not* used — exhaustion is `null`, not
+  an error; we reserve errors for failures.)
+
+### `Stream.reopen()` — retry a failed turn after a config swap
+
+When `next()` surfaces a **terminal** error at the provider-open boundary
+(e.g. an HTTP 400 because the chosen reasoning/thinking mode is unsupported by
+the model), the conversation is left untouched — the user prompt that `run()`
+appended is intact and no assistant message was committed. `reopen()` resets
+that failed stream back to its turn-open state so the embedder can:
+
+1. catch the error from `next()`,
+2. change the agent config (`set_config`), and
+3. call `reopen()` and resume `next()` — re-running the SAME turn against the
+   new config **without re-appending the user message**.
+
+This is the binding-portable alternative to calling `run()` again (which would
+append a duplicate prompt and persist it). It errors if the stream has not
+failed. Exposed at every layer: Zig `Stream.reopen() !void`, C
+`panto_stream_reopen(PantoStream*) -> PantoStatus`, Go `(*Stream).Reopen()
+error`, Lua `stream:reopen()`. No embedder is privileged: the panto CLI's
+adaptive-thinking fallback is built entirely on this public method.
+
+### `Event` spans the whole turn, not one HTTP response
+
+Verified from the code: `Agent.runStep` wraps **multiple** `streamStep` calls
+in a tool-using turn (one provider stream per assistant message), with
+concurrent tool dispatch *between* them (`agent.zig`). The `Stream` therefore
+spans the entire `run()` loop — provider streaming **and** tool dispatch — not
+a single SSE response. The `Event` union must express both layers.
+
+The exact variant list is to be finalized against what the provider loop and
+agent loop emit today (see `ReceiverVTable` in `provider.zig` and the dispatch
+path in `agent.zig`), but it covers at least:
+
+- message lifecycle: `MessageStart`, `MessageComplete`
+- block lifecycle: `BlockStart`, `BlockComplete`
+- content: `ContentDelta`
+- tool identity: `ToolDetails` (id + name)
+- tool dispatch boundaries: tool-call start / tool-result (so Go/Python
+  consumers can observe the agent running tools between provider turns)
+- provider retry notices: `ProviderRetry` (informational; today `onProviderRetry`)
+
+`Event` is an `enum union`; it is the single type every binding marshals into
+its native form. Define it once, in `libpanto`.
+
+## The hard part of Phase 0: making `Stream.next()` resumable
+
+Verified from `provider_openai_chat.zig`: the current provider loop is a
+single-stack-frame loop —
+
+```
+readVec(body_reader) -> parser.feed(chunk) -> handleEvent(...) -> receiver.*(...)
+```
+
+— with all state (the HTTP body reader + `transfer_buf`, the `SSEParser`, the
+per-response `StreamState`) living **on the stack**. A pull `next()` must
+*return* between events, so that state has to move into the `Stream` handle.
+Three ways to get there:
+
+1. **State machine (recommended target).** The `Stream` owns the socket/body
+   reader, the `SSEParser`, and the decode state; `next()` reads/parses just
+   enough to produce one `Event` and returns. No threads. Cleanest and most
+   portable across all four packages. Most upfront refactoring effort, since
+   the provider loop is inverted into a resumable step function.
+
+2. **Thread + queue inside the handle.** Keep the existing
+   `streamStep(receiver)` loop verbatim; run it on a thread whose receiver
+   pushes events into a bounded queue; `next()` pops. Minimal change to proven
+   code, but costs a thread per active stream plus shutdown/backpressure care.
+   A reasonable interim if (1) proves invasive.
+
+3. **Zig async/suspend.** Language-level coroutines. Availability depends on the
+   toolchain's async status at the time of implementation; treat as
+   not-reliably-available without checking first.
+
+The internal mechanism (1 vs 2) is swappable later **without changing the
+exported contract** — the public surface is pull either way. Recommendation:
+target (1); fall back to (2) only if the provider-loop inversion is too costly
+for v1.
+
+> Expectation: **Phase 0 is a net-red diff.** Deleting `Receiver` from the
+> public API, the `CompactionCapture` no-op receiver, and the CLI's
+> `CLIReceiver` vtable plumbing should remove more than the `Stream` machinery
+> adds.
+
+---
+
+## Phase 0 — internal refactor of `libpanto` to a pull API **(COMPLETE)**
+
+The real work. Everything after this is wrapping.
+
+1. **Define the `Event` union** (success-only `enum union`) from the current
+   `ReceiverVTable` callbacks plus the agent's tool-dispatch boundaries. This
+   is the type every binding marshals.
+2. **Define the `Stream` type**: a resumable handle owning the provider read
+   loop's state (body reader, `SSEParser`, decode state) and the agent's
+   tool-dispatch position. `fn next(self: *Stream) !?Event`. This is where the
+   state-machine-vs-thread decision lands.
+3. **`Agent.run() -> Stream`.** With `conversation` owned by the `Agent` (per
+   `docs/pluggable-session-store.md`) and `receiver` removed, `run()` takes
+   only `self`. The agent loop (provider stream → tool dispatch → repeat)
+   becomes the thing `Stream.next()` drives incrementally.
+4. **Delete `Receiver` from the public API** (`root.zig`, `provider.zig`).
+   Internal seams that genuinely want push (if any) get the trivial pull→push
+   wrapper, in caller code, not in the library surface.
+5. **Rebuild the CLI on the pull loop.** `src/main.zig`'s `CLIReceiver`
+   collapses into a `while (try stream.next()) |ev| render(ev)` loop. Prove the
+   refactor end-to-end against the existing test suite; CLI behavior unchanged.
+
+**Exit criteria:** `Receiver` gone from `root.zig`; `Stream.next()` is the only
+streaming primitive; CLI output and the full test suite are unchanged.
+
+> Sequencing with `docs/pluggable-session-store.md` is **flexible**, not a
+> dependency. That plan moves `Conversation` onto the `Agent`; this one removes
+> `receiver`. The two are orthogonal:
+> - If the session-store work lands first, `run()` takes only `self`.
+> - If this work lands first, `run(self, conv)` takes a single `conv` argument
+>   (just `runStep` minus `receiver`), and the session-store refactor drops
+>   `conv` later.
+>
+> Either way `receiver` is gone after Phase 0. Don't block on ordering.
+
+## Phase 1 — `libpanto-c` (C ABI)
+
+6. **Opaque handles** for the major types: `PantoAgent`, `PantoStream` (and
+   whatever construction requires — config, conversation/session store).
+   Consumers never see the layout.
+7. **`panto_next_event(stream, *PantoEvent) -> status`** is the C projection of
+   `!?Event`. The `!?Event` triple maps onto a status enum plus an out-param:
+
+   | Zig `next()`            | C status     | out-param        |
+   | ----------------------- | ------------ | ---------------- |
+   | `Event` value           | `EVENT` (0)  | filled           |
+   | `null`                  | `DONE` (1)   | untouched        |
+   | `error.X`               | `ERROR` (2)  | error detail     |
+
+8. **`@export` wrappers** for construction, `run()`, `next_event`, teardown,
+   and a free function for any owned event payloads.
+9. **The C header is committed and hand-maintained, not generated by
+   `build.zig`.** (Open question 1 below: we are *not* having `build.zig` emit
+   `panto.h`.) A stable, hand-written `include/panto.h` is the ABI contract —
+   it should change deliberately, be reviewable in diffs, and not be a build
+   artifact. `build.zig` emits the shared library and stages the committed
+   header; it does not author it.
+10. **Define the event-marshalling C structs once.** `PantoEvent` (a tagged
+    union mirroring the Zig `Event`) is read by every downstream binding.
+
+## Phase 2 — `libpanto-go` (validates `libpanto-c`)
+
+11. **cgo bindings**: `Agent`, `Stream`, and the raw
+    `Stream.Next() (Event, bool, error)` (or `(Event, error)` with a separate
+    done signal) mapping `EVENT`/`DONE`/`ERROR` onto Go's idioms.
+12. **`Stream.Iter() iter.Seq[Event]`** — the modern range-over-func form, so
+    consumers write `for ev := range stream.Iter { ... }`. Single goroutine,
+    auto-terminates after yielding `MessageComplete`, surfaces a failure via a
+    trailing `stream.Err()` after the range (the idiomatic Go pattern since
+    `bufio.Scanner`). This is the primary Go surface.
+13. **A channel wrapper too.** Go users expect channels; a goroutine drains
+    `Next()` into a channel. Cheap over a pull core. Ship both the raw
+    iterator and the channel form.
+
+`libpanto-go` is the validation harness for the `libpanto-c` ABI — if Go can
+drive a full streaming turn idiomatically, the C surface is sound.
+
+## Phase 3 — `libpanto-py` (pure Zig CPython extension)
+
+14. **`build.zig` does `@cImport(@cInclude("Python.h"))`** and emits
+    `_panto.so` (a loadable extension module). No C in this package. Decide
+    **stable ABI (`abi3` / limited API)** here to collapse the
+    CPython-version × platform wheel matrix to one artifact per platform.
+15. **`module.zig`** implements `PyModuleDef`, `PyMethodDef`, and an `Agent`
+    type and `Stream` type as `PyTypeObject`s, calling the **Zig** `libpanto`
+    API directly. The `Stream` type's `tp_iternext` calls Zig `Stream.next()`:
+    - `Event` → build and return the `PyObject` event.
+    - `null` → set `StopIteration` (return `NULL` with no error set).
+    - `error.X` → set the mapped Python exception.
+    - Wrap the blocking `next()` in `Py_BEGIN_ALLOW_THREADS` /
+      `Py_END_ALLOW_THREADS` so streaming doesn't serialize the whole
+      interpreter; re-acquire the GIL to build the event `PyObject`.
+16. **`panto/__init__.py`** is a thin Pythonic surface over the native
+    `_panto`: a real exception hierarchy, context managers, idiomatic event
+    objects. **The async generator is pure Python** — wrap the sync iterator
+    with `asyncio.to_thread` (blocking `next()` runs in a worker thread). No
+    file descriptors, no ABI change. (See "Out of scope for v1" below.)
+
+## Phase 4 — `libpanto-lua` (pure Zig Lua C-module) + CLI integration
+
+Targets **Lua 5.4** only (see version-targeting note up top).
+
+17. **`build.zig` does `@cImport(@cInclude("lua.h"))`** (plus `lauxlib.h`,
+    `lualib.h`) and emits `panto.so` — no `lib` prefix — exporting
+    `luaopen_panto`. No C in this package. Build against Lua 5.4 headers;
+    `luaL_checkversion` in the init path makes a wrong-version load fail loud.
+18. **`module.zig`** implements the `luaL_Reg` module table plus an `Agent` and
+    a `Stream` as `luaL_newmetatable` userdata types, calling the **Zig**
+    `libpanto` API directly. `luaopen_panto` returns the module table. The
+    `Stream` userdata exposes an iterator step (a `__call` closure, or a
+    `stream:iter()` returning the standard `iterator, state, control` triple)
+    that calls Zig `Stream.next()`:
+    - `Event` → build and return the event as a Lua table.
+    - `null` → return `nil` (ends the `for` loop).
+    - `error.X` → `lua_error` with the mapped message.
+    Wrap blocking `next()` outside the Lua lock if/when a threaded host needs
+    it; for the single-threaded embedded CLI VM this is a plain call.
+19. **CLI integration: load the native module via `package.preload`, then
+    augment it in Zig (Option B).** The panto CLI embeds Lua and exposes the
+    `panto` table to extensions through `require('panto')` — there is no
+    `panto` global. The host installs a loader into `package.preload['panto']`
+    (`src/lua_bridge.zig`) that **calls the native `luaopen_panto` itself**,
+    takes the fresh table it returns, **adds `panto.ext`** (tool/command
+    registration, `on`, `emit`) to that same table, and returns it. So the CLI
+    and standalone Lua share the *identical* native agent/stream surface; the
+    CLI's copy merely carries an extra `ext` field. Standalone Lua has no
+    preload entry and falls through to `panto.so` on `cpath` (see next
+    section).
+
+## Lua: one `require('panto')`, two surfaces
+
+The goal: any Lua code does `require('panto')` to reach libpanto, and inside the
+panto CLI's embedded VM that *same* require additionally yields `panto.ext` for
+authoring tools/commands/event handlers. One name, resolved two ways:
+
+- **Standalone Lua** (a user's own 5.4 interpreter) → `require('panto')` finds
+  `panto.so` on `package.cpath`, runs `luaopen_panto`, and returns the
+  **agent/stream API only**. No `panto.ext` — there is no live CLI context to
+  register tools against.
+- **panto CLI embedded VM** → `require('panto')` returns the **same agent/stream
+  surface plus `panto.ext`**, because `panto.ext` needs the host's live
+  `Context` (registry, `EventBus`, session manager) which only exists inside
+  the CLI process.
+
+The mechanism for the second case is **`package.preload['panto']` + in-Zig
+augmentation (Option B)**. The CLI host installs a loader into
+`package.preload` *before* running extension scripts; because the preload
+searcher is `package.searchers[1]`, it **always wins over `cpath`**, so in the
+CLI VM `require('panto')` never reaches `panto.so` directly. Instead the loader:
+
+1. calls the native `luaopen_panto` to build the agent/stream table, then
+2. attaches `panto.ext` to that table, and
+3. returns it.
+
+This:
+
+- removes the injected `panto` global in favor of explicit `require` (matching
+  how every other Lua module is consumed);
+- keeps `panto.ext` available **only** where it's meaningful (the CLI VM),
+  since standalone Lua has no preload entry and falls through to the native
+  `panto.so`, which has no `ext`;
+- reuses the *exact* native agent/stream surface in both environments — no
+  second implementation, no drift. The CLI's table is the native table plus an
+  `ext` field.
+
+**Why mutating the returned table is safe (not "modifying an external
+dependency").** `luaopen_panto` builds and returns a **fresh Lua table** on each
+call — standard Lua C-module behavior, *not* a shared singleton handed back by
+reference. The host owns that table the instant it is returned; adding `ext`
+mutates the host's own copy and touches neither the `.so`'s code nor any other
+`lua_State`. `require` caches per-`(state, name)` in `package.loaded`, so the
+one CLI VM caches the augmented table while a standalone interpreter (a
+different state, no preload entry) independently gets the plain native table.
+The one invariant `libpanto-lua` must preserve: `luaopen_panto` returns a fresh
+table, never a process-shared one.
+
+**Everything host-side stays in Zig.** The preload loader, the call into
+`luaopen_panto`, and the construction of `panto.ext` are all Zig C-API code in
+`src/lua_bridge.zig` — no Lua glue file. `panto.ext` needs the live CLI
+`Context` (registry, `EventBus`, session manager); the loader closes over it the
+way today's `installEmit` already does — the `Context` rides as a
+light-userdata upvalue on the registration C-closures — so no Lua-level wiring
+or host-supplied hook is required.
+
+> Sequencing: the native `libpanto-lua` module (steps 17–18) and the CLI's
+> `package.preload['panto']` wiring (step 19) are separable. The CLI provides
+> `require('panto')` via preload independent of the native module landing, and
+> the native module can ship for standalone use independently. Neither blocks
+> the other.
+
+## The one contract that unifies all packages
+
+| layer    | progress / terminal       | exhausted          | failure                |
+| -------- | ------------------------- | ------------------ | ---------------------- |
+| Zig      | `Event` / `MessageComplete` | `null`           | `error.X`              |
+| C        | `PantoEvent` / status `EVENT` | status `DONE`  | status `ERROR`         |
+| Go       | `Event` (Iter yields)     | Iter stops         | `error` / `stream.Err()` |
+| Python   | event object (yielded)    | `StopIteration`    | raised exception       |
+
+Design every binding to this single table. Pull-shaped, success-only events,
+terminal-by-`MessageComplete`, exhaustion-by-`null`, failure-by-error.
+
+Lua slots in exactly like Python (it is a pull-iterator language too): a stream
+is a Lua iterator (a `for ev in stream` via a `__call`/closure or `pairs`-style
+driver) whose underlying step calls Zig `Stream.next()`:
+
+| layer    | progress / terminal           | exhausted          | failure                  |
+| -------- | ----------------------------- | ------------------ | ------------------------ |
+| Lua      | event table (yielded)         | iterator ends      | `error(...)` / `pcall` false |
+
+- `Event` → push the event as a Lua table and return it from the iterator step.
+- `null` → the iterator returns `nil`, ending the `for` loop (the 5-line
+  terminal-by-`MessageComplete` discipline holds: well-behaved code stops after
+  consuming `MessageComplete` and never materializes the `nil`).
+- `error.X` → `lua_error` with a mapped message, catchable via `pcall`.
+
+## Out of scope for v1 (deliberately deferred)
+
+- **A pollable fd / `panto_step_poll(timeout)`.** This is the only thing that
+  makes async *natively* non-blocking on both Go (`select`) and Python
+  (`await` on readiness). It is real work in `libpanto-c` and unnecessary for
+  v1. **Design the C ABI so a pollable fd can be added later without breaking
+  the existing pull surface**, but do not build it now.
+- **A native async Python API.** Without an fd, async Python is *just* the sync
+  pull API wrapped in `asyncio.to_thread`. That is pure-Python glue in
+  `panto/__init__.py`; no native or ABI work, so it isn't a binding deliverable.
+- **A `libpanto-lua` LuaJIT (Lua 5.1 ABI) artifact.** Important for the
+  LuaJIT-based ecosystem, but a separate per-version binary requiring
+  `LUA_VERSION_NUM`-guarded conditional compilation. Crucially this is a
+  *LuaJIT-target* shim, not clean Lua 5.1: LuaJIT selectively backports bits of
+  5.2/5.3 and those backports aren't reliably detectable at preprocess time, so
+  it must be validated against actual LuaJIT. Defer.
+- **A `libpanto-lua` Lua 5.5 artifact.** Another separate per-version binary.
+  When this is built, **seriously consider bumping the CLI's embedded Lua to
+  5.5** at the same time so the embedded VM and the standalone module track the
+  same modern version. Defer. (5.3, 5.2, and earlier are *not* deferred items —
+  they are out of scope entirely; all EOL.)
+- **Languages beyond Go, Python, and Lua.** `libpanto-c` is the reuse point for
+  any future cgo-style or cffi-style consumer (Ruby, Node N-API, …); none are
+  in scope now.
+
+## Open questions / decisions to finalize before coding
+
+1. **`panto.h` generation — resolved: no.** The header is committed and
+   hand-maintained as the ABI contract; `build.zig` stages it but does not emit
+   it. (Captured in Phase 1, step 9.)
+2. **Resumable-handle strategy (state machine vs. thread+queue)** — target the
+   state machine; decide finally once the provider-loop inversion is scoped.
+3. **Exact `Event` variants** — finalize against `ReceiverVTable`
+   (`provider.zig`) and the tool-dispatch path (`agent.zig`).
+4. **What moves onto `Agent` vs. stays per-`run()`** — driven by
+   `docs/pluggable-session-store.md`'s final shape; `conversation` is moving,
+   confirm nothing else needs to.
+5. **CPython stable-ABI (`abi3`) commitment** — decide in Phase 3 to fix the
+   wheel matrix early.
+6. **CLI bridge migration shape (`package.preload`)** — confirm the preload
+   loader integrates cleanly with the existing `src/lua_bridge.zig` install
+   path and the luarocks bootstrap ordering
+   (`docs/archive/pluggable-session-store.md`). The bare `panto` global is
+   dropped; extensions must `require('panto')`.
diff --git a/docs/todos.md b/docs/todos.md
new file mode 100644
index 0000000..8dd9f71
--- /dev/null
+++ b/docs/todos.md
@@ -0,0 +1,16 @@
+## libpanto
+
+- [x] user-provided system prompt
+- [x] polish zig API
+- [x] C ABI
+- [x] Agent compaction with custom compaction prompts
+- [x] Agent auto-compaction
+- [x] image upload support
+- [ ] google gemini native provider
+- [x] openai responses API native provider
+- [ ] non-streaming
+- [ ] one-shot simple API
+- [x] message-level error retries
+- [x] abort/cancellation
+- [ ] user message queueing: steering, follow-up
+- [ ] step cap, stop conditions