diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/libpanto-bindings.md | 517 | ||||
| -rw-r--r-- | docs/todos.md | 17 |
2 files changed, 0 insertions, 534 deletions
diff --git a/docs/libpanto-bindings.md b/docs/libpanto-bindings.md deleted file mode 100644 index 2872fde..0000000 --- a/docs/libpanto-bindings.md +++ /dev/null @@ -1,517 +0,0 @@ -# Plan: `libpanto` language bindings (Go + Python + Lua) - -## Goal - -Expose `libpanto` to other languages, targeting **Go**, **Python**, and **Lua** -first. The bindings are organized as a family of sibling packages, named -uniformly: - -| package | language | role | -| -------------- | -------- | ------------------------------------------------------------- | -| `libpanto` | Zig | the core library (exists today) | -| `libpanto-c` | Zig | a C-ABI shared library + header wrapping the public Zig API | -| `libpanto-go` | Go | idiomatic Go bindings over `libpanto-c` via cgo | -| `libpanto-py` | Zig | a CPython extension implemented in pure Zig (`@cImport`) | -| `libpanto-lua` | Zig | a Lua C-module implemented in pure Zig (`@cImport`) | - -Three consumers, two paths to the core: - -- **Go → `libpanto-c` → `libpanto`.** cgo is effectively the only option, and - cgo can only call C. So a C ABI is mandatory. -- **Python → `libpanto`** directly. `libpanto-py` is a native CPython - extension written *in Zig*: it `@cImport`s `Python.h`, builds `PyObject` - glue against the translated C types, and `build.zig` emits the loadable - `.so`. It calls the Zig API directly and **does not** depend on `libpanto-c`. -- **Lua → `libpanto`** directly. `libpanto-lua` is the same pattern as - `libpanto-py`, one rung over: a native Lua C-module written *in Zig* that - `@cImport`s `lua.h` / `lauxlib.h`, builds the `luaL_Reg` tables and userdata - types against the translated C types, and `build.zig` emits a loadable - `panto.so` (no `lib` prefix) exporting `luaopen_panto`. Discovered via - `package.cpath`, loaded by `require('panto')`. It calls the Zig API directly - and **does not** depend on `libpanto-c`. This is also the bindings package - that the panto CLI's *own* embedded Lua extension environment consumes — see - "Lua: one `require('panto')`, two surfaces" below. - -### Lua version targeting: no stable ABI, so pick a version - -Unlike CPython (where `abi3` collapses the version axis to one artifact), -**Lua has no stable-ABI escape hatch.** Lua guarantees binary compatibility -only across *bugfix* releases of one version (e.g. 5.4.1 ↔ 5.4.8); across -versions there is **no ABI compatibility and the C API signatures themselves -differ** (`lua_resume`, `lua_pcall`/`lua_pcallk`, `lua_load` gained params; the -globals model changed from `_G` to `_ENV` upvalues in 5.2+). A module compiled -for the wrong version fails loud at `require` time (typically -`undefined symbol: luaL_checkversion`), not silently. So the artifact matrix is -`{Lua version} × platform`, and supporting more than one version means -conditional compilation guarded by `LUA_VERSION_NUM` (plus a `lua-compat`-style -shim layer) producing **separate per-version binaries** — never one fat object. - -**v1 targets Lua 5.4 only.** That is the established current version (5.5.0 -shipped 22 Dec 2025 and is too new to matter yet) and, critically, it is the -version of the panto CLI's embedded Lua, so it directly unblocks libpanto -features for CLI extensions. **5.3, 5.2, and earlier are explicitly out of -scope** — all EOL, with adoption draining to 5.4. - -Two shim efforts are **deferred** (see "Out of scope for v1"): - -- a **LuaJIT (Lua 5.1 ABI)** artifact, important for the large LuaJIT-based - ecosystem; and -- a **Lua 5.5** artifact, at which point we'll seriously consider bumping the - CLI's embedded Lua to 5.5 as well. - -Note on the LuaJIT shim specifically: LuaJIT is ABI-compatible with Lua 5.1 but -*selectively backports* pieces of 5.2/5.3 (`goto`, `load()`, some C API like -`luaL_setfuncs`), and **preprocessor detection of which backports are present -is not reliable**. So that deferred work is a *LuaJIT-target* shim validated -against actual LuaJIT — not a clean textbook "Lua 5.1.5" build. - -### Why this split (decisions settled) - -- **`libpanto-c` is built regardless**, because Go requires it. It doubles as - the stable-ABI validation platform for the C surface. -- **No cffi.** A pure-Python cffi binding was considered and rejected. Its only - real appeal (no native build step, trivial wheels) holds solidly *only* in - ABI mode, which is also where streaming callbacks and silent struct/ABI - drift bite hardest; API mode reintroduces the C-compile/per-platform-wheel - cost without the control of a real native module. Either way it would be a - throwaway second validator of the same C surface that `libpanto-go` already - exercises. Net: redundant. We'd reimplement it natively on the soon side. -- **`libpanto-py` is pure Zig, not C-over-`libpanto-c`.** The decisive win is - that a Zig→`.so` compilation involves **no C translation units in the repo at - all** — one toolchain, header translation done at compile time by `@cImport`. - Going straight to the Zig API (rather than through `libpanto-c`) also skips a - redundant marshalling layer. -- **`libpanto-lua` is pure Zig too, same reasoning as `libpanto-py`.** Lua C - modules are structurally the CPython-extension pattern: one shared object - exporting a fixed-name init function (`luaopen_panto`), discovered on - `package.cpath`, loaded by `require`. Zig `@cImport`s the Lua headers and - emits the `.so` directly against the Zig API — no C translation units, no - `libpanto-c` dependency. The extra payoff unique to Lua: the panto CLI - *already* embeds Lua, so this package is not just an external binding but the - thing the CLI's own extension VM loads (below). - -## The core architectural decision: ship a *pull* streaming API - -This is the spine of the whole project, and it requires an **internal refactor -of `libpanto` first** (Phase 0). Everything else wraps it. - -### Why pull, not push - -`libpanto` today streams via a **push** `Receiver` vtable -(`provider.zig`): the provider loop calls `onMessageStart`, `onContentDelta`, -`onBlockComplete`, `onMessageComplete`, etc. Push is the wrong primitive to -*export*, for two independent reasons that land on the same conclusion: - -1. **It doesn't compose into the idiomatic surfaces we want.** - - Go: the idiomatic streaming shape is a single goroutine driving a - range-over-func iterator (`for ev := range stream.Iter`) and/or a channel. - cgo callbacks *into* Go are slow and awkward (`//export`, pointer rules, - no closures). A goroutine draining a pull API is the clean build. - - Python: a (sync or async) generator **is** a pull construct — - `__next__` *is* "give me the next event." A push callback can't `yield`; - to turn pushed values into a `for ev in stream`, you must run the producer - and consumer in different stack frames concurrently (a thread + queue). - So push → generator *forces* a thread+queue regardless. Pull → generator - is a 1:1 mapping with zero adaptation. - -2. **Pull is the more primitive primitive.** Push composes trivially on top of - pull (`while (try s.next()) |ev| receiver.onEvent(ev);` — five lines). - Pull does **not** compose cheaply on top of push: to expose `next()` over a - `Receiver`-driven core you must suspend *inside* a callback and resume the - provider loop later, which means a thread+queue or hand-rolled coroutine - state — i.e. you rebuild pull anyway, the hard way. - -**Conclusion:** ship only the pull API. `Receiver` leaves the public surface -entirely. Anyone who wants push writes the five-line wrapper themselves. - -### The Zig interface - -```zig -// run() drops the receiver parameter entirely and returns a Stream. -// Whether it also drops `conv` depends on sequencing against -// docs/pluggable-session-store.md (see note below) — either form is fine. -pub fn run(self: *Agent) !Stream // or: run(self: *Agent, conv: *Conversation) !Stream - -// Stream is a resumable handle, closely linked to SSEParser + the provider -// read loop. next() yields events until the turn is exhausted. -pub fn next(self: *Stream) !?Event -``` - -`Stream.next()` returns `!?Event`, which gives **three orthogonal channels**: - -| signal | meaning | -| ------------------ | ---------------------------------------------------- | -| `Event` (a value) | streaming progress, including the terminal event | -| `null` | the stream is exhausted (already past the terminal) | -| `error.X` | a genuine failure (network, parse, provider error) | - -### The terminal-event invariant (documented contract) - -> Every `next()` call on a stream **after** it has yielded a `MessageComplete` -> event returns `null`, and `null` is **never** returned before -> `MessageComplete`. - -Consequences, and why this exact shape: - -- **`MessageComplete` is a normal `Event` variant**, not an error and not - signalled by `null`. It is the in-band success terminal. -- **Intelligent consumers never observe `null`.** They stop after *consuming* - `MessageComplete`. `null` exists only as the defensive answer to "you called - `next()` one time too many," distinct from a real `error`. -- Python: emit `MessageComplete`, then raise `StopIteration` on the *next* - call (which sees `null`) — or, more precisely, the generator stops right - after yielding `MessageComplete`, so well-behaved code raises `StopIteration` - without ever materializing a `None`. -- Go: the `Iter` range-over-func stops *after* yielding `MessageComplete`, not - by waiting to observe a `null` sentinel. -- Mid-stream **provider errors surface as a Zig `error` from `next()`** (the - `!` in `!?Event`), not as an `Event.Error` variant. This keeps the `Event` - union success-only and maps cleanly to a raised Python exception and a Go - `error`. (`error.StreamExhausted` is *not* used — exhaustion is `null`, not - an error; we reserve errors for failures.) - -### `Stream.reopen()` — retry a failed turn after a config swap - -When `next()` surfaces a **terminal** error at the provider-open boundary -(e.g. an HTTP 400 because the chosen reasoning/thinking mode is unsupported by -the model), the conversation is left untouched — the user prompt that `run()` -appended is intact and no assistant message was committed. `reopen()` resets -that failed stream back to its turn-open state so the embedder can: - -1. catch the error from `next()`, -2. change the agent config (`set_config`), and -3. call `reopen()` and resume `next()` — re-running the SAME turn against the - new config **without re-appending the user message**. - -This is the binding-portable alternative to calling `run()` again (which would -append a duplicate prompt and persist it). It errors if the stream has not -failed. Exposed at every layer: Zig `Stream.reopen() !void`, C -`panto_stream_reopen(PantoStream*) -> PantoStatus`, Go `(*Stream).Reopen() -error`, Lua `stream:reopen()`. No embedder is privileged: the panto CLI's -adaptive-thinking fallback is built entirely on this public method. - -### `Event` spans the whole turn, not one HTTP response - -Verified from the code: `Agent.runStep` wraps **multiple** `streamStep` calls -in a tool-using turn (one provider stream per assistant message), with -concurrent tool dispatch *between* them (`agent.zig`). The `Stream` therefore -spans the entire `run()` loop — provider streaming **and** tool dispatch — not -a single SSE response. The `Event` union must express both layers. - -The exact variant list is to be finalized against what the provider loop and -agent loop emit today (see `ReceiverVTable` in `provider.zig` and the dispatch -path in `agent.zig`), but it covers at least: - -- message lifecycle: `MessageStart`, `MessageComplete` -- block lifecycle: `BlockStart`, `BlockComplete` -- content: `ContentDelta` -- tool identity: `ToolDetails` (id + name) -- tool dispatch boundaries: tool-call start / tool-result (so Go/Python - consumers can observe the agent running tools between provider turns) -- provider retry notices: `ProviderRetry` (informational; today `onProviderRetry`) - -`Event` is an `enum union`; it is the single type every binding marshals into -its native form. Define it once, in `libpanto`. - -## The hard part of Phase 0: making `Stream.next()` resumable - -Verified from `provider_openai_chat.zig`: the current provider loop is a -single-stack-frame loop — - -``` -readVec(body_reader) -> parser.feed(chunk) -> handleEvent(...) -> receiver.*(...) -``` - -— with all state (the HTTP body reader + `transfer_buf`, the `SSEParser`, the -per-response `StreamState`) living **on the stack**. A pull `next()` must -*return* between events, so that state has to move into the `Stream` handle. -Three ways to get there: - -1. **State machine (recommended target).** The `Stream` owns the socket/body - reader, the `SSEParser`, and the decode state; `next()` reads/parses just - enough to produce one `Event` and returns. No threads. Cleanest and most - portable across all four packages. Most upfront refactoring effort, since - the provider loop is inverted into a resumable step function. - -2. **Thread + queue inside the handle.** Keep the existing - `streamStep(receiver)` loop verbatim; run it on a thread whose receiver - pushes events into a bounded queue; `next()` pops. Minimal change to proven - code, but costs a thread per active stream plus shutdown/backpressure care. - A reasonable interim if (1) proves invasive. - -3. **Zig async/suspend.** Language-level coroutines. Availability depends on the - toolchain's async status at the time of implementation; treat as - not-reliably-available without checking first. - -The internal mechanism (1 vs 2) is swappable later **without changing the -exported contract** — the public surface is pull either way. Recommendation: -target (1); fall back to (2) only if the provider-loop inversion is too costly -for v1. - -> Expectation: **Phase 0 is a net-red diff.** Deleting `Receiver` from the -> public API, the `CompactionCapture` no-op receiver, and the CLI's -> `CLIReceiver` vtable plumbing should remove more than the `Stream` machinery -> adds. - ---- - -## Phase 0 — internal refactor of `libpanto` to a pull API **(COMPLETE)** - -The real work. Everything after this is wrapping. - -1. **Define the `Event` union** (success-only `enum union`) from the current - `ReceiverVTable` callbacks plus the agent's tool-dispatch boundaries. This - is the type every binding marshals. -2. **Define the `Stream` type**: a resumable handle owning the provider read - loop's state (body reader, `SSEParser`, decode state) and the agent's - tool-dispatch position. `fn next(self: *Stream) !?Event`. This is where the - state-machine-vs-thread decision lands. -3. **`Agent.run() -> Stream`.** With `conversation` owned by the `Agent` (per - `docs/pluggable-session-store.md`) and `receiver` removed, `run()` takes - only `self`. The agent loop (provider stream → tool dispatch → repeat) - becomes the thing `Stream.next()` drives incrementally. -4. **Delete `Receiver` from the public API** (`root.zig`, `provider.zig`). - Internal seams that genuinely want push (if any) get the trivial pull→push - wrapper, in caller code, not in the library surface. -5. **Rebuild the CLI on the pull loop.** `src/main.zig`'s `CLIReceiver` - collapses into a `while (try stream.next()) |ev| render(ev)` loop. Prove the - refactor end-to-end against the existing test suite; CLI behavior unchanged. - -**Exit criteria:** `Receiver` gone from `root.zig`; `Stream.next()` is the only -streaming primitive; CLI output and the full test suite are unchanged. - -> Sequencing with `docs/pluggable-session-store.md` is **flexible**, not a -> dependency. That plan moves `Conversation` onto the `Agent`; this one removes -> `receiver`. The two are orthogonal: -> - If the session-store work lands first, `run()` takes only `self`. -> - If this work lands first, `run(self, conv)` takes a single `conv` argument -> (just `runStep` minus `receiver`), and the session-store refactor drops -> `conv` later. -> -> Either way `receiver` is gone after Phase 0. Don't block on ordering. - -## Phase 1 — `libpanto-c` (C ABI) - -6. **Opaque handles** for the major types: `PantoAgent`, `PantoStream` (and - whatever construction requires — config, conversation/session store). - Consumers never see the layout. -7. **`panto_next_event(stream, *PantoEvent) -> status`** is the C projection of - `!?Event`. The `!?Event` triple maps onto a status enum plus an out-param: - - | Zig `next()` | C status | out-param | - | ----------------------- | ------------ | ---------------- | - | `Event` value | `EVENT` (0) | filled | - | `null` | `DONE` (1) | untouched | - | `error.X` | `ERROR` (2) | error detail | - -8. **`@export` wrappers** for construction, `run()`, `next_event`, teardown, - and a free function for any owned event payloads. -9. **The C header is committed and hand-maintained, not generated by - `build.zig`.** (Open question 1 below: we are *not* having `build.zig` emit - `panto.h`.) A stable, hand-written `include/panto.h` is the ABI contract — - it should change deliberately, be reviewable in diffs, and not be a build - artifact. `build.zig` emits the shared library and stages the committed - header; it does not author it. -10. **Define the event-marshalling C structs once.** `PantoEvent` (a tagged - union mirroring the Zig `Event`) is read by every downstream binding. - -## Phase 2 — `libpanto-go` (validates `libpanto-c`) - -11. **cgo bindings**: `Agent`, `Stream`, and the raw - `Stream.Next() (Event, bool, error)` (or `(Event, error)` with a separate - done signal) mapping `EVENT`/`DONE`/`ERROR` onto Go's idioms. -12. **`Stream.Iter() iter.Seq[Event]`** — the modern range-over-func form, so - consumers write `for ev := range stream.Iter { ... }`. Single goroutine, - auto-terminates after yielding `MessageComplete`, surfaces a failure via a - trailing `stream.Err()` after the range (the idiomatic Go pattern since - `bufio.Scanner`). This is the primary Go surface. -13. **A channel wrapper too.** Go users expect channels; a goroutine drains - `Next()` into a channel. Cheap over a pull core. Ship both the raw - iterator and the channel form. - -`libpanto-go` is the validation harness for the `libpanto-c` ABI — if Go can -drive a full streaming turn idiomatically, the C surface is sound. - -## Phase 3 — `libpanto-py` (pure Zig CPython extension) - -14. **`build.zig` does `@cImport(@cInclude("Python.h"))`** and emits - `_panto.so` (a loadable extension module). No C in this package. Decide - **stable ABI (`abi3` / limited API)** here to collapse the - CPython-version × platform wheel matrix to one artifact per platform. -15. **`module.zig`** implements `PyModuleDef`, `PyMethodDef`, and an `Agent` - type and `Stream` type as `PyTypeObject`s, calling the **Zig** `libpanto` - API directly. The `Stream` type's `tp_iternext` calls Zig `Stream.next()`: - - `Event` → build and return the `PyObject` event. - - `null` → set `StopIteration` (return `NULL` with no error set). - - `error.X` → set the mapped Python exception. - - Wrap the blocking `next()` in `Py_BEGIN_ALLOW_THREADS` / - `Py_END_ALLOW_THREADS` so streaming doesn't serialize the whole - interpreter; re-acquire the GIL to build the event `PyObject`. -16. **`panto/__init__.py`** is a thin Pythonic surface over the native - `_panto`: a real exception hierarchy, context managers, idiomatic event - objects. **The async generator is pure Python** — wrap the sync iterator - with `asyncio.to_thread` (blocking `next()` runs in a worker thread). No - file descriptors, no ABI change. (See "Out of scope for v1" below.) - -## Phase 4 — `libpanto-lua` (pure Zig Lua C-module) + CLI integration - -Targets **Lua 5.4** only (see version-targeting note up top). - -17. **`build.zig` does `@cImport(@cInclude("lua.h"))`** (plus `lauxlib.h`, - `lualib.h`) and emits `panto.so` — no `lib` prefix — exporting - `luaopen_panto`. No C in this package. Build against Lua 5.4 headers; - `luaL_checkversion` in the init path makes a wrong-version load fail loud. -18. **`module.zig`** implements the `luaL_Reg` module table plus an `Agent` and - a `Stream` as `luaL_newmetatable` userdata types, calling the **Zig** - `libpanto` API directly. `luaopen_panto` returns the module table. The - `Stream` userdata exposes an iterator step (a `__call` closure, or a - `stream:iter()` returning the standard `iterator, state, control` triple) - that calls Zig `Stream.next()`: - - `Event` → build and return the event as a Lua table. - - `null` → return `nil` (ends the `for` loop). - - `error.X` → `lua_error` with the mapped message. - Wrap blocking `next()` outside the Lua lock if/when a threaded host needs - it; for the single-threaded embedded CLI VM this is a plain call. -19. **CLI integration: load the native module via `package.preload`, then - augment it in Zig (Option B).** The panto CLI embeds Lua and exposes the - `panto` table to extensions through `require('panto')` — there is no - `panto` global. The host installs a loader into `package.preload['panto']` - (`src/lua_bridge.zig`) that **calls the native `luaopen_panto` itself**, - takes the fresh table it returns, **adds `panto.ext`** (tool/command - registration, `on`, `emit`) to that same table, and returns it. So the CLI - and standalone Lua share the *identical* native agent/stream surface; the - CLI's copy merely carries an extra `ext` field. Standalone Lua has no - preload entry and falls through to `panto.so` on `cpath` (see next - section). - -## Lua: one `require('panto')`, two surfaces - -The goal: any Lua code does `require('panto')` to reach libpanto, and inside the -panto CLI's embedded VM that *same* require additionally yields `panto.ext` for -authoring tools/commands/event handlers. One name, resolved two ways: - -- **Standalone Lua** (a user's own 5.4 interpreter) → `require('panto')` finds - `panto.so` on `package.cpath`, runs `luaopen_panto`, and returns the - **agent/stream API only**. No `panto.ext` — there is no live CLI context to - register tools against. -- **panto CLI embedded VM** → `require('panto')` returns the **same agent/stream - surface plus `panto.ext`**, because `panto.ext` needs the host's live - `Context` (registry, `EventBus`, session manager) which only exists inside - the CLI process. - -The mechanism for the second case is **`package.preload['panto']` + in-Zig -augmentation (Option B)**. The CLI host installs a loader into -`package.preload` *before* running extension scripts; because the preload -searcher is `package.searchers[1]`, it **always wins over `cpath`**, so in the -CLI VM `require('panto')` never reaches `panto.so` directly. Instead the loader: - -1. calls the native `luaopen_panto` to build the agent/stream table, then -2. attaches `panto.ext` to that table, and -3. returns it. - -This: - -- removes the injected `panto` global in favor of explicit `require` (matching - how every other Lua module is consumed); -- keeps `panto.ext` available **only** where it's meaningful (the CLI VM), - since standalone Lua has no preload entry and falls through to the native - `panto.so`, which has no `ext`; -- reuses the *exact* native agent/stream surface in both environments — no - second implementation, no drift. The CLI's table is the native table plus an - `ext` field. - -**Why mutating the returned table is safe (not "modifying an external -dependency").** `luaopen_panto` builds and returns a **fresh Lua table** on each -call — standard Lua C-module behavior, *not* a shared singleton handed back by -reference. The host owns that table the instant it is returned; adding `ext` -mutates the host's own copy and touches neither the `.so`'s code nor any other -`lua_State`. `require` caches per-`(state, name)` in `package.loaded`, so the -one CLI VM caches the augmented table while a standalone interpreter (a -different state, no preload entry) independently gets the plain native table. -The one invariant `libpanto-lua` must preserve: `luaopen_panto` returns a fresh -table, never a process-shared one. - -**Everything host-side stays in Zig.** The preload loader, the call into -`luaopen_panto`, and the construction of `panto.ext` are all Zig C-API code in -`src/lua_bridge.zig` — no Lua glue file. `panto.ext` needs the live CLI -`Context` (registry, `EventBus`, session manager); the loader closes over it the -way today's `installEmit` already does — the `Context` rides as a -light-userdata upvalue on the registration C-closures — so no Lua-level wiring -or host-supplied hook is required. - -> Sequencing: the native `libpanto-lua` module (steps 17–18) and the CLI's -> `package.preload['panto']` wiring (step 19) are separable. The CLI provides -> `require('panto')` via preload independent of the native module landing, and -> the native module can ship for standalone use independently. Neither blocks -> the other. - -## The one contract that unifies all packages - -| layer | progress / terminal | exhausted | failure | -| -------- | ------------------------- | ------------------ | ---------------------- | -| Zig | `Event` / `MessageComplete` | `null` | `error.X` | -| C | `PantoEvent` / status `EVENT` | status `DONE` | status `ERROR` | -| Go | `Event` (Iter yields) | Iter stops | `error` / `stream.Err()` | -| Python | event object (yielded) | `StopIteration` | raised exception | - -Design every binding to this single table. Pull-shaped, success-only events, -terminal-by-`MessageComplete`, exhaustion-by-`null`, failure-by-error. - -Lua slots in exactly like Python (it is a pull-iterator language too): a stream -is a Lua iterator (a `for ev in stream` via a `__call`/closure or `pairs`-style -driver) whose underlying step calls Zig `Stream.next()`: - -| layer | progress / terminal | exhausted | failure | -| -------- | ----------------------------- | ------------------ | ------------------------ | -| Lua | event table (yielded) | iterator ends | `error(...)` / `pcall` false | - -- `Event` → push the event as a Lua table and return it from the iterator step. -- `null` → the iterator returns `nil`, ending the `for` loop (the 5-line - terminal-by-`MessageComplete` discipline holds: well-behaved code stops after - consuming `MessageComplete` and never materializes the `nil`). -- `error.X` → `lua_error` with a mapped message, catchable via `pcall`. - -## Out of scope for v1 (deliberately deferred) - -- **A pollable fd / `panto_step_poll(timeout)`.** This is the only thing that - makes async *natively* non-blocking on both Go (`select`) and Python - (`await` on readiness). It is real work in `libpanto-c` and unnecessary for - v1. **Design the C ABI so a pollable fd can be added later without breaking - the existing pull surface**, but do not build it now. -- **A native async Python API.** Without an fd, async Python is *just* the sync - pull API wrapped in `asyncio.to_thread`. That is pure-Python glue in - `panto/__init__.py`; no native or ABI work, so it isn't a binding deliverable. -- **A `libpanto-lua` LuaJIT (Lua 5.1 ABI) artifact.** Important for the - LuaJIT-based ecosystem, but a separate per-version binary requiring - `LUA_VERSION_NUM`-guarded conditional compilation. Crucially this is a - *LuaJIT-target* shim, not clean Lua 5.1: LuaJIT selectively backports bits of - 5.2/5.3 and those backports aren't reliably detectable at preprocess time, so - it must be validated against actual LuaJIT. Defer. -- **A `libpanto-lua` Lua 5.5 artifact.** Another separate per-version binary. - When this is built, **seriously consider bumping the CLI's embedded Lua to - 5.5** at the same time so the embedded VM and the standalone module track the - same modern version. Defer. (5.3, 5.2, and earlier are *not* deferred items — - they are out of scope entirely; all EOL.) -- **Languages beyond Go, Python, and Lua.** `libpanto-c` is the reuse point for - any future cgo-style or cffi-style consumer (Ruby, Node N-API, …); none are - in scope now. - -## Open questions / decisions to finalize before coding - -1. **`panto.h` generation — resolved: no.** The header is committed and - hand-maintained as the ABI contract; `build.zig` stages it but does not emit - it. (Captured in Phase 1, step 9.) -2. **Resumable-handle strategy (state machine vs. thread+queue)** — target the - state machine; decide finally once the provider-loop inversion is scoped. -3. **Exact `Event` variants** — finalize against `ReceiverVTable` - (`provider.zig`) and the tool-dispatch path (`agent.zig`). -4. **What moves onto `Agent` vs. stays per-`run()`** — driven by - `docs/pluggable-session-store.md`'s final shape; `conversation` is moving, - confirm nothing else needs to. -5. **CPython stable-ABI (`abi3`) commitment** — decide in Phase 3 to fix the - wheel matrix early. -6. **CLI bridge migration shape (`package.preload`)** — confirm the preload - loader integrates cleanly with the existing `src/lua_bridge.zig` install - path and the luarocks bootstrap ordering - (`docs/archive/pluggable-session-store.md`). The bare `panto` global is - dropped; extensions must `require('panto')`. diff --git a/docs/todos.md b/docs/todos.md index 96a5d86..433e344 100644 --- a/docs/todos.md +++ b/docs/todos.md @@ -1,20 +1,3 @@ -## libpanto - -- [x] user-provided system prompt (design: docs/archive/system-prompt.md) -- [x] polish zig API -- [x] C ABI -- [x] Agent compaction with custom compaction prompts -- [x] Agent auto-compaction -- [x] image upload support -- [ ] google gemini native provider -- [x] openai responses API native provider -- [ ] non-streaming -- [ ] one-shot simple API -- [x] message-level error retries -- [x] abort/cancellation -- [ ] user message queueing: steering, follow-up -- [ ] step cap, stop conditions - ## panto cli - [ ] tui |