diff options
Diffstat (limited to 'docs/libpanto-bindings.md')
| -rw-r--r-- | docs/libpanto-bindings.md | 202 |
1 files changed, 188 insertions, 14 deletions
diff --git a/docs/libpanto-bindings.md b/docs/libpanto-bindings.md index 3c1a355..8d803d1 100644 --- a/docs/libpanto-bindings.md +++ b/docs/libpanto-bindings.md @@ -1,18 +1,20 @@ -# Plan: `libpanto` language bindings (Go + Python) +# Plan: `libpanto` language bindings (Go + Python + Lua) ## Goal -Expose `libpanto` to other languages, targeting **Go** and **Python** first. -The bindings are organized as a family of sibling packages, named uniformly: +Expose `libpanto` to other languages, targeting **Go**, **Python**, and **Lua** +first. The bindings are organized as a family of sibling packages, named +uniformly: -| package | language | role | -| ------------- | -------- | ------------------------------------------------------------- | -| `libpanto` | Zig | the core library (exists today) | -| `libpanto-c` | Zig | a C-ABI shared library + header wrapping the public Zig API | -| `libpanto-go` | Go | idiomatic Go bindings over `libpanto-c` via cgo | -| `libpanto-py` | Zig | a CPython extension implemented in pure Zig (`@cImport`) | +| package | language | role | +| -------------- | -------- | ------------------------------------------------------------- | +| `libpanto` | Zig | the core library (exists today) | +| `libpanto-c` | Zig | a C-ABI shared library + header wrapping the public Zig API | +| `libpanto-go` | Go | idiomatic Go bindings over `libpanto-c` via cgo | +| `libpanto-py` | Zig | a CPython extension implemented in pure Zig (`@cImport`) | +| `libpanto-lua` | Zig | a Lua C-module implemented in pure Zig (`@cImport`) | -Two consumers, two paths to the core: +Three consumers, two paths to the core: - **Go → `libpanto-c` → `libpanto`.** cgo is effectively the only option, and cgo can only call C. So a C ABI is mandatory. @@ -20,6 +22,48 @@ Two consumers, two paths to the core: extension written *in Zig*: it `@cImport`s `Python.h`, builds `PyObject` glue against the translated C types, and `build.zig` emits the loadable `.so`. It calls the Zig API directly and **does not** depend on `libpanto-c`. +- **Lua → `libpanto`** directly. `libpanto-lua` is the same pattern as + `libpanto-py`, one rung over: a native Lua C-module written *in Zig* that + `@cImport`s `lua.h` / `lauxlib.h`, builds the `luaL_Reg` tables and userdata + types against the translated C types, and `build.zig` emits a loadable + `panto.so` (no `lib` prefix) exporting `luaopen_panto`. Discovered via + `package.cpath`, loaded by `require('panto')`. It calls the Zig API directly + and **does not** depend on `libpanto-c`. This is also the bindings package + that the panto CLI's *own* embedded Lua extension environment consumes — see + "Lua: one `require('panto')`, two surfaces" below. + +### Lua version targeting: no stable ABI, so pick a version + +Unlike CPython (where `abi3` collapses the version axis to one artifact), +**Lua has no stable-ABI escape hatch.** Lua guarantees binary compatibility +only across *bugfix* releases of one version (e.g. 5.4.1 ↔ 5.4.8); across +versions there is **no ABI compatibility and the C API signatures themselves +differ** (`lua_resume`, `lua_pcall`/`lua_pcallk`, `lua_load` gained params; the +globals model changed from `_G` to `_ENV` upvalues in 5.2+). A module compiled +for the wrong version fails loud at `require` time (typically +`undefined symbol: luaL_checkversion`), not silently. So the artifact matrix is +`{Lua version} × platform`, and supporting more than one version means +conditional compilation guarded by `LUA_VERSION_NUM` (plus a `lua-compat`-style +shim layer) producing **separate per-version binaries** — never one fat object. + +**v1 targets Lua 5.4 only.** That is the established current version (5.5.0 +shipped 22 Dec 2025 and is too new to matter yet) and, critically, it is the +version of the panto CLI's embedded Lua, so it directly unblocks libpanto +features for CLI extensions. **5.3, 5.2, and earlier are explicitly out of +scope** — all EOL, with adoption draining to 5.4. + +Two shim efforts are **deferred** (see "Out of scope for v1"): + +- a **LuaJIT (Lua 5.1 ABI)** artifact, important for the large LuaJIT-based + ecosystem; and +- a **Lua 5.5** artifact, at which point we'll seriously consider bumping the + CLI's embedded Lua to 5.5 as well. + +Note on the LuaJIT shim specifically: LuaJIT is ABI-compatible with Lua 5.1 but +*selectively backports* pieces of 5.2/5.3 (`goto`, `load()`, some C API like +`luaL_setfuncs`), and **preprocessor detection of which backports are present +is not reliable**. So that deferred work is a *LuaJIT-target* shim validated +against actual LuaJIT — not a clean textbook "Lua 5.1.5" build. ### Why this split (decisions settled) @@ -37,6 +81,14 @@ Two consumers, two paths to the core: all** — one toolchain, header translation done at compile time by `@cImport`. Going straight to the Zig API (rather than through `libpanto-c`) also skips a redundant marshalling layer. +- **`libpanto-lua` is pure Zig too, same reasoning as `libpanto-py`.** Lua C + modules are structurally the CPython-extension pattern: one shared object + exporting a fixed-name init function (`luaopen_panto`), discovered on + `package.cpath`, loaded by `require`. Zig `@cImport`s the Lua headers and + emits the `.so` directly against the Zig API — no C translation units, no + `libpanto-c` dependency. The extra payoff unique to Lua: the panto CLI + *already* embeds Lua, so this package is not just an external binding but the + thing the CLI's own extension VM loads (below). ## The core architectural decision: ship a *pull* streaming API @@ -281,7 +333,99 @@ drive a full streaming turn idiomatically, the C surface is sound. with `asyncio.to_thread` (blocking `next()` runs in a worker thread). No file descriptors, no ABI change. (See "Out of scope for v1" below.) -## The one contract that unifies all four packages +## Phase 4 — `libpanto-lua` (pure Zig Lua C-module) + CLI integration + +Targets **Lua 5.4** only (see version-targeting note up top). + +17. **`build.zig` does `@cImport(@cInclude("lua.h"))`** (plus `lauxlib.h`, + `lualib.h`) and emits `panto.so` — no `lib` prefix — exporting + `luaopen_panto`. No C in this package. Build against Lua 5.4 headers; + `luaL_checkversion` in the init path makes a wrong-version load fail loud. +18. **`module.zig`** implements the `luaL_Reg` module table plus an `Agent` and + a `Stream` as `luaL_newmetatable` userdata types, calling the **Zig** + `libpanto` API directly. `luaopen_panto` returns the module table. The + `Stream` userdata exposes an iterator step (a `__call` closure, or a + `stream:iter()` returning the standard `iterator, state, control` triple) + that calls Zig `Stream.next()`: + - `Event` → build and return the event as a Lua table. + - `null` → return `nil` (ends the `for` loop). + - `error.X` → `lua_error` with the mapped message. + Wrap blocking `next()` outside the Lua lock if/when a threaded host needs + it; for the single-threaded embedded CLI VM this is a plain call. +19. **CLI integration: load the native module via `package.preload`, then + augment it in Zig (Option B).** The panto CLI embeds Lua and exposes the + `panto` table to extensions through `require('panto')` — there is no + `panto` global. The host installs a loader into `package.preload['panto']` + (`src/lua_bridge.zig`) that **calls the native `luaopen_panto` itself**, + takes the fresh table it returns, **adds `panto.ext`** (tool/command + registration, `on`, `emit`) to that same table, and returns it. So the CLI + and standalone Lua share the *identical* native agent/stream surface; the + CLI's copy merely carries an extra `ext` field. Standalone Lua has no + preload entry and falls through to `panto.so` on `cpath` (see next + section). + +## Lua: one `require('panto')`, two surfaces + +The goal: any Lua code does `require('panto')` to reach libpanto, and inside the +panto CLI's embedded VM that *same* require additionally yields `panto.ext` for +authoring tools/commands/event handlers. One name, resolved two ways: + +- **Standalone Lua** (a user's own 5.4 interpreter) → `require('panto')` finds + `panto.so` on `package.cpath`, runs `luaopen_panto`, and returns the + **agent/stream API only**. No `panto.ext` — there is no live CLI context to + register tools against. +- **panto CLI embedded VM** → `require('panto')` returns the **same agent/stream + surface plus `panto.ext`**, because `panto.ext` needs the host's live + `Context` (registry, `EventBus`, session manager) which only exists inside + the CLI process. + +The mechanism for the second case is **`package.preload['panto']` + in-Zig +augmentation (Option B)**. The CLI host installs a loader into +`package.preload` *before* running extension scripts; because the preload +searcher is `package.searchers[1]`, it **always wins over `cpath`**, so in the +CLI VM `require('panto')` never reaches `panto.so` directly. Instead the loader: + +1. calls the native `luaopen_panto` to build the agent/stream table, then +2. attaches `panto.ext` to that table, and +3. returns it. + +This: + +- removes the injected `panto` global in favor of explicit `require` (matching + how every other Lua module is consumed); +- keeps `panto.ext` available **only** where it's meaningful (the CLI VM), + since standalone Lua has no preload entry and falls through to the native + `panto.so`, which has no `ext`; +- reuses the *exact* native agent/stream surface in both environments — no + second implementation, no drift. The CLI's table is the native table plus an + `ext` field. + +**Why mutating the returned table is safe (not "modifying an external +dependency").** `luaopen_panto` builds and returns a **fresh Lua table** on each +call — standard Lua C-module behavior, *not* a shared singleton handed back by +reference. The host owns that table the instant it is returned; adding `ext` +mutates the host's own copy and touches neither the `.so`'s code nor any other +`lua_State`. `require` caches per-`(state, name)` in `package.loaded`, so the +one CLI VM caches the augmented table while a standalone interpreter (a +different state, no preload entry) independently gets the plain native table. +The one invariant `libpanto-lua` must preserve: `luaopen_panto` returns a fresh +table, never a process-shared one. + +**Everything host-side stays in Zig.** The preload loader, the call into +`luaopen_panto`, and the construction of `panto.ext` are all Zig C-API code in +`src/lua_bridge.zig` — no Lua glue file. `panto.ext` needs the live CLI +`Context` (registry, `EventBus`, session manager); the loader closes over it the +way today's `installEmit` already does — the `Context` rides as a +light-userdata upvalue on the registration C-closures — so no Lua-level wiring +or host-supplied hook is required. + +> Sequencing: the native `libpanto-lua` module (steps 17–18) and the CLI's +> `package.preload['panto']` wiring (step 19) are separable. The CLI provides +> `require('panto')` via preload independent of the native module landing, and +> the native module can ship for standalone use independently. Neither blocks +> the other. + +## The one contract that unifies all packages | layer | progress / terminal | exhausted | failure | | -------- | ------------------------- | ------------------ | ---------------------- | @@ -293,6 +437,20 @@ drive a full streaming turn idiomatically, the C surface is sound. Design every binding to this single table. Pull-shaped, success-only events, terminal-by-`MessageComplete`, exhaustion-by-`null`, failure-by-error. +Lua slots in exactly like Python (it is a pull-iterator language too): a stream +is a Lua iterator (a `for ev in stream` via a `__call`/closure or `pairs`-style +driver) whose underlying step calls Zig `Stream.next()`: + +| layer | progress / terminal | exhausted | failure | +| -------- | ----------------------------- | ------------------ | ------------------------ | +| Lua | event table (yielded) | iterator ends | `error(...)` / `pcall` false | + +- `Event` → push the event as a Lua table and return it from the iterator step. +- `null` → the iterator returns `nil`, ending the `for` loop (the 5-line + terminal-by-`MessageComplete` discipline holds: well-behaved code stops after + consuming `MessageComplete` and never materializes the `nil`). +- `error.X` → `lua_error` with a mapped message, catchable via `pcall`. + ## Out of scope for v1 (deliberately deferred) - **A pollable fd / `panto_step_poll(timeout)`.** This is the only thing that @@ -303,9 +461,20 @@ terminal-by-`MessageComplete`, exhaustion-by-`null`, failure-by-error. - **A native async Python API.** Without an fd, async Python is *just* the sync pull API wrapped in `asyncio.to_thread`. That is pure-Python glue in `panto/__init__.py`; no native or ABI work, so it isn't a binding deliverable. -- **Languages beyond Go and Python.** `libpanto-c` is the reuse point for any - future cgo-style or cffi-style consumer (Ruby, Node N-API, …); none are in - scope now. +- **A `libpanto-lua` LuaJIT (Lua 5.1 ABI) artifact.** Important for the + LuaJIT-based ecosystem, but a separate per-version binary requiring + `LUA_VERSION_NUM`-guarded conditional compilation. Crucially this is a + *LuaJIT-target* shim, not clean Lua 5.1: LuaJIT selectively backports bits of + 5.2/5.3 and those backports aren't reliably detectable at preprocess time, so + it must be validated against actual LuaJIT. Defer. +- **A `libpanto-lua` Lua 5.5 artifact.** Another separate per-version binary. + When this is built, **seriously consider bumping the CLI's embedded Lua to + 5.5** at the same time so the embedded VM and the standalone module track the + same modern version. Defer. (5.3, 5.2, and earlier are *not* deferred items — + they are out of scope entirely; all EOL.) +- **Languages beyond Go, Python, and Lua.** `libpanto-c` is the reuse point for + any future cgo-style or cffi-style consumer (Ruby, Node N-API, …); none are + in scope now. ## Open questions / decisions to finalize before coding @@ -321,3 +490,8 @@ terminal-by-`MessageComplete`, exhaustion-by-`null`, failure-by-error. confirm nothing else needs to. 5. **CPython stable-ABI (`abi3`) commitment** — decide in Phase 3 to fix the wheel matrix early. +6. **CLI bridge migration shape (`package.preload`)** — confirm the preload + loader integrates cleanly with the existing `src/lua_bridge.zig` install + path and the luarocks bootstrap ordering + (`docs/archive/pluggable-session-store.md`). The bare `panto` global is + dropped; extensions must `require('panto')`. |