diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/step19-cli-panto-module-plan.md | 144 |
1 files changed, 144 insertions, 0 deletions
diff --git a/docs/step19-cli-panto-module-plan.md b/docs/step19-cli-panto-module-plan.md new file mode 100644 index 0000000..f162403 --- /dev/null +++ b/docs/step19-cli-panto-module-plan.md @@ -0,0 +1,144 @@ +# Step 19 plan: CLI consumes `libpanto-lua` as a Lua module (Option A) + +Goal: inside the panto CLI's embedded Lua VM, `require('panto')` returns the +**native** `libpanto-lua` agent/stream table **plus** `panto.ext`. The CLI +obtains that table only through Lua's normal module machinery (`require`) — it +never calls `luaopen_panto` directly and knows nothing of the module's +internals. The single invariant relied upon: `luaopen_panto` returns a fresh +table (already true). + +## Mechanism + +1. **Embed `panto.so` in the CLI binary and stage it at bootstrap** into the + versioned rocks tree's C-module dir (`<tree>/lib/lua/5.4/panto.so`), which + is already on `package.cpath` (see `configurePackagePaths`). The staged + `.so` leaves `lua_*` undefined and resolves them against the host binary at + `dlopen` time — identical to how `luv.so` already works (`exe.rdynamic = + true`). + +2. **After luarocks bootstrap configures cpath**, the CLI: + ```lua + local panto = require('panto') -- finds staged panto.so + panto.ext = <ext table built in Zig> -- augment host's fresh copy + package.preload['panto'] = function() return panto end -- any later require gets ext too + ``` + `require` caches the augmented table in `package.loaded['panto']`, and the + preload entry (searcher slot 1) makes any *future* `require('panto')` — e.g. + from an extension — return the same augmented table without re-hitting cpath. + +## File-by-file changes + +### `build.zig` (CLI) +- Add `libpanto-lua` as a path dependency (`build.zig.zon`). +- Get its compiled shared object via `dep.artifact("panto").getEmittedBin()`. +- New codegen step `gen_panto_so_embed.zig`: emit a Zig module that + `@embedFile`s the `.so` bytes (binary embed is fine). Mirror + `generateLuaHeadersEmbed`. Materialize the `.so` next to the generated file + via `addCopyFile` so `@embedFile` can reach it. +- Add the embed module import to `exe_mod` and `test_mod` as + `embedded_panto_so`. + +### new `src/embedded_panto_so` import +- A single `pub const bytes: []const u8 = @embedFile("panto.so");` style module + (generated). Name: `embedded_panto_so`. + +### `src/luarocks_runtime.zig` +- New step `stagePantoModule`: write `embedded_panto_so.bytes` to + `<lib_lua_dir>/panto.so` via `writeIfDifferent`. Call it from `bootstrap` + after the tree dirs exist (alongside `stageLuaHeaders`). +- `lib_lua_dir` must exist (create if missing) before writing. + +### `src/lua_bridge.zig` +- **Stop building the native `panto` table and stop installing the preload + loader.** `install()` becomes "create the registrations tables + build the + `ext` table, stash `ext` in a registry slot" — it no longer fabricates a + `panto` table or touches `package.preload`. +- Keep all `ext` thunks (`register_tool`/`register_command`/`on`/`emit`) and + the registrations-table machinery unchanged. +- `pushPantoTable` is repurposed (or replaced by `pushExtTable`): internal Zig + code that today adds members to `panto` (`installEmit`, `_record_result`, + wrapper closure) must target the **augmented native** table after it is + obtained. Decision below. + +### `src/lua_runtime.zig` +- `create()` no longer relies on a Zig-built `panto` table. Split startup: + - `create()` opens the state, `openlibs`, builds the `ext` table + regs + tables (via the slimmed `lua_bridge.install`), creates the EventBridge. + **Defer** `installEmit` until the native table exists. + - New `installPantoModule(self)`: run the `require('panto')` + attach-`ext` + + set-`preload` sequence in Zig (a `luaL_dostring`-style snippet plus stack + surgery), then stash the augmented table in the registry slot so + `pushPantoTable` keeps working, then `installEmit`. +- `installScheduler` already runs after bootstrap; `installPantoModule` runs + between bootstrap and `installScheduler` (it needs cpath configured; the + scheduler's `_record_result`/wrapper need the table present). + +### `src/main.zig` +- After `luarocks_runtime.bootstrap(...)` returns (cpath configured, `panto.so` + staged) and before `rt.installScheduler()`, call `try rt.installPantoModule()`. + +## Sequencing (the hazard) + +Today, ordering is: `create` (builds table + preload + installEmit) → +bootstrap → installScheduler (adds `_record_result`, wrapper, caches uv.run). + +New ordering: +1. `create` — state, libs, `ext` + regs tables, EventBridge. **No table, no + preload, no emit yet.** +2. `bootstrap` — stages `panto.so`, configures cpath/path, installs luv. +3. `installPantoModule` — `require('panto')`, attach `ext`, set preload, stash + in registry slot, `installEmit`. +4. `installScheduler` — `_record_result` + wrapper closure (both reach the now- + augmented native table via the registry slot) + cache uv.run. +5. extension loading — `require('panto')` returns the augmented table. + +Everything that previously assumed "the `panto` table exists from `create`" +moves to step 3. The only consumers are `installEmit` and the scheduler, both +already post-bootstrap-or-movable. + +## Open decision: registry slot semantics + +`lua_bridge.panto_table_key` currently holds the Zig-built table. Two choices: + +- **(chosen)** Keep the slot; in `installPantoModule`, after building the + augmented table, `lua_rawsetp` it into `panto_table_key`. `pushPantoTable` + is unchanged and now yields the native+ext table. `installEmit` / + `_record_result` / wrapper all keep using `pushPantoTable`. Minimal churn. +- Alternative: split into `pushExtTable` (for `ext` members) vs `pushPantoTable` + (whole module). More precise but more edits. Not needed — `ext` lives on the + module table, so one slot pointing at the module suffices. + +## Tests to update / add + +- `lua_bridge.zig` tests assume `install` builds a `panto` table reachable via + `require('panto')`/`pushPantoTable`. These must change: either keep a tiny + test-only table builder, or rewrite the tests to drive `ext` directly. Since + `ext` is what those tests actually exercise (register_tool/command/on), point + them at the `ext` table via a test helper that fabricates a minimal module + table `{ ext = <ext> }` and stashes it — preserving coverage without the + native `.so`. +- `lua_runtime.zig` tests call `LuaRuntime.create` and `loadExtension` without + bootstrap, so `require('panto')` would fail there (no staged `.so`, no + preload). Add a test-only `installTestPantoTable(self)` that fabricates the + `{ ext = <ext> }` module table and registers the preload loader, mirroring + what the old `create` did — so the runtime tests keep passing without a real + native module. `writeTempScript` already prepends `require('panto')`. +- New integration check (best-effort, may be manual): in a built CLI, + `require('panto')` yields a table with both `agent` (native) and `ext`. + +## Risks + +- **Binary size**: embedding `panto.so` (~5.6 MB debug) bloats the CLI binary. + ReleaseFast/strip will shrink it; acceptable (luv and the agent tree are also + embedded). Confirm size is tolerable; consider stripping the `.so` in the + module's ReleaseSafe/Fast build. +- **Two Lua builds must be ABI-identical**: the CLI's statically-linked Lua + (5.4.7) and the headers `panto.so` was compiled against (5.4.7, same pin). + Both come from the same `lua_src` tarball hash — consistent by construction. +- **`dep.artifact("panto")`**: `libpanto-lua/build.zig` currently emits the + `.so` through `addLibrary` + a manual install-file rename. `artifact("panto")` + returns the `addLibrary` Compile step (named `panto`), whose `getEmittedBin()` + is `libpanto.so`/`libpanto.dylib`. Embedding by bytes is name-agnostic; we + stage it as `panto.so`. Verify the dependency exposes the artifact (it must be + `b.installArtifact`'d or at least addressable; may need a small tweak to + `libpanto-lua/build.zig` to expose it cleanly to dependents). |