1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
|
# Step 19 plan: CLI consumes `libpanto-lua` as a Lua module (Option A)
Goal: inside the panto CLI's embedded Lua VM, `require('panto')` returns the
**native** `libpanto-lua` agent/stream table **plus** `panto.ext`. The CLI
obtains that table only through Lua's normal module machinery (`require`) — it
never calls `luaopen_panto` directly and knows nothing of the module's
internals. The single invariant relied upon: `luaopen_panto` returns a fresh
table (already true).
## Mechanism
1. **Embed `panto.so` in the CLI binary and stage it at bootstrap** into the
versioned rocks tree's C-module dir (`<tree>/lib/lua/5.4/panto.so`), which
is already on `package.cpath` (see `configurePackagePaths`). The staged
`.so` leaves `lua_*` undefined and resolves them against the host binary at
`dlopen` time — identical to how `luv.so` already works (`exe.rdynamic =
true`).
2. **After luarocks bootstrap configures cpath**, the CLI:
```lua
local panto = require('panto') -- finds staged panto.so
panto.ext = <ext table built in Zig> -- augment host's fresh copy
package.preload['panto'] = function() return panto end -- any later require gets ext too
```
`require` caches the augmented table in `package.loaded['panto']`, and the
preload entry (searcher slot 1) makes any *future* `require('panto')` — e.g.
from an extension — return the same augmented table without re-hitting cpath.
## File-by-file changes
### `build.zig` (CLI)
- Add `libpanto-lua` as a path dependency (`build.zig.zon`).
- Get its compiled shared object via `dep.artifact("panto").getEmittedBin()`.
- New codegen step `gen_panto_so_embed.zig`: emit a Zig module that
`@embedFile`s the `.so` bytes (binary embed is fine). Mirror
`generateLuaHeadersEmbed`. Materialize the `.so` next to the generated file
via `addCopyFile` so `@embedFile` can reach it.
- Add the embed module import to `exe_mod` and `test_mod` as
`embedded_panto_so`.
### new `src/embedded_panto_so` import
- A single `pub const bytes: []const u8 = @embedFile("panto.so");` style module
(generated). Name: `embedded_panto_so`.
### `src/luarocks_runtime.zig`
- New step `stagePantoModule`: write `embedded_panto_so.bytes` to
`<lib_lua_dir>/panto.so` via `writeIfDifferent`. Call it from `bootstrap`
after the tree dirs exist (alongside `stageLuaHeaders`).
- `lib_lua_dir` must exist (create if missing) before writing.
### `src/lua_bridge.zig`
- **Stop building the native `panto` table and stop installing the preload
loader.** `install()` becomes "create the registrations tables + build the
`ext` table, stash `ext` in a registry slot" — it no longer fabricates a
`panto` table or touches `package.preload`.
- Keep all `ext` thunks (`register_tool`/`register_command`/`on`/`emit`) and
the registrations-table machinery unchanged.
- `pushPantoTable` is repurposed (or replaced by `pushExtTable`): internal Zig
code that today adds members to `panto` (`installEmit`, `_record_result`,
wrapper closure) must target the **augmented native** table after it is
obtained. Decision below.
### `src/lua_runtime.zig`
- `create()` no longer relies on a Zig-built `panto` table. Split startup:
- `create()` opens the state, `openlibs`, builds the `ext` table + regs
tables (via the slimmed `lua_bridge.install`), creates the EventBridge.
**Defer** `installEmit` until the native table exists.
- New `installPantoModule(self)`: run the `require('panto')` + attach-`ext` +
set-`preload` sequence in Zig (a `luaL_dostring`-style snippet plus stack
surgery), then stash the augmented table in the registry slot so
`pushPantoTable` keeps working, then `installEmit`.
- `installScheduler` already runs after bootstrap; `installPantoModule` runs
between bootstrap and `installScheduler` (it needs cpath configured; the
scheduler's `_record_result`/wrapper need the table present).
### `src/main.zig`
- After `luarocks_runtime.bootstrap(...)` returns (cpath configured, `panto.so`
staged) and before `rt.installScheduler()`, call `try rt.installPantoModule()`.
## Sequencing (the hazard)
Today, ordering is: `create` (builds table + preload + installEmit) →
bootstrap → installScheduler (adds `_record_result`, wrapper, caches uv.run).
New ordering:
1. `create` — state, libs, `ext` + regs tables, EventBridge. **No table, no
preload, no emit yet.**
2. `bootstrap` — stages `panto.so`, configures cpath/path, installs luv.
3. `installPantoModule` — `require('panto')`, attach `ext`, set preload, stash
in registry slot, `installEmit`.
4. `installScheduler` — `_record_result` + wrapper closure (both reach the now-
augmented native table via the registry slot) + cache uv.run.
5. extension loading — `require('panto')` returns the augmented table.
Everything that previously assumed "the `panto` table exists from `create`"
moves to step 3. The only consumers are `installEmit` and the scheduler, both
already post-bootstrap-or-movable.
## Open decision: registry slot semantics
`lua_bridge.panto_table_key` currently holds the Zig-built table. Two choices:
- **(chosen)** Keep the slot; in `installPantoModule`, after building the
augmented table, `lua_rawsetp` it into `panto_table_key`. `pushPantoTable`
is unchanged and now yields the native+ext table. `installEmit` /
`_record_result` / wrapper all keep using `pushPantoTable`. Minimal churn.
- Alternative: split into `pushExtTable` (for `ext` members) vs `pushPantoTable`
(whole module). More precise but more edits. Not needed — `ext` lives on the
module table, so one slot pointing at the module suffices.
## Tests to update / add
- `lua_bridge.zig` tests assume `install` builds a `panto` table reachable via
`require('panto')`/`pushPantoTable`. These must change: either keep a tiny
test-only table builder, or rewrite the tests to drive `ext` directly. Since
`ext` is what those tests actually exercise (register_tool/command/on), point
them at the `ext` table via a test helper that fabricates a minimal module
table `{ ext = <ext> }` and stashes it — preserving coverage without the
native `.so`.
- `lua_runtime.zig` tests call `LuaRuntime.create` and `loadExtension` without
bootstrap, so `require('panto')` would fail there (no staged `.so`, no
preload). Add a test-only `installTestPantoTable(self)` that fabricates the
`{ ext = <ext> }` module table and registers the preload loader, mirroring
what the old `create` did — so the runtime tests keep passing without a real
native module. `writeTempScript` already prepends `require('panto')`.
- New integration check (best-effort, may be manual): in a built CLI,
`require('panto')` yields a table with both `agent` (native) and `ext`.
## Risks
- **Binary size**: embedding `panto.so` (~5.6 MB debug) bloats the CLI binary.
ReleaseFast/strip will shrink it; acceptable (luv and the agent tree are also
embedded). Confirm size is tolerable; consider stripping the `.so` in the
module's ReleaseSafe/Fast build.
- **Two Lua builds must be ABI-identical**: the CLI's statically-linked Lua
(5.4.7) and the headers `panto.so` was compiled against (5.4.7, same pin).
Both come from the same `lua_src` tarball hash — consistent by construction.
- **`dep.artifact("panto")`**: `libpanto-lua/build.zig` currently emits the
`.so` through `addLibrary` + a manual install-file rename. `artifact("panto")`
returns the `addLibrary` Compile step (named `panto`), whose `getEmittedBin()`
is `libpanto.so`/`libpanto.dylib`. Embedding by bytes is name-agnostic; we
stage it as `panto.so`. Verify the dependency exposes the artifact (it must be
`b.installArtifact`'d or at least addressable; may need a small tweak to
`libpanto-lua/build.zig` to expose it cleanly to dependents).
|
