From b788eb05c6d194b91fdc141b6655e61ccaa76ddb Mon Sep 17 00:00:00 2001 From: T Date: Tue, 26 May 2026 14:33:29 -0600 Subject: lua makeover project doc --- LUA_MAKEOVER.md | 524 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 524 insertions(+) create mode 100644 LUA_MAKEOVER.md (limited to 'LUA_MAKEOVER.md') diff --git a/LUA_MAKEOVER.md b/LUA_MAKEOVER.md new file mode 100644 index 0000000..416468c --- /dev/null +++ b/LUA_MAKEOVER.md @@ -0,0 +1,524 @@ +# Lua Runtime Makeover + +Side project, separate from the main phase plan. Reworks panto's Lua +embedding into something that fits Lua's actual concurrency model and +opens the door to a luarocks-based extension ecosystem. + +Phase 3 as shipped is fine and stays. This document describes what +replaces it. + +## Why + +The current Lua embedding has three problems, in increasing order of +how badly they constrain us: + +1. **One `lua_State` per tool call.** Every `LuaTool.invoke` builds and + tears down a fresh interpreter. Module-global state is impossible. + Top-level extension code runs over and over. The API only makes + sense for stateless single-shot handlers, which is not how Lua + wants to be used. +2. **The `Tool` contract is "thread-safe."** Right for native + extensions; wrong for Lua. Lua's concurrency primitive is the + coroutine, not the OS thread. A single `lua_State` is not safe for + concurrent host entry, so the only way to honor "thread-safe" with + Lua is one state per call — which we have, badly. +3. **No path to a real Lua ecosystem.** The current setup discovers + `.lua` files on disk and that's it. There's no answer to "how do + extension authors depend on lua-cjson" or "how does an HTTP-using + tool work." The eventual answer to both is luarocks, and the + current architecture has no place for it. + +## Shape of the fix + +Three independent pieces that compose: + +1. **`ToolSource` in libpanto.** A new kind of registration alongside + `Tool`. A source owns one or more tools and receives all calls + targeting them, as a batch, on one thread per turn. Different + sources still run in parallel. Lua becomes one source. +2. **Long-lived `lua_State` with cooperative scheduling.** The panto + CLI maintains exactly one `lua_State` for its entire lifetime. + Extension top-level code runs once at startup. Each tool call is a + coroutine. A libuv event loop drives suspended coroutines. +3. **luarocks as the package manager.** Both panto's own runtime + batteries (luv, coro-*, future additions) and user-installed + extensions come through luarocks, installed into a tree under + `$XDG_DATA_HOME/panto/`. luarocks itself is embedded in the panto + binary as Lua source. + +## libpanto: `ToolSource` + +Native tools keep the existing `Tool` API unchanged. Adapters that +back multiple tools through a shared runtime use the new `ToolSource`: + +```zig +pub const ToolSource = struct { + name: []const u8, // diagnostic only ("panto-lua") + tools: []const Tool.Decl, // metadata only; no per-tool vtable + ctx: *anyopaque, + vtable: *const VTable, + + pub const VTable = struct { + /// libpanto guarantees: for a given turn, all ToolUse calls + /// whose tool name belongs to this source are delivered in + /// one invoke_batch call, on one thread. Different sources + /// still execute in parallel. + /// + /// The source decides internal scheduling (coroutines, + /// sequential, internal worker pool). + invoke_batch: *const fn ( + ctx: *anyopaque, + calls: []const Call, + results: []CallResult, // parallel array, pre-allocated + allocator: Allocator, + ) anyerror!void, + + deinit: *const fn (ctx: *anyopaque, allocator: Allocator) void, + }; + + pub const Call = struct { + tool_name: []const u8, + input: []const u8, + }; + + pub const CallResult = union(enum) { + ok: []u8, // owned by allocator + err: anyerror, + }; +}; + +pub const Tool.Decl = struct { + name: []const u8, + description: []const u8, + schema_json: []const u8, +}; +``` + +`ToolRegistry` indexes by tool name with a tagged value: +`{ .single: *Tool }` or `{ .source: *ToolSource, .tool_index: usize }`. + +`Agent.registerToolSource(src: ToolSource) !void` is the new entry +point. + +### Agent loop change + +In `runStep`, after collecting ToolUse blocks for a turn: + +1. Group them by owning source. Single-`Tool` entries form + single-entry groups. +2. Spawn one OS thread per group. +3. Each thread calls either `tool.vtable.invoke` (single) or + `source.vtable.invoke_batch` (batched). +4. Join. Assemble ToolResult blocks in the original order. + +**Concurrency contract becomes:** different groups run in parallel; a +single group is the source's problem. The "thread-safe" promise still +holds for native `Tool`s. For Lua, it relaxes to "coroutine-safe +within the panto-lua runtime." + +## Lua runtime + +### One `lua_State`, many coroutines, one event loop + +The panto CLI creates a single `lua_State` at startup. Every Lua +extension is loaded into it. All top-level extension code runs once. +Module-global state is real and persistent across calls. + +When `invoke_batch` fires for the panto-lua source: + +``` +1. for each call: coroutine.create(handler), coroutine.resume(co, args) +2. uv.run() until all coroutines have completed (or errored) +3. collect results into the CallResult array +``` + +That's the entire scheduler. The work happens inside libuv: when a +coroutine calls a yield-aware libuv operation (HTTP, fs, subprocess, +sleep) it suspends; libuv resumes it when the event fires. + +A wrapper layer translates libuv's callback shape into coroutine +yields. The Luvit project's `coro-*` modules (coro-fs, coro-net, +coro-http, coro-channel, coro-spawn) do this upstream for the common +operations. Where they don't cover something, we write small wrappers +in panto's own Lua code. The pattern is ~10 lines: + +```lua +local function fs_open(path, flags, mode) + local co = coroutine.running() + uv.fs_open(path, flags, mode, function(err, fd) + coroutine.resume(co, err, fd) + end) + return coroutine.yield() +end +``` + +### What this gets us + +- True cooperative parallel I/O within a batch. Three concurrent + `web_fetch` calls go through three concurrent sockets; total + latency is `max(req1, req2, req3)`, not sum. +- First-class async subprocess. A `bash`-style tool that runs three + commands at once does it without blocking the runtime. +- Module-global state for extensions that want it (rate limiters, + caches, lazy connection pools, etc.). +- Extension top-level code runs once. Initialization is real. + +### The honest caveat + +Cooperative scheduling only helps when handlers yield. A handler that +calls a non-yielding C function — raw `os.execute`, `io.read` against +a slow file, an FFI call — blocks its siblings until it returns. +Document loudly. The escape hatch is "use native extensions for +work that can't yield." + +This is the same trade Python's asyncio makes with `requests` vs +`aiohttp`. Panto's recommended posture: handlers should use libuv via +the coro-* wrappers (or other luv-aware libraries) for I/O. Pure +compute is fine. Calling `socket.http.request` or `os.execute` will +work but blocks the batch. + +### Why libuv (not cqueues) + +Considered cqueues + lua-http. Better HTTP story (HTTP/2), +coroutine-native API. Lost on **subprocess**, which has no +maintained cqueues binding and which matters a lot for a coding +agent. Also lost on familiarity — luv is what every Neovim user has +seen. + +Trade accepted: HTTP/1.1 only via coro-http, plus a small wrapper +layer panto maintains for the libuv operations the coro-* set +doesn't cover. + +## luarocks as the package manager + +### Distribution model + +luarocks is embedded in the panto binary as `@embedFile`'d Lua +source. At startup the runtime: + +1. Computes `$PANTO_HOME = $XDG_DATA_HOME/panto` (default + `~/.local/share/panto`). +2. Configures the embedded `lua_State`'s `package.path` and + `package.cpath` to look under `$PANTO_HOME/share/lua/5.4/` and + `$PANTO_HOME/lib/lua/5.4/`. +3. Bootstraps the embedded luarocks against + `--tree=$PANTO_HOME`. +4. Reconciles a "runtime batteries" manifest (luv, coro-fs, + coro-http, coro-net, coro-channel, coro-spawn, plus any future + additions) — installs missing rocks, no-ops if present. +5. Iterates user extensions from config. For `luarocks:foo`-style + references, ensures `foo` is installed. +6. Hands control to the agent loop. + +luarocks 3.12+ no-ops on already-installed rocks and caches the +upstream manifest, so steps 4–5 are cheap on every run after the +first. + +Network failures on later runs are swallowed: the rocks are already +there. First-run-with-no-network degrades to "no Lua extensions +work" — native panto features and the agent loop are unaffected. + +### Why fully luarocks-based + +- One mechanism for everything Lua-distribution. Runtime batteries + and user extensions install the same way. +- Single small binary. No vendored libuv, no vendored luv, no + `@embedFile`'d coro-* sources. luarocks itself is ~1MB of Lua + source that compresses well. +- Version flexibility for batteries without re-shipping panto. +- Matches Neovim's rocks.nvim direction — the most relevant + ecosystem signal for "Lua as a serious distribution target." +- User extension story is genuinely the same as the batteries + story. No special cases. + +### Distributable artifact + +Single `panto` binary contains: + +- Zig CLI + libpanto +- Lua 5.4 (already vendored) +- luarocks Lua source, embedded via `@embedFile` +- A small Zig-side bootstrap that configures `package.path` for the + embedded luarocks code + +Everything else lives under `$PANTO_HOME` and is installed on first +run. + +## Migration shape + +Independent chunks of work, roughly in order: + +1. **`ToolSource` in libpanto.** Add the type, registry tagging, and + the per-source-thread fan-out in `runStep`. `Tool` unchanged. + Native extensions keep working. +2. **Long-lived `lua_State` runtime in the CLI.** New module + (`lua_runtime.zig`). Loads all discovered Lua extensions once. + Registers itself as one `ToolSource` named `panto-lua`. + `invoke_batch` runs each call in a coroutine and drives + `uv.run()`. No batteries yet — handlers run sync. +3. **Embed luarocks.** `build.zig.zon` fetches luarocks source. + `build.zig` embeds it via `@embedFile`. Runtime bootstraps it at + startup against `$PANTO_HOME`. +4. **Install luv as the first battery.** Verify the cooperative + scheduler actually works end-to-end with a real yield-aware + library. +5. **Install coro-* batteries.** Wire them as the default + I/O surface for extension authors. +6. **User extension config syntax.** `luarocks:foo` references in + panto's config file; lockfile equivalent for reproducibility. + Mostly orthogonal to everything above. +7. **Delete `LuaTool` and per-call `lua_State` machinery.** Phase 3 + code retires. + +Documentation updates: phase-3.md gains a "superseded by +LUA_MAKEOVER.md for the Lua runtime; native extension contract +unchanged" note. The contract for native extensions ("thread-safe") +stays as-is. + +## Open questions + +These came up during design and need resolution before +implementation. We'll edit answers in here as decisions land. + +### Q1: luarocks's own C dependencies + +luarocks attempts to `require` several optional Lua modules via +`pcall` and falls back to shelling out or to its own pure-Lua +implementations when they're missing. The optional set: + +- **LuaSocket** (`socket.http`, `socket.ftp`) — for HTTP/FTP + downloads. Without it: shell out to a configured downloader + (`curl` or `wget`). +- **LuaSec** (`ssl.https`) — for HTTPS. Without it *and* without + the LuaSocket+luarocks-internal HTTPS path: must use `curl` or + `wget` for HTTPS. **luarocks.org is HTTPS-only**, so this is + effectively mandatory in some form. +- **LuaFileSystem** (`lfs`) — for directory operations, + `chdir`, file attributes. Without it: degraded fallbacks using + only `io.*` and `os.*`. Some operations become impossible. +- **lua-bz2** (`bz2`) — for `.bz2` archives. Almost never + encountered; rocks ship as `.tar.gz` or `.zip`. +- **LuaPosix** (`posix`) — for chmod and other POSIX ops. + Without it: fall back to shelling out to `chmod` etc. +- **md5** — for checksums. luarocks has a pure-Lua fallback. +- **`luarocks.tools.zip`** (bundled with luarocks itself) — pure + Lua zip/gzip. No external dependency. +- **`luarocks.tools.tar`** (bundled) — pure Lua tar. + +**Correction:** there is no `--with-lua=embedded` flag — that was a +hallucination from earlier in the design conversation. luarocks's +`./configure` accepts `--with-lua=DIR`, `--with-lua-include=DIR`, +`--with-lua-lib=DIR`, `--with-lua-interpreter=NAME`, +`--lua-version=VERSION`. These point luarocks at *a* Lua install +(which can be ours under `$PANTO_HOME`); they don't enable a +separate "embedded" mode. + +**Decision:** minimize luarocks's optional Lua deps. Bootstrap runs +luarocks in its degraded-but-functional mode using its bundled +pure-Lua `tools.zip` and `tools.tar`, plus shell-out to `curl` (or +`wget`) for HTTPS downloads. If any of the optional deps turn out +to be effectively mandatory in practice, we statically link the +native C library and embed the Lua wrapper as `@embedFile` in +`panto` (sibling to Lua itself in `build.zig`). Likely candidates: +LuaFileSystem (small, pure C wrapper around POSIX, very widely +used), and possibly LuaSocket+LuaSec if shelling out to `curl` +proves too clunky. + +We also depend on `curl` (or `wget`) being on PATH for downloads. +This is universal on Unix dev machines and we accept the +dependency. If it's missing, bootstrap surfaces a clear error. + +### Q2: C toolchain on first run + +luv has a C component. Building it requires `cc`, `make`, and Lua +headers. On dev machines (panto's target audience) these are +universal. On bare end-user machines they aren't. + +**Decision:** add a `panto bootstrap` subcommand. It's effectively +a no-op `panto` invocation that exercises the same +fetch-and-install path that every normal startup runs, just +without entering the agent loop afterwards. On a clean machine +it's where the slow first-run download-and-compile happens with +visible output; on subsequent runs it's a fast no-op equivalent +to what every `panto` startup already does. + +Every normal `panto` startup runs the same sync logic. The +`bootstrap` subcommand isn't a *separate* mechanism — it's a way +to run the sync explicitly without the agent loop, for users who +want to do setup ahead of time, or for CI/scripted installs. + +When the toolchain is missing, surface a friendly error from +the bootstrap code path before luarocks itself barfs. "You need +a C compiler and make installed to compile Lua extensions" or +similar. Native panto features keep working regardless — only +the Lua tool runtime is gated on successful bootstrap. + +### Q3: Lua headers for C rocks + +Building C rocks against panto's embedded Lua requires the Lua +headers to be findable on disk. + +**Decision:** drop the headers into `$PANTO_HOME/rocks/lua-X.Y.Z/include/` +at bootstrap time, where `X.Y.Z` is the Lua version panto is +built against. Embed the header sources via `@embedFile` (they're +already available via the `lua_src` build dep). Bootstrap writes +them out on first run and on any panto upgrade that changes the +Lua version. + +**Why a versioned subdirectory:** rocks compiled against Lua +5.4.7's headers are not safe to load into a Lua 5.5 interpreter +(ABI changes happen across minor versions). The whole rock tree +lives under `$PANTO_HOME/rocks/lua-X.Y.Z/` and each Lua version +gets its own. A panto upgrade that bumps Lua creates a new tree +and reinstalls everything against it. The old tree is left in +place for rollback; users (or a future `panto gc` command) can +delete stale ones. + +Directory layout: + +``` +$PANTO_HOME/ + rocks/ + lua-5.4.7/ + include/ ← Lua headers + share/lua/5.4/ ← installed pure-Lua rocks + lib/lua/5.4/ ← installed C rocks + ...luarocks metadata... + lua-5.5.0/ ← after a future upgrade + ... +``` + +luarocks is invoked with `--tree=$PANTO_HOME/rocks/lua-5.4.7` and +configured (via its config file or CLI flags) to know that the +Lua headers live at `$PANTO_HOME/rocks/lua-5.4.7/include/`. The +tree contains everything needed for that Lua version including +the headers, which keeps rebuilds reproducible and rollback +clean. + +### Q4: The `lua` interpreter that luarocks expects on PATH + +luarocks uses an external `lua` binary for some operations (running +rockspec build scripts, primarily). It needs to be on PATH and +needs to be the same version as the embedded interpreter. + +**Decision:** `panto lua` is a first-class user-visible subcommand +that wraps the **upstream `lua.c` standalone interpreter** with +panto's environment pre-configured. + +Mechanically: + +- Compile `lua.c` (the upstream standalone interpreter, ~600 lines) + into the panto binary as a subcommand entry point. Currently + excluded from `lua_files` in `build.zig`; include it for the + `lua` subcommand path. +- `panto lua` arguments pass through to `lua.c`'s normal + command-line handling (`-i`, `-l`, `-e`, `script.lua args...`, + etc.). Full standalone-interpreter behavior, not a luarocks-only + subset. +- Before handing control to `lua.c`'s main, panto's subcommand + setup runs the same bootstrap as `panto run` (verify batteries + installed, install missing rocks, configure `package.path` / + `package.cpath` to find `$PANTO_HOME/rocks/lua-X.Y.Z/...`). +- Configure luarocks (via its config file written to + `$PANTO_HOME/rocks/lua-X.Y.Z/config.lua`) to use + ` lua` as its Lua interpreter. luarocks's + `--with-lua-interpreter=...` flag accepts an executable name; + we either symlink or use the full argv mechanism. + +This gives users a real `lua` they can use to test their +extensions in panto's environment — `require "luv"` and +`require "coro-http"` work, plus anything else they've installed +via `panto lua -e 'require("luarocks.cmd").run(...)'` or similar. + +**Side benefit (Q7-related):** until we have a proper user-facing +`panto rocks install foo` command, `panto lua` is also the user's +escape hatch for installing extra rocks into `$PANTO_HOME`. We +can invoke luarocks itself through it. + +### Q5: Reproducibility / lockfile + +luarocks installs latest-matching by default. For a CLI tool we +want reproducible: the same panto version installs the same +battery versions on every fresh machine. + +**Decision:** pin exact versions in a panto-internal manifest +shipped with the binary. No user-facing `panto lock` or +`panto sync` commands — sync is what every startup (and +`panto bootstrap`) does automatically. + +A manifest file (likely `runtime-batteries.zon` or similar in the +panto source tree) lists exact versions: + +```zig +.{ + .lua_version = "5.4.7", + .luarocks_version = "3.12.2", + .batteries = .{ + .luv = "1.51.0-1", + .{ .@"coro-fs" = "3.0.4-1" }, + .{ .@"coro-http" = "3.2.1-1" }, + .{ .@"coro-net" = "3.2.1-1" }, + .{ .@"coro-channel" = "3.0.4-1" }, + .{ .@"coro-spawn" = "3.2.1-1" }, + }, +} +``` + +Bumping any of these is a deliberate edit + commit + version bump +of panto itself. Each panto release pins one consistent set. + +Bootstrap reads the embedded manifest and ensures the tree matches: +any rock not present at the pinned version gets installed; any +stale versions get removed. A panto upgrade that bumps Lua +creates an entirely new tree (per Q3) and installs everything +fresh against it. + +### Q6: Where in the agent loop does the runtime live + +**Decision:** CLI-side. libpanto continues to be native-only and +Lua-unaware. + +The long-lived `lua_State` is constructed in the panto CLI's +`main` (or a module it calls) before the `Agent` is built. +Bootstrap runs first, then the runtime loads all discovered +Lua extensions into the state, then the runtime registers itself +with the `Agent` as a single `ToolSource` named `panto-lua`. +The source's `ctx` holds the runtime; the `lua_State` lives +inside it. + +libpanto's only concept is `Tool` and `ToolSource`. It has no +idea that one of its sources happens to be Lua-backed. + +### Q7: What "user extension" actually means in the new world + +**Decision (for now):** keep the phase-3 directory-based discovery +as the only user extension mechanism. Local `.lua` files in +`~/.config/panto/extensions/` and `./.panto/extensions/`. No +config file, no `luarocks:foo` references yet. + +Directory-discovered extensions get access to whatever's in +`$PANTO_HOME/rocks/lua-X.Y.Z/` — the runtime batteries (luv, +coro-*, future additions) and nothing else by default. They can +`require` any of those modules and pure-Lua code they write +themselves; that's the supported surface. + +**Escape hatch for extra rocks:** users who need a third-party +Lua library (lua-cjson, lpeg, etc.) for their local extension +can install it manually via `panto lua` + the embedded luarocks: + +``` +panto lua -e 'require("luarocks.cmd").run("install", "lua-cjson")' +``` + +or more sugared if we feel like making that look better. The +rock lands in `$PANTO_HOME/rocks/lua-X.Y.Z/` and survives across +panto runs. + +**Future work (not part of this makeover):** a proper config +file with `luarocks:panto-subagents`-style references, where +panto-published extensions can declare their own Lua library +dependencies in their rockspec and bootstrap installs the +whole graph automatically. The infrastructure built here +(luarocks-as-runtime, `$PANTO_HOME` tree, version pinning) +directly enables this — it's just an orthogonal piece of +user-facing surface that hasn't been designed yet. -- cgit v1.3