diff options
Diffstat (limited to 'LUA_MAKEOVER.md')
| -rw-r--r-- | LUA_MAKEOVER.md | 524 |
1 files changed, 0 insertions, 524 deletions
diff --git a/LUA_MAKEOVER.md b/LUA_MAKEOVER.md deleted file mode 100644 index 416468c..0000000 --- a/LUA_MAKEOVER.md +++ /dev/null @@ -1,524 +0,0 @@ -# Lua Runtime Makeover - -Side project, separate from the main phase plan. Reworks panto's Lua -embedding into something that fits Lua's actual concurrency model and -opens the door to a luarocks-based extension ecosystem. - -Phase 3 as shipped is fine and stays. This document describes what -replaces it. - -## Why - -The current Lua embedding has three problems, in increasing order of -how badly they constrain us: - -1. **One `lua_State` per tool call.** Every `LuaTool.invoke` builds and - tears down a fresh interpreter. Module-global state is impossible. - Top-level extension code runs over and over. The API only makes - sense for stateless single-shot handlers, which is not how Lua - wants to be used. -2. **The `Tool` contract is "thread-safe."** Right for native - extensions; wrong for Lua. Lua's concurrency primitive is the - coroutine, not the OS thread. A single `lua_State` is not safe for - concurrent host entry, so the only way to honor "thread-safe" with - Lua is one state per call — which we have, badly. -3. **No path to a real Lua ecosystem.** The current setup discovers - `.lua` files on disk and that's it. There's no answer to "how do - extension authors depend on lua-cjson" or "how does an HTTP-using - tool work." The eventual answer to both is luarocks, and the - current architecture has no place for it. - -## Shape of the fix - -Three independent pieces that compose: - -1. **`ToolSource` in libpanto.** A new kind of registration alongside - `Tool`. A source owns one or more tools and receives all calls - targeting them, as a batch, on one thread per turn. Different - sources still run in parallel. Lua becomes one source. -2. **Long-lived `lua_State` with cooperative scheduling.** The panto - CLI maintains exactly one `lua_State` for its entire lifetime. - Extension top-level code runs once at startup. Each tool call is a - coroutine. A libuv event loop drives suspended coroutines. -3. **luarocks as the package manager.** Both panto's own runtime - batteries (luv, coro-*, future additions) and user-installed - extensions come through luarocks, installed into a tree under - `$XDG_DATA_HOME/panto/`. luarocks itself is embedded in the panto - binary as Lua source. - -## libpanto: `ToolSource` - -Native tools keep the existing `Tool` API unchanged. Adapters that -back multiple tools through a shared runtime use the new `ToolSource`: - -```zig -pub const ToolSource = struct { - name: []const u8, // diagnostic only ("panto-lua") - tools: []const Tool.Decl, // metadata only; no per-tool vtable - ctx: *anyopaque, - vtable: *const VTable, - - pub const VTable = struct { - /// libpanto guarantees: for a given turn, all ToolUse calls - /// whose tool name belongs to this source are delivered in - /// one invoke_batch call, on one thread. Different sources - /// still execute in parallel. - /// - /// The source decides internal scheduling (coroutines, - /// sequential, internal worker pool). - invoke_batch: *const fn ( - ctx: *anyopaque, - calls: []const Call, - results: []CallResult, // parallel array, pre-allocated - allocator: Allocator, - ) anyerror!void, - - deinit: *const fn (ctx: *anyopaque, allocator: Allocator) void, - }; - - pub const Call = struct { - tool_name: []const u8, - input: []const u8, - }; - - pub const CallResult = union(enum) { - ok: []u8, // owned by allocator - err: anyerror, - }; -}; - -pub const Tool.Decl = struct { - name: []const u8, - description: []const u8, - schema_json: []const u8, -}; -``` - -`ToolRegistry` indexes by tool name with a tagged value: -`{ .single: *Tool }` or `{ .source: *ToolSource, .tool_index: usize }`. - -`Agent.registerToolSource(src: ToolSource) !void` is the new entry -point. - -### Agent loop change - -In `runStep`, after collecting ToolUse blocks for a turn: - -1. Group them by owning source. Single-`Tool` entries form - single-entry groups. -2. Spawn one OS thread per group. -3. Each thread calls either `tool.vtable.invoke` (single) or - `source.vtable.invoke_batch` (batched). -4. Join. Assemble ToolResult blocks in the original order. - -**Concurrency contract becomes:** different groups run in parallel; a -single group is the source's problem. The "thread-safe" promise still -holds for native `Tool`s. For Lua, it relaxes to "coroutine-safe -within the panto-lua runtime." - -## Lua runtime - -### One `lua_State`, many coroutines, one event loop - -The panto CLI creates a single `lua_State` at startup. Every Lua -extension is loaded into it. All top-level extension code runs once. -Module-global state is real and persistent across calls. - -When `invoke_batch` fires for the panto-lua source: - -``` -1. for each call: coroutine.create(handler), coroutine.resume(co, args) -2. uv.run() until all coroutines have completed (or errored) -3. collect results into the CallResult array -``` - -That's the entire scheduler. The work happens inside libuv: when a -coroutine calls a yield-aware libuv operation (HTTP, fs, subprocess, -sleep) it suspends; libuv resumes it when the event fires. - -A wrapper layer translates libuv's callback shape into coroutine -yields. The Luvit project's `coro-*` modules (coro-fs, coro-net, -coro-http, coro-channel, coro-spawn) do this upstream for the common -operations. Where they don't cover something, we write small wrappers -in panto's own Lua code. The pattern is ~10 lines: - -```lua -local function fs_open(path, flags, mode) - local co = coroutine.running() - uv.fs_open(path, flags, mode, function(err, fd) - coroutine.resume(co, err, fd) - end) - return coroutine.yield() -end -``` - -### What this gets us - -- True cooperative parallel I/O within a batch. Three concurrent - `web_fetch` calls go through three concurrent sockets; total - latency is `max(req1, req2, req3)`, not sum. -- First-class async subprocess. A `bash`-style tool that runs three - commands at once does it without blocking the runtime. -- Module-global state for extensions that want it (rate limiters, - caches, lazy connection pools, etc.). -- Extension top-level code runs once. Initialization is real. - -### The honest caveat - -Cooperative scheduling only helps when handlers yield. A handler that -calls a non-yielding C function — raw `os.execute`, `io.read` against -a slow file, an FFI call — blocks its siblings until it returns. -Document loudly. The escape hatch is "use native extensions for -work that can't yield." - -This is the same trade Python's asyncio makes with `requests` vs -`aiohttp`. Panto's recommended posture: handlers should use libuv via -the coro-* wrappers (or other luv-aware libraries) for I/O. Pure -compute is fine. Calling `socket.http.request` or `os.execute` will -work but blocks the batch. - -### Why libuv (not cqueues) - -Considered cqueues + lua-http. Better HTTP story (HTTP/2), -coroutine-native API. Lost on **subprocess**, which has no -maintained cqueues binding and which matters a lot for a coding -agent. Also lost on familiarity — luv is what every Neovim user has -seen. - -Trade accepted: HTTP/1.1 only via coro-http, plus a small wrapper -layer panto maintains for the libuv operations the coro-* set -doesn't cover. - -## luarocks as the package manager - -### Distribution model - -luarocks is embedded in the panto binary as `@embedFile`'d Lua -source. At startup the runtime: - -1. Computes `$PANTO_HOME = $XDG_DATA_HOME/panto` (default - `~/.local/share/panto`). -2. Configures the embedded `lua_State`'s `package.path` and - `package.cpath` to look under `$PANTO_HOME/share/lua/5.4/` and - `$PANTO_HOME/lib/lua/5.4/`. -3. Bootstraps the embedded luarocks against - `--tree=$PANTO_HOME`. -4. Reconciles a "runtime batteries" manifest (luv, coro-fs, - coro-http, coro-net, coro-channel, coro-spawn, plus any future - additions) — installs missing rocks, no-ops if present. -5. Iterates user extensions from config. For `luarocks:foo`-style - references, ensures `foo` is installed. -6. Hands control to the agent loop. - -luarocks 3.12+ no-ops on already-installed rocks and caches the -upstream manifest, so steps 4–5 are cheap on every run after the -first. - -Network failures on later runs are swallowed: the rocks are already -there. First-run-with-no-network degrades to "no Lua extensions -work" — native panto features and the agent loop are unaffected. - -### Why fully luarocks-based - -- One mechanism for everything Lua-distribution. Runtime batteries - and user extensions install the same way. -- Single small binary. No vendored libuv, no vendored luv, no - `@embedFile`'d coro-* sources. luarocks itself is ~1MB of Lua - source that compresses well. -- Version flexibility for batteries without re-shipping panto. -- Matches Neovim's rocks.nvim direction — the most relevant - ecosystem signal for "Lua as a serious distribution target." -- User extension story is genuinely the same as the batteries - story. No special cases. - -### Distributable artifact - -Single `panto` binary contains: - -- Zig CLI + libpanto -- Lua 5.4 (already vendored) -- luarocks Lua source, embedded via `@embedFile` -- A small Zig-side bootstrap that configures `package.path` for the - embedded luarocks code - -Everything else lives under `$PANTO_HOME` and is installed on first -run. - -## Migration shape - -Independent chunks of work, roughly in order: - -1. **`ToolSource` in libpanto.** Add the type, registry tagging, and - the per-source-thread fan-out in `runStep`. `Tool` unchanged. - Native extensions keep working. -2. **Long-lived `lua_State` runtime in the CLI.** New module - (`lua_runtime.zig`). Loads all discovered Lua extensions once. - Registers itself as one `ToolSource` named `panto-lua`. - `invoke_batch` runs each call in a coroutine and drives - `uv.run()`. No batteries yet — handlers run sync. -3. **Embed luarocks.** `build.zig.zon` fetches luarocks source. - `build.zig` embeds it via `@embedFile`. Runtime bootstraps it at - startup against `$PANTO_HOME`. -4. **Install luv as the first battery.** Verify the cooperative - scheduler actually works end-to-end with a real yield-aware - library. -5. **Install coro-* batteries.** Wire them as the default - I/O surface for extension authors. -6. **User extension config syntax.** `luarocks:foo` references in - panto's config file; lockfile equivalent for reproducibility. - Mostly orthogonal to everything above. -7. **Delete `LuaTool` and per-call `lua_State` machinery.** Phase 3 - code retires. - -Documentation updates: phase-3.md gains a "superseded by -LUA_MAKEOVER.md for the Lua runtime; native extension contract -unchanged" note. The contract for native extensions ("thread-safe") -stays as-is. - -## Open questions - -These came up during design and need resolution before -implementation. We'll edit answers in here as decisions land. - -### Q1: luarocks's own C dependencies - -luarocks attempts to `require` several optional Lua modules via -`pcall` and falls back to shelling out or to its own pure-Lua -implementations when they're missing. The optional set: - -- **LuaSocket** (`socket.http`, `socket.ftp`) — for HTTP/FTP - downloads. Without it: shell out to a configured downloader - (`curl` or `wget`). -- **LuaSec** (`ssl.https`) — for HTTPS. Without it *and* without - the LuaSocket+luarocks-internal HTTPS path: must use `curl` or - `wget` for HTTPS. **luarocks.org is HTTPS-only**, so this is - effectively mandatory in some form. -- **LuaFileSystem** (`lfs`) — for directory operations, - `chdir`, file attributes. Without it: degraded fallbacks using - only `io.*` and `os.*`. Some operations become impossible. -- **lua-bz2** (`bz2`) — for `.bz2` archives. Almost never - encountered; rocks ship as `.tar.gz` or `.zip`. -- **LuaPosix** (`posix`) — for chmod and other POSIX ops. - Without it: fall back to shelling out to `chmod` etc. -- **md5** — for checksums. luarocks has a pure-Lua fallback. -- **`luarocks.tools.zip`** (bundled with luarocks itself) — pure - Lua zip/gzip. No external dependency. -- **`luarocks.tools.tar`** (bundled) — pure Lua tar. - -**Correction:** there is no `--with-lua=embedded` flag — that was a -hallucination from earlier in the design conversation. luarocks's -`./configure` accepts `--with-lua=DIR`, `--with-lua-include=DIR`, -`--with-lua-lib=DIR`, `--with-lua-interpreter=NAME`, -`--lua-version=VERSION`. These point luarocks at *a* Lua install -(which can be ours under `$PANTO_HOME`); they don't enable a -separate "embedded" mode. - -**Decision:** minimize luarocks's optional Lua deps. Bootstrap runs -luarocks in its degraded-but-functional mode using its bundled -pure-Lua `tools.zip` and `tools.tar`, plus shell-out to `curl` (or -`wget`) for HTTPS downloads. If any of the optional deps turn out -to be effectively mandatory in practice, we statically link the -native C library and embed the Lua wrapper as `@embedFile` in -`panto` (sibling to Lua itself in `build.zig`). Likely candidates: -LuaFileSystem (small, pure C wrapper around POSIX, very widely -used), and possibly LuaSocket+LuaSec if shelling out to `curl` -proves too clunky. - -We also depend on `curl` (or `wget`) being on PATH for downloads. -This is universal on Unix dev machines and we accept the -dependency. If it's missing, bootstrap surfaces a clear error. - -### Q2: C toolchain on first run - -luv has a C component. Building it requires `cc`, `make`, and Lua -headers. On dev machines (panto's target audience) these are -universal. On bare end-user machines they aren't. - -**Decision:** add a `panto bootstrap` subcommand. It's effectively -a no-op `panto` invocation that exercises the same -fetch-and-install path that every normal startup runs, just -without entering the agent loop afterwards. On a clean machine -it's where the slow first-run download-and-compile happens with -visible output; on subsequent runs it's a fast no-op equivalent -to what every `panto` startup already does. - -Every normal `panto` startup runs the same sync logic. The -`bootstrap` subcommand isn't a *separate* mechanism — it's a way -to run the sync explicitly without the agent loop, for users who -want to do setup ahead of time, or for CI/scripted installs. - -When the toolchain is missing, surface a friendly error from -the bootstrap code path before luarocks itself barfs. "You need -a C compiler and make installed to compile Lua extensions" or -similar. Native panto features keep working regardless — only -the Lua tool runtime is gated on successful bootstrap. - -### Q3: Lua headers for C rocks - -Building C rocks against panto's embedded Lua requires the Lua -headers to be findable on disk. - -**Decision:** drop the headers into `$PANTO_HOME/rocks/lua-X.Y.Z/include/` -at bootstrap time, where `X.Y.Z` is the Lua version panto is -built against. Embed the header sources via `@embedFile` (they're -already available via the `lua_src` build dep). Bootstrap writes -them out on first run and on any panto upgrade that changes the -Lua version. - -**Why a versioned subdirectory:** rocks compiled against Lua -5.4.7's headers are not safe to load into a Lua 5.5 interpreter -(ABI changes happen across minor versions). The whole rock tree -lives under `$PANTO_HOME/rocks/lua-X.Y.Z/` and each Lua version -gets its own. A panto upgrade that bumps Lua creates a new tree -and reinstalls everything against it. The old tree is left in -place for rollback; users (or a future `panto gc` command) can -delete stale ones. - -Directory layout: - -``` -$PANTO_HOME/ - rocks/ - lua-5.4.7/ - include/ ← Lua headers - share/lua/5.4/ ← installed pure-Lua rocks - lib/lua/5.4/ ← installed C rocks - ...luarocks metadata... - lua-5.5.0/ ← after a future upgrade - ... -``` - -luarocks is invoked with `--tree=$PANTO_HOME/rocks/lua-5.4.7` and -configured (via its config file or CLI flags) to know that the -Lua headers live at `$PANTO_HOME/rocks/lua-5.4.7/include/`. The -tree contains everything needed for that Lua version including -the headers, which keeps rebuilds reproducible and rollback -clean. - -### Q4: The `lua` interpreter that luarocks expects on PATH - -luarocks uses an external `lua` binary for some operations (running -rockspec build scripts, primarily). It needs to be on PATH and -needs to be the same version as the embedded interpreter. - -**Decision:** `panto lua` is a first-class user-visible subcommand -that wraps the **upstream `lua.c` standalone interpreter** with -panto's environment pre-configured. - -Mechanically: - -- Compile `lua.c` (the upstream standalone interpreter, ~600 lines) - into the panto binary as a subcommand entry point. Currently - excluded from `lua_files` in `build.zig`; include it for the - `lua` subcommand path. -- `panto lua` arguments pass through to `lua.c`'s normal - command-line handling (`-i`, `-l`, `-e`, `script.lua args...`, - etc.). Full standalone-interpreter behavior, not a luarocks-only - subset. -- Before handing control to `lua.c`'s main, panto's subcommand - setup runs the same bootstrap as `panto run` (verify batteries - installed, install missing rocks, configure `package.path` / - `package.cpath` to find `$PANTO_HOME/rocks/lua-X.Y.Z/...`). -- Configure luarocks (via its config file written to - `$PANTO_HOME/rocks/lua-X.Y.Z/config.lua`) to use - `<absolute panto path> lua` as its Lua interpreter. luarocks's - `--with-lua-interpreter=...` flag accepts an executable name; - we either symlink or use the full argv mechanism. - -This gives users a real `lua` they can use to test their -extensions in panto's environment — `require "luv"` and -`require "coro-http"` work, plus anything else they've installed -via `panto lua -e 'require("luarocks.cmd").run(...)'` or similar. - -**Side benefit (Q7-related):** until we have a proper user-facing -`panto rocks install foo` command, `panto lua` is also the user's -escape hatch for installing extra rocks into `$PANTO_HOME`. We -can invoke luarocks itself through it. - -### Q5: Reproducibility / lockfile - -luarocks installs latest-matching by default. For a CLI tool we -want reproducible: the same panto version installs the same -battery versions on every fresh machine. - -**Decision:** pin exact versions in a panto-internal manifest -shipped with the binary. No user-facing `panto lock` or -`panto sync` commands — sync is what every startup (and -`panto bootstrap`) does automatically. - -A manifest file (likely `runtime-batteries.zon` or similar in the -panto source tree) lists exact versions: - -```zig -.{ - .lua_version = "5.4.7", - .luarocks_version = "3.12.2", - .batteries = .{ - .luv = "1.51.0-1", - .{ .@"coro-fs" = "3.0.4-1" }, - .{ .@"coro-http" = "3.2.1-1" }, - .{ .@"coro-net" = "3.2.1-1" }, - .{ .@"coro-channel" = "3.0.4-1" }, - .{ .@"coro-spawn" = "3.2.1-1" }, - }, -} -``` - -Bumping any of these is a deliberate edit + commit + version bump -of panto itself. Each panto release pins one consistent set. - -Bootstrap reads the embedded manifest and ensures the tree matches: -any rock not present at the pinned version gets installed; any -stale versions get removed. A panto upgrade that bumps Lua -creates an entirely new tree (per Q3) and installs everything -fresh against it. - -### Q6: Where in the agent loop does the runtime live - -**Decision:** CLI-side. libpanto continues to be native-only and -Lua-unaware. - -The long-lived `lua_State` is constructed in the panto CLI's -`main` (or a module it calls) before the `Agent` is built. -Bootstrap runs first, then the runtime loads all discovered -Lua extensions into the state, then the runtime registers itself -with the `Agent` as a single `ToolSource` named `panto-lua`. -The source's `ctx` holds the runtime; the `lua_State` lives -inside it. - -libpanto's only concept is `Tool` and `ToolSource`. It has no -idea that one of its sources happens to be Lua-backed. - -### Q7: What "user extension" actually means in the new world - -**Decision (for now):** keep the phase-3 directory-based discovery -as the only user extension mechanism. Local `.lua` files in -`~/.config/panto/extensions/` and `./.panto/extensions/`. No -config file, no `luarocks:foo` references yet. - -Directory-discovered extensions get access to whatever's in -`$PANTO_HOME/rocks/lua-X.Y.Z/` — the runtime batteries (luv, -coro-*, future additions) and nothing else by default. They can -`require` any of those modules and pure-Lua code they write -themselves; that's the supported surface. - -**Escape hatch for extra rocks:** users who need a third-party -Lua library (lua-cjson, lpeg, etc.) for their local extension -can install it manually via `panto lua` + the embedded luarocks: - -``` -panto lua -e 'require("luarocks.cmd").run("install", "lua-cjson")' -``` - -or more sugared if we feel like making that look better. The -rock lands in `$PANTO_HOME/rocks/lua-X.Y.Z/` and survives across -panto runs. - -**Future work (not part of this makeover):** a proper config -file with `luarocks:panto-subagents`-style references, where -panto-published extensions can declare their own Lua library -dependencies in their rockspec and bootstrap installs the -whole graph automatically. The infrastructure built here -(luarocks-as-runtime, `$PANTO_HOME` tree, version pinning) -directly enables this — it's just an orthogonal piece of -user-facing surface that hasn't been designed yet. |