lua makeover project doc

1 files changed, 524 insertions, 0 deletions

diff --git a/LUA_MAKEOVER.md b/LUA_MAKEOVER.md
new file mode 100644
index 0000000..416468c
--- /dev/null
+++ b/LUA_MAKEOVER.md
@@ -0,0 +1,524 @@
+# Lua Runtime Makeover
+
+Side project, separate from the main phase plan. Reworks panto's Lua
+embedding into something that fits Lua's actual concurrency model and
+opens the door to a luarocks-based extension ecosystem.
+
+Phase 3 as shipped is fine and stays. This document describes what
+replaces it.
+
+## Why
+
+The current Lua embedding has three problems, in increasing order of
+how badly they constrain us:
+
+1. **One `lua_State` per tool call.** Every `LuaTool.invoke` builds and
+   tears down a fresh interpreter. Module-global state is impossible.
+   Top-level extension code runs over and over. The API only makes
+   sense for stateless single-shot handlers, which is not how Lua
+   wants to be used.
+2. **The `Tool` contract is "thread-safe."** Right for native
+   extensions; wrong for Lua. Lua's concurrency primitive is the
+   coroutine, not the OS thread. A single `lua_State` is not safe for
+   concurrent host entry, so the only way to honor "thread-safe" with
+   Lua is one state per call — which we have, badly.
+3. **No path to a real Lua ecosystem.** The current setup discovers
+   `.lua` files on disk and that's it. There's no answer to "how do
+   extension authors depend on lua-cjson" or "how does an HTTP-using
+   tool work." The eventual answer to both is luarocks, and the
+   current architecture has no place for it.
+
+## Shape of the fix
+
+Three independent pieces that compose:
+
+1. **`ToolSource` in libpanto.** A new kind of registration alongside
+   `Tool`. A source owns one or more tools and receives all calls
+   targeting them, as a batch, on one thread per turn. Different
+   sources still run in parallel. Lua becomes one source.
+2. **Long-lived `lua_State` with cooperative scheduling.** The panto
+   CLI maintains exactly one `lua_State` for its entire lifetime.
+   Extension top-level code runs once at startup. Each tool call is a
+   coroutine. A libuv event loop drives suspended coroutines.
+3. **luarocks as the package manager.** Both panto's own runtime
+   batteries (luv, coro-*, future additions) and user-installed
+   extensions come through luarocks, installed into a tree under
+   `$XDG_DATA_HOME/panto/`. luarocks itself is embedded in the panto
+   binary as Lua source.
+
+## libpanto: `ToolSource`
+
+Native tools keep the existing `Tool` API unchanged. Adapters that
+back multiple tools through a shared runtime use the new `ToolSource`:
+
+```zig
+pub const ToolSource = struct {
+    name: []const u8,             // diagnostic only ("panto-lua")
+    tools: []const Tool.Decl,     // metadata only; no per-tool vtable
+    ctx: *anyopaque,
+    vtable: *const VTable,
+
+    pub const VTable = struct {
+        /// libpanto guarantees: for a given turn, all ToolUse calls
+        /// whose tool name belongs to this source are delivered in
+        /// one invoke_batch call, on one thread. Different sources
+        /// still execute in parallel.
+        ///
+        /// The source decides internal scheduling (coroutines,
+        /// sequential, internal worker pool).
+        invoke_batch: *const fn (
+            ctx: *anyopaque,
+            calls: []const Call,
+            results: []CallResult,    // parallel array, pre-allocated
+            allocator: Allocator,
+        ) anyerror!void,
+
+        deinit: *const fn (ctx: *anyopaque, allocator: Allocator) void,
+    };
+
+    pub const Call = struct {
+        tool_name: []const u8,
+        input: []const u8,
+    };
+
+    pub const CallResult = union(enum) {
+        ok: []u8,            // owned by allocator
+        err: anyerror,
+    };
+};
+
+pub const Tool.Decl = struct {
+    name: []const u8,
+    description: []const u8,
+    schema_json: []const u8,
+};
+```
+
+`ToolRegistry` indexes by tool name with a tagged value:
+`{ .single: *Tool }` or `{ .source: *ToolSource, .tool_index: usize }`.
+
+`Agent.registerToolSource(src: ToolSource) !void` is the new entry
+point.
+
+### Agent loop change
+
+In `runStep`, after collecting ToolUse blocks for a turn:
+
+1. Group them by owning source. Single-`Tool` entries form
+   single-entry groups.
+2. Spawn one OS thread per group.
+3. Each thread calls either `tool.vtable.invoke` (single) or
+   `source.vtable.invoke_batch` (batched).
+4. Join. Assemble ToolResult blocks in the original order.
+
+**Concurrency contract becomes:** different groups run in parallel; a
+single group is the source's problem. The "thread-safe" promise still
+holds for native `Tool`s. For Lua, it relaxes to "coroutine-safe
+within the panto-lua runtime."
+
+## Lua runtime
+
+### One `lua_State`, many coroutines, one event loop
+
+The panto CLI creates a single `lua_State` at startup. Every Lua
+extension is loaded into it. All top-level extension code runs once.
+Module-global state is real and persistent across calls.
+
+When `invoke_batch` fires for the panto-lua source:
+
+```
+1. for each call: coroutine.create(handler), coroutine.resume(co, args)
+2. uv.run() until all coroutines have completed (or errored)
+3. collect results into the CallResult array
+```
+
+That's the entire scheduler. The work happens inside libuv: when a
+coroutine calls a yield-aware libuv operation (HTTP, fs, subprocess,
+sleep) it suspends; libuv resumes it when the event fires.
+
+A wrapper layer translates libuv's callback shape into coroutine
+yields. The Luvit project's `coro-*` modules (coro-fs, coro-net,
+coro-http, coro-channel, coro-spawn) do this upstream for the common
+operations. Where they don't cover something, we write small wrappers
+in panto's own Lua code. The pattern is ~10 lines:
+
+```lua
+local function fs_open(path, flags, mode)
+  local co = coroutine.running()
+  uv.fs_open(path, flags, mode, function(err, fd)
+    coroutine.resume(co, err, fd)
+  end)
+  return coroutine.yield()
+end
+```
+
+### What this gets us
+
+- True cooperative parallel I/O within a batch. Three concurrent
+  `web_fetch` calls go through three concurrent sockets; total
+  latency is `max(req1, req2, req3)`, not sum.
+- First-class async subprocess. A `bash`-style tool that runs three
+  commands at once does it without blocking the runtime.
+- Module-global state for extensions that want it (rate limiters,
+  caches, lazy connection pools, etc.).
+- Extension top-level code runs once. Initialization is real.
+
+### The honest caveat
+
+Cooperative scheduling only helps when handlers yield. A handler that
+calls a non-yielding C function — raw `os.execute`, `io.read` against
+a slow file, an FFI call — blocks its siblings until it returns.
+Document loudly. The escape hatch is "use native extensions for
+work that can't yield."
+
+This is the same trade Python's asyncio makes with `requests` vs
+`aiohttp`. Panto's recommended posture: handlers should use libuv via
+the coro-* wrappers (or other luv-aware libraries) for I/O. Pure
+compute is fine. Calling `socket.http.request` or `os.execute` will
+work but blocks the batch.
+
+### Why libuv (not cqueues)
+
+Considered cqueues + lua-http. Better HTTP story (HTTP/2),
+coroutine-native API. Lost on **subprocess**, which has no
+maintained cqueues binding and which matters a lot for a coding
+agent. Also lost on familiarity — luv is what every Neovim user has
+seen.
+
+Trade accepted: HTTP/1.1 only via coro-http, plus a small wrapper
+layer panto maintains for the libuv operations the coro-* set
+doesn't cover.
+
+## luarocks as the package manager
+
+### Distribution model
+
+luarocks is embedded in the panto binary as `@embedFile`'d Lua
+source. At startup the runtime:
+
+1. Computes `$PANTO_HOME = $XDG_DATA_HOME/panto` (default
+   `~/.local/share/panto`).
+2. Configures the embedded `lua_State`'s `package.path` and
+   `package.cpath` to look under `$PANTO_HOME/share/lua/5.4/` and
+   `$PANTO_HOME/lib/lua/5.4/`.
+3. Bootstraps the embedded luarocks against
+   `--tree=$PANTO_HOME`.
+4. Reconciles a "runtime batteries" manifest (luv, coro-fs,
+   coro-http, coro-net, coro-channel, coro-spawn, plus any future
+   additions) — installs missing rocks, no-ops if present.
+5. Iterates user extensions from config. For `luarocks:foo`-style
+   references, ensures `foo` is installed.
+6. Hands control to the agent loop.
+
+luarocks 3.12+ no-ops on already-installed rocks and caches the
+upstream manifest, so steps 4–5 are cheap on every run after the
+first.
+
+Network failures on later runs are swallowed: the rocks are already
+there. First-run-with-no-network degrades to "no Lua extensions
+work" — native panto features and the agent loop are unaffected.
+
+### Why fully luarocks-based
+
+- One mechanism for everything Lua-distribution. Runtime batteries
+  and user extensions install the same way.
+- Single small binary. No vendored libuv, no vendored luv, no
+  `@embedFile`'d coro-* sources. luarocks itself is ~1MB of Lua
+  source that compresses well.
+- Version flexibility for batteries without re-shipping panto.
+- Matches Neovim's rocks.nvim direction — the most relevant
+  ecosystem signal for "Lua as a serious distribution target."
+- User extension story is genuinely the same as the batteries
+  story. No special cases.
+
+### Distributable artifact
+
+Single `panto` binary contains:
+
+- Zig CLI + libpanto
+- Lua 5.4 (already vendored)
+- luarocks Lua source, embedded via `@embedFile`
+- A small Zig-side bootstrap that configures `package.path` for the
+  embedded luarocks code
+
+Everything else lives under `$PANTO_HOME` and is installed on first
+run.
+
+## Migration shape
+
+Independent chunks of work, roughly in order:
+
+1. **`ToolSource` in libpanto.** Add the type, registry tagging, and
+   the per-source-thread fan-out in `runStep`. `Tool` unchanged.
+   Native extensions keep working.
+2. **Long-lived `lua_State` runtime in the CLI.** New module
+   (`lua_runtime.zig`). Loads all discovered Lua extensions once.
+   Registers itself as one `ToolSource` named `panto-lua`.
+   `invoke_batch` runs each call in a coroutine and drives
+   `uv.run()`. No batteries yet — handlers run sync.
+3. **Embed luarocks.** `build.zig.zon` fetches luarocks source.
+   `build.zig` embeds it via `@embedFile`. Runtime bootstraps it at
+   startup against `$PANTO_HOME`.
+4. **Install luv as the first battery.** Verify the cooperative
+   scheduler actually works end-to-end with a real yield-aware
+   library.
+5. **Install coro-* batteries.** Wire them as the default
+   I/O surface for extension authors.
+6. **User extension config syntax.** `luarocks:foo` references in
+   panto's config file; lockfile equivalent for reproducibility.
+   Mostly orthogonal to everything above.
+7. **Delete `LuaTool` and per-call `lua_State` machinery.** Phase 3
+   code retires.
+
+Documentation updates: phase-3.md gains a "superseded by
+LUA_MAKEOVER.md for the Lua runtime; native extension contract
+unchanged" note. The contract for native extensions ("thread-safe")
+stays as-is.
+
+## Open questions
+
+These came up during design and need resolution before
+implementation. We'll edit answers in here as decisions land.
+
+### Q1: luarocks's own C dependencies
+
+luarocks attempts to `require` several optional Lua modules via
+`pcall` and falls back to shelling out or to its own pure-Lua
+implementations when they're missing. The optional set:
+
+- **LuaSocket** (`socket.http`, `socket.ftp`) — for HTTP/FTP
+  downloads. Without it: shell out to a configured downloader
+  (`curl` or `wget`).
+- **LuaSec** (`ssl.https`) — for HTTPS. Without it *and* without
+  the LuaSocket+luarocks-internal HTTPS path: must use `curl` or
+  `wget` for HTTPS. **luarocks.org is HTTPS-only**, so this is
+  effectively mandatory in some form.
+- **LuaFileSystem** (`lfs`) — for directory operations,
+  `chdir`, file attributes. Without it: degraded fallbacks using
+  only `io.*` and `os.*`. Some operations become impossible.
+- **lua-bz2** (`bz2`) — for `.bz2` archives. Almost never
+  encountered; rocks ship as `.tar.gz` or `.zip`.
+- **LuaPosix** (`posix`) — for chmod and other POSIX ops.
+  Without it: fall back to shelling out to `chmod` etc.
+- **md5** — for checksums. luarocks has a pure-Lua fallback.
+- **`luarocks.tools.zip`** (bundled with luarocks itself) — pure
+  Lua zip/gzip. No external dependency.
+- **`luarocks.tools.tar`** (bundled) — pure Lua tar.
+
+**Correction:** there is no `--with-lua=embedded` flag — that was a
+hallucination from earlier in the design conversation. luarocks's
+`./configure` accepts `--with-lua=DIR`, `--with-lua-include=DIR`,
+`--with-lua-lib=DIR`, `--with-lua-interpreter=NAME`,
+`--lua-version=VERSION`. These point luarocks at *a* Lua install
+(which can be ours under `$PANTO_HOME`); they don't enable a
+separate "embedded" mode.
+
+**Decision:** minimize luarocks's optional Lua deps. Bootstrap runs
+luarocks in its degraded-but-functional mode using its bundled
+pure-Lua `tools.zip` and `tools.tar`, plus shell-out to `curl` (or
+`wget`) for HTTPS downloads. If any of the optional deps turn out
+to be effectively mandatory in practice, we statically link the
+native C library and embed the Lua wrapper as `@embedFile` in
+`panto` (sibling to Lua itself in `build.zig`). Likely candidates:
+LuaFileSystem (small, pure C wrapper around POSIX, very widely
+used), and possibly LuaSocket+LuaSec if shelling out to `curl`
+proves too clunky.
+
+We also depend on `curl` (or `wget`) being on PATH for downloads.
+This is universal on Unix dev machines and we accept the
+dependency. If it's missing, bootstrap surfaces a clear error.
+
+### Q2: C toolchain on first run
+
+luv has a C component. Building it requires `cc`, `make`, and Lua
+headers. On dev machines (panto's target audience) these are
+universal. On bare end-user machines they aren't.
+
+**Decision:** add a `panto bootstrap` subcommand. It's effectively
+a no-op `panto` invocation that exercises the same
+fetch-and-install path that every normal startup runs, just
+without entering the agent loop afterwards. On a clean machine
+it's where the slow first-run download-and-compile happens with
+visible output; on subsequent runs it's a fast no-op equivalent
+to what every `panto` startup already does.
+
+Every normal `panto` startup runs the same sync logic. The
+`bootstrap` subcommand isn't a *separate* mechanism — it's a way
+to run the sync explicitly without the agent loop, for users who
+want to do setup ahead of time, or for CI/scripted installs.
+
+When the toolchain is missing, surface a friendly error from
+the bootstrap code path before luarocks itself barfs. "You need
+a C compiler and make installed to compile Lua extensions" or
+similar. Native panto features keep working regardless — only
+the Lua tool runtime is gated on successful bootstrap.
+
+### Q3: Lua headers for C rocks
+
+Building C rocks against panto's embedded Lua requires the Lua
+headers to be findable on disk.
+
+**Decision:** drop the headers into `$PANTO_HOME/rocks/lua-X.Y.Z/include/`
+at bootstrap time, where `X.Y.Z` is the Lua version panto is
+built against. Embed the header sources via `@embedFile` (they're
+already available via the `lua_src` build dep). Bootstrap writes
+them out on first run and on any panto upgrade that changes the
+Lua version.
+
+**Why a versioned subdirectory:** rocks compiled against Lua
+5.4.7's headers are not safe to load into a Lua 5.5 interpreter
+(ABI changes happen across minor versions). The whole rock tree
+lives under `$PANTO_HOME/rocks/lua-X.Y.Z/` and each Lua version
+gets its own. A panto upgrade that bumps Lua creates a new tree
+and reinstalls everything against it. The old tree is left in
+place for rollback; users (or a future `panto gc` command) can
+delete stale ones.
+
+Directory layout:
+
+```
+$PANTO_HOME/
+  rocks/
+    lua-5.4.7/
+      include/              ← Lua headers
+      share/lua/5.4/        ← installed pure-Lua rocks
+      lib/lua/5.4/          ← installed C rocks
+      ...luarocks metadata...
+    lua-5.5.0/              ← after a future upgrade
+      ...
+```
+
+luarocks is invoked with `--tree=$PANTO_HOME/rocks/lua-5.4.7` and
+configured (via its config file or CLI flags) to know that the
+Lua headers live at `$PANTO_HOME/rocks/lua-5.4.7/include/`. The
+tree contains everything needed for that Lua version including
+the headers, which keeps rebuilds reproducible and rollback
+clean.
+
+### Q4: The `lua` interpreter that luarocks expects on PATH
+
+luarocks uses an external `lua` binary for some operations (running
+rockspec build scripts, primarily). It needs to be on PATH and
+needs to be the same version as the embedded interpreter.
+
+**Decision:** `panto lua` is a first-class user-visible subcommand
+that wraps the **upstream `lua.c` standalone interpreter** with
+panto's environment pre-configured.
+
+Mechanically:
+
+- Compile `lua.c` (the upstream standalone interpreter, ~600 lines)
+  into the panto binary as a subcommand entry point. Currently
+  excluded from `lua_files` in `build.zig`; include it for the
+  `lua` subcommand path.
+- `panto lua` arguments pass through to `lua.c`'s normal
+  command-line handling (`-i`, `-l`, `-e`, `script.lua args...`,
+  etc.). Full standalone-interpreter behavior, not a luarocks-only
+  subset.
+- Before handing control to `lua.c`'s main, panto's subcommand
+  setup runs the same bootstrap as `panto run` (verify batteries
+  installed, install missing rocks, configure `package.path` /
+  `package.cpath` to find `$PANTO_HOME/rocks/lua-X.Y.Z/...`).
+- Configure luarocks (via its config file written to
+  `$PANTO_HOME/rocks/lua-X.Y.Z/config.lua`) to use
+  `<absolute panto path> lua` as its Lua interpreter. luarocks's
+  `--with-lua-interpreter=...` flag accepts an executable name;
+  we either symlink or use the full argv mechanism.
+
+This gives users a real `lua` they can use to test their
+extensions in panto's environment — `require "luv"` and
+`require "coro-http"` work, plus anything else they've installed
+via `panto lua -e 'require("luarocks.cmd").run(...)'` or similar.
+
+**Side benefit (Q7-related):** until we have a proper user-facing
+`panto rocks install foo` command, `panto lua` is also the user's
+escape hatch for installing extra rocks into `$PANTO_HOME`. We
+can invoke luarocks itself through it.
+
+### Q5: Reproducibility / lockfile
+
+luarocks installs latest-matching by default. For a CLI tool we
+want reproducible: the same panto version installs the same
+battery versions on every fresh machine.
+
+**Decision:** pin exact versions in a panto-internal manifest
+shipped with the binary. No user-facing `panto lock` or
+`panto sync` commands — sync is what every startup (and
+`panto bootstrap`) does automatically.
+
+A manifest file (likely `runtime-batteries.zon` or similar in the
+panto source tree) lists exact versions:
+
+```zig
+.{
+    .lua_version = "5.4.7",
+    .luarocks_version = "3.12.2",
+    .batteries = .{
+        .luv = "1.51.0-1",
+        .{ .@"coro-fs" = "3.0.4-1" },
+        .{ .@"coro-http" = "3.2.1-1" },
+        .{ .@"coro-net" = "3.2.1-1" },
+        .{ .@"coro-channel" = "3.0.4-1" },
+        .{ .@"coro-spawn" = "3.2.1-1" },
+    },
+}
+```
+
+Bumping any of these is a deliberate edit + commit + version bump
+of panto itself. Each panto release pins one consistent set.
+
+Bootstrap reads the embedded manifest and ensures the tree matches:
+any rock not present at the pinned version gets installed; any
+stale versions get removed. A panto upgrade that bumps Lua
+creates an entirely new tree (per Q3) and installs everything
+fresh against it.
+
+### Q6: Where in the agent loop does the runtime live
+
+**Decision:** CLI-side. libpanto continues to be native-only and
+Lua-unaware.
+
+The long-lived `lua_State` is constructed in the panto CLI's
+`main` (or a module it calls) before the `Agent` is built.
+Bootstrap runs first, then the runtime loads all discovered
+Lua extensions into the state, then the runtime registers itself
+with the `Agent` as a single `ToolSource` named `panto-lua`.
+The source's `ctx` holds the runtime; the `lua_State` lives
+inside it.
+
+libpanto's only concept is `Tool` and `ToolSource`. It has no
+idea that one of its sources happens to be Lua-backed.
+
+### Q7: What "user extension" actually means in the new world
+
+**Decision (for now):** keep the phase-3 directory-based discovery
+as the only user extension mechanism. Local `.lua` files in
+`~/.config/panto/extensions/` and `./.panto/extensions/`. No
+config file, no `luarocks:foo` references yet.
+
+Directory-discovered extensions get access to whatever's in
+`$PANTO_HOME/rocks/lua-X.Y.Z/` — the runtime batteries (luv,
+coro-*, future additions) and nothing else by default. They can
+`require` any of those modules and pure-Lua code they write
+themselves; that's the supported surface.
+
+**Escape hatch for extra rocks:** users who need a third-party
+Lua library (lua-cjson, lpeg, etc.) for their local extension
+can install it manually via `panto lua` + the embedded luarocks:
+
+```
+panto lua -e 'require("luarocks.cmd").run("install", "lua-cjson")'
+```
+
+or more sugared if we feel like making that look better. The
+rock lands in `$PANTO_HOME/rocks/lua-X.Y.Z/` and survives across
+panto runs.
+
+**Future work (not part of this makeover):** a proper config
+file with `luarocks:panto-subagents`-style references, where
+panto-published extensions can declare their own Lua library
+dependencies in their rockspec and bootstrap installs the
+whole graph automatically. The infrastructure built here
+(luarocks-as-runtime, `$PANTO_HOME` tree, version pinning)
+directly enables this — it's just an orthogonal piece of
+user-facing surface that hasn't been designed yet.