finish lua runtime makeover

- new multi-tool registration via ToolSource
- thread per source-or-standalone-tool
- switched to zig 0.16 Io threading interface
- cli: include `luv` package and run concurrent lua tools via libuv
- one single long-lived lua_State for the whole cli program

1 files changed, 0 insertions, 524 deletions

diff --git a/LUA_MAKEOVER.md b/LUA_MAKEOVER.md
deleted file mode 100644
index 416468c..0000000
--- a/LUA_MAKEOVER.md
+++ /dev/null
@@ -1,524 +0,0 @@
-# Lua Runtime Makeover
-
-Side project, separate from the main phase plan. Reworks panto's Lua
-embedding into something that fits Lua's actual concurrency model and
-opens the door to a luarocks-based extension ecosystem.
-
-Phase 3 as shipped is fine and stays. This document describes what
-replaces it.
-
-## Why
-
-The current Lua embedding has three problems, in increasing order of
-how badly they constrain us:
-
-1. **One `lua_State` per tool call.** Every `LuaTool.invoke` builds and
-   tears down a fresh interpreter. Module-global state is impossible.
-   Top-level extension code runs over and over. The API only makes
-   sense for stateless single-shot handlers, which is not how Lua
-   wants to be used.
-2. **The `Tool` contract is "thread-safe."** Right for native
-   extensions; wrong for Lua. Lua's concurrency primitive is the
-   coroutine, not the OS thread. A single `lua_State` is not safe for
-   concurrent host entry, so the only way to honor "thread-safe" with
-   Lua is one state per call — which we have, badly.
-3. **No path to a real Lua ecosystem.** The current setup discovers
-   `.lua` files on disk and that's it. There's no answer to "how do
-   extension authors depend on lua-cjson" or "how does an HTTP-using
-   tool work." The eventual answer to both is luarocks, and the
-   current architecture has no place for it.
-
-## Shape of the fix
-
-Three independent pieces that compose:
-
-1. **`ToolSource` in libpanto.** A new kind of registration alongside
-   `Tool`. A source owns one or more tools and receives all calls
-   targeting them, as a batch, on one thread per turn. Different
-   sources still run in parallel. Lua becomes one source.
-2. **Long-lived `lua_State` with cooperative scheduling.** The panto
-   CLI maintains exactly one `lua_State` for its entire lifetime.
-   Extension top-level code runs once at startup. Each tool call is a
-   coroutine. A libuv event loop drives suspended coroutines.
-3. **luarocks as the package manager.** Both panto's own runtime
-   batteries (luv, coro-*, future additions) and user-installed
-   extensions come through luarocks, installed into a tree under
-   `$XDG_DATA_HOME/panto/`. luarocks itself is embedded in the panto
-   binary as Lua source.
-
-## libpanto: `ToolSource`
-
-Native tools keep the existing `Tool` API unchanged. Adapters that
-back multiple tools through a shared runtime use the new `ToolSource`:
-
-```zig
-pub const ToolSource = struct {
-    name: []const u8,             // diagnostic only ("panto-lua")
-    tools: []const Tool.Decl,     // metadata only; no per-tool vtable
-    ctx: *anyopaque,
-    vtable: *const VTable,
-
-    pub const VTable = struct {
-        /// libpanto guarantees: for a given turn, all ToolUse calls
-        /// whose tool name belongs to this source are delivered in
-        /// one invoke_batch call, on one thread. Different sources
-        /// still execute in parallel.
-        ///
-        /// The source decides internal scheduling (coroutines,
-        /// sequential, internal worker pool).
-        invoke_batch: *const fn (
-            ctx: *anyopaque,
-            calls: []const Call,
-            results: []CallResult,    // parallel array, pre-allocated
-            allocator: Allocator,
-        ) anyerror!void,
-
-        deinit: *const fn (ctx: *anyopaque, allocator: Allocator) void,
-    };
-
-    pub const Call = struct {
-        tool_name: []const u8,
-        input: []const u8,
-    };
-
-    pub const CallResult = union(enum) {
-        ok: []u8,            // owned by allocator
-        err: anyerror,
-    };
-};
-
-pub const Tool.Decl = struct {
-    name: []const u8,
-    description: []const u8,
-    schema_json: []const u8,
-};
-```
-
-`ToolRegistry` indexes by tool name with a tagged value:
-`{ .single: *Tool }` or `{ .source: *ToolSource, .tool_index: usize }`.
-
-`Agent.registerToolSource(src: ToolSource) !void` is the new entry
-point.
-
-### Agent loop change
-
-In `runStep`, after collecting ToolUse blocks for a turn:
-
-1. Group them by owning source. Single-`Tool` entries form
-   single-entry groups.
-2. Spawn one OS thread per group.
-3. Each thread calls either `tool.vtable.invoke` (single) or
-   `source.vtable.invoke_batch` (batched).
-4. Join. Assemble ToolResult blocks in the original order.
-
-**Concurrency contract becomes:** different groups run in parallel; a
-single group is the source's problem. The "thread-safe" promise still
-holds for native `Tool`s. For Lua, it relaxes to "coroutine-safe
-within the panto-lua runtime."
-
-## Lua runtime
-
-### One `lua_State`, many coroutines, one event loop
-
-The panto CLI creates a single `lua_State` at startup. Every Lua
-extension is loaded into it. All top-level extension code runs once.
-Module-global state is real and persistent across calls.
-
-When `invoke_batch` fires for the panto-lua source:
-
-```
-1. for each call: coroutine.create(handler), coroutine.resume(co, args)
-2. uv.run() until all coroutines have completed (or errored)
-3. collect results into the CallResult array
-```
-
-That's the entire scheduler. The work happens inside libuv: when a
-coroutine calls a yield-aware libuv operation (HTTP, fs, subprocess,
-sleep) it suspends; libuv resumes it when the event fires.
-
-A wrapper layer translates libuv's callback shape into coroutine
-yields. The Luvit project's `coro-*` modules (coro-fs, coro-net,
-coro-http, coro-channel, coro-spawn) do this upstream for the common
-operations. Where they don't cover something, we write small wrappers
-in panto's own Lua code. The pattern is ~10 lines:
-
-```lua
-local function fs_open(path, flags, mode)
-  local co = coroutine.running()
-  uv.fs_open(path, flags, mode, function(err, fd)
-    coroutine.resume(co, err, fd)
-  end)
-  return coroutine.yield()
-end
-```
-
-### What this gets us
-
-- True cooperative parallel I/O within a batch. Three concurrent
-  `web_fetch` calls go through three concurrent sockets; total
-  latency is `max(req1, req2, req3)`, not sum.
-- First-class async subprocess. A `bash`-style tool that runs three
-  commands at once does it without blocking the runtime.
-- Module-global state for extensions that want it (rate limiters,
-  caches, lazy connection pools, etc.).
-- Extension top-level code runs once. Initialization is real.
-
-### The honest caveat
-
-Cooperative scheduling only helps when handlers yield. A handler that
-calls a non-yielding C function — raw `os.execute`, `io.read` against
-a slow file, an FFI call — blocks its siblings until it returns.
-Document loudly. The escape hatch is "use native extensions for
-work that can't yield."
-
-This is the same trade Python's asyncio makes with `requests` vs
-`aiohttp`. Panto's recommended posture: handlers should use libuv via
-the coro-* wrappers (or other luv-aware libraries) for I/O. Pure
-compute is fine. Calling `socket.http.request` or `os.execute` will
-work but blocks the batch.
-
-### Why libuv (not cqueues)
-
-Considered cqueues + lua-http. Better HTTP story (HTTP/2),
-coroutine-native API. Lost on **subprocess**, which has no
-maintained cqueues binding and which matters a lot for a coding
-agent. Also lost on familiarity — luv is what every Neovim user has
-seen.
-
-Trade accepted: HTTP/1.1 only via coro-http, plus a small wrapper
-layer panto maintains for the libuv operations the coro-* set
-doesn't cover.
-
-## luarocks as the package manager
-
-### Distribution model
-
-luarocks is embedded in the panto binary as `@embedFile`'d Lua
-source. At startup the runtime:
-
-1. Computes `$PANTO_HOME = $XDG_DATA_HOME/panto` (default
-   `~/.local/share/panto`).
-2. Configures the embedded `lua_State`'s `package.path` and
-   `package.cpath` to look under `$PANTO_HOME/share/lua/5.4/` and
-   `$PANTO_HOME/lib/lua/5.4/`.
-3. Bootstraps the embedded luarocks against
-   `--tree=$PANTO_HOME`.
-4. Reconciles a "runtime batteries" manifest (luv, coro-fs,
-   coro-http, coro-net, coro-channel, coro-spawn, plus any future
-   additions) — installs missing rocks, no-ops if present.
-5. Iterates user extensions from config. For `luarocks:foo`-style
-   references, ensures `foo` is installed.
-6. Hands control to the agent loop.
-
-luarocks 3.12+ no-ops on already-installed rocks and caches the
-upstream manifest, so steps 4–5 are cheap on every run after the
-first.
-
-Network failures on later runs are swallowed: the rocks are already
-there. First-run-with-no-network degrades to "no Lua extensions
-work" — native panto features and the agent loop are unaffected.
-
-### Why fully luarocks-based
-
-- One mechanism for everything Lua-distribution. Runtime batteries
-  and user extensions install the same way.
-- Single small binary. No vendored libuv, no vendored luv, no
-  `@embedFile`'d coro-* sources. luarocks itself is ~1MB of Lua
-  source that compresses well.
-- Version flexibility for batteries without re-shipping panto.
-- Matches Neovim's rocks.nvim direction — the most relevant
-  ecosystem signal for "Lua as a serious distribution target."
-- User extension story is genuinely the same as the batteries
-  story. No special cases.
-
-### Distributable artifact
-
-Single `panto` binary contains:
-
-- Zig CLI + libpanto
-- Lua 5.4 (already vendored)
-- luarocks Lua source, embedded via `@embedFile`
-- A small Zig-side bootstrap that configures `package.path` for the
-  embedded luarocks code
-
-Everything else lives under `$PANTO_HOME` and is installed on first
-run.
-
-## Migration shape
-
-Independent chunks of work, roughly in order:
-
-1. **`ToolSource` in libpanto.** Add the type, registry tagging, and
-   the per-source-thread fan-out in `runStep`. `Tool` unchanged.
-   Native extensions keep working.
-2. **Long-lived `lua_State` runtime in the CLI.** New module
-   (`lua_runtime.zig`). Loads all discovered Lua extensions once.
-   Registers itself as one `ToolSource` named `panto-lua`.
-   `invoke_batch` runs each call in a coroutine and drives
-   `uv.run()`. No batteries yet — handlers run sync.
-3. **Embed luarocks.** `build.zig.zon` fetches luarocks source.
-   `build.zig` embeds it via `@embedFile`. Runtime bootstraps it at
-   startup against `$PANTO_HOME`.
-4. **Install luv as the first battery.** Verify the cooperative
-   scheduler actually works end-to-end with a real yield-aware
-   library.
-5. **Install coro-* batteries.** Wire them as the default
-   I/O surface for extension authors.
-6. **User extension config syntax.** `luarocks:foo` references in
-   panto's config file; lockfile equivalent for reproducibility.
-   Mostly orthogonal to everything above.
-7. **Delete `LuaTool` and per-call `lua_State` machinery.** Phase 3
-   code retires.
-
-Documentation updates: phase-3.md gains a "superseded by
-LUA_MAKEOVER.md for the Lua runtime; native extension contract
-unchanged" note. The contract for native extensions ("thread-safe")
-stays as-is.
-
-## Open questions
-
-These came up during design and need resolution before
-implementation. We'll edit answers in here as decisions land.
-
-### Q1: luarocks's own C dependencies
-
-luarocks attempts to `require` several optional Lua modules via
-`pcall` and falls back to shelling out or to its own pure-Lua
-implementations when they're missing. The optional set:
-
-- **LuaSocket** (`socket.http`, `socket.ftp`) — for HTTP/FTP
-  downloads. Without it: shell out to a configured downloader
-  (`curl` or `wget`).
-- **LuaSec** (`ssl.https`) — for HTTPS. Without it *and* without
-  the LuaSocket+luarocks-internal HTTPS path: must use `curl` or
-  `wget` for HTTPS. **luarocks.org is HTTPS-only**, so this is
-  effectively mandatory in some form.
-- **LuaFileSystem** (`lfs`) — for directory operations,
-  `chdir`, file attributes. Without it: degraded fallbacks using
-  only `io.*` and `os.*`. Some operations become impossible.
-- **lua-bz2** (`bz2`) — for `.bz2` archives. Almost never
-  encountered; rocks ship as `.tar.gz` or `.zip`.
-- **LuaPosix** (`posix`) — for chmod and other POSIX ops.
-  Without it: fall back to shelling out to `chmod` etc.
-- **md5** — for checksums. luarocks has a pure-Lua fallback.
-- **`luarocks.tools.zip`** (bundled with luarocks itself) — pure
-  Lua zip/gzip. No external dependency.
-- **`luarocks.tools.tar`** (bundled) — pure Lua tar.
-
-**Correction:** there is no `--with-lua=embedded` flag — that was a
-hallucination from earlier in the design conversation. luarocks's
-`./configure` accepts `--with-lua=DIR`, `--with-lua-include=DIR`,
-`--with-lua-lib=DIR`, `--with-lua-interpreter=NAME`,
-`--lua-version=VERSION`. These point luarocks at *a* Lua install
-(which can be ours under `$PANTO_HOME`); they don't enable a
-separate "embedded" mode.
-
-**Decision:** minimize luarocks's optional Lua deps. Bootstrap runs
-luarocks in its degraded-but-functional mode using its bundled
-pure-Lua `tools.zip` and `tools.tar`, plus shell-out to `curl` (or
-`wget`) for HTTPS downloads. If any of the optional deps turn out
-to be effectively mandatory in practice, we statically link the
-native C library and embed the Lua wrapper as `@embedFile` in
-`panto` (sibling to Lua itself in `build.zig`). Likely candidates:
-LuaFileSystem (small, pure C wrapper around POSIX, very widely
-used), and possibly LuaSocket+LuaSec if shelling out to `curl`
-proves too clunky.
-
-We also depend on `curl` (or `wget`) being on PATH for downloads.
-This is universal on Unix dev machines and we accept the
-dependency. If it's missing, bootstrap surfaces a clear error.
-
-### Q2: C toolchain on first run
-
-luv has a C component. Building it requires `cc`, `make`, and Lua
-headers. On dev machines (panto's target audience) these are
-universal. On bare end-user machines they aren't.
-
-**Decision:** add a `panto bootstrap` subcommand. It's effectively
-a no-op `panto` invocation that exercises the same
-fetch-and-install path that every normal startup runs, just
-without entering the agent loop afterwards. On a clean machine
-it's where the slow first-run download-and-compile happens with
-visible output; on subsequent runs it's a fast no-op equivalent
-to what every `panto` startup already does.
-
-Every normal `panto` startup runs the same sync logic. The
-`bootstrap` subcommand isn't a *separate* mechanism — it's a way
-to run the sync explicitly without the agent loop, for users who
-want to do setup ahead of time, or for CI/scripted installs.
-
-When the toolchain is missing, surface a friendly error from
-the bootstrap code path before luarocks itself barfs. "You need
-a C compiler and make installed to compile Lua extensions" or
-similar. Native panto features keep working regardless — only
-the Lua tool runtime is gated on successful bootstrap.
-
-### Q3: Lua headers for C rocks
-
-Building C rocks against panto's embedded Lua requires the Lua
-headers to be findable on disk.
-
-**Decision:** drop the headers into `$PANTO_HOME/rocks/lua-X.Y.Z/include/`
-at bootstrap time, where `X.Y.Z` is the Lua version panto is
-built against. Embed the header sources via `@embedFile` (they're
-already available via the `lua_src` build dep). Bootstrap writes
-them out on first run and on any panto upgrade that changes the
-Lua version.
-
-**Why a versioned subdirectory:** rocks compiled against Lua
-5.4.7's headers are not safe to load into a Lua 5.5 interpreter
-(ABI changes happen across minor versions). The whole rock tree
-lives under `$PANTO_HOME/rocks/lua-X.Y.Z/` and each Lua version
-gets its own. A panto upgrade that bumps Lua creates a new tree
-and reinstalls everything against it. The old tree is left in
-place for rollback; users (or a future `panto gc` command) can
-delete stale ones.
-
-Directory layout:
-
-```
-$PANTO_HOME/
-  rocks/
-    lua-5.4.7/
-      include/              ← Lua headers
-      share/lua/5.4/        ← installed pure-Lua rocks
-      lib/lua/5.4/          ← installed C rocks
-      ...luarocks metadata...
-    lua-5.5.0/              ← after a future upgrade
-      ...
-```
-
-luarocks is invoked with `--tree=$PANTO_HOME/rocks/lua-5.4.7` and
-configured (via its config file or CLI flags) to know that the
-Lua headers live at `$PANTO_HOME/rocks/lua-5.4.7/include/`. The
-tree contains everything needed for that Lua version including
-the headers, which keeps rebuilds reproducible and rollback
-clean.
-
-### Q4: The `lua` interpreter that luarocks expects on PATH
-
-luarocks uses an external `lua` binary for some operations (running
-rockspec build scripts, primarily). It needs to be on PATH and
-needs to be the same version as the embedded interpreter.
-
-**Decision:** `panto lua` is a first-class user-visible subcommand
-that wraps the **upstream `lua.c` standalone interpreter** with
-panto's environment pre-configured.
-
-Mechanically:
-
-- Compile `lua.c` (the upstream standalone interpreter, ~600 lines)
-  into the panto binary as a subcommand entry point. Currently
-  excluded from `lua_files` in `build.zig`; include it for the
-  `lua` subcommand path.
-- `panto lua` arguments pass through to `lua.c`'s normal
-  command-line handling (`-i`, `-l`, `-e`, `script.lua args...`,
-  etc.). Full standalone-interpreter behavior, not a luarocks-only
-  subset.
-- Before handing control to `lua.c`'s main, panto's subcommand
-  setup runs the same bootstrap as `panto run` (verify batteries
-  installed, install missing rocks, configure `package.path` /
-  `package.cpath` to find `$PANTO_HOME/rocks/lua-X.Y.Z/...`).
-- Configure luarocks (via its config file written to
-  `$PANTO_HOME/rocks/lua-X.Y.Z/config.lua`) to use
-  `<absolute panto path> lua` as its Lua interpreter. luarocks's
-  `--with-lua-interpreter=...` flag accepts an executable name;
-  we either symlink or use the full argv mechanism.
-
-This gives users a real `lua` they can use to test their
-extensions in panto's environment — `require "luv"` and
-`require "coro-http"` work, plus anything else they've installed
-via `panto lua -e 'require("luarocks.cmd").run(...)'` or similar.
-
-**Side benefit (Q7-related):** until we have a proper user-facing
-`panto rocks install foo` command, `panto lua` is also the user's
-escape hatch for installing extra rocks into `$PANTO_HOME`. We
-can invoke luarocks itself through it.
-
-### Q5: Reproducibility / lockfile
-
-luarocks installs latest-matching by default. For a CLI tool we
-want reproducible: the same panto version installs the same
-battery versions on every fresh machine.
-
-**Decision:** pin exact versions in a panto-internal manifest
-shipped with the binary. No user-facing `panto lock` or
-`panto sync` commands — sync is what every startup (and
-`panto bootstrap`) does automatically.
-
-A manifest file (likely `runtime-batteries.zon` or similar in the
-panto source tree) lists exact versions:
-
-```zig
-.{
-    .lua_version = "5.4.7",
-    .luarocks_version = "3.12.2",
-    .batteries = .{
-        .luv = "1.51.0-1",
-        .{ .@"coro-fs" = "3.0.4-1" },
-        .{ .@"coro-http" = "3.2.1-1" },
-        .{ .@"coro-net" = "3.2.1-1" },
-        .{ .@"coro-channel" = "3.0.4-1" },
-        .{ .@"coro-spawn" = "3.2.1-1" },
-    },
-}
-```
-
-Bumping any of these is a deliberate edit + commit + version bump
-of panto itself. Each panto release pins one consistent set.
-
-Bootstrap reads the embedded manifest and ensures the tree matches:
-any rock not present at the pinned version gets installed; any
-stale versions get removed. A panto upgrade that bumps Lua
-creates an entirely new tree (per Q3) and installs everything
-fresh against it.
-
-### Q6: Where in the agent loop does the runtime live
-
-**Decision:** CLI-side. libpanto continues to be native-only and
-Lua-unaware.
-
-The long-lived `lua_State` is constructed in the panto CLI's
-`main` (or a module it calls) before the `Agent` is built.
-Bootstrap runs first, then the runtime loads all discovered
-Lua extensions into the state, then the runtime registers itself
-with the `Agent` as a single `ToolSource` named `panto-lua`.
-The source's `ctx` holds the runtime; the `lua_State` lives
-inside it.
-
-libpanto's only concept is `Tool` and `ToolSource`. It has no
-idea that one of its sources happens to be Lua-backed.
-
-### Q7: What "user extension" actually means in the new world
-
-**Decision (for now):** keep the phase-3 directory-based discovery
-as the only user extension mechanism. Local `.lua` files in
-`~/.config/panto/extensions/` and `./.panto/extensions/`. No
-config file, no `luarocks:foo` references yet.
-
-Directory-discovered extensions get access to whatever's in
-`$PANTO_HOME/rocks/lua-X.Y.Z/` — the runtime batteries (luv,
-coro-*, future additions) and nothing else by default. They can
-`require` any of those modules and pure-Lua code they write
-themselves; that's the supported surface.
-
-**Escape hatch for extra rocks:** users who need a third-party
-Lua library (lua-cjson, lpeg, etc.) for their local extension
-can install it manually via `panto lua` + the embedded luarocks:
-
-```
-panto lua -e 'require("luarocks.cmd").run("install", "lua-cjson")'
-```
-
-or more sugared if we feel like making that look better. The
-rock lands in `$PANTO_HOME/rocks/lua-X.Y.Z/` and survives across
-panto runs.
-
-**Future work (not part of this makeover):** a proper config
-file with `luarocks:panto-subagents`-style references, where
-panto-published extensions can declare their own Lua library
-dependencies in their rockspec and bootstrap installs the
-whole graph automatically. The infrastructure built here
-(luarocks-as-runtime, `$PANTO_HOME` tree, version pinning)
-directly enables this — it's just an orthogonal piece of
-user-facing surface that hasn't been designed yet.