path: root/docs/archive/LUA_MAKEOVER.md

# Lua Runtime Makeover

Side project, separate from the main phase plan. Reworks panto's Lua
embedding into something that fits Lua's actual concurrency model and
opens the door to a luarocks-based extension ecosystem.

Phase 3 as shipped is fine and stays. This document describes what
replaces it.

## Why

The current Lua embedding has three problems, in increasing order of
how badly they constrain us:

1. **One `lua_State` per tool call.** Every `LuaTool.invoke` builds and
   tears down a fresh interpreter. Module-global state is impossible.
   Top-level extension code runs over and over. The API only makes
   sense for stateless single-shot handlers, which is not how Lua
   wants to be used.
2. **The `Tool` contract is "thread-safe."** Right for native
   extensions; wrong for Lua. Lua's concurrency primitive is the
   coroutine, not the OS thread. A single `lua_State` is not safe for
   concurrent host entry, so the only way to honor "thread-safe" with
   Lua is one state per call — which we have, badly.
3. **No path to a real Lua ecosystem.** The current setup discovers
   `.lua` files on disk and that's it. There's no answer to "how do
   extension authors depend on lua-cjson" or "how does an HTTP-using
   tool work." The eventual answer to both is luarocks, and the
   current architecture has no place for it.

## Shape of the fix

Three independent pieces that compose:

1. **`ToolSource` in libpanto.** A new kind of registration alongside
   `Tool`. A source owns one or more tools and receives all calls
   targeting them, as a batch, on one thread per turn. Different
   sources still run in parallel. Lua becomes one source.
2. **Long-lived `lua_State` with cooperative scheduling.** The panto
   CLI maintains exactly one `lua_State` for its entire lifetime.
   Extension top-level code runs once at startup. Each tool call is a
   coroutine. A libuv event loop drives suspended coroutines.
3. **luarocks as the package manager.** Both panto's own runtime
   battery (luv — see note below) and user-installed extensions
   come through luarocks, installed into a tree under
   `$XDG_DATA_HOME/panto/`. luarocks itself is embedded in the panto
   binary as Lua source.

   Originally planned to ship `coro-*` (coro-fs, coro-http, coro-net,
   coro-channel, coro-spawn) as additional batteries; that plan was
   dropped during implementation. luv alone provides the libuv
   surface (callback-shaped, but coroutine-friendly with a `co =
   coroutine.running(); ...; coroutine.yield()` pattern); shipping
   the smallest possible set keeps the makeover focused and lets us
   write panto's own `std` tools directly against luv. Users who
   want coro-* helpers can install them via `panto lua -e ...`
   against the embedded luarocks.

## libpanto: `ToolSource`

Native tools keep the existing `Tool` API unchanged. Adapters that
back multiple tools through a shared runtime use the new `ToolSource`:

```zig
pub const ToolSource = struct {
    name: []const u8,             // diagnostic only ("panto-lua")
    tools: []const Tool.Decl,     // metadata only; no per-tool vtable
    ctx: *anyopaque,
    vtable: *const VTable,

    pub const VTable = struct {
        /// libpanto guarantees: for a given turn, all ToolUse calls
        /// whose tool name belongs to this source are delivered in
        /// one invoke_batch call, on one thread. Different sources
        /// still execute in parallel.
        ///
        /// The source decides internal scheduling (coroutines,
        /// sequential, internal worker pool).
        invoke_batch: *const fn (
            ctx: *anyopaque,
            calls: []const Call,
            results: []CallResult,    // parallel array, pre-allocated
            allocator: Allocator,
        ) anyerror!void,

        deinit: *const fn (ctx: *anyopaque, allocator: Allocator) void,
    };

    pub const Call = struct {
        tool_name: []const u8,
        input: []const u8,
    };

    pub const CallResult = union(enum) {
        ok: []u8,            // owned by allocator
        err: anyerror,
    };
};

pub const Tool.Decl = struct {
    name: []const u8,
    description: []const u8,
    schema_json: []const u8,
};
```

`ToolRegistry` indexes by tool name with a tagged value:
`{ .single: *Tool }` or `{ .source: *ToolSource, .tool_index: usize }`.

`Agent.registerToolSource(src: ToolSource) !void` is the new entry
point.

### Agent loop change

In `runStep`, after collecting ToolUse blocks for a turn:

1. Group them by owning source. Single-`Tool` entries form
   single-entry groups.
2. Spawn one OS thread per group.
3. Each thread calls either `tool.vtable.invoke` (single) or
   `source.vtable.invoke_batch` (batched).
4. Join. Assemble ToolResult blocks in the original order.

**Concurrency contract becomes:** different groups run in parallel; a
single group is the source's problem. The "thread-safe" promise still
holds for native `Tool`s. For Lua, it relaxes to "coroutine-safe
within the panto-lua runtime."

## Lua runtime

### One `lua_State`, many coroutines, one event loop

The panto CLI creates a single `lua_State` at startup. Every Lua
extension is loaded into it. All top-level extension code runs once.
Module-global state is real and persistent across calls.

When `invoke_batch` fires for the panto-lua source:

```
1. for each call: coroutine.create(handler), coroutine.resume(co, args)
2. uv.run() until all coroutines have completed (or errored)
3. collect results into the CallResult array
```

That's the entire scheduler. The work happens inside libuv: when a
coroutine calls a yield-aware libuv operation (HTTP, fs, subprocess,
sleep) it suspends; libuv resumes it when the event fires.

A wrapper layer translates libuv's callback shape into coroutine
yields. The Luvit project's `coro-*` modules (coro-fs, coro-net,
coro-http, coro-channel, coro-spawn) do this upstream for the common
operations. Where they don't cover something, we write small wrappers
in panto's own Lua code. The pattern is ~10 lines:

```lua
local function fs_open(path, flags, mode)
  local co = coroutine.running()
  uv.fs_open(path, flags, mode, function(err, fd)
    coroutine.resume(co, err, fd)
  end)
  return coroutine.yield()
end
```

### What this gets us

- True cooperative parallel I/O within a batch. Three concurrent
  `web_fetch` calls go through three concurrent sockets; total
  latency is `max(req1, req2, req3)`, not sum.
- First-class async subprocess. A `bash`-style tool that runs three
  commands at once does it without blocking the runtime.
- Module-global state for extensions that want it (rate limiters,
  caches, lazy connection pools, etc.).
- Extension top-level code runs once. Initialization is real.

### The honest caveat

Cooperative scheduling only helps when handlers yield. A handler that
calls a non-yielding C function — raw `os.execute`, `io.read` against
a slow file, an FFI call — blocks its siblings until it returns.
Document loudly. The escape hatch is "use native extensions for
work that can't yield."

This is the same trade Python's asyncio makes with `requests` vs
`aiohttp`. Panto's recommended posture: handlers should use libuv via
the coro-* wrappers (or other luv-aware libraries) for I/O. Pure
compute is fine. Calling `socket.http.request` or `os.execute` will
work but blocks the batch.

### Why libuv (not cqueues)

Considered cqueues + lua-http. Better HTTP story (HTTP/2),
coroutine-native API. Lost on **subprocess**, which has no
maintained cqueues binding and which matters a lot for a coding
agent. Also lost on familiarity — luv is what every Neovim user has
seen.

Trade accepted: HTTP only via luv's lower-level TCP + manual
framing (or a user-installed rock), and a callback-shaped async
surface rather than a coroutine-native one. Coroutine ergonomics
are an extension-author concern, not a runtime concern.

## luarocks as the package manager

### Distribution model

luarocks is embedded in the panto binary as `@embedFile`'d Lua
source. At startup the runtime:

1. Computes `$PANTO_HOME = $XDG_DATA_HOME/panto` (default
   `~/.local/share/panto`).
2. Configures the embedded `lua_State`'s `package.path` and
   `package.cpath` to look under `$PANTO_HOME/share/lua/5.4/` and
   `$PANTO_HOME/lib/lua/5.4/`.
3. Bootstraps the embedded luarocks against
   `--tree=$PANTO_HOME`.
4. Reconciles a "runtime batteries" manifest (currently just
   luv) — installs missing rocks, no-ops if present.
5. Iterates user extensions from config. For `luarocks:foo`-style
   references, ensures `foo` is installed.
6. Hands control to the agent loop.

luarocks 3.12+ no-ops on already-installed rocks and caches the
upstream manifest, so steps 4–5 are cheap on every run after the
first.

Network failures on later runs are swallowed: the rocks are already
there. First-run-with-no-network degrades to "no Lua extensions
work" — native panto features and the agent loop are unaffected.

### Why fully luarocks-based

- One mechanism for everything Lua-distribution. Runtime batteries
  and user extensions install the same way.
- Single small binary. No vendored libuv, no vendored luv, no
  `@embedFile`'d coro-* sources. luarocks itself is ~1MB of Lua
  source that compresses well.
- Version flexibility for batteries without re-shipping panto.
- Matches Neovim's rocks.nvim direction — the most relevant
  ecosystem signal for "Lua as a serious distribution target."
- User extension story is genuinely the same as the batteries
  story. No special cases.

### Distributable artifact

Single `panto` binary contains:

- Zig CLI + libpanto
- Lua 5.4 (already vendored)
- luarocks Lua source, embedded via `@embedFile`
- A small Zig-side bootstrap that configures `package.path` for the
  embedded luarocks code

Everything else lives under `$PANTO_HOME` and is installed on first
run.

## Migration shape

Independent chunks of work, roughly in order:

1. **`ToolSource` in libpanto.** ✅ **Done.** Type lives in
   `libpanto/src/tool_source.zig`; registry tagging in
   `tool_registry.zig`; per-group `std.Io.Group` fan-out in
   `agent.zig`. Native `Tool` API unchanged.
2. **Long-lived `lua_State` runtime in the CLI.** ✅ **Done.**
   `src/lua_runtime.zig` owns one `lua_State` for the process
   lifetime, loads all discovered extensions once, stores handlers
   as `luaL_ref` slots, and runs each call as a coroutine.
   Registers itself with libpanto as the `panto-lua` `ToolSource`.
   Step 5 added the libuv-driven scheduler that makes yields
   productive.
3. **Embed luarocks.** ✅ **Done.** `build.zig.zon` fetches
   luarocks 3.13.0 source. `build/gen_luarocks_embed.zig` enumerates
   the Lua sources at build time and produces a Zig module that
   `@embedFile`s each entry. `src/luarocks_runtime.zig` installs an
   embedded-source `package.searcher` so `require("luarocks.*")`
   resolves without disk I/O, injects `luarocks.core.hardcoded`,
   configures `package.path`/`cpath`, stages Lua headers under
   `<tree>/include/`, writes `<tree>/etc/luarocks/config-5.4.lua`,
   and materializes a `<tree>/bin/lua` shell wrapper that `exec`s
   `panto lua` (the interpreter luarocks shells out to for rockspec
   build scripts).
4. **Install luv as the first battery.** ✅ **Done.** Manifest
   pins `luv 1.52.1-0` (`src/manifest.zig`). Bootstrap reconciles
   in-process by `load`ing the embedded `src/bin/luarocks` driver
   and calling it with `install <name> <version>` args. luv's
   rockspec bundles libuv as a git submodule and builds it via
   CMake, so first-run toolchain requirement is `cc`, `make`, and
   `cmake`. Subsequent runs detect the version-stamped install
   metadata under `<tree>/lib/luarocks/rocks-5.4/<name>/<version>/`
   and no-op fast.
5. **Cooperative scheduler around `uv.run()`.** ✅ **Done.**
   `LuaRuntime.installScheduler` registers `panto._record_result`
   (a C function with the runtime pointer as upvalue), installs a
   wrapper closure that runs handlers under `pcall` and reports
   results, and caches `require("luv").run` in the registry. The
   new `runBatch` path creates one coroutine per call, resumes
   each once, then ticks `uv.run("once")` between resumes until
   every coroutine has terminated. Coroutines that yield without
   any pending libuv handle are surfaced as a deadlock error.

   We deliberately ship no `coro-*` helpers — luv is the entire
   async surface. Extension authors `require("luv")` and use its
   native callback-shaped APIs; `panto lua -e ...` (plus the
   embedded luarocks) is the escape hatch for anything else.
6. ~~User extension config syntax.~~ **Out of scope for this
   makeover.** Deferred; see Q7 "Future work".
7. **Delete `LuaTool` and per-call `lua_State` machinery.** ✅
   **Done** as part of step 2 — `LuaTool` and `LuaStatePool` are
   gone from the tree.

Additional work surfaced by Q1–Q7 that lands alongside steps 3–5:

- **`panto lua` subcommand.** ✅ **Done.** `build/panto_lua_repl.c`
   includes upstream `lua.c` with `#define static` stripped so we
   can call `pmain` directly. `src/subcommand.zig` opens a fresh
   `lua_State`, runs the same bootstrap pipeline as `panto run`,
   then hands the state to `panto_lua_pmain(L, argc, argv)` so
   upstream argv handling (`-i`, `-l`, `-e`, `script.lua args...`)
   works verbatim. (Q4)
- **`panto bootstrap` subcommand.** ✅ **Done.** Same bootstrap path
   as the agent and `panto lua`; just exits cleanly afterwards
   instead of entering an interactive surface. Idempotent: a clean
   tree on subsequent runs no-ops in milliseconds. (Q2)

   Supports `--force`: wipes the per-Lua-version tree
   (`<home>/rocks/lua-X.Y.Z/`) before running the regular bootstrap.
   Useful for recovering from a corrupted installation or testing
   the cold-start path. Sibling trees from other Lua versions are
   not touched.
- **Batteries manifest + reconcile.** ✅ **Done.** `src/manifest.zig`
   ships pinned versions; `reconcileBatteries` runs in-process by
   `load`ing the embedded luarocks driver, calling its `cmd.run_command`
   with `install <name> <version>` for any rock that isn't already
   on disk. (Q5)
- **Lua headers staged at `$PANTO_HOME/rocks/lua-X.Y.Z/include/`.**
   ✅ **Done.** `build/gen_lua_headers_embed.zig` embeds the headers
   from the Lua source tarball at build time; `stageLuaHeaders` in
   `luarocks_runtime.zig` writes them to disk on first run and
   when the contents differ (e.g. a panto upgrade that bumped
   the bundled Lua version). (Q3)

Documentation updates: phase-3.md gains a "superseded by
LUA_MAKEOVER.md for the Lua runtime; native extension contract
unchanged" note. The contract for native extensions ("thread-safe")
stays as-is.

## Open questions

These came up during design and need resolution before
implementation. We'll edit answers in here as decisions land.

### Q1: luarocks's own C dependencies

luarocks attempts to `require` several optional Lua modules via
`pcall` and falls back to shelling out or to its own pure-Lua
implementations when they're missing. The optional set:

- **LuaSocket** (`socket.http`, `socket.ftp`) — for HTTP/FTP
  downloads. Without it: shell out to a configured downloader
  (`curl` or `wget`).
- **LuaSec** (`ssl.https`) — for HTTPS. Without it *and* without
  the LuaSocket+luarocks-internal HTTPS path: must use `curl` or
  `wget` for HTTPS. **luarocks.org is HTTPS-only**, so this is
  effectively mandatory in some form.
- **LuaFileSystem** (`lfs`) — for directory operations,
  `chdir`, file attributes. Without it: degraded fallbacks using
  only `io.*` and `os.*`. Some operations become impossible.
- **lua-bz2** (`bz2`) — for `.bz2` archives. Almost never
  encountered; rocks ship as `.tar.gz` or `.zip`.
- **LuaPosix** (`posix`) — for chmod and other POSIX ops.
  Without it: fall back to shelling out to `chmod` etc.
- **md5** — for checksums. luarocks has a pure-Lua fallback.
- **`luarocks.tools.zip`** (bundled with luarocks itself) — pure
  Lua zip/gzip. No external dependency.
- **`luarocks.tools.tar`** (bundled) — pure Lua tar.

**Correction:** there is no `--with-lua=embedded` flag — that was a
hallucination from earlier in the design conversation. luarocks's
`./configure` accepts `--with-lua=DIR`, `--with-lua-include=DIR`,
`--with-lua-lib=DIR`, `--with-lua-interpreter=NAME`,
`--lua-version=VERSION`. These point luarocks at *a* Lua install
(which can be ours under `$PANTO_HOME`); they don't enable a
separate "embedded" mode.

**Decision:** minimize luarocks's optional Lua deps. Bootstrap runs
luarocks in its degraded-but-functional mode using its bundled
pure-Lua `tools.zip` and `tools.tar`, plus shell-out to `curl` (or
`wget`) for HTTPS downloads. If any of the optional deps turn out
to be effectively mandatory in practice, we statically link the
native C library and embed the Lua wrapper as `@embedFile` in
`panto` (sibling to Lua itself in `build.zig`). Likely candidates:
LuaFileSystem (small, pure C wrapper around POSIX, very widely
used), and possibly LuaSocket+LuaSec if shelling out to `curl`
proves too clunky.

We also depend on `curl` (or `wget`) being on PATH for downloads.
This is universal on Unix dev machines and we accept the
dependency. If it's missing, bootstrap surfaces a clear error.

### Q2: C toolchain on first run

luv has a C component. Building it requires `cc`, `make`, and Lua
headers. On dev machines (panto's target audience) these are
universal. On bare end-user machines they aren't.

**Decision:** add a `panto bootstrap` subcommand. It's effectively
a no-op `panto` invocation that exercises the same
fetch-and-install path that every normal startup runs, just
without entering the agent loop afterwards. On a clean machine
it's where the slow first-run download-and-compile happens with
visible output; on subsequent runs it's a fast no-op equivalent
to what every `panto` startup already does.

Every normal `panto` startup runs the same sync logic. The
`bootstrap` subcommand isn't a *separate* mechanism — it's a way
to run the sync explicitly without the agent loop, for users who
want to do setup ahead of time, or for CI/scripted installs.

When the toolchain is missing, surface a friendly error from
the bootstrap code path before luarocks itself barfs. "You need
a C compiler and make installed to compile Lua extensions" or
similar. Native panto features keep working regardless — only
the Lua tool runtime is gated on successful bootstrap.

### Q3: Lua headers for C rocks

Building C rocks against panto's embedded Lua requires the Lua
headers to be findable on disk.

**Decision:** drop the headers into `$PANTO_HOME/rocks/lua-X.Y.Z/include/`
at bootstrap time, where `X.Y.Z` is the Lua version panto is
built against. Embed the header sources via `@embedFile` (they're
already available via the `lua_src` build dep). Bootstrap writes
them out on first run and on any panto upgrade that changes the
Lua version.

**Why a versioned subdirectory:** rocks compiled against Lua
5.4.7's headers are not safe to load into a Lua 5.5 interpreter
(ABI changes happen across minor versions). The whole rock tree
lives under `$PANTO_HOME/rocks/lua-X.Y.Z/` and each Lua version
gets its own. A panto upgrade that bumps Lua creates a new tree
and reinstalls everything against it. The old tree is left in
place for rollback; users (or a future `panto gc` command) can
delete stale ones.

Directory layout:

```
$PANTO_HOME/
  rocks/
    lua-5.4.7/
      include/              ← Lua headers
      share/lua/5.4/        ← installed pure-Lua rocks
      lib/lua/5.4/          ← installed C rocks
      ...luarocks metadata...
    lua-5.5.0/              ← after a future upgrade
      ...
```

luarocks is invoked with `--tree=$PANTO_HOME/rocks/lua-5.4.7` and
configured (via its config file or CLI flags) to know that the
Lua headers live at `$PANTO_HOME/rocks/lua-5.4.7/include/`. The
tree contains everything needed for that Lua version including
the headers, which keeps rebuilds reproducible and rollback
clean.

### Q4: The `lua` interpreter that luarocks expects on PATH

luarocks uses an external `lua` binary for some operations (running
rockspec build scripts, primarily). It needs to be on PATH and
needs to be the same version as the embedded interpreter.

**Decision:** `panto lua` is a first-class user-visible subcommand
that wraps the **upstream `lua.c` standalone interpreter** with
panto's environment pre-configured.

Mechanically:

- Compile `lua.c` (the upstream standalone interpreter, ~600 lines)
  into the panto binary as a subcommand entry point. Currently
  excluded from `lua_files` in `build.zig`; include it for the
  `lua` subcommand path.
- `panto lua` arguments pass through to `lua.c`'s normal
  command-line handling (`-i`, `-l`, `-e`, `script.lua args...`,
  etc.). Full standalone-interpreter behavior, not a luarocks-only
  subset.
- Before handing control to `lua.c`'s main, panto's subcommand
  setup runs the same bootstrap as `panto run` (verify batteries
  installed, install missing rocks, configure `package.path` /
  `package.cpath` to find `$PANTO_HOME/rocks/lua-X.Y.Z/...`).
- Configure luarocks (via its config file written to
  `$PANTO_HOME/rocks/lua-X.Y.Z/config.lua`) to use
  `<absolute panto path> lua` as its Lua interpreter. luarocks's
  `--with-lua-interpreter=...` flag accepts an executable name;
  we either symlink or use the full argv mechanism.

This gives users a real `lua` they can use to test their
extensions in panto's environment — `require "luv"` and
`require "coro-http"` work, plus anything else they've installed
via `panto lua -e 'require("luarocks.cmd").run(...)'` or similar.

**Side benefit (Q7-related):** until we have a proper user-facing
`panto rocks install foo` command, `panto lua` is also the user's
escape hatch for installing extra rocks into `$PANTO_HOME`. We
can invoke luarocks itself through it.

### Q5: Reproducibility / lockfile

luarocks installs latest-matching by default. For a CLI tool we
want reproducible: the same panto version installs the same
battery versions on every fresh machine.

**Decision:** pin exact versions in a panto-internal manifest
shipped with the binary. No user-facing `panto lock` or
`panto sync` commands — sync is what every startup (and
`panto bootstrap`) does automatically.

A manifest file (likely `runtime-batteries.zon` or similar in the
panto source tree) lists exact versions:

```zig
.{
    .lua_version = "5.4.7",
    .luarocks_version = "3.12.2",
    .batteries = .{
        .luv = "1.51.0-1",
        .{ .@"coro-fs" = "3.0.4-1" },
        .{ .@"coro-http" = "3.2.1-1" },
        .{ .@"coro-net" = "3.2.1-1" },
        .{ .@"coro-channel" = "3.0.4-1" },
        .{ .@"coro-spawn" = "3.2.1-1" },
    },
}
```

Bumping any of these is a deliberate edit + commit + version bump
of panto itself. Each panto release pins one consistent set.

Bootstrap reads the embedded manifest and ensures the tree matches:
any rock not present at the pinned version gets installed; any
stale versions get removed. A panto upgrade that bumps Lua
creates an entirely new tree (per Q3) and installs everything
fresh against it.

### Q6: Where in the agent loop does the runtime live

**Decision:** CLI-side. libpanto continues to be native-only and
Lua-unaware.

The long-lived `lua_State` is constructed in the panto CLI's
`main` (or a module it calls) before the `Agent` is built.
Bootstrap runs first, then the runtime loads all discovered
Lua extensions into the state, then the runtime registers itself
with the `Agent` as a single `ToolSource` named `panto-lua`.
The source's `ctx` holds the runtime; the `lua_State` lives
inside it.

libpanto's only concept is `Tool` and `ToolSource`. It has no
idea that one of its sources happens to be Lua-backed.

### Q7: What "user extension" actually means in the new world

**Decision (for now):** keep the phase-3 directory-based discovery
as the only user extension mechanism. Local `.lua` files in
`~/.config/panto/extensions/` and `./.panto/extensions/`. No
config file, no `luarocks:foo` references yet.

Directory-discovered extensions get access to whatever's in
`$PANTO_HOME/rocks/lua-X.Y.Z/` — the runtime batteries (luv,
coro-*, future additions) and nothing else by default. They can
`require` any of those modules and pure-Lua code they write
themselves; that's the supported surface.

**Escape hatch for extra rocks:** users who need a third-party
Lua library (lua-cjson, lpeg, etc.) for their local extension
can install it manually via `panto lua` + the embedded luarocks:

```
panto lua -e 'require("luarocks.cmd").run("install", "lua-cjson")'
```

or more sugared if we feel like making that look better. The
rock lands in `$PANTO_HOME/rocks/lua-X.Y.Z/` and survives across
panto runs.

**Future work (not part of this makeover):** a proper config
file with `luarocks:panto-subagents`-style references, where
panto-published extensions can declare their own Lua library
dependencies in their rockspec and bootstrap installs the
whole graph automatically. The infrastructure built here
(luarocks-as-runtime, `$PANTO_HOME` tree, version pinning)
directly enables this — it's just an orthogonal piece of
user-facing surface that hasn't been designed yet.