# Lua Runtime Makeover

Side project, separate from the main phase plan. Reworks panto's Lua
embedding into something that fits Lua's actual concurrency model and
opens the door to a luarocks-based extension ecosystem.

Phase 3 as shipped is fine and stays. This document describes what
replaces it.

## Why

The current Lua embedding has three problems, in increasing order of
how badly they constrain us:

1. **One `lua_State` per tool call.** Every `LuaTool.invoke` builds and
   tears down a fresh interpreter. Module-global state is impossible.
   Top-level extension code runs over and over. The API only makes
   sense for stateless single-shot handlers, which is not how Lua
   wants to be used.
2. **The `Tool` contract is "thread-safe."** Right for native
   extensions; wrong for Lua. Lua's concurrency primitive is the
   coroutine, not the OS thread. A single `lua_State` is not safe for
   concurrent host entry, so the only way to honor "thread-safe" with
   Lua is one state per call — which we have, badly.
3. **No path to a real Lua ecosystem.** The current setup discovers
   `.lua` files on disk and that's it. There's no answer to "how do
   extension authors depend on lua-cjson" or "how does an HTTP-using
   tool work." The eventual answer to both is luarocks, and the
   current architecture has no place for it.

## Shape of the fix

Three independent pieces that compose:

1. **`ToolSource` in libpanto.** A new kind of registration alongside
   `Tool`. A source owns one or more tools and receives all calls
   targeting them, as a batch, on one thread per turn. Different
   sources still run in parallel. Lua becomes one source.
2. **Long-lived `lua_State` with cooperative scheduling.** The panto
   CLI maintains exactly one `lua_State` for its entire lifetime.
   Extension top-level code runs once at startup. Each tool call is a
   coroutine. A libuv event loop drives suspended coroutines.
3. **luarocks as the package manager.** Both panto's own runtime
   batteries (luv, coro-*, future additions) and user-installed
   extensions come through luarocks, installed into a tree under
   `$XDG_DATA_HOME/panto/`. luarocks itself is embedded in the panto
   binary as Lua source.

## libpanto: `ToolSource`

Native tools keep the existing `Tool` API unchanged. Adapters that
back multiple tools through a shared runtime use the new `ToolSource`:

```zig
pub const ToolSource = struct {
    name: []const u8,             // diagnostic only ("panto-lua")
    tools: []const Tool.Decl,     // metadata only; no per-tool vtable
    ctx: *anyopaque,
    vtable: *const VTable,

    pub const VTable = struct {
        /// libpanto guarantees: for a given turn, all ToolUse calls
        /// whose tool name belongs to this source are delivered in
        /// one invoke_batch call, on one thread. Different sources
        /// still execute in parallel.
        ///
        /// The source decides internal scheduling (coroutines,
        /// sequential, internal worker pool).
        invoke_batch: *const fn (
            ctx: *anyopaque,
            calls: []const Call,
            results: []CallResult,    // parallel array, pre-allocated
            allocator: Allocator,
        ) anyerror!void,

        deinit: *const fn (ctx: *anyopaque, allocator: Allocator) void,
    };

    pub const Call = struct {
        tool_name: []const u8,
        input: []const u8,
    };

    pub const CallResult = union(enum) {
        ok: []u8,            // owned by allocator
        err: anyerror,
    };
};

pub const Tool.Decl = struct {
    name: []const u8,
    description: []const u8,
    schema_json: []const u8,
};
```

`ToolRegistry` indexes by tool name with a tagged value:
`{ .single: *Tool }` or `{ .source: *ToolSource, .tool_index: usize }`.

`Agent.registerToolSource(src: ToolSource) !void` is the new entry
point.

### Agent loop change

In `runStep`, after collecting ToolUse blocks for a turn:

1. Group them by owning source. Single-`Tool` entries form
   single-entry groups.
2. Spawn one OS thread per group.
3. Each thread calls either `tool.vtable.invoke` (single) or
   `source.vtable.invoke_batch` (batched).
4. Join. Assemble ToolResult blocks in the original order.

**Concurrency contract becomes:** different groups run in parallel; a
single group is the source's problem. The "thread-safe" promise still
holds for native `Tool`s. For Lua, it relaxes to "coroutine-safe
within the panto-lua runtime."

## Lua runtime

### One `lua_State`, many coroutines, one event loop

The panto CLI creates a single `lua_State` at startup. Every Lua
extension is loaded into it. All top-level extension code runs once.
Module-global state is real and persistent across calls.

When `invoke_batch` fires for the panto-lua source:

```
1. for each call: coroutine.create(handler), coroutine.resume(co, args)
2. uv.run() until all coroutines have completed (or errored)
3. collect results into the CallResult array
```

That's the entire scheduler. The work happens inside libuv: when a
coroutine calls a yield-aware libuv operation (HTTP, fs, subprocess,
sleep) it suspends; libuv resumes it when the event fires.

A wrapper layer translates libuv's callback shape into coroutine
yields. The Luvit project's `coro-*` modules (coro-fs, coro-net,
coro-http, coro-channel, coro-spawn) do this upstream for the common
operations. Where they don't cover something, we write small wrappers
in panto's own Lua code. The pattern is ~10 lines:

```lua
local function fs_open(path, flags, mode)
  local co = coroutine.running()
  uv.fs_open(path, flags, mode, function(err, fd)
    coroutine.resume(co, err, fd)
  end)
  return coroutine.yield()
end
```

### What this gets us

- True cooperative parallel I/O within a batch. Three concurrent
  `web_fetch` calls go through three concurrent sockets; total
  latency is `max(req1, req2, req3)`, not sum.
- First-class async subprocess. A `bash`-style tool that runs three
  commands at once does it without blocking the runtime.
- Module-global state for extensions that want it (rate limiters,
  caches, lazy connection pools, etc.).
- Extension top-level code runs once. Initialization is real.

### The honest caveat

Cooperative scheduling only helps when handlers yield. A handler that
calls a non-yielding C function — raw `os.execute`, `io.read` against
a slow file, an FFI call — blocks its siblings until it returns.
Document loudly. The escape hatch is "use native extensions for
work that can't yield."

This is the same trade Python's asyncio makes with `requests` vs
`aiohttp`. Panto's recommended posture: handlers should use libuv via
the coro-* wrappers (or other luv-aware libraries) for I/O. Pure
compute is fine. Calling `socket.http.request` or `os.execute` will
work but blocks the batch.

### Why libuv (not cqueues)

Considered cqueues + lua-http. Better HTTP story (HTTP/2),
coroutine-native API. Lost on **subprocess**, which has no
maintained cqueues binding and which matters a lot for a coding
agent. Also lost on familiarity — luv is what every Neovim user has
seen.

Trade accepted: HTTP/1.1 only via coro-http, plus a small wrapper
layer panto maintains for the libuv operations the coro-* set
doesn't cover.

## luarocks as the package manager

### Distribution model

luarocks is embedded in the panto binary as `@embedFile`'d Lua
source. At startup the runtime:

1. Computes `$PANTO_HOME = $XDG_DATA_HOME/panto` (default
   `~/.local/share/panto`).
2. Configures the embedded `lua_State`'s `package.path` and
   `package.cpath` to look under `$PANTO_HOME/share/lua/5.4/` and
   `$PANTO_HOME/lib/lua/5.4/`.
3. Bootstraps the embedded luarocks against
   `--tree=$PANTO_HOME`.
4. Reconciles a "runtime batteries" manifest (luv, coro-fs,
   coro-http, coro-net, coro-channel, coro-spawn, plus any future
   additions) — installs missing rocks, no-ops if present.
5. Iterates user extensions from config. For `luarocks:foo`-style
   references, ensures `foo` is installed.
6. Hands control to the agent loop.

luarocks 3.12+ no-ops on already-installed rocks and caches the
upstream manifest, so steps 4–5 are cheap on every run after the
first.

Network failures on later runs are swallowed: the rocks are already
there. First-run-with-no-network degrades to "no Lua extensions
work" — native panto features and the agent loop are unaffected.

### Why fully luarocks-based

- One mechanism for everything Lua-distribution. Runtime batteries
  and user extensions install the same way.
- Single small binary. No vendored libuv, no vendored luv, no
  `@embedFile`'d coro-* sources. luarocks itself is ~1MB of Lua
  source that compresses well.
- Version flexibility for batteries without re-shipping panto.
- Matches Neovim's rocks.nvim direction — the most relevant
  ecosystem signal for "Lua as a serious distribution target."
- User extension story is genuinely the same as the batteries
  story. No special cases.

### Distributable artifact

Single `panto` binary contains:

- Zig CLI + libpanto
- Lua 5.4 (already vendored)
- luarocks Lua source, embedded via `@embedFile`
- A small Zig-side bootstrap that configures `package.path` for the
  embedded luarocks code

Everything else lives under `$PANTO_HOME` and is installed on first
run.

## Migration shape

Independent chunks of work, roughly in order:

1. **`ToolSource` in libpanto.** Add the type, registry tagging, and
   the per-source-thread fan-out in `runStep`. `Tool` unchanged.
   Native extensions keep working.
2. **Long-lived `lua_State` runtime in the CLI.** New module
   (`lua_runtime.zig`). Loads all discovered Lua extensions once.
   Registers itself as one `ToolSource` named `panto-lua`.
   `invoke_batch` runs each call in a coroutine and drives
   `uv.run()`. No batteries yet — handlers run sync.
3. **Embed luarocks.** `build.zig.zon` fetches luarocks source.
   `build.zig` embeds it via `@embedFile`. Runtime bootstraps it at
   startup against `$PANTO_HOME`.
4. **Install luv as the first battery.** Verify the cooperative
   scheduler actually works end-to-end with a real yield-aware
   library.
5. **Install coro-* batteries.** Wire them as the default
   I/O surface for extension authors.
6. **User extension config syntax.** `luarocks:foo` references in
   panto's config file; lockfile equivalent for reproducibility.
   Mostly orthogonal to everything above.
7. **Delete `LuaTool` and per-call `lua_State` machinery.** Phase 3
   code retires.

Documentation updates: phase-3.md gains a "superseded by
LUA_MAKEOVER.md for the Lua runtime; native extension contract
unchanged" note. The contract for native extensions ("thread-safe")
stays as-is.

## Open questions

These came up during design and need resolution before
implementation. We'll edit answers in here as decisions land.

### Q1: luarocks's own C dependencies

luarocks attempts to `require` several optional Lua modules via
`pcall` and falls back to shelling out or to its own pure-Lua
implementations when they're missing. The optional set:

- **LuaSocket** (`socket.http`, `socket.ftp`) — for HTTP/FTP
  downloads. Without it: shell out to a configured downloader
  (`curl` or `wget`).
- **LuaSec** (`ssl.https`) — for HTTPS. Without it *and* without
  the LuaSocket+luarocks-internal HTTPS path: must use `curl` or
  `wget` for HTTPS. **luarocks.org is HTTPS-only**, so this is
  effectively mandatory in some form.
- **LuaFileSystem** (`lfs`) — for directory operations,
  `chdir`, file attributes. Without it: degraded fallbacks using
  only `io.*` and `os.*`. Some operations become impossible.
- **lua-bz2** (`bz2`) — for `.bz2` archives. Almost never
  encountered; rocks ship as `.tar.gz` or `.zip`.
- **LuaPosix** (`posix`) — for chmod and other POSIX ops.
  Without it: fall back to shelling out to `chmod` etc.
- **md5** — for checksums. luarocks has a pure-Lua fallback.
- **`luarocks.tools.zip`** (bundled with luarocks itself) — pure
  Lua zip/gzip. No external dependency.
- **`luarocks.tools.tar`** (bundled) — pure Lua tar.

**Correction:** there is no `--with-lua=embedded` flag — that was a
hallucination from earlier in the design conversation. luarocks's
`./configure` accepts `--with-lua=DIR`, `--with-lua-include=DIR`,
`--with-lua-lib=DIR`, `--with-lua-interpreter=NAME`,
`--lua-version=VERSION`. These point luarocks at *a* Lua install
(which can be ours under `$PANTO_HOME`); they don't enable a
separate "embedded" mode.

**Decision:** minimize luarocks's optional Lua deps. Bootstrap runs
luarocks in its degraded-but-functional mode using its bundled
pure-Lua `tools.zip` and `tools.tar`, plus shell-out to `curl` (or
`wget`) for HTTPS downloads. If any of the optional deps turn out
to be effectively mandatory in practice, we statically link the
native C library and embed the Lua wrapper as `@embedFile` in
`panto` (sibling to Lua itself in `build.zig`). Likely candidates:
LuaFileSystem (small, pure C wrapper around POSIX, very widely
used), and possibly LuaSocket+LuaSec if shelling out to `curl`
proves too clunky.

We also depend on `curl` (or `wget`) being on PATH for downloads.
This is universal on Unix dev machines and we accept the
dependency. If it's missing, bootstrap surfaces a clear error.

### Q2: C toolchain on first run

luv has a C component. Building it requires `cc`, `make`, and Lua
headers. On dev machines (panto's target audience) these are
universal. On bare end-user machines they aren't.

**Decision:** add a `panto bootstrap` subcommand. It's effectively
a no-op `panto` invocation that exercises the same
fetch-and-install path that every normal startup runs, just
without entering the agent loop afterwards. On a clean machine
it's where the slow first-run download-and-compile happens with
visible output; on subsequent runs it's a fast no-op equivalent
to what every `panto` startup already does.

Every normal `panto` startup runs the same sync logic. The
`bootstrap` subcommand isn't a *separate* mechanism — it's a way
to run the sync explicitly without the agent loop, for users who
want to do setup ahead of time, or for CI/scripted installs.

When the toolchain is missing, surface a friendly error from
the bootstrap code path before luarocks itself barfs. "You need
a C compiler and make installed to compile Lua extensions" or
similar. Native panto features keep working regardless — only
the Lua tool runtime is gated on successful bootstrap.

### Q3: Lua headers for C rocks

Building C rocks against panto's embedded Lua requires the Lua
headers to be findable on disk.

**Decision:** drop the headers into `$PANTO_HOME/rocks/lua-X.Y.Z/include/`
at bootstrap time, where `X.Y.Z` is the Lua version panto is
built against. Embed the header sources via `@embedFile` (they're
already available via the `lua_src` build dep). Bootstrap writes
them out on first run and on any panto upgrade that changes the
Lua version.

**Why a versioned subdirectory:** rocks compiled against Lua
5.4.7's headers are not safe to load into a Lua 5.5 interpreter
(ABI changes happen across minor versions). The whole rock tree
lives under `$PANTO_HOME/rocks/lua-X.Y.Z/` and each Lua version
gets its own. A panto upgrade that bumps Lua creates a new tree
and reinstalls everything against it. The old tree is left in
place for rollback; users (or a future `panto gc` command) can
delete stale ones.

Directory layout:

```
$PANTO_HOME/
  rocks/
    lua-5.4.7/
      include/              ← Lua headers
      share/lua/5.4/        ← installed pure-Lua rocks
      lib/lua/5.4/          ← installed C rocks
      ...luarocks metadata...
    lua-5.5.0/              ← after a future upgrade
      ...
```

luarocks is invoked with `--tree=$PANTO_HOME/rocks/lua-5.4.7` and
configured (via its config file or CLI flags) to know that the
Lua headers live at `$PANTO_HOME/rocks/lua-5.4.7/include/`. The
tree contains everything needed for that Lua version including
the headers, which keeps rebuilds reproducible and rollback
clean.

### Q4: The `lua` interpreter that luarocks expects on PATH

luarocks uses an external `lua` binary for some operations (running
rockspec build scripts, primarily). It needs to be on PATH and
needs to be the same version as the embedded interpreter.

**Decision:** `panto lua` is a first-class user-visible subcommand
that wraps the **upstream `lua.c` standalone interpreter** with
panto's environment pre-configured.

Mechanically:

- Compile `lua.c` (the upstream standalone interpreter, ~600 lines)
  into the panto binary as a subcommand entry point. Currently
  excluded from `lua_files` in `build.zig`; include it for the
  `lua` subcommand path.
- `panto lua` arguments pass through to `lua.c`'s normal
  command-line handling (`-i`, `-l`, `-e`, `script.lua args...`,
  etc.). Full standalone-interpreter behavior, not a luarocks-only
  subset.
- Before handing control to `lua.c`'s main, panto's subcommand
  setup runs the same bootstrap as `panto run` (verify batteries
  installed, install missing rocks, configure `package.path` /
  `package.cpath` to find `$PANTO_HOME/rocks/lua-X.Y.Z/...`).
- Configure luarocks (via its config file written to
  `$PANTO_HOME/rocks/lua-X.Y.Z/config.lua`) to use
  `<absolute panto path> lua` as its Lua interpreter. luarocks's
  `--with-lua-interpreter=...` flag accepts an executable name;
  we either symlink or use the full argv mechanism.

This gives users a real `lua` they can use to test their
extensions in panto's environment — `require "luv"` and
`require "coro-http"` work, plus anything else they've installed
via `panto lua -e 'require("luarocks.cmd").run(...)'` or similar.

**Side benefit (Q7-related):** until we have a proper user-facing
`panto rocks install foo` command, `panto lua` is also the user's
escape hatch for installing extra rocks into `$PANTO_HOME`. We
can invoke luarocks itself through it.

### Q5: Reproducibility / lockfile

luarocks installs latest-matching by default. For a CLI tool we
want reproducible: the same panto version installs the same
battery versions on every fresh machine.

**Decision:** pin exact versions in a panto-internal manifest
shipped with the binary. No user-facing `panto lock` or
`panto sync` commands — sync is what every startup (and
`panto bootstrap`) does automatically.

A manifest file (likely `runtime-batteries.zon` or similar in the
panto source tree) lists exact versions:

```zig
.{
    .lua_version = "5.4.7",
    .luarocks_version = "3.12.2",
    .batteries = .{
        .luv = "1.51.0-1",
        .{ .@"coro-fs" = "3.0.4-1" },
        .{ .@"coro-http" = "3.2.1-1" },
        .{ .@"coro-net" = "3.2.1-1" },
        .{ .@"coro-channel" = "3.0.4-1" },
        .{ .@"coro-spawn" = "3.2.1-1" },
    },
}
```

Bumping any of these is a deliberate edit + commit + version bump
of panto itself. Each panto release pins one consistent set.

Bootstrap reads the embedded manifest and ensures the tree matches:
any rock not present at the pinned version gets installed; any
stale versions get removed. A panto upgrade that bumps Lua
creates an entirely new tree (per Q3) and installs everything
fresh against it.

### Q6: Where in the agent loop does the runtime live

**Decision:** CLI-side. libpanto continues to be native-only and
Lua-unaware.

The long-lived `lua_State` is constructed in the panto CLI's
`main` (or a module it calls) before the `Agent` is built.
Bootstrap runs first, then the runtime loads all discovered
Lua extensions into the state, then the runtime registers itself
with the `Agent` as a single `ToolSource` named `panto-lua`.
The source's `ctx` holds the runtime; the `lua_State` lives
inside it.

libpanto's only concept is `Tool` and `ToolSource`. It has no
idea that one of its sources happens to be Lua-backed.

### Q7: What "user extension" actually means in the new world

**Decision (for now):** keep the phase-3 directory-based discovery
as the only user extension mechanism. Local `.lua` files in
`~/.config/panto/extensions/` and `./.panto/extensions/`. No
config file, no `luarocks:foo` references yet.

Directory-discovered extensions get access to whatever's in
`$PANTO_HOME/rocks/lua-X.Y.Z/` — the runtime batteries (luv,
coro-*, future additions) and nothing else by default. They can
`require` any of those modules and pure-Lua code they write
themselves; that's the supported surface.

**Escape hatch for extra rocks:** users who need a third-party
Lua library (lua-cjson, lpeg, etc.) for their local extension
can install it manually via `panto lua` + the embedded luarocks:

```
panto lua -e 'require("luarocks.cmd").run("install", "lua-cjson")'
```

or more sugared if we feel like making that look better. The
rock lands in `$PANTO_HOME/rocks/lua-X.Y.Z/` and survives across
panto runs.

**Future work (not part of this makeover):** a proper config
file with `luarocks:panto-subagents`-style references, where
panto-published extensions can declare their own Lua library
dependencies in their rockspec and bootstrap installs the
whole graph automatically. The infrastructure built here
(luarocks-as-runtime, `$PANTO_HOME` tree, version pinning)
directly enables this — it's just an orthogonal piece of
user-facing surface that hasn't been designed yet.