Finish the hard parts of the lua makeover

- bundle luarocks source in the panto binary
- bootstrap process (intended for first `panto` run):
  - make ~/.local/share/panto/...
  - write out luarocks sources into it
  - run luarocks to install luv
- new `panto bootstrap` command just runs the bootstrap
- `panto bootstrap --force` removes everything and re-bootstraps
- new `panto lua` command just runs panto's embedded lua

1 files changed, 391 insertions, 0 deletions

diff --git a/docs/archive/phase-3.md b/docs/archive/phase-3.md
new file mode 100644
index 0000000..eb52ee9
--- /dev/null
+++ b/docs/archive/phase-3.md
@@ -0,0 +1,391 @@
+# Phase 3: Extension System
+
+> **Status:** Complete. The Lua runtime described in this doc has been **superseded by [`LUA_MAKEOVER.md`](../LUA_MAKEOVER.md)**. The current `panto` CLI uses a long-lived `lua_State` exposed to libpanto as a single `ToolSource`; the per-call `LuaStatePool` / `LuaTool` design described below no longer exists. The **native extension contract (`Tool`) is unchanged** — the rest of this document remains accurate for that path.
+
+## Goal
+
+Introduce a native-code tool extension API on `libpanto` and a Lua extension runtime in the `panto` CLI. Together these transform `pantograph` from a chat client into an agent that can act on the world. The extension system is the primary mechanism for adding capability — tools are extensions, not built-ins.
+
+This phase also marks the point where the `Agent` abstraction starts earning its keep. Until now, `agent.zig` has been a thin pass-through to the provider. In phase 3 the agent grows into the thing that owns the tool registry and drives the tool-call loop. Providers stay dumb (stream blocks); the agent is what makes a conversation iterative rather than one-shot.
+
+## Deliverable
+
+Two layered deliverables:
+
+1. **libpanto** — a Zig-native tool extension API. `Agent` exposes `registerTool(Tool)` and runs a tool-call loop in `runStep`. The agent loop detects ToolUse blocks in LLM responses, dispatches to registered tools, feeds ToolResult blocks back, and repeats until the model stops calling tools. libpanto has no awareness of Lua, Python, or any specific extension language.
+
+2. **panto CLI** — a Lua extension runtime that discovers Lua scripts on disk, loads them, and registers each declared tool with `libpanto` via a `LuaTool` adapter that satisfies libpanto's `Tool` interface.
+
+At the end of this phase, you can:
+
+- Embed `libpanto` in a Zig program, implement a `Tool` struct natively, and register it with the agent — no Lua involved.
+- Write a Lua extension that registers a tool, place it in `.agent/extensions/` (or `~/.config/panto/extensions/`), and have the `panto` CLI discover and load it.
+- Have a conversation where the LLM calls your tool and receives the result.
+- See tool calls execute in parallel when the LLM returns multiple ToolUse blocks.
+- See a meaningful error message when a Lua extension crashes, instead of a process abort.
+
+## What is usable at the end
+
+| Capability | How to exercise it |
+|---|---|
+| Native tool registration | In a Zig embedder, construct a `Tool` and call `agent.registerTool(tool)` |
+| Write a Lua tool extension | Create `.agent/extensions/mytool.lua` calling `panto.register_tool(...)` |
+| Discover Lua extensions | Place `.lua` files or directories in extension directories; `panto` CLI loads them on startup |
+| LLM calls a tool | Ask the LLM to use a registered tool; it emits a ToolUse block; the agent dispatches to the registered handler |
+| Tool result fed back | The tool's return bytes become a ToolResult block sent back to the LLM |
+| Parallel tool calls | LLM returns multiple ToolUse blocks; they execute concurrently |
+| Lua crash handling | A crashing Lua handler prints `the "mytool" extension crashed: <trace>` and aborts the turn |
+
+## What is explicitly out of scope
+
+- Core tools as extensions (phase 5 — `std.read`, `std.write`, etc.)
+- Conversation serialization / disk persistence (phase 4)
+- C ABI distribution of libpanto for external consumers (future)
+- GitHub or luarocks extension loaders (future — local filesystem only in phase 3)
+- Shared-object extensions (future)
+- Extension sandboxing beyond `xpcall` crash protection in the Lua adapter (future)
+- Config file for specifying which extensions to load (phase 6 — phase 3 loads everything it discovers)
+- Non-Lua extension runtimes (future — the libpanto API is general, but only the Lua adapter ships in phase 3)
+
+---
+
+## libpanto: Native Tool API
+
+### The `Tool` interface
+
+`libpanto` defines a `Tool` as an opaque-context value with a vtable. This is the boundary between the agent loop and any extension runtime.
+
+```zig
+pub const Tool = struct {
+    name: []const u8,           // borrowed; lifetime owned by the registrar
+    schema_json: []const u8,    // borrowed JSON Schema bytes; lifetime owned by the registrar
+    ctx: *anyopaque,
+    vtable: *const VTable,
+
+    pub const VTable = struct {
+        /// Invoke the tool. MUST be thread-safe — the agent may call `invoke`
+        /// concurrently from multiple threads when the LLM returns multiple
+        /// ToolUse blocks in a single response.
+        ///
+        /// `input` is the raw JSON bytes the provider sent. The tool is
+        /// responsible for parsing them if it cares about their structure.
+        ///
+        /// Returns owned bytes allocated with `allocator`. These bytes become
+        /// the `content` of the ToolResult block.
+        ///
+        /// Returning an error aborts the current turn. The agent will surface
+        /// the error to the user. Native tool implementations are responsible
+        /// for catching their own panics — a panic in `invoke` will crash the
+        /// process. Adapters that bridge to safer languages (Lua, Python, Go)
+        /// should convert panics/exceptions into errors here.
+        invoke: *const fn (ctx: *anyopaque, input: []const u8, allocator: std.mem.Allocator) anyerror![]u8,
+
+        /// Called when the tool is removed from the registry or the agent is
+        /// torn down. Frees any resources owned by `ctx`.
+        deinit: *const fn (ctx: *anyopaque, allocator: std.mem.Allocator) void,
+    };
+};
+```
+
+The contract:
+
+- **Thread-safety is mandatory.** Tool implementations promise that concurrent `invoke` calls are safe. This pushes the burden onto adapters (where it belongs — the Lua adapter solves it with a `lua_State` pool; a native Zig tool may just be re-entrant) and out of the agent loop.
+- **Input is opaque bytes.** libpanto never parses tool input. The tool decides whether to parse JSON, treat it as a string, or ignore it.
+- **Output is opaque bytes.** Same principle in reverse.
+- **Errors abort the turn.** No partial recovery in phase 3.
+- **Panics crash the process.** Native code that wants safety implements it itself. This is trivially obvious from the API surface: it's a function pointer with `anyerror!` return — there's no magic. Adapters for safer languages can wrap panics/exceptions and convert them to errors.
+
+### `Agent` changes
+
+```zig
+pub const Agent = struct {
+    provider: provider.Provider,
+    allocator: std.mem.Allocator,
+    registry: ToolRegistry,
+
+    pub fn init(allocator: Allocator, prov: provider.Provider) Agent { ... }
+    pub fn deinit(self: *Agent) void { ... }
+
+    pub fn registerTool(self: *Agent, tool: Tool) !void { ... }
+    pub fn unregisterTool(self: *Agent, name: []const u8) void { ... }
+
+    pub fn runStep(self: *Agent, conv: *Conversation, receiver: *Receiver) !void { ... }
+};
+```
+
+`registerTool` takes ownership of the `Tool` value (the agent calls `tool.vtable.deinit` on teardown or unregister). Tool name uniqueness is enforced — registering a name that already exists is an error.
+
+### `ToolRegistry`
+
+A small internal module:
+
+```
+ToolRegistry = struct {
+    tools: std.StringHashMap(Tool),
+    allocator: Allocator,
+
+    fn register(name, tool) !void
+    fn unregister(name) void
+    fn lookup(name) ?*const Tool
+    fn iterator() Iterator           // for serializing into provider requests
+    fn deinit() void                 // invokes each tool's vtable.deinit
+};
+```
+
+### Agent loop with tools
+
+`runStep` gains the tool-call loop:
+
+```
+runStep(conv, receiver):
+  loop:
+    provider.streamStep(conv, receiver)
+    inspect the assistant message just appended to conv for ToolUse blocks
+    if none: return — turn complete
+    for each ToolUse block, in parallel:
+      tool = registry.lookup(block.name) or error
+      result_bytes = tool.vtable.invoke(tool.ctx, block.input, allocator)
+      build a ToolResult block { tool_use_id = block.id, content = result_bytes }
+    append a user Message containing all ToolResult blocks to conv
+    continue loop
+```
+
+A single `runStep` may invoke the provider multiple times if the LLM chains tool calls.
+
+### Parallel execution
+
+When a response contains multiple ToolUse blocks, the agent dispatches them concurrently. Implementation likely uses a small thread pool or `std.Thread.spawn` per call (TBD during implementation). Because `Tool.invoke` is declared thread-safe, the agent loop has no further constraints — it just fans out and joins.
+
+### Tool serialization in provider requests
+
+When the agent calls the provider, the provider receives the registry (or an iterator over it) and emits the tools array. This means provider request serialization gains a tools-emitting branch when the registry is non-empty.
+
+**OpenAI**:
+```json
+{
+  "tools": [
+    { "type": "function",
+      "function": { "name": "echo", "parameters": <schema_json> } }
+  ]
+}
+```
+
+**Anthropic**:
+```json
+{
+  "tools": [
+    { "name": "echo", "input_schema": <schema_json> }
+  ]
+}
+```
+
+The `schema_json` bytes are emitted verbatim. Description fields are deferred — `Tool` does not yet carry a description (see open questions).
+
+### ToolUse decoding in responses
+
+Already handled by the existing Receiver callback sequence from phase 1/2:
+
+- OpenAI: `delta.tool_calls` → `onBlockStart(.ToolUse, ...)` with `meta.tool_id` and `meta.tool_name`, then `onContentDelta` with JSON argument fragments.
+- Anthropic: `content_block_start` with `type: "tool_use"` → `onBlockStart(.ToolUse, ...)`, then `content_block_delta` with `input_json_delta` fragments.
+
+The assembled ToolUseBlock contains `id`, `name`, and `input` (TextualBlock with the full JSON string).
+
+### ToolResult encoding in requests
+
+**OpenAI** — each ToolResult block becomes a separate top-level `tool` message:
+```json
+{ "role": "tool", "tool_call_id": "<id>", "content": "<result>" }
+```
+
+**Anthropic** — ToolResult blocks are content blocks on a user message:
+```json
+{ "role": "user", "content": [
+  { "type": "tool_result", "tool_use_id": "<id>", "content": "<result>" }
+] }
+```
+
+---
+
+## panto CLI: Lua Extension Runtime
+
+The Lua runtime lives entirely in the CLI. libpanto has no Lua dependency.
+
+### Extension discovery
+
+The CLI scans two directories in order:
+
+1. `.panto/extensions/` — project-local, relative to current working directory
+2. `~/.config/panto/extensions/` — user-level
+
+`.panto/` follows the established per-agent pattern (`.claude/`, `.opencode/`, `.pi/`).
+
+### Naming and structure
+
+- **Single-file**: `<name>.lua` → extension name is `<name>`
+- **Directory**: `<name>/init.lua` → extension name is `<name>`
+
+Hierarchical names via dot/directory mapping, mirroring `require("a.b.c")`:
+
+- `utils/json.lua` → `utils.json`
+- `coding/edit/init.lua` → `coding.edit`
+
+Sub-modules under a directory extension (e.g., `coding/edit/helpers.lua`) are the extension's internal business — the CLI only loads the top-level entry point.
+
+### Loading behavior
+
+- Recursively scan both directories.
+- Construct extension names from relative paths.
+- For each unique extension name, load its entry point into the `lua_State` pool (see below).
+- Project-local entries shadow user-level entries of the same name.
+- Extension top-level code runs at load time; it is expected to call `panto.register_tool(...)`.
+
+### The Lua bridge
+
+A small Zig module (`lua_bridge.zig`) registers Zig functions into a `lua_State`. The bridge exposes:
+
+#### `panto.register_tool(name, schema, handler)`
+
+- `name` (string) — tool name, e.g. `"bash"`
+- `schema` (table) — JSON Schema as a Lua table. The bridge serializes this to JSON bytes at registration time.
+- `handler` (function) — invoked when the LLM calls this tool. Receives a single argument: a Lua table parsed from the input JSON. Must return a string.
+
+```lua
+panto.register_tool("echo", {
+    type = "object",
+    properties = {
+        message = { type = "string", description = "The message to echo back" }
+    },
+    required = { "message" }
+}, function(input)
+    return input.message
+end)
+```
+
+The bridge stores the handler in the Lua registry (`luaL_ref`) and constructs a `LuaTool` (see below) that it then registers with the agent.
+
+### `LuaTool` — the adapter to libpanto
+
+```zig
+const LuaTool = struct {
+    pool: *LuaStatePool,
+    handler_ref: i32,         // registry ref to the Lua function
+    name_owned: []u8,
+    schema_owned: []u8,
+
+    // Implements Tool.VTable.invoke:
+    //   1. pool.acquire() → lua_State
+    //   2. lua_rawgeti(L, LUA_REGISTRYINDEX, handler_ref) — push handler
+    //   3. parse input bytes into a Lua table via std.json
+    //   4. xpcall the handler with the table
+    //   5. read returned string (or capture trace on error)
+    //   6. pool.release(L)
+    //   7. return result bytes or error
+};
+```
+
+The adapter is what makes the CLI side responsible for Lua-specific concerns:
+
+- **JSON → Lua table conversion**: happens here, not in libpanto.
+- **`xpcall` crash protection**: wraps the handler call, captures `debug.traceback`, returns an error so libpanto aborts the turn cleanly with `the "<tool_name>" extension crashed: <trace>`.
+- **Thread-safety**: provided by the `lua_State` pool. Each concurrent `invoke` checks out a separate state. libpanto's contract is satisfied.
+
+### `LuaStatePool`
+
+On-demand pool of `lua_State` instances:
+
+```
+LuaStatePool = struct {
+    states: std.ArrayList(*lua_State),
+    available: std.BitSet,
+    allocator: Allocator,
+    extension_paths: []const []const u8,    // all discovered entry points
+
+    fn acquire() *lua_State          // returns a free state, or creates a new one
+    fn release(L: *lua_State) void   // returns L to the pool
+    fn deinit() void                 // closes all states
+};
+```
+
+- States are created lazily — no pre-allocation.
+- Each new state loads all discovered extensions identically, so any state can handle any tool.
+- A state is checked out for the duration of a single `invoke` call.
+
+There's a coupling here: the handler `luaL_ref` is per-`lua_State`. The pool must guarantee that the same registry slot in every state points at the same handler. The cleanest way is: when an extension is first loaded into the prototype state, the bridge records the registry index. When a new state is constructed, the bridge re-loads all extensions in the same order, producing the same ref indices. This needs care during implementation but is mechanically straightforward.
+
+---
+
+## Module Changes
+
+### libpanto
+
+New:
+- `libpanto/src/tool.zig` — `Tool` and `VTable` definitions
+- `libpanto/src/tool_registry.zig` — `ToolRegistry`
+
+Modified:
+- `libpanto/src/agent.zig` — owns registry, gains `registerTool`, drives tool-call loop in `runStep`
+- `libpanto/src/provider.zig` — provider receives a way to see registered tools (likely a `*const ToolRegistry` on the agent passed via `streamStep` context, or a callback)
+- `libpanto/src/provider_openai_chat.zig` — request serialization emits `tools` array
+- `libpanto/src/provider_anthropic_messages.zig` — request serialization emits `tools` array
+- `libpanto/src/openai_chat_json.zig` — ToolResult message encoding
+- `libpanto/src/anthropic_messages_json.zig` — ToolResult content block encoding
+- `libpanto/src/root.zig` — re-exports `Tool`, `ToolRegistry`
+
+### panto CLI
+
+New:
+- `src/lua_bridge.zig` — Zig functions registered into Lua states; `panto.register_tool` implementation
+- `src/lua_tool.zig` — `LuaTool` adapter implementing libpanto's `Tool` interface
+- `src/lua_state_pool.zig` — on-demand `lua_State` pool
+- `src/extension_loader.zig` — directory scanning, name construction, loading
+
+Modified:
+- `src/main.zig` — wires extension discovery + loading into agent startup
+
+### External dependency
+
+Lua 5.4 linked into the `panto` binary. Zig's build system can fetch and compile Lua from source (~30KLOC C). No system dependency required. libpanto does not link Lua.
+
+---
+
+## Testing Strategy
+
+### Unit tests (libpanto)
+
+| What | How |
+|---|---|
+| Tool registration | Construct a native `Tool`, register with `Agent`, verify lookup |
+| Duplicate registration | Verify registering the same name twice returns an error |
+| Tool dispatch | Hand-craft a conversation with a ToolUse block, run `runStep` against a stub provider, verify the registered tool's `invoke` is called with the right bytes |
+| ToolResult round-trip | Verify the ToolResult message appended to the conversation has the right `tool_use_id` and `content` |
+| Parallel dispatch | Register two tools with sleep+timestamps, verify they ran concurrently (elapsed wall time < sum of individual times) |
+| Provider request serialization | Snapshot tests on OpenAI and Anthropic request bodies with tools registered |
+
+### Unit tests (panto CLI)
+
+| What | How |
+|---|---|
+| Extension discovery | Create temp directory with single-file and directory extensions, verify names constructed correctly |
+| Project shadows user | Same extension name in both directories — project wins |
+| Lua bridge input | Feed JSON bytes through `LuaTool.invoke`, verify the Lua handler sees the expected table |
+| Lua bridge output | Handler returns a string, verify it round-trips to bytes |
+| `xpcall` crash protection | Extension whose handler throws — verify `invoke` returns an error and includes the trace |
+| Concurrent `invoke` | Two threads call `invoke` on the same `LuaTool`, verify they don't share a `lua_State` |
+
+### Integration tests (manual)
+
+- `echo.lua` extension: ask the LLM to use it, verify result is fed back, LLM continues.
+- `crash.lua` extension: verify the crash is caught, printed, turn aborts gracefully.
+- Multi-tool response: ask the LLM to call two tools in one message, verify both execute concurrently.
+
+---
+
+## Resolved Decisions
+
+- **Project-local extension directory**: `.panto/extensions/`, matching `.claude/`, `.opencode/`, `.pi/`.
+- **Lua version**: Lua 5.4. LuaJIT is stuck at 5.1 semantics and the JIT speedup is largely irrelevant for tool handlers (which spend most of their time in Zig-side I/O). 5.4 keeps us aligned with current luarocks and Lua documentation.
+- **Tool description**: a top-level `description: []const u8` field on `Tool`. The Lua bridge gains it as a parameter: `panto.register_tool(name, description, schema, handler)`. Schema describes input only; description is the human-readable purpose.
+- **Provider sees the registry**: `provider.streamStep` gains a `*const ToolRegistry` parameter. Embedders normally pass the same registry on every call.
+- **Parallel dispatch**: `std.Thread.spawn` per ToolUse block. Thread creation is sub-millisecond on Linux/macOS; tool calls are orders of magnitude longer. Recycling buys nothing meaningful.
+- **Streaming tool results**: deferred, possibly indefinitely. Provider APIs only accept a final result payload, so streaming would be display-only — not worth the vtable complication in phase 3.
+- **Handler timeout / cancellation**: deferred. POSIX has no clean way to cancel a thread (`pthread_kill` with SIGKILL terminates the whole process; `pthread_cancel` is unusable in practice; signal-based interruption is not async-signal-safe for arbitrary code). Lua's `lua_sethook` only fires between VM steps, so it can't interrupt a handler blocked in a C call. A real cancellation primitive requires process isolation, which is a phase of its own — see the Punted list in [overview.md](overview.md). Phase 3 ships with the honest contract: **tool handlers run to completion. If a tool hangs, the user kills the `panto` process.**