path: root/docs/phase-3.md

# Phase 3: Extension System

## Goal

Introduce a Lua extension runtime and tool registration/execution, transforming `pantograph` from a chat client into an agent that can act on the world. The extension system is the primary mechanism for adding capability — tools are extensions, not built-ins.

## Deliverable

A working extension system where Lua scripts can register tools and handle tool-use requests. The agent loop detects ToolUse blocks in LLM responses, executes the corresponding tool handlers, and feeds ToolResult blocks back. At the end of this phase, you can:

- Write a Lua extension that registers a tool and handles invocations.
- Place it in `~/.config/panto/extensions/` (or `.panto/extensions/`) and have `pantograph` discover and load it.
- Have a conversation where the LLM calls your tool and receives the result.
- See tool calls execute in parallel when the LLM returns multiple ToolUse blocks.
- See a meaningful error message when an extension crashes, instead of a process abort.

## What is usable at the end

| Capability | How to exercise it |
|---|---|
| Write a tool extension | Create `~/.config/panto/extensions/mytool.lua` calling `panto.register_tool(...)` |
| Discover extensions | Place `.lua` files or directories in extension directories; `pantograph` loads them on startup |
| LLM calls a tool | Ask the LLM to use a registered tool; it emits a ToolUse block; pantograph executes the handler |
| Tool result fed back | The tool handler's return value becomes a ToolResult block sent back to the LLM |
| Parallel tool calls | LLM returns multiple ToolUse blocks; they execute concurrently |
| Extension crash handling | A crashing tool handler prints `the "mytool" extension crashed: <trace>` and aborts the turn |

## What is explicitly out of scope

- Core tools as extensions (phase 5 — `std.read`, `std.write`, etc.)
- Conversation serialization / disk persistence (phase 4)
- C ABI distribution of libpanto for external consumers (future)
- GitHub or luarocks extension loaders (future — local filesystem only in phase 3)
- Shared-object extensions (future)
- Extension sandboxing beyond `xpcall` crash protection (future)
- Config file for specifying which extensions to load (phase 6 — phase 3 loads everything it discovers)

---

## Extension Discovery

### Directory locations

`pantograph` scans two directories in order:

1. `.panto/extensions/` — project-local extensions (relative to current working directory)
2. `~/.config/panto/extensions/` — user-level extensions

### Naming and structure

Extensions are identified by name, derived from their path. Two formats:

- **Single-file**: `<name>.lua` → extension name is `<name>`
- **Directory**: `<name>/init.lua` → extension name is `<name>`

Names can be hierarchical using dots as separators, mapping to directory nesting:

- `utils/json.lua` → extension name is `utils.json`
- `coding/edit/init.lua` → extension name is `coding.edit`

This convention mirrors Lua's `require("a.b.c")` path resolution. Extension sub-modules (e.g., `coding/edit/helpers.lua`) are the extension's internal business — pantograph only loads the top-level entry point (`init.lua` or the single file).

### Loading behavior

- Scan both directories recursively.
- Construct extension names from relative paths using dot separators.
- Load each discovered entry point file into a fresh `lua_State`.
- After loading, the extension's top-level code runs, which should call `panto.register_tool(...)` to register its tools.
- Duplicate extension names: project-local takes precedence over user-level.

---

## Lua Bridge

The Lua bridge is a Zig module (`lua_bridge.zig`) that registers panto functions into the Lua state and handles translation between Zig types and Lua types. It is compiled into the `panto` binary — it is not a separate library.

### Functions exposed to Lua

#### `panto.register_tool(name, schema, handler)`

Registers a tool with the agent.

- `name` (string) — tool name, e.g. `"bash"`
- `schema` (table) — tool input schema as a Lua table (JSON Schema format). Serialized to JSON bytes for inclusion in provider requests.
- `handler` (function) — called when the LLM invokes this tool. Receives a single argument: a Lua table parsed from the tool input JSON. Must return a string (the tool result content).

Example:

```lua
panto.register_tool("echo", {
    type = "object",
    properties = {
        message = { type = "string", description = "The message to echo back" }
    },
    required = { "message" }
}, function(input)
    return input.message
end)
```

### Input parsing at the bridge boundary

Tool input arrives in libpanto as raw JSON bytes (stored in the ToolUseBlock's TextualBlock). At the Lua bridge boundary, libpanto parses these bytes into a Lua table using `std.json`, then passes the table to the handler. This is a convenience service for extension authors — internally, libpanto still treats tool input as opaque bytes. The round-trip guarantee: the JSON bytes the provider sent are faithfully represented in the Lua table.

### Output from handlers

The handler returns a string. This string becomes the `content` of a ToolResult block. It is stored as raw bytes; libpanto does not interpret or parse it.

---

## Tool Registration (Internal)

When `panto.register_tool()` is called from Lua, the bridge stores:

```
RegisteredTool = struct {
    name: []const u8,           // owned copy
    input_schema: []const u8,   // JSON bytes, owned copy (serialized from the Lua table)
    lua_handler_ref: i32,       // Lua registry reference to the handler function
};
```

A global tool registry maps tool names to `RegisteredTool` entries. The agent loop consults this registry when it encounters ToolUse blocks.

The `input_schema` bytes are included verbatim in provider requests when tools are listed. Both OpenAI and Anthropic accept JSON Schema objects for tool input definitions.

---

## Tool Execution

### Agent loop extension

The agent loop gains tool-call handling after each provider response:

```
runStep(conversation, receiver):
  1. Call provider.streamStep(conversation, receiver)
  2. Examine the completed message for ToolUse blocks
  3. If ToolUse blocks present:
     a. For each ToolUse block, look up the tool in the registry
     b. Execute all tool handlers (see parallel execution below)
     c. Collect ToolResult blocks
     d. Construct a user Message containing the ToolResult blocks
     e. Append to conversation
     f. Go to step 1 (call provider again with the updated conversation)
  4. If no ToolUse blocks: done — the turn is complete
```

A single `runStep` may invoke the provider multiple times if the LLM chains tool calls.

### Parallel execution

Multiple ToolUse blocks in a single response are executed concurrently. This is a documented part of the extension API: **tool handlers may be called concurrently in separate threads.** Extension authors must ensure their handlers are thread-safe.

Implementation: on-demand `lua_State` pool.

```
LuaStatePool = struct {
    states: std.ArrayList(*lua_State),
    available: std.BitSet,     // which states are free
    allocator: std.mem.Allocator,
    extension_dirs: []const []const u8,

    pub fn acquire(self) *lua_State    // returns an existing free state, or creates a new one
    pub fn release(self, *lua_State)   // returns state to the pool
    pub fn deinit(self) void           // destroys all states
};
```

- `acquire()`: if a free state exists, return it. Otherwise, create a fresh `lua_State`, load all discovered extensions into it (so the handler function references are valid), and return it.
- States are created lazily, not pre-allocated.
- Each state has all extensions loaded identically, so any state can handle any tool.
- When tool execution completes, the state is returned to the pool for reuse.

### Crash protection

Every tool handler invocation is wrapped in `xpcall` with a traceback handler:

```lua
xpcall(handler_fn, function(err)
    return debug.traceback(err)
end, input_table)
```

If the handler crashes:

- The error and stack trace are captured as a string.
- pantograph prints: `the "<tool_name>" extension crashed: <trace>`
- The current LLM turn is aborted — no ToolResult is generated for this tool call.
- Other concurrent tool calls in the same batch are not affected (they run in separate `lua_State` instances).

---

## Tool Serialization in Provider Requests

When tools are registered, the provider requests must include them. Both providers have a `tools` field in the request body.

### OpenAI

```
{
  "model": ...,
  "stream": true,
  "messages": [...],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "echo",
        "description": "...",        // not yet supported, phase 5+ when we add descriptions
        "parameters": { <input_schema> }
      }
    }
  ]
}
```

### Anthropic

```
{
  "model": ...,
  "system": ...,
  "stream": true,
  "messages": [...],
  "tools": [
    {
      "name": "echo",
      "description": "...",
      "input_schema": { <input_schema> }
    }
  ]
}
```

The `input_schema` bytes stored in the registry are emitted verbatim into the `parameters` (OpenAI) or `input_schema` (Anthropic) field.

### ToolUse in responses

Both providers return tool-call information in their streaming responses. This is already handled by the existing Receiver callback sequence:

- OpenAI: `delta.tool_calls` triggers `onBlockStart(.ToolUse, ...)` with `meta.tool_id` and `meta.tool_name`, then `onContentDelta` with JSON argument fragments.
- Anthropic: `content_block_start` with `type: "tool_use"` triggers `onBlockStart(.ToolUse, ...)`, then `content_block_delta` with `input_json_delta` fragments.

The assembled ToolUseBlock contains `id`, `name`, and `input` (TextualBlock with the full JSON string). The agent loop reads `name` to look up the registered tool, reads `input.content()` to get the JSON string, and passes it through the Lua bridge.

### ToolResult in requests

After tool execution, a user Message containing ToolResult blocks is appended to the conversation. Serialization differs by provider:

**OpenAI**: Each ToolResult block becomes a separate message:
```json
{ "role": "tool", "tool_call_id": "<tool_use_id>", "content": "<result string>" }
```

**Anthropic**: ToolResult blocks are content blocks on a user message:
```json
{ "role": "user", "content": [
  { "type": "tool_result", "tool_use_id": "...", "content": "..." }
] }
```

---

## Module Changes

### New files

```
src/lua_bridge.zig        // Zig functions registered into Lua state, type translation
src/tool_registry.zig     // RegisteredTool storage, lookup by name
src/lua_state_pool.zig    // On-demand pool of lua_State instances
src/extension_loader.zig  // Directory scanning, extension discovery and loading
```

### Modified files

- `agent.zig` — runStep gains the tool-call loop (detect ToolUse → execute → feed results → repeat)
- `provider_openai.zig` — request serialization includes `tools` array when tools are registered
- `provider_anthropic.zig` — request serialization includes `tools` array when tools are registered
- `json.zig` — serialization for ToolResult blocks (OpenAI and Anthropic formats)
- `config.zig` — extension directories added to config

### External dependency

Lua interpreter linked into the `panto` binary. Zig's build system can fetch and compile Lua from source (Lua is a small C codebase, ~30KLOC). No system dependency required.

---

## Testing Strategy

### Unit tests

| What | How |
|---|---|
| Extension discovery | Create temp directory with single-file and directory extensions, verify names constructed correctly |
| Tool registration | Load a Lua extension that registers a tool, verify entry appears in registry with correct name and schema |
| Lua bridge input parsing | Feed JSON strings through the bridge, verify correct Lua tables produced |
| Lua bridge output | Call a tool handler that returns a string, verify it becomes a ToolResult with correct content |
| Crash protection | Load an extension whose handler throws an error, verify xpcall catches it and returns trace |

### Integration test (manual)

- Write a simple `echo.lua` extension, place it in extension directory
- Start `panto`, ask the LLM to use the echo tool
- Verify the tool is called, result is fed back, LLM continues
- Write a `crash.lua` extension that throws an error
- Verify the crash is caught and printed with context, turn aborts gracefully
- Ask the LLM to use two tools in one response, verify both execute

---

## Open Questions (to resolve during implementation)

1. **Lua version**: Lua 5.4 is current. Luau (Roblox's fork) has performance improvements but diverges. Stick with standard Lua 5.4 for compatibility with luarocks and existing ecosystem?
2. **Handler timeout**: Should tool handlers have a timeout? A hung tool call blocks the agent loop. Could add a configurable timeout with abort.
3. **Streaming tool results**: Some tools (e.g., `bash` running a long command) produce output incrementally. Phase 3 handlers return a single string. Streaming results would require a different handler interface — possibly a callback the handler calls to emit partial output. Defer to a later phase?
4. **Tool description field**: The `panto.register_tool()` call in phase 3 includes a schema but no explicit description string. Provider APIs require a description. Options: add a `description` parameter, or extract it from the schema. Probably simplest to add it as a parameter.