diff options
Diffstat (limited to 'docs/archive')
| -rw-r--r-- | docs/archive/LUA_MAKEOVER.md | 594 | ||||
| -rw-r--r-- | docs/archive/ideas.md | 38 | ||||
| -rw-r--r-- | docs/archive/phase-1.md | 449 | ||||
| -rw-r--r-- | docs/archive/phase-2.md | 157 | ||||
| -rw-r--r-- | docs/archive/phase-3.md | 391 |
5 files changed, 1629 insertions, 0 deletions
diff --git a/docs/archive/LUA_MAKEOVER.md b/docs/archive/LUA_MAKEOVER.md new file mode 100644 index 0000000..f701a29 --- /dev/null +++ b/docs/archive/LUA_MAKEOVER.md @@ -0,0 +1,594 @@ +# Lua Runtime Makeover + +Side project, separate from the main phase plan. Reworks panto's Lua +embedding into something that fits Lua's actual concurrency model and +opens the door to a luarocks-based extension ecosystem. + +Phase 3 as shipped is fine and stays. This document describes what +replaces it. + +## Why + +The current Lua embedding has three problems, in increasing order of +how badly they constrain us: + +1. **One `lua_State` per tool call.** Every `LuaTool.invoke` builds and + tears down a fresh interpreter. Module-global state is impossible. + Top-level extension code runs over and over. The API only makes + sense for stateless single-shot handlers, which is not how Lua + wants to be used. +2. **The `Tool` contract is "thread-safe."** Right for native + extensions; wrong for Lua. Lua's concurrency primitive is the + coroutine, not the OS thread. A single `lua_State` is not safe for + concurrent host entry, so the only way to honor "thread-safe" with + Lua is one state per call — which we have, badly. +3. **No path to a real Lua ecosystem.** The current setup discovers + `.lua` files on disk and that's it. There's no answer to "how do + extension authors depend on lua-cjson" or "how does an HTTP-using + tool work." The eventual answer to both is luarocks, and the + current architecture has no place for it. + +## Shape of the fix + +Three independent pieces that compose: + +1. **`ToolSource` in libpanto.** A new kind of registration alongside + `Tool`. A source owns one or more tools and receives all calls + targeting them, as a batch, on one thread per turn. Different + sources still run in parallel. Lua becomes one source. +2. **Long-lived `lua_State` with cooperative scheduling.** The panto + CLI maintains exactly one `lua_State` for its entire lifetime. + Extension top-level code runs once at startup. Each tool call is a + coroutine. A libuv event loop drives suspended coroutines. +3. **luarocks as the package manager.** Both panto's own runtime + battery (luv — see note below) and user-installed extensions + come through luarocks, installed into a tree under + `$XDG_DATA_HOME/panto/`. luarocks itself is embedded in the panto + binary as Lua source. + + Originally planned to ship `coro-*` (coro-fs, coro-http, coro-net, + coro-channel, coro-spawn) as additional batteries; that plan was + dropped during implementation. luv alone provides the libuv + surface (callback-shaped, but coroutine-friendly with a `co = + coroutine.running(); ...; coroutine.yield()` pattern); shipping + the smallest possible set keeps the makeover focused and lets us + write panto's own `std` tools directly against luv. Users who + want coro-* helpers can install them via `panto lua -e ...` + against the embedded luarocks. + +## libpanto: `ToolSource` + +Native tools keep the existing `Tool` API unchanged. Adapters that +back multiple tools through a shared runtime use the new `ToolSource`: + +```zig +pub const ToolSource = struct { + name: []const u8, // diagnostic only ("panto-lua") + tools: []const Tool.Decl, // metadata only; no per-tool vtable + ctx: *anyopaque, + vtable: *const VTable, + + pub const VTable = struct { + /// libpanto guarantees: for a given turn, all ToolUse calls + /// whose tool name belongs to this source are delivered in + /// one invoke_batch call, on one thread. Different sources + /// still execute in parallel. + /// + /// The source decides internal scheduling (coroutines, + /// sequential, internal worker pool). + invoke_batch: *const fn ( + ctx: *anyopaque, + calls: []const Call, + results: []CallResult, // parallel array, pre-allocated + allocator: Allocator, + ) anyerror!void, + + deinit: *const fn (ctx: *anyopaque, allocator: Allocator) void, + }; + + pub const Call = struct { + tool_name: []const u8, + input: []const u8, + }; + + pub const CallResult = union(enum) { + ok: []u8, // owned by allocator + err: anyerror, + }; +}; + +pub const Tool.Decl = struct { + name: []const u8, + description: []const u8, + schema_json: []const u8, +}; +``` + +`ToolRegistry` indexes by tool name with a tagged value: +`{ .single: *Tool }` or `{ .source: *ToolSource, .tool_index: usize }`. + +`Agent.registerToolSource(src: ToolSource) !void` is the new entry +point. + +### Agent loop change + +In `runStep`, after collecting ToolUse blocks for a turn: + +1. Group them by owning source. Single-`Tool` entries form + single-entry groups. +2. Spawn one OS thread per group. +3. Each thread calls either `tool.vtable.invoke` (single) or + `source.vtable.invoke_batch` (batched). +4. Join. Assemble ToolResult blocks in the original order. + +**Concurrency contract becomes:** different groups run in parallel; a +single group is the source's problem. The "thread-safe" promise still +holds for native `Tool`s. For Lua, it relaxes to "coroutine-safe +within the panto-lua runtime." + +## Lua runtime + +### One `lua_State`, many coroutines, one event loop + +The panto CLI creates a single `lua_State` at startup. Every Lua +extension is loaded into it. All top-level extension code runs once. +Module-global state is real and persistent across calls. + +When `invoke_batch` fires for the panto-lua source: + +``` +1. for each call: coroutine.create(handler), coroutine.resume(co, args) +2. uv.run() until all coroutines have completed (or errored) +3. collect results into the CallResult array +``` + +That's the entire scheduler. The work happens inside libuv: when a +coroutine calls a yield-aware libuv operation (HTTP, fs, subprocess, +sleep) it suspends; libuv resumes it when the event fires. + +A wrapper layer translates libuv's callback shape into coroutine +yields. The Luvit project's `coro-*` modules (coro-fs, coro-net, +coro-http, coro-channel, coro-spawn) do this upstream for the common +operations. Where they don't cover something, we write small wrappers +in panto's own Lua code. The pattern is ~10 lines: + +```lua +local function fs_open(path, flags, mode) + local co = coroutine.running() + uv.fs_open(path, flags, mode, function(err, fd) + coroutine.resume(co, err, fd) + end) + return coroutine.yield() +end +``` + +### What this gets us + +- True cooperative parallel I/O within a batch. Three concurrent + `web_fetch` calls go through three concurrent sockets; total + latency is `max(req1, req2, req3)`, not sum. +- First-class async subprocess. A `bash`-style tool that runs three + commands at once does it without blocking the runtime. +- Module-global state for extensions that want it (rate limiters, + caches, lazy connection pools, etc.). +- Extension top-level code runs once. Initialization is real. + +### The honest caveat + +Cooperative scheduling only helps when handlers yield. A handler that +calls a non-yielding C function — raw `os.execute`, `io.read` against +a slow file, an FFI call — blocks its siblings until it returns. +Document loudly. The escape hatch is "use native extensions for +work that can't yield." + +This is the same trade Python's asyncio makes with `requests` vs +`aiohttp`. Panto's recommended posture: handlers should use libuv via +the coro-* wrappers (or other luv-aware libraries) for I/O. Pure +compute is fine. Calling `socket.http.request` or `os.execute` will +work but blocks the batch. + +### Why libuv (not cqueues) + +Considered cqueues + lua-http. Better HTTP story (HTTP/2), +coroutine-native API. Lost on **subprocess**, which has no +maintained cqueues binding and which matters a lot for a coding +agent. Also lost on familiarity — luv is what every Neovim user has +seen. + +Trade accepted: HTTP only via luv's lower-level TCP + manual +framing (or a user-installed rock), and a callback-shaped async +surface rather than a coroutine-native one. Coroutine ergonomics +are an extension-author concern, not a runtime concern. + +## luarocks as the package manager + +### Distribution model + +luarocks is embedded in the panto binary as `@embedFile`'d Lua +source. At startup the runtime: + +1. Computes `$PANTO_HOME = $XDG_DATA_HOME/panto` (default + `~/.local/share/panto`). +2. Configures the embedded `lua_State`'s `package.path` and + `package.cpath` to look under `$PANTO_HOME/share/lua/5.4/` and + `$PANTO_HOME/lib/lua/5.4/`. +3. Bootstraps the embedded luarocks against + `--tree=$PANTO_HOME`. +4. Reconciles a "runtime batteries" manifest (currently just + luv) — installs missing rocks, no-ops if present. +5. Iterates user extensions from config. For `luarocks:foo`-style + references, ensures `foo` is installed. +6. Hands control to the agent loop. + +luarocks 3.12+ no-ops on already-installed rocks and caches the +upstream manifest, so steps 4–5 are cheap on every run after the +first. + +Network failures on later runs are swallowed: the rocks are already +there. First-run-with-no-network degrades to "no Lua extensions +work" — native panto features and the agent loop are unaffected. + +### Why fully luarocks-based + +- One mechanism for everything Lua-distribution. Runtime batteries + and user extensions install the same way. +- Single small binary. No vendored libuv, no vendored luv, no + `@embedFile`'d coro-* sources. luarocks itself is ~1MB of Lua + source that compresses well. +- Version flexibility for batteries without re-shipping panto. +- Matches Neovim's rocks.nvim direction — the most relevant + ecosystem signal for "Lua as a serious distribution target." +- User extension story is genuinely the same as the batteries + story. No special cases. + +### Distributable artifact + +Single `panto` binary contains: + +- Zig CLI + libpanto +- Lua 5.4 (already vendored) +- luarocks Lua source, embedded via `@embedFile` +- A small Zig-side bootstrap that configures `package.path` for the + embedded luarocks code + +Everything else lives under `$PANTO_HOME` and is installed on first +run. + +## Migration shape + +Independent chunks of work, roughly in order: + +1. **`ToolSource` in libpanto.** ✅ **Done.** Type lives in + `libpanto/src/tool_source.zig`; registry tagging in + `tool_registry.zig`; per-group `std.Io.Group` fan-out in + `agent.zig`. Native `Tool` API unchanged. +2. **Long-lived `lua_State` runtime in the CLI.** ✅ **Done.** + `src/lua_runtime.zig` owns one `lua_State` for the process + lifetime, loads all discovered extensions once, stores handlers + as `luaL_ref` slots, and runs each call as a coroutine. + Registers itself with libpanto as the `panto-lua` `ToolSource`. + Step 5 added the libuv-driven scheduler that makes yields + productive. +3. **Embed luarocks.** ✅ **Done.** `build.zig.zon` fetches + luarocks 3.13.0 source. `build/gen_luarocks_embed.zig` enumerates + the Lua sources at build time and produces a Zig module that + `@embedFile`s each entry. `src/luarocks_runtime.zig` installs an + embedded-source `package.searcher` so `require("luarocks.*")` + resolves without disk I/O, injects `luarocks.core.hardcoded`, + configures `package.path`/`cpath`, stages Lua headers under + `<tree>/include/`, writes `<tree>/etc/luarocks/config-5.4.lua`, + and materializes a `<tree>/bin/lua` shell wrapper that `exec`s + `panto lua` (the interpreter luarocks shells out to for rockspec + build scripts). +4. **Install luv as the first battery.** ✅ **Done.** Manifest + pins `luv 1.52.1-0` (`src/manifest.zig`). Bootstrap reconciles + in-process by `load`ing the embedded `src/bin/luarocks` driver + and calling it with `install <name> <version>` args. luv's + rockspec bundles libuv as a git submodule and builds it via + CMake, so first-run toolchain requirement is `cc`, `make`, and + `cmake`. Subsequent runs detect the version-stamped install + metadata under `<tree>/lib/luarocks/rocks-5.4/<name>/<version>/` + and no-op fast. +5. **Cooperative scheduler around `uv.run()`.** ✅ **Done.** + `LuaRuntime.installScheduler` registers `panto._record_result` + (a C function with the runtime pointer as upvalue), installs a + wrapper closure that runs handlers under `pcall` and reports + results, and caches `require("luv").run` in the registry. The + new `runBatch` path creates one coroutine per call, resumes + each once, then ticks `uv.run("once")` between resumes until + every coroutine has terminated. Coroutines that yield without + any pending libuv handle are surfaced as a deadlock error. + + We deliberately ship no `coro-*` helpers — luv is the entire + async surface. Extension authors `require("luv")` and use its + native callback-shaped APIs; `panto lua -e ...` (plus the + embedded luarocks) is the escape hatch for anything else. +6. ~~User extension config syntax.~~ **Out of scope for this + makeover.** Deferred; see Q7 "Future work". +7. **Delete `LuaTool` and per-call `lua_State` machinery.** ✅ + **Done** as part of step 2 — `LuaTool` and `LuaStatePool` are + gone from the tree. + +Additional work surfaced by Q1–Q7 that lands alongside steps 3–5: + +- **`panto lua` subcommand.** ✅ **Done.** `build/panto_lua_repl.c` + includes upstream `lua.c` with `#define static` stripped so we + can call `pmain` directly. `src/subcommand.zig` opens a fresh + `lua_State`, runs the same bootstrap pipeline as `panto run`, + then hands the state to `panto_lua_pmain(L, argc, argv)` so + upstream argv handling (`-i`, `-l`, `-e`, `script.lua args...`) + works verbatim. (Q4) +- **`panto bootstrap` subcommand.** ✅ **Done.** Same bootstrap path + as the agent and `panto lua`; just exits cleanly afterwards + instead of entering an interactive surface. Idempotent: a clean + tree on subsequent runs no-ops in milliseconds. (Q2) + + Supports `--force`: wipes the per-Lua-version tree + (`<home>/rocks/lua-X.Y.Z/`) before running the regular bootstrap. + Useful for recovering from a corrupted installation or testing + the cold-start path. Sibling trees from other Lua versions are + not touched. +- **Batteries manifest + reconcile.** ✅ **Done.** `src/manifest.zig` + ships pinned versions; `reconcileBatteries` runs in-process by + `load`ing the embedded luarocks driver, calling its `cmd.run_command` + with `install <name> <version>` for any rock that isn't already + on disk. (Q5) +- **Lua headers staged at `$PANTO_HOME/rocks/lua-X.Y.Z/include/`.** + ✅ **Done.** `build/gen_lua_headers_embed.zig` embeds the headers + from the Lua source tarball at build time; `stageLuaHeaders` in + `luarocks_runtime.zig` writes them to disk on first run and + when the contents differ (e.g. a panto upgrade that bumped + the bundled Lua version). (Q3) + +Documentation updates: phase-3.md gains a "superseded by +LUA_MAKEOVER.md for the Lua runtime; native extension contract +unchanged" note. The contract for native extensions ("thread-safe") +stays as-is. + +## Open questions + +These came up during design and need resolution before +implementation. We'll edit answers in here as decisions land. + +### Q1: luarocks's own C dependencies + +luarocks attempts to `require` several optional Lua modules via +`pcall` and falls back to shelling out or to its own pure-Lua +implementations when they're missing. The optional set: + +- **LuaSocket** (`socket.http`, `socket.ftp`) — for HTTP/FTP + downloads. Without it: shell out to a configured downloader + (`curl` or `wget`). +- **LuaSec** (`ssl.https`) — for HTTPS. Without it *and* without + the LuaSocket+luarocks-internal HTTPS path: must use `curl` or + `wget` for HTTPS. **luarocks.org is HTTPS-only**, so this is + effectively mandatory in some form. +- **LuaFileSystem** (`lfs`) — for directory operations, + `chdir`, file attributes. Without it: degraded fallbacks using + only `io.*` and `os.*`. Some operations become impossible. +- **lua-bz2** (`bz2`) — for `.bz2` archives. Almost never + encountered; rocks ship as `.tar.gz` or `.zip`. +- **LuaPosix** (`posix`) — for chmod and other POSIX ops. + Without it: fall back to shelling out to `chmod` etc. +- **md5** — for checksums. luarocks has a pure-Lua fallback. +- **`luarocks.tools.zip`** (bundled with luarocks itself) — pure + Lua zip/gzip. No external dependency. +- **`luarocks.tools.tar`** (bundled) — pure Lua tar. + +**Correction:** there is no `--with-lua=embedded` flag — that was a +hallucination from earlier in the design conversation. luarocks's +`./configure` accepts `--with-lua=DIR`, `--with-lua-include=DIR`, +`--with-lua-lib=DIR`, `--with-lua-interpreter=NAME`, +`--lua-version=VERSION`. These point luarocks at *a* Lua install +(which can be ours under `$PANTO_HOME`); they don't enable a +separate "embedded" mode. + +**Decision:** minimize luarocks's optional Lua deps. Bootstrap runs +luarocks in its degraded-but-functional mode using its bundled +pure-Lua `tools.zip` and `tools.tar`, plus shell-out to `curl` (or +`wget`) for HTTPS downloads. If any of the optional deps turn out +to be effectively mandatory in practice, we statically link the +native C library and embed the Lua wrapper as `@embedFile` in +`panto` (sibling to Lua itself in `build.zig`). Likely candidates: +LuaFileSystem (small, pure C wrapper around POSIX, very widely +used), and possibly LuaSocket+LuaSec if shelling out to `curl` +proves too clunky. + +We also depend on `curl` (or `wget`) being on PATH for downloads. +This is universal on Unix dev machines and we accept the +dependency. If it's missing, bootstrap surfaces a clear error. + +### Q2: C toolchain on first run + +luv has a C component. Building it requires `cc`, `make`, and Lua +headers. On dev machines (panto's target audience) these are +universal. On bare end-user machines they aren't. + +**Decision:** add a `panto bootstrap` subcommand. It's effectively +a no-op `panto` invocation that exercises the same +fetch-and-install path that every normal startup runs, just +without entering the agent loop afterwards. On a clean machine +it's where the slow first-run download-and-compile happens with +visible output; on subsequent runs it's a fast no-op equivalent +to what every `panto` startup already does. + +Every normal `panto` startup runs the same sync logic. The +`bootstrap` subcommand isn't a *separate* mechanism — it's a way +to run the sync explicitly without the agent loop, for users who +want to do setup ahead of time, or for CI/scripted installs. + +When the toolchain is missing, surface a friendly error from +the bootstrap code path before luarocks itself barfs. "You need +a C compiler and make installed to compile Lua extensions" or +similar. Native panto features keep working regardless — only +the Lua tool runtime is gated on successful bootstrap. + +### Q3: Lua headers for C rocks + +Building C rocks against panto's embedded Lua requires the Lua +headers to be findable on disk. + +**Decision:** drop the headers into `$PANTO_HOME/rocks/lua-X.Y.Z/include/` +at bootstrap time, where `X.Y.Z` is the Lua version panto is +built against. Embed the header sources via `@embedFile` (they're +already available via the `lua_src` build dep). Bootstrap writes +them out on first run and on any panto upgrade that changes the +Lua version. + +**Why a versioned subdirectory:** rocks compiled against Lua +5.4.7's headers are not safe to load into a Lua 5.5 interpreter +(ABI changes happen across minor versions). The whole rock tree +lives under `$PANTO_HOME/rocks/lua-X.Y.Z/` and each Lua version +gets its own. A panto upgrade that bumps Lua creates a new tree +and reinstalls everything against it. The old tree is left in +place for rollback; users (or a future `panto gc` command) can +delete stale ones. + +Directory layout: + +``` +$PANTO_HOME/ + rocks/ + lua-5.4.7/ + include/ ← Lua headers + share/lua/5.4/ ← installed pure-Lua rocks + lib/lua/5.4/ ← installed C rocks + ...luarocks metadata... + lua-5.5.0/ ← after a future upgrade + ... +``` + +luarocks is invoked with `--tree=$PANTO_HOME/rocks/lua-5.4.7` and +configured (via its config file or CLI flags) to know that the +Lua headers live at `$PANTO_HOME/rocks/lua-5.4.7/include/`. The +tree contains everything needed for that Lua version including +the headers, which keeps rebuilds reproducible and rollback +clean. + +### Q4: The `lua` interpreter that luarocks expects on PATH + +luarocks uses an external `lua` binary for some operations (running +rockspec build scripts, primarily). It needs to be on PATH and +needs to be the same version as the embedded interpreter. + +**Decision:** `panto lua` is a first-class user-visible subcommand +that wraps the **upstream `lua.c` standalone interpreter** with +panto's environment pre-configured. + +Mechanically: + +- Compile `lua.c` (the upstream standalone interpreter, ~600 lines) + into the panto binary as a subcommand entry point. Currently + excluded from `lua_files` in `build.zig`; include it for the + `lua` subcommand path. +- `panto lua` arguments pass through to `lua.c`'s normal + command-line handling (`-i`, `-l`, `-e`, `script.lua args...`, + etc.). Full standalone-interpreter behavior, not a luarocks-only + subset. +- Before handing control to `lua.c`'s main, panto's subcommand + setup runs the same bootstrap as `panto run` (verify batteries + installed, install missing rocks, configure `package.path` / + `package.cpath` to find `$PANTO_HOME/rocks/lua-X.Y.Z/...`). +- Configure luarocks (via its config file written to + `$PANTO_HOME/rocks/lua-X.Y.Z/config.lua`) to use + `<absolute panto path> lua` as its Lua interpreter. luarocks's + `--with-lua-interpreter=...` flag accepts an executable name; + we either symlink or use the full argv mechanism. + +This gives users a real `lua` they can use to test their +extensions in panto's environment — `require "luv"` and +`require "coro-http"` work, plus anything else they've installed +via `panto lua -e 'require("luarocks.cmd").run(...)'` or similar. + +**Side benefit (Q7-related):** until we have a proper user-facing +`panto rocks install foo` command, `panto lua` is also the user's +escape hatch for installing extra rocks into `$PANTO_HOME`. We +can invoke luarocks itself through it. + +### Q5: Reproducibility / lockfile + +luarocks installs latest-matching by default. For a CLI tool we +want reproducible: the same panto version installs the same +battery versions on every fresh machine. + +**Decision:** pin exact versions in a panto-internal manifest +shipped with the binary. No user-facing `panto lock` or +`panto sync` commands — sync is what every startup (and +`panto bootstrap`) does automatically. + +A manifest file (likely `runtime-batteries.zon` or similar in the +panto source tree) lists exact versions: + +```zig +.{ + .lua_version = "5.4.7", + .luarocks_version = "3.12.2", + .batteries = .{ + .luv = "1.51.0-1", + .{ .@"coro-fs" = "3.0.4-1" }, + .{ .@"coro-http" = "3.2.1-1" }, + .{ .@"coro-net" = "3.2.1-1" }, + .{ .@"coro-channel" = "3.0.4-1" }, + .{ .@"coro-spawn" = "3.2.1-1" }, + }, +} +``` + +Bumping any of these is a deliberate edit + commit + version bump +of panto itself. Each panto release pins one consistent set. + +Bootstrap reads the embedded manifest and ensures the tree matches: +any rock not present at the pinned version gets installed; any +stale versions get removed. A panto upgrade that bumps Lua +creates an entirely new tree (per Q3) and installs everything +fresh against it. + +### Q6: Where in the agent loop does the runtime live + +**Decision:** CLI-side. libpanto continues to be native-only and +Lua-unaware. + +The long-lived `lua_State` is constructed in the panto CLI's +`main` (or a module it calls) before the `Agent` is built. +Bootstrap runs first, then the runtime loads all discovered +Lua extensions into the state, then the runtime registers itself +with the `Agent` as a single `ToolSource` named `panto-lua`. +The source's `ctx` holds the runtime; the `lua_State` lives +inside it. + +libpanto's only concept is `Tool` and `ToolSource`. It has no +idea that one of its sources happens to be Lua-backed. + +### Q7: What "user extension" actually means in the new world + +**Decision (for now):** keep the phase-3 directory-based discovery +as the only user extension mechanism. Local `.lua` files in +`~/.config/panto/extensions/` and `./.panto/extensions/`. No +config file, no `luarocks:foo` references yet. + +Directory-discovered extensions get access to whatever's in +`$PANTO_HOME/rocks/lua-X.Y.Z/` — the runtime batteries (luv, +coro-*, future additions) and nothing else by default. They can +`require` any of those modules and pure-Lua code they write +themselves; that's the supported surface. + +**Escape hatch for extra rocks:** users who need a third-party +Lua library (lua-cjson, lpeg, etc.) for their local extension +can install it manually via `panto lua` + the embedded luarocks: + +``` +panto lua -e 'require("luarocks.cmd").run("install", "lua-cjson")' +``` + +or more sugared if we feel like making that look better. The +rock lands in `$PANTO_HOME/rocks/lua-X.Y.Z/` and survives across +panto runs. + +**Future work (not part of this makeover):** a proper config +file with `luarocks:panto-subagents`-style references, where +panto-published extensions can declare their own Lua library +dependencies in their rockspec and bootstrap installs the +whole graph automatically. The infrastructure built here +(luarocks-as-runtime, `$PANTO_HOME` tree, version pinning) +directly enables this — it's just an orthogonal piece of +user-facing surface that hasn't been designed yet. diff --git a/docs/archive/ideas.md b/docs/archive/ideas.md new file mode 100644 index 0000000..e83d543 --- /dev/null +++ b/docs/archive/ideas.md @@ -0,0 +1,38 @@ +# pantograph Ideas + +`pantograph` is a minimal coding agent inspired by `pi`, with a focus on performance, efficiency, correctness, and a small core that can be extended deliberately. + +## Core architecture + +- The core agent loop will be written in Zig for performance, efficiency, and a small runtime footprint. +- In addition to the CLI/application, the project should expose a `libpanto` library through a C ABI so other programs can embed or build on the agent functionality. + +## Extension system + +- Extensions will initially be written in Lua, chosen for speed, simplicity, and low overhead. +- Because poorly written extensions can easily destabilize the host, `pantograph` should explore ways to improve extension correctness and crash protection. +- Future extension support should include loading shared object libraries through a C ABI, allowing extensions to be written in Zig, Rust, C, C++, or other native languages. + +## Minimal built-in feature set + +- `pantograph` should follow `pi`'s philosophy of avoiding unnecessary built-ins such as native subagents, MCP support, and permission systems. +- `pantograph` will go further by not building in AGENTS.md automation, skills, or customizable `/prompts`. +- Those features should be possible to implement as extensions rather than being part of the core runtime. + +## Provider API support + +- Provider support should be careful and conservative. +- The core should support Anthropic-shaped and OpenAI-shaped APIs with arbitrary base URLs. +- The goal is to avoid the common failure mode where provider integrations exist but only partially work, break important agent features, or crash the process. + +## Server/proxy mode + +- `pantograph` should support running as a server that exposes OpenAI-compatible and Anthropic-compatible APIs. +- In this mode, `pantograph` can act as a lightweight provider router/proxy to its configured backends. +- This is intended to provide the useful parts of tools like `omniroute` while avoiding excessive memory usage, fragile integrations, and runtime instability. + +## Core tools as extensions + +- Basic tools such as `read`, `write`, `edit`, and `bash` should be supplied by extensions rather than hardcoded into the core. +- These tools can be included in the standard distribution but should be individually disableable. +- Once shared object extension support exists, the standard core tools should be ported to Zig/native extensions. diff --git a/docs/archive/phase-1.md b/docs/archive/phase-1.md new file mode 100644 index 0000000..6b149aa --- /dev/null +++ b/docs/archive/phase-1.md @@ -0,0 +1,449 @@ +# Phase 1: libpanto — Minimal Chat Library + +**Status: complete.** Streaming chat works end-to-end against OpenAI-compatible APIs; conversation history persists across turns; thinking blocks (`reasoning_content` / `reasoning`) are streamed; `reasoning_effort` is configurable. Open questions resolved: thinking support implemented; mid-stream errors propagate via Zig errors; connections are one-per-turn intentionally (uniform reentry into each turn); long-conversation memory deferred to a later phase. Session persistence is phase 4. + +## Goal + +A Zig library that can hold a streaming conversation with an LLM via an OpenAI-compatible API. No tools, no extensions — just chat. Includes a minimal CLI for live testing. + +## Deliverable + +A `libpanto` Zig module importable by other Zig code, plus a `panto` binary that wires it into a basic read/print loop. At the end of this phase, you can: + +- Start `panto`, type a message, and receive a streamed response from an OpenAI-compatible LLM. +- Send follow-up messages that include full conversation history. +- See thinking tokens and text tokens stream in as they arrive. +- Have the complete conversation available in memory for the duration of the session. + +## What is usable at the end + +| Capability | How to exercise it | +|---|---| +| Open a conversation | `libpanto.conversation.Conversation.init(allocator)` | +| Add a user message | `conversation.addUserMessage("hello")` | +| Run an agent step (streaming) | `agent.runStep(conversation, &receiver)` | +| See streamed output | CLI prints thinking/text chunks as they arrive | +| Conversation persists across turns | Follow-up messages include prior history | + +## What is explicitly out of scope + +- Tools and tool-use (phase 3+) +- Extensions and extension API (phase 3+) +- C ABI (phase 3+, when needed for Lua extensions) +- Anthropic provider (phase 2) +- Disk persistence / session save (later phase) +- Server/proxy mode (future, undefined phase) +- System prompt construction framework (later phase — the model supports system messages, but no opinionated assembly system yet) + +--- + +## Data Model + +### TextualBlock (shared streaming buffer) + +``` +TextualBlock = struct { + buf: std.ArrayList(u8), + allocator: std.mem.Allocator, + + pub fn init(allocator: std.mem.Allocator) TextualBlock + pub fn content(self: *const TextualBlock) []const u8 // self.buf.items + pub fn append(self: *TextualBlock, delta: []const u8) !void + pub fn deinit(self: *TextualBlock) void // self.buf.deinit() +} +``` + +`Text` and `Thinking` blocks both use `TextualBlock` as their payload. They share the same append-based streaming behavior — deltas arrive incrementally and are appended via an internal `ArrayList(u8)`, giving amortized O(1) appends and avoiding the O(n²) re-copying that would result from storing `[]const u8` slices. + +The `TextualBlock` stores its own allocator reference so that `deinit()` needs no external context. Each `TextualBlock` has an `init()`/`deinit()` pair. This also means a `ContentBlock` can clean itself up without the caller providing an allocator. + +### ContentBlock (tagged union) + +``` +ContentBlock = union(enum) { + Text: TextualBlock, + Thinking: TextualBlock, + ToolUse: ToolUseBlock, // phase 3+ + ToolResult: ToolResultBlock, // phase 3+ + + pub fn deinit(self: *ContentBlock) void { + switch (self.*) { + .Text, .Thinking => |*b| b.deinit(), + .ToolUse => |b| { /* free id, name, input */ }, + .ToolResult => |b| { /* free tool_use_id, content */ }, + } + } +} + +ToolUseBlock = struct { + id: []const u8, // owned copy + name: []const u8, // owned copy + input: []const u8, // raw JSON bytes, owned copy +} + +ToolResultBlock = struct { + tool_use_id: []const u8, // owned copy + content: []const u8, // owned copy +} +``` + +`ToolUse` and `ToolResult` are defined in the model now but not populated or processed until the extensions phase. This avoids a model refactor later — the types exist, we just never encounter them in phase 1. + +The `input` field of `ToolUse` is stored as raw JSON bytes (`[]const u8`) rather than a parsed structure. We are not in the business of understanding tool input schemas; we pass them through. + +`ToolUse` blocks also stream incrementally — both providers send tool input as JSON fragments across multiple deltas. Therefore `ToolUse.input` also uses `TextualBlock` for assembly. `id` and `name` arrive at block-start time and are stored as owned `[]const u8` copies. + +`ToolResult` blocks are constructed by `pantograph` itself (not streamed from a provider), so `content` could be a simple `[]const u8`. However, for consistency and to allow progressive construction of results, it also uses `TextualBlock`. + +Updated types: +``` +ToolUseBlock = struct { + id: []const u8, // owned copy, from onBlockStart metadata + name: []const u8, // owned copy, from onBlockStart metadata + input: TextualBlock, // accumulated from onContentDelta +} + +ToolResultBlock = struct { + tool_use_id: []const u8, // owned copy + content: TextualBlock, // accumulated content +} +``` + +**Memory discipline**: When a `ContentBlock` is moved into a `Message`'s content list (stored in `std.ArrayList(ContentBlock)`), the TextualBlock's internal ArrayList buffer pointer remains valid — it points to the same heap allocation. The caller must ensure each block's `deinit()` is called exactly once, and must not copy a ContentBlock without clearing the source (standard Zig move semantics). + +### Message + +``` +Message = { + role: enum { system, user, assistant }, + content: []ContentBlock, +} +``` + +A system message may contain multiple `Text` blocks. When serializing to Anthropic's API (phase 2), these are concatenated into the single system prompt string. + +An assistant message is assembled incrementally during streaming. A user message containing tool results (phase 3+) naturally groups multiple `ToolResult` blocks. + +### Conversation + +``` +Conversation = { + messages: std.ArrayList(Message), + allocator: std.mem.Allocator, +} +``` + +Ordered list of messages. Methods: + +- `init(allocator)` → Conversation +- `addSystemMessage(text)` → appends `Message{ .system, [TextBlock(text)] }` +- `addUserMessage(text)` → appends `Message{ .user, [TextBlock(text)] }` +- `addAssistantMessage(blocks)` → appends `Message{ .assistant, blocks }` (called by agent loop after streaming completes) +- `deinit()` → frees all owned memory + +All `[]const u8` fields in ContentBlocks and Messages are owned by the Conversation and freed on `deinit()`. Content is stored as copies, not slices into external buffers. `TextualBlock` fields back their content with a heap-allocated `ArrayList(u8)` that grows incrementally during streaming and is freed on `deinit()`. + +--- + +## Module Structure + +``` +src/ + root.zig // public API re-exports + conversation.zig // Message, ContentBlock, Conversation + provider.zig // Provider interface, StreamEvent, StreamResult + provider_openai.zig // OpenAI-compatible implementation + sse.zig // SSE line parser + agent.zig // Agent loop: runStep, Receiver interface + config.zig // Config struct (api_key, base_url, model) + json.zig // Serialization helpers (model → wire JSON, deltas → ContentBlocks) +``` + +### `conversation.zig` + +Defines `Message`, `ContentBlock`, `Conversation`. All serialization to/from provider wire formats lives in `json.zig` — conversation.zig is pure data structure. + +Tests: create conversations, add messages, verify content, free without leaks. + +### `provider.zig` + +Defines the `Provider` interface: + +``` +Provider = struct { + ptr: *anyopaque, + vtable: *const VTable, + + VTable = struct { + streamStep: *const fn(*anyopaque, conversation: *Conversation, receiver: *Receiver) anyerror!void, + deinit: *const fn(*anyopaque) void, + }; + + pub fn streamStep(self, conversation, receiver) !void + pub fn deinit(self) void +}; +``` + +And the `Receiver` interface for streaming callbacks: + +``` +Receiver = struct { + ptr: *anyopaque, + vtable: *const ReceiverVTable, + + ReceiverVTable = struct { + onMessageStart: *const fn(*anyopaque, role: MessageRole) void, + onBlockStart: *const fn(*anyopaque, block_type: ContentBlockType, index: usize, meta: ?BlockMeta) void, + onContentDelta: *const fn(*anyopaque, block_index: usize, delta: []const u8) void, + onBlockComplete: *const fn(*anyopaque, block_index: usize, block: ContentBlock) void, + onMessageComplete:*const fn(*anyopaque, message: Message) void, + }; + + pub fn onMessageStart(self, role) void + pub fn onBlockStart(self, block_type, index, meta) void + pub fn onContentDelta(self, block_index, delta) void + pub fn onBlockComplete(self, block_index, block) void + pub fn onMessageComplete(self, message) void +}; + +BlockMeta = struct { + // Only populated for ToolUse blocks. Null for Text/Thinking. + tool_id: ?[]const u8, + tool_name: ?[]const u8, +}; +``` + +**Callback contract:** + +- Callbacks are always invoked in this order for every block, regardless of which provider is active. +- `onMessageStart` is called when the stream begins delivering a new message. +- `onBlockStart` is called when a new content block begins. `meta` carries block-type-specific metadata (tool id/name for ToolUse, null for Text/Thinking). +- `onContentDelta` is called zero or more times per block with raw byte fragments. For Text/Thinking these are word fragments; for ToolUse these are JSON fragments. The receiver does not need to interpret them. `delta` is a `[]const u8` — libpanto does not parse tool input content, it passes bytes through. +- `onBlockComplete` is called when a block is finished. The `block` parameter contains the fully assembled ContentBlock. The receiver that only needs complete content can ignore deltas and use this. +- `onMessageComplete` is called when the stream ends. The `message` parameter contains the fully assembled Message with all blocks. +- Providers guarantee that `onBlockComplete`'s `block` and `onMessageComplete`'s `message` are always fully assembled and valid. + +This uniform callback sequence means the TUI and agent loop don't need to know which provider is active. Anthropic (phase 2) maps its structured events directly to these callbacks; OpenAI synthesizes block boundaries from delta field transitions (see below). + +### `provider_openai.zig` + +Implements `Provider` for OpenAI-compatible APIs. + +- Converts `Conversation` → OpenAI wire JSON (see [OpenAI serialization](#openai-serialization) below) +- Makes HTTP POST to `{base_url}/chat/completions` with `stream: true` +- Reads SSE events, parses each `data: {...}` line as complete JSON +- Synthesizes block boundaries from delta field transitions (OpenAI has no explicit block boundary events) +- Calls the full Receiver callback sequence (onMessageStart → onBlockStart → onContentDelta → onBlockComplete → onMessageComplete) +- Accumulates deltas into ContentBlocks via TextualBlocks + +Construction: + +``` +OpenAIProvider.init(allocator, config) !OpenAIProvider +``` + +### OpenAI block boundary synthesis + +OpenAI's streaming deltas have no explicit block boundaries. The provider tracks a state machine to infer when blocks start and end: + +``` +StreamingState = struct { + active_block_type: enum { none, thinking, text, tool_use }, + active_block_index: usize, + // assembly buffers per block +} + +On each SSE event: + 1. If delta.role == "assistant" → emit onMessageStart(.assistant) + 2. If delta.reasoning_content present: + - If active_block_type != .thinking: + - If active_block_type != .none → emit onBlockComplete for prior block + - Emit onBlockStart(.Thinking, index, null) + - active_block_type = .thinking + - Emit onContentDelta(index, delta.reasoning_content) + 3. If delta.content present: + - If active_block_type != .text: + - If active_block_type != .none → emit onBlockComplete for prior block + - Emit onBlockStart(.Text, index, null) + - active_block_type = .text + - Emit onContentDelta(index, delta.content) + 4. If delta.tool_calls present: + - If active_block_type != .tool_use: + - If active_block_type != .none → emit onBlockComplete for prior block + - Emit onBlockStart(.ToolUse, index, .{ .tool_id = ..., .tool_name = ... }) + - active_block_type = .tool_use + - Emit onContentDelta(index, delta.tool_calls[].function.arguments) + 5. If finish_reason != null: + - If active_block_type != .none → emit onBlockComplete for current block + - Emit onMessageComplete(assembled_message) +``` + +A transition in `active_block_type` means the previous block is done and a new one has started. The state machine also handles the case where the same block type appears again after an intervening type (e.g., thinking → text → thinking), which would open a new Thinking block at a new index. + +### `sse.zig` + +Incremental SSE line parser. The HTTP client delivers arbitrary-sized read buffers; this module reassembles them into complete `data: ...\n\n` events. + +``` +SSEParser = struct { + buf: std.ArrayList(u8), + + pub fn init(allocator) SSEParser + pub fn feed(self, chunk: []const u8) ![]const []const u8 // returns slice of complete event strings + pub fn deinit(self) void +}; +``` + +`feed()` may return zero events (partial line buffered) or multiple events (chunk contained several). The caller does not need to worry about line boundaries. + +Tests: feed partial chunks, verify events emitted at correct boundaries; multi-event in single chunk; empty lines; `data: [DONE]`. + +### `json.zig` + +Two responsibilities: + +1. **Serialize Conversation → OpenAI request body** — Convert our `Message`/`ContentBlock` model into the JSON shape OpenAI expects. See below. +2. **Parse SSE chunk deltas → ContentBlock updates** — Each SSE event's JSON contains a `choices[0].delta` object. Extract text/thinking content from it. + +### `agent.zig` + +The agent loop. In phase 1, it's simple: + +``` +Agent = struct { + provider: Provider, + allocator: std.mem.Allocator, + + pub fn init(allocator, provider) Agent + pub fn runStep(self, conversation: *Conversation, receiver: *Receiver) !void + pub fn deinit(self) void +}; +``` + +`runStep` does: +1. Call `provider.streamStep(conversation, receiver)` — this streams the response and calls the full Receiver callback sequence on the receiver +2. The `onMessageComplete` callback appends the finished Message to the conversation (the agent itself can wire this, or the caller handles it — TBD during implementation) + +In later phases, `runStep` gains the tool-call loop: check for ToolUse blocks, execute them, feed results back, call provider again. But the shape stays the same — one `runStep` invocation carries the conversation through a full agent turn. + +### `config.zig` + +``` +Config = struct { + api_key: []const u8, + base_url: []const u8, // e.g. "https://api.openai.com/v1" + model: []const u8, // e.g. "gpt-4o" +}; +``` + +Populated from environment variables (`PANTO_API_KEY`, `PANTO_BASE_URL`, `PANTO_MODEL`) with defaults for base_url and model. + +### `root.zig` + +Public API. Re-exports the types and functions that external Zig code needs: + +``` +pub const conversation = @import("conversation.zig"); +pub const provider = @import("provider.zig"); +pub const agent = @import("agent.zig"); +pub const config = @import("config.zig"); +``` + +Does not re-export provider_openai, sse, or json — those are internal. + +--- + +## OpenAI Serialization + +### Request + +Our `Conversation` → OpenAI `chat/completions` request body: + +``` +{ + "model": config.model, + "stream": true, + "messages": [ + // For each Message in conversation: + // + // role=.system → { "role": "system", "content": "<concatenated text blocks>" } + // role=.user → { "role": "user", "content": "<concatenated text blocks>" } + // (ToolResult blocks pulled out into separate role:tool messages in phase 3+) + // role=.assistant → { "role": "assistant", "content": [ + // ...text blocks as { "type": "text", "text": "..." }, + // ...thinking blocks as { "type": "thinking", "thinking": "..." }, + // ...tool_use blocks become function_call/function entries in phase 3+ + // ] } + ] +} +``` + +For phase 1, all content blocks we encounter are `Text` or `Thinking`, so serialization is straightforward. + +### Response (streaming) + +Each SSE event is a complete JSON object: + +``` +data: {"id":"chatcmpl-123","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]} +data: {"id":"chatcmpl-123","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]} +data: {"id":"chatcmpl-123","choices":[{"index":0,"delta":{"content":" world"},"finish_reason":null}]} +data: {"id":"chatcmpl-123","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} +data: [DONE] +``` + +We parse each event's `choices[0].delta` and drive the block boundary state machine: +- `delta.role == "assistant"` → emit onMessageStart, marks the start of a new assistant message +- `delta.reasoning_content` → transition to Thinking block if needed, append via onContentDelta +- `delta.content` → transition to Text block if needed, append via onContentDelta +- `delta.tool_calls` → transition to ToolUse block if needed, append arguments via onContentDelta (phase 3+) + +The `finish_reason: "stop"` signals stream end → emit onBlockComplete for any active block, then onMessageComplete. + +--- + +## Minimal CLI + +``` +panto/ + src/ + main.zig // CLI entry point +``` + +Behavior: +1. Read `PANTO_API_KEY`, `PANTO_BASE_URL`, `PANTO_MODEL` from environment +2. Create a Conversation, add a system message (default: "You are a helpful assistant.") +3. Print a prompt (`> `), read a line from stdin +4. Add user message, call `agent.runStep()`, print streamed deltas to stdout +5. Repeat step 3 until EOF (Ctrl+D) + +There is no line editing, no scrolling, no syntax highlighting. Just `readline` → `print`. The sole purpose is exercising libpanto against a real API. + +--- + +## Testing Strategy + +### Unit tests (automated, per module) + +| Module | What to test | +|---|---| +| `conversation.zig` | Create conversation, add messages of each role, verify content block storage, free without leaks | +| `sse.zig` | Feed partial chunks, verify event boundaries; multi-event chunks; `data: [DONE]`; empty lines between events | +| `json.zig` | Serialize conversation → OpenAI JSON; parse delta JSON objects → content updates | +| `config.zig` | Parse from env vars; defaults for missing optional fields | + +### Integration test (manual) + +- Run `panto` binary with a real API key +- Hold a multi-turn conversation +- Verify responses stream to stdout +- Verify follow-up messages include prior context (ask the model "what did I just say?") + +--- + +## Open Questions (to resolve during implementation) + +1. **Thinking token support in OpenAI API**: OpenAI's `reasoning_content` field in streaming deltas is not universally present across models/endpoints. We need to handle its absence gracefully (just skip it, don't crash). +2. **Error handling in streams**: Mid-stream HTTP errors, rate limiting, truncated responses. How do we represent these to the caller? An `onError` callback on the Receiver seems likely. +3. **HTTP connection lifecycle**: Does `std.http.Client` support long-lived streaming connections cleanly? We may need to manage connection pooling or timeouts. +4. **Memory strategy for long conversations**: We're storing full message content in memory. For phase 1 this is fine, but we should define the interface so a later phase can introduce message summarization or offloading without changing the agent loop. diff --git a/docs/archive/phase-2.md b/docs/archive/phase-2.md new file mode 100644 index 0000000..4cb7ed9 --- /dev/null +++ b/docs/archive/phase-2.md @@ -0,0 +1,157 @@ +# Phase 2: Anthropic Provider + +**Status: complete.** Anthropic Messages API streams end-to-end alongside the phase-1 OpenAI provider; the Receiver callback sequence is identical across both; system prompts extract to the top-level field; thinking blocks round-trip via captured `signature` deltas. Refinements that diverged from the original plan: file/type names use the explicit dialect (`provider_anthropic_messages`, `AnthropicMessagesConfig`) to leave room for `openai_responses` and future Anthropic shapes; `Config` is a tagged union keyed by `APIStyle` rather than separate types at call sites; concrete provider types are internal — callers construct via `provider.Provider.init(allocator, io, config)`. Wire-level gotchas worth remembering: both providers send `accept-encoding: identity` because gzip buffers small SSE frames and defeats streaming; the read loop uses `readVec` rather than `readSliceShort` because the latter fills its buffer before returning, which also defeats streaming. Tool use / tool result blocks remain stubbed for phase 3+. + +## Goal + +Add Anthropic as a second provider, validating that the Provider abstraction and internal conversation model are genuinely provider-agnostic — not just OpenAI in disguise. + +## Deliverable + +A working `provider_anthropic.zig` that can hold a streaming conversation via Anthropic's API. At the end of this phase, you can: + +- Switch between OpenAI and Anthropic providers with no changes to the agent loop or conversation model. +- Stream responses from either provider and see the same callback sequence (onMessageStart, onBlockStart, onContentDelta, onBlockComplete, onMessageComplete). +- Observe thinking and text content streaming correctly from Anthropic models. + +## What is usable at the end + +| Capability | How to exercise it | +|---|---| +| Create an Anthropic provider | `AnthropicProvider.init(allocator, config)` | +| Run a chat via Anthropic | Same `agent.runStep()` call, different provider | +| Stream Anthropic responses | Same Receiver interface, same callback sequence | +| System prompts with Anthropic | System messages extracted and sent as top-level system field | + +## What is explicitly out of scope + +- Tools and tool-use (phase 3+) +- Extensions (phase 3+) +- Conversation serialization / disk persistence (phase 4) +- Server/proxy mode (future) +- Google API provider (future) + +--- + +## Receiver Interface + +The Receiver interface with the full 5-callback lifecycle is defined in phase 1. Both providers must produce the same callback sequence: + +- onMessageStart → onBlockStart → onContentDelta(s) → onBlockComplete → ... → onMessageComplete + +See `phase-1.md` for the full definition and contract. + +### How OpenAI synthesizes the callbacks + +Defined in phase 1. OpenAI has no explicit block boundaries; the provider infers them via a state machine that tracks the active block type and emits start/complete callbacks on transitions. + +### How Anthropic maps to the callbacks + +Anthropic's structured events map directly: + +| Anthropic event | Callback | +|---|---| +| `message_start` | onMessageStart(.assistant) | +| `content_block_start` | onBlockStart(type, index, meta if ToolUse) | +| `content_block_delta` | onContentDelta(index, delta bytes) | +| `content_block_stop` | onBlockComplete(index, assembled block) | +| `message_delta` + `message_stop` | onMessageComplete(assembled message) | + +No inference needed — Anthropic gives us explicit boundaries. + +--- + +## Anthropic Request Serialization + +### Wire format differences from OpenAI + +| Aspect | OpenAI | Anthropic | +|---|---|---| +| System prompt | Messages with `role: "system"` | Top-level `system` field (string) | +| Content shape | String or array of parts | Always array of content blocks | +| Tool results | Separate `role: "tool"` messages | Content blocks on `role: "user"` messages | +| Auth | `Authorization: Bearer <key>` | `x-api-key: <key>` + `anthropic-version` header | +| Streaming | `stream: true` in request body | `stream: true` in request body | + +### Serialization rules + +**System messages**: Extract all `role=.system` messages from the conversation. Concatenate their Text block contents into a single string. Set as the top-level `system` field. Do not include them in the messages array. + +**User messages**: Emit as `role: "user"`. Content blocks become Anthropic content block format: +- Text → `{ "type": "text", "text": "..." }` +- ToolResult → `{ "type": "tool_result", "tool_use_id": "...", "content": "..." }` (phase 3+) + +**Assistant messages**: Emit as `role: "assistant"`. Content blocks: +- Text → `{ "type": "text", "text": "..." }` +- Thinking → `{ "type": "thinking", "thinking": "..." }` +- ToolUse → `{ "type": "tool_use", "id": "...", "name": "...", "input": {...} }` (phase 3+) + +Note: Anthropic expects ToolUse's `input` as a parsed JSON object, not a string. Since we store `input` as raw bytes in a TextualBlock, we will need to parse it into a `std.json.Value` during Anthropic serialization. This is the one place where libpanto does parse tool input JSON — it's a serialization requirement, not an interpretation of the tool schema. The round-trip guarantee is: the bytes we stored serialize back to equivalent JSON when sent to Anthropic. + +--- + +## Anthropic Streaming Event Parser + +Each SSE event is a complete JSON object. The event type is in a top-level `type` field. + +### Event types and handling + +| Event type | What we extract | Action | +|---|---|---| +| `message_start` | `message.role`, `message.id`, `message.model` | Emit onMessageStart; begin assembling Message | +| `content_block_start` | `content_block.type`, `content_block.index`, `content_block.id`, `content_block.name` | Emit onBlockStart; create new TextualBlock or ToolUseBlock | +| `content_block_delta` | `delta.type` (text_delta, thinking_delta, input_json_delta), `delta.text` or `delta.thinking` or `delta.partial_json` | Append to current block's buffer; emit onContentDelta | +| `content_block_stop` | `index` | Emit onBlockComplete with assembled block | +| `message_delta` | `delta.stop_reason`, `usage` | Track stop reason for onMessageComplete | +| `message_stop` | (none) | Emit onMessageComplete with fully assembled Message | + +The parser is a separate concern from the SSE line parser (`sse.zig`). The SSE parser reassembles byte chunks into complete `data: {...}` events. The Anthropic event parser interprets the JSON of each event. They compose: `HTTP read → SSE parser → event strings → Anthropic event parser → callbacks`. + +--- + +## Module Changes + +### New files + +``` +src/provider_anthropic.zig // Anthropic provider implementation +``` + +### Modified files + +- `provider.zig` — ReceiverVTable expanded to the 5-callback lifecycle +- `provider_openai.zig` — No changes needed (block synthesis already built in phase 1) +- `agent.zig` — Any changes needed for expanded Receiver (should be minimal since Receiver is an interface) + +### Config + +``` +AnthropicConfig = struct { + api_key: []const u8, + base_url: []const u8, // e.g. "https://api.anthropic.com" + model: []const u8, // e.g. "claude-sonnet-4-20250514" + api_version: []const u8, // e.g. "2023-06-01" +}; +``` + +Separate from `OpenAIConfig`. The CLI and any higher-level config can unify them; libpanto treats them as distinct. + +--- + +## Testing Strategy + +### Unit tests + +| What | How | +|---|---| +| Anthropic serialization | Create conversations with system/user/assistant messages, serialize to Anthropic JSON, verify structure and system prompt extraction | +| Streaming event parser | Feed canned Anthropic SSE events (message_start, content_block_start, etc.) and verify correct callback sequence and assembled output | +| Block boundary synthesis | Feed OpenAI-style deltas (reasoning → content transitions) and verify onBlockStart/onBlockComplete emitted correctly | +| Receiver contract | Verify that both providers produce the same callback sequence for equivalent conversations | + +### Integration test (manual) + +- Run `panto` binary against Anthropic API with a real API key +- Hold a multi-turn conversation with thinking model +- Verify thinking and text stream correctly +- Switch to OpenAI provider, verify same experience diff --git a/docs/archive/phase-3.md b/docs/archive/phase-3.md new file mode 100644 index 0000000..eb52ee9 --- /dev/null +++ b/docs/archive/phase-3.md @@ -0,0 +1,391 @@ +# Phase 3: Extension System + +> **Status:** Complete. The Lua runtime described in this doc has been **superseded by [`LUA_MAKEOVER.md`](../LUA_MAKEOVER.md)**. The current `panto` CLI uses a long-lived `lua_State` exposed to libpanto as a single `ToolSource`; the per-call `LuaStatePool` / `LuaTool` design described below no longer exists. The **native extension contract (`Tool`) is unchanged** — the rest of this document remains accurate for that path. + +## Goal + +Introduce a native-code tool extension API on `libpanto` and a Lua extension runtime in the `panto` CLI. Together these transform `pantograph` from a chat client into an agent that can act on the world. The extension system is the primary mechanism for adding capability — tools are extensions, not built-ins. + +This phase also marks the point where the `Agent` abstraction starts earning its keep. Until now, `agent.zig` has been a thin pass-through to the provider. In phase 3 the agent grows into the thing that owns the tool registry and drives the tool-call loop. Providers stay dumb (stream blocks); the agent is what makes a conversation iterative rather than one-shot. + +## Deliverable + +Two layered deliverables: + +1. **libpanto** — a Zig-native tool extension API. `Agent` exposes `registerTool(Tool)` and runs a tool-call loop in `runStep`. The agent loop detects ToolUse blocks in LLM responses, dispatches to registered tools, feeds ToolResult blocks back, and repeats until the model stops calling tools. libpanto has no awareness of Lua, Python, or any specific extension language. + +2. **panto CLI** — a Lua extension runtime that discovers Lua scripts on disk, loads them, and registers each declared tool with `libpanto` via a `LuaTool` adapter that satisfies libpanto's `Tool` interface. + +At the end of this phase, you can: + +- Embed `libpanto` in a Zig program, implement a `Tool` struct natively, and register it with the agent — no Lua involved. +- Write a Lua extension that registers a tool, place it in `.agent/extensions/` (or `~/.config/panto/extensions/`), and have the `panto` CLI discover and load it. +- Have a conversation where the LLM calls your tool and receives the result. +- See tool calls execute in parallel when the LLM returns multiple ToolUse blocks. +- See a meaningful error message when a Lua extension crashes, instead of a process abort. + +## What is usable at the end + +| Capability | How to exercise it | +|---|---| +| Native tool registration | In a Zig embedder, construct a `Tool` and call `agent.registerTool(tool)` | +| Write a Lua tool extension | Create `.agent/extensions/mytool.lua` calling `panto.register_tool(...)` | +| Discover Lua extensions | Place `.lua` files or directories in extension directories; `panto` CLI loads them on startup | +| LLM calls a tool | Ask the LLM to use a registered tool; it emits a ToolUse block; the agent dispatches to the registered handler | +| Tool result fed back | The tool's return bytes become a ToolResult block sent back to the LLM | +| Parallel tool calls | LLM returns multiple ToolUse blocks; they execute concurrently | +| Lua crash handling | A crashing Lua handler prints `the "mytool" extension crashed: <trace>` and aborts the turn | + +## What is explicitly out of scope + +- Core tools as extensions (phase 5 — `std.read`, `std.write`, etc.) +- Conversation serialization / disk persistence (phase 4) +- C ABI distribution of libpanto for external consumers (future) +- GitHub or luarocks extension loaders (future — local filesystem only in phase 3) +- Shared-object extensions (future) +- Extension sandboxing beyond `xpcall` crash protection in the Lua adapter (future) +- Config file for specifying which extensions to load (phase 6 — phase 3 loads everything it discovers) +- Non-Lua extension runtimes (future — the libpanto API is general, but only the Lua adapter ships in phase 3) + +--- + +## libpanto: Native Tool API + +### The `Tool` interface + +`libpanto` defines a `Tool` as an opaque-context value with a vtable. This is the boundary between the agent loop and any extension runtime. + +```zig +pub const Tool = struct { + name: []const u8, // borrowed; lifetime owned by the registrar + schema_json: []const u8, // borrowed JSON Schema bytes; lifetime owned by the registrar + ctx: *anyopaque, + vtable: *const VTable, + + pub const VTable = struct { + /// Invoke the tool. MUST be thread-safe — the agent may call `invoke` + /// concurrently from multiple threads when the LLM returns multiple + /// ToolUse blocks in a single response. + /// + /// `input` is the raw JSON bytes the provider sent. The tool is + /// responsible for parsing them if it cares about their structure. + /// + /// Returns owned bytes allocated with `allocator`. These bytes become + /// the `content` of the ToolResult block. + /// + /// Returning an error aborts the current turn. The agent will surface + /// the error to the user. Native tool implementations are responsible + /// for catching their own panics — a panic in `invoke` will crash the + /// process. Adapters that bridge to safer languages (Lua, Python, Go) + /// should convert panics/exceptions into errors here. + invoke: *const fn (ctx: *anyopaque, input: []const u8, allocator: std.mem.Allocator) anyerror![]u8, + + /// Called when the tool is removed from the registry or the agent is + /// torn down. Frees any resources owned by `ctx`. + deinit: *const fn (ctx: *anyopaque, allocator: std.mem.Allocator) void, + }; +}; +``` + +The contract: + +- **Thread-safety is mandatory.** Tool implementations promise that concurrent `invoke` calls are safe. This pushes the burden onto adapters (where it belongs — the Lua adapter solves it with a `lua_State` pool; a native Zig tool may just be re-entrant) and out of the agent loop. +- **Input is opaque bytes.** libpanto never parses tool input. The tool decides whether to parse JSON, treat it as a string, or ignore it. +- **Output is opaque bytes.** Same principle in reverse. +- **Errors abort the turn.** No partial recovery in phase 3. +- **Panics crash the process.** Native code that wants safety implements it itself. This is trivially obvious from the API surface: it's a function pointer with `anyerror!` return — there's no magic. Adapters for safer languages can wrap panics/exceptions and convert them to errors. + +### `Agent` changes + +```zig +pub const Agent = struct { + provider: provider.Provider, + allocator: std.mem.Allocator, + registry: ToolRegistry, + + pub fn init(allocator: Allocator, prov: provider.Provider) Agent { ... } + pub fn deinit(self: *Agent) void { ... } + + pub fn registerTool(self: *Agent, tool: Tool) !void { ... } + pub fn unregisterTool(self: *Agent, name: []const u8) void { ... } + + pub fn runStep(self: *Agent, conv: *Conversation, receiver: *Receiver) !void { ... } +}; +``` + +`registerTool` takes ownership of the `Tool` value (the agent calls `tool.vtable.deinit` on teardown or unregister). Tool name uniqueness is enforced — registering a name that already exists is an error. + +### `ToolRegistry` + +A small internal module: + +``` +ToolRegistry = struct { + tools: std.StringHashMap(Tool), + allocator: Allocator, + + fn register(name, tool) !void + fn unregister(name) void + fn lookup(name) ?*const Tool + fn iterator() Iterator // for serializing into provider requests + fn deinit() void // invokes each tool's vtable.deinit +}; +``` + +### Agent loop with tools + +`runStep` gains the tool-call loop: + +``` +runStep(conv, receiver): + loop: + provider.streamStep(conv, receiver) + inspect the assistant message just appended to conv for ToolUse blocks + if none: return — turn complete + for each ToolUse block, in parallel: + tool = registry.lookup(block.name) or error + result_bytes = tool.vtable.invoke(tool.ctx, block.input, allocator) + build a ToolResult block { tool_use_id = block.id, content = result_bytes } + append a user Message containing all ToolResult blocks to conv + continue loop +``` + +A single `runStep` may invoke the provider multiple times if the LLM chains tool calls. + +### Parallel execution + +When a response contains multiple ToolUse blocks, the agent dispatches them concurrently. Implementation likely uses a small thread pool or `std.Thread.spawn` per call (TBD during implementation). Because `Tool.invoke` is declared thread-safe, the agent loop has no further constraints — it just fans out and joins. + +### Tool serialization in provider requests + +When the agent calls the provider, the provider receives the registry (or an iterator over it) and emits the tools array. This means provider request serialization gains a tools-emitting branch when the registry is non-empty. + +**OpenAI**: +```json +{ + "tools": [ + { "type": "function", + "function": { "name": "echo", "parameters": <schema_json> } } + ] +} +``` + +**Anthropic**: +```json +{ + "tools": [ + { "name": "echo", "input_schema": <schema_json> } + ] +} +``` + +The `schema_json` bytes are emitted verbatim. Description fields are deferred — `Tool` does not yet carry a description (see open questions). + +### ToolUse decoding in responses + +Already handled by the existing Receiver callback sequence from phase 1/2: + +- OpenAI: `delta.tool_calls` → `onBlockStart(.ToolUse, ...)` with `meta.tool_id` and `meta.tool_name`, then `onContentDelta` with JSON argument fragments. +- Anthropic: `content_block_start` with `type: "tool_use"` → `onBlockStart(.ToolUse, ...)`, then `content_block_delta` with `input_json_delta` fragments. + +The assembled ToolUseBlock contains `id`, `name`, and `input` (TextualBlock with the full JSON string). + +### ToolResult encoding in requests + +**OpenAI** — each ToolResult block becomes a separate top-level `tool` message: +```json +{ "role": "tool", "tool_call_id": "<id>", "content": "<result>" } +``` + +**Anthropic** — ToolResult blocks are content blocks on a user message: +```json +{ "role": "user", "content": [ + { "type": "tool_result", "tool_use_id": "<id>", "content": "<result>" } +] } +``` + +--- + +## panto CLI: Lua Extension Runtime + +The Lua runtime lives entirely in the CLI. libpanto has no Lua dependency. + +### Extension discovery + +The CLI scans two directories in order: + +1. `.panto/extensions/` — project-local, relative to current working directory +2. `~/.config/panto/extensions/` — user-level + +`.panto/` follows the established per-agent pattern (`.claude/`, `.opencode/`, `.pi/`). + +### Naming and structure + +- **Single-file**: `<name>.lua` → extension name is `<name>` +- **Directory**: `<name>/init.lua` → extension name is `<name>` + +Hierarchical names via dot/directory mapping, mirroring `require("a.b.c")`: + +- `utils/json.lua` → `utils.json` +- `coding/edit/init.lua` → `coding.edit` + +Sub-modules under a directory extension (e.g., `coding/edit/helpers.lua`) are the extension's internal business — the CLI only loads the top-level entry point. + +### Loading behavior + +- Recursively scan both directories. +- Construct extension names from relative paths. +- For each unique extension name, load its entry point into the `lua_State` pool (see below). +- Project-local entries shadow user-level entries of the same name. +- Extension top-level code runs at load time; it is expected to call `panto.register_tool(...)`. + +### The Lua bridge + +A small Zig module (`lua_bridge.zig`) registers Zig functions into a `lua_State`. The bridge exposes: + +#### `panto.register_tool(name, schema, handler)` + +- `name` (string) — tool name, e.g. `"bash"` +- `schema` (table) — JSON Schema as a Lua table. The bridge serializes this to JSON bytes at registration time. +- `handler` (function) — invoked when the LLM calls this tool. Receives a single argument: a Lua table parsed from the input JSON. Must return a string. + +```lua +panto.register_tool("echo", { + type = "object", + properties = { + message = { type = "string", description = "The message to echo back" } + }, + required = { "message" } +}, function(input) + return input.message +end) +``` + +The bridge stores the handler in the Lua registry (`luaL_ref`) and constructs a `LuaTool` (see below) that it then registers with the agent. + +### `LuaTool` — the adapter to libpanto + +```zig +const LuaTool = struct { + pool: *LuaStatePool, + handler_ref: i32, // registry ref to the Lua function + name_owned: []u8, + schema_owned: []u8, + + // Implements Tool.VTable.invoke: + // 1. pool.acquire() → lua_State + // 2. lua_rawgeti(L, LUA_REGISTRYINDEX, handler_ref) — push handler + // 3. parse input bytes into a Lua table via std.json + // 4. xpcall the handler with the table + // 5. read returned string (or capture trace on error) + // 6. pool.release(L) + // 7. return result bytes or error +}; +``` + +The adapter is what makes the CLI side responsible for Lua-specific concerns: + +- **JSON → Lua table conversion**: happens here, not in libpanto. +- **`xpcall` crash protection**: wraps the handler call, captures `debug.traceback`, returns an error so libpanto aborts the turn cleanly with `the "<tool_name>" extension crashed: <trace>`. +- **Thread-safety**: provided by the `lua_State` pool. Each concurrent `invoke` checks out a separate state. libpanto's contract is satisfied. + +### `LuaStatePool` + +On-demand pool of `lua_State` instances: + +``` +LuaStatePool = struct { + states: std.ArrayList(*lua_State), + available: std.BitSet, + allocator: Allocator, + extension_paths: []const []const u8, // all discovered entry points + + fn acquire() *lua_State // returns a free state, or creates a new one + fn release(L: *lua_State) void // returns L to the pool + fn deinit() void // closes all states +}; +``` + +- States are created lazily — no pre-allocation. +- Each new state loads all discovered extensions identically, so any state can handle any tool. +- A state is checked out for the duration of a single `invoke` call. + +There's a coupling here: the handler `luaL_ref` is per-`lua_State`. The pool must guarantee that the same registry slot in every state points at the same handler. The cleanest way is: when an extension is first loaded into the prototype state, the bridge records the registry index. When a new state is constructed, the bridge re-loads all extensions in the same order, producing the same ref indices. This needs care during implementation but is mechanically straightforward. + +--- + +## Module Changes + +### libpanto + +New: +- `libpanto/src/tool.zig` — `Tool` and `VTable` definitions +- `libpanto/src/tool_registry.zig` — `ToolRegistry` + +Modified: +- `libpanto/src/agent.zig` — owns registry, gains `registerTool`, drives tool-call loop in `runStep` +- `libpanto/src/provider.zig` — provider receives a way to see registered tools (likely a `*const ToolRegistry` on the agent passed via `streamStep` context, or a callback) +- `libpanto/src/provider_openai_chat.zig` — request serialization emits `tools` array +- `libpanto/src/provider_anthropic_messages.zig` — request serialization emits `tools` array +- `libpanto/src/openai_chat_json.zig` — ToolResult message encoding +- `libpanto/src/anthropic_messages_json.zig` — ToolResult content block encoding +- `libpanto/src/root.zig` — re-exports `Tool`, `ToolRegistry` + +### panto CLI + +New: +- `src/lua_bridge.zig` — Zig functions registered into Lua states; `panto.register_tool` implementation +- `src/lua_tool.zig` — `LuaTool` adapter implementing libpanto's `Tool` interface +- `src/lua_state_pool.zig` — on-demand `lua_State` pool +- `src/extension_loader.zig` — directory scanning, name construction, loading + +Modified: +- `src/main.zig` — wires extension discovery + loading into agent startup + +### External dependency + +Lua 5.4 linked into the `panto` binary. Zig's build system can fetch and compile Lua from source (~30KLOC C). No system dependency required. libpanto does not link Lua. + +--- + +## Testing Strategy + +### Unit tests (libpanto) + +| What | How | +|---|---| +| Tool registration | Construct a native `Tool`, register with `Agent`, verify lookup | +| Duplicate registration | Verify registering the same name twice returns an error | +| Tool dispatch | Hand-craft a conversation with a ToolUse block, run `runStep` against a stub provider, verify the registered tool's `invoke` is called with the right bytes | +| ToolResult round-trip | Verify the ToolResult message appended to the conversation has the right `tool_use_id` and `content` | +| Parallel dispatch | Register two tools with sleep+timestamps, verify they ran concurrently (elapsed wall time < sum of individual times) | +| Provider request serialization | Snapshot tests on OpenAI and Anthropic request bodies with tools registered | + +### Unit tests (panto CLI) + +| What | How | +|---|---| +| Extension discovery | Create temp directory with single-file and directory extensions, verify names constructed correctly | +| Project shadows user | Same extension name in both directories — project wins | +| Lua bridge input | Feed JSON bytes through `LuaTool.invoke`, verify the Lua handler sees the expected table | +| Lua bridge output | Handler returns a string, verify it round-trips to bytes | +| `xpcall` crash protection | Extension whose handler throws — verify `invoke` returns an error and includes the trace | +| Concurrent `invoke` | Two threads call `invoke` on the same `LuaTool`, verify they don't share a `lua_State` | + +### Integration tests (manual) + +- `echo.lua` extension: ask the LLM to use it, verify result is fed back, LLM continues. +- `crash.lua` extension: verify the crash is caught, printed, turn aborts gracefully. +- Multi-tool response: ask the LLM to call two tools in one message, verify both execute concurrently. + +--- + +## Resolved Decisions + +- **Project-local extension directory**: `.panto/extensions/`, matching `.claude/`, `.opencode/`, `.pi/`. +- **Lua version**: Lua 5.4. LuaJIT is stuck at 5.1 semantics and the JIT speedup is largely irrelevant for tool handlers (which spend most of their time in Zig-side I/O). 5.4 keeps us aligned with current luarocks and Lua documentation. +- **Tool description**: a top-level `description: []const u8` field on `Tool`. The Lua bridge gains it as a parameter: `panto.register_tool(name, description, schema, handler)`. Schema describes input only; description is the human-readable purpose. +- **Provider sees the registry**: `provider.streamStep` gains a `*const ToolRegistry` parameter. Embedders normally pass the same registry on every call. +- **Parallel dispatch**: `std.Thread.spawn` per ToolUse block. Thread creation is sub-millisecond on Linux/macOS; tool calls are orders of magnitude longer. Recycling buys nothing meaningful. +- **Streaming tool results**: deferred, possibly indefinitely. Provider APIs only accept a final result payload, so streaming would be display-only — not worth the vtable complication in phase 3. +- **Handler timeout / cancellation**: deferred. POSIX has no clean way to cancel a thread (`pthread_kill` with SIGKILL terminates the whole process; `pthread_cancel` is unusable in practice; signal-based interruption is not async-signal-safe for arbitrary code). Lua's `lua_sethook` only fires between VM steps, so it can't interrupt a handler blocked in a C call. A real cancellation primitive requires process isolation, which is a phase of its own — see the Punted list in [overview.md](overview.md). Phase 3 ships with the honest contract: **tool handlers run to completion. If a tool hangs, the user kills the `panto` process.** |