diff options
Diffstat (limited to 'docs/phase-1.md')
| -rw-r--r-- | docs/phase-1.md | 449 |
1 files changed, 0 insertions, 449 deletions
diff --git a/docs/phase-1.md b/docs/phase-1.md deleted file mode 100644 index 6b149aa..0000000 --- a/docs/phase-1.md +++ /dev/null @@ -1,449 +0,0 @@ -# Phase 1: libpanto — Minimal Chat Library - -**Status: complete.** Streaming chat works end-to-end against OpenAI-compatible APIs; conversation history persists across turns; thinking blocks (`reasoning_content` / `reasoning`) are streamed; `reasoning_effort` is configurable. Open questions resolved: thinking support implemented; mid-stream errors propagate via Zig errors; connections are one-per-turn intentionally (uniform reentry into each turn); long-conversation memory deferred to a later phase. Session persistence is phase 4. - -## Goal - -A Zig library that can hold a streaming conversation with an LLM via an OpenAI-compatible API. No tools, no extensions — just chat. Includes a minimal CLI for live testing. - -## Deliverable - -A `libpanto` Zig module importable by other Zig code, plus a `panto` binary that wires it into a basic read/print loop. At the end of this phase, you can: - -- Start `panto`, type a message, and receive a streamed response from an OpenAI-compatible LLM. -- Send follow-up messages that include full conversation history. -- See thinking tokens and text tokens stream in as they arrive. -- Have the complete conversation available in memory for the duration of the session. - -## What is usable at the end - -| Capability | How to exercise it | -|---|---| -| Open a conversation | `libpanto.conversation.Conversation.init(allocator)` | -| Add a user message | `conversation.addUserMessage("hello")` | -| Run an agent step (streaming) | `agent.runStep(conversation, &receiver)` | -| See streamed output | CLI prints thinking/text chunks as they arrive | -| Conversation persists across turns | Follow-up messages include prior history | - -## What is explicitly out of scope - -- Tools and tool-use (phase 3+) -- Extensions and extension API (phase 3+) -- C ABI (phase 3+, when needed for Lua extensions) -- Anthropic provider (phase 2) -- Disk persistence / session save (later phase) -- Server/proxy mode (future, undefined phase) -- System prompt construction framework (later phase — the model supports system messages, but no opinionated assembly system yet) - ---- - -## Data Model - -### TextualBlock (shared streaming buffer) - -``` -TextualBlock = struct { - buf: std.ArrayList(u8), - allocator: std.mem.Allocator, - - pub fn init(allocator: std.mem.Allocator) TextualBlock - pub fn content(self: *const TextualBlock) []const u8 // self.buf.items - pub fn append(self: *TextualBlock, delta: []const u8) !void - pub fn deinit(self: *TextualBlock) void // self.buf.deinit() -} -``` - -`Text` and `Thinking` blocks both use `TextualBlock` as their payload. They share the same append-based streaming behavior — deltas arrive incrementally and are appended via an internal `ArrayList(u8)`, giving amortized O(1) appends and avoiding the O(n²) re-copying that would result from storing `[]const u8` slices. - -The `TextualBlock` stores its own allocator reference so that `deinit()` needs no external context. Each `TextualBlock` has an `init()`/`deinit()` pair. This also means a `ContentBlock` can clean itself up without the caller providing an allocator. - -### ContentBlock (tagged union) - -``` -ContentBlock = union(enum) { - Text: TextualBlock, - Thinking: TextualBlock, - ToolUse: ToolUseBlock, // phase 3+ - ToolResult: ToolResultBlock, // phase 3+ - - pub fn deinit(self: *ContentBlock) void { - switch (self.*) { - .Text, .Thinking => |*b| b.deinit(), - .ToolUse => |b| { /* free id, name, input */ }, - .ToolResult => |b| { /* free tool_use_id, content */ }, - } - } -} - -ToolUseBlock = struct { - id: []const u8, // owned copy - name: []const u8, // owned copy - input: []const u8, // raw JSON bytes, owned copy -} - -ToolResultBlock = struct { - tool_use_id: []const u8, // owned copy - content: []const u8, // owned copy -} -``` - -`ToolUse` and `ToolResult` are defined in the model now but not populated or processed until the extensions phase. This avoids a model refactor later — the types exist, we just never encounter them in phase 1. - -The `input` field of `ToolUse` is stored as raw JSON bytes (`[]const u8`) rather than a parsed structure. We are not in the business of understanding tool input schemas; we pass them through. - -`ToolUse` blocks also stream incrementally — both providers send tool input as JSON fragments across multiple deltas. Therefore `ToolUse.input` also uses `TextualBlock` for assembly. `id` and `name` arrive at block-start time and are stored as owned `[]const u8` copies. - -`ToolResult` blocks are constructed by `pantograph` itself (not streamed from a provider), so `content` could be a simple `[]const u8`. However, for consistency and to allow progressive construction of results, it also uses `TextualBlock`. - -Updated types: -``` -ToolUseBlock = struct { - id: []const u8, // owned copy, from onBlockStart metadata - name: []const u8, // owned copy, from onBlockStart metadata - input: TextualBlock, // accumulated from onContentDelta -} - -ToolResultBlock = struct { - tool_use_id: []const u8, // owned copy - content: TextualBlock, // accumulated content -} -``` - -**Memory discipline**: When a `ContentBlock` is moved into a `Message`'s content list (stored in `std.ArrayList(ContentBlock)`), the TextualBlock's internal ArrayList buffer pointer remains valid — it points to the same heap allocation. The caller must ensure each block's `deinit()` is called exactly once, and must not copy a ContentBlock without clearing the source (standard Zig move semantics). - -### Message - -``` -Message = { - role: enum { system, user, assistant }, - content: []ContentBlock, -} -``` - -A system message may contain multiple `Text` blocks. When serializing to Anthropic's API (phase 2), these are concatenated into the single system prompt string. - -An assistant message is assembled incrementally during streaming. A user message containing tool results (phase 3+) naturally groups multiple `ToolResult` blocks. - -### Conversation - -``` -Conversation = { - messages: std.ArrayList(Message), - allocator: std.mem.Allocator, -} -``` - -Ordered list of messages. Methods: - -- `init(allocator)` → Conversation -- `addSystemMessage(text)` → appends `Message{ .system, [TextBlock(text)] }` -- `addUserMessage(text)` → appends `Message{ .user, [TextBlock(text)] }` -- `addAssistantMessage(blocks)` → appends `Message{ .assistant, blocks }` (called by agent loop after streaming completes) -- `deinit()` → frees all owned memory - -All `[]const u8` fields in ContentBlocks and Messages are owned by the Conversation and freed on `deinit()`. Content is stored as copies, not slices into external buffers. `TextualBlock` fields back their content with a heap-allocated `ArrayList(u8)` that grows incrementally during streaming and is freed on `deinit()`. - ---- - -## Module Structure - -``` -src/ - root.zig // public API re-exports - conversation.zig // Message, ContentBlock, Conversation - provider.zig // Provider interface, StreamEvent, StreamResult - provider_openai.zig // OpenAI-compatible implementation - sse.zig // SSE line parser - agent.zig // Agent loop: runStep, Receiver interface - config.zig // Config struct (api_key, base_url, model) - json.zig // Serialization helpers (model → wire JSON, deltas → ContentBlocks) -``` - -### `conversation.zig` - -Defines `Message`, `ContentBlock`, `Conversation`. All serialization to/from provider wire formats lives in `json.zig` — conversation.zig is pure data structure. - -Tests: create conversations, add messages, verify content, free without leaks. - -### `provider.zig` - -Defines the `Provider` interface: - -``` -Provider = struct { - ptr: *anyopaque, - vtable: *const VTable, - - VTable = struct { - streamStep: *const fn(*anyopaque, conversation: *Conversation, receiver: *Receiver) anyerror!void, - deinit: *const fn(*anyopaque) void, - }; - - pub fn streamStep(self, conversation, receiver) !void - pub fn deinit(self) void -}; -``` - -And the `Receiver` interface for streaming callbacks: - -``` -Receiver = struct { - ptr: *anyopaque, - vtable: *const ReceiverVTable, - - ReceiverVTable = struct { - onMessageStart: *const fn(*anyopaque, role: MessageRole) void, - onBlockStart: *const fn(*anyopaque, block_type: ContentBlockType, index: usize, meta: ?BlockMeta) void, - onContentDelta: *const fn(*anyopaque, block_index: usize, delta: []const u8) void, - onBlockComplete: *const fn(*anyopaque, block_index: usize, block: ContentBlock) void, - onMessageComplete:*const fn(*anyopaque, message: Message) void, - }; - - pub fn onMessageStart(self, role) void - pub fn onBlockStart(self, block_type, index, meta) void - pub fn onContentDelta(self, block_index, delta) void - pub fn onBlockComplete(self, block_index, block) void - pub fn onMessageComplete(self, message) void -}; - -BlockMeta = struct { - // Only populated for ToolUse blocks. Null for Text/Thinking. - tool_id: ?[]const u8, - tool_name: ?[]const u8, -}; -``` - -**Callback contract:** - -- Callbacks are always invoked in this order for every block, regardless of which provider is active. -- `onMessageStart` is called when the stream begins delivering a new message. -- `onBlockStart` is called when a new content block begins. `meta` carries block-type-specific metadata (tool id/name for ToolUse, null for Text/Thinking). -- `onContentDelta` is called zero or more times per block with raw byte fragments. For Text/Thinking these are word fragments; for ToolUse these are JSON fragments. The receiver does not need to interpret them. `delta` is a `[]const u8` — libpanto does not parse tool input content, it passes bytes through. -- `onBlockComplete` is called when a block is finished. The `block` parameter contains the fully assembled ContentBlock. The receiver that only needs complete content can ignore deltas and use this. -- `onMessageComplete` is called when the stream ends. The `message` parameter contains the fully assembled Message with all blocks. -- Providers guarantee that `onBlockComplete`'s `block` and `onMessageComplete`'s `message` are always fully assembled and valid. - -This uniform callback sequence means the TUI and agent loop don't need to know which provider is active. Anthropic (phase 2) maps its structured events directly to these callbacks; OpenAI synthesizes block boundaries from delta field transitions (see below). - -### `provider_openai.zig` - -Implements `Provider` for OpenAI-compatible APIs. - -- Converts `Conversation` → OpenAI wire JSON (see [OpenAI serialization](#openai-serialization) below) -- Makes HTTP POST to `{base_url}/chat/completions` with `stream: true` -- Reads SSE events, parses each `data: {...}` line as complete JSON -- Synthesizes block boundaries from delta field transitions (OpenAI has no explicit block boundary events) -- Calls the full Receiver callback sequence (onMessageStart → onBlockStart → onContentDelta → onBlockComplete → onMessageComplete) -- Accumulates deltas into ContentBlocks via TextualBlocks - -Construction: - -``` -OpenAIProvider.init(allocator, config) !OpenAIProvider -``` - -### OpenAI block boundary synthesis - -OpenAI's streaming deltas have no explicit block boundaries. The provider tracks a state machine to infer when blocks start and end: - -``` -StreamingState = struct { - active_block_type: enum { none, thinking, text, tool_use }, - active_block_index: usize, - // assembly buffers per block -} - -On each SSE event: - 1. If delta.role == "assistant" → emit onMessageStart(.assistant) - 2. If delta.reasoning_content present: - - If active_block_type != .thinking: - - If active_block_type != .none → emit onBlockComplete for prior block - - Emit onBlockStart(.Thinking, index, null) - - active_block_type = .thinking - - Emit onContentDelta(index, delta.reasoning_content) - 3. If delta.content present: - - If active_block_type != .text: - - If active_block_type != .none → emit onBlockComplete for prior block - - Emit onBlockStart(.Text, index, null) - - active_block_type = .text - - Emit onContentDelta(index, delta.content) - 4. If delta.tool_calls present: - - If active_block_type != .tool_use: - - If active_block_type != .none → emit onBlockComplete for prior block - - Emit onBlockStart(.ToolUse, index, .{ .tool_id = ..., .tool_name = ... }) - - active_block_type = .tool_use - - Emit onContentDelta(index, delta.tool_calls[].function.arguments) - 5. If finish_reason != null: - - If active_block_type != .none → emit onBlockComplete for current block - - Emit onMessageComplete(assembled_message) -``` - -A transition in `active_block_type` means the previous block is done and a new one has started. The state machine also handles the case where the same block type appears again after an intervening type (e.g., thinking → text → thinking), which would open a new Thinking block at a new index. - -### `sse.zig` - -Incremental SSE line parser. The HTTP client delivers arbitrary-sized read buffers; this module reassembles them into complete `data: ...\n\n` events. - -``` -SSEParser = struct { - buf: std.ArrayList(u8), - - pub fn init(allocator) SSEParser - pub fn feed(self, chunk: []const u8) ![]const []const u8 // returns slice of complete event strings - pub fn deinit(self) void -}; -``` - -`feed()` may return zero events (partial line buffered) or multiple events (chunk contained several). The caller does not need to worry about line boundaries. - -Tests: feed partial chunks, verify events emitted at correct boundaries; multi-event in single chunk; empty lines; `data: [DONE]`. - -### `json.zig` - -Two responsibilities: - -1. **Serialize Conversation → OpenAI request body** — Convert our `Message`/`ContentBlock` model into the JSON shape OpenAI expects. See below. -2. **Parse SSE chunk deltas → ContentBlock updates** — Each SSE event's JSON contains a `choices[0].delta` object. Extract text/thinking content from it. - -### `agent.zig` - -The agent loop. In phase 1, it's simple: - -``` -Agent = struct { - provider: Provider, - allocator: std.mem.Allocator, - - pub fn init(allocator, provider) Agent - pub fn runStep(self, conversation: *Conversation, receiver: *Receiver) !void - pub fn deinit(self) void -}; -``` - -`runStep` does: -1. Call `provider.streamStep(conversation, receiver)` — this streams the response and calls the full Receiver callback sequence on the receiver -2. The `onMessageComplete` callback appends the finished Message to the conversation (the agent itself can wire this, or the caller handles it — TBD during implementation) - -In later phases, `runStep` gains the tool-call loop: check for ToolUse blocks, execute them, feed results back, call provider again. But the shape stays the same — one `runStep` invocation carries the conversation through a full agent turn. - -### `config.zig` - -``` -Config = struct { - api_key: []const u8, - base_url: []const u8, // e.g. "https://api.openai.com/v1" - model: []const u8, // e.g. "gpt-4o" -}; -``` - -Populated from environment variables (`PANTO_API_KEY`, `PANTO_BASE_URL`, `PANTO_MODEL`) with defaults for base_url and model. - -### `root.zig` - -Public API. Re-exports the types and functions that external Zig code needs: - -``` -pub const conversation = @import("conversation.zig"); -pub const provider = @import("provider.zig"); -pub const agent = @import("agent.zig"); -pub const config = @import("config.zig"); -``` - -Does not re-export provider_openai, sse, or json — those are internal. - ---- - -## OpenAI Serialization - -### Request - -Our `Conversation` → OpenAI `chat/completions` request body: - -``` -{ - "model": config.model, - "stream": true, - "messages": [ - // For each Message in conversation: - // - // role=.system → { "role": "system", "content": "<concatenated text blocks>" } - // role=.user → { "role": "user", "content": "<concatenated text blocks>" } - // (ToolResult blocks pulled out into separate role:tool messages in phase 3+) - // role=.assistant → { "role": "assistant", "content": [ - // ...text blocks as { "type": "text", "text": "..." }, - // ...thinking blocks as { "type": "thinking", "thinking": "..." }, - // ...tool_use blocks become function_call/function entries in phase 3+ - // ] } - ] -} -``` - -For phase 1, all content blocks we encounter are `Text` or `Thinking`, so serialization is straightforward. - -### Response (streaming) - -Each SSE event is a complete JSON object: - -``` -data: {"id":"chatcmpl-123","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]} -data: {"id":"chatcmpl-123","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]} -data: {"id":"chatcmpl-123","choices":[{"index":0,"delta":{"content":" world"},"finish_reason":null}]} -data: {"id":"chatcmpl-123","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} -data: [DONE] -``` - -We parse each event's `choices[0].delta` and drive the block boundary state machine: -- `delta.role == "assistant"` → emit onMessageStart, marks the start of a new assistant message -- `delta.reasoning_content` → transition to Thinking block if needed, append via onContentDelta -- `delta.content` → transition to Text block if needed, append via onContentDelta -- `delta.tool_calls` → transition to ToolUse block if needed, append arguments via onContentDelta (phase 3+) - -The `finish_reason: "stop"` signals stream end → emit onBlockComplete for any active block, then onMessageComplete. - ---- - -## Minimal CLI - -``` -panto/ - src/ - main.zig // CLI entry point -``` - -Behavior: -1. Read `PANTO_API_KEY`, `PANTO_BASE_URL`, `PANTO_MODEL` from environment -2. Create a Conversation, add a system message (default: "You are a helpful assistant.") -3. Print a prompt (`> `), read a line from stdin -4. Add user message, call `agent.runStep()`, print streamed deltas to stdout -5. Repeat step 3 until EOF (Ctrl+D) - -There is no line editing, no scrolling, no syntax highlighting. Just `readline` → `print`. The sole purpose is exercising libpanto against a real API. - ---- - -## Testing Strategy - -### Unit tests (automated, per module) - -| Module | What to test | -|---|---| -| `conversation.zig` | Create conversation, add messages of each role, verify content block storage, free without leaks | -| `sse.zig` | Feed partial chunks, verify event boundaries; multi-event chunks; `data: [DONE]`; empty lines between events | -| `json.zig` | Serialize conversation → OpenAI JSON; parse delta JSON objects → content updates | -| `config.zig` | Parse from env vars; defaults for missing optional fields | - -### Integration test (manual) - -- Run `panto` binary with a real API key -- Hold a multi-turn conversation -- Verify responses stream to stdout -- Verify follow-up messages include prior context (ask the model "what did I just say?") - ---- - -## Open Questions (to resolve during implementation) - -1. **Thinking token support in OpenAI API**: OpenAI's `reasoning_content` field in streaming deltas is not universally present across models/endpoints. We need to handle its absence gracefully (just skip it, don't crash). -2. **Error handling in streams**: Mid-stream HTTP errors, rate limiting, truncated responses. How do we represent these to the caller? An `onError` callback on the Receiver seems likely. -3. **HTTP connection lifecycle**: Does `std.http.Client` support long-lived streaming connections cleanly? We may need to manage connection pooling or timeouts. -4. **Memory strategy for long conversations**: We're storing full message content in memory. For phase 1 this is fine, but we should define the interface so a later phase can introduce message summarization or offloading without changing the agent loop. |