diff options
Diffstat (limited to 'docs/archive')
| -rw-r--r-- | docs/archive/image-reads.md | 200 |
1 files changed, 200 insertions, 0 deletions
diff --git a/docs/archive/image-reads.md b/docs/archive/image-reads.md new file mode 100644 index 0000000..ab65ee1 --- /dev/null +++ b/docs/archive/image-reads.md @@ -0,0 +1,200 @@ +# Plan: image (and file) upload support for tool results + +## Goal + +Let tools return binary attachments (images, PDFs) — not just text — so the +`read` tool can hand image/PDF files to the model. Anthropic and OpenAI wrap +such content differently in their request JSON, so the tool-result data model +must grow from "one text string" to "a list of typed parts." This is a +first-class libpanto (Zig/C) feature, not a Lua-only convenience. + +## Background / constraints + +- A Lua tool handler currently can only return a string + (`lua_bridge.readHandlerResult` rejects non-strings), and + `conversation.ToolResultBlock.content` is a single `TextualBlock`. +- **Anthropic** allows `tool_result.content` to be an *array* of typed blocks + (`{type:"text"}`, `{type:"image",source:{type:"base64",media_type,data}}`, + and `{type:"document",...}` for PDFs). +- **OpenAI Chat Completions** does **not** allow images in `tool` (or system / + assistant) messages — only in `user` messages. So a tool-returned image + cannot ride inside the `role:"tool"` message; it must be emitted as a + synthetic follow-up `role:"user"` message. +- pi (reference impl) stores image data **inline as base64** in the session + log (no sidecar files) and treats each image as a fixed ~4800-char estimate + for compaction token math. We follow the same approach. + +## Locked decisions + +1. **Module graph flip.** `tool.zig` is the more atomic module; move `ToolDecl` + into `tool.zig` and have `tool_source.zig` import it from there (flips the + current `tool_source -> tool` edge to `tool -> ... ` removed; new edge is + `tool_source -> tool`). The new `ResultPart` type also lives in `tool.zig`. + `conversation.zig` (currently import-free) imports `ResultPart` from + `tool.zig`. Verify no cycle: `tool.zig` must not import `conversation.zig`. + +2. **One `ResultPart` type, two roles.** `ResultPart` is both: + - the element type of `Tool.invoke`'s return value (the widened contract), and + - the in-memory storage in `ToolResultBlock.parts: []ResultPart` + (replacing the single `content: TextualBlock`). + +3. **Contract widening (native Zig/C feature).** + - `Tool.invoke` returns `[]ResultPart` instead of `[]u8`. + - `ToolSource` `CallResult.ok` carries `[]ResultPart` instead of `[]u8`. + - Every native tool and adapter updates to the new return shape. + +4. **Detection: magic bytes** (not file extension). Detect PNG / JPEG / GIF / + BMP / WEBP / PDF from leading bytes; this drives both "is this an + attachment?" and the `media_type` string. + +5. **Resize: libpanto-native, 2000x2000 max, skip when already small.** + - stb single-header trio (`stb_image.h`, `stb_image_resize2.h`, + `stb_image_write.h`) for JPEG / PNG / GIF / BMP — **preserve input codec** + on re-encode. + - `jebp.h` (single-header, **decode-only**) for WEBP. Resized WEBP is + re-encoded as **JPEG (~q80)** since there is no small single-header WEBP + encoder and fidelity is secondary to token size for LLM input. + - **TODO comment** at the WEBP re-encode site: consider re-encoding to PNG + instead of JPEG when the source WEBP has an alpha layer (JPEG has no + transparency; flattening alpha can look wrong for screenshots/diagrams). + - PDFs pass through unresized. + - Skip the decode/resize/encode round-trip entirely when both dimensions + are already <= 2000 (avoid quality loss + CPU). + +6. **Storage: inline base64** (pi-style), no sidecar files. Compaction sizes + each image as a fixed ~4800-char estimate so base64 blobs don't distort the + retention window. + +## ResultPart shape (proposed) + +```zig +// in tool.zig +pub const MediaPart = struct { + media_type: []const u8, // e.g. "image/png", "application/pdf" + data: []const u8, // base64-encoded bytes +}; + +pub const ResultPart = union(enum) { + text: []const u8, + media: MediaPart, +}; +``` + +In-memory conversation storage uses the streaming `TextualBlock` form for the +text part to preserve incremental-append semantics: + +```zig +// in conversation.zig +pub const ResultPartStored = union(enum) { + text: TextualBlock, + media: struct { media_type: []const u8, data: TextualBlock }, +}; +pub const ToolResultBlock = struct { + tool_use_id: []const u8, + parts: std.ArrayList(ResultPartStored), + // ... deinit frees tool_use_id + every part +}; +``` + +(Exact split between the borrowed `ResultPart` contract type and the owned +stored type to be finalized during implementation — the key constraint is +`Tool.invoke` returns owned bytes the agent takes ownership of.) + +## Work breakdown (dependency order) + +### 1. Module graph + types (`tool.zig`, `tool_source.zig`, `conversation.zig`) +- Move `ToolDecl` from `tool_source.zig` to `tool.zig`; update the import edge. +- Add `MediaPart` + `ResultPart` to `tool.zig`. +- Rework `ToolResultBlock` in `conversation.zig` to hold `parts`. Update its + `deinit`. Update `cloneBlock` (agent.zig ~line 86). + +### 2. Widen the native contract (`tool.zig`, `tool_source.zig`) +- `Tool.invoke` return: `[]u8` -> `[]ResultPart`. +- `ToolSource.CallResult.ok`: `[]u8` -> `[]ResultPart`. +- Update vtable doc comments (ownership of part bytes transfers to the agent). + +### 3. Agent assembly (`agent.zig` ~line 550-595) +- Build `ToolResultBlock.parts` from the returned `[]ResultPart` instead of + copying a single byte slice into `content`. +- `FlatCall.result` type follows the contract change. + +### 4. Lua bridge (`lua_bridge.zig`, `lua_runtime.zig`) +- `readHandlerResult`: accept a string (-> one `text` part) OR a table + `{ text = "...", attachments = { { media_type = "...", data = "..." }, ... } }` + (-> one optional text part + media parts). +- Thread parts through `Slot` / `recordResultC` / the legacy sync path + (`invokeCoroutineSync`) instead of `[]u8`. + +### 5. Native image processing (libpanto, new module e.g. `image.zig` + C deps) +- Vendor stb headers + `jebp.h`; wire into `libpanto/build.zig` and + `build.zig.zon` (matching the existing C-dep vendoring pattern). +- `detectMediaType(bytes) -> ?[]const u8` via magic bytes. +- `maybeResize(bytes, media_type) -> owned bytes`: + - non-raster (PDF) or already <=2000x2000: return as-is. + - stb-supported: decode -> resize (stb_image_resize2, Mitchell) -> re-encode + in same codec (stb_image_write). + - WEBP: jebp decode -> resize -> JPEG ~q80 (with the alpha-layer TODO). +- The `read` tool calls this before base64-encoding. + +### 6. Serializers +- **Anthropic** (`anthropic_messages_json.zig`, `ToolResult` branch): emit + `content` as an array — `{type:"text"}` for text parts; `{type:"image", + source:{type:"base64",media_type,data}}` for image media; `{type:"document", + source:{...}}` for PDFs. +- **OpenAI** (`openai_chat_json.zig`): `role:"tool"` message carries text parts + only (plus a short placeholder note when media exists); emit a synthetic + follow-up `role:"user"` message whose content array holds `image_url` + data-URL parts for the media. + +### 7. Session round-trip (`session.zig`) +- `DiskToolResultBlock` gains a `parts` array; `writeDiskBlock` / + `parseDiskBlock` handle text + media (`{type:"image",mimeType,data}` style). + Inline base64, no sidecar. +- Round-trip tests for a tool result containing text + an image part. + +### 8. Compaction (`compaction.zig` / wherever tool-result size is estimated) +- Count a fixed ~4800-char estimate per image part rather than base64 length, + so images don't distort the retention window. **Verify** the exact site + where pantograph sizes tool-result content. + +### 9. read tool (`agent/tools/read.lua`) +- After reading bytes, magic-byte detect. On a recognized binary type, hand the + raw bytes to the libpanto image path (resize happens natively), base64-encode + the result, and return `{ attachments = { { media_type=..., data=... } } }`. + Text path unchanged. +- NOTE: detection + resize is native; the Lua side mostly forwards bytes and + receives processed bytes back. Exact Lua<->libpanto surface for invoking the + native image step to be settled in step 5 (a new `panto.*` bridge fn vs. + doing detection/resize entirely inside a native read tool). + +## Open items — RESOLVED during implementation +- Owned-vs-borrowed split: `tool.ResultPart` (contract) borrows-then-transfers + owned bytes. `media: { media_type: ?[]const u8, data: []const u8 }` carries + **raw, un-encoded file bytes**; `media_type` is an optional hint. The agent + frees via `tool.freeResultParts`. The stored conversation type + `conversation.ResultPartStored` uses `TextualBlock` for text (append + semantics) and resolved `media_type` + base64 `TextualBlock` data for media. +- **libpanto does the heavy lifting, tools stay dumb.** Tools return raw bytes + (optional `media_type` hint). At tool-result assembly (`agent.zig` + `dispatchToolCalls`), libpanto calls `image.process(bytes, hint)` which + magic-byte-detects the type when the hint is absent, resizes large rasters, + and the agent base64-encodes the result for storage. There is **no** + `panto.process_image` bridge — `read.lua` only does a minimal magic-byte + sniff (`is_attachment`) to choose binary-vs-text, then returns raw bytes. + Unrecognized bytes are dropped with a text note rather than aborting the turn. +- WEBP-with-alpha: still re-encodes to JPEG (TODO comment retained at the + re-encode site in `image.zig`). + +## Status +All 9 work-breakdown steps implemented. C deps vendored under +`libpanto/src/cdeps/` (stb trio + jebp), compiled via `image_impl.c` wired into +`libpanto/build.zig`. `zig build` + `zig build test` green at both the libpanto +and top-level layers. + +## Test plan +- Unit: magic-byte detection; resize skip vs. apply; codec preservation; + WEBP->JPEG path. +- Bridge: handler returning string; handler returning `{text,attachments}`. +- Serializers: Anthropic block-array shape; OpenAI tool+synthetic-user split. +- Session: round-trip text+image tool result. +- Contract: a native Zig tool returning `[]ResultPart` with a media part. |