image uploads

1 files changed, 200 insertions, 0 deletions

diff --git a/docs/archive/image-reads.md b/docs/archive/image-reads.md
new file mode 100644
index 0000000..ab65ee1
--- /dev/null
+++ b/docs/archive/image-reads.md
@@ -0,0 +1,200 @@
+# Plan: image (and file) upload support for tool results
+
+## Goal
+
+Let tools return binary attachments (images, PDFs) — not just text — so the
+`read` tool can hand image/PDF files to the model. Anthropic and OpenAI wrap
+such content differently in their request JSON, so the tool-result data model
+must grow from "one text string" to "a list of typed parts." This is a
+first-class libpanto (Zig/C) feature, not a Lua-only convenience.
+
+## Background / constraints
+
+- A Lua tool handler currently can only return a string
+  (`lua_bridge.readHandlerResult` rejects non-strings), and
+  `conversation.ToolResultBlock.content` is a single `TextualBlock`.
+- **Anthropic** allows `tool_result.content` to be an *array* of typed blocks
+  (`{type:"text"}`, `{type:"image",source:{type:"base64",media_type,data}}`,
+  and `{type:"document",...}` for PDFs).
+- **OpenAI Chat Completions** does **not** allow images in `tool` (or system /
+  assistant) messages — only in `user` messages. So a tool-returned image
+  cannot ride inside the `role:"tool"` message; it must be emitted as a
+  synthetic follow-up `role:"user"` message.
+- pi (reference impl) stores image data **inline as base64** in the session
+  log (no sidecar files) and treats each image as a fixed ~4800-char estimate
+  for compaction token math. We follow the same approach.
+
+## Locked decisions
+
+1. **Module graph flip.** `tool.zig` is the more atomic module; move `ToolDecl`
+   into `tool.zig` and have `tool_source.zig` import it from there (flips the
+   current `tool_source -> tool` edge to `tool -> ... ` removed; new edge is
+   `tool_source -> tool`). The new `ResultPart` type also lives in `tool.zig`.
+   `conversation.zig` (currently import-free) imports `ResultPart` from
+   `tool.zig`. Verify no cycle: `tool.zig` must not import `conversation.zig`.
+
+2. **One `ResultPart` type, two roles.** `ResultPart` is both:
+   - the element type of `Tool.invoke`'s return value (the widened contract), and
+   - the in-memory storage in `ToolResultBlock.parts: []ResultPart`
+     (replacing the single `content: TextualBlock`).
+
+3. **Contract widening (native Zig/C feature).**
+   - `Tool.invoke` returns `[]ResultPart` instead of `[]u8`.
+   - `ToolSource` `CallResult.ok` carries `[]ResultPart` instead of `[]u8`.
+   - Every native tool and adapter updates to the new return shape.
+
+4. **Detection: magic bytes** (not file extension). Detect PNG / JPEG / GIF /
+   BMP / WEBP / PDF from leading bytes; this drives both "is this an
+   attachment?" and the `media_type` string.
+
+5. **Resize: libpanto-native, 2000x2000 max, skip when already small.**
+   - stb single-header trio (`stb_image.h`, `stb_image_resize2.h`,
+     `stb_image_write.h`) for JPEG / PNG / GIF / BMP — **preserve input codec**
+     on re-encode.
+   - `jebp.h` (single-header, **decode-only**) for WEBP. Resized WEBP is
+     re-encoded as **JPEG (~q80)** since there is no small single-header WEBP
+     encoder and fidelity is secondary to token size for LLM input.
+   - **TODO comment** at the WEBP re-encode site: consider re-encoding to PNG
+     instead of JPEG when the source WEBP has an alpha layer (JPEG has no
+     transparency; flattening alpha can look wrong for screenshots/diagrams).
+   - PDFs pass through unresized.
+   - Skip the decode/resize/encode round-trip entirely when both dimensions
+     are already <= 2000 (avoid quality loss + CPU).
+
+6. **Storage: inline base64** (pi-style), no sidecar files. Compaction sizes
+   each image as a fixed ~4800-char estimate so base64 blobs don't distort the
+   retention window.
+
+## ResultPart shape (proposed)
+
+```zig
+// in tool.zig
+pub const MediaPart = struct {
+    media_type: []const u8, // e.g. "image/png", "application/pdf"
+    data: []const u8,       // base64-encoded bytes
+};
+
+pub const ResultPart = union(enum) {
+    text: []const u8,
+    media: MediaPart,
+};
+```
+
+In-memory conversation storage uses the streaming `TextualBlock` form for the
+text part to preserve incremental-append semantics:
+
+```zig
+// in conversation.zig
+pub const ResultPartStored = union(enum) {
+    text: TextualBlock,
+    media: struct { media_type: []const u8, data: TextualBlock },
+};
+pub const ToolResultBlock = struct {
+    tool_use_id: []const u8,
+    parts: std.ArrayList(ResultPartStored),
+    // ... deinit frees tool_use_id + every part
+};
+```
+
+(Exact split between the borrowed `ResultPart` contract type and the owned
+stored type to be finalized during implementation — the key constraint is
+`Tool.invoke` returns owned bytes the agent takes ownership of.)
+
+## Work breakdown (dependency order)
+
+### 1. Module graph + types (`tool.zig`, `tool_source.zig`, `conversation.zig`)
+- Move `ToolDecl` from `tool_source.zig` to `tool.zig`; update the import edge.
+- Add `MediaPart` + `ResultPart` to `tool.zig`.
+- Rework `ToolResultBlock` in `conversation.zig` to hold `parts`. Update its
+  `deinit`. Update `cloneBlock` (agent.zig ~line 86).
+
+### 2. Widen the native contract (`tool.zig`, `tool_source.zig`)
+- `Tool.invoke` return: `[]u8` -> `[]ResultPart`.
+- `ToolSource.CallResult.ok`: `[]u8` -> `[]ResultPart`.
+- Update vtable doc comments (ownership of part bytes transfers to the agent).
+
+### 3. Agent assembly (`agent.zig` ~line 550-595)
+- Build `ToolResultBlock.parts` from the returned `[]ResultPart` instead of
+  copying a single byte slice into `content`.
+- `FlatCall.result` type follows the contract change.
+
+### 4. Lua bridge (`lua_bridge.zig`, `lua_runtime.zig`)
+- `readHandlerResult`: accept a string (-> one `text` part) OR a table
+  `{ text = "...", attachments = { { media_type = "...", data = "..." }, ... } }`
+  (-> one optional text part + media parts).
+- Thread parts through `Slot` / `recordResultC` / the legacy sync path
+  (`invokeCoroutineSync`) instead of `[]u8`.
+
+### 5. Native image processing (libpanto, new module e.g. `image.zig` + C deps)
+- Vendor stb headers + `jebp.h`; wire into `libpanto/build.zig` and
+  `build.zig.zon` (matching the existing C-dep vendoring pattern).
+- `detectMediaType(bytes) -> ?[]const u8` via magic bytes.
+- `maybeResize(bytes, media_type) -> owned bytes`:
+  - non-raster (PDF) or already <=2000x2000: return as-is.
+  - stb-supported: decode -> resize (stb_image_resize2, Mitchell) -> re-encode
+    in same codec (stb_image_write).
+  - WEBP: jebp decode -> resize -> JPEG ~q80 (with the alpha-layer TODO).
+- The `read` tool calls this before base64-encoding.
+
+### 6. Serializers
+- **Anthropic** (`anthropic_messages_json.zig`, `ToolResult` branch): emit
+  `content` as an array — `{type:"text"}` for text parts; `{type:"image",
+  source:{type:"base64",media_type,data}}` for image media; `{type:"document",
+  source:{...}}` for PDFs.
+- **OpenAI** (`openai_chat_json.zig`): `role:"tool"` message carries text parts
+  only (plus a short placeholder note when media exists); emit a synthetic
+  follow-up `role:"user"` message whose content array holds `image_url`
+  data-URL parts for the media.
+
+### 7. Session round-trip (`session.zig`)
+- `DiskToolResultBlock` gains a `parts` array; `writeDiskBlock` /
+  `parseDiskBlock` handle text + media (`{type:"image",mimeType,data}` style).
+  Inline base64, no sidecar.
+- Round-trip tests for a tool result containing text + an image part.
+
+### 8. Compaction (`compaction.zig` / wherever tool-result size is estimated)
+- Count a fixed ~4800-char estimate per image part rather than base64 length,
+  so images don't distort the retention window. **Verify** the exact site
+  where pantograph sizes tool-result content.
+
+### 9. read tool (`agent/tools/read.lua`)
+- After reading bytes, magic-byte detect. On a recognized binary type, hand the
+  raw bytes to the libpanto image path (resize happens natively), base64-encode
+  the result, and return `{ attachments = { { media_type=..., data=... } } }`.
+  Text path unchanged.
+- NOTE: detection + resize is native; the Lua side mostly forwards bytes and
+  receives processed bytes back. Exact Lua<->libpanto surface for invoking the
+  native image step to be settled in step 5 (a new `panto.*` bridge fn vs.
+  doing detection/resize entirely inside a native read tool).
+
+## Open items — RESOLVED during implementation
+- Owned-vs-borrowed split: `tool.ResultPart` (contract) borrows-then-transfers
+  owned bytes. `media: { media_type: ?[]const u8, data: []const u8 }` carries
+  **raw, un-encoded file bytes**; `media_type` is an optional hint. The agent
+  frees via `tool.freeResultParts`. The stored conversation type
+  `conversation.ResultPartStored` uses `TextualBlock` for text (append
+  semantics) and resolved `media_type` + base64 `TextualBlock` data for media.
+- **libpanto does the heavy lifting, tools stay dumb.** Tools return raw bytes
+  (optional `media_type` hint). At tool-result assembly (`agent.zig`
+  `dispatchToolCalls`), libpanto calls `image.process(bytes, hint)` which
+  magic-byte-detects the type when the hint is absent, resizes large rasters,
+  and the agent base64-encodes the result for storage. There is **no**
+  `panto.process_image` bridge — `read.lua` only does a minimal magic-byte
+  sniff (`is_attachment`) to choose binary-vs-text, then returns raw bytes.
+  Unrecognized bytes are dropped with a text note rather than aborting the turn.
+- WEBP-with-alpha: still re-encodes to JPEG (TODO comment retained at the
+  re-encode site in `image.zig`).
+
+## Status
+All 9 work-breakdown steps implemented. C deps vendored under
+`libpanto/src/cdeps/` (stb trio + jebp), compiled via `image_impl.c` wired into
+`libpanto/build.zig`. `zig build` + `zig build test` green at both the libpanto
+and top-level layers.
+
+## Test plan
+- Unit: magic-byte detection; resize skip vs. apply; codec preservation;
+  WEBP->JPEG path.
+- Bridge: handler returning string; handler returning `{text,attachments}`.
+- Serializers: Anthropic block-array shape; OpenAI tool+synthetic-user split.
+- Session: round-trip text+image tool result.
+- Contract: a native Zig tool returning `[]ResultPart` with a media part.