# Plan: image (and file) upload support for tool results

## Goal

Let tools return binary attachments (images, PDFs) — not just text — so the
`read` tool can hand image/PDF files to the model. Anthropic and OpenAI wrap
such content differently in their request JSON, so the tool-result data model
must grow from "one text string" to "a list of typed parts." This is a
first-class libpanto (Zig/C) feature, not a Lua-only convenience.

## Background / constraints

- A Lua tool handler currently can only return a string
  (`lua_bridge.readHandlerResult` rejects non-strings), and
  `conversation.ToolResultBlock.content` is a single `TextualBlock`.
- **Anthropic** allows `tool_result.content` to be an *array* of typed blocks
  (`{type:"text"}`, `{type:"image",source:{type:"base64",media_type,data}}`,
  and `{type:"document",...}` for PDFs).
- **OpenAI Chat Completions** does **not** allow images in `tool` (or system /
  assistant) messages — only in `user` messages. So a tool-returned image
  cannot ride inside the `role:"tool"` message; it must be emitted as a
  synthetic follow-up `role:"user"` message.
- pi (reference impl) stores image data **inline as base64** in the session
  log (no sidecar files) and treats each image as a fixed ~4800-char estimate
  for compaction token math. We follow the same approach.

## Locked decisions

1. **Module graph flip.** `tool.zig` is the more atomic module; move `ToolDecl`
   into `tool.zig` and have `tool_source.zig` import it from there (flips the
   current `tool_source -> tool` edge to `tool -> ... ` removed; new edge is
   `tool_source -> tool`). The new `ResultPart` type also lives in `tool.zig`.
   `conversation.zig` (currently import-free) imports `ResultPart` from
   `tool.zig`. Verify no cycle: `tool.zig` must not import `conversation.zig`.

2. **One `ResultPart` type, two roles.** `ResultPart` is both:
   - the element type of `Tool.invoke`'s return value (the widened contract), and
   - the in-memory storage in `ToolResultBlock.parts: []ResultPart`
     (replacing the single `content: TextualBlock`).

3. **Contract widening (native Zig/C feature).**
   - `Tool.invoke` returns `[]ResultPart` instead of `[]u8`.
   - `ToolSource` `CallResult.ok` carries `[]ResultPart` instead of `[]u8`.
   - Every native tool and adapter updates to the new return shape.

4. **Detection: magic bytes** (not file extension). Detect PNG / JPEG / GIF /
   BMP / WEBP / PDF from leading bytes; this drives both "is this an
   attachment?" and the `media_type` string.

5. **Resize: libpanto-native, 2000x2000 max, skip when already small.**
   - stb single-header trio (`stb_image.h`, `stb_image_resize2.h`,
     `stb_image_write.h`) for JPEG / PNG / GIF / BMP — **preserve input codec**
     on re-encode.
   - `jebp.h` (single-header, **decode-only**) for WEBP. Resized WEBP is
     re-encoded as **JPEG (~q80)** since there is no small single-header WEBP
     encoder and fidelity is secondary to token size for LLM input.
   - **TODO comment** at the WEBP re-encode site: consider re-encoding to PNG
     instead of JPEG when the source WEBP has an alpha layer (JPEG has no
     transparency; flattening alpha can look wrong for screenshots/diagrams).
   - PDFs pass through unresized.
   - Skip the decode/resize/encode round-trip entirely when both dimensions
     are already <= 2000 (avoid quality loss + CPU).

6. **Storage: inline base64** (pi-style), no sidecar files. Compaction sizes
   each image as a fixed ~4800-char estimate so base64 blobs don't distort the
   retention window.

## ResultPart shape (proposed)

```zig
// in tool.zig
pub const MediaPart = struct {
    media_type: []const u8, // e.g. "image/png", "application/pdf"
    data: []const u8,       // base64-encoded bytes
};

pub const ResultPart = union(enum) {
    text: []const u8,
    media: MediaPart,
};
```

In-memory conversation storage uses the streaming `TextualBlock` form for the
text part to preserve incremental-append semantics:

```zig
// in conversation.zig
pub const ResultPartStored = union(enum) {
    text: TextualBlock,
    media: struct { media_type: []const u8, data: TextualBlock },
};
pub const ToolResultBlock = struct {
    tool_use_id: []const u8,
    parts: std.ArrayList(ResultPartStored),
    // ... deinit frees tool_use_id + every part
};
```

(Exact split between the borrowed `ResultPart` contract type and the owned
stored type to be finalized during implementation — the key constraint is
`Tool.invoke` returns owned bytes the agent takes ownership of.)

## Work breakdown (dependency order)

### 1. Module graph + types (`tool.zig`, `tool_source.zig`, `conversation.zig`)
- Move `ToolDecl` from `tool_source.zig` to `tool.zig`; update the import edge.
- Add `MediaPart` + `ResultPart` to `tool.zig`.
- Rework `ToolResultBlock` in `conversation.zig` to hold `parts`. Update its
  `deinit`. Update `cloneBlock` (agent.zig ~line 86).

### 2. Widen the native contract (`tool.zig`, `tool_source.zig`)
- `Tool.invoke` return: `[]u8` -> `[]ResultPart`.
- `ToolSource.CallResult.ok`: `[]u8` -> `[]ResultPart`.
- Update vtable doc comments (ownership of part bytes transfers to the agent).

### 3. Agent assembly (`agent.zig` ~line 550-595)
- Build `ToolResultBlock.parts` from the returned `[]ResultPart` instead of
  copying a single byte slice into `content`.
- `FlatCall.result` type follows the contract change.

### 4. Lua bridge (`lua_bridge.zig`, `lua_runtime.zig`)
- `readHandlerResult`: accept a string (-> one `text` part) OR a table
  `{ text = "...", attachments = { { media_type = "...", data = "..." }, ... } }`
  (-> one optional text part + media parts).
- Thread parts through `Slot` / `recordResultC` / the legacy sync path
  (`invokeCoroutineSync`) instead of `[]u8`.

### 5. Native image processing (libpanto, new module e.g. `image.zig` + C deps)
- Vendor stb headers + `jebp.h`; wire into `libpanto/build.zig` and
  `build.zig.zon` (matching the existing C-dep vendoring pattern).
- `detectMediaType(bytes) -> ?[]const u8` via magic bytes.
- `maybeResize(bytes, media_type) -> owned bytes`:
  - non-raster (PDF) or already <=2000x2000: return as-is.
  - stb-supported: decode -> resize (stb_image_resize2, Mitchell) -> re-encode
    in same codec (stb_image_write).
  - WEBP: jebp decode -> resize -> JPEG ~q80 (with the alpha-layer TODO).
- The `read` tool calls this before base64-encoding.

### 6. Serializers
- **Anthropic** (`anthropic_messages_json.zig`, `ToolResult` branch): emit
  `content` as an array — `{type:"text"}` for text parts; `{type:"image",
  source:{type:"base64",media_type,data}}` for image media; `{type:"document",
  source:{...}}` for PDFs.
- **OpenAI** (`openai_chat_json.zig`): `role:"tool"` message carries text parts
  only (plus a short placeholder note when media exists); emit a synthetic
  follow-up `role:"user"` message whose content array holds `image_url`
  data-URL parts for the media.

### 7. Session round-trip (`session.zig`)
- `DiskToolResultBlock` gains a `parts` array; `writeDiskBlock` /
  `parseDiskBlock` handle text + media (`{type:"image",mimeType,data}` style).
  Inline base64, no sidecar.
- Round-trip tests for a tool result containing text + an image part.

### 8. Compaction (`compaction.zig` / wherever tool-result size is estimated)
- Count a fixed ~4800-char estimate per image part rather than base64 length,
  so images don't distort the retention window. **Verify** the exact site
  where pantograph sizes tool-result content.

### 9. read tool (`agent/tools/read.lua`)
- After reading bytes, magic-byte detect. On a recognized binary type, hand the
  raw bytes to the libpanto image path (resize happens natively), base64-encode
  the result, and return `{ attachments = { { media_type=..., data=... } } }`.
  Text path unchanged.
- NOTE: detection + resize is native; the Lua side mostly forwards bytes and
  receives processed bytes back. Exact Lua<->libpanto surface for invoking the
  native image step to be settled in step 5 (a new `panto.*` bridge fn vs.
  doing detection/resize entirely inside a native read tool).

## Open items — RESOLVED during implementation
- Owned-vs-borrowed split: `tool.ResultPart` (contract) borrows-then-transfers
  owned bytes. `media: { media_type: ?[]const u8, data: []const u8 }` carries
  **raw, un-encoded file bytes**; `media_type` is an optional hint. The agent
  frees via `tool.freeResultParts`. The stored conversation type
  `conversation.ResultPartStored` uses `TextualBlock` for text (append
  semantics) and resolved `media_type` + base64 `TextualBlock` data for media.
- **libpanto does the heavy lifting, tools stay dumb.** Tools return raw bytes
  (optional `media_type` hint). At tool-result assembly (`agent.zig`
  `dispatchToolCalls`), libpanto calls `image.process(bytes, hint)` which
  magic-byte-detects the type when the hint is absent, resizes large rasters,
  and the agent base64-encodes the result for storage. There is **no**
  `panto.process_image` bridge — `read.lua` only does a minimal magic-byte
  sniff (`is_attachment`) to choose binary-vs-text, then returns raw bytes.
  Unrecognized bytes are dropped with a text note rather than aborting the turn.
- WEBP-with-alpha: still re-encodes to JPEG (TODO comment retained at the
  re-encode site in `image.zig`).

## Status
All 9 work-breakdown steps implemented. C deps vendored under
`libpanto/src/cdeps/` (stb trio + jebp), compiled via `image_impl.c` wired into
`libpanto/build.zig`. `zig build` + `zig build test` green at both the libpanto
and top-level layers.

## Test plan
- Unit: magic-byte detection; resize skip vs. apply; codec preservation;
  WEBP->JPEG path.
- Bridge: handler returning string; handler returning `{text,attachments}`.
- Serializers: Anthropic block-array shape; OpenAI tool+synthetic-user split.
- Session: round-trip text+image tool result.
- Contract: a native Zig tool returning `[]ResultPart` with a media part.