1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
|
# Plan: image (and file) upload support for tool results
## Goal
Let tools return binary attachments (images, PDFs) — not just text — so the
`read` tool can hand image/PDF files to the model. Anthropic and OpenAI wrap
such content differently in their request JSON, so the tool-result data model
must grow from "one text string" to "a list of typed parts." This is a
first-class libpanto (Zig/C) feature, not a Lua-only convenience.
## Background / constraints
- A Lua tool handler currently can only return a string
(`lua_bridge.readHandlerResult` rejects non-strings), and
`conversation.ToolResultBlock.content` is a single `TextualBlock`.
- **Anthropic** allows `tool_result.content` to be an *array* of typed blocks
(`{type:"text"}`, `{type:"image",source:{type:"base64",media_type,data}}`,
and `{type:"document",...}` for PDFs).
- **OpenAI Chat Completions** does **not** allow images in `tool` (or system /
assistant) messages — only in `user` messages. So a tool-returned image
cannot ride inside the `role:"tool"` message; it must be emitted as a
synthetic follow-up `role:"user"` message.
- pi (reference impl) stores image data **inline as base64** in the session
log (no sidecar files) and treats each image as a fixed ~4800-char estimate
for compaction token math. We follow the same approach.
## Locked decisions
1. **Module graph flip.** `tool.zig` is the more atomic module; move `ToolDecl`
into `tool.zig` and have `tool_source.zig` import it from there (flips the
current `tool_source -> tool` edge to `tool -> ... ` removed; new edge is
`tool_source -> tool`). The new `ResultPart` type also lives in `tool.zig`.
`conversation.zig` (currently import-free) imports `ResultPart` from
`tool.zig`. Verify no cycle: `tool.zig` must not import `conversation.zig`.
2. **One `ResultPart` type, two roles.** `ResultPart` is both:
- the element type of `Tool.invoke`'s return value (the widened contract), and
- the in-memory storage in `ToolResultBlock.parts: []ResultPart`
(replacing the single `content: TextualBlock`).
3. **Contract widening (native Zig/C feature).**
- `Tool.invoke` returns `[]ResultPart` instead of `[]u8`.
- `ToolSource` `CallResult.ok` carries `[]ResultPart` instead of `[]u8`.
- Every native tool and adapter updates to the new return shape.
4. **Detection: magic bytes** (not file extension). Detect PNG / JPEG / GIF /
BMP / WEBP / PDF from leading bytes; this drives both "is this an
attachment?" and the `media_type` string.
5. **Resize: libpanto-native, 2000x2000 max, skip when already small.**
- stb single-header trio (`stb_image.h`, `stb_image_resize2.h`,
`stb_image_write.h`) for JPEG / PNG / GIF / BMP — **preserve input codec**
on re-encode.
- `jebp.h` (single-header, **decode-only**) for WEBP. Resized WEBP is
re-encoded as **JPEG (~q80)** since there is no small single-header WEBP
encoder and fidelity is secondary to token size for LLM input.
- **TODO comment** at the WEBP re-encode site: consider re-encoding to PNG
instead of JPEG when the source WEBP has an alpha layer (JPEG has no
transparency; flattening alpha can look wrong for screenshots/diagrams).
- PDFs pass through unresized.
- Skip the decode/resize/encode round-trip entirely when both dimensions
are already <= 2000 (avoid quality loss + CPU).
6. **Storage: inline base64** (pi-style), no sidecar files. Compaction sizes
each image as a fixed ~4800-char estimate so base64 blobs don't distort the
retention window.
## ResultPart shape (proposed)
```zig
// in tool.zig
pub const MediaPart = struct {
media_type: []const u8, // e.g. "image/png", "application/pdf"
data: []const u8, // base64-encoded bytes
};
pub const ResultPart = union(enum) {
text: []const u8,
media: MediaPart,
};
```
In-memory conversation storage uses the streaming `TextualBlock` form for the
text part to preserve incremental-append semantics:
```zig
// in conversation.zig
pub const ResultPartStored = union(enum) {
text: TextualBlock,
media: struct { media_type: []const u8, data: TextualBlock },
};
pub const ToolResultBlock = struct {
tool_use_id: []const u8,
parts: std.ArrayList(ResultPartStored),
// ... deinit frees tool_use_id + every part
};
```
(Exact split between the borrowed `ResultPart` contract type and the owned
stored type to be finalized during implementation — the key constraint is
`Tool.invoke` returns owned bytes the agent takes ownership of.)
## Work breakdown (dependency order)
### 1. Module graph + types (`tool.zig`, `tool_source.zig`, `conversation.zig`)
- Move `ToolDecl` from `tool_source.zig` to `tool.zig`; update the import edge.
- Add `MediaPart` + `ResultPart` to `tool.zig`.
- Rework `ToolResultBlock` in `conversation.zig` to hold `parts`. Update its
`deinit`. Update `cloneBlock` (agent.zig ~line 86).
### 2. Widen the native contract (`tool.zig`, `tool_source.zig`)
- `Tool.invoke` return: `[]u8` -> `[]ResultPart`.
- `ToolSource.CallResult.ok`: `[]u8` -> `[]ResultPart`.
- Update vtable doc comments (ownership of part bytes transfers to the agent).
### 3. Agent assembly (`agent.zig` ~line 550-595)
- Build `ToolResultBlock.parts` from the returned `[]ResultPart` instead of
copying a single byte slice into `content`.
- `FlatCall.result` type follows the contract change.
### 4. Lua bridge (`lua_bridge.zig`, `lua_runtime.zig`)
- `readHandlerResult`: accept a string (-> one `text` part) OR a table
`{ text = "...", attachments = { { media_type = "...", data = "..." }, ... } }`
(-> one optional text part + media parts).
- Thread parts through `Slot` / `recordResultC` / the legacy sync path
(`invokeCoroutineSync`) instead of `[]u8`.
### 5. Native image processing (libpanto, new module e.g. `image.zig` + C deps)
- Vendor stb headers + `jebp.h`; wire into `libpanto/build.zig` and
`build.zig.zon` (matching the existing C-dep vendoring pattern).
- `detectMediaType(bytes) -> ?[]const u8` via magic bytes.
- `maybeResize(bytes, media_type) -> owned bytes`:
- non-raster (PDF) or already <=2000x2000: return as-is.
- stb-supported: decode -> resize (stb_image_resize2, Mitchell) -> re-encode
in same codec (stb_image_write).
- WEBP: jebp decode -> resize -> JPEG ~q80 (with the alpha-layer TODO).
- The `read` tool calls this before base64-encoding.
### 6. Serializers
- **Anthropic** (`anthropic_messages_json.zig`, `ToolResult` branch): emit
`content` as an array — `{type:"text"}` for text parts; `{type:"image",
source:{type:"base64",media_type,data}}` for image media; `{type:"document",
source:{...}}` for PDFs.
- **OpenAI** (`openai_chat_json.zig`): `role:"tool"` message carries text parts
only (plus a short placeholder note when media exists); emit a synthetic
follow-up `role:"user"` message whose content array holds `image_url`
data-URL parts for the media.
### 7. Session round-trip (`session.zig`)
- `DiskToolResultBlock` gains a `parts` array; `writeDiskBlock` /
`parseDiskBlock` handle text + media (`{type:"image",mimeType,data}` style).
Inline base64, no sidecar.
- Round-trip tests for a tool result containing text + an image part.
### 8. Compaction (`compaction.zig` / wherever tool-result size is estimated)
- Count a fixed ~4800-char estimate per image part rather than base64 length,
so images don't distort the retention window. **Verify** the exact site
where pantograph sizes tool-result content.
### 9. read tool (`agent/tools/read.lua`)
- After reading bytes, magic-byte detect. On a recognized binary type, hand the
raw bytes to the libpanto image path (resize happens natively), base64-encode
the result, and return `{ attachments = { { media_type=..., data=... } } }`.
Text path unchanged.
- NOTE: detection + resize is native; the Lua side mostly forwards bytes and
receives processed bytes back. Exact Lua<->libpanto surface for invoking the
native image step to be settled in step 5 (a new `panto.*` bridge fn vs.
doing detection/resize entirely inside a native read tool).
## Open items — RESOLVED during implementation
- Owned-vs-borrowed split: `tool.ResultPart` (contract) borrows-then-transfers
owned bytes. `media: { media_type: ?[]const u8, data: []const u8 }` carries
**raw, un-encoded file bytes**; `media_type` is an optional hint. The agent
frees via `tool.freeResultParts`. The stored conversation type
`conversation.ResultPartStored` uses `TextualBlock` for text (append
semantics) and resolved `media_type` + base64 `TextualBlock` data for media.
- **libpanto does the heavy lifting, tools stay dumb.** Tools return raw bytes
(optional `media_type` hint). At tool-result assembly (`agent.zig`
`dispatchToolCalls`), libpanto calls `image.process(bytes, hint)` which
magic-byte-detects the type when the hint is absent, resizes large rasters,
and the agent base64-encodes the result for storage. There is **no**
`panto.process_image` bridge — `read.lua` only does a minimal magic-byte
sniff (`is_attachment`) to choose binary-vs-text, then returns raw bytes.
Unrecognized bytes are dropped with a text note rather than aborting the turn.
- WEBP-with-alpha: still re-encodes to JPEG (TODO comment retained at the
re-encode site in `image.zig`).
## Status
All 9 work-breakdown steps implemented. C deps vendored under
`libpanto/src/cdeps/` (stb trio + jebp), compiled via `image_impl.c` wired into
`libpanto/build.zig`. `zig build` + `zig build test` green at both the libpanto
and top-level layers.
## Test plan
- Unit: magic-byte detection; resize skip vs. apply; codec preservation;
WEBP->JPEG path.
- Bridge: handler returning string; handler returning `{text,attachments}`.
- Serializers: Anthropic block-array shape; OpenAI tool+synthetic-user split.
- Session: round-trip text+image tool result.
- Contract: a native Zig tool returning `[]ResultPart` with a media part.
|
