path: root/docs/archive/retries.md

# Message-level error retries

## Summary

The agent loop should make an explicit decision for every failure that occurs while producing the next conversation message. Depending on the failure, the agent should either:

1. retry the same provider request with backoff,
2. compact/rewrite the conversation and retry the same provider request,
3. append conversation-visible tool error results and continue the agent loop, or
4. hard-fail and return the error to the embedder.

The important distinction is between provider/API failures and tool-call failures:

- Provider/API failures usually mean the assistant message was not produced. The correct recovery is to retry the same request without mutating the conversation.
- Tool-call failures usually mean the model made a recoverable choice. The correct recovery is to append `ToolResult` blocks describing the failures, then let the model continue from there.

This plan treats retries as **message-level**: a failed provider attempt should not create a partial duplicate message, and a tool failure should be represented as the next conversation message in the normal assistant/tool-result exchange.

## Existing behavior

We already have several special cases:

- Context overflow errors are handled by automatic compaction followed by one retry of the same request.
- Max-tokens overflow during tool argument generation can surface as a partially generated tool input; this is currently handled as invalid/incomplete JSON-ish tool input and sent back as a tool result.
- Some tool-result post-processing failures are already degraded into text, e.g. unrecognized binary attachments are dropped with a note.

However, error handling is not yet holistic:

- Provider HTTP failures are often collapsed into broad errors such as `HttpError`, losing retryability information.
- Transient provider/API failures do not have a general backoff retry loop.
- Tool execution errors usually abort the agent loop instead of becoming tool error results.
- Retry attempts are not surfaced to the UI, which makes coding-agent behavior opaque during provider outages or rate limits.

## Error actions

### 1. Retry the same provider request

Use this when the provider request appears transiently unavailable or unreliable, and the conversation has not gained a completed assistant message.

Examples:

- HTTP 429 rate limit.
- HTTP 408, 409, 425 where retry is appropriate.
- HTTP 500, 502, 503, 504.
- Connection reset, DNS/connect failure, TLS failure, timeout.
- Stream interruption or malformed provider stream before a complete assistant message.
- Possibly an empty assistant response, with a small retry cap.

Behavior:

- Do not append or modify conversation messages.
- Retry the same request against the same conversation snapshot.
- Use exponential backoff with jitter.
- Honor `Retry-After` when the provider sends it.
- Stop after a configurable attempt cap and then hard-fail.
- Notify the receiver/UI before each delayed retry.

### 2. Compact/rewrite context, then retry the same provider request

Use this when the provider rejects the request because the input context is too large.

Examples:

- OpenAI-compatible `context_length_exceeded` / `maximum context length` responses.
- Anthropic `prompt is too long` responses.
- Other provider-specific context-window diagnostics.

Behavior:

- Run automatic compaction if configured.
- Rewrite the conversation into the compacted form.
- Retry the same request once against the compacted conversation.
- If compaction cannot shed enough context, or if the retry also overflows, hard-fail.

This is already mostly implemented; it should be integrated into the same provider-error decision path as ordinary retries.

### 3. Append tool error results and continue

Use this when the model emitted a tool call and the failure is meaningful to the model.

Examples:

- Unknown tool name.
- Invalid/incomplete tool input.
- Future schema validation failures.
- Native tool handler returned an error.
- Lua/runtime exception.
- `ToolSource` per-call error.
- `ToolSource` whole-batch failure, mapped to each affected call.
- Tool output shape/encoding problems that can be explained to the model.
- Media processing failures where the rest of the agent is healthy.

Behavior:

- Append a user message containing one `ToolResult` for every `ToolUse` block, preserving original call order.
- Successful calls keep their normal results.
- Failed calls get model-readable error text.
- The agent loop continues, allowing the model to correct arguments, try a different tool, or explain the failure to the user.
- Prefer adding an internal `is_error` flag to tool results, serialized where supported.

Example error text:

```text
Tool execution failed for `std.read`: file not found: src/foo.zig
You may fix the arguments, try a different tool, or explain the failure to the user.
```

Provider serialization:

- Anthropic supports an `is_error`-style tool result marker; serialize it when present.
- OpenAI Chat Completions has no direct equivalent; include the error as textual tool content.

### 4. Hard-fail

Use this when retrying would be unsafe or misleading, or when the failure belongs to the host application rather than the model/provider interaction.

Examples:

- User cancellation / abort.
- Out of memory.
- Receiver/UI/session persistence failure.
- Invalid local configuration: missing API key, malformed URL, unknown model alias.
- Provider authentication/authorization failure: HTTP 401/403.
- Non-retryable provider request errors: most HTTP 400s except context overflow.
- Internal invariant violation / conversation corruption.
- Concurrency unavailable, unless we add a sequential fallback.

Behavior:

- Return the error to the embedder.
- Do not synthesize a model-visible tool result for host/system failures.

## Provider retry notifications

Retry visibility should be front-loaded. Coding-agent UIs need to show that the agent is waiting on the provider rather than silently hanging.

Add a receiver callback for provider retry scheduling, something like:

```zig
onProviderRetry(info: ProviderRetryInfo) void
```

Possible fields:

```zig
const ProviderRetryInfo = struct {
    attempt: usize,          // next attempt number, 1-based or 0-based; define clearly
    max_attempts: usize,
    delay_ms: u64,
    err: anyerror,
    status_code: ?u16 = null,
    retry_after_ms: ?u64 = null,
    message: ?[]const u8 = null,
};
```

Semantics:

- Fire after a provider attempt fails and before sleeping for the next attempt.
- Fire for ordinary provider retry/backoff.
- Fire for context-overflow compaction retry too, either through this callback or through a sibling callback that says compaction is being attempted.
- Do not fire for tool-call failures that are being sent back as tool results; those are normal conversation continuations.

Initial CLI behavior can be simple, e.g. print a dim status line:

```text
provider unavailable: retrying in 2.4s (attempt 2/4)
```

## Provider error classification

The providers should stop collapsing all HTTP/API failures into a single broad error when the agent needs to distinguish retryable from non-retryable failures.

Minimum useful public errors:

```zig
error.ProviderRateLimited
error.ProviderUnavailable
error.ProviderServerError
error.ProviderTransport
error.ProviderStreamMalformed
error.ProviderAuthFailed
error.ProviderBadRequest
error.ProviderModelNotFound
error.ContextOverflow
```

Suggested HTTP mapping:

- `400` + context marker: `ContextOverflow`
- other `400`: `ProviderBadRequest`
- `401`, `403`: `ProviderAuthFailed`
- `404`: `ProviderModelNotFound` or `ProviderBadRequest`, depending on provider/API shape
- `408`: retryable transport/request timeout
- `409`, `425`: retryable if provider semantics indicate temporary conflict/not-ready
- `429`: `ProviderRateLimited`
- `500..599`: `ProviderServerError` / `ProviderUnavailable`

Longer term, `anyerror` alone is too lossy for status code, provider message, and `Retry-After`. A richer provider error/reporting channel would be useful. For the first implementation, distinct error names plus logging may be enough.

## Agent retry policy

Add a retry policy to config, with conservative defaults.

Possible shape:

```zig
const RetryConfig = struct {
    max_attempts: usize = 4,
    initial_delay_ms: u64 = 500,
    max_delay_ms: u64 = 10_000,
    multiplier: f64 = 2.0,
    jitter: bool = true,
};
```

Rules:

- Attempt count includes the first attempt; `max_attempts = 4` means one initial try plus up to three retries.
- Retry only errors classified as retryable provider failures.
- Never retry cancellation, local receiver errors, auth failures, or bad local configuration.
- If the provider supplies `Retry-After`, prefer that delay, capped by policy.
- Keep context-overflow compaction retry as a separate one-shot path, not an unbounded retry loop.

## Tool error result policy

Change tool dispatch so normal tool failures do not abort the turn.

Current behavior to replace:

- Unknown tool returns `error.UnknownTool`.
- `Tool.invoke` error records `FlatCall.err`; after dispatch, the first error aborts the turn.
- `ToolSource` per-call errors and whole-batch errors abort the turn.

Desired behavior:

- Unknown tool becomes an error `ToolResult`.
- Invalid/incomplete input remains an error `ToolResult`.
- Native tool errors become error `ToolResult`s unless classified as hard host failures.
- Source per-call errors become per-call error `ToolResult`s.
- Source whole-batch errors become error `ToolResult`s for every member call unless the batch failed due to hard host failure.
- Successful calls in the same assistant message should still return successful results.

Hard host failures should still abort, e.g. cancellation and out-of-memory.

This requires a helper like:

```zig
fn classifyToolError(err: anyerror) ToolErrorAction {
    return switch (err) {
        error.Canceled, error.OutOfMemory => .hard_fail,
        else => .tool_result,
    };
}
```

The exact list will need auditing across native tools, Lua tools, source runtimes, and media processing.

## Data model changes

Consider extending `ToolResultBlock` with an error marker:

```zig
pub const ToolResultBlock = struct {
    tool_use_id: []const u8,
    parts: std.ArrayList(ResultPartStored),
    is_error: bool = false,
};
```

Update:

- conversation cloning/deinit paths,
- session persistence format,
- Anthropic serializer/parser,
- OpenAI serializer/parser,
- tests.

Backward compatibility:

- Missing `is_error` in old sessions should default to `false`.
- OpenAI Chat serialization can ignore the flag except for the textual content already present.

## Implementation plan

### Phase 1: Classify provider errors

- Expand provider HTTP status handling to return specific provider errors.
- Preserve context-overflow detection.
- Add tests for status-to-error mapping where practical.

### Phase 2: Add provider retry loop

- Add retry config defaults.
- Implement `streamWithRetries` around `stream_fn` in `Agent.runStep`.
- Retry only classified transient provider errors.
- Add exponential backoff with jitter and `Retry-After` support if available.
- Ensure failed attempts do not mutate the conversation.

### Phase 3: Add retry notifications

- Extend `ReceiverVTable` with provider retry notification.
- Update all receivers, test receivers, and CLI receivers.
- Print visible retry status in the CLI.

### Phase 4: Convert tool failures to tool results

- Add helper to build textual error `ResultPart`s.
- Change unknown-tool handling to append an error result instead of aborting.
- Change `FlatCall.err` assembly to synthesize error results for model-visible failures.
- Keep hard host failures as returned errors.
- Update tool API docs to say returned errors normally become model-visible tool failures.

### Phase 5: Add tool-result error marker

- Add `is_error` to `ToolResultBlock`.
- Serialize to Anthropic where supported.
- Preserve compatibility for OpenAI and old session files.
- Update tests.

### Phase 6: Holistic tests

Add tests for:

- provider 429 retries and then succeeds without duplicate messages,
- provider 500 retries with backoff notification,
- provider auth failure does not retry,
- context overflow still compacts once and retries,
- unknown tool produces an error tool result and the agent continues,
- tool handler error produces an error tool result and the model gets another turn,
- source per-call and batch errors produce error results,
- cancellation/OOM-like errors still hard-fail,
- retry notification callbacks are delivered before delay.

## Open questions

- Should malformed provider streams always be retryable, or only before any complete content block has been committed?
- Do we need a richer provider error object sooner rather than later, mainly for `Retry-After` and UI diagnostics?
- Should tool implementations have a first-class `model_error` return path distinct from host errors, instead of overloading `anyerror`?
- Should `ConcurrencyUnavailable` fall back to sequential tool execution instead of hard-failing?
- How should retry notifications be represented in persisted sessions, if at all? Initial answer: probably not persisted; they are UI/runtime events, not conversation history.