From 3f1ace16afc7877b0bfad374cb286d4d84140960 Mon Sep 17 00:00:00 2001 From: t Date: Thu, 4 Jun 2026 09:46:49 -0600 Subject: failure retries scheme --- docs/archive/retries.md | 341 ++++++++++++++++++++++++++++++++++++++++++++++++ docs/todos.md | 5 +- 2 files changed, 344 insertions(+), 2 deletions(-) create mode 100644 docs/archive/retries.md (limited to 'docs') diff --git a/docs/archive/retries.md b/docs/archive/retries.md new file mode 100644 index 0000000..26aced1 --- /dev/null +++ b/docs/archive/retries.md @@ -0,0 +1,341 @@ +# Message-level error retries + +## Summary + +The agent loop should make an explicit decision for every failure that occurs while producing the next conversation message. Depending on the failure, the agent should either: + +1. retry the same provider request with backoff, +2. compact/rewrite the conversation and retry the same provider request, +3. append conversation-visible tool error results and continue the agent loop, or +4. hard-fail and return the error to the embedder. + +The important distinction is between provider/API failures and tool-call failures: + +- Provider/API failures usually mean the assistant message was not produced. The correct recovery is to retry the same request without mutating the conversation. +- Tool-call failures usually mean the model made a recoverable choice. The correct recovery is to append `ToolResult` blocks describing the failures, then let the model continue from there. + +This plan treats retries as **message-level**: a failed provider attempt should not create a partial duplicate message, and a tool failure should be represented as the next conversation message in the normal assistant/tool-result exchange. + +## Existing behavior + +We already have several special cases: + +- Context overflow errors are handled by automatic compaction followed by one retry of the same request. +- Max-tokens overflow during tool argument generation can surface as a partially generated tool input; this is currently handled as invalid/incomplete JSON-ish tool input and sent back as a tool result. +- Some tool-result post-processing failures are already degraded into text, e.g. unrecognized binary attachments are dropped with a note. + +However, error handling is not yet holistic: + +- Provider HTTP failures are often collapsed into broad errors such as `HttpError`, losing retryability information. +- Transient provider/API failures do not have a general backoff retry loop. +- Tool execution errors usually abort the agent loop instead of becoming tool error results. +- Retry attempts are not surfaced to the UI, which makes coding-agent behavior opaque during provider outages or rate limits. + +## Error actions + +### 1. Retry the same provider request + +Use this when the provider request appears transiently unavailable or unreliable, and the conversation has not gained a completed assistant message. + +Examples: + +- HTTP 429 rate limit. +- HTTP 408, 409, 425 where retry is appropriate. +- HTTP 500, 502, 503, 504. +- Connection reset, DNS/connect failure, TLS failure, timeout. +- Stream interruption or malformed provider stream before a complete assistant message. +- Possibly an empty assistant response, with a small retry cap. + +Behavior: + +- Do not append or modify conversation messages. +- Retry the same request against the same conversation snapshot. +- Use exponential backoff with jitter. +- Honor `Retry-After` when the provider sends it. +- Stop after a configurable attempt cap and then hard-fail. +- Notify the receiver/UI before each delayed retry. + +### 2. Compact/rewrite context, then retry the same provider request + +Use this when the provider rejects the request because the input context is too large. + +Examples: + +- OpenAI-compatible `context_length_exceeded` / `maximum context length` responses. +- Anthropic `prompt is too long` responses. +- Other provider-specific context-window diagnostics. + +Behavior: + +- Run automatic compaction if configured. +- Rewrite the conversation into the compacted form. +- Retry the same request once against the compacted conversation. +- If compaction cannot shed enough context, or if the retry also overflows, hard-fail. + +This is already mostly implemented; it should be integrated into the same provider-error decision path as ordinary retries. + +### 3. Append tool error results and continue + +Use this when the model emitted a tool call and the failure is meaningful to the model. + +Examples: + +- Unknown tool name. +- Invalid/incomplete tool input. +- Future schema validation failures. +- Native tool handler returned an error. +- Lua/runtime exception. +- `ToolSource` per-call error. +- `ToolSource` whole-batch failure, mapped to each affected call. +- Tool output shape/encoding problems that can be explained to the model. +- Media processing failures where the rest of the agent is healthy. + +Behavior: + +- Append a user message containing one `ToolResult` for every `ToolUse` block, preserving original call order. +- Successful calls keep their normal results. +- Failed calls get model-readable error text. +- The agent loop continues, allowing the model to correct arguments, try a different tool, or explain the failure to the user. +- Prefer adding an internal `is_error` flag to tool results, serialized where supported. + +Example error text: + +```text +Tool execution failed for `std.read`: file not found: src/foo.zig +You may fix the arguments, try a different tool, or explain the failure to the user. +``` + +Provider serialization: + +- Anthropic supports an `is_error`-style tool result marker; serialize it when present. +- OpenAI Chat Completions has no direct equivalent; include the error as textual tool content. + +### 4. Hard-fail + +Use this when retrying would be unsafe or misleading, or when the failure belongs to the host application rather than the model/provider interaction. + +Examples: + +- User cancellation / abort. +- Out of memory. +- Receiver/UI/session persistence failure. +- Invalid local configuration: missing API key, malformed URL, unknown model alias. +- Provider authentication/authorization failure: HTTP 401/403. +- Non-retryable provider request errors: most HTTP 400s except context overflow. +- Internal invariant violation / conversation corruption. +- Concurrency unavailable, unless we add a sequential fallback. + +Behavior: + +- Return the error to the embedder. +- Do not synthesize a model-visible tool result for host/system failures. + +## Provider retry notifications + +Retry visibility should be front-loaded. Coding-agent UIs need to show that the agent is waiting on the provider rather than silently hanging. + +Add a receiver callback for provider retry scheduling, something like: + +```zig +onProviderRetry(info: ProviderRetryInfo) void +``` + +Possible fields: + +```zig +const ProviderRetryInfo = struct { + attempt: usize, // next attempt number, 1-based or 0-based; define clearly + max_attempts: usize, + delay_ms: u64, + err: anyerror, + status_code: ?u16 = null, + retry_after_ms: ?u64 = null, + message: ?[]const u8 = null, +}; +``` + +Semantics: + +- Fire after a provider attempt fails and before sleeping for the next attempt. +- Fire for ordinary provider retry/backoff. +- Fire for context-overflow compaction retry too, either through this callback or through a sibling callback that says compaction is being attempted. +- Do not fire for tool-call failures that are being sent back as tool results; those are normal conversation continuations. + +Initial CLI behavior can be simple, e.g. print a dim status line: + +```text +provider unavailable: retrying in 2.4s (attempt 2/4) +``` + +## Provider error classification + +The providers should stop collapsing all HTTP/API failures into a single broad error when the agent needs to distinguish retryable from non-retryable failures. + +Minimum useful public errors: + +```zig +error.ProviderRateLimited +error.ProviderUnavailable +error.ProviderServerError +error.ProviderTransport +error.ProviderStreamMalformed +error.ProviderAuthFailed +error.ProviderBadRequest +error.ProviderModelNotFound +error.ContextOverflow +``` + +Suggested HTTP mapping: + +- `400` + context marker: `ContextOverflow` +- other `400`: `ProviderBadRequest` +- `401`, `403`: `ProviderAuthFailed` +- `404`: `ProviderModelNotFound` or `ProviderBadRequest`, depending on provider/API shape +- `408`: retryable transport/request timeout +- `409`, `425`: retryable if provider semantics indicate temporary conflict/not-ready +- `429`: `ProviderRateLimited` +- `500..599`: `ProviderServerError` / `ProviderUnavailable` + +Longer term, `anyerror` alone is too lossy for status code, provider message, and `Retry-After`. A richer provider error/reporting channel would be useful. For the first implementation, distinct error names plus logging may be enough. + +## Agent retry policy + +Add a retry policy to config, with conservative defaults. + +Possible shape: + +```zig +const RetryConfig = struct { + max_attempts: usize = 4, + initial_delay_ms: u64 = 500, + max_delay_ms: u64 = 10_000, + multiplier: f64 = 2.0, + jitter: bool = true, +}; +``` + +Rules: + +- Attempt count includes the first attempt; `max_attempts = 4` means one initial try plus up to three retries. +- Retry only errors classified as retryable provider failures. +- Never retry cancellation, local receiver errors, auth failures, or bad local configuration. +- If the provider supplies `Retry-After`, prefer that delay, capped by policy. +- Keep context-overflow compaction retry as a separate one-shot path, not an unbounded retry loop. + +## Tool error result policy + +Change tool dispatch so normal tool failures do not abort the turn. + +Current behavior to replace: + +- Unknown tool returns `error.UnknownTool`. +- `Tool.invoke` error records `FlatCall.err`; after dispatch, the first error aborts the turn. +- `ToolSource` per-call errors and whole-batch errors abort the turn. + +Desired behavior: + +- Unknown tool becomes an error `ToolResult`. +- Invalid/incomplete input remains an error `ToolResult`. +- Native tool errors become error `ToolResult`s unless classified as hard host failures. +- Source per-call errors become per-call error `ToolResult`s. +- Source whole-batch errors become error `ToolResult`s for every member call unless the batch failed due to hard host failure. +- Successful calls in the same assistant message should still return successful results. + +Hard host failures should still abort, e.g. cancellation and out-of-memory. + +This requires a helper like: + +```zig +fn classifyToolError(err: anyerror) ToolErrorAction { + return switch (err) { + error.Canceled, error.OutOfMemory => .hard_fail, + else => .tool_result, + }; +} +``` + +The exact list will need auditing across native tools, Lua tools, source runtimes, and media processing. + +## Data model changes + +Consider extending `ToolResultBlock` with an error marker: + +```zig +pub const ToolResultBlock = struct { + tool_use_id: []const u8, + parts: std.ArrayList(ResultPartStored), + is_error: bool = false, +}; +``` + +Update: + +- conversation cloning/deinit paths, +- session persistence format, +- Anthropic serializer/parser, +- OpenAI serializer/parser, +- tests. + +Backward compatibility: + +- Missing `is_error` in old sessions should default to `false`. +- OpenAI Chat serialization can ignore the flag except for the textual content already present. + +## Implementation plan + +### Phase 1: Classify provider errors + +- Expand provider HTTP status handling to return specific provider errors. +- Preserve context-overflow detection. +- Add tests for status-to-error mapping where practical. + +### Phase 2: Add provider retry loop + +- Add retry config defaults. +- Implement `streamWithRetries` around `stream_fn` in `Agent.runStep`. +- Retry only classified transient provider errors. +- Add exponential backoff with jitter and `Retry-After` support if available. +- Ensure failed attempts do not mutate the conversation. + +### Phase 3: Add retry notifications + +- Extend `ReceiverVTable` with provider retry notification. +- Update all receivers, test receivers, and CLI receivers. +- Print visible retry status in the CLI. + +### Phase 4: Convert tool failures to tool results + +- Add helper to build textual error `ResultPart`s. +- Change unknown-tool handling to append an error result instead of aborting. +- Change `FlatCall.err` assembly to synthesize error results for model-visible failures. +- Keep hard host failures as returned errors. +- Update tool API docs to say returned errors normally become model-visible tool failures. + +### Phase 5: Add tool-result error marker + +- Add `is_error` to `ToolResultBlock`. +- Serialize to Anthropic where supported. +- Preserve compatibility for OpenAI and old session files. +- Update tests. + +### Phase 6: Holistic tests + +Add tests for: + +- provider 429 retries and then succeeds without duplicate messages, +- provider 500 retries with backoff notification, +- provider auth failure does not retry, +- context overflow still compacts once and retries, +- unknown tool produces an error tool result and the agent continues, +- tool handler error produces an error tool result and the model gets another turn, +- source per-call and batch errors produce error results, +- cancellation/OOM-like errors still hard-fail, +- retry notification callbacks are delivered before delay. + +## Open questions + +- Should malformed provider streams always be retryable, or only before any complete content block has been committed? +- Do we need a richer provider error object sooner rather than later, mainly for `Retry-After` and UI diagnostics? +- Should tool implementations have a first-class `model_error` return path distinct from host errors, instead of overloading `anyerror`? +- Should `ConcurrencyUnavailable` fall back to sequential tool execution instead of hard-failing? +- How should retry notifications be represented in persisted sessions, if at all? Initial answer: probably not persisted; they are UI/runtime events, not conversation history. diff --git a/docs/todos.md b/docs/todos.md index 55a97fc..1388ecf 100644 --- a/docs/todos.md +++ b/docs/todos.md @@ -5,13 +5,14 @@ - [ ] C ABI - [x] Agent compaction with custom compaction prompts - [x] Agent auto-compaction -- [ ] image upload support +- [x] image upload support - [ ] google gemini native provider - [ ] openai responses API native provider - [ ] non-streaming - [ ] one-shot simple API -- [ ] message-level error retries +- [x] message-level error retries - [ ] abort/cancellation +- [ ] user message queueing: steering, follow-up - [ ] step cap, stop conditions ## panto cli -- cgit v1.3