path: root/docs/oauth-provider-auth.md

# OAuth provider auth

Goal: make provider authentication a core, reusable capability instead of
embedding auth details directly in each provider entry. A provider should name
the auth session it uses; core resolves that auth session into request-ready
credentials before a provider stream opens.

This covers today's static API-key providers, GitHub Copilot subscription
login, and OpenAI Codex/ChatGPT subscription login with one configuration
shape:

```toml
[providers.openai]
style = "openai_chat"
base_url = "https://api.openai.com/v1"
auth = "openai_api"

[auth.openai_api]
type = "api_key"
key_env_var = "OPENAI_API_KEY"
```

Providers no longer carry `api_key` or `api_key_env_var` directly. All
providers use `auth = "<name>"`; all credentials live under `[auth.<name>]`.
OAuth is just one family of auth configuration types in that namespace.

---

## Why config-only core auth

The earlier Copilot design put most auth behavior in Lua: HTTP requests, token
storage, refresh-before-turn, config mutation, and retry-after-401. With named
`[auth.<name>]` blocks, that split no longer buys enough. Auth can be entirely
described in config and executed by core.

Core should own:

- loading and validating named auth blocks
- API-key resolution from literal config or environment
- OAuth login flows
- token persistence under `$PANTO_HOME`
- refresh-before-turn with an expiry margin
- refresh-after-401/403 with one retry
- final request-header injection
- dynamic `base_url` updates when an auth exchange returns one

Lua is not required for API-key, GitHub Copilot, or OpenAI Codex auth
correctness. Extensions may still package optional provider presets or UI
polish, but the flagship auth stories should work from config alone.

---

## OAuth flow choices

`oauth_pkce` means OAuth 2.0 Authorization Code flow with PKCE.

PKCE is "Proof Key for Code Exchange." The client creates a one-time secret
called a `code_verifier`, sends only a hashed `code_challenge` in the browser
authorization URL, then later proves it owns the original verifier when it
exchanges the returned authorization code for tokens. This protects native
apps and CLIs that cannot safely keep a client secret.

For pantograph, `oauth_pkce` is the browser-login flow:

1. Start a short-lived localhost callback server.
2. Generate PKCE verifier/challenge and state.
3. Open or print an authorization URL.
4. Receive the callback with `code` and `state`.
5. Exchange the code + verifier at the token endpoint.
6. Persist returned tokens and refresh them later.

This is different from `oauth_device`, which prints a URL and user code, then
polls until the browser-side authorization completes. Device flow is better for
headless or remote terminals because it needs no local callback server.

For the first implementation, prefer `oauth_device` if it can cover both
flagship user stories:

1. It avoids callback URLs, so it works naturally through SSH and remote
   servers where `localhost` is ambiguous.
2. It avoids implementing a local Zig web server in the first pass.
3. GitHub officially positions device flow for headless apps and CLIs.
4. OpenAI Codex already supports device-code login for headless environments.

Keep `oauth_pkce` in the design as a known later extension, not as a first-pass
requirement, unless Codex subscription auth proves device flow is unavailable
for the accounts we need to support.

---

## Configuration model

### API key auth

```toml
[auth.openai_api]
type = "api_key"
key_env_var = "OPENAI_API_KEY"
```

or:

```toml
[auth.local_proxy_key]
type = "api_key"
key = "sk-..."
```

Rules:

- `key` wins over `key_env_var` if both are present.
- if `key_env_var` is set but absent from the environment, the auth session is
  unresolved
- a provider with unresolved auth may remain visible/selectable, but the first
  request should fail with a clear auth error unless the auth type can launch an
  interactive login
- literal `key` is supported for completeness, but env vars remain preferred

### OAuth device auth

GitHub Copilot:

```toml
[auth.github_copilot]
type = "oauth_device"
dialect = "token"
client_id = "Iv1.b507a08c87ecfe98"
device_code_url = "https://github.com/login/device/code"
token_url = "https://github.com/login/oauth/access_token"
scope = "read:user"
token_request_format = "form" # or "json"; default TBD
```

OpenAI Codex:

```toml
[auth.openai_codex]
type = "oauth_device"
dialect = "codex"
client_id = "app_EMoamEEZ73f0CkXaXp7hrann"
issuer = "https://auth.openai.com"
device_code_url = "https://auth.openai.com/api/accounts/deviceauth/usercode"
device_poll_url = "https://auth.openai.com/api/accounts/deviceauth/token"
verification_url = "https://auth.openai.com/codex/device"
token_url = "https://auth.openai.com/oauth/token"
```

Core owns requesting the device code, presenting `verification_uri` and
`user_code`, polling on `interval`, and storing the durable OAuth token.
`dialect` selects the device-flow completion shape:

- `token` — standard OAuth device flow. The poll endpoint returns the OAuth
  token response directly, including `access_token` and optional
  `refresh_token` / `id_token`.
- `codex` — OpenAI Codex device flow. The poll endpoint returns an
  authorization code plus PKCE verifier data; core exchanges those at
  `token_url` to obtain `id_token`, `access_token`, and `refresh_token`.

### OAuth PKCE auth (deferred)

```toml
[auth.openai_codex]
type = "oauth_pkce"
client_id = "app_EMoamEEZ73f0CkXaXp7hrann"
issuer = "https://auth.openai.com"
authorize_url = "https://auth.openai.com/oauth/authorize"
token_url = "https://auth.openai.com/oauth/token"
redirect_port = 1455
scopes = [
  "openid",
  "profile",
  "email",
  "offline_access",
  "api.connectors.read",
  "api.connectors.invoke",
]

[auth.openai_codex.authorize_params]
id_token_add_organizations = "true"
codex_cli_simplified_flow = "true"
```

Core can later own PKCE generation, callback handling, code exchange, token
persistence, JWT expiry parsing, and refresh-token grants. This is explicitly
not required for the first pass if `oauth_device` covers Copilot and Codex.

### Optional token exchange

Some OAuth flows do not return the token that should be sent to the model API.
GitHub Copilot is the important example: GitHub device auth returns `ghu_...`,
then Copilot requires a second exchange to get a short-lived chat token and API
endpoint.

```toml
[auth.github_copilot.exchange]
method = "GET"
url = "https://api.github.com/copilot_internal/v2/token"
bearer = "oauth_access_token"
token_json_path = "token"
expires_at_json_path = "expires_at"
base_url_json_path = "endpoints.api"

[auth.github_copilot.exchange.headers]
User-Agent = "GitHubCopilotChat/0.26.7"
Editor-Version = "vscode/1.99.0"
Editor-Plugin-Version = "copilot-chat/0.26.7"
Copilot-Integration-Id = "vscode-chat"
```

The exchange result becomes the request credential used by the provider. It may
also override the provider `base_url`.

### Provider reference

```toml
[providers.copilot]
style = "openai_chat"
base_url = "https://api.individual.githubcopilot.com" # fallback/placeholder
auth = "github_copilot"

[providers.copilot.extra_headers]
User-Agent = "GitHubCopilotChat/0.26.7"
Editor-Version = "vscode/1.99.0"
Editor-Plugin-Version = "copilot-chat/0.26.7"
Copilot-Integration-Id = "vscode-chat"
X-Initiator = "user"
```

```toml
[providers.codex]
style = "openai_chat" # pending endpoint verification
base_url = "https://chatgpt.com/backend-api" # placeholder
auth = "openai_codex"
```

`extra_headers` remains a provider capability because it applies to the model
request, not necessarily to the auth endpoints. Auth exchanges can have their
own headers.

---

## GitHub Copilot auth

Verified against opencode and copilot-api reference implementations.

Two tokens, two phases:

1. **One-time browser login (OAuth device flow).** Yields a long-lived OAuth
   token (`ghu_...`). This is the durable credential; persist it. The device
   flow needs no redirect URI and no local HTTP server.
2. **Per-~30-min token exchange.** GET the Copilot token endpoint with the
   `ghu_` token as bearer; receive a short-lived API token, its expiry, and the
   API base URL. Re-run with the same `ghu_` token to refresh.

Constants:

```text
CLIENT_ID        = "Iv1.b507a08c87ecfe98"
DEVICE_CODE_URL  = "https://github.com/login/device/code"
ACCESS_TOKEN_URL = "https://github.com/login/oauth/access_token"
TOKEN_URL        = "https://api.github.com/copilot_internal/v2/token"
SCOPE            = "read:user"
GRANT_TYPE       = "urn:ietf:params:oauth:grant-type:device_code"
```

Device-code request:

```json
{ "client_id": CLIENT_ID, "scope": "read:user" }
```

Poll request:

```json
{ "client_id": CLIENT_ID, "device_code": "...", "grant_type": GRANT_TYPE }
```

Copilot exchange response:

```json
{
  "token": "...",
  "expires_at": 1700000000,
  "refresh_in": 1500,
  "endpoints": {
    "api": "https://api.individual.githubcopilot.com"
  }
}
```

`token` is opaque and becomes the provider bearer credential.
`endpoints.api` becomes the runtime `base_url`.

Required Copilot headers ride on the token exchange and chat requests:

```text
User-Agent:            GitHubCopilotChat/<ver>
Editor-Version:        vscode/<ver>
Editor-Plugin-Version: copilot-chat/<ver>
Copilot-Integration-Id: vscode-chat
```

Chat requests commonly also send `X-Initiator: user` or `agent`.

---

## OpenAI Codex auth

Official Codex documentation describes two OpenAI sign-in methods:

- ChatGPT sign-in for subscription access
- API-key sign-in for usage-based access

The open-source Codex client currently implements ChatGPT sign-in with OAuth
PKCE and also supports a device-code variant for headless environments. For
pantograph, prefer the device-code variant first.

Device-code login uses the ChatGPT auth issuer plus Codex-specific device auth
endpoints:

```text
issuer/client auth base: https://auth.openai.com
device user-code URL:    https://auth.openai.com/api/accounts/deviceauth/usercode
device poll URL:         https://auth.openai.com/api/accounts/deviceauth/token
browser verification:    https://auth.openai.com/codex/device
token endpoint:          https://auth.openai.com/oauth/token
client_id:               app_EMoamEEZ73f0CkXaXp7hrann
```

The device poll response returns an authorization code plus PKCE verifier data;
Codex then exchanges that code at the token endpoint. That means OpenAI's
device flow is not identical to GitHub's, but it still has the CLI-friendly
property we care about: no local callback URL and no local web server.

Browser/PKCE login uses:

```text
issuer/client auth base: https://auth.openai.com
token endpoint:          https://auth.openai.com/oauth/token
client_id:               app_EMoamEEZ73f0CkXaXp7hrann
default callback port:   1455
fallback callback port:  1457
```

The token exchange returns:

```json
{
  "id_token": "...",
  "access_token": "...",
  "refresh_token": "..."
}
```

Core should persist these as one auth session. Refresh uses
`grant_type = "refresh_token"` against the same token endpoint. The access
token is a JWT in normal ChatGPT auth sessions, so core can refresh
proactively when its `exp` claim is within the safety margin.

Codex-backed requests use:

```text
Authorization: Bearer <access_token>
ChatGPT-Account-ID: <account_id>        # when present
X-OpenAI-Fedramp: true                  # when the ID token says FedRAMP
```

Open question: the exact provider wire endpoint pantograph should call for
ChatGPT subscription-backed Codex use. The auth machinery is clear; the model
request shape still needs verification before we commit to treating this as a
plain `openai_chat` provider.

---

## Core implementation plan

### C1. Parse named auth blocks

Add `Config.auths: []AuthConfig` and make `Provider.auth` required for
networked providers. Move `api_key` / `api_key_env_var` out of providers and
into `[auth.<name>]`.

Compatibility path: accept provider-level `api_key` and `api_key_env_var` for
one release by synthesizing hidden auth entries, then warn.

### C2. Resolve provider auth before each request

The active provider config should be built from:

- provider transport fields (`style`, `base_url`, `extra_headers`)
- selected model alias
- resolved auth session (`api_key`, OAuth access token, exchanged token, extra
  auth-derived headers, dynamic base URL)

This replaces the old "drop providers whose env var is absent" behavior. A
provider can survive config resolution even if its auth is not currently
resolved.

### C3. Core HTTP client

Core auth needs HTTPS for OAuth and token exchange. Implement a small
request/response HTTP helper backed by Zig `std.http.Client`. Because this is
core Zig auth, it does not need the Lua coroutine surface from the old design.
If Lua still needs HTTP later, expose a separate `panto.http` wrapper.

### C4. Token storage

Store under `$PANTO_HOME/auth/<name>.json` initially:

```json
{
  "type": "oauth_device",
  "access_token": "...",
  "refresh_token": "...",
  "id_token": "...",
  "expires_at": 1700000000,
  "exchange": {
    "token": "...",
    "expires_at": 1700000000,
    "base_url": "https://api.individual.githubcopilot.com"
  }
}
```

Only fields relevant to the auth type are present. Treat these files like
passwords. A future credential-store backend can keep the same logical API.

### C5. Refresh lifecycle

Before opening a provider stream:

1. Load the named auth session.
2. If absent and interactive login is possible, run login.
3. If token is within the refresh margin, refresh.
4. If an exchange is configured and stale, run the exchange.
5. Build request auth headers and dynamic provider config.

On 401/403:

1. Force one refresh/exchange.
2. Retry the same turn once.
3. Surface the original provider error if it still fails.

### C6. Extra headers

Add generic `extra_headers` to `OpenAIChatConfig` and
`AnthropicMessagesConfig`, then thread config TOML through libpanto.

Auth-derived headers and provider `extra_headers` should merge deterministically.
Provider request-specific defaults should not leak into OAuth endpoints unless
configured under `[auth.<name>.exchange.headers]`.

### C7. No Lua dependency

Built-in auth should not depend on Lua. Optional Lua surfaces can come later if
they are useful:

- auth status query
- command to trigger login/logout
- UI overrides for device-code prompts
- provider packages that install config/model defaults

None of these are required for Copilot or Codex auth correctness.

---

## Tests

- API-key auth: literal key, env var present, env var absent, provider survives
  with clear unresolved-auth error.
- Config migration: provider-level legacy key fields synthesize auth entries.
- OAuth device: device-code response parsing, polling pending/success/error,
  persisted token shape.
- OAuth device: GitHub-style direct device-token polling.
- OAuth device: OpenAI Codex-style device polling followed by authorization-code
  exchange.
- OAuth PKCE later: authorize URL construction, state validation, code
  exchange, token persistence.
- JWT expiry parsing: refresh inside safety margin, no-op when fresh.
- Token exchange: Copilot `token`, `expires_at`, and `endpoints.api` mapping.
- Request headers: bearer auth, `ChatGPT-Account-ID`, FedRAMP header,
  provider `extra_headers`, exchange-only headers.
- 401 retry: refresh/exchange once and retry the same turn.

## Open questions

- Exact TOML names: `key_env_var` vs. current `api_key_env_var`; this doc uses
  `key_env_var` because the `[auth]` section already establishes the domain.
- Whether `auth = "<name>"` should be mandatory for no-auth local providers or
  whether `auth = null` / omitted means unauthenticated.
- How much of OpenAI Codex device auth to model generically versus as an
  OpenAI-specific device-flow dialect.
- The correct Codex subscription-backed model request endpoint and wire API.
- Whether to add OS keychain storage in the first pass or after file-backed
  auth works end to end.

---

## Implementation status (as built)

The first pass landed as a clean break (no compatibility shim): every networked
provider must name an `[auth.<name>]` session via `auth = "<name>"`;
provider-level `api_key`/`api_key_env_var` are gone. Resolved decisions for the
open questions above: `auth` is required for networked providers; Codex device
auth is a `dialect = "codex"` variant of the generic `oauth_device` type;
keychain storage is deferred (file-backed `$PANTO_HOME/auth/<name>.json`,
owner-only `0600`).

The `[auth.<name>]` schema is deliberately flat and minimal:

- **`type` is inferred** — `key` ⇒ `api_key`, `client_id` ⇒ `oauth_device`
  (explicit `type` still works).
- **`${...}` substitution** on every value — `${env:VAR}` reads the
  environment, `${sibling}` reads another key in the same section. The common
  api-key session is one line: `key = "${env:OPENAI_API_KEY}"`. An unset
  `${env:VAR}` resolves to empty, and a provider whose `api_key` resolves empty
  is **dropped** (reproducing "export your key or the provider disappears").
  GitHub Enterprise is a one-liner: set `domain = "company.ghe.com"` and the
  `${domain}`-templated URLs follow.
- **The exchange is flat** — `exchange_url`, `exchange_method`,
  `exchange_token_path`, `exchange_expires_path`, `exchange_base_url_path` (no
  `[auth.<name>.exchange]` sub-table); presence of `exchange_url` enables it.
- **No auth-level headers** — the provider's `extra_headers` (the client
  identity, e.g. Copilot's editor headers) are reused on the auth HTTP calls
  (device/poll/exchange) as well as the model request, so they're written once.

### What core owns (libpanto)

- `auth.zig` — auth config types (`AuthConfig`), the persisted `TokenSet`, token
  storage (load/save/delete), the device flow (both dialects), refresh, the
  Copilot exchange, JWT `exp` parsing, Codex account-id extraction, and the
  pure credential/refresh-predicate helpers. No Lua dependency.
- `http_helper.zig` — a non-streaming request/response helper over the
  process-global `std.http.Client`, plus dotted-JSON-path readers.
- Providers gained `extra_headers`, merged onto the built-in request headers.

### What the CLI owns (`src/`)

- `config_file.zig` parses `[auth.<name>]` and the `auth = "<name>"` reference,
  runs `${...}`/`${env:VAR}` substitution, infers `type`, and resolves
  `api_key` sessions eagerly. A provider whose `api_key` resolves empty is
  dropped; OAuth providers always survive (resolved at turn time).
- `auth_manager.zig` resolves the active provider's auth before each turn
  (load → login → refresh → exchange → build credential), patching the live
  `ProviderConfig` (bearer, dynamic `base_url`, merged headers).
- The TUI runs an inline device login when no token is stored, and forces one
  refresh/exchange retry on a `ProviderAuthFailed` (401/403).
- `panto auth status | login <name> | logout <name>` subcommands provide a
  line-based login path outside the TUI.

### Device-flow specifics (verified against reference clients)

- **GitHub Copilot** (`dialect = "token"`): request `/login/device/code`, poll
  `/login/oauth/access_token` until the OAuth token (`ghu_…`) arrives, then a
  per-~30-min exchange (GET `/copilot_internal/v2/token`) yields the chat token,
  its expiry, and `endpoints.api` (the runtime `base_url`). Copilot speaks the
  OpenAI Chat Completions wire format, so it is a plain `openai_chat` provider —
  this path is structurally complete.
- **OpenAI Codex** (`dialect = "codex"`): POST
  `…/deviceauth/usercode` → poll `…/deviceauth/token` (403/404 = pending). The
  poll **returns a server-generated PKCE `code_verifier` alongside the
  authorization code**, so the client does *not* generate PKCE for device flow;
  it relays the verifier when exchanging the code at `/oauth/token` for
  `{access,refresh,id}_token`. Refresh uses `grant_type=refresh_token`. The
  account id comes from the `id_token` JWT claim `https://api.openai.com/auth`
  → `chatgpt_account_id`, sent as `chatgpt-account-id`.

### Codex model endpoint — resolved open question

Research confirmed the Codex subscription endpoint is **not** Chat Completions:
it is the OpenAI **Responses API** at
`https://chatgpt.com/backend-api/codex/responses` (request body uses `input`
items, `reasoning`, `include: ["reasoning.encrypted_content"]`, `store: false`;
headers `Authorization: Bearer`, `chatgpt-account-id`,
`OpenAI-Beta: responses=experimental`, `originator: codex_cli_rs`). That is a
new wire dialect, implemented as the **`openai_responses` provider style**
(`provider_openai_responses.zig` + `openai_responses_json.zig`), distinct from
`openai_chat`. The default config ships a commented `[providers.codex]` /
`[auth.openai_codex]` example. The serializer maps system→`instructions`,
user/assistant text→message items, tool calls→`function_call` /
`function_call_output` items, and tools→flat function entries; the stream
parser handles `response.output_text.delta`,
`response.reasoning_summary_text.delta`, `response.output_item.added/done` +
`response.function_call_arguments.delta`, and `response.completed` (usage).
Known limitation: encrypted reasoning items are requested via `include` but not
persisted across turns, so multi-turn reasoning continuity is approximate.

### Verification status

Unit tests cover config parsing, token storage round-trips, JWT/account-id
extraction, credential building, refresh/exchange predicates, form encoding,
and header merging. The HTTP-dependent flows (device login, refresh, exchange,
Responses streaming) follow verified reference implementations but require live
credentials (a GitHub Copilot subscription / a ChatGPT plan) to confirm
end-to-end against the real endpoints.