Repository navigation
Feature: fallback engines for resilient agentic workflow execution #48348
Description
Summary
Today each workflow is compiled against exactly one engine (engine: claude, engine: copilot, etc.). If that engine is unavailable — quota exhausted, service outage, auth failure — the entire job fails with no recovery path. A fallback-engines list would let users declare an ordered preference and have gh-aw automatically retry with the next engine when the primary one cannot run.
Proposed syntax
engine: claude
fallback-engines:
- copilot
- codexResolution order: claude → copilot → codex. The first engine that succeeds wins; failures are logged but do not surface as job failures until all options are exhausted.
Behaviour spec
| Scenario | Behaviour |
|---|---|
| Primary engine succeeds | No fallback triggered |
| Primary fails with transient error (rate limit, 5xx) | Try next engine in list |
| Primary fails with config/auth error | Try next engine in list |
| All engines exhausted | Job fails with aggregated error messages |
fallback-engines omitted |
No change from today |
Why this matters
- Multi-tenant runners commonly have uneven quota across providers; a fallback avoids hard stops.
- Copilot BYOK setups may have primary keys scoped to one provider; a cheaper fallback engine can handle overflow.
- Scheduled / unattended workflows (no human to re-trigger) benefit most — they need resilience baked in.
Implementation notes
The ResolvedEngineTarget chain already lives in pkg/workflow/engine_definition.go. A fallback-engines field on the workflow spec could be parsed alongside engine and compiled into a retry loop at the runner level rather than re-generating the full workflow YAML per fallback. The log-parsing fallback path in pkg/cli/logs_parsing_engines.go and the existing engine resolution fallback (runtime-ID prefix match) show patterns that could be adapted.
Alternatives considered
- User-side retry workflow: callers can wrap the workflow in a retry loop, but this duplicates boilerplate across every workflow and doesn't capture transient errors that surface inside the compiled YAML.
- Single engine with model fallback: each engine already supports env-var model fallback, but that doesn't help when the whole provider is down.
Activity
theletterf commentedon Jul 27, 2026
@pelikhan Could be a good QoL feature!
pelikhan commentedon Jul 27, 2026
yes thinking.
theletterf commentedon Jul 28, 2026
Thanks! :)
theletterf commentedon Aug 11, 2026
Any news on this one?
pelikhan commentedon Oct 7, 2026
we are looking into fallback-models, the next logical step is fallback engine. it's a much bigger change...
pelikhan commentedon Oct 7, 2026
Status (2026-10-07; Maintainer direction): On October 7, a maintainer said fallback models are under investigation and fallback engines would be a much bigger next step. The next step is to scope engine-fallback behavior and failure handling after that model work is clearer.
pelikhan commentedon Oct 16, 2026
Community issue refresher — 2026-10-16: Partially addressed through model fallback; engine fallback remains relevant.
Since the October 7 maintainer reply, #66593 merged ordered
engine.fallback-modelsusing native AWF request-level recovery. The current validation requires the firewall, compatible AWF/API-proxy versions, concrete model IDs, and currently one inference provider. Regression coverage covers this model contract.That is useful resilience, but it is not the proposed
claude → copilot → codexruntime failover. Switching a model within an existing provider/request does not recover from an unavailable engine executable, engine-specific authentication/setup failure, or a whole-provider outage.The remaining request is ordered engine execution with safe retry boundaries, per-engine credentials/tools/log handling, and aggregated terminal failure reporting. It should be scoped separately from model fallback rather than considered implemented by it. This refresher is based on compiler code, tests, and merge history; no provider outage or engine failover was exercised.
pelikhan commentedon Oct 22, 2026
Still relevant: ordered cross-engine execution recovery is not implemented by the merged model-fallback work. Reviewed remote
mainatffdd9c9.#66593 merged
engine.fallback-models, but current documentation describes AWF retrying an individual request without restarting the agent, on the same provider. It explicitly excludes authentication failures and HTTP 429 from model switching. Compiler validation still rejects cross-provider chains. Neither this nor the engine catalog’s ID-prefix resolution fallback executesclaude → copilot → codexafter a failed engine.Next action: specify and implement a runner-level ordered engine-attempt contract: primary success skips fallbacks; eligible transient/quota/auth/config failures advance to the next configured engine; exhaustion reports all attempt failures; omission preserves existing behavior. Define per-engine setup/credentials/tool compatibility and a safe restart boundary so partially completed attempts cannot duplicate safe outputs or other side effects. Acceptance coverage must exercise those scenarios across different engines, not merely different models or harness retries. The request remains open; no outage/failover run was performed for this assessment.