Building a multi-provider fallback chain that actually fails over

Why fallback matters

LLM providers rate-limit you. They go down. They have per-model quotas. If your product is built on a single provider, any of these will take you down with them. A fallback chain means your system degrades gracefully instead of failing completely.

At Meraki, we ran multi-provider AI call coaching for 100+ sales reps. A provider outage during peak calling hours would have meant zero coaching reports. That was not acceptable.

The single-provider problem

The naive implementation is a try/except around a single API call. That works until it does not — and when it fails, it fails loudly in production, often during the worst possible moment.

The real problem is not just rate limits. It is also: different providers have different cost curves, different latency profiles, and different strengths. You want to route intelligently, not just fall back blindly.

Designing the chain

Our chain: Claude first (best quality), Gemini second (good quality, lower cost), OpenAI last (fallback of last resort). Each hop happens only on a RateLimit or timeout exception — not on content errors.

async def generate(prompt: str, context: dict) -> str:
    for provider in [claude_client, gemini_client, openai_client]:
        try:
            return await provider.complete(
                prompt=prompt,
                context=context,
                cache_ttl=86400,  # 24h prompt cache
                timeout=8.0
            )
        except (RateLimitError, TimeoutError):
            logger.warning(f"{provider.name} unavailable, trying next")
            continue
    raise AllProvidersExhausted()

Prompt caching

The real cost win was not the fallback — it was caching. Our system prompt is large (coaching rubrics, rep context, product knowledge). Caching it 24h cut generation cost by ~95% on cache hits.

The key insight: the system prompt changes rarely. The user turn changes every call. Cache the stable part aggressively.

Partial failures

Rate limits are not binary. A provider might accept the request but return a partial response, or time out mid-stream. We handle these separately from hard failures: partial responses get logged and discarded, triggering a retry on the next provider.

Observability

Every call logs: which provider was used, whether it was a cache hit, latency, token count, and whether it was a primary or fallback invocation. This lets you see provider health trends before they become outages.

What I'd change

I would add circuit breakers per provider so a flapping provider does not cause latency spikes from repeated timeouts. And I would add p95 latency tracking per provider to inform which one becomes the primary dynamically rather than always starting with Claude.