Resilient, free AI: LLM fallback that never goes down

The AI assistant living in the terminal of this portfolio answers questions about my experience and my projects. It runs on no paid API. And yet it doesn’t go down when a provider is saturated, out of quota, or simply slow. The piece that makes that possible isn’t a better model: it’s a fallback chain across free LLMs, designed to degrade gracefully instead of returning an error.

This article is about how it’s actually wired — the decisions, the numbers, and the real potholes — because “resilient” reads well in a README but is earned in the details.

The problem: free tiers go down, and they go down badly

Free LLM tiers are generous, but they have three habitual ways of failing you, almost always at the worst possible moment:

• Quota exhausted. You hit the per-minute or per-day request limit and start getting 429s.
• Model saturation. The endpoint returns 503 or just hangs because half the world is hammering the same :free model.
• Unpredictable latency. The provider “works,” but it takes 12 seconds. In a serverless function with an execution cap, that’s a failure every bit as fatal as a 500.

If your app leans on a single free provider, any one of those three takes it down. The naive answer — “I’ll wrap it in a try/catch and show an error message” — turns every provider hiccup into a bad experience for whoever is visiting the site. The engineering answer is different: assume every layer will fail, and design so the next one picks up the baton.

The idea: with_fallbacks over free layers

The pattern is the one many frameworks call with_fallbacks: you define a sequence of providers ordered by preference and, on the first failure, you move to the next one without the user noticing. The name doesn’t matter; the property it gives you does: the system only goes down when ALL layers go down at once, far less likely than any single layer failing.

My chain, fastest to last resort:

1. Groq — primary. Absurdly fast inference (Llama 3.3 70B). When it’s available, the answer lands before the user has finished reading their own question.
2. OpenRouter — secondary. Aggregates several :free models (gpt-oss-120b, Llama 3.3 70B, Qwen3). There’s a nuance here I’ll get to below: the fallback is doubled.
3. Pollinations — safety net. No API key. It’s the link that guarantees the chain is never empty.

There’s a natural fourth layer that slots in without touching the architecture: Gemini has a solid free tier and speaks the same dialect, so adding it is a matter of one more entry in the list. The design is built for exactly that — the layers are interchangeable.

Why Pollinations goes last (and why that’s the point)

Groq and OpenRouter need an API key. If the key isn’t configured, or the provider is down, that layer simply doesn’t exist for that request. Pollinations asks for no key, so it’s always there. This changes a fundamental property of the system: the list of active providers can never be empty. There is always at least one candidate to ask. Without that floor, a day with no keys or two providers down would leave you with no AI at all; with it, the worst case is that you answer a little slower.

The detail that keeps it simple: everyone speaks OpenAI

The reason this chain fits into a single function rather than three separate integrations is that all three providers expose the OpenAI API format (/chat/completions, the same messages shape, the same way to read choices[0].message.content). That means one single ask(provider, messages) function serves all three. Only the URL, the auth header, and little else change.

async function ask(provider, messages) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), TIMEOUT_MS);
  try {
    const res = await fetch(provider.url, {
      method: "POST",
      headers: authHeaders(provider),
      body: JSON.stringify({ model: provider.model, messages, temperature: 0.6 }),
      signal: controller.signal,
    });
    if (!res.ok) throw new Error(`${provider.name} HTTP ${res.status}`);
    const content = (await res.json())?.choices?.[0]?.message?.content;
    if (typeof content !== "string" || !content.trim()) {
      throw new Error(`${provider.name} empty response`);
    }
    return content.trim();
  } finally {
    clearTimeout(timeout);
  }
}

This is DRY taken seriously: the diversity of providers is absorbed into configuration (a list of objects), not into code. Adding Gemini is a new entry in that list, not a new branch in the logic.

The fallback loop is just as boring, which is exactly what you want in a critical piece:

for (const provider of activeProviders()) {
  try {
    return ok(await ask(provider, [system, ...history]));
  } catch {
    // Provider down or out of quota → next layer. Silent on purpose.
  }
}
return ok(ERRORS[lang].unavailable); // everything fell: final degradation

The timeout: the least obvious decision and the most important one

Here’s the pothole almost nobody anticipates. On Vercel (Hobby plan) a serverless function has a hard execution limit of ~10 seconds. My first instinct was to give each provider a comfortable 15-second margin. The result was counterintuitive: the fallback chain never reached Pollinations. With three sequential 15s attempts, the platform killed the function with a 504 before the code could move to the third layer. The fallback existed on paper, but the infrastructure was killing it.

The fix was to budget time backwards: if I have ~10s and three attempts, each provider gets 3 seconds. Three attempts × 3s fit into ~9s, with room to spare. An AbortController per request cuts off any provider that doesn’t answer in time and treats it as just another failure, moving on to the next. A slow provider stops being a hang: it’s just one more layer that gets skipped.

The lesson: in serverless, the timeout isn’t a reliability parameter, it’s a budget constraint. If the sum of your retries doesn’t fit inside the platform’s limit, your fallback is decorative.

Fallback inside the fallback

OpenRouter has its own roulette: one :free model can be saturated while another answers without a hitch. That’s why that layer doesn’t point at a single model but at a list of up to three (gpt-oss-120b, Llama 3.3 70B, Qwen3). OpenRouter tries the first and, if it can’t, drops to the next, all inside the same call. It’s a nested fallback: layers of models inside a layer of provider.

Graceful degradation: the context adapts to each layer

Not every free model accepts the same prompt size. The big ones (Groq, OpenRouter) swallow a ~18,000-character system context without blinking; the Pollinations model cuts the input much sooner, around ~7,000. If I send it the full context, it fails on length and I lose my safety net exactly when I need it most.

The fix is that each provider declares its own maxContext, and the portfolio context — generated dynamically from my real projects and experience — is trimmed to that size before being sent. Pollinations gets a condensed version; Groq, the full one. Degrading isn’t only “use another provider”: it’s adjusting what you ask for to what that provider can actually give.

That same robustness philosophy lives in the practical RAG that feeds the assistant: if a content collection fails to read, the system degrades to a reduced context instead of propagating a non-JSON 500 that would break the client. Every layer assumes the one below it can fail.

What this has to do with shipping AI to production

This chain is small, but it distills how I approach any real AI system: the model is the easy part; reliability is the product. The same mindset — assume failure, budget your resources, degrade gracefully, never fake what you don’t have — is what I apply on more serious work, like the DevFlow AI suite or the semantic matcher in CV Finder AI, where a late or wrong answer isn’t a cosmetic detail but a bad decision.

Building an LLM demo is an afternoon’s work. Building one that’s still standing when your favorite provider has a bad day, without spending a euro on APIs, is another thing entirely. And that’s almost always the one that matters.

---

Got an AI system that buckles under quota, overspends, or hangs in production? That’s exactly the kind of problem I like to solve. Let’s talk and I’ll walk you through how I’d approach it.