How To Fix OpenAI Rate Limits & Timeout Errors

LLMs are used across a wide variety of production use cases — translation, sentiment analysis, code generation, content creation, and more.

However, integrating any LLM API directly into production has challenges. Rate limits, timeout errors, and availability issues are a fact of life. OpenAI, Anthropic, and other providers all impose rate limits on tokens per second and requests per second.

Here are the key techniques to keep your LLM-powered features reliable.

Exponential Backoff

Exponential backoff is a strategy that gradually increases the delay between retries when you hit a rate limit. Below is an example in Node.js:

const BASE_URL = 'https://api.openai.com/v1/chat/completions';

async function makeRequestWithBackoff(endpoint, params, retries = 3, backoffDelay = 500) {
  try {
    const response = await fetch(endpoint, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
      },
      body: JSON.stringify(params),
    });

    if (!response.ok) {
      const status = response.status;
      if (status === 429 && retries > 0) {
        const delay = Math.random() * backoffDelay;
        console.log(`Rate limit hit, retrying in ${delay}ms`);
        await new Promise((resolve) => setTimeout(resolve, delay));
        return makeRequestWithBackoff(endpoint, params, retries - 1, backoffDelay * 2);
      }
      throw new Error(`HTTP ${status}: ${response.statusText}`);
    }

    return response.json();
  } catch (error) {
    if (retries > 0 && error instanceof TypeError) {
      const delay = Math.random() * backoffDelay;
      await new Promise((resolve) => setTimeout(resolve, delay));
      return makeRequestWithBackoff(endpoint, params, retries - 1, backoffDelay * 2);
    }
    throw error;
  }
}

const params = {
  messages: [
    { role: "user", content: "Hi, Who are you?" }
  ],
  max_tokens: 50,
  model: "gpt-4o-mini"
};

makeRequestWithBackoff(BASE_URL, params)
  .then(data => console.log(data))
  .catch(error => console.error(error));

You can adapt this to linear or randomized backoff depending on your traffic patterns.

Batching

If you’re hitting requests-per-second limits but have headroom on tokens, batch multiple prompts into fewer API calls. OpenAI’s Batch API lets you submit groups of requests at a discount, with results returned within 24 hours — useful for non-latency-sensitive workloads like evaluation, data processing, or content generation.

Caching

Users frequently query the same things. A caching layer above your API calls saves both cost and latency:

• Exact-match caching — hash the prompt and model params; serve from cache on match
• Semantic caching — embed the query and check similarity against recent requests; serve a cached response if the intent matches

Either approach reduces calls to the LLM provider and improves response times.

Multi-Provider Load Balancing

You can set up multiple providers (OpenAI, Azure OpenAI, Anthropic, Google) and load-balance across them. If one is throttled or down, traffic shifts to another. Tools like LiteLLM and OpenRouter make this straightforward.

Always Stream Responses

Streaming ensures that the model begins delivering tokens within seconds. Without streaming, you don’t know whether a request will time out until the full response duration elapses.

With streaming, a delay beyond the first 5–6 seconds is a clear signal that something is wrong — you can terminate and retry immediately rather than waiting for a full timeout.

Setting Up Fallbacks

For use cases where response quality can flex, set up model fallbacks. If your primary model (e.g., gpt-4o) is unavailable or slow, fall back to a lighter model (gpt-4o-mini, claude-3.5-haiku, gemini-2.0-flash). This keeps the feature running even during provider issues.

Putting It All Together

These techniques work best in combination. A production-grade LLM integration typically layers them:

1. Streaming — always on, for early failure detection
2. Caching — to avoid redundant calls entirely
3. Exponential backoff — for transient rate limits
4. Multi-provider routing — for availability during outages
5. Model fallbacks — as a last resort to keep the feature alive

If you’re building systems that rely on coding agents and need them to behave reliably, the same principles apply to structuring how you work with AI agents — reliability comes from systems, not hope.

---

Thanks for reading. These are patterns we used extensively at Merlin to keep AI features reliable at scale (20M+ users). If you’re building production LLM systems and want to discuss approaches, feel free to reach out.