flo2 blog

Fixing 'Context Length Exceeded' Errors in LLM APIs

Fix 'context length exceeded' errors: what causes them, token budgeting, trimming/RAG/chunking, lowering max_tokens, and routing to bigger-window models.

Fixing LLM 401/403 Errors: Invalid API Key & Auth Problems

Fix LLM 401/403 errors: 401 (bad key) vs 403 (forbidden), a debugging checklist, safe logging, and centralizing keys behind a gateway.

Best Cheap, Fast LLMs (2026): Speed + Low Cost Together

The best cheap, fast LLMs in 2026: small models + fast providers, how to evaluate cost/speed/quality, and racing + A/B to land the best one.

Gemini API Free Tier (2026): What's Free & the Limits

Google's Gemini API free tier: per-minute/day limits, the data-handling caveat, getting a key, and extending free usage behind a gateway.

RAG vs Long Context: Which Should You Use (or Both)?

RAG vs long context: the cost, latency, quality and scale trade-offs, when to use each, hybrid patterns, and routing by job.

Embeddings API Guide: Turning Text into Vectors for Search & RAG

Embeddings API guide: what text embeddings are, what they power (search, RAG), the endpoint with examples, choosing a model, and production tips.

System Prompts: How to Write Them (With Examples)

System prompts explained: what to put in them, a concrete example, best practices (including caching-friendly ordering), and common mistakes.

LLM Temperature & top_p Explained: Controlling Randomness

LLM temperature and top_p explained: what they do to sampling, how they differ, and which settings to use per task (code, extraction, creative).

LLM Function Calling (Tool Use): How It Works, With Examples

LLM function calling (tool use) explained: the request/response flow with a code example, OpenAI vs Anthropic differences, and best practices.

Anthropic API Guide: Claude, Messages API & First Request

Anthropic API guide: get a key, call the Messages endpoint with curl + Python, and Claude's content blocks, tool use, streaming and prompt caching.

OpenAI API Guide: Keys, Chat Completions & First Request

OpenAI API guide: get a key, authenticate, call Chat Completions with curl + Python, and the basics of streaming, tools, JSON mode and security.

Qwen API Guide: Access Qwen Models (Open + Hosted)

How to access Qwen models (Alibaba's open-weight family) via DashScope or hosts like DeepInfra/Together — with OpenAI-SDK examples and routing.

Fireworks AI API Guide: Fast Open-Model Inference & Setup

Fireworks AI API guide: fast open-model inference, getting a key, the OpenAI-compatible endpoint with examples, pricing, and gateway routing.

Open-Weight vs Frontier LLMs: The Cost Difference, Explained

The cost gap between open-weight and frontier LLMs: why it exists, when cheap open models are good enough, and a tiered routing strategy.

Cheapest LLM for Summarization: High Volume, Low Cost

The cheapest LLMs for summarization at scale: why small models fit, what to measure, top low-cost options, and chunking/caching/batch techniques.

Databricks (Mosaic) AI Gateway Explained: Features & Fit

What the Databricks/Mosaic AI Gateway is, its Unity Catalog governance strengths, and where a standalone zero-markup BYOK gateway fits.

Azure AI Gateway (API Management): GenAI Gateway Capabilities

What the Azure AI gateway is (API Management's GenAI policies): token rate limiting, load balancing, caching, and where a provider-agnostic gateway fits.

Kong AI Gateway Explained: Plugins, Use Cases & Alternatives

What Kong AI Gateway is — AI plugins for routing, semantic caching, token rate limiting and prompt guards — and a lighter zero-markup alternative.

flo2 vs Vercel AI Gateway: A Framework-Agnostic Alternative

flo2 vs Vercel AI Gateway: integration model, provider coverage, pricing, routing features, and which suits which stack — a framework-agnostic alternative.

OpenRouter Status & Uptime: Checking It and Staying Resilient

Check OpenRouter status, understand why any provider can have downtime, and build resilience with timeouts, retries, fallback and multi-provider routing.

OpenRouter Chat Completions API: Endpoint, Setup & Example

The OpenRouter /v1/chat/completions endpoint: base URL, auth, request/response shape, model selection, streaming, and curl + Python examples.

OpenRouter Free Tier Limits (2026): What You Actually Get

How OpenRouter's free tier and :free models actually work — rate limits, daily caps, data terms, and how to go beyond them with multi-provider stacks.

Using DeepSeek on OpenRouter (and Cheaper BYOK Options)

How to call DeepSeek chat and reasoning models via OpenRouter — model slugs, a code example, cost trade-offs vs direct/BYOK, and data notes.

LLM JSON Mode & Structured Outputs: Reliable Machine-Readable Replies

LLM JSON mode and structured outputs: response_format, JSON schema, tool calling, provider differences, and best practices for valid JSON.

Gemini Flash vs GPT Mini: Picking a Cheap, Fast Workhorse

Gemini Flash vs GPT mini: comparing the cheap, fast workhorse tier on cost, speed, context and quality — and why to A/B test.

Together AI API Guide: Open Models, Setup & Pricing

Together AI API guide: open models, getting a key, the OpenAI-compatible endpoint with curl + Python, pricing, and gateway routing.

Handling LLM Timeouts: Sensible Limits, Streaming & Fallback

Handle LLM timeouts: why requests are slow, connect vs read timeouts, streaming to avoid spurious failures, and timeout + fallback.

Batch API Discounts: Cheaper LLM Calls for Async Workloads

Batch API discounts: how OpenAI/Anthropic async batch APIs cut per-token cost, when batching fits, and combining it with prompt caching.

LLM Cost Calculator: Estimate Your Token Spend Accurately

Calculate LLM API cost accurately: the token-price formula, estimation heuristics, monthly scaling, hidden costs, and logging actual spend.

Use Mistral with the OpenAI SDK: Compatible API & Base URL

Use Mistral with the OpenAI SDK: point the client at api.mistral.ai/v1, swap key and model, with examples, streaming and gotchas.

Use DeepInfra with the OpenAI SDK: Compatible API & Base URL

Use DeepInfra with the OpenAI SDK: the compatible base URL, curl + Python examples, streaming, gotchas, and gateway routing.

Helicone vs flo2: Observability-First vs Routing-First Gateways

Helicone vs flo2: observability-first vs routing-first. Where they overlap and differ, who each suits, and when to use both.

Portkey Alternative: What to Look For & Top Options

Looking for a Portkey alternative? What to evaluate (coverage, routing, caching, cost, pricing model), a fair take on Portkey, and a zero-markup BYOK option.

OpenRouter Credits & Fees: How Billing Works (2026)

How OpenRouter credits and fees work: prepaid balance, top-up/payment fees, the BYOK option, and how paying providers directly compares.

Using OpenRouter (or a Gateway) with Claude Code

Route Claude Code through OpenRouter or an Anthropic-compatible gateway by overriding the base URL — the approach, env vars, and compatibility trade-offs.

Groq Rate Limits: RPM/TPM, 429s & How to Scale Past Them

Groq rate limits explained: RPM/TPM and daily caps, reading 429s, backoff and pacing, and scaling past one key with multi-key fallback.

BYOK vs Credits: Two Ways to Pay for LLM Access

BYOK vs prepaid credits for LLM access: pay providers directly at list price vs one prepaid balance — the trade-offs and when each fits.

Self-Hosted vs Hosted LLM Gateway: Which Should You Run?

Self-hosted vs hosted LLM gateway: the ops, compliance, scale and cost trade-offs, with a comparison table to help you choose.

LiteLLM vs flo2: Self-Hosted Proxy vs Hosted Zero-Markup Gateway

LiteLLM vs flo2: self-hosted open-source proxy vs hosted zero-markup gateway — setup, ops, features, pricing, and who each suits.

Use Cerebras with the OpenAI SDK: Compatible API & Base URL

Use Cerebras Inference with the OpenAI SDK: the compatible base URL, curl + Python examples, streaming, gotchas, and gateway fallback.

Use Gemini with the OpenAI SDK: The Compatibility Endpoint

Use Gemini with the OpenAI SDK via Google's compatibility endpoint: base_url, key and model changes, curl + Python, support and gotchas.

Changing the OpenAI Base URL: Point the SDK Anywhere

How to change the OpenAI base URL in Python and Node to point the SDK at a proxy, another provider, or a local model — with examples.

Anthropic Messages vs OpenAI Chat Completions: Mapping the APIs

Anthropic Messages vs OpenAI Chat Completions: system prompts, content blocks, tool calls, streaming and usage fields — mapped, with bridging.

OpenAI Responses API Explained (vs Chat Completions)

The OpenAI Responses API vs Chat Completions: how /v1/responses differs, built-in tools and state, a code example, and when to use each.

Prompt Caching: How It Cuts LLM Cost & Latency

Prompt caching explained: how cached prompt prefixes cut LLM cost and latency, how to structure prompts for hits, and provider differences.

LLM API Pricing Compared (2026): How to Read & Compare Costs

How LLM API pricing works: input vs output tokens, cached-input discounts, context and batch costs, and how to compare true cost per task.

Cheapest LLM for Coding (2026): Strong Code at Low Cost

The cheapest LLMs for coding in 2026: what to look for, strong low-cost options, measuring cost per accepted suggestion, and tiered routing.

Zero-Markup LLM Gateway: Pay Providers Directly with BYOK

A zero-markup LLM gateway routes requests without reselling tokens — BYOK, pay list price, keep discounts. How it differs from credit aggregators.

max_tokens Explained: Control LLM Output Length & Cost

max_tokens explained: it caps output length, not the context window. Handle finish_reason length, max_completion_tokens, and set it for cost.

Input vs Output Tokens: Why They're Priced Differently

Input vs output tokens: why output is priced higher, how cached-input pricing works, and how to estimate, track and cut token costs.

Tokens Per Second (tok/s): What It Means for LLM Speed

Tokens per second (tok/s) explained: prefill vs decode, how tok/s and TTFT set total latency, what drives output speed, and how to measure it.

LLM A/B Testing: Find the Best Model & Prompt for Each Task

LLM A/B testing: compare models, prompts and providers by evidence — methodology, LLM judging, model-task fit, pitfalls, and gateway automation.

Multi-Provider LLM Strategy: Resilience, Cost & No Lock-In

A multi-provider LLM strategy for resilience, cost and no lock-in: why diversify, the challenges, and how a gateway layer makes it clean.

Streaming + Fallback: Reliable LLM Streams Across Providers

Streaming + fallback for LLMs: why fallback breaks mid-stream, the buffer-before-forward pattern, SSE errors, and resilient streams via a gateway.

LLM Response Caching: Cut Cost & Latency on Repeat Calls

LLM response caching: cut cost and latency on repeat calls — when it's safe, exact vs semantic caching, cache keys and TTL, and gateway caching.

Best Open-Source LLM APIs (2026): Models & Where to Run Them

The best open-source LLM APIs in 2026: top open-weight models, where to run them (Groq, Cerebras, DeepInfra), and how to choose and route.

Fastest LLM Inference: What Makes Models Fast & How to Get It

Fastest LLM inference explained: what makes models fast, which providers lead, and how to get the fastest result with streaming and racing.

GPT vs Claude Pricing: How OpenAI & Anthropic Costs Compare

GPT vs Claude pricing compared: how OpenAI and Anthropic structure token costs, why output dominates, and how to compare true cost per task.

DeepSeek vs Llama: Which Open Model Should You Use?

DeepSeek vs Llama compared for developers: reasoning, coding, cost, context, licensing, where to run each, and how to route between them.

OpenRouter vs LiteLLM: Hosted Aggregator vs Self-Hosted Proxy

OpenRouter vs LiteLLM: hosted credit-based aggregator vs self-hosted open-source proxy. Setup, pricing, control, lock-in, and who each suits.

Groq vs Cerebras: Speed, Models & Pricing Compared

Groq vs Cerebras for developers: LPU vs wafer-scale hardware, model lineups, throughput, pricing and OpenAI-compatible access — plus racing for speed.

Unified LLM API: One Interface for Every Model

A unified LLM API gives you one interface for every model: how a gateway normalizes formats, the benefits, and a before/after code example.

Time to First Token (TTFT): What It Is & How to Improve It

Time to first token (TTFT) explained: what drives first-token latency, why it matters for chat and agents, and how to cut it with streaming and caching.

LLM Context Windows Explained: Tokens, Limits & Cost

LLM context windows explained: what they are, how tokens and the limit work, why bigger isn't always better, and how to manage context length and cost.

BYOK for LLMs: Bring Your Own Key, Explained

BYOK for LLMs explained: bring your own provider keys to a zero-markup gateway, the benefits and trade-offs, and BYOK vs prepaid-credit aggregators.

LLM Observability: Logging, Tracing & Cost Tracking

LLM observability for developers: what to log per call (tokens, cost, latency, fallbacks), tracing agent/RAG chains, dashboards, and PII handling.

LLM Retries Done Right: Exponential Backoff & Jitter

Retry LLM calls correctly: which errors to retry vs fail fast, exponential backoff with jitter, honoring Retry-After, idempotency and circuit breakers.

LLM Model Routing: Send Each Request to the Right Model

LLM model routing explained: send each request to the right model by cost, quality, latency or task — with rules, a router model, and fallback.

xAI Grok API Guide: Models, Pricing & OpenAI-Compatible Setup

Use the xAI Grok API: get a key, call the OpenAI-compatible api.x.ai/v1 endpoint with curl + Python, how pricing works, and routing Grok behind a gateway.

DeepInfra API Guide: Cheap Open-Model Inference & Setup

Use the DeepInfra API for cheap open-model inference: get a key, call the OpenAI-compatible endpoint (curl + Python), pricing, and reliable routing.

Mistral API Guide: Models, Pricing & OpenAI-Compatible Setup

Use the Mistral API: model families, a key from La Plateforme, the OpenAI-compatible base URL with curl + Python, pricing, and routing behind a gateway.

What Is an LLM Proxy? Use Cases, Benefits & How to Run One

What is an LLM proxy? The server between your app and LLM provider APIs: what it adds (keys, logging, caching, routing, cost, security), proxy vs gateway vs router, self-host vs hosted.

LLM Load Balancing: Spread Requests Across Keys & Providers

A developer's guide to LLM load balancing: round-robin, weighted, least-in-flight and cost-aware routing across keys and providers, with 429/5xx failover and a code example.

DeepSeek API Guide: Models, Pricing & OpenAI-Compatible Setup

A developer's guide to the DeepSeek API: chat and reasoning models, getting a key, the OpenAI-compatible endpoint with curl + Python, low-cost pricing, and routing behind a gateway.

Cerebras Inference API Guide: Speed, Models, Pricing & Setup

A developer's guide to the Cerebras Inference API: wafer-scale speed, OpenAI-compatible setup with curl + Python, pricing and rate limits, and fallback for latency-critical apps.

Portkey AI Gateway: Overview, Features & Alternatives

An overview of the Portkey AI gateway: unified API, routing, observability, guardrails and prompt management, open-source vs hosted, and a fair zero-markup BYOK alternative.

LiteLLM Alternative: Hosted, Zero-Markup LLM Gateways Compared

Looking for a LiteLLM alternative? Why teams want a hosted LLM gateway instead of self-hosting the proxy, what to evaluate, a fair comparison, and a zero-markup BYOK option.

OpenRouter API Key: How to Get, Use & Secure It

Create and use an OpenRouter API key: sign-in steps, bearer-token auth, the OpenAI-compatible base URL, curl + Python examples, security best practices, and the BYOK alternative.

OpenRouter Rate Limits: Free vs Paid & How to Avoid 429s

How OpenRouter rate limits work in 2026: why free/:free models and low balances get throttled, what a 429 means, and how to avoid them with backoff, BYOK and gateway fallback.

Use Groq with the OpenAI SDK: Base URL & Compatible API

Groq is OpenAI-compatible at api.groq.com/openai/v1. curl, Python & Node examples for /chat/completions, streaming, what's supported, and migrating an OpenAI app to Groq.

What Is an AI Gateway? Definition, Features & Why You Need One

What is an AI gateway? The managed layer between your app and AI providers: definition, core features, AI gateway vs LLM gateway vs proxy, and when you need one.

Google Gemini API Guide: Keys, Free Tier, Pricing & Setup

A developer's guide to the Gemini API: get a key from Google AI Studio, choose Pro vs Flash, understand the free tier and pricing, and call it via the OpenAI-compatible endpoint.

Groq API Guide (2026): Setup, Speed, Pricing & Rate Limits

A developer's guide to the Groq API: create a key, the OpenAI-compatible base URL, curl + Python examples, what fast LPU inference is best for, and handling rate limits with fallback.

Cloudflare AI Gateway Explained: Caching, Analytics & Limits

What is Cloudflare AI Gateway? How it works, its edge caching, analytics and rate limits, what to confirm on pricing, and a fair comparison to a routing-first gateway.

Vercel AI Gateway Explained: Features, Pricing & Alternatives

What Vercel AI Gateway is, how it fits Next.js and the Vercel AI SDK, how its pricing model works, and how a framework-agnostic, zero-markup BYOK alternative compares.

OpenRouter Free Models (2026): What's Free, the Limits & Smarter Free Options

How OpenRouter free models work in 2026: the :free tier, rate limits and data terms, how to find the current free models, and stacking free tiers behind a gateway.

What Is OpenRouter? How It Works + the BYOK Alternative

What is OpenRouter? How the unified LLM API works (credits, routing, OpenAI-compatible), what it's great at, the reseller trade-off, and the zero-markup BYOK alternative.

How to Reduce LLM Latency: TTFT, Streaming, Racing & Caching

Cut LLM latency with practical levers: stream tokens to lower TTFT, pick faster inference providers, trim output, cache prefixes and responses, and race the tail.

Fixing LLM 429 Rate Limit Errors: Backoff, Retries & Fallback

Hitting HTTP 429 from an LLM API? Learn why RPM/TPM limits trigger 429s, how to read Retry-After headers, and fix them with backoff, jitter, queues and fallback.

AI Tokenomics: A Practical Framework to Cut Your LLM Bill

AI tokenomics explained: input vs output vs cached pricing, the six levers to reduce LLM costs, and how to measure true cost per successful task.

OpenAI-Compatible API: One Base URL for Every Model

What an OpenAI-compatible API is, how to switch providers with a base_url override, parity gotchas, and routing every model through one endpoint.

What Is LiteLLM? The Open-Source LLM Proxy, Explained

What is LiteLLM? A developer's guide to the open-source LLM proxy and Python SDK, its routing and fallback features, self-host tradeoffs, and hosted alternatives.

OpenRouter Pricing Explained (2026): Credits, Fees & BYOK

How OpenRouter pricing really works in 2026: credits, fees, BYOK surcharges and where markup hides. Compare true per-token cost vs a zero-markup BYOK gateway.

Ollama's OpenAI-Compatible API: Call /v1/chat/completions

Call a local Ollama model with the OpenAI SDK. curl + Python examples for /v1/chat/completions, streaming, base URL, gotchas, and local+cloud routing.

Free LLM APIs in 2026: Real Free Tiers, Limits & How to Use Them

A developer's guide to free LLM APIs in 2026: real free tiers (Gemini, Groq, Mistral), self-hosting, the catches, and how to stack keys to stay free.

Best LLM Gateway in 2026: flo2 vs Vercel AI Gateway vs Cloudflare vs Portkey

A fair buyer's guide to LLM / AI gateways — Vercel, Cloudflare, Portkey, Helicone, LiteLLM, OpenRouter and flo2 — across pricing, markup, routing, caching and cost accounting.

LLM Fallback and Racing: Build an LLM API That Never Goes Down

Retries, fallback chains, load balancing and racing — how to design a production LLM API that survives provider outages, rate limits and slow-tail latency.

The Best OpenRouter Alternative in 2026: Bring Your Own Keys, Zero Markup

Looking for an OpenRouter alternative? Compare resold-credit aggregators vs bring-your-own-key, zero-markup gateways on pricing, transparency, routing and lock-in.

The Cheapest LLM API in 2026: How to Actually Pay Less per Token

There is no single cheapest LLM API — only the cheapest model per task. The 2026 price tiers and the routing, caching and BYOK tactics that cut your LLM bill.

What Is an LLM Gateway? A Developer's Guide for 2026

An LLM gateway puts one API in front of every model provider — routing, fallback, caching and cost accounting. What it is, the problems it solves, and how to choose one.