The 5 Best LiteLLM Alternatives: Figure Out Which Job You're Replacing First

Let's be honest up front: if your LiteLLM setup is running fine, you probably don't need to read this whole thing — skip to the last section. What actually makes people want to rip it out is never that it's "not powerful enough." It's the specific, annoying moments — a proxy worker chewing through RAM at 2am until the OOM killer takes it down; RSS creeping up on Kubernetes to the point where the official production docs literally tell you to recycle workers every N requests; a config.yaml that has ballooned to 200+ lines nobody dares touch; a minor version bump that quietly changed your fallback behavior and you only found out in prod. Those are what make someone type "should we switch gateways?" into Slack.

So this piece isn't going to crown a single winner. "LiteLLM alternatives" is a slightly misleading phrase — almost nobody replaces LiteLLM wholesale. You replace one or two of the jobs it does. Once that clicks, the whole decision gets a lot simpler. If you're not deeply familiar with LiteLLM yet, our complete LiteLLM guide is the primer; this article assumes you've already run it in anger.

1. First: which LiteLLM job are you actually replacing?

LiteLLM is quietly doing four jobs for you at once, plus a fifth it doesn't touch but you'll eventually have to deal with:

• Unified interface — flattening the mismatched SDKs of OpenAI, Anthropic, Gemini, and a pile of Chinese models into one OpenAI-shaped call. This is its core value, and the hardest thing to replace cleanly.
• Cost tracking — who spent how much, which key blew its budget, an itemized ledger.
• Failover — auto-switching to a backup when the primary route dies, rotating keys to dodge rate limits.
• Observability — logs, latency, errors, per-call token counts.
• Owning inference — the one job LiteLLM doesn't do: whose GPU is actually running these models?

Each of the five "alternatives" below bites off exactly one or two of those. OpenRouter takes over "unified interface + failover," fully managed. Portkey grabs "unified interface + observability + governance/guardrails." LangSmith really only does the "observability/eval" job — treating it as a gateway replacement is a misread. Cloudflare AI Gateway gives you "observability + caching + cost tracking" at the edge, but as a passthrough. vLLM/Ollama does the entirely separate fifth job: owning inference. Once you see the division of labor, you stop agonizing over "which is best" and just ask "which one is missing for me?"

2. The comparison table, at a glance

The column worth remembering is the second one. Notice that none of them lifts all four of LiteLLM's jobs at once — because LiteLLM is itself a bundle of several things stitched together. Now the walkthroughs. Where a tool deserves more, it gets more; the edge cases get two sentences.

3. OpenRouter: if you just want it handled, this is it

What it is: a managed model aggregator. One API key, one base_url, and behind it sit 400–500+ models across 60-odd providers. You maintain nothing. When the primary route slows or dies, it fails over to another provider at the edge, and your app barely notices.

The setup is almost comically easy — often it really is one environment variable:

# OpenRouter: swap the base_url and your OpenAI SDK just works
export OPENAI_API_KEY="sk-or-..."
export OPENAI_BASE_URL="https://openrouter.ai/api/v1"

Performance is better than people expect. It runs at the edge, so routing adds only about 15–25 ms of overhead; in one public latency benchmark in 2026, OpenRouter's time-to-first-token actually beat calling OpenAI directly by roughly 70 ms, because it sits closer to the user and picks the fastest live provider instance. Trailing-90-day uptime is 99.99%. For a team that just wants to bounce between GPT and Claude reliably, it's genuinely painless.

Now the annoying bits. First, that 5.5% isn't skimmed off your tokens — it's skimmed off the money you load in. Every credit top-up loses 5.5% via card (5% crypto), with an $0.80 minimum per transaction. That's real margin, and it stings at volume. Second, the BYOK math (bring your own upstream key) changed on July 14, 2026: it used to be a free request-count allowance (1M/mo on Pay-as-you-go, 5M on Enterprise), and now it's measured as list-price inference — free up to $25,000/mo on PAYG ($200,000 on Enterprise), and past that you pay 5% of what the call would have cost on OpenRouter's own platform. If your whole reason for BYOK was to save money, redo the math. Third, and this is the one an engineer should care about most: the routing is a black box. You don't fully control which provider — or which quantization — is serving you. Same model name, potentially different real-world quality depending on the backend it landed on.

One line: if you just want to switch between a couple of closed frontier models without running anything — OpenRouter, don't overthink it. If you're sensitive about cost granularity or about knowing exactly who's serving you, that black box will bug you. For direct-connect relay options there's more choice; see our AI API relay comparison alongside this.

4. Portkey: the most complete — but it has a new owner

What it is: gateway + observability + guardrails in one. It's long been the most complete option for "I want a unified gateway and enterprise-grade observability and guardrails" — routing, rate limiting, policy enforcement, and full visibility into every model interaction in a single product, reportedly processing trillions of tokens a month. On capability alone it's more enterprise-shaped than OpenRouter and more gateway-shaped than LangSmith.

One line: on capability it's still the most complete gateway-type option of the five. But at this point in mid-2026 I wouldn't anchor a brand-new, vendor-neutral project on it — wait and see where the post-acquisition roadmap lands. Conversely, if you're already inside Palo Alto's ecosystem and guardrails/compliance are hard requirements, the acquisition works in your favor.

5. LangChain / LangSmith: calling it a gateway replacement is the wrong axis

Let's clear up a common mix-up: putting LangChain/LangSmith on a "LiteLLM alternatives" list is picking the wrong axis. LangChain is an app-orchestration framework; LangSmith is a tracing/eval/observability platform. Neither is a gateway, and neither does the "flatten every SDK into one format" job that is LiteLLM's whole reason to exist.

But if the thing you actually wanted from LiteLLM and never quite got is the observability job — you want to see the full call chain of every request, you want a real eval harness — then LangSmith is a legitimate answer. Its trace trees and evaluation tooling go much deeper than LiteLLM's built-in logging.

The cost comes in two layers. One, it's not a transparent proxy — you have to instrument your code, so it's not the painless base_url swap the others are. Two, the pricing bites. The free Developer tier is 5,000 traces/month, 14-day retention, one seat. Plus is $39/seat/month with 10,000 base traces included, overage at $2.50 per 1,000, and if you want 400-day extended retention that's $5.00 per 1,000. Seats stack linearly — ten people is $390/month before any overage. The real trap is trace overage: a five-person team producing 2 million traces a month can run north of $5,000/month. Estimate your call volume before you instrument, not when the invoice lands.

One line: choose LangSmith when what you're missing is "seeing what actually happened on every call" plus an eval loop. If you wanted a gateway, it's answering a different question.

6. Cloudflare AI Gateway: the underrated near-freebie

What it is: a control plane at the edge that gives you observability, caching, rate limiting, and cost tracking. It's the easiest one to overlook here and, in the right situation, the best value of the lot.

The economics are pretty. Core gateway features are free with no per-call fee; the free tier includes 100,000 logs/month, and past that Workers Paid ($5/month) gets you 1 million. The new-for-2026 Unified Billing even lets you fold third-party model usage (OpenAI and friends) straight onto your Cloudflare invoice, at the cost of a 5% fee on credits. The caching genuinely saves money — a hit returns straight from the edge and never touches the upstream.

One line: if you're already on Cloudflare and what you want is exactly observability, caching, and a rate-limit choke point — this layer is nearly free and close to zero-ops. Just don't expect it to hand you LiteLLM's "one format for every provider."

7. Self-hosted vLLM / Ollama: a completely different axis

Strictly speaking these two aren't "LiteLLM alternatives" at all — they replace that fifth job, owning inference, i.e. running the model yourself. In real engineering you usually don't swap vLLM in for LiteLLM; you run the model on vLLM and put a LiteLLM (or another gateway) in front of it. They stack; it's not either/or.

Ollama is the king of local and prototype work: five minutes to a running model on your own machine, lovely with Open WebUI or a RAG that doesn't need concurrency. But its ceiling is hard. By default it handles roughly 4 concurrent requests (OLLAMA_NUM_PARALLEL defaults to 1, sometimes 4 on high-VRAM boxes), and under load it settles around 41 tokens/second. At 128 concurrent users its P99 latency spikes to 673 ms while vLLM in the same test stays under 100 ms. It also has a nasty quirk: when multiple users hit the same model, it pins the load to a single GPU and leaves the other cards idle. The blunt takeaway: Ollama belongs on your own machine, not carrying production concurrency.

vLLM is the real answer for self-hosted production. PagedAttention plus continuous batching push throughput up to 24x over naive implementations, and by 2026 it's the baseline for any serious inference stack — born at UC Berkeley's Sky Computing Lab, now a core PyTorch-ecosystem project, run in production at LinkedIn and Uber. Stripe cut inference cost 73% after migrating to it, serving 50 million calls a day on a third of the GPU fleet. The price is ops: you own the GPUs, the scheduling, the autoscaling, the monitoring. This is not a change-one-base_url job.

One line: go self-hosted on vLLM when data can't leave the building, when scale makes running it yourself cheaper, or when you need your own fine-tuned model — and keep Ollama for your laptop and demos. And remember: once it's running, you'll probably still want a gateway in front of it, which loops you back to the options above.

8. Which one, in plain English

• You just want to switch between OpenAI and Claude with zero ops → OpenRouter, don't overthink it. Just accept the 5.5% top-up fee.
• You're already on Cloudflare and want observability + caching + rate limiting → Cloudflare AI Gateway, near-freebie, wired up in a day.
• Enterprise, need guardrails / compliance / audit, especially if you're already a Palo Alto customer → Portkey. If you're a new project that values vendor neutrality, wait and watch.
• What you actually lack is "seeing every call + running evals" → LangSmith, but size up your trace volume first and keep an eye on that ballooning invoice.
• Data can't leave / scale makes self-hosting cheaper / you run fine-tuned models → self-host vLLM with a gateway in front; keep Ollama for local dev.

9. The honest conclusion: a lot of the time, don't switch

After all of that, the most contrarian — and most worth-saying — line is this: a lot of the time the best "alternative" is not switching at all.

The memory problem that makes people want to bail has a known fix. LiteLLM's own production docs spell it out: configure worker recycling — restart each worker after it has served N requests to keep the slow memory creep in check — pin your version so a minor bump doesn't surprise you, and put a memory limit on the process. Roughly this:

# Run the LiteLLM proxy under gunicorn and recycle workers after a fixed
# number of requests to keep the slow memory growth in check (tune to your load)
litellm --config config.yaml \
  --num_workers 4 \
  --run_gunicorn --max_requests 1200 --max_requests_jitter 200

And worth noting: LiteLLM is itself moving to a Rust core (its own tagline is now "Rust core with Python SDK"), so the high-concurrency overhead that used to be stuck behind Python's GIL is steadily getting solved — as of v1.78.5 the median added overhead at 1,000 concurrent requests is down to 8 ms. In other words, the problem you hit today might be gone next release.

Here's the slightly rude version: most people switch gateways because of one bad afternoon, not because they hit a real architectural ceiling. Work out whether you hit the ceiling or just a config accident before you commit to a migration. Migrations aren't free either — don't trade a pitfall you already know how to route around for a brand-new one.