Building an LLM Gateway with LiteLLM (2026)

You've got Claude Code, Codex CLI, a DeepSeek API key, and a free GLM quota — but the keys live in four different places, every tool is configured differently, and your billing is a mess to track. LiteLLM Proxy turns those four into one: a local HTTP gateway with a unified OpenAI-compatible interface. Every AI tool points at the same address, and you manage all your keys and routing rules in one config.yaml.

This is a hands-on tutorial — no theory, straight to building. Starting from zero, by the end you'll have:

• Claude Code routing through LiteLLM Proxy to DeepSeek / GLM (direct-connect from mainland China, lower cost)
• OpenAI Codex CLI using the same Proxy to mix Claude Sonnet 5 and DeepSeek for coding
• Automatic failover when your primary route goes down, and automatic key rotation across multiple keys (to dodge rate limits)
• Virtual keys you can hand out to teammates, each with its own spending cap
• A production Docker deployment with PostgreSQL + Redis

1. Architecture: What You're Building

┌─────────────────────────────────────────────────┐
│           Your local machine / server            │
│                                                 │
│  Claude Code ──┐                                │
│  Codex CLI   ──┼──► LiteLLM Proxy :4000 ──────► │──► Anthropic API
│  any script  ──┘   (routing managed by config)   │──► OpenAI API
│                     │                           │──► DeepSeek API (direct connect)
│                     ├── load balancing           │──► Zhipu GLM API (direct connect)
│                     ├── failover                 │──► SiliconFlow and other relays
│                     ├── virtual key management   │
│                     └── cost tracking            │
└─────────────────────────────────────────────────┘

Every AI tool's requests hit http://localhost:4000, and LiteLLM handles:

• Translating Anthropic-format requests (from Claude Code) into OpenAI / DeepSeek / GLM format
• Routing OpenAI-format requests (from Codex) to any provider
• Automatically falling back to a backup when the primary route fails
• Rotating across multiple API keys so a single key doesn't get rate-limited

2. Installing LiteLLM

# Recommended: uv (10x faster)
uv tool install 'litellm[proxy]'

# Or pip
pip install 'litellm[proxy]'

# Verify
litellm --version
# Should print v1.91.0 or higher

3. The Core config.yaml Structure

Every setting for LiteLLM Proxy lives in a single config.yaml file. Here's the skeleton:

model_list:              # the "model menu" you expose to clients
  - model_name: xxx      # the name clients call
    litellm_params:
      model: provider/model-id   # the actual provider + model being called
      api_key: os.environ/XXX    # read the key from an env var, never hardcode it

litellm_settings:        # global behavior
  drop_params: true      # silently drop params the target model doesn't support
  num_retries: 3

general_settings:        # proxy server settings
  master_key: sk-1234    # the key clients must pass to call the proxy

router_settings:         # routing strategy
  routing_strategy: simple-shuffle

model_name is the name external tools see — call it whatever you want. litellm_params.model is what LiteLLM actually forwards the request to.

4. Wiring Up Claude Code

Claude Code speaks the Anthropic Messages API format. LiteLLM automatically translates that format to whatever the target provider expects, then translates the response back.

4.1 How it works

Claude Code reads two environment variables:

• ANTHROPIC_BASE_URL: where it sends API requests (point this at your LiteLLM address)
• ANTHROPIC_AUTH_TOKEN: the Authorization Bearer token (use LiteLLM's master_key)

4.2 Setup steps

Step 1: In config.yaml, configure the models you want Claude Code to use:

model_list:
  # Primary: Claude Sonnet 5 (official Anthropic)
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

  # Cheap fallback: DeepSeek V4 Flash (direct connect, ~90% cheaper)
  - model_name: deepseek-flash
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_API_KEY

  # Free fallback: GLM-4.7-Flash (permanently free)
  - model_name: glm-flash
    litellm_params:
      model: zai/glm-4.7-flash
      api_key: os.environ/ZAI_API_KEY

general_settings:
  master_key: sk-my-proxy-key

Step 2: Start the LiteLLM Proxy:

litellm --config config.yaml
# Output: LiteLLM Proxy running on http://0.0.0.0:4000

Step 3: Set the environment variables, then launch Claude Code:

# Set env vars (add to ~/.zshrc or ~/.bashrc to make them permanent)
export ANTHROPIC_BASE_URL="http://localhost:4000"
export ANTHROPIC_AUTH_TOKEN="sk-my-proxy-key"

# Use native Anthropic Claude (default)
claude

# Switch to DeepSeek (cheaper)
claude --model deepseek-flash

# Switch to free GLM
claude --model glm-flash

# Enable in-panel /model switching (requires Claude Code v2.1.129+)
export CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1
claude  # once inside, use /model to list and switch models

4.3 Verifying it works

# curl directly to check the Proxy is forwarding Claude-Code-format requests
curl http://localhost:4000/v1/messages \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-my-proxy-key" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "deepseek-flash",
    "max_tokens": 50,
    "messages": [{"role": "user", "content": "reply with ok"}]
  }'
# Should return DeepSeek's response, but in Anthropic Messages format

5. Wiring Up OpenAI Codex CLI

Codex CLI (@openai/codex) speaks the OpenAI Chat Completions format. It reads two variables: OPENAI_BASE_URL and OPENAI_API_KEY.

5.1 Install Codex CLI

npm install -g @openai/codex
# or
yarn global add @openai/codex

5.2 Configuration

Codex uses the OpenAI format directly against the /v1/chat/completions endpoint — no extra translation layer needed:

# Point Codex at the LiteLLM Proxy (OpenAI-compatible endpoint)
export OPENAI_BASE_URL="http://localhost:4000"
export OPENAI_API_KEY="sk-my-proxy-key"   # LiteLLM master_key

# Run Codex on Claude Sonnet 5 (forwarded through the Proxy)
codex --model claude-sonnet-5

# Run Codex on DeepSeek (cheaper)
codex --model deepseek-flash --full-auto

# Run Codex on the free GLM tier
codex --model glm-flash

5.3 A persistent Codex config.toml

# ~/.codex/config.toml
model = "deepseek-flash"          # default to DeepSeek
provider = "openai"
base-url = "http://localhost:4000"

[model-options]
temperature = 0.2

Once this is set, just run codex directly — no need to pass --model every time.

6. Wiring Up DeepSeek (Multi-Key Load Balancing)

DeepSeek's official API (api.deepseek.com) isn't directly reachable from mainland China, so it's typically accessed through a mainland relay like SiliconFlow. Multiple account keys let you spread rate-limit pressure across accounts.

6.1 Through SiliconFlow (direct connect)

model_list:
  # SiliconFlow key 1 (primary)
  - model_name: deepseek-flash
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_KEY_1
      rpm: 50   # this key: max 50 requests per minute

  # SiliconFlow key 2 (spreads the load)
  - model_name: deepseek-flash
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_KEY_2
      rpm: 50

  # DeepSeek flagship (higher quality, pricier)
  - model_name: deepseek-pro
    litellm_params:
      model: openai/deepseek-v4-pro
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_KEY_1

With two entries sharing model_name: deepseek-flash, LiteLLM automatically load-balances between them — when key 1 gets rate-limited, it transparently switches to key 2.

6.2 Through DeepSeek's official API (requires a proxy/VPN)

  - model_name: deepseek-official
    litellm_params:
      model: deepseek/deepseek-chat   # litellm's built-in deepseek/ prefix
      api_key: os.environ/DEEPSEEK_API_KEY

6.3 Setting environment variables

# In a .env file, or export directly
export SILICONFLOW_KEY_1="sk-sf-xxxxxxxx"
export SILICONFLOW_KEY_2="sk-sf-yyyyyyyy"
export DEEPSEEK_API_KEY="sk-deepseek-zzzzzzzz"

7. Wiring Up Zhipu GLM (Including the Free Tier)

Zhipu AI (bigmodel.cn) has a direct-connect API from mainland China, and GLM-4.7-Flash is permanently free — a 31B MoE model, 59.2% on SWE-bench, 200K context — a solid low-cost fallback.

7.1 Sign up and get a key

1. Register at bigmodel.cn
2. Complete identity verification to get 20 million free tokens
3. Generate an API key from the console

7.2 config.yaml setup

model_list:
  # Free GLM flagship-tier (permanently free, strong at coding)
  - model_name: glm-flash
    litellm_params:
      model: zai/glm-4.7-flash    # LiteLLM's built-in zai/ prefix supports Zhipu
      api_key: os.environ/ZAI_API_KEY

  # GLM paid flagship (stronger, paid)
  - model_name: glm-pro
    litellm_params:
      model: zai/glm-5.2
      api_key: os.environ/ZAI_API_KEY

  # GLM vision model (free)
  - model_name: glm-vision
    litellm_params:
      model: zai/glm-4.6v-flash
      api_key: os.environ/ZAI_API_KEY

export ZAI_API_KEY="your-glm-api-key"

7.3 Verifying GLM connectivity

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-my-proxy-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-flash",
    "messages": [{"role": "user", "content": "Introduce yourself in one sentence"}]
  }'

8. Failover: Auto-Switch When the Primary Goes Down

This is one of LiteLLM's most useful features: when your primary model fails (rate-limited, down, erroring), it automatically switches to a backup — completely invisible to the caller.

8.1 Setting priority with the order parameter

model_list:
  # Primary: Claude Sonnet 5 (strongest, tried first)
  - model_name: best-model
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
      order: 1   # highest priority

  # Backup 1: DeepSeek (cheap, direct connect)
  - model_name: best-model
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_KEY_1
      order: 2   # used if Claude goes down

  # Backup 2: free GLM (last resort)
  - model_name: best-model
    litellm_params:
      model: zai/glm-4.7-flash
      api_key: os.environ/ZAI_API_KEY
      order: 3   # final fallback

All entries share the same model_name, so callers just use best-model. LiteLLM tries order=1 first, escalates to order=2 on failure, then order=3.

8.2 Cross-group fallbacks with the fallbacks parameter

litellm_settings:
  # Cross-group fallback when the primary model fails
  fallbacks:
    - claude-sonnet-5: ["deepseek-flash", "glm-flash"]
    - deepseek-flash: ["glm-flash"]

  # Dedicated fallback for rate limits (429 errors)
  content_policy_fallbacks:
    - claude-sonnet-5: ["deepseek-flash"]

  # Fallback when the context window is exceeded
  context_window_fallbacks:
    - claude-sonnet-5: ["glm-flash"]   # GLM has a 200K context window

  num_retries: 3        # retry each deployment 3 times before switching
  allowed_fails: 3      # more than 3 failures in 1 minute triggers cooldown
  cooldown_time: 30     # cooldown for 30 seconds

9. The Full config.yaml (All Four, Combined)

Merging everything above into one complete, ready-to-use config.yaml:

# config.yaml — LiteLLM all-in-one gateway config
# Claude Code + Codex CLI + DeepSeek + GLM

model_list:
  # ── Claude Sonnet 5 (official Anthropic) ─────────────────
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

  # ── DeepSeek Flash (via SiliconFlow, direct connect, dual-key rotation) ──
  - model_name: deepseek-flash
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_KEY_1
      rpm: 50

  - model_name: deepseek-flash
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_KEY_2
      rpm: 50

  # ── DeepSeek flagship (higher quality) ───────────────────
  - model_name: deepseek-pro
    litellm_params:
      model: openai/deepseek-v4-pro
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_KEY_1

  # ── GLM free tier (permanently free fallback) ────────────
  - model_name: glm-flash
    litellm_params:
      model: zai/glm-4.7-flash
      api_key: os.environ/ZAI_API_KEY

  # ── GLM paid flagship ─────────────────────────────────────
  - model_name: glm-pro
    litellm_params:
      model: zai/glm-5.2
      api_key: os.environ/ZAI_API_KEY

  # ── best-model: a priority-ordered auto-failover group ───
  - model_name: best-model
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
      order: 1

  - model_name: best-model
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://api.siliconflow.cn/v1
      api_key: os.environ/SILICONFLOW_KEY_1
      order: 2

  - model_name: best-model
    litellm_params:
      model: zai/glm-4.7-flash
      api_key: os.environ/ZAI_API_KEY
      order: 3

litellm_settings:
  drop_params: true
  num_retries: 3
  request_timeout: 60
  allowed_fails: 3
  cooldown_time: 30
  fallbacks:
    - claude-sonnet-5: ["deepseek-flash", "glm-flash"]
    - deepseek-flash: ["glm-flash"]
  context_window_fallbacks:
    - claude-sonnet-5: ["glm-flash"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  alerting: ["slack"]    # optional, Slack alerts for slow requests/errors

router_settings:
  routing_strategy: simple-shuffle

# .env file (same directory)
ANTHROPIC_API_KEY=sk-ant-xxx
SILICONFLOW_KEY_1=sk-sf-xxx
SILICONFLOW_KEY_2=sk-sf-yyy
ZAI_API_KEY=your-glm-key
LITELLM_MASTER_KEY=sk-my-proxy-2026

# Start it (automatically reads .env from the current directory)
litellm --config config.yaml

# Verify every model is reachable
curl http://localhost:4000/models \
  -H "Authorization: Bearer sk-my-proxy-2026" | python3 -m json.tool

10. Virtual Keys and Team Budgets

If you're running this for a team, don't hand out the master_key to everyone — give each person a virtual key with its own limit:

10.1 Creating a virtual key

# Create a virtual key for developer Alice ($5/month cap, restricted to deepseek-flash and glm-flash only)
curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-my-proxy-2026" \
  -H "Content-Type: application/json" \
  -d '{
    "team_id": "dev-team",
    "max_budget": 5.0,
    "budget_duration": "30d",
    "models": ["deepseek-flash", "glm-flash"],
    "metadata": {"user": "alice@company.com"}
  }'

# Returns: {"key": "sk-virtual-abc123", "expires": "2026-08-08", ...}

# Alice uses her virtual key with Codex — she can only call the allowed models, and requests are auto-rejected once she's over budget
export OPENAI_API_KEY="sk-virtual-abc123"
export OPENAI_BASE_URL="http://localhost:4000"
codex --model deepseek-flash

10.2 Checking spend

# Check spend details for a specific key
curl http://localhost:4000/key/info?key=sk-virtual-abc123 \
  -H "Authorization: Bearer sk-my-proxy-2026"

# Check aggregate spend across all keys
curl http://localhost:4000/global/spend \
  -H "Authorization: Bearer sk-my-proxy-2026"

11. Production Docker Deployment

11.1 Quick single-container deployment

# docker-compose.yml (minimal)
version: '3.8'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    env_file:
      - .env
    command: --config /app/config.yaml --port 4000 --num_workers 8
    restart: unless-stopped

docker compose up -d
docker compose logs -f litellm   # tail the logs

11.2 Full production deployment (PostgreSQL + Redis)

For production traffic at 1000+ RPM, you need PostgreSQL to persist virtual keys and spend data, and Redis to share rate-limit state across instances:

# docker-compose.prod.yml
version: '3.8'
services:
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${'{POSTGRES_PASSWORD}'}
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 10s

  redis:
    image: redis:7-alpine
    volumes:
      - redis-data:/data

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    env_file:
      - .env
    environment:
      - DATABASE_URL=postgresql://litellm:${'{POSTGRES_PASSWORD}'}@postgres:5432/litellm
      - REDIS_HOST=redis
      - REDIS_PORT=6379
    command: --config /app/config.yaml --port 4000 --num_workers 8
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    restart: unless-stopped

volumes:
  postgres-data:
  redis-data:

Enable Redis under router_settings in config.yaml:

router_settings:
  routing_strategy: usage-based-routing  # this strategy requires Redis
  redis_host: redis
  redis_port: 6379
  cache_responses: true   # identical requests are served straight from cache, cutting cost

12. Debugging and Troubleshooting

12.1 Turning on verbose logging

# Start in debug mode
litellm --config config.yaml --detailed_debug

# Shows: routing decisions, retry logs, and the actual provider used for every request

12.2 Health checks

# Check the Proxy's own status
curl http://localhost:4000/health

# Check whether every configured model is reachable
curl http://localhost:4000/health/liveliness \
  -H "Authorization: Bearer sk-my-proxy-2026"

12.3 Common errors, quick reference

13. FAQ

Q: My Claude Code has always talked to Anthropic directly — will routing it through the Proxy break tool use?

A: LiteLLM has full support for Claude Code's Anthropic Messages API, including tool_use, system prompts, vision, and streaming. If the target model is native Anthropic (the anthropic/ prefix), the request passes through with zero loss. If the target is DeepSeek or GLM, LiteLLM converts Anthropic's tool_use format into OpenAI's function_calling format — most tool-use scenarios work fine, though a handful of Anthropic-specific tool features may need adjustment.

Q: Is running Codex CLI's --full-auto mode through the Proxy safe?

A: Safety depends on the model and your system, not on the Proxy. --full-auto lets Codex execute file operations automatically — we'd recommend running it in an isolated Docker container or VM regardless of whether you're using a Proxy.

Q: Is the free GLM-4.7-Flash good enough for Claude Code coding work?

A: GLM-4.7-Flash scores 59.2% on SWE-bench, roughly on par with early Claude 3.5 Sonnet — it handles medium-sized tasks just fine. For complex multi-file refactors, switch back to Claude Sonnet 5; for everyday code generation, explanations, and bug fixes, GLM-Flash is plenty and costs nothing.

Q: Does the Proxy store my request content?

A: Not by default. It only logs content if you enable a callback like Langfuse or MLflow. Virtual-key spend records (token counts, cost) are stored in PostgreSQL, but that doesn't include the request content itself.

Q: Can I deploy the Proxy in the cloud so my whole team can use it?

A: Yes — that's the classic use case for Proxy mode. Deploy it to a cloud server, change ANTHROPIC_BASE_URL=https://your-domain, and add HTTPS (Nginx/Caddy) and a firewall. Hand out virtual keys to each person — no need to distribute real API keys.