Heretek-AI/arcane-repo
1
0
Fork 0
mirror of https://github.com/Heretek-AI/arcane-repo.git synced 2026-07-01 18:25:50 -04:00
Code Issues Packages Projects Releases Wiki Activity

feat(litellm): align with upstream — add PostgreSQL + Prometheus to compose

Browse Source 
- docker-compose.yml: add bundled PostgreSQL 16 (db) and Prometheus services,
  healthchecks on both, SERVICE_CONNECTED condition before starting litellm,
  STORE_MODEL_IN_DB=True, hardcoded DATABASE_URL pointing to db container
- prometheus.yml: new file — scrape litellm:4000 every 15s with 15d retention
- .env.example: remove redundant DATABASE_URL docs, add port variable hints
- README.md: document all 3 services, their ports, Prometheus queries,
  and all-in-one quick start
This commit is contained in:
John Doe
2026-05-10 16:23:00 -04:00
parent 5fed07433e
commit 1738bdb9d8
4 changed files with 134 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
templates/litellm/.env.example
+10 -7
View File
@@ -10,11 +10,11 @@ LITELLM_MASTER_KEY=sk-liteLLM-master-key-change-me
# Salt for encrypting stored credentials.
LITELLM_SALT_KEY=sk-liteLLM-salt-key-change-me
# ── DATABASE (optional) ───────────────────────────────────────────────────────
# Leave blank to use SQLite inside the container (data persists in the litellm_data volume).
# To use PostgreSQL, set the full connection string:
# ── DATABASE ────────────────────────────────────────────────────────────────
# PostgreSQL is bundled and managed by Docker Compose (litellm_db on port 5432).
# The DATABASE_URL below is hardcoded in docker-compose.yml — no need to set it here.
# To use an external database instead, uncomment and edit:
# DATABASE_URL=postgresql://user:password@host:5432/litellm
DATABASE_URL=
# ── PROVIDER API KEYS ────────────────────────────────────────────────────────
# Set at least one. LiteLLM reads these automatically — no config changes needed.
@@ -39,10 +39,13 @@ COHERE_API_KEY=
# Together AI (Llama, Qwen, DeepSeek, etc.)
TOGETHERAI_API_KEY=
# ── PROXY SETTINGS (optional) ─────────────────────────────────────────────────
# Port exposed on the host (default: 4000)
LITELLM_PORT=4000
# ── PORTS (optional) ────────────────────────────────────────────────────────
# Override defaults if ports conflict with other services.
# LITELLM_PORT=4000 # Proxy API (default: 4000)
# LITELLM_DB_PORT=5432 # PostgreSQL (default: 5432)
# LITELLM_PROM_PORT=9090 # Prometheus (default: 9090)
# ── PROXY SETTINGS (optional) ─────────────────────────────────────────────────
# Drop unsupported params from requests to providers
LITELLM_DROP_PARAMS=false
templates/litellm/README.md
+68 -39
View File
@@ -2,6 +2,14 @@
[LiteLLM](https://github.com/BerriAI/litellm) is a lightweight proxy that provides an OpenAI-compatible API endpoint for 100+ LLM providers. Route requests to OpenAI, Anthropic, Google Gemini, open-source models via Together AI, local models via Ollama, and many more — all through a single endpoint with consistent error handling, rate limiting, and spend tracking.
## What's Included (All-in-One)
| Service | Port | Description |
|-------------|-------|------------------------------------------------|
| LiteLLM | 4000 | OpenAI-compatible proxy API |
| PostgreSQL | 5432 | Persistent database for spend logs and keys |
| Prometheus | 9090 | Metrics and observability dashboard |
## Quick Start (3 Steps)
### 1. Create your environment file
@@ -12,13 +20,13 @@ cp .env.example .env
Open `.env` and set at least one provider API key. The defaults are fine to get started locally.
### 2. Start the proxy
### 2. Start everything
```bash
docker compose up -d
```
That's it. LiteLLM is running at `http://localhost:4000`.
Docker Compose starts all three services (LiteLLM, PostgreSQL, Prometheus) and waits for the database to be healthy before starting the proxy.
### 3. Test it
@@ -34,35 +42,56 @@ curl http://localhost:4000/v1/chat/completions \
> **Tip:** Get an API key from [platform.openai.com](https://platform.openai.com) (or [console.anthropic.com](https://console.anthropic.com) for Claude) and paste it into `.env` before testing.
## Services
### LiteLLM Proxy (port 4000)
OpenAI-compatible API. Health check endpoint:
```bash
curl http://localhost:4000/health/liveliness
```
Admin UI at `http://localhost:4000/ui` — use your `LITELLM_MASTER_KEY` to log in.
### PostgreSQL (port 5432)
Bundled PostgreSQL 16 stores spend logs, API keys, and model configs. Data persists in the `litellm_postgres_data` named volume.
### Prometheus (port 9090)
Pre-configured to scrape LiteLLM metrics. Access the Prometheus UI at `http://localhost:9090`. Metrics include request latency, token usage, error rates, and cost per model.
## Environment Variables
| Variable | Default | Description |
|----------------------|--------------------------------------|--------------------------------------------------|
| `LITELLM_MASTER_KEY` | `sk-liteLLM-master-key-change-me` | Admin auth key — change this in production! |
| `LITELLM_SALT_KEY` | `sk-liteLLM-salt-key-change-me` | Salt for encrypting stored credentials |
| `DATABASE_URL` | *(blank — uses SQLite)* | PostgreSQL connection string for persisting data |
| `OPENAI_API_KEY` | *(blank)* | OpenAI API key |
| `ANTHROPIC_API_KEY` | *(blank)* | Anthropic API key (Claude) |
| `GEMINI_API_KEY` | *(blank)* | Google Gemini API key |
| `LITELLM_PORT` | `4000` | Host port for the proxy |
| Variable | Default | Description |
|----------------------|----------------------------------|---------------------------------------------|
| `LITELLM_MASTER_KEY` | `sk-liteLLM-master-key-change-me` | Admin auth key — change this in production! |
| `LITELLM_SALT_KEY` | `sk-liteLLM-salt-key-change-me` | Salt for encrypting stored credentials |
| `OPENAI_API_KEY` | *(blank)* | OpenAI API key |
| `ANTHROPIC_API_KEY` | *(blank)* | Anthropic API key (Claude) |
| `GEMINI_API_KEY` | *(blank)* | Google Gemini API key |
| `LITELLM_PORT` | `4000` | Proxy host port |
| `LITELLM_DB_PORT` | `5432` | PostgreSQL host port |
| `LITELLM_PROM_PORT` | `9090` | Prometheus host port |
## Provider API Keys
Set at least one key in `.env`. LiteLLM auto-detects providers — no config changes needed:
LiteLLM auto-detects providers from environment variables — no config changes needed:
| Variable | Provider | Sign up |
|---------------------|-------------------|-----------------------------------------|
| Variable | Provider | Sign up |
|---------------------|-------------------|--------------------------------------------|
| `OPENAI_API_KEY` | OpenAI | [platform.openai.com](https://platform.openai.com) |
| `ANTHROPIC_API_KEY` | Anthropic | [console.anthropic.com](https://console.anthropic.com) |
| `GEMINI_API_KEY` | Google | [aistudio.google.com](https://aistudio.google.com) |
| `COHERE_API_KEY` | Cohere | [cohere.com](https://cohere.com) |
| `TOGETHERAI_API_KEY` | Together AI | [together.ai](https://together.ai) |
| `COHERE_API_KEY` | Cohere | [cohere.com](https://cohere.com) |
| `TOGETHERAI_API_KEY` | Together AI | [together.ai](https://together.ai) |
## Managing the Proxy
Full provider list: [docs.litellm.ai/docs/providers](https://docs.litellm.ai/docs/providers)
## Managing the Stack
**View logs:**
```bash
docker compose logs -f litellm
docker compose logs -f litellm # Proxy logs
docker compose logs -f db # Database logs
docker compose logs -f prometheus # Prometheus logs
```
**Restart after changing `.env`:**
@@ -70,44 +99,44 @@ docker compose logs -f litellm
docker compose restart litellm
```
**Stop the proxy:**
**Stop everything:**
```bash
docker compose down
```
**Check health:**
**Re-build:**
```bash
curl http://localhost:4000/health/liveliness
docker compose up -d --build
```
## Prometheus Metrics
Prometheus scrapes LiteLLM every 15 seconds with 15-day retention. Example queries in the Prometheus UI:
```
# Request rate per model
rate(litellm_requests_total[5m])
# Average latency per model
rate(litellm_request_duration_seconds_sum[5m]) / rate(litellm_request_duration_seconds_count[5m])
# Spend per key
litellm_spend_usd_total
```
## OpenAI-Compatible Endpoints
Once running, use any OpenAI SDK or HTTP client:
```
POST /v1/chat/completions Chat completions
POST /v1/completions Text completions
POST /v1/embeddings Embeddings
POST /v1/embeddings Embeddings
GET /v1/models List available models
GET /health/liveliness Liveness check
GET /health/readiness Readiness check
GET /health/liveliness Liveness check
GET /health/readiness Readiness check
```
Full API reference: [docs.litellm.ai/docs/proxy/api](https://docs.litellm.ai/docs/proxy/api)
## Admin UI
Access the LiteLLM admin dashboard at `http://localhost:4000/ui` — use your `LITELLM_MASTER_KEY` to log in. View key management, spend logs, and model configs.
## Persistence
Without `DATABASE_URL`, LiteLLM stores data in an internal SQLite database. Set it to a PostgreSQL connection string to persist across container recreations:
```bash
echo "DATABASE_URL=postgresql://user:password@host:5432/litellm" >> .env
docker compose restart litellm
```
## Hardened / Offline Deployments
LiteLLM also ships a hardened compose profile for restricted environments (read-only rootfs, no outbound network). See the [upstream Docker guide](https://github.com/BerriAI/litellm/tree/litellm_internal_staging/docker) for details.
templates/litellm/docker-compose.yml
+49 -20
View File
@@ -1,5 +1,3 @@
version: '3.8'
services:
litellm:
image: ghcr.io/berriai/litellm:main-latest
@@ -10,26 +8,57 @@ services:
volumes:
- litellm_data:/app/data
restart: unless-stopped
env_file:
- .env
depends_on:
db:
condition: service_healthy
environment:
# ── Required ──────────────────────────────────────────────────────────
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-liteLLM-master-key-change-me}
- LITELLM_SALT_KEY=${LITELLM_SALT_KEY:-sk-liteLLM-salt-key-change-me}
# ── Database (optional — SQLite used if unset) ───────────────────────
- DATABASE_URL=${DATABASE_URL:-}
# ── Provider API Keys (set at least one) ────────────────────────────
- OPENAI_API_KEY=${OPENAI_API_KEY:-}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
- GEMINI_API_KEY=${GEMINI_API_KEY:-}
- AZURE_OPENAI_API_KEY=${AZURE_OPENAI_API_KEY:-}
- AZURE_OPENAI_API_BASE=${AZURE_OPENAI_API_BASE:-}
- COHERE_API_KEY=${COHERE_API_KEY:-}
- TOGETHERAI_API_KEY=${TOGETHERAI_API_KEY:-}
- OLLAMA_BASE_URL=${OLLAMA_BASE_URL:-}
# ── Proxy Settings ───────────────────────────────────────────────────
- LITELLM_DROP_PARAMS=${LITELLM_DROP_PARAMS:-false}
- LITELLM_MAX_PARALLEL_REQUESTS=${LITELLM_MAX_PARALLEL_REQUESTS:-}
- LITELLM_REQUEST_TIMEOUT=${LITELLM_REQUEST_TIMEOUT:-60}
- DATABASE_URL=postgresql://llmproxy:dbpassword9090@db:5432/litellm
- STORE_MODEL_IN_DB=True
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
db:
image: postgres:16
restart: always
container_name: litellm_db
environment:
POSTGRES_DB: litellm
POSTGRES_USER: llmproxy
POSTGRES_PASSWORD: dbpassword9090
ports:
- "${LITELLM_DB_PORT:-5432}:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
interval: 1s
timeout: 5s
retries: 10
prometheus:
image: prom/prometheus
container_name: litellm_prometheus
volumes:
- prometheus_data:/prometheus
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
ports:
- "${LITELLM_PROM_PORT:-9090}:9090"
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=15d"
restart: unless-stopped
volumes:
prometheus_data:
name: litellm_prometheus_data
postgres_data:
name: litellm_postgres_data
litellm_data:
name: litellm_data
templates/litellm/prometheus.yml
+7
View File
@@ -0,0 +1,7 @@
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'litellm'
static_configs:
- targets: ['litellm:4000']