LLMProxyAS

OpenAI-compatible LLM reverse proxy written in Go.

Repository: https://github.com/dev2k6/LLMProxyAS

Point any OpenAI SDK or HTTP client at LLMProxyAS. The proxy validates models against an allow-list, injects upstream credentials when needed, and forwards traffic to any OpenAI-compatible provider — OpenAI, Azure OpenAI, Groq, Together, DeepSeek, vLLM, Ollama, and more.


Default port	`20182`
Client base URL	`http://<host>:20182/v1`
Health	`GET /health`
Entrypoint	`cmd/server`
Config	`configs/config.json`, `configs/allowed_models.json`
License	MIT
GitHub	dev2k6/LLMProxyAS

Features

Area	Details
OpenAI-compatible API	Chat completions, embeddings, legacy completions, models list; passthrough for images, audio, moderations
Streaming	Full SSE (`stream=true`) with flush, idle timeout, and max duration
Tools & vision	Bodies forwarded as-is (function calling, `image_url`, multimodal messages)
Model allow-list	Only IDs in `allowed_models.json` are accepted; `GET /v1/models` reflects that list
Credential injection	Upstream `api_key` from config so apps can call the proxy without a provider key
Provider switch	Change `base_url` + `api_key` only — no rebuild for config-only deploys with volume mounts
Resilience	Panic recovery, body caps, dial/TLS/header timeouts, stream guards, graceful shutdown
Ops	JSON structured logs, Docker multi-stage image, Compose healthcheck, non-root container user

How it works

┌──────────────┐   OpenAI HTTP API    ┌──────────────┐    HTTPS     ┌─────────────────┐
│  Your app    │ ───────────────────► │  LLMProxyAS  │ ───────────► │ Upstream LLM    │
│  SDK / curl  │   :20182/v1/...      │  (this repo) │              │ OpenAI / Groq…  │
└──────────────┘                      └──────────────┘              └─────────────────┘
                                            │
                    ┌───────────────────────┼───────────────────────┐
                    │                       │                       │
              model allow-list      API key injection         timeouts / SSE
              (local filter)        (config or client)        logging / recovery

Client uses base_url = http://host:20182/v1 (any official OpenAI SDK works).
Proxy checks model against allowed_models.json (when present in the body).
Proxy forwards to base_url from config.json, attaching authentication.
Response — including SSE chunks — is returned to the client.

Requirements

Mode	Requirement
Local binary	Go 1.22+, network access to upstream
Docker	Docker Engine + Docker Compose v2
Upstream	OpenAI-compatible HTTP API; API key if the provider requires one

Quick start

1. Clone and configure

git clone https://github.com/dev2k6/LLMProxyAS.git
cd LLMProxyAS

cp configs/config.example.json configs/config.json

Edit configs/config.json:

{
  "base_url": "https://api.openai.com/v1",
  "api_key": "sk-your-real-upstream-key",
  "port": 20182,
  "log_level": "info"
}

Edit configs/allowed_models.json to the models you allow.

Never commit real API keys. Keep secrets in local files, env-mounted secrets, or a secret manager.

2. Run

Local

go mod tidy
go run ./cmd/server
# or: make run

Docker Compose

docker compose up -d --build
# or: make docker-up

3. Verify

curl -s http://localhost:20182/health
curl -s http://localhost:20182/v1/models

Expected health payload:

{
  "status": "ok",
  "service": "LLMProxyAS"
}

4. First chat call

curl -s http://localhost:20182/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello from LLMProxyAS"}]
  }'

If api_key is set in config, no Authorization header is required from the client.

Authentication

Hop	Behavior
Client → LLMProxyAS	No proxy API key required. Any client that can reach the port may call the API.
LLMProxyAS → upstream	Forwards client `Authorization` if present; otherwise sends `Bearer <api_key>` from config.

Recommended pattern

Store the real provider key only in configs/config.json (or a mounted secret).
Point internal apps at http://llmproxyas:20182/v1.
Clients use a dummy SDK key (e.g. not-needed) when the SDK requires the field.

Production warning: LLMProxyAS does not authenticate callers. Expose it only on trusted networks, or put an authenticating gateway (API key, mTLS, OAuth2, IP allow-list) in front. See SECURITY.md.

Configuration

CLI flags

Flag	Default	Description
`-config`	`configs/config.json`	Upstream and server settings
`-models`	`configs/allowed_models.json`	Model allow-list

Path resolution (first existing file wins):

Path passed on the CLI
configs/<basename>
<basename> in the working directory
Same candidates relative to the executable directory

`configs/config.json`

Template: configs/config.example.json

Field	Default	Description
`base_url`	(required)	Upstream base URL (typically ends with `/v1`)
`api_key`	`""`	Upstream key when client omits `Authorization`
`port`	`20182`	Listen port
`timeout_seconds`	`120`	Timeout for non-streaming upstream requests
`log_level`	`info`	`debug` \| `info` \| `warn` \| `error`
`log_body`	`false`	Reserved for future body logging
`max_body_bytes`	`33554432`	Max request body (32 MiB; vision-friendly)
`response_header_timeout_seconds`	`60`	Max wait for upstream response headers
`stream_idle_timeout_seconds`	`120`	Abort stream if upstream is silent this long
`stream_max_seconds`	`600`	Hard cap per stream (10 minutes)
`read_header_timeout_seconds`	`10`	HTTP server read-header timeout
`idle_timeout_seconds`	`120`	Keep-alive idle timeout

Example full config:

{
  "base_url": "https://api.openai.com/v1",
  "api_key": "sk-xxxxxxxxxxxxxxxx",
  "timeout_seconds": 120,
  "port": 20182,
  "log_level": "info",
  "log_body": false,
  "max_body_bytes": 33554432,
  "response_header_timeout_seconds": 60,
  "stream_idle_timeout_seconds": 120,
  "stream_max_seconds": 600,
  "read_header_timeout_seconds": 10,
  "idle_timeout_seconds": 120
}

`configs/allowed_models.json`

{
  "models": [
    "gpt-4o",
    "gpt-4o-mini",
    "gpt-4-turbo",
    "claude-3-5-sonnet-20241022",
    "deepseek-chat",
    "qwen2.5-72b-instruct"
  ]
}

Rule	Result
`model` not in list	HTTP `400`, `error.code = model_not_allowed`
`GET /v1/models`	Returns only allow-listed IDs (`owned_by`: `LLMProxyAS`)
`GET /v1/models/{id}` unknown	HTTP `404`, `model_not_found`

Provider examples

OpenAI

{
  "base_url": "https://api.openai.com/v1",
  "api_key": "sk-..."
}

Groq

{
  "base_url": "https://api.groq.com/openai/v1",
  "api_key": "gsk_..."
}

DeepSeek

{
  "base_url": "https://api.deepseek.com/v1",
  "api_key": "sk-..."
}

Local OpenAI-compatible (vLLM / Ollama / LiteLLM)

{
  "base_url": "http://127.0.0.1:8000/v1",
  "api_key": "ollama"
}

Update allowed_models.json to match the upstream model IDs. Restart the process after config changes (Compose volume mounts need container restart, not rebuild).

Docker Compose environment

Optional .env.example → .env:

Variable	Default	Description
`HOST_PORT`	`20182`	Host port mapped to container `20182`
`IMAGE_TAG`	`latest`	Image tag
`TZ`	`UTC`	Container timezone

HTTP API

Method	Path	Auth filter	Description
`GET`	`/health`	—	Liveness
`GET`	`/v1/models`	—	List allow-listed models (local)
`GET`	`/v1/models/{id}`	—	Get one allow-listed model
`POST`	`/v1/chat/completions`	model filter	Chat, tools, vision, stream
`POST`	`/v1/completions`	model filter	Legacy completions
`POST`	`/v1/embeddings`	model filter	Embeddings
`POST`	`/v1/images/generations`	model filter*	Images
`POST`	`/v1/images/edits`	model filter*	Image edits
`POST`	`/v1/images/variations`	model filter*	Image variations
`POST`	`/v1/audio/speech`	model filter*	TTS
`POST`	`/v1/audio/transcriptions`	model filter*	STT
`POST`	`/v1/audio/translations`	model filter*	Audio translation
`POST`	`/v1/moderations`	model filter*	Moderations

* Model filter applies when the JSON body includes a non-empty model field.

Unsupported or future /v1/* POST routes may still be forwarded via the catch-all handler with the same model filter rules.

Client examples

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:20182/v1",
    api_key="not-needed",
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Streaming

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Count to five"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Tools / function calling

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather by city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Weather in Hanoi?"}],
    tools=tools,
)
print(resp.choices[0].message)

Vision (image URL)

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is in this image?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}},
        ],
    }],
)

Node.js

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:20182/v1",
  apiKey: "not-needed",
});

const resp = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Hello" }],
});
console.log(resp.choices[0].message.content);

cURL

# Models
curl -s http://localhost:20182/v1/models | jq .

# Chat (key optional if configured server-side)
curl -s http://localhost:20182/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hi"}]
  }' | jq .

# Stream
curl -N http://localhost:20182/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "stream": true,
    "messages": [{"role": "user", "content": "Say hello slowly"}]
  }'

Rejected model (example)

curl -s http://localhost:20182/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"not-allowed","messages":[{"role":"user","content":"x"}]}'

{
  "error": {
    "message": "model 'not-allowed' is not allowed by this proxy; see GET /v1/models for permitted models",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_allowed"
  }
}

Docker

Compose (recommended)

# Configure configs/config.json and allowed_models.json first
docker compose up -d --build

docker compose ps
docker compose logs -f
curl -s http://localhost:20182/health

docker compose down

Item	Value
URL	`http://localhost:20182`
Health	`http://localhost:20182/health`
Config volume	`./configs` → `/app/configs` (read-only)
Image	`llmproxyas:latest`
User	non-root (`appuser`, uid `10001`)

Custom host port:

# bash
HOST_PORT=8080 docker compose up -d --build

# PowerShell
$env:HOST_PORT = "8080"; docker compose up -d --build

Build & run image

docker build -t llmproxyas:latest .

# Linux / macOS
docker run --rm -p 20182:20182 \
  -v "$(pwd)/configs:/app/configs:ro" \
  llmproxyas:latest

# Windows PowerShell
docker run --rm -p 20182:20182 `
  -v "${PWD}/configs:/app/configs:ro" `
  llmproxyas:latest

Docker files

File	Purpose
`Dockerfile`	Multi-stage Go build → Alpine runtime, healthcheck
`docker-compose.yml`	Ports, volume, restart, log rotation, healthcheck
`.dockerignore`	Lean, secret-safe build context
`.env.example`	Optional Compose variables

Prebuilt binaries

Release binaries are published in the bin/ directory of this repository for convenience:

File	Platform
`bin/LLMProxyAS.exe`	Windows x64 (shortcut)
`bin/LLMProxyAS-windows-amd64.exe`	Windows x64
`bin/LLMProxyAS-windows-arm64.exe`	Windows ARM64
`bin/LLMProxyAS-linux-amd64`	Linux x64
`bin/LLMProxyAS-linux-arm64`	Linux ARM64
`bin/LLMProxyAS-darwin-amd64`	macOS Intel
`bin/LLMProxyAS-darwin-arm64`	macOS Apple Silicon

Also included: bin/configs/ (config.json, allowed_models.json) and bin/README.txt.

Run (Windows example) — execute from bin/ so local configs resolve:

cd bin
.\LLMProxyAS.exe

Edit bin/configs/config.json (api_key, base_url) before production use.

Rebuild all platforms:

powershell -ExecutionPolicy Bypass -File .\scripts\build-all.ps1

Direct download (raw GitHub):

https://github.com/dev2k6/LLMProxyAS/raw/main/bin/LLMProxyAS-windows-amd64.exe
https://github.com/dev2k6/LLMProxyAS/raw/main/bin/LLMProxyAS-linux-amd64
https://github.com/dev2k6/LLMProxyAS/raw/main/bin/LLMProxyAS-darwin-arm64

(Adjust branch name if not main.)

Build from source

go mod tidy

# Linux / macOS
go build -o bin/LLMProxyAS ./cmd/server
./bin/LLMProxyAS -config configs/config.json -models configs/allowed_models.json

# Windows
go build -o bin/LLMProxyAS.exe ./cmd/server
.\bin\LLMProxyAS.exe -config configs\config.json -models configs\allowed_models.json

Cross-compile example (Linux binary from any host with Go):

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 \
  go build -trimpath -ldflags="-s -w" -o bin/LLMProxyAS ./cmd/server

Or build every target into bin/:

powershell -ExecutionPolicy Bypass -File .\scripts\build-all.ps1

Makefile targets

Target	Description
`make build`	Build binary into `bin/`
`make run`	`go run ./cmd/server`
`make tidy`	`go mod tidy`
`make test`	`go test ./...`
`make clean`	Remove build artifacts
`make docker-build`	`docker compose build`
`make docker-up`	Build and start detached
`make docker-down`	Stop stack
`make docker-logs`	Follow logs
`make help`	List targets

Project layout

.
├── cmd/server/                 # Process entrypoint (flags, exit codes)
├── internal/
│   ├── app/                    # DI, lifecycle, config path resolution
│   ├── config/                 # config.json + allowed_models loaders
│   ├── handler/                # Proxy, models, health handlers
│   ├── httpserver/             # Gin router + http.Server wrapper
│   ├── logging/                # slog JSON logger
│   ├── middleware/             # Recovery, request log, model filter
│   └── oaierr/                 # OpenAI error JSON helpers
├── configs/
│   ├── config.example.json     # Safe template for distribution
│   ├── config.json             # Local runtime config (keep secrets out of git)
│   └── allowed_models.json     # Model allow-list
├── Dockerfile
├── docker-compose.yml
├── Makefile
├── LICENSE
├── CHANGELOG.md
├── CONTRIBUTING.md
├── SECURITY.md
└── README.md

Package	Responsibility
`cmd/server`	Thin main, CLI flags, fatal recovery
`internal/app`	Wire dependencies; run / signal shutdown
`internal/httpserver`	Routes, listen, graceful stop
`internal/handler`	Reverse proxy and API handlers
`internal/middleware`	Cross-cutting HTTP middleware
`internal/config`	Load and validate JSON configuration
`internal/oaierr`	Safe, consistent client errors
`internal/logging`	Log level and JSON handler setup

Resilience

Concern	Behavior
Panic in handler	Recovered; OpenAI-style `500` if body not started
Oversized body	Rejected (`413` / `body_too_large`)
Upstream dial / TLS	Transport-level timeouts
Upstream header stall	`ResponseHeaderTimeout` → gateway error / timeout
Non-stream hang	`http.Client.Timeout` = `timeout_seconds`
Stream silence	`stream_idle_timeout_seconds` + best-effort SSE error event
Stream too long	`stream_max_seconds` context cancel
Client disconnect	Upstream request cancelled via request context
Process stop	`SIGINT` / `SIGTERM` → graceful `Shutdown`, then force `Close`
Double-write guard	JSON errors skipped if response already started (e.g. mid-SSE)

Logging

Library: log/slog with JSON handler to stdout
Levels: debug, info, warn, error via log_level
Request logs include method, path, status, latency, client IP, model (when set)

Example line:

{
  "time": "2026-07-19T12:00:00Z",
  "level": "INFO",
  "msg": "request",
  "method": "POST",
  "path": "/v1/chat/completions",
  "status": 200,
  "latency_ms": 342,
  "model": "gpt-4o-mini",
  "client_ip": "127.0.0.1"
}

Error format

All proxy-generated errors follow the OpenAI envelope:

{
  "error": {
    "message": "human-readable message",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_allowed"
  }
}

Common code values:

Code	Meaning
`model_not_allowed`	Model not in allow-list
`model_not_found`	Model id not listed / unknown
`body_too_large`	Request body over `max_body_bytes`
`upstream_error`	Upstream request failed
`upstream_timeout`	Upstream timed out
`upstream_unreachable`	Dial / DNS failure
`stream_interrupted`	Stream ended with error after headers were sent
`internal_error`	Unexpected panic / internal failure

Upstream HTTP error bodies are forwarded when the upstream responds with a normal HTTP response.

Security

Summary:

Do not commit secrets — use config.example.json; keep real keys local or in secret stores.
No client authentication — protect the port with network policy or a gateway.
Allow-list models — reduce cost and abuse surface.
Prefer private networks for production deployments.
Container defaults — non-root user, read-only config mount in Compose.

Full policy: SECURITY.md.

Troubleshooting

Symptom	Things to check
`failed to load config`	Path to `configs/config.json`; run from repo root or pass `-config`
`connection refused` on client	Process/container not listening; wrong host/port
Upstream `401` / `403`	Invalid `api_key` or client-forwarded token
`model_not_allowed`	Add model id to `allowed_models.json` and restart
Stream stops / idle timeout	Increase `stream_idle_timeout_seconds` or check upstream
Long jobs cut off	Increase `stream_max_seconds` / `timeout_seconds`
Docker health unhealthy	Logs via `docker compose logs`; ensure port `20182` inside container
Body too large	Raise `max_body_bytes` for large base64 vision payloads

Debug logging:

"log_level": "debug"

Roadmap

Ideas under consideration (not commitments):

Client authentication middleware (static keys / bearer)
Model alias map (client id → upstream id)
Rate limiting and basic quotas
Multi-upstream failover
Prometheus metrics endpoint
Optional request/response body logging (log_body)
Hot-reload of allow-list / config

Contributions welcome — see CONTRIBUTING.md.

Contributing

See CONTRIBUTING.md for setup, coding guidelines, and PR expectations.

Changelog

See CHANGELOG.md for release history.

License

This project is licensed under the MIT License — see LICENSE for the full text.

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
bin		bin
cmd/server		cmd/server
configs		configs
internal		internal
scripts		scripts
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.yml		docker-compose.yml
go.mod		go.mod
go.sum		go.sum

Folders and files

Latest commit

History

Repository files navigation

LLMProxyAS

Table of contents

Features

How it works

Requirements

Quick start

1. Clone and configure

2. Run

3. Verify

4. First chat call

Authentication

Configuration

CLI flags

configs/config.json

configs/allowed_models.json

Provider examples

Docker Compose environment

HTTP API

Client examples

Python (OpenAI SDK)

Streaming

Tools / function calling

Vision (image URL)

Node.js

cURL

Rejected model (example)

Docker

Compose (recommended)

Build & run image

Docker files

Prebuilt binaries

Build from source

Makefile targets

Project layout

Resilience

Logging

Error format

Security

Troubleshooting

Roadmap

Contributing

Changelog

License

About

Topics

Resources

License

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 1

Contributors

Uh oh!

Languages

`configs/config.json`

`configs/allowed_models.json`