ai-gateway

togo-framework/ai-gateway

An AI gateway for togo — rate limits, spend caps, caching, prompt management & provider fallback.

Install

A control plane in front of LLM providers (Cortex-style). Wrap the togo AI kit with per-key rate limits + spend caps, response caching, prompt-template management, and provider routing/fallback so you can expose AI safely to many consumers and bill it.

Quick start

Features

• Per-key rate limiting — sliding window, requests/min (429 over limit).
• Spend caps — monthly + hard caps enforced before the provider call (402 over quota), with a per-model price book and accumulated spend.
• Response caching — SHA-256 over (model | messages | temperature | max_tokens), TTL (default 1h); cache hits are returned instantly and billed at zero.
• Prompt management — named, versioned templates with {{var}} interpolation (RenderPrompt), activate any version.
• Provider routing + fallback — tries registered providers in order; falls back to the next on error.
• API keys — hashed (SHA-256), per-key allowed-models + limits, revocation; raw key shown once.

Provider interface

The gateway is decoupled from any SDK — implement a tiny adapter over your togo ai driver:

REST API

/complete reads the key from Authorization: Bearer <key>.

Method	Path	Description
POST	/api/ai-gateway/complete	gated completion
GET/POST/DELETE	/api/ai-gateway/keys[/{id}]	API-key management
GET/POST	/api/ai-gateway/prompts[/{name}]	prompt templates + versions
GET	/api/ai-gateway/usage?key=	usage/spend records

Configuration

No required env. Set per-model prices with g.SetPrice(model, aigateway.Price{Input, Output}); default cache TTL is 1h. Swap the in-memory stores via the exported seams for DB persistence.

---

Premium sponsors

Support togo — become a sponsor.