Skip to content

README.md

Latest commit

 

History

History
76 lines (56 loc) · 3.61 KB

README.md

File metadata and controls

76 lines (56 loc) · 3.61 KB
Raw

agentic-server

Standalone, OpenAI-compatible LLM gateway — a stateless multi-provider proxy with pluggable inference metering.

Overview

agentic-server is a small Express app that accepts OpenAI-compatible requests, routes them to a configured LLM provider (OpenAI, Anthropic, or Ollama), normalizes the response back to the OpenAI shape, and records usage through an injected sink. It is backend-agnostic: the gateway never imports a concrete telemetry/billing implementation — callers inject an InferenceSink, so the same gateway can meter to compute_log, a billing service, or nothing at all.

Usage

import { createAgenticServer } from 'agentic-server';

const app = createAgenticServer({
  providerType: 'ollama',
  providerBaseUrl: 'http://localhost:11434',
  defaultModel: 'llama3',
  // Optional: record usage. Fire-and-forget; implementations must never throw.
  inferenceSink: { logInference: (entry) => myBackend.record(entry) },
});

app.listen(3001, () => {
  console.log('agentic-server running on :3001');
});

Endpoints

Method Path Description
POST /v1/chat/completions OpenAI-compatible chat completion (multi-provider)
POST /v1/embeddings OpenAI-compatible embeddings (multi-provider)
POST /v1/usage Submit an external usage report to the metering sink
GET /v1/providers List configured providers
GET /healthz Health check

Identity & tenancy

Requests carry tenant identity via headers: X-Database-Id (required — requests without it are rejected 400), X-Entity-Id, and X-Actor-Id. Routing to a specific provider can be forced with X-LLM-Provider.

isPublic controls trust: when false (default) the server sits on a private network and trusts these headers directly; when true the identity headers are stripped from incoming requests (external clients cannot set tenant context), so such requests fail loud rather than being mis-attributed.

Metering

Metering is injected, not built in:

interface InferenceSink {
  logInference(entry: InferenceEntry): void; // fire-and-forget; must not throw
}

The gateway calls logInference after each chat/embedding (and for /v1/usage reports). Consumers own the concrete implementation — e.g. constructive-db injects a compute_log-backed sink built on its ModuleLoader.

Architecture

Client → POST /v1/chat/completions → agentic-server
                                          │
                        ┌─────────────────┼─────────────────┐
                        │                 │                 │
                 resolveProvider     LLM provider      InferenceSink
                 + transforms      (OpenAI/Ollama/…)   (injected metering)