Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# nebari-llm-serving-pack
# llm-serving-pack
#
# Tasks for the Astro/Starlight documentation site (lives in docs/).
# Run `make help` to list targets.
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# nebari-llm-serving-pack
# llm-serving-pack

A [Nebari](https://github.com/nebari-dev/nebari-infrastructure-core) software pack for serving LLMs. Deploys a Kubernetes operator that manages LLM model serving via [llm-d](https://llm-d.ai), with per-model access control, API key management, and Envoy AI Gateway integration for token counting and rate limiting.

Expand All @@ -19,8 +19,8 @@ Models can be loaded from HuggingFace (default) or mounted as OCI/modelcar image
- Kubernetes 1.28+ cluster with [Nebari Infrastructure Core](https://github.com/nebari-dev/nebari-infrastructure-core) deployed
- [nebari-operator](https://github.com/nebari-dev/nebari-operator) running
- NVIDIA GPU Operator installed (auto-discovers GPU nodes and manages the device plugin). **Note**: nebari-infrastructure-core does not install this automatically yet - tracked in [nebari-dev/nebari-infrastructure-core#232](https://github.com/nebari-dev/nebari-infrastructure-core/issues/232). Until that is done, install it manually as an ArgoCD app (see [examples/nvidia-gpu-operator.yaml](examples/nvidia-gpu-operator.yaml)).
- **Envoy Gateway installed and configured for AI Gateway integration** - `extensionApis.enableBackend`, `extensionManager` pointing at the AI Gateway controller service, and `backendResources` allowing `inference.networking.k8s.io/InferencePool`. This is a **hard requirement**; without it, the routing layer 404s at runtime. Ready-to-apply example in [`examples/envoy-gateway.yaml`](examples/envoy-gateway.yaml); see the [Installation guide](https://nebari-dev.github.io/nebari-llm-serving-pack/installation/#6-reconfigure-envoy-gateway-with-ai-gateway-extension-wiring) for details.
- Envoy AI Gateway installed (v0.5.0+). **Note**: the `envoyAIGateway.install` flag in this chart is not yet implemented - tracked in [#44](https://github.com/nebari-dev/nebari-llm-serving-pack/issues/44). Until that is done, install it manually as an ArgoCD app (see [examples/envoy-ai-gateway.yaml](examples/envoy-ai-gateway.yaml)).
- **Envoy Gateway installed and configured for AI Gateway integration** - `extensionApis.enableBackend`, `extensionManager` pointing at the AI Gateway controller service, and `backendResources` allowing `inference.networking.k8s.io/InferencePool`. This is a **hard requirement**; without it, the routing layer 404s at runtime. Ready-to-apply example in [`examples/envoy-gateway.yaml`](examples/envoy-gateway.yaml); see the [Installation guide](https://nebari-dev.github.io/llm-serving-pack/installation/#6-reconfigure-envoy-gateway-with-ai-gateway-extension-wiring) for details.
- Envoy AI Gateway installed (v0.5.0+). **Note**: the `envoyAIGateway.install` flag in this chart is not yet implemented - tracked in [#44](https://github.com/nebari-dev/llm-serving-pack/issues/44). Until that is done, install it manually as an ArgoCD app (see [examples/envoy-ai-gateway.yaml](examples/envoy-ai-gateway.yaml)).
- [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension) (GIE) installed (InferencePool / InferenceModel CRDs).
- A cert-manager `ClusterIssuer` the operator can use for the shared-hostname Certificate. Default expected name is `letsencrypt-production`; override with `platform.tls.clusterIssuer` in the chart values.
- DNS for `llm.<baseDomain>` and `llm-internal.<baseDomain>` resolving to the shared Gateway's load balancer (a wildcard CNAME on the base domain is the simplest way). Required for HTTP-01 issuance on the shared Certificate.
Expand Down Expand Up @@ -253,7 +253,7 @@ The pack expects the following to be available on the cluster:

## Development

See the [Local Development guide](https://nebari-dev.github.io/nebari-llm-serving-pack/local-development/) for a full walkthrough of the local dev environment.
See the [Local Development guide](https://nebari-dev.github.io/llm-serving-pack/local-development/) for a full walkthrough of the local dev environment.

```bash
# Create kind cluster with all dependencies
Expand Down Expand Up @@ -326,15 +326,15 @@ This pack is at alpha maturity (`v0.1.0-alpha.x`). The following limitations app

- **Single-namespace model:** All LLMModels must be in the operator's own namespace. Per-team isolation requires running separate pack installs (one namespace per team). This is a hard constraint imposed by Envoy Gateway's `apiKeyAuth`, which does not support cross-namespace Secret references.
- **API keys are not continuously tied to group membership.** Keys are issued based on the user's OIDC groups at creation time. If a user later loses group access, existing keys continue to work. The periodic audit that revokes such keys (default interval: 5 minutes) is **off by default** - it only runs when `keyManager.oidcUserinfoURL` is configured, which ships empty. Without it, keys are never revoked on group change. Even when enabled, this is eventual consistency, not real-time revocation.
- **JWKS path is Keycloak-specific.** The internal endpoint's JWT SecurityPolicy constructs the JWKS URI as `<issuerURL>/protocol/openid-connect/certs`. Non-Keycloak OIDC providers will not work out of the box. Tracked in [#61](https://github.com/nebari-dev/nebari-llm-serving-pack/issues/61).
- **JWKS path is Keycloak-specific.** The internal endpoint's JWT SecurityPolicy constructs the JWKS URI as `<issuerURL>/protocol/openid-connect/certs`. Non-Keycloak OIDC providers will not work out of the box. Tracked in [#61](https://github.com/nebari-dev/llm-serving-pack/issues/61).
- **NVIDIA GPUs only.** AMD and Intel accelerators are not supported in v0.1.
- **No scale-to-zero.** Idle model pods are not scaled down automatically.
- **No per-key rate limiting or token quotas.** Rate limiting is applied at the model level via Envoy AI Gateway, not per individual API key.
- **No API key expiration.** Keys do not expire on a schedule; revocation requires either manual deletion or - only when the audit is enabled via `keyManager.oidcUserinfoURL` - the audit detecting lost group access.
- **No team-level shared API keys.** Each key is tied to an individual user's identity.
- **OCI model loading uses init-container copy.** Kubernetes image volumes (alpha/beta) are not used; every pod start copies files from the OCI image to a shared emptyDir. On pods with `storage.type: emptyDir` and `preload: true`, every restart triggers a full re-download.
- **API key and metadata storage is limited to ~1 MiB per model** (Kubernetes Secret/ConfigMap size limit). This supports several thousand keys per model; it is a known scaling ceiling for v0.1.
- **`envoyAIGateway.install` flag is not yet implemented.** Envoy AI Gateway must be installed separately. Tracked in [#44](https://github.com/nebari-dev/nebari-llm-serving-pack/issues/44).
- **`envoyAIGateway.install` flag is not yet implemented.** Envoy AI Gateway must be installed separately. Tracked in [#44](https://github.com/nebari-dev/llm-serving-pack/issues/44).
- **Per-model subdomains are not yet wired into routing.** The `endpoints.external.subdomain` field is validated but unused. All models share `llm.<baseDomain>`; per-model dispatch is by the `model` field in the request body.

## License
Expand Down
4 changes: 2 additions & 2 deletions docs/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/package.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "nebari-llm-serving-pack-docs-astro",
"name": "llm-serving-pack-docs-astro",
"type": "module",
"version": "0.0.1",
"scripts": {
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/local-development.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Local Development
---
This guide walks through setting up a local development environment using [kind](https://kind.sigs.k8s.io/) to test the nebari-llm-serving-pack without a full Nebari deployment.
This guide walks through setting up a local development environment using [kind](https://kind.sigs.k8s.io/) to test the llm-serving-pack without a full Nebari deployment.

> **Scope of this dev path.** The local `kind` setup exercises the
> operator, key manager, CRD, webhook, and reconciler logic against a
Expand Down
2 changes: 1 addition & 1 deletion examples/envoy-ai-gateway.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
#
# This manual installation is a workaround until the chart's
# envoyAIGateway.install flag is implemented:
# https://github.com/nebari-dev/nebari-llm-serving-pack/issues/44
# https://github.com/nebari-dev/llm-serving-pack/issues/44
#
# References:
# - https://aigateway.envoyproxy.io/docs/getting-started/installation/
Expand Down
2 changes: 1 addition & 1 deletion examples/envoy-gateway.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Envoy Gateway ArgoCD Application with AI-Gateway-compatible values.
#
# The nebari-llm-serving-pack relies on Envoy AI Gateway's ext_proc
# The llm-serving-pack relies on Envoy AI Gateway's ext_proc
# filter to extract the `model` field from OpenAI-style request bodies
# and dispatch per-model. For that filter to be programmed into
# Envoy's HCM, Envoy Gateway MUST be installed with:
Expand Down
10 changes: 5 additions & 5 deletions key-manager/cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ import (
"k8s.io/client-go/rest"
"sigs.k8s.io/controller-runtime/pkg/client"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
llmv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1"

"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/api"
"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/audit"
"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/secrets"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/api"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/audit"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/secrets"
)

func main() {
Expand Down
6 changes: 3 additions & 3 deletions key-manager/go.mod
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
module github.com/nebari-dev/nebari-llm-serving-pack/key-manager
module github.com/nebari-dev/llm-serving-pack/key-manager

go 1.25.8

replace github.com/nebari-dev/nebari-llm-serving-pack/operator => ../operator
replace github.com/nebari-dev/llm-serving-pack/operator => ../operator

require (
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/nebari-dev/nebari-llm-serving-pack/operator v0.0.0
github.com/nebari-dev/llm-serving-pack/operator v0.0.0
golang.org/x/sync v0.18.0
k8s.io/api v0.35.0
k8s.io/apimachinery v0.35.0
Expand Down
4 changes: 2 additions & 2 deletions key-manager/internal/api/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import (
"log/slog"
"net/http"

"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/secrets"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/secrets"
)

// ModelLister provides model listing for the handler.
Expand Down
4 changes: 2 additions & 2 deletions key-manager/internal/api/handler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ import (
"testing"
"time"

"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/secrets"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/secrets"
)

// discardLogger returns a slog logger that discards all output, for tests.
Expand Down
4 changes: 2 additions & 2 deletions key-manager/internal/audit/auditor.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ import (
"log/slog"
"time"

"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/secrets"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/secrets"
)

// UserGroupsLookup returns the current OIDC groups for a username.
Expand Down
6 changes: 3 additions & 3 deletions key-manager/internal/audit/auditor_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ import (
"testing"
"time"

"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/audit"
"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/secrets"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/audit"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/models"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/secrets"
)

// mockModelLister is a test double for ModelLister.
Expand Down
2 changes: 1 addition & 1 deletion key-manager/internal/models/watcher.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (

"sigs.k8s.io/controller-runtime/pkg/client"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
llmv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1"
)

// ModelInfo is a simplified view of an LLMModel or PassthroughModel for
Expand Down
4 changes: 2 additions & 2 deletions key-manager/internal/models/watcher_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ import (
"k8s.io/apimachinery/pkg/runtime"
"sigs.k8s.io/controller-runtime/pkg/client/fake"

"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/models"
llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/models"
llmv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1"
)

func boolPtr(b bool) *bool { return &b }
Expand Down
2 changes: 1 addition & 1 deletion key-manager/internal/secrets/manager_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client/fake"

"github.com/nebari-dev/nebari-llm-serving-pack/key-manager/internal/secrets"
"github.com/nebari-dev/llm-serving-pack/key-manager/internal/secrets"
)

// sanitizedPrefix mirrors the production sanitizeUsernameForKey helper so
Expand Down
4 changes: 2 additions & 2 deletions operator/PROJECT
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ domain: nebari.dev
layout:
- go.kubebuilder.io/v4
projectName: operator
repo: github.com/nebari-dev/nebari-llm-serving-pack/operator
repo: github.com/nebari-dev/llm-serving-pack/operator
resources:
- api:
crdVersion: v1
Expand All @@ -16,7 +16,7 @@ resources:
domain: nebari.dev
group: llm
kind: LLMModel
path: github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1
path: github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1
version: v1alpha1
webhooks:
validation: true
Expand Down
10 changes: 5 additions & 5 deletions operator/cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,10 @@ import (
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
"sigs.k8s.io/controller-runtime/pkg/webhook"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/controller"
webhookv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/webhook/v1alpha1"
llmv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/llm-serving-pack/operator/internal/controller"
webhookv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/internal/webhook/v1alpha1"
// +kubebuilder:scaffold:imports
)

Expand Down Expand Up @@ -221,7 +221,7 @@ func main() {
// Deployment; the webhook uses it to enforce that LLMModels are
// created in the operator's namespace so their API-key Secrets land
// in the same namespace as the SecurityPolicies that reference them.
// See https://github.com/nebari-dev/nebari-llm-serving-pack/issues/59.
// See https://github.com/nebari-dev/llm-serving-pack/issues/59.
operatorNamespace := os.Getenv("POD_NAMESPACE")
if err := webhookv1alpha1.SetupLLMModelWebhookWithManager(mgr, operatorNamespace); err != nil {
setupLog.Error(err, "unable to create webhook", "webhook", "LLMModel")
Expand Down
2 changes: 1 addition & 1 deletion operator/go.mod
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module github.com/nebari-dev/nebari-llm-serving-pack/operator
module github.com/nebari-dev/llm-serving-pack/operator

go 1.24.5

Expand Down
4 changes: 2 additions & 2 deletions operator/internal/controller/clustertls_singleton.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client"
logf "sigs.k8s.io/controller-runtime/pkg/log"

"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/controller/reconcilers"
"github.com/nebari-dev/llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/llm-serving-pack/operator/internal/controller/reconcilers"
)

// defaultClusterTLSResyncInterval is the background resync cadence for the
Expand Down
4 changes: 2 additions & 2 deletions operator/internal/controller/clustertls_singleton_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"

"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/controller/reconcilers"
"github.com/nebari-dev/llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/llm-serving-pack/operator/internal/controller/reconcilers"
)

// newFakeClientWithGateway builds a fake client that knows about the
Expand Down
6 changes: 3 additions & 3 deletions operator/internal/controller/llmmodel_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ import (
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
logf "sigs.k8s.io/controller-runtime/pkg/log"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/controller/reconcilers"
llmv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/llm-serving-pack/operator/internal/controller/reconcilers"
)

const (
Expand Down
4 changes: 2 additions & 2 deletions operator/internal/controller/llmmodel_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/reconcile"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/config"
llmv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/llm-serving-pack/operator/internal/config"
)

// testConfig returns a minimal OperatorConfig suitable for envtest.
Expand Down
6 changes: 3 additions & 3 deletions operator/internal/controller/passthroughmodel_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ import (
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
logf "sigs.k8s.io/controller-runtime/pkg/log"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/nebari-llm-serving-pack/operator/internal/controller/reconcilers"
llmv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1"
"github.com/nebari-dev/llm-serving-pack/operator/internal/config"
"github.com/nebari-dev/llm-serving-pack/operator/internal/controller/reconcilers"
)

// Condition types reported on PassthroughModel status.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import (
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/reconcile"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
llmv1alpha1 "github.com/nebari-dev/llm-serving-pack/operator/api/v1alpha1"
)

// newPassthroughReconciler returns a reconciler wired to the envtest client.
Expand Down
Loading
Loading