Skip to content

Commit f9cd0f7

Browse files
kristin-kronstain-brownkristin-kronstain-brown
authored
Merge branch 'main' into kkb-fault-injection
2 parents 464e3e2 + d2e11df commit f9cd0f7

11 files changed

Lines changed: 1044 additions & 16 deletions

File tree

‎assets/agw-docs/pages/agentgateway/llm/budget-limits.md‎

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,43 @@ spec:
135135
unit: Hours
136136
```
137137

138+
{{< doc-test paths="budget-limits" >}}
139+
kubectl apply -f- <<EOF
140+
apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}}
141+
kind: {{< reuse "agw-docs/snippets/policy.md" >}}
142+
metadata:
143+
name: local-token-budget
144+
namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
145+
spec:
146+
targetRefs:
147+
- group: gateway.networking.k8s.io
148+
kind: Gateway
149+
name: agentgateway-proxy
150+
traffic:
151+
rateLimit:
152+
local:
153+
- tokens: 10000
154+
unit: Hours
155+
EOF
156+
157+
YAMLTest -f - <<'EOF'
158+
- name: local-token-budget policy is accepted
159+
wait:
160+
target:
161+
kind: {{< reuse "agw-docs/snippets/policy.md" >}}
162+
metadata:
163+
namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
164+
name: local-token-budget
165+
jsonPath: "$.status.ancestors[0].conditions[?(@.type=='Accepted')].status"
166+
jsonPathExpectation:
167+
comparator: equals
168+
value: "True"
169+
polling:
170+
timeoutSeconds: 60
171+
intervalSeconds: 2
172+
EOF
173+
{{< /doc-test >}}
174+
138175
## Monitor budget usage
139176

140177
Track how much of each user's budget has been consumed using Prometheus metrics.
@@ -195,6 +232,8 @@ cost = (50,000 / 1,000,000 × $30) + (50,000 / 1,000,000 × $60)
195232

196233
For more information on cost calculation, see the [cost tracking guide]({{< link-hextra path="/llm/cost-controls/cost-tracking/" >}}).
197234

235+
<!--TODO dollar budget
236+
198237
## Enforce a dollar budget
199238

200239
The budgets in the previous sections are measured in *tokens*. If you configure a [model cost catalog]({{< link-hextra path="/llm/cost-controls/costs/" >}}), {{< reuse "agw-docs/snippets/agentgateway.md" >}} computes the realized USD cost of each request and exposes it to CEL as `llm.cost`. You can then rate limit on *dollars* directly, which enforces a true spend cap regardless of which model or input/output token mix each user hits.
@@ -230,7 +269,7 @@ The budgets in the previous sections are measured in *tokens*. If you configure
230269
backendRef:
231270
kind: Service
232271
name: ratelimit
233-
namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
272+
namespace: ratelimit
234273
port: 8081
235274
descriptors:
236275
- entries:
@@ -258,6 +297,8 @@ The budgets in the previous sections are measured in *tokens*. If you configure
258297
To budget on a single cost component instead of the total, use that field in the expression. For example, `cost: 'uint(llm.cost.output * 1000000)'` budgets only on output-token spend.
259298
{{< /callout >}}
260299
300+
-->
301+
261302
## Cleanup
262303
263304
{{< reuse "agw-docs/snippets/cleanup.md" >}}

‎assets/agw-docs/pages/agentgateway/llm/virtual-keys.md‎

Lines changed: 127 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ flowchart LR
3030

3131
When a request arrives:
3232

33-
1. {{< reuse "agw-docs/snippets/agentgateway-capital.md" >}} validates the API key, extracts the user ID from a request header, and checks the request against the user's token budget.
33+
1. {{< reuse "agw-docs/snippets/agentgateway-capital.md" >}} validates the API key, extracts the user ID from the key's metadata, and checks the request against the user's token budget.
3434
2. If budget is available:
3535
1. The request proceeds to the LLM.
3636
2. Agentgateway tracks token usage and deducts tokens from the user's budget.
@@ -53,6 +53,60 @@ This example creates two virtual keys (for Alice and Bob) with independent daily
5353

5454
### Create API keys for users
5555

56+
{{< version exclude-if="1.3.x,1.2.x,1.1.x,1.0.x,2.2.x" >}}
57+
Store your virtual keys in a Kubernetes ConfigMap, with one entry per user. Because a ConfigMap is not confidential, each entry stores a SHA-256 *hash* of the API key (`keyHash`) rather than the raw key. Clients still send the raw key in the `Authorization: Bearer` header. {{< reuse "agw-docs/snippets/agentgateway-capital.md" >}} hashes the presented key and compares it to the stored hash, so the plaintext key never has to exist in the cluster.
58+
59+
> [!IMPORTANT]
60+
> ConfigMaps with hashed keys (as opposed to Secrets) are the recommended way to store virtual keys. If you need to use Kubernetes Secrets, refer to the [API key authentication guide]({{< link-hextra path="/security/apikey/" >}}) for an example.
61+
62+
1. Generate a `sha256:<hex>` hash for each user's API key. The hash is computed over the exact key bytes, so do not include a trailing newline.
63+
64+
```sh
65+
printf '%s' 'sk-alice-abc123def456' | sha256sum | awk '{print "sha256:"$1}' # alice
66+
printf '%s' 'sk-bob-xyz789uvw012' | sha256sum | awk '{print "sha256:"$1}' # bob
67+
```
68+
69+
2. Create a ConfigMap that stores the hash and metadata for each user. Add a label so that the authentication policy can select the ConfigMap in the next step.
70+
71+
```yaml,paths="virtual-keys"
72+
kubectl apply -f- <<EOF
73+
apiVersion: v1
74+
kind: ConfigMap
75+
metadata:
76+
name: llm-api-keys
77+
namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
78+
labels:
79+
agentgateway.dev/apikey: "true"
80+
data:
81+
alice: |
82+
{
83+
"keyHash": "sha256:477fe4bee6e6d46cc4fa5827b4c375f045b01701de265deea7c26a74d4f86d1f",
84+
"metadata": {
85+
"user_id": "alice"
86+
}
87+
}
88+
bob: |
89+
{
90+
"keyHash": "sha256:b2bd46e01122f51b540cc49cfdd020484776e67f5d3966c099e689eac4b8d93e",
91+
"metadata": {
92+
"user_id": "bob"
93+
}
94+
}
95+
EOF
96+
```
97+
98+
{{% reuse "agw-docs/snippets/review-table.md" %}}
99+
100+
| Setting | Description |
101+
|-------------|-------------|
102+
| `metadata.labels` | A label that the authentication policy uses to select this ConfigMap. Because ConfigMap sources are selected by label, this label is required. |
103+
| `data.<name>` | Each entry in `data` represents a user. The value is a JSON object with a `keyHash` and optional `metadata`. |
104+
| `keyHash` | A SHA-256 hash of the user's API key, in `sha256:<hex>` format. The raw key is never stored. |
105+
| `metadata.user_id` | The user identifier that rate limiting and metrics policies read (as `apiKey.user_id`) to attribute usage per user. |
106+
107+
{{< /version >}}
108+
109+
{{< version include-if="1.3.x,1.2.x,1.1.x,1.0.x,2.2.x" >}}
56110
Create an API key secret that stores keys and metadata for each user.
57111

58112
```yaml,paths="virtual-keys"
@@ -88,9 +142,44 @@ EOF
88142
| `stringData.<name>` | Each key in `stringData` represents a user. The value is a JSON object containing the API key and metadata. |
89143
| `key` | The API key value that users include in their `Authorization: Bearer` header. |
90144
| `metadata.user_id` | The user identifier extracted by rate limiting policies to enforce per-user budgets. |
145+
{{< /version >}}
91146

92147
### Configure API key authentication
93148

149+
{{< version exclude-if="1.3.x,1.2.x,1.1.x,1.0.x,2.2.x" >}}
150+
Create an {{< reuse "agw-docs/snippets/policy.md" >}} that requires API key authentication for all requests to the gateway. Unlike Secrets, which you can reference directly by name, ConfigMaps can be selected only by label. Use `configMapSelector` to match every ConfigMap that carries the label you added in the previous step, whether that is a single ConfigMap or several.
151+
152+
```yaml,paths="virtual-keys"
153+
kubectl apply -f- <<EOF
154+
apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}}
155+
kind: {{< reuse "agw-docs/snippets/policy.md" >}}
156+
metadata:
157+
name: api-key-auth
158+
namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
159+
spec:
160+
targetRefs:
161+
- group: gateway.networking.k8s.io
162+
kind: Gateway
163+
name: agentgateway-proxy
164+
traffic:
165+
apiKeyAuthentication:
166+
mode: Strict
167+
configMapSelector:
168+
matchLabels:
169+
agentgateway.dev/apikey: "true"
170+
EOF
171+
```
172+
173+
{{% reuse "agw-docs/snippets/review-table.md" %}}
174+
175+
| Setting | Description |
176+
|-------------|-------------|
177+
| `targetRefs` | Apply the policy to the entire Gateway so all routes require API keys. |
178+
| `apiKeyAuthentication.mode` | Set to `Strict` to require a valid API key for all requests. |
179+
| `configMapSelector.matchLabels` | Selects all ConfigMaps that carry the given labels, combining their keys. Every entry in a selected ConfigMap must use `keyHash`; a raw `key` value is rejected because ConfigMaps are not confidential. |
180+
{{< /version >}}
181+
182+
{{< version include-if="1.3.x,1.2.x,1.1.x,1.0.x,2.2.x" >}}
94183
Create an {{< reuse "agw-docs/snippets/policy.md" >}} that requires API key authentication for all requests to the gateway. You can source the API keys from a single Secret with `secretRef`, or from multiple Secrets selected by label with `secretSelector`. Use `secretSelector` when you want to spread keys across many Secrets, such as one Secret per team or tenant, instead of maintaining a single Secret.
95184

96185
{{< tabs >}}
@@ -161,6 +250,7 @@ EOF
161250
| `apiKeyAuthentication.mode` | Set to `Strict` to require a valid API key for all requests. |
162251
| `secretRef.name` | References a single Secret containing API keys and user metadata. Use this or `secretSelector`, not both. |
163252
| `secretSelector.matchLabels` | Selects all Secrets that carry the given labels, combining their keys. Use instead of `secretRef` when keys are spread across multiple Secrets. Secret-only. |
253+
{{< /version >}}
164254

165255
### Configure per-key token budgets
166256

@@ -687,8 +777,40 @@ For more information on cost tracking, see the [cost tracking guide]({{< link-he
687777

688778
Provide different budget tiers for free, standard, and premium users.
689779

690-
1. Add tier metadata to each API key in the Secret.
780+
1. Add tier metadata to each API key.
781+
782+
{{< version exclude-if="1.3.x,1.2.x,1.1.x,1.0.x,2.2.x" >}}
783+
Store the keys in a ConfigMap, using the `keyHash` of each key. Generate each hash with `printf '%s' '<key>' | sha256sum | awk '{print "sha256:"$1}'`.
784+
785+
```yaml
786+
apiVersion: v1
787+
kind: ConfigMap
788+
metadata:
789+
name: llm-api-keys
790+
namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
791+
labels:
792+
agentgateway.dev/apikey: "true"
793+
data:
794+
alice: |
795+
{
796+
"keyHash": "sha256:477fe4bee6e6d46cc4fa5827b4c375f045b01701de265deea7c26a74d4f86d1f",
797+
"metadata": {
798+
"user_id": "alice",
799+
"tier": "premium"
800+
}
801+
}
802+
charlie: |
803+
{
804+
"keyHash": "sha256:1dfa4e699a31bc3bdcb34c74b43f7a4e58d38c51cec243b4add27b16de599355",
805+
"metadata": {
806+
"user_id": "charlie",
807+
"tier": "free"
808+
}
809+
}
810+
```
811+
{{< /version >}}
691812
813+
{{< version include-if="1.3.x,1.2.x,1.1.x,1.0.x,2.2.x" >}}
692814
```yaml
693815
apiVersion: v1
694816
kind: Secret
@@ -714,6 +836,7 @@ Provide different budget tiers for free, standard, and premium users.
714836
}
715837
}
716838
```
839+
{{< /version >}}
717840
718841
2. Configure rate limiting to use the tier and user_id from API key metadata.
719842
@@ -813,7 +936,8 @@ For more advanced rate limiting patterns, see the [budget and spend limits guide
813936
814937
```sh
815938
kubectl delete {{< reuse "agw-docs/snippets/policy.md" >}} api-key-auth per-user-metrics -n {{< reuse "agw-docs/snippets/namespace.md" >}} --ignore-not-found
816-
kubectl delete secret llm-api-keys -n {{< reuse "agw-docs/snippets/namespace.md" >}}
939+
kubectl delete configmap llm-api-keys -n {{< reuse "agw-docs/snippets/namespace.md" >}} --ignore-not-found
940+
kubectl delete secret llm-api-keys -n {{< reuse "agw-docs/snippets/namespace.md" >}} --ignore-not-found
817941
kubectl delete httproute openai -n {{< reuse "agw-docs/snippets/namespace.md" >}}
818942
kubectl delete {{< reuse "agw-docs/snippets/backend.md" >}} openai -n {{< reuse "agw-docs/snippets/namespace.md" >}}
819943
```

0 commit comments

Comments
 (0)