Merge pull request #776 from agentgateway/kkb-fault-injection

Added fault injection topics

Author: kristin-kronstain-brown

assets/agw-docs/pages/resiliency/fault-injection.md

@@ -0,0 +1,205 @@
+Inject artificial latency into requests with an {{< reuse "agw-docs/snippets/policy.md" >}} to test how your clients and upstream services handle slow responses. Fault injection is useful for verifying that timeouts, retries, and client-side deadlines behave as expected. To learn more, you can use fault injection alongside [Timeouts]({{< link-hextra path="/resiliency/timeouts/" >}}) and [Retries]({{< link-hextra path="/resiliency/retry/" >}}).
+
+The injected delay counts against the request timeout. If the delay is longer than the configured [request timeout]({{< link-hextra path="/resiliency/timeouts/" >}}), the request times out.
+
+{{< reuse "agw-docs/snippets/agentgateway/prereq.md" >}}
+
+## Inject a delay {#inject-a-delay}
+
+1. Create an HTTPRoute for the httpbin app.
+   ```yaml {paths="delay-in-trafficpolicy"}
+   kubectl apply -n httpbin -f- <<EOF
+   apiVersion: gateway.networking.k8s.io/v1
+   kind: HTTPRoute
+   metadata:
+     name: httpbin-delay
+     namespace: httpbin
+   spec:
+     hostnames:
+     - faultinjection.example
+     parentRefs:
+     - name: agentgateway-proxy
+       namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
+     rules:
+     - backendRefs:
+       - kind: Service
+         name: httpbin
+         port: 8000
+   EOF
+   ```
+
+2. Create an {{< reuse "agw-docs/snippets/policy.md" >}} that injects a fixed 2-second delay into the requests that the HTTPRoute handles.
+   ```yaml {paths="delay-in-trafficpolicy"}
+   kubectl apply -f- <<EOF
+   apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}}
+   kind: {{< reuse "agw-docs/snippets/policy.md" >}}
+   metadata:
+     name: httpbin-delay
+     namespace: httpbin
+   spec:
+     targetRefs:
+     - group: gateway.networking.k8s.io
+       kind: HTTPRoute
+       name: httpbin-delay
+     traffic:
+       delay:
+         duration: 2s
+   EOF
+   ```
+
+   {{< reuse "agw-docs/snippets/review-table.md" >}}
+
+   | Field | Description |
+   | -- | -- |
+   | `targetRefs` | The route, gateway, or listener to apply the delay to. In this example, the policy targets the `httpbin-delay` HTTPRoute. |
+   | `traffic.delay.duration` | The latency to inject before the request is forwarded to the backend. Set either a duration string, such as `2s` or `500ms`, or a CEL expression that is evaluated against the request. For more information, see [Inject a probabilistic or random delay](#inject-a-probabilistic-or-random-delay). |
+
+3. Send a request along the route and verify that the response is delayed by about 2 seconds.
+   {{< tabs >}}
+   {{% tab name="Cloud Provider LoadBalancer" %}}
+   ```sh
+   curl -s -o /dev/null -w "%{time_total}s\n" http://$INGRESS_GW_ADDRESS/get -H "host: faultinjection.example"
+   ```
+   {{% /tab %}}
+   {{% tab name="Port-forward for local testing" %}}
+   ```sh
+   curl -s -o /dev/null -w "%{time_total}s\n" localhost:8080/get -H "host: faultinjection.example"
+   ```
+   {{% /tab %}}
+   {{< /tabs >}}
+
+   Example output:
+   ```console
+   2.01s
+   ```
+
+4. Optional: Verify that the delay policy is applied in the proxy configuration.
+
+   1. Port-forward the gateway proxy on port 15000.
+      ```sh
+      kubectl port-forward deploy/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} 15000
+      ```
+
+   2. Find the delay policy in the config dump.
+      ```sh
+      curl -s http://localhost:15000/config_dump | jq '[.policies[] | select(.policy.traffic.delay?)] | .[0]'
+      ```
+
+## Inject a probabilistic or random delay {#inject-a-probabilistic-or-random-delay}
+
+Because `duration` accepts a CEL expression, you can inject latency into only a subset of requests, or add jitter. The expression is evaluated against each request and returns either a duration or a number that is interpreted as milliseconds. A non-positive result injects no delay.
+
+| Expression | Effect |
+| -- | -- |
+| `duration("500ms")` | A fixed 500ms delay, expressed as a CEL duration. |
+| `random() < 0.1 ? 500 : 0` | A 500ms delay on approximately 10% of requests, and no delay otherwise. |
+| `int(random() * 500.0)` | A random delay between 0 and 500ms (jitter) on every request. |
+
+For example, the following policy delays approximately 10% of requests by 500ms.
+
+```yaml
+kubectl apply -f- <<EOF
+apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}}
+kind: {{< reuse "agw-docs/snippets/policy.md" >}}
+metadata:
+  name: httpbin-delay
+  namespace: httpbin
+spec:
+  targetRefs:
+  - group: gateway.networking.k8s.io
+    kind: HTTPRoute
+    name: httpbin-delay
+  traffic:
+    delay:
+      duration: "random() < 0.1 ? 500 : 0"
+EOF
+```
+
+{{< doc-test paths="delay-in-trafficpolicy" >}}
+# WHAT THIS TEST VALIDATES:
+#   * The httpbin-delay HTTPRoute (host faultinjection.example) is Accepted.
+#   * The httpbin-delay AgentgatewayPolicy (traffic.delay.duration: 2s) is Accepted.
+#   * A request to /get with host faultinjection.example returns 200 (the injected
+#     delay does not break the request).
+#   * The delay policy is present in the proxy config dump.
+# WHAT THIS TEST DOES NOT VALIDATE (and why):
+#   * That the response is actually delayed by ~2s — Different layer: YAMLTest has
+#     no response-latency assertion, so timing is not measured.
+#   * The probabilistic/random CEL delay example — Display-only block, not tagged.
+# Warm up the new faultinjection.example hostname before asserting (two-phase proxy warmup).
+for i in $(seq 1 60); do
+  curl -s --max-time 5 -o /dev/null "http://${INGRESS_GW_ADDRESS}:80/get" -H "host: faultinjection.example" && break
+  sleep 2
+done
+{{< /doc-test >}}
+
+{{< doc-test paths="delay-in-trafficpolicy" >}}
+YAMLTest -f - <<'EOF'
+- name: wait for httpbin-delay HTTPRoute to be accepted
+  wait:
+    target:
+      kind: HTTPRoute
+      metadata:
+        namespace: httpbin
+        name: httpbin-delay
+    jsonPath: "$.status.parents[0].conditions[?(@.type=='Accepted')].status"
+    jsonPathExpectation:
+      comparator: equals
+      value: "True"
+    polling:
+      timeoutSeconds: 300
+      intervalSeconds: 5
+- name: wait for httpbin-delay AgentgatewayPolicy to be accepted
+  wait:
+    target:
+      kind: AgentgatewayPolicy
+      metadata:
+        namespace: httpbin
+        name: httpbin-delay
+    jsonPath: "$.status.ancestors[0].conditions[?(@.type=='Accepted')].status"
+    jsonPathExpectation:
+      comparator: equals
+      value: "True"
+    polling:
+      timeoutSeconds: 300
+      intervalSeconds: 5
+- name: verify request through the delayed route returns 200
+  retries: 1
+  http:
+    url: "http://${INGRESS_GW_ADDRESS}:80"
+    path: /get
+    method: GET
+    headers:
+      host: "faultinjection.example"
+  source:
+    type: local
+  expect:
+    statusCode: 200
+- name: verify delay policy in config dump
+  http:
+    url: http://localhost:15000
+    skipSslVerification: true
+    method: GET
+    path: /config_dump
+  source:
+    type: pod
+    usePortForward: true
+    selector:
+      kind: Deployment
+      metadata:
+        namespace: agentgateway-system
+        name: agentgateway-proxy
+  expect:
+    bodyContains:
+    - '"duration"'
+EOF
+{{< /doc-test >}}
+
+## Cleanup
+
+{{< reuse "agw-docs/snippets/cleanup.md" >}} Run the following commands.
+
+```sh
+kubectl delete httproute httpbin-delay -n httpbin
+kubectl delete {{< reuse "agw-docs/snippets/policy.md" >}} httpbin-delay -n httpbin
+```

content/docs/kubernetes/main/resiliency/fault-injection.md

@@ -0,0 +1,16 @@
+---
+title: Fault injection
+weight: 20
+description: Inject artificial latency into requests to test how your clients and services handle slow responses.
+test:
+  delay-in-trafficpolicy:
+  - file: ${versionRoot}/quickstart/install.md
+    path: experimental
+  - file: ${versionRoot}/setup/gateway.md
+    path: all
+  - file: ${versionRoot}/install/sample-app.md
+    path: install-httpbin
+  - path: delay-in-trafficpolicy
+---
+
+{{< reuse "agw-docs/pages/resiliency/fault-injection.md" >}}

content/docs/standalone/main/configuration/resiliency/fault-injection.md

@@ -0,0 +1,144 @@
+---
+title: Fault injection
+weight: 20
+description: Inject artificial latency into requests to test how your clients and services handle slow responses.
+# TODO: re-enable once the agentgateway:latest-dev image includes the standalone
+# `delay` policy (added in agentgateway#2613 / commit c844384d). Until then the
+# --validate-only doc-test fails with "unknown field `delay`".
+# test:
+#   fault-injection:
+#   - file: ${versionRoot}/configuration/resiliency/fault-injection.md
+#     path: fault-injection
+---
+
+Attaches to: {{< badge content="Route" path="/configuration/routes/">}}
+
+{{< reuse "agw-docs/snippets/config-styles-note.md" >}}
+
+{{< doc-test paths="fault-injection" >}}
+{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}}
+{{< /doc-test >}}
+
+Fault injection adds artificial latency to matching requests before agentgateway forwards them to the backend. Use it to test how your clients and upstream services behave when responses are slow, such as verifying that timeouts, retries, and client-side deadlines work as expected.
+
+The injected delay counts against the request timeout. If the delay is longer than the configured [request timeout]({{< link-hextra path="/configuration/resiliency/timeouts/" >}}), the request times out.
+
+## Inject a delay
+
+Set `delay.duration` in the route policies. The `duration` can be either of the following values.
+
+| Value | Description |
+| -- | -- |
+| A duration string | A fixed latency to inject, such as `2s` or `500ms`. |
+| A CEL expression | An expression that is evaluated against each request and returns a duration or a number of milliseconds. Use this option for conditional, probabilistic, or randomized delays. A non-positive result injects no delay. |
+
+The following example injects a fixed 2-second delay before agentgateway forwards requests to the backend.
+
+{{< tabs >}}
+{{< tab name="Simplified (MCP)" >}}
+```yaml
+# yaml-language-server: $schema=https://agentgateway.dev/schema/config
+mcp:
+  port: 3000
+  policies:
+    delay:
+      duration: 2s
+  targets:
+  - name: everything
+    stdio:
+      cmd: npx
+      args: ["@modelcontextprotocol/server-everything"]
+```
+{{< /tab >}}
+{{< tab name="Routing-based" >}}
+```yaml
+# yaml-language-server: $schema=https://agentgateway.dev/schema/config
+binds:
+- port: 3000
+  listeners:
+  - routes:
+    - policies:
+        delay:
+          duration: 2s
+      backends:
+      - host: localhost:8080
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+## Inject a probabilistic or random delay
+
+Because `duration` accepts a CEL expression, you can inject latency into only a subset of requests, or add jitter. The expression returns either a duration or a number that is interpreted as milliseconds.
+
+| Expression | Effect |
+| -- | -- |
+| `duration("500ms")` | A fixed 500ms delay, expressed as a CEL duration. |
+| `random() < 0.1 ? 500 : 0` | A 500ms delay on approximately 10% of requests, and no delay otherwise. |
+| `int(random() * 500.0)` | A random delay between 0 and 500ms (jitter) on every request. |
+
+The following example delays approximately 10% of requests by 500ms.
+
+```yaml
+# yaml-language-server: $schema=https://agentgateway.dev/schema/config
+binds:
+- port: 3000
+  listeners:
+  - routes:
+    - policies:
+        delay:
+          duration: "random() < 0.1 ? 500 : 0"
+      backends:
+      - host: localhost:8080
+```
+
+{{< doc-test paths="fault-injection" >}}
+# WHAT THIS TEST VALIDATES:
+#   * The delay (fault injection) policy is accepted by agentgateway in the
+#     routing-based, simplified MCP, and CEL-expression forms.
+# WHAT THIS TEST DOES NOT VALIDATE (and why):
+#   * That the latency is actually injected at runtime — verifying that would
+#     require timing a live request against a running backend, which this page
+#     omits.
+cat <<'EOF' > config.yaml
+# yaml-language-server: $schema=https://agentgateway.dev/schema/config
+binds:
+- port: 3000
+  listeners:
+  - routes:
+    - policies:
+        delay:
+          duration: 2s
+      backends:
+      - host: localhost:8080
+EOF
+agentgateway -f config.yaml --validate-only
+
+cat <<'EOF' > config-mcp.yaml
+# yaml-language-server: $schema=https://agentgateway.dev/schema/config
+mcp:
+  port: 3000
+  policies:
+    delay:
+      duration: 2s
+  targets:
+  - name: everything
+    stdio:
+      cmd: npx
+      args: ["@modelcontextprotocol/server-everything"]
+EOF
+agentgateway -f config-mcp.yaml --validate-only
+
+cat <<'EOF' > config-cel.yaml
+# yaml-language-server: $schema=https://agentgateway.dev/schema/config
+binds:
+- port: 3000
+  listeners:
+  - routes:
+    - policies:
+        delay:
+          duration: "random() < 0.1 ? 500 : 0"
+      backends:
+      - host: localhost:8080
+EOF
+agentgateway -f config-cel.yaml --validate-only
+{{< /doc-test >}}