Skip to content

docs(egress): egress observability guide for metrics and access logging#2108

Open
maksymvavilov wants to merge 2 commits into
Kuadrant:mainfrom
maksymvavilov:gh-2074
Open

docs(egress): egress observability guide for metrics and access logging#2108
maksymvavilov wants to merge 2 commits into
Kuadrant:mainfrom
maksymvavilov:gh-2074

Conversation

@maksymvavilov

@maksymvavilov maksymvavilov commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • New guide at doc/user-guides/egress/observability.md covering Istio proxy metrics and Envoy access logs for egress gateway traffic
  • Metric label values, access log field mappings, and troubleshooting scenarios validated on a live Kind cluster with hack/setup-egress.sh
  • Cross-references added from egress-gateway.md (Next Steps) and observability/README.md (Overview list)

What the guide covers

  • Which standard Istio metrics are emitted by the egress gateway and how label values differ from ingress
  • PromQL queries for egress traffic rate, error rate, latency percentiles, and no-route detection
  • Access log field reference with egress-specific interpretation (upstream host, downstream remote address, response flags)
  • Troubleshooting scenarios: external service errors with per-endpoint upstream stats, missing routes, latency isolation (gateway overhead vs external service), client timeouts
  • Access log filtering via Istio Telemetry API

What it does not cover (separate issues)

This guide is the first deliverable for the egress observability epic (#2083). It will be extended in follow-up tasks:

Test plan

  • Verify guide renders correctly on the docs site
  • Verify all internal links resolve (egress-gateway.md, observability/README.md, tracing.md, telemetrypolicy.md, metrics.md, envoy-access-logs.md)
  • Walk through the guide commands on a Kind cluster with hack/setup-egress.sh

Closes #2074

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added a comprehensive guide to monitoring egress gateway traffic using Prometheus metrics and Envoy access logs.
    • Added PromQL examples for request rates, errors, latency, transferred bytes, and missing routes.
    • Added troubleshooting guidance for external-service failures, routing issues, and gateway latency.
    • Added instructions for enabling, filtering, and formatting egress access logs.
  • Documentation

    • Added links to the new egress observability guide from the observability overview and egress gateway documentation.

Add a new egress observability guide covering Istio proxy metrics and
Envoy access logs for egress gateway traffic. All metric label values
and access log examples are validated on a live Kind cluster.

This guide is the first deliverable for the egress observability epic
(Kuadrant#2083) and will be extended in follow-up tasks with distributed
tracing (Kuadrant#2084), TelemetryPolicy custom labels (Kuadrant#2086), and
TokenRateLimitPolicy (Kuadrant#2085) content.

Closes Kuadrant#2074

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Maskym Vavilov <mvavilov@redhat.com>
@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@maksymvavilov, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 39 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ac7b9511-dab9-4062-9f7b-e24150439fc8

📥 Commits

Reviewing files that changed from the base of the PR and between 44666ee and 670d8de.

📒 Files selected for processing (1)
  • doc/user-guides/egress/observability.md
📝 Walkthrough

Walkthrough

Adds an egress observability guide covering Prometheus metrics, Envoy access logs, troubleshooting scenarios, JSON logging, scrape verification, and follow-up references. Existing observability and egress gateway documentation now link to the guide.

Changes

Egress observability documentation

Layer / File(s) Summary
Metrics and egress observability guide
doc/user-guides/egress/observability.md
Adds prerequisites, egress-specific metric labels, PromQL examples, and guidance for identifying egress proxy traffic.
Access logs and troubleshooting
doc/user-guides/egress/observability.md
Documents Envoy access logs, response flags, filtering, JSON formatting, and troubleshooting for external-service, routing, timeout, and latency cases.
Verification and documentation navigation
doc/user-guides/egress/observability.md, doc/observability/README.md, doc/user-guides/egress/egress-gateway.md
Explains Prometheus scrape verification, adds references and next steps, and links the guide from existing documentation.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

  • #2083 — Covers the egress metrics, access logging, and observability guide documented here.
  • #2076 — Relates to extending the egress observability guide with tracing details.
  • #2080 — Relates to egress gateway metrics, Prometheus queries, and relevant labels.

Possibly related PRs

Suggested labels: area/doc, documentation

Suggested reviewers: mikenairn

Poem

I’m a rabbit with metrics to share,
Prometheus hops through egress air.
Logs tell tales of routes and bytes,
Flags reveal the gateway’s sights.
Trace the paths, both near and far—
Documentation shines like a star!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: a new egress observability guide for metrics and access logging.
Linked Issues check ✅ Passed The new guide and cross-references cover egress metrics, labels, Telemetry needs, access logs, and worked troubleshooting examples.
Out of Scope Changes check ✅ Passed No unrelated changes are evident; the added tracing and rate-limit references are framed as follow-up context, not scope creep.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@maksymvavilov maksymvavilov moved this to Ready For Review in Kuadrant Jul 14, 2026
@maksymvavilov maksymvavilov self-assigned this Jul 14, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (1)
doc/user-guides/egress/observability.md (1)

78-82: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add language identifiers to diagnostic code fences.

These plain fenced blocks trigger markdownlint MD040. Mark log and command output blocks as text so the documentation passes the configured lint rule.

Also applies to: 168-173, 199-203, 222-235, 243-247, 265-267

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@doc/user-guides/egress/observability.md` around lines 78 - 82, Add the text
language identifier to the fenced diagnostic output blocks in the observability
guide, including the shown metrics block and the additional referenced blocks,
while leaving their contents unchanged.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@doc/user-guides/egress/observability.md`:
- Line 254: Update the troubleshooting text for the NR condition to state only
that no HTTPRoute matched the request. Separate the HTTPRoute and ServiceEntry
checks, and mention a missing ServiceEntry only when evidence specifically
indicates that resource is absent.
- Around line 134-138: Update “Identifying the Egress Gateway in Metrics” to
clearly separate Kubernetes pod-label filtering from PromQL metric filtering:
describe gateway.networking.k8s.io/gateway-name=kuadrant-egressgateway only for
kubectl or log queries, and retain
source_workload="kuadrant-egressgateway-istio" as the istio_requests_total
PromQL selector.
- Around line 297-310: Update the accessLogging filter expression in the
egress-access-logs Telemetry example to retain entries where response.code is
unset as well as responses with status codes 400 or higher, using the required
unset-or-error condition.
- Around line 164-191: Correct the field mappings in the “Reading Egress Access
Logs” table to match the actual Istio/Envoy access-log format: identify field 12
as X-Forwarded-For, map 10.244.0.18:43722 to the downstream remote address, and
adjust all subsequent entries accordingly. Prefer named Envoy fields over
numeric positions where possible, while preserving the documented egress
troubleshooting guidance.
- Around line 258-269: Update the access-log timing explanation around
“Duration” and “Upstream service time” to describe their difference as
approximate non-upstream latency, including proxy processing, network transit,
and queueing. Replace claims of pure gateway overhead and statements that close
values prove the external service is the bottleneck with appropriately qualified
guidance.

---

Nitpick comments:
In `@doc/user-guides/egress/observability.md`:
- Around line 78-82: Add the text language identifier to the fenced diagnostic
output blocks in the observability guide, including the shown metrics block and
the additional referenced blocks, while leaving their contents unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1ca82c83-07cd-4e4c-b52e-62e227fbb125

📥 Commits

Reviewing files that changed from the base of the PR and between 21b1869 and 44666ee.

📒 Files selected for processing (3)
  • doc/observability/README.md
  • doc/user-guides/egress/egress-gateway.md
  • doc/user-guides/egress/observability.md

Comment thread doc/user-guides/egress/observability.md Outdated
Comment thread doc/user-guides/egress/observability.md Outdated
Comment thread doc/user-guides/egress/observability.md Outdated
Comment thread doc/user-guides/egress/observability.md Outdated
Comment thread doc/user-guides/egress/observability.md Outdated
- Separate Kubernetes pod label from PromQL metric label in gateway
  identification section
- Fix access log field table: use named format variables instead of
  incorrect positional numbers, add X-Forwarded-For field
- NR response flag indicates missing HTTPRoute, not necessarily missing
  ServiceEntry
- Latency difference is non-upstream latency (proxy, network, queueing),
  not pure gateway overhead
- Error-only filter uses !has(response.code) to capture connection
  failures where no HTTP response code is generated
- Add language specifiers to all fenced code blocks

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Maskym Vavilov <mvavilov@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Ready For Review

Development

Successfully merging this pull request may close these issues.

Validate egress metrics and access logging, write observability guide

1 participant