Fix incorrect Content-Type on /metrics endpoint#2418
Open
fishi0x01 wants to merge 1 commit into
Open
Conversation
common/status_app.py's metrics handler returned generate_latest() bytes via web.Response(body=...) without an explicit content_type, so aiohttp defaulted the response to "application/octet-stream". Legacy Prometheus (2.x) parses the body as classic exposition text regardless, but Prometheus 3.x rejects ambiguous Content-Type outright unless a fallback_scrape_protocol is configured - which caused these targets (listener, taskomatic, grouper, notificator, evaluator-upload, evaluator-recalc) to show as down when scraped by App-SRE's newer Cluster Observability Operator Prometheus stack, while working fine under the legacy stack. Fix: explicitly set the Content-Type header to CONTENT_TYPE_LATEST, matching prometheus_client's own documented usage pattern. Ref: https://redhat.atlassian.net/browse/APPSRE-13041
Reviewer's guide (collapsed on small PRs)Reviewer's GuideSets the Prometheus /metrics endpoint to return the correct text-based metrics Content-Type instead of the default application/octet-stream, by importing prometheus_client.CONTENT_TYPE_LATEST and applying it explicitly on the aiohttp response. Sequence diagram for updated /metrics Prometheus scrape responsesequenceDiagram
actor Prometheus
participant StatusApp
participant prometheus_client
participant aiohttp_web as web
Prometheus->>StatusApp: GET /metrics
StatusApp->>StatusApp: metrics(request)
StatusApp->>prometheus_client: generate_latest()
prometheus_client-->>StatusApp: metrics_bytes
StatusApp->>aiohttp_web: Response(body=metrics_bytes, headers={Content-Type: CONTENT_TYPE_LATEST})
aiohttp_web-->>Prometheus: 200 OK Content-Type=CONTENT_TYPE_LATEST
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
common/status_app.py's/metricshandler returnedgenerate_latest()bytes viaweb.Response(body=...)without an explicitcontent_type, so aiohttp defaulted the response'sContent-Typeheader toapplication/octet-stream.Why
Legacy Prometheus (2.x) parses the response body as the classic exposition text format regardless of Content-Type, so this went unnoticed. Prometheus 3.x is stricter and rejects ambiguous Content-Type outright unless a
fallback_scrape_protocolis configured on the scrape side. App-SRE's newer Cluster Observability Operator stack runs Prometheus 3.x, so scrapes againstlistener,taskomatic,grouper,notificator,evaluator-upload, andevaluator-recalc(all of which use this sharedstatus_appmodule) were failing with:while working fine under the legacy stack.
Fix
Explicitly set the
Content-Typeheader toCONTENT_TYPE_LATEST, matchingprometheus_client's own documented usage pattern (see exposition.py).Ref: APPSRE-13041
Summary by Sourcery
Bug Fixes: