feat: Instrument pre-completion chat availability

[randy-concepcion]

What's New

Adds pre-completion coverage to mlpa_chat_availability_total, so auth rejections and user-provisioning failures show up in the chat availability signal.

A chat request terminates at one of four points and records exactly one outcome:

1. Auth (authorize_chat_request) - all excluded:
   • invalid_service_type_for_model
   • auth_rejected
   • invalid_auth_request
2. User provisioning (get_or_create_user_for_completion):
   • signup_cap_exceeded (excluded)
   • provisioning_failure (failure, 5xx only)
3. Blocked check: blocked (excluded)
4. Model call (get_completion / stream_completion): already instrumented

Notes

• The auth wrapper catches only HTTPException and records only 401 and 400. Everything else, including App Attest's explicit 500, is re-raised without recording.
• Shared-call 400s use invalid_auth_request rather than invalid_purpose because malformed App Attest base64 reaches the same path as purpose validation.
• auth_system_failure is defined but not emitted. See the limitation below.
• Reference: Availability RFC and SLI Inventory.

Known Limitation

This is an interim measurement. For FxA and Play Integrity, the same 401 response is returned whether the request was correctly rejected (such as an expired token) or the authentication system itself had a problem (such as an outage). Since they can't be distinguished, all of these are counted as expected rejections and left out of the ratio for now.

This means that real authentication-system failures are not yet counted as failures. This is a known gap, already noted in the Availability RFC and the SLI Inventory, and it will be addressed once those failures can be identified separately.

The availability ratio is success / (success + failure), and excluded and abort are held out.

[randy-concepcion]

/build-pr

[randy-concepcion]

Manual verification done in dev, in addition to the unit tests.

Verified manually

• The three auth-stage excluded reasons, triggered with crafted requests and confirmed recording with the right labels and outcome="excluded"
  • auth_rejected (valid service-type, invalid auth)
  • invalid_service_type_for_model (valid model with a service-type it does not allow)
  • invalid_auth_request (invalid purpose)
• Real dev traffic exercised success and upstream_error, so the availability ratio is live, currently around 99% with dips that line up with upstream_error clusters
• Confirmed the counter is scraped into Prometheus and queryable

Not verified manually
provisioning_failure, signup_cap_exceeded, and blocked need specific user or system state, so they were not triggered in dev. They are covered by the unit tests. auth_system_failure is defined but not emitted, which is the known auth limitation in the description.

Dashboard
Test dashboard built in Yardstick to validate the metric and compare against the existing success-rate panels.
[TEST] MLPA Availability

[randy-concepcion]

I've also added the same panels to our main Grafana dashboard so once these changes are deployed, we can start seeing them in Prod.

• MLPA Chat Availability (Rate)
• MLPA Chat Availability (Time Series)
• Exclusions/Failures Table

MLPA Availability Panels (Prod) - No data available until deployed.

[noahpodgurski]

Looks really good 🙌 just a few comments :)

[randy-concepcion]

Thanks for the review @noahpodgurski! I replied to each comment inline. I'll make some of these changes now and adjust according to your responses.

[randy-concepcion]

@noahpodgurski Thanks again for the feedback! I've pushed changes and is ready for another round of review when you have a chance! 🙏

[randy-concepcion]

/build-pr

[randy-concepcion]

Manually verified on dev. Grafana link for 2026-06-17 time window.

Prometheus: Metrics Explorer

MLPA Availability in Grafana (includes failures)