vttablet: improve health check resilience for MySQL connection limits

[tanjinx]

Description

Improve vttablet health check resilience so transient MySQL connection issues don't unnecessarily take the tablet out of service.

Problem

When max_user_connections is exceeded for the app user (e.g. vt_app), vttablet's CheckMySQL incorrectly concludes MySQL is unreachable and transitions the tablet to Not connected to mysql, taking it entirely out of service. This can cause prolonged outages from a transient resource exhaustion event, with cascading effects as load shifts to other tablets.

Three issues contribute:

1. IsMySQLReachable uses the app user (AppWithDB) to check MySQL liveness — the same user that just hit its connection limit.
2. IsTooManyConnectionsErr only recognizes the global max_connections handshake error (CRServerHandshakeErr + "Too many connections"), not per-user limits (ERTooManyUserConnections errno 1203, ERUserLimitReached errno 1226).
3. Transient socket errors (errno 2002/2003) from brief backlog saturation immediately fail the health check with no retry.

Changes

• Switch IsMySQLReachable to use DbaWithDB() instead of AppWithDB() so the health check doesn't compete with app connection limits.
• Extend IsTooManyConnectionsErr to recognize errno 1203 and 1226 as "MySQL is reachable but busy."
• Add retry logic with exponential backoff (100ms, 200ms, 400ms) for transient socket errors (errno 2002/2003).

Related Issue(s)

#19951

Checklist

• Tests were added or are not required
• Did the new or modified tests all pass locally?
• Documentation was added or is not required

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[Copilot]

Pull request overview

This PR improves vttablet’s MySQL health check behavior so transient connection exhaustion/connection errors don’t incorrectly transition a tablet to “Not connected to mysql” and take it out of service.

Changes:

• Switch health-check connectivity probe to use DBA credentials (DbaWithDB) instead of the app user.
• Expand “too many connections” detection to include per-user connection/resource limits (errno 1203/1226).
• Add short exponential-backoff retries for transient connect errors (errno 2002/2003) and unit tests for the new behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
go/vt/vttablet/tabletserver/query_engine.go	Refactors IsMySQLReachable to use a retrying helper and switches to DBA credentials.
go/vt/vttablet/tabletserver/query_engine_test.go	Adds tests covering retry behavior, transient errors, and connection-limit handling.
go/mysql/sqlerror/constants.go	Extends IsTooManyConnectionsErr to recognize per-user connection limit/resource errors.
go/mysql/sqlerror/sql_error_test.go	Adds unit tests validating the expanded IsTooManyConnectionsErr logic.

[codecov]

Codecov Report

❌ Patch coverage is 96.77419% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 78.12%. Comparing base (70c7a72) to head (6c3aee5).
⚠️ Report is 220 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/vttablet/tabletserver/query_engine.go	96.29%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #20008       +/-   ##
===========================================
+ Coverage   69.67%   78.12%    +8.45%     
===========================================
  Files        1614      103     -1511     
  Lines      216793    15722   -201071     
===========================================
- Hits       151044    12283   -138761     
+ Misses      65749     3439    -62310     

Flag	Coverage Δ
partial	78.12% <96.77%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.