Express checkup#21
Closed
NikolayS wants to merge 89 commits into
Closed
Conversation
- Add D004_SETTINGS, F001_SETTINGS, G001_SETTINGS constants for filtering - Add generate_d004_from_a003, generate_f001_from_a003, generate_g001_from_a003 methods - Add filter_a003_settings and extract_postgres_version_from_a003 helpers - Add report_to_markdown method with report-specific markdown generators - Add --format option to CLI (json/markdown/both) - Update generate_all_reports to use A003 data for derived reports - Update tests to reflect new architecture
Instead of relying only on postgres_version field, now also extracts version info from server_version and server_version_num settings which are part of the A003 data itself.
Add `postgresai checkup` command for direct PostgreSQL health checks without requiring Prometheus. Generates JSON reports that comply with defined schemas. Features: - A002: Postgres major version - A003: Postgres settings (reuses query from metrics.yml) - A013: Postgres minor version (new schema added) Usage: postgresai checkup postgresql://user:pass@host/db postgresai checkup --check-id A003 --json The express mode runs SQL queries directly against PostgreSQL, useful for quick health checks without full monitoring setup.
Add comprehensive tests for the checkup module including: - parseVersionNum function with various PG versions - createBaseReport structure validation - CHECK_INFO and REPORT_GENERATORS constants - Mock client tests for getPostgresVersion, getSettings - Report generators (A002, A003, A013) - CLI help output and error handling
…checkup Add two new reports to the express checkup command: - A004: Cluster information (database sizes, connection stats, cache hit ratio) - A007: Altered settings (non-default PostgreSQL settings) New SQL queries added: - alteredSettings: filters pg_settings for non-default values - databaseSizes: queries pg_database for size info - clusterStats: aggregates pg_stat_database metrics - connectionStates: groups pg_stat_activity by state - uptimeInfo: gets server start time and uptime Also adds comprehensive unit tests for all new functionality.
- Remove --json flag (JSON is now the default output) - --output still writes JSON files to a directory - --upload outputs summary to stderr, JSON to stdout - Simplifies CLI usage for express checkup
- H001: Invalid indexes (indisvalid = false) - H002: Unused indexes (idx_scan = 0, excludes unique/PK indexes) - H003: Non-indexed foreign keys (FK constraints without supporting indexes) Added SQL queries, data fetching functions, generators, and unit tests.
- Added import for node:crypto (needed for generateDefaultProjectName) - Resolved import conflicts from auth module - Fixed stray conflict marker in postgres-ai.ts - Added deleteConfigKeys call on OAuth re-auth to clear stale project
- Add METRICS_SQL.redundantIndexes SQL query - Add RedundantIndex interface - Add getRedundantIndexes data fetcher function - Add generateH004 generator function - Register in REPORT_GENERATORS and CHECK_INFO - Update build script to also compile lib/checkup.ts - Add comprehensive unit tests
These JSON files were test artifacts that shouldn't be tracked in git.
Add JSON schema for H003 report to match the express checkup implementation.
The sync_github_prs_to_gitlab_mrs.py tool is no longer used.
- Convert checkup.test.cjs to checkup.test.ts using Bun's test framework - Remove H003 (non-indexed foreign keys) - will be added separately - Remove init.test.cjs (init.test.ts exists in main) - Remove H003.schema.json - Restore original build script (no extra lib/checkup.ts step) - Add structure validation tests for H001, H002, H004 reports
- Structure data by database name (matches reporter output format) - H001: Add relation_name, supports_fk fields - H002: Add index_definition, idx_is_btree fields, stats_reset object - H004: Add relation_name, index_definition fields - Add database_size_bytes/pretty to all H00x reports - Add statsReset SQL query for H002 - Update tests to match new schema-compliant structure
- Add ajv dependency for JSON schema validation - Validate generated reports against reporter/schemas/*.schema.json - Test both empty and populated report data - Use Ajv2020 for JSON Schema draft-2020-12 support
- Remove report_to_markdown() and related helper methods - Remove unused --format argument (only JSON is generated) - Fix bug: json_filename was defined but output_filename was used - Reporter now only generates JSON reports as intended
Add module docstring explaining that postgres_reports.py: - ONLY generates JSON reports with raw Observations - Does NOT convert to other formats (Markdown, HTML, etc.) - Does NOT generate Conclusions or Recommendations - Serves as input for downstream processing components
Follow SQL style guide rule: use lowercase SQL keywords (not uppercase). - select, from, join, where, and, order by, limit, with, as, on, etc. - Updated all METRICS_SQL queries in checkup.ts - Updated test mock client patterns to match
The epoch parameter is deprecated and no longer required by the API. Removed from createCheckupReport function and uploadCfg.
- Merge 22 individual CHECK_INFO/REPORT_GENERATORS tests into 3 parameterized tests - Remove 3 'has correct top-level structure' tests (covered by schema-validation.test.ts) - Reduce checkup.test.ts from 1162 to 966 lines (~17% reduction) - Reduce test count from 108 to 85 while maintaining coverage
- Extract shared mock client to test-utils.ts for reuse - Parameterize schema validation tests (272 → 81 lines, -70%) - Consolidate CHECK_INFO tests into single parameterized test - Remove duplicate structure tests (schema validation covers them) - Net reduction: ~271 lines while maintaining full test coverage
Resolve conflict in reporter/postgres_reports.py docstring: - Keep detailed scope documentation from our branch - Add new check types (K004-K008) from main branch
- Validates CLI-generated reports against JSON schemas from reporter/schemas/ - Tests all 11 check types (A002, A003, A004, A007, A013, D004, F001, G001, H001, H002, H004) - Ensures compatibility between express and full (monitoring) modes - Uses real PostgreSQL via pg_tmp for realistic testing
- Add generation_mode field to K008.schema.json (was missing after merge) - Add K008 to expected_report_codes in test_generators_unit.py
Address code review feedback: the source interface used Record<string, string> while the generated output used Record<number, string>. Since PG major versions are semantically numbers and YAML parses numeric keys as numbers, align both to use Record<number, string> for consistency.
Address code review feedback: reorder stop() to clear interval first, preventing any theoretical race with pending render callbacks.
Address code review feedback: add isValidRedundantToItem type guard to validate JSON structure before processing. Filter out invalid items instead of blindly casting to any. Use nullish coalescing for safer property access.
Address code review feedback: add databaseName option to MockClientOptions instead of hardcoding 'testdb'. This allows tests to verify behavior with different database names if needed.
Address code review feedback: use console.warn for truly unexpected errors (path.resolve failing) while keeping DEBUG-only logging for expected fallbacks (package.json not found in bundled environments).
Address code review feedback: - Add settled flag to prevent multiple resolve/reject calls - Call req.destroy() as backup when timeout fires - Reject promise directly in timeout handler (don't rely only on abort signal) - Use wrapper functions settledReject/settledResolve for safe promise settlement
Address code review feedback: add documentation explaining the two error handling patterns used in the module: - Propagating: core data functions throw on error (report should fail) - Graceful degradation: optional queries catch errors (include in output) Also add JSDoc @throws annotations to key functions.
Address code review feedback: add MS_PER_SECOND constant and use existing SECONDS_PER_MINUTE. Add comment explaining the threshold logic for formatting time values (ms/s/min based on magnitude).
Address code review feedback: checkup.test.ts now imports createMockClient from test-utils.ts instead of having its own duplicate implementation. - Added versionRows option to shared MockClientOptions - Updated all test calls to use new signature - Reduced checkup.test.ts by ~110 lines
The npm test script runs embed-metrics.ts first, which generates the metrics-embedded.ts file required by metrics-loader.ts. Running 'bun test' directly bypasses this step, causing module not found errors in CI.
- Extract checkup command helpers (prepareOutputDirectory, prepareUploadConfig, uploadCheckupReports, writeReportFiles, printUploadSummary) reducing action handler from ~230 to ~90 lines - Add JSDoc documentation to public API functions in checkup-api.ts - Add bunfig.toml with default 30s test timeout - Remove redundant individual timeout settings from init.integration.test.ts
- Moved spinner.stop() from catch block to finally block - Ensures interval cleanup in all edge cases (idempotent call) - Prevents potential memory leak if exception bypasses catch
- Replaced manual escaping with JSON.stringify for SQL strings - Handles all edge cases (unicode, control characters, etc.) - More robust than manual backslash/backtick/dollar escaping
- Clear timeout once response headers are received (callback invoked) - Prevents confusing 'timed out' errors when server is mid-response - Timeout now only applies to connection establishment phase - Updated error message to clarify 'no response' timeout
- Added optional redundant_to_parse_error field to RedundantIndex - Errors now captured in report output (not just console log) - Makes data quality issues visible in report JSON - Updated H004 schema to allow the new optional field
- Added optional postmaster_startup_error field to StatsReset - Consistent with documented error handling strategy (graceful degradation) - Errors now captured in report output for visibility - Updated H002 schema to allow the new optional field
- Added optional postmaster_startup_error field to StatsReset interface - Errors now captured in report output (consistent with documented pattern) - Updated H002 schema to allow the new optional field - Aligns with documented graceful degradation error handling strategy
- Added toBool() helper for converting various truthy values - Replaced manual boolean checks with toBool() for consistency - Handles numeric 0/1, string 't'/'f', and boolean values
- Added tests for timeout and ECONNRESET error retry behavior - Fixed isRetryableError to check error codes before message - Added 'timed out' to retryable error patterns - Added CI warning when integration tests are skipped - Helps catch CI configuration issues early
- Added JSDoc to key exported functions (generateAllReports, getInvalidIndexes, etc.) - Clarified API key dual-send pattern in checkup-api.ts (header for gateway, body for RPC) - Documents defense-in-depth security pattern for API authentication
- Added generateVersionReport for A002, A013 (version-only reports) - Added generateSettingsReport for A003, A007 (settings-based reports) - Added generateIndexReport for H001, H002, H004 (index reports with totals) - Reduced ~150 lines to ~85 lines (43% reduction) - Keeps A004 explicit due to unique structure
- Reordered stop() to set stopped=true before clearInterval() - Added comment explaining single-threaded safety model - Queued render() calls now see flag immediately on execution
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.