Skip to content

feat: Add JSON output format for CI/CD integration (closes #66)#99

Merged
ajitpratap0 merged 10 commits intomainfrom
feat/json-output-format-issue-66
Nov 16, 2025
Merged

feat: Add JSON output format for CI/CD integration (closes #66)#99
ajitpratap0 merged 10 commits intomainfrom
feat/json-output-format-issue-66

Conversation

@ajitpratap0
Copy link
Copy Markdown
Owner

Summary

Adds JSON output format support to validate and parse CLI commands for automation and CI/CD integration.

Changes

New Files

  • cmd/gosqlx/internal/output/json.go - JSON formatting functions
  • cmd/gosqlx/internal/output/json_test.go - Comprehensive test suite

Modified Commands

  • validate: Added --output-format json flag
  • parse: Added -f json format option

JSON Output Structure

Validation Output:

{
  "command": "validate",
  "input": {"type": "file", "files": ["test.sql"], "count": 1},
  "status": "success",
  "results": {
    "valid": true,
    "total_files": 1,
    "valid_files": 1,
    "invalid_files": 0
  },
  "errors": [],
  "stats": {
    "duration": "217.959µs",
    "throughput_files_per_sec": 4588.0
  }
}

Parse Output:

{
  "command": "parse",
  "status": "success",
  "results": {
    "ast": {"type": "AST", "statements": [...]},
    "metadata": {
      "parser_version": "2.0.0-alpha",
      "sql_compliance": "~80-85% SQL-99"
    }
  }
}

Features

✅ Error categorization (tokenization, parsing, syntax, io)
✅ Input type detection (file, files, stdin)
✅ Optional performance statistics
✅ Structured error reporting
✅ AST details and metadata
✅ Backward compatible (text format default)

Usage Examples

# Validate with JSON output
gosqlx validate --output-format json query.sql

# Parse with JSON output  
gosqlx parse -f json "SELECT * FROM users"

# CI/CD pipeline
gosqlx validate --output-format json --output-file results.json queries/

# Stdin support
cat query.sql | gosqlx validate --output-format json

Testing

✅ All JSON format tests pass
✅ All existing CLI tests pass (100+ tests)
✅ Pre-commit hooks passed

CI/CD Integration Benefits

  • Machine-readable output for automated tooling
  • Structured error reporting for GitHub Actions
  • Problem matchers for IDE integration
  • Performance metrics for monitoring
  • Standardized format for multiple tools

Closes #66

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

Ajit Pratap Singh and others added 10 commits November 16, 2025 21:36
Implement comprehensive stdin/stdout pipeline support for all CLI commands
(validate, format, analyze, parse) with Unix pipeline conventions and
cross-platform compatibility.

Features:
- Auto-detection: Commands automatically detect piped input
- Explicit stdin: Support "-" as stdin marker for all commands
- Input redirection: Full support for "< file.sql" syntax
- Broken pipe handling: Graceful handling of Unix EPIPE errors
- Security: 10MB input limit to prevent DoS attacks
- Cross-platform: Works on Unix/Linux/macOS and Windows PowerShell

Implementation:
- Created stdin_utils.go with pipeline utilities:
  - IsStdinPipe(): Detects piped input using golang.org/x/term
  - ReadFromStdin(): Reads from stdin with size limits
  - GetInputSource(): Unified input detection (stdin/file/direct SQL)
  - WriteOutput(): Handles stdout and file output with broken pipe detection
  - DetectInputMode(): Determines input mode based on args and stdin state
  - ValidateStdinInput(): Security validation for stdin content

- Updated all commands with stdin support:
  - validate.go: Stdin validation with temp file approach
  - format.go: Stdin formatting (blocks -i flag appropriately)
  - analyze.go: Stdin analysis with direct content processing
  - parse.go: Stdin parsing with direct content processing

- Dependencies:
  - Added golang.org/x/term for stdin detection

- Testing:
  - Unit tests: stdin_utils_test.go with comprehensive coverage
  - Integration tests: pipeline_integration_test.go for real pipeline testing
  - Manual testing: Validated echo, cat, and redirect operations

- Documentation:
  - Updated README.md with comprehensive pipeline examples
  - Unix/Linux/macOS and Windows PowerShell examples
  - Git hooks integration examples

Usage Examples:
  echo "SELECT * FROM users" | gosqlx validate
  cat query.sql | gosqlx format
  gosqlx validate -
  gosqlx format < query.sql
  cat query.sql | gosqlx format | gosqlx validate

Cross-platform:
  # Unix/Linux/macOS
  cat query.sql | gosqlx format | tee formatted.sql | gosqlx validate

  # Windows PowerShell
  Get-Content query.sql | gosqlx format | Set-Content formatted.sql
  "SELECT * FROM users" | gosqlx validate

Security:
- 10MB stdin size limit (MaxStdinSize constant)
- Binary data detection (null byte check)
- Input validation before processing
- Temporary file cleanup in validate command

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Resolved dependency conflicts in go.mod and go.sum:
- Kept newer golang.org/x/sys v0.38.0 (was v0.13.0 in main)
- Kept golang.org/x/term v0.37.0 (required for stdin/stdout pipeline)
- Added fsnotify v1.9.0 from watch mode feature
- Reorganized dependencies after go mod tidy

All tests passing after merge.
Fixed 3 critical issues causing all CI builds/tests to fail:

1. Go Version Format (Fixes: Build, Test, Vulnerability Check failures)
   - Changed go.mod from 'go 1.24.0' (three-part) to 'go 1.24' (two-part)
   - Three-part format not supported by Go 1.19/1.20 toolchains in CI
   - Error: 'invalid go version 1.24.0: must match format 1.23'

2. Lint Error SA9003 (Fixes: Lint job failure)
   - Fixed empty else branch in cmd/gosqlx/cmd/format.go:169-173
   - Removed unnecessary else block while preserving same behavior
   - Staticcheck SA9003: empty branch warning resolved

3. Workflow Go Version Mismatch (Fixes: Security scan failures)
   - Updated .github/workflows/security.yml to use Go 1.24
   - Both GoSec and GovulnCheck jobs now use Go 1.24
   - Matches project requirements for golang.org/x/term v0.37.0

All changes maintain backward compatibility and functionality.

Related: #65 (stdin/stdout pipeline feature)
Updated Go version across all GitHub Actions workflows to match go.mod requirements:

- .github/workflows/go.yml: Changed build matrix from [1.19, 1.20, 1.21] to [1.24]
- .github/workflows/test.yml: Changed test matrix from [1.19, 1.20, 1.21] to [1.24]
- .github/workflows/test.yml: Changed benchmark job from 1.21 to 1.24
- .github/workflows/lint.yml: Changed from 1.21 to 1.24

This fixes all remaining CI failures caused by incompatibility between:
- Project dependencies (golang.org/x/term v0.37.0) requiring Go 1.24
- Old workflow configurations using Go 1.19-1.21

Related: PR #97, Issue #65
Running go mod tidy updates go.mod format to go 1.24.0 (three-part)
which is the standard format for Go 1.24+. This resolves build failures
caused by out-of-sync go.mod and go.sum files.

Note: Go 1.24 supports both two-part (1.24) and three-part (1.24.0)
formats, but go mod tidy standardizes on three-part format.
- Replace hardcoded /tmp/ path with os.TempDir()
- Add path/filepath import for filepath.Join
- Fixes Windows test failure in TestWriteOutput
Add JSON output format support for validate and parse commands to enable
CI/CD integration, automation, and IDE problem matchers.

Changes:
- Add JSON output format structures in cmd/gosqlx/internal/output/json.go
  * JSONValidationOutput: Structured validation results
  * JSONParseOutput: Structured parse results with AST representation
  * Support for error categorization and performance statistics

- Update validate command (cmd/gosqlx/cmd/validate.go)
  * Add --output-format json flag (text/json/sarif)
  * Auto-enable quiet mode when using JSON format
  * Include stats in JSON when --stats flag is used
  * Support both file and stdin input

- Update parse command (cmd/gosqlx/cmd/parser_cmd.go)
  * Add -f json format option
  * Use standardized JSON output structure
  * Maintain backward compatibility with existing formats

- Add comprehensive test coverage (cmd/gosqlx/internal/output/json_test.go)
  * Validation JSON output tests (success/failure cases)
  * Parse JSON output tests
  * Error categorization tests
  * Input type detection tests
  * Statement conversion tests

JSON Output Features:
- Command executed
- Input file/query information
- Success/failure status
- Detailed error messages with type categorization
- Results (AST structure, validation results)
- Optional performance statistics

Example JSON output:
{
  "command": "validate",
  "input": {"type": "file", "files": ["test.sql"], "count": 1},
  "status": "success",
  "results": {
    "valid": true,
    "total_files": 1,
    "valid_files": 1,
    "invalid_files": 0
  }
}

All tests passing. Ready for CI/CD integration.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Resolved conflicts in validate.go
- Kept JSON output implementation from feature branch
- Integrated with stdin/stdout pipeline support from main
- All tests passing
@claude
Copy link
Copy Markdown

claude Bot commented Nov 16, 2025

Claude finished @ajitpratap0's task —— View job


PR Review: JSON Output Format Implementation

✅ Code Review Complete

This is a well-implemented JSON output format feature that adds significant value for CI/CD integration. Here's my detailed analysis:

🌟 Strengths

1. Excellent Code Quality

  • Clean Architecture: Well-structured JSON output package with clear separation of concerns
  • Type Safety: Comprehensive struct definitions with proper JSON tags
  • Error Handling: Robust error categorization and position tracking
  • Memory Management: Proper use of object pooling patterns (defer tokenizer.PutTokenizer(tkz))

2. Comprehensive Test Coverage

  • 409 lines of tests covering all major scenarios
  • Edge case testing: Empty files, stdin input, error conditions
  • Integration tests: Real SQL parsing with tokenizer and parser
  • Error categorization tests: Proper classification of different error types

3. Strong JSON Schema Design

// Well-designed output structure
type JSONValidationOutput struct {
    Command string                `json:"command"`
    Input   JSONInputInfo         `json:"input"`
    Status  string                `json:"status"` 
    Results JSONValidationResults `json:"results"`
    Errors  []JSONValidationError `json:"errors,omitempty"`
    Stats   *JSONValidationStats  `json:"stats,omitempty"`
}

4. Production-Ready Features

  • Multiple input types: File, files, stdin, directory detection
  • Performance metrics: Throughput calculation and duration tracking
  • Backward compatibility: Text format remains default
  • Security: Proper file permission handling (0600)

🔍 Minor Issues & Suggestions

1. String Matching Performance (json.go:291-302)

// Current implementation - could be optimized
func contains(s string, substrings ...string) bool {
    for _, substr := range substrings {
        if len(s) >= len(substr) {
            for i := 0; i <= len(s)-len(substr); i++ {
                if s[i:i+len(substr)] == substr {
                    return true
                }
            }
        }
    }
    return false
}

Suggestion: Consider using strings.Contains() for better performance:

func contains(s string, substrings ...string) bool {
    for _, substr := range substrings {
        if strings.Contains(strings.ToLower(s), substr) {
            return true
        }
    }
    return false
}

2. Error Message Truncation (json.go:270-274)

errorLower := errorMsg
if len(errorLower) > 100 {
    errorLower = errorLower[:100]
}

Issue: Variable name errorLower is misleading since no lowercasing happens.
Suggestion: Either actually lowercase or rename the variable.

3. Hardcoded Parser Version (json.go:182)

ParserVersion: "2.0.0-alpha",

Suggestion: Consider making this configurable or importing from a version constant.

🚀 Performance Considerations

Positive Aspects:

  • ✅ Proper object pooling usage
  • ✅ Efficient JSON marshaling with MarshalIndent
  • ✅ Optional statistics to avoid unnecessary computation

Potential Optimizations:

  • JSON encoding could be cached for repeated similar outputs
  • Consider streaming JSON for very large result sets

🔒 Security Assessment

✅ Good Security Practices:

  • File permissions set to 0600 for output files
  • Proper input validation and error handling
  • No sensitive information logged
  • Safe string operations

No Security Concerns Found

📊 Test Coverage Analysis

Excellent Coverage:

  • ✅ Success and failure scenarios
  • ✅ Different input types (file, files, stdin)
  • ✅ Error categorization testing
  • ✅ JSON structure validation
  • ✅ AST conversion testing
  • ✅ Integration with real parser components

Test Quality:

// Example of good test structure
func TestFormatValidationJSON_WithErrors(t *testing.T) {
    result := &ValidationResult{
        TotalFiles:   2,
        ValidFiles:   1, 
        InvalidFiles: 1,
        // ... detailed setup
    }
    
    jsonData, err := FormatValidationJSON(result, []string{"test1.sql", "test2.sql"}, false)
    if err != nil {
        t.Fatalf("FormatValidationJSON failed: %v", err)
    }
    
    // Comprehensive validation of output structure
}

🎯 Integration Quality

CLI Integration (validate.go & parser_cmd.go):

  • ✅ Clean flag handling (--output-format json)
  • ✅ Proper quiet mode activation for JSON output
  • ✅ Consistent error handling across commands
  • ✅ Good separation of concerns

📝 Minor Suggestions

  1. Add JSON Schema validation for output format consistency
  2. Consider adding a --json-compact flag for minified output
  3. Documentation: Consider adding godoc examples for public functions
  4. Version handling: Make parser version configurable

🏆 Overall Assessment

Rating: ⭐⭐⭐⭐⭐ (Excellent)

This is a production-ready implementation that:

  • ✅ Follows Go best practices and project conventions
  • ✅ Has comprehensive test coverage
  • ✅ Provides valuable CI/CD integration features
  • ✅ Maintains backward compatibility
  • ✅ Uses proper error handling and security practices

Recommendation: APPROVE - This PR significantly enhances the project's automation capabilities while maintaining high code quality standards.

🚦 CI/CD Benefits Delivered

The JSON output format will enable:

  • Automated SQL validation in GitHub Actions
  • Integration with problem matchers for IDE support
  • Structured error reporting for monitoring tools
  • Performance metrics collection for optimization

This is exactly what Issue #66 requested and is implemented excellently.

@ajitpratap0 ajitpratap0 merged commit 265cf4f into main Nov 16, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI-007: Output Format Options

1 participant