|
| 1 | +# libsqlglot Testing Framework |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +libsqlglot has a comprehensive testing framework with **492 tests** covering all SQL dialects, features, and edge cases. The framework ensures robust, secure, and standards-compliant SQL parsing and transpilation across 45+ database dialects. |
| 6 | + |
| 7 | +## Test Structure |
| 8 | + |
| 9 | +### Test Files (53 total) |
| 10 | + |
| 11 | +``` |
| 12 | +tests/ |
| 13 | +├── Core Functionality |
| 14 | +│ ├── test_parser.cpp - Parser fundamentals |
| 15 | +│ ├── test_tokenizer.cpp - Tokenizer correctness |
| 16 | +│ ├── test_transpiler.cpp - Cross-dialect transpilation |
| 17 | +│ ├── test_generator.cpp - SQL generation |
| 18 | +│ └── test_optimizer.cpp - Query optimization |
| 19 | +│ |
| 20 | +├── Dialect Coverage |
| 21 | +│ ├── test_all_dialects_comprehensive.cpp - Universal SQL across all 45 dialects |
| 22 | +│ ├── test_dialect_transpilation.cpp - Dialect-to-dialect conversions |
| 23 | +│ ├── test_dialect_transforms.cpp - Dialect-specific transformations |
| 24 | +│ ├── test_dialect_coverage.cpp - Feature matrix validation |
| 25 | +│ └── test_dialect_feature_combinations.cpp - Feature combination fuzzing (NEW) |
| 26 | +│ |
| 27 | +├── SQL Features |
| 28 | +│ ├── test_advanced_sql.cpp - Complex queries (CTEs, window functions) |
| 29 | +│ ├── test_cte_windows_subqueries.cpp - CTE and window function edge cases |
| 30 | +│ ├── test_dml_statements.cpp - INSERT, UPDATE, DELETE, MERGE |
| 31 | +│ ├── test_grant_revoke.cpp - Security/permissions (GRANT, REVOKE) |
| 32 | +│ ├── test_utility_statements.cpp - ANALYZE, VACUUM, DELIMITER |
| 33 | +│ └── test_scalar_functions.cpp - Built-in functions |
| 34 | +│ |
| 35 | +├── Procedural SQL |
| 36 | +│ ├── test_stored_procedures.cpp - CREATE PROCEDURE, functions |
| 37 | +│ ├── test_begin_end_blocks.cpp - BEGIN/END statement blocks |
| 38 | +│ ├── test_if_statement.cpp - Conditional logic |
| 39 | +│ ├── test_for_loop.cpp - FOR loops |
| 40 | +│ ├── test_while_loop.cpp - WHILE loops |
| 41 | +│ ├── test_for_while_transpile.cpp - Loop transpilation |
| 42 | +│ ├── test_cursors.cpp - Cursor operations |
| 43 | +│ ├── test_exceptions.cpp - Exception handling |
| 44 | +│ ├── test_raise.cpp - RAISE statements |
| 45 | +│ ├── test_assignments.cpp - Variable assignments |
| 46 | +│ ├── test_declare_keyword.cpp - DECLARE statements |
| 47 | +│ ├── test_loop_break_continue.cpp - Loop control |
| 48 | +│ └── test_returns_comprehensive.cpp - RETURN statements |
| 49 | +│ |
| 50 | +├── Edge Cases & Security |
| 51 | +│ ├── test_error_recovery.cpp - Error handling |
| 52 | +│ ├── test_error_messages.cpp - Error message quality |
| 53 | +│ ├── test_security.cpp - SQL injection patterns |
| 54 | +│ ├── test_fk_check_constraints.cpp - Constraint validation |
| 55 | +│ └── test_mad_queries.cpp - Stress testing |
| 56 | +│ |
| 57 | +├── Memory & Performance |
| 58 | +│ ├── test_arena.cpp - Arena allocator correctness |
| 59 | +│ ├── test_intern.cpp - String interning |
| 60 | +│ └── test_performance.cpp - Benchmarking |
| 61 | +│ |
| 62 | +└── End-to-End |
| 63 | + └── test_end_to_end.cpp - Full stack integration |
| 64 | +``` |
| 65 | + |
| 66 | +## Test Categories |
| 67 | + |
| 68 | +### 1. Universal SQL Tests |
| 69 | + |
| 70 | +Test basic SQL statements across **all 45 dialects**: |
| 71 | + |
| 72 | +- `SELECT`, `INSERT`, `UPDATE`, `DELETE` |
| 73 | +- `JOIN` (INNER, LEFT, RIGHT, FULL, CROSS) |
| 74 | +- `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT` |
| 75 | +- Subqueries, `UNION`, `INTERSECT`, `EXCEPT` |
| 76 | + |
| 77 | +**Coverage**: Every dialect must parse these correctly. |
| 78 | + |
| 79 | +### 2. Dialect-Specific Feature Tests |
| 80 | + |
| 81 | +Test features unique to specific dialects: |
| 82 | + |
| 83 | +| Dialect | Features Tested | |
| 84 | +|---------|----------------| |
| 85 | +| **PostgreSQL** | ILIKE, array literals, JSON operators, PL/pgSQL | |
| 86 | +| **MySQL** | Backtick identifiers, LIMIT syntax, stored procedures | |
| 87 | +| **SQL Server** | TOP, bracket identifiers, T-SQL, BEGIN/END | |
| 88 | +| **BigQuery** | QUALIFY, three-part names (project.dataset.table) | |
| 89 | +| **Oracle** | FETCH FIRST, ROWNUM, PL/SQL, dual table | |
| 90 | +| **Snowflake** | QUALIFY, ILIKE, array functions | |
| 91 | +| **DuckDB** | PostgreSQL+ extensions, PIVOT | |
| 92 | + |
| 93 | +### 3. Feature Combination Tests (NEW) |
| 94 | + |
| 95 | +Test all combinations of SQL features: |
| 96 | + |
| 97 | +- **70+ SQL templates** × **45 dialects** = 3,150+ test cases |
| 98 | +- WHERE clause variations (=, >, <, IN, BETWEEN, LIKE, NULL, AND, OR) |
| 99 | +- JOIN combinations (2-way, 3-way, multi-table) |
| 100 | +- Aggregation + GROUP BY + HAVING combinations |
| 101 | +- Subqueries in WHERE/FROM/SELECT |
| 102 | +- CTE variations (simple, multiple, recursive) |
| 103 | + |
| 104 | +**Goal**: Ensure feature interactions work correctly. |
| 105 | + |
| 106 | +### 4. Fuzzing Tests |
| 107 | + |
| 108 | +#### SQL Injection Resistance |
| 109 | +```cpp |
| 110 | +"SELECT * FROM users WHERE id = 1 OR 1=1" |
| 111 | +"SELECT * FROM users; DROP TABLE users--" |
| 112 | +"SELECT * FROM users WHERE id = 1'; DROP TABLE users--" |
| 113 | +``` |
| 114 | +**Requirement**: Parser must handle gracefully without crashes. |
| 115 | + |
| 116 | +#### Edge Cases |
| 117 | +- **Deep nesting**: 50+ levels of subqueries (stack overflow protection) |
| 118 | +- **Long identifiers**: 1000+ character names |
| 119 | +- **Special characters**: Unicode, spaces, dashes in identifiers |
| 120 | +- **Large queries**: 10,000+ token queries |
| 121 | + |
| 122 | +#### Random Query Generation |
| 123 | +- Generates 100+ random valid SQL queries |
| 124 | +- Tests parser robustness against unexpected combinations |
| 125 | + |
| 126 | +### 5. Security Tests |
| 127 | + |
| 128 | +#### GRANT/REVOKE Comprehensive Coverage |
| 129 | +- Basic permissions: `SELECT`, `INSERT`, `UPDATE`, `DELETE` |
| 130 | +- Advanced: `EXECUTE`, `ALL`, `ALL PRIVILEGES` |
| 131 | +- Column-level: `GRANT SELECT (id, name) ON users` |
| 132 | +- Role management: `GRANT admin_role TO alice` |
| 133 | +- Options: `WITH GRANT OPTION`, `WITH ADMIN OPTION` |
| 134 | +- Revocation: `REVOKE GRANT OPTION FOR`, `REVOKE ADMIN OPTION FOR` |
| 135 | + |
| 136 | +#### Utility Statements |
| 137 | +- `ANALYZE` (statistics gathering) |
| 138 | +- `VACUUM` (storage optimization) |
| 139 | +- `DELIMITER` (batch script support) |
| 140 | + |
| 141 | +### 6. Cross-Dialect Transpilation |
| 142 | + |
| 143 | +Test **N × N dialect pairs** (selective sampling to avoid 45×45=2025 combinations): |
| 144 | + |
| 145 | +```cpp |
| 146 | +MySQL → PostgreSQL → BigQuery → Snowflake → Redshift |
| 147 | +Oracle → SQL Server → DuckDB → ClickHouse |
| 148 | +Hive → Spark → Presto → Trino |
| 149 | +``` |
| 150 | + |
| 151 | +**Round-trip testing**: Parse → Transpile → Transpile back → Verify semantics preserved |
| 152 | + |
| 153 | +### 7. Procedural SQL |
| 154 | + |
| 155 | +Full coverage of stored procedure features: |
| 156 | + |
| 157 | +- **Control flow**: IF/ELSE, CASE, FOR, WHILE, LOOP |
| 158 | +- **Variables**: DECLARE, assignments (`:=`, `=`, `SET`) |
| 159 | +- **Cursors**: DECLARE, OPEN, FETCH, CLOSE |
| 160 | +- **Exceptions**: BEGIN/EXCEPTION/END, RAISE |
| 161 | +- **Functions**: CREATE FUNCTION, RETURN |
| 162 | +- **Procedures**: CREATE PROCEDURE, CALL |
| 163 | + |
| 164 | +Tested across: **PostgreSQL** (PL/pgSQL), **MySQL** (stored procedures), **Oracle** (PL/SQL), **SQL Server** (T-SQL) |
| 165 | + |
| 166 | +## Running Tests |
| 167 | + |
| 168 | +### All Tests |
| 169 | +```bash |
| 170 | +docker compose -f docker/docker-compose.yml run --rm test |
| 171 | +``` |
| 172 | + |
| 173 | +### Specific Test Categories |
| 174 | +```bash |
| 175 | +# Dialect tests only |
| 176 | +./build/tests/libsqlglot_tests "[dialects]" |
| 177 | + |
| 178 | +# Fuzzing tests only |
| 179 | +./build/tests/libsqlglot_tests "[fuzzing]" |
| 180 | + |
| 181 | +# Security tests |
| 182 | +./build/tests/libsqlglot_tests "[security]" |
| 183 | + |
| 184 | +# Performance tests |
| 185 | +./build/tests/libsqlglot_tests "[performance]" |
| 186 | +``` |
| 187 | + |
| 188 | +### Random Fuzzing (excluded by default) |
| 189 | +```bash |
| 190 | +./build/tests/libsqlglot_tests --run-excluded-tests "[.]" |
| 191 | +``` |
| 192 | + |
| 193 | +## Test Metrics |
| 194 | + |
| 195 | +| Metric | Value | |
| 196 | +|--------|-------| |
| 197 | +| **Total Tests** | 492 | |
| 198 | +| **Test Files** | 53 | |
| 199 | +| **Dialects Covered** | 45 | |
| 200 | +| **SQL Templates** | 70+ | |
| 201 | +| **Feature Combinations** | 3,150+ | |
| 202 | +| **Pass Rate** | 100% | |
| 203 | + |
| 204 | +## Code Quality Standards |
| 205 | + |
| 206 | +### Memory Safety |
| 207 | +✅ Arena allocator with: |
| 208 | +- Integer overflow protection |
| 209 | +- Maximum allocation limits (1GB) |
| 210 | +- Alignment safety checks |
| 211 | +- RAII with `std::unique_ptr` |
| 212 | + |
| 213 | +✅ No manual memory management: |
| 214 | +- No `malloc/free` |
| 215 | +- No `new/delete` (except placement new in arena) |
| 216 | +- No unsafe C functions (`strcpy`, `strcat`, `sprintf`) |
| 217 | + |
| 218 | +### Parser Safety |
| 219 | +✅ Bounds checking on all token access: |
| 220 | +```cpp |
| 221 | +void advance() { |
| 222 | + if (!is_at_end()) pos_++; // Safe advancement |
| 223 | +} |
| 224 | +``` |
| 225 | + |
| 226 | +✅ Null pointer checks before string construction: |
| 227 | +```cpp |
| 228 | +if (is_at_end() || current().text == nullptr) { |
| 229 | + break; |
| 230 | +} |
| 231 | +``` |
| 232 | + |
| 233 | +### Build Configuration Separation |
| 234 | + |
| 235 | +#### Release Build (Production) |
| 236 | +```cmake |
| 237 | +-O3 # Full optimization |
| 238 | +-march=x86-64 # Portable (no AVX/AVX2 for wheels) |
| 239 | +-mtune=generic # Generic tuning |
| 240 | +-flto # Link-time optimization |
| 241 | +-fmerge-all-constants # Constant merging |
| 242 | +-fvisibility=hidden # Symbol hiding |
| 243 | +``` |
| 244 | + |
| 245 | +Optional: `-march=native -mtune=native` for local builds with `LIBSQLGLOT_ENABLE_NATIVE=ON` |
| 246 | + |
| 247 | +#### Debug Build (Development) |
| 248 | +```cmake |
| 249 | +-O0 # No optimization |
| 250 | +-g # Debug symbols |
| 251 | +``` |
| 252 | + |
| 253 | +**Guarantee**: Debug builds never contaminate release artifacts. |
| 254 | + |
| 255 | +## Best Practices |
| 256 | + |
| 257 | +### Adding New Tests |
| 258 | + |
| 259 | +1. **Create test file**: `tests/test_<feature>.cpp` |
| 260 | +2. **Use descriptive tags**: `[category][subcategory]` |
| 261 | + ```cpp |
| 262 | + TEST_CASE("Feature description", "[category][subcategory]") { |
| 263 | + // Test code |
| 264 | + } |
| 265 | + ``` |
| 266 | +3. **Test all dialects**: Loop through `all_dialects` vector |
| 267 | +4. **Add fuzzing**: Include edge cases and random generation |
| 268 | +5. **Update this document**: Add to appropriate section |
| 269 | +
|
| 270 | +### Test Organization |
| 271 | +
|
| 272 | +- **One concept per file**: Don't mix unrelated features |
| 273 | +- **Progressive complexity**: Basic → Intermediate → Advanced → Edge cases |
| 274 | +- **Clear failure messages**: Use `INFO()` for context |
| 275 | +- **Comprehensive coverage**: Test both success and failure paths |
| 276 | +
|
| 277 | +### Dialect-Specific Tests |
| 278 | +
|
| 279 | +When adding dialect-specific features: |
| 280 | +
|
| 281 | +1. Add to `DialectFeatures` in `dialects.h` |
| 282 | +2. Create test in `test_dialect_feature_combinations.cpp` |
| 283 | +3. Verify feature flag matches actual support |
| 284 | +4. Test transpilation to/from other dialects |
| 285 | +
|
| 286 | +## Future Enhancements |
| 287 | +
|
| 288 | +- [ ] Property-based testing (QuickCheck-style) |
| 289 | +- [ ] Mutation testing (inject bugs, verify tests catch them) |
| 290 | +- [ ] Coverage analysis (code coverage metrics) |
| 291 | +- [ ] Performance regression testing |
| 292 | +- [ ] Continuous fuzzing integration |
| 293 | +- [ ] Dialect compliance testing against vendor documentation |
| 294 | +
|
| 295 | +## Contributing |
| 296 | +
|
| 297 | +When submitting code: |
| 298 | +
|
| 299 | +1. ✅ **All 492 tests must pass** |
| 300 | +2. ✅ Add tests for new features |
| 301 | +3. ✅ Add fuzzing tests for edge cases |
| 302 | +4. ✅ Update this document |
| 303 | +5. ✅ Follow security best practices |
| 304 | +6. ✅ Maintain code quality standards |
| 305 | +
|
| 306 | +## Continuous Integration |
| 307 | +
|
| 308 | +Tests run automatically on: |
| 309 | +- Every commit |
| 310 | +- Every pull request |
| 311 | +- Nightly builds |
| 312 | +- Release candidates |
| 313 | +
|
| 314 | +**Zero tolerance**: No failing tests in main branch. |
| 315 | +
|
| 316 | +--- |
| 317 | +
|
| 318 | +Last updated: 2026-03-22 |
| 319 | +Version: 0.4.1 |
| 320 | +Test count: 492 |
0 commit comments