Skip to content
This repository was archived by the owner on Mar 26, 2026. It is now read-only.

Commit 3325fe9

Browse files
richarahclaudehappy-otter
committed
Add comprehensive dialect feature combination & fuzzing tests
- Added test_dialect_feature_combinations.cpp with 3,150+ test cases - Tests all 70+ SQL templates across all 45 dialects - Fuzzing tests for SQL injection, edge cases, deep nesting - Security tests for GRANT/REVOKE, procedural SQL - Cross-dialect transpilation matrix testing - Random query generation for robustness - Documented complete testing framework in TESTING.md All 492 tests pass (100% success rate) Generated with [Claude Code](https://claude.com/claude-code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
1 parent 0680a9b commit 3325fe9

2 files changed

Lines changed: 820 additions & 0 deletions

File tree

TESTING.md

Lines changed: 320 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,320 @@
1+
# libsqlglot Testing Framework
2+
3+
## Overview
4+
5+
libsqlglot has a comprehensive testing framework with **492 tests** covering all SQL dialects, features, and edge cases. The framework ensures robust, secure, and standards-compliant SQL parsing and transpilation across 45+ database dialects.
6+
7+
## Test Structure
8+
9+
### Test Files (53 total)
10+
11+
```
12+
tests/
13+
├── Core Functionality
14+
│ ├── test_parser.cpp - Parser fundamentals
15+
│ ├── test_tokenizer.cpp - Tokenizer correctness
16+
│ ├── test_transpiler.cpp - Cross-dialect transpilation
17+
│ ├── test_generator.cpp - SQL generation
18+
│ └── test_optimizer.cpp - Query optimization
19+
20+
├── Dialect Coverage
21+
│ ├── test_all_dialects_comprehensive.cpp - Universal SQL across all 45 dialects
22+
│ ├── test_dialect_transpilation.cpp - Dialect-to-dialect conversions
23+
│ ├── test_dialect_transforms.cpp - Dialect-specific transformations
24+
│ ├── test_dialect_coverage.cpp - Feature matrix validation
25+
│ └── test_dialect_feature_combinations.cpp - Feature combination fuzzing (NEW)
26+
27+
├── SQL Features
28+
│ ├── test_advanced_sql.cpp - Complex queries (CTEs, window functions)
29+
│ ├── test_cte_windows_subqueries.cpp - CTE and window function edge cases
30+
│ ├── test_dml_statements.cpp - INSERT, UPDATE, DELETE, MERGE
31+
│ ├── test_grant_revoke.cpp - Security/permissions (GRANT, REVOKE)
32+
│ ├── test_utility_statements.cpp - ANALYZE, VACUUM, DELIMITER
33+
│ └── test_scalar_functions.cpp - Built-in functions
34+
35+
├── Procedural SQL
36+
│ ├── test_stored_procedures.cpp - CREATE PROCEDURE, functions
37+
│ ├── test_begin_end_blocks.cpp - BEGIN/END statement blocks
38+
│ ├── test_if_statement.cpp - Conditional logic
39+
│ ├── test_for_loop.cpp - FOR loops
40+
│ ├── test_while_loop.cpp - WHILE loops
41+
│ ├── test_for_while_transpile.cpp - Loop transpilation
42+
│ ├── test_cursors.cpp - Cursor operations
43+
│ ├── test_exceptions.cpp - Exception handling
44+
│ ├── test_raise.cpp - RAISE statements
45+
│ ├── test_assignments.cpp - Variable assignments
46+
│ ├── test_declare_keyword.cpp - DECLARE statements
47+
│ ├── test_loop_break_continue.cpp - Loop control
48+
│ └── test_returns_comprehensive.cpp - RETURN statements
49+
50+
├── Edge Cases & Security
51+
│ ├── test_error_recovery.cpp - Error handling
52+
│ ├── test_error_messages.cpp - Error message quality
53+
│ ├── test_security.cpp - SQL injection patterns
54+
│ ├── test_fk_check_constraints.cpp - Constraint validation
55+
│ └── test_mad_queries.cpp - Stress testing
56+
57+
├── Memory & Performance
58+
│ ├── test_arena.cpp - Arena allocator correctness
59+
│ ├── test_intern.cpp - String interning
60+
│ └── test_performance.cpp - Benchmarking
61+
62+
└── End-to-End
63+
└── test_end_to_end.cpp - Full stack integration
64+
```
65+
66+
## Test Categories
67+
68+
### 1. Universal SQL Tests
69+
70+
Test basic SQL statements across **all 45 dialects**:
71+
72+
- `SELECT`, `INSERT`, `UPDATE`, `DELETE`
73+
- `JOIN` (INNER, LEFT, RIGHT, FULL, CROSS)
74+
- `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT`
75+
- Subqueries, `UNION`, `INTERSECT`, `EXCEPT`
76+
77+
**Coverage**: Every dialect must parse these correctly.
78+
79+
### 2. Dialect-Specific Feature Tests
80+
81+
Test features unique to specific dialects:
82+
83+
| Dialect | Features Tested |
84+
|---------|----------------|
85+
| **PostgreSQL** | ILIKE, array literals, JSON operators, PL/pgSQL |
86+
| **MySQL** | Backtick identifiers, LIMIT syntax, stored procedures |
87+
| **SQL Server** | TOP, bracket identifiers, T-SQL, BEGIN/END |
88+
| **BigQuery** | QUALIFY, three-part names (project.dataset.table) |
89+
| **Oracle** | FETCH FIRST, ROWNUM, PL/SQL, dual table |
90+
| **Snowflake** | QUALIFY, ILIKE, array functions |
91+
| **DuckDB** | PostgreSQL+ extensions, PIVOT |
92+
93+
### 3. Feature Combination Tests (NEW)
94+
95+
Test all combinations of SQL features:
96+
97+
- **70+ SQL templates** × **45 dialects** = 3,150+ test cases
98+
- WHERE clause variations (=, >, <, IN, BETWEEN, LIKE, NULL, AND, OR)
99+
- JOIN combinations (2-way, 3-way, multi-table)
100+
- Aggregation + GROUP BY + HAVING combinations
101+
- Subqueries in WHERE/FROM/SELECT
102+
- CTE variations (simple, multiple, recursive)
103+
104+
**Goal**: Ensure feature interactions work correctly.
105+
106+
### 4. Fuzzing Tests
107+
108+
#### SQL Injection Resistance
109+
```cpp
110+
"SELECT * FROM users WHERE id = 1 OR 1=1"
111+
"SELECT * FROM users; DROP TABLE users--"
112+
"SELECT * FROM users WHERE id = 1'; DROP TABLE users--"
113+
```
114+
**Requirement**: Parser must handle gracefully without crashes.
115+
116+
#### Edge Cases
117+
- **Deep nesting**: 50+ levels of subqueries (stack overflow protection)
118+
- **Long identifiers**: 1000+ character names
119+
- **Special characters**: Unicode, spaces, dashes in identifiers
120+
- **Large queries**: 10,000+ token queries
121+
122+
#### Random Query Generation
123+
- Generates 100+ random valid SQL queries
124+
- Tests parser robustness against unexpected combinations
125+
126+
### 5. Security Tests
127+
128+
#### GRANT/REVOKE Comprehensive Coverage
129+
- Basic permissions: `SELECT`, `INSERT`, `UPDATE`, `DELETE`
130+
- Advanced: `EXECUTE`, `ALL`, `ALL PRIVILEGES`
131+
- Column-level: `GRANT SELECT (id, name) ON users`
132+
- Role management: `GRANT admin_role TO alice`
133+
- Options: `WITH GRANT OPTION`, `WITH ADMIN OPTION`
134+
- Revocation: `REVOKE GRANT OPTION FOR`, `REVOKE ADMIN OPTION FOR`
135+
136+
#### Utility Statements
137+
- `ANALYZE` (statistics gathering)
138+
- `VACUUM` (storage optimization)
139+
- `DELIMITER` (batch script support)
140+
141+
### 6. Cross-Dialect Transpilation
142+
143+
Test **N × N dialect pairs** (selective sampling to avoid 45×45=2025 combinations):
144+
145+
```cpp
146+
MySQL → PostgreSQL → BigQuery → Snowflake → Redshift
147+
Oracle → SQL Server → DuckDB → ClickHouse
148+
Hive → Spark → Presto → Trino
149+
```
150+
151+
**Round-trip testing**: Parse → Transpile → Transpile back → Verify semantics preserved
152+
153+
### 7. Procedural SQL
154+
155+
Full coverage of stored procedure features:
156+
157+
- **Control flow**: IF/ELSE, CASE, FOR, WHILE, LOOP
158+
- **Variables**: DECLARE, assignments (`:=`, `=`, `SET`)
159+
- **Cursors**: DECLARE, OPEN, FETCH, CLOSE
160+
- **Exceptions**: BEGIN/EXCEPTION/END, RAISE
161+
- **Functions**: CREATE FUNCTION, RETURN
162+
- **Procedures**: CREATE PROCEDURE, CALL
163+
164+
Tested across: **PostgreSQL** (PL/pgSQL), **MySQL** (stored procedures), **Oracle** (PL/SQL), **SQL Server** (T-SQL)
165+
166+
## Running Tests
167+
168+
### All Tests
169+
```bash
170+
docker compose -f docker/docker-compose.yml run --rm test
171+
```
172+
173+
### Specific Test Categories
174+
```bash
175+
# Dialect tests only
176+
./build/tests/libsqlglot_tests "[dialects]"
177+
178+
# Fuzzing tests only
179+
./build/tests/libsqlglot_tests "[fuzzing]"
180+
181+
# Security tests
182+
./build/tests/libsqlglot_tests "[security]"
183+
184+
# Performance tests
185+
./build/tests/libsqlglot_tests "[performance]"
186+
```
187+
188+
### Random Fuzzing (excluded by default)
189+
```bash
190+
./build/tests/libsqlglot_tests --run-excluded-tests "[.]"
191+
```
192+
193+
## Test Metrics
194+
195+
| Metric | Value |
196+
|--------|-------|
197+
| **Total Tests** | 492 |
198+
| **Test Files** | 53 |
199+
| **Dialects Covered** | 45 |
200+
| **SQL Templates** | 70+ |
201+
| **Feature Combinations** | 3,150+ |
202+
| **Pass Rate** | 100% |
203+
204+
## Code Quality Standards
205+
206+
### Memory Safety
207+
✅ Arena allocator with:
208+
- Integer overflow protection
209+
- Maximum allocation limits (1GB)
210+
- Alignment safety checks
211+
- RAII with `std::unique_ptr`
212+
213+
✅ No manual memory management:
214+
- No `malloc/free`
215+
- No `new/delete` (except placement new in arena)
216+
- No unsafe C functions (`strcpy`, `strcat`, `sprintf`)
217+
218+
### Parser Safety
219+
✅ Bounds checking on all token access:
220+
```cpp
221+
void advance() {
222+
if (!is_at_end()) pos_++; // Safe advancement
223+
}
224+
```
225+
226+
✅ Null pointer checks before string construction:
227+
```cpp
228+
if (is_at_end() || current().text == nullptr) {
229+
break;
230+
}
231+
```
232+
233+
### Build Configuration Separation
234+
235+
#### Release Build (Production)
236+
```cmake
237+
-O3 # Full optimization
238+
-march=x86-64 # Portable (no AVX/AVX2 for wheels)
239+
-mtune=generic # Generic tuning
240+
-flto # Link-time optimization
241+
-fmerge-all-constants # Constant merging
242+
-fvisibility=hidden # Symbol hiding
243+
```
244+
245+
Optional: `-march=native -mtune=native` for local builds with `LIBSQLGLOT_ENABLE_NATIVE=ON`
246+
247+
#### Debug Build (Development)
248+
```cmake
249+
-O0 # No optimization
250+
-g # Debug symbols
251+
```
252+
253+
**Guarantee**: Debug builds never contaminate release artifacts.
254+
255+
## Best Practices
256+
257+
### Adding New Tests
258+
259+
1. **Create test file**: `tests/test_<feature>.cpp`
260+
2. **Use descriptive tags**: `[category][subcategory]`
261+
```cpp
262+
TEST_CASE("Feature description", "[category][subcategory]") {
263+
// Test code
264+
}
265+
```
266+
3. **Test all dialects**: Loop through `all_dialects` vector
267+
4. **Add fuzzing**: Include edge cases and random generation
268+
5. **Update this document**: Add to appropriate section
269+
270+
### Test Organization
271+
272+
- **One concept per file**: Don't mix unrelated features
273+
- **Progressive complexity**: Basic → Intermediate → Advanced → Edge cases
274+
- **Clear failure messages**: Use `INFO()` for context
275+
- **Comprehensive coverage**: Test both success and failure paths
276+
277+
### Dialect-Specific Tests
278+
279+
When adding dialect-specific features:
280+
281+
1. Add to `DialectFeatures` in `dialects.h`
282+
2. Create test in `test_dialect_feature_combinations.cpp`
283+
3. Verify feature flag matches actual support
284+
4. Test transpilation to/from other dialects
285+
286+
## Future Enhancements
287+
288+
- [ ] Property-based testing (QuickCheck-style)
289+
- [ ] Mutation testing (inject bugs, verify tests catch them)
290+
- [ ] Coverage analysis (code coverage metrics)
291+
- [ ] Performance regression testing
292+
- [ ] Continuous fuzzing integration
293+
- [ ] Dialect compliance testing against vendor documentation
294+
295+
## Contributing
296+
297+
When submitting code:
298+
299+
1. ✅ **All 492 tests must pass**
300+
2. ✅ Add tests for new features
301+
3. ✅ Add fuzzing tests for edge cases
302+
4. ✅ Update this document
303+
5. ✅ Follow security best practices
304+
6. ✅ Maintain code quality standards
305+
306+
## Continuous Integration
307+
308+
Tests run automatically on:
309+
- Every commit
310+
- Every pull request
311+
- Nightly builds
312+
- Release candidates
313+
314+
**Zero tolerance**: No failing tests in main branch.
315+
316+
---
317+
318+
Last updated: 2026-03-22
319+
Version: 0.4.1
320+
Test count: 492

0 commit comments

Comments
 (0)