Add next-test command and adapt CLAUDE.md for this repo

- Create cmd/next-test to find the next todo test (shortest query first)
- Rewrite CLAUDE.md to match ClickHouse parser workflow:
  - Tests use explain.txt (not ast.json)
  - Remove TsqlAstParser section (not applicable)
  - Document metadata options

Author: claude

CLAUDE.md

@@ -13,11 +13,11 @@ This tool finds all tests with `todo: true` in their metadata and returns the on
 ## Workflow
 
 1. Run `go run ./cmd/next-test` to find the next test to implement
-2. Check the test's `query.sql` to understand what SQL needs parsing
-3. Check the test's `ast.json` to understand the expected output format
+2. Check the test's `query.sql` to understand what ClickHouse SQL needs parsing
+3. Check the test's `explain.txt` to understand the expected EXPLAIN output
 4. Implement the necessary AST types in `ast/`
 5. Add parser logic in `parser/parser.go`
-6. Add JSON marshaling functions in `parser/parser.go`
+6. Update the `Explain()` function if needed to match ClickHouse's output format
 7. Enable the test by removing `todo: true` from its `metadata.json` (set it to `{}`)
 8. Run `go test ./parser/... -timeout 5s` to verify
 9. Check if other todo tests now pass (see below)
@@ -37,79 +37,26 @@ The tests are very fast. If a test is timing out, it indicates a bug (likely an
 After implementing parser changes, run:
 
 ```bash
-go test ./parser/... -only-todo -v 2>&1 | grep "PASS:"
+go test ./parser/... -check-skipped -v 2>&1 | grep "PASSES NOW"
 ```
 
-This shows any todo tests that now pass. Enable those tests by removing `todo: true` from their `metadata.json`.
-
-Available test flags:
-
-- `-only-todo` - Run only todo/invalid_syntax tests (find newly passing tests)
-- `-run-todo` - Run todo/invalid_syntax tests along with normal tests
+Tests that output `PASSES NOW` can have their `todo` flag removed from `metadata.json`. This helps identify when parser improvements fix multiple tests at once.
 
 ## Test Structure
 
 Each test in `parser/testdata/` contains:
 
-- `metadata.json` - `{}` for enabled tests, `{"todo": true}` for pending tests, or `{"invalid_syntax": true}` for tests with invalid SQL
-- `query.sql` - T-SQL to parse
-- `ast.json` - Expected AST output
-
-## Important Rules
-
-**NEVER modify `ast.json` files** - These are golden files containing the expected output. If tests fail due to JSON mismatches, fix the Go code to match the expected output, not the other way around.
-
-## Generating ast.json with TsqlAstParser
-
-The `TsqlAstParser/` directory contains a C# tool that generates `ast.json` files using Microsoft's official T-SQL parser (ScriptDom).
-
-### Prerequisites
-
-1. Install .NET 8.0 SDK:
-
-```bash
-curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --channel 8.0 --install-dir ~/.dotnet
-```
-
-2. Download the NuGet package (if `packages/` directory is empty):
-
-```bash
-mkdir -p packages
-curl -L -o packages/microsoft.sqlserver.transactsql.scriptdom.170.128.0.nupkg \
-  "https://api.nuget.org/v3-flatcontainer/microsoft.sqlserver.transactsql.scriptdom/170.128.0/microsoft.sqlserver.transactsql.scriptdom.170.128.0.nupkg"
-```
-
-3. Build the tool:
-
-```bash
-~/.dotnet/dotnet build TsqlAstParser -c Release
-```
-
-### Usage
+- `metadata.json` - `{}` for enabled tests, `{"todo": true}` for pending tests
+- `query.sql` - ClickHouse SQL to parse
+- `explain.txt` - Expected EXPLAIN AST output (matches ClickHouse's format)
 
-Generate `ast.json` for a single test:
+### Metadata Options
 
-```bash
-~/.dotnet/dotnet run --project TsqlAstParser -c Release -- parser/testdata/TestName/query.sql parser/testdata/TestName/ast.json
-```
+- `todo: true` - Test is pending implementation
+- `skip: true` - Skip test entirely (e.g., causes infinite loop)
+- `explain: false` - Skip test (e.g., ClickHouse couldn't parse it)
+- `parse_error: true` - Query is intentionally invalid SQL
 
-Generate `ast.json` for all tests missing it:
-
-```bash
-for dir in parser/testdata/*/; do
-  if [ -f "$dir/query.sql" ] && [ ! -f "$dir/ast.json" ]; then
-    ~/.dotnet/dotnet run --project TsqlAstParser -c Release -- "$dir/query.sql" "$dir/ast.json"
-  fi
-done
-```
-
-### Limitations
-
-TsqlAstParser uses TSql160Parser (SQL Server 2022) and cannot parse:
-
-- SQL Server 170+ features (VECTOR indexes, AI functions, JSON enhancements)
-- Fabric DW-specific syntax (CLONE TABLE, CLUSTER BY)
-- Deprecated syntax removed in newer versions
-- Intentionally invalid SQL (error test cases)
+## Important Rules
 
-Tests for unsupported syntax will not have `ast.json` files generated.
+**NEVER modify `explain.txt` files** - These are golden files containing the expected output from ClickHouse. If tests fail due to output mismatches, fix the Go code to match the expected output, not the other way around.

cmd/next-test/main.go

@@ -0,0 +1,103 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+)
+
+type testMetadata struct {
+	Todo       bool  `json:"todo,omitempty"`
+	Explain    *bool `json:"explain,omitempty"`
+	Skip       bool  `json:"skip,omitempty"`
+	ParseError bool  `json:"parse_error,omitempty"`
+}
+
+type todoTest struct {
+	name      string
+	querySize int
+}
+
+func main() {
+	testdataDir := "parser/testdata"
+	entries, err := os.ReadDir(testdataDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error reading testdata: %v\n", err)
+		os.Exit(1)
+	}
+
+	var todoTests []todoTest
+
+	for _, entry := range entries {
+		if !entry.IsDir() {
+			continue
+		}
+
+		testDir := filepath.Join(testdataDir, entry.Name())
+		metadataPath := filepath.Join(testDir, "metadata.json")
+
+		// Read metadata
+		metadataBytes, err := os.ReadFile(metadataPath)
+		if err != nil {
+			continue
+		}
+
+		var metadata testMetadata
+		if err := json.Unmarshal(metadataBytes, &metadata); err != nil {
+			continue
+		}
+
+		// Only include tests marked as todo
+		if !metadata.Todo {
+			continue
+		}
+
+		// Skip tests with skip or explain=false or parse_error
+		if metadata.Skip || (metadata.Explain != nil && !*metadata.Explain) || metadata.ParseError {
+			continue
+		}
+
+		// Read query to get its size
+		queryPath := filepath.Join(testDir, "query.sql")
+		queryBytes, err := os.ReadFile(queryPath)
+		if err != nil {
+			continue
+		}
+
+		todoTests = append(todoTests, todoTest{
+			name:      entry.Name(),
+			querySize: len(queryBytes),
+		})
+	}
+
+	if len(todoTests) == 0 {
+		fmt.Println("No todo tests found!")
+		return
+	}
+
+	// Sort by query size (shortest first)
+	sort.Slice(todoTests, func(i, j int) bool {
+		return todoTests[i].querySize < todoTests[j].querySize
+	})
+
+	// Print the shortest one
+	next := todoTests[0]
+	testDir := filepath.Join(testdataDir, next.name)
+
+	fmt.Printf("Next test: %s\n\n", next.name)
+
+	// Print query.sql contents
+	queryPath := filepath.Join(testDir, "query.sql")
+	queryBytes, _ := os.ReadFile(queryPath)
+	fmt.Printf("Query (%d bytes):\n%s\n", next.querySize, string(queryBytes))
+
+	// Print explain.txt contents if it exists
+	explainPath := filepath.Join(testDir, "explain.txt")
+	if explainBytes, err := os.ReadFile(explainPath); err == nil {
+		fmt.Printf("\nExpected EXPLAIN output:\n%s\n", string(explainBytes))
+	}
+
+	fmt.Printf("\nRemaining todo tests: %d\n", len(todoTests))
+}