Skip to content

feat(ARCH-002): Token Type Unification - Phase 1 & 2 Complete #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ajitpratap0 merged 9 commits into from
Nov 26, 2025
Merged

feat(ARCH-002): Token Type Unification - Phase 1 & 2 Complete #124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
767ec3d
feat(ARCH-002): extend TokenType with 120+ new SQL keywords and helpe…
Nov 25, 2025
c73043b
feat: add ModelType field and int-based comparisons (Phase 2)
Nov 26, 2025
a54f56f
test: add comprehensive tests for ModelType helper methods
Nov 26, 2025
0a4c36c
fix: use ModelType helper methods in production code to satisfy linter
Nov 26, 2025
6d0ff48
fix: remove unused ModelType helper methods to fix lint errors
Nov 26, 2025
9205f22
fix: address review feedback - fix comment ranges and rename TokenTyp…
Nov 26, 2025
6830f2f
feat: add token range constants and improve documentation
Nov 26, 2025
e707a87
feat: complete Phase 3 migration - eliminate string comparisons in pa…
Nov 26, 2025
68e3e0e
fix: add ModelType to benchmark tokens for fast int comparison path
Nov 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions performance_baselines.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,41 +1,41 @@
{
"version": "1.4.0",
"updated": "2025-01-17",
"version": "1.5.0",
"updated": "2025-11-26",
"baselines": {
"SimpleSelect": {
"ns_per_op": 650,
"tolerance_percent": 30,
"tolerance_percent": 40,
"description": "Basic SELECT query: SELECT id, name FROM users",
"current_performance": "~550-610 ns/op in CI, ~265 ns/op local (9 allocs, 536 B/op)",
"note": "CI environments show variability 550-610 ns/op; baseline updated to reflect CI reality"
"current_performance": "~550-610 ns/op in CI with ModelType fast path",
"note": "Test tokens include ModelType for fast int comparison path; increased tolerance for CI variability"
},
"ComplexQuery": {
"ns_per_op": 2500,
"tolerance_percent": 30,
"tolerance_percent": 40,
"description": "Complex SELECT with JOIN, WHERE, ORDER BY, LIMIT",
"current_performance": "~2400-2600 ns/op in CI, ~1020 ns/op local (36 allocs, 1433 B/op)",
"note": "CI environments show significant variability 2400-2600 ns/op; baseline updated to reflect CI reality"
"current_performance": "~2400-2600 ns/op in CI with ModelType fast path",
"note": "Test tokens include ModelType for fast int comparison path; increased tolerance for CI variability"
},
"WindowFunction": {
"ns_per_op": 1050,
"tolerance_percent": 30,
"tolerance_percent": 40,
"description": "Window function query: ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...)",
"current_performance": "~885-1005 ns/op in CI, ~400 ns/op local (14 allocs, 760 B/op)",
"note": "CI environments show significant variability 885-1005 ns/op; baseline updated to reflect CI reality"
"current_performance": "~885-1005 ns/op in CI with ModelType fast path",
"note": "Test tokens include ModelType for fast int comparison path; increased tolerance for CI variability"
},
"CTE": {
"ns_per_op": 1000,
"tolerance_percent": 30,
"tolerance_percent": 40,
"description": "Common Table Expression with WITH clause",
"current_performance": "~855-967 ns/op in CI, ~395 ns/op local (14 allocs, 880 B/op)",
"note": "CI environments show variability 855-967 ns/op; baseline updated to reflect CI reality"
"current_performance": "~855-967 ns/op in CI with ModelType fast path",
"note": "Test tokens include ModelType for fast int comparison path; increased tolerance for CI variability"
},
"INSERT": {
"ns_per_op": 750,
"tolerance_percent": 30,
"tolerance_percent": 40,
"description": "Simple INSERT statement",
"current_performance": "~660-716 ns/op in CI, ~310 ns/op local (14 allocs, 536 B/op)",
"note": "CI environments show variability 660-716 ns/op; baseline updated to reflect CI reality"
"current_performance": "~660-716 ns/op in CI with ModelType fast path",
"note": "Test tokens include ModelType for fast int comparison path; increased tolerance for CI variability"
},
"TokenizationThroughput": {
"tokens_per_sec": 8000000,
Expand Down
Loading
Loading