Skip to content

feat: add semantic context and SQL intent planning to the generation pipeline#42

Merged
luiscarbonel1991 merged 1 commit into
mainfrom
feat/semantic-context-intent-planning
Apr 7, 2026
Merged

feat: add semantic context and SQL intent planning to the generation pipeline#42
luiscarbonel1991 merged 1 commit into
mainfrom
feat/semantic-context-intent-planning

Conversation

@luiscarbonel1991

Copy link
Copy Markdown
Owner

Summary

This PR upgrades nlp2sql from a mostly schema-and-question driven pipeline to a business-aware generation pipeline:

question -> semantic resolution -> schema/examples retrieval -> SQL intent plan -> prompt assembly -> SQL -> semantic/execution validation -> optional repair

It introduces a first-class semantic layer, structured intent planning before generation, semantic validation after generation, CLI support for semantic/example artifacts, and stronger public/local validation around the repository's e-commerce example domain.

What changed

Semantic domain model

Added first-class semantic entities:

  • SemanticContext
  • SemanticEntityMapping
  • MetricDefinition
  • DimensionDefinition
  • DomainRule
  • CanonicalQueryPattern
  • SqlIntentPlan
  • SemanticIssue
  • SemanticValidationResult

Ports and adapters

Added new extension points:

  • SemanticResolverPort
  • SemanticValidatorPort

Added initial adapters:

  • NoOpSemanticResolver
  • NoOpSemanticValidator
  • DictSemanticResolver
  • FileSemanticResolver

Service orchestration

Upgraded QueryGenerationService to orchestrate:

  • query analysis
  • semantic resolution
  • schema retrieval
  • example selection
  • SQL intent planning
  • prompt assembly
  • generation
  • semantic validation
  • optional execution validation and repair

Added supporting services:

  • SemanticResolutionService
  • SqlIntentPlanningService
  • SemanticValidationService
  • PromptAssemblyService
  • ExampleSelectionService
  • QueryAnalysisService
  • QueryRepairService

Public API

Exposed semantic support through the DSL:

  • connect(..., semantic_hooks=..., semantic_context=...)
  • nlp.ask(..., semantic_context=...)

Kept execution hooks and semantic hooks separate so downstream services can control them independently.

CLI

Added support for:

  • --semantic-context-file
  • --semantic-context-json
  • --examples-file
  • --examples-json
  • --show-semantic-context
  • --show-sql-intent-plan
  • --show-selected-examples
  • --validate
  • --repair

Centralized artifact loading in utils/artifact_loader.py.

Prompting and metadata

Improved provider prompt rendering with richer business context and structured SQL intent plan metadata.

Added richer result metadata including:

  • semantic_context
  • sql_intent_plan
  • selected_examples
  • repair_attempts
  • execution_validation

Local integration validation

Expanded the local PostgreSQL e-commerce schema to better validate semantic disambiguation.

Added regression-style integration coverage showing semantic context can move generation from a plausible transactional path to the intended aggregate fact table.

Sanitized automated tests and public docs to avoid private warehouse identifiers.

Docs and versioning

Refreshed public docs so they match the current DSL-first, semantic-aware architecture.

Bumped version to 0.2.0rc13.

Why

Few-shot examples alone were not enough to capture business meaning such as:

  • canonical fact tables
  • required filters
  • required dimensions
  • disallowed tables
  • preferred query patterns

This PR adds the missing semantic/business layer directly to the library while preserving the hexagonal architecture and keeping the public API ergonomic.

Validation

Ran:

  • uv lock
  • uv sync
  • uv run python -c "from nlp2sql import __version__; print(__version__)"
  • uv run pytest -m "integration and not llm"
  • uv run pytest -m "integration and llm"

Results:

  • version verified as 0.2.0rc13
  • integration and not llm: 21 passed, 6 skipped
  • integration and llm: 3 passed

Notes:

  • the skipped tests are LocalStack Redshift integration tests when local Redshift is not running
  • local PostgreSQL semantic/e-commerce integration flow passed

Follow-up

After this PR, the next step is to consume nlp2sql through a downstream uv source dependency and verify compatibility with an existing integration.

Closes #41
Closes #34

…pipeline

Introduce a first-class semantic business layer, structured SQL intent
planning, semantic validation, CLI artifact loading, and local e-commerce
validation for the DSL-first nlp2sql workflow.

- add SemanticContext, SqlIntentPlan, SemanticValidationResult, and related entities
- add semantic resolver and validator ports plus dict/file/noop adapters
- upgrade QueryGenerationService with semantic resolution, intent planning,
  semantic validation, and repair-aware orchestration
- expose semantic_hooks and semantic_context through connect() and ask()
- add CLI support for semantic/example artifacts and runtime metadata display
- expand local e-commerce integration coverage and sanitize public tests/docs
- bump version to 0.2.0rc13

Closes #41
Closes #34
@luiscarbonel1991 luiscarbonel1991 added enhancement New feature or request ai context labels Apr 6, 2026
@luiscarbonel1991
luiscarbonel1991 merged commit 5590825 into main Apr 7, 2026
1 check passed
@luiscarbonel1991
luiscarbonel1991 deleted the feat/semantic-context-intent-planning branch April 7, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai context enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add semantic context and SQL intent planning to the nlp2sql generation pipeline feat: add --examples-file flag to CLI for few-shot examples

1 participant