Skip to content

Commit ffbf975

Browse files
committed
feat(ai): align Gemma 4 model IDs with platform-specific formats
Add GemmaModel.to_backend_id() for per-platform model ID conversion (Google AI Studio, LM Studio, Ollama, OpenRouter). Update model fallback chain to skip cloud-unavailable 12B variant. Fix ruff RET504/RET505 lint errors. Update all AGENTS.md/CLAUDE.md/GEMINI.md docs with corrected model tables, env vars, and function names.
1 parent 10f0b83 commit ffbf975

14 files changed

Lines changed: 175 additions & 80 deletions

File tree

AGENTS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ sqlseed/
6666

6767
- **Provider fallback chain**: mimesis → faker → base (auto-degrades)
6868
- **AI backend fallback chain**: Google AI Studio → LM Studio → Ollama (multi-backend)
69-
- **Gemma 4 Native Function Calling**: `GEMMA_TOOLS` (analyze_schema, generate_column_values) with auto-fallback to JSON mode
69+
- **Gemma 4 Native Function Calling**: `GEMMA_TOOLS` (analyze_schema) with auto-fallback to JSON mode
7070
- **Context manager pattern**: `DataOrchestrator` is a context manager
7171
- **Plugin mediation**: `PluginMediator` bridges plugins and core (not direct calls)
7272
- **DAG-based column ordering**: `ColumnDAG` handles derive_from dependencies

CLAUDE.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ _utils/ → (no internal deps, used by all layers)
6464
### Key Modules
6565

6666
- **`core/orchestrator.py`**`DataOrchestrator` is the central coordinator. Uses `CoreCtx` (db, schema, mapper, relation, shared_pool) and `ExtCtx` (registry, plugins, mediator, enrichment, metrics) dataclasses. `fill_table()` is the main entry point; `fill()` is its alias.
67-
- **`core/mapper.py`**`ColumnMapper` with 9-level strategy chain (autoincrement PK → user config → exact match → default check → pattern match → nullable → type fallback). 74 exact rules, 26 regex patterns.
67+
- **`core/mapper.py`**`ColumnMapper` with 9-level strategy chain (autoincrement PK → user config → exact match → default check → pattern match → nullable → type fallback). 74 exact rules, 27 regex patterns.
6868
- **`core/schema.py`**`SchemaInferrer` reads SQLite schema + `CREATE TABLE` SQL for autoincrement detection.
6969
- **`core/relation.py`**`RelationResolver` + `SharedPool` for cross-table FK integrity. Implicit associations via name matching, explicit via `ColumnAssociation` config.
7070
- **`core/column_dag.py`** — Topological sort for `derive_from` column dependencies.
@@ -74,7 +74,8 @@ _utils/ → (no internal deps, used by all layers)
7474
- **`database/`**`DatabaseAdapter` protocol with two implementations: `SQLiteUtilsAdapter` (default, requires `sqlite-utils`) and `RawSQLiteAdapter` (fallback). `_compat.py` controls `HAS_SQLITE_UTILS` flag.
7575
- **`plugins/`** — 11 pluggy hooks. `PluginManager` + `PluginMediator` bridge plugins and core.
7676
- **`config/`** — Pydantic models (`GeneratorConfig`, `TableConfig`, `ColumnConfig`, `ColumnAssociation`), YAML/JSON loader, `SnapshotManager`.
77-
- **`cli/main.py`** — Click commands: `fill`, `preview`, `inspect`, `ai-suggest`, `config-generate`. CLI log level via `SQLSEED_LOG_LEVEL` env var (default `WARNING`). Cache dir via `SQLSEED_CACHE_DIR` env var.
77+
- **`cli/main.py`** — Click commands: `fill`, `preview`, `inspect`, `init`, `replay`, `ai-suggest`. CLI log level via `SQLSEED_LOG_LEVEL` env var (default `WARNING`). Cache dir via `SQLSEED_CACHE_DIR` env var.
78+
- **`_utils/paths.py`** — Platform-aware cache directory resolution (`get_cache_dir()`). Supports macOS/Linux/Windows, overridable via `SQLSEED_CACHE_DIR` env var.
7879

7980
### Public API (`src/sqlseed/__init__.py`)
8081

@@ -154,7 +155,7 @@ Run `pytest tests/test_doc_sync.py` to verify doc sync after changes.
154155

155156
## Testing
156157

157-
- Fixtures in `tests/conftest.py`: `tmp_db` (users + orders tables), `tmp_db_with_data`, `unique_test_db`, `create_project_info_db()` (project_info table with unique indexes), `raw_adapter`, `raw_adapter_with_data`
158+
- Fixtures in `tests/conftest.py`: `tmp_db` (users + orders tables), `tmp_db_with_data`, `unique_test_db`, `create_project_info_db()` (project_info table with unique indexes), `raw_adapter`, `raw_adapter_with_data`. Helper: `make_column_info()` factory for `ColumnInfo` objects.
158159
- Use real SQLite via `tmp_path` fixture, never mock the database layer
159160
- CLI tests: use `click.testing.CliRunner`, never subprocess
160161
- AI plugin tests: `pytest.importorskip("sqlseed_ai")`
@@ -206,6 +207,6 @@ When preparing a new version release:
206207

207208
## Dependencies
208209

209-
**Core**: sqlite-utils, pydantic, pluggy, structlog, pyyaml, click, rich, typing_extensions, simpleeval, rstr
210-
**Optional**: faker (`sqlseed[faker]`), mimesis (`sqlseed[mimesis]`)
210+
**Core**: pydantic, pluggy, structlog, pyyaml, click, rich, typing_extensions, simpleeval, rstr
211+
**Optional**: sqlite-utils (`sqlseed[sqlite-utils]`), faker (`sqlseed[faker]`), mimesis (`sqlseed[mimesis]`), tqdm (`sqlseed[tqdm]`)
211212
**Dev**: pytest, pytest-cov, pytest-asyncio, pytest-benchmark, ruff, mypy, pre-commit

GEMINI.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,9 @@ Core features include:
2323
- **`src/sqlseed/plugins/`**: Plugin management and hook specification definitions based on `pluggy` (`hookspecs.py` and `manager.py`).
2424
- **`src/sqlseed/config/`**: Configuration management using `pydantic` models, YAML/JSON loader (`loader.py`, `models.py`), and runtime snapshots supporting CLI `replay` command (`snapshot.py`).
2525
- **`src/sqlseed/cli/`**: `click`-based command-line interface (`main.py` providing fill, preview, inspect, init, replay, ai-suggest).
26-
- **`src/sqlseed/_utils/`**: Internal utilities including SQL safety (`sql_safe.py`), shared schema helpers (`schema_helpers.py`), performance metrics collection (`metrics.py`), progress bar wrapper (`progress.py`, based on `rich`), and logging wrapper (`logger.py`, based on `structlog`).
26+
- **`src/sqlseed/_utils/`**: Internal utilities including SQL safety (`sql_safe.py`), shared schema helpers (`schema_helpers.py`), performance metrics collection (`metrics.py`), progress bar wrapper (`progress.py`, based on `rich`), logging wrapper (`logger.py`, based on `structlog`), and platform-aware cache directory path utilities (`paths.py`).
2727
- **`plugins/sqlseed-ai/`**: Standalone package providing OpenAI-compatible LLM-driven generation. Contains `analyzer.py` (LLM table-level analysis), `refiner.py` (self-correction loop), `errors.py` (error summary), `examples.py` (few-shot examples), `config.py` (AIConfig model), `_client.py` (API client), `_json_utils.py` (JSON parsing), and `_model_selector.py` (Gemma 4 model selection, multi-backend support).
28-
- **`plugins/mcp-server-sqlseed/`**: MCP server based on FastMCP providing one Resource (`sqlseed://schema/{db_path}/{table_name}`) and three core Tools (`sqlseed_inspect_schema`, `sqlseed_generate_yaml`, `sqlseed_execute_fill`) for seamless AI assistant integration (driven by `server.py` and `config.py`).
28+
- **`plugins/mcp-server-sqlseed/`**: MCP server based on FastMCP providing one Resource (`sqlseed://schema/{db_path}/{table_name}`) and six core Tools (`sqlseed_inspect_schema`, `sqlseed_generate_yaml`, `sqlseed_execute_fill`, `sqlseed_gemma4_analyze`, `sqlseed_gemma4_agent_fill`, `sqlseed_list_gemma_models`) for seamless AI assistant integration (driven by `server.py` and `config.py`).
2929
- **`examples/`**: Contains tutorial Jupyter notebooks (`notebooks/`) covering quickstart, column mapping, generators, DB association, expression/DAGs, AI configuration, MCP server, and testing patterns. Also includes demo DB builder script (`build_demo_db.py`).
3030
- **`docs/`**: Project documentation including architecture diagrams (`architecture.md`).
3131

examples/notebooks/07-ai-plugin.ipynb

Lines changed: 11 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -367,30 +367,23 @@
367367
"name": "stdout",
368368
"output_type": "stream",
369369
"text": [
370-
"偏好免费模型 (12 个):\n",
371-
" 1. nvidia/nemotron-3-super-120b-a12b:free\n",
372-
" 2. tencent/hy3-preview:free\n",
373-
" 3. inclusionai/ling-2.6-1t:free\n",
374-
" 4. inclusionai/ling-2.6-flash:free\n",
375-
" 5. z-ai/glm-4.5-air:free\n",
376-
" 6. minimax/minimax-m2.5:free\n",
377-
" 7. openai/gpt-oss-120b:free\n",
378-
" 8. nvidia/nemotron-3-nano-30b-a3b:free\n",
379-
" 9. google/gemma-4-31b-it:free\n",
380-
" 10. nvidia/nemotron-nano-9b-v2:free\n",
381-
" 11. openai/gpt-oss-20b:free\n",
382-
" 12. google/gemma-4-26b-a4b-it:free\n",
370+
"Gemma 4 模型优先级 (5 个):\n",
371+
" 1. gemma-4-31b-it (Gemma 4 31B Dense)\n",
372+
" 2. gemma-4-26b-a4b-it (Gemma 4 26B A4B MoE (Recommended))\n",
373+
" 3. gemma-4-12b-it (Gemma 4 12B Unified (Laptop))\n",
374+
" 4. gemma-4-e4b-it (Gemma 4 E4B (4B Effective, Edge))\n",
375+
" 5. gemma-4-e2b-it (Gemma 4 E2B (2B Effective, Edge))\n",
383376
"\n",
384-
"select_best_free_model() 按优先级尝试, 自动回退\n"
377+
"select_gemma_model() 按优先级尝试, 自动回退\n"
385378
]
386379
}
387380
],
388381
"source": [
389-
"from sqlseed_ai._model_selector import PREFERRED_FREE_MODELS\n",
382+
"from sqlseed_ai._model_selector import _GEMMA_MODEL_PRIORITY\n",
390383
"\n",
391-
"print(f'偏好免费模型 ({len(PREFERRED_FREE_MODELS)} 个):')\n",
392-
"for i, model in enumerate(PREFERRED_FREE_MODELS, 1):\n",
393-
" print(f' {i}. {model}')\n",
384+
"print(f'Gemma 4 模型优先级 ({len(_GEMMA_MODEL_PRIORITY)} 个):')\n",
385+
"for i, model in enumerate(_GEMMA_MODEL_PRIORITY, 1):\n",
386+
" print(f' {i}. {model.value} ({model.display_name})')\n",
394387
"\n",
395388
"print('\\nselect_gemma_model() 按优先级尝试, 自动回退')\n",
396389
"\n"

plugins/mcp-server-sqlseed/AGENTS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ mcp-server-sqlseed/
1313
├── __init__.py # main() entry point
1414
├── __main__.py # python -m support
1515
├── config.py # MCPServerConfig (Pydantic)
16-
└── server.py # FastMCP server, 3 tools (190 lines)
16+
└── server.py # FastMCP server, 6 tools (349 lines)
1717
```
1818

1919
## WHERE TO LOOK

plugins/mcp-server-sqlseed/src/mcp_server_sqlseed/AGENTS.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ FastMCP 服务器实现。为 AI 助手提供 sqlseed 的数据生成工具。
3131
| `sqlseed_inspect_schema` | `db_path: str`, `table_name: str | None = None` | `dict[str, Any]` | 检查数据库 schema(含 schema_hash) |
3232
| `sqlseed_generate_yaml` | `db_path: str`, `table_name: str`, `max_retries: int = 3`, `api_key: str | None = None`, `base_url: str | None = None`, `model: str | None = None` | `str` (YAML 或错误文本) | AI 生成 YAML 配置 |
3333
| `sqlseed_execute_fill` | `db_path: str`, `table_name: str`, `count: int = 1000`, `yaml_config: str | None = None`, `enrich: bool = False` | `dict[str, Any]` | 执行数据填充 |
34+
| `sqlseed_gemma4_analyze` | `db_path: str`, `table_name: str`, `model: str | None = None`, `backend: str | None = None` | `dict[str, Any]` | Gemma 4 分析表结构并推荐配置 |
35+
| `sqlseed_gemma4_agent_fill` | `db_path: str`, `table_name: str`, `count: int = 1000`, `model: str | None = None`, `backend: str | None = None`, `max_retries: int = 3` | `dict[str, Any]` | Gemma 4 端到端:分析→生成→填充 |
36+
| `sqlseed_list_gemma_models` | (无参数) | `dict[str, Any]` | 列出 Gemma 4 模型变体和后端 |
3437

3538
- `_validate_db_path()` 验证扩展名必须为 `.db``.sqlite``.sqlite3`
3639
- `_MAX_YAML_CONFIG_SIZE = 256 * 1024`(256KB)限制 YAML 配置大小

plugins/sqlseed-ai/AGENTS.md

Lines changed: 33 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## OVERVIEW
44

5-
LLM-powered schema analysis and template generation. Separate package with own pyproject.toml.
5+
LLM-powered schema analysis and template generation. Separate package with own pyproject.toml. Supports 4 backends (Google AI Studio, LM Studio, Ollama, OpenAI-compatible) and 5 Gemma 4 model variants.
66

77
## STRUCTURE
88

@@ -11,24 +11,26 @@ sqlseed-ai/
1111
├── pyproject.toml # Separate package: sqlseed>=0.1.0, openai>=1.0
1212
└── src/sqlseed_ai/
1313
├── __init__.py # AISqlseedPlugin, plugin instance, hookimpl registration
14-
├── analyzer.py # SchemaAnalyzer — LLM schema analysis
15-
├── refiner.py # AiConfigRefiner — post-generation refinement and self-correction loop
16-
├── config.py # AIConfig — env-based OpenAI config
17-
├── errors.py # Custom exceptions
18-
├── _client.py # OpenAI client wrapper
19-
├── _model_selector.py # Model selection logic
20-
├── _json_utils.py # JSON parsing utilities
21-
└── examples.py # Usage examples
14+
├── analyzer.py # SchemaAnalyzer — LLM schema analysis, streaming, tool calling
15+
├── refiner.py # AiConfigRefiner — post-generation refinement, self-correction, streaming
16+
├── config.py # AIConfig — env-based config, GemmaModel enum, AIBackend enum
17+
├── errors.py # Error classification (7 processors)
18+
├── _client.py # OpenAI client wrapper, httpx timeout config
19+
├── _model_selector.py # Gemma 4 model selection and fallback chain
20+
├── _json_utils.py # JSON parsing utilities (3-strategy fallback)
21+
└── examples.py # Few-shot examples for prompts
2222
```
2323

2424
## WHERE TO LOOK
2525

2626
| Task | Location | Notes |
2727
|------|----------|-------|
2828
| Add hook | `__init__.py` | Decorate with `@hookimpl` |
29-
| Modify LLM calls | `_client.py` | OpenAI client wrapper |
30-
| Change model selection | `_model_selector.py` | Model picker logic |
31-
| Add config option | `config.py` | AIConfig.from_env() |
29+
| Modify LLM calls | `analyzer.py` | `call_llm()`, `call_llm_streaming()`, `_call_llm_once()` |
30+
| Change model selection | `_model_selector.py` | `select_gemma_model()`, `select_next_gemma_model()` |
31+
| Add config option | `config.py` | `AIConfig.from_env()`, `GemmaModel`, `AIBackend` |
32+
| Modify prompt templates | `analyzer.py` | `_SYSTEM_PROMPT`, `_COMPACT_SYSTEM_PROMPT`, `_ULTRA_COMPACT_SYSTEM_PROMPT` |
33+
| Change error handling | `errors.py` | `summarize_error()` with 7 processors |
3234

3335
## CONVENTIONS
3436

@@ -44,3 +46,22 @@ sqlseed-ai/
4446
- **NEVER** raise from hook methods → return None on failure
4547
- **ALWAYS** use `AIConfig.from_env()` for configuration
4648
- **ALWAYS** cap template generation at 50 values (`min(count, 50)`)
49+
50+
## Gemma 4 Model Variants
51+
52+
| Enum | Model ID | Use Case |
53+
|------|----------|----------|
54+
| `GEMMA_4_E2B` | `gemma-4-e2b-it` | Ultra-light edge, Ollama/LM Studio |
55+
| `GEMMA_4_E4B` | `gemma-4-e4b-it` | Lightweight local, LM Studio |
56+
| `GEMMA_4_12B` | `gemma-4-12b-it` | Balanced, LM Studio/Ollama |
57+
| `GEMMA_4_26B_A4B` | `gemma-4-26b-a4b-it` | High quality, recommended |
58+
| `GEMMA_4_31B` | `gemma-4-31b-it` | Best quality, Google AI Studio |
59+
60+
## Backend Configuration
61+
62+
| Backend | Default Base URL | Notes |
63+
|---------|-----------------|-------|
64+
| `google_ai_studio` | `https://generativelanguage.googleapis.com/v1beta/openai/` | Cloud, supports tool calling |
65+
| `lm_studio` | `http://127.0.0.1:1234/v1` | Local, auto-detect models |
66+
| `ollama` | `http://localhost:11434/v1` | Local, offline |
67+
| `openai_compat` | (must set `SQLSEED_AI_BASE_URL`) | Generic OpenAI-compatible |

plugins/sqlseed-ai/src/sqlseed_ai/AGENTS.md

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ AI 数据生成插件的实现。通过 OpenAI 兼容 API 分析数据库模式
1515
| `refiner.py` | `AiConfigRefiner` 配置优化器,基于 AI 建议优化现有配置,支持自纠正 |
1616
| `config.py` | `AIConfig` AI 配置(API key, model, base_url 等),支持环境变量 |
1717
| `_client.py` | OpenAI 客户端工厂,支持自定义 base_url |
18-
| `_model_selector.py` | `select_next_free_model()` 免费模型自动选择与降级 |
18+
| `_model_selector.py` | `select_next_gemma_model()` Gemma 模型自动选择与降级 |
1919
| `_json_utils.py` | JSON 响应容错解析,`_sanitize_names()` 名称清洗 |
2020
| `errors.py` | 错误类型定义,`ErrorSummary`/`summarize_error()` 错误汇总 |
2121
| `examples.py` | Few-shot 示例,用于 AI 提示词 |
@@ -25,10 +25,11 @@ AI 数据生成插件的实现。通过 OpenAI 兼容 API 分析数据库模式
2525

2626
| 环境变量 | 字段 | 回退 |
2727
|----------|------|------|
28-
| `SQLSEED_AI_API_KEY` | `api_key` | `OPENAI_API_KEY` |
29-
| `SQLSEED_AI_BASE_URL` | `base_url` | `OPENAI_BASE_URL` |
30-
| `SQLSEED_AI_MODEL` | `model` ||
28+
| `SQLSEED_AI_API_KEY` | `api_key` | `GOOGLE_API_KEY``OPENAI_API_KEY` |
29+
| `SQLSEED_AI_BASE_URL` | `base_url` | `OPENAI_BASE_URL`(按后端自动设置) |
30+
| `SQLSEED_AI_MODEL` | `model` |(自动检测本地模型) |
3131
| `SQLSEED_AI_TIMEOUT` | `timeout` | 默认 60.0 |
32+
| `SQLSEED_AI_BACKEND` | `backend` | 自动检测(`google_ai_studio`, `lm_studio`, `ollama`, `openai_compat`|
3233

3334
## LLM 调用与回退机制
3435

@@ -65,13 +66,15 @@ AI 数据生成插件的实现。通过 OpenAI 兼容 API 分析数据库模式
6566

6667
### Working In This Directory
6768

68-
- `AISqlseedPlugin` 实现 `hookimpl``sqlseed_ai_analyze_table`(分析整张表)和 `sqlseed_pre_generate_templates`(为非简单列生成模板值)
69+
- `AISqlseedPlugin` 实现 `hookimpl``sqlseed_ai_analyze_table`(分析整张表)和 `sqlseed_pre_generate_templates`(为非简单列生成模板值)。不实现 `sqlseed_register_providers``sqlseed_register_column_mappers`
6970
- 简单列(name, email, phone 等)通过 `_SIMPLE_COL_RE` 正则跳过 AI 调用,不要为简单列浪费 LLM token
70-
- `_model_selector.py` 维护免费模型列表,自动选择可用模型,支持降级,模型列表可能需要定期更新
71-
- JSON 解析必须使用 `_json_utils.py` 的容错逻辑,不要直接 `json.loads`,LLM 返回的 JSON 格式可能不规范
71+
- `_model_selector.py` 维护 Gemma 4 模型列表,`select_gemma_model()` 自动选择,`select_next_gemma_model()` 降级
72+
- JSON 解析必须使用 `_json_utils.py` 的容错逻辑(3 策略:直接解析 → 围栏清理 → raw_decode),不要直接 `json.loads`
7273
- 所有 AI 调用需处理 `APIConnectionError`/`APITimeoutError`/`APIError`
7374
- `refiner.py` 的自纠正流程:生成 → 验证 → 修正,最多重试若干次
74-
- `config.py``AIConfig` 支持环境变量(`SQLSEED_AI_API_KEY`, `SQLSEED_AI_MODEL`, `SQLSEED_AI_BASE_URL`
75+
- `config.py``AIConfig` 支持多后端自动检测,关键方法:`resolve_model()`, `resolve_base_url()`, `resolve_api_key()`, `resolve_max_tokens()`, `should_use_streaming()`, `should_use_ultra_compact()`
76+
- 流式调用:`call_llm_streaming()` + `generate_and_refine_streaming()`,E2B/E4B 模型自动禁用流式
77+
- Prompt 降级:normal → compact → ultra-compact,小模型(E2B/E4B)自动启用 ultra-compact
7578

7679
### Testing Requirements
7780

0 commit comments

Comments
 (0)