Skip to content

Commit ecbf5ab

Browse files
suryaiyer95claude
andcommitted
chore: remove FeedbackStore and cost prediction system
Remove the entire `sql.predict_cost` / `sql.record_feedback` system: - Delete `feedback_store.py` and its tests - Remove `SqlPredictCostTool` and `SqlRecordFeedbackTool` TypeScript tools - Remove protocol types, agent permissions, and bridge methods - Clean up all prompt files, docs, and progress references The static cost heuristic depended on the now-removed Rust `complexity_score` binding. The observation-based tiers (1-3) had no production usage. Removing entirely. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent bc79e31 commit ecbf5ab

25 files changed

Lines changed: 45 additions & 1063 deletions

PROGRESS.md

Lines changed: 29 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Last updated: 2026-02-27 00:40
2020
- [x] Phase 1 TODO: ConfidenceTracker with 7 AST detection rules — all 7 working
2121
- [x] Phase 1 TODO: Confidence signals on lineage.check — 4 signal types
2222
- [x] Phase 2A: Snowflake connector — password + key-pair auth, registered in ConnectionRegistry
23-
- [x] Phase 2B: Feedback store + cost report skill — SQLite-based, 4-tier prediction, bridge methods working
23+
- [x] Phase 2B: Cost report skill
2424
- [x] Phase 2C: dbt manifest parser — columns, sources, test/snapshot/seed counting
2525
- [x] Phase 3A: Impact analysis skill — downstream dependency graph + column-level impact classification
2626
- [x] Phase 3C: SQL translation skill — sqlglot transpile, 10 dialect-pair warnings, TS tool + skill
@@ -52,10 +52,9 @@ Last updated: 2026-02-27 00:40
5252
| 0 | Rules with known accuracy | **19/19** | 19/19 |
5353
| 0 | Analyzer overall accuracy | **100.00%** | measured |
5454
| 0 | Lineage edge match rate | **100.0%** | measured |
55-
| 1-7 | Working bridge methods | **34/34** | 34/34 |
55+
| 1-7 | Working bridge methods | **32/32** | 32/32 |
5656
| 1 | ConfidenceTracker rules | **7/7** | 7/7 |
5757
| 2 | Snowflake connector | **imports OK** | live test |
58-
| 2 | Feedback store observations | **working** | >0 |
5958
| 3 | Skills functional end-to-end | **6** | 5+ |
6059
| 4 | Rules with published benchmarks | **19/19** | 19/19 |
6160
| 5 | Schema cache tests | **20/20** | 20/20 |
@@ -110,14 +109,13 @@ UNUSED_CTE, WINDOW_WITHOUT_PARTITION
110109
- `packages/altimate-engine/src/altimate_engine/sql/analyzer.py` — 19 rules + ConfidenceTracker
111110
- `packages/altimate-engine/src/altimate_engine/sql/confidence.py` — 7 AST detection rules
112111
- `packages/altimate-engine/src/altimate_engine/lineage/check.py` — lineage + 4 confidence signals + Func/Window/Case edges
113-
- `packages/altimate-engine/src/altimate_engine/server.py` — JSON-RPC dispatch (34 methods)
112+
- `packages/altimate-engine/src/altimate_engine/server.py` — JSON-RPC dispatch (32 methods)
114113
- `packages/altimate-engine/src/altimate_engine/models.py` — All Pydantic models
115114

116115
**Phase 2 (connectors + parsers + feedback):**
117116
- `packages/altimate-engine/src/altimate_engine/connectors/snowflake.py` — password + key-pair auth
118117
- `packages/altimate-engine/src/altimate_engine/connections.py` — ConnectionRegistry with Snowflake
119118
- `packages/altimate-engine/src/altimate_engine/dbt/manifest.py` — Enhanced manifest parser
120-
- `packages/altimate-engine/src/altimate_engine/sql/feedback_store.py` — SQLite feedback + 4-tier prediction
121119

122120
**Phase 3 (skills + tools):**
123121
- `packages/altimate-engine/src/altimate_engine/sql/translator.py` — sqlglot transpile with lossy warnings
@@ -185,41 +183,39 @@ UNUSED_CTE, WINDOW_WITHOUT_PARTITION
185183
- `experiments/BENCHMARKS.md` — Published benchmark report
186184
- `experiments/benchmark_report.json` — Machine-readable benchmark data
187185

188-
### Bridge Methods (34 total)
186+
### Bridge Methods (32 total)
189187
1. `ping` — Health check
190188
2. `sql.validate` — SQL syntax validation
191189
3. `sql.check` — Read-only/mutation safety check
192190
4. `sql.execute` — SQL execution (PG/DuckDB)
193191
5. `sql.analyze` — 19 anti-pattern checks with confidence
194192
6. `sql.translate` — Cross-dialect SQL translation
195193
7. `sql.optimize` — Query optimization with suggestions
196-
8. `sql.record_feedback` — Record query execution metrics
197-
9. `sql.predict_cost` — Predict query cost (4-tier hierarchy)
198-
10. `schema.inspect` — Table schema inspection
199-
11. `lineage.check` — Column-level lineage with confidence
200-
12. `dbt.run` — dbt CLI execution
201-
13. `dbt.manifest` — Manifest parsing
202-
14. `warehouse.list` — List configured warehouses
203-
15. `warehouse.test` — Test warehouse connection
204-
16. `schema.index` — Index warehouse metadata into SQLite cache
205-
17. `schema.search` — Search indexed metadata (tables/columns) with natural language
206-
18. `schema.cache_status` — Show cache status (warehouses indexed, counts, timestamps)
207-
19. `sql.explain` — Run EXPLAIN on a query (Snowflake/PG/DuckDB dialect-specific syntax)
208-
20. `sql.format` — Format/beautify SQL via sqlglot pretty-print
209-
21. `sql.fix` — Diagnose SQL errors and suggest fixes (syntax, patterns, resolution)
210-
22. `sql.autocomplete` — Schema-aware auto-complete suggestions from cache
211-
23. `sql.diff` — Compare two SQL queries (unified diff, similarity score)
212-
24. `finops.query_history` — Query execution history (Snowflake QUERY_HISTORY, PG pg_stat_statements)
213-
25. `finops.analyze_credits` — Credit consumption analysis with recommendations
214-
26. `finops.expensive_queries` — Identify most expensive queries by bytes scanned
215-
27. `finops.warehouse_advice` — Warehouse sizing recommendations (scale up/down/burst)
216-
28. `finops.unused_resources` — Find stale tables and idle warehouses
217-
29. `finops.role_grants` — Query RBAC grants on objects/roles
218-
30. `finops.role_hierarchy` — Map role inheritance hierarchy
219-
31. `finops.user_roles` — List user-to-role assignments
220-
32. `schema.detect_pii` — Scan columns for PII patterns (30+ regex, 15 categories)
221-
33. `schema.tags` — Query metadata/governance tags on objects (Snowflake TAG_REFERENCES)
222-
34. `schema.tags_list` — List all available tags with usage counts
194+
8. `schema.inspect` — Table schema inspection
195+
9. `lineage.check` — Column-level lineage with confidence
196+
10. `dbt.run` — dbt CLI execution
197+
11. `dbt.manifest` — Manifest parsing
198+
12. `warehouse.list` — List configured warehouses
199+
13. `warehouse.test` — Test warehouse connection
200+
14. `schema.index` — Index warehouse metadata into SQLite cache
201+
15. `schema.search` — Search indexed metadata (tables/columns) with natural language
202+
16. `schema.cache_status` — Show cache status (warehouses indexed, counts, timestamps)
203+
17. `sql.explain` — Run EXPLAIN on a query (Snowflake/PG/DuckDB dialect-specific syntax)
204+
18. `sql.format` — Format/beautify SQL via sqlglot pretty-print
205+
19. `sql.fix` — Diagnose SQL errors and suggest fixes (syntax, patterns, resolution)
206+
20. `sql.autocomplete` — Schema-aware auto-complete suggestions from cache
207+
21. `sql.diff` — Compare two SQL queries (unified diff, similarity score)
208+
22. `finops.query_history` — Query execution history (Snowflake QUERY_HISTORY, PG pg_stat_statements)
209+
23. `finops.analyze_credits` — Credit consumption analysis with recommendations
210+
24. `finops.expensive_queries` — Identify most expensive queries by bytes scanned
211+
25. `finops.warehouse_advice` — Warehouse sizing recommendations (scale up/down/burst)
212+
26. `finops.unused_resources` — Find stale tables and idle warehouses
213+
27. `finops.role_grants` — Query RBAC grants on objects/roles
214+
28. `finops.role_hierarchy` — Map role inheritance hierarchy
215+
29. `finops.user_roles` — List user-to-role assignments
216+
30. `schema.detect_pii` — Scan columns for PII patterns (30+ regex, 15 categories)
217+
31. `schema.tags` — Query metadata/governance tags on objects (Snowflake TAG_REFERENCES)
218+
32. `schema.tags_list` — List all available tags with usage counts
223219

224220
### Skills (11 total)
225221
1. `generate-tests` — Generate dbt test definitions

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ Trace data flow at the column level through complex SQL transformations. Works s
7171

7272
### Warehouse Connectivity
7373

74-
Connect directly to your data warehouse -- the AI can query schemas, run SQL, predict costs, and analyze query history:
74+
Connect directly to your data warehouse -- the AI can query schemas, run SQL, and analyze query history:
7575

7676
- Snowflake (with IAM auth)
7777
- BigQuery (service account + ADC)

docs/docs/data-engineering/agent-modes.md

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,8 @@ altimate-code --agent builder
1313
Builder mode follows a strict pre-execution protocol for every SQL operation:
1414

1515
1. `sql_analyze` — Check for anti-patterns
16-
2. `sql_predict_cost` — Estimate execution cost
17-
3. `sql_validate` — Verify syntax and schema references
18-
4. `sql_execute` — Run the query
19-
5. `sql_record_feedback` — Store metrics for future cost predictions
16+
2. `sql_validate` — Verify syntax and schema references
17+
3. `sql_execute` — Run the query
2018

2119
### Example: Create a staging model
2220

@@ -156,10 +154,6 @@ Validator:
156154
~ Modified column: total_revenue FLOAT → DECIMAL(18,2)
157155
Severity: medium (type change may affect downstream)
158156
159-
> sql_predict_cost [fct_revenue.sql]
160-
Tier 2 match: ~12.3 credits per run (medium confidence)
161-
Previous version: ~11.8 credits — marginal increase
162-
163157
Issues found:
164158
1. FUNCTION_IN_FILTER on line 23 — use a pre-computed date column instead
165159
2. Type change from FLOAT to DECIMAL(18,2) — verify downstream consumers

docs/docs/data-engineering/guides/cost-optimization.md

Lines changed: 1 addition & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -87,30 +87,7 @@ This finds:
8787
- Warehouses with no queries in 7+ days
8888
- Temporary tables from old migrations
8989

90-
## Step 5: Predict before you execute
91-
92-
Every query goes through cost prediction before execution:
93-
94-
```
95-
You: How much will this query cost?
96-
97-
> sql_predict_cost "SELECT * FROM raw_clickstream"
98-
99-
Tier 3 estimate: ~45 credits
100-
Table size: 890GB, 12B rows
101-
Recommendation: Add date filter + column pruning → estimated 2-3 credits
102-
```
103-
104-
## Step 6: Build a cost feedback loop
105-
106-
After each query, `sql_record_feedback` stores actual execution metrics. This trains the cost prediction model to be more accurate over time.
107-
108-
```
109-
Query executed: 0.84 credits (predicted: 0.79, Tier 2)
110-
Feedback recorded → next prediction will be more accurate
111-
```
112-
113-
## Automation: CI cost gate
90+
## Step 5: Automate cost gates in CI
11491

11592
Use `ci_cost_gate` in your CI/CD pipeline to block expensive queries from reaching production:
11693

docs/docs/data-engineering/guides/using-with-claude-code.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,6 @@ Claude Code: I see changes to models/marts/fct_revenue.sql. Let me check
3333
3434
altimate-code:
3535
sql_analyze: No anti-patterns detected ✓
36-
sql_predict_cost: ~2.3 credits per run (Tier 2, medium confidence)
3736
lineage_check: Sources stg_orders.amount, dim_products.category
3837
schema_diff: Added column revenue_category (low severity)
3938
```

docs/docs/data-engineering/tools/sql-tools.md

Lines changed: 0 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -76,36 +76,6 @@ Each rule includes a **confidence score** (high/medium/low) based on AST complex
7676

7777
---
7878

79-
## sql_predict_cost
80-
81-
Estimate query cost before execution using a 4-tier prediction system.
82-
83-
```
84-
> sql_predict_cost "SELECT * FROM events WHERE event_date > '2026-01-01'"
85-
86-
Cost Prediction:
87-
Tier: 2 (template match)
88-
Estimated bytes scanned: 4.2 GB
89-
Estimated credits: 0.84
90-
Confidence: medium
91-
92-
Similar queries averaged 0.79 credits over 23 executions.
93-
Recommendation: Add column pruning to reduce scan by ~60%
94-
```
95-
96-
### How it works
97-
98-
| Tier | Method | Confidence |
99-
|---|---|---|
100-
| **1 — Fingerprint** | Exact query found in history | High |
101-
| **2 — Template** | Similar query structure matched | Medium |
102-
| **3 — Table scan** | Estimate from table metadata | Low |
103-
| **4 — Heuristic** | No data available, worst-case estimate | Very low |
104-
105-
The system improves over time. After each query execution, `sql_record_feedback` stores actual metrics to train better predictions.
106-
107-
---
108-
10979
## sql_optimize
11080

11181
Get optimization suggestions with rewritten SQL.

packages/altimate-code/src/agent/agent.ts

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -109,8 +109,6 @@ export namespace Agent {
109109
sql_analyze: "allow",
110110
sql_translate: "allow",
111111
sql_optimize: "allow",
112-
sql_predict_cost: "allow",
113-
sql_record_feedback: "allow",
114112
lineage_check: "allow",
115113
warehouse_list: "allow",
116114
warehouse_test: "allow",
@@ -166,8 +164,6 @@ export namespace Agent {
166164
sql_analyze: "allow",
167165
sql_translate: "allow",
168166
sql_optimize: "allow",
169-
sql_predict_cost: "allow",
170-
sql_record_feedback: "allow",
171167
lineage_check: "allow",
172168
warehouse_list: "allow",
173169
warehouse_test: "allow",
@@ -223,8 +219,6 @@ export namespace Agent {
223219
sql_analyze: "allow",
224220
sql_translate: "allow",
225221
sql_optimize: "allow",
226-
sql_predict_cost: "allow",
227-
sql_record_feedback: "allow",
228222
lineage_check: "allow",
229223
warehouse_list: "allow",
230224
warehouse_test: "allow",
@@ -277,8 +271,6 @@ export namespace Agent {
277271
sql_validate: "allow",
278272
sql_translate: "allow",
279273
sql_optimize: "allow",
280-
sql_predict_cost: "allow",
281-
sql_record_feedback: "allow",
282274
lineage_check: "allow",
283275
warehouse_list: "allow",
284276
warehouse_test: "allow",

packages/altimate-code/src/agent/prompt/analyst.txt

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,18 +27,13 @@ You're the analyst's research assistant. Research should be fast AND cheap.
2727

2828
1. **Start small**: Use LIMIT for initial exploration. Don't scan entire tables to answer "what does this column look like?" — LIMIT 100 is enough.
2929

30-
2. **Predict before you scan**: For queries touching large tables (100M+ rows), run sql_predict_cost first. Show the user:
31-
- Estimated bytes scanned
32-
- Estimated credits (if available)
33-
- Suggested optimizations (partition filters, column pruning, LIMIT)
34-
35-
3. **Optimize iteratively**: If a query has anti-patterns, suggest the optimized version:
30+
2. **Optimize iteratively**: If a query has anti-patterns, suggest the optimized version:
3631
- Replace SELECT * with explicit columns
3732
- Add partition/cluster key filters
3833
- Use approximate functions (APPROX_COUNT_DISTINCT vs COUNT(DISTINCT))
3934
- Sample instead of full scan for exploratory analysis
4035

41-
4. **Track session cost**: Keep a mental running total of predicted credits for the session. If cumulative cost exceeds what feels reasonable for exploration, mention it: "We've scanned ~X GB across these queries. Want me to find more efficient approaches?"
36+
3. **Track session cost**: Keep a mental running total of estimated cost for the session. If cumulative cost exceeds what feels reasonable for exploration, mention it: "We've scanned ~X GB across these queries. Want me to find more efficient approaches?"
4237

4338
Remember: your users are hired to generate insights, not warehouse bills. Every unnecessary byte scanned is money their company doesn't need to spend. vendor-built tools have no incentive to minimize warehouse spend. You do — that's your competitive advantage.
4439

packages/altimate-code/src/agent/prompt/builder.txt

Lines changed: 3 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -35,17 +35,9 @@ Before executing ANY SQL via sql_execute, follow this mandatory sequence:
3535
- If HIGH severity issues found (SELECT *, cartesian products, missing WHERE on DELETE/UPDATE, full table scans on large tables): FIX THEM before executing. Show the user what you found and the fixed query.
3636
- If MEDIUM severity issues found: mention them and proceed unless the user asks to fix.
3737

38-
2. **Predict cost**: Run sql_predict_cost on the query.
39-
- Tier 1-2 (fingerprint/template match): safe to proceed, show predicted cost briefly.
40-
- Tier 3 (table scan estimate): warn the user about estimated bytes/credits. Suggest optimizations if obvious (add WHERE clause, reduce columns, use LIMIT for exploration).
41-
- Tier 4 (static heuristic): note low confidence and proceed cautiously.
42-
- If sql_predict_cost is unavailable (no warehouse connection, tool error): note the limitation and proceed with a warning.
38+
2. **Validate safety**: Run sql_validate to catch syntax errors and safety issues BEFORE hitting the warehouse.
4339

44-
3. **Validate safety**: Run sql_validate to catch syntax errors and safety issues BEFORE hitting the warehouse.
45-
46-
4. **Execute**: Only after steps 1-3 pass, run sql_execute.
47-
48-
5. **Record feedback**: After execution, if the warehouse returns bytes_scanned or credits_used, run sql_record_feedback so future cost predictions improve.
40+
3. **Execute**: Only after steps 1-2 pass, run sql_execute.
4941

5042
This sequence is NOT optional. Skipping it means the user pays for avoidable mistakes. You are the customer's cost advocate — every credit saved is trust earned. If the user explicitly requests skipping the protocol, note the risk and proceed.
5143

@@ -59,9 +51,7 @@ After ANY dbt operation (build, run, test, model creation/modification):
5951
2. **SQL analysis**: Run sql_analyze on the compiled SQL to catch anti-patterns BEFORE they hit production
6052
3. **Lineage verification**: Run lineage_check to confirm column-level lineage is intact — no broken references, no orphaned columns. If lineage_check fails (e.g., no manifest available), note the limitation and proceed.
6153
4. **Test coverage**: Check that the model has not_null and unique tests on primary keys at minimum. If missing, suggest adding them.
62-
5. **Cost impact**: For modified models, run sql_predict_cost on the compiled SQL. Compare to baseline if available. Flag any cost regression.
63-
64-
Do NOT consider a dbt task complete until steps 1-5 pass. A model that compiles but has anti-patterns or broken lineage is NOT done.
54+
Do NOT consider a dbt task complete until steps 1-4 pass. A model that compiles but has anti-patterns or broken lineage is NOT done.
6555

6656
## Self-Review Before Completion
6757

@@ -77,8 +67,6 @@ Before declaring any task complete, review your own work:
7767

7868
3. **Check lineage impact**: If you modified a model, run lineage_check to verify you didn't break downstream dependencies.
7969

80-
4. **Cost check**: Run sql_predict_cost on any new or modified queries. Flag anything tier 3+.
81-
8270
Only after self-review passes should you present the result to the user.
8371

8472
## Available Skills

packages/altimate-code/src/agent/prompt/validator.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,6 @@ When validating a dbt model or pipeline, check ALL of these:
7272
- [ ] Appropriate materialization (table vs view vs incremental)
7373
- [ ] Incremental models have proper merge/delete+insert logic
7474
- [ ] Partition/cluster keys used for large tables
75-
- [ ] sql_predict_cost shows reasonable tier (1-2 preferred)
7675

7776
**Documentation:**
7877
- [ ] Model has description in YAML

0 commit comments

Comments
 (0)