Fork test: PR #1129 - feat(anomaly): add LLM-generated AI explanations for has_no_row_anomalies#1238
Closed
mwojtyczka wants to merge 41 commits into
Closed
Fork test: PR #1129 - feat(anomaly): add LLM-generated AI explanations for has_no_row_anomalies#1238mwojtyczka wants to merge 41 commits into
mwojtyczka wants to merge 41 commits into
Conversation
…_anomaly function
…s/dqx into ai-explanations-clean
…+ fix no-cheat CI Pivot the has_no_row_anomalies AI explanations to a single ai_query (Spark SQL) execution path and remove the DSPy/driver executor, per review direction. - Delete the DSPy/driver path (build_language_model, _explain_one_group, _call_llm_for_groups, _add_explanation_column_driver, dspy signature) and the LLMModelConfig.executor field; AI explanations need no extra dependency. - UUID-suffix the (segment, pattern) working column via ScoringOutputColumns so it can never collide with a user column (threaded through ExplanationContext). - Rename the ai_explanation struct field pattern -> top_features. - Redact segment values listed in redact_columns; escape backslash + quote in the redact SQL literal; build ai_query responseFormat and confidence thresholds from single sources of truth; warn when max_groups < eligible segments. - Drop model_name from the prompt and add anti-pattern guidance + two few-shot exemplars. - Fix the failing no-cheat / no-pylint-disable CI the right way: remove the redundant per-line disables and add protected-access to pylint per-file-ignores (no CHEAT bypass). import-outside-toplevel disappears with the driver path. - Replace driver/DSPy tests with ai_query unit + integration tests, including a prompt-quality guard (dominant feature referenced + length caps). Signed SHAP display and config/prompt knobs are intentionally deferred (see PR). Co-authored-by: Isaac
mwojtyczka
added
do-not-merge
…uery responseFormat, demo label - LLMModelConfig.__post_init__ now validates max_tokens (positive int) and temperature (non-negative) alongside max_retries, so a bad value raises a clean InvalidParameterError instead of a raw TypeError at ai_query SQL-build time (and no non-positive max_tokens / out-of-range temperature reaches the endpoint). - Escape _AI_QUERY_RESPONSE_FORMAT via _sql_string_literal when interpolating it into the ai_query SQL — a no-op today (the JSON carries no quotes) but hardens against a future _PROMPT_OUTPUT_FIELDS entry that contains a single quote. - Rename the demo's "Section 5" -> "Section 5a" now that "Section 5b: AI explanations" was added. Co-authored-by: Isaac
… test The rendered prompt is the contract the model sees; an edit to any _PROMPT_* table silently changes every explanation. Add a fast unit test (no LLM, no workspace, ms) asserting _render_ai_query_prompt_header() matches a committed golden snapshot (tests/resources/ai_query_prompt_header.txt), so prompt changes must be deliberate and surface in review as a diff. Full output-quality evaluation stays with #1030. Co-authored-by: Isaac
…document pattern collisions - _sanitize now trims an overrun LLM output field back to a word boundary (drops the trailing partial word) instead of cutting mid-word, when it exceeds _LLM_FIELD_MAX_LEN. - Document the (segment, top-2) pattern-collision trade-off in the Row Anomaly guide: groups sharing the same top-2 contributors merge into one explanation — intentional, keeps the LLM-call count (and cost) low at the price of a less specific narrative. Co-authored-by: Isaac
…ful degradation Make enable_contributions and enable_ai_explanation default to True so row anomaly checks produce SHAP contributions and plain-language AI explanations out of the box, reducing setup friction. - check_funcs / scoring_config: flip both defaults to True - rename public param llm_model_config -> ai_explanation_llm_model_config - degrade instead of erroring: when enable_contributions is False, AI explanations are disabled with a warning (not an exception) - graceful endpoint guard: probe the Model Serving endpoint once before issuing ai_query; if unreachable, skip explanations with a warning and let scoring complete (no run failure when the endpoint/Foundation Model APIs are unavailable) - document the ai_query requirement (serverless or DBR 15.4 LTS+) in the guide, reference, and demo Section 5b - tests: unit coverage for defaults + downgrade behaviour; integration coverage for default-on, graceful degradation, and contributions-off - conftest: opt the broad anomaly suite out of SHAP/LLM cost by default Co-authored-by: Isaac
…partition warning The per-pattern top-N selection used an unpartitioned Window (global totals + row_number) and returned that windowed DataFrame as the group projection that ai_query consumes downstream. Because callers re-action the scored DataFrame (e.g. the demo counts anomalies at several thresholds), Spark re-analysed the window on every action and re-emitted "No Partition Defined for Window operation" outside the local suppression block — noisy for users, though harmless (the window only ran over the small per-pattern aggregate). Collect the top-N keys + totals once (under the existing suppression), then rebuild the kept projection by filtering the per-pattern aggregate to those known keys with isin(...). Equivalent selection, no window in the returned lineage, so no warning on downstream actions. Co-authored-by: Isaac
…un, not per action The per-group ai_query results were left-joined lazily onto the scored DataFrame, so every downstream action (count, display, write) re-executed the lineage and re-invoked ai_query for every group — multiplying LLM cost and latency per action and contradicting the documented "one call per group per scoring run" cost model. A notebook that counts anomalies at several thresholds paid a full LLM pass per count. Materialise the per-group results once (collect + rebuild as a local DataFrame) before the join. The payload is small and bounded: at most max_groups rows, each with three length-capped text fields. Co-authored-by: Isaac
…atch baseline stats Three scoring/training-path performance improvements (no feature changes): 1. SHAP only for anomalous rows. Contributions were computed for every row inside the scoring UDFs even though they are only surfaced for rows at or above the threshold. The UDFs now recompute severity from the model's quantile points in numpy (epsilon-over-inclusive) and run TreeSHAP only on that subset; add_info_column gates the struct field so the observable contract is exact — anomalous rows always carry contributions, other rows a null map. SHAP cost now scales with the anomaly count, not table size. 2. No more scoring self-join. Scorers joined the scored frame (derived from the input) back onto the input on a monotonically_increasing_id — double computation, a full shuffle, and a join keyed on a non-deterministic id. The original row now rides through feature engineering inside a collision-proof struct column and is restored with a select after scoring. 3. Baseline drift statistics in two actions. compute_baseline_statistics ran two Spark actions (full scans) per column; now one combined aggregate select plus one multi-column approxQuantile regardless of column count. Docs updated for the contributions-only-for-anomalous-rows contract (and two stale "disabled by default" lines from the default-on change); threshold exploration guidance added — severity is still computed for every row, score at a lower threshold if sub-threshold contributions are needed. Co-authored-by: Isaac
…or tests that don't use it
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated sync from fork PR for CI testing.
Original PR: #1129
All tests, including unit and integration tests run on this PR (they are skipped for fork PRs).