App MVP

[laurencewells]

Changes

Linked issues

This pull request introduces major improvements to the DQX App's documentation and deployment configuration, focusing on clarifying the architecture, authentication model, and deployment steps, as well as enhancing the Databricks Asset Bundle (databricks.yml) to provision all required resources automatically. The changes also update development dependencies and provide a comprehensive product overview in a new CLAUDE.md file.

Key changes include:

1. Documentation and Product Context

• Added CLAUDE.md with a detailed product overview, architecture diagram, personas, user journeys, and internal storage design for the DQX App. This helps new contributors and users understand the app's purpose, scope, and design decisions.
• Expanded and reorganized README.md to clarify the authentication model (OBO vs. SP), async job pattern for Spark operations, deployment steps, troubleshooting, and removed outdated configuration steps. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]

2. Deployment Automation and Resource Provisioning

• Enhanced databricks.yml to:
  • Define variables for catalog and schema names, and centralize app environment configuration.
  • Automatically provision required resources: SQL warehouse, Delta schema, and Databricks Job for async Spark tasks (profiler and dry-run), and wire their IDs into the app config. [1] [2]
  • Grant necessary permissions and set up API scopes for workspace and catalog access.

3. Dependency Updates

• Replaced databricks-connect with databricks-sql-connector[pyarrow] in pyproject.toml, reflecting the new architecture where Spark operations are offloaded to a Databricks Job rather than running in-process.

4. Development and Troubleshooting

• Updated documentation to cover the new async job pattern, environment variables, and troubleshooting steps for profiler/dry-run operations. [1] [2]

These changes make the app easier to deploy, maintain, and understand, and set a strong foundation for future development.

---

References:
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]

Resolves #..

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[github-actions]

✅ 722/722 passed, 1 flaky, 39 skipped, 5h20m41s total

Flaky tests:

• 🤪 test_e2e_workflow_for_patterns_exclude_patterns (4m13.413s)

_{Running from acceptance #4537}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.11%. Comparing base (c83bbe7) to head (c0f094b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1090      +/-   ##
==========================================
+ Coverage   91.86%   92.11%   +0.25%     
==========================================
  Files         101      101              
  Lines        9546     9546              
==========================================
+ Hits         8769     8793      +24     
+ Misses        777      753      -24     

Flag	Coverage Δ
anomaly	53.54% <ø> (+30.68%)	⬆️
anomaly-serverless	53.55% <ø> (+30.63%)	⬆️
integration	50.36% <ø> (+0.84%)	⬆️
integration-serverless	50.61% <ø> (+0.24%)	⬆️
unit	55.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[laurencewells]

Few issues to sort outstanding
The App needs to give user Create table permissions to the temp schema
One more scope niggle in the LLM Generate rules for live deployment

[mwojtyczka]

LGTM