Add Lakemeter OSS-safe cost estimation tool

[CheeYuTan]

Summary

Adds Lakemeter, a full-stack Databricks App for workload sizing, cost estimation, and demand planning.

Lakemeter v1 provides an AI-assisted sizing calculator for most Databricks workloads. Lakemeter v2 adds demand plan views and scenario tiers to help teams compare adoption paths, budget ranges, and workload growth scenarios.

This supersedes the earlier closed Lakemeter submission in #698 with the updated OSS-safe codebase.

Why OSS

Lakemeter addresses a recurring field problem: sizing and cost-planning conversations are often bespoke, hard to standardize, and difficult to share externally across workloads.

Open sourcing Lakemeter gives customers, partners, and field teams a transparent, self-serve tool for Databricks cost modeling, reduces one-off sizing work, and creates a cleaner path for customer feedback and community contribution.

OSS Safety Work Completed

• Removed Databricks-internal benchmark-derived sizing logic, formulas, and prompt content.
• Replaced sensitive sizing assumptions with user-provided inputs and public pricing guidance.
• Updated tests to reflect the OSS-safe input model.
• Redeployed and tested the app after redaction.
• Validated that the assistant no longer exposes internal benchmark methodology.

What's Included

• FastAPI backend
• React/TypeScript frontend
• Databricks Apps deployment config
• AI-assisted workload configuration
• Excel export
• Pricing/reference data handling
• Tests and documentation

Test Plan

• Databricks pre-commit and pre-push secret scans passed
• Python compile check passed for backend/app and tests/ai_assistant/test_accept_all_types.py
• Searched for removed internal benchmark surfaces; remaining matches are prompt guardrails/public CU-hour pricing references
• Deployed and tested sanitized app before submission
• Targeted pytest command attempted, but local sandbox Python does not have pytest installed

This pull request was AI-assisted by Cursor.

[CLAassistant]

All committers have signed the CLA.