ci(ud): expand UD compatibility workflow scope + fix bugs + reproducible UD build

[sfc-gh-fpawlowski]

Incremental improvements on top of ud-ci-workflow-auto-triggered-full (base branch, so the diff is scoped to just these changes).

What changes

• Scope — matrix groups unit-integ / scala / modin / datasource / doctest, each run against UD. At-risk groups use continue-on-error so every gap is visible in one run. (unit-integ already covers UDF via the or udf marker.)
• Bug fixes — correct doctest tox env name (py313, not py3.13); split scala by path (tests/integ/scala) instead of the -k scala name filter that dropped scala-dir tests from both groups.
• Reproducible UD build — build-ud builds the UD wheel once from a pinned ref (hatch build), caches by resolved commit, uploads as an artifact; test legs install it via the existing ud_connector_path swap (deps first, UD force-reinstalled last so it wins regardless of connector version range) — no per-leg Rust rebuild.
• Triggers / artifacts — workflow_dispatch + nightly schedule + concurrency; pull_request gated to the UD integration branches; upload JUnit XML + a failing-test-name list per group.
• Honest counts for odbc-reports — disable reruns via UD_RERUN_FLAGS, keep the pytest terminal summary line in the log (odbc-reports sums those across group logs). tox.ini gains backward-compatible UD_RERUN_FLAGS and JUNIT_REPORT_DIR hooks (no-ops when unset).

Deferred per discussion

Differential legacy-vs-UD baseline, parsing JUnit into our own summary, known-divergence xfail list, and switching odbc-reports to consume artifacts.

To verify on first dispatch run

• build-ud: hatch build in universal-driver/python succeeds with rustup-stable + apt protobuf-compiler (may need a pinned/newer protoc).
• UD_REF still defaults to the floating snowpark-compatibility branch — pin a SHA for cross-run reproducibility.
• modin group: empty RERUN_FLAGS doesn't break run_snowparkpandasnotdoctest.sh arg handling.
• datasource carved out of unit-integ; confirm nothing else in tests/integ imports DB drivers at collection.

Follow-up

Once green enough, categorize the most common UD-vs-legacy failure causes from the uploaded results (missing connector private symbols like _describe_internal, Arrow/fetch_pandas mismatches, PUT/GET, etc.) as the backlog for fixing UD or Snowpark.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.29%. Comparing base (32defa2) to head (b8b2a87).

Additional details and impacted files

@@                          Coverage Diff                           @@
##           ud-ci-workflow-auto-triggered-full    #4275      +/-   ##
======================================================================
- Coverage                               95.52%   94.29%   -1.23%     
======================================================================
  Files                                     171      171              
  Lines                                   44299    44299              
  Branches                                 7564     7564              
======================================================================
- Hits                                    42316    41772     -544     
- Misses                                   1221     1665     +444     
- Partials                                  762      862     +100     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.