docs(delta): session-5b — A.1 upstream apache#4700, A.2 carved + reviewed clean (fork #4)

Author: schenksj

DELTA_PR_SPLIT_PLAN.md

@@ -220,6 +220,10 @@ Two file classes:
 **A.5 — CDF**
 - `CometDeltaCdfScanExec.scala`, the `CometExecRule` CDF hook hunk (deferred from A.2),
   CDF wiring in the contrib (the Rust `TableChanges` path already landed in A.3x).
+- **Also deferred from A.2 (review round 2):** `DeltaIntegration`'s CDF members
+  (`isCdfRelation`, `transformCdf`, `convertCdfBinding` + its cache, and the
+  `RowDataSourceScanExec` import) — A.2 stripped them as dead code (no caller until the CDF
+  hook lands). Re-add them here alongside the CometExecRule CDF arm.
 - Tests: `CometDeltaCdcSuite`, `CometDeltaCdfReflectionReproSuite`.
 
 **A.6a — Test battery** *(gated on all extraction PRs + %-path fix being merged)*
@@ -544,6 +548,31 @@ Append-only. Newest entry at the top. Entry template:
 - Next action:
 ```
 
+### 2026-06-20 (session 5b — Opus 4.8) — A.1 opened upstream (#4700); A.2 carved + reviewed clean
+- **A.1:** opened upstream as **#4700** (with AI-disclosure footer; new standing rule: every PR body
+  ends with it). Apache CI is `action_required` (awaiting a maintainer to approve workflow runs for a
+  non-committer PR — NOT a failure). Fork review-draft #3 closed as superseded.
+- **Loose end fixed:** fork PR #2 (monolith, `feat/delta-kernel-read`) was stranded on the pre-main-sync
+  commit `f442361c3` (I'd pushed the merge to `contrib-delta-direct`/#4366 but not `feat/delta-kernel-read`).
+  Pushed `9f29732a8` (FF) → #2 MERGEABLE again. Its stale `delta-contrib/*` CI failures were from the old
+  commit; A.1/#4700 is unaffected (zero Delta surface).
+- **A.2 carved** onto `pr/delta-A1-spi` → branch `pr/delta-A2-buildgate` @ **`f2ad00c29`**, fork review
+  draft **#4** (base = A.1 branch, A.2-only diff). 20 files +4020/-14. An Explore agent produced the
+  precise carve map (whole-checkout vs partial-carve vs stub vs copy); see `.delta-split/review-log-A2.md`.
+  Proto `delta_scan = 118`. Stub contrib crate authored (real impl deferred). pom.xml carries
+  `delta.version` only (NO 4.1.1 pin, §10.1). CometExecRule = marker hook only (CDF → A.5).
+- **§5 (all green):** default + gated native build, clippy ×2 feature states, gate-verify script (all
+  layers), gated + default JVM compile, spotless/scalastyle (both profiles), cargo fmt.
+- **Review round (manual + /code-review high, 28 agents, 9 refuted / 6 survived) — 5 fixed, 2 accepted:**
+  fixed a VACUOUS cargo-tree gate check (no anti-vacuous guard → false pass on cargo failure), a
+  pipefail+`grep -q` SIGPIPE misfire in the gate script (→ here-strings), dead CDF block in
+  DeltaIntegration (→ stripped to A.5, see A.5 note), scanHandler per-call MODULE$ re-resolve
+  (→ @volatile cache), raw Class.forName (→ ClassLoaders.loadClass). Accepted+documented: a Spark-3.4
+  diagnostic-only fallback-reason string change (rare metadata-col + AQE overlap, matches monolith) and
+  DeltaConf's unread-in-A.2 config entries (the contrib's config leaf).
+- **Next action:** carve **A.3a** (Rust driver side: error/engine/predicate/scan/jni) onto
+  `pr/delta-A2-buildgate`, OR wait for #4700 review. Open A.2 upstream once A.1 is approved/merged (§8).
+
 ### 2026-06-20 (session 5 — Opus 4.8) — maintainers OK'd the plan; #4366 re-synced; A.1 refreshed onto current main + review round 2 clean
 - **Phase/unit:** A.1 (Core SPI) refreshed onto post-#4535 main, re-reviewed clean. Maintainers
   comfortable with the split plan — execution greenlit. A.1 still fork-local (PR #3); upstream open

review-log-A2.md

@@ -0,0 +1,51 @@
+# Unit A.2 — Build gate + inert wiring — review log
+
+Branch: `pr/delta-A2-buildgate` (based on `pr/delta-A1-spi`).
+Commit: `f2ad00c29` (single commit; review fixes amended in).
+
+## Carve (from `feat/delta-kernel-read` @ 9f29732a8 against A.1 base)
+20 files, +4020/-14. Wholesale-A.2 checkouts (spark/pom.xml, native Cargo.toml ×2,
+operator.proto [delta_scan=118], planner.rs, operator_registry.rs, jni_api.rs, CometScanRule.scala);
+whole-file copies (delta_scan.rs shim, DeltaIntegration.scala, DeltaConf.scala, gate script);
+authored STUB contrib crate (contrib/delta/native/{Cargo.toml,lib.rs,planner.rs} -> plan_delta_scan
+returns DataFusionError::NotImplemented); partial carves (pom.xml = delta.version only, NOT the
+4.1.1 pin; CometExecRule = marker hook only, CDF hook -> A.5); authored minimal delta_build_gate.yml.
+
+## §5 verification (all green)
+default + gated native build; clippy both feature states; gate-verify script (all layers);
+gated + default JVM test-compile; spotless + scalastyle (both profiles); cargo fmt (native + contrib).
+
+## Review round (manual /review-comet-pr + /code-review high, 28-agent workflow: 9 refuted, 6 survived)
+
+**Found+fixed (folded into the commit):**
+- **[gate script] cargo-tree check was vacuous** (CONFIRMED): a failing `cargo tree` -> empty output
+  -> leak grep finds nothing -> false PASS (the Maven gate had an anti-vacuous guard, the cargo one
+  didn't). Added a `datafusion-comet ` anchor assertion. Unit-tested: empty input now fails, real
+  tree passes.
+- **[gate script] pipefail + `grep -q` SIGPIPE** (found during verification): `echo "$BIG" | grep -q`
+  under `set -o pipefail` returns 141 (grep exits early -> echo SIGPIPE), misfiring the `if !` guard
+  on the 1842-line effective-pom. Converted the three guards to here-strings and replaced
+  `head -1` (same early-close hazard) with `sed -n '1s//p'`.
+- **[DeltaIntegration] dead CDF block** (CONFIRMED): `isCdfRelation`/`transformCdf`/`convertCdfBinding`
+  (~65 lines) had no caller in A.2 (the CDF hook is A.5). Stripped to A.5 (plan §4 A.5 updated).
+- **[DeltaIntegration] scanHandler re-resolved MODULE$ per call** (CONFIRMED): violated the repo's
+  cache-reflection convention. Added a @volatile `scanHandlerCache` (memoised once per JVM).
+- **[DeltaIntegration] raw Class.forName** (PLAUSIBLE): swapped to `ClassLoaders.loadClass`
+  (thread-context-classloader-first), matching IcebergReflection's convention.
+
+**Accepted / documented (not fixed — match the green monolith, default behavior unchanged):**
+- **metadata-col fallback REASON string on Spark<3.5** (CONFIRMED, diagnostic-only): a V1 scan with
+  BOTH a metadata column AND an AQE DPP filter now reports "AQE Dynamic Partition Pruning requires
+  Spark 3.5+" instead of "Metadata column is not supported". Both still fall back to Spark (no
+  behavior change); the string differs only in this rare Spark-3.4 overlap. Inherent to letting Delta
+  claim metadata-col scans (the reorder is required); the metadata-col guard is re-applied inside
+  transformV1Scan so non-overlap cases are identical. Matches the monolith.
+- **DeltaConf's 4 entries are unread in A.2** (PLAUSIBLE): they're the contrib's config surface,
+  consumed by later units. DeltaConf is the leaf the plan deliberately ships so the contrib
+  add-source dir compiles; trimming would fragment the config object or need an artificial placeholder.
+
+**Refuted (9):** style dups (parameterize the two near-identical reflection caches), `isAvailable`
+no-caller (public util), `isDeltaScanMarker` getClass.getName per node (inherent to marker matching),
+compiled-class gate "vacuous on compile failure" (the find-based check is fine), et al.
+
+**State:** A.2 review clean. Ready to open upstream once A.1 (#4700) is approved/merged (per §8).