docs: accurate PR description and code comments for sort pushdown

Update architecture comments and module docs to reflect actual behavior:
- Core value: fix wrong file order via statistics sorting → Exact upgrade
- EnforceSorting already handles correct file order cases
- PushdownSort only triggers when byte-range split hasn't happened
- Removed misleading claims about multi-partition optimization

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: zhuqi-lucas

datafusion/datasource/src/file_scan_config.rs

@@ -898,57 +898,51 @@ impl DataSource for FileScanConfig {
 
     /// Push sort requirements into file-based data sources.
     ///
-    /// # Sort Pushdown Architecture
+    /// # When does this matter?
     ///
-    /// ```text
-    /// Query: SELECT ... ORDER BY col ASC [LIMIT N]
+    /// `EnforceSorting` (which runs before `PushdownSort`) already eliminates
+    /// `SortExec` when `validated_output_ordering()` confirms the file order is
+    /// correct. However, if files are listed in wrong order (e.g., alphabetical
+    /// order doesn't match sort key order), `validated_output_ordering()` strips
+    /// the ordering and `EnforceSorting` cannot help.
     ///
-    ///   PushdownSort optimizer
-    ///         │
-    ///         ▼
-    ///   FileScanConfig::try_pushdown_sort()
-    ///         │
-    ///         ├─► FileSource::try_pushdown_sort()
-    ///         │     │
-    ///         │     ├─ natural ordering matches? ──► Exact
-    ///         │     │   (e.g. Parquet WITH ORDER)     │
-    ///         │     │                                  ▼
-    ///         │     │                      rebuild_with_source(exact=true)
-    ///         │     │                        ├─ sort files by stats within groups
-    ///         │     │                        ├─ verify non-overlapping
-    ///         │     │                        └─► keep output_ordering → SortExec removed
-    ///         │     │
-    ///         │     ├─ reversed ordering matches? ──► Inexact
-    ///         │     │   (reverse_row_groups=true)      │
-    ///         │     │                                   ▼
-    ///         │     │                       rebuild_with_source(exact=false)
-    ///         │     │                         ├─ sort files by stats
-    ///         │     │                         └─► clear output_ordering → SortExec kept
-    ///         │     │
-    ///         │     └─ neither ──► Unsupported
-    ///         │
-    ///         └─► try_sort_file_groups_by_statistics()
-    ///               (best-effort: reorder files by min/max stats)
-    ///               └─► Inexact if reordered, Unsupported if already in order
-    /// ```
+    /// This is where `PushdownSort` adds value: it **sorts files by statistics**
+    /// to fix the ordering, then re-checks — enabling sort elimination even when
+    /// files were originally in wrong order.
     ///
-    /// # Result Plans
+    /// # Architecture
     ///
     /// ```text
-    /// Exact (single partition):     DataSourceExec [files sorted, non-overlapping]
-    /// Exact (multi partition):      SPM ─► DataSourceExec [group 0] | [group 1]
-    /// Inexact (reverse scan):       SortExec ─► DataSourceExec [reverse_row_groups]
-    /// Inexact (stats reorder):      SortExec ─► DataSourceExec [files reordered]
+    /// PushdownSort optimizer finds SortExec
+    ///   │
+    ///   ▼
+    /// FileScanConfig::try_pushdown_sort()
+    ///   │
+    ///   ├─► FileSource returns Exact
+    ///   │     (natural ordering already satisfies request)
+    ///   │     → rebuild_with_source: sort files by stats, verify non-overlapping
+    ///   │     → SortExec removed, fetch (LIMIT) pushed to DataSourceExec
+    ///   │
+    ///   ├─► FileSource returns Inexact
+    ///   │     (reverse_row_groups=true)
+    ///   │     → SortExec kept, scan optimized
+    ///   │
+    ///   └─► FileSource returns Unsupported
+    ///         (ordering was stripped because files in wrong order)
+    ///         → try_sort_file_groups_by_statistics():
+    ///           1. Sort files within groups by min/max statistics
+    ///           2. Re-check: are files now non-overlapping + ordering valid?
+    ///              YES → upgrade to Exact → SortExec removed
+    ///              NO  → Inexact (files reordered but Sort stays)
     /// ```
     ///
-    /// # Trade-offs
+    /// # Note on multi-partition plans
     ///
-    /// - **Exact + single partition**: No sort, no merge, but sequential I/O only.
-    ///   Best for LIMIT queries (reads minimal data and stops).
-    /// - **Exact + multi partition**: No sort per partition, cheap O(n) SPM merge,
-    ///   parallel I/O. Best for full scans on large datasets.
-    /// - **Inexact**: Sort still required, but statistics-aware file ordering helps
-    ///   TopK discard data earlier via dynamic filters.
+    /// In the default configuration, `EnforceDistribution` byte-range splits
+    /// files into single-file groups before `PushdownSort` runs. Single-file
+    /// groups pass `validated_output_ordering()` trivially, so `EnforceSorting`
+    /// already eliminates `SortExec`. In this case, `PushdownSort` finds no
+    /// `SortExec` and does nothing.
     fn try_pushdown_sort(
         &self,
         order: &[PhysicalSortExpr],

datafusion/physical-optimizer/src/pushdown_sort.rs

@@ -37,15 +37,13 @@
 //!
 //! ## Capabilities
 //!
-//! - **Sort elimination**: when data source's natural ordering already satisfies the
-//!   request (e.g., Parquet files with matching `WITH ORDER`), return `Exact` and
-//!   remove the `SortExec` entirely
+//! - **Statistics-based file sorting + sort elimination**: when files within a
+//!   partition are non-overlapping and internally sorted but listed in wrong order,
+//!   sort them by min/max statistics to fix the ordering. After sorting, the
+//!   ordering becomes valid and `SortExec` can be removed entirely. Also preserves
+//!   `fetch` (LIMIT) from the eliminated `SortExec` for early termination.
 //! - **Reverse scan optimization**: when required sort is the reverse of the data source's
 //!   natural ordering, enable reverse scanning (reading row groups in reverse order)
-//! - **Statistics-based file reordering**: sort files within each group by their min/max
-//!   statistics to approximate the requested order, improving TopK and limit performance
-//! - **Non-overlapping detection**: when files have non-overlapping ranges and matching
-//!   within-file ordering, the combined scan is `Exact` (sort eliminated)
 //! - **Prefix matching**: if data has ordering [A DESC, B ASC] and query needs
 //!   [A DESC], the existing ordering satisfies the requirement (`Exact`).
 //!   If the query needs [A ASC] (reverse of the prefix), a reverse scan is