feat: introduce pluggable SpillFile trait and TempFileFactory for custom spill backends (apache#21882)

## Which issue does this PR close?

- Closes apache#21215, depends on apache#22230

## Rationale for this change

DataFusion’s spill infrastructure is tightly coupled to OS-level files,
with no extension points for alternative storage backends. `DiskManager`
cannot be customized for file creation, and `IPCStreamWriter` depends on
OS file paths.
This prevents integration in environments where temporary storage must
be managed by the host system. For example, Postgres extensions (e.g.,
ParadeDB) require spill files to go through `BufFile` APIs to respect
`temp_tablespaces`, enforce `temp_file_limit`, and integrate with
transaction-scoped cleanup. Since `BufFile` has no OS-visible path, it
cannot work with the current design.
A secondary motivation raised by @alamb is supporting object storage
backends (S3, GCS) for spilling, which require async IO and cannot use
`std::io::Write` or `std::io::Read`.

## What changes are included in this PR?

- Introduced `SpillFile`, `SpillWriter`, and `TempFileFactory` traits to
abstract spill file handling
- Added `DiskManagerMode::Custom` to allow pluggable backends
- Updated `DiskManager` to return `Arc<dyn SpillFile>` instead of
OS-bound types
- Refactored write path using `SpillWriteAdapter` to bridge sync Arrow
writers with backend-agnostic writers
- Refactored read path to use async streaming (`Stream<Item =
Result<Bytes>>`) instead of blocking state machines
- Updated spill-related components to operate on `Arc<dyn SpillFile>`
- Migrated the Sort-Merge Join (SMJ) operator to use the async spill
abstraction

## Are these changes tested?

Yes. Existing spill tests cover the full read/write flow.

- Fixed `test_disk_usage_decreases_as_files_consumed` by correcting a
pre-existing off-by-one assumption in file rotation
- Fixed `test_preserve_order_with_spilling` by just asserting spilling
occurs (`spill_count>0`) and output batches are sorted

## Are there any user-facing changes?

Yes this introduces API changes:

- Spill-related APIs now use `Arc<dyn SpillFile>` instead of
`RefCountedTempFile`
- New public traits: `SpillFile`, `SpillWriter`, `TempFileFactory`
- Added `DiskManagerMode::Custom` for custom backends

Custom spill backends can now be implemented and plugged in via
`DiskManager`.

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Author: pantShrey

Cargo.lock

@@ -55,6 +55,7 @@ sql = []
 arrow = { workspace = true }
 arrow-buffer = { workspace = true }
 async-trait = { workspace = true }
+bytes = { workspace = true }
 dashmap = { workspace = true }
 datafusion-common = { workspace = true, default-features = false }
 datafusion-expr = { workspace = true, default-features = false }
@@ -66,7 +67,11 @@ parking_lot = { workspace = true }
 parquet = { workspace = true, optional = true }
 rand = { workspace = true }
 tempfile = { workspace = true }
+tokio = { workspace = true }
+tokio-util = { workspace = true, features = ["io"] }
 url = { workspace = true }
+[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
+tokio = { workspace = true, features = ["fs"] }
 
 [dev-dependencies]
 chrono = { workspace = true }