Skip to content

Commit 7cecc18

Browse files
authored
doc: upgrade guide for new compression option for spill files (#16472)
* doc: upgrade guide for new compression option for spill files * fix CI
1 parent a4f4b17 commit 7cecc18

1 file changed

Lines changed: 31 additions & 0 deletions

File tree

docs/source/library-user-guide/upgrading.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,37 @@ See [#16317] for details.
6565

6666
[#16317]: https://github.com/apache/datafusion/pull/16317
6767

68+
### New `datafusion.execution.spill_compression` configuration option
69+
70+
DataFusion 49.0.0 adds support for compressing spill files when data is written to disk during spilling query execution. A new configuration option `datafusion.execution.spill_compression` controls the compression codec used.
71+
72+
**Configuration:**
73+
74+
- **Key**: `datafusion.execution.spill_compression`
75+
- **Default**: `uncompressed`
76+
- **Valid values**: `uncompressed`, `lz4_frame`, `zstd`
77+
78+
**Usage:**
79+
80+
```rust
81+
# /* comment to avoid running
82+
use datafusion::prelude::*;
83+
use datafusion_common::config::SpillCompression;
84+
85+
let config = SessionConfig::default()
86+
.with_spill_compression(SpillCompression::Zstd);
87+
let ctx = SessionContext::new_with_config(config);
88+
# */
89+
```
90+
91+
Or via SQL:
92+
93+
```sql
94+
SET datafusion.execution.spill_compression = 'zstd';
95+
```
96+
97+
For more details about this configuration option, including performance trade-offs between different compression codecs, see the [Configuration Settings](../user-guide/configs.md) documentation.
98+
6899
## DataFusion `48.0.0`
69100

70101
### `Expr::Literal` has optional metadata

0 commit comments

Comments
 (0)