Skip to content

Commit ce3c5fb

Browse files
[Pipelines] Document minimum roll-interval values for R2 sinks (#30883)
* [Pipelines] Document minimum roll-interval values for R2 sinks * Update src/content/docs/pipelines/sinks/available-sinks/r2-data-catalog.mdx Co-authored-by: ask-bonk[bot] <249159057+ask-bonk[bot]@users.noreply.github.com> --------- Co-authored-by: ask-bonk[bot] <249159057+ask-bonk[bot]@users.noreply.github.com>
1 parent d68df35 commit ce3c5fb

2 files changed

Lines changed: 5 additions & 3 deletions

File tree

src/content/docs/pipelines/sinks/available-sinks/r2-data-catalog.mdx

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,12 +60,14 @@ Control when data is written to Iceberg tables. Configure based on your needs:
6060

6161
### Roll interval
6262

63-
Set how often files are written (default: 300 seconds):
63+
Set how often files are written (default: 300 seconds, minimum: 60 seconds):
6464

6565
```bash
6666
--roll-interval 60 # Write files every 60 seconds
6767
```
6868

69+
The minimum interval for R2 Data Catalog sinks is 60 seconds to prevent compaction issues. Iceberg tables require periodic compaction to merge small files into larger ones for optimal query performance. Writing too often creates merge conflicts with the compaction process.
70+
6971
### Roll size
7072

7173
Set maximum file size in MB before creating a new file:

src/content/docs/pipelines/sinks/available-sinks/r2.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,10 +88,10 @@ Control when files are written to R2. Configure based on your needs:
8888

8989
### Roll interval
9090

91-
Set how often files are written (default: 300 seconds):
91+
Set how often files are written (default: 300 seconds, minimum: 10 seconds):
9292

9393
```bash
94-
--roll-interval 60 # Write files every 60 seconds
94+
--roll-interval 10 # Write files every 10 seconds
9595
```
9696

9797
### Roll size

0 commit comments

Comments
 (0)