Skip to content

feat(query): Add stream backlog admin API#20104

Open
zhyass wants to merge 3 commits into
databendlabs:mainfrom
zhyass:add_option
Open

feat(query): Add stream backlog admin API#20104
zhyass wants to merge 3 commits into
databendlabs:mainfrom
zhyass:add_option

Conversation

@zhyass

@zhyass zhyass commented Jul 3, 2026

Copy link
Copy Markdown
Member

fix

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Adds a stream backlog admin API for estimating pending stream changes from metadata.

API

  • GET /v1/stream_backlog?stream_name=&database=
  • GET /v1/tenants/:tenant/stream_backlog?stream_name=&database=

stream_name is required. database is optional and defaults to default.

Result
The API returns:

{
    "rows_added": 3,
    "rows_removed": 0,
    "estimated_rows": 3,
    "estimated_bytes": 120
}
  • rows_added: metadata-derived rows from latest-side changes.
  • rows_removed: metadata-derived rows from base-side changes. Append-only streams report 0.
  • estimated_rows: estimated rows the stream may need to process.
  • estimated_bytes: estimated uncompressed bytes for estimated_rows.

The values are metadata-derived estimates, not exact stream output row counts.

Changes

  • Computes backlog from Fuse snapshot and block metadata.
  • Handles append-only streams, standard update/delete/truncate cases, and compact/recluster-style physical rewrites.
  • Extends management tests to cover API output and result fields across append, update, delete, truncate, and compaction behavior.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

fix
@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Jul 3, 2026
@zhyass

zhyass commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 11644d5619

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant