Skip to content

docs(datafusion): add user documentation for DataFusion integration#2029

Closed
xbattlax wants to merge 5 commits intoapache:mainfrom
xbattlax:docs_datafusion_integration
Closed

docs(datafusion): add user documentation for DataFusion integration#2029
xbattlax wants to merge 5 commits intoapache:mainfrom
xbattlax:docs_datafusion_integration

Conversation

@xbattlax
Copy link
Copy Markdown
Contributor

@xbattlax xbattlax commented Jan 15, 2026

Summary

Add comprehensive user documentation for the DataFusion integration, addressing issue #2027.

Changes

  • Add website/src/datafusion.md documentation page covering:

    • Catalog-based access with IcebergCatalogProvider
    • SQL operations: CREATE TABLE, INSERT INTO, SELECT
    • Metadata tables ($snapshots, $manifests)
    • External tables via IcebergTableProviderFactory
    • Partitioned tables and write modes (fanout vs clustered)
    • Query optimization (projection, filter, LIMIT pushdown)
    • Configuration options and limitations

  • Add crates/examples/src/datafusion_integration.rs working example with annotated code sections for documentation includes

  • Update website/src/SUMMARY.md to include new documentation page

  • Add required dependencies to crates/examples/Cargo.toml

Notes

This addresses the documentation request from #2027, which noted substantial progress in DataFusion integration (per epic #1382). The documentation covers all major features merged including:

Closes #2027

@xbattlax xbattlax mentioned this pull request Jan 15, 2026
Open
@xbattlax
docs(datafusion): add user documentation for DataFusion integration
4830a74 
Add comprehensive user documentation for the DataFusion integration
that covers SQL-based table operations, catalog integration, and
query optimization features.

Changes:
- Add datafusion.md documentation page with setup, SQL operations,
  metadata tables, partitioned tables, and configuration options
- Add datafusion_integration.rs example with annotated code sections
- Update SUMMARY.md to include new documentation page
- Add required dependencies to examples crate

Closes apache#2027
@xbattlax xbattlax force-pushed the docs_datafusion_integration branch from 853219a to 4830a74 Compare January 15, 2026 08:31
liurenjie1024
liurenjie1024 reviewed Jan 16, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xbattlax for this pr, just finished first round of review. cc @CTTY to also take a look.

Comment thread crates/examples/src/datafusion_integration.rs Outdated
Comment thread crates/examples/src/datafusion_integration.rs Outdated
Comment thread website/src/datafusion.md Outdated
Comment thread website/src/datafusion.md Outdated
Comment thread website/src/datafusion.md Outdated
@xbattlax
Address PR review feedback
94a836d 
- Change catalog name from "iceberg" to "my_catalog" to clarify it has no special meaning
- Move external table code to separate example file
- Remove sections not suited for end-users: External Tables, Table Provider Types, Creating Partitioned Tables (Rust API), Query Optimization
- Add clarification that table properties must be set via Iceberg catalog API
@xbattlax
Copy link
Copy Markdown
Contributor Author

xbattlax commented Jan 16, 2026 •
edited
Loading

Thanks @xbattlax for this pr, just finished first round of review. cc @CTTY to also take a look.

Thank you two for the review! First round done :)

@xbattlax xbattlax force-pushed the docs_datafusion_integration branch 3 times, most recently from 68f9f95 to 6cb8fbd Compare January 16, 2026 10:25
@xbattlax xbattlax force-pushed the docs_datafusion_integration branch from 6cb8fbd to 7f6bfda Compare January 16, 2026 14:59
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 28, 2026

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Feb 28, 2026
@xbattlax
Copy link
Copy Markdown
Contributor Author

xbattlax commented Mar 3, 2026

@liurenjie1024 have you some other points on this PR ?

@github-actions github-actions Bot removed the stale label Mar 4, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 3, 2026

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Apr 3, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 10, 2026

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Apr 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CTTY CTTY CTTY left review comments

+1 more reviewer

@liurenjie1024 liurenjie1024 liurenjie1024 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Update user doc for datafusion integration.

3 participants

@xbattlax @liurenjie1024 @CTTY