Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
0c84aaf
CI: Disable caching (#2268)
Fokko Aug 3, 2025
ccaa15c
Bump Poetry to 2.1.3 (#2267)
Fokko Aug 4, 2025
14ee8da
Avro: Fix tests (#2265)
Fokko Aug 4, 2025
3e391a7
Clear updates/requirements after commit (#1961)
Fokko Aug 4, 2025
632d783
Adding support for Google AuthManager (#2072)
rambleraptor Aug 4, 2025
b5e059e
Build: Bump deptry from 0.23.0 to 0.23.1 (#2273)
dependabot[bot] Aug 5, 2025
55c02e4
Build: Bump mypy-boto3-glue from 1.39.12 to 1.40.0 (#2274)
dependabot[bot] Aug 5, 2025
6d9f2e8
Build: Bump coverage from 7.10.1 to 7.10.2 (#2276)
dependabot[bot] Aug 5, 2025
00ef3ce
Build: Bump fastavro from 1.11.1 to 1.12.0 (#2277)
dependabot[bot] Aug 5, 2025
ef2010f
Build: Bump docutils from 0.21.2 to 0.22 (#2278)
dependabot[bot] Aug 5, 2025
38b05a2
Build: Bump polars from 1.31.0 to 1.32.0 (#2279)
dependabot[bot] Aug 5, 2025
d8ee913
Build: Bump daft from 0.5.14 to 0.5.16 (#2281)
dependabot[bot] Aug 5, 2025
5e9b3d3
Build: Bump mkdocstrings from 0.29.1 to 0.30.0 (#2282)
dependabot[bot] Aug 5, 2025
5c604c2
Add `__eq__` and `__hash__` methods to `ManifestFile` (#2233)
smaheshwar-pltr Aug 5, 2025
cd7d8c7
Fix projected fields predicate evaluation (#2029)
Erigara Aug 5, 2025
2a77513
Build: Bump mypy-boto3-dynamodb from 1.39.0 to 1.40.0 (#2283)
dependabot[bot] Aug 5, 2025
e97e275
Build: Bump pypa/cibuildwheel from 3.0.1 to 3.1.3 (#2285)
dependabot[bot] Aug 5, 2025
7b0c856
Build: Bump griffe from 1.7.3 to 1.9.0 (#2280)
dependabot[bot] Aug 5, 2025
aa8c6af
Remove unused `AddFileTask` (#2284)
Fokko Aug 5, 2025
d1c6005
Build: Bump mkdocs-material from 9.6.15 to 9.6.16 (#2286)
dependabot[bot] Aug 5, 2025
f21e2a1
Build: Bump pyiceberg-core from 0.5.1 to 0.6.0 (#2275)
dependabot[bot] Aug 6, 2025
2a11b25
Bump Iceberg in CI (#2288)
Fokko Aug 6, 2025
a32c3dc
Use `rest` shorthand in `conftest.py` (#2289)
Fokko Aug 6, 2025
45f66da
Add test for migrated tables (#2290)
Fokko Aug 6, 2025
b6a45ed
fix: sanitize invalid Avro field names in manifest file (#2245)
kris-gaudel Aug 6, 2025
8042d82
Convert `_get_column_projection_values` to use Field-IDs (#2293)
Fokko Aug 7, 2025
00222a4
Pass in type explicitly for `initial-default` (#2296)
Fokko Aug 7, 2025
b67ef2e
Bump Poetry to 2.1.4 (#2297)
Fokko Aug 7, 2025
5902602
Enable add tests migrated Hive tables (#2295)
Fokko Aug 7, 2025
b435c1b
Minor cleanup (#2298)
Fokko Aug 7, 2025
dab2536
Update `__repr__` for `initial-default` and `write-default` (#2287)
rambleraptor Aug 7, 2025
24b12dd
Support reading nanoseconds from PyArrow (#2294)
rambleraptor Aug 7, 2025
0d463b9
added support for ADLS_CREDENTIAL (#2299)
tbirch-cyber Aug 10, 2025
8052652
Add missing Arrow AzureFileSystem parameters (#2301)
kevinjqliu Aug 11, 2025
7a8134e
Build: Bump polars from 1.32.0 to 1.32.2 (#2320)
dependabot[bot] Aug 12, 2025
aa48b69
Build: Bump pre-commit from 4.2.0 to 4.3.0 (#2319)
dependabot[bot] Aug 12, 2025
29f71fb
Build: Bump rich from 14.0.0 to 14.1.0 (#2318)
dependabot[bot] Aug 12, 2025
979babb
Build: Bump moto from 5.1.8 to 5.1.10 (#2317)
dependabot[bot] Aug 12, 2025
c9cd95d
Build: Bump boto3 from 1.38.46 to 1.39.11 (#2316)
dependabot[bot] Aug 12, 2025
0ca2ce9
Build: Bump huggingface-hub from 0.33.4 to 0.34.4 (#2315)
dependabot[bot] Aug 12, 2025
514f6f6
Build: Bump mypy-boto3-glue from 1.40.0 to 1.40.5 (#2313)
dependabot[bot] Aug 12, 2025
91a91a9
Build: Bump daft from 0.5.16 to 0.5.18 (#2312)
dependabot[bot] Aug 12, 2025
e6324f4
Build: Bump actions/checkout from 4 to 5 (#2311)
dependabot[bot] Aug 12, 2025
88234ca
Build: Bump sqlalchemy from 2.0.41 to 2.0.43 (#2308)
dependabot[bot] Aug 12, 2025
8a98deb
Build: Bump bodo from 2025.7.5 to 2025.8.1 (#2309)
dependabot[bot] Aug 12, 2025
06caec2
Build: Bump actions/download-artifact from 4 to 5 (#2310)
dependabot[bot] Aug 12, 2025
0d95576
Build: Bump coverage from 7.10.2 to 7.10.3 (#2314)
dependabot[bot] Aug 12, 2025
47637d3
Build: Bump mmh3 from 5.1.0 to 5.2.0 (#2321)
dependabot[bot] Aug 12, 2025
76a6451
Build: Bump griffe from 1.9.0 to 1.11.1 (#2322)
dependabot[bot] Aug 12, 2025
4b961f7
refactor: consolidate snapshot expiration into MaintenanceTable (#2143)
ForeverAngry Aug 12, 2025
640c592
Document `older_than` of ExpireSnapshots (#2324)
Fokko Aug 13, 2025
0d1de4e
support adls.token
kevinjqliu Aug 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/license_check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,5 @@ jobs:
rat:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
- run: dev/check-license
4 changes: 2 additions & 2 deletions .github/workflows/nightly-pypi-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
outputs:
VERSION: ${{ steps.set-version.outputs.VERSION }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
with:
fetch-depth: 1

Expand Down Expand Up @@ -71,7 +71,7 @@ jobs:

steps:
- name: Download all the artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v5
with:
merge-multiple: true
path: dist/
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pypi-build-artifacts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
os: [ ubuntu-22.04, windows-2022, macos-13, macos-14 ]

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
with:
fetch-depth: 1

Expand All @@ -62,7 +62,7 @@ jobs:
if: startsWith(matrix.os, 'ubuntu')

- name: Build wheels
uses: pypa/cibuildwheel@v3.0.1
uses: pypa/cibuildwheel@v3.1.3
with:
output-dir: wheelhouse
config-file: "pyproject.toml"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-ci-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
- name: Install poetry
run: make install-poetry
- uses: actions/setup-python@v5
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/python-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,12 @@ jobs:
python: ['3.9', '3.10', '3.11', '3.12']

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
- name: Install poetry
run: make install-poetry
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
cache: poetry
cache-dependency-path: ./poetry.lock
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Install
Expand All @@ -76,7 +74,7 @@ jobs:
python: ['3.9', '3.10', '3.11', '3.12']

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Install
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-release-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
- name: Install poetry
run: make install-poetry
- uses: actions/setup-python@v5
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ jobs:
needs:
- validate-inputs
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
with:
fetch-depth: 1

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/svn-build-artifacts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
os: [ ubuntu-22.04, windows-2022, macos-13, macos-14 ]

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
with:
fetch-depth: 1

Expand All @@ -57,7 +57,7 @@ jobs:
if: startsWith(matrix.os, 'ubuntu')

- name: Build wheels
uses: pypa/cibuildwheel@v3.0.1
uses: pypa/cibuildwheel@v3.1.3
with:
output-dir: wheelhouse
config-file: "pyproject.toml"
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ else
TEST_RUNNER = poetry run
endif

POETRY_VERSION = 2.1.1
POETRY_VERSION = 2.1.4

# ============
# Help Section
Expand Down
44 changes: 44 additions & 0 deletions mkdocs/docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1287,6 +1287,50 @@ with table.manage_snapshots() as ms:
ms.create_branch(snapshot_id1, "Branch_A").create_tag(snapshot_id2, "tag789")
```

## Table Maintenance

PyIceberg provides table maintenance operations through the `table.maintenance` API. This provides a clean interface for performing maintenance tasks like snapshot expiration.

### Snapshot Expiration

Expire old snapshots to clean up table metadata and reduce storage costs:

```python
# Expire snapshots older than three days
from datetime import datetime, timedelta
table.maintenance.expire_snapshots().older_than(
datetime.now() - timedelta(days=3)
).commit()

# Expire a specific snapshot by ID
table.maintenance.expire_snapshots().by_id(12345).commit()

# Context manager usage (recommended for multiple operations)
with table.maintenance.expire_snapshots() as expire:
expire.by_id(12345)
expire.by_id(67890)
# Automatically commits when exiting the context
```

#### Real-world Example

```python
def cleanup_old_snapshots(table_name: str, snapshot_ids: list[int]):
"""Remove specific snapshots from a table."""
catalog = load_catalog("production")
table = catalog.load_table(table_name)

# Use context manager for safe transaction handling
with table.maintenance.expire_snapshots() as expire:
for snapshot_id in snapshot_ids:
expire.by_id(snapshot_id)

print(f"Expired {len(snapshot_ids)} snapshots from {table_name}")

# Usage
cleanup_old_snapshots("analytics.user_events", [12345, 67890, 11111])
```

## Views

PyIceberg supports view operations.
Expand Down
2 changes: 2 additions & 0 deletions mkdocs/docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,7 @@ The RESTCatalog supports pluggable authentication via the `auth` configuration b
- `noop`: No authentication (no Authorization header sent).
- `basic`: HTTP Basic authentication.
- `custom`: Custom authentication manager (requires `auth.impl`).
- `google`: Google Authentication support

###### Configuration Properties

Expand All @@ -414,6 +415,7 @@ catalog:
| `auth.impl` | Conditionally | The fully qualified class path for a custom AuthManager. Required if `auth.type` is `custom`. |
| `auth.basic` | If type is `basic` | Block containing `username` and `password` for HTTP Basic authentication. |
| `auth.custom` | If type is `custom` | Block containing configuration for the custom AuthManager. |
| `auth.google` | If type is `google` | Block containing `credentials_path` to a service account file (if using). Will default to using Application Default Credentials. |

###### Examples

Expand Down
Loading
Loading