chore: Upgrade all Python dependencies to latest versions

[martyngigg]

Summary

Upgrades all deployed Python dependencies with the exception of pyarrow. Pyarrow pinned @ 21.0.0 due to bug with s3 uploads. See #183.

Fixes #178
Fixes #183

Summary by CodeRabbit

• Chores
  • Updated project dependencies for improved stability and compatibility.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

📝 Walkthrough

Walkthrough

The pull request adds pyarrow==21.0.0 as a project dependency in elt-common/pyproject.toml, positioned alongside existing dlt and pyiceberg dependencies. A comment documents a checksum-related issue affecting pyarrow versions 22.0.0 and above.

Changes

Cohort / File(s)	Summary
Dependency Version Pinning elt-common/pyproject.toml	Added pyarrow==21.0.0 to [project] dependencies with accompanying comment explaining checksum-related incompatibility with pyarrow >= 22.0.0. Positioned between dlt[parquet,s3]~=1.20.0 and pyiceberg~=0.9.1.

Poem

🐰 When pyarrow's twenty-two tripped on checksums,
We hopped back to twenty-one, setting new sums!
Parquet uploads sail to S3 with ease,
No multipart failures, no XAmz to displease,
This rabbit's rejoiced by such steady release! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The PR title claims to upgrade all dependencies to latest versions, but the actual change only pins pyarrow at 21.0.0 (an older version) without upgrading other dependencies like requests, ruff, or pytest mentioned in issue #178.	Update the PR title to accurately reflect that only pyarrow is pinned to 21.0.0, or ensure all other dependencies listed in issue #178 are also upgraded as the title suggests.
Linked Issues check	⚠️ Warning	The PR only addresses the pyarrow pin from issue #183 but fails to implement security vulnerability fixes required by issue #178: requests should be ≥2.32.4, ruff should be ≥0.14.10, and pytest should be considered for upgrade to 9.0.2.	Upgrade requests to ≥2.32.4, ruff to ≥0.14.10, and pytest to 9.0.2 in pyproject.toml to fully address both linked issues #178 and #183.
Out of Scope Changes check	⚠️ Warning	The pyarrow pin is in scope for issue #183, but the PR does not include the security dependency updates required by linked issue #178, making the overall PR incomplete relative to its stated objectives.	Include all dependency updates from issue #178 (requests, ruff, pytest) in addition to the pyarrow pin to fully satisfy the objectives of both linked issues.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

elt-common/pyproject.toml (1)

29-39: Critical: Dev dependencies do not meet security requirements from issue #178.

The PR claims to fix issue #178, but the dev dependencies listed do not satisfy the security requirements specified in that issue:

• Line 36: requests~=2.32.3 — Issue #178 requires >= 2.32.4 (target 2.32.5) to address CVE-2024-47081.
• Line 38: ruff~=0.11.11 — Issue #178 requires >= 0.14.10 to address an improper neutralisation vulnerability.
• Line 33: pytest~=8.3.5 — Issue #178 recommends upgrading to 9.0.2.

These security vulnerabilities remain unaddressed despite the PR claiming to resolve them.

🔒 Proposed fix to address security vulnerabilities

 dev = [
   "minio~=7.2.16",
   "prek>=0.1.6",
   "pydantic-settings~=2.10.1",
-  "pytest~=8.3.5",
+  "pytest~=9.0.2",
   "pytest-httpx~=0.35.0",
   "pytest-mock~=3.14.1",
-  "requests~=2.32.3",
+  "requests~=2.32.5",
   "requests-mock~=1.12.1",
-  "ruff~=0.11.11",
+  "ruff~=0.14.10",
 ]

🤖 Fix all issues with AI agents

In @elt-common/pyproject.toml:
- Around line 12-14: pyiceberg is still pinned to "pyiceberg~=0.9.1" in
pyproject.toml but should be upgraded to 0.10.0; update the dependency entry to
reference version 0.10.0 (e.g., "pyiceberg~=0.10.0"), regenerate the lock file
(poetry.lock or equivalent), and run the test suite and any packaging/CI checks
to ensure compatibility with the new pyiceberg release.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to data retention organization setting

📥 Commits

Reviewing files that changed from the base of the PR and between 2d810b0 and f1933a9.

⛔ Files ignored due to path filters (5)

• elt-common/uv.lock is excluded by !**/*.lock
• warehouses/accelerator/extract_load/accelerator_sharepoint/extract_and_load.py.lock is excluded by !**/*.lock
• warehouses/accelerator/extract_load/electricity_sharepoint/extract_and_load.py.lock is excluded by !**/*.lock
• warehouses/accelerator/extract_load/opralogweb/extract_and_load.py.lock is excluded by !**/*.lock
• warehouses/accelerator/extract_load/statusdisplay/extract_and_load.py.lock is excluded by !**/*.lock

📒 Files selected for processing (1)

• elt-common/pyproject.toml

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: warehouses end-to-end tests
• GitHub Check: elt-common end-to-end tests