[RUNTIME] Problem when "running" package from source using `uv run --with=`

[amotl]

Problem

git clone https://github.com/crate/cratedb-toolkit
cd cratedb-toolkit
uv run --with='.[kinesis,test]' pytest -m kinesis

  × No solution found when resolving dependencies for split (markers: python_full_version >= '3.10'):
  ╰─▶ Because cratedb-toolkit[io-ingestr] depends on sqlalchemy>=1,<2 and cratedb-toolkit[all] depends on sqlalchemy>=2, we can conclude that
      cratedb-toolkit[all] and cratedb-toolkit[io-ingestr] are incompatible.
      And because your project requires cratedb-toolkit[all] and cratedb-toolkit[io-ingestr], we can conclude that your project's requirements are
      unsatisfiable.

[hampsterx]

was curious heh

The root cause seems to be that ingestr hard-pins sqlalchemy==1.4.52 (requirements.in), making io-ingest fundamentally incompatible with any extra that requires SA2 (io, influxdb, full, etc.). When uv run --with='.[kinesis,test]' resolves from source, it sees the conflict across extras.

This isn't a uv bug or a kinesis-specific issue - it's a known dependency split.

• Support for SQLAlchemy 2.x bruin-data/ingestr#229

I see you raised that issue eh tracking SA2 migration, but it's blocked by the dialect ecosystem:

• dlt itself supports sqlalchemy>=1.4 (SA2-compatible) - not the blocker
• clickhouse-sqlalchemy - partially migrated to SA2 (sync only, async still unstable)
• sqlalchemy-redshift - still pins <2.0.0, last commit April 2023, effectively stalled
• Several other dialects bundled by ingestr have varying SA2 readiness

Until ingestr migrates, the workaround is to avoid combining io-ingest with SA2-requiring extras. For the specific case in the issue (.[kinesis,test]), this should actually work fine since neither extra pulls in io-ingest - the error may be caused by uv resolving all extras when running from source. A uv run --resolution=lowest-direct or installing only the needed extras explicitly might help.

[amotl]

Hi. Thank you for investigating this. Unfortunately, the --resolution=lowest-direct option did not help.

uv run --resolution=lowest-direct --with='.[kinesis,test]' pytest -m kinesis

[amotl]

Thank you also for analyzing the dependency chains. As I can learn from your analysis, the most prominent blocker is sqlalchemy-redshift. Colleagues have been sending patches, but they are not merged, so, well, effectively stalled, yes.

• Fixes for SQLAlchemy V2 sqlalchemy-redshift/sqlalchemy-redshift#307
• fix: Fixes for reflection in SQLAlchemy 2.0  sqlalchemy-redshift/sqlalchemy-redshift#314
• Fix SQLAlchemy V2 support sqlalchemy-redshift/sqlalchemy-redshift#319
• Migration to SQLAlchemy 2.0 and Python 3.10+ sqlalchemy-redshift/sqlalchemy-redshift#326

[amotl]

This patch might finally modernize sqlalchemy-redshift. Thank you, @awsmadi.

• Build modernization - Python 3.10+, SA 2.0, remove pkg_resources sqlalchemy-redshift/sqlalchemy-redshift#327

[amotl]

A patch was merged to modernize sqlalchemy-redshift, a release is pending.

• Migration to SQLAlchemy 2.0 and Python 3.10+ sqlalchemy-redshift/sqlalchemy-redshift#326
• Build modernization - Python 3.10+, SA 2.0, remove pkg_resources sqlalchemy-redshift/sqlalchemy-redshift#327 (comment)
• support sqlalchemy 2.0 sqlalchemy-redshift/sqlalchemy-redshift#264 (comment)

[amotl]

When the blockade by ingestr will be resolved with GH-842, now that omniload will understand SQLAlchemy 2.x, other requirements from the pymongo extra will take its role, which also adds a dependency constraint on sqlalchemy<2. 💥

  × No solution found when resolving dependencies for split (markers: python_full_version >= '3.10'):
  ╰─▶ Because cratedb-toolkit[pymongo] depends on sqlalchemy<2 and cratedb-toolkit[deltalake] depends on sqlalchemy>=2, we can conclude that
      cratedb-toolkit[deltalake] and cratedb-toolkit[pymongo] are incompatible.
      And because your project requires cratedb-toolkit[deltalake] and cratedb-toolkit[pymongo], we can conclude that your project's requirements are
      unsatisfiable.