Upgrade Python Packages

[tino097]

This PR introduces managed dependency resolution using uv lock, upgrading all Python packages to their latest compatible versions. This directly addresses the growing number of Snyk and Dependabot vulnerability reports caused by unpinned transitive dependencies.

https://linear.app/mindsdb/issue/FQE-1840/pin-dependencies-in-requirementstxt-from-to-for-stability

What changed:

• Cleaned up pyproject.toml: separated direct dependencies from transitive ones, applied loose version constraints on direct deps only, and organized optional dependency groups (agents, kb, opentelemetry, handlers, dev, test)
• Generated uv.lock as the cross-platform lockfile (single file covers both Linux and Windows, including platform-specific deps like duckdb)
• Exported requirements-*.txt files from uv.lock for Snyk/Dependabot/Vanta compatibility, since native uv.lock scanning is not yet available (expected April 2026)
• Added .snyk configuration file
• Local snyk scan: Tested 14 projects, 2 contained vulnerable paths.

Steps for verification:

• Create snyk config file
• Create uv.lock file
• Generate requirements-*.txt files
• Pass all tests

---

Confidence Score: 5/5 - Safe to Merge

• No review comments were generated, indicating the PR appears clean from automated analysis
• No critical, significant, or high-risk issues were identified in the heuristic analysis
• No existing unresolved comments to address
• Coverage was limited to 2/11 files, but with no issues found in reviewed files and no flags raised, confidence remains high

[ea-rus]

@tino097, as I understand uv.lock and *.txt files were generated, right? How they were generated?
If I want to I add dependency or update version what should I do?

[tino097]

Adding new packages would be done with the pyproject.toml

As for the uv.lock would be used for developers to "lock" the versions generated in requirements.txt no matter of platform they are using.

And for the generating the requirements-*.txt files the following command can be executed

uv export --format requirements-txt --no-hashes > requirements.txt

@ea-rus any suggestions/concerns over this ?
cc @StpMax @martyna-mindsdb

[StpMax]

Just some thoughts:
I'm not sure the changes in this PR will actually solve the problem of growing number of Snyk and Dependabot vulnerability reports caused by unpinned transitive dependencies. All the Snyk-based changes that have been merged into the project so far constrain libraries with >= only. Pinning the versions of all shared libraries won't reduce the number of PRs from Snyk.
I'm also concerned that we're ending up with some kind of mix between pyproject.toml-style and setup.py-style. Ideally we should pick one or the other (i'd prefer pyproject.toml, since setup.py is legacy). We now have two sources of truth for the dependency list: pyproject.toml and requirements-*.txt. Moreover, the values in pyproject.toml aren't really "truth" — they're more like "suggestions", since for example it says pydantic ~= 2.12 while requirements.txt has pydantic==2.12.5. This is confusing.
Maybe we should get rid of the requirements-*.txt files and setup.py entirely, and keep only pyproject.toml? And pin specific dependency versions there. That would make everything much simpler.

[tino097]

Hey @StpMax, as we discussed

The intent is:

• pyproject.toml will be human-maintained constraints (loose, intentional)
• uv.lock will be machine-generated lockfile (exact, authoritative)
• requirements-*.txt are the exports from the lockfile for tool compatibility. This can be deprecated in future.

So there's really one source of truth (uv.lock), with requirements-*.txt being derived artifacts.

Regarding the Snyk scenarios as an example:

mindsdb  <-------  litellm ~= 1.x
                               └── opentelemetry-sdk == 1.25.0   ← CVE here

If we install opentelemetry-sdk 1.26.0 that fixes the CVE, but litellm 1.x hasn't released a version that allows 1.26.0 yet.
This means that we can set the floor to be on
opentelemetry-sdk >= 1.25.1 and update the .snyk file with the CVE information

ignore:
  CVE-2024-XXXX:
    - opentelemetry-sdk:
        reason: Blocked by litellm compatibility, tracking issue #YYYY
        expires: '2026-06-01'

So when the litellm will publish a updated version we can use
uv lock --upgrade-package opentelemetry-sdk

and we can re-export the requirements files with those changes

cc @martyna-mindsdb @ea-rus

[StpMax]

@tino097 I still hold the opinion that having requirements*.txt in this case does not solve the Snyk CVE problem. Let's walk through the given example. If opentelemetry-sdk==1.25.0 has a vulnerability and we need to install opentelemetry-sdk>=1.25.1, the workflow would be:

• Add opentelemetry-sdk>=1.25.1 to pyproject.toml and run uv lock
  • If lock succeeds → nothing else to do, the dependency is already pinned in pyproject.toml
  • If dependency resolution fails, then:
    • Remove opentelemetry-sdk>=1.25.1 from pyproject.toml
    • Add an ignore entry for opentelemetry-sdk to .snyk. At this step, to fill in the reason field, we might need the name of the library that prevents resolving a compatible version of opentelemetry-sdk.

So in this case, requirements-*.txt is only needed to write a comment for the Snyk rule. But wouldn't uv lock failing with a resolution error provide exactly the same information?

Speaking of CVE reports in general — most of them look like Vulnerability in library XYZ fixed in version 1.1. In that case, it doesn't matter how or where the dependency is declared — we'll still get the report and the PR.

If all our dependencies are pinned, the only scenario where we wouldn't get a report/PR is if a vulnerability is introduced in a newer version of a library that is higher than what we have pinned.

So, as i see it, the proposed changes won't help solve the problem of the "growing number of Snyk and Dependabot vulnerability reports caused by unpinned transitive dependencies".

On top of that, we will have new issues (as i described above) — for example, mismatch between declared dependencies (in pyproject.toml) and actual ones (in requirements-*.txt).

As a temporary solution, while Snyk doesn't yet support reading uv.lock, this might be ok (but pydantic-ai should be moved to the main dependencies). But finally in future we'll need to keep just one source of truth — ideally pyproject.toml only.

[tino097]

Hey @StpMax, good points overall, let me try to clarify a few things.

On requirements-*.txt as a source of truth

We dont manually edit these files at all they are fully generated from uv.lock via uv export --frozen.
The final outcome will be pyproject.toml only, once Snyk start with uv.lock support, we will remove requirements-*.txt files from the repo

On the .snyk ignore workflow

Agreed and the workflow is:

• Snyk reports a CVE on a transitive dep
• We investigate if does it actually affect our codebase?
• If yes and fixable, upgrade the floor in pyproject.toml, run uv lock
• If not fixable yet because is blocked by a direct dep or not affecting us, than we add a ignore to .snyk. But we have something that we can reference to review it again in some time

The value of having pinned transitive deps in the exported requirements files is exactly what you said, it will give Snyk accurate version information to report against, rather than guessing.
https://docs.snyk.io/scan-with-snyk/snyk-open-source/manage-vulnerabilities/differences-in-open-source-vulnerability-counts-across-environments

So as a conclusion, maybe the statement that we wont get many reports is not practically correct but we will have a clear pathway of improving the dependency management and handling the reports

edit: i will fix the typos and move pydantic-ai

[hamishfagg]

@tino097 we might want to update tests/scripts/check_requirements.py as well to suit these changes and work off pyproject.toml. But I think we might be able to simplify it considerably by setting all handlers except the one we're checking as a dev depend group: https://deptry.com/usage/#optional-dependencies-dev-groups

Im also hoping we can get rid of all of the cross-requirement.txt checks and just run validate-pyproject or something

[tino097]

@tino097 we might want to update tests/scripts/check_requirements.py as well to suit these changes and work off pyproject.toml. But I think we might be able to simplify it considerably by setting all handlers except the one we're checking as a dev depend group: https://deptry.com/usage/#optional-dependencies-dev-groups

Im also hoping we can get rid of all of the cross-requirement.txt checks and just run validate-pyproject or something

I've missed this comment, but its really helpful, thanks @hamishfagg !!!

Additionally we will remove community handlers with #12250

[tino097]

This PR didn't helped a lot with its intention, it will be closed