Adopt uv and pyproject.toml for dependency management - #223
Merged
Conversation
Dependencies move from requirements.txt into pyproject.toml, with uv.lock pinning an exact, reproducible set. CI installs from the lock instead of re-resolving on every run. This is the failure mode that has bitten this project repeatedly: siminn's venv was built in Oct 2023, drifted five months behind its own code, and failed on every daily restart for over two years without anyone noticing. A lockfile makes "the installed environment matches the declared one" a checkable property rather than an assumption. - pyproject.toml: dependencies translated verbatim from requirements.txt, keeping the PyPy-specific constraint comments (openai<1.40 needs no Rust for jiter, httpx<0.28, setuptools<82 for pkg_resources). requires-python is >=3.11, matching the interpreter matrix. - tool.uv package = false: GreynirServer is an application whose modules live at the repository root and run in place, not an installable library, so uv manages the environment without trying to build the project. - tool.uv.sources pins icespeak to the 'greynir' branch, replacing the git+https direct reference (verified: 'greynir' is a branch, not a tag). - The SQLAlchemy type stubs move to a dev dependency group. They are never imported at runtime, and sqlalchemy-stubs declares mypy as a dependency, which drags a Rust/PyO3 extension into what production installs. - CI runs `uv sync --frozen` bound to the setup-python interpreter, then `uv run` for pytest, curlylint and scraper.py --init. - .gitignore: add .venv/, which the existing venv* pattern does not match. requirements.txt is deliberately left in place, with a header explaining why: scripts/deploy.sh still runs `pip install -r requirements.txt` against the production venv, which is still PyPy 3.10. Regenerating it from a lock resolved for >=3.11 could break a deploy. It should be deleted once deploy.sh moves to `uv sync` and production is rebuilt on PyPy 3.11. Verified locally on CPython 3.11: uv lock resolves 91 packages with no conflicts, uv sync installs them in 10s entirely from wheels with no build failures, and the resulting environment imports cleanly with the PyPy pins intact (openai 1.39.0, httpx 0.27.2). The test suite was NOT run locally -- it imports main and would connect to the production scraper database on this host. CI is the correct place to verify it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the suite locally was unsafe: the database name is hardcoded to "scraper" in db/__init__.py, so a local run cannot be isolated by database name, and pytest would connect to the production cluster on port 5432 and write to its query log. scripts/test_local.sh mirrors the CI steps against a disposable PostgreSQL cluster of its own -- initdb into a temp directory, start it on a spare port, create the reynir role and scraper database, populate from tests/files/populate_testdb.sql, run pytest, then tear the whole thing down. It needs neither sudo nor Docker, and refuses outright to run against 5432. Also adds [tool.pytest.ini_options] testpaths = ["tests"]. Without it pytest walks the entire repository from its rootdir, so any local interpreter or virtualenv directory (a pypy/ or venv/ tree) gets collected, putting vendored third-party packages on sys.path where they shadow the real ones. This produced 17 collection errors locally. CI never hit it because its checkout is clean, but it broke local runs for anyone with a local interpreter directory. Current local result: 34 passed, 7 skipped, 19 failed. All 19 failures are "RuntimeError: TTS with Azure failed" -- icespeak asserts at import time that a speech engine is configured, so the script writes structurally valid placeholder keys to satisfy the import. Real synthesis needs real credentials, which CI injects from repository secrets. Drop a real key into resources/AzureSpeechServerKey.json for full parity; the script will not overwrite a non-empty key file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
icespeak defaults ICESPEAK_KEYS_DIR to a relative "keys" directory. This project has never had one -- the API keys live in resources/, and no keys/ directory exists in the repository or in either deployment. icespeak reads the variable at import time and asserts that at least one speech engine is configured, so without it every import of icespeak fails outright with "No voices available". Deployments papered over this by setting ICESPEAK_KEYS_DIR in .env, with the result that a fresh checkout could not start the web server at all: main.py imports routes, which imports icespeak, which asserted and died before Flask ever started. The tests only worked because CI passed the variable explicitly. utility.py now sets it unconditionally to the absolute RESOURCES_DIR, so it holds regardless of the working directory and regardless of whether a .env exists. utility is imported by main.py well before routes, which is what guarantees it lands before icespeak is first imported. Removes the now-redundant ICESPEAK_KEYS_DIR from the CI workflow and from scripts/test_local.sh, where it would silently be overwritten anyway. Verified: the web server starts with no ICESPEAK_KEYS_DIR in the environment and serves pages, and the full suite passes locally (53 passed, 7 skipped) with the variable unset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
config/Greynir.conf set host = 0.0.0.0, and the config file overrides the GREYNIR_HOST environment variable (settings.py:183), so there was no way to narrow the binding without editing this file. Running main.py on a host with a public IP therefore exposed the Flask development server, with full access to the production database, on every interface. This only affects the development server: production serves through gunicorn, which takes its bind address from config/gunicorn_config.py, and deploy.sh does not copy config/Greynir.conf to deployments. Anyone who does want to reach the dev server from another machine can either change this value locally or forward a port over SSH. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dependencies move from
requirements.txtintopyproject.toml, withuv.lockpinning an exact, reproducible set. CI installs from the lock instead of re-resolving on every run.This targets a failure mode that has bitten this project repeatedly: siminn's venv was built in Oct 2023, drifted five months behind its own code, and failed on every daily restart for over two years without anyone noticing. A lockfile makes "the installed environment matches the declared one" a checkable property rather than an assumption.
Changes
pyproject.toml— dependencies translated verbatim fromrequirements.txt, keeping the PyPy-specific constraint comments (openai<1.40so jiter/Rust isn't needed,httpx<0.28,setuptools<82forpkg_resources).requires-python = ">=3.11", matching the interpreter matrix.[tool.uv] package = false— GreynirServer is an application whose modules live at the repository root and run in place, not an installable library. uv manages the environment without trying to build the project, so nosrc/restructure is needed.[tool.uv.sources]pinsicespeakto thegreynirbranch, replacing thegit+httpsdirect reference. Verified against the remote thatgreyniris a branch, not a tag.sqlalchemy-stubsdeclaresmypyas a dependency — which is what dragged the Rust/PyO3 extension into the install and set the 3.11 floor. Production no longer installs them.uv sync --frozenbound to the setup-python interpreter, thenuv runfor pytest, curlylint andscraper.py --init.--frozenmeans CI fails loudly if the lock drifts frompyproject.tomlrather than silently re-resolving..gitignore— add.venv/, which the existingvenv*pattern does not match (leading dot).requirements.txtis deliberately keptscripts/deploy.shstill runspip install -r requirements.txtagainst the production venv, which is still PyPy 3.10. Regenerating that file from a lock resolved for>=3.11could break a deploy. It carries a header explaining the situation and should be deleted oncedeploy.shmoves touv sync --frozen --no-devand production is rebuilt on PyPy 3.11.Verification
Locally on CPython 3.11 with uv 0.12.0 (the same version CI installs):
uv lockresolves 91 packages, no conflictsuv sync --frozeninstalls in 10.5 s, entirely from wheels, no build failuresuv lock --checkconfirms the lock matchespyproject.tomlThe test suite was not run locally. It imports
mainand would connect to the productionscraperdatabase on this host. CI is the correct place to verify it.Out of scope
vectors/keeps its ownrequirements.txtand CPython environment (currently 3.9.4, gensim 3.8.2). It is a separate, incompatible runtime — numpy and gensim don't work adequately under PyPy — so it wants its own uv project rather than a workspace member. Separate work.🤖 Generated with Claude Code