Merge pull request #1 from lancedb/claude/project-improvement-areas-rdxfow

Harden CI/security, fill doc & test gaps, add governance files

Author: justinrmiller

.github/dependabot.yml

@@ -0,0 +1,33 @@
+version: 2
+updates:
+  # Keep the SHA-pinned GitHub Actions current (Dependabot bumps both the SHA and
+  # the trailing version comment).
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    groups:
+      github-actions:
+        patterns: ["*"]
+    commit-message:
+      prefix: "ci"
+
+  # Python dependencies via uv (reads pyproject.toml + uv.lock). The cluster-matched
+  # betas are intentionally pinned to the deployed Geneva/LanceDB build, and
+  # pyarrow/numpy are left to geneva's own constraints — Dependabot must not touch
+  # those, only the loose deps (typer, gradio, transformers, etc.).
+  - package-ecosystem: "uv"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    groups:
+      python:
+        patterns: ["*"]
+    commit-message:
+      prefix: "deps"
+    ignore:
+      - dependency-name: "geneva"
+      - dependency-name: "lancedb"
+      - dependency-name: "pylance"
+      - dependency-name: "pyarrow"
+      - dependency-name: "numpy"

.github/workflows/ci.yml

@@ -16,19 +16,23 @@ jobs:
   test:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
 
       - name: Install uv
-        uses: astral-sh/setup-uv@v8.2.0
+        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
         with:
           enable-cache: true
 
       # Installs the full runtime stack + the dev group. geneva/lancedb/pylance
       # come from the public Gemfury indexes declared in pyproject.toml; if those
       # ever require auth, add UV_INDEX_LANCEDB_PASSWORD / UV_INDEX_LANCE_FORMAT_PASSWORD
       # repo secrets and pass them here as env.
-      - name: Sync dependencies
-        run: uv sync --group dev
+      #
+      # --locked asserts uv.lock is in sync with pyproject.toml and fails the job
+      # on drift (e.g. a dependency edit without `uv lock`) instead of silently
+      # updating it — guarding the deliberate cluster pins.
+      - name: Sync dependencies (lockfile must be up to date)
+        run: uv sync --locked --group dev
 
       - name: Ruff lint
         run: uv run ruff check --output-format=github
@@ -47,8 +51,45 @@ jobs:
 
       - name: Upload coverage report
         if: always()
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
         with:
           name: coverage-xml
           path: coverage.xml
           if-no-files-found: ignore
+
+  # Security hardening: a dependency CVE scan and a secret scan, run in parallel
+  # with `test`. The dependency audit is advisory (continue-on-error) — like the
+  # `ty` step — because the geneva/lancedb/pylance betas come from Gemfury and may
+  # not be known to the advisory DB. The secret scan IS allowed to fail the job.
+  audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          fetch-depth: 0  # the secret scan walks git history
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        with:
+          enable-cache: true
+
+      # Export the locked dependency set (without this project) and audit the
+      # pinned versions against the PyPI advisory DB.
+      - name: Dependency audit (pip-audit, non-blocking)
+        run: |
+          uv export --frozen --no-emit-project --no-hashes \
+            --format requirements-txt -o requirements.txt
+          uvx pip-audit -r requirements.txt
+        continue-on-error: true
+
+      # Secret scan with TruffleHog (Apache-2.0, fully open source). Run the
+      # binary directly — no Action wrapper, no license — and walk git history,
+      # failing the job only on actively verified secrets to avoid noise.
+      - name: Secret scan (trufflehog)
+        env:
+          TRUFFLEHOG_VERSION: 3.95.6
+        run: |
+          curl -sSLf \
+            "https://github.com/trufflesecurity/trufflehog/releases/download/v${TRUFFLEHOG_VERSION}/trufflehog_${TRUFFLEHOG_VERSION}_linux_amd64.tar.gz" \
+            | tar -xz -C /tmp trufflehog
+          /tmp/trufflehog git file://. --only-verified --fail --no-update

.gitignore

@@ -47,6 +47,9 @@ ENV/
 env.bak/
 venv.bak/
 
+# requirements.txt exported by the CI dependency-audit job (uv.lock is the source of truth).
+requirements.txt
+
 # --- Test / coverage / type-check caches ---
 .pytest_cache/
 .tox/

.pre-commit-config.yaml

@@ -13,6 +13,7 @@ repos:
       - id: check-yaml
       - id: check-toml
       - id: check-merge-conflict
+      - id: detect-private-key  # block accidental private-key commits
       - id: debug-statements
       - id: mixed-line-ending
       - id: check-added-large-files

CONTRIBUTING.md

@@ -0,0 +1,77 @@
+# Contributing to geneva-examples
+
+Thanks for improving the Geneva UDF examples. This guide covers the local setup,
+the project's conventions, and the workflow for adding a new UDF or stage.
+
+## Local setup
+
+Requires Python ≥ 3.12 and [`uv`](https://docs.astral.sh/uv/).
+
+```bash
+make install   # uv sync --group dev + install the git pre-commit hook
+make check     # the full CI gate: ruff lint + format-check + pytest (90% coverage)
+```
+
+`geneva`, `lancedb`, and `pylance` are pinned betas served from Gemfury indexes
+(declared in [`pyproject.toml`](pyproject.toml)). If your environment can't reach
+those indexes, `uv sync` will fail on those packages — request access or run in an
+environment that has it.
+
+Useful targets (see `make help`): `make lint-fix`, `make format`, `make test`,
+`make typecheck`, `make precommit`.
+
+## Conventions
+
+- **Formatting & linting:** `ruff` (config in `pyproject.toml`) is the single
+  source of truth and gates every commit via pre-commit. Run `make format` before
+  pushing. Line length is 88; existing `# noqa` suppressions are kept honest by
+  `RUF100`, so don't add ones the selected rules won't use.
+- **Type checking:** `ty` runs in pre-commit and CI but is **non-blocking** by
+  design — it's a preview tool with many false positives on the untyped ML deps.
+  Prefer precise annotations; for the opaque Geneva/LanceDB runtime objects, use
+  the structural `Protocol`s in
+  [`geneva_examples/core/_types.py`](geneva_examples/core/_types.py) under a
+  `TYPE_CHECKING` guard instead of importing the beta runtime types. Promoting
+  `ty` to a blocking gate is deliberately deferred until its false-positive rate
+  drops.
+- **Imports in UDF bodies:** a UDF/chunker body is a self-contained closure that
+  ships to the remote workers. Nest its imports and helpers *inside* the factory
+  function so they serialize with it; keep the driver/CLI code lightweight.
+
+## Adding a new UDF
+
+1. **Prototype in UDF Studio.** Run `uv run udf-studio`, pick a template, point it
+   at sample data in `studio_data/`, and iterate on your `transform(value)` (UDF)
+   or `chunk(value)` (chunker) locally — no cluster, GPU, or Ray. See the
+   [README](README.md#udf-studio).
+2. **Add a factory + manifest.** Create a module in
+   [`geneva_examples/udfs/`](geneva_examples/udfs/) following
+   [`imageinfo.py`](geneva_examples/udfs/imageinfo.py): export a `build_*_udf(...)`
+   factory and a `*_RUNTIME_PIP` list pinning the worker-side packages
+   (env-overridable, like `GENEVA_PACKAGE_SPEC`).
+3. **Wire a stage CLI.** Add a Typer CLI under
+   [`geneva_examples/pipeline/stages/`](geneva_examples/pipeline/stages/) modeled on
+   [`lightweight.py`](geneva_examples/pipeline/stages/lightweight.py): load config,
+   `connect`, build a `GenevaManifest`, build the UDF(s), and call the shared
+   [`backfill_column()`](geneva_examples/pipeline/stages/_runner.py) runner. Add a
+   `project.scripts` entry in `pyproject.toml`.
+
+## Testing & the coverage policy
+
+The suite enforces a **90% coverage gate**, but the pieces that need a live
+cluster, GPU, or model weights are listed in `[tool.coverage.run] omit` in
+`pyproject.toml` (the model UDFs, the pipeline/ops CLIs, the Gradio wiring). Their
+*pure* helpers are still unit-tested — they just don't inflate the percentage.
+
+When you add code:
+
+- Unit-test pure helpers directly (see `tests/test_udfs.py`,
+  `tests/test_pipeline_runner.py`, `tests/test_ops_*.py`).
+- For CLI *wiring* that would otherwise hit a cluster, add a mocked smoke test in
+  the style of [`tests/test_pipeline_smoke.py`](tests/test_pipeline_smoke.py): use
+  `typer.testing.CliRunner` and monkeypatch `load_config`/`connect` plus an
+  injected fake `geneva` module.
+- Reuse the synthetic-media fixtures in `tests/conftest.py`
+  (`make_png`, `make_mp4`, `data_dir`).
+
+Run `make check` before opening a PR.