Skip to content

Adding lexical/semantic eval tools to Ollama Docker image#364

Draft
tefirman wants to merge 2 commits into
mainfrom
add-eval-deps-ollama
Draft

Adding lexical/semantic eval tools to Ollama Docker image#364
tefirman wants to merge 2 commits into
mainfrom
add-eval-deps-ollama

Conversation

@tefirman
Copy link
Copy Markdown
Member

Type of Change

  • Version update (extends an existing image's bundled tooling)
  • Documentation update

Description

Adds the Python dependencies needed by the WDL functional-eval pipeline (lexical/semantic similarity scoring and HuggingFace embedding loading) directly into the getwilds/ollama image so callers no longer have to pip install them at runtime. Previously, wilds-wdl-writer's wdl_writer/benchmarking.py (run via benchmarking.wdl) crashed on import because the image only shipped ollama and chromadb.

New packages, with pins matching a fresh install today:

  • rapidfuzz==3.14.5 — fast string similarity for lexical eval
  • sentence-transformers==5.5.1 — embedding models for semantic similarity
  • llama-index-embeddings-huggingface==0.7.0 — the adapter benchmarking.py imports directly
  • torch==2.12.0+cpu — installed first from https://download.pytorch.org/whl/cpu so the CUDA wheels (multi-GB) are skipped. Ollama owns the GPU; the embedding model runs on CPU. The smoke test asserts not torch.cuda.is_available() to catch regressions if a future change accidentally pulls in the CUDA build.

Applied identically to Dockerfile_0.21.0 and Dockerfile_latest. README updated: intro mentions lexical/semantic eval support, Image Details and Dockerfile Structure list the new packages, and citations were added for sentence-transformers and PyTorch.

Testing

How did you test these changes?

  • make lint IMAGE=ollama — hadolint passes on both Dockerfiles.
  • Built the image locally and confirmed it comes up cleanly. The expanded smoke test exercises every new import (rapidfuzz, torch + CPU-only assertion, sentence_transformers, from llama_index.embeddings.huggingface import HuggingFaceEmbedding) as part of the build.

Did the tests pass?

Yes — lint clean and local build succeeded.

Checklist

  • Dockerfile follows naming convention (Dockerfile_X.Y.Z or Dockerfile_latest)
  • All required OCI metadata labels are present and accurate
  • README.md is included/updated in the tool directory
  • Tested locally with make validate IMAGE=toolname (or manually built and verified)
  • Image builds successfully for target platform(s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant