Switch to AIPCC base image and enable fully hermetic builds

[ckhordiasma]

https://redhat.atlassian.net/browse/RHOAIENG-47507

Summary

• Switch Dockerfile.konflux from UBI9 python-312 to AIPCC CPU base image (quay.io/aipcc/base-images/cpu:3.5.0-ea.1) with pre-installed system libraries and pre-configured Python index
• Compile requirements.txt against the AIPCC index using uv pip compile with --default-index, --prerelease=if-necessary, and --no-strip-markers for multi-arch compatibility
• Add requirements-build.txt generated via pip-compile --build-deps-for=wheel --only-build-deps for the poetry-core build backend
• Add hermetic model prefetch — replace the runtime pre_download_required_models.py network download with prefetched artifacts via the Konflux generic fetcher, enabling full --network none builds
• Enable hermetic: true in the PR pipeline

Hermetic model prefetch

The previous approach downloaded ML models from HuggingFace/GitHub at build time, which was the last network dependency blocking hermetic builds. This PR replaces it with:

• artifacts.lock.yaml — generic fetcher lockfile with download URLs and checksums for all model files, generated by scripts/generate_model_lockfile.py
• scripts/setup_prefetched_models.sh — reconstructs HF cache directory structures from the flat prefetched files during Docker build
• requirements-models.txt — spacy en-core-web-lg model as a pip URL dependency (handled by the pip prefetcher)

Models prefetched:

Model	Type	Source
sentence-transformers/all-MiniLM-L6-v2	Sentence Transformers (PyTorch)	HuggingFace Hub
qdrant/all-MiniLM-L6-v2-onnx	FastEmbed (ONNX)	HuggingFace Hub
en-core-web-lg 3.8.0	SpaCy	GitHub releases (pip)
punkt	NLTK tokenizer	GitHub nltk_data

To regenerate the lockfile when models change:

uv run --with pyyaml --with fastembed scripts/generate_model_lockfile.py

Multi-arch verification

Hermetic builds (--network none) tested and model loading validated on all four target architectures:

Arch	Host	Build	Models
aarch64	local (Apple Silicon)	✅ Pass	✅ All 4 load offline
x86_64	Beaker VM	✅ Pass	✅ All 4 load offline
s390x	Beaker VM	✅ Pass	✅ All 4 load offline
ppc64le	Beaker VM	✅ Pass	✅ All 4 load offline

Key decisions

• hermetic: true — all dependencies (pip wheels, spacy model, HuggingFace models, NLTK data) are now prefetched. No network access needed during build.
• Generic fetcher for HF models — each model file is listed individually in artifacts.lock.yaml with SHA256 checksums. The setup script reconstructs the HF Hub cache structure (models--org--repo/snapshots/{commit}/) at build time.
• Separate pip install for requirements-models.txt — the spacy model URL dependency includes a --hash, which triggers pip's --require-hashes mode. Installing it separately prevents hash-mode from propagating to the main requirements.txt.
• Profile-gated setup — both generate_model_lockfile.py and setup_prefetched_models.sh accept GUARDRAILS_PROFILE and fail with a clear error for unsupported profiles (only opensource is implemented).
• --prerelease=if-necessary instead of --prerelease=allow — avoids picking RC versions that may lack wheels on some architectures
• --no-strip-markers — preserves platform markers so arch-specific deps like triton are skipped where not needed

Test plan

• Hermetic build passes on all 4 architectures (aarch64, x86_64, s390x, ppc64le)
• All models load offline (Sentence Transformers, FastEmbed, SpaCy, NLTK) on all 4 architectures
• Verify prefetched dependencies match what's installed in the image
• PR pipeline builds successfully on all 4 architectures
• Container starts and serves the guardrails API on port 8000

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.02%. Comparing base (618be51) to head (63058ae).
⚠️ Report is 268 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #245      +/-   ##
==========================================
+ Coverage   76.91%   86.02%   +9.11%     
==========================================
  Files         200      250      +50     
  Lines       20473    25297    +4824     
==========================================
+ Hits        15747    21762    +6015     
+ Misses       4726     3535    -1191     

Flag	Coverage Δ
python	86.02% <ø> (+9.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 158 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[satyamg1620]

@ckhordiasma for s390x, loading spacy models causing endianess issue. for that thinc package has to be build from source and have to patch the fix. @modassarrana89-new do you want to add something?

[m-misiura]

AIPCC base image seems to have a different naming convention for the certificates inside /etc/pki/tls i.e. they are called ca-certificates.crt but Python ssl library expects them to be called ca-bundle.crt

I think we may need a symlink in the Dockerfile.konflux

[ckhordiasma]

removed s390x as a build target due to endianness issue in upstream thinc package. Awaiting explosion/thinc#970

[satyamg1620]

@ckhordiasma Can you switch index-url to https://console.redhat.com/api/pypi/public-rhai/rhoai/3.5/cpu-ubi9/simple/ in requirements.txt as for thinc package for s390x, it works well https://console.redhat.com/api/pypi/public-rhai/rhoai/3.5/cpu-ubi9/simple/thinc/

cc: @modassarrana89-new

[ckhordiasma]

/retest