feat: implement rag-optimize-ci MVP with CI gating and provider adapters

Author: Harmeet Singh

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,20 @@
+---
+name: Bug report
+about: Report a defect in rag-optimize-ci
+title: "[bug] "
+labels: bug
+assignees: ""
+---
+
+## Summary
+
+## Steps to reproduce
+
+## Expected behavior
+
+## Actual behavior
+
+## Environment
+- Python:
+- OS:
+- Command/config used:

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,15 @@
+---
+name: Feature request
+about: Propose a new feature for rag-optimize-ci
+title: "[feat] "
+labels: enhancement
+assignees: ""
+---
+
+## Problem
+
+## Proposed solution
+
+## Alternatives considered
+
+## Acceptance criteria

.github/workflows/ci.yml

@@ -0,0 +1,32 @@
+name: ci
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Unit tests
+        run: python -m unittest discover -s tests -p 'test_*.py'
+
+  sample-run:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Run sample
+        run: |
+          python -m ragopt.cli run --config examples/ragopt.yaml --report artifacts/report.md
+          test -f artifacts/report.md

.github/workflows/release.yml

@@ -0,0 +1,17 @@
+name: release
+
+on:
+  push:
+    tags:
+      - 'v*'
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Test
+        run: python -m unittest discover -s tests -p 'test_*.py'

.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+.pytest_cache/
+.venv/
+artifacts/
+*.pyc
+*.pyo
+*.pyd
+.DS_Store

ARCHITECTURE.md

@@ -0,0 +1,31 @@
+# Architecture
+
+## Flow
+
+1. Load `ragopt.yaml`.
+2. Load benchmark dataset.
+3. For each candidate:
+   - run generation over each query
+   - compute per-case metrics
+   - aggregate metrics
+   - apply hard constraints
+4. Rank candidates by weighted score.
+5. Persist JSON artifact + markdown report.
+6. Optionally post markdown to GitHub PR.
+
+## Modules
+
+- `ragopt/models.py`: schema and result types
+- `ragopt/config.py`: config and dataset loading
+- `ragopt/adapters.py`: generation provider abstraction
+- `ragopt/metrics.py`: metric and scoring functions
+- `ragopt/engine.py`: orchestration and comparison
+- `ragopt/reporting.py`: markdown outputs
+- `ragopt/github.py`: PR comment helper
+- `ragopt/cli.py`: public command interface
+
+## Extension points
+
+- Add providers in `adapters.py`.
+- Add new metrics in `metrics.py` and wire into engine.
+- Add policy checks in `engine.py` compare/run paths.

CASE_STUDY.md

@@ -0,0 +1,31 @@
+# Case Study: rag-optimize-ci
+
+## Problem
+Teams building RAG apps ship prompt/model/retrieval changes quickly but lack a practical CI gate for quality, latency, and cost tradeoffs.
+
+## Solution
+`rag-optimize-ci` runs candidate configs against a benchmark dataset and returns:
+
+- candidate scorecard
+- recommendation with constraints
+- regression verdict vs baseline
+
+## Architecture
+
+- `ragopt/config.py`: config + dataset loading
+- `ragopt/adapters.py`: provider abstraction
+- `ragopt/metrics.py`: quality, groundedness, citation, latency/cost scoring
+- `ragopt/engine.py`: run pipeline, recommendation, regression comparison
+- `ragopt/cli.py`: run/compare/recommend interfaces
+- `action.yml`: GitHub Action integration
+
+## Core tradeoffs
+
+- Started with deterministic metrics and mock adapter to ensure reliability and reproducibility.
+- Deferred complex online judge models and dashboards until CI workflow was complete.
+
+## Impact goals
+
+- Reduce bad RAG releases by catching regressions in PR checks.
+- Standardize model/prompt/retriever decisions with transparent scoring.
+- Enable lean teams to enforce quality without heavy platform investment.

CONTRIBUTING.md

@@ -0,0 +1,27 @@
+# Contributing
+
+## Setup
+
+```bash
+python3 -m unittest discover -s tests -p 'test_*.py'
+```
+
+## Development workflow
+
+1. Open an issue describing the bug/feature.
+2. Add or update tests first where possible.
+3. Keep PRs focused and small.
+4. Ensure tests pass before opening PR.
+
+## Good first issues
+
+- Add real YAML parser support.
+- Add OpenAI adapter behind env vars.
+- Add richer citation matching (semantic not index-only).
+- Add configurable latency percentile scoring.
+
+## Code style
+
+- Type hints for public functions.
+- Keep functions small and deterministic.
+- Add comments only where logic is non-obvious.

DECISIONS.md

@@ -0,0 +1,21 @@
+# Design Decisions
+
+## 1. CI-first before dashboard
+Decision: prioritize GitHub Action + CLI over web UI.
+Reason: strongest immediate value for startup teams and highest hiring signal.
+
+## 2. RAG-only scope
+Decision: optimize only RAG QA workflows in v1.
+Reason: clear product identity beats broad generic evaluation.
+
+## 3. Weighted score + hard constraints
+Decision: rank with weighted score after enforcing quality/latency/cost thresholds.
+Reason: mirrors real shipping criteria where safety rails must be non-negotiable.
+
+## 4. Mock adapter in v1
+Decision: include deterministic mock provider first.
+Reason: keeps project reproducible, low-cost, and testable in CI without external API keys.
+
+## 5. JSON artifacts as source of truth
+Decision: persist run artifacts as JSON in `artifacts/`.
+Reason: easy regression diffing, reproducibility, and auditability.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Harmeet Singh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.