Skip to content

Commit b38bf08

Browse files
feat(evaluators): new api from runners-api (#242)
## Summary - Refactored Galileo Luna invocation so Agent Control calls runners-api directly at `/api/v1/scorers/invoke`. - Switched Luna auth to internal JWT only, signed with `GALILEO_API_SECRET_KEY` or `GALILEO_API_SECRET`. - Made `scorer_id` the required runtime identity; `scorer_label` is optional metadata and `scorer_version_id` is optional pinning. - Ensured requests always include `config: {}` and never forward `Galileo-API-Key` to runners-api. ## Scope - User-facing/API changes: - Luna configs now require `scorer_id`; label-only and version-only configs are rejected. - Galileo Luna examples/docs now document `GALILEO_RUNNERS_API_URL`, `GALILEO_LUNA_SCORER_ID`, and internal secret auth. - Internal changes: - Replaced API `/scorers/invoke` and `/internal/scorers/invoke` client routing with runners-api routing. - Removed Luna auth mode selection and API URL/console URL derivation. - Added tests for endpoint, JWT payload, scorer ID validation, version forwarding, API-key stripping, and request body shape. - Out of scope: - Agent Control UI changes. - runners-api/API service implementation changes. - Scorer hydration, version resolution, caching, retries, or protect execution logic inside Agent Control. ## Risk and Rollout - **Compatibility note: `galileo.luna` now requires `scorer_id`. `scorer_label` is optional display/metadata and `scorer_version_id` is an optional version pin. Any saved Luna controls that only have `scorer_label` or `scorer_version_id` must be re-saved through the UI or manually updated to include the resolved `scorer_id` before they can be evaluated with this version. The enterprise UI already resolves the selected scorer and saves `scorer_id`.** - Risk level: medium - Rollback plan: - Revert this PR to restore the previous API-mediated Luna invocation path. - If rollout exposes saved controls without `scorer_id`, migrate those controls or temporarily roll back before retrying cutover. ## Testing - [x] Added or updated automated tests - [ ] Ran `make check` (targeted package validation was run instead) - [x] Manually verified behavior Targeted validation: - `make -C evaluators/contrib/galileo test` -> 86 passed - `make -C evaluators/contrib/galileo lint` -> passed - `make -C evaluators/contrib/galileo typecheck` -> passed - `git diff --check` -> clean ## Checklist - [x] Linked issue/spec: RFC on direct runners-api scorer invoke - [x] Updated docs/examples for user-facing changes - [x] Included required follow-up tasks: confirm/migrate any legacy saved Luna controls missing `scorer_id` --------- Co-authored-by: abhinav-galileo <abhinav-galileo@users.noreply.github.com>
1 parent b374734 commit b38bf08

13 files changed

Lines changed: 601 additions & 743 deletions

File tree

README.md

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ Enforce runtime guardrails through a centralized control layer—configure once
4141

4242
## Quick Start
4343

44-
Prerequisites: Docker and Python 3.12+.
44+
Prerequisites: Docker (or Podman, see [Podman setup](#podman-setup)) and Python 3.12+.
4545

4646
Quick start flow:
4747

@@ -292,6 +292,51 @@ Explore working examples for popular frameworks.
292292
- [AWS Strands](examples/strands_agents/) - protect Strands workflows and tool calls
293293
- [Google ADK Decorator](examples/google_adk_decorator/) - add controls with `@control()`
294294

295+
## Podman Setup
296+
297+
If Docker Desktop is not available, you can use [Podman](https://podman-desktop.io) as a drop-in replacement. No changes to repo files are needed — the setup below makes `docker` and `docker compose` transparently resolve to Podman.
298+
299+
**One-time setup:**
300+
301+
1. Install [Podman Desktop](https://podman-desktop.io) and create a machine from its UI (start it before continuing).
302+
303+
2. Install `podman-compose`:
304+
305+
```bash
306+
brew install podman-compose
307+
```
308+
309+
3. Create a `docker` shim that routes `docker compose` to `podman-compose` and everything else to `podman`:
310+
311+
```bash
312+
mkdir -p ~/.local/bin
313+
cat > ~/.local/bin/docker << 'EOF'
314+
#!/bin/zsh
315+
if [[ "$1" == "compose" ]]; then
316+
shift
317+
exec podman-compose "$@"
318+
fi
319+
exec podman "$@"
320+
EOF
321+
chmod +x ~/.local/bin/docker
322+
```
323+
324+
4. Add `~/.local/bin` early in your PATH (if not already):
325+
326+
```bash
327+
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
328+
source ~/.zshrc
329+
```
330+
331+
**Verify:**
332+
333+
```bash
334+
docker ps
335+
docker compose version
336+
```
337+
338+
After this, all existing `docker`/`docker compose` commands and `make` targets work as-is.
339+
295340
## How It Works
296341

297342
![Agent Control Architecture](docs/images/Architecture.png)

evaluators/contrib/galileo/README.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,13 @@ Integration package for Galileo Luna evaluator.
66

77
The `galileo.luna2` evaluator ID has been removed. Existing controls that use
88
`galileo.luna2` should migrate to `galileo.luna` and update their evaluator
9-
configuration to the direct Luna scorer fields (`scorer_label`, `scorer_id`, or
10-
`scorer_version_id`, plus `threshold` and `operator`). If you still need the
11-
legacy Luna2 evaluator, pin `agent-control-evaluator-galileo <8`.
9+
configuration to use the direct Luna scorer fields. `scorer_id` is required;
10+
`scorer_label` and `scorer_version_id` are optional. The evaluator calls the
11+
URL configured by `GALILEO_LUNA_INVOKE_URL`; the target must support the Luna
12+
scorer invoke request/response contract and internal Galileo secret auth. Also
13+
set `threshold` and `operator` as needed. If you still need the legacy Luna2
14+
evaluator, pin
15+
`agent-control-evaluator-galileo <8`.
1216

1317
## Install
1418

0 commit comments

Comments
 (0)