Commit b38bf08
feat(evaluators): new api from runners-api (#242)
## Summary
- Refactored Galileo Luna invocation so Agent Control calls runners-api
directly at `/api/v1/scorers/invoke`.
- Switched Luna auth to internal JWT only, signed with
`GALILEO_API_SECRET_KEY` or `GALILEO_API_SECRET`.
- Made `scorer_id` the required runtime identity; `scorer_label` is
optional metadata and `scorer_version_id` is optional pinning.
- Ensured requests always include `config: {}` and never forward
`Galileo-API-Key` to runners-api.
## Scope
- User-facing/API changes:
- Luna configs now require `scorer_id`; label-only and version-only
configs are rejected.
- Galileo Luna examples/docs now document `GALILEO_RUNNERS_API_URL`,
`GALILEO_LUNA_SCORER_ID`, and internal secret auth.
- Internal changes:
- Replaced API `/scorers/invoke` and `/internal/scorers/invoke` client
routing with runners-api routing.
- Removed Luna auth mode selection and API URL/console URL derivation.
- Added tests for endpoint, JWT payload, scorer ID validation, version
forwarding, API-key stripping, and request body shape.
- Out of scope:
- Agent Control UI changes.
- runners-api/API service implementation changes.
- Scorer hydration, version resolution, caching, retries, or protect
execution logic inside Agent Control.
## Risk and Rollout
- **Compatibility note: `galileo.luna` now requires `scorer_id`.
`scorer_label` is optional display/metadata and `scorer_version_id` is
an optional version pin. Any saved Luna controls that only have
`scorer_label` or `scorer_version_id` must be re-saved through the UI or
manually updated to include the resolved `scorer_id` before they can be
evaluated with this version. The enterprise UI already resolves the
selected scorer and saves `scorer_id`.**
- Risk level: medium
- Rollback plan:
- Revert this PR to restore the previous API-mediated Luna invocation
path.
- If rollout exposes saved controls without `scorer_id`, migrate those
controls or temporarily roll back before retrying cutover.
## Testing
- [x] Added or updated automated tests
- [ ] Ran `make check` (targeted package validation was run instead)
- [x] Manually verified behavior
Targeted validation:
- `make -C evaluators/contrib/galileo test` -> 86 passed
- `make -C evaluators/contrib/galileo lint` -> passed
- `make -C evaluators/contrib/galileo typecheck` -> passed
- `git diff --check` -> clean
## Checklist
- [x] Linked issue/spec: RFC on direct runners-api scorer invoke
- [x] Updated docs/examples for user-facing changes
- [x] Included required follow-up tasks: confirm/migrate any legacy
saved Luna controls missing `scorer_id`
---------
Co-authored-by: abhinav-galileo <abhinav-galileo@users.noreply.github.com>1 parent b374734 commit b38bf08
13 files changed
Lines changed: 601 additions & 743 deletions
File tree
- evaluators/contrib/galileo
- src/agent_control_evaluator_galileo/luna
- tests
- examples
- galileo_luna
- models
- server/tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| |||
292 | 292 | | |
293 | 293 | | |
294 | 294 | | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
295 | 340 | | |
296 | 341 | | |
297 | 342 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
10 | | - | |
11 | | - | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
12 | 16 | | |
13 | 17 | | |
14 | 18 | | |
| |||
0 commit comments