refactor(evaluators)!: reorganize into builtin + extra tiers

Split evaluators into two packages:
- builtin (`agent-control-evaluators`): Core infrastructure + regex, list, json, sql
- extra/galileo (`agent-control-evaluator-galileo`): Luna2 evaluator (calls external API)

BREAKING CHANGES:
- Luna2 import path changed from `agent_control_evaluators.galileo_luna2`
  to `agent_control_evaluator_galileo.luna2`
- External evaluator names use dot notation instead of slash
  (e.g., `galileo.luna2` instead of `galileo/luna2`)
- SDK and server now depend on `agent-control-evaluators` as a runtime
  dependency (not vendored) to avoid duplicate module conflicts

Key changes:
- Move builtin evaluators to `evaluators/builtin/`
- Create `evaluators/extra/galileo/` as separate package
- Add entry points for plugin discovery (`agent_control.evaluators`)
- Update workspace to include only builtin (extras excluded for perf)
- Add CI workflow for testing extra packages
- Add template scaffold for creating new evaluator packages
- Server build script no longer vendors evaluators

Author: abhinav-galileo

.github/workflows/test-extras.yml

@@ -0,0 +1,52 @@
+name: Test Extras
+
+on:
+  push:
+    paths:
+      # Trigger on extra changes
+      - 'evaluators/extra/**'
+      # Also trigger on core changes that could break extras
+      - 'evaluators/builtin/**'
+      - 'models/**'
+      - 'engine/**'
+      - 'server/**'
+      - 'sdks/python/**'
+  pull_request:
+    paths:
+      - 'evaluators/extra/**'
+      - 'evaluators/builtin/**'
+      - 'models/**'
+      - 'engine/**'
+      - 'server/**'
+      - 'sdks/python/**'
+
+jobs:
+  test-galileo:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup uv and Python
+        uses: astral-sh/setup-uv@v3
+        with:
+          python-version: "3.12"
+
+      - name: Sync workspace
+        run: make sync
+
+      - name: Install galileo extra
+        run: cd evaluators/extra/galileo && uv pip install -e .
+
+      - name: Lint galileo
+        run: cd evaluators/extra/galileo && uv run ruff check --config ../../../pyproject.toml src/
+
+      - name: Typecheck galileo
+        run: cd evaluators/extra/galileo && uv run mypy --config-file ../../../pyproject.toml src/
+
+      - name: Test galileo
+        run: cd evaluators/extra/galileo && uv run pytest
+
+      - name: Verify SDK integration
+        run: |
+          cd sdks/python
+          uv run pytest tests/test_luna2_smoke.py

AGENTS.md

@@ -26,7 +26,8 @@ Forwarded targets:
 - `engine/`: **control evaluation engine and evaluator system** — all evaluation logic, evaluator discovery, and evaluator orchestration lives here (`engine/src/agent_control_engine/`)
 - `server/`: FastAPI server (`server/src/agent_control_server/`)
 - `sdks/python/`: Python SDK — uses engine for evaluation (`sdks/python/src/agent_control/`)
-- `evaluators/`: evaluator implementations (`evaluators/src/agent_control_evaluators/`)
+- `evaluators/builtin/`: builtin evaluator implementations (`evaluators/builtin/src/agent_control_evaluators/`)
+- `evaluators/extra/`: optional evaluator packages (e.g., `evaluators/extra/galileo/`)
 - `ui/`: Nextjs based web app to manage agent controls 
 - `examples/`: runnable examples (ruff has relaxed import rules here)
 
@@ -66,13 +67,19 @@ All testing guidance (including “behavior changes require tests”) lives in `
   4) add SDK wrapper in `sdks/python/src/agent_control/`
   5) add tests (server + SDK) and update docs/examples if user-facing
 
-- Add a new evaluator:
-  1) implement evaluator class extending `Evaluator` in `evaluators/src/agent_control_evaluators/`
+- Add a new builtin evaluator:
+  1) implement evaluator class extending `Evaluator` in `evaluators/builtin/src/agent_control_evaluators/`
   2) use `@register_evaluator` decorator (from `agent_control_evaluators`)
-  3) add entry point in `evaluators/pyproject.toml` for auto-discovery
-  4) add tests in the evaluators package
+  3) add entry point in `evaluators/builtin/pyproject.toml` for auto-discovery
+  4) add tests in the evaluators/builtin package
   5) evaluator is automatically available to server and SDK via `discover_evaluators()`
 
+- Add an external evaluator package:
+  1) copy `evaluators/extra/template/` as a starting point
+  2) implement evaluator class extending `Evaluator` from `agent_control_evaluators`
+  3) add entry point using `org.name` format (e.g., `galileo.luna2`)
+  4) package is discovered automatically when installed alongside agent-control
+
 ## Git/PR workflow
 
 - Branch naming: `feature/...`, `fix/...`, `refactor/...`

Makefile

@@ -1,16 +1,18 @@
-.PHONY: help sync test test-models test-sdk lint lint-fix typecheck check build build-models build-server build-sdk publish publish-models publish-server publish-sdk hooks-install hooks-uninstall prepush
+.PHONY: help sync test test-models test-sdk lint lint-fix typecheck check build build-models build-server build-sdk publish publish-models publish-server publish-sdk hooks-install hooks-uninstall prepush evaluators-test evaluators-lint evaluators-lint-fix evaluators-typecheck evaluators-build
 
 # Workspace package names
 PACK_MODELS := agent-control-models
 PACK_SERVER := agent-control-server
 PACK_SDK    := agent-control
 PACK_ENGINE := agent-control-engine
+PACK_EVALUATORS := agent-control-evaluators
 
 # Directories
 MODELS_DIR := models
 SERVER_DIR := server
 SDK_DIR    := sdks/python
 ENGINE_DIR := engine
+EVALUATORS_DIR := evaluators/builtin
 
 help:
 	@echo "Agent Control - Makefile commands"
@@ -56,7 +58,7 @@ sync:
 # Test
 # ---------------------------
 
-test: server-test engine-test sdk-test
+test: server-test engine-test sdk-test evaluators-test
 
 # Run tests, lint, and typecheck
 check: test lint typecheck
@@ -65,17 +67,17 @@ check: test lint typecheck
 # Quality
 # ---------------------------
 
-lint: engine-lint
+lint: engine-lint evaluators-lint
 	uv run --package $(PACK_MODELS) ruff check --config pyproject.toml models/src
 	uv run --package $(PACK_SERVER) ruff check --config pyproject.toml server/src
 	uv run --package $(PACK_SDK) ruff check --config pyproject.toml sdks/python/src
 
-lint-fix: engine-lint-fix
+lint-fix: engine-lint-fix evaluators-lint-fix
 	uv run --package $(PACK_MODELS) ruff check --config pyproject.toml --fix models/src
 	uv run --package $(PACK_SERVER) ruff check --config pyproject.toml --fix server/src
 	uv run --package $(PACK_SDK) ruff check --config pyproject.toml --fix sdks/python/src
 
-typecheck: engine-typecheck
+typecheck: engine-typecheck evaluators-typecheck
 	uv run --package $(PACK_MODELS) mypy --config-file pyproject.toml models/src
 	uv run --package $(PACK_SERVER) mypy --config-file pyproject.toml server/src
 	uv run --package $(PACK_SDK) mypy --config-file pyproject.toml sdks/python/src
@@ -84,7 +86,7 @@ typecheck: engine-typecheck
 # Build / Publish
 # ---------------------------
 
-build: build-models build-server build-sdk engine-build
+build: build-models build-server build-sdk engine-build evaluators-build
 
 build-models:
 	cd $(MODELS_DIR) && uv build
@@ -130,6 +132,21 @@ engine-%:
 sdk-%:
 	$(MAKE) -C $(SDK_DIR) $(patsubst sdk-%,%,$@)
 
+evaluators-test:
+	$(MAKE) -C $(EVALUATORS_DIR) test
+
+evaluators-lint:
+	$(MAKE) -C $(EVALUATORS_DIR) lint
+
+evaluators-lint-fix:
+	$(MAKE) -C $(EVALUATORS_DIR) lint-fix
+
+evaluators-typecheck:
+	$(MAKE) -C $(EVALUATORS_DIR) typecheck
+
+evaluators-build:
+	$(MAKE) -C $(EVALUATORS_DIR) build
+
 .PHONY: server-%
 server-%:
 	$(MAKE) -C $(SERVER_DIR) $(patsubst server-%,%,$@)

evaluators/README.md

@@ -0,0 +1,33 @@
+.PHONY: help sync test lint lint-fix typecheck build publish
+
+PACKAGE := agent-control-evaluators
+
+help:
+	@echo "Agent Control Evaluators - Makefile commands"
+	@echo ""
+	@echo "  make test            - run pytest"
+	@echo "  make lint            - run ruff check"
+	@echo "  make lint-fix        - run ruff check --fix"
+	@echo "  make typecheck       - run mypy"
+	@echo "  make build           - build package"
+
+sync:
+	uv sync
+
+test:
+	uv run pytest --cov=src --cov-report=xml:../../coverage-evaluators.xml -q
+
+lint:
+	uv run ruff check --config ../../pyproject.toml src/
+
+lint-fix:
+	uv run ruff check --config ../../pyproject.toml --fix src/
+
+typecheck:
+	uv run mypy --config-file ../../pyproject.toml src/
+
+build:
+	uv build
+
+publish:
+	uv publish

evaluators/builtin/Makefile

@@ -0,0 +1,48 @@
+# Agent Control Evaluators
+
+Built-in evaluators for agent-control.
+
+## Installation
+
+```bash
+pip install agent-control-evaluators
+```
+
+## Available Evaluators
+
+| Name | Description |
+|------|-------------|
+| `regex` | Regular expression pattern matching |
+| `list` | List-based value matching (allow/deny) |
+| `json` | JSON validation (schema, required fields, types) |
+| `sql` | SQL query validation |
+
+## Usage
+
+Evaluators are automatically discovered via Python entry points:
+
+```python
+from agent_control_evaluators import discover_evaluators, list_evaluators
+
+# Load all available evaluators
+discover_evaluators()
+
+# See what's available
+print(list_evaluators())
+# {'regex': <class 'RegexEvaluator'>, 'list': ..., 'json': ..., 'sql': ...}
+```
+
+## External Evaluators
+
+Additional evaluators are available via separate packages:
+
+- `agent-control-evaluator-galileo` - Galileo Luna2 evaluator
+
+Install convenience extras:
+```bash
+pip install agent-control-evaluators[galileo]
+```
+
+## Creating Custom Evaluators
+
+See [AGENTS.md](../../AGENTS.md) for guidance on creating new evaluators.

evaluators/builtin/README.md

@@ -1,7 +1,7 @@
 [project]
 name = "agent-control-evaluators"
-version = "2.1.0"
-description = "Evaluator implementations for agent-control"
+version = "3.0.0"
+description = "Builtin evaluators for agent-control"
 readme = "README.md"
 requires-python = ">=3.12"
 license = { text = "Apache-2.0" }
@@ -15,16 +15,15 @@ dependencies = [
 ]
 
 [project.optional-dependencies]
-luna2 = ["httpx>=0.24.0"]
-all = ["httpx>=0.24.0"]
+# NOTE: galileo extra commented out during local dev - package not yet on PyPI
+# galileo = ["agent-control-evaluator-galileo>=3.0.0"]
 dev = ["pytest>=8.0.0", "pytest-asyncio>=0.23.0"]
 
 [project.entry-points."agent_control.evaluators"]
 regex = "agent_control_evaluators.regex:RegexEvaluator"
 list = "agent_control_evaluators.list:ListEvaluator"
 json = "agent_control_evaluators.json:JSONEvaluator"
 sql = "agent_control_evaluators.sql:SQLEvaluator"
-"galileo/luna2" = "agent_control_evaluators.galileo_luna2:Luna2Evaluator"
 
 [build-system]
 requires = ["hatchling"]

evaluators/pyproject.toml evaluators/builtin/pyproject.tomlevaluators/pyproject.toml renamed to evaluators/builtin/pyproject.toml

@@ -1,6 +1,6 @@
 """Agent Control Evaluators.
 
-This package contains evaluator implementations for agent-control.
+This package contains builtin evaluator implementations for agent-control.
 Built-in evaluators (regex, list, json, sql) are registered automatically on import.
 
 Available evaluators:
@@ -10,15 +10,12 @@
         - json: JSON validation
         - sql: SQL query validation
 
-    External (provider/name format):
-        - galileo/luna2: Galileo Luna-2 runtime protection
-          (pip install agent-control-evaluators[luna2])
-
 Naming convention:
     - Built-in: "regex", "list", "json", "sql"
-    - External: "provider/name" (e.g., "galileo/luna2")
+    - External: "provider.name" (e.g., "galileo.luna2")
     - Agent-scoped: "agent:name" (custom code deployed with agent)
 
+External evaluators are installed via separate packages (e.g., agent-control-evaluator-galileo).
 Custom evaluators are Evaluator classes deployed with the engine.
 Their schemas are registered via initAgent for validation purposes.
 """
@@ -45,7 +42,7 @@
 from agent_control_evaluators.regex import RegexEvaluator, RegexEvaluatorConfig
 from agent_control_evaluators.sql import SQLEvaluator, SQLEvaluatorConfig
 
-__version__ = "0.1.0"
+__version__ = "3.0.0"
 
 __all__ = [
     # Core infrastructure