feat(evaluators): add defenseclaw evaluator (#248)

## Summary

Added two global external evaluator integrations:
- `defenseclaw.rule_pack`
- `defenseclaw.opa_policy`

- Added strict, typed Pydantic configuration models and distinct JSON
Schemas.
- Added dedicated Agent Control UI forms with canonical serialization of
hidden protocol fields.
- Both evaluator execution paths are intentionally no-ops and do not
install or invoke DefenseClaw OSS packages.

## Scope

**User-facing/API changes:**
- Both evaluators appear independently in `GET /api/v1/evaluators`.
- Added typed UI forms for rule-pack and OPA-policy configuration.
- Rule-pack forms support adding and removing rules.
- Fixed protocol fields remain hidden in Form mode but are always
serialized.
- DefenseClaw configurations are preserved in `get_server_controls()`
output.

**Internal changes:**
- Added the `agent-control-evaluator-defenseclaw` contrib package.
- Added evaluator entry-point registration and package/release wiring.
- Added strict nested validation, including unique rule IDs and
confidence bounds.
- Added default configurations and UI registry entries.
- Added focused backend and UI tests.

**Out of scope:**
- DefenseClaw transport, authentication, or provider invocation.
- Installing or calling DefenseClaw OSS packages.
- Changes to the separate `~/code/ui` Galileo UI repository.
- Orbit production packaging changes.

## Risk and Rollout

**Risk level:** medium

**Primary risk:**
- The evaluator package must be installed in the server runtime for
catalog discovery and server execution.
- It must also be installed in SDK runtimes for controls using
`execution: "sdk"`.

**Rollout requirements:**
- Publish `agent-control-evaluator-defenseclaw`.
- Add the package to the Orbit Agent Control image dependencies.
- Verify both evaluator identifiers appear in `GET /api/v1/evaluators`
before activating controls.

**Rollback plan:**
- Remove or disable controls referencing the DefenseClaw evaluator
names.
- Remove the DefenseClaw package from runtime dependencies.
- Revert the evaluator registry and UI form registration changes.

## Testing

- [x] Added or updated automated tests
  - Ran `make check` (full repository check was not run).
  - Ran 26 focused Python tests successfully.
  - Ran Ruff and mypy successfully.
  - Ran TypeScript, ESLint, and Prettier checks successfully.
- Verified contrib package wiring, evaluator registration, and server
catalog schemas.
- Playwright could not run in the sandbox because it could not bind the
local UI server port.

- [x] Manually verified behavior
- Registration and generated catalog schemas were verified
programmatically.
- Full browser-based Form/JSON interaction should be verified locally
using the supplied testing guide.

## Checklist

- [ ] **Linked issue/spec**
  - Technical spec: `defenseclaw-external-evaluators-technical-spec.md`
  - Add the appropriate issue/spec URL before merging.

- [x] **Updated docs/examples for user-facing changes**
  - Added the contrib package README.
  - Created a first-time local setup and testing guide.

- [x] **Included required follow-up tasks**
  - Add DefenseClaw packaging to `~/code/orbit`.
  - Implement the corresponding forms in `~/code/ui`.
  - Run the full Playwright suite.
- Install the evaluator package in every SDK runtime that evaluates
DefenseClaw controls.

Author: namrataghadi-galileo

.gitignore

@@ -73,6 +73,9 @@ server/.generated/
 tmp/
 temp/
 *.tmp
+.codex-doc-qa/
+.codex-doc-work/
+.pnpm-store/
 
 # OS
 .DS_Store
@@ -87,4 +90,4 @@ Thumbs.db
 # Local notes
 rearch_plan.md
 
-node_modules
+node_modules

evaluators/builtin/pyproject.toml

@@ -18,6 +18,7 @@ dependencies = [
 galileo = ["agent-control-evaluator-galileo>=7.5.0"]
 budget = ["agent-control-evaluator-budget>=7.5.0"]
 cisco = ["agent-control-evaluator-cisco>=7.5.0"]
+defenseclaw = ["agent-control-evaluator-defenseclaw>=8.2.0"]
 dev = ["pytest>=8.0.0", "pytest-asyncio>=0.23.0"]
 
 [project.entry-points."agent_control.evaluators"]
@@ -39,3 +40,4 @@ agent-control-models = { workspace = true }
 agent-control-evaluator-galileo = { path = "../contrib/galileo", editable = true }
 agent-control-evaluator-budget = { path = "../contrib/budget", editable = true }
 agent-control-evaluator-cisco = { path = "../contrib/cisco", editable = true }
+agent-control-evaluator-defenseclaw = { path = "../contrib/defenseclaw", editable = true }

evaluators/builtin/tests/test_contrib_packages.py

@@ -39,6 +39,7 @@ def test_discover_contrib_packages_returns_expected_metadata() -> None:
     assert [(package.name, package.package, package.extra) for package in packages] == [
         ("budget", "agent-control-evaluator-budget", "budget"),
         ("cisco", "agent-control-evaluator-cisco", "cisco"),
+        ("defenseclaw", "agent-control-evaluator-defenseclaw", "defenseclaw"),
         ("galileo", "agent-control-evaluator-galileo", "galileo"),
     ]
 

evaluators/contrib/defenseclaw/Makefile

@@ -0,0 +1,33 @@
+.PHONY: help sync test lint lint-fix typecheck check build
+
+PACKAGE := agent-control-evaluator-defenseclaw
+
+help:
+	@echo "Agent Control Evaluator - DefenseClaw - Makefile commands"
+	@echo "  make sync       - sync package dependencies"
+	@echo "  make test       - run pytest"
+	@echo "  make lint       - run ruff check"
+	@echo "  make lint-fix   - run ruff check --fix"
+	@echo "  make typecheck  - run mypy"
+	@echo "  make check      - run tests, lint, and typecheck"
+	@echo "  make build      - build package"
+
+sync:
+	uv sync
+
+test:
+	uv run --with pytest --with pytest-asyncio --with pytest-cov --package $(PACKAGE) pytest tests --cov=src --cov-report=xml:../../../coverage-evaluators-defenseclaw.xml -q
+
+lint:
+	uv run --with ruff --package $(PACKAGE) ruff check --config ../../../pyproject.toml src/ tests/
+
+lint-fix:
+	uv run --with ruff --package $(PACKAGE) ruff check --config ../../../pyproject.toml --fix src/ tests/
+
+typecheck:
+	uv run --with mypy --package $(PACKAGE) mypy --config-file ../../../pyproject.toml src/
+
+check: test lint typecheck
+
+build:
+	uv build

evaluators/contrib/defenseclaw/README.md

@@ -0,0 +1,16 @@
+# DefenseClaw evaluators
+
+This package exposes two global external evaluators for Agent Control:
+
+- `defenseclaw.rule_pack` validates a versioned DefenseClaw rule pack.
+- `defenseclaw.opa_policy` validates a versioned DefenseClaw OPA policy configuration.
+
+Install both evaluators with:
+
+```bash
+pip install "agent-control-evaluators[defenseclaw]"
+```
+
+The configuration contracts and JSON Schemas are implemented and discoverable. Both evaluator
+classes intentionally execute as no-ops: they return `matched=False` without contacting or
+installing any DefenseClaw runtime or OSS package.

evaluators/contrib/defenseclaw/pyproject.toml

@@ -0,0 +1,42 @@
+[project]
+name = "agent-control-evaluator-defenseclaw"
+version = "8.2.0"
+description = "DefenseClaw evaluators for agent-control"
+readme = "README.md"
+requires-python = ">=3.12"
+license = { text = "Apache-2.0" }
+authors = [{ name = "Agent Control Team" }]
+dependencies = [
+    "agent-control-evaluators>=8.2.0",
+    "agent-control-models>=8.2.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-asyncio>=0.23.0",
+    "pytest-cov>=4.0.0",
+    "ruff>=0.1.0",
+    "mypy>=1.8.0",
+]
+
+[project.entry-points."agent_control.evaluators"]
+"defenseclaw.rule_pack" = "agent_control_evaluator_defenseclaw.rule_pack:DefenseClawRulePackEvaluator"
+"defenseclaw.opa_policy" = "agent_control_evaluator_defenseclaw.opa_policy:DefenseClawOpaPolicyEvaluator"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/agent_control_evaluator_defenseclaw"]
+
+[tool.uv.sources]
+agent-control-evaluators = { path = "../../builtin", editable = true }
+agent-control-models = { path = "../../../models", editable = true }
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.2",
+    "pytest-asyncio>=1.3.0",
+]

evaluators/contrib/defenseclaw/src/agent_control_evaluator_defenseclaw/__init__.py

@@ -0,0 +1,28 @@
+"""Agent Control DefenseClaw external evaluators."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("agent-control-evaluator-defenseclaw")
+except PackageNotFoundError:
+    __version__ = "0.0.0.dev"
+
+from .common import Severity
+from .opa_policy import DefenseClawOpaPolicyConfig, DefenseClawOpaPolicyEvaluator, OpaPolicy
+from .rule_pack import (
+    DefenseClawRulePackConfig,
+    DefenseClawRulePackEvaluator,
+    RuleConfig,
+    RulePack,
+)
+
+__all__ = [
+    "DefenseClawOpaPolicyConfig",
+    "DefenseClawOpaPolicyEvaluator",
+    "DefenseClawRulePackConfig",
+    "DefenseClawRulePackEvaluator",
+    "OpaPolicy",
+    "RuleConfig",
+    "RulePack",
+    "Severity",
+]

evaluators/contrib/defenseclaw/src/agent_control_evaluator_defenseclaw/common.py

@@ -0,0 +1,22 @@
+"""Shared DefenseClaw evaluator types and runtime behavior."""
+
+from __future__ import annotations
+
+from typing import Annotated, Literal
+
+from agent_control_models import EvaluatorResult
+from pydantic import ConfigDict, StringConstraints
+
+Severity = Literal["LOW", "MEDIUM", "HIGH", "CRITICAL"]
+NonEmptyString = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1)]
+
+STRICT_PROVIDER_CONFIG = ConfigDict(extra="forbid")
+
+
+def no_op_result() -> EvaluatorResult:
+    """Return the intentional no-op evaluation result."""
+    return EvaluatorResult(
+        matched=False,
+        confidence=1.0,
+        message="DefenseClaw configuration accepted; evaluator execution is a no-op",
+    )

evaluators/contrib/defenseclaw/src/agent_control_evaluator_defenseclaw/opa_policy/__init__.py

@@ -0,0 +1,6 @@
+"""DefenseClaw OPA-policy evaluator exports."""
+
+from .config import DefenseClawOpaPolicyConfig, OpaPolicy
+from .evaluator import DefenseClawOpaPolicyEvaluator
+
+__all__ = ["DefenseClawOpaPolicyConfig", "DefenseClawOpaPolicyEvaluator", "OpaPolicy"]

evaluators/contrib/defenseclaw/src/agent_control_evaluator_defenseclaw/opa_policy/config.py

@@ -0,0 +1,34 @@
+"""Typed configuration for the DefenseClaw OPA-policy evaluator."""
+
+from __future__ import annotations
+
+from typing import Literal
+
+from agent_control_evaluators import EvaluatorConfig
+from pydantic import BaseModel
+
+from ..common import STRICT_PROVIDER_CONFIG, Severity
+
+
+class OpaPolicy(BaseModel):
+    """DefenseClaw OPA policy settings supported by the v1 contract.
+
+    See https://cisco-ai-defense.github.io/docs/defenseclaw/policy for the policy contract.
+    """
+
+    model_config = STRICT_PROVIDER_CONFIG
+
+    domain: Literal["guardrail"] = "guardrail"
+    block_at: Severity = "HIGH"
+    alert_at: Severity = "MEDIUM"
+    cisco_trust_level: Literal["full"] = "full"
+
+
+class DefenseClawOpaPolicyConfig(EvaluatorConfig):
+    """Configuration envelope for `defenseclaw.opa_policy`."""
+
+    # Reject unknown configuration fields so malformed DefenseClaw payloads fail validation.
+    model_config = STRICT_PROVIDER_CONFIG
+
+    schema_version: Literal[1] = 1
+    policy: OpaPolicy