Skip to content

Commit 573602a

Browse files
asamal4asamal4
committed
specs: agentic openshift lightspeed evaluation
1 parent 72eeedd commit 573602a

1 file changed

Lines changed: 250 additions & 0 deletions

File tree

‎specs/agentic_openshift_lightspeed_evaluation.md‎

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
# Event-Driven Agent Evaluation: Openshift Agentic Lightspeed
2+
3+
Asutosh Samal (@asamal4), Carmelo Riolo (@rioloc), Alberto Falossi (@falox)
4+
Rev. 0.1 (Apr 27, 2026)
5+
Rev. 0.2 (May 12, 2026 — agents config override)
6+
7+
**Scope:** Openshift Agentic Lightspeed integration, generic agent framework, evaluation mechanisms
8+
9+
## 1\. Overview
10+
11+
lightspeed-eval assumes synchronous HTTP request-response. Openshift Agentic Lightspeed and similar systems are event-driven: CRDs applied, workflows executed, cluster state changed. The "answer" is a trajectory of events and a final cluster state.
12+
13+
**Solution:** Introduce a generic agents configuration layer. HTTP API becomes one agent type. Openshift Agentic Lightspeed is the first non-HTTP agent. The framework evaluates agent results using deterministic assertions (parity with operator eval) and, in future, LLM-as-judge.
14+
15+
## 2\. Key Decisions & Open Questions
16+
17+
| \# | Decision | Choice |
18+
| :---- | :---- | :---- |
19+
| D1 | Config structure | agents: top-level with default.agent (selection) \+ default.agent\_config (fallback properties) and named agent definitions. Same `agent` \+ `agent_config` field names in eval\_data for consistency. |
20+
| D2 | CRD operations approach | *K8s Python client OR kubectl/oc subprocess ??* |
21+
| D3 | Proposal input | Inline spec dict in eval\_data (same fields as operator EvalSuite) |
22+
| D4 | Evaluation metric | Single custom:proposal\_status with all assertion checks |
23+
| D5 | query for Openshift Agentic Lightspeed | request accepted as alias for query; driver injects into proposal CR |
24+
| D6 | LLM-as-judge | Needed for behavioural testing \- TBD |
25+
| D7 | Polling vs Watch | Simple polling |
26+
| D8 | Backward compatibility | api: block auto-migrates to agents |
27+
28+
## 3\. Configuration Architecture
29+
30+
### 3.1 Agents Block (system.yaml)
31+
32+
```
33+
agents:
34+
enabled: true # Master switch — disables all agent execution when false
35+
36+
default:
37+
agent: ols_api # Used when eval_data doesn't specify
38+
agent_config: # Fallback properties for agents that don't set their own
39+
timeout: 600
40+
retry: 3
41+
42+
ols_api:
43+
type: http_api
44+
api_base: http://localhost:8080
45+
endpoint_type: streaming
46+
provider: openai
47+
model: gpt-4o
48+
49+
openshift_agentic_lightspeed:
50+
type: kubernetes_crd
51+
kubeconfig: ${KUBECONFIG}
52+
namespace: openshift-lightspeed
53+
auto_approve: true
54+
cleanup_proposals: true # Delete eval proposals after status captured
55+
timeout: 900 # Explicit — ignores default.agent_config.timeout
56+
poll_interval: 2
57+
```
58+
59+
**Structure:** `enabled` is a master switch at agents level — controls whether any agent execution happens (vs using pre-filled data). In future, per-agent `enabled` can be considered to allow enabling/disabling individual agents independently. `default` holds agent selection (`agent`) and fallback properties (`agent_config`). Everything else with a `type:` field is an agent definition. CRD coordinates (crd\_group, crd\_version, crd\_plural, crd\_kind) are configurable for other CRD-based agents.
60+
61+
**`agent` + `agent_config` consistency:** The same field names are used in both system.yaml (`default.agent`, `default.agent_config`) and eval\_data (`agent`, `agent_config`) for clarity.
62+
63+
### 3.2 Config Resolution
64+
65+
```
66+
eval_data.agent_config > agents.<name> typed fields > default.agent_config
67+
(highest priority) (agent-specific) (fallback for unset fields only)
68+
```
69+
70+
**Note:** `default.agent_config` only applies to fields the agent didn't explicitly set. This prevents system defaults from silently overriding agent-specific values.
71+
72+
### 3.3 Backward Compatibility
73+
74+
The existing api: block should auto-migrate to agents: via a Pydantic model\_validator. Migration only fires when agents: is absent. When both exist, agents: takes precedence.
75+
76+
```
77+
# Migration output
78+
agents:
79+
enabled: true/false # From api.enabled
80+
default:
81+
agent: api # Key name, distinct from type
82+
api: # Named agent definition
83+
type: http_api # Type is separate
84+
api_base: ...
85+
```
86+
87+
### 3.4 eval\_data Agent Selection
88+
89+
```
90+
conversation_groups:
91+
- conversation_group_id: legacy_tests # No agent = uses default
92+
turns: [...]
93+
94+
- conversation_group_id: openshift_agentic_lightspeed_tests # Explicit agent
95+
agent: openshift_agentic_lightspeed
96+
turns: [...]
97+
98+
- conversation_group_id: openshift_agentic_lightspeed_custom # Per-group config override
99+
agent: openshift_agentic_lightspeed
100+
agent_config:
101+
namespace: custom-namespace
102+
timeout: 1200
103+
turns: [...]
104+
```
105+
106+
## 4\. Agent Driver Architecture
107+
108+
### 4.1 Driver Interface
109+
110+
```
111+
class AgentDriver(ABC):
112+
@abstractmethod
113+
def execute_turn(self, turn_data: TurnData, config: dict) -> Optional[str]:
114+
"""Enrich turn_data in-place. Return error message or None."""
115+
...
116+
117+
@abstractmethod
118+
def validate_config(self, config: dict) -> None: ...
119+
```
120+
121+
**Caching:** There is a plan to move caching configuration to a framework level, applied uniformly to all components (agents, judge LLM, embedding model) rather than at individual component level.
122+
123+
### 4.2 Driver Registry & Pipeline Integration
124+
125+
```
126+
AGENT_DRIVERS = {
127+
"http_api": HttpApiDriver, # Wraps existing APIDataAmender
128+
# "kubernetes_crd": KubernetesCrdDriver,
129+
"subprocess": SubprocessDriver, # CLI-based proposal lifecycle via kubectl/oc
130+
}
131+
```
132+
133+
Pipeline change: the driver should replace the amender call. Metrics are agent-agnostic.
134+
135+
```
136+
Current: processor -> APIDataAmender -> MetricsEvaluator
137+
New: processor -> AgentDriver.execute_turn() -> MetricsEvaluator
138+
```
139+
140+
## 5\. Openshift Agentic Lightspeed Flow
141+
142+
### 5.1 Lifecycle
143+
144+
```
145+
1. Build Proposal CR ← Merge proposal_spec + request + agent config
146+
2. Create CR on cluster ← Auto-generated name: eval-<uuid8>
147+
3. Poll status ← Loop every poll_interval seconds
148+
4. Auto-approve ← If phase == Proposed and auto_approve enabled
149+
5. Terminal phase ← Completed / Failed / Denied / Escalated
150+
6. Populate turn_data ← proposal_status (full dict) + response (summary text)
151+
7. Cleanup proposal CR ← Delete the created CR (if cleanup_proposals enabled)
152+
8. Metrics evaluate ← custom:proposal_status on enriched data
153+
```
154+
The driver manages the full proposal lifecycle — create through cleanup. Setup/cleanup scripts are only needed for **infrastructure** (deploying agent, llmprovider, sandbox and needed CRs to the cluster).
155+
156+
### 5.2 Data Model
157+
158+
TurnData new fields:
159+
160+
| Field | Type | Source | Purpose |
161+
| :---- | :---- | :---- | :---- |
162+
| description | Optional\[str\] | User | Human-readable label for reports. Falls back to query. |
163+
| proposal\_spec | Optional\[dict\] | User | Inline proposal spec |
164+
| expected\_proposal\_status | Optional\[dict\] | User | Assertions to check against proposal\_status |
165+
| proposal\_status | Optional\[dict\] | Framework | Raw CRD status populated by driver. Saved in amended data. |
166+
167+
**query** remains required. **request** is accepted as alias. For Openshift Agentic Lightspeed, the driver injects a query/request into the proposal CR's request field.
168+
169+
EvaluationData new fields: agent: Optional\[str\], agent\_config: Optional\[dict\]
170+
171+
## 6\. Open Decision: K8s Python Client vs kubectl/oc
172+
173+
Both implement the same AgentDriver interface. The rest of the framework is unaffected.
174+
175+
| Factor | K8s Python Client | kubectl/oc Subprocess |
176+
| :---- | :---- | :---- |
177+
| New dependency | kubernetes package (\~50MB) | None (oc already needed for setup) |
178+
| Auth | Kubeconfig loading in code | Inherits from shell |
179+
| Code | \~200 LOC | \~100 LOC |
180+
| Debugging | Inspect Python objects | Copy-paste commands to terminal |
181+
| Consistency | Different tool than setup scripts | Same tool as setup scripts |
182+
| Errors | Python exceptions | Parse stderr \+ exit codes |
183+
184+
Evaluation/assertion logic (**custom:proposal\_status**) will be Python regardless. This decision only affects CRD lifecycle operations (create, poll, approve, fetch).
185+
186+
Recommendation: Lean toward kubectl/oc for consistency and fewer dependencies. ??
187+
188+
## 7\. Evaluation: custom:proposal\_status
189+
190+
### 7.1 Architecture
191+
192+
A single metric that should run all assertion checks from expected\_proposal\_status in sequence, failing fast at the first failure. Mirrors the operator's EvalSuite: one Expect block per case, one result.
193+
194+
Checks should run in order: phase → timing → analysis → components → execution → verification. Each check returns None (no expectation, skip), (True, reason), or (False, reason).
195+
196+
### 7.2 expected\_proposal\_status Structure
197+
198+
```
199+
expected_proposal_status:
200+
phase: Completed # Exact match
201+
phase_in: [Completed, Escalated] # Alternative: any of these
202+
max_duration: "5m"
203+
max_attempts: 3
204+
analysis:
205+
min_options: 1
206+
options:
207+
- risk_in: [low, medium]
208+
confidence_in: [medium, high]
209+
diagnosis_contains: ["scale", "replicas"]
210+
components:
211+
- type: remediation_summary
212+
match: { action: Scale, replicas: 3 }
213+
- type: risk_assessment
214+
match_contains: { summary: "low risk" }
215+
required: [mitigation_steps]
216+
- type: destructive_action
217+
absent: true
218+
execution:
219+
phase: Succeeded
220+
verification:
221+
passed: true
222+
summary_contains: "3 replicas running"
223+
```
224+
225+
This structure should map 1:1 to the operator's Expectations Go struct (camelCase → snake\_case).
226+
227+
### 7.3 LLM-as-Judge (Future)
228+
229+
LLM-based quality evaluation is a future phase. Approach TBD — may use existing metrics (e.g., custom:answer\_correctness on the remediation summary), a new metric, or a combination.
230+
231+
### 7.4 Comparison: eval\_data vs Operator EvalSuite
232+
233+
| Aspect | Operator EvalSuite | lightspeed-eval |
234+
| :---- | :---- | :---- |
235+
| Input | case.workflow \+ case.request inline | request (alias for query) \+ proposal\_spec.workflow |
236+
| Label | case.name | turn.description |
237+
| Assertions | case.expect (single block) | turn.expected\_proposal\_status (same semantics) |
238+
| Naming | camelCase (minOptions) | snake\_case (min\_options) |
239+
| Scope | One suite \= one workflow | Mixed agents in one eval run |
240+
| Extra | NA | Future: LLM-as-judge |
241+
242+
## 8\. Dependencies
243+
244+
If **K8s Python client** (Approach A): New dependency kubernetes\>=28.0.0
245+
246+
If **kubectl/oc subprocess** (Approach B): No new Python dependencies. oc/kubectl already required for setup scripts.
247+
248+
Always required: Cluster access, RBAC permissions, operator installed for real evaluations.
249+
250+
Not required: Operator eval CLI (lightspeed-eval drives the flow).

0 commit comments

Comments
 (0)