|
1 | 1 | --- |
2 | 2 | permalink: /09-testing.html |
| 3 | +title: Testing |
3 | 4 | --- |
4 | 5 |
|
5 | | -# 09. FACET v2.1.3 Testing Guide |
6 | | -**Reading Time:** 15-20 minutes | **Difficulty:** Intermediate | **Previous:** [08-lenses.md](08-lenses.html) | **Next:** [10-performance.md](10-performance.html) |
7 | | -**Specification Compliance:** Implements FACET v2.1.3 specification Section 13 (@test blocks) with extended features. |
| 6 | +# 09. Testing with `@test` (Hypervisor) |
8 | 7 |
|
9 | | -## Table of Contents |
| 8 | +`@test` is required for Hypervisor profile conformance. |
10 | 9 |
|
11 | | -- [Overview](#overview) |
12 | | -- [Syntax](#syntax) |
13 | | -- [Variable Overrides](#variable-overrides) |
14 | | -- [Mocking](#mocking) |
15 | | -- [Assertions](#assertions) |
16 | | -- [Running Tests](#running-tests) |
17 | | -- [Test Results](#test-results) |
18 | | -- [Best Practices](#best-practices) |
19 | | -- [Examples](#examples) |
20 | | -- [Reference](#reference) |
21 | | - |
22 | | ---- |
23 | | - |
24 | | -## Overview |
25 | | - |
26 | | -FACET v2.1.3 includes a built-in testing system using `@test` blocks. These blocks allow you to: |
27 | | -- Define test inputs and variable overrides |
28 | | -- Mock external interfaces and lenses |
29 | | -- Assert expected outputs and telemetry |
30 | | -- Run automated test suites |
31 | | - |
32 | | -## Syntax |
| 10 | +## Minimal syntax |
33 | 11 |
|
34 | 12 | ```facet |
35 | | -@test "test name" |
| 13 | +@test "basic" |
36 | 14 | vars: |
37 | | - # Override @input values |
38 | | - variable_name: value |
39 | | - |
| 15 | + username: "TestUser" |
| 16 | + input: |
| 17 | + query: "hello" |
40 | 18 | mock: |
41 | | - # Mock interface calls |
42 | | - InterfaceName.method: { return: value } |
43 | | - |
44 | | - # Mock lens calls |
45 | | - lens_name: { return: value } |
46 | | - |
| 19 | + WeatherAPI.get_current: { temp: 10, condition: "Rain" } |
47 | 20 | assert: |
48 | | - # Assertions on output and telemetry |
49 | | - - output contains "expected text" |
50 | | - - cost < 0.01 |
51 | | - - tokens < 100 |
52 | | - - sentiment "positive" |
| 21 | + - canonical.messages[0].role == "system" |
| 22 | + - canonical contains "hello" |
| 23 | + - telemetry.gas_used < 5000 |
53 | 24 | ``` |
54 | 25 |
|
55 | | -## Test Sections |
56 | | - |
57 | | -### vars Section |
58 | | -Override `@input` values for the test: |
59 | | - |
60 | | -```facet |
61 | | -@test "user input test" |
62 | | - vars: |
63 | | - user_query: "What is the weather today?" |
64 | | - user_location: "New York" |
65 | | -``` |
66 | | - |
67 | | -### mock Section |
68 | | -Define mock return values for interfaces and lenses: |
69 | | - |
70 | | -```facet |
71 | | -@test "with mocked weather API" |
72 | | - mock: |
73 | | - WeatherAPI.get_current: { temp: 25, condition: "Sunny" } |
74 | | - WeatherAPI.get_forecast: { |
75 | | - forecast: ["sunny", "sunny", "cloudy"] |
76 | | - } |
77 | | - llm_complete: { text: "The weather will be sunny." } |
78 | | -``` |
79 | | - |
80 | | -### assert Section |
81 | | -Define assertions to validate test results: |
82 | | - |
83 | | -#### Output Assertions |
84 | | -- `contains` - Check if output contains text |
85 | | -- `not_contains` - Check if output doesn't contain text |
86 | | -- `equals` - Check exact match |
87 | | -- `matches` - Regex pattern match |
88 | | - |
89 | | -#### Telemetry Assertions |
90 | | -- `cost < value` - Check estimated cost |
91 | | -- `tokens < value` - Check token usage |
92 | | -- `time < value` - Check execution time (ms) |
93 | | -- `gas < value` - Check gas consumption |
94 | | - |
95 | | -#### Sentiment Assertion |
96 | | -- `sentiment "positive|negative|neutral"` - Basic sentiment analysis |
| 26 | +## Semantics |
97 | 27 |
|
98 | | -#### Boolean Assertions |
99 | | -- `true` - Value should be truthy |
100 | | -- `false` - Value should be falsy |
101 | | -- `null` - Value should be null |
102 | | -- `not_null` - Value should not be null |
| 28 | +- each test runs in isolation |
| 29 | +- `vars` overrides variables in test environment (still type-checked) |
| 30 | +- `input` supplies runtime values for `@input` |
| 31 | +- `mock` intercepts interface calls by fully-qualified name |
| 32 | +- assertions run against: |
| 33 | + - `canonical` |
| 34 | + - `telemetry` |
| 35 | + - `execution` (recommended, if artifact exposed) |
103 | 36 |
|
104 | | -## Examples |
| 37 | +## Example file |
105 | 38 |
|
106 | | -### Basic Test |
107 | 39 | ```facet |
108 | 40 | @vars |
109 | | - user_query: @input(type="string") |
110 | | - |
111 | | -@system |
112 | | - role: "helpful assistant" |
113 | | - |
114 | | -@user |
115 | | - $user_query |
| 41 | + query: @input(type="string") |
116 | 42 |
|
117 | | -@test "basic response" |
118 | | - vars: |
119 | | - user_query: "Hello, world!" |
120 | | - |
121 | | - assert: |
122 | | - - output contains "Hello" |
123 | | - - output sentiment "positive" |
124 | | - - cost < 0.01 |
125 | | -``` |
126 | | - |
127 | | -### Test with Mocks |
128 | | -```facet |
129 | | -@vars |
130 | | - user_query: @input(type="string") |
131 | | - location: @input(type="string") |
132 | | - |
133 | 43 | @system |
134 | | - role: "weather assistant" |
135 | | - tools: [$WeatherAPI] |
136 | | - |
137 | | -@user |
138 | | - "What's the weather in $location?" |
| 44 | + content: "You are helpful." |
139 | 45 |
|
140 | | -@test "weather query" |
141 | | - vars: |
142 | | - user_query: "What's the weather in Tokyo?" |
143 | | - location: "Tokyo" |
144 | | - |
145 | | - mock: |
146 | | - WeatherAPI.get_current: { temp: 22, condition: "Clear" } |
147 | | - |
148 | | - assert: |
149 | | - - output contains "22" |
150 | | - - output contains "Clear" |
151 | | - - tokens < 50 |
152 | | -``` |
| 46 | +@user |
| 47 | + content: $query |
153 | 48 |
|
154 | | -### Complex Test with Multiple Assertions |
155 | | -```facet |
156 | | -@test "comprehensive test" |
157 | | - vars: |
158 | | - prompt: "Summarize this text" |
159 | | - text: "This is a very long text that needs summarization..." |
160 | | - |
161 | | - mock: |
162 | | - llm_summarize: { |
163 | | - summary: "This text is about summarization." |
164 | | - tokens_used: 150 |
165 | | - } |
166 | | - |
| 49 | +@test "query-roundtrip" |
| 50 | + input: |
| 51 | + query: "hello" |
167 | 52 | assert: |
168 | | - - output contains "summarization" |
169 | | - - output not contains "very long text" |
170 | | - - tokens < 200 |
171 | | - - cost < 0.05 |
172 | | - - time < 5000 |
173 | | - - sentiment "neutral" |
| 53 | + - canonical.messages[1].role == "user" |
| 54 | + - canonical contains "hello" |
174 | 55 | ``` |
175 | 56 |
|
176 | | -## Running Tests |
| 57 | +Run: |
177 | 58 |
|
178 | | -### Command Line |
179 | 59 | ```bash |
180 | | -# Run all tests |
181 | | -fct test -i document.facet |
182 | | - |
183 | | -# Run specific test |
184 | | -fct test -i document.facet --filter "weather query" |
185 | | - |
186 | | -# Verbose output |
187 | | -fct test -i document.facet --output verbose |
188 | | - |
189 | | -# JSON output for CI/CD |
190 | | -fct test -i document.facet --output json |
191 | | - |
192 | | -# Custom resource limits |
193 | | -fct test -i document.facet --budget 8192 --gas-limit 20000 |
194 | | -``` |
195 | | - |
196 | | -### Output Formats |
197 | | - |
198 | | -#### Summary (default) |
199 | | -``` |
200 | | -Test Results: |
201 | | - ✓ basic response |
202 | | - ✓ weather query |
203 | | - ✓ comprehensive test |
204 | | -
|
205 | | -Total: 3 |
206 | | -Passed: 3 |
207 | | -``` |
208 | | - |
209 | | -#### Verbose |
| 60 | +facet-fct test --input test_example.facet |
210 | 61 | ``` |
211 | | -🧪 Test: basic response |
212 | | -✅ PASSED (150ms, 45 tokens, $0.0012) |
213 | | - ✓ Expected output to contain 'Hello' |
214 | | - ✓ Expected sentiment 'positive' |
215 | | - ✓ Expected cost < 0.01 |
216 | 62 |
|
217 | | -🧪 Test: weather query |
218 | | -✅ PASSED (200ms, 52 tokens, $0.0015) |
219 | | - ✓ Expected output to contain '22' |
220 | | - ✓ Expected output to contain 'Clear' |
221 | | - ✓ Expected tokens < 50 |
222 | | -
|
223 | | -Summary: 2/2 passed |
224 | | -``` |
225 | | - |
226 | | -#### JSON |
227 | | -```json |
228 | | -{ |
229 | | - "tests": [ |
230 | | - { |
231 | | - "name": "basic response", |
232 | | - "passed": true, |
233 | | - "assertions": [...], |
234 | | - "telemetry": {...}, |
235 | | - "error": null |
236 | | - } |
237 | | - ], |
238 | | - "summary": { |
239 | | - "total": 2, |
240 | | - "passed": 2, |
241 | | - "failed": 0 |
242 | | - } |
243 | | -} |
244 | | -``` |
245 | | - |
246 | | -## Best Practices |
247 | | - |
248 | | -1. **Descriptive Test Names** - Use clear, descriptive names |
249 | | -2. **Specific Assertions** - Test both positive and negative cases |
250 | | -3. **Mock External Dependencies** - Ensure tests are deterministic |
251 | | -4. **Resource Limits** - Set reasonable cost and token limits |
252 | | -5. **Isolation** - Each test should be independent |
253 | | - |
254 | | -## Integration with CI/CD |
255 | | - |
256 | | -```yaml |
257 | | -# GitHub Actions example |
258 | | -- name: Run FACET tests |
259 | | - run: | |
260 | | - fct test -i document.facet --output json > test-results.json |
261 | | - |
262 | | -- name: Check test results |
263 | | - run: | |
264 | | - jq -e '.summary.failed == 0' test-results.json |
265 | | -``` |
| 63 | +## Pure mode expectations |
266 | 64 |
|
267 | | -## Troubleshooting |
| 65 | +In pure mode tests: |
268 | 66 |
|
269 | | -### Common Issues |
270 | | -
|
271 | | -1. **Mock Not Found** |
272 | | - ``` |
273 | | - Error: Missing mock for Level 1 lens: llm_complete |
274 | | - ``` |
275 | | - Solution: Add mock for the lens in the test's `mock` section |
276 | | - |
277 | | -2. **Assertion Failed** |
278 | | - ``` |
279 | | - ✗ Expected output to contain 'umbrella' |
280 | | - Actual: It's sunny today! |
281 | | - ``` |
282 | | - Solution: Check mock values or adjust assertion |
283 | | -
|
284 | | -3. **Resource Limit Exceeded** |
285 | | - ``` |
286 | | - ✗ Expected cost < 0.01 |
287 | | - Actual: 0.015 |
288 | | - ``` |
289 | | - Solution: Adjust test expectations or optimize the FACET document |
290 | | -
|
291 | | -## Next Steps |
292 | | -
|
293 | | -🎯 **Testing in Practice:** |
294 | | -- **[05-examples-guide.md](05-examples-guide.html)** - Complete testing example |
295 | | -- **[07-api-reference.md](07-api-reference.html)** - TestRunner API |
296 | | -- **[10-performance.md](10-performance.html)** - Testing performance |
297 | | -
|
298 | | -🔧 **Advanced Testing:** |
299 | | -- **[11-security.md](11-security.html)** - Security testing |
300 | | -- **[12-errors.md](12-errors.html)** - Test-related errors |
301 | | -- **[13-import-system.md](13-import-system.html)** - Testing imported modules |
302 | | -
|
303 | | -📚 **Resources:** |
304 | | -- **[test_example.facet](../examples/test_example.facet)** - Complete test example |
305 | | -- **[faq.md](faq.html)** - Testing FAQs |
306 | | -
|
307 | | ---- |
308 | | -
|
309 | | -## Reference |
310 | | -
|
311 | | -See [FACET v2.1.3 Specification](https://github.com/rokoss21/facet-compiler/blob/master/FACET-v2.1.3-Production-Language-Specification.md) for complete details on: |
312 | | -- [Section 13: Testing (@test)](../FACET-v2.1.3-Production-Language-Specification.md#13-testing-test) |
313 | | -- [Error Codes](../FACET-v2.1.3-Production-Language-Specification.md#appendix-c-normative-error-code-catalog) |
314 | | -- [Execution Model](../FACET-v2.1.3-Production-Language-Specification.md#7-execution-model) |
315 | | - ``` |
316 | | - ✗ Expected output to contain 'umbrella' |
317 | | - Actual: It's sunny today! |
318 | | - ``` |
319 | | - Solution: Check mock values or adjust assertion |
320 | | -
|
321 | | -3. **Resource Limit Exceeded** |
322 | | - ``` |
323 | | - ✗ Expected cost < 0.01 |
324 | | - Actual: 0.015 |
325 | | - ``` |
326 | | - Solution: Adjust test expectations or optimize the FACET document |
327 | | -
|
328 | | -## Next Steps |
329 | | -
|
330 | | -🎯 **Testing in Practice:** |
331 | | -- **[05-examples-guide.md](05-examples-guide.html)** - Complete testing example |
332 | | -- **[07-api-reference.md](07-api-reference.html)** - TestRunner API |
333 | | -- **[10-performance.md](10-performance.html)** - Testing performance |
334 | | -
|
335 | | -🔧 **Advanced Testing:** |
336 | | -- **[11-security.md](11-security.html)** - Security testing |
337 | | -- **[12-errors.md](12-errors.html)** - Test-related errors |
338 | | -- **[13-import-system.md](13-import-system.html)** - Testing imported modules |
339 | | -
|
340 | | -📚 **Resources:** |
341 | | -- **[test_example.facet](../examples/test_example.facet)** - Complete test example |
342 | | -- **[faq.md](faq.html)** - Testing FAQs |
343 | | -
|
344 | | ---- |
| 67 | +- Level-1 cache miss → `F803` |
| 68 | +- disallowed I/O → `F801` |
| 69 | +- policy/guard deny/undecidable → `F454` / `F455` |
345 | 70 |
|
346 | | -## Reference |
| 71 | +## Practical tips |
347 | 72 |
|
348 | | -See [FACET v2.1.3 Specification](https://github.com/rokoss21/facet-compiler/blob/master/FACET-v2.1.3-Production-Language-Specification.md) for complete details on: |
349 | | -- [Section 13: Testing (@test)](../FACET-v2.1.3-Production-Language-Specification.md#13-testing-test) |
350 | | -- [Error Codes](../FACET-v2.1.3-Production-Language-Specification.md#appendix-c-normative-error-code-catalog) |
351 | | -- [Execution Model](../FACET-v2.1.3-Production-Language-Specification.md#7-execution-model) |
| 73 | +- keep tests deterministic (fixed inputs/mocks) |
| 74 | +- assert on canonical structure and key values |
| 75 | +- add policy tests for allow/deny behavior |
| 76 | +- use `facet-fct inspect` when debugging failing assertions |
0 commit comments