Skip to content

Commit 77de796

Browse files
committed
docs: align docs strictly with FACET v2.1.3 spec (rust branch sync)
1 parent 9bbec5a commit 77de796

22 files changed

Lines changed: 947 additions & 11539 deletions

docs/01-quickstart.md

Lines changed: 44 additions & 474 deletions
Large diffs are not rendered by default.

docs/02-tutorial.md

Lines changed: 116 additions & 684 deletions
Large diffs are not rendered by default.

docs/03-architecture.md

Lines changed: 61 additions & 794 deletions
Large diffs are not rendered by default.

docs/04-type-system.md

Lines changed: 59 additions & 824 deletions
Large diffs are not rendered by default.

docs/05-examples-guide.md

Lines changed: 49 additions & 991 deletions
Large diffs are not rendered by default.

docs/06-cli.md

Lines changed: 53 additions & 698 deletions
Large diffs are not rendered by default.

docs/07-api-reference.md

Lines changed: 74 additions & 687 deletions
Large diffs are not rendered by default.

docs/08-lenses.md

Lines changed: 50 additions & 1493 deletions
Large diffs are not rendered by default.

docs/09-testing.md

Lines changed: 43 additions & 318 deletions
Original file line numberDiff line numberDiff line change
@@ -1,351 +1,76 @@
11
---
22
permalink: /09-testing.html
3+
title: Testing
34
---
45

5-
# 09. FACET v2.1.3 Testing Guide
6-
**Reading Time:** 15-20 minutes | **Difficulty:** Intermediate | **Previous:** [08-lenses.md](08-lenses.html) | **Next:** [10-performance.md](10-performance.html)
7-
**Specification Compliance:** Implements FACET v2.1.3 specification Section 13 (@test blocks) with extended features.
6+
# 09. Testing with `@test` (Hypervisor)
87

9-
## Table of Contents
8+
`@test` is required for Hypervisor profile conformance.
109

11-
- [Overview](#overview)
12-
- [Syntax](#syntax)
13-
- [Variable Overrides](#variable-overrides)
14-
- [Mocking](#mocking)
15-
- [Assertions](#assertions)
16-
- [Running Tests](#running-tests)
17-
- [Test Results](#test-results)
18-
- [Best Practices](#best-practices)
19-
- [Examples](#examples)
20-
- [Reference](#reference)
21-
22-
---
23-
24-
## Overview
25-
26-
FACET v2.1.3 includes a built-in testing system using `@test` blocks. These blocks allow you to:
27-
- Define test inputs and variable overrides
28-
- Mock external interfaces and lenses
29-
- Assert expected outputs and telemetry
30-
- Run automated test suites
31-
32-
## Syntax
10+
## Minimal syntax
3311

3412
```facet
35-
@test "test name"
13+
@test "basic"
3614
vars:
37-
# Override @input values
38-
variable_name: value
39-
15+
username: "TestUser"
16+
input:
17+
query: "hello"
4018
mock:
41-
# Mock interface calls
42-
InterfaceName.method: { return: value }
43-
44-
# Mock lens calls
45-
lens_name: { return: value }
46-
19+
WeatherAPI.get_current: { temp: 10, condition: "Rain" }
4720
assert:
48-
# Assertions on output and telemetry
49-
- output contains "expected text"
50-
- cost < 0.01
51-
- tokens < 100
52-
- sentiment "positive"
21+
- canonical.messages[0].role == "system"
22+
- canonical contains "hello"
23+
- telemetry.gas_used < 5000
5324
```
5425

55-
## Test Sections
56-
57-
### vars Section
58-
Override `@input` values for the test:
59-
60-
```facet
61-
@test "user input test"
62-
vars:
63-
user_query: "What is the weather today?"
64-
user_location: "New York"
65-
```
66-
67-
### mock Section
68-
Define mock return values for interfaces and lenses:
69-
70-
```facet
71-
@test "with mocked weather API"
72-
mock:
73-
WeatherAPI.get_current: { temp: 25, condition: "Sunny" }
74-
WeatherAPI.get_forecast: {
75-
forecast: ["sunny", "sunny", "cloudy"]
76-
}
77-
llm_complete: { text: "The weather will be sunny." }
78-
```
79-
80-
### assert Section
81-
Define assertions to validate test results:
82-
83-
#### Output Assertions
84-
- `contains` - Check if output contains text
85-
- `not_contains` - Check if output doesn't contain text
86-
- `equals` - Check exact match
87-
- `matches` - Regex pattern match
88-
89-
#### Telemetry Assertions
90-
- `cost < value` - Check estimated cost
91-
- `tokens < value` - Check token usage
92-
- `time < value` - Check execution time (ms)
93-
- `gas < value` - Check gas consumption
94-
95-
#### Sentiment Assertion
96-
- `sentiment "positive|negative|neutral"` - Basic sentiment analysis
26+
## Semantics
9727

98-
#### Boolean Assertions
99-
- `true` - Value should be truthy
100-
- `false` - Value should be falsy
101-
- `null` - Value should be null
102-
- `not_null` - Value should not be null
28+
- each test runs in isolation
29+
- `vars` overrides variables in test environment (still type-checked)
30+
- `input` supplies runtime values for `@input`
31+
- `mock` intercepts interface calls by fully-qualified name
32+
- assertions run against:
33+
- `canonical`
34+
- `telemetry`
35+
- `execution` (recommended, if artifact exposed)
10336

104-
## Examples
37+
## Example file
10538

106-
### Basic Test
10739
```facet
10840
@vars
109-
user_query: @input(type="string")
110-
111-
@system
112-
role: "helpful assistant"
113-
114-
@user
115-
$user_query
41+
query: @input(type="string")
11642
117-
@test "basic response"
118-
vars:
119-
user_query: "Hello, world!"
120-
121-
assert:
122-
- output contains "Hello"
123-
- output sentiment "positive"
124-
- cost < 0.01
125-
```
126-
127-
### Test with Mocks
128-
```facet
129-
@vars
130-
user_query: @input(type="string")
131-
location: @input(type="string")
132-
13343
@system
134-
role: "weather assistant"
135-
tools: [$WeatherAPI]
136-
137-
@user
138-
"What's the weather in $location?"
44+
content: "You are helpful."
13945
140-
@test "weather query"
141-
vars:
142-
user_query: "What's the weather in Tokyo?"
143-
location: "Tokyo"
144-
145-
mock:
146-
WeatherAPI.get_current: { temp: 22, condition: "Clear" }
147-
148-
assert:
149-
- output contains "22"
150-
- output contains "Clear"
151-
- tokens < 50
152-
```
46+
@user
47+
content: $query
15348
154-
### Complex Test with Multiple Assertions
155-
```facet
156-
@test "comprehensive test"
157-
vars:
158-
prompt: "Summarize this text"
159-
text: "This is a very long text that needs summarization..."
160-
161-
mock:
162-
llm_summarize: {
163-
summary: "This text is about summarization."
164-
tokens_used: 150
165-
}
166-
49+
@test "query-roundtrip"
50+
input:
51+
query: "hello"
16752
assert:
168-
- output contains "summarization"
169-
- output not contains "very long text"
170-
- tokens < 200
171-
- cost < 0.05
172-
- time < 5000
173-
- sentiment "neutral"
53+
- canonical.messages[1].role == "user"
54+
- canonical contains "hello"
17455
```
17556

176-
## Running Tests
57+
Run:
17758

178-
### Command Line
17959
```bash
180-
# Run all tests
181-
fct test -i document.facet
182-
183-
# Run specific test
184-
fct test -i document.facet --filter "weather query"
185-
186-
# Verbose output
187-
fct test -i document.facet --output verbose
188-
189-
# JSON output for CI/CD
190-
fct test -i document.facet --output json
191-
192-
# Custom resource limits
193-
fct test -i document.facet --budget 8192 --gas-limit 20000
194-
```
195-
196-
### Output Formats
197-
198-
#### Summary (default)
199-
```
200-
Test Results:
201-
✓ basic response
202-
✓ weather query
203-
✓ comprehensive test
204-
205-
Total: 3
206-
Passed: 3
207-
```
208-
209-
#### Verbose
60+
facet-fct test --input test_example.facet
21061
```
211-
🧪 Test: basic response
212-
✅ PASSED (150ms, 45 tokens, $0.0012)
213-
✓ Expected output to contain 'Hello'
214-
✓ Expected sentiment 'positive'
215-
✓ Expected cost < 0.01
21662

217-
🧪 Test: weather query
218-
✅ PASSED (200ms, 52 tokens, $0.0015)
219-
✓ Expected output to contain '22'
220-
✓ Expected output to contain 'Clear'
221-
✓ Expected tokens < 50
222-
223-
Summary: 2/2 passed
224-
```
225-
226-
#### JSON
227-
```json
228-
{
229-
"tests": [
230-
{
231-
"name": "basic response",
232-
"passed": true,
233-
"assertions": [...],
234-
"telemetry": {...},
235-
"error": null
236-
}
237-
],
238-
"summary": {
239-
"total": 2,
240-
"passed": 2,
241-
"failed": 0
242-
}
243-
}
244-
```
245-
246-
## Best Practices
247-
248-
1. **Descriptive Test Names** - Use clear, descriptive names
249-
2. **Specific Assertions** - Test both positive and negative cases
250-
3. **Mock External Dependencies** - Ensure tests are deterministic
251-
4. **Resource Limits** - Set reasonable cost and token limits
252-
5. **Isolation** - Each test should be independent
253-
254-
## Integration with CI/CD
255-
256-
```yaml
257-
# GitHub Actions example
258-
- name: Run FACET tests
259-
run: |
260-
fct test -i document.facet --output json > test-results.json
261-
262-
- name: Check test results
263-
run: |
264-
jq -e '.summary.failed == 0' test-results.json
265-
```
63+
## Pure mode expectations
26664

267-
## Troubleshooting
65+
In pure mode tests:
26866

269-
### Common Issues
270-
271-
1. **Mock Not Found**
272-
```
273-
Error: Missing mock for Level 1 lens: llm_complete
274-
```
275-
Solution: Add mock for the lens in the test's `mock` section
276-
277-
2. **Assertion Failed**
278-
```
279-
✗ Expected output to contain 'umbrella'
280-
Actual: It's sunny today!
281-
```
282-
Solution: Check mock values or adjust assertion
283-
284-
3. **Resource Limit Exceeded**
285-
```
286-
✗ Expected cost < 0.01
287-
Actual: 0.015
288-
```
289-
Solution: Adjust test expectations or optimize the FACET document
290-
291-
## Next Steps
292-
293-
🎯 **Testing in Practice:**
294-
- **[05-examples-guide.md](05-examples-guide.html)** - Complete testing example
295-
- **[07-api-reference.md](07-api-reference.html)** - TestRunner API
296-
- **[10-performance.md](10-performance.html)** - Testing performance
297-
298-
🔧 **Advanced Testing:**
299-
- **[11-security.md](11-security.html)** - Security testing
300-
- **[12-errors.md](12-errors.html)** - Test-related errors
301-
- **[13-import-system.md](13-import-system.html)** - Testing imported modules
302-
303-
📚 **Resources:**
304-
- **[test_example.facet](../examples/test_example.facet)** - Complete test example
305-
- **[faq.md](faq.html)** - Testing FAQs
306-
307-
---
308-
309-
## Reference
310-
311-
See [FACET v2.1.3 Specification](https://github.com/rokoss21/facet-compiler/blob/master/FACET-v2.1.3-Production-Language-Specification.md) for complete details on:
312-
- [Section 13: Testing (@test)](../FACET-v2.1.3-Production-Language-Specification.md#13-testing-test)
313-
- [Error Codes](../FACET-v2.1.3-Production-Language-Specification.md#appendix-c-normative-error-code-catalog)
314-
- [Execution Model](../FACET-v2.1.3-Production-Language-Specification.md#7-execution-model)
315-
```
316-
✗ Expected output to contain 'umbrella'
317-
Actual: It's sunny today!
318-
```
319-
Solution: Check mock values or adjust assertion
320-
321-
3. **Resource Limit Exceeded**
322-
```
323-
✗ Expected cost < 0.01
324-
Actual: 0.015
325-
```
326-
Solution: Adjust test expectations or optimize the FACET document
327-
328-
## Next Steps
329-
330-
🎯 **Testing in Practice:**
331-
- **[05-examples-guide.md](05-examples-guide.html)** - Complete testing example
332-
- **[07-api-reference.md](07-api-reference.html)** - TestRunner API
333-
- **[10-performance.md](10-performance.html)** - Testing performance
334-
335-
🔧 **Advanced Testing:**
336-
- **[11-security.md](11-security.html)** - Security testing
337-
- **[12-errors.md](12-errors.html)** - Test-related errors
338-
- **[13-import-system.md](13-import-system.html)** - Testing imported modules
339-
340-
📚 **Resources:**
341-
- **[test_example.facet](../examples/test_example.facet)** - Complete test example
342-
- **[faq.md](faq.html)** - Testing FAQs
343-
344-
---
67+
- Level-1 cache miss → `F803`
68+
- disallowed I/O → `F801`
69+
- policy/guard deny/undecidable → `F454` / `F455`
34570

346-
## Reference
71+
## Practical tips
34772

348-
See [FACET v2.1.3 Specification](https://github.com/rokoss21/facet-compiler/blob/master/FACET-v2.1.3-Production-Language-Specification.md) for complete details on:
349-
- [Section 13: Testing (@test)](../FACET-v2.1.3-Production-Language-Specification.md#13-testing-test)
350-
- [Error Codes](../FACET-v2.1.3-Production-Language-Specification.md#appendix-c-normative-error-code-catalog)
351-
- [Execution Model](../FACET-v2.1.3-Production-Language-Specification.md#7-execution-model)
73+
- keep tests deterministic (fixed inputs/mocks)
74+
- assert on canonical structure and key values
75+
- add policy tests for allow/deny behavior
76+
- use `facet-fct inspect` when debugging failing assertions

0 commit comments

Comments
 (0)