Skip to content

Commit eec5179

Browse files
AlexanderLaninAlexanderLanin
committed
docs: overhaul ITF documentation
- Tutorial: remove concept intro and advanced steps (DLT, traceability); sharpen to workspace setup → first test → capability guards - How-to: replace duplicated Docker/QEMU recipes with upstream links; keep S-CORE-specific sections (DLT, traceability, session targets) - Reference: replace plugin table with upstream links for full CLI/attr docs; keep QEMU config JSON example - Explanation: add plugin architecture, capability abstraction, and host/target diagrams from ITF presentations; document three-repo model
1 parent 22be9c2 commit eec5179

4 files changed

Lines changed: 92 additions & 109 deletions

File tree

‎docs/explanation/04-testing-infrastructure.md‎

Lines changed: 78 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,38 @@ traceability metadata`"]
9393

9494
The central abstraction is the capability-based `Target` model. Each target exposes capabilities such as `exec`, `file_transfer`, `restart`, `ssh`, and `sftp`. Tests declare which capabilities they require using the `@requires_capabilities` decorator, and the framework skips tests when the active target does not provide them. This design lets the same test code run against different target environments — a Docker container locally, a QEMU image in CI, a real board in the lab — with the plugin determining which capabilities are available.
9595

96-
The plugin model is what makes ITF extensible. Four built-in plugins cover the current target types and supporting concerns:
96+
The plugin model is what makes ITF extensible. Plugins fall into two categories:
97+
98+
```{mermaid}
99+
graph TB
100+
CORE["ITF Core Framework"]
101+
102+
subgraph "Target Plugins"
103+
QEMU["QEMU Plugin"]
104+
DOCKER["Docker Plugin"]
105+
FUTURE_T["..."]
106+
end
107+
108+
subgraph "Non-Target Plugins"
109+
DLT["DLT Plugin"]
110+
FUTURE_NT["..."]
111+
end
112+
113+
CORE --- QEMU
114+
CORE --- DOCKER
115+
CORE --- FUTURE_T
116+
CORE --- DLT
117+
CORE --- FUTURE_NT
118+
119+
style CORE fill:#4a90d9,color:#fff,stroke:#2a6cb8
120+
style QEMU fill:#7cb342,color:#fff,stroke:#5a8f2a
121+
style DOCKER fill:#7cb342,color:#fff,stroke:#5a8f2a
122+
style FUTURE_T fill:#7cb342,color:#fff,stroke:#5a8f2a
123+
style DLT fill:#f4a460,color:#fff,stroke:#d4845a
124+
style FUTURE_NT fill:#f4a460,color:#fff,stroke:#d4845a
125+
```
126+
127+
**Target plugins** override the `target_init` fixture to yield a `Target` subclass — they determine what environment the test runs against. Only one target plugin is active at a time. **Non-target plugins** (like DLT) provide independent fixtures and compose freely with any target plugin. Four built-in plugins cover the current scope:
97128

98129
| Plugin | Bazel label | Purpose |
99130
|---|---|---|
@@ -102,7 +133,52 @@ The plugin model is what makes ITF extensible. Four built-in plugins cover the c
102133
| DLT | `@score_itf//score/itf/plugins:dlt_plugin` | Diagnostic Log and Trace message capture and query |
103134
| Attribute | `@score_itf//score/itf/plugins:attribute_plugin` | Requirement traceability metadata in JUnit XML |
104135

105-
Each plugin is a Bazel target (built via the `py_itf_plugin` rule) that handles provisioning, communication, and teardown. New target types can be added by implementing the `Target` abstract class and a `target_init` pytest fixture in a custom plugin, without changing the ITF core or the test-writing interface.
136+
New target types can be added by implementing the `Target` abstract class and a `target_init` pytest fixture in a new plugin — the ITF core and existing tests require no changes.
137+
138+
The capability abstraction is what makes the same test file portable across environments:
139+
140+
```{mermaid}
141+
graph LR
142+
TEST["Test Code"]
143+
CAP["Capabilities\nexecute, upload,\ndownload, restart"]
144+
PLUGIN["Plugin"]
145+
TARGET["Concrete Target"]
146+
147+
TEST -->|"calls"| CAP
148+
CAP -->|"dispatches to"| PLUGIN
149+
PLUGIN -->|"controls"| TARGET
150+
151+
style TEST fill:#4a90d9,color:#fff,stroke:#2a6cb8
152+
style CAP fill:#f4a460,color:#fff,stroke:#d4845a
153+
style PLUGIN fill:#7cb342,color:#fff,stroke:#5a8f2a
154+
style TARGET fill:#888,color:#fff,stroke:#666
155+
```
156+
157+
ITF runs entirely on the host. It never communicates with the target directly — the plugin handles all provisioning, communication, and teardown. Adding support for a new hardware target (Raspberry Pi, Qualcomm board, etc.) requires only a new plugin:
158+
159+
```{mermaid}
160+
graph LR
161+
subgraph "Host"
162+
ITF["ITF + pytest"]
163+
PLUGIN["Target Plugin"]
164+
end
165+
166+
ITF --> PLUGIN
167+
168+
PLUGIN -->|"today"| QEMU["QEMU VM"]
169+
PLUGIN -->|"today"| DOCKER["Docker Container"]
170+
PLUGIN -.->|"future"| RASPI["Raspberry Pi"]
171+
PLUGIN -.->|"future"| QUALCOMM["Qualcomm Board"]
172+
173+
style ITF fill:#4a90d9,color:#fff,stroke:#2a6cb8
174+
style PLUGIN fill:#7cb342,color:#fff,stroke:#5a8f2a
175+
style QEMU fill:#f4a460,color:#fff,stroke:#d4845a
176+
style DOCKER fill:#f4a460,color:#fff,stroke:#d4845a
177+
style RASPI fill:#ccc,color:#666,stroke:#999,stroke-dasharray: 5 5
178+
style QUALCOMM fill:#ccc,color:#666,stroke:#999,stroke-dasharray: 5 5
179+
```
180+
181+
ITF is structured across three Bazel modules. `score_itf` is the open-source core (plugin contract, Bazel rules, pytest integration). Private or proprietary plugins live in separate repositories (e.g. `vsps_itf_plugins` for ETAS-internal targets) and are consumed via `git_override` or `archive_override` in the test consumer's `MODULE.bazel`. This separation means the open-source core can evolve independently from private plugin implementations.
106182

107183
For traceability, the attribute plugin provides an `@add_test_properties` decorator that writes metadata such as `fully_verifies`, `test_type`, and `derivation_technique` into JUnit XML output. This connects ITF test results to the traceability model described in [section 4.2](#test-traceability).
108184

‎docs/how-to/testing.md‎

Lines changed: 4 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ py_test(
110110

111111
## Write an ITF integration test
112112

113-
[ITF](https://github.com/eclipse-score/itf) runs tests against real environments — Docker containers, QEMU VMs, or hardware — through a unified `Target` interface. The same test file can target different environments using `@requires_capabilities` decorators. See [Tutorial: Write Your First ITF Integration Test](../tutorials/itf-integration-test.md) for a guided introduction.
113+
[ITF](https://github.com/eclipse-score/itf) runs tests against real environments — Docker containers, QEMU VMs, or hardware — through a unified `Target` interface. See [Tutorial: Write Your First ITF Integration Test](../tutorials/itf-integration-test.md) for a step-by-step introduction.
114114

115115
### Add ITF to your workspace
116116

@@ -120,53 +120,12 @@ In `MODULE.bazel`:
120120
bazel_dep(name = "score_itf", version = "0.2.0")
121121
```
122122

123-
### Docker test
124-
125-
```python
126-
def test_hello(target):
127-
exit_code, output = target.execute("echo 'Hello from target!'")
128-
assert exit_code == 0
129-
assert b"Hello from target!" in output
130-
```
131-
132-
```starlark
133-
load("@score_itf//:defs.bzl", "py_itf_test")
134-
135-
py_itf_test(
136-
name = "test_hello",
137-
srcs = ["test_hello.py"],
138-
args = ["--docker-image=ubuntu:24.04"],
139-
plugins = ["@score_itf//score/itf/plugins:docker_plugin"],
140-
)
141-
```
142-
143-
### QEMU test
144-
145-
```python
146-
def test_qemu_ssh(target):
147-
with target.ssh(username="root", password="") as ssh:
148-
exit_code = ssh.execute_command("uname -a")
149-
assert exit_code == 0
150-
```
151-
152-
```starlark
153-
py_itf_test(
154-
name = "test_qemu",
155-
srcs = ["test_qemu.py"],
156-
args = [
157-
"--qemu-image=$(location //path:qemu_image)",
158-
"--qemu-config=$(location qemu_config.json)",
159-
],
160-
data = [
161-
"//path:qemu_image",
162-
"qemu_config.json",
163-
],
164-
plugins = ["@score_itf//score/itf/plugins:qemu_plugin"],
165-
)
166-
```
123+
For Docker and QEMU test recipes, see the [upstream how-to](https://eclipse-score.github.io/itf/main/how-to/write_tests.html). For full plugin CLI args and capabilities, see the [upstream plugin reference](https://eclipse-score.github.io/itf/main/how-to/plugins.html).
167124

168125
### Capability guards
169126

127+
Tests declare which target capabilities they need. The framework skips tests when the active target does not provide them:
128+
170129
```python
171130
from score.itf.plugins.core import requires_capabilities
172131

‎docs/reference/configuration.md‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -82,16 +82,9 @@ py_itf_test(
8282
)
8383
```
8484

85-
Available plugins:
85+
For the full `py_itf_test` attribute reference and `py_itf_plugin` rule, see the [upstream Bazel macros reference](https://eclipse-score.github.io/itf/main/reference/bazel_macros.html).
8686

87-
| Plugin label | Purpose |
88-
|---|---|
89-
| `@score_itf//score/itf/plugins:docker_plugin` | Docker container targets |
90-
| `@score_itf//score/itf/plugins:qemu_plugin` | QEMU virtual machine targets |
91-
| `@score_itf//score/itf/plugins:dlt_plugin` | DLT log capture |
92-
| `@score_itf//score/itf/plugins:attribute_plugin` | Requirement traceability metadata |
93-
94-
Source: [eclipse-score/itf](https://github.com/eclipse-score/itf)
87+
For per-plugin CLI args and capabilities, see the [upstream plugin reference](https://eclipse-score.github.io/itf/main/reference/plugins.html).
9588

9689
## QEMU Configuration (ITF)
9790

‎docs/tutorials/itf-integration-test.md‎

Lines changed: 8 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,11 @@
11
# Tutorial: Write Your First ITF Integration Test
22

3-
This tutorial walks through writing a working integration test using the [ITF framework](https://github.com/eclipse-score/itf). By the end, you will have a test that runs a command inside a Docker container through Bazel, and you will understand why ITF is structured the way it is.
3+
This tutorial walks through writing a working integration test using the [ITF framework](https://github.com/eclipse-score/itf). By the end, you will have a test that runs a command inside a Docker container through Bazel.
44

55
**What you need:** a working devcontainer or local environment with Bazel and Docker available. See [Getting Started](getting-started.md) if you have not set that up yet.
66

77
---
88

9-
## What ITF does
10-
11-
Most S-CORE tests are unit tests: they compile and run inside the Bazel sandbox on the build host. ITF tests are different — they run code against a real environment (a Docker container, a QEMU virtual machine, or a hardware device) through a unified plugin interface. This is the right tool for testing behavior that depends on a real OS, a real network, or a real device.
12-
13-
The same test file can target different environments. The `@requires_capabilities` decorator tells ITF which environment capabilities a test needs, and the framework skips tests that the active target cannot fulfill. Write once, run on Docker locally and on hardware in CI.
14-
15-
---
16-
179
## Step 1: Add ITF to your workspace
1810

1911
In `MODULE.bazel`, add the dependency:
@@ -71,7 +63,7 @@ py_itf_test(
7163
bazel test //tests:test_hello --test_output=all
7264
```
7365

74-
You should see Docker pulling the image (once), the container starting, your `echo` command running, and a green test result. If something goes wrong, `--test_output=all` shows the full pytest output.
66+
You should see Docker pulling the image (once), the container starting, your `echo` command running, and a green test result.
7567

7668
To force re-execution even if Bazel cached a previous pass:
7769

@@ -83,7 +75,7 @@ bazel test //tests:test_hello --nocache_test_results --test_output=all
8375

8476
## Step 5: Make the test portable with capability guards
8577

86-
Add a second test that uses a capability not all targets provide. Docker supports `exec` and `file_transfer`; QEMU additionally supports `ssh` and `sftp`. Guard tests so they are skipped rather than failed on the wrong target:
78+
The same test file can run against different target types. Guard tests on the capabilities they actually need so they are skipped rather than failed on targets that do not provide them:
8779

8880
```python
8981
from score.itf.plugins.core import requires_capabilities
@@ -100,44 +92,7 @@ def test_network_only(target):
10092
assert exit_code == 0
10193
```
10294

103-
Run against the Docker target — `test_network_only` will be skipped because Docker does not expose `ssh`. Switch to a QEMU target and it runs. The same file works in both cases without modification.
104-
105-
---
106-
107-
## Step 6: Add traceability metadata (optional)
108-
109-
When a test verifies specific requirements, the attribute plugin can write that information into the JUnit XML report so the docs-as-code traceability system can link back to the requirement objects:
110-
111-
```python
112-
from score.itf.plugins.core import requires_capabilities
113-
from attribute_plugin import add_test_properties
114-
115-
@add_test_properties(
116-
fully_verifies=["REQ-HELLO-001"],
117-
test_type="requirements-based",
118-
derivation_technique="requirements-analysis",
119-
)
120-
def test_hello(target):
121-
exit_code, output = target.execute("echo 'Hello from ITF!'")
122-
assert exit_code == 0
123-
assert b"Hello from ITF!" in output
124-
```
125-
126-
Update the BUILD target to include the attribute plugin:
127-
128-
```starlark
129-
py_itf_test(
130-
name = "test_hello",
131-
srcs = ["test_hello.py"],
132-
args = ["--docker-image=ubuntu:24.04"],
133-
plugins = [
134-
"@score_itf//score/itf/plugins:docker_plugin",
135-
"@score_itf//score/itf/plugins:attribute_plugin",
136-
],
137-
)
138-
```
139-
140-
The metadata appears as JUnit XML attributes on the test case element and can be consumed by Sphinx-based traceability tooling without a separate database or postprocessing step.
95+
Run against the Docker target — `test_network_only` is skipped because Docker does not expose `ssh`. Switch to a QEMU target and it runs. The same file works in both cases without modification.
14196

14297
---
14398

@@ -148,12 +103,12 @@ You now have a working ITF test that:
148103
- runs inside a Docker container through `bazel test`
149104
- participates in Bazel's incremental build and result caching
150105
- guards itself with `@requires_capabilities` so it can run against multiple target types
151-
- optionally writes requirement traceability into the test result report
152106

153107
---
154108

155109
## Next steps
156110

157-
- [Write an ITF test against a QEMU target](../how-to/testing.md#write-an-itf-integration-test) — SSH-based interaction with a VM
158-
- [Run QNX unit tests](../how-to/testing.md#advanced-qnx-unit-tests) — cross-compiled tests in a QNX microvm
159-
- [ITF framework explained](../explanation/04-testing-infrastructure.md#itf-framework) — plugin model, capability system, and traceability integration
111+
- [How-to: Testing](../how-to/testing.md#write-an-itf-integration-test) — DLT log capture, requirement traceability, session-scoped targets
112+
- [ITF: Write Tests](https://eclipse-score.github.io/itf/main/how-to/write_tests.html) — QEMU tests, SSH on Docker, full capability reference (upstream docs)
113+
- [ITF: Using Plugins](https://eclipse-score.github.io/itf/main/how-to/plugins.html) — all built-in plugins with full CLI args (upstream docs)
114+
- [Explanation: ITF Framework](../explanation/04-testing-infrastructure.md#itf-framework) — plugin model, capability system, and architecture

0 commit comments

Comments
 (0)