Skip to content

Commit 3538908

Browse files
ArneTRArneTR
committed
Added machine baseline checks and phase_padding info
1 parent 307a523 commit 3538908

3 files changed

Lines changed: 243 additions & 1 deletion

File tree

‎content/en/docs/cluster/machine-baseline-checks.md‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
---
2+
title: "Machine Baseline Checks"
3+
description: "Optional hardware and OS checks that GMT verifies before each measurement run"
4+
date: 2026-07-02T00:00:00+00:00
5+
weight: 1005
6+
toc: false
7+
---
8+
9+
GMT can verify a set of hardware and OS properties before every measurement run. All checks are configured under the `machine:` key in `config.yml`.
10+
11+
Every check is **opt-in**: omitting a key (or setting it to `null` / `False`) silently skips that check. A failed check emits a `WARN` and, depending on `system_check_threshold`, may abort the run.
12+
13+
## Temperature
14+
15+
GMT reads the current temperature from a hardware sensor chip and compares it against a configured baseline. If the machine is outside the expected range the run is held until the temperature stabilises.
16+
17+
```yaml
18+
machine:
19+
base_temperature_chip: "acpitz-acpi-0" # sensor chip name (from `sensors`)
20+
base_temperature_feature: "temp1" # feature name on that chip
21+
base_temperature_value: 65 # maximum acceptable °C
22+
```
23+
24+
| Condition | Behaviour |
25+
|---|---|
26+
| `current > base_temperature_value` | `cooldown` status event; client sleeps 60 s and retries |
27+
| `current ≤ base_temperature_value − 10` | `warmup` status event; client spins all CPU cores for 5 min then retries |
28+
| More than 10 consecutive failures | Fatal error; cluster process exits |
29+
30+
The current temperature is always written to `machines.current_temperature` in the database. Status events (`cooldown`, `warmup`) include the exact temperature reading in their `data` column.
31+
32+
To find the right chip and feature names:
33+
34+
```sh
35+
sensors
36+
# or for JSON output:
37+
sensors -j
38+
```
39+
40+
Also see [Accuracy Control →]({{< relref "accuracy-control" >}}).
41+
42+
## RAPL Power Capping
43+
44+
Verifies that RAPL power limits are at or below the values you have locked on the machine. Ensures a consistent thermal envelope across cluster nodes.
45+
46+
```yaml
47+
machine:
48+
rapl_power_capping:
49+
package: 35 # Watts — per-socket CPU package limit
50+
dram: 10 # Watts — DRAM controller limit
51+
psys: 65 # Watts — platform-wide limit (if available)
52+
```
53+
54+
Each sub-key is independent; omit any you do not want to enforce. Values are compared against the `constraint_0_power_limit_uw` sysfs files under `/sys/devices/virtual/powercap/intel-rapl/`.
55+
56+
To read the current limits on your machine:
57+
58+
```sh
59+
cat /sys/devices/virtual/powercap/intel-rapl/intel-rapl:*/constraint_0_power_limit_uw
60+
```
61+
62+
Also see [NOP Linux →]({{< relref "nop-linux" >}}) for how to make these limits persistent across reboots.
63+
64+
## Docker Registry Mirror
65+
66+
Confirms that the Docker daemon is configured to use a specific registry mirror (e.g. a local cache).
67+
68+
```yaml
69+
machine:
70+
docker_registry_url: "https://registry.example.com"
71+
```
72+
73+
The check runs `docker info` and looks for the URL in the `Registry Mirrors` section.
74+
Also see [Container Registry →]({{< relref "container-registry" >}}).
75+
76+
## CPU Core Count
77+
78+
Verifies the total number of logical CPUs (threads, including SMT/HT siblings).
79+
80+
```yaml
81+
machine:
82+
cpu_cores: 16
83+
```
84+
85+
Detects hot-plug events or unexpected changes in Hyper-Threading state. If `cpu_smt` is also configured, the two checks complement each other.
86+
87+
## Installed RAM
88+
89+
Verifies total installed RAM in whole gigabytes.
90+
91+
```yaml
92+
machine:
93+
dram_gb: 64
94+
```
95+
96+
Detects failed or removed DIMMs.
97+
98+
## USB Device Allowlist
99+
100+
Warns when any connected USB device is **not** present in the allowlist. Each entry is matched as a substring against `lsusb` output lines.
101+
102+
```yaml
103+
machine:
104+
usb_devices:
105+
- "8087:0026" # Intel USB hub
106+
- "046d:c52b" # Logitech Unifying Receiver
107+
- "Linux Foundation" # internal root hubs
108+
```
109+
110+
Run `lsusb` on your machine to collect the expected entries. The check is skipped on macOS and Windows.
111+
112+
## PCI Device Allowlist
113+
114+
Warns when any connected PCI device is **not** present in the allowlist. Each entry is matched as a substring against `lspci` output lines.
115+
116+
```yaml
117+
machine:
118+
pci_devices:
119+
- "Network controller"
120+
- "SATA controller"
121+
- "VGA compatible controller"
122+
```
123+
124+
Run `lspci` on your machine to collect the expected entries. The check is skipped on macOS and Windows.
125+
126+
## CPU Scaling Governor
127+
128+
Verifies that **all** CPU cores use the expected frequency scaling governor. Governor changes after a reboot are one of the most common silent causes of measurement variance.
129+
130+
```yaml
131+
machine:
132+
cpu_governor: "performance"
133+
```
134+
135+
Reads `/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`. To lock the governor:
136+
137+
```sh
138+
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
139+
# or
140+
sudo cpupower frequency-set -g performance
141+
```
142+
143+
Also see [NOP Linux →]({{< relref "nop-linux" >}}).
144+
145+
## CPU Frequency
146+
147+
Verifies that the current frequency of **every** CPU core is within ±10 MHz of the configured value. Useful when frequency scaling is fully locked (e.g. `intel_pstate` with min = max).
148+
149+
```yaml
150+
machine:
151+
cpu_frequency_mhz: 3000
152+
```
153+
154+
## CPU Scaling Driver
155+
156+
Verifies that the cpufreq scaling driver matches the expected value on all cores.
157+
158+
```yaml
159+
machine:
160+
cpu_scaling_driver: "intel_pstate" # or "acpi-cpufreq", "cppc_cpufreq", etc.
161+
```
162+
163+
A different driver can apply different power and frequency policies even when the governor setting appears identical.
164+
165+
## SMT / Hyper-Threading
166+
167+
Verifies whether Simultaneous Multi-Threading (Intel HT / AMD SMT) is enabled or disabled.
168+
169+
```yaml
170+
machine:
171+
cpu_smt: false # true = SMT must be enabled, false = SMT must be disabled
172+
```
173+
174+
Reads `/sys/devices/system/cpu/smt/active`. To disable SMT:
175+
176+
```sh
177+
echo off | sudo tee /sys/devices/system/cpu/smt/control
178+
# For a permanent change add "nosmt" to GRUB_CMDLINE_LINUX in /etc/default/grub
179+
```
180+
181+
## CPU Turbo Boost
182+
183+
Verifies whether CPU frequency boost (Intel Turbo Boost / AMD Boost) is enabled or disabled.
184+
185+
```yaml
186+
machine:
187+
cpu_turbo_boost: false # true = boost must be on, false = boost must be off
188+
```
189+
190+
To disable Turbo Boost (Intel):
191+
192+
```sh
193+
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
194+
```
195+
196+
## System-wide Systemd Timers
197+
198+
This check runs as root and warns when any active system-wide `systemd` timer is detected. Timers can wake the system and create measurement noise.
199+
200+
This check is not configured via `config.yml` — it always runs on Linux. To inspect active timers:
201+
202+
```sh
203+
systemctl --all list-timers
204+
```
205+
206+
Also see [NOP Linux →]({{< relref "nop-linux" >}}) for a full list of services and timers to disable on cluster machines.
207+
208+
---
209+
210+
## All checks at a glance
211+
212+
| Config key | Default | Severity |
213+
|---|---|---|
214+
| `base_temperature_{chip,feature,value}` | `False` (skipped) | WARN (blocks run until stable) |
215+
| `rapl_power_capping.package` | `null` (skipped) | WARN |
216+
| `rapl_power_capping.dram` | `null` (skipped) | WARN |
217+
| `rapl_power_capping.psys` | `null` (skipped) | WARN |
218+
| `docker_registry_url` | `null` (skipped) | WARN |
219+
| `cpu_cores` | `null` (skipped) | WARN |
220+
| `dram_gb` | `null` (skipped) | WARN |
221+
| `usb_devices` | absent (skipped) | WARN |
222+
| `pci_devices` | absent (skipped) | WARN |
223+
| `cpu_governor` | `null` (skipped) | WARN |
224+
| `cpu_frequency_mhz` | `null` (skipped) | WARN |
225+
| `cpu_scaling_driver` | `null` (skipped) | WARN |
226+
| `cpu_smt` | `null` (skipped) | WARN |
227+
| `cpu_turbo_boost` | `null` (skipped) | WARN |
228+
| systemd timers (root, always on) | — | WARN |

‎content/en/docs/measuring/configuration.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,10 @@ Please see [cluster installation →]({{< relref "/docs/cluster/installation" >}
119119

120120
Also see [Resource Limits]({{< relref "/docs/measuring/resource-limits" >}}) to better understand how GMT enforces resource limits on its orchestrated containers.
121121

122+
GMT can also verify a range of optional hardware and OS properties before each measurement run — CPU governor, turbo boost, SMT state, installed RAM, connected USB/PCI devices, RAPL power limits, and more. All are opt-in via keys under `machine:` in `config.yml`.
123+
124+
See [Machine Baseline Checks →]({{< relref "/docs/cluster/machine-baseline-checks" >}}) for the full list and configuration reference.
125+
122126
### measurement
123127

124128
- `full_docker_prune_whitelist` **[list]**: A list of image names (without tag) or image IDs (short form) that will be whitelisted when `--full-docker-prune` is active. Images listed here will not be pruned. Useful for cluster installations where non security critical images shall be kept that take long to download.
@@ -188,7 +192,6 @@ For local installations these are to be found under [https://metrics.green-codin
188192
+ Example: *NetworkConnectionsProxyContainerProvider*
189193
- `flow-process-duration` **[integer]**: Max. duration in seconds for how long one flow should take. Timeout-Exception is thrown if exceeded.
190194
- `total-duration` **[integer]**: Max. duration in seconds for how long the whole run may take. Including building containers, baseline, idle, runtime and removal phases.
191-
- `phase-padding` **[integer]**: Phase padding is by default applied to the end of the phase to capture the last sampling tick, which might be cut-off. GMT applies one extra tick to the end of the phase. If your phase cut-offs must me microsecond exact you can turn this off. Typically not recommended and should be left on. See [https://github.com/green-coding-solutions/green-metrics-tool/issues/1129](https://github.com/green-coding-solutions/green-metrics-tool/issues/1129) for details.
192195
- `dev-no-sleeps` **[integer]**: Does not sleep in between phases and for cool-down periods. Beware that this will speed up runs on the cluster but render them invalid.
193196
- `dev-no-optimizations` **[integer]**: De-activates running the optimizations after a measurement.
194197

‎content/en/docs/prologue/measurement-phases.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,3 +59,14 @@ the `usage-scenario.yml` are run within the orchestrated application.
5959
In this phase the application architecture is being taken down
6060
and metric providers are being stopped.
6161
The system returns back to baseline and the measurement process is finished.
62+
63+
## Phase Boundaries and Sampling
64+
65+
Metric providers sample continuously in the background at their own configured
66+
interval, independent of when a phase starts or ends. Because a phase boundary
67+
can fall anywhere between two sampling ticks, the very last reading before the
68+
boundary could otherwise be cut off and lost from the phase it actually belongs to.
69+
70+
To avoid this, GMT automatically pads the end of every phase by one extra
71+
sampling tick, so that a reading which happened just before the boundary is
72+
still included. This is done transparently and needs no configuration.

0 commit comments

Comments
 (0)