Skip to content

Commit 792e1a4

Browse files
midgemacfMacFarland, Midgie
midgemacf
and
MacFarland, Midgie
authored
Release/1.1.4 (#10)
## [1.1.4] - 2025-08-22 ### Added - 🔥 Random data generation for all supported probes - 🔥 Public and "all" view in Grafana Dashboards ### Fixed - 🩹 Autopopulation of unnamed probe identifiers in Grafana Dashboards Co-authored-by: MacFarland, Midgie <macfarlandmj@ornl.gov>
1 parent 70c7e3f commit 792e1a4

14 files changed

Lines changed: 3805 additions & 21 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,15 @@ This project adheres to [Semantic Versioning](https://semver.org/).
3737
*Unreleased* versions radiate potential—-and dread. Once you merge an infernal PR, move its bullet under a new version heading with the actual release date.*
3838
3939
-->
40-
## [1.1.3] - 2025-08
40+
## [1.1.4] - 2025-08-22
41+
### Added
42+
- 🔥 Random data generation for all supported probes
43+
- 🔥 Public and "all" view in Grafana Dashboards
44+
45+
### Fixed
46+
- 🩹 Autopopulation of unnamed probe identifiers in Grafana Dashboards
47+
48+
## [1.1.3] - 2025-08-18
4149
### Added
4250
- 🔥 Microchip TP4100 Load
4351
- 🔥 Microchip TP4100 Collection

‎docs/guides/random-data-generation.md‎

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
# Random Data Generation
2+
3+
The `opensampl load random` command allows you to generate synthetic test data for all supported probe types. This is useful for testing, development, and demonstration purposes.
4+
5+
The data will be uploaded to your database in whichever way is configured with opensampl config, as with any load command.
6+
7+
## Usage
8+
9+
```bash
10+
opensampl load random <PROBE_TYPE> [OPTIONS]
11+
```
12+
13+
Where `<PROBE_TYPE>` can be any supported vendor:
14+
- `ADVA` - ADVA clock probes
15+
- `MicrochipTWST` - Microchip TWST modems
16+
- `MicrochipTP4100` - Microchip TP4100 GPS receivers
17+
18+
## Basic Examples
19+
20+
Generate data for a single ADVA probe with default settings:
21+
```bash
22+
opensampl load random adva
23+
```
24+
25+
Generate data for multiple probes:
26+
```bash
27+
opensampl load random adva --num-probes 5
28+
```
29+
30+
Generate 24 hours of data with specific parameters:
31+
```bash
32+
opensampl load random adva --duration 24 --sample-interval 0.5
33+
```
34+
35+
## Configuration Options
36+
37+
### Basic Parameters
38+
39+
| Option | Type | Default | Description |
40+
|--------|---|---------|-------------|
41+
| `--num-probes` | int | 1 | Number of probes to generate data for |
42+
| `--duration` | float | 1.0 | Duration of data in hours |
43+
| `--seed` | int | None | Random seed for reproducible results |
44+
| `--sample-interval` | float | 1.0 | Sample interval in seconds |
45+
46+
### Time Series Parameters
47+
48+
| Option | Type | Description |
49+
|--------|------|-------------|
50+
| `--base-value` | float | Base value for time offset measurements |
51+
| `--noise-amplitude` | float | Noise amplitude/standard deviation |
52+
| `--drift-rate` | float | Linear drift rate per second |
53+
| `--outlier-probability` | float | Probability of outliers per sample (default: 0.01) |
54+
| `--outlier-multiplier` | float | Multiplier for outlier noise amplitude (default: 10.0) |
55+
56+
### Probe Identity Parameters
57+
58+
| Option | Type | Description |
59+
|--------|------|-----------------------------------------------------|
60+
| `--probe-ip` | str | Specific IP address for the probe (randomly generated if not provided) |
61+
| `--probe-id` | str | Specific probe ID (available for ADVA and MicrochipTP4100; randomly generated/incremented if not provided) |
62+
63+
### Microchip TWST Specific Parameters
64+
65+
| Option | Type | Description |
66+
|--------|------|-------------|
67+
| `--num-channels` | int | Number of remote channels to generate data for (default: 4) |
68+
| `--ebno-base-value` | float | Base value for Eb/No measurements in dB |
69+
| `--ebno-noise-amplitude` | float | Noise amplitude/standard deviation for Eb/No measurements in dB |
70+
| `--ebno-drift-rate` | float | Linear drift rate per second for Eb/No measurements in dB/s |
71+
72+
## Configuration Files
73+
74+
You can use YAML configuration files to specify complex parameters, which are all named identically to the cli options:
75+
76+
```bash
77+
opensampl load random adva --config my_test_config.yaml
78+
```
79+
80+
Example configuration file (`test_config.yaml`):
81+
```yaml
82+
# Basic settings
83+
num_probes: 3
84+
duration_hours: 2.0
85+
seed: 42
86+
87+
# Time series parameters
88+
sample_interval: 0.5
89+
base_value: 1.5e-7
90+
noise_amplitude: 2.3e-9
91+
drift_rate: -5.2e-13
92+
outlier_probability: 0.02
93+
outlier_multiplier: 15.0
94+
```
95+
96+
Command-line options take precedence over configuration file values.
97+
98+
## How It Works
99+
100+
### Data Generation Process
101+
102+
1. **Probe Identity**: Each probe is assigned a unique combination of IP address and probe ID
103+
- IP addresses are randomly generated in the format `X.X.X.X` if not specified
104+
- Probe IDs are automatically incremented (1, 2, 3...) if not specified, or based on the provided probe ID for multiple probes
105+
106+
2. **Time Series Generation**: Creates realistic time offset measurements with:
107+
- **Base Value**: Starting offset value (vendor-specific defaults)
108+
- **Linear Drift**: Systematic change over time
109+
- **Random Noise**: Gaussian noise around the drift line
110+
- **Outliers**: Occasional large deviations for realism
111+
112+
3. **Database Storage**:
113+
- Probe metadata is automatically generated and stored
114+
- Time series data is sent directly to the database
115+
- All data uses appropriate database schemas for each vendor
116+
117+
### Vendor-Specific Defaults
118+
119+
Each probe type has realistic default values:
120+
121+
**ADVA Probes:**
122+
- Base value: Random between -1µs to 1µs
123+
- Noise amplitude: Random between 1ns to 10ns
124+
- Drift rate: Random between -1e-12 to 1e-12 s/s
125+
126+
**Microchip TWST Modems:**
127+
- Base value: Random between -10ns to 10ns
128+
- Noise amplitude: Random between 0.1ns to 1ns
129+
- Drift rate: Random between -1e-12 to 1e-12 s/s
130+
- Eb/No base value: Random between 10.0 to 20.0 dB
131+
- Eb/No noise amplitude: Random between 0.5 to 2.0 dB
132+
- Eb/No drift rate: Random between -0.01 to 0.01 dB/s
133+
- Default channels: 4 remote channels
134+
- Fixed probe ID: "modem"
135+
136+
**Microchip TP4100 GPS:**
137+
- Base value: Random between -500ns to 500ns
138+
- Noise amplitude: Random between 10ns to 50ns
139+
- Drift rate: Random between -1e-10 to 1e-10 s/s
140+
- Measurement type: "time-error (ns)" vs GNSS reference
141+
- Supports custom probe IDs
142+
143+
### Reproducibility
144+
145+
Use the `--seed` parameter for reproducible data:
146+
```bash
147+
# These commands will generate identical data
148+
opensampl load random adva --seed 123 --num-probes 2
149+
opensampl load random adva --seed 123 --num-probes 2
150+
```
151+
152+
When generating multiple probes with a seed, each probe gets a unique but deterministic seed (original + probe index).
153+
154+
## Advanced Usage
155+
156+
### Multiple Probe Types
157+
Generate data for different vendors:
158+
```bash
159+
# Generate ADVA data
160+
opensampl load random adva --num-probes 2 --duration 12
161+
162+
# Generate Microchip TWST data
163+
opensampl load random twst --num-probes 3 --duration 8
164+
165+
# Generate TP4100 GPS data
166+
opensampl load random tp4100 --num-probes 1 --duration 24
167+
```
168+
169+
### High-Frequency Data
170+
Generate high-resolution data:
171+
```bash
172+
opensampl load random adva --duration 0.1 --sample-interval 0.01 --num-probes 1
173+
```
174+
This creates 6 minutes of data sampled every 10ms (36,000 samples).
175+
176+
### Testing Scenarios
177+
Create specific test scenarios:
178+
```bash
179+
# High drift scenario
180+
opensampl load random adva --drift-rate 1e-10 --duration 24
181+
182+
# High noise scenario
183+
opensampl load random adva --noise-amplitude 1e-6 --outlier-probability 0.1
184+
185+
# Stable reference scenario
186+
opensampl load random adva --drift-rate 0 --noise-amplitude 1e-12 --outlier-probability 0
187+
```
188+
189+
### Microchip TWST Examples
190+
Generate TWST modem data with specific parameters:
191+
```bash
192+
# Generate data for 8 channels with high Eb/No
193+
opensampl load random twst --num-channels 8 --ebno-base-value 25.0 --duration 12
194+
195+
# Low Eb/No scenario with more noise
196+
opensampl load random twst --ebno-base-value 12.0 --ebno-noise-amplitude 3.0 --ebno-drift-rate -0.05
197+
198+
# Multiple probes with specific configuration
199+
opensampl load random twst --num-probes 3 --num-channels 6 --duration 4
200+
```
201+
202+
### Microchip TP4100 Examples
203+
Generate TP4100 GPS receiver data:
204+
```bash
205+
# Generate data with specific probe ID
206+
opensampl load random tp4100 --probe-id "GPS-Site-A" --duration 24
207+
208+
# High precision scenario
209+
opensampl load random tp4100 --base-value 0 --noise-amplitude 5e-9 --drift-rate 0
210+
211+
# Multiple GPS receivers
212+
opensampl load random tp4100 --num-probes 5 --probe-id "GPS-Array" --duration 48
213+
```
214+
215+
## Output
216+
217+
After successful generation, you'll see a summary:
218+
```
219+
=== Generated 3 ADVA probes ===
220+
- 192.168.1.100:1
221+
- 10.0.0.85:2
222+
- 172.16.42.200:3
223+
```
224+
225+
The generated data is immediately available in your database and can be accessed through the regular openSAMPL APIs and analysis tools.

‎mkdocs.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ nav:
1414
- Create: guides/create_probe_type.md
1515
- Server: guides/opensampl-server.md
1616
- Collect: guides/collect.md
17+
- Random Data: guides/random-data-generation.md
1718
- API:
1819
- index: api/index.md
1920
- ats6502:

‎opensampl/cli.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,8 +179,14 @@ def load():
179179
"""Load data into database"""
180180

181181

182+
@load.group(cls=CaseInsensitiveGroup)
183+
def random():
184+
"""Generate and send random test data to the database"""
185+
186+
182187
for vendor in VENDORS.all():
183188
load.add_command(vendor.get_parser().get_cli_command(), name=vendor.name)
189+
random.add_command(vendor.get_parser().get_random_data_cli_command(), name=vendor.name)
184190

185191

186192
def path_or_string(value: str) -> Union[dict, list]:

0 commit comments

Comments
 (0)