Skip to content

Commit a1df8d6

Browse files
ktro2828ktro2828
authored
docs: update documents for sanity check (#221)
Signed-off-by: ktro2828 <kotaro.uetake@tier4.jp>
1 parent 72683e6 commit a1df8d6

6 files changed

Lines changed: 143 additions & 54 deletions

File tree

‎docs/apis/common.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,6 @@
77

88
::: t4_devkit.common.io
99

10-
::: t4_devkit.common.sanity
11-
1210
::: t4_devkit.common.serialize
1311

1412
::: t4_devkit.common.timestamp

‎docs/apis/sanity.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# `sanity`
2+
3+
<!-- prettier-ignore-start -->
4+
::: t4_devkit.sanity
5+
options:
6+
filters: ["!FMT*", "!REF*", "!REC*", "!STR*", "!TIV*"]
7+
show_bases: false
8+
<!-- prettier-ignore-end -->

‎docs/cli/t4sanity.md‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,10 +61,21 @@ $ t4sanity <DATA_ROOT>
6161
+-----------+---------+--------+--------+---------+----------+
6262
| DatasetID | Version | Passed | Failed | Skipped | Warnings |
6363
+-----------+---------+--------+--------+---------+----------+
64-
| dataset1 | | 49 | 0 | 2 | 3 |
64+
| dataset1 | 0 | 49 | 0 | 2 | 3 |
6565
+-----------+---------+--------+--------+---------+----------+
6666
```
6767
68+
### Exit Status Logic
69+
70+
`t4sanity` CLI returns the exit code based on the following conditions:
71+
72+
| Condition | `--strict` | Exit Code | Notes |
73+
| ----------------------------------------------------------------------- | ----------------- | --------- | --------------------------------------------------- |
74+
| At least one `Severity.ERROR` rule failed | N/A | 1 | Always fails the run |
75+
| At least one `Severity.WARNING` rule failed, no `Severity.ERROR` failed | `False` (default) | 0 | Run is considered successful, warnings are reported |
76+
| At least one `Severity.WARNING` rule failed, no `Severity.ERROR` failed | `True` | 1 | Treat warnings as failures; exit with failure |
77+
| All rules passed or skipped | N/A | 0 | Run is considered successful |
78+
6879
### Dump Results as JSON
6980
7081
To dump results into JSON, use the `-o; --output` option:
@@ -92,6 +103,18 @@ Then a JSON file named `result.json` will be generated as follows:
92103
}
93104
```
94105
106+
Here is the description of the JSON format:
107+
108+
- `dataset_id`: The ID of the dataset.
109+
- `version`: The version of the dataset.
110+
- `reports`: An array of rule reports.
111+
- `id`: The ID of the rule.
112+
- `name`: The name of the rule.
113+
- `severity`: How important a rule is.
114+
- `description`: A description of the rule.
115+
- `status`: What happened when it ran.
116+
- `reasons`: An array of reasons for failure or skipped rules, null if passed.
117+
95118
### Exclude Checks
96119
97120
With `-e; --excludes` option enables us to exclude specific checks by specifying the **rule IDs or groups**:

‎docs/schema/requirement.md‎

Lines changed: 51 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -2,74 +2,74 @@
22

33
## Structure (`STR`)
44

5-
| ID | Name | Severity | Description |
6-
| -------- | ----------------------------- | -------- | -------------------------------------------------------------------- |
7-
| `STR001` | `version-dir-presence` | `Warn` | `version/` directory exists under the dataset root directory. |
8-
| `STR002` | `annotation-dir-presence` | `Error` | `annotation/` directory exists under the dataset root directory. |
9-
| `STR003` | `data-dir-presence` | `Error` | `data/` directory exists under the dataset root directory. |
10-
| `STR004` | `map-dir-presence` | `Warn` | `map/` directory exists under the dataset root directory. |
11-
| `STR005` | `bag-dir-presence` | `Warn` | `input_bag/` directory exists under the dataset root directory. |
12-
| `STR006` | `status-file-presence` | `Warn` | `status.json` file exists under the dataset root directory. |
13-
| `STR007` | `schema-files-presence` | `Error` | Mandatory schema JSON files exist under the `annotation/` directory. |
14-
| `STR008` | `lanelet-file-presence` | `Warn` | `lanelet2_map.osm` file exists under the `map/` directory. |
15-
| `STR009` | `pointcloud-map-dir-presence` | `Warn` | `pointcloud_map.pcd` directory exists under the `map/` directory. |
5+
| ID | Name | Severity | Description |
6+
| -------- | ----------------------------- | --------- | -------------------------------------------------------------------- |
7+
| `STR001` | `version-dir-presence` | `WARNING` | `version/` directory exists under the dataset root directory. |
8+
| `STR002` | `annotation-dir-presence` | `ERROR` | `annotation/` directory exists under the dataset root directory. |
9+
| `STR003` | `data-dir-presence` | `ERROR` | `data/` directory exists under the dataset root directory. |
10+
| `STR004` | `map-dir-presence` | `WARNING` | `map/` directory exists under the dataset root directory. |
11+
| `STR005` | `bag-dir-presence` | `WARNING` | `input_bag/` directory exists under the dataset root directory. |
12+
| `STR006` | `status-file-presence` | `WARNING` | `status.json` file exists under the dataset root directory. |
13+
| `STR007` | `schema-files-presence` | `ERROR` | Mandatory schema JSON files exist under the `annotation/` directory. |
14+
| `STR008` | `lanelet-file-presence` | `WARNING` | `lanelet2_map.osm` file exists under the `map/` directory. |
15+
| `STR009` | `pointcloud-map-dir-presence` | `WARNING` | `pointcloud_map.pcd` directory exists under the `map/` directory. |
1616

1717
## Schema Record (`REC`)
1818

1919
| ID | Name | Severity | Description |
2020
| -------- | ----------------------------- | -------- | --------------------------------------- |
21-
| `REC001` | `scene-single` | `Error` | `Scene` record is a single. |
22-
| `REC002` | `sample-not-empty` | `Error` | `Sample` record is not empty. |
23-
| `REC003` | `sample-data-not-empty` | `Error` | `SampleData` record is not empty. |
24-
| `REC004` | `ego-pose-not-empty` | `Error` | `EgoPose` record is not empty. |
25-
| `REC005` | `calibrated-sensor-non-empty` | `Error` | `CalibratedSensor` record is not empty. |
26-
| `REC006` | `instance-not-empty` | `Error` | `Instance` record is not empty. |
21+
| `REC001` | `scene-single` | `ERROR` | `Scene` record is a single. |
22+
| `REC002` | `sample-not-empty` | `ERROR` | `Sample` record is not empty. |
23+
| `REC003` | `sample-data-not-empty` | `ERROR` | `SampleData` record is not empty. |
24+
| `REC004` | `ego-pose-not-empty` | `ERROR` | `EgoPose` record is not empty. |
25+
| `REC005` | `calibrated-sensor-non-empty` | `ERROR` | `CalibratedSensor` record is not empty. |
26+
| `REC006` | `instance-not-empty` | `ERROR` | `Instance` record is not empty. |
2727

2828
## Reference (`REF`)
2929

3030
| ID | Name | Severity | Description |
3131
| -------- | ------------------------------------- | -------- | ------------------------------------------------------------------------- |
32-
| `REF001` | `scene-to-log` | `Error` | `Scene.log_token` refers to `Log` record. |
33-
| `REF002` | `scene-to-first-sample` | `Error` | `Scene.first_sample_token` refers to `Sample` record. |
34-
| `REF003` | `scene-to-last-sample` | `Error` | `Scene.last_sample_token` refers to `Sample` record. |
35-
| `REF004` | `sample-to-scene` | `Error` | `Sample.scene_token` refers to `Scene` record. |
36-
| `REF005` | `sample-data-to-sample` | `Error` | `SampleData.sample_token` refers to `Sample` record. |
37-
| `REF006` | `sample-data-to-ego-pose` | `Error` | `SampleData.ego_pose_token` refers to `EgoPose` record. |
38-
| `REF007` | `sample-data-to-calibrated-sensor` | `Error` | `SampleData.calibrated_sensor_token` refers to `CalibratedSensor` record. |
39-
| `REF008` | `calibrated-sensor-to-sensor` | `Error` | `CalibratedSensor.sensor_token` refers to `Sensor` record. |
40-
| `REF009` | `instance-to-category` | `Error` | `Instance.category_token` refers to `Category` record. |
41-
| `REF010` | `instance-to-first-sample-annotation` | `Error` | `Instance.first_annotation_token` refers to `SampleAnnotation` record. |
42-
| `REF011` | `instance-to-last-sample-annotation` | `Error` | `Instance.last_annotation_token` refers to `SampleAnnotation` record. |
43-
| `REF012` | `lidarseg-to-sample-data` | `Error` | `LidarSeg.sample_data_token` refers to `SampleData` record. |
44-
| `REF013` | `sample-data-filename-presence` | `Error` | `SampleData.filename` exists. |
45-
| `REF014` | `sample-data-info-filename-presence` | `Error` | `SampleData.info_filename` exists if it is not `None`. |
46-
| `REF015` | `lidarseg-filename-presence` | `Error` | `LidarSeg.filename` exists if `lidarseg.json` exists. |
32+
| `REF001` | `scene-to-log` | `ERROR` | `Scene.log_token` refers to `Log` record. |
33+
| `REF002` | `scene-to-first-sample` | `ERROR` | `Scene.first_sample_token` refers to `Sample` record. |
34+
| `REF003` | `scene-to-last-sample` | `ERROR` | `Scene.last_sample_token` refers to `Sample` record. |
35+
| `REF004` | `sample-to-scene` | `ERROR` | `Sample.scene_token` refers to `Scene` record. |
36+
| `REF005` | `sample-data-to-sample` | `ERROR` | `SampleData.sample_token` refers to `Sample` record. |
37+
| `REF006` | `sample-data-to-ego-pose` | `ERROR` | `SampleData.ego_pose_token` refers to `EgoPose` record. |
38+
| `REF007` | `sample-data-to-calibrated-sensor` | `ERROR` | `SampleData.calibrated_sensor_token` refers to `CalibratedSensor` record. |
39+
| `REF008` | `calibrated-sensor-to-sensor` | `ERROR` | `CalibratedSensor.sensor_token` refers to `Sensor` record. |
40+
| `REF009` | `instance-to-category` | `ERROR` | `Instance.category_token` refers to `Category` record. |
41+
| `REF010` | `instance-to-first-sample-annotation` | `ERROR` | `Instance.first_annotation_token` refers to `SampleAnnotation` record. |
42+
| `REF011` | `instance-to-last-sample-annotation` | `ERROR` | `Instance.last_annotation_token` refers to `SampleAnnotation` record. |
43+
| `REF012` | `lidarseg-to-sample-data` | `ERROR` | `LidarSeg.sample_data_token` refers to `SampleData` record. |
44+
| `REF013` | `sample-data-filename-presence` | `ERROR` | `SampleData.filename` exists. |
45+
| `REF014` | `sample-data-info-filename-presence` | `ERROR` | `SampleData.info_filename` exists if it is not `None`. |
46+
| `REF015` | `lidarseg-filename-presence` | `ERROR` | `LidarSeg.filename` exists if `lidarseg.json` exists. |
4747

4848
## Format (`FMT`)
4949

5050
| ID | Name | Severity | Description |
5151
| -------- | ------------------------- | -------- | ------------------------------------------------- |
52-
| `FMT001` | `attribute-field` | `Error` | All types of `Attribute` fields are valid. |
53-
| `FMT002` | `calibrated-sensor-field` | `Error` | All types of `CalibratedSensor` fields are valid. |
54-
| `FMT003` | `category-field` | `Error` | All types of `Category` fields are valid. |
55-
| `FMT004` | `ego-pose-field` | `Error` | All types of `EgoPose` fields are valid. |
56-
| `FMT005` | `instance-field` | `Error` | All types of `Instance` fields are valid. |
57-
| `FMT006` | `log-field` | `Error` | All types of `Log` fields are valid. |
58-
| `FMT007` | `map-field` | `Error` | All types of `Map` fields are valid. |
59-
| `FMT008` | `sample-field` | `Error` | All types of `Sample` fields are valid. |
60-
| `FMT009` | `sample-annotation-field` | `Error` | All types of `SampleAnnotation` fields are valid. |
61-
| `FMT010` | `sample-data-field` | `Error` | All types of `SampleData` fields are valid. |
62-
| `FMT011` | `scene-field` | `Error` | All types of `Scene` fields are valid. |
63-
| `FMT012` | `sensor-field` | `Error` | All types of `Sensor` fields are valid. |
64-
| `FMT013` | `visibility-field` | `Error` | All types of `Visibility` fields are valid. |
65-
| `FMT014` | `lidarseg-field` | `Error` | All types of `Lidarseg` fields are valid. |
66-
| `FMT015` | `object-ann-field` | `Error` | All types of `ObjectAnn` fields are valid. |
67-
| `FMT016` | `surface-ann-field` | `Error` | All types of `SurfaceAnn` fields are valid. |
68-
| `FMT017` | `keypoint-field` | `Error` | All types of `Keypoint` fields are valid. |
69-
| `FMT018` | `vehicle-state-field` | `Error` | All types of `VehicleState` fields are valid. |
52+
| `FMT001` | `attribute-field` | `ERROR` | All types of `Attribute` fields are valid. |
53+
| `FMT002` | `calibrated-sensor-field` | `ERROR` | All types of `CalibratedSensor` fields are valid. |
54+
| `FMT003` | `category-field` | `ERROR` | All types of `Category` fields are valid. |
55+
| `FMT004` | `ego-pose-field` | `ERROR` | All types of `EgoPose` fields are valid. |
56+
| `FMT005` | `instance-field` | `ERROR` | All types of `Instance` fields are valid. |
57+
| `FMT006` | `log-field` | `ERROR` | All types of `Log` fields are valid. |
58+
| `FMT007` | `map-field` | `ERROR` | All types of `Map` fields are valid. |
59+
| `FMT008` | `sample-field` | `ERROR` | All types of `Sample` fields are valid. |
60+
| `FMT009` | `sample-annotation-field` | `ERROR` | All types of `SampleAnnotation` fields are valid. |
61+
| `FMT010` | `sample-data-field` | `ERROR` | All types of `SampleData` fields are valid. |
62+
| `FMT011` | `scene-field` | `ERROR` | All types of `Scene` fields are valid. |
63+
| `FMT012` | `sensor-field` | `ERROR` | All types of `Sensor` fields are valid. |
64+
| `FMT013` | `visibility-field` | `ERROR` | All types of `Visibility` fields are valid. |
65+
| `FMT014` | `lidarseg-field` | `ERROR` | All types of `Lidarseg` fields are valid. |
66+
| `FMT015` | `object-ann-field` | `ERROR` | All types of `ObjectAnn` fields are valid. |
67+
| `FMT016` | `surface-ann-field` | `ERROR` | All types of `SurfaceAnn` fields are valid. |
68+
| `FMT017` | `keypoint-field` | `ERROR` | All types of `Keypoint` fields are valid. |
69+
| `FMT018` | `vehicle-state-field` | `ERROR` | All types of `VehicleState` fields are valid. |
7070

7171
## Tier4 Instance (`TIV`)
7272

7373
| ID | Name | Severity | Description |
7474
| -------- | ------------ | -------- | ----------------------------------------------- |
75-
| `TIV001` | `load-tier4` | `Error` | Ensure `Tier4` instance is loaded successfully. |
75+
| `TIV001` | `load-tier4` | `ERROR` | Ensure `Tier4` instance is loaded successfully. |

‎docs/tutorials/sanity.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
## Sanity Check
2+
3+
### Quick Start
4+
5+
`sanity_check(...)` function performs a series of sanity checks to ensure the integrity of the dataset.
6+
7+
About the defined rules, please refer to [requirement.md](../schema/requirement.md).
8+
9+
```python
10+
from t4_devkit.common.io import save_json
11+
from t4_devkit.common.serialize import serialize_dataclass
12+
from t4_devkit.sanity import sanity_check, print_sanity_result
13+
14+
15+
result = sanity_check("<path/to/dataset>")
16+
17+
# display detailed results and summary
18+
print_sanity_result(result)
19+
20+
# save result to JSON file if you want
21+
save_json(serialize_dataclass(result), "result.json")
22+
```
23+
24+
### How to Add New Checkers
25+
26+
All checkers must follow:
27+
28+
- Implement a class that inherits from `Checker` class.
29+
- Its ID must be unique and belong to one of `RuleGroup` enum.
30+
- Override the `check() -> list[Reason] | None` method to perform the specific check.
31+
- Register the checker using `CHECKERS.register()` decorator.
32+
33+
```python
34+
from __future__ import annotations
35+
36+
from typing import TYPE_CHECKING
37+
38+
from t4_devkit.sanity.checker import Checker, RuleID, RuleName, Severity
39+
from t4_devkit.sanity.registry import CHECKERS
40+
from t4_devkit.sanity.result import Reason
41+
42+
if TYPE_CHECKING:
43+
from t4_devkit.sanity.context import SanityContext
44+
45+
46+
@CHECKERS.register()
47+
class STR000(Checker):
48+
"""This is a custom checker."""
49+
50+
id = RuleID("STR000")
51+
name = RuleName("my-custom-checker")
52+
severity = Severity.ERROR
53+
description = "This is a custom checker."
54+
55+
def check(self, context: SanityContext) -> list[Reason] | None:
56+
# Return a list of reasons if the check fails, or None if it passes.
57+
return None
58+
```

‎mkdocs.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,15 @@ nav:
1515
- Initialization: tutorials/initialize.md
1616
- Visualization: tutorials/render.md
1717
- Schema Customization: tutorials/customize.md
18+
- Sanity Check: tutorials/sanity.md
1819
- CLI:
1920
- Home: cli/index.md
2021
- t4viz: cli/t4viz.md
2122
- t4sanity: cli/t4sanity.md
2223
- API References:
2324
- t4_devkit.tier4: apis/tier4.md
2425
- t4_devkit.helper: apis/helper.md
26+
- t4_devkit.sanity: apis/sanity.md
2527
- t4_devkit.schema:
2628
- Home: apis/schema/index.md
2729
- Schema Names: apis/schema/name.md

0 commit comments

Comments
 (0)