Skip to content

Commit 15db771

Browse files
yarikopticclaude
andcommitted
refactor: rename "sidecar" to "companion" to avoid BIDS confusion
In BIDS, "sidecar" specifically refers to .json files accompanying data files. Rename all internal references to the _validation.jsonl file from "sidecar" to "companion" to avoid confusion: - validation_sidecar_path() → validation_companion_path() - _auto_save_sidecar() → _auto_save_companion() - Variable names, docstrings, comments, and spec prose The only remaining "sidecar" reference is in validate/types.py where it correctly describes BIDS sidecar JSON files. Co-Authored-By: Claude Code 2.1.81 / Claude Opus 4.6 <noreply@anthropic.com>
1 parent 6284d1a commit 15db771

6 files changed

Lines changed: 99 additions & 95 deletions

File tree

.specify/specs/validate-machine-readable-output.md

Lines changed: 61 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -98,13 +98,13 @@ stage 2 and makes stage 3 work from either stage 1 (live) or stage 2 (loaded).
9898
```
9999
Live validation: validate() → [ValidationResult] → filter/group → render
100100
101-
save(sidecar)
101+
save(companion)
102102
103103
Stored results: load(files) → [ValidationResult] → filter/group → render
104104
105-
Upload results: upload() → [ValidationResult] → save(sidecar)
105+
Upload results: upload() → [ValidationResult] → save(companion)
106106
107-
load(sidecar) → filter/group → render
107+
load(companion) → filter/group → render
108108
```
109109

110110
### Uniform output across all formats — no envelope/non-envelope split
@@ -413,8 +413,8 @@ def load_validation_jsonl(*paths: str | Path) -> list[ValidationResult]:
413413
return results
414414

415415

416-
def validation_sidecar_path(logfile: str | Path) -> Path:
417-
"""Derive the validation sidecar path from a log file path.
416+
def validation_companion_path(logfile: str | Path) -> Path:
417+
"""Derive the validation companion path from a log file path.
418418
419419
Given ``2026.03.19-14.30.00Z-12345.log``, returns
420420
``2026.03.19-14.30.00Z-12345_validation.jsonl`` in the same directory.
@@ -489,9 +489,9 @@ The `validate()` CLI function collects all results into a list (already does
489489
for filtering), then dispatches to `display_errors()` (human) or structured
490490
formatter.
491491

492-
### Phase 1b: File output (`-o/--output`) + auto-save sidecar
492+
### Phase 1b: File output (`-o/--output`) + auto-save companion
493493

494-
**Goal**: Write results to file and auto-save a `_validation.jsonl` sidecar
494+
**Goal**: Write results to file and auto-save a `_validation.jsonl` companion
495495
alongside the `.log` file.
496496

497497
#### CLI changes
@@ -512,11 +512,11 @@ Validation:
512512
`.yaml`/`.yml``yaml`). If extension is unrecognized, raise
513513
`click.UsageError`
514514

515-
#### Auto-save sidecar
515+
#### Auto-save companion
516516

517-
`dandi validate` writes a `_validation.jsonl` sidecar next to its log file
517+
`dandi validate` writes a `_validation.jsonl` companion next to its log file
518518
**by default** whenever there are results (any severity), regardless of
519-
`--format`. The sidecar is **skipped** only when:
519+
`--format`. The companion is **skipped** only when:
520520
- `--output` is specified (user already has structured results in a file)
521521
- `--load` is used (re-rendering existing results, not a fresh run)
522522
- No results are produced (clean validation)
@@ -525,21 +525,21 @@ Validation:
525525
# After collecting and filtering results, regardless of --format:
526526
if not load and not output_file and filtered:
527527
if hasattr(ctx, "obj") and ctx.obj is not None:
528-
_auto_save_sidecar(filtered, ctx.obj.logfile)
528+
_auto_save_companion(filtered, ctx.obj.logfile)
529529
```
530530

531-
**Important**: The sidecar must be written in ALL output format branches
531+
**Important**: The companion must be written in ALL output format branches
532532
(human, structured-to-stdout, structured-to-file), not just in the
533-
structured-to-stdout branch. A bug existed where the sidecar was only saved
533+
structured-to-stdout branch. A bug existed where the companion was only saved
534534
in the `else` branch (structured output to stdout), missing the most common
535535
case (human output, the default).
536536

537-
The sidecar accumulation rate matches the already-existing `.log` files.
537+
The companion accumulation rate matches the already-existing `.log` files.
538538

539539
The `validate` command's help text should mention this behavior so users
540540
know where to find saved results:
541541

542-
> Validation results are automatically saved as a JSONL sidecar next to the
542+
> Validation results are automatically saved as a JSONL companion next to the
543543
> dandi-cli log file (unless --output is specified). Use ``dandi validate
544544
> --load <path>`` to re-render saved results with different options.
545545
@@ -616,7 +616,7 @@ if load and paths:
616616
#### Example workflows
617617

618618
```bash
619-
# Run validation (auto-saves _validation.jsonl sidecar)
619+
# Run validation (auto-saves _validation.jsonl companion)
620620
dandi validate /data/dandiset001
621621

622622
# Re-render the auto-saved results with different filters
@@ -636,7 +636,7 @@ Note: shell glob expansion is done by the shell, not by Click. `--load
636636
results/*.jsonl` works because the shell expands the glob into multiple
637637
`--load` arguments before the CLI sees them.
638638

639-
### Phase 3: Upload validation sidecar — addresses #1753
639+
### Phase 3: Upload validation companion — addresses #1753
640640

641641
**Goal**: Persist all validation results from `dandi upload` for later inspection.
642642

@@ -659,20 +659,20 @@ function's validation happens inside a deeply nested generator (`_upload_item`).
659659
@click.option(
660660
"--validation-log",
661661
help="Path for writing validation results in JSONL format. "
662-
"Defaults to a sidecar file next to the dandi-cli log file. "
662+
"Defaults to a companion file next to the dandi-cli log file. "
663663
"Pass empty string to disable.",
664664
type=click.Path(dir_okay=False),
665665
default=None,
666666
)
667667
```
668668

669669
Behavior:
670-
- **Not specified** (default): derive sidecar from `ctx.obj.logfile` via
671-
`validation_sidecar_path()` — same as current implicit behavior
670+
- **Not specified** (default): derive companion from `ctx.obj.logfile` via
671+
`validation_companion_path()` — same as current implicit behavior
672672
- **Explicit path**: use that path directly as `validation_log_path`
673673
- **Empty string `""`**: pass `None` to `upload()`, disabling validation logging
674674

675-
This makes the validation sidecar feature visible in `dandi upload --help`
675+
This makes the validation companion feature visible in `dandi upload --help`
676676
and allows users to control where results are saved or opt out entirely.
677677

678678
#### Python API parameter
@@ -696,15 +696,15 @@ resolved path (or `None`) to `upload()`.
696696
# In cmd_upload.py:
697697
if validation_log is not None:
698698
# Explicit --validation-log: use as-is (empty string → disable)
699-
sidecar = Path(validation_log) if validation_log else None
699+
companion = Path(validation_log) if validation_log else None
700700
else:
701701
# Default: derive from logfile
702702
ctx = click.get_current_context()
703-
sidecar = None
703+
companion = None
704704
if ctx.obj is not None:
705-
sidecar = validation_sidecar_path(ctx.obj.logfile)
705+
companion = validation_companion_path(ctx.obj.logfile)
706706

707-
upload_(..., validation_log_path=sidecar)
707+
upload_(..., validation_log_path=companion)
708708

709709
# In upload.py, inside the upload loop (already implemented):
710710
if validation_log_path is not None and validation_statuses:
@@ -726,11 +726,11 @@ Uses `append_validation_jsonl()` from `dandi/validate/io.py` — the file is
726726
opened in append mode for each batch, allowing incremental writes as files are
727727
validated during upload without holding all results in memory.
728728

729-
#### Testing the upload validation sidecar
729+
#### Testing the upload validation companion
730730

731731
See **Step 3** in the Implementation Order section for detailed test descriptions.
732732

733-
Key insight: to test the default sidecar-next-to-logfile behavior, CLI tests
733+
Key insight: to test the default companion-next-to-logfile behavior, CLI tests
734734
must invoke `main` (the Click group), not `upload` directly, because `main()`
735735
sets up `ctx.obj.logfile`. Example:
736736

@@ -866,17 +866,17 @@ cat results/*.jsonl | jq -s '
866866
- Reuse `formatter.py` infrastructure for JSON/YAML
867867
- Tests: CliRunner for each format, round-trip serialization
868868

869-
### Step 1b: `--output` + auto-save sidecar
869+
### Step 1b: `--output` + auto-save companion
870870

871871
- Add `--output` option, auto-detect format from extension (`.json`
872872
`json_pp`, `.jsonl``json_lines`, `.yaml`/`.yml``yaml`); error
873873
if extension unrecognized and `--format` not given
874874
- Create `dandi/validate/io.py` with shared `write_validation_jsonl()`,
875875
`append_validation_jsonl()`, `load_validation_jsonl()`,
876-
`validation_sidecar_path()`
877-
- Auto-save `_validation.jsonl` sidecar when results are non-empty and
876+
`validation_companion_path()`
877+
- Auto-save `_validation.jsonl` companion when results are non-empty and
878878
`--output` is not specified (user already has their output file)
879-
- Tests: file creation, sidecar naming, empty-results no-op, sidecar
879+
- Tests: file creation, companion naming, empty-results no-op, companion
880880
suppressed when `--output` is used
881881

882882
### Step 1c: `--summary`
@@ -895,11 +895,11 @@ cat results/*.jsonl | jq -s '
895895
- Tests: load + filter, multi-file concatenation, mutual exclusivity error,
896896
forward-compatible loading of unknown versions
897897

898-
### Step 3: Upload validation sidecar — addresses #1753
898+
### Step 3: Upload validation companion — addresses #1753
899899

900900
#### 3a: Add `--validation-log` CLI option to `dandi upload`
901901

902-
Currently the upload command writes a validation sidecar derived from
902+
Currently the upload command writes a validation companion derived from
903903
`ctx.obj.logfile` (the dandi-cli log file), but this is invisible to the
904904
user — there is no `--help` text about it, no way to control the path, and
905905
no way to disable it.
@@ -910,15 +910,15 @@ Add an explicit `--validation-log` option:
910910
@click.option(
911911
"--validation-log",
912912
help="Path for writing validation results in JSONL format. "
913-
"Defaults to a sidecar file next to the dandi-cli log file. "
913+
"Defaults to a companion file next to the dandi-cli log file. "
914914
"Pass empty string to disable.",
915915
type=click.Path(dir_okay=False),
916916
default=None,
917917
)
918918
```
919919

920920
Behavior:
921-
- **Not specified** (default): derive sidecar from `ctx.obj.logfile` as today
921+
- **Not specified** (default): derive companion from `ctx.obj.logfile` as today
922922
- **Explicit path**: use that path directly
923923
- **Empty string `""`**: disable validation logging entirely
924924

@@ -931,39 +931,39 @@ needing to go through `main()` to get a logfile.
931931
- CLI wrapper resolves the `--validation-log` option → path or `None` and
932932
passes it through
933933

934-
#### 3c: Testing strategy for upload validation sidecar
934+
#### 3c: Testing strategy for upload validation companion
935935

936936
**Problem**: Existing upload tests call `SampleDandiset.upload()` which
937937
invokes the Python API directly. The Python API's `validation_log_path`
938938
parameter works, but the real end-to-end flow — where `dandi upload`
939-
automatically derives the sidecar from its log file — is untested. Also,
939+
automatically derives the companion from its log file — is untested. Also,
940940
invoking `CliRunner().invoke(upload, ...)` on the subcommand alone does not
941941
set `ctx.obj.logfile` (that is done by `main()`).
942942

943943
**Approach — two layers of tests**:
944944

945945
1. **Python API tests** (`dandi/tests/test_upload.py`): Extend existing
946946
tests that already exercise `SampleDandiset.upload()` to pass
947-
`validation_log_path` and verify sidecar contents. These test the core
947+
`validation_log_path` and verify companion contents. These test the core
948948
write logic without CLI overhead.
949949

950-
- `test_upload_bids_invalid`: pass `validation_log_path`, verify sidecar
950+
- `test_upload_bids_invalid`: pass `validation_log_path`, verify companion
951951
exists with ERROR-level results after `UploadError`
952952
- `test_upload_invalid_metadata`: same pattern for NWB validation errors
953-
- New `test_upload_validation_sidecar_clean`: upload valid NWB with
954-
`validation_log_path`, verify sidecar absent or contains only
953+
- New `test_upload_validation_companion_clean`: upload valid NWB with
954+
`validation_log_path`, verify companion absent or contains only
955955
sub-ERROR results
956956

957957
2. **CLI integration tests** (`dandi/cli/tests/test_cmd_upload.py` — new file):
958958
Use `CliRunner().invoke(main, ["upload", ...])` to go through `main()`
959959
so `ctx.obj.logfile` is set. These require the docker-compose archive
960960
fixture. Tests verify:
961961

962-
- Default behavior: `_validation.jsonl` sidecar appears next to the log
963-
file (derived via `validation_sidecar_path(ctx.obj.logfile)`)
964-
- `--validation-log /path/to/custom.jsonl`: sidecar at specified path
965-
- `--validation-log ""`: no sidecar written
966-
- Sidecar content is loadable via `load_validation_jsonl()` and contains
962+
- Default behavior: `_validation.jsonl` companion appears next to the log
963+
file (derived via `validation_companion_path(ctx.obj.logfile)`)
964+
- `--validation-log /path/to/custom.jsonl`: companion at specified path
965+
- `--validation-log ""`: no companion written
966+
- Companion content is loadable via `load_validation_jsonl()` and contains
967967
expected severity levels
968968

969969
The log file path can be discovered from the CliRunner output (dandi
@@ -972,10 +972,10 @@ set `ctx.obj.logfile` (that is done by `main()`).
972972

973973
```python
974974
@pytest.mark.ai_generated
975-
def test_upload_cli_validation_sidecar(
975+
def test_upload_cli_validation_companion(
976976
new_dandiset: SampleDandiset, tmp_path: Path
977977
) -> None:
978-
"""CLI upload creates validation sidecar next to log file."""
978+
"""CLI upload creates validation companion next to log file."""
979979
# ... set up dandiset with a file that has validation warnings/errors ...
980980
runner = CliRunner()
981981
r = runner.invoke(main, [
@@ -988,9 +988,9 @@ set `ctx.obj.logfile` (that is done by `main()`).
988988
log_files = sorted(logdir.glob("*.log"))
989989
assert log_files # at least one log was created
990990
latest_log = log_files[-1]
991-
sidecar = validation_sidecar_path(latest_log)
992-
if sidecar.exists():
993-
results = load_validation_jsonl(sidecar)
991+
companion = validation_companion_path(latest_log)
992+
if companion.exists():
993+
results = load_validation_jsonl(companion)
994994
assert len(results) > 0
995995
```
996996

@@ -999,7 +999,7 @@ set `ctx.obj.logfile` (that is done by `main()`).
999999
def test_upload_cli_validation_log_option(
10001000
new_dandiset: SampleDandiset, tmp_path: Path
10011001
) -> None:
1002-
"""--validation-log sends sidecar to explicit path."""
1002+
"""--validation-log sends companion to explicit path."""
10031003
custom_log = tmp_path / "my-validation.jsonl"
10041004
runner = CliRunner()
10051005
r = runner.invoke(main, [
@@ -1019,7 +1019,7 @@ set `ctx.obj.logfile` (that is done by `main()`).
10191019
def test_upload_cli_validation_log_disabled(
10201020
new_dandiset: SampleDandiset, tmp_path: Path
10211021
) -> None:
1022-
"""--validation-log '' disables sidecar."""
1022+
"""--validation-log '' disables companion."""
10231023
runner = CliRunner()
10241024
r = runner.invoke(main, [
10251025
"upload",
@@ -1050,7 +1050,7 @@ set `ctx.obj.logfile` (that is done by `main()`).
10501050
- Exit code semantics preserved: non-zero if any ERROR-severity issues
10511051
- When using `--format` other than `human`, colored output is suppressed
10521052
- When using `--load`, exit code reflects loaded results
1053-
- The auto-save sidecar is the only new side effect; it writes when there
1053+
- The auto-save companion is the only new side effect; it writes when there
10541054
are results and `--output` is not specified (for `validate`), or always
10551055
(for `upload`). Follows the same lifecycle as the existing `.log` files
10561056
- The `dandi/validate/` subpackage `__init__.py` re-exports all public API,
@@ -1067,9 +1067,9 @@ set `ctx.obj.logfile` (that is done by `main()`).
10671067
| `_record_version` | Unit | `validate/tests/test_types.py` | Serialization includes field, loader handles missing/unknown |
10681068
| `--load` | CLI unit | `cli/tests/test_cmd_validate.py` | Load from fixture files, multi-file concat, mutual exclusivity |
10691069
| `--output` | CLI unit | `cli/tests/test_cmd_validate.py` | Verify file creation, content matches stdout format |
1070-
| Sidecar auto-save (`validate`) | CLI unit | `cli/tests/test_cmd_validate.py` | Verify `_validation.jsonl` created next to mock logfile |
1071-
| Upload sidecar (Python API) | Integration | `tests/test_upload.py` | Pass `validation_log_path` to `SampleDandiset.upload()`, verify file + contents |
1072-
| Upload sidecar (CLI, default) | CLI integration | `cli/tests/test_cmd_upload.py` | `CliRunner().invoke(main, ["upload", ...])` — verify `_validation.jsonl` next to log file |
1070+
| Companion auto-save (`validate`) | CLI unit | `cli/tests/test_cmd_validate.py` | Verify `_validation.jsonl` created next to mock logfile |
1071+
| Upload companion (Python API) | Integration | `tests/test_upload.py` | Pass `validation_log_path` to `SampleDandiset.upload()`, verify file + contents |
1072+
| Upload companion (CLI, default) | CLI integration | `cli/tests/test_cmd_upload.py` | `CliRunner().invoke(main, ["upload", ...])` — verify `_validation.jsonl` next to log file |
10731073
| Upload `--validation-log` option | CLI integration | `cli/tests/test_cmd_upload.py` | Custom path, empty-string disable |
10741074
| Extended grouping | CLI unit | `cli/tests/test_cmd_validate.py` | Each grouping value, section headers, counts |
10751075
| Summary | CLI unit | `cli/tests/test_cmd_validate.py` | Verify counts match actual results |
@@ -1095,11 +1095,11 @@ All new tests marked `@pytest.mark.ai_generated`.
10951095
| `dandi/validate/tests/test_types.py` | **Renamed** from `dandi/tests/test_validate_types.py` |
10961096
| `dandi/validate/tests/test_io.py` | **New** — tests for I/O utilities |
10971097
| `dandi/cli/cmd_validate.py` | Refactor + add `--format`, `--output`, `--load`, `--summary`, grouping extensions |
1098-
| `dandi/upload.py` | Add `validation_log_path` parameter, write sidecar |
1099-
| `dandi/cli/cmd_upload.py` | Add `--validation-log` option, pass sidecar path to `upload()` |
1100-
| `dandi/cli/tests/test_cmd_upload.py` | **New** — CLI integration tests for upload validation sidecar |
1098+
| `dandi/upload.py` | Add `validation_log_path` parameter, write companion |
1099+
| `dandi/cli/cmd_upload.py` | Add `--validation-log` option, pass companion path to `upload()` |
1100+
| `dandi/cli/tests/test_cmd_upload.py` | **New** — CLI integration tests for upload validation companion |
11011101
| `dandi/cli/tests/test_cmd_validate.py` | Extend with format/load/output/summary tests |
1102-
| `dandi/tests/test_upload.py` | Extend existing tests to pass `validation_log_path` and verify sidecar |
1102+
| `dandi/tests/test_upload.py` | Extend existing tests to pass `validation_log_path` and verify companion |
11031103
| `dandi/pynwb_utils.py` | Update imports |
11041104
| `dandi/organize.py` | Update imports |
11051105
| `dandi/files/bases.py` | Update imports |

dandi/cli/cmd_upload.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ def upload(
9797
"""
9898
# Avoid heavy imports by importing with function:
9999
from ..upload import upload as upload_
100-
from ..validate.io import validation_sidecar_path
100+
from ..validate.io import validation_companion_path
101101

102102
if jobs_pair is None:
103103
jobs = None
@@ -106,7 +106,9 @@ def upload(
106106
jobs, jobs_per_file = jobs_pair
107107

108108
ctx = click.get_current_context()
109-
sidecar = validation_sidecar_path(ctx.obj.logfile) if ctx.obj is not None else None
109+
companion = (
110+
validation_companion_path(ctx.obj.logfile) if ctx.obj is not None else None
111+
)
110112

111113
upload_(
112114
paths,
@@ -119,5 +121,5 @@ def upload(
119121
jobs=jobs,
120122
jobs_per_file=jobs_per_file,
121123
sync=sync,
122-
validation_log_path=sidecar,
124+
validation_log_path=companion,
123125
)

0 commit comments

Comments
 (0)