Barseq blackbox acquisition by dougollerenshaw · Pull Request #1781 · AllenNeuralDynamics/aind-data-schema

dougollerenshaw · 2026-03-20T20:44:48Z

Creates simple black-box style acquisition.json files for barseq acquisitions for subjects 780345 and 780346.

Note that I had a much more involved/complicated PR that attempted to capture the actual acquisition details here: #1690

I abandoned that effort after Polina shared the raw data paths and @saskiad and I spent some time trying to understand the structure of the actual raw acquisition files. We realized there was simply too much we didn't understand about how the image files were acquired and subsequently processed for us to be able to accurately capture the process for now.

This PR closes https://github.com/AllenNeuralDynamics/aind-scientific-computing/issues/569 and closes #1761

…cquisition for two subjects, plus json outputs

saskiad · 2026-03-20T21:20:58Z

+      "https://www.protocols.io/view/barseq-2-5-kqdg3ke9qv25/v1"
+   ],
+   "ethics_review_id": null,
+   "instrument_id": "Dogwood",


there shouldn't be an instrument for black box

Are you sure here? I can understand instrument not being required in a black-box acquisition. But in this case we have the instrument name and the corresponding instrument.json. I don't see a downside to including it. I can remove if you'd still prefer, but I just wanted to double check first.

yeah, I'm sure. We are treating this as we send it to a team and they send us something back. What they do is a black box.

saskiad · 2026-03-20T21:28:32Z

+        # Source: /allen/aind/stage/barseq/20250303_780345_slide7_maxprojection/experiment_detail.txt
+        #         /allen/aind/stage/barseq/20250305_780345_slide8_maxprojection/experiment_detail.txt
+        #         /allen/aind/stage/barseq/20250317_780345_slide9_maxprojection/experiment_detail.txt
+        "experimenters": ["Mara Rue", "Shannon Khem", "Tracy"],


if we're doing black box, I'd have this be Barseq team and not name people.

Done. See 0272459

saskiad · 2026-03-20T21:28:58Z

+
+# Instrument ID for the Nikon Ti2-E "Dogwood" spinning disk confocal.
+# Source: aind-data-schema barseq instrument PR #1685
+INSTRUMENT_ID = "Dogwood"


no instrument id for black box.

See my comment above: #1781 (comment)

dbirman

Seems fine from my perspective. I wouldn't include paths to S3, relative, or VAST locations in general

…nNeuralDynamics/aind-data-schema into barseq_blackbox_acquisition

…ed example barseq

…nNeuralDynamics/aind-data-schema into barseq_blackbox_acquisition

dougollerenshaw · 2026-04-01T20:22:23Z

This PR now includes a new "ExternalDataStream" class that makes instrument_id optional.

Note that the two json files in the examples/2026.01.16_barseq_acquisition will be removed before merging. These are only here to verify that the actual acquisition files are reasonable.

dbirman

Requesting some changes, see comments.

Also, I think you still need to add a test in test_metadata.py that creates a Metadata object with an acquisition that has only an external data stream and no instrument. This should crash the Metadata-level validator that checks the core file requirements and then we can modify how the requirements work to allow this.

…o instrument warning is raised for external acquisitions.

dougollerenshaw · 2026-04-03T00:40:53Z

Requesting some changes, see comments.

Also, I think you still need to add a test in test_metadata.py that creates a Metadata object with an acquisition that has only an external data stream and no instrument. This should crash the Metadata-level validator that checks the core file requirements and then we can modify how the requirements work to allow this.

See this commit: aa5f520

dougollerenshaw · 2026-04-03T00:41:06Z

OK, I think everything is addressed. Mind another look?

dbirman

This looks good to me, hard to know whether every edge case is handled for data_stream having two kinds of objects now but we'll find out pretty quickly if there are issues!

A few minor comments on wording, not critical to change them but I think it will be a tiny bit easier for users with the changes.

dbirman · 2026-04-03T01:01:21Z

 | `protocol_id` | `Optional[List[str]]` | Protocol ID (DOI for protocols.io) |
 | `ethics_review_id` | `Optional[List[str]]` | Ethics review ID  |
-| `instrument_id` | `str` | Instrument ID (Should match the Instrument.instrument_id) |
+| `instrument_id` | `Optional[str]` | Instrument ID (Should match the Instrument.instrument_id. Required when data_streams contains a DataStream.) |


Not blocking but I think my preferred language is "Required when instrument metadata is available."

See d29c08d

dbirman · 2026-04-03T01:02:06Z

 | `calibrations` | List[[Calibration](components/measurements.md#calibration) or [VolumeCalibration](components/measurements.md#volumecalibration) or [PowerCalibration](components/measurements.md#powercalibration)] | Calibrations (List of calibration measurements taken prior to acquisition.) |
 | `maintenance` | List[[Maintenance](components/measurements.md#maintenance)] | Maintenance (List of maintenance on instrument prior to acquisition.) |
-| `data_streams` | List[[DataStream](acquisition.md#datastream)] | Data streams (A data stream is a collection of devices that are acquiring data simultaneously. Each acquisition can include multiple streams. Streams should be split when configurations are changed.) |
+| `data_streams` | List[[DataStream](acquisition.md#datastream) or [ExternalDataStream](acquisition.md#externaldatastream)] | Data streams (A data stream is a collection of devices that are acquiring data simultaneously. Each acquisition can include multiple streams. Streams should be split when configurations are changed. Use ExternalDataStream for acquisitions performed externally where full instrument metadata is unavailable.) |


Same comment as above here "for acquisitions where instrument metadata is unavailable."

see d29c08d

dbirman · 2026-04-03T01:02:35Z


+### ExternalDataStream
+
+A simplified data stream for acquisitions performed externally, where full instrument metadata is unavailable.


You can drop the "full". If there's partial instrument metadata we would still want it.

See d29c08d

Stale review. Dan has approved after updates.

* Allow specimen_id to be a list (#1770) * Allow specimen_id to be a list * Removed nonetype test * Fixed specimen names in test * Updated specimen_id description to clarify expectation during in-vivo sessions * Spread args over multi lines to avoid lint fail * docs: rebuild docs (#1771) * docs: minor fixes to acquisition.md * docs: remove bad directives and links * docs: improve description for AcquisitionSubjectDetails (#1772) * docs: improve description for AcquisitionSubjectDetails * docs: rebuild docs * fix: throw warnings when users include multiple identically named devices in the Instrument (#1773) * fix: allow both planar and regular sectioning in sectioning procedures (#1777) * fix: allow NHPSubject as subject_details option (#1778) * docs: auto-build docs on opening PR (#1760) * docs: auto-build docs on opening PR * update docs [skip actions] * docs: update contribution.md * update docs [skip actions] * chore: remove all unnecessary graphviz dependencies * update docs [skip actions] * update docs [skip actions] --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Handle lists of specimen IDs in procedures validator (#1783) * Handle lists of specimen IDs in procedures validator * Fix to properly handle individual str as well as lists of str * refactor: simplifying code a tiny bit --------- Co-authored-by: Dan Birman <danbirman@gmail.com> * Add ticket linking workflow (#1785) * Barseq blackbox acquisition (#1781) * First pass at minimal blackbox acquisition. Has file that generates acquisition for two subjects, plus json outputs * Added reference to output files * Added specimen IDs to barseq acquisitions * Removed readme file * Updating examples/barseq_acquisition.py docstring * Cleaned up comments in examples/barseq_acquisition.py * Set timestamps to pacific with offset * Changed experimenters to 'Barseq team' * Removed references to local paths, removed duplicate jsons * Added ExternalDataStream, made instrument optional, added test, updated example barseq * Updated example json outputs for barseq subjects * update docs [skip actions] * Linting fixes * Used DiscriminatedList * Clean up stream adding logic * Simplify barseq acquisition example * Removed jsons * clean up test and import properly * Add ExternalDataStream support to Metadata validators and test that no instrument warning is raised for external acquisitions. * Wording updates as suggested by Dan --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * 1792 quality control validator has a bug at line 307 where it fails to check metrics before doing metrics0 (#1793) * fix: validator would trigger on empty metrics * fix: do this upfront by bypassing the validator, adds a test * chore: delete CHANGELOG.md (#1799) * chore: delete CHANGELOG.md * update docs [skip actions] --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * refactor: replace root calls to logging with module-level loggers (#1800) * fix: issue with actions being skipped unnecessarily * refactor: replace root logger with local loggers * test: fixes to logger patches * docs: improve docstring (#1748) * docs: improve docstring * docs: update specimen ID * docs: clarify suffix * docs: better writing * update docs * fix: bad merge * update docs --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * refactor: only allow wrapped NonSurgeryInjections in Procedures.subject_procedures (#1802) * refactor: only allow Injections in Procedures.subject_procedures when wrapped with an InjectionProcedure * chore: add ethics_review_id/protocol_id * fix: only allow Injection, BrainInjection requires a full CS from Surgery * update docs [skip actions] * refactor: NonSurgicalInjection * update docs --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add missing `tzdata` dependency (#1811) * Add git hash to Code model (#1813) * Add git hash to Code model * update docs * Increase hash length upper bound * Change property name and emit warning if hash and version are not defined * Refactor type alias * update docs * Break lines * Update tests to reflect longer hashes --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * feat: add barseq instrument (#1685) * feat: add barseq instrument * refactor: SlapPlane -> Slap2Plane * Revert "refactor: SlapPlane -> Slap2Plane" This reverts commit 6f193b8. * chore: lint * refactor: fix serialization code in example * fix: fix issues blocking build * chore: fix write function for instrument example * fix: specimen_ids validator correctly flattens lists (#1763) * feat: working on procedures_sectioning example * feat: continuing to develop chunk sections * chore: embed slide regions in python code * chore: lint * chore: docstrings * chore: lint * chore: remove full sectioning file --------- Co-authored-by: Doug Ollerenshaw <doug.ollerenshaw@alleninstitute.org> * chore: bump version * bump schema version [skip actions] * chore: bump metadata * bump schema version [skip actions] --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Doug Ollerenshaw <doug.ollerenshaw@alleninstitute.org> Co-authored-by: gouwens <ngouwens@gmail.com> Co-authored-by: Bruno Cruz <7049351+bruno-f-cruz@users.noreply.github.com>

dougollerenshaw added 6 commits March 20, 2026 12:39

First pass at minimal blackbox acquisition. Has file that generates a…

dfa78a5

…cquisition for two subjects, plus json outputs

Added reference to output files

8d33892

Added specimen IDs to barseq acquisitions

e4b3cb2

Removed readme file

508e540

Updating examples/barseq_acquisition.py docstring

96b40ae

Cleaned up comments in examples/barseq_acquisition.py

7275b29

dougollerenshaw requested a review from saskiad March 20, 2026 20:51

saskiad reviewed Mar 20, 2026

View reviewed changes

Comment thread examples/barseq_780345_acquisition.json Outdated

saskiad reviewed Mar 20, 2026

View reviewed changes

saskiad previously requested changes Mar 20, 2026

View reviewed changes

dougollerenshaw added 3 commits March 25, 2026 12:10

Set timestamps to pacific with offset

62a9dce

Changed experimenters to 'Barseq team'

0272459

Merge branch 'dev' into barseq_blackbox_acquisition

20947ed

dougollerenshaw requested review from dbirman and saskiad March 25, 2026 19:18

dbirman approved these changes Mar 25, 2026

View reviewed changes

Comment thread examples/barseq_780346_acquisition.json Outdated

dougollerenshaw and others added 9 commits March 25, 2026 12:36

Removed references to local paths, removed duplicate jsons

289eec5

Merge branch 'barseq_blackbox_acquisition' of https://github.com/Alle…

7ae12b8

…nNeuralDynamics/aind-data-schema into barseq_blackbox_acquisition

Merge branch 'dev' into barseq_blackbox_acquisition

c6a9639

Added ExternalDataStream, made instrument optional, added test, updat…

ecd9e1f

…ed example barseq

Updated example json outputs for barseq subjects

cdab299

Merge branch 'barseq_blackbox_acquisition' of https://github.com/Alle…

873f691

…nNeuralDynamics/aind-data-schema into barseq_blackbox_acquisition

update docs [skip actions]

41d3e9a

Linting fixes

db81c5a

Merge branch 'barseq_blackbox_acquisition' of https://github.com/Alle…

e77770c

…nNeuralDynamics/aind-data-schema into barseq_blackbox_acquisition

dougollerenshaw requested a review from dbirman April 1, 2026 20:20

dbirman requested changes Apr 1, 2026

View reviewed changes

dougollerenshaw added 6 commits April 2, 2026 14:47

Used DiscriminatedList

b303d4b

Clean up stream adding logic

89250f2

Simplify barseq acquisition example

ca8c683

Removed jsons

2ab00e1

clean up test and import properly

6fc71ae

Add ExternalDataStream support to Metadata validators and test that n…

aa5f520

…o instrument warning is raised for external acquisitions.

dougollerenshaw requested a review from dbirman April 3, 2026 00:41

dbirman approved these changes Apr 3, 2026

View reviewed changes

Wording updates as suggested by Dan

d29c08d

dougollerenshaw enabled auto-merge April 3, 2026 01:56

dougollerenshaw added this pull request to the merge queue Apr 3, 2026

dougollerenshaw removed the request for review from saskiad April 3, 2026 01:57

Merged via the queue into dev with commit 1f0a7d9 Apr 3, 2026
6 checks passed

dougollerenshaw deleted the barseq_blackbox_acquisition branch April 3, 2026 01:59


		### ExternalDataStream

		A simplified data stream for acquisitions performed externally, where full instrument metadata is unavailable.

Uh oh!

Conversation

dougollerenshaw commented Mar 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbirman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dougollerenshaw commented Apr 1, 2026

Uh oh!

dbirman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dougollerenshaw commented Apr 3, 2026

Uh oh!

dougollerenshaw commented Apr 3, 2026

Uh oh!

dbirman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants