doc: Document current dataset lifecycle

docs/conf.py

@@ -14,6 +14,7 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
 
 extensions = [
+    'sphinxcontrib.mermaid',
     'myst_parser',
     'sphinx_copybutton',
     'sphinxext.rediraffe',

docs/dataset-lifecycle.md

@@ -0,0 +1,268 @@
+---
+name: Dataset Lifecycle
+route: /dataset-lifecycle
+---
+
+# Dataset Lifecycle
+
+Datasets are git/git-annex repositories managed by datalad-service. This
+document describes the states a dataset, its snapshots, and their DOIs
+pass through, and the mutations that drive transitions between them.
+
+## Dataset states
+
+A dataset's state is determined by two fields: whether it has any
+snapshots and whether it is public.
+
+```{mermaid}
+stateDiagram-v2
+    [*] --> Draft
+    Draft --> Embargoed : createSnapshot()
+    Embargoed --> Public : publishDataset()
+    Public --> Embargoed : updatePublic(false) [admin]
+    Draft --> Deleted : deleteDataset()
+    Embargoed --> Deleted : deleteDataset()
+    Public --> Deleted : deleteDataset()
+```
+
+| State         | Determination        | DB representation                                  |
+| ------------- | -------------------- | -------------------------------------------------- |
+| **Draft**     | Unversioned, private | `Dataset.public = false`, no `Snapshot` docs exist |
+| **Embargoed** | Versioned, private   | `Dataset.public = false`, ≥1 `Snapshot` doc exists |
+| **Public**    | Versioned, public    | `Dataset.public = true`, `Dataset.publishDate` set |
+| **Deleted**   | Dataset removed      | `Deletion` doc created                             |
+
+These states are implicit — there is no `Dataset.status` field. The code
+does not use the term "embargoed"; both Draft and Embargoed datasets have
+`public = false` and are distinguished only by the presence of snapshots.
+
+**Key constraints:**
+
+- Draft → Embargoed is irreversible. Once a snapshot exists, the dataset
+  cannot return to Draft.
+- Public → Embargoed requires admin privileges.
+
+### Creation
+
+`createDataset()` creates a new `Dataset` document with `public: false`
+and fires a `{ type: "created" }` event.
+
+- **JS:** `packages/openneuro-server/src/datalad/dataset.ts`
+
+### Snapshotting (Draft → Embargoed, or new version)
+
+`createSnapshot()` tags the current HEAD as a new version. The first
+snapshot transitions the dataset from Draft to Embargoed. Subsequent
+snapshots do not change the dataset state.
+
+- **JS:** `packages/openneuro-server/src/datalad/snapshots.ts`
+- **Python:** `services/datalad/datalad_service/tasks/snapshots.py`
+- **Event:** `{ type: "versioned", version: tag }`
+
+**Sequence:**
+
+1. Acquire distributed lock (`lockSnapshot()`, 30-min TTL)
+2. Create `"versioned"` event
+3. Mint DOI via DataCite MDS API (`createIfNotExistsDoi()`)
+   — format: `doi:{prefix}/openneuro.{datasetId}.v{tag}`
+4. Update `dataset_description.json` with `DatasetDOI` field
+5. Update `CHANGES` file with version and changelog entries
+6. POST to datalad-service to create git tag
+7. Store `Snapshot` document in MongoDB
+8. Clear Redis caches
+9. Queue for Elasticsearch indexing
+10. Notify dataset followers
+
+**Preconditions:** BIDS validation and git-annex fsck should pass before
+creating a snapshot. This is currently enforced in the UI, not in the
+`createSnapshot()` API.
+
+### Publishing (Embargoed → Public)
+
+`publishDataset()` sets `Dataset.public = true` and triggers export to
+S3 and GitHub.
+
+- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/publish.ts`
+- **JS function:** `packages/openneuro-server/src/datalad/dataset.ts` — `updatePublic()`
+- **Python task:** `services/datalad/datalad_service/tasks/publish.py`
+- **Event:** `{ type: "published", public: true }`
+
+**Sequence:**
+
+1. Check write permissions
+2. Set `Dataset.public = true` and `Dataset.publishDate = new Date()`
+3. Create `"published"` event
+4. POST to datalad-service `/datasets/{id}/publish`
+5. Create S3 and GitHub remotes (`create_remotes_and_export()`)
+6. Export dataset to S3-PUBLIC and GitHub (`export_dataset()`)
+7. Set S3 access tags to "public"
+8. Run remote fsck to verify exported data
+9. Drop local annexed files after verification
+
+### Re-embargo (Public → Embargoed)
+
+`updatePublic(datasetId, false, user)` sets `Dataset.public = false` and
+updates S3 access tags to "private".
+
+- **JS:** `packages/openneuro-server/src/datalad/dataset.ts` — `updatePublic()`
+- **Event:** `{ type: "published", public: false }`
+
+**Known issue:** This also sets `Dataset.publishDate = new Date()` rather
+than clearing it.
+
+### Deletion
+
+`deleteDataset()` removes the dataset from the git backend and
+Elasticsearch. Currently, datasets with snapshots require admin action to
+delete.
+
+- **JS:** `packages/openneuro-server/src/datalad/dataset.ts` — `deleteDataset()`
+- **Event:** `{ type: "deleted" }`
+- Creates a `Deletion` document (datasetId, user, reason, optional
+  redirect URL)
+
+## Snapshot status
+
+Snapshots can be independently deprecated without changing the dataset's
+state.
+
+```{mermaid}
+stateDiagram-v2
+    [*] --> Active : createSnapshot()
+    Active --> Deprecated : deprecateSnapshot()
+    Deprecated --> Active : undoDeprecateSnapshot()
+```
+
+| Status         | DB representation                                           |
+| -------------- | ----------------------------------------------------------- |
+| **Active**     | `Snapshot` doc exists, no matching `DeprecatedSnapshot` doc |
+| **Deprecated** | `DeprecatedSnapshot` doc with user, reason, timestamp       |
+
+- **JS:** `packages/openneuro-server/src/graphql/resolvers/snapshots.ts`
+
+## DOI assignment
+
+Each snapshot is assigned a DOI via the DataCite MDS API at creation
+time. DOIs are minted immediately as findable — there is no state
+management after creation.
+
+| Field   | Value                                       |
+| ------- | ------------------------------------------- |
+| Format  | `doi:{prefix}/openneuro.{datasetId}.v{tag}` |
+| API     | DataCite MDS (XML metadata + `PUT /doi`)    |
+| Created | During `createSnapshot()` (step 3)          |
+| State   | Not tracked locally                         |
+
+- **JS:** `packages/openneuro-server/src/libs/doi/index.ts`
+- **Handler:** `packages/openneuro-server/src/handlers/doi.ts`
+- **Model:** `packages/openneuro-server/src/models/doi.ts`
+  — fields: `datasetId`, `snapshotId`, `doi`
+
+DOI metadata includes creators, title, publisher ("Openneuro"),
+publication year, and resource type. The DOI string is written into
+`dataset_description.json` before the git tag is created.
+
+DOI state is never updated after creation. The DOI remains findable at
+DataCite regardless of whether the dataset is later re-embargoed, the
+snapshot is deprecated, or the dataset is deleted.
+
+## Working tree
+
+The working tree (git HEAD) can diverge from the latest snapshot when
+users upload or edit files. Draft datasets are always diverged (no
+snapshot exists). For Embargoed and Public datasets:
+
+| Status       | Determination                   |
+| ------------ | ------------------------------- |
+| **Clean**    | `HEAD == latestSnapshot.hexsha` |
+| **Diverged** | `HEAD != latestSnapshot.hexsha` |
+
+Divergence does not change the dataset's lifecycle state. A new
+`createSnapshot()` call is required to tag the current HEAD.
+
+- **JS:** `packages/openneuro-server/src/datalad/draft.ts`
+  — `getDraftRevision()`, `getDraftInfo()`, `resetDraft()`
+
+### Data retention
+
+When the working tree diverges from the latest snapshot, a notification
+schedule enforces data retention:
+
+- 14 days diverged: first warning email
+- 21 days: second warning
+- 28 days: deletion notice
+- No snapshot within 24h of first upload: one-time reminder
+
+- **JS:** `packages/openneuro-server/src/datalad/dataRetentionNotifications.ts`
+
+## Validation
+
+Validation is a precondition for snapshotting, not a persistent state.
+Two checks are involved:
+
+### BIDS schema validation
+
+Runs the BIDS validator (deno-based, v2.3.2) against a specific commit.
+
+- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/validation.ts` — `revalidate()`
+- **Python task:** `services/datalad/datalad_service/tasks/validator.py` — `validate_dataset()`
+- **Storage:** `Validation` collection
+
+### git-annex fsck
+
+Checks data consistency of annexed files.
+
+- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/fileCheck.ts` — `fsckDataset()`
+- **Python task:** `services/datalad/datalad_service/tasks/fsck.py` — `git_annex_fsck_local()`
+- Runs `git-annex fsck -J4` with incremental 45-day schedule
+- **Storage:** `FileCheck` collection
+
+## Data model reference
+
+### MongoDB collections
+
+| Collection           | File                                        | Key fields                                     |
+| -------------------- | ------------------------------------------- | ---------------------------------------------- |
+| `Dataset`            | `packages/.../models/dataset.ts`            | `id, public, publishDate, created, modified`   |
+| `Snapshot`           | `packages/.../models/snapshot.ts`           | `datasetId, tag, hexsha, created`              |
+| `DeprecatedSnapshot` | `packages/.../models/deprecatedSnapshot.ts` | `id, user, reason, timestamp`                  |
+| `DatasetEvent`       | `packages/.../models/datasetEvents.ts`      | `datasetId, userId, event, success, timestamp` |
+| `Doi`                | `packages/.../models/doi.ts`                | `datasetId, snapshotId, doi`                   |
+| `Validation`         | `packages/.../models/validation.ts`         | `id (ref), datasetId, issues, codeMessages`    |
+| `FileCheck`          | `packages/.../models/fileCheck.ts`          | `datasetId, hexsha, refs, annexFsck[]`         |
+| `Deletion`           | `packages/.../models/deletion.ts`           | `datasetId, user, reason, redirect`            |
+
+All model files are under `packages/openneuro-server/src/models/`.
+
+### GraphQL types
+
+Defined in `packages/openneuro-server/src/graphql/schema.ts`:
+
+- `Dataset` — core entity with `public`, `publishDate`, `draft`, `snapshots`
+- `Draft` — working tree with `modified`, `summary`, `validation`, `fileCheck`
+- `Snapshot` — tagged version with `tag`, `hexsha`, `deprecated`
+- `DeprecatedSnapshot` — deprecation record
+- `DatasetValidation` — validation results with error/warning counts
+
+### Mutations
+
+| Mutation                | Effect                                              |
+| ----------------------- | --------------------------------------------------- |
+| `createDataset`         | Creates dataset in Draft state                      |
+| `createSnapshot`        | Creates snapshot + DOI; Draft → Embargoed on first  |
+| `publishDataset`        | Embargoed → Public; exports to S3/GitHub            |
+| `updatePublic(false)`   | Public → Embargoed (admin only)                     |
+| `deprecateSnapshot`     | Marks a snapshot as deprecated                      |
+| `undoDeprecateSnapshot` | Removes deprecation                                 |
+| `deleteDataset`         | Removes dataset (admin required if snapshots exist) |
+| `fsckDataset`           | Triggers git-annex fsck                             |
+| `revalidate`            | Triggers BIDS validation                            |
+
+### Event types
+
+Dataset events are stored in the `DatasetEvent` collection and track all
+state transitions:
+
+`created` · `versioned` · `deleted` · `published` · `permissionChange` ·
+`git` · `upload` · `note` · `contributorRequest` · `contributorCitation` ·
+`contributorRequestResponse` · `contributorCitationResponse`

docs/index.md

@@ -90,6 +90,7 @@ structure of our package and hosting setup may be interested in the following se
 :maxdepth: 1
 
 architecture
+dataset-lifecycle
 maintenance
 Source repository (GitHub) <https://github.com/OpenNeuroOrg/openneuro>
 ```

docs/requirements.txt

@@ -4,3 +4,4 @@ myst-parser >=0.18.1
 sphinx-copybutton >=0.5.0
 pygments-graphql-lexer >=0.1.0
 sphinxext-rediraffe >=0.2.7
+sphinxcontrib-mermaid >=2.0.1