Skip to content

Commit f0f412f

Browse files
dansanddansand
committed
Generic dataset-crate template
0 parents  commit f0f412f

11 files changed

Lines changed: 203 additions & 0 deletions

File tree

‎.devcontainer/devcontainer.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"name": "Dataset crate (mate toolkit)",
3+
"image": "mcr.microsoft.com/devcontainers/miniconda:3",
4+
"features": {
5+
"ghcr.io/devcontainers/features/github-cli:1": {}
6+
},
7+
// Build the tracked env on create (installs the mate toolkit from git + quarto), and make
8+
// new terminals open in it. Same environment.yml used locally and in CI — one definition.
9+
"postCreateCommand": "conda env create -f environment.yml || conda env update -f environment.yml; grep -q 'conda activate mate' ~/.bashrc || echo 'conda activate mate' >> ~/.bashrc",
10+
"postAttachCommand": "echo '\\n Dataset crate ready. Try:\\n mate build . # scan the repo into the crate\\n mate enrich . # resolve ORCIDs / DOIs\\n mate validate . # check it\\n mate render . -o _site\\n'",
11+
"customizations": {
12+
"vscode": {
13+
"extensions": ["ms-python.python"]
14+
}
15+
}
16+
}

‎.github/ISSUE_TEMPLATE/new-dataset.yml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
name: New dataset
2+
description: Describe a dataset; an action writes it into the crate.
3+
title: '[dataset] '
4+
labels:
5+
- dataset
6+
body:
7+
- type: markdown
8+
attributes:
9+
value: Describe your dataset. On submit, an automated action writes these values
10+
into the RO-Crate (`ro-crate-metadata.json`) — the single source of truth. **Edit
11+
the crate afterwards (CLI / editor), not by reopening this issue.**
12+
- type: input
13+
id: title
14+
attributes:
15+
label: Title
16+
validations:
17+
required: true
18+
- type: textarea
19+
id: description
20+
attributes:
21+
label: Short description
22+
validations:
23+
required: true
24+
- type: input
25+
id: license
26+
attributes:
27+
label: License (SPDX id or URL)
28+
validations:
29+
required: true
30+
- type: textarea
31+
id: creators
32+
attributes:
33+
label: Creator(s) — ORCID iD or 'Family, Given'
34+
description: 'One per line: an ORCID iD (e.g. 0000-0002-1270-4377), or "Family,
35+
Given".'
36+
validations:
37+
required: true
38+
- type: input
39+
id: keywords
40+
attributes:
41+
label: Keywords
42+
description: Comma-separated.
43+
validations:
44+
required: false
45+
- type: input
46+
id: publication
47+
attributes:
48+
label: Related publication (DOI)
49+
validations:
50+
required: false
51+
- type: dropdown
52+
id: payload_backend
53+
attributes:
54+
label: External data payload — backend (optional)
55+
options:
56+
- (none)
57+
- git
58+
- zenodo
59+
- url
60+
validations:
61+
required: false
62+
- type: input
63+
id: payload_ref
64+
attributes:
65+
label: Payload reference (e.g. Zenodo record id, or a URL)
66+
validations:
67+
required: false

‎.github/workflows/build.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Thin caller: all logic lives in the toolkit's reusable workflow, pinned to a version.
2+
# Auto-regenerates the crate on every push (the crate must track the filesystem); the manual
3+
# trigger is just an on-demand re-run. The toolkit's commit uses [skip ci] to avoid a loop.
4+
name: Build dataset crate
5+
6+
on:
7+
push:
8+
branches: [main]
9+
paths-ignore:
10+
- "_site/**"
11+
- "ro-crate-metadata.json" # the build's own regen-commit shouldn't re-fire it
12+
workflow_dispatch:
13+
14+
jobs:
15+
mate:
16+
uses: ModelAtlasofTheEarth/mate-toolkit/.github/workflows/build.yml@v0
17+
permissions:
18+
contents: write

‎.github/workflows/on-issue.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# When a "New dataset" issue is opened/edited, write its answers into the crate's root entity.
2+
# The issue is the SEED on-ramp only — it sets the model-as-a-whole (root + references), never
3+
# per-file detail. Thin caller; logic is in the toolkit's reusable workflow.
4+
name: Apply dataset submission
5+
6+
on:
7+
issues:
8+
types: [opened, edited]
9+
10+
jobs:
11+
apply:
12+
# Gate on the title prefix the issue form sets ("[dataset] …"). Labels aren't copied by
13+
# "Use this template", so a title prefix is the reliable trigger.
14+
if: ${{ startsWith(github.event.issue.title, '[dataset]') }}
15+
uses: ModelAtlasofTheEarth/mate-toolkit/.github/workflows/build.yml@v0
16+
with:
17+
issue_body: ${{ github.event.issue.body }}
18+
issue_number: ${{ github.event.issue.number }}
19+
issue_title: ${{ github.event.issue.title }}
20+
permissions:
21+
contents: write
22+
issues: write

‎.gitignore‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Large data does not live in git — declare it as an external payload (a reference in the
2+
# crate); it appears as an external data entity, not a tracked file.
3+
large_data/*
4+
!large_data/.gitkeep
5+
6+
# rendered site (regenerated by CI / `mate render`)
7+
_site/
8+
# Crate-O writes this on save; the toolkit regenerates the page anyway
9+
ro-crate-preview.html
10+
11+
.DS_Store
12+
__pycache__/

‎.mate/metadata.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# ── Seed metadata (the one-shot bootstrap of the crate's root entity) ──
2+
# This is the authored INPUT. `mate build` reads it into ro-crate-metadata.json (the single
3+
# source of truth) and then never parses it back. After the first build, edit the CRATE
4+
# (via `mate describe` / Crate-O), not this file. Fields are defined by the active profile
5+
# (default: the generic `base` profile — schema.org only).
6+
title: ""
7+
description: ""
8+
license: "" # SPDX id or URL, e.g. CC-BY-4.0
9+
creators: [] # one per line: an ORCID iD, or "Family, Given"
10+
keywords: []
11+
publication: "" # related publication DOI (optional)

‎.rocrateignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Repo infrastructure that is present but is NOT part of the described dataset.
2+
# (Layered on top of .gitignore + the toolkit's hard-skips.)
3+
.github/
4+
.devcontainer/
5+
.mate/
6+
environment.yml

‎README.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Dataset crate (template)
2+
3+
A self-describing dataset repository. Its metadata lives in **`ro-crate-metadata.json`** — an
4+
[RO-Crate](https://www.researchobject.org/ro-crate/) built *dynamically from whatever is in
5+
this repo*. That crate is the **single source of truth**; this README and the published page
6+
are **generated projections** of it.
7+
8+
This template is domain-free: it produces a generic [schema.org](https://schema.org) `Dataset`
9+
crate. A discipline (e.g. geoscience) is opted into by dropping a `.mate/profile.yml` in the
10+
repo — the engine (`mate` toolkit) is identical either way.
11+
12+
## How it works
13+
14+
- **Seed it once.** Fill `.mate/metadata.yml` (or open a *New dataset* issue). On push, an
15+
action writes those values into the crate's root entity.
16+
- **Add data by just adding files.** `mate build` scans the filesystem and mirrors every file
17+
and folder into the crate as data entities — automatically, on every push.
18+
- **Enrich.** Best-effort resolution of any DOIs / ORCIDs you provided.
19+
- **Describe specifics.** Per-file/-folder detail (a different author, a description, a type)
20+
is an editing act: `mate describe <path>` locally, or Crate-O — *not* the issue.
21+
22+
## Local use
23+
24+
```bash
25+
conda env create -f environment.yml # installs the mate toolkit + quarto
26+
conda activate mate
27+
mate build . # scan repo -> crate (preserves authored content, refreshes the derived layer)
28+
mate enrich . # resolve ORCIDs / DOIs (best-effort)
29+
mate validate . # check the crate
30+
mate render . -o _site
31+
```
32+
33+
> Do not hand-edit `ro-crate-metadata.json`'s derived layer or the generated page — they are
34+
> regenerated on every push.

‎data/.gitkeep‎

Whitespace-only changes.

‎environment.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# The dataset's tracked environment — SAME file used locally and in CI (the reusable workflow
2+
# loads it via micromamba), so behaviour is identical. This is the SINGLE place the toolkit
3+
# version is pinned (the @v0 tag); bump it here to move to a newer toolkit.
4+
#
5+
# conda env create -f environment.yml && conda activate mate
6+
# mate build . && mate validate . && mate render . -o _site
7+
#
8+
# For local dev against a checkout, install the toolkit editable instead:
9+
# pip install -e ../mate-toolkit
10+
name: mate
11+
channels: [conda-forge]
12+
dependencies:
13+
- python=3.11
14+
- quarto
15+
- pip
16+
- pip:
17+
- "mate-toolkit @ git+https://github.com/ModelAtlasofTheEarth/mate-toolkit@v0"

0 commit comments

Comments
 (0)