Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ lc status --json # machine-readable output
lc verify # recompute hashes and walk the provenance chain
lc build # pre-build container images from Containerfiles
lc build --force --runtime podman # force rebuild and pin runtime
lc export wrroc # export a Workflow Run RO-Crate bundle for publication
lc export wrroc --zip -o run.zip # zip the bundle for upload to Zenodo / WorkflowHub
```

## Capabilities
Expand Down Expand Up @@ -110,6 +112,10 @@ Jobs always dispatch through a Dask cluster: a `LocalCluster` on a workstation,

Every materialized output gets a `.lightcone-manifest.json` capturing `code_version` (sha256 of recipe + container + decisions), `data_version` (sha256 of the output directory contents), input manifest versions, git SHA, lc version, and host. `lc verify` recomputes hashes and walks the chain — failures surface as `tampered_data`, `broken_chain`, or `missing_manifest`. `lc status` reads only manifests, so it works offline.

### Publishing analyses

`lc export wrroc` walks your manifests and emits a [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/) bundle — a JSON-LD package readable by WorkflowHub, Zenodo's RO-Crate plugin, and any RO-Crate-aware archive. Each materialization becomes a `CreateAction` with `object` (inputs, including upstream datasets via stable `@id` references) and `result` (the output dataset); decisions become `PropertyValue` entities; the workflow is captured as a `ComputationalWorkflow`. The lightcone manifest format on disk is unchanged — WRROC is the publication view, generated on demand. Use `--metadata-only` to ship only the provenance graph (useful when data files are huge), `--zip` to package the bundle for upload, or `-u <universe>` to restrict to specific universes.

### Telemetry

Claude Code sessions are traced to Langfuse with full conversation structure, tool calls, and git commit linking. Disable with `TRACE_TO_LANGFUSE=false` in `.claude/settings.local.json`.
Expand Down
15 changes: 15 additions & 0 deletions claude/lightcone/guides/lightcone-cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ lc run [OUTPUTS...] [--universe NAME] [--force] [--verbose] [--rerun-triggers TR
lc build [--force] [--runtime docker] # Build container images from specs
lc status [--universe NAME] [--json] # Materialization status (text or JSON)
lc verify [--universe NAME] # Recompute hashes and walk the provenance chain
lc export wrroc [--output PATH] [--universe NAME] [--zip] [--metadata-only] [--author "NAME <EMAIL>"] # Export Workflow Run RO-Crate bundle
lc eval {run,report,compare} # Run/inspect eval suites (requires the 'eval' extra)
```

Expand Down Expand Up @@ -72,3 +73,17 @@ Outputs land at `results/<universe>/<output_id>/`, with the per-output manifest
- **`lc verify` failure** — `missing_manifest` (output dir exists with no `.lightcone-manifest.json`), `tampered_data` (bytes on disk no longer hash to the recorded `data_version`), or `broken_chain` (an upstream's `data_version` drifted from what this output's manifest recorded). Re-run the affected output with `lc run` to repair.

After failure: fix, then `lc run <output_id> --universe <name>`.

## Publishing Analyses

`lc export wrroc` bundles the project's manifests, workflow definition, decisions, and (optionally) data files into a [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/) — a JSON-LD package readable by RO-Crate-aware archives (WorkflowHub, Zenodo's RO-Crate plugin, etc.). The lightcone manifest format on disk is unchanged; the bundle is generated on demand.

```bash
lc export wrroc # ./wrroc/ directory
lc export wrroc -o run.zip --zip # zip bundle
lc export wrroc --metadata-only # provenance graph + manifests only (no data files)
lc export wrroc -u baseline -u alt_method # restrict to specific universes
lc export wrroc --author "Name <email@host>" # override git config
```

The bundle's `@graph` contains a `ComputationalWorkflow` (the astra.yaml), one `CreateAction` per materialized output (with `object` referencing upstream datasets and external inputs, `result` referencing the produced dataset, and `instrument` referencing the recipe `SoftwareApplication`), `PropertyValue` entries for decisions and provenance metadata (`code_version`, `data_version`), and a `Person` for the author.
106 changes: 106 additions & 0 deletions docs/cli/export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# lc export

Export project artifacts in interoperable formats. Currently the only
exporter is `wrroc` (Workflow Run RO-Crate); the group is shaped to host
future formats without breaking the CLI surface.

## Synopsis

```
lc export wrroc [OPTIONS]
```

## lc export wrroc

Walk the project's per-output `.lightcone-manifest.json` sidecars and
emit a [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/)
bundle — a JSON-LD package readable by WorkflowHub, Zenodo's RO-Crate
plugin, and any RO-Crate-aware archive. The on-disk manifest format is
unchanged; the bundle is the *publication view*, generated on demand.

### Options

| Option | Default | Effect |
|--------|---------|--------|
| `--output`, `-o PATH` | `./wrroc` | Bundle directory, or `.zip` path when `--zip` is set. |
| `--universe`, `-u NAME` | every universe with materialized outputs | Restrict to specific universes. Repeatable. |
| `--author "NAME <email>"` | git `user.name` / `user.email`, then `LIGHTCONE_AUTHOR` env | Override the author recorded in the bundle. |
| `--license URL` | `https://creativecommons.org/licenses/by/4.0/` | License URL or SPDX identifier for the bundle. Required by the WRROC profile. |
| `--zip` / `--no-zip` | `--no-zip` | Package the bundle as a single `.zip` after building. |
| `--metadata-only` | off | Skip data files; bundle manifests + `astra.yaml` + universe files only. |

### What gets bundled

- `astra.yaml` → `ComputationalWorkflow` (`programmingLanguage: snakemake`).
- Each materialized output → `Dataset` with `version = data_version`.
- Each recipe execution → `CreateAction` with `object` (inputs, both upstream datasets via stable `@id` and external files), `result` (the produced dataset), and `instrument` (the recipe `SoftwareApplication`).
- Each container → `SoftwareApplication` referenced via `softwareRequirements`.
- Each decision → `FormalParameter` + per-run `PropertyValue`.
- Author → `Person`.

Provenance metadata (`code_version`, `data_version`, `git_sha`, `lc_version`, host) lands as `PropertyValue` entries on the relevant entities.

### Examples

```bash
lc export wrroc # ./wrroc/ directory
lc export wrroc -o run.zip --zip # zip bundle for upload
lc export wrroc --metadata-only # provenance graph only, no data files
lc export wrroc -u baseline -u alt_method # restrict to specific universes
lc export wrroc --author "Ada Lovelace <ada@example.org>"
lc export wrroc --license https://opensource.org/licenses/MIT
```

### Output

```
✓ Wrote WRROC directory: ./wrroc
Captured 7 runs across universes: baseline, alt_method
```

If no materialized outputs are found, the bundle still writes — but only
contains the workflow definition, and a warning is printed:

```
✓ Wrote WRROC directory: ./wrroc
Warning: no materialized outputs were found — the bundle contains only
the workflow definition.
This usually means recipes haven't been run yet (try lc run) or the
.lightcone-manifest.json sidecars are missing.
Workflow-only bundles will not pass strict Provenance Run Crate
validation; that profile requires at least one materialized run.
```

### Failure modes

| Error | Cause |
|---|---|
| `No astra.yaml at <path>; cannot export.` | The cwd is not inside an ASTRA project. |
| `<path> is non-empty; refuse to clobber.` | The target directory already has contents. Pass a fresh path or remove the existing one. |
| `<path> is an existing directory; cannot overwrite with a zip.` | `--zip` was requested but the output path resolves to a directory. Use a file path like `bundle.zip`. |

Manifests that exist but are unreadable (e.g. cross-user symlinks under
`results/` with permission denied) are warned about and skipped — they
do not abort the export.

### Validation

The bundle conforms to the [Provenance Run Crate 0.5](https://w3id.org/ro/wfrun/provenance/0.5)
profile. To validate locally:

```bash
pip install git+https://github.com/crs4/rocrate-validator.git
rocrate-validator -y validate ./wrroc/
```

### When to run

- Before submitting a paper or depositing artifacts in Zenodo / WorkflowHub.
- After a clean run (`lc verify` clean) on the final commit you intend to publish.
- For external collaborators who don't have `lc` installed but need to inspect provenance.

### Related

- [`lc verify`](verify.md) — confirm the manifest chain is intact before exporting.
- [`lc status`](status.md) — see which outputs will be captured by the export.
- [api/manifest](../api/manifest.md) — the on-disk format the export reads from.
1 change: 1 addition & 0 deletions docs/cli/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Code skills, with the CLI as the durable, scriptable backstop.
| [`lc build`](build.md) | Build container images declared in `astra.yaml`. |
| [`lc status`](status.md) | Manifest-driven status report. No Snakemake import needed. |
| [`lc verify`](verify.md) | Recompute hashes, walk the input chain, surface tampering. |
| [`lc export`](export.md) | Emit interoperable bundles (Workflow Run RO-Crate) for publication. |
| [`lc setup`](setup.md) | Write a minimal `~/.lightcone/config.yaml`. |

## Global options
Expand Down
5 changes: 5 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ dependencies = [
"dask>=2024.1",
"distributed>=2024.1",
"langfuse>=2.0",
"rocrate>=0.11",
]

[dependency-groups]
Expand Down Expand Up @@ -88,6 +89,10 @@ module = ["dask.*", "distributed.*"]
ignore_missing_imports = true
follow_untyped_imports = true

[[tool.mypy.overrides]]
module = ["rocrate.*", "astra.*"]
ignore_missing_imports = true

[tool.pytest.ini_options]
testpaths = ["tests"]
markers = [
Expand Down
107 changes: 107 additions & 0 deletions src/lightcone/cli/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -827,6 +827,113 @@ def _ensure_images(project: Path, *, runtime: str, force: bool = False) -> None:
raise click.ClickException(str(e))


# =============================================================================
# lc export
# =============================================================================


@main.group()
def export() -> None:
"""Export project artifacts in interoperable formats."""


@export.command("wrroc")
@click.option(
"--output",
"-o",
type=click.Path(path_type=Path),
default=Path("./wrroc"),
help="Bundle directory (or .zip path with --zip).",
show_default=True,
)
@click.option(
"--universe",
"-u",
multiple=True,
help="Restrict to specific universes (default: all).",
)
@click.option(
"--author",
default=None,
help="Author override, e.g. \"Name <email@host>\". Default: git config.",
)
@click.option(
"--license",
"license_url",
default=None,
help="License URL or SPDX identifier. Default: CC-BY-4.0.",
)
@click.option(
"--zip/--no-zip",
"zip_bundle",
default=False,
help="Package the bundle as a .zip after building.",
)
@click.option(
"--metadata-only",
is_flag=True,
help="Skip data files; bundle manifests + astra.yaml + universes only.",
)
def export_wrroc_cmd(
output: Path,
universe: tuple[str, ...],
author: str | None,
license_url: str | None,
zip_bundle: bool,
metadata_only: bool,
) -> None:
"""Export a Workflow Run RO-Crate (WRROC) bundle.

The bundle is suitable for upload to WorkflowHub, Zenodo (with the
RO-Crate plugin), or any RO-Crate-aware archive. The lightcone
manifest format on disk is unchanged — this is a publication view
generated on demand.

Examples:

lc export wrroc # ./wrroc/ directory
lc export wrroc -o my-run.zip --zip # zip bundle
lc export wrroc --metadata-only # provenance, no data
lc export wrroc -u baseline -u alt # specific universes
"""
from lightcone.engine.wrroc import export_wrroc

project = _project_root()

try:
result = export_wrroc(
project_path=project,
output_path=output,
universes=list(universe) or None,
author=author,
license=license_url,
zip_bundle=zip_bundle,
include_data=not metadata_only,
)
except FileExistsError as e:
raise click.ClickException(str(e))

flavor = "zip bundle" if result.is_zip else "directory"
console.print(
f"[green]✓[/green] Wrote WRROC {flavor}: [cyan]{result.bundle_path}[/cyan]"
)
if result.runs_included == 0:
console.print(
"[yellow]Warning:[/yellow] no materialized outputs were found — "
"the bundle contains only the workflow definition.\n"
" This usually means recipes haven't been run yet (try [cyan]lc run[/cyan]) "
"or the [cyan].lightcone-manifest.json[/cyan] sidecars are missing.\n"
" Workflow-only bundles will not pass strict Provenance Run Crate "
"validation; that profile requires at least one materialized run."
)
else:
u_list = ", ".join(result.universes_included)
console.print(
f" Captured [bold]{result.runs_included}[/bold] runs across "
f"universes: [cyan]{u_list}[/cyan]"
)


# Register eval subgroup (requires optional 'eval' extra)
try:
from lightcone.eval.cli import eval_group
Expand Down
4 changes: 4 additions & 0 deletions src/lightcone/engine/manifest.py
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,10 @@ def write_manifest(
"decisions": cfg.get("decisions", {}),
"input_versions": input_versions,
"git_sha": cfg.get("git_sha"),
# URL of the git origin remote at the time of materialization.
# Optional/additive — older manifests without this field still
# parse. Surfaced by ``lc export wrroc`` as a CodeRepository entity.
"git_remote": cfg.get("git_remote"),
"lc_version": cfg.get("lc_version"),
"finished_at": time.time(),
"host": socket.gethostname(),
Expand Down
34 changes: 34 additions & 0 deletions src/lightcone/engine/snakefile.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,38 @@ def _git_sha(project_path: Path) -> str | None:
return None


def _git_remote(project_path: Path) -> str | None:
"""URL of the ``origin`` git remote, if the project is a git clone.

Captured into the manifest alongside ``git_sha`` so a published
bundle can identify *which repository* the commit belongs to.
SSH URLs (``git@host:owner/repo.git``) are normalised to ``https``
form so consumers (e.g. WorkflowHub) can render them as clickable
links.
"""
try:
out = subprocess.run(
["git", "-C", str(project_path),
"config", "--get", "remote.origin.url"],
capture_output=True,
text=True,
check=False,
)
if out.returncode != 0:
return None
url = out.stdout.strip()
if not url:
return None
if url.startswith("git@"):
host_path = url.removeprefix("git@").replace(":", "/", 1)
url = f"https://{host_path}"
if url.endswith(".git"):
url = url.removesuffix(".git")
return url
except FileNotFoundError:
return None


def _lc_version() -> str:
try:
from importlib.metadata import version
Expand Down Expand Up @@ -317,6 +349,7 @@ def generate(
cfg: dict[str, dict[str, dict[str, Any]]] = {}

git_sha = _git_sha(project_path)
git_remote = _git_remote(project_path)
lc_version = _lc_version()
resolve_image = make_image_tag_resolver(project_path, project_name)

Expand Down Expand Up @@ -411,6 +444,7 @@ def generate(
"decisions": scoped_decisions,
"code_version": cv,
"git_sha": git_sha,
"git_remote": git_remote,
"lc_version": lc_version,
}

Expand Down
Loading
Loading