lc export wrroc — Workflow Run RO-Crate publication

[dkn16]

This closes #85

What

Adds lc export wrroc — walks the project's per-output .lightcone-manifest.json sidecars and emits a Workflow Run RO-Crate bundle (JSON-LD via rocrate-py) suitable for upload to WorkflowHub, Zenodo's RO-Crate plugin, or any RO-Crate-aware archive.

Why

The lightcone manifest format is great for internal provenance but not portable. WRROC is the standard interop format for workflow runs — accepted by archives, citable, and machine-readable by other tools. This is the publication view; the on-disk manifest format is unchanged.

CLI

lc export wrroc                                # ./wrroc/ directory
lc export wrroc -o run.zip --zip               # zip bundle
lc export wrroc --metadata-only                # provenance only, no data
lc export wrroc -u baseline -u alt             # filter universes
lc export wrroc --author "Name <email>"        # override git config
lc export wrroc --license <url>                # default: CC-BY-4.0

Entity mapping

• astra.yaml → ComputationalWorkflow (lang: snakemake)
• each materialized output → Dataset (with schema.org version = data_version)
• each recipe execution → CreateAction (object/result/instrument)
• each container → SoftwareApplication (softwareRequirements)
• each decision → FormalParameter + per-run PropertyValue
• author → Person

Validation

Verified against the official rocrate-validator (CRS4) at REQUIRED severity:

─────── [OK] RO-Crate is a valid provenance-run-crate-0.5 !!!  ────────
PASSED Checks: 83 / 83

Tested end-to-end on a real multi-output project (fmood, 7 runs) and a complex multiverse with sub-analyses (DESI BAO project, 49 outputs, 19 decisions across root + 2 sub-analyses).

Tested

• 33 tests, all passing
• Sub-analyses: dataset @ids are path-rooted (analyses/sub/results/.../), CreateAction @ids qualified to avoid collisions
• Permission-denied results dirs (e.g. cross-user symlinks): warn + skip, don't abort
• Round-trip through rocrate.Crate(path) confirms structural validity
• Workflow-only bundles (no runs yet) succeed with a clear warning

Files changed

File	LoC
src/lightcone/engine/wrroc.py (new)	~660
src/lightcone/cli/commands.py (export group + wrroc command)	+95
tests/test_wrroc.py (new)	~470
pyproject.toml (rocrate>=0.11 + mypy override)	+5
README.md + claude/lightcone/guides/lightcone-cli-reference.md	+30

Commits

1. feat: add lc export wrroc for Workflow Run RO-Crate publication
2. fix(wrroc): pass official Provenance Run Crate 0.5 validator (83/83)
3. fix(wrroc): handle ASTRA's actual decisions schema (dict-of-dicts)
4. fix(wrroc): tolerate unreadable manifests; document workflow-only limits

Out of scope

• Signing (SLSA / in-toto attestations)
• Round-trip import (lc import wrroc)
• BioSchemas tool annotations
• Continuous bundling — the exporter is one-shot by design

How to test

git checkout lc-export
pip install -e .
cd path/to/your/project
lc export wrroc -o bundle/
# (optional) pip install git+https://github.com/crs4/rocrate-validator.git
# rocrate-validator -y validate bundle/

[dkn16]

@copilot can you review it?

[github-actions]

✅ Eval Results

Metric	Value
Score	1.00
Build complete	✅
Cost	$0.74
Turns	32
Duration	322s
lightcone-cli	0.3.5.dev15+gb45180f5b (b45180f5)
Results	Download

Graders

✅ spec_valid (1.00)
✅ all_materialized (1.00)

Full output

18:21:17 lightcone.eval.build Building lightcone-cli wheel from /home/runner/work/lightcone-cli/lightcone-cli ...
18:21:22 lightcone.eval.build Built lightcone_cli-0.3.5.dev15+gb45180f5b-py3-none-any.whl (commit b45180f5)
18:21:25 lightcone.eval.sandbox Created sandbox f6f2159f-bc00-4b2d-8190-7d53c45c13af for trial build-snae-0
18:21:26 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/f6f2159f-bc00-4b2d-8190-7d53c45c13af/files/bulk-upload "HTTP/1.1 200 OK"
18:21:26 lightcone.eval.sandbox Installed wheels: ['lightcone_cli-0.3.5.dev15+gb45180f5b-py3-none-any.whl']
18:21:27 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/f6f2159f-bc00-4b2d-8190-7d53c45c13af/files/bulk-upload "HTTP/1.1 200 OK"
18:21:27 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/f6f2159f-bc00-4b2d-8190-7d53c45c13af/files/bulk-upload "HTTP/1.1 200 OK"
18:21:27 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/f6f2159f-bc00-4b2d-8190-7d53c45c13af/files/bulk-upload "HTTP/1.1 200 OK"
18:21:28 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/f6f2159f-bc00-4b2d-8190-7d53c45c13af/files/bulk-upload "HTTP/1.1 200 OK"
18:26:51 lightcone.eval.sandbox Deleted sandbox for trial build-snae-0
  snae trial 0: score=1.00 complete

lightcone-cli: 0.3.5.dev15+gb45180f5b (HEAD b45180f5)
ASTRA: 0.2.7

  Eval Results: Scores  
┏━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Task ┃     Score     ┃
┡━━━━━━╇━━━━━━━━━━━━━━━┩
│ snae │ 1.00 +/- 0.00 │
│      │ pass@k: 100%  │
└──────┴───────────────┘

   Eval Results: Cost &   
         Duration         
┏━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Task ┃ Cost / Duration ┃
┡━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ snae │      $0.74      │
│      │      322s       │
└──────┴─────────────────┘

Total: 1 trials, $0.74, 322s

Results saved to: eval-results/build-b45180f5/results.json

[Copilot]

@copilot can you review it?

Here's the review. The implementation is solid — well-structured, good test coverage (33 tests, 83/83 validator), and the entity mapping is correct. Two issues found and fixed in 78870de:

Fixed:

1. Ruff F841 — original = manifest_mod.read_manifest in TestUnreadableManifest was assigned but never used (monkeypatch handles restore automatically). Removed.
2. Redundant temp-dir write in zip path — crate.write(bundle_dir) created a temp directory that was immediately deleted after crate.write_zip(). rocrate-py's write_zip() operates independently without a prior write() call, so this was extra disk I/O with no benefit. Removed the dead code and the now-unused shutil import.

No action needed (design choices worth noting):

• from rocrate.model import ContextEntity is imported inside methods/functions rather than at module top — this is the intentional lazy-import pattern documented in the module docstring so the module stays importable without rocrate installed.
• _detect_author runs two git config subprocesses sequentially — fine for CLI, but could be combined into one call (git config --list) if it ever becomes a performance concern.
• The conformsTo override on crate.metadata to set both 1.1 and 1.2 is deliberately belt-and-suspenders for cross-validator compatibility.

CodeQL scan: 0 alerts. All 33 tests pass.

[dkn16]

@EiffL this is the mapping between current manifest file and WRROC. Looks like the overlap is minimal, I would suggest freeze the internal structure of manifest file for now, maybe we will want to revisit it in the future.

Field-by-field mapping

Real lightcone manifest from fmood/results/baseline/auroc_chi2/:

Manifest field	WRROC equivalent	Type/shape match?	Notes
output_id	Dataset.@id (path-derived)	✗ different model	Manifest uses opaque ID, WRROC uses path-as-identity
universe_id	encoded in @id path (results/<u>/<o>/)	✗ different model	WRROC has no first-class universe concept
schema_version	RO-Crate conformsTo	✗ different scope	Manifest = our schema, conformsTo = upstream RO-Crate spec
code_version	exported as PropertyValue on action	✗ no native equivalent	Custom hash; WRROC uses mainEntity.version differently
data_version	Dataset.version	✓ direct	Same hash, different field name
container_image	SoftwareApplication.softwareVersion (on the container entity)	≈ same concept	Image tag → version string
recipe	SoftwareApplication.description (the recipe entity)	✓ direct	The command text
decisions	list of PropertyValue on CreateAction	✗ different shape	Dict → entity graph
input_versions	CreateAction.object (list of @id refs)	✗ different model	Hash-chain vs reference-chain
git_sha	(no native; could be CodeRepository)	✗ no native	Currently dropped from export
lc_version	(no native; could be agent property)	✗ no native	Currently dropped from export
finished_at	CreateAction.endTime	≈ direct	Unix ts → ISO 8601 (one-liner conversion)
host	(no native; could be agent.location)	✗ no native	Currently dropped from export
slurm_job_id	(no native; could be actionApplication)	✗ no native	Currently dropped from export

Direct or near-direct matches: 3 (data_version, recipe, finished_at).
Different shape/model: 6 (output_id, universe_id, decisions, input_versions, plus the _id/_version/_sha semantic mismatches).
No WRROC equivalent: 4 (git_sha, lc_version, host, slurm_job_id).

[dkn16]

Follow-up: git_remote + tool_name (commit 16c9db1)

Two additive fields added on top of the WRROC export, addressing the two clearest naming/usability gaps surfaced by the alignment investigation. No SCHEMA_VERSION bump — older manifests still parse and behave identically.

git_remote (manifest field)

• Captured at run time in snakefile.py via _git_remote(): shells out to git config --get remote.origin.url, normalises ssh URLs to https form (git@github.com:foo/bar.git → https://github.com/foo/bar), strips trailing .git.
• Threaded into cfg alongside git_sha; written into every new manifest by write_manifest().
• Surfaced by the WRROC exporter as a CodeRepository entity (de-duplicated across all CreateActions) and linked from the workflow via the schema.org codeRepository property.
• Old manifests without the field → no CodeRepository entity emitted (silent, expected).

tool_name (optional recipe.tool_name in astra.yaml)

Resolves SoftwareApplication.name in priority order:

1. recipe.tool_name if set in the spec — author-chosen, citable
2. Heuristic from the command (e.g. python scripts/analyze.py … → scripts/analyze.py)
3. output_id as last resort

Replaces the previous behaviour of using a 60-char truncation of the bash command, which produced ugly identical-looking entries for distinct recipes.

Files changed in this commit

• src/lightcone/engine/snakefile.py — _git_remote() helper + cfg threading
• src/lightcone/engine/manifest.py — additive git_remote field
• src/lightcone/engine/wrroc.py — _link_code_repository() builder, _heuristic_tool_name() helper, name resolution chain
• tests/test_manifest.py, tests/test_snakefile.py, tests/test_wrroc.py — 10 new tests covering precedence, URL normalisation, entity emission, and the no-git-remote pass-through

End-to-end demo

SoftwareApplication
  @id:  #recipe-0
  name: scripts/analyze.py        ← was: "python scripts/analyze.py --n_radial_bins {params.cfg[decisi"
  description: <full command>     (unchanged)

CodeRepository
  @id:  https://github.com/dkn16/fmood
  name: fmood
  url:  https://github.com/dkn16/fmood

Workflow.codeRepository.@id: https://github.com/dkn16/fmood

Validator: 83/83 REQUIRED checks pass ✓

Total tests now 331 (was 316). Lint + mypy clean.

[dkn16]

@claude could you review this PR?

[claude]

Claude finished @dkn16's task in 2m 4s —— View job

---

PR Review: lc export wrroc — Workflow Run RO-Crate publication

• Gather context and read PR files
• Review src/lightcone/engine/wrroc.py
• Review src/lightcone/cli/commands.py
• Review src/lightcone/engine/snakefile.py and manifest.py
• Review tests/test_wrroc.py
• Review pyproject.toml changes
• Post comprehensive review

---

This is a well-structured, production-quality addition. Entity mapping is correct, the 83/83 validator passes, test coverage is solid (33 tests including edge cases), and the architectural decisions (one-shot exporter, lazy imports, additive manifest fields) are all sound. Copilot already caught and fixed two regressions (unused var + redundant zip write). Below is what remains.

One actual bug

output_path.unlink() on a directory crashes in zip mode (wrroc.py:222)

if zip_bundle:
    if output_path.exists():
        output_path.unlink()   # ← IsADirectoryError if user passes a dir path

Path.unlink() raises IsADirectoryError (errno 21) on POSIX when the path is a directory. If a user runs lc export wrroc -o mydir --zip and mydir already exists as a directory (or a previous non-zip export landed there), they get a traceback instead of a clear error message. The non-zip path handles this correctly with a FileExistsError + message. The zip path should do the same:

if zip_bundle:
    if output_path.exists():
        if output_path.is_dir():
            raise FileExistsError(f"{output_path} is an existing directory; cannot overwrite with a zip.")
        output_path.unlink()

Fix this →

There's no test covering this case in TestZipBundle.

Design gaps worth discussing

Silent tie-breaking for conflicting git_remote values (wrroc.py:596-611)

_link_code_repository() is idempotent via if self._code_repo_id is not None: return — meaning only the first non-None git_remote seen across all manifests wins. If a project's outputs were produced at different times from different forks/remotes, the later one is silently dropped. This is almost certainly fine in practice (a project has one origin), but a logger.debug on the ignored case would help during debugging:

if self._code_repo_id is not None:
    if self._code_repo_id != git_remote:
        logger.debug("Ignoring conflicting git_remote %s (already have %s)", git_remote, self._code_repo_id)
    return

data_version recorded twice (wrroc.py:517-524)

data_version is already set on the Dataset as version (line 429). It's then also attached to the CreateAction.object as a PropertyValue (lines 517-524). The code_version PropertyValue makes sense (no WRROC native), but the data_version PropertyValue is redundant. If this is intentional (e.g. for round-trip tooling), a comment explaining why would help.

Minor/nitpicks

SSH URL normalization misses the ssh:// prefix form (snakefile.py:156-160)

if url.startswith("git@"):
    host_path = url.removeprefix("git@").replace(":", "/", 1)
    url = f"https://{host_path}"

This handles git@github.com:owner/repo.git correctly. But ssh://git@github.com/owner/repo (less common, used by some CI systems and enterprise hosts) isn't normalised. Not a blocker, but worth a # TODO: handle ssh:// comment or a second branch.

_heuristic_tool_name() could match URL tokens (wrroc.py:779-801)

A command like curl https://example.com/data.csv -o out.csv would have _heuristic_tool_name return https://example.com/data.csv because it contains /. This is a tool used as a downloader, and the output_id fallback would be more useful here. A simple guard — skip tokens that match https?:// — would fix it. Low severity since tool_name in the spec overrides the heuristic.

export_wrroc() contains a redundant spec_path.is_file() check (wrroc.py:131-135)

_project_root() in the CLI already guarantees astra.yaml exists before calling export_wrroc. The internal FileNotFoundError guard is dead code on the CLI path. If this is meant as a guard for programmatic callers, it's fine — but the docstring could note that.

_detect_author() makes two git subprocesses (wrroc.py:857-865)

Copilot noted this. One git config --list pipe would be cleaner. Not a correctness issue.

Tests

Coverage is excellent. A few gaps worth noting:

• No test for the IsADirectoryError bug in zip mode (described above)
• No test for LIGHTCONE_AUTHOR env var path through _detect_author()
• TestGitRemote.test_no_git_remote_no_repo_entity uses minimal_project which produces manifests via _materialize() — that fixture doesn't include git_remote, so this is testing the right thing, but it's implicit; a comment would help
• TestSubAnalyses.subanalysis_project fixture calls write_manifest directly with string recipe (line 428), not a dict — this tests that the string form of recipe still works, which is good

Manifest / snakefile changes

The git_remote field is cleanly additive (no SCHEMA_VERSION bump, older manifests parse with None). The _git_remote() helper properly tolerates missing git, non-zero returncode, and empty output. SSH→HTTPS normalisation is correct for the common case.

The tool_name precedence chain is clean and well-tested.

Summary

This PR is ready to merge with the zip-mode directory crash fixed. Everything else is either already solid or low-priority polish. The entity mapping, validator compliance, and test structure are all in good shape.

[dkn16]

@claude could you fix the bug? 👀

[claude]

Claude finished @dkn16's task in 1m 12s —— View job

---

Fixed zip-mode directory crash

• Understand the bug
• Read wrroc.py to find the exact location
• Apply the fix
• Add a test for the new guard
• Commit and push

What changed (commit 50f3fb8):

In export_wrroc() (wrroc.py:220), the zip branch was calling output_path.unlink() unconditionally, which raises IsADirectoryError on POSIX when the path is an existing directory. Added an is_dir() guard that raises a descriptive FileExistsError instead:

if zip_bundle:
    if output_path.exists():
        if output_path.is_dir():
            raise FileExistsError(
                f"{output_path} is an existing directory; cannot overwrite with a zip. "
                "Pass a file path (e.g. bundle.zip) or remove the existing directory."
            )
        output_path.unlink()

Added TestZipBundle.test_zip_raises_on_existing_directory to cover this case.
| Branch: lc-export

[EiffL]

LGTM

[github-actions]

Developer Certificate of Origin

This PR has been approved. Before it can be merged, all contributors must sign the Developer Certificate of Origin.

Status

• ✅ @dkn16
• ✅ @EiffL
• ❌ @Copilot

How to sign

Post the following comment exactly as written:

I have read the Developer Certificate of Origin and I hereby sign the DCO for this PR

[dkn16]

I have read the Developer Certificate of Origin and I hereby sign the DCO for this PR

[EiffL]

I have read the Developer Certificate of Origin and I hereby sign the DCO for this PR

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
❌ Deployment failed View logs	lightcone-cli	492917e	May 07 2026, 06:22 PM