Embed source RDI in ARC via ISA Investigation Comments

[Zalfsten]

Problem

Source RDI (Research Data Infrastructure) context is today outside the ARC payload:

• API request body (CreateArcRequest.rdi) or harvest document
• CouchDB envelope (ArcDocument.rdi)
• Celery task payload (ArcSyncTask.rdi)
• ArcStore.create_or_update(..., rdi=rdi) for GitLab project topics

The ARC itself only carries investigation metadata (identifier, name/title, description). When an ARC leaves the middleware (Git clone, export), it does not record which RDI it belongs to, nor infrastructure description or URL.

Goal

Embed source RDI inside the ARC in a way that:

• survives the middleware Git sync path (ARC.from_rocrate_json_string → arc.Write())
• is standards-aligned (ISA / RO-Crate), not a proprietary sidecar format
• carries at minimum: RDI identifier, description, and canonical URL
• allows the middleware to remain the security authority for which RDI an ingest is allowed

Preferred approach: ISA Investigation Comments

Why not RO-Crate Organization / publisher?

RO-Crate contextual entities (Organization + isPartOf / publisher) are semantically correct, and RoCratePayload preserves extra @graph nodes in CouchDB.

However, arctrl drops them on round-trip:

ARC.from_rocrate_json_string(json_with_Organization) → ToROCrateJsonString()
→ publisher / isPartOf / Organization nodes are gone

The Git sync path uses arctrl for writing the scaffold, so RO-Crate-only enrichment would not survive Git push.

Why not proprietary sidecar files?

arc.Write() produces the ISA scaffold (isa.investigation.xlsx, studies/, …) and does not emit ro-crate-metadata.json at the Git repo root.

The middleware Git sync wipes the repo (except .git) before each arc.Write(). Arbitrary files such as fairagro-rdi.json would be deleted unless they are part of the ARC object.

Sidecar files would also be invisible to standard ARC tooling.

Why not ISA Publication or Publisher?

The ISA Abstract Model Investigation has Publications (scholarly articles), Contacts (people with Affiliation as free text), but no Publisher field for hosting infrastructure.

Comment[Funder] (ISA-Tab Scientific Data extension) targets funding agencies, not RDI hosting — semantically wrong for our use case.

ISA Comment[<name>] — the sweet spot

ISA-XLSX allows arbitrary investigation-level comments. arctrl supports them via Comment.create(name, value):

from arctrl import Comment

inv.Comments.append(Comment.create("RDI", "edaphobase"))
inv.Comments.append(Comment.create("RDI Description", "Edaphobase soil biodiversity database"))
inv.Comments.append(Comment.create("RDI URL", "https://www.edaphobase.org"))

Verified behaviour:

Layer	Result
RO-Crate	@type: Comment nodes linked from root Dataset via comment
ISA-XLSX	Comment[RDI] | edaphobase row in isa.investigation.xlsx
arctrl round-trip	Preserved (unlike Organization nodes)
Git sync (arc.Write)	Written into investigation workbook

Proposed comment names (to standardise)

Comment name	Content	Example
RDI	RDI identifier (must match authorized ingest RDI)	edaphobase
RDI Description	Human-readable infrastructure summary	Edaphobase soil biodiversity database
RDI URL	Canonical infrastructure URL	https://www.edaphobase.org

Names are open for bikeshedding; consider a short FAIRagro convention doc once agreed.

Authority model (recommended: middleware enrich + validate)

Source	Role
API rdi / harvest document rdi	Authoritative for auth, arc_id, CouchDB envelope
Deployment RDI registry (config)	Source for description + URL
ISA Comments in ARC	Mirror for portability; optional client pre-fill

Rules to define:

1. On ingest, middleware injects or updates the three Comment[...] fields from the authorized RDI + registry.
2. If the client already supplied RDI comments, they must match the authorized RDI (and registry values, or middleware overwrites) — otherwise 422.
3. Harvest path (POST /v3/harvests/{harvest_id}/arcs): RDI from harvest document; enrichment same as standalone upload.
4. arc_id = sha256(f"{identifier}:{rdi}") unchanged; RDI comment must stay consistent with envelope rdi.

Out of scope / explicitly rejected

• Proprietary sidecar JSON/YAML in the ARC scaffold
• RO-Crate Organization as the only carrier (until arctrl preserves it on round-trip)
• Treating ARC-embedded RDI as the security boundary (certificate allowlist remains external)

Implementation phases

Phase 1 — Design & config

• Agree comment names and validation rules (spec in middleware/api/spec/)
• Extend deployment config: RDI registry beyond known_rdis: list[str] (identifier + description + url)
• Document hybrid authority model (middleware vs. client pre-fill)

Phase 2 — Ingest enrichment

• After RO-Crate validation, inject/update ISA Comments on the arc_content dict or via arctrl after ARC.from_rocrate_json_string
• Validate client-supplied RDI comments against authorized rdi
• Unit tests: enrichment, conflict, harvest path, idempotent re-submit

Phase 3 — Read path & ArcStore decoupling

• Extract RDI from Comments in git_project_metadata_from_arc (fallback to rdi parameter during migration)
• Remove separate rdi argument from ArcStore.create_or_update once reliable
• GitLab topic still derived from Comment[RDI] value

Phase 4 — Client / API docs

• Clarify: rdi in CreateArcRequest remains required for authorization (even when Comments are present)
• api_client README: optional pre-fill of RDI comments

Acceptance criteria

• CouchDB arc_content contains standardized RDI Comments after ingest
• Git repo isa.investigation.xlsx contains Comment[RDI*] rows after sync
• RO-Crate JSON (when generated) includes linked Comment entities
• Authorized RDI from API/harvest/cert cannot be bypassed via ARC content alone
• Re-submitting unchanged ARC does not break idempotency or arc_id
• Behaviour documented when client RDI comments conflict with request RDI

Technical notes

• RoCratePayload already allows extra @graph nodes; Comments are also representable as RO-Crate Comment type after arctrl serialization.
• GitRepo._create_or_update wipes non-.git files before arc.Write() — any solution must go through the ARC object, not loose files.
• Today known_rdis is identifier-only (middleware/api/src/middleware/api/config.py); description/URL need a registry.
• Background: refactoring git_metadata off ArcStore exposed rdi as the last middleware-only context not on the arctrl ARC object.

Related code

• middleware/api/src/middleware/api/business_logic/arc_manager.py
• middleware/api/src/middleware/api/document_store/arc_document.py
• middleware/api/src/middleware/api/arc_store/remote_git_provider.py (git_project_metadata_from_arc)
• middleware/shared/src/middleware/shared/api_models/common/rocrate.py
• middleware/api/spec/arc-upload/spec.md, harvest-arc-upload/spec.md

References

• ISA Abstract Model — Investigation
• ISA-XLSX specification (Comments)
• RO-Crate Contextual Entities (semantically correct but arctrl round-trip issue)
• arctrl skill: .agents/skills/arctrl/SKILL.md