Feature: supplementary file payload alongside RO-Crate JSON on ARC upload

[Zalfsten]

Problem

Clients (e.g. the INSPIRE harvester in m4.2_middleware_harvester) build ARCs with
arctrl and upload them via POST /v3/arcs or POST /v3/harvests/{harvest_id}/arcs.
The request body currently carries only RO-Crate JSON-LD (arc).

Some harvesters must attach supplementary files that are not part of the ISA model
(e.g. raw ISO 19139 XML as iso19115.xml for INSPIRE records). With arctrl 3.1:

Mechanism	Effect
FileSystem.AddFile("iso19115.xml")	Path in FileSystem / GetAdditionalPayload() only
GetWriteContracts()	Does not include supplementary files
ToROCrateJsonString()	Does not include File entities or file bytes in @graph
WriteAsync()	Writes ISA scaffold only; supplementary files are not created

Workarounds considered and rejected:

• Patch RO-Crate JSON with embedded file content → content is lost on
  from_rocrate_json_string() roundtrip; requires extracting patched fields from the
  request before deserialization (fragile, semantically wrong).
• Register files in the ARC object (AddFile) and rely on WriteAsync() → still
  no write contracts; does not write supplementary files.

DataHUB synchronization uses git, not RO-Crate transport. What matters is the
on-disk ARC folder the API materializes. Supplementary files must land in that
scaffold as real files before git push.

Proposal

Extend ARC upload endpoints to accept an optional files payload alongside arc
(RO-Crate JSON-LD). The API writes the ISA scaffold from arc, then writes each
supplementary file into the ARC directory.

Request shape (sketch)

{
  "rdi": "my-rdi",
  "arc": {
    "@context": "...",
    "@graph": [ "... ISA metadata ..." ]
  },
  "files": {
    "iso19115.xml": {
      "content": "PGJhc2U2ND4uLi48L2Jhc2U2ND4=",
      "encoding": "base64",
      "media_type": "application/xml"
    }
  }
}

• files is optional; omitted or {} preserves current behaviour (backward compatible).
• Keys are relative paths from the ARC root (validate: no .., no absolute paths).
• encoding: start with "base64"; plain "utf-8" text may be added later if needed.
• media_type is optional metadata (logging / validation).

Server-side write flow (recommended: two-step)

1. arc_obj = ARC.from_rocrate_json_string(request.arc)
2. WriteAsync(arc_dir)                         → ISA scaffold (xlsx, folders, README, …)
3. for (path, file) in request.files:
       write path under arc_dir with decoded content
4. git commit + push                           → DataHUB

Note: Registering supplementary files in the in-memory ARC object (AddFile) before
WriteAsync() is not required and does not change ISA xlsx content. Direct filesystem
writes after the scaffold are sufficient and simpler.

Alternative (equivalent on disk): fulfill manual Contract.create_create() entries via
full_fill_contract_batch_async() instead of plain write_bytes().

Affected endpoints

• POST /v3/arcs (CreateArcRequest)
• POST /v3/harvests/{harvest_id}/arcs (SubmitHarvestArcRequest)

Related packages (follow-up, can be separate PRs)

Package	Change
middleware.shared	Pydantic models: ArcFilePayload, extend CreateArcRequest / SubmitHarvestArcRequest
middleware.api_client	_serialize_arc() / harvest submission accepts { "arc": ..., "files": ... }
m4.2_middleware_harvester	INSPIRE plugin yields iso19115.xml in files when record.raw_xml is set

Acceptance criteria

• Upload with files creates supplementary files in the ARC directory with correct content.
• Upload without files behaves exactly as today.
• Path validation rejects traversal (../, absolute paths).
• ISA xlsx files are unchanged compared to upload without files (same RO-Crate input).
• Git/DataHUB sync includes supplementary files in the committed tree.
• OpenAPI / shared models document the new field.

References

• Harvester spec: attach iso19115.xml per record (middleware/inspire/spec/inspire-to-arc-mapping/spec.md).
• ARCtrl behaviour documented in harvester repo: .agents/skills/arctrl/SKILL.md (supplementary files section).
• arctrl 3.1: GetWriteContracts() scope excludes AddFile() paths.

Non-goals

• Embedding supplementary file bytes inside RO-Crate @graph (non-standard, lost on roundtrip).
• Changes to arctrl itself (may be pursued separately upstream).