Adds WORM evidence storage for history events and snapshots#345
Conversation
- Add WriteHistoryEvidence function to verify history ranges and store event/snapshot artifacts. - Introduce EventArtifactCandidate and SnapshotArtifactCandidate structures for managing artifacts. - Implement functions to load event and snapshot artifact candidates from the database. - Create HistoryManifest and associated functions for building and validating manifests. - Add ManifestJWSSigner for signing manifests with RSA keys. - Implement artifact storage and retrieval mechanisms for both events and snapshots. - Ensure proper error handling and validation throughout the evidence storage process.
There was a problem hiding this comment.
Pull request overview
This PR introduces a WORM (write-once, read-many) evidence pipeline for BaSyx history data, enabling deterministic per-row history_event artifacts plus optional snapshot checkpoint and signed/unsigned range manifests stored in an S3-compatible object store, with PostgreSQL cataloging and verification tooling.
Changes:
- Add core evidence types, manifest generation/signing/decoding, and PostgreSQL catalog persistence + verification for history ranges.
- Implement an S3-backed
EvidenceStorewith Object Lock retention requirements and wire synchronous evidence writes into the history append transaction path. - Add configuration surface (YAML + env), schema migration
v1.1.3, a CLI (cmd/historyevidenceverifier), updated docs/examples, and CI coverage for the new history package tests.
Reviewed changes
Copilot reviewed 34 out of 35 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/common/jws/jws.go | Add RS256 JWS signing helper used for manifest signing. |
| internal/common/history/snapshot_artifact.go | Build and load snapshot checkpoint artifacts for evidence storage. |
| internal/common/history/manifest.go | Generate canonical history range manifests and deterministic range digests. |
| internal/common/history/manifest_artifact.go | Serialize manifests as unsigned JSON or compact RS256 JWS and decode stored artifacts. |
| internal/common/history/history_event_artifact.go | Build/load per-row history_event evidence artifacts from DB history rows. |
| internal/common/history/evidence_writer.go | Orchestrate verification, artifact storage (events/snapshots/manifests), and catalog recording. |
| internal/common/history/evidence_verifier.go | Verify hash chains, manifest consistency, evidence receipts, and optional object-store hashes. |
| internal/common/history/evidence_types.go | Define evidence store interfaces, artifact/receipt/reference structs, manifest structs. |
| internal/common/history/evidence_test.go | Add unit tests for manifest determinism/signing, S3 receipt behavior, writer/verifier paths. |
| internal/common/history/evidence_store_s3.go | Implement S3 evidence backend with Object Lock retention + verification reads. |
| internal/common/history/evidence_hash.go | Provide SHA-256 helper for evidence artifact hashing. |
| internal/common/history/evidence_catalog.go | Persist evidence manifest/artifact receipts into PostgreSQL catalog tables. |
| internal/common/history/config.go | Add evidence configuration (enabled/provider/store/timeout) and provider initialization helper. |
| internal/common/history/append.go | Integrate synchronous evidence write + catalog insert into history append transaction flow. |
| internal/common/history/append_test.go | Add tests asserting evidence writes happen pre-commit and failures roll back DB tx. |
| internal/common/database.go | Bump schema version constant to v1.1.3. |
| internal/common/configuration.go | Add config schema/env overrides/defaults/validation for evidence + future integrity anchor. |
| internal/common/configuration_test.go | Add tests for evidence defaults, env overrides, and validation behavior. |
| go.mod | Add AWS SDK v2 deps and require gopkg.in/go-jose/go-jose.v2. |
| go.sum | Record checksums for newly added/updated module dependencies. |
| examples/BaSyxHistoryAuditGuardedExample/README.md | Document local MinIO Object Lock setup and evidence publish/verify workflow. |
| examples/BaSyxHistoryAuditGuardedExample/docker-compose.yml | Add MinIO + init container and wire evidence env vars into the example. |
| docu/user/aas_api_v3_2.md | Document new evidence-related environment variables and YAML configuration. |
| docu/developer/aas_v3_2_runtime.md | Describe evidence pipeline, schema migration 1_1_3.sql, and runtime sequence changes. |
| database/patches/1_1_3.sql | Add evidence catalog tables/indexes and bump DB schema to v1.1.3. |
| cmd/submodelrepositoryservice/main.go | Initialize evidence store at service startup via history.ConfigureEvidence. |
| cmd/historyevidenceverifier/main.go | Add CLI to write evidence artifacts and/or verify ranges and stored manifests. |
| cmd/digitaltwinregistryservice/main.go | Initialize evidence store at service startup via history.ConfigureEvidence. |
| cmd/conceptdescriptionrepositoryservice/main.go | Initialize evidence store at service startup via history.ConfigureEvidence. |
| cmd/basyxconfigurationservice/main.go | Register schema patch 1_1_3.sql and adjust 1_1_2.sql version registration. |
| cmd/aasrepositoryservice/main.go | Initialize evidence store at service startup via history.ConfigureEvidence. |
| cmd/aasregistryservice/main.go | Initialize evidence store at service startup via history.ConfigureEvidence. |
| cmd/aasenvironmentservice/main.go | Initialize evidence store at service startup via history.ConfigureEvidence. |
| .github/workflows/examples-smoke.yml | Add smoke job entry for the evidence-enabled history guarded example compose file. |
| .github/workflows/common-tests.yml | Add dedicated go test coverage run for ./internal/common/history. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ory_event artifacts for improved audit attribution
Martin187187
left a comment
Martin187187
left a comment
There was a problem hiding this comment.
is audit information intentionally not added yet?
i get this in minio: "audit":{"actor_issuer":"","actor_subject":"","authorization_result":"","client_id":"","correlation_id":"","endpoint":"","http_method":"","matched_rule_id":"","operation":"","policy_id":"","request_id":"","source_ip":"","user_agent":""}
Deleted Objects are shown no matter if "Show deleted objects" is ticked or not. Maybe this can be configured when requesting the histroy update to minio?
|
Yes, that will be introduced with the next PR |
Deleted here means deleted from the WORM storage, not deleted in Terms of an AAS was deleted. The WORM storage is also append only. You can't delete stuff from there. And even if you do, you can not fully delete it and will always see it if you check this checkbox. Btw, history ist not fetched from the WORM storage. It always comes from the PostgreDB. The WORM storage is a recovery and traceability storage, not for providing the data for the /$history endpoints |
|
Error updating Property "PcfCO2eq"! Status: 500 I get thiss error when i start the backend wihtout minio running and trying to change a value in a submodel. Maybe 503 Service Unavailable is more fitting? |
Summary
Adds WORM-compatible history evidence support for AAS v3.2 history.
The default integrity architecture is now:
PostgreSQL history tables -> hash chain -> signed/unsigned manifest -> S3-compatible WORM object storageThis keeps the existing
$historyand$recent-changesHTTP APIs unchanged, while adding external evidence artifacts for tamper detection, verification, and bounded recovery.Changes
EvidenceStoreabstraction for immutable evidence artifacts.history_eventartifact writes during history append:history_eventartifacts containing:effective_difffor attribution, even when the stored recovery payload is a full snapshot1_1_3.sql.cmd/historyevidenceverifier.BaSyxHistoryAuditGuardedExample.IntegrityAnchor/external_anchoras optional future extension point for immudb, Rekor, Trillian, or timestamping.Important Behavior
When evidence storage is enabled, successful API writes require both PostgreSQL and WORM evidence storage to succeed. If the S3-compatible backend is unavailable, rejects retention metadata, or times out, the mutation fails and PostgreSQL rolls back.
Recovery from WORM evidence is bounded by
history.fullSnapshotInterval:1: every row is recoverable as a full WORM snapshot>1: recovery starts from the latest WORM snapshot and replays WORM-stored diffsThe
effective_difffield prevents attribution confusion when a full snapshot checkpoint is written for a small user change.Security / Compliance Notes
This feature strengthens integrity protection, traceability, auditability, recovery evidence, and unauthorized modification detection. It does not by itself make a deployment NIS2-compliant.
Production deployments still need correctly operated WORM storage, restricted IAM, retention policy, key management, monitoring, audit logs, backups, and operational procedures.
MinIO is included only as a local/test S3-compatible Object Lock backend. Production targets should use an operated S3-compatible WORM service, for example AWS S3 Object Lock.