You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have searched existing issues and this is not a duplicate
I understand this issue needs status:approved before a PR can be opened
π Bug Description
Server chunk-ingest (WriteChunk β materializedChunkMutations) materializes relation mutations into cloud_mutations without validating relation required-fields, unlike the mutation-push path, which rejects incomplete relations with a 400. A relation payload that /sync/mutations/push refuses is silently accepted via the chunk-upload path and later applied into memory_relations with empty judgment metadata, because the consumer apply path validates only the FK, not the payload fields. This is a server-side validation bypass: the chunk endpoint is not the required-field boundary the push endpoint is.
This is the validation sibling of #502. #502 (PR #600) correctly restores relation presence in chunk-ingest; this issue is the validation parity that the chunk path still lacks and becomes reachable once relations flow through it.
Root cause
Relation required-field validation lives only in the push HTTP handler, never in the chunk-ingest layer:
Push path (validates, atomic reject):handleMutationPush (internal/cloud/cloudserver/mutations.go:64) runs every entry through validateMutationEntry (:346) β validateRelationPayload (:317) against relationRequiredFields (:303: sync_id, source_id, target_id, relation, judgment_status, marked_by_actor, marked_by_kind) and rejects the entire batch with 400 on any missing field, before InsertMutationBatch.
Chunk-ingest path (no field validation):materializedChunkMutations (internal/cloud/cloudstore/cloudstore.go:357, relation loop at :402) only trims, defaults opβupsert and empty payloadβ{}, and appends. It never checks relation required-fields.
Apply path (FK-only):applyRelationUpsertTx (internal/store/store.go:5632) guards decode + non-empty source_id/target_id (ErrApplyDead) + referenced observations existing (ErrRelationFKMissing). It never checks judgment_status/marked_by_actor/marked_by_kind, so an incomplete-but-FK-valid relation upserts into memory_relations with empty metadata.
Build a chunk whose mutations[] contains a relation entry missing judgment_status / marked_by_actor / marked_by_kind but with valid, resolvable source_id / target_id.
Upload via the chunk path (engram sync --cloud). The relation lands in cloud_mutations (control: the same payload sent to /sync/mutations/push returns 400 invalid relation payload).
A consumer pulls via /sync/mutations/pull β applyRelationUpsertTx upserts the relation into memory_relations with empty judgment metadata, no error.
Unit-level: a materializedChunkMutations test with an incomplete relation currently returns it as a materialized entry; after the fix it should be rejected/quarantined by the same rule the push path applies.
β Expected Behavior
Chunk-ingest should enforce the same relation required-field contract as the push path, so an incomplete relation cannot enter cloud_mutations via the chunk endpoint and corrupt the consumer relation graph.
β Actual Behavior
Incomplete relations bypass validation through the chunk endpoint and are applied with empty judgment metadata. No immediate data loss, but the server-side relation graph is corrupted with metadata-less rows, and the two ingest endpoints disagree on what a valid relation is.
Operating System
macOS
Engram Version
main (validation asymmetry is live once relation chunk-ingest materialization from #600 is present)
Agent / Client
Claude Code
π Relevant Logs
# same relation payload, two endpoints:
POST /sync/mutations/push -> 400 {"error":"invalid relation payload","reason_code":"validation_error", ...} # validateRelationPayload rejects
engram sync --cloud (chunk path) -> relation materialized into cloud_mutations, no error # materializedChunkMutations does not validate
# consumer apply: upserted into memory_relations with empty judgment_status/marked_by_* (applyRelationUpsertTx guards FK only)
π‘ Additional Context
Suggested fix (with the three constraints verified against main):
Use the push path validator (relationRequiredFields / validateRelationPayload, 7 fields, no project), not store.ValidateSyncMutationPayload β the latter is a doctor-only diagnostic and additionally requires project, which the framework deliberately leaves empty for REQ-011 source-missing relations (store/relations.go:696, :897), so it would reject legitimate relations.
Decide per-row skip/quarantine vs. atomic whole-batch rejection. A naive return nil, err inside materializedChunkMutations aborts the whole chunk (it runs at cloudstore.go:246, before BeginTx at :251), dropping every valid session/observation/prompt/relation in the chunk. The push path uses atomic-batch rejection; the chunk path should either match that explicitly or skip+log the offending relation.
Factor the push validator so both endpoints share one relation contract, to prevent the two paths from drifting again.
Related:#502 / #600 (relation presence in chunk-ingest β this is its validation counterpart); #576 (client-side chunk-import relation FK-miss deferral β a different sibling). No overlap in code with either.
Severity: medium β no data loss, narrow trigger (requires a fabricated/legacy/buggy chunk with an incomplete relation; normally-synced relations are always judged and complete), but it is a real server-side validation bypass that corrupts the relation graph and diverges from the push contract.
π Pre-flight Checks
status:approvedbefore a PR can be openedπ Bug Description
Server chunk-ingest (
WriteChunkβmaterializedChunkMutations) materializes relation mutations intocloud_mutationswithout validating relation required-fields, unlike the mutation-push path, which rejects incomplete relations with a400. A relation payload that/sync/mutations/pushrefuses is silently accepted via the chunk-upload path and later applied intomemory_relationswith empty judgment metadata, because the consumer apply path validates only the FK, not the payload fields. This is a server-side validation bypass: the chunk endpoint is not the required-field boundary the push endpoint is.This is the validation sibling of #502. #502 (PR #600) correctly restores relation presence in chunk-ingest; this issue is the validation parity that the chunk path still lacks and becomes reachable once relations flow through it.
Root cause
Relation required-field validation lives only in the push HTTP handler, never in the chunk-ingest layer:
handleMutationPush(internal/cloud/cloudserver/mutations.go:64) runs every entry throughvalidateMutationEntry(:346) βvalidateRelationPayload(:317) againstrelationRequiredFields(:303:sync_id, source_id, target_id, relation, judgment_status, marked_by_actor, marked_by_kind) and rejects the entire batch with400on any missing field, beforeInsertMutationBatch.materializedChunkMutations(internal/cloud/cloudstore/cloudstore.go:357, relation loop at:402) only trims, defaultsopβupsertand empty payloadβ{}, and appends. It never checks relation required-fields.applyRelationUpsertTx(internal/store/store.go:5632) guards decode + non-emptysource_id/target_id(ErrApplyDead) + referenced observations existing (ErrRelationFKMissing). It never checksjudgment_status/marked_by_actor/marked_by_kind, so an incomplete-but-FK-valid relation upserts intomemory_relationswith empty metadata.π Steps to Reproduce
main(with fix(cloud): materialize relation mutations on chunk-ingest (WriteChunk)Β #600 / relation chunk-ingest materialization present).mutations[]contains arelationentry missingjudgment_status/marked_by_actor/marked_by_kindbut with valid, resolvablesource_id/target_id.engram sync --cloud). The relation lands incloud_mutations(control: the same payload sent to/sync/mutations/pushreturns400 invalid relation payload)./sync/mutations/pullβapplyRelationUpsertTxupserts the relation intomemory_relationswith empty judgment metadata, no error.Unit-level: a
materializedChunkMutationstest with an incomplete relation currently returns it as a materialized entry; after the fix it should be rejected/quarantined by the same rule the push path applies.β Expected Behavior
Chunk-ingest should enforce the same relation required-field contract as the push path, so an incomplete relation cannot enter
cloud_mutationsvia the chunk endpoint and corrupt the consumer relation graph.β Actual Behavior
Incomplete relations bypass validation through the chunk endpoint and are applied with empty judgment metadata. No immediate data loss, but the server-side relation graph is corrupted with metadata-less rows, and the two ingest endpoints disagree on what a valid relation is.
Operating System
macOS
Engram Version
main(validation asymmetry is live once relation chunk-ingest materialization from #600 is present)Agent / Client
Claude Code
π Relevant Logs
π‘ Additional Context
Suggested fix (with the three constraints verified against
main):relationRequiredFields/validateRelationPayload, 7 fields, noproject), notstore.ValidateSyncMutationPayloadβ the latter is a doctor-only diagnostic and additionally requiresproject, which the framework deliberately leaves empty for REQ-011 source-missing relations (store/relations.go:696,:897), so it would reject legitimate relations.return nil, errinsidematerializedChunkMutationsaborts the whole chunk (it runs atcloudstore.go:246, beforeBeginTxat:251), dropping every valid session/observation/prompt/relation in the chunk. The push path uses atomic-batch rejection; the chunk path should either match that explicitly or skip+log the offending relation.Related: #502 / #600 (relation presence in chunk-ingest β this is its validation counterpart); #576 (client-side chunk-import relation FK-miss deferral β a different sibling). No overlap in code with either.
Severity: medium β no data loss, narrow trigger (requires a fabricated/legacy/buggy chunk with an incomplete relation; normally-synced relations are always judged and complete), but it is a real server-side validation bypass that corrupts the relation graph and diverges from the push contract.