feat(storage): exclusive field-reference file ownership — ADR-0104 D3 wave 2 (PR-3) (#3527)

* feat(storage): exclusive field-reference file ownership — ADR-0104 D3 wave 2 (PR-3)

A file/image/avatar/video/audio field holding a sys_file id now records its
owner on the file itself: sys_file.ref_object / ref_id / ref_field name the
single (object, record, field) slot that references it, maintained on the
engine write path — claimed on insert, reconciled on update, released when the
owning record is deleted.

Field references are EXCLUSIVE, unlike attachments. The attachments surface
deliberately shares one file across many sys_attachment join rows; a field
reference is owned by at most one slot, and writing an already-owned id into a
second slot copies the bytes into a fresh sys_file rather than sharing the row.

Two properties follow, and both exist so a mistake cannot become unrecoverable:

  1. A file's read authorisation derives from exactly one parent record instead
     of the union of every referrer's — so copying a private record's file id
     into a world-readable record, which generated code does without any
     visible error, cannot silently widen who can read the bytes.
  2. Release is an OBSERVED transition ("my one owner let go"), never an
     inferred absence ("I counted and found nobody"). With no count, no
     miscount can authorise a delete — and a freshly-built ledger's ignorance
     of pre-existing references cannot either.

Deletes nothing. This records and releases ownership; it never tombstones, and
the scope==='attachments' guardrail that keeps field-referenced files out of
the reap entirely is untouched, so shipping it cannot reclaim any bytes.
Enabling collection must extend releaseOwnership AND the sys_file reap guard's
sweep-time re-verify in the same change — half of it is worse than none, since
tombstoning while the guard still re-verifies only sys_attachment (always empty
for a field file) makes every release a guaranteed byte delete.

Also lifts isFileIdToken into @objectstack/spec/data as the single arbiter of
"is this stored string an opaque file id or a legacy/external URL?", replacing
the private copy in the engine read resolver so the reader and the writer
cannot drift into claiming files nobody expands, or vice versa.

Dormant until a field actually holds an id token: objects without file-class
fields, inline-blob values and URL-shaped values all exit before any I/O.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHpGw3GBA9aFpfwVArRWfd

* fix(storage): write \u0000-free same-slot check

check:nul-bytes caught a raw NUL used as an impossible-record-id sentinel in
the copy-on-claim same-slot comparison. A raw NUL makes ripgrep treat the whole
file as binary, so it silently drops out of code search and every grep-based
lint. Replaced with an explicit null check plus a named isSameSlot helper,
which says what it means and is reused by claimFile.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHpGw3GBA9aFpfwVArRWfd

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/adr-0104-d3w2-pr3-file-ownership.md

@@ -0,0 +1,34 @@
+---
+"@objectstack/service-storage": minor
+"@objectstack/spec": minor
+---
+
+feat(storage): exclusive field-reference file ownership — ADR-0104 D3 wave 2 (PR-3)
+
+A `file`/`image`/`avatar`/`video`/`audio` field that holds a `sys_file` id now
+records its owner on the file: `sys_file.ref_object` / `ref_id` / `ref_field`
+name the single `(object, record, field)` slot that references it, maintained on
+the engine write path — claimed on insert, reconciled on update, released when
+the owning record is deleted.
+
+**Field references are exclusive, unlike attachments.** The attachments surface
+deliberately shares one file across many `sys_attachment` join rows; a field
+reference is owned by at most one slot, and writing an already-owned id into a
+second slot **copies the bytes into a fresh `sys_file`** rather than sharing the
+row. That keeps a file's read authorisation derived from exactly one parent
+record instead of the union of every referrer's — so copying a private record's
+file id into a world-readable one cannot silently widen access — and it removes
+reference counting from the lifecycle entirely: a file is released because its
+one owner let go, never because a count came back zero.
+
+**Deletes nothing.** This records and releases ownership; it never tombstones,
+and the `scope === 'attachments'` guardrail that keeps field-referenced files
+out of the reap is untouched. Collection is a separate, gated change that must
+also extend the reap guard's sweep-time re-verify in the same commit.
+
+Also exports `isFileIdToken` from `@objectstack/spec/data` as the single arbiter
+of "is this stored string an opaque file id, or a legacy/external URL?", now
+shared by the read resolver and the write claimer so the two cannot drift.
+
+Dormant until a field actually holds an id token: objects without file-class
+fields, inline-blob values and URL-shaped values all exit before any I/O.

docs/adr/0104-field-runtime-value-shape-contract.md

@@ -1,8 +1,9 @@
 # ADR-0104: Field runtime value-shape as a first-class contract — spec-owned value schemas, typed action handlers, file-as-reference
 
 - **Status**: Accepted (2026-07-22) — staged per D4. D1 landed (#3429), D2
-  landed (#3432); D3 refined into two waves by the 2026-07-24 addendum below.
-  Strict-default flip of D1/D2 tracked in #3438.
+  landed (#3432); D3 refined into two waves by the 2026-07-24 addendum below,
+  and wave 2's ownership model settled by the 2026-07-27 addendum.
+  Strict-default flip of D1/D2 tracked in #3438; wave 2 sequence in #3459.
 - **Date**: 2026-07-22
 - **Issue**: design follow-up generalizing #3405 / #3406 (inline lookup param
   silently stripped); relates #3407 (silently dropped writes), #1878 / #1891
@@ -481,12 +482,119 @@ rode v16/v17) to amortise the migration cost:
   under AI authoring this is *desirable*: it disambiguates "managed file" from
   "external link" so the AI cannot conflate them.
 
+## Addendum (2026-07-27) — wave 2's ownership model: field references are exclusive
+
+The 2026-07-24 addendum left one thing open, and it turned out to be the
+decision the rest of wave 2 hangs off: **when a field holds a `sys_file` id,
+what is the relationship between file and record?** Implementing the reference
+rows forced the answer.
+
+### Two lineages, and why we take one of each
+
+Enterprise platforms solve file lifecycle two ways:
+
+- **Junction-for-everything** (Salesforce `ContentDocumentLink`, SAP GOS): every
+  reference is a row in one polymorphic link table; a file is shared, and dies
+  when the last link does. Sharing is the feature.
+- **Column-as-reference** (Dataverse File/Image columns): the column *is* the
+  reference; lifecycle follows the row; no sharing.
+
+ObjectStack already runs the first for its attachments surface (`sys_attachment`
++ the ADR-0057 tombstone/reap), and that is right — attaching one document to
+several records is exactly what that surface is for, and the user performs the
+attach explicitly.
+
+**Field references take the second model.** A `product.image` is a *property of
+that product*, and properties are copied, not shared. Concretely: at most one
+`(object, record, field)` slot owns a given `sys_file`; the owner is recorded on
+`sys_file.ref_object` / `ref_id` / `ref_field`; writing an already-owned id into
+a second slot **copies the bytes into a fresh `sys_file`** rather than sharing
+the row.
+
+### Why exclusivity, decided on failure modes rather than elegance
+
+Sharing is more storage-efficient and matches the better-known lineage. We take
+exclusivity anyway, because under AI-authored metadata the two models fail
+differently and the difference is not symmetric.
+
+The naive generated clone — `insert({...src, id: undefined})` — spreads a file
+id into a second record. Every model must let that code *work*; rejecting it
+would force AI authors to learn a bespoke file API, which is precisely the kind
+of special case that produces mistakes. So the real question is what happens
+*after* it works:
+
+| | worst case | recoverable? |
+|---|---|---|
+| Shared + reference-counted | file's readable-by set becomes the **union of every referrer's** — copying a private record's id into a world-readable record silently widens access; and a single miscount authorises an **irreversible byte delete** | no |
+| Exclusive + copy-on-claim | duplicated bytes | yes — it costs money, not data |
+
+Read authorisation for attachment files already derives from the parent record.
+Under sharing, "the parent record" is ambiguous, and the safe reading of an
+ambiguous ACL is the union — which means generated code can leak data while
+containing no visible error. Exclusivity removes the ambiguity by construction:
+one file, one parent, one ACL.
+
+The storage cost this concedes belongs at a different layer anyway: **dedup the
+bytes, not the rows.** Content-hash dedup behind the storage adapter (`etag` is
+already carried on `sys_file`) recovers the savings while leaving every row its
+own owner and its own ACL — the lifecycle never has to reason about sharing.
+A genuinely shared library asset should be modelled as its own object with a
+lookup, or attached through the attachments surface; steering AI authors there
+is the correct outcome, not a limitation.
+
+### The safety principle this buys: observed transition, never inferred absence
+
+> A file becomes collectable only because the platform **observed its one owner
+> let go** — never because a scan **inferred** that nobody references it.
+
+The existing attachment path already obeys this (the tombstone is set by the
+hook that watched the join row die, not by a sweep that found none). Exclusive
+ownership extends the same discipline to fields, and eliminates counting
+entirely: with no count, no miscount can authorise a delete. It also defuses the
+cold-start hazard a new ledger would otherwise carry — a freshly-built reference
+table knows nothing of references that predate it, so trusting its zeroes would
+mass-delete history.
+
+### Consequent resequencing of wave 2
+
+The 2026-07-24 plan packaged the write cutover and the GC enable together, and
+placed the backfill after. That order is wrong: **a ledger may not authorise
+deletion until it has been backfilled and reconciled.** Wave 2 therefore
+sequences as:
+
+1. **Ownership bookkeeping** — `ref_*` columns, claim/release on the write path,
+   copy-on-claim. Records ownership; deletes nothing. Dormant until a field
+   actually holds an id, so it lands safely ahead of the cutover.
+2. **Governed download** — read authorisation derived from the one owning
+   record; anonymous capability URL demoted to opt-in `acl: 'public_read'`.
+3. **Write cutover** (protocol major) — stored form narrows to an id;
+   `accept` / `maxSize` enforced. Still deletes nothing.
+4. **Backfill** — `os migrate` converts legacy inline blobs to `sys_file` rows
+   and claims them; external URLs reported, not converted.
+5. **Reconciliation soak** — `os storage verify-references` compares what
+   records actually hold against recorded ownership, reporting *over-claim*
+   (safe: file retained longer than needed), *under-claim* (**blocking**: a held
+   file with no owner would be treated as free), and unclaimed orphans.
+6. **GC enable** — *gated, irreversible*. Relaxing the `scope === 'attachments'`
+   tombstone guardrail and extending the `sys_file` reap guard's sweep-time
+   re-verify to the ownership columns **must ship in the same change**. Half of
+   it is worse than none: tombstoning released files while the guard still
+   re-verifies only `sys_attachment` — always empty for a field file — turns
+   every release into a guaranteed byte delete rather than a risky one.
+
+R4's acceptance gate is now executable rather than aspirational: step 6 may not
+merge until step 5 reports **zero under-claims for ≥7 consecutive days** on real
+tenant data. R5 (public-posture inventory) and R6 (sub-key read scan) stand.
+
 ### Why this stays inside ADR-0104 rather than a new ADR
 
 D3 is already this ADR's third phase; the two-wave split and the
 enforcement-point principle are a **refinement of the D4 rollout**, not a new
 decision, so they live here. Wave 2's migration mechanics (dual-read window,
 `os migrate` backfill, the R4/R5/R6 gates) are specified in §D3 above and need
-no separate record. Should wave 2's implementation surface a genuinely new
-decision (e.g. the reference-table shape, or a chunked-migration protocol), that
-specific choice — not file-as-reference as a whole — would earn its own ADR.
+no separate record. Wave 2's implementation did surface one genuinely new
+decision — the exclusive-ownership model — and it is recorded as the 2026-07-27
+addendum above rather than a separate ADR, because it settles *how* D3's
+already-accepted "field values point into `sys_file`" behaves rather than
+revisiting whether to do it. A future choice that stands on its own (say, a
+chunked-migration protocol, or byte-layer content dedup) would earn its own ADR.

packages/objectql/src/engine.ts

@@ -12,7 +12,7 @@ import {
   type DroppedFieldsEvent
 } from '@objectstack/spec/data';
 import type { WriteObservabilityOptions } from '@objectstack/spec/contracts';
-import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, FILE_REFERENCE_TYPES } from '@objectstack/spec/data';
+import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, FILE_REFERENCE_TYPES, isFileIdToken } from '@objectstack/spec/data';
 import { ExecutionContext, ExecutionContextSchema } from '@objectstack/spec/kernel';
 import { IDataDriver, IDataEngine, Logger, createLogger, withTransientRetry, type RetryOptions } from '@objectstack/core';
 import { SummaryRecomputeError, type SummaryRecomputeFailure } from './summary-errors.js';
@@ -2142,14 +2142,6 @@ export class ObjectQL implements IDataEngine {
     return records;
   }
 
-  /**
-   * Whether a value is an opaque `sys_file` id token — the minted uuid/nanoid
-   * form (letters, digits, `_`, `-`, bounded length), and crucially NOT a URL:
-   * a `https://…` / `/api/…` / `data:…` / `blob:…` file value carries `:`, `/`
-   * or `.` and is left untouched (it is a legacy/external URL, not a reference).
-   */
-  private static readonly FILE_ID_RE = /^[A-Za-z0-9_-]{1,64}$/;
-
   /**
    * Resolve file-field id references to their expanded `FileValueSchema` form
    * (ADR-0104 D3 wave 2). A `file`/`image`/`avatar`/`video`/`audio` value
@@ -2190,7 +2182,7 @@ export class ObjectQL implements IDataEngine {
     // what keeps a seeded `data:`/CDN image value from firing a bogus lookup.
     const candidateIds: string[] = [];
     const addCandidate = (v: unknown) => {
-      if (typeof v === 'string' && ObjectQL.FILE_ID_RE.test(v)) candidateIds.push(v);
+      if (isFileIdToken(v)) candidateIds.push(v);
     };
     for (const record of records) {
       for (const fieldName of fileFields) {