objectstack-ai
diff --git a/‎.changeset/adr-0104-d3w2-pr6-backfill.md‎
Lines changed: 34 additions & 0 deletions b/‎.changeset/adr-0104-d3w2-pr6-backfill.md‎
Lines changed: 34 additions & 0 deletions
diff --git a/‎docs/adr/0104-field-runtime-value-shape-contract.md‎
Lines changed: 27 additions & 9 deletions b/‎docs/adr/0104-field-runtime-value-shape-contract.md‎
Lines changed: 27 additions & 9 deletions
diff --git a/‎packages/services/service-storage/src/backfill-file-references.test.ts‎
Lines changed: 299 additions & 0 deletions b/‎packages/services/service-storage/src/backfill-file-references.test.ts‎
Lines changed: 299 additions & 0 deletions
@@ -0,0 +1,34 @@
+---
+"@objectstack/service-storage": minor
+---
+
+feat(storage): legacy file-value backfill — ADR-0104 D3 wave 2 (PR-6)
+
+`backfillFileReferences()` converts the pre-reference forms a `file`/`image`/
+`avatar`/`video`/`audio` field may hold — an inline metadata blob
+(`{url, name, size, …}`) or a bare URL string — into the reference form: an
+opaque `sys_file` id, owned by the record's field.
+
+What it will and will not convert:
+
+- **A URL naming this platform's own resolver** (`…/storage/files/:id`) already
+  identifies a `sys_file`; the field is rewritten to the bare id and no bytes
+  move.
+- **A `data:` URI** carries its bytes inline; they are uploaded, a `sys_file` is
+  registered, and the field is rewritten to its id.
+- **An external URL** is reported, never converted. Re-hosting third-party
+  content is a bandwidth, licensing and privacy decision that is not a
+  migration's to make — ADR-0104 R7 retires these toward an explicit `url`
+  field, which under AI authoring is the point: it stops "managed file" and
+  "external link" being the same declaration.
+
+**Dry run by default** — nothing is written unless `apply` is set, and the
+dry-run report has the same shape as the applied one so the plan can be reviewed
+and diffed. **Idempotent** — a value already in reference form is recorded and
+left alone, so a partially-completed run is safe to repeat.
+
+The backfill never writes the ownership columns itself: it rewrites the record,
+and the claim hooks observe that write and record ownership. One claiming path,
+so there is nothing that can disagree with itself. Run
+`verifyFileReferences()` afterwards to confirm the two agree — that
+reconciliation is the gate the irreversible collection change must pass.
@@ -567,24 +567,42 @@ sequences as:
    actually holds an id, so it lands safely ahead of the cutover.
 2. **Governed download** — read authorisation derived from the one owning
    record; anonymous capability URL demoted to opt-in `acl: 'public_read'`.
-3. **Write cutover** (protocol major) — stored form narrows to an id;
+3. **Backfill** — converts legacy inline blobs and own-resolver URLs to
+   `sys_file` references, which the claim hooks then own; external URLs are
+   reported, never re-hosted. Dry-run by default, idempotent.
+4. **Write cutover** (protocol major) — stored form narrows to an id;
    `accept` / `maxSize` enforced. Still deletes nothing.
-4. **Backfill** — `os migrate` converts legacy inline blobs to `sys_file` rows
-   and claims them; external URLs reported, not converted.
-5. **Reconciliation soak** — `os storage verify-references` compares what
-   records actually hold against recorded ownership, reporting *over-claim*
-   (safe: file retained longer than needed), *under-claim* (**blocking**: a held
-   file with no owner would be treated as free), and unclaimed orphans.
+5. **Reconciliation soak** — `verify-references` compares what records actually
+   hold against recorded ownership, and splits disagreements by whether they can
+   cause data loss: *blocking* (a held file nothing owns; a file held by a slot
+   that does not own it; one file held by two slots) versus *advisory* (owned
+   but no longer held — fails toward retention; unreferenced committed files —
+   storage cost only).
 6. **GC enable** — *gated, irreversible*. Relaxing the `scope === 'attachments'`
    tombstone guardrail and extending the `sys_file` reap guard's sweep-time
    re-verify to the ownership columns **must ship in the same change**. Half of
    it is worse than none: tombstoning released files while the guard still
    re-verifies only `sys_attachment` — always empty for a field file — turns
    every release into a guaranteed byte delete rather than a risky one.
 
+Backfill precedes the cutover for the same reason it precedes collection, one
+step further back: narrowing the accepted stored form while records still hold
+legacy values would reject any update that rewrites such a field, breaking
+working apps until the backfill catches up. Backfilling first leaves the cutover
+nothing legacy to reject.
+
 R4's acceptance gate is now executable rather than aspirational: step 6 may not
-merge until step 5 reports **zero under-claims for ≥7 consecutive days** on real
-tenant data. R5 (public-posture inventory) and R6 (sub-key read scan) stand.
+merge until step 5 reports **zero blocking discrepancies for ≥7 consecutive
+days** on real tenant data. R5 (public-posture inventory) and R6 (sub-key read
+scan) stand.
+
+Steps 4 and 6 are the two that can break something. Step 4 is a declared
+protocol major and breaks *loudly* — a rejected write, recoverable. Step 6
+breaks *silently and permanently* — deleted bytes. They are therefore held to
+different standards: step 4 rides the planned v17 window paired with the client
+adoption that consumes the new form, while step 6 additionally requires the
+reconciliation evidence above, and neither the enable nor the migration run
+should be performed without an explicit human decision.
 
 ### Why this stays inside ADR-0104 rather than a new ADR
 
 
@@ -0,0 +1,299 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+
+import { describe, it, expect, vi } from 'vitest';
+import {
+  backfillFileReferences,
+  formatBackfillReport,
+  type BackfillEngine,
+} from './backfill-file-references.js';
+
+const silentLogger = () => ({ info: vi.fn(), warn: vi.fn(), debug: vi.fn() });
+
+const REGISTRY: Record<string, any> = {
+  sys_file: { fields: { id: { type: 'text' } } },
+  product: {
+    fields: {
+      id: { type: 'text' },
+      name: { type: 'text' },
+      image: { type: 'image' },
+      gallery: { type: 'image', multiple: true },
+    },
+  },
+  tag: { fields: { id: { type: 'text' }, label: { type: 'text' } } },
+};
+
+function fakeEngine(tables: Record<string, Array<Record<string, unknown>>>) {
+  const calls: Array<{ op: string; object: string; arg: any }> = [];
+  const engine: BackfillEngine & { tables: typeof tables; calls: typeof calls } = {
+    getObject: (name) => REGISTRY[name],
+    getConfigs: () => REGISTRY,
+    async find(object, options: any) {
+      const rows = tables[object] ?? [];
+      const start = typeof options?.offset === 'number' ? options.offset : 0;
+      const end = typeof options?.limit === 'number' ? start + options.limit : undefined;
+      return rows.slice(start, end);
+    },
+    async insert(object, data: any) {
+      calls.push({ op: 'insert', object, arg: data });
+      (tables[object] ??= []).push({ ...data });
+      return data;
+    },
+    async update(object, data: any) {
+      calls.push({ op: 'update', object, arg: data });
+      const row = (tables[object] ?? []).find((r) => String(r.id) === String(data.id));
+      if (row) Object.assign(row, data);
+      return row;
+    },
+    tables,
+    calls,
+  };
+  return engine;
+}
+
+const fakeStorage = () =>
+  ({
+    upload: vi.fn(async () => {}),
+    download: vi.fn(async () => Buffer.from('x')),
+    delete: vi.fn(async () => {}),
+    exists: vi.fn(async () => true),
+    getInfo: vi.fn(async () => ({ key: 'k', size: 1, contentType: 'text/plain', lastModified: new Date() })),
+  }) as any;
+
+const run = (engine: any, opts: any = {}, storage: any = fakeStorage()) =>
+  backfillFileReferences(engine, () => storage, silentLogger(), opts);
+
+describe('backfillFileReferences (ADR-0104 D3 wave 2)', () => {
+  it('rewrites a URL that already names a sys_file to the bare id, moving no bytes', async () => {
+    const engine = fakeEngine({
+      product: [{ id: 'p1', image: 'https://app.example.com/api/v1/storage/files/file_a' }],
+      sys_file: [],
+    });
+    const storage = fakeStorage();
+
+    const report = await run(engine, { apply: true }, storage);
+
+    expect(engine.tables.product[0].image).toBe('file_a');
+    expect(report.converted).toBe(1);
+    expect(storage.upload).not.toHaveBeenCalled();
+    expect(report.actions[0].kind).toBe('resolved_local_url');
+  });
+
+  it('resolves the same URL when it is wrapped in a legacy inline blob', async () => {
+    const engine = fakeEngine({
+      product: [
+        { id: 'p1', image: { url: '/api/v1/storage/files/file_b', name: 'b.png', size: 12 } },
+      ],
+      sys_file: [],
+    });
+
+    await run(engine, { apply: true });
+
+    expect(engine.tables.product[0].image).toBe('file_b');
+  });
+
+  it('uploads inline data: bytes and rewrites the field to the new id', async () => {
+    const engine = fakeEngine({
+      product: [{ id: 'p1', image: { url: 'data:image/png;base64,aGVsbG8=', name: 'hi.png' } }],
+      sys_file: [],
+    });
+    const storage = fakeStorage();
+
+    const report = await run(engine, { apply: true }, storage);
+
+    expect(storage.upload).toHaveBeenCalledTimes(1);
+    const created = engine.tables.sys_file[0];
+    expect(created).toMatchObject({ name: 'hi.png', mime_type: 'image/png', status: 'committed' });
+    expect(created.size).toBe(5); // "hello"
+    expect(engine.tables.product[0].image).toBe(created.id);
+    expect(report.converted).toBe(1);
+  });
+
+  /**
+   * Converting an external URL would mean fetching third-party content and
+   * silently re-hosting it — a bandwidth/licensing/privacy decision that is not
+   * a migration's to make.
+   */
+  it('reports an external URL and never re-hosts it', async () => {
+    const engine = fakeEngine({
+      product: [{ id: 'p1', image: 'https://cdn.example.com/logo.png' }],
+      sys_file: [],
+    });
+    const storage = fakeStorage();
+
+    const report = await run(engine, { apply: true }, storage);
+
+    expect(engine.tables.product[0].image).toBe('https://cdn.example.com/logo.png');
+    expect(storage.upload).not.toHaveBeenCalled();
+    expect(report.externalUrls).toBe(1);
+    expect(report.converted).toBe(0);
+    expect(formatBackfillReport(report)).toContain('NOT re-hosted');
+  });
+
+  it('handles a multiple:true field element-wise', async () => {
+    const engine = fakeEngine({
+      product: [
+        {
+          id: 'p1',
+          gallery: [
+            '/api/v1/storage/files/file_a',
+            'https://cdn.example.com/x.png',
+            'file_already',
+          ],
+        },
+      ],
+      sys_file: [],
+    });
+
+    const report = await run(engine, { apply: true });
+
+    expect(engine.tables.product[0].gallery).toEqual([
+      'file_a',
+      'https://cdn.example.com/x.png',
+      'file_already',
+    ]);
+    expect(report.converted).toBe(1);
+    expect(report.externalUrls).toBe(1);
+    expect(report.alreadyReferences).toBe(1);
+  });
+
+  // ── Dry run ───────────────────────────────────────────────────────
+  it('writes nothing on a dry run but reports the same plan', async () => {
+    const engine = fakeEngine({
+      product: [
+        { id: 'p1', image: '/api/v1/storage/files/file_a' },
+        { id: 'p2', image: 'data:text/plain;base64,aGk=' },
+      ],
+      sys_file: [],
+    });
+    const storage = fakeStorage();
+
+    const report = await run(engine, {}, storage);
+
+    expect(report.applied).toBe(false);
+    expect(report.converted).toBe(2);
+    expect(engine.calls).toHaveLength(0);
+    expect(engine.tables.product[0].image).toBe('/api/v1/storage/files/file_a');
+    expect(storage.upload).not.toHaveBeenCalled();
+    expect(formatBackfillReport(report)).toContain('DRY RUN');
+  });
+
+  // ── Idempotence / safety ──────────────────────────────────────────
+  it('is idempotent — a second run converts nothing', async () => {
+    const engine = fakeEngine({
+      product: [{ id: 'p1', image: '/api/v1/storage/files/file_a' }],
+      sys_file: [],
+    });
+
+    await run(engine, { apply: true });
+    const second = await run(engine, { apply: true });
+
+    expect(second.converted).toBe(0);
+    expect(second.alreadyReferences).toBe(1);
+  });
+
+  it('never writes the ref_* columns itself — claiming stays on one path', async () => {
+    const engine = fakeEngine({
+      product: [{ id: 'p1', image: '/api/v1/storage/files/file_a' }],
+      sys_file: [],
+    });
+
+    await run(engine, { apply: true });
+
+    // The record write is what the claim hooks observe; the backfill must not
+    // record ownership behind their back, or there would be two claiming paths
+    // that could disagree.
+    for (const c of engine.calls) {
+      expect(c.arg).not.toHaveProperty('ref_object');
+      expect(c.arg).not.toHaveProperty('ref_id');
+      expect(c.arg).not.toHaveProperty('ref_field');
+    }
+    expect(engine.calls.filter((c) => c.object === 'sys_file')).toHaveLength(0);
+  });
+
+  it('leaves data: values alone when no storage service is available', async () => {
+    const engine = fakeEngine({
+      product: [{ id: 'p1', image: 'data:text/plain;base64,aGk=' }],
+      sys_file: [],
+    });
+
+    const report = await backfillFileReferences(engine, () => null, silentLogger(), { apply: true });
+
+    expect(report.unresolvable).toBe(1);
+    expect(report.converted).toBe(0);
+    expect(engine.tables.product[0].image).toBe('data:text/plain;base64,aGk=');
+  });
+
+  it('skips objects with no file-class fields', async () => {
+    const engine = fakeEngine({ tag: [{ id: 't1', label: 'x' }], product: [], sys_file: [] });
+
+    const report = await run(engine, { apply: true });
+
+    expect(report.scannedObjects).toEqual(['product']);
+  });
+
+  /**
+   * The values scanned here come straight out of tenant records, so URL
+   * matching must not be able to backtrack. A pattern anchored only at the tail
+   * retries from every start position, which a value repeating the matched
+   * prefix turns into a polynomial blowup (CodeQL js/polynomial-redos).
+   */
+  it('matches resolver URLs in linear time on an adversarial value', async () => {
+    // Many repetitions of the matched prefix, ending in a segment that is not
+    // an id token — the shape that makes a tail-anchored pattern retry from
+    // every start position.
+    const hostile = '/storage/files/'.repeat(6000) + '!';
+    const engine = fakeEngine({ product: [{ id: 'p1', image: hostile }], sys_file: [] });
+
+    const started = process.hrtime.bigint();
+    const report = await run(engine, {});
+    const elapsedMs = Number(process.hrtime.bigint() - started) / 1e6;
+
+    // Not a reference (the trailing segment is not an id token) — and reaching
+    // that verdict must not depend on how many times the prefix repeats.
+    expect(report.converted).toBe(0);
+    expect(elapsedMs).toBeLessThan(1000);
+  });
+
+  it('accepts the resolver URL shapes and rejects near-misses', async () => {
+    const engine = fakeEngine({
+      product: [
+        { id: 'p1', image: 'https://app.example.com/api/v1/storage/files/aaa' },
+        { id: 'p2', image: '/api/v1/storage/files/bbb/url' },
+        { id: 'p3', image: '/api/v1/storage/files/ccc?v=2' },
+        { id: 'p4', image: '/api/v1/storage/files/ddd/' },
+        { id: 'p5', image: 'https://cdn.example.com/files/eee' }, // no /storage
+        { id: 'p6', image: '/api/v1/storage/files/' }, // no id
+      ],
+      sys_file: [],
+    });
+
+    await run(engine, { apply: true });
+
+    expect(engine.tables.product.map((r) => r.image)).toEqual([
+      'aaa',
+      'bbb',
+      'ccc',
+      'ddd',
+      'https://cdn.example.com/files/eee',
+      '/api/v1/storage/files/',
+    ]);
+  });
+
+  it('pages through large objects and flags truncation at the bound', async () => {
+    const many = Array.from({ length: 500 }, (_, i) => ({
+      id: `p${i}`,
+      image: `/api/v1/storage/files/file_${i}`,
+    }));
+    const engine = fakeEngine({ product: many, sys_file: [] });
+
+    const full = await run(engine, {});
+    expect(full.scannedRecords).toBe(500);
+    expect(full.truncated).toBe(false);
+
+    const bounded = await run(fakeEngine({ product: many, sys_file: [] }), {
+      maxRecordsPerObject: 200,
+    });
+    expect(bounded.truncated).toBe(true);
+    expect(formatBackfillReport(bounded)).toContain('truncated');
+  });
+});