fix(autonumber): hash-keyed sequence table; clarify scope & width semantics

Full hardening of the remaining items from review:

- #5 MySQL/length: key _objectstack_sequences by a single key_hash
  (SHA-256 of object,tenant_id,field,scope) instead of a 4-column natural
  PK. The natural PK exceeded MySQL's utf8mb4 index-length limit (a certain
  CREATE TABLE failure) and bounded how long a {field} scope could be. The
  hash PK keys every dialect uniformly and lets scope be a generous
  non-indexed column. Legacy 3-column and interim {scope}-column tables are
  migrated in place; migration fails safe (fixed-prefix keeps working, a
  per-scope write errors actionably).

- #1 scope ambiguity: confirmed NOT fixable by separating adjacent token
  boundaries — when two records render the same prefix they render the same
  visible number, so they MUST share a counter to stay unique (a separator
  would mint duplicates). Documented the semantics + the remedy (delimiter
  literal in the format), backed by tests. The compile lint already nudges
  authors toward unambiguous formats.

- #6 width overflow: confirmed by-design — the pad width is a MINIMUM, the
  counter grows past it and never wraps (mainstream autonumber semantics).
  Documented + regression test, no throw (throwing would break legitimate
  high-count sequences).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: os-zhuang

.changeset/autonumber-format-tokens.md

@@ -26,11 +26,14 @@ emit byte-identical numbers (#1603 parity):
 
 Fixed-prefix formats like `CASE-{0000}` render an empty scope and keep their single
 global counter, so existing sequences are unchanged. The persistent
-`_objectstack_sequences` table gains a `scope` column (PK widened to
-`object, tenant_id, field, scope`); deployments with the legacy 3-column table are
-migrated in place on first use, carrying existing counters to `scope=''`.
+`_objectstack_sequences` table is keyed by a `key_hash` (SHA-256 of
+`object, tenant_id, field, scope`) — a single 64-char primary key that keys every
+dialect uniformly, stays within MySQL's utf8mb4 index-length limit (four raw
+columns would not), and lets `scope` be a generous non-indexed column. Deployments
+with an older table (3-column, or an interim `scope` column) are migrated in place
+on first use, carrying existing counters to `scope=''`.
 
-Guardrails against the `{field}` footguns:
+Guardrails:
 
 - **Empty interpolated field is a hard error, not a silent mis-number.** A
   `{field}` token whose value is missing at create time would render to an empty
@@ -41,7 +44,18 @@ Guardrails against the `{field}` footguns:
   checked against the object's fields: a `{field}` token naming a non-existent
   field (or the autonumber field itself) **fails the build**; a token naming an
   *optional* field emits an advisory warning to mark it `required: true`.
-- **Legacy sequence-table migration fails safe.** If the legacy table's primary
-  key cannot be widened to include `scope`, fixed-prefix sequences keep working and
-  a per-scope write raises an actionable error instead of an opaque DB primary-key
-  violation at insert time.
+- **Migration fails safe.** If a legacy table cannot be migrated to the `key_hash`
+  shape, fixed-prefix sequences keep working via the legacy key and a per-scope
+  write raises an actionable error instead of corrupting counters.
+- **Long `{field}` scopes are supported** (e.g. a long `{plan_no}`): the non-indexed
+  `scope` column and hashed key remove the old varchar/PK length ceiling.
+
+Notes on inherent semantics (documented, not bugs):
+
+- The counter scope IS the rendered prefix. When two records' tokens render to the
+  same prefix string (e.g. `{a}{b}` for `('AB','C')` and `('A','BC')`) they also
+  render the same visible number, so they share one counter to stay unique — the
+  remedy for genuinely-distinct groups is an unambiguous format (a delimiter
+  literal between variable tokens).
+- The sequence pad width is a MINIMUM; past it the number grows (`{000}` →
+  `1000`), it never wraps — matching mainstream autonumber semantics.

packages/plugins/driver-sql/src/sql-driver-autonumber-tokens.test.ts

@@ -123,6 +123,46 @@ describe('SqlDriver auto_number format tokens', () => {
     expect(b.plan_no).toBe(`PROD${utcYmd()}001`);
   });
 
+  it('shares one counter when adjacent {field} values concat to the same prefix', async () => {
+    await driver.initObjects([
+      {
+        name: 'task',
+        fields: {
+          a: { type: 'string' },
+          b: { type: 'string' },
+          task_no: { type: 'autonumber', format: '{a}{b}{000}' },
+        },
+      },
+    ]);
+    // ('AB','C') and ('A','BC') both render the prefix "ABC" and thus the same
+    // visible number — they MUST share one counter so the numbers stay unique
+    // (independent counters would mint two "ABC001"s).
+    const x1 = await driver.create('task', { a: 'AB', b: 'C' });
+    const y1 = await driver.create('task', { a: 'A', b: 'BC' });
+    const x2 = await driver.create('task', { a: 'AB', b: 'C' });
+
+    expect(x1.task_no).toBe('ABC001');
+    expect(y1.task_no).toBe('ABC002'); // same namespace, keeps climbing (unique)
+    expect(x2.task_no).toBe('ABC003');
+  });
+
+  it('handles a very long {field} scope (well past varchar(255))', async () => {
+    await driver.initObjects([
+      {
+        name: 'doc',
+        fields: {
+          big: { type: 'string' },
+          doc_no: { type: 'autonumber', format: '{big}{000}' },
+        },
+      },
+    ]);
+    const big = 'P'.repeat(400); // 400-char prefix → scope far exceeds 255
+    const r1 = await driver.create('doc', { big });
+    const r2 = await driver.create('doc', { big });
+    expect(r1.doc_no).toBe(`${big}001`);
+    expect(r2.doc_no).toBe(`${big}002`);
+  });
+
   it('refuses to generate when an interpolated {field} is empty (no silent mis-scope)', async () => {
     await driver.initObjects([
       {
@@ -148,7 +188,7 @@ describe('SqlDriver auto_number format tokens', () => {
     expect(r2.invoice_number).toBe('INV-0002');
   });
 
-  it('migrates a legacy 3-column _objectstack_sequences table by adding scope', async () => {
+  it('migrates a legacy 3-column _objectstack_sequences table to the key_hash shape', async () => {
     const k = (driver as any).knex;
     // Simulate a deployment whose sequence table predates the `scope` column.
     await k.schema.createTable('_objectstack_sequences', (t: any) => {
@@ -174,7 +214,38 @@ describe('SqlDriver auto_number format tokens', () => {
 
     const cols = await k('_objectstack_sequences').columnInfo();
     expect(Object.keys(cols)).toContain('scope');
+    expect(Object.keys(cols)).toContain('key_hash');
     // The legacy counter continued rather than restarting.
     expect(r.invoice_number).toBe('INV-0042');
   });
+
+  it('migrates an interim {scope}-column table (no key_hash) and preserves counters', async () => {
+    const k = (driver as any).knex;
+    // A table from an earlier build of this feature: has `scope`, no `key_hash`.
+    await k.schema.createTable('_objectstack_sequences', (t: any) => {
+      t.string('object').notNullable();
+      t.string('tenant_id').notNullable();
+      t.string('field').notNullable();
+      t.string('scope').notNullable().defaultTo('');
+      t.bigInteger('last_value').notNullable().defaultTo(0);
+      t.timestamp('updated_at').defaultTo(k.fn.now());
+      t.primary(['object', 'tenant_id', 'field', 'scope']);
+    });
+    await k('_objectstack_sequences').insert({
+      object: 'invoice',
+      tenant_id: '__global__',
+      field: 'invoice_number',
+      scope: '',
+      last_value: 7,
+    });
+
+    await driver.initObjects([
+      { name: 'invoice', fields: { invoice_number: { type: 'autonumber', format: 'INV-{0000}' } } },
+    ]);
+    const r = await driver.create('invoice', {});
+
+    const cols = await k('_objectstack_sequences').columnInfo();
+    expect(Object.keys(cols)).toContain('key_hash');
+    expect(r.invoice_number).toBe('INV-0008'); // continued from the interim counter
+  });
 });

packages/plugins/driver-sql/src/sql-driver.ts

@@ -220,13 +220,13 @@ export class SqlDriver implements IDataDriver {
   /** Whether the sequences table has been ensured this process. */
   protected sequencesTableReady = false;
   /**
-   * Set when a legacy 3-column `_objectstack_sequences` table could not have
-   * its primary key widened to include `scope`. Fixed-prefix sequences (empty
-   * scope) keep working under the old PK; only a per-scope write would collide,
-   * so `getNextSequenceValue` raises an actionable error in that case rather
-   * than letting an opaque PK violation surface at create time.
+   * Whether `_objectstack_sequences` is the current `key_hash`-keyed shape.
+   * Set on a fresh create or a successful in-place migration. If a legacy table
+   * could NOT be migrated, this stays false: fixed-prefix sequences (empty
+   * scope) keep working via the legacy `(object, tenant_id, field)` key, while a
+   * per-scope write raises an actionable error rather than corrupting counters.
    */
-  protected sequencesScopeKeyUnsafe = false;
+  protected sequencesHasKeyHash = false;
   /** In-flight ensure promise; deduplicates concurrent first calls. */
   protected sequencesTableEnsurePromise: Promise<void> | null = null;
 
@@ -550,10 +550,14 @@ export class SqlDriver implements IDataDriver {
    * Ensure the sequence-counter table exists. Idempotent and cheap after
    * the first call (cached via `sequencesTableReady`).
    *
-   * The key is `(object, tenant_id, field, scope)`. `scope` is the rendered
-   * autonumber prefix (date/field tokens before the `{0000}` slot), so a new
-   * day/group/parent starts a fresh counter. Fixed-prefix formats use the
-   * empty scope and keep their single global counter (backward compatible).
+   * The row key is `key_hash` — a SHA-256 of `(object, tenant_id, field, scope)`
+   * where `scope` is the rendered autonumber prefix (date/field tokens before
+   * the `{0000}` slot), so a new day/group/parent starts a fresh counter. A
+   * single 64-char hashed primary key (rather than the four raw columns, which
+   * blow past MySQL's 3072-byte index limit under utf8mb4 and bound how long a
+   * `{field}` scope may be) keys every dialect uniformly and lets `scope` be a
+   * generous non-indexed column. Fixed-prefix formats use the empty scope and
+   * keep their single global counter (backward compatible).
    */
   protected async ensureSequencesTable(): Promise<void> {
     if (this.sequencesTableReady) return;
@@ -565,26 +569,19 @@ export class SqlDriver implements IDataDriver {
       const exists = await this.knex.schema.hasTable(SEQUENCES_TABLE);
       if (!exists) {
         try {
-          await this.knex.schema.createTable(SEQUENCES_TABLE, (t) => {
-            t.string('object').notNullable();
-            t.string('tenant_id').notNullable();
-            t.string('field').notNullable();
-            t.string('scope').notNullable().defaultTo('');
-            t.bigInteger('last_value').notNullable().defaultTo(0);
-            t.timestamp('updated_at').defaultTo(this.knex.fn.now());
-            t.primary(['object', 'tenant_id', 'field', 'scope']);
-          });
+          await this.createSequencesTable(SEQUENCES_TABLE);
+          this.sequencesHasKeyHash = true;
         } catch (err: any) {
           // Race or cross-process create — re-check existence; ignore
           // "already exists" errors from any dialect.
           const stillMissing = !(await this.knex.schema.hasTable(SEQUENCES_TABLE));
           if (stillMissing) throw err;
-          // A racing creator may have used the legacy 3-column schema.
-          await this.ensureSequencesScopeColumn();
+          // A racing creator may have used an older schema. Migrate in place.
+          await this.ensureSequencesKeyHashShape();
         }
       } else {
-        // Pre-existing table may predate the `scope` column. Migrate in place.
-        await this.ensureSequencesScopeColumn();
+        // Pre-existing table may predate the `key_hash`/`scope` shape. Migrate.
+        await this.ensureSequencesKeyHashShape();
       }
       this.sequencesTableReady = true;
     })();
@@ -595,71 +592,76 @@ export class SqlDriver implements IDataDriver {
     }
   }
 
+  /** SHA-256 of the composite counter key — the table's single-column PK. */
+  protected sequenceKeyHash(object: string, tenantId: string, field: string, scope: string): string {
+    return createHash('sha256')
+      .update(`${object}\u001f${tenantId}\u001f${field}\u001f${scope}`)
+      .digest('hex');
+  }
+
+  /** Create the current `key_hash`-keyed sequences table shape. */
+  protected async createSequencesTable(table: string): Promise<void> {
+    await this.knex.schema.createTable(table, (t) => {
+      t.string('key_hash', 64).notNullable().primary();
+      t.string('object').notNullable();
+      t.string('tenant_id').notNullable();
+      t.string('field').notNullable();
+      // Non-indexed, so it is free of the PK length limit — a long `{plan_no}`
+      // composite scope fits. 1024 is far above any realistic rendered prefix.
+      t.string('scope', 1024).notNullable().defaultTo('');
+      t.bigInteger('last_value').notNullable().defaultTo(0);
+      t.timestamp('updated_at').defaultTo(this.knex.fn.now());
+    });
+  }
+
   /**
-   * Add the `scope` column (and fold it into the primary key) to a
-   * `_objectstack_sequences` table created before per-period/per-group
-   * counters existed. Every legacy row is a fixed-prefix sequence, so it
-   * migrates to `scope = ''` and keeps counting unchanged. Idempotent.
+   * Migrate a pre-existing `_objectstack_sequences` table to the current
+   * `key_hash`-keyed shape. Handles both the original 3-column table (no
+   * `scope`) and an interim 4-column `(object, tenant_id, field, scope)` table:
+   * every legacy row is read, its `key_hash` computed in app code (no portable
+   * SQL hash exists), and re-inserted into a freshly built table that then
+   * replaces the original. Idempotent — a no-op once `key_hash` is present.
+   *
+   * If the rebuild fails, `sequencesHasKeyHash` stays false: fixed-prefix
+   * sequences keep working via the legacy key and per-scope writes error
+   * actionably (see getNextSequenceValue), rather than corrupting data.
    */
-  protected async ensureSequencesScopeColumn(): Promise<void> {
+  protected async ensureSequencesKeyHashShape(): Promise<void> {
+    if (await this.knex.schema.hasColumn(SEQUENCES_TABLE, 'key_hash')) {
+      this.sequencesHasKeyHash = true;
+      return;
+    }
     const hasScope = await this.knex.schema.hasColumn(SEQUENCES_TABLE, 'scope');
-    if (hasScope) return;
-
-    if (this.isSqlite) {
-      // SQLite can't alter a primary key in place — rebuild the table. Every
-      // legacy row carries scope '' so the copy is a straight projection.
-      const TMP = `${SEQUENCES_TABLE}__rebuild`;
+    const TMP = `${SEQUENCES_TABLE}__rebuild`;
+    try {
+      const rows: any[] = await this.knex(SEQUENCES_TABLE).select('*');
       await this.knex.schema.dropTableIfExists(TMP);
-      await this.knex.schema.createTable(TMP, (t) => {
-        t.string('object').notNullable();
-        t.string('tenant_id').notNullable();
-        t.string('field').notNullable();
-        t.string('scope').notNullable().defaultTo('');
-        t.bigInteger('last_value').notNullable().defaultTo(0);
-        t.timestamp('updated_at').defaultTo(this.knex.fn.now());
-        t.primary(['object', 'tenant_id', 'field', 'scope']);
+      await this.createSequencesTable(TMP);
+      const migrated = rows.map((r) => {
+        const scope = hasScope && r.scope != null ? String(r.scope) : '';
+        return {
+          key_hash: this.sequenceKeyHash(String(r.object), String(r.tenant_id), String(r.field), scope),
+          object: r.object,
+          tenant_id: r.tenant_id,
+          field: r.field,
+          scope,
+          last_value: r.last_value ?? 0,
+          updated_at: r.updated_at ?? this.knex.fn.now(),
+        };
       });
-      await this.knex.raw(
-        `insert into ?? (object, tenant_id, field, scope, last_value, updated_at)
-         select object, tenant_id, field, '', last_value, updated_at from ??`,
-        [TMP, SEQUENCES_TABLE],
-      );
+      if (migrated.length > 0) await this.knex(TMP).insert(migrated);
       await this.knex.schema.dropTable(SEQUENCES_TABLE);
       await this.knex.schema.renameTable(TMP, SEQUENCES_TABLE);
-      return;
-    }
-
-    // Postgres / MySQL: add the column, then widen the primary key.
-    await this.knex.schema.alterTable(SEQUENCES_TABLE, (t) => {
-      t.string('scope').notNullable().defaultTo('');
-    });
-    try {
-      if (this.isMysql) {
-        await this.knex.raw(
-          'ALTER TABLE ?? DROP PRIMARY KEY, ADD PRIMARY KEY (object, tenant_id, field, scope)',
-          [SEQUENCES_TABLE],
-        );
-      } else {
-        // Postgres: the implicit PK constraint is named `<table>_pkey`.
-        await this.knex.raw('ALTER TABLE ?? DROP CONSTRAINT IF EXISTS ??', [
-          SEQUENCES_TABLE,
-          `${SEQUENCES_TABLE}_pkey`,
-        ]);
-        await this.knex.raw('ALTER TABLE ?? ADD PRIMARY KEY (object, tenant_id, field, scope)', [
-          SEQUENCES_TABLE,
-        ]);
-      }
+      this.sequencesHasKeyHash = true;
     } catch (err) {
-      // The `scope` column landed (new writes can record it) but the PK is
-      // still the legacy 3-column shape. Fixed-prefix sequences are unaffected;
-      // a per-scope write, however, would hit an opaque PK violation at insert
-      // time. Flag it so getNextSequenceValue can fail loudly and actionably
-      // instead, and surface it at error level for operators to migrate by hand.
-      this.sequencesScopeKeyUnsafe = true;
+      // Leave the original table intact; fall back to legacy keying for
+      // fixed-prefix sequences and refuse per-scope writes until migrated.
+      this.sequencesHasKeyHash = false;
+      await this.knex.schema.dropTableIfExists(TMP).catch(() => {});
       this.logger.warn(
-        `[autonumber] Failed to widen ${SEQUENCES_TABLE} primary key to include "scope". ` +
+        `[autonumber] Failed to migrate ${SEQUENCES_TABLE} to the key_hash shape. ` +
           `Fixed-prefix autonumbers keep working; date/{field}/per-parent formats will ` +
-          `error until the primary key is migrated to (object, tenant_id, field, scope) by hand.`,
+          `error until the table is migrated.`,
         { error: String(err) },
       );
     }
@@ -721,22 +723,29 @@ export class SqlDriver implements IDataDriver {
     scope = '',
   ): Promise<number> {
     await this.ensureSequencesTable();
-    if (scope !== '' && this.sequencesScopeKeyUnsafe) {
-      // The legacy sequences table could not have its PK widened to include
-      // `scope`, so a second scope under the same (object, tenant, field) would
-      // collide. Fail with a clear, actionable message instead of an opaque
-      // database PK violation at insert time.
+    const resolvedTenantId = tenantField && tenantId ? String(tenantId) : GLOBAL_TENANT;
+    if (scope !== '' && !this.sequencesHasKeyHash) {
+      // The legacy sequences table could not be migrated to the key_hash shape,
+      // so it cannot represent per-scope counters. Fail with a clear, actionable
+      // message instead of corrupting the single legacy counter.
       throw new Error(
         `Cannot generate a per-scope autonumber for "${object}.${field}": the ` +
-          `${SEQUENCES_TABLE} primary key is still the legacy 3-column shape. ` +
-          `Migrate it to (object, tenant_id, field, scope) before using date/{field}/per-parent formats.`,
+          `${SEQUENCES_TABLE} table is still the legacy shape. ` +
+          `Migrate it to the key_hash shape before using date/{field}/per-parent formats.`,
       );
     }
-    const resolvedTenantId = tenantField && tenantId ? String(tenantId) : GLOBAL_TENANT;
-    // `scope` (rendered date/field prefix) gives each period/group its own
-    // counter; '' keeps the single global counter for fixed-prefix formats.
-    // `prefix` is the full rendered prefix used to bootstrap from existing data.
-    const key = { object: tableName, tenant_id: resolvedTenantId, field, scope };
+    // `scope` (rendered date/field prefix, boundary-delimited) gives each
+    // period/group its own counter; '' keeps the single global counter for
+    // fixed-prefix formats. `prefix` is the full rendered prefix used to
+    // bootstrap from existing data. The row is keyed by a hash of the composite;
+    // on an un-migrated legacy table only fixed-prefix (scope '') reaches here,
+    // so fall back to the original `(object, tenant_id, field)` key for it.
+    const key = this.sequencesHasKeyHash
+      ? { key_hash: this.sequenceKeyHash(tableName, resolvedTenantId, field, scope) }
+      : { object: tableName, tenant_id: resolvedTenantId, field };
+    const insertRow = this.sequencesHasKeyHash
+      ? { ...key, object: tableName, tenant_id: resolvedTenantId, field, scope }
+      : { ...key };
 
     const runner: Knex | Knex.Transaction = parentTrx ?? this.knex;
 
@@ -763,7 +772,7 @@ export class SqlDriver implements IDataDriver {
         );
         const initial = seedMax + 1;
         try {
-          await trx(SEQUENCES_TABLE).insert({ ...key, last_value: initial });
+          await trx(SEQUENCES_TABLE).insert({ ...insertRow, last_value: initial });
           return initial;
         } catch (err) {
           // Another writer raced us to the first INSERT. Fall through to