Skip to content

Commit 6793097

Browse files
claudeclaude
committed
Merge origin/main — reconcile generateSql with #3652's rendering
#3652 rewrote `ObjectQLStrategy.generateSql` (date_trunc for bucketed dimensions, COUNT DISTINCT, ORDER BY/LIMIT/OFFSET) and made `execute()` echo the rendered SQL on the result. Kept all of that. Resolved the WHERE clause in favour of this branch's version, which renders the same caller filters plus the two things #3652's inline loop does not: - the READ SCOPE (`compileScopedFilterToSql`), the point of #3602 residual 2 — without it the echoed string still reads as an unscoped table scan while the aggregate is scoped; - `assertJoinedScopesEnforceable`, so it never renders SQL for a query `execute()` would reject. Its filter rendering is also closer to what runs: `contains` binds `%v%` (a bare comparand under LIKE would only match exactly), and values go through `coerceFilterValueForObjectQL`, so the comparand shown is the one this path hands the engine. Scope VALUES stay in `params`, which the `execute()` echo discards — only the predicate shape reaches the browser, same as NativeSQL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013st2KArjmLQSuVzpDS91jj
2 parents 354935b + 9aa5510 commit 6793097

68 files changed

Lines changed: 4529 additions & 82 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.changeset/adr-0104-deployment-migration-gate.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
---
2+
"@objectstack/spec": minor
3+
"@objectstack/platform-objects": minor
4+
"@objectstack/service-storage": minor
5+
"@objectstack/objectql": minor
6+
"@objectstack/cli": minor
7+
---
8+
9+
feat(migrate): `os migrate files-to-references` — a data migration with a self-check, gated per deployment (#3617)
10+
11+
The ADR-0104 file-as-reference migration ships as a command a deployment runs
12+
against its own database, and the deployment-level flag it records is what may
13+
later authorise irreversible behaviour — never the platform version.
14+
15+
```bash
16+
os migrate files-to-references # dry run: reports, writes nothing
17+
os migrate files-to-references --apply # converts, verifies, records the flag
18+
```
19+
20+
The run backfills legacy file-field values (inline metadata blobs, own-resolver
21+
URLs, `data:` URIs) into owned `sys_file` references, reconciles the ownership
22+
ledger against what records actually hold, and — only on an `--apply` run whose
23+
reconciliation reports **zero blocking discrepancies** — records
24+
`sys_migration { id: 'adr-0104-file-references', verified_at, blocking: 0 }`.
25+
26+
**Why a flag rather than a release note.** ObjectStack is a development
27+
platform: third-party deployments upgrade on their own schedule and their data
28+
is not observable by anyone else, so no release-side soak can vouch for them.
29+
The evidence has to be produced where the data is. Consequences:
30+
31+
- Installing a new version never starts deleting bytes. Running the migration
32+
and passing its self-check is the consent.
33+
- Not run, or not passed → files are retained forever. Wasted storage, zero
34+
data loss.
35+
- A later failing run **clears** `verified_at`: a deployment whose data has
36+
drifted closes its own gate.
37+
- A dry run writes nothing at all — not the conversions, and not the flag,
38+
even when the self-check would pass.
39+
- External URLs stay advisory. They are not `sys_file`s, so they can never
40+
enter collection; whether to remodel them as a `url` field is the app
41+
author's decision (ADR-0104 R7), not a gate.
42+
43+
Ships alongside:
44+
45+
- `@objectstack/spec` — `DataMigrationFlagSchema`, `FILE_REFERENCES_MIGRATION_ID`,
46+
and the single `isDataMigrationFlagVerified` predicate both future consumers
47+
(collection #3459, strict value-shape #3438) read, so the two gates cannot
48+
disagree about the same fact.
49+
- `@objectstack/platform-objects` — the `sys_migration` object plus
50+
`readDataMigrationFlag` / `isDataMigrationVerified` / `recordDataMigrationRun`.
51+
Reads fail toward "not verified": a gate that cannot read its evidence stays
52+
closed.
53+
- `@objectstack/objectql` — a read may now opt out of file-reference expansion
54+
via the spec's `RAW_FILE_VALUES_CONTEXT_KEY`, and the storage service's
55+
bookkeeping/scan reads do. Without it the read resolver rewrites stored ids to
56+
their expanded form before the reconciliation sees them, which reports held
57+
references as absent — noisy `stale_owner` findings, and a missed
58+
`unowned_reference` would have been a false pass of the collection gate.

‎.changeset/analytics-widget-query-options.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
---
2+
"@objectstack/spec": patch
3+
"@objectstack/service-analytics": patch
4+
"@objectstack/rest": patch
5+
---
6+
7+
feat(analytics): honour widget `dateGranularity`, `sortBy`/`sortOrder`, and `limit` in the dataset query (#3588)
8+
9+
Three presentation options were accepted by the metadata layer and then dropped
10+
by the analytics query builder. They reached no SQL, produced no error, and the
11+
only way to notice was to read the `sql` a dataset response echoes — so a
12+
dashboard could declare `dateGranularity: 'month'` and quietly render one bar
13+
per record.
14+
15+
- **`dateGranularity` now buckets.** `DatasetSelection` gained an optional
16+
`dateGranularity`, applied to every selected `date` dimension. Precedence per
17+
dimension: an explicit `timeDimensions` granularity, then the selection's,
18+
then the dataset dimension's own default. A widget can bucket a trend by month
19+
without the dataset committing every other consumer to that granularity.
20+
- **`order` / `limit` / `offset` now apply on every path.** They are applied to
21+
the ASSEMBLED grid — after measure-scoped sub-queries merge, after `compareTo`
22+
columns attach, and after derived measures are computed — so a derived measure
23+
is a valid sort key and the ObjectQL aggregate path (which has no ordering
24+
grammar, and which native SQL hands every date-bucketed query to) orders
25+
identically to native SQL. A single-query selection still pushes the window
26+
down into the statement. An `order` key that names nothing the selection
27+
projects is now rejected (400) rather than silently ignored.
28+
- **`limit` is deterministic.** Without an `order`, a limit orders by the
29+
selected dimensions first, so it truncates a reproducible window instead of an
30+
arbitrary subset.
31+
- **Widget `options` is a contract again.** The four query-affecting keys
32+
(`dateGranularity`, `sortBy`, `sortOrder`, `limit`) plus `stageOrder` are
33+
declared on `DashboardWidgetOptionsSchema`, so a typo like `sortDirection` is
34+
an author-time error. The bag stays open — renderer extras (`icon`, `columns`,
35+
`striped`, …) pass through untouched.
36+
37+
Two latent bugs surfaced while fixing the above and are fixed here too:
38+
39+
- `order`/`limit` were forwarded to EVERY sub-query. A measure-scoped
40+
supplementary query selects one measure, so an inherited `ORDER BY` named a
41+
column it never selected, and an inherited `LIMIT` truncated it before the
42+
merge — dropping rows from the assembled grid. Nothing hit this only because
43+
nothing passed `order`.
44+
- The `compareTo` pass built its query by hand and skipped granularity
45+
resolution, so a month-bucketed primary grid was merged against raw-timestamp
46+
comparison rows. No dimension key matched and every `<measure>__compare`
47+
column came back empty.
48+
49+
`ObjectQLStrategy` now also echoes a representative `sql` (with `date_trunc`,
50+
`WHERE`, `ORDER BY`, and `LIMIT`; filter values parameterized, never inlined).
51+
Previously the `sql` field simply vanished from the response whenever a query
52+
was date-bucketed, leaving an author unable to tell "not implemented" from "this
53+
strategy doesn't report".

‎.changeset/client-url-conformance-capstone.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
"@objectstack/client": patch
3+
---
4+
5+
test(client): close the route audit's reverse direction — every SDK URL must match a route some surface mounts (#3642)
6+
7+
The capstone of the #3563 route audit. The dispatcher (#3563), REST (#3587) and
8+
service-mount (#3636) ledgers all run server → client: enumerate what a surface
9+
mounts, demand a reviewed disposition, and for `sdk` rows demand the named
10+
client method exists. None of them asked the reverse question — does the URL
11+
the client *builds* match anything a server *mounts*? — so a method could name
12+
a real function, carry a green ledger row, and 404 everywhere.
13+
14+
That shipped four times, found one at a time by hand: `analytics.explain` and
15+
`analytics.meta` (#3584), `meta.getView` (#3611), and `i18n.getTranslations` /
16+
`getFieldLabels` (#3636) — the last pair having carried green `sdk` rows since
17+
tranche 1.
18+
19+
`client-url-conformance.test.ts` drives every method on a real client with a
20+
recording `fetch` and matches each captured URL against the union of all four
21+
ledgers. A real drive rather than a hand-written "method X targets route Y"
22+
table, because such a table is an assertion *about* the code that the code can
23+
drift away from — the exact failure being fixed. Mutation-checked: re-injecting
24+
the #3636 dialect bug fails the suite.
25+
26+
The sweep's own completeness is asserted, since that is what rots silently — a
27+
new method must be driven or declared `NON_HTTP` with a reason; a driven method
28+
emitting zero requests fails (stale placeholder args are how a sweep quietly
29+
stops covering anything); a URL containing `undefined` fails; and the
30+
`__api-endpoint` `(unmatched)` catch-all is excluded from the pattern set so it
31+
cannot match everything and make the suite vacuous.
32+
33+
196 of ~219 methods matched. Two bounds are reported rather than papered over:
34+
`/api/v1/cloud/*` (23 `projects.*` methods) belongs to the sibling `cloud` repo
35+
and is exempt by prefix, bounded so no other namespace can use it (#3655); and
36+
60 of ~196 matched calls rest only on a `**` prefix claim rather than a
37+
resolvable route — 54 of those on `* /auth/**` — a count the guard ratchets so
38+
it can only shrink (#3656).
39+
40+
No runtime change: this is a guard plus the ledger-header and audit-doc notes
41+
recording what it does and does not cover.

‎.changeset/page-header-i18n-3589.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
"@objectstack/spec": minor
3+
"@objectstack/platform-objects": minor
4+
"@objectstack/cli": minor
5+
"@objectstack/rest": patch
6+
---
7+
8+
feat(spec): resolve page metadata i18n — `page:header` title/subtitle (#3589)
9+
10+
Custom system pages authored as metadata (Installed Apps, Cloud Connection,
11+
Connect an Agent) hard-code their `page:header` copy in
12+
`properties.title` / `properties.subtitle`. Every other metadata type is
13+
localized at the REST boundary, but `page` was not: the `pages` namespace
14+
existed only on `AppTranslationBundleSchema` — a schema no runtime reads —
15+
with no resolver behind it, so those headers stayed English in every locale
16+
while the matching nav labels translated correctly.
17+
18+
- `TranslationDataSchema` (the shape the i18n service actually serves) gains a
19+
`pages` namespace: `pages.<name>.{label,description,title,subtitle}`.
20+
- New `translatePage` in `@objectstack/spec/system` translates a page's own
21+
`label` / `description` and overlays `title` / `subtitle` onto every
22+
`page:header` in the page's regions. Registered in
23+
`translateMetadataDocument`, so it rides the existing read path.
24+
- `page` added to the REST boundary's `TRANSLATABLE_META_TYPES`. Locale
25+
extraction, the locale-keyed ETag, and `Vary: Accept-Language` already
26+
covered every metadata type — no new plumbing.
27+
- `objectstack i18n extract` now emits page entries, including the
28+
`page:header` copy, so the new namespace is not invisible to the tooling.
29+
- zh-CN / ja-JP / es-ES translations shipped for the three Setup pages, plus
30+
the missing `nav_cloud_connection` / `nav_connect_agent` nav labels (these
31+
existed only in zh-CN).
32+
33+
Header copy is keyed by **page name**, not by component id: `page:header`
34+
instances carry no stable id. `title` falls back to `pages.<name>.label`, since
35+
a page's header title and its nav label are normally the same string.
36+
37+
Authoring is unchanged and English literals stay in metadata as the fallback —
38+
a page with no `pages` entry renders exactly as before. Consumers of
39+
`@object-ui` need no change: pages arrive already localized from the server.

‎.changeset/scoped-invitation-placement.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
"@objectstack/platform-objects": minor
3+
"@objectstack/plugin-auth": minor
4+
"@objectstack/plugin-security": minor
5+
---
6+
7+
feat(authz): scoped invitations — placement intent on an invitation, gated by
8+
the issuer's adminScope and applied on acceptance (ADR-0105 D8)
9+
10+
An invitation may now carry PLACEMENT INTENT — the business unit the invitee
11+
lands in and the positions they are assigned — so a delegated (plant) admin's
12+
invitee arrives already in the right unit and role instead of waiting on a
13+
platform admin. This closes the structural gap ADR-0105 D8 names for
14+
`single`-posture deployments and is the natural admission path under `group`.
15+
16+
The two halves ship together, deliberately:
17+
18+
- **Issuance is authorized** against the ISSUER's `adminScope` (ADR-0090 D12),
19+
by dry-running the existing `DelegatedAdminGate` against the very
20+
`sys_user_position` rows the acceptance would write. The gate is reused
21+
verbatim — no second copy of the subtree/allowlist logic to drift — so an
22+
invitation can never place what its issuer could not have assigned directly.
23+
Without that gate the feature would be an escalation hole: the built-in
24+
`organization_admin` is deliberately read-only on the RBAC tables precisely
25+
so a fresh org admin cannot rebind themselves, and applying an unchecked
26+
invitation payload under system context would hand that authority straight
27+
back.
28+
- **Acceptance applies it**, idempotently and failure-isolated: a replayed
29+
acceptance converges instead of duplicating assignments, and a placement
30+
miss never undoes a valid membership.
31+
32+
Surface:
33+
34+
- `sys_invitation` gains `business_unit_id` + `positions` (ADR-0092 extension
35+
fields, registered in the D7 collision-guarded whitelist; NOT generically
36+
editable — placement is set only at issuance, through the gate).
37+
- `@objectstack/plugin-security` registers the `invitation-placement` service
38+
(`assertIssuable` / `apply`).
39+
- `@objectstack/plugin-auth` wires better-auth's `beforeCreateInvitation` /
40+
`afterAcceptInvitation` to it. **Fail closed**: an invitation that requests
41+
placement in a deployment without the delegated-administration runtime is
42+
refused, never silently placed unchecked.
43+
44+
Existing invitations are unaffected — an invitation without placement intent
45+
never consults the gate and behaves exactly as before.

‎.changeset/two-factor-lockout-and-object-translations.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
"@objectstack/platform-objects": patch
3+
---
4+
5+
fix(i18n): ship the missing object-translation keys for the better-auth 1.7 and ADR-0105 D6 fields (#3624 follow-up)
6+
7+
The generated object-translation bundles predate two rounds of field additions,
8+
so six fields had no entry in **any** locale and fell back to their raw schema
9+
labels in every UI surface that reads the bundle:
10+
11+
- `sys_team.member_count`, `sys_team_member.membership_key`,
12+
`sys_two_factor.failed_verification_count` / `locked_until` — the better-auth
13+
1.7 columns provisioned in #3647.
14+
- `sys_organization.parent_organization_id` / `sort_order` — the same gap left
15+
by the earlier ADR-0105 D6 group-structure work.
16+
17+
Regenerated with `os i18n extract` (merge mode, so every existing translation is
18+
preserved — the diff is purely additive). No API or schema change; the fields
19+
themselves already shipped.

‎content/docs/deployment/cli.mdx‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -517,6 +517,48 @@ first. It never drops a table that is absent from your metadata, and on SQLite
517517
it reconciles via a table rebuild (copy → swap) that preserves your data.
518518
</Callout>
519519

520+
#### Data migrations
521+
522+
The commands above reconcile **schema**. A *data* migration rewrites rows, and
523+
whether it is done is a fact about **your** database, not about the platform
524+
version you installed — so each deployment runs it, and its result is recorded
525+
where the data lives.
526+
527+
| Command | Description |
528+
|---------|-------------|
529+
| `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag |
530+
531+
```bash
532+
os migrate files-to-references # Dry run: full report, writes nothing
533+
os migrate files-to-references --apply # Convert, verify, record the flag (prompts)
534+
os migrate files-to-references --apply --yes --json # CI / scripts
535+
os migrate files-to-references --object product # Restrict to one object (repeatable)
536+
```
537+
538+
A `file` / `image` / `avatar` / `video` / `audio` field value is an opaque
539+
`sys_file` id that the platform owns. Values written before that (an inline
540+
`{url, name, …}` blob, or a URL naming this platform's own
541+
`…/storage/files/:id` resolver) are converted in place; **external URLs are
542+
reported, never re-hosted** — re-hosting third-party content is a licensing and
543+
privacy decision, and the right fix is usually to model the field as a `url`
544+
field instead.
545+
546+
The run then reconciles what records actually hold against what `sys_file`
547+
records as each file's owner. Zero blocking discrepancies is what records the
548+
flag — and that flag, not the version number, is what later enables features
549+
that act irreversibly on migrated data (reclaiming the bytes of released files;
550+
rejecting rather than warning about legacy media values). Never running it is
551+
safe: files are simply retained forever.
552+
553+
Exit status is `0` only when the self-check passes, so CI can gate on it.
554+
555+
<Callout type="warn">
556+
A dry run writes **nothing** — not the conversions, and not the flag either,
557+
even when the self-check would pass. `--apply` is the only writing mode. A
558+
later run that *fails* its self-check clears the flag's verified state, so a
559+
database that has drifted closes its own gate.
560+
</Callout>
561+
520562
### Scaffolding
521563

522564
| Command | Alias | Description |

‎content/docs/references/system/migration.mdx‎

Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,37 @@ description: Migration protocol schemas
55

66
{/* ⚠️ AUTO-GENERATED — DO NOT EDIT. Run build-docs.ts to regenerate. Hand-written docs live in the module folders under content/docs/. */}
77

8+
Migration protocol — the two kinds of migration, kept apart on purpose.
9+
10+
A **schema migration** (`ChangeSet` + its atomic operations) reshapes the
11+
12+
physical database to match metadata: add a field, change a type, create an
13+
14+
object, run SQL. It is derivable from the metadata and applied by
15+
16+
`os migrate plan` / `os migrate apply`.
17+
18+
A **data migration** rewrites the rows themselves, and whether it is done is
19+
20+
a property of ONE DEPLOYMENT's database rather than of the installed code
21+
22+
version — so it cannot be expressed as a ChangeSet, and its completion cannot
23+
24+
be inferred from a release. `DataMigrationFlag` is the per-deployment record
25+
26+
that one ran here and its self-check passed; consumers that would act
27+
28+
irreversibly on migrated data gate on the flag instead of the version.
29+
830
<Callout type="info">
931
**Source:** `packages/spec/src/system/migration.zod.ts`
1032
</Callout>
1133

1234
## TypeScript Usage
1335

1436
```typescript
15-
import { AddFieldOperation, ChangeSet, CreateObjectOperation, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependency, MigrationOperation, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
16-
import type { AddFieldOperation, ChangeSet, CreateObjectOperation, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependency, MigrationOperation, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
37+
import { AddFieldOperation, ChangeSet, CreateObjectOperation, DataMigrationFlag, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependency, MigrationOperation, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
38+
import type { AddFieldOperation, ChangeSet, CreateObjectOperation, DataMigrationFlag, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependency, MigrationOperation, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
1739

1840
// Validate data
1941
const result = AddFieldOperation.parse(data);
@@ -69,6 +91,25 @@ Create a new object
6991
| **object** | `{ name: string; label?: string; pluralLabel?: string; description?: string; … }` | ✅ | Full object definition to create |
7092

7193

94+
---
95+
96+
## DataMigrationFlag
97+
98+
Deployment-level record that a data migration ran here and its self-check passed — the evidence gate consumers read instead of the platform version
99+
100+
### Properties
101+
102+
| Property | Type | Required | Description |
103+
| :--- | :--- | :--- | :--- |
104+
| **id** | `string` | ✅ | Migration id (e.g. adr-0104-file-references) — one row per data migration |
105+
| **last_run_at** | `string` | ✅ | When this migration last completed a gated (apply-mode) run on this deployment |
106+
| **verified_at** | `string \| null` | optional | When the self-check last PASSED. Null/absent until it does — and cleared again by a later failing run, so a regression closes the gate |
107+
| **applied_at** | `string \| null` | optional | When the backfill last ran in apply mode (writes enabled) |
108+
| **blocking** | `integer` | ✅ | Blocking discrepancies reported by the last self-check. The gate requires 0 |
109+
| **advisory** | `integer` | optional | Advisory findings from the last run (external URLs, stale owners, …) — cost storage or need a modelling decision, never block the gate |
110+
| **details** | `string` | optional | JSON-encoded counts from the last run, for diagnostics |
111+
112+
72113
---
73114

74115
## DeleteObjectOperation

0 commit comments

Comments
 (0)