Skip to content

[catalog] convert mz_audit_events to BuiltinMaterializedView#37281

Draft
mtabebe wants to merge 2 commits into
MaterializeInc:mainfrom
mtabebe:ma/convert-builtins/mz-audit-log
Draft

[catalog] convert mz_audit_events to BuiltinMaterializedView#37281
mtabebe wants to merge 2 commits into
MaterializeInc:mainfrom
mtabebe:ma/convert-builtins/mz-audit-log

Conversation

@mtabebe

@mtabebe mtabebe commented Jun 24, 2026

Copy link
Copy Markdown
Contributor
Problem:
mz_catalog.mz_audit_events is a BuiltinTable. This requires coordinated
writes from a single environmentd process, conflicting with the
multi-envd goal. The table also carried a special-case bootstrap path
(`bootstrap_audit_log_table` in `coord.rs`) that diffed the in-memory
`AuditLogIterator` against the table contents on every boot.

Solution:
Convert mz_audit_events to a BuiltinMaterializedView backed by a query
over mz_internal.mz_catalog_raw, and delete the bootstrap path.

Each AuditLog row in the durable catalog has a `proto::AuditLogKey`
whose `event_type` and `object_type` are numeric `Serialize_repr` codes
that the MV maps back to the kebab-case strings produced by the prior
populator (`audit_log::{EventType,ObjectType}` Display). `details` goes
through a new SQL helper:
  mz_internal.parse_catalog_audit_log_details(jsonb) -> jsonb

Breaking change:
The `mz_audit_events.details` JSON shape changes for variants where the
proto and audit-log structs diverge beyond `#[serde(flatten)]` /
`StringWrapper`.
queries:

Testing:
- New `src/expr/src/scalar/func/impls/jsonb.rs#tests` Rust unit tests
  cover variant strip, flatten, recursive flatten, StringWrapper
  unwrap, null option, non-FullNameV1 negative cases, and error inputs.
- test/sqllogictest/mz_audit_events.slt new lockdown test (schema,
  OID stability, item type, NOT-NULL invariants, monotonicity, every
  reachable (event_type, object_type) pair, representative details
  extractions, NULL user, deterministic occurred_at, PUBLIC_SELECT).
- test/sqllogictest/singlereplica_audit_log.slt updated for new shape.
- test/sqllogictest/{information_schema_tables,oid,materialized_views,
  cluster,catalog_server_explain,mz_catalog_server_index_accounting,
  rbac_mcp_agent,rename,transform/predicate_pushdown,
  autogenerated/mz_catalog}.slt all pass.
- test/testdrive/{catalog,audit-log}.td updated for new shape.

Problem:
mz_internal.mz_comments is a BuiltinTable. This requires coordinated
writes from a single environmentd process, conflicting with the
multi-envd goal.

Solution:
Convert mz_comments to a BuiltinMaterializedView backed by a query over
mz_internal.mz_catalog_raw.

Each Comment row in the durable catalog has a `CommentObject` key
whose variant is the object type and whose inner value carries the id.

Testing:
- test/sqllogictest/mz_comments.slt lockdown test
- test/sqllogictest/comment.slt, oid.slt, information_schema_tables.slt,
  replacement-materialized-views.slt, mz_catalog_server_index_accounting.slt,
  catalog_server_explain.slt all pass.
@mtabebe mtabebe force-pushed the ma/convert-builtins/mz-audit-log branch from becf59e to 4b62e21 Compare June 24, 2026 20:23
Problem:
mz_catalog.mz_audit_events is a BuiltinTable. This requires coordinated
writes from a single environmentd process, conflicting with the
multi-envd goal. The table also carried a special-case bootstrap path
(`bootstrap_audit_log_table` in `coord.rs`) that diffed the in-memory
`AuditLogIterator` against the table contents on every boot.

Solution:
Convert mz_audit_events to a BuiltinMaterializedView backed by a query
over mz_internal.mz_catalog_raw, and delete the bootstrap path.

Each AuditLog row in the durable catalog has a `proto::AuditLogKey`
whose `event_type` and `object_type` are numeric `Serialize_repr` codes
that the MV maps back to the kebab-case strings produced by the prior
populator (`audit_log::{EventType,ObjectType}` Display). `details` goes
through a new SQL helper:
  mz_internal.parse_catalog_audit_log_details(jsonb) -> jsonb

Breaking change:
The `mz_audit_events.details` JSON shape changes for variants where the
proto and audit-log structs diverge beyond `#[serde(flatten)]` /
`StringWrapper`.
queries:

Testing:
- New `src/expr/src/scalar/func/impls/jsonb.rs#tests` Rust unit tests
  cover variant strip, flatten, recursive flatten, StringWrapper
  unwrap, null option, non-FullNameV1 negative cases, and error inputs.
- test/sqllogictest/mz_audit_events.slt new lockdown test (schema,
  OID stability, item type, NOT-NULL invariants, monotonicity, every
  reachable (event_type, object_type) pair, representative details
  extractions, NULL user, deterministic occurred_at, PUBLIC_SELECT).
- test/sqllogictest/singlereplica_audit_log.slt updated for new shape.
- test/sqllogictest/{information_schema_tables,oid,materialized_views,
  cluster,catalog_server_explain,mz_catalog_server_index_accounting,
  rbac_mcp_agent,rename,transform/predicate_pushdown,
  autogenerated/mz_catalog}.slt all pass.
- test/testdrive/{catalog,audit-log}.td updated for new shape.
@mtabebe mtabebe force-pushed the ma/convert-builtins/mz-audit-log branch from 4b62e21 to e66bff2 Compare June 25, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant