Commit 8857dba
authored
clickhouse new syncs and verify-data (#1304)
<!--
Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md
-->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* External DB sync now covers teams, team members, permissions,
invitations, email outbox, session replays, refresh tokens, and
connected accounts.
* New sequence ID fields and automatic change-flagging added to many
record types to enable incremental sync.
* **Improvements**
* Added concurrent indexes, faster/parallelized sync pipelines,
verification tooling, and richer observability.
* Dashboard sequencer stats expanded and end-to-end sync tests
significantly extended.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->1 parent 5d9c7d9 commit 8857dba
38 files changed
Lines changed: 6074 additions & 264 deletions
File tree
- apps
- backend
- prisma
- migrations
- 20260316000000_add_team_team_member_sequence_columns
- 20260316000001_add_email_outbox_sequence_columns
- 20260316000002_add_session_replay_sequence_columns
- 20260317000000_add_team_permission_invitation_sequence_columns
- 20260317000001_add_project_permission_notification_preference_sequence_columns
- 20260318000000_add_sequence_id_to_refresh_tokens_and_oauth_accounts
- 20260318000001_add_sequence_indexes_concurrently
- scripts
- verify-data-integrity
- src
- app/api/latest
- auth
- password/update
- sessions
- current
- emails
- notification-preference
- outbox
- unsubscribe-link
- internal/external-db-sync
- sequencer
- status
- oauth-providers
- session-replays/batch
- team-member-profiles
- team-memberships
- teams
- users
- lib
- oauth
- route-handlers
- dashboard/src/app/(main)/(protected)/projects/[projectId]/external-db-sync
- e2e/tests/backend/endpoints/api/v1
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
55 | 56 | | |
56 | 57 | | |
57 | 58 | | |
| |||
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
Lines changed: 3 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
Lines changed: 3 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
Lines changed: 154 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
179 | 179 | | |
180 | 180 | | |
181 | 181 | | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
182 | 185 | | |
183 | 186 | | |
184 | 187 | | |
185 | 188 | | |
186 | 189 | | |
| 190 | + | |
| 191 | + | |
187 | 192 | | |
188 | 193 | | |
189 | 194 | | |
| |||
205 | 210 | | |
206 | 211 | | |
207 | 212 | | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
208 | 216 | | |
209 | 217 | | |
210 | 218 | | |
| |||
213 | 221 | | |
214 | 222 | | |
215 | 223 | | |
| 224 | + | |
| 225 | + | |
216 | 226 | | |
217 | 227 | | |
218 | 228 | | |
| |||
226 | 236 | | |
227 | 237 | | |
228 | 238 | | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
229 | 242 | | |
| 243 | + | |
| 244 | + | |
230 | 245 | | |
231 | 246 | | |
232 | 247 | | |
| |||
241 | 256 | | |
242 | 257 | | |
243 | 258 | | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
244 | 262 | | |
| 263 | + | |
| 264 | + | |
245 | 265 | | |
246 | 266 | | |
247 | 267 | | |
| |||
343 | 363 | | |
344 | 364 | | |
345 | 365 | | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
346 | 369 | | |
347 | 370 | | |
348 | 371 | | |
| 372 | + | |
| 373 | + | |
349 | 374 | | |
350 | 375 | | |
351 | 376 | | |
| |||
361 | 386 | | |
362 | 387 | | |
363 | 388 | | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
364 | 392 | | |
365 | 393 | | |
366 | 394 | | |
| |||
371 | 399 | | |
372 | 400 | | |
373 | 401 | | |
| 402 | + | |
| 403 | + | |
374 | 404 | | |
375 | 405 | | |
376 | 406 | | |
| |||
612 | 642 | | |
613 | 643 | | |
614 | 644 | | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
615 | 648 | | |
| 649 | + | |
| 650 | + | |
616 | 651 | | |
617 | 652 | | |
618 | 653 | | |
| |||
655 | 690 | | |
656 | 691 | | |
657 | 692 | | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
658 | 696 | | |
659 | 697 | | |
660 | 698 | | |
| 699 | + | |
661 | 700 | | |
662 | 701 | | |
663 | 702 | | |
| |||
1012 | 1051 | | |
1013 | 1052 | | |
1014 | 1053 | | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
1015 | 1057 | | |
1016 | 1058 | | |
1017 | 1059 | | |
1018 | 1060 | | |
1019 | 1061 | | |
1020 | 1062 | | |
1021 | 1063 | | |
| 1064 | + | |
| 1065 | + | |
1022 | 1066 | | |
1023 | 1067 | | |
1024 | 1068 | | |
| |||
1078 | 1122 | | |
1079 | 1123 | | |
1080 | 1124 | | |
| 1125 | + | |
| 1126 | + | |
| 1127 | + | |
1081 | 1128 | | |
1082 | 1129 | | |
| 1130 | + | |
| 1131 | + | |
1083 | 1132 | | |
1084 | 1133 | | |
1085 | 1134 | | |
| |||
0 commit comments