Commit f693f13
committed
feat(events): add hard cap to event write
This applies to both client side and server side events.
Two write points for events to clickhouse-batch route for client side events, and logEvent for server side.
Max batch size for client side events is 500.
We decrease item quantity after batch is uploaded for clickhouse. The alternative is to either do a check for eventsItem.quantity -batchSize <=0 before the upload and block upsert based on it (which would result in users not getting their full limit) or doing some weird partial batch upload.
At most, with this approach, we would give users ~500 extra events. This is 0.5% extra on free plan, so we judge that it is ok.
frontend-banners pop up on analytics when at 80%+ and when you hit your limit, with an option to upgrade your plan.
We deliberated using tryDecreaseQuantity instead to both check and debit items, but there is potential for debit to happen even when clickhouse is down with that approach. This would be worse for users, so we accept the slight race condition.
Manual testing: on local, changed events for free and team plan to smaller numbers. Checked analytics tables to verify no new events being added after limit reached.1 parent e527e6c commit f693f13
7 files changed
Lines changed: 291 additions & 2 deletions
File tree
- apps
- backend/src
- app/api/latest/analytics/events/batch
- lib
- dashboard/src/app/(main)/(protected)/projects/[projectId]/analytics
- queries
- replays
- tables
- e2e/tests/backend/endpoints/api/v1
Lines changed: 18 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
| 4 | + | |
3 | 5 | | |
4 | 6 | | |
5 | 7 | | |
| 8 | + | |
6 | 9 | | |
7 | 10 | | |
8 | 11 | | |
| |||
61 | 64 | | |
62 | 65 | | |
63 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
64 | 77 | | |
65 | 78 | | |
66 | 79 | | |
| |||
89 | 102 | | |
90 | 103 | | |
91 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
92 | 110 | | |
93 | 111 | | |
94 | 112 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
3 | 5 | | |
| 6 | + | |
4 | 7 | | |
5 | 8 | | |
6 | | - | |
| 9 | + | |
7 | 10 | | |
8 | 11 | | |
9 | 12 | | |
| |||
264 | 267 | | |
265 | 268 | | |
266 | 269 | | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
267 | 295 | | |
268 | 296 | | |
269 | 297 | | |
| |||
377 | 405 | | |
378 | 406 | | |
379 | 407 | | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
380 | 415 | | |
381 | 416 | | |
382 | 417 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
838 | 839 | | |
839 | 840 | | |
840 | 841 | | |
| 842 | + | |
841 | 843 | | |
842 | 844 | | |
843 | 845 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
1384 | 1385 | | |
1385 | 1386 | | |
1386 | 1387 | | |
| 1388 | + | |
1387 | 1389 | | |
1388 | 1390 | | |
1389 | 1391 | | |
| |||
Lines changed: 84 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
17 | 20 | | |
18 | 21 | | |
19 | 22 | | |
| 23 | + | |
20 | 24 | | |
21 | 25 | | |
22 | 26 | | |
| |||
316 | 320 | | |
317 | 321 | | |
318 | 322 | | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| |||
592 | 593 | | |
593 | 594 | | |
594 | 595 | | |
| 596 | + | |
595 | 597 | | |
596 | 598 | | |
597 | 599 | | |
| |||
0 commit comments