Commit 06dbd13
authored
Feature/list actors scale w/ pagination (agent-substrate#156)
# Implement Pagination and Pipelining for ListActors
## Overview
**Fixes:** agent-substrate#153
This PR introduces robust, memory-safe pagination to the `ListActors`
operation and optimizes the Valkey retrieval layer.
Previously, `ListActors` performed an unbounded parallel scatter-gather
across all Valkey cluster shards, pulling all actors into memory at
once. At scale (millions/billions of actors), this would cause OOM
errors and severe latency.
This update replaces the concurrent scatter-gather with a deterministic,
sequential shard scanning approach, adhering to
[AIP-158](https://google.aip.dev/158) pagination standards.
## Changes Included
- **gRPC API (`ateapi.proto`)**: Added `page_size`, `page_token`, and
`next_page_token` fields.
- **Client (`kubectl-ate`)**: Updated to seamlessly loop through
`next_page_token` values until all actors are fetched, presenting a
continuous stream to the user without blowing up memory on a single
request.
- **API Handler (`controlapi`)**: Bounded requests with a max
`page_size` limit of `1000`.
- **Persistence (`ateredis`)**:
- Implemented `listActorsPageToken` to safely serialize and encode
cross-shard cursors.
- Replaced `ForEachMaster` concurrent execution with deterministic
sequential iteration.
- Implemented Valkey Pipeline batching (`Pipelined`) for `GET` requests
to eliminate N+1 network latency.
- **Testing**: Added `TestListActors_Pagination` at both the integration
level (`functional_test.go`) and data layer (`ateredis_test.go`).
Refactored sorting to use `cmpopts.SortSlices`.
## Design Decisions
During the design phase, several explicit trade-offs were made to
balance performance with API simplicity:
1. **Opaque Tokens (AIP-158):** The `page_token` is strictly an opaque
string at the `ateapi` layer. The client (`kubectl-ate`) is completely
unaware that it contains base64-encoded JSON mapping to `{"shard",
"cursor"}`. This keeps our API database-agnostic.
2. **Empty Pages & Client-Side Looping:** The persistence layer is kept
simple and "dumb". If a shard scan returns 0 keys but the cursor is not
`0`, we return an empty page. We rely on the client `kubectl-ate` to
`for` loop until `next_page_token` is completely exhausted. Update: I
have server side looping across shards until pagesize is met to limit
round trips due to high shard count or low actor count on distributed
shards.
3. **"Soft" Consistency (Missing/Duplicate Data):** `ListActors` is an
infrequent operations command, not on the critical path. Taking a global
cluster lock for iteration would severely degrade performance. We accept
eventual consistency here: if actors migrate shards during an active
scan, they might be duplicated or missed.
4. **Topology Flux Errors:** If a client passes a valid token pointing
to a shard address that no longer exists (due to a cluster resharding
mid-scan), the API simply returns an `Aborted` error. We do not attempt
complex recovery; the client must just restart the `ListActors` command.
5. **Bounded Page Size:** Requests are currently capped at a maximum of
`1000` to prevent single massive payloads. We can revisit/change the max
later.
NOTE: Actor filtering will be implemented later.
## Architecture Flow (Valkey)
Note: If there are future plugin-db models they will need to handle the
cusor/pages at their own implementation layer per the store.go
interface.
The system now seamlessly iterates across shards, handing off cursors
safely between nodes:
```mermaid
sequenceDiagram
participant CLI as kubectl-ate
participant API as ateapi
participant Valkey as Valkey Cluster (Multiple Shards)
CLI->>API: ListActors(page_size=1000, page_token="")
Note over API: Request 1: Empty Token
API->>API: Decode page_token (empty -> start fresh)
API->>Valkey: Fetch all Master Nodes
Valkey-->>API: [Master C, Master A, Master B]
API->>API: Sort Masters alphabetically by IP/Port
Note over API: Sorted: [Master A, Master B, Master C]
API->>Valkey: SCAN (Master A, cursor=0, count=1000)
Valkey-->>API: keys=[k1..k300], next_cursor=45
Note over API: OPTIMIZATION: Pipelining avoids N+1 problem
API->>Valkey: PIPELINE GET (k1..k300) directly on Master A
Valkey-->>API: values=[v1..v300]
API->>API: Decode Protobufs
API->>API: Encode page_token={"shard": "Master A", "cursor": 45}
API-->>CLI: 300 Actors, next_page_token="base64(eyJzaGFyZ..."
%% SECOND REQUEST
CLI->>API: ListActors(page_size=1000, page_token="base64(...)")
Note over API: Request 2: Resuming Mid-Shard
API->>API: Decode page_token -> Shard="Master A", Cursor=45
API->>API: Fast-forward directly to Master A
API->>Valkey: SCAN (Master A, cursor=45, count=1000)
Valkey-->>API: keys=[k301..k400], next_cursor=0
API->>Valkey: PIPELINE GET (k301..k400) directly on Master A
Valkey-->>API: values=[v301..v400]
Note over API: Shard A is empty. Token points to Shard B!
API->>API: Encode page_token={"shard": "Master B", "cursor": 0}
API-->>CLI: 100 Actors, next_page_token="base64(eyJzaGFyZ..."
%% THIRD REQUEST
CLI->>API: ListActors(page_size=1000, page_token="base64(...)")
Note over API: Request 3: Starting New Shard
API->>API: Decode page_token -> Shard="Master B", Cursor=0
Note over API: Continues scanning Master B...
```
## Testing:
```
> ./bin/kubectl-ate get actors
NAMESPACE TEMPLATE ID STATUS ATEOM POD ATEOM IP VERSION
ate-demo-counter counter 61b478cc-2e44-4ec7-ac6c-1af3f385de86 STATUS_SUSPENDED <none> 5
```
```
~/go/bin/grpcurl -insecure -d '{"page_size": 2}' localhost:9090 ateapi.Control/ListActors
{
"actors": [
{
"actorId": "test-counter-4",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjE5OjYzNzkiLCJjdXJzb3IiOjI1MjU1fQ=="
}
```
<details>
```
bash -c '
export NEXT_TOKEN=""
while :; do
echo "--- Fetching Page ---"
# Fetch the data and store the response
RESPONSE=$(~/go/bin/grpcurl -insecure -d "{\"page_size\": 2, \"page_token\": \"$NEXT_TOKEN\"}" localhost:9090 ateapi.Control/ListActors)
# Print the response to the terminal
echo "$RESPONSE"
# Extract the token without needing jq
NEXT_TOKEN=$(echo "$RESPONSE" | grep -o "\"nextPageToken\": \"[^\"]*\"" | cut -d"\"" -f4)
# If no token is found, we reached the end
if [ -z "$NEXT_TOKEN" ]; then
echo "--- Reached the end! ---"
break
fi
echo ""
read -p "Press Enter to fetch the next page using the new token..."
done
'
```
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-4",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjE5OjYzNzkiLCJjdXJzb3IiOjI1MjU1fQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjB9"
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-1",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjQ1NjAwfQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-5",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjYyMDg2fQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjI1OjYzNzkiLCJjdXJzb3IiOjB9"
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-2",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjI1OjYzNzkiLCJjdXJzb3IiOjE2OTkzfQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-3",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
},
{
"actorId": "61b478cc-2e44-4ec7-ac6c-1af3f385de86",
"version": "5",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED",
"lastSnapshot":
"gs://ate-snapshots/ate-demo-counter/61b478cc-2e44-4ec7-ac6c-1af3f385de86/2026-06-03T03:17:34Z-UMYVUZOWW6LDPS5X5FSG3KC4VV"
}
]
}
--- Reached the end! ---
</details>1 parent f46c67f commit 06dbd13
9 files changed
Lines changed: 401 additions & 52 deletions
File tree
- cmd
- ateapi/internal
- controlapi
- store
- ateredis
- atenet/internal/app/router
- kubectl-ate/internal/cmd
- pkg/proto/ateapipb
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
24 | 23 | | |
25 | 24 | | |
26 | 25 | | |
| |||
36 | 35 | | |
37 | 36 | | |
38 | 37 | | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| |||
619 | 619 | | |
620 | 620 | | |
621 | 621 | | |
622 | | - | |
623 | | - | |
624 | | - | |
625 | | - | |
626 | 622 | | |
627 | 623 | | |
628 | 624 | | |
629 | 625 | | |
630 | | - | |
631 | | - | |
632 | | - | |
633 | 626 | | |
634 | | - | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
635 | 635 | | |
636 | 636 | | |
637 | 637 | | |
638 | 638 | | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
639 | 717 | | |
640 | 718 | | |
641 | 719 | | |
| |||
1203 | 1281 | | |
1204 | 1282 | | |
1205 | 1283 | | |
1206 | | - | |
| 1284 | + | |
1207 | 1285 | | |
1208 | 1286 | | |
1209 | 1287 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
| 26 | + | |
| 27 | + | |
24 | 28 | | |
25 | 29 | | |
26 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
27 | 35 | | |
28 | | - | |
| 36 | + | |
| 37 | + | |
29 | 38 | | |
30 | 39 | | |
31 | 40 | | |
32 | 41 | | |
33 | | - | |
| 42 | + | |
| 43 | + | |
34 | 44 | | |
35 | 45 | | |
36 | 46 | | |
37 | | - | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
38 | 55 | | |
39 | 56 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
44 | 48 | | |
45 | 49 | | |
| 50 | + | |
46 | 51 | | |
47 | 52 | | |
48 | 53 | | |
| |||
393 | 398 | | |
394 | 399 | | |
395 | 400 | | |
396 | | - | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
397 | 445 | | |
398 | | - | |
| 446 | + | |
| 447 | + | |
399 | 448 | | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
406 | | - | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
407 | 475 | | |
408 | 476 | | |
409 | | - | |
410 | | - | |
411 | | - | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
412 | 481 | | |
413 | 482 | | |
414 | | - | |
415 | | - | |
416 | | - | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
417 | 489 | | |
418 | 490 | | |
419 | | - | |
420 | | - | |
421 | | - | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
422 | 502 | | |
423 | | - | |
424 | | - | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
425 | 507 | | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
426 | 514 | | |
427 | | - | |
| 515 | + | |
428 | 516 | | |
429 | | - | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
430 | 567 | | |
431 | 568 | | |
432 | 569 | | |
| |||
0 commit comments