Skip to content

Commit 06dbd13

Browse files
authored
Feature/list actors scale w/ pagination (agent-substrate#156)
# Implement Pagination and Pipelining for ListActors ## Overview **Fixes:** agent-substrate#153 This PR introduces robust, memory-safe pagination to the `ListActors` operation and optimizes the Valkey retrieval layer. Previously, `ListActors` performed an unbounded parallel scatter-gather across all Valkey cluster shards, pulling all actors into memory at once. At scale (millions/billions of actors), this would cause OOM errors and severe latency. This update replaces the concurrent scatter-gather with a deterministic, sequential shard scanning approach, adhering to [AIP-158](https://google.aip.dev/158) pagination standards. ## Changes Included - **gRPC API (`ateapi.proto`)**: Added `page_size`, `page_token`, and `next_page_token` fields. - **Client (`kubectl-ate`)**: Updated to seamlessly loop through `next_page_token` values until all actors are fetched, presenting a continuous stream to the user without blowing up memory on a single request. - **API Handler (`controlapi`)**: Bounded requests with a max `page_size` limit of `1000`. - **Persistence (`ateredis`)**: - Implemented `listActorsPageToken` to safely serialize and encode cross-shard cursors. - Replaced `ForEachMaster` concurrent execution with deterministic sequential iteration. - Implemented Valkey Pipeline batching (`Pipelined`) for `GET` requests to eliminate N+1 network latency. - **Testing**: Added `TestListActors_Pagination` at both the integration level (`functional_test.go`) and data layer (`ateredis_test.go`). Refactored sorting to use `cmpopts.SortSlices`. ## Design Decisions During the design phase, several explicit trade-offs were made to balance performance with API simplicity: 1. **Opaque Tokens (AIP-158):** The `page_token` is strictly an opaque string at the `ateapi` layer. The client (`kubectl-ate`) is completely unaware that it contains base64-encoded JSON mapping to `{"shard", "cursor"}`. This keeps our API database-agnostic. 2. **Empty Pages & Client-Side Looping:** The persistence layer is kept simple and "dumb". If a shard scan returns 0 keys but the cursor is not `0`, we return an empty page. We rely on the client `kubectl-ate` to `for` loop until `next_page_token` is completely exhausted. Update: I have server side looping across shards until pagesize is met to limit round trips due to high shard count or low actor count on distributed shards. 3. **"Soft" Consistency (Missing/Duplicate Data):** `ListActors` is an infrequent operations command, not on the critical path. Taking a global cluster lock for iteration would severely degrade performance. We accept eventual consistency here: if actors migrate shards during an active scan, they might be duplicated or missed. 4. **Topology Flux Errors:** If a client passes a valid token pointing to a shard address that no longer exists (due to a cluster resharding mid-scan), the API simply returns an `Aborted` error. We do not attempt complex recovery; the client must just restart the `ListActors` command. 5. **Bounded Page Size:** Requests are currently capped at a maximum of `1000` to prevent single massive payloads. We can revisit/change the max later. NOTE: Actor filtering will be implemented later. ## Architecture Flow (Valkey) Note: If there are future plugin-db models they will need to handle the cusor/pages at their own implementation layer per the store.go interface. The system now seamlessly iterates across shards, handing off cursors safely between nodes: ```mermaid sequenceDiagram participant CLI as kubectl-ate participant API as ateapi participant Valkey as Valkey Cluster (Multiple Shards) CLI->>API: ListActors(page_size=1000, page_token="") Note over API: Request 1: Empty Token API->>API: Decode page_token (empty -> start fresh) API->>Valkey: Fetch all Master Nodes Valkey-->>API: [Master C, Master A, Master B] API->>API: Sort Masters alphabetically by IP/Port Note over API: Sorted: [Master A, Master B, Master C] API->>Valkey: SCAN (Master A, cursor=0, count=1000) Valkey-->>API: keys=[k1..k300], next_cursor=45 Note over API: OPTIMIZATION: Pipelining avoids N+1 problem API->>Valkey: PIPELINE GET (k1..k300) directly on Master A Valkey-->>API: values=[v1..v300] API->>API: Decode Protobufs API->>API: Encode page_token={"shard": "Master A", "cursor": 45} API-->>CLI: 300 Actors, next_page_token="base64(eyJzaGFyZ..." %% SECOND REQUEST CLI->>API: ListActors(page_size=1000, page_token="base64(...)") Note over API: Request 2: Resuming Mid-Shard API->>API: Decode page_token -> Shard="Master A", Cursor=45 API->>API: Fast-forward directly to Master A API->>Valkey: SCAN (Master A, cursor=45, count=1000) Valkey-->>API: keys=[k301..k400], next_cursor=0 API->>Valkey: PIPELINE GET (k301..k400) directly on Master A Valkey-->>API: values=[v301..v400] Note over API: Shard A is empty. Token points to Shard B! API->>API: Encode page_token={"shard": "Master B", "cursor": 0} API-->>CLI: 100 Actors, next_page_token="base64(eyJzaGFyZ..." %% THIRD REQUEST CLI->>API: ListActors(page_size=1000, page_token="base64(...)") Note over API: Request 3: Starting New Shard API->>API: Decode page_token -> Shard="Master B", Cursor=0 Note over API: Continues scanning Master B... ``` ## Testing: ``` > ./bin/kubectl-ate get actors NAMESPACE TEMPLATE ID STATUS ATEOM POD ATEOM IP VERSION ate-demo-counter counter 61b478cc-2e44-4ec7-ac6c-1af3f385de86 STATUS_SUSPENDED <none> 5 ``` ``` ~/go/bin/grpcurl -insecure -d '{"page_size": 2}' localhost:9090 ateapi.Control/ListActors { "actors": [ { "actorId": "test-counter-4", "version": "1", "actorTemplateNamespace": "ate-demo-counter", "actorTemplateName": "counter", "status": "STATUS_SUSPENDED" } ], "nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjE5OjYzNzkiLCJjdXJzb3IiOjI1MjU1fQ==" } ``` <details> ``` bash -c ' export NEXT_TOKEN="" while :; do echo "--- Fetching Page ---" # Fetch the data and store the response RESPONSE=$(~/go/bin/grpcurl -insecure -d "{\"page_size\": 2, \"page_token\": \"$NEXT_TOKEN\"}" localhost:9090 ateapi.Control/ListActors) # Print the response to the terminal echo "$RESPONSE" # Extract the token without needing jq NEXT_TOKEN=$(echo "$RESPONSE" | grep -o "\"nextPageToken\": \"[^\"]*\"" | cut -d"\"" -f4) # If no token is found, we reached the end if [ -z "$NEXT_TOKEN" ]; then echo "--- Reached the end! ---" break fi echo "" read -p "Press Enter to fetch the next page using the new token..." done ' ``` --- Fetching Page --- { "actors": [ { "actorId": "test-counter-4", "version": "1", "actorTemplateNamespace": "ate-demo-counter", "actorTemplateName": "counter", "status": "STATUS_SUSPENDED" } ], "nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjE5OjYzNzkiLCJjdXJzb3IiOjI1MjU1fQ==" } Press Enter to fetch the next page using the new token... --- Fetching Page --- { "nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjB9" } Press Enter to fetch the next page using the new token... --- Fetching Page --- { "actors": [ { "actorId": "test-counter-1", "version": "1", "actorTemplateNamespace": "ate-demo-counter", "actorTemplateName": "counter", "status": "STATUS_SUSPENDED" } ], "nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjQ1NjAwfQ==" } Press Enter to fetch the next page using the new token... --- Fetching Page --- { "actors": [ { "actorId": "test-counter-5", "version": "1", "actorTemplateNamespace": "ate-demo-counter", "actorTemplateName": "counter", "status": "STATUS_SUSPENDED" } ], "nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjYyMDg2fQ==" } Press Enter to fetch the next page using the new token... --- Fetching Page --- { "nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjI1OjYzNzkiLCJjdXJzb3IiOjB9" } Press Enter to fetch the next page using the new token... --- Fetching Page --- { "actors": [ { "actorId": "test-counter-2", "version": "1", "actorTemplateNamespace": "ate-demo-counter", "actorTemplateName": "counter", "status": "STATUS_SUSPENDED" } ], "nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjI1OjYzNzkiLCJjdXJzb3IiOjE2OTkzfQ==" } Press Enter to fetch the next page using the new token... --- Fetching Page --- { "actors": [ { "actorId": "test-counter-3", "version": "1", "actorTemplateNamespace": "ate-demo-counter", "actorTemplateName": "counter", "status": "STATUS_SUSPENDED" }, { "actorId": "61b478cc-2e44-4ec7-ac6c-1af3f385de86", "version": "5", "actorTemplateNamespace": "ate-demo-counter", "actorTemplateName": "counter", "status": "STATUS_SUSPENDED", "lastSnapshot": "gs://ate-snapshots/ate-demo-counter/61b478cc-2e44-4ec7-ac6c-1af3f385de86/2026-06-03T03:17:34Z-UMYVUZOWW6LDPS5X5FSG3KC4VV" } ] } --- Reached the end! --- </details>
1 parent f46c67f commit 06dbd13

9 files changed

Lines changed: 401 additions & 52 deletions

File tree

cmd/ateapi/internal/controlapi/functional_test.go

Lines changed: 88 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ import (
2020
"net"
2121
"os"
2222
"os/exec"
23-
"sort"
2423
"strings"
2524
"sync"
2625
"testing"
@@ -36,6 +35,7 @@ import (
3635
"github.com/agent-substrate/substrate/pkg/proto/ateapipb"
3736
"github.com/alicebob/miniredis/v2"
3837
"github.com/google/go-cmp/cmp"
38+
"github.com/google/go-cmp/cmp/cmpopts"
3939
"github.com/redis/go-redis/v9"
4040
"google.golang.org/grpc"
4141
"google.golang.org/grpc/codes"
@@ -619,23 +619,101 @@ func TestListActors(t *testing.T) {
619619
t.Fatalf("expected 2 actors, got %d", len(listResp.Actors))
620620
}
621621

622-
sort.Slice(listResp.Actors, func(i, j int) bool {
623-
return listResp.Actors[i].ActorId < listResp.Actors[j].ActorId
624-
})
625-
626622
want := []*ateapipb.Actor{
627623
resp1.GetActor(),
628624
resp2.GetActor(),
629625
}
630-
sort.Slice(want, func(i, j int) bool {
631-
return want[i].ActorId < want[j].ActorId
632-
})
633626

634-
if diff := cmp.Diff(want, listResp.Actors, protocmp.Transform()); diff != "" {
627+
opts := []cmp.Option{
628+
protocmp.Transform(),
629+
cmpopts.SortSlices(func(a, b *ateapipb.Actor) bool {
630+
return a.ActorId < b.ActorId
631+
}),
632+
}
633+
634+
if diff := cmp.Diff(want, listResp.Actors, opts...); diff != "" {
635635
t.Errorf("ListActors response mismatch (-want +got):\n%s", diff)
636636
}
637637
}
638638

639+
// TestListActors_Pagination tests that ListActors correctly paginates results.
640+
func TestListActors_Pagination(t *testing.T) {
641+
ns := namespaceForTest("ns-list-actors-pagination")
642+
tc := setupTest(t, ns)
643+
defer tc.cleanup()
644+
645+
createTemplate(t, tc, ns)
646+
647+
var want []*ateapipb.Actor
648+
for i := 0; i < 5; i++ {
649+
resp, err := tc.client.CreateActor(context.Background(), &ateapipb.CreateActorRequest{
650+
ActorTemplateNamespace: ns,
651+
ActorTemplateName: "tmpl1",
652+
ActorId: fmt.Sprintf("id%d", i),
653+
})
654+
if err != nil {
655+
t.Fatalf("CreateActor %d failed: %v", i, err)
656+
}
657+
want = append(want, resp.GetActor())
658+
}
659+
660+
var allActors []*ateapipb.Actor
661+
pageToken := ""
662+
663+
for {
664+
listResp, err := tc.client.ListActors(context.Background(), &ateapipb.ListActorsRequest{
665+
PageSize: 2,
666+
PageToken: pageToken,
667+
})
668+
if err != nil {
669+
t.Fatalf("ListActors failed: %v", err)
670+
}
671+
672+
allActors = append(allActors, listResp.Actors...)
673+
pageToken = listResp.GetNextPageToken()
674+
if pageToken == "" {
675+
break
676+
}
677+
}
678+
679+
if len(allActors) != 5 {
680+
t.Fatalf("expected 5 actors total, got %d", len(allActors))
681+
}
682+
683+
opts := []cmp.Option{
684+
protocmp.Transform(),
685+
cmpopts.SortSlices(func(a, b *ateapipb.Actor) bool {
686+
return a.ActorId < b.ActorId
687+
}),
688+
}
689+
690+
if diff := cmp.Diff(want, allActors, opts...); diff != "" {
691+
t.Errorf("ListActors pagination response mismatch (-want +got):\n%s", diff)
692+
}
693+
}
694+
695+
func TestListActors_PageSizeValidation(t *testing.T) {
696+
ns := namespaceForTest("ns-list-actors-validation")
697+
tc := setupTest(t, ns)
698+
defer tc.cleanup()
699+
700+
// 1. Negative page size
701+
_, err := tc.client.ListActors(context.Background(), &ateapipb.ListActorsRequest{
702+
PageSize: -1,
703+
})
704+
if status.Code(err) != codes.InvalidArgument {
705+
t.Errorf("expected InvalidArgument error for negative page_size, got: %v", err)
706+
}
707+
708+
// 2. Page size exceeding maxPageSize (1000)
709+
_, err = tc.client.ListActors(context.Background(), &ateapipb.ListActorsRequest{
710+
PageSize: 1001,
711+
})
712+
if status.Code(err) != codes.InvalidArgument {
713+
t.Errorf("expected InvalidArgument error for page_size > 1000, got: %v", err)
714+
}
715+
}
716+
639717
// TestListWorkers tests that workers mirrored to Redis are listed.
640718
// Workflow:
641719
// 1. Creates a mock worker Pod in Kubernetes.
@@ -1203,7 +1281,7 @@ func TestSuspendActor_DanglingWorker(t *testing.T) {
12031281
deleteWorkerPod(t, tc, ns, "worker-1")
12041282

12051283
// 3. Call SuspendActor -> Should succeed (our fix skips missing pod execution)
1206-
actors, _ := tc.persistence.ListActors(context.Background())
1284+
actors, _, _ := tc.persistence.ListActors(context.Background(), maxPageSize, "")
12071285
t.Logf("Actors in Redis before Suspend: %d", len(actors))
12081286
for _, a := range actors {
12091287
t.Logf(" Actor: %s/%s/%s", a.GetActorTemplateNamespace(), a.GetActorTemplateName(), a.GetActorId())

cmd/ateapi/internal/controlapi/list_actors.go

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,21 +19,38 @@ import (
1919
"fmt"
2020

2121
"github.com/agent-substrate/substrate/pkg/proto/ateapipb"
22+
"google.golang.org/grpc/codes"
23+
"google.golang.org/grpc/status"
2224
)
2325

26+
const maxPageSize = 1000
27+
2428
func (s *Service) ListActors(ctx context.Context, req *ateapipb.ListActorsRequest) (*ateapipb.ListActorsResponse, error) {
2529
if err := validateListActorsRequest(req); err != nil {
26-
return nil, err
30+
return nil, status.Error(codes.InvalidArgument, err.Error())
31+
}
32+
pageSize := req.GetPageSize()
33+
if pageSize == 0 {
34+
pageSize = maxPageSize
2735
}
28-
actors, err := s.persistence.ListActors(ctx)
36+
37+
actors, nextToken, err := s.persistence.ListActors(ctx, pageSize, req.GetPageToken())
2938
if err != nil {
3039
return nil, fmt.Errorf("while listing actors in db: %w", err)
3140
}
3241
return &ateapipb.ListActorsResponse{
33-
Actors: actors,
42+
Actors: actors,
43+
NextPageToken: nextToken,
3444
}, nil
3545
}
3646

37-
func validateListActorsRequest(_ *ateapipb.ListActorsRequest) error {
47+
func validateListActorsRequest(req *ateapipb.ListActorsRequest) error {
48+
pageSize := req.GetPageSize()
49+
if pageSize < 0 {
50+
return fmt.Errorf("page_size cannot be negative")
51+
}
52+
if pageSize > maxPageSize {
53+
return fmt.Errorf("page_size %d exceeds maximum page size %d", pageSize, maxPageSize)
54+
}
3855
return nil
3956
}

cmd/ateapi/internal/store/ateredis/ateredis.go

Lines changed: 159 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,13 @@ package ateredis
4141

4242
import (
4343
"context"
44+
"crypto/sha256"
45+
"encoding/base64"
46+
"encoding/hex"
47+
"encoding/json"
4448
"errors"
4549
"fmt"
50+
"sort"
4651
"strings"
4752
"sync"
4853
"time"
@@ -393,40 +398,172 @@ func (s *Persistence) ListWorkers(ctx context.Context) ([]*ateapipb.Worker, erro
393398
return result, nil
394399
}
395400

396-
func (s *Persistence) ListActors(ctx context.Context) ([]*ateapipb.Actor, error) {
401+
type listActorsPageToken struct {
402+
ShardHash string `json:"shard_hash"`
403+
Cursor uint64 `json:"cursor"`
404+
}
405+
406+
func encodePageToken(token listActorsPageToken) string {
407+
b, _ := json.Marshal(token)
408+
return base64.StdEncoding.EncodeToString(b)
409+
}
410+
411+
func decodePageToken(tokenStr string) (listActorsPageToken, error) {
412+
var token listActorsPageToken
413+
if tokenStr == "" {
414+
return token, nil
415+
}
416+
b, err := base64.StdEncoding.DecodeString(tokenStr)
417+
if err != nil {
418+
return token, err
419+
}
420+
err = json.Unmarshal(b, &token)
421+
return token, err
422+
}
423+
424+
func hashShardAddr(addr string) string {
425+
h := sha256.Sum256([]byte(addr))
426+
return hex.EncodeToString(h[:])
427+
}
428+
429+
func (s *Persistence) ListActors(ctx context.Context, pageSize int32, pageTokenStr string) ([]*ateapipb.Actor, string, error) {
430+
token, err := decodePageToken(pageTokenStr)
431+
if err != nil {
432+
return nil, "", fmt.Errorf("invalid page token: %w", err)
433+
}
434+
435+
masters, err := s.getSortedMasters(ctx)
436+
if err != nil {
437+
return nil, "", err
438+
}
439+
440+
startIndex, err := findStartingShard(masters, token.ShardHash)
441+
if err != nil {
442+
return nil, "", err
443+
}
444+
397445
var result []*ateapipb.Actor
398-
var mu sync.Mutex
446+
var nextToken string
447+
stop := false
399448

400-
err := s.rdb.ForEachMaster(ctx, func(ctx context.Context, master *redis.Client) error {
401-
iter := master.Scan(ctx, 0, "actor:*", 0).Iterator()
402-
for iter.Next(ctx) {
403-
actorKey := iter.Val()
404-
parts := strings.Split(actorKey, ":")
405-
if len(parts) != 2 {
406-
return fmt.Errorf("bad key format %q", actorKey)
449+
for i := startIndex; i < len(masters) && !stop; i++ {
450+
master := masters[i]
451+
shardAddr := master.Options().Addr
452+
cursor := uint64(0)
453+
if i == startIndex && token.ShardHash != "" {
454+
cursor = token.Cursor
455+
}
456+
457+
for {
458+
remaining := int(pageSize) - len(result)
459+
if remaining <= 0 {
460+
if cursor != 0 {
461+
nextToken = encodePageToken(listActorsPageToken{
462+
ShardHash: hashShardAddr(shardAddr),
463+
Cursor: cursor,
464+
})
465+
} else if i+1 < len(masters) {
466+
nextToken = encodePageToken(listActorsPageToken{
467+
ShardHash: hashShardAddr(masters[i+1].Options().Addr),
468+
Cursor: 0,
469+
})
470+
} else {
471+
nextToken = ""
472+
}
473+
stop = true
474+
break
407475
}
408476

409-
getCmd := master.Get(ctx, actorKey)
410-
if getCmd.Err() != nil {
411-
return fmt.Errorf("while getting actor %q: %w", actorKey, getCmd.Err())
477+
var keys []string
478+
keys, cursor, err = master.Scan(ctx, cursor, "actor:*", int64(remaining)).Result()
479+
if err != nil {
480+
return nil, "", fmt.Errorf("while scanning shard %s: %w", shardAddr, err)
412481
}
413482

414-
actor := &ateapipb.Actor{}
415-
if err := protojson.Unmarshal([]byte(getCmd.Val()), actor); err != nil {
416-
return fmt.Errorf("in protojson.Unmarshal: %w", err)
483+
if len(keys) > 0 {
484+
actors, err := s.fetchActors(ctx, master, keys)
485+
if err != nil {
486+
return nil, "", err
487+
}
488+
result = append(result, actors...)
417489
}
418490

419-
mu.Lock()
420-
result = append(result, actor)
421-
mu.Unlock()
491+
if cursor == 0 {
492+
if i+1 < len(masters) {
493+
nextToken = encodePageToken(listActorsPageToken{
494+
ShardHash: hashShardAddr(masters[i+1].Options().Addr),
495+
Cursor: 0,
496+
})
497+
} else {
498+
nextToken = ""
499+
}
500+
break
501+
}
422502
}
423-
return iter.Err()
424-
})
503+
}
504+
505+
return result, nextToken, nil
506+
}
425507

508+
func (s *Persistence) getSortedMasters(ctx context.Context) ([]*redis.Client, error) {
509+
var masters []*redis.Client
510+
err := s.rdb.ForEachMaster(ctx, func(ctx context.Context, master *redis.Client) error {
511+
masters = append(masters, master)
512+
return nil
513+
})
426514
if err != nil {
427-
return nil, fmt.Errorf("while iterating all redis master: %w", err)
515+
return nil, fmt.Errorf("while listing redis masters: %w", err)
428516
}
429-
return result, nil
517+
518+
sort.Slice(masters, func(i, j int) bool {
519+
return masters[i].Options().Addr < masters[j].Options().Addr
520+
})
521+
return masters, nil
522+
}
523+
524+
func findStartingShard(masters []*redis.Client, shardHash string) (int, error) {
525+
if shardHash == "" {
526+
return 0, nil
527+
}
528+
for i, m := range masters {
529+
if hashShardAddr(m.Options().Addr) == shardHash {
530+
return i, nil
531+
}
532+
}
533+
return 0, fmt.Errorf("topology changed: shard with hash %s not found (aborted)", shardHash)
534+
}
535+
536+
func (s *Persistence) fetchActors(ctx context.Context, master *redis.Client, keys []string) ([]*ateapipb.Actor, error) {
537+
cmds, err := master.Pipelined(ctx, func(pipe redis.Pipeliner) error {
538+
for _, key := range keys {
539+
pipe.Get(ctx, key)
540+
}
541+
return nil
542+
})
543+
if err != nil && !errors.Is(err, redis.Nil) {
544+
return nil, fmt.Errorf("while fetching keys in shard %s: %w", master.Options().Addr, err)
545+
}
546+
547+
var actors []*ateapipb.Actor
548+
for _, cmd := range cmds {
549+
getCmd, ok := cmd.(*redis.StringCmd)
550+
if !ok {
551+
continue
552+
}
553+
if getCmd.Err() != nil {
554+
if errors.Is(getCmd.Err(), redis.Nil) {
555+
continue
556+
}
557+
return nil, fmt.Errorf("while getting actor: %w", getCmd.Err())
558+
}
559+
560+
actor := &ateapipb.Actor{}
561+
if err := protojson.Unmarshal([]byte(getCmd.Val()), actor); err != nil {
562+
return nil, fmt.Errorf("in protojson.Unmarshal: %w", err)
563+
}
564+
actors = append(actors, actor)
565+
}
566+
return actors, nil
430567
}
431568

432569
func (s *Persistence) AcquireLock(ctx context.Context, key string, value string, ttl time.Duration) (bool, error) {

0 commit comments

Comments
 (0)