Skip to content

Commit 96ab8ff

Browse files
kvapsclaude
andcommitted
refactor(kv): replace ConfigMap-per-instance with KVEntry CRD per (instance, key)
The old implementation stored every key under one corev1.ConfigMap per LINSTOR instance. ConfigMap.data is etcd-validated to <=1 MiB, which is a silent scaling cliff: 5–10k entries (entirely realistic on prod cozystack clusters where linstor-csi pushes per-PVC bookkeeping into the "csi-volumes" instance) starts rejecting writes with "request is too large". This slice replaces that with a per-(instance, key) `KVEntry` CRD: - api/v1alpha1/kventry_types.go: cluster-scoped CRD with spec.instance, spec.key, spec.value (all free-form). status reserved. - pkg/store/k8s/kv_store.go: rewritten against the CRD. - metadata.name = "kv-" + sha256(instance + "\0" + key)[:16] (opaque, deterministic, k8s-safe). - Label blockstor.io/kv-instance-hash = sha256(instance)[:8] indexes server-side; the in-memory loop after re-checks Spec.Instance to defend against the once-in-2^64 hash collision. - SetKeys upserts per key (Get → Update or Create), so two writers on the same key get a clean conflict the caller can retry. No more monolithic ConfigMap rewrites. - DeleteNamespace iterates the instance's entries and drops anything matching ns or ns/* — same semantics as before, no '/' encoding now that keys live in spec, not in ConfigMap.data. - The old keyEncodeMarker hack ("__slash__") is gone; LINSTOR keys round-trip verbatim. - pkg/store/k8s/k8s_test.go: wipeAllKV switches to DeleteAllOf KVEntry. - The store.KeyValueStore interface and the REST contract (/v1/key-value-store/{instance}) are unchanged — golinstor and linstor-csi see the same wire shape. Shared storetest/RunKeyValueStore (6 cases incl. delete-namespace path) runs unchanged against both InMemory and the new CRD-backed store; both green. The DeleteNamespace test that originally forced the encoding hack now passes because it never crosses ConfigMap key validation in the first place. Per-key writes are also a fix to a less obvious bug: a future writer adding one key no longer races a concurrent writer adding another key (they used to collide on ConfigMap ResourceVersion); now they hit distinct CRDs. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
1 parent 81ec4da commit 96ab8ff

13 files changed

Lines changed: 607 additions & 88 deletions

PROJECT

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,4 +57,11 @@ resources:
5757
kind: Snapshot
5858
path: github.com/cozystack/blockstor/api/v1alpha1
5959
version: v1alpha1
60+
- api:
61+
crdVersion: v1
62+
domain: blockstor.io
63+
group: blockstor.io
64+
kind: KVEntry
65+
path: github.com/cozystack/blockstor/api/v1alpha1
66+
version: v1alpha1
6067
version: "3"

api/v1alpha1/kventry_types.go

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
/*
2+
Copyright 2026 Cozystack contributors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package v1alpha1
18+
19+
import (
20+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
21+
)
22+
23+
// KVEntrySpec is one entry in the LINSTOR-style key/value store.
24+
//
25+
// Upstream LINSTOR groups entries by an "instance" (e.g. "csi-volumes")
26+
// and addresses them by a free-form key (e.g. "Aux/csi-volume-annotations").
27+
// We store one CRD per (instance, key) pair so neither half of the key
28+
// runs into k8s naming restrictions, and so the per-write blast radius is
29+
// a single object instead of an entire ConfigMap.
30+
type KVEntrySpec struct {
31+
// instance is the LINSTOR KV instance name. Free-form string.
32+
// +required
33+
Instance string `json:"instance"`
34+
35+
// key is the free-form key inside the instance. May contain '/', '.',
36+
// uppercase, etc. — anything golinstor sends.
37+
// +required
38+
Key string `json:"key"`
39+
40+
// value is the opaque payload.
41+
// +optional
42+
Value string `json:"value,omitempty"`
43+
}
44+
45+
// KVEntryStatus is currently empty: KVEntries are pure config and have no
46+
// observed state.
47+
type KVEntryStatus struct {
48+
// conditions represent the current state of the KVEntry. Reserved for
49+
// future use; populated by reconcilers if any are introduced.
50+
// +listType=map
51+
// +listMapKey=type
52+
// +optional
53+
Conditions []metav1.Condition `json:"conditions,omitempty"`
54+
}
55+
56+
// +kubebuilder:object:root=true
57+
// +kubebuilder:subresource:status
58+
// +kubebuilder:resource:scope=Cluster
59+
60+
// KVEntry is the Schema for the kventries API
61+
type KVEntry struct {
62+
metav1.TypeMeta `json:",inline"`
63+
64+
// metadata is a standard object metadata
65+
// +optional
66+
metav1.ObjectMeta `json:"metadata,omitzero"`
67+
68+
// spec defines the desired state of KVEntry
69+
// +required
70+
Spec KVEntrySpec `json:"spec"`
71+
72+
// status defines the observed state of KVEntry
73+
// +optional
74+
Status KVEntryStatus `json:"status,omitzero"`
75+
}
76+
77+
// +kubebuilder:object:root=true
78+
79+
// KVEntryList contains a list of KVEntry
80+
type KVEntryList struct {
81+
metav1.TypeMeta `json:",inline"`
82+
metav1.ListMeta `json:"metadata,omitzero"`
83+
Items []KVEntry `json:"items"`
84+
}
85+
86+
func init() {
87+
SchemeBuilder.Register(&KVEntry{}, &KVEntryList{})
88+
}

api/v1alpha1/zz_generated.deepcopy.go

Lines changed: 96 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
---
2+
apiVersion: apiextensions.k8s.io/v1
3+
kind: CustomResourceDefinition
4+
metadata:
5+
annotations:
6+
controller-gen.kubebuilder.io/version: v0.20.1
7+
name: kventries.blockstor.io.blockstor.io
8+
spec:
9+
group: blockstor.io.blockstor.io
10+
names:
11+
kind: KVEntry
12+
listKind: KVEntryList
13+
plural: kventries
14+
singular: kventry
15+
scope: Cluster
16+
versions:
17+
- name: v1alpha1
18+
schema:
19+
openAPIV3Schema:
20+
description: KVEntry is the Schema for the kventries API
21+
properties:
22+
apiVersion:
23+
description: |-
24+
APIVersion defines the versioned schema of this representation of an object.
25+
Servers should convert recognized schemas to the latest internal value, and
26+
may reject unrecognized values.
27+
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
28+
type: string
29+
kind:
30+
description: |-
31+
Kind is a string value representing the REST resource this object represents.
32+
Servers may infer this from the endpoint the client submits requests to.
33+
Cannot be updated.
34+
In CamelCase.
35+
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
36+
type: string
37+
metadata:
38+
type: object
39+
spec:
40+
description: spec defines the desired state of KVEntry
41+
properties:
42+
instance:
43+
description: instance is the LINSTOR KV instance name. Free-form string.
44+
type: string
45+
key:
46+
description: |-
47+
key is the free-form key inside the instance. May contain '/', '.',
48+
uppercase, etc. — anything golinstor sends.
49+
type: string
50+
value:
51+
description: value is the opaque payload.
52+
type: string
53+
required:
54+
- instance
55+
- key
56+
type: object
57+
status:
58+
description: status defines the observed state of KVEntry
59+
properties:
60+
conditions:
61+
description: |-
62+
conditions represent the current state of the KVEntry. Reserved for
63+
future use; populated by reconcilers if any are introduced.
64+
items:
65+
description: Condition contains details for one aspect of the current
66+
state of this API Resource.
67+
properties:
68+
lastTransitionTime:
69+
description: |-
70+
lastTransitionTime is the last time the condition transitioned from one status to another.
71+
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
72+
format: date-time
73+
type: string
74+
message:
75+
description: |-
76+
message is a human readable message indicating details about the transition.
77+
This may be an empty string.
78+
maxLength: 32768
79+
type: string
80+
observedGeneration:
81+
description: |-
82+
observedGeneration represents the .metadata.generation that the condition was set based upon.
83+
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
84+
with respect to the current state of the instance.
85+
format: int64
86+
minimum: 0
87+
type: integer
88+
reason:
89+
description: |-
90+
reason contains a programmatic identifier indicating the reason for the condition's last transition.
91+
Producers of specific condition types may define expected values and meanings for this field,
92+
and whether the values are considered a guaranteed API.
93+
The value should be a CamelCase string.
94+
This field may not be empty.
95+
maxLength: 1024
96+
minLength: 1
97+
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
98+
type: string
99+
status:
100+
description: status of the condition, one of True, False, Unknown.
101+
enum:
102+
- "True"
103+
- "False"
104+
- Unknown
105+
type: string
106+
type:
107+
description: type of condition in CamelCase or in foo.example.com/CamelCase.
108+
maxLength: 316
109+
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
110+
type: string
111+
required:
112+
- lastTransitionTime
113+
- message
114+
- reason
115+
- status
116+
- type
117+
type: object
118+
type: array
119+
x-kubernetes-list-map-keys:
120+
- type
121+
x-kubernetes-list-type: map
122+
type: object
123+
required:
124+
- spec
125+
type: object
126+
served: true
127+
storage: true
128+
subresources:
129+
status: {}

config/crd/kustomization.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ resources:
88
- bases/blockstor.io.blockstor.io_resourcedefinitions.yaml
99
- bases/blockstor.io.blockstor.io_resources.yaml
1010
- bases/blockstor.io.blockstor.io_snapshots.yaml
11+
- bases/blockstor.io.blockstor.io_kventries.yaml
1112
# +kubebuilder:scaffold:crdkustomizeresource
1213

1314
patches:

config/rbac/kustomization.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ resources:
2222
# default, aiding admins in cluster management. Those roles are
2323
# not used by the blockstor itself. You can comment the following lines
2424
# if you do not want those helpers be installed with your Project.
25+
- kventry_admin_role.yaml
26+
- kventry_editor_role.yaml
27+
- kventry_viewer_role.yaml
2528
- snapshot_admin_role.yaml
2629
- snapshot_editor_role.yaml
2730
- snapshot_viewer_role.yaml
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# This rule is not used by the project blockstor itself.
2+
# It is provided to allow the cluster admin to help manage permissions for users.
3+
#
4+
# Grants full permissions ('*') over blockstor.io.blockstor.io.
5+
# This role is intended for users authorized to modify roles and bindings within the cluster,
6+
# enabling them to delegate specific permissions to other users or groups as needed.
7+
8+
apiVersion: rbac.authorization.k8s.io/v1
9+
kind: ClusterRole
10+
metadata:
11+
labels:
12+
app.kubernetes.io/name: blockstor
13+
app.kubernetes.io/managed-by: kustomize
14+
name: kventry-admin-role
15+
rules:
16+
- apiGroups:
17+
- blockstor.io.blockstor.io
18+
resources:
19+
- kventries
20+
verbs:
21+
- '*'
22+
- apiGroups:
23+
- blockstor.io.blockstor.io
24+
resources:
25+
- kventries/status
26+
verbs:
27+
- get

0 commit comments

Comments
 (0)