V1.2.9 (NFS-backed snapshot RESTORE hits the bug described here)

[ravi100k]

Hammerspace CSI: NFS-backed snapshot restore fails with NotFound desc = could not find source snapshots

Repo: hammer-space/csi-plugin
Affected: restore-from-snapshot on NFS-backed (fsType: nfs) StorageClasses
Plugin image tested: hammerspaceinc/csi-plugin:v1.2.9
Not affected: file-backed snapshot/restore (fsType: ext4 + mountBackingShareName) — that path works correctly once the separate RBAC issue is fixed (see companion report hs-csi-snapshot-rbac-bug.md).

TL;DR

NFS-backed snapshots are created correctly on the Anvil (POST /share-snapshots/snapshot-create/... returns 200, VolumeSnapshot reaches readyToUse: true). But when a new PVC is created with that VolumeSnapshot as its dataSource, the driver's CreateVolume RPC fails with NotFound desc = could not find source snapshots, even though the snapshot exists on the Anvil and is enumerable via the same driver's own GET to /share-snapshots/snapshot-list/<share>.

Root cause is a string-format mismatch in the driver's handling of the snapshot_id for share-snapshots: the snapshot_id encodes the snapshot name with literal double-quote characters around the timestamp segment, but the Anvil's snapshot-list response returns the snapshot names as plain JSON strings (no surrounding quote characters after JSON decode). The driver's lookup compares the quote-wrapped value against the unquoted list entries and never matches.

The restore PVC hangs in Pending indefinitely with the external-provisioner retrying every ~30 s; each retry produces the same NotFound.

Symptom

$ kubectl describe pvc restore -n hs-nfs-snapshot-e2e
...
Status:        Pending
StorageClass:  hs-nfs-snapshot-e2e
DataSource:
  APIGroup:  snapshot.storage.k8s.io
  Kind:      VolumeSnapshot
  Name:      src-snap
Events:
  Warning  ProvisioningFailed  (x9 over 4m27s)
    failed to provision volume with StorageClass "hs-nfs-snapshot-e2e":
    rpc error: code = NotFound desc = could not find source snapshots

The driver retries forever; user-visible behaviour is a PVC stuck in Pending.

Evidence

Snapshot creation succeeded — snapshot does exist on Anvil

Anvil received the create call and returned 200; the VolumeSnapshot CR reached readyToUse: true:

$ kubectl get volumesnapshot src-snap -n hs-nfs-snapshot-e2e
NAME       READYTOUSE   SOURCEPVC   SNAPSHOTCLASS         SNAPSHOTCONTENT                     AGE
src-snap   true         src         hs-nfs-snapshot-e2e   snapcontent-6fea8502-d66d-...       5s

From the driver's own log (the received response from the Anvil to its /share-snapshots/snapshot-list/<share> GET, verbatim):

"body": "[\"2026-06-29T19-23-50-0\",\"current\"]"

After JSON-decoding that body, the snapshots-on-this-share array is the two strings 2026-06-29T19-23-50-0 and current. Both are returned as plain JSON strings — i.e. their values do not contain literal " characters.

Restore CreateVolume — driver-side snapshot_id encoding

From the driver's gRPC log (verbatim), for the failing restore:

gRPCCall: {
  "Method": "/csi.v1.Controller/CreateVolume",
  "Request": {
    "name": "pvc-277dae43-9dab-4bf0-83d5-a6e5ed9a315a",
    "capacity_range": {"required_bytes": 1073741824},
    "volume_capabilities": [{"AccessType": {"Mount": {"fs_type": "nfs"}},
                             "access_mode": {"mode": 5}}],
    "parameters": {
      "fsType": "nfs",
      "exportOptions": "*,RW,false",
      "objectives": "keep-online",
      "volumeNameFormat": "nfs-%s",
      ...
    },
    "volume_content_source": {
      "Type": {
        "Snapshot": {
          "snapshot_id": "\"2026-06-29T19-23-50-0\"|/nfs-pvc-599e10c3-4a95-40e7-8fab-03f05408a12c"
        }
      }
    }
  },
  "Response": null,
  "Error": "rpc error: code = NotFound desc = could not find source snapshots"
}

After JSON-unescape, the actual snapshot_id string is:

"2026-06-29T19-23-50-0"|/nfs-pvc-599e10c3-4a95-40e7-8fab-03f05408a12c

The literal first and last characters of the left half of | are " characters. That's where the snapshot_id was constructed (either during CreateSnapshot response composition or when it was persisted into the VolumeSnapshotContent.status.snapshotHandle).

The mismatch

Side	Value	Source
Anvil's snapshot-list entry	2026-06-29T19-23-50-0	JSON string, no surrounding quotes after decode
Driver's snapshot_id (left of |)	"2026-06-29T19-23-50-0"	Literal quote characters present in the string

When the driver iterates over the snapshot-list response and compares each entry to the parsed snapshot_id, the comparison fails because of those quotes. Result: NotFound.

Why this only affects NFS-backed restore

The file-backed snapshot path uses different REST endpoints (/file-snapshots/list?filename-expression=...) and a different snapshot_id encoding. We've separately verified end-to-end file-backed snapshot+restore on the same cluster, plugin version, and Anvil — it works correctly. The NFS-restore code path is the only one that has the quote-mismatch defect.

Suggested fix

The bug is in the share-snapshots restore code path of the driver. Either:

1. Strip the surrounding quotes when parsing the snapshot_id before comparing against snapshot-list entries. Lowest-risk fix because it preserves compatibility with existing snapshots already created with the quoted format.
2. Stop wrapping the timestamp in quote characters when composing the snapshot_id in the CreateSnapshot response. Cleaner long-term, but pre-existing VolumeSnapshotContent objects with the old quoted format won't be restorable until they're recreated.

A defensive combination of both is probably best: emit the clean format going forward, and on lookup strip optional surrounding quotes for backward compatibility with snapshots created by older plugin versions.

Workaround

There is no production-acceptable workaround at the user level — the snapshot_id is stored verbatim in VolumeSnapshotContent.status.snapshotHandle and is generated entirely by the driver. Hand-editing the status field doesn't survive snapshot-controller reconciles.

Options for the meantime:

• Use file-backed StorageClasses (fsType: ext4 + mountBackingShareName) if snapshot/restore is required. That path is working end-to-end.
• For NFS-backed storage, create new PVCs by hand and re-populate from a known-good source (e.g., kubectl cp from the source pod to the destination pod) — skips the CSI restore path entirely.

Neither is acceptable for production snapshot-based workflows; both are stopgaps until the driver fix lands.

Reproduction

Prerequisites:

• Kubernetes ≥ 1.25 with snapshot CRDs + snapshot-controller (kubernetes-csi/external-snapshotter v8.x)
• Hammerspace CSI plugin installed
• The companion RBAC fix from hs-csi-snapshot-rbac-bug.md applied (otherwise snapshot CREATE doesn't work either, and you can't get to this bug)
• Hammerspace Anvil reachable; CSI credentials configured

Steps:

# 1. NFS-backed StorageClass + VolumeSnapshotClass
kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: hs-nfs-snap-test}
provisioner: com.hammerspace.csi
allowVolumeExpansion: true
parameters:
  fsType: "nfs"
  objectives: "keep-online"
  exportOptions: "*,RW,false"
  volumeNameFormat: "nfs-%s"
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata: {name: hs-nfs-snap-test}
driver: com.hammerspace.csi
deletionPolicy: Delete
EOF

# 2. Source PVC + a write pod to put a marker on it
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: src, namespace: default}
spec:
  accessModes: [ReadWriteMany]
  storageClassName: hs-nfs-snap-test
  resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: writer, namespace: default}
spec:
  restartPolicy: Never
  containers:
  - name: c
    image: busybox:1.36
    command: [sh, -c, "echo marker > /d/m.txt && sync"]
    volumeMounts: [{name: d, mountPath: /d}]
  volumes:
  - name: d
    persistentVolumeClaim: {claimName: src}
EOF
kubectl wait pod/writer --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s

# 3. Snapshot
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: {name: src-snap, namespace: default}
spec:
  volumeSnapshotClassName: hs-nfs-snap-test
  source: {persistentVolumeClaimName: src}
EOF
kubectl wait volumesnapshot/src-snap \
  --for=jsonpath='{.status.readyToUse}'=true --timeout=300s
# ^ this PASSES — snapshot is real, exists on Anvil

# 4. Restore PVC from the snapshot
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: restore, namespace: default}
spec:
  accessModes: [ReadWriteMany]
  storageClassName: hs-nfs-snap-test
  dataSource: {name: src-snap, kind: VolumeSnapshot,
               apiGroup: snapshot.storage.k8s.io}
  resources: {requests: {storage: 1Gi}}
EOF

# 5. Wait and observe: restore PVC stays Pending forever
kubectl describe pvc restore
# Events show:
#   ProvisioningFailed ... rpc error: code = NotFound desc = could not find source snapshots

To see the snapshot_id format mismatch directly:

# Driver's view (note the literal \" around the timestamp):
kubectl -n kube-system logs csi-provisioner-0 -c hs-csi-plugin-controller \
  | grep -A1 'CreateVolume' | grep snapshot_id

# Anvil's view (no quotes inside the JSON-decoded strings):
ssh serviceadmin@<anvil-ip> \
  'grep "snapshot-list/<source-share>" /var/trace/mgmt/mgmt-rest.log | tail -1'

Test environment for this report

Component	Version
Kubernetes	v1.36.2 (kubeadm, 1 master + 1 worker)
Container runtime	containerd 2.2.5
Plugin manifest	kubernetes-1.29/plugin.yaml (with the companion RBAC fix applied)
hammerspaceinc/csi-plugin	v1.2.9
Snapshot CRDs + controller	kubernetes-csi/external-snapshotter v8.5.0
Anvil	Hammerspace 5.3.0-111 (tf-anvil2.lab.hammer.space)

The bug is in the driver's share-snapshots code path and is independent of Kubernetes version, container runtime, or Anvil release; reproduces against any cluster where NFS-backed snapshot CREATE is working but RESTORE is attempted.

Relationship to the RBAC bug

This is a separate, independent bug from the missing-ClusterRole RBAC issue documented in hs-csi-snapshot-rbac-bug.md. The RBAC bug blocks snapshot CREATE on all backends (file-backed and NFS-backed) until the missing ClusterRole is supplied. After the RBAC fix:

• File-backed snapshot CREATE and RESTORE both work end-to-end.
• NFS-backed snapshot CREATE works.
• NFS-backed snapshot RESTORE hits the bug described here.

Both bugs surface in the same operational test (a Job that creates → snapshots → restores a PVC), so triage may benefit from referencing the two reports together.