Skip to content

PBM-1689 Restore stuck on chunks download - #1340

Merged
jcechace merged 2 commits into
percona:devfrom
jcechace:PBM-1689-stuck-on-chunks-download
Jun 26, 2026
Merged

PBM-1689 Restore stuck on chunks download#1340
jcechace merged 2 commits into
percona:devfrom
jcechace:PBM-1689-stuck-on-chunks-download

Conversation

@jcechace

@jcechace jcechace commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

ticket: https://perconadev.atlassian.net/browse/PBM-1689

before:

pbm restore 2026-06-25T11:27:22Z -w -y

Starting restore 2026-06-25T11:47:05.333362065Z from '2026-06-25T11:27:22Z'..........Started logical restore.
Waiting to finish.......................................................

and agent is stuck at

2026-06-25T11:47:53Z W [rs1/rs101:27017] [restore/2026-06-25T11:47:05.333362065Z] errGetObj Err: operation error S3: GetObject, https response error StatusCode: 403, RequestID: 3ZXPW115VWPJSY92, HostID: qEzQvClNLGWp+XJcLOAR9JWL/hDs+8jW67B49Jk4vpZesh9u14hpl6ZNxsLxo6VWQ0Eylte3imIf25bkLMPl/qclG1qG4eKS, api error AccessDenied: User: arn:aws:iam::...:user/pbm-jakub is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:eu-central-1:...with an explicit deny in an identity-based policy
2026-06-25T11:47:53Z W [rs1/rs101:27017] [restore/2026-06-25T11:47:05.333362065Z] failed to download chunk 0-8388607
2026-06-25T11:47:53Z W [rs1/rs101:27017] [restore/2026-06-25T11:47:05.333362065Z] retryChunk got failed to download chunk 0-8388607 (of 9841) after 2 retries: operation error S3: GetObject, https response error StatusCode: 403, RequestID: 3ZXPW115VWPJSY92, HostID: qEzQvClNLGWp+XJcLOAR9JWL/hDs+8jW67B49Jk4vpZesh9u14hpl6ZNxsLxo6VWQ0Eylte3imIf25bkLMPl/qclG1qG4eKS, api error AccessDenied: User: arn:aws:iam::...:user/pbm-jakub is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:eu-central-1:119175775298:key/5ecc3fe9-f6ff-4265-9439-294bf25ed72b with an explicit deny in an identity-based policy, try to reconnect in 9s
2026-06-25T11:48:02Z I [rs1/rs101:27017] [restore/2026-06-25T11:47:05.333362065Z] session recreated, resuming download

after:

pbm restore 2026-06-25T11:27:22Z -w -y

Starting restore 2026-06-25T11:42:04.867306196Z from '2026-06-25T11:27:22Z'...........Started logical restore.
Waiting to finish......................................Error: operation failed with: mongorestore: restore mongo dump (successes: 0 / fails: 0): I/O failure reading beginning of archive: compose: metadata: new metafile reader: download: "metadata.json": SourceReader: download 'pbm-jakub/2026-06-25T11:27:22Z/rs1/metadata.json': operation error S3: GetObject, https response error StatusCode: 403, RequestID: 3NF83S9TMEST23RB, HostID: iIZdTq9sxmPDM+iJ5a74vIodVOdhYDoIsMNXs4uNm5/7Qd2jTmNCFIWwA7fGWcfhDkV31xIUyD5Pp9oPwfplor8n5izan3fM, api error AccessDenied: User: arn:aws:iam::...:user/pbm-jakub is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:eu-central-1:... with an explicit deny in an identity-based policy

jcechace added 2 commits June 24, 2026 11:30
Use errors.As to detect GetObjError in tryChunk instead of relying on a custom errors.Is implementation. This restores the intended retry layering: object-open failures skip the inner same-session retry loop and are handled by the outer retryChunk session re-creation path.

Also fix GetObjError unwrapping so provider errors remain inspectable through the standard error chain, and remove the now-unused custom Is method.
@jcechace
jcechace marked this pull request as ready for review June 25, 2026 11:53
@jcechace
jcechace requested a review from boris-ilijic as a code owner June 25, 2026 11:53
@jcechace
jcechace merged commit 052d024 into percona:dev Jun 26, 2026
37 of 39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants