Upgrade to 26.4.2 hanging on Retrying failed request: /?lifecycle

[arkadicolson]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

Running ./install.sh

Expected Result

upgrade succesful

Actual Result

Container sentry-self-hosted-postgres-1 Healthy
Container sentry-self-hosted-seaweedfs-1 Healthy
( 1/26) Installing libbz2 (1.0.8-r6)
( 2/26) Installing libexpat (2.7.5-r0)
( 3/26) Installing libffi (3.5.2-r0)
( 4/26) Installing gdbm (1.26-r0)
( 5/26) Installing xz-libs (5.8.3-r0)
( 6/26) Installing libgcc (15.2.0-r2)
( 7/26) Installing libstdc++ (15.2.0-r2)
( 8/26) Installing mpdecimal (4.0.1-r0)
( 9/26) Installing ncurses-terminfo-base (6.5_p20251123-r0)
(10/26) Installing libncursesw (6.5_p20251123-r0)
(11/26) Installing libpanelw (6.5_p20251123-r0)
(12/26) Installing readline (8.3.1-r0)
(13/26) Installing sqlite-libs (3.51.2-r0)
(14/26) Installing python3 (3.12.13-r0)
(15/26) Installing python3-pycache-pyc0 (3.12.13-r0)
(16/26) Installing pyc (3.12.13-r0)
(17/26) Installing s3cmd-pyc (2.4.0-r1)
(18/26) Installing python3-pyc (3.12.13-r0)
(19/26) Installing py3-six (1.17.0-r0)
(20/26) Installing py3-six-pyc (1.17.0-r0)
(21/26) Installing py3-dateutil (2.9.0-r2)
(22/26) Installing py3-dateutil-pyc (2.9.0-r2)
(23/26) Installing libmagic (5.46-r2)
(24/26) Installing py3-magic (0.4.27-r3)
(25/26) Installing py3-magic-pyc (0.4.27-r3)
(26/26) Installing s3cmd (2.4.0-r1)
Executing busybox-1.37.0-r30.trigger
OK: 64.1 MiB in 55 packages
Node store already exists, skipping creation...
Making sure the bucket lifecycle policy is all set up correctly...

WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 6 sec...

Product Area

Issues

Link

No response

DSN

No response

Version

26.4.1

[linear-code]

ENG-7647

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[tuobi2]

Just to narrow this down: are you using the default bundled seaweedfs or an external S3-compatible store? If it’s seaweedfs, can you check the logs of the sentry-self-hosted-seaweedfs-1 container around the time of the timeout? It might be rejecting or hanging on the lifecycle request, which would explain the retry loop.

Note: AI-assisted.

[arkadicolson]

We are using the default bundled seaweedfs.

More output:

(22/26) Installing py3-dateutil-pyc (2.9.0-r2)
(23/26) Installing libmagic (5.46-r2)
(24/26) Installing py3-magic (0.4.27-r3)
(25/26) Installing py3-magic-pyc (0.4.27-r3)
(26/26) Installing s3cmd (2.4.0-r1)
Executing busybox-1.37.0-r30.trigger
OK: 64.1 MiB in 55 packages
Node store already exists, skipping creation...
Making sure the bucket lifecycle policy is all set up correctly...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 15 sec...
ERROR: S3 Temporary Error: Request failed for: /?lifecycle. Please try again later.

Error in install/bootstrap-s3-nodestore.sh:89.
'$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' setlifecycle /tmp/nodestore-lifecycle-policy.xml s3://nodestore' exited with status 75
-> ./install.sh:main:39
--> install/bootstrap-s3-nodestore.sh:source:89

install/error-handling.sh: line 81: /usr/bin/docker: Argument list too long

seaweed logs:

seaweedfs-1 | file /buckets/nodestore/nodestore/i-o:fff94f67f01da804142798e4a0e7b706: UpdateEntry: rpc error: code = Unknown desc = not found /buckets/nodestore/nodestore/i-o:fff94f67f01da804142798e4a0e7b706: filer: no entry is found in filer store
seaweedfs-1 | file /buckets/nodestore/nodestore/i-o:fffa1b862586e46dc257f862e6f15e7b: UpdateEntry: rpc error: code = Unknown desc = not found /buckets/nodestore/nodestore/i-o:fffa1b862586e46dc257f862e6f15e7b: filer: no entry is found in filer store
seaweedfs-1 | file /buckets/nodestore/nodestore/i-o:fffd7a44e9881c423d780a6f430b9751: UpdateEntry: rpc error: code = Unknown desc = not found /buckets/nodestore/nodestore/i-o:fffd7a44e9881c423d780a6f430b9751: filer: no entry is found in filer store
seaweedfs-1 | file /buckets/nodestore/nodestore/i-o:fffe65d3df6c66554141629ee47a081b: UpdateEntry: rpc error: code = Unknown desc = not found /buckets/nodestore/nodestore/i-o:fffe65d3df6c66554141629ee47a081b: filer: no entry is found in filer store
seaweedfs-1 | file /buckets/nodestore/nodestore/i-o:fffe8a5c9ad12de517da6f4e284b8afa: UpdateEntry: rpc error: code = Unknown desc = not found /buckets/nodestore/nodestore/i-o:fffe8a5c9ad12de517da6f4e284b8afa: filer: no entry is found in filer store
seaweedfs-1 | file /buckets/nodestore/nodestore/i-o:fffea3869635c12d7be6410f58ae42e7: UpdateEntry: rpc error: code = Unknown desc = not found /buckets/nodestore/nodestore/i-o:fffea3869635c12d7be6410f58ae42e7: filer: no entry is found in filer store
seaweedfs-1 | I0513 07:34:50.683879 filer_on_meta_event.go:86 procesing directory:"/etc/seaweedfs" event_notification:{old_entry:{name:"filer.conf" attributes:{file_size:968 mtime:1778657654 file_mode:420 crtime:1759590535} content:"{\n "version": 0,\n "locations": [\n {\n "locationPrefix": "/buckets/nodestore/",\n "collection": "nodestore",\n "replication": "000",\n "ttl": "30d",\n "diskType": "",\n "fsync": false,\n "volumeGrowthCount": 1,\n "readOnly": false,\n "dataCenter": "",\n "rack": "",\n "dataNode": "",\n "maxFileNameLength": 0,\n "disableChunkDeletion": false,\n "worm": false,\n "wormGracePeriodSeconds": "0",\n "wormRetentionTimeSeconds": "0"\n },\n {\n "locationPrefix": "/buckets/profiles/",\n "collection": "profiles",\n "replication": "",\n "ttl": "90d",\n "diskType": "",\n "fsync": false,\n "volumeGrowthCount": 0,\n "readOnly": false,\n "dataCenter": "",\n "rack": "",\n "dataNode": "",\n "maxFileNameLength": 0,\n "disableChunkDeletion": false,\n "worm": false,\n "wormGracePeriodSeconds": "0",\n "wormRetentionTimeSeconds": "0"\n }\n ]\n}"} new_entry:{name:"filer.conf" attributes:{file_size:968 mtime:1778657690 file_mode:420 crtime:1759590535} content:"{\n "version": 0,\n "locations": [\n {\n "locationPrefix": "/buckets/nodestore/",\n "collection": "nodestore",\n "replication": "000",\n "ttl": "30d",\n "diskType": "",\n "fsync": false,\n "volumeGrowthCount": 1,\n "readOnly": false,\n "dataCenter": "",\n "rack": "",\n "dataNode": "",\n "maxFileNameLength": 0,\n "disableChunkDeletion": false,\n "worm": false,\n "wormGracePeriodSeconds": "0",\n "wormRetentionTimeSeconds": "0"\n },\n {\n "locationPrefix": "/buckets/profiles/",\n "collection": "profiles",\n "replication": "",\n "ttl": "90d",\n "diskType": "",\n "fsync": false,\n "volumeGrowthCount": 0,\n "readOnly": false,\n "dataCenter": "",\n "rack": "",\n "dataNode": "",\n "maxFileNameLength": 0,\n "disableChunkDeletion": false,\n "worm": false,\n "wormGracePeriodSeconds": "0",\n "wormRetentionTimeSeconds": "0"\n }\n ]\n}"} delete_chunks:true new_parent_path:"/etc/seaweedfs" signatures:-223324364} ts_ns:1778657690683480424
seaweedfs-1 | I0513 07:34:51.512087 empty_folder_cleaner.go:311 EmptyFolderCleaner: folder /buckets/nodestore/nodestore (triggered by f7ea08e01a5c90eaf3e643aa8308dfda) has 1000 items, not empty
seaweedfs-1 | I0513 07:34:56.369041 filer_on_meta_event.go:86 procesing directory:"/etc/seaweedfs" event_notification:{old_entry:{name:"filer.conf" attributes:{file_size:968 mtime:1778657690 file_mode:420 crtime:1759590535} content:"{\n "version": 0,\n "locations": [\n {\n "locationPrefix": "/buckets/nodestore/",\n "collection": "nodestore",\n "replication": "000",\n "ttl": "30d",\n "diskType": "",\n "fsync": false,\n "volumeGrowthCount": 1,\n "readOnly": false,\n "dataCenter": "",\n "rack": "",\n "dataNode": "",\n "maxFileNameLength": 0,\n "disableChunkDeletion": false,\n "worm": false,\n "wormGracePeriodSeconds": "0",\n "wormRetentionTimeSeconds": "0"\n },\n {\n "locationPrefix": "/buckets/profiles/",\n "collection": "profiles",\n "replication": "",\n "ttl": "90d",\n "diskType": "",\n "fsync": false,\n "volumeGrowthCount": 0,\n "readOnly": false,\n "dataCenter": "",\n "rack": "",\n "dataNode": "",\n "maxFileNameLength": 0,\n "disableChunkDeletion": false,\n "worm": false,\n "wormGracePeriodSeconds": "0",\n "wormRetentionTimeSeconds": "0"\n }\n ]\n}"} new_entry:{name:"filer.conf" attributes:{file_size:968 mtime:1778657696 file_mode:420 crtime:1759590535} content:"{\n "version": 0,\n "locations": [\n {\n "locationPrefix": "/buckets/nodestore/",\n "collection": "nodestore",\n "replication": "000",\n "ttl": "30d",\n "diskType": "",\n "fsync": false,\n "volumeGrowthCount": 1,\n "readOnly": false,\n "dataCenter": "",\n "rack": "",\n "dataNode": "",\n "maxFileNameLength": 0,\n "disableChunkDeletion": false,\n "worm": false,\n "wormGracePeriodSeconds": "0",\n "wormRetentionTimeSeconds": "0"\n },\n {\n "locationPrefix": "/buckets/profiles/",\n "collection": "profiles",\n "replication": "",\n "ttl": "90d",\n "diskType": "",\n "fsync": false,\n "volumeGrowthCount": 0,\n "readOnly": false,\n "dataCenter": "",\n "rack": "",\n "dataNode": "",\n "maxFileNameLength": 0,\n "disableChunkDeletion": false,\n "worm": false,\n "wormGracePeriodSeconds": "0",\n "wormRetentionTimeSeconds": "0"\n }\n ]\n}"} delete_chunks:true new_parent_path:"/etc/seaweedfs" signatures:-223324364} ts_ns:1778657696291056065
seaweedfs-1 | I0513 07:36:31.718736 master_client.go:24 the cluster has 1 filer

[aldy505]

What is your previous version that you're upgrading from?