Add functionality for verifying tarball signatures to automated ingestion scripts

[bedroge]

Still testing it, but first tests look good. I'm running a separate instance/fork of the automated ingestion on the Stratum 0, configured to use a test bucket and non-existing target repo. This successfully picked up the tarball+metadata file and their signatures from the test bucket (uploaded by @trz42 in EESSI/software-layer#968):

-rw-r--r--. 1 eessi eessi 124619768 Mar 20 16:12 eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz
-rw-r--r--. 1 eessi eessi       663 Mar 20 16:12 eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz.meta.txt
-rw-r--r--. 1 eessi eessi       878 Mar 20 16:12 eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz.meta.txt.sig
-rw-r--r--. 1 eessi eessi       878 Mar 20 16:12 eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz.sig

The log shows that the signatures were successfully verified:

2025-03-20 16:12:59,176 - DEBUG - Signature for /srv/tmp/tarballs/eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz successfully verified.
2025-03-20 16:12:59,186 - DEBUG - Signature for /srv/tmp/tarballs/eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz.meta.txt successfully verified.

We should do the same for a non-valid signature (and make sure they fail), and for tarballs without a signature file (this should succeed if signatures_required = no).

[bedroge]

The staging PR for the test tarball was opened: https://github.com/EESSI/staging/pull/2544. Before merging that PR, I've manually changed the signature of the metadata file to verify that the ingestion would fail, and it did:

2025-03-20 16:28:32,061 - INFO - Tarball 2023.06/software/linux/aarch64/nvidia/grace/1741954310/eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz is ready to be ingested.
2025-03-20 16:28:32,061 - INFO - Verifying its signature...
2025-03-20 16:28:32,276 - DEBUG - Signature for /srv/tmp/tarballs/eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz successfully verified.
2025-03-20 16:28:32,285 - ERROR - Failed to verify signature for /srv/tmp/tarballs/eessi-2023.06-software-linux-aarch64-nvidia-grace-1741954310.tar.gz.meta.txt.
2025-03-20 16:28:32,285 - ERROR - Signature of tarball (or its metadata file) could not be verified!

Right now it only prints this in the log file, and continues. It should also open an issue in the staging repo, I'll have to add that.

[bedroge]

It now opens an issue if the verification fails, see https://github.com/EESSI/staging/issues/2545.

[bedroge]

Another test: I've uploaded a tarball and metadata file without signatures, and set signatures_required = yes. This will print errors to the log and it will ignore the tarball. It doesn't open an issue, as we could possibly get a whole lot of them (but it can be easily added later if we want this).

[trz42]

Looks good.

• A few suggestions for naming attributes/variables. (if done would require renaming across the whole script.)
• A tiny typo.
• Question about logic when to return False (two places).

[trz42]

Maybe worth to rename elements to clearly notify what they refer to and have the naming consistent? For example,

(self.object_remote_path, self.object_local_path, self.object_sig_remote_path, self.object_sig_local_path),
(self.metadata_remote_path, self.metadata_local_path, self.metadata_sig_remote_path, self.metadata_sig_local_path),

[bedroge]

This originated from the object terminology used by the boto client, but I agree that it would be more clear to use the names like you proposed. Is it okay to do that in a follow-up PR, as it would require changes across the entire script?

[trz42]

Yes, perfectly fine to do it in a follow-up PR.

[trz42]

Possibly synchronise naming with potential changes in files array, e.g.,

for (remote_path, local_path, sig_remote_path, sig_local_path) in files:

[trz42]

lgtm