The original problem, reported by one customer, was that they got non retryable

errors during CompleteMPU calls. In this case, their client would follow up
with a call to the AbortMPU API, logically.

We made a mistake during the last bugfix:

- We incorrectly thought that the reason of the error was a failure during
  the extra part deletion. And fixed it by cleaning everything during AbortMPU.
  But this is wrong: when an error happens in this step of the CompleteMPU API,
  it means the S3 object was properly created AND the overview keys and parts
  metadata were already cleaned up. Aborting in such case would just return
  a 404 error, but won't do anything. Instead, the state that the customer
  encountered was different: they failed at the `deletePartsMetadata` step.
  This is something we support with the `CompleteMultiPartUpload` retry logic.

The clean up in the AbortMPU API is actually great: when we end up in this
situation, the client has two available (and valid) choices:

- They retry the CompleteMultiPartUpload.
- They call the AbortMPU API.

The latter is now possible, since the cleanup introduction. In this case, we
detect that the object is in an inconsistent state and proceed with the
removal of everything (both in the S3 bucket and the mpuShadowBucket), so
the result is consistent: the MPU is aborted and the customer must re-upload
the whole object.

The first option is preferred, as it keeps the object in the bucket and avoids
the need to re-upload.

But the problem here is that we do not guarantee that a retryable error is
returned to the client: in case of MD update failure, the returned error
might not be retryable: we can have DeleteConflict or NoSuchKey errors.
Other usually are retryable (InternalError, etc).

So we want to make sure of two behaviors:
- In the batchDeleteObjectMetadata error callback case, we should always
  return a retryable error. It means that in case of a DeleteConflict, we
  should always retry, and in case of a NoSuchKey, we should not retry.
- During the batchDeleteExtraParts, we already cleaned up the mpu bucket,
  so returning an error to the client if the operation fails will wrongly
  lead to a completeMPU retry or ABort, while the operation succeeded, and
  we only created ghosts. In any case, the ghosts are permanent: without
  the mpu bucket, we cannot clean them in subsequent calls.

Issue: CLDSRV-669

Author: Seragonia

lib/api/completeMultipartUpload.js

@@ -524,15 +524,64 @@ function completeMultipartUpload(authInfo, request, log, callback) {
         function deletePartsMetadata(mpuBucket, keysToDelete, aggregateETag,
             extraPartLocations, destinationBucket, generatedVersionId, droppedMPUSize, next) {
             services.batchDeleteObjectMetadata(mpuBucket.getName(),
-                keysToDelete, log, err => next(err, extraPartLocations,
-                    destinationBucket, aggregateETag, generatedVersionId, droppedMPUSize));
+                keysToDelete, log, err => {
+                    if (err) {
+                        log.error('error deleting MPU metadata during completeMPU', {
+                            method: 'completeMultipartUpload',
+                            mpuBucket: mpuBucket.getName(),
+                            keysToDeleteCount: keysToDelete.length,
+                            error: err,
+                        });
+
+                        // Handle specific error cases according to retry strategy
+                        if (err.is?.DeleteConflict) {
+                            // DeleteConflict should trigger automatic retry
+                            // Convert to InternalError to make it retryable
+                            return next(errors.InternalError, extraPartLocations,
+                                destinationBucket, aggregateETag, generatedVersionId, droppedMPUSize);
+                        }
+
+                        // For NoSuchKey and other errors, return them as-is
+                        // NoSuchKey is non-retryable, InternalError and others are retryable
+                        return next(err, extraPartLocations,
+                            destinationBucket, aggregateETag, generatedVersionId, droppedMPUSize);
+                    }
+
+                    log.debug('successfully deleted MPU metadata during completeMPU', {
+                        method: 'completeMultipartUpload',
+                        mpuBucket: mpuBucket.getName(),
+                        keysToDeleteCount: keysToDelete.length,
+                    });
+                    return next(null, extraPartLocations,
+                        destinationBucket, aggregateETag, generatedVersionId, droppedMPUSize);
+                });
         },
         function batchDeleteExtraParts(extraPartLocations, destinationBucket,
             aggregateETag, generatedVersionId, droppedMPUSize, next) {
             if (extraPartLocations && extraPartLocations.length > 0) {
                 return data.batchDelete(extraPartLocations, request.method, null, log, err => {
                     if (err) {
-                        return next(err);
+                        // Extra part deletion failure should not fail the operation
+                        // The S3 object was created successfully and MPU metadata was cleaned up
+                        // Orphaned extra parts are acceptable since the main operation succeeded
+                        log.warn('failed to delete extra parts, keeping orphan but returning success', {
+                            method: 'completeMultipartUpload',
+                            extraPartLocationsCount: extraPartLocations.length,
+                            error: err,
+                        });
+
+                        // Continue with quota validation but don't fail if it errors either
+                        return validateQuotas(request, destinationBucket, request.accountQuotas,
+                            ['objectDelete'], 'objectDelete', -droppedMPUSize, false, log, quotaErr => {
+                                if (quotaErr) {
+                                    log.warn('failed to update inflights after extra part deletion failure', {
+                                        method: 'completeMultipartUpload',
+                                        error: quotaErr,
+                                    });
+                                }
+                                // Always succeed - the core operation was successful
+                                return next(null, destinationBucket, aggregateETag, generatedVersionId);
+                            });
                     }
 
                     return validateQuotas(request, destinationBucket, request.accountQuotas,