S3 Multipart Upload Results

I accidentally set off a huge multipart upload. I was afraid to cancel the upload midway through as I didn’t know how to remove the uploaded chunks and to stop being charged for a partial upload. So I let it run and recorded the results. The upload errored at 56 minutes (The operation couldn’t be completed. (AWSSDKSwiftCore.AWSResponseError error 1). Now I need to find where the partial upload is in Amazon console…

Results

Details: File Size - 17.91GB Speedtest by Ookla

Server: TNP Ltd. - Manchester ISP: Virgin Media Latency: 14.68 ms (11.32 ms jitter) Download: 99.53 Mbps (data used: 171.5 MB)
Upload: 16.40 Mbps (data used: 27.0 MB)
Packet Loss: 0.0%

Times: Start - 13:23pm Finish - 14:17 Duration - 56 minutes

Update

So it turns out the Amazon Console doesn’t show the partially uploaded chunks nor a way for them to be deleted from the console… that I could find. I hastily coded up an abort() function and luckily was about to extract the required values from the multipart upload request.

Lessons Learnt

  • Make sure you log the bucket name, key and upload id somewhere so if this ever happens you can abort the multipart upload.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
      @IBAction func abort(_ sender: Any) {
        let s3 = S3(accessKeyId: accessKeyID.stringValue, secretAccessKey: secretAccessKey.stringValue, sessionToken: sessionToken.stringValue, region: .euwest2)
        let abortRequest = S3.AbortMultipartUploadRequest(bucket: bucketName.stringValue, key: key.stringValye, uploadId: uploadID.stringValue)
        let _ = s3.abortMultipartUpload(abortRequest).always { result in
          switch result {
          case .success(let output):
            print(output)
          case .failure(let uploadError):
            print(uploadError.localizedDescription)
          }
          print(result)
        }
      }
    
  • Compress the videos beforehand.

  • I think the upload stopped because the credentials to access the S3 bucket only last 1hr. It would be worth pursuing whether that theory is correct.

References

  • Amazon S3 multipart upload limits. [online]. Available at: https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html [accessed 3 December 2020].