Copy_from from S3 to Linode fails

Automatic translation by Google

hello

Currently, we are developing using Linode, but we are creating Lambda with Ruby to migrate data from existing S3 to Linode.

I'm trying to copy using the Ruby SDK copy_from to migrate from S3 to Linode, but it fails with a key violation.

However, there is no mistake in the key, and it actually succeeds in copying from S3 to another S3 bucket using the same information.

Are there any conditions to use copy_from for Linode?

primary_client = Aws::S3::Client.new

second_client = Aws::S3::Client.new(

      endpoint: 'https://ap-south-1.linodeobjects.com',

      region: 'default',

      access_key_id: secondary_access_key,

      secret_access_key: secondary_access_sec,

      http_continue_timeout: 0

    )

primary_bucket = Aws::S3::Resource.new(client: primary_client).bucket('s3mybucket')

secondary_bucket = Aws::S3::Resource.new(client: second_client).bucket('mybucket')


from_key = 'testObject.txt' # is 5MB

to_key = 'testObject.txt'

if primary_bucket.object(from_key).exists?

  to_obj = secondary_bucket.object(to_key)

  to_obj.copy_from(primary_bucket.object(from_key), copy_source_client: primary_client, multipart_copy: true, acl: 'public-read')

  puts 'is ok.'

else

  puts 'not found.'

end

Result

#<Thread:0x00007f651a5e1178 /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.96.1/lib/aws-sdk-s3/object_multipart_copier.rb:62 run> terminated with exception (report_on_exception is true):

Traceback (most recent call last):

        13: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.96.1/lib/aws-sdk-s3/object_multipart_copier.rb:66:in `block in copy_part_thread'

        12: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.96.1/lib/aws-sdk-s3/object_multipart_copier.rb:78:in `copy_part'

        11: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.96.1/lib/aws-sdk-s3/client.rb:13615:in `upload_part_copy'

        10: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/seahorse/client/request.rb:72:in `send_request'

         9: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/seahorse/client/plugins/response_target.rb:24:in `call'

         8: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'

         7: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/seahorse/client/plugins/request_callback.rb:71:in `call'

         6: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'

         5: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'

         4: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'

         3: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/accelerate.rb:50:in `call'

         2: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/dualstack.rb:36:in `call'

         1: from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'

/mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Aws::S3::Errors::NoSuchKey (Aws::S3::Errors::NoSuchKey)

Aws::S3::Errors::NoSuchKey: Aws::S3::Errors::NoSuchKey

from /mnt/c/work/myfans/wcl/myfans/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.114.2/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'

For different buckets in the same S3

primary_client = Aws::S3::Client.new
primary_bucket = Aws::S3::Resource.new(client: primary_client).bucket('s3mybucket')

from_key = 'testObject.txt' # is 5MB

to_key = 'testObject2.txt'

if primary_bucket.object(from_key).exists?

  to_obj = primary_bucket.object(to_key) # primary to primary

  to_obj.copy_from(primary_bucket.object(from_key), copy_source_client: primary_client, multipart_copy: true, acl: 'public-read')

  puts 'is ok.'

else

  puts 'not found.'

end

Result

=> is ok.

Thank you.