Problems creating object storage buckets using AWS SDK
I am using the c# implementation of the AWS SDK, and struggling to get expected behaviour out of the Linode object storage endpoints. Here is a little code to show what I'm on about:
if (!await AmazonS3Util.DoesS3BucketExistV2Async(s3Client, BucketName))
{
var Request = new PutBucketRequest { BucketName = BucketName, BucketRegionName = "us-east-1", UseClientRegion = false };
PutBucketResponse putBucketResponse = await s3Client.PutBucketAsync(Request, stoppingToken);
}
So - the only way that I can create a bucket in any cluster is to specify BucketRegionName as "us-east-1" - if I do that, then the bucket is created on the cluster to which I connected - so if I connect to eu-central-1 the bucket will be created there regardless of the setting in UseClientRegion.
If I use anything other than "us-east-1" for the bucket region name, having connected to any of the three available clusters, my PutBucketAsync call will fail. When it fails, its with a 400 bad request and nothing useful in the body response.
I can create the buckets just fine in the Linode UI so I am assuming this is some nicety of the AWS SDK or some minor incompatibility between the two - but I cannot make peace with specifying "us-east-1" as my BucketRegionName when I want to make a bucket on a different cluster entirely.
I am hopeful that someone can say "hey stupid, you're doing it wrong" with a bit of guidance on how to do things differently…..just please don't say AWS SDK is not going to work with Linode because my use case involves compatibility with multiple object storage providers.
Thank you everyone!
7 Replies
I’m pretty sure you can only create buckets on the cluster you are connected/authenticated to.
When you create your connection (or your instance of the S3 client), are you also providing the “endpoint URL” - i.e. https://eu-central-1.linodeobjects.com)?
The Linode clusters function independently. So if you authenticate against eu-central-1, you’ll only be able to create buckets in eu-central-1.
Ok ignore what I just said!
Just tried this using rclone, with a location constraint of “eu-central-1” with an endpoint URL of https://eu-central-1.linodeobjects.com/ and this does indeed fail with a 400:
Failed to sync: InvalidLocationConstraint: The specified location-constraint is not valid
status code: 400, request id: , host id:
My rclone config is:
endpoint = https://eu-central-1.linodeobjects.com/
location_constraint = eu-central-1
I was able to get this working by removing the location_constraint line, which I assume doesn’t pass any region to the storage system, and creates the bucket in eu-central-1.
Thank you Andy for taking the time and trouble to validate what I was experiencing. Yes, when I connect to object storage I specify eu central endpoint URL:
var s3Config = new AmazonS3Config { ServiceURL = "https://eu-central-1.linodeobjects.com/" };
var s3Client = new AmazonS3Client(AccessKeyId, SecretAccessKey, s3Config);
The absence of the location constraint apparently being the secret to success here, I dug into the AWS SDK to see what its doing:
string regionCode = null;
var region = putBucketRequest.BucketRegion;
if (region != null && !string.IsNullOrEmpty(region.Value))
{
regionCode = region.Value;
}
else if (!string.IsNullOrEmpty(putBucketRequest.BucketRegionName))
{
if (putBucketRequest.BucketRegionName == "eu-west-1")
regionCode = "EU";
else if (putBucketRequest.BucketRegionName != "us-east-1")
regionCode = putBucketRequest.BucketRegionName;
}
if (regionCode != null)
{
xmlWriter.WriteStartElement("CreateBucketConfiguration", "");
xmlWriter.WriteElementString("LocationConstraint", "", regionCode);
xmlWriter.WriteEndElement();
}
So the LocationConstraint is only passed to the object storage service if I actively specify a BucketRegion or BucketRegionName - if I leave them out I should in theory have a request formatted the same as your Rclone example - and yet it fails. And further this doesn't explain why specifying the location constraint as "us-east-1" causes my PutBucket request to always succeed - I suspect you would see the same behaviour.
I am really at a loss with this, everything else works a treat using the AWS SDK.
Do you reckon Linode support could find my PutBucket requests in their logging somewhere and identify what's actually Bad about my Bad Request?
Very strange.
I had tried setting my location constraint to us-east-1 in rclone (but still authenticating against eu-central-1, and this also failed with a 400.)
I could only get it working by leaving out the location constraint altogether, which is different to what you are seeing.
I think it’s certainly one to query with the support team; this behaviour does not seem “correct”.
Thank you Andy I will raise it with them and report back - if anyone else finds this thread with the same problem I would hate for it to hang here without a resolution good or otherwise!
