Object Storage Errors During Docker Push to Private Registry

Hi!

I've been stuck on this problem for a bit and could really use any help. I followed the tutorial from the guides to setup a docker registry in my LKE instance. This instance is behind traefik and is served using IngressRoutes.

Unfortunately, when I push docker images some of the larger layers fail once their progress is complete. For instance I get

The push refers to repository [registry.example.io/nginx]
14773070094d: Layer already exists 
7d2fd59c368c: Layer already exists 
56f8fe6aedcd: Layer already exists 
9f4d73e635f1: Layer already exists 
747b290aeba8: Layer already exists 
fc1cf9ca5139: Pushing [==============================================>]  112.8MB/112.8MB
5d4427064ecc: Layer already exists

which then becomes

The push refers to repository [registry.example.io/nginx]
14773070094d: Layer already exists 
7d2fd59c368c: Layer already exists 
56f8fe6aedcd: Layer already exists 
9f4d73e635f1: Layer already exists 
747b290aeba8: Layer already exists 
fc1cf9ca5139: Retrying in 3...
5d4427064ecc: Layer already exists

I collected the debug logs for this push

time="2024-05-28T14:03:13.606970118Z" level=debug msg="s3aws.List("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/hashstates/sha256")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=24.453643ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).List" trace.id=90bdd5b8-9abe-4908-89c3-bf7ac6fb4a8f trace.line=170 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:13.668351034Z" level=debug msg="s3aws.GetContent("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/hashstates/sha256/41830539")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=61.097633ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=dfb68cb4-5286-46f6-a8fb-c3f6ec7c1445 trace.line=95 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:13.700038399Z" level=debug msg="s3aws.Stat("/docker/registry/v2/blobs/sha256/a1/a11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f/data")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=31.604724ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=6570a986-b045-4c5c-b010-7524cd64a26f trace.line=155 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:13.776392541Z" level=debug msg="s3aws.Stat("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/data")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=76.277352ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=227cb759-83a5-4774-b257-5271d3df813e trace.line=155 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:13.830245483Z" level=debug msg="s3aws.Move("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/data", "/docker/registry/v2/blobs/sha256/a1/a11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f/data"" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=53.78626ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Move" trace.id=4c2e1f6a-3adb-4c7f-8961-6335f795abd5 trace.line=185 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:13.830684868Z" level=error msg="unknown error completing upload: s3aws: AccessDenied: 
    status code: 403, request id: tx000004a35fd15941e7b6e-006655e421-43b66fa8-default, host id: " go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:13.831060303Z" level=debug msg="(*blobWriter).Cancel" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:13.831474068Z" level=error msg="error canceling upload after error: already closed" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:13.88813793Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/hashstates/sha256/41830539")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=56.283379ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=1118bb09-154a-4e81-9e8e-8d67208bb12c trace.line=110 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
10.2.0.48 - - [28/May/2024:14:03:13 +0000] "PUT /v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f HTTP/1.1" 500 123 "" "docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \\(linux\\))"
time="2024-05-28T14:03:13.888893089Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: AccessDenied: 
    status code: 403, request id: tx000004a35fd15941e7b6e-006655e421-43b66fa8-default, host id: " err.message="unknown error" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=500.852778ms http.response.status=500 http.response.written=123 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1 
time="2024-05-28T14:03:19.228783344Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx 
time="2024-05-28T14:03:19.229440381Z" level=debug msg="(*linkedBlobStore).Writer" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx 
time="2024-05-28T14:03:19.255164369Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/startedat")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=25.452036ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=c1a25f06-3be0-48cf-9132-536c63b94553 trace.line=110 vars.name=nginx 
time="2024-05-28T14:03:19.276360046Z" level=debug msg="s3aws.Writer("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/data", false)" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=20.73266ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=673f1b71-8141-465e-a5c9-a77a5632fc6b trace.line=142 vars.name=nginx 
10.2.0.48 - - [28/May/2024:14:03:19 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \\(linux\\))"
time="2024-05-28T14:03:19.293999062Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/hashstates/sha256/0")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=17.018809ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=269ad40b-7504-4169-b0d8-f69d7e71f68b trace.line=110 vars.name=nginx 
time="2024-05-28T14:03:19.294390297Z" level=info msg="response completed" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" http.response.duration=66.800618ms http.response.status=202 http.response.written=0 
time="2024-05-28T14:03:19.598258313Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59 
time="2024-05-28T14:03:19.598848119Z" level=debug msg="(*linkedBlobStore).Resume" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59 
time="2024-05-28T14:03:19.613508844Z" level=debug msg="s3aws.GetContent("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/startedat")" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=14.428311ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=1de9a74e-9e7d-461c-9faa-e25532759437 trace.line=95 vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59 
time="2024-05-28T14:03:19.651367197Z" level=debug msg="s3aws.Writer("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/data", true)" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=37.50012ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=49e64727-d465-447f-bcfe-7475f5378895 trace.line=142 vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59 
45.56.84.131 - - [28/May/2024:14:03:21 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
45.56.84.131 - - [28/May/2024:14:03:21 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
time="2024-05-28T14:03:21.6682567Z" level=debug msg="s3aws.Stat("/")" go.version=go1.20.8 instance.id=44c9e61c-ee58-46f0-88b1-905693931aab service=registry trace.duration=85.561306ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=b7199a81-3b9f-42fb-ba36-1a488fddc877 trace.line=155 version=2.8.3 
45.56.84.131 - - [28/May/2024:14:03:31 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
45.56.84.131 - - [28/May/2024:14:03:31 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
time="2024-05-28T14:03:31.631863116Z" level=debug msg="s3aws.Stat("/")" go.version=go1.20.8 instance.id=44c9e61c-ee58-46f0-88b1-905693931aab service=registry trace.duration=49.216971ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=67a51442-0a44-4444-b9b9-d0f96049f126 trace.line=155 version=2.8.3 
time="2024-05-28T14:03:41.031824169Z" level=error msg="client disconnected during blob PATCH" contentLength=-1 copied=39026688 error="unexpected EOF" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59 
time="2024-05-28T14:03:41.063136268Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/hashstates/sha256/39026688")" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=31.141198ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=20fc326f-441f-491c-bd44-7fe5ad152839 trace.line=110 vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59 
45.56.84.131 - - [28/May/2024:14:03:41 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
45.56.84.131 - - [28/May/2024:14:03:41 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
10.2.0.48 - - [28/May/2024:14:03:19 +0000] "PATCH /v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59? HTTP/1.1" 500 89 "" "docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \\(linux\\))"

and it would appear that the bucket sends back a 403 error when moving some blobs and then this makes the registry raise a 500 error. This makes no sense as my authentication to bucket was sufficient to write the other layers. This appears to be an issue only with the larger layers. This happens immediately after the following request:

PUT /v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1

This is very frustrating because I'd like to move on to deploying images from the registry. Any help would be tremendous.

Thank you so much for any help,
Adrian