Object Storage Errors During Docker Push to Private Registry
Hi!
I've been stuck on this problem for a bit and could really use any help. I followed the tutorial from the guides to setup a docker registry in my LKE instance. This instance is behind traefik and is served using IngressRoute
s.
Unfortunately, when I push docker images some of the larger layers fail once their progress is complete. For instance I get
The push refers to repository [registry.example.io/nginx]
14773070094d: Layer already exists
7d2fd59c368c: Layer already exists
56f8fe6aedcd: Layer already exists
9f4d73e635f1: Layer already exists
747b290aeba8: Layer already exists
fc1cf9ca5139: Pushing [==============================================>] 112.8MB/112.8MB
5d4427064ecc: Layer already exists
which then becomes
The push refers to repository [registry.example.io/nginx]
14773070094d: Layer already exists
7d2fd59c368c: Layer already exists
56f8fe6aedcd: Layer already exists
9f4d73e635f1: Layer already exists
747b290aeba8: Layer already exists
fc1cf9ca5139: Retrying in 3...
5d4427064ecc: Layer already exists
I collected the debug logs for this push
time="2024-05-28T14:03:13.606970118Z" level=debug msg="s3aws.List("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/hashstates/sha256")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=24.453643ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).List" trace.id=90bdd5b8-9abe-4908-89c3-bf7ac6fb4a8f trace.line=170 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:13.668351034Z" level=debug msg="s3aws.GetContent("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/hashstates/sha256/41830539")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=61.097633ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=dfb68cb4-5286-46f6-a8fb-c3f6ec7c1445 trace.line=95 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:13.700038399Z" level=debug msg="s3aws.Stat("/docker/registry/v2/blobs/sha256/a1/a11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f/data")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=31.604724ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=6570a986-b045-4c5c-b010-7524cd64a26f trace.line=155 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:13.776392541Z" level=debug msg="s3aws.Stat("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/data")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=76.277352ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=227cb759-83a5-4774-b257-5271d3df813e trace.line=155 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:13.830245483Z" level=debug msg="s3aws.Move("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/data", "/docker/registry/v2/blobs/sha256/a1/a11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f/data"" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=53.78626ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Move" trace.id=4c2e1f6a-3adb-4c7f-8961-6335f795abd5 trace.line=185 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:13.830684868Z" level=error msg="unknown error completing upload: s3aws: AccessDenied:
status code: 403, request id: tx000004a35fd15941e7b6e-006655e421-43b66fa8-default, host id: " go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:13.831060303Z" level=debug msg="(*blobWriter).Cancel" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:13.831474068Z" level=error msg="error canceling upload after error: already closed" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:13.88813793Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/nginx/_uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1/hashstates/sha256/41830539")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=56.283379ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=1118bb09-154a-4e81-9e8e-8d67208bb12c trace.line=110 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
10.2.0.48 - - [28/May/2024:14:03:13 +0000] "PUT /v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f HTTP/1.1" 500 123 "" "docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \\(linux\\))"
time="2024-05-28T14:03:13.888893089Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: AccessDenied:
status code: 403, request id: tx000004a35fd15941e7b6e-006655e421-43b66fa8-default, host id: " err.message="unknown error" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=f05d20fa-386d-4038-9e3a-7daaa1e48162 http.request.method=PUT http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1?&digest=sha256%3Aa11fc495bafd95699c7cb83ca0878f63f94e34c28837c1da8ae7c9879343604f" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=500.852778ms http.response.status=500 http.response.written=123 vars.name=nginx vars.uuid=1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
time="2024-05-28T14:03:19.228783344Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx
time="2024-05-28T14:03:19.229440381Z" level=debug msg="(*linkedBlobStore).Writer" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx
time="2024-05-28T14:03:19.255164369Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/startedat")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=25.452036ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=c1a25f06-3be0-48cf-9132-536c63b94553 trace.line=110 vars.name=nginx
time="2024-05-28T14:03:19.276360046Z" level=debug msg="s3aws.Writer("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/data", false)" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=20.73266ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=673f1b71-8141-465e-a5c9-a77a5632fc6b trace.line=142 vars.name=nginx
10.2.0.48 - - [28/May/2024:14:03:19 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \\(linux\\))"
time="2024-05-28T14:03:19.293999062Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/hashstates/sha256/0")" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=17.018809ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=269ad40b-7504-4169-b0d8-f69d7e71f68b trace.line=110 vars.name=nginx
time="2024-05-28T14:03:19.294390297Z" level=info msg="response completed" go.version=go1.20.8 http.request.host=registry.acederberg.io http.request.id=69183607-5b29-48df-88ea-c9ecc24a22c3 http.request.method=POST http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" http.response.duration=66.800618ms http.response.status=202 http.response.written=0
time="2024-05-28T14:03:19.598258313Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59
time="2024-05-28T14:03:19.598848119Z" level=debug msg="(*linkedBlobStore).Resume" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59
time="2024-05-28T14:03:19.613508844Z" level=debug msg="s3aws.GetContent("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/startedat")" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=14.428311ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=1de9a74e-9e7d-461c-9faa-e25532759437 trace.line=95 vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59
time="2024-05-28T14:03:19.651367197Z" level=debug msg="s3aws.Writer("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/data", true)" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=37.50012ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=49e64727-d465-447f-bcfe-7475f5378895 trace.line=142 vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59
45.56.84.131 - - [28/May/2024:14:03:21 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
45.56.84.131 - - [28/May/2024:14:03:21 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
time="2024-05-28T14:03:21.6682567Z" level=debug msg="s3aws.Stat("/")" go.version=go1.20.8 instance.id=44c9e61c-ee58-46f0-88b1-905693931aab service=registry trace.duration=85.561306ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=b7199a81-3b9f-42fb-ba36-1a488fddc877 trace.line=155 version=2.8.3
45.56.84.131 - - [28/May/2024:14:03:31 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
45.56.84.131 - - [28/May/2024:14:03:31 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
time="2024-05-28T14:03:31.631863116Z" level=debug msg="s3aws.Stat("/")" go.version=go1.20.8 instance.id=44c9e61c-ee58-46f0-88b1-905693931aab service=registry trace.duration=49.216971ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=67a51442-0a44-4444-b9b9-d0f96049f126 trace.line=155 version=2.8.3
time="2024-05-28T14:03:41.031824169Z" level=error msg="client disconnected during blob PATCH" contentLength=-1 copied=39026688 error="unexpected EOF" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59
time="2024-05-28T14:03:41.063136268Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/nginx/_uploads/16d00965-9dec-4f19-a3ec-30a04c968d59/hashstates/sha256/39026688")" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.acederberg.io http.request.id=06f701fc-f912-4889-9b31-b84798e936ed http.request.method=PATCH http.request.remoteaddr=45.56.84.131 http.request.uri="/v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59?" http.request.useragent="docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \(linux\))" trace.duration=31.141198ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=20fc326f-441f-491c-bd44-7fe5ad152839 trace.line=110 vars.name=nginx vars.uuid=16d00965-9dec-4f19-a3ec-30a04c968d59
45.56.84.131 - - [28/May/2024:14:03:41 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
45.56.84.131 - - [28/May/2024:14:03:41 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.29"
10.2.0.48 - - [28/May/2024:14:03:19 +0000] "PATCH /v2/nginx/blobs/uploads/16d00965-9dec-4f19-a3ec-30a04c968d59? HTTP/1.1" 500 89 "" "docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.5.0-35-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.2 \\(linux\\))"
and it would appear that the bucket sends back a 403 error when moving some blobs and then this makes the registry raise a 500 error. This makes no sense as my authentication to bucket was sufficient to write the other layers. This appears to be an issue only with the larger layers. This happens immediately after the following request:
PUT /v2/nginx/blobs/uploads/1d030deb-7a1d-4c0e-b6e7-8dd8beb50aa1
This is very frustrating because I'd like to move on to deploying images from the registry. Any help would be tremendous.
Thank you so much for any help,
Adrian
2 Replies
For anybody who reads this or for those who find themselves here (if anyone ever does) the following docker compose project will reproduce the issue.
version: '3'
services:
registry:
container_name: registry
restart: always
image: registry:2
ports:
- 5000:5000
environment:
REGISTRY_HTTP_TLS_CERTIFICATE: /certs/local.crt
REGISTRY_HTTP_TLS_KEY: /certs/local.key
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
REGISTRY_STORAGE: 's3'
REGISTRY_STORAGE_S3_ACCESSKEY: REDACTED_1234
REGISTRY_STORAGE_S3_SECRETKEY: REDACTED_1234
REGISTRY_STORAGE_S3_BUCKET: my-object-storage
REGISTRY_STORAGE_S3_SECURE: 'true'
REGISTRY_STORAGE_S3_REGION: us-lax-1
REGISTRY_STORAGE_S3_REGIONENDPOINT: us-lax-1.linodeobjects.com
volumes:
# - /path/data:/var/lib/registry
- ./certs:/certs
- ./auth:/auth
To run this (bring your own certs and htpasswd as shown in the volumes) do
docker compose up
The reproduction steps are the same as in the initial comment but with the registry name replaced:
docker pull nginx
docker login localhost:5000
docker image tag nginx localhost:5000/nginx:latest
docker push localhost:5000/nginx:latest
From this it is easy to see that traefik/kubernetes is not the source of this issue.
For anybody who reads this or for those who find themselves here (if anyone ever does) the following docker compose project will reproduce the issue.
version: '3'
services:
registry:
container_name: registry
restart: always
image: registry:2
ports:
- 5000:5000
environment:
REGISTRY_HTTP_TLS_CERTIFICATE: /certs/local.crt
REGISTRY_HTTP_TLS_KEY: /certs/local.key
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
REGISTRY_STORAGE: 's3'
REGISTRY_STORAGE_S3_ACCESSKEY: REDACTED_1234
REGISTRY_STORAGE_S3_SECRETKEY: REDACTED_1234
REGISTRY_STORAGE_S3_BUCKET: my-object-storage
REGISTRY_STORAGE_S3_SECURE: 'true'
REGISTRY_STORAGE_S3_REGION: us-lax-1
REGISTRY_STORAGE_S3_REGIONENDPOINT: us-lax-1.linodeobjects.com
volumes:
# - /path/data:/var/lib/registry
- ./certs:/certs
- ./auth:/auth
To run this (bring your own certs and htpasswd as shown in the volumes) do
docker compose up
The reproduction steps are the same as in the initial comment but with the registry name replaced:
docker pull nginx
docker login localhost:5000
docker image tag nginx localhost:5000/nginx:latest
docker push localhost:5000/nginx:latest
From this it is easy to see that traefik/kubernetes is not the source of this issue.