Infrastructure/fedora-infrastructure
1
0
Fork 0
Code Issues 70 Pull requests Projects Releases Packages Wiki Activity Actions

registry.fp.o multiarch manifest lists seem to not work #7223

New issue
Closed
opened 2018-09-05 14:32:33 +00:00 by dustymabe · 19 comments
dustymabe commented 2018-09-05 14:32:33 +00:00
Copy link
  • Describe what you need us to do:

Manifest lists don't seem to work.

[root@localhost ~]# podman pull registry.fedoraproject.org/fedora:latest                                                                                                                                                                   
Trying to pull registry.fedoraproject.org/fedora:latest...Failed
error pulling image "registry.fedoraproject.org/fedora:latest": unable to pull registry.fedoraproject.org/fedora:latest: unable to pull image, or you do not have pull access                                                              
[root@localhost ~]#
[root@localhost ~]# uname -m
aarch64
[root@localhost ~]#

pulling the explicit image seems to get farther but there might be a bug in podman for aarch64 (i'm using podman-0.8.4-1.git9f9b8cf.fc29.aarch64)

[root@localhost ~]#
[root@localhost ~]#  podman pull registry.fedoraproject.org/fedora:28-aarch64
Trying to pull registry.fedoraproject.org/fedora:28-aarch64...Getting image source signatures
Copying blob sha256:a050f587f7bc76897c77936a232ceb1db95f69c9949a8699de3eb1fa68b704c3
 84.86 MB / 84.86 MB [===================================================] 1m43s
Failed
error pulling image "registry.fedoraproject.org/fedora:28-aarch64": unable to pull registry.fedoraproject.org/fedora:28-aarch64: unable to pull image, or you do not have pull access
  • When do you need this? (YYYY/MM/DD)

iot edition would like this soonish

  • When is this no longer needed or useful? (YYYY/MM/DD)

  • If we cannot complete your request, what is the impact?

aarch64 machines don't get images easily

* Describe what you need us to do: Manifest lists don't seem to work. ``` [root@localhost ~]# podman pull registry.fedoraproject.org/fedora:latest Trying to pull registry.fedoraproject.org/fedora:latest...Failed error pulling image "registry.fedoraproject.org/fedora:latest": unable to pull registry.fedoraproject.org/fedora:latest: unable to pull image, or you do not have pull access [root@localhost ~]# [root@localhost ~]# uname -m aarch64 [root@localhost ~]# ``` pulling the explicit image seems to get farther but there might be a bug in podman for aarch64 (i'm using `podman-0.8.4-1.git9f9b8cf.fc29.aarch64`) ``` [root@localhost ~]# [root@localhost ~]# podman pull registry.fedoraproject.org/fedora:28-aarch64 Trying to pull registry.fedoraproject.org/fedora:28-aarch64...Getting image source signatures Copying blob sha256:a050f587f7bc76897c77936a232ceb1db95f69c9949a8699de3eb1fa68b704c3 84.86 MB / 84.86 MB [===================================================] 1m43s Failed error pulling image "registry.fedoraproject.org/fedora:28-aarch64": unable to pull registry.fedoraproject.org/fedora:28-aarch64: unable to pull image, or you do not have pull access ``` * When do you need this? (YYYY/MM/DD) iot edition would like this soonish * When is this no longer needed or useful? (YYYY/MM/DD) * If we cannot complete your request, what is the impact? aarch64 machines don't get images easily
dustymabe commented 2018-09-05 14:51:58 +00:00
Author
Copy link

note skopeo gives us a bit more information for registry.fedoraproject.org/fedora:latest

[root@localhost ~]# skopeo copy docker://registry.fedoraproject.org/fedora:latest dir:/tmp/boo/
FATA[0001] Error choosing an image from manifest list docker://registry.fedoraproject.org/fedora:latest: no image found in manifest list for architecture arm64, OS linux
note skopeo gives us a bit more information for `registry.fedoraproject.org/fedora:latest` ``` [root@localhost ~]# skopeo copy docker://registry.fedoraproject.org/fedora:latest dir:/tmp/boo/ FATA[0001] Error choosing an image from manifest list docker://registry.fedoraproject.org/fedora:latest: no image found in manifest list for architecture arm64, OS linux ```
cverna commented 2018-09-05 17:30:11 +00:00
Copy link

Metadata Update from @cverna:

  • Issue assigned to cverna
**Metadata Update from @cverna**: - Issue assigned to cverna
kevin commented 2018-09-05 22:08:28 +00:00
Copy link

Metadata Update from @kevin:

  • Issue priority set to: Waiting on Assignee (was: Needs Review)
**Metadata Update from @kevin**: - Issue priority set to: Waiting on Assignee (was: Needs Review)
cverna commented 2018-09-06 17:34:46 +00:00
Copy link

I have opened https://pagure.io/releng/pull-request/7773 to fix this.

I have opened https://pagure.io/releng/pull-request/7773 to fix this.
dustymabe commented 2018-09-07 01:26:21 +00:00
Author
Copy link

you asked me for this today.. here is the output. I just ran the command:

[root@localhost ~]# skopeo copy docker://registry.fedoraproject.org/fedora:rawhide dir:/tmp/coo/
FATA[0001] Error choosing an image from manifest list docker://registry.fedoraproject.org/fedora:rawhide: no image found in manifest list for architecture arm64, OS linux
you asked me for this today.. here is the output. I just ran the command: ``` [root@localhost ~]# skopeo copy docker://registry.fedoraproject.org/fedora:rawhide dir:/tmp/coo/ FATA[0001] Error choosing an image from manifest list docker://registry.fedoraproject.org/fedora:rawhide: no image found in manifest list for architecture arm64, OS linux ```
puiterwijk commented 2018-09-10 13:26:09 +00:00
Copy link

Well, doing a lookup on the registry, I am only seeing a raw manifest, and not a manifest list, for fedora:rawhide: "curl https://registry.fedoraproject.org/v2/fedora/manifests/rawhide", which only has the amd64 architecture?
i.e. it doesn't look like the multi-arch manifest list is being pushed?
I do see a bunch of tags like "28-armhfp" and "28-ppc64le" in the tag list, but still no manifest lists.

Well, doing a lookup on the registry, I am only seeing a raw manifest, and not a manifest list, for fedora:rawhide: "curl https://registry.fedoraproject.org/v2/fedora/manifests/rawhide", which only has the amd64 architecture? i.e. it doesn't look like the multi-arch manifest list is being pushed? I do see a bunch of tags like "28-armhfp" and "28-ppc64le" in the tag list, but still no manifest lists.
cverna commented 2018-09-10 13:29:52 +00:00
Copy link

Well, doing a lookup on the registry, I am only seeing a raw manifest, and not a manifest list, for fedora:rawhide: "curl https://registry.fedoraproject.org/v2/fedora/manifests/rawhide", which only has the amd64 architecture?
i.e. it doesn't look like the multi-arch manifest list is being pushed?
I do see a bunch of tags like "28-armhfp" and "28-ppc64le" in the tag list, but still no manifest lists.

You need to pass the -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" to the curl command to get the manifest list

> Well, doing a lookup on the registry, I am only seeing a raw manifest, and not a manifest list, for fedora:rawhide: "curl https://registry.fedoraproject.org/v2/fedora/manifests/rawhide", which only has the amd64 architecture? > i.e. it doesn't look like the multi-arch manifest list is being pushed? > I do see a bunch of tags like "28-armhfp" and "28-ppc64le" in the tag list, but still no manifest lists. You need to pass the -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" to the curl command to get the manifest list
puiterwijk commented 2018-09-10 13:32:58 +00:00
Copy link

Ah. Thanks.
So, then I notice that your PR made things get pushed as "arm64v8", while Dusty's error says it can't find "arm64", which to me indeed don't seem like the same string at all?

Ah. Thanks. So, then I notice that your PR made things get pushed as "arm64v8", while Dusty's error says it can't find "arm64", which to me indeed don't seem like the same string at all?
dustymabe commented 2018-09-10 13:38:57 +00:00
Author
Copy link

Ah. Thanks.
So, then I notice that your PR made things get pushed as "arm64v8", while Dusty's error says it can't find "arm64", which to me indeed don't seem like the same string at all?

It's possible the client (podman) is to blame. Does anyone know what the value should be and then we can determine if its the server or the client that is at fault?

> Ah. Thanks. > So, then I notice that your PR made things get pushed as "arm64v8", while Dusty's error says it can't find "arm64", which to me indeed don't seem like the same string at all? It's possible the client (podman) is to blame. Does anyone know what the value *should* be and then we can determine if its the server or the client that is at fault?
cverna commented 2018-09-10 13:39:41 +00:00
Copy link

I used https://github.com/docker-library/official-images/blob/master/library/fedora#L13 as an example.
I could not find a proper documentation on what the registry expect. I fine with testing "arm64" instead of "arm64v8". It would be nice to know if there are more info on the registry logs.

I used https://github.com/docker-library/official-images/blob/master/library/fedora#L13 as an example. I could not find a proper documentation on what the registry expect. I fine with testing "arm64" instead of "arm64v8". It would be nice to know if there are more info on the registry logs.
puiterwijk commented 2018-09-10 13:43:21 +00:00
Copy link

I believe most of Docker uses the GOARCH terms, which are: https://golang.org/doc/install/source#environment .
So I guess that podman defaults to that too.
However, the Docker official-images program uses arm64v8: https://github.com/docker-library/official-images#architectures-other-than-amd64 .

I believe most of Docker uses the GOARCH terms, which are: https://golang.org/doc/install/source#environment . So I guess that podman defaults to that too. However, the Docker official-images program uses arm64v8: https://github.com/docker-library/official-images#architectures-other-than-amd64 .
puiterwijk commented 2018-09-10 13:46:26 +00:00
Copy link

@cverna I just see a 200 against the manifest list, and then a 200 against the actual manifest, and that's it.
All of the decisions for which arches to support or pull are on the client-side.

@cverna I just see a 200 against the manifest list, and then a 200 against the actual manifest, and that's it. All of the decisions for which arches to support or pull are on the client-side.
puiterwijk commented 2018-09-10 13:49:55 +00:00
Copy link

(@cverna, note that this is not meant as a "you can't see the logs", I'd be totally fine with that, just giving you the info you asked for until someone takes care of your ticket for that access)

(@cverna, note that this is not meant as a "you can't see the logs", I'd be totally fine with that, just giving you the info you asked for until someone takes care of your ticket for that access)
cverna commented 2018-09-10 13:50:18 +00:00
Copy link

Let give "arm64" a try then 😄

Let give "arm64" a try then :smile:
cverna commented 2018-09-10 13:56:00 +00:00
Copy link

(@cverna, note that this is not meant as a "you can't see the logs", I'd be totally fine with that, just giving you the info you asked for until someone takes care of your ticket for that access)

Sure thanks for helping :)

> (@cverna, note that this is not meant as a "you can't see the logs", I'd be totally fine with that, just giving you the info you asked for until someone takes care of your ticket for that access) Sure thanks for helping :)
cverna commented 2018-09-10 14:06:05 +00:00
Copy link

OK so that the manifest list from the docker hub

{
'schemaVersion': 2, 
'mediaType': 'application/vnd.docker.distribution.manifest.list.v2+json', 
'manifests': [
{'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', 'size': 529, 'digest': 'sha256:6
fb84ba634fe68572a2ac99741062695db24b921d0aa72e61ee669902f88c187', 'platform': {'architecture': 'amd64', 'os': 'linux'}}, 
{'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', 'size': 529, 'digest
': 'sha256:dadbe43c2d90b68cd9e9ba7bdd9907f32842f3b93bf66d8f22e1fd12ac5cefcf', 'platform': {'architecture': 'arm', 'os': 'linux', 'variant': 'v7'}}, 
{'mediaType': 'application/vnd.docker.distribution.manifest.v2+
json', 'size': 529, 'digest': 'sha256:a2adc1621935869bcd51a84b007b3ee6aa8aa9aef0bf30a947f7312fd43e5b0b', 'platform': {'architecture': 'arm64', 'os': 'linux', 'variant': 'v8'}}, 
{'mediaType': 'application/vnd.doc
ker.distribution.manifest.v2+json', 'size': 529, 'digest': 'sha256:74026e25854714e616f46182b4f41a1b25a90a76b484e30142d073bec6b861df', 'platform': {'architecture': 'ppc64le', 'os': 'linux'}}]}

So it looks like arm and arm64 is the way to go

OK so that the manifest list from the docker hub ``` { 'schemaVersion': 2, 'mediaType': 'application/vnd.docker.distribution.manifest.list.v2+json', 'manifests': [ {'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', 'size': 529, 'digest': 'sha256:6 fb84ba634fe68572a2ac99741062695db24b921d0aa72e61ee669902f88c187', 'platform': {'architecture': 'amd64', 'os': 'linux'}}, {'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', 'size': 529, 'digest ': 'sha256:dadbe43c2d90b68cd9e9ba7bdd9907f32842f3b93bf66d8f22e1fd12ac5cefcf', 'platform': {'architecture': 'arm', 'os': 'linux', 'variant': 'v7'}}, {'mediaType': 'application/vnd.docker.distribution.manifest.v2+ json', 'size': 529, 'digest': 'sha256:a2adc1621935869bcd51a84b007b3ee6aa8aa9aef0bf30a947f7312fd43e5b0b', 'platform': {'architecture': 'arm64', 'os': 'linux', 'variant': 'v8'}}, {'mediaType': 'application/vnd.doc ker.distribution.manifest.v2+json', 'size': 529, 'digest': 'sha256:74026e25854714e616f46182b4f41a1b25a90a76b484e30142d073bec6b861df', 'platform': {'architecture': 'ppc64le', 'os': 'linux'}}]} ``` So it looks like `arm` and `arm64` is the way to go
cverna commented 2018-09-10 17:26:57 +00:00
Copy link

So now it is picking the correct image but there is an issue with the certificates.

sudo docker -D pull registry.fedoraproject.org/fedora:rawhide
Trying to pull repository registry.fedoraproject.org/fedora ...
sha256:d1df13939e52b24fef32271b8f1543e90c38a1a47e84c859db45d437e463d0f3: Pulling from registry.fedoraproject.org/fedora                                                                                           
0fe73490aedd: Pulling fs layer
error pulling image configuration: Get https://cdn.registry.fedoraproject.org/v2/fedora/blobs/sha256:d00f2170d181b5605ef7b165a88be0a67200dd8f2878811df76572338d86f974: x509: certificate has expired or is not yet
valid

The same command works just fine on x86_64.

So now it is picking the correct image but there is an issue with the certificates. ``` sudo docker -D pull registry.fedoraproject.org/fedora:rawhide Trying to pull repository registry.fedoraproject.org/fedora ... sha256:d1df13939e52b24fef32271b8f1543e90c38a1a47e84c859db45d437e463d0f3: Pulling from registry.fedoraproject.org/fedora 0fe73490aedd: Pulling fs layer error pulling image configuration: Get https://cdn.registry.fedoraproject.org/v2/fedora/blobs/sha256:d00f2170d181b5605ef7b165a88be0a67200dd8f2878811df76572338d86f974: x509: certificate has expired or is not yet valid ``` The same command works just fine on x86_64.
cverna commented 2018-09-10 17:39:25 +00:00
Copy link

So now it is picking the correct image but there is an issue with the certificates.
sudo docker -D pull registry.fedoraproject.org/fedora:rawhide
Trying to pull repository registry.fedoraproject.org/fedora ...
sha256:d1df13939e52b24fef32271b8f1543e90c38a1a47e84c859db45d437e463d0f3: Pulling from registry.fedoraproject.org/fedora
0fe73490aedd: Pulling fs layer
error pulling image configuration: Get https://cdn.registry.fedoraproject.org/v2/fedora/blobs/sha256:d00f2170d181b5605ef7b165a88be0a67200dd8f2878811df76572338d86f974: x509: certificate has expired or is not yet
valid

The same command works just fine on x86_64.

It was just an issue with the date of the aarch64 box I was doing the testing from :D. So this should work now, it can be tested using the rawhide image.

I will coordinate with releng to do a base image release this week.

> So now it is picking the correct image but there is an issue with the certificates. > sudo docker -D pull registry.fedoraproject.org/fedora:rawhide > Trying to pull repository registry.fedoraproject.org/fedora ... > sha256:d1df13939e52b24fef32271b8f1543e90c38a1a47e84c859db45d437e463d0f3: Pulling from registry.fedoraproject.org/fedora > 0fe73490aedd: Pulling fs layer > error pulling image configuration: Get https://cdn.registry.fedoraproject.org/v2/fedora/blobs/sha256:d00f2170d181b5605ef7b165a88be0a67200dd8f2878811df76572338d86f974: x509: certificate has expired or is not yet > valid > > > > The same command works just fine on x86_64. It was just an issue with the date of the aarch64 box I was doing the testing from :D. So this should work now, it can be tested using the rawhide image. I will coordinate with releng to do a base image release this week.
cverna commented 2018-09-10 17:40:03 +00:00
Copy link

Metadata Update from @cverna:

  • Issue close_status updated to: Fixed
  • Issue status updated to: Closed (was: Open)
**Metadata Update from @cverna**: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
announcement

   
authentication

Authentication related issues

  
automate

   
aws

Issues related to Amazon Web Services

  
backlog

Tasks that will take longer then ~4 hours to do

  
blocked

Issue is blocked

  
bodhi

Issues with Bodhi

  
ci

Continuous Integration

  
Closed As
Duplicate
Closed with the reason of Duplicate

  
Closed As
Fixed
Closed with the reason of Fixed

  
Closed As
Fixed with Explanation
Closed with the reason of Fixed with Explanation

  
Closed As
Initiative Worthy
Closed with the reason of Initiative Worthy

  
Closed As
Insufficient data
Closed with the reason of Insufficient data

  
Closed As
Invalid
Closed with the reason of Invalid

  
Closed As
Spam
Closed with the reason of Spam

  
Closed As
Upstream
Closed with the reason of Upstream

  
Closed As/Will Not
Can Not fix
Closed with the reason of Will Not/Can Not fix

  
cloud

Items in FedoraInfraCloud or COmmunishift

  
communishift

issues/questions about os.edorainfracloud.org ( Community OpenShift)

  
copr

COPR system (run by other team)

  
database

Issues related to databases

  
deprecated

Tools which Infrastructure may not be able to fix

  
dev

Tasks requiring some dev work

  
discourse

outsourced tool

  
dns

Changes or problems with DNS

  
downloads

rsync,dl.fp.o,AWS,torrents

  
easyfix

   
epel

Issues relating to epel

  
factory2

Generic issues for the factory2 team

  
firmitas

   
gitlab

   
greenwave

Issues with greenwave

  
hardware

Problem is with Fedora Infrastructure Owned/Leased Hardware

  
help wanted

   
high-gain

task that gets us a lot / is important / makes many people happy

  
high-trouble

task that is complex/long/difficult

  
iad2

Something to do with the iad2 datacenter

  
koji

Issues with the build system

  
koschei

Items needing Koschei team

  
lists

issues with lists

  
low-gain

tasks that gets us little / isn't very important / makes only 1 person or few happy

  
low-trouble

task that is easy/short

  
mbs

Issues with mbs.fedoraproject.org

  
medium-gain

task that gets us some / is somewhat important / makes some people happy

  
medium-trouble

task that is medium complex / difficult

  
mini-initiative

Something that is bigger than one person but not a full initiative

  
mirrorlists

Way to tell systems where downloads are

  
monitoring

   
Needs investigation

   
notifier

   
odcs

On Demand Compose Service issues

  
OpenShift

Issues with OpenShift

  
ops

ops problem now...

  
OSBS

OpenShift Build System issues in Fedora

  
outage

   
packager_workflow_blocker

Blocks the packager workflow

  
pagure

Issues with pagure.io

  
permissions

Items which need permissions set in outside tools

  
Priority
Needs Review
Priority of Needs Review

  
Priority
Next Meeting
Priority of Next Meeting

  
Priority
🔥 URGENT 🔥
Priority of 🔥 URGENT 🔥

  
Priority
Waiting on Assignee
Priority of Waiting on Assignee

  
Priority
Waiting on External
Priority of Waiting on External

  
Priority
Waiting on Reporter
Priority of Waiting on Reporter

  
rabbitmq

issues with the message broker

  
rdu-cc

Items doing with RDU Community Cage

  
release-monitoring

release-monitoring.org (Anitya)

  
releng

For problems which should be in the releng ticket system

  
repoSpanner

Issues with repoSpanner

  
request-for-resources

   
s390x

The secondary architecture hardware and problems related with it

  
security

   
SMTP

Email issues outside of lists (DMARC, SPF, aliases)

  
src.fp.o

Issues related to src.fedoraproject.org

  
staging

Issues in staging

  
taiga

outsourced tool

  
unfreeze

Patches to be applied after freeze is lifted

  
waiverdb

Issues with waiverdb

  
websites-general

General items with a website focus

  
wiki

Issues with the Wiki

No labels
announcement
authentication
automate
aws
backlog
blocked
bodhi
ci
Closed As
Duplicate
Closed As
Fixed
Closed As
Fixed with Explanation
Closed As
Initiative Worthy
Closed As
Insufficient data
Closed As
Invalid
Closed As
Spam
Closed As
Upstream
Closed As/Will Not
Can Not fix
cloud
communishift
copr
database
deprecated
dev
discourse
dns
downloads
easyfix
epel
factory2
firmitas
gitlab
greenwave
hardware
help wanted
high-gain
high-trouble
iad2
koji
koschei
lists
low-gain
low-trouble
mbs
medium-gain
medium-trouble
mini-initiative
mirrorlists
monitoring
Needs investigation
notifier
odcs
OpenShift
ops
OSBS
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
rdu-cc
release-monitoring
releng
repoSpanner
request-for-resources
s390x
security
SMTP
src.fp.o
staging
taiga
unfreeze
waiverdb
websites-general
wiki
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Infrastructure/fedora-infrastructure#7223
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?