tar errors out when creating hard links (aarch64 flatpak issue) #9532
Labels
No labels
announcement
authentication
automate
aws
backlog
blocked
bodhi
ci
Closed As
Duplicate
Closed As
Fixed
Closed As
Fixed with Explanation
Closed As
Initiative Worthy
Closed As
Insufficient data
Closed As
Invalid
Closed As
Spam
Closed As
Upstream
Closed As
Will Not or Can Not fix
cloud
communishift
copr
database
deprecated
dev
discourse
dns
downloads
easyfix
epel
factory2
firmitas
gitlab
greenwave
hardware
help wanted
high-gain
high-trouble
iad2
koji
koschei
lists
low-gain
low-trouble
mbs
medium-gain
medium-trouble
mini-initiative
mirrorlists
monitoring
Needs investigation
notifier
odcs
OpenShift
ops
OSBS
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
rdu-cc
release-monitoring
releng
repoSpanner
request-for-resources
s390x
security
SMTP
src.fp.o
staging
taiga
unfreeze
waiverdb
websites-general
wiki
No milestone
No project
No assignees
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: Infrastructure/fedora-infrastructure#9532
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
I ran into an issue with aarch64 flatpaks where tar is erroring out when creating hard links. The koji task is https://koji.fedoraproject.org/koji/taskinfo?taskID=57503246 and the matching x86_64 build went fine. aarch64 reproducibly errors out with:
I am not sure what's going on: could it be something to do with how the file system is mounted that causes tar to error out? Is it a file system that doesn't support hard links (or doesn't support hard links that point to a different directory)?
Metadata Update from @mobrien:
I don't know enough about how OSBS (Open Shift Build System) composes flatpacks for any architecture. @otaylor can you give some pointed on what to look at?
Metadata Update from @smooge:
I'm just going to have a look at this now to see if it was caused by the installation but my knowledge of flatpaks is limited so any assistance is appreciated
Thanks! I'm not very knowledgeable how atomic reactor works, but when looking through the two logs (x86_64.log that succeeded and aarch64.log that failed) it stands out that the failed one says that docker uses:
"Driver": "overlay2",
and the one that succeeds says:
Driver": "devicemapper",
I don't know if that's what causing the issue, but might be related? Could it be that overlayfs doesn't support all the hard link features that Fedora rpm contents uses?
@kalev that is likely the issue, we came to a similar conclusion independently. Not sure if this is something we can fix easily. I'll do some investigation
There is a reasonably high amount of work involved in changing it so I wouldn't be able to promise the change anytime soon. Its something that would need to be discussed as it would probably need an outage as well
Fair enough, thanks. I'll do some tests in the mean time and try to figure out how many flatpaks are affected.
I've found two more flatpaks that fail the same way: gnome-music and gedit. That is out of ~30 I've tried to build so far. It should be easy enough to work this around in individual flatpak container.yaml by adding
... so it's not super urgent to get the infra side fixed :)
Thanks for looking into this @kalev hopefully we will get it resolved in the new year
Talking about this with kalev, we figured out what is going wrong - it's a question of how flatpak-module-tools manipulates python tarfile.TarInfo headers when rewriting paths. If a path goes from > 100 to < 100 bytes, then the member.name field is changed, but member.pax_headers is left untouched and ends up overriding the name.
I'm not sure yet why we're seeing this:
But I do know why it it's not happening with local builds - the paths we rewrite in that case are different and shorter. In any case, I'll come up with a fixed version of flatpak-module-tools and that should resolve this - nothing to do with filesystem drivers :-)
Probably this is actually "only with Python-3.8 and newer" - in Python-3.8, the default format for the tarfile module was changed from GNU_FORMAT to PAX_FORMAT
OK, fixed the bug upstream, built a new version for Fedora, submitted f32/f33 updates:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-c4773405c8
When you have a chance, please update the buildroot to contain this python3-flatpak-module-tools - hopefully that will resolve this issue.
@otaylor I've updated the buildroot container on the aarch64 builders with the updated version of python3-flatpak-module-tools. Let me know if it solves the issue.
https://koji.fedoraproject.org/koji/buildinfo?buildID=1660933 completed successfully on aarch64, so I think we can consider this fixed.
Could we get this updated for x86_64 as well? #9538 and #9542 are manifestations of the same issue on x86_64.
Ok great! I have added the change to the x86_64 nodes now too.
Metadata Update from @mobrien:
Issue status updated to: Open (was: Closed)
Issue status updated to: Closed (was: Open)
Issue close_status updated to: Fixed