infrastructure/fedora-infrastructure

Fork 0

Koji builder upgrade to Fedora 31 breaks MBS #8410

New issue

Closed

opened 2019-11-25 11:37:56 +00:00 by mbooth · 23 comments

mbooth commented

2019-11-25 11:37:56 +00:00

I originally filed this problem against MBS here: https://pagure.io/fm-orchestrator/issue/1525

But it occurs to me that it could coincide with recent infra changes.

@mizdebsk pointed out to me that the failures are happening only on build hosts that are upgraded to Fedora > 29 and since I was only seeing the problem from late last week, this might coincide with the change to infra: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=a816316

It feels pretty urgent because I have so far not managed to build any archful modular packages since last week.

I originally filed this problem against MBS here: https://pagure.io/fm-orchestrator/issue/1525 But it occurs to me that it could coincide with recent infra changes. @mizdebsk pointed out to me that the failures are happening only on build hosts that are upgraded to Fedora > 29 and since I was only seeing the problem from late last week, this might coincide with the change to infra: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=a816316 It feels pretty urgent because I have so far not managed to build any archful modular packages since last week.

mizdebsk commented

2019-11-25 11:39:40 +00:00

Metadata Update from @mizdebsk:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: koji, mbs

**Metadata Update from @mizdebsk**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: koji, mbs

mizdebsk commented

2019-11-25 11:43:06 +00:00

On Fedora 31 mock chroot fails to initialize with the following DNF error:

No available modular metadata for modular package 'module-build-macros-0.1-1.module_f30+7100+0fc9a5b6.noarch', it cannot be installed on the system

On Fedora 29 mock chroot initializes fine.

Fedora 29 builders have:

dnf-4.2.5-5.fc29.noarch
libdnf-0.31.0-10.fc29.x86_64

Fedora 31 builders have:

dnf-4.2.15-2.fc31.noarch
libdnf-0.37.2-2.fc31.x86_64

On Fedora 31 mock chroot fails to initialize with the following DNF error: `No available modular metadata for modular package 'module-build-macros-0.1-1.module_f30+7100+0fc9a5b6.noarch', it cannot be installed on the system` On Fedora 29 mock chroot initializes fine. Fedora 29 builders have: - dnf-4.2.5-5.fc29.noarch - libdnf-0.31.0-10.fc29.x86_64 Fedora 31 builders have: - dnf-4.2.15-2.fc31.noarch - libdnf-0.37.2-2.fc31.x86_64

mizdebsk commented

2019-11-25 12:17:06 +00:00

fail-safe mechanism of DNF 4.2.7 is causing this. Koji build repos lack modular metadata, but DNF sees that module-build-macros is a modular package and fails to install it.

[fail-safe mechanism](https://bugzilla.redhat.com/show_bug.cgi?id=1616167) of DNF 4.2.7 is causing this. Koji build repos lack modular metadata, but DNF sees that module-build-macros is a modular package and fails to install it.

mizdebsk commented

2019-11-25 12:37:38 +00:00

I've temporary disabled F29 staging builders to ease reproducing this issue in staging Koji:

Mon Nov 25 13:35:06 2019 host configuration for buildvm-05.stg.phx2.fedoraproject.org altered by mizdebsk
    enabled: True -> False
Mon Nov 25 13:35:06 2019 host configuration for buildvm-03.stg.phx2.fedoraproject.org altered by mizdebsk
    enabled: True -> False
Mon Nov 25 13:35:24 2019 host configuration for buildvm-aarch64-01.stg.arm.fedoraproject.org altered by mizdebsk
    enabled: True -> False
Mon Nov 25 13:35:24 2019 host configuration for buildvm-armv7-01.stg.arm.fedoraproject.org altered by mizdebsk
    enabled: True -> False

I've temporary disabled F29 staging builders to ease reproducing this issue in staging Koji: ``` Mon Nov 25 13:35:06 2019 host configuration for buildvm-05.stg.phx2.fedoraproject.org altered by mizdebsk enabled: True -> False Mon Nov 25 13:35:06 2019 host configuration for buildvm-03.stg.phx2.fedoraproject.org altered by mizdebsk enabled: True -> False Mon Nov 25 13:35:24 2019 host configuration for buildvm-aarch64-01.stg.arm.fedoraproject.org altered by mizdebsk enabled: True -> False Mon Nov 25 13:35:24 2019 host configuration for buildvm-armv7-01.stg.arm.fedoraproject.org altered by mizdebsk enabled: True -> False ```

mizdebsk commented

2019-11-25 12:48:20 +00:00

The issue is reproducible in staging Koji, but only after I upgraded DNF to latest 4.2.15 as follows:

buildvm-01.stg.phx2.fedoraproject.org | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    }, 
    "changed": true, 
    "msg": "", 
    "rc": 0, 
    "results": [
        "Installed: dnf", 
        "Installed: sqlite-3.30.0-1.fc31.x86_64", 
        "Installed: yum-4.2.15-2.fc31.noarch", 
        "Installed: python3-hawkey-0.37.2-2.fc31.x86_64", 
        "Installed: libdnf-0.37.2-2.fc31.x86_64", 
        "Installed: python3-libdnf-0.37.2-2.fc31.x86_64", 
        "Installed: librepo-1.11.0-1.fc31.x86_64", 
        "Installed: dnf-4.2.15-2.fc31.noarch", 
        "Installed: dnf-automatic-4.2.15-2.fc31.noarch", 
        "Installed: dnf-data-4.2.15-2.fc31.noarch", 
        "Installed: python3-librepo-1.11.0-1.fc31.x86_64", 
        "Installed: python3-dnf-4.2.15-2.fc31.noarch", 
        "Removed: yum-4.2.9-5.fc31.noarch", 
        "Removed: libdnf-0.35.3-6.fc31.x86_64", 
        "Removed: dnf-4.2.9-5.fc31.noarch", 
        "Removed: librepo-1.10.5-1.fc31.x86_64", 
        "Removed: dnf-automatic-4.2.9-5.fc31.noarch", 
        "Removed: dnf-data-4.2.9-5.fc31.noarch", 
        "Removed: python3-dnf-4.2.9-5.fc31.noarch", 
        "Removed: python3-hawkey-0.35.3-6.fc31.x86_64", 
        "Removed: python3-libdnf-0.35.3-6.fc31.x86_64", 
        "Removed: python3-librepo-1.10.5-1.fc31.x86_64"
    ]
}

The same component build succeeded with older DNF 4.2.9-5

The issue is [reproducible in staging Koji](https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90208181), but only after I upgraded DNF to latest 4.2.15 as follows: ``` buildvm-01.stg.phx2.fedoraproject.org | CHANGED => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": true, "msg": "", "rc": 0, "results": [ "Installed: dnf", "Installed: sqlite-3.30.0-1.fc31.x86_64", "Installed: yum-4.2.15-2.fc31.noarch", "Installed: python3-hawkey-0.37.2-2.fc31.x86_64", "Installed: libdnf-0.37.2-2.fc31.x86_64", "Installed: python3-libdnf-0.37.2-2.fc31.x86_64", "Installed: librepo-1.11.0-1.fc31.x86_64", "Installed: dnf-4.2.15-2.fc31.noarch", "Installed: dnf-automatic-4.2.15-2.fc31.noarch", "Installed: dnf-data-4.2.15-2.fc31.noarch", "Installed: python3-librepo-1.11.0-1.fc31.x86_64", "Installed: python3-dnf-4.2.15-2.fc31.noarch", "Removed: yum-4.2.9-5.fc31.noarch", "Removed: libdnf-0.35.3-6.fc31.x86_64", "Removed: dnf-4.2.9-5.fc31.noarch", "Removed: librepo-1.10.5-1.fc31.x86_64", "Removed: dnf-automatic-4.2.9-5.fc31.noarch", "Removed: dnf-data-4.2.9-5.fc31.noarch", "Removed: python3-dnf-4.2.9-5.fc31.noarch", "Removed: python3-hawkey-0.35.3-6.fc31.x86_64", "Removed: python3-libdnf-0.35.3-6.fc31.x86_64", "Removed: python3-librepo-1.10.5-1.fc31.x86_64" ] } ``` The same component build [succeeded](https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90208117) with older DNF 4.2.9-5

mprahl commented

2019-11-25 13:22:59 +00:00

@mizdebsk we had the same problem for MBS local builds and the following PR resolved it:
https://pagure.io/fm-orchestrator/pull-request/1494

Is there a way that MBS could configure DNF in Koji to use module_hotfixes=true when resolving dependencies?

@mizdebsk we had the same problem for MBS local builds and the following PR resolved it: https://pagure.io/fm-orchestrator/pull-request/1494 Is there a way that MBS could configure DNF in Koji to use `module_hotfixes=true` when resolving dependencies?

mbooth commented

2019-11-25 13:27:24 +00:00

Author

So it is because the repo generated by koji does not contain modularity metadata?

Here is what I do when I test build locally. After building module-build-macros, and then after the completion of each rank of components I run this script: https://github.com/mbooth101/modularity-misc/blob/master/update_repo.py

It regenerates the repo and updates the modulemd metadata to include all the latest RPM artifacts -- I'm surprised this is not what is happening in MBS/koji; I guess there is no mechanism for MBS to customise the intermediate repositories that are generated by koji?

So it is because the repo generated by koji does not contain modularity metadata? Here is what I do when I test build locally. After building module-build-macros, and then after the completion of each rank of components I run this script: https://github.com/mbooth101/modularity-misc/blob/master/update_repo.py It regenerates the repo and updates the modulemd metadata to include all the latest RPM artifacts -- I'm surprised this is not what is happening in MBS/koji; I guess there is no mechanism for MBS to customise the intermediate repositories that are generated by koji?

mizdebsk commented

2019-11-25 13:28:29 +00:00

So it is because the repo generated by koji does not contain modularity metadata?

Correct. Koji repos don't contain modular metadata.

> So it is because the repo generated by koji does not contain modularity metadata? Correct. Koji repos don't contain modular metadata.

mizdebsk commented

2019-11-25 13:34:00 +00:00

Metadata Update from @mizdebsk:

Issue tagged with: factory2

**Metadata Update from @mizdebsk**: - Issue tagged with: factory2

jkaluza commented

2019-11-25 13:37:21 +00:00

@mprahl, @mizdebsk , we should set:

KOJI_TAG_EXTRA_OPTS = {
  "mock.package_manager": "dnf"
  "repo_include_all": True
  "mock.new_chroot": 0
  "mock.yum.module_hotfixes": 1
}

in the roles/mbs/common/templates/config.py and run the playbook. That should fix it.

@mprahl, @mizdebsk , we should set: ``` KOJI_TAG_EXTRA_OPTS = { "mock.package_manager": "dnf" "repo_include_all": True "mock.new_chroot": 0 "mock.yum.module_hotfixes": 1 } ``` in the `roles/mbs/common/templates/config.py` and run the playbook. That should fix it.

jkaluza commented

2019-11-25 13:37:21 +00:00

Metadata Update from @jkaluza:

Issue untagged with: factory2, koji, mbs

**Metadata Update from @jkaluza**: - Issue **un**tagged with: factory2, koji, mbs

jkaluza commented

2019-11-25 13:38:46 +00:00

Also note that if we ever start using modular RPMs in non-modular buildroots (for example when f31-build includes modular RPMs, because thy would be default for f31), then f31-build also needs to set mock.yum.module_hotfixes: 1.

jkaluza commented

2019-11-25 13:39:12 +00:00

Metadata Update from @jkaluza:

Issue tagged with: factory2, koji, mbs

**Metadata Update from @jkaluza**: - Issue tagged with: factory2, koji, mbs

mizdebsk commented

2019-11-25 13:41:01 +00:00

I already tested module_hotfixes=1 and it didn't work. I will try again.

I already tested `module_hotfixes=1` and it didn't work. I will try again.

mprahl commented

2019-11-25 13:41:51 +00:00

@jkaluza thanks. I'll give that a try.

Edit: Nevermind. It looks like @mizdebsk is already on it.

@jkaluza thanks. I'll give that a try. Edit: Nevermind. It looks like @mizdebsk is already on it.

mizdebsk commented

2019-11-25 13:48:26 +00:00

I've added KOJI_TAG_EXTRA_OPTS to staging MBS config and I'm running testmodule build to check whether it fixes the issue.

mizdebsk commented

2019-11-25 14:01:00 +00:00

Thanks @jkaluza , it looks like module_hotfixes=1 fixed the issue in staging. I'll apply the fix in production too.

Thanks @jkaluza , it looks like `module_hotfixes=1` fixed the issue in staging. I'll apply the fix in production too.

mizdebsk commented

2019-11-25 14:07:58 +00:00

I've applied the fix proposed by @jkaluza in production. @mbooth, Please confirm whether the issue is fixed.

mizdebsk commented

2019-11-25 14:07:59 +00:00

Metadata Update from @mizdebsk:

Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)

**Metadata Update from @mizdebsk**: - Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)

mbooth commented

2019-11-25 14:53:16 +00:00

Author

Hey all, it seems to be behaving for me now, many thanks.

My random failures are now limited to s390x as is tradition :-)

Hey all, it seems to be behaving for me now, many thanks. My random failures are now limited to s390x [as is tradition](https://pagure.io/koji/issue/1418) :-)

mizdebsk commented

2019-11-25 14:55:52 +00:00

Thanks for confirmation. I'm closing this issue as fixed then.
I have re-enabled Fedora 29 builders in staging as investigation is over.

Thanks for confirmation. I'm closing this issue as fixed then. I have re-enabled Fedora 29 builders in staging as investigation is over.

mizdebsk commented

2019-11-25 14:55:53 +00:00

Metadata Update from @mizdebsk: