Infrastructure/arc
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

fix parsing errors and sphinx warnings

Browse source 
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
This commit is contained in:
Ryan Lercho 2023-11-16 08:02:56 +10:00 • committed by zlopez
parent 8fb9b2fdf0
commit ba720c3d77
98 changed files with 4799 additions and 4788 deletions
Download patch file Download diff file Expand all files Collapse all files

321
docs/pdc/endpoints.rst
View file

@ -1,195 +1,194 @@
Endpoints
=========
Based on the users we identified, we have identified the endpoints our scripts and services are actually using.
Based on the users we identified, we have identified the endpoints our scripts and
services are actually using.
/rest_api/v1/global-components/?name={{name}}
-----------------------
---------------------------------------------
::
.. code-block::
{
"dist_git_path": null, # dont use these
"dist_git_web_url": "https://src.fedoraproject.org/cgit/rpms/gnupg2.git", # generated from name
"id": 20,
"labels": [], # we don't use these
"name": "gnupg2",
"upstream": null # we don't use this
}
{
"dist_git_path": null, # dont use these
"dist_git_web_url": "https://src.fedoraproject.org/cgit/rpms/gnupg2.git", # generated from name
"id": 20,
"labels": [], # we don't use these
"name": "gnupg2",
"upstream": null # we don't use this
}
/rest_api/v1/component-branches/
--------------------
--------------------------------
::
.. code-block::
{
"active": true, # I don't know where do we store this?
"critical_path": false,
"global_component": "0ad",
"id": 501964,
"name": "f34",
"slas": [
{
"eol": "2022-03-08",
"id": 1003101,
"sla": "security_fixes"
},
{
"eol": "2022-03-08",
"id": 1003102,
"sla": "bug_fixes"
}
],
"type": "rpm" # | rpm | container | zip | iso | composite | module | flatpak
{
"active": true, # I don't know where do we store this?
"critical_path": false,
"global_component": "0ad",
"id": 501964,
"name": "f34",
"slas": [
{
"eol": "2022-03-08",
"id": 1003101,
"sla": "security_fixes"
},
{
"eol": "2022-03-08",
"id": 1003102,
"sla": "bug_fixes"
}
],
"type": "rpm" # | rpm | container | zip | iso | composite | module | flatpak
}
rest_api/v1/component-branch-slas/
----------------------
----------------------------------
Couldn't find the shape without authenticating
rest_api/v1/compose-images/
---------------
---------------------------
stores `composeinfo.json` and `image-manifest.json`
gives ability to retrieve list of images in specified compose
stores `composeinfo.json` and `image-manifest.json` gives ability to retrieve list
of images in specified compose
::
.. code-block::
{
"header": {
"type": "productmd.images",
"version": "1.2"
},
"payload": {
"compose": {
"date": "20160416",
"id": "Fedora-Rawhide-20160416.n.0",
"respin": 0,
"type": "nightly"
{
"header": {
"type": "productmd.images",
"version": "1.2"
},
"images": {
"Workstation": {
"x86_64": [
{
"arch": "x86_64",
"bootable": true,
"checksums": {
"sha256": "a7cbd69cc5c96dbf0e0f39d29b09c27844af677a602f512672527a5f7a91000c"
},
"disc_count": 1,
"disc_number": 1,
"format": "iso",
"implant_md5": "ea67ef79f3374d91735008148de4146e",
"mtime": 1460789082,
"path": "Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-Rawhide-20160416.n.0.iso",
"size": 426770432,
"subvariant": "Workstation",
"type": "boot",
"volume_id": "Fedora-WS-dvd-x86_64-rawh"
}
]
"payload": {
"compose": {
"date": "20160416",
"id": "Fedora-Rawhide-20160416.n.0",
"respin": 0,
"type": "nightly"
},
"images": {
"Workstation": {
"x86_64": [
{
"arch": "x86_64",
"bootable": true,
"checksums": {
"sha256": "a7cbd69cc5c96dbf0e0f39d29b09c27844af677a602f512672527a5f7a91000c"
},
"disc_count": 1,
"disc_number": 1,
"format": "iso",
"implant_md5": "ea67ef79f3374d91735008148de4146e",
"mtime": 1460789082,
"path": "Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-Rawhide-20160416.n.0.iso",
"size": 426770432,
"subvariant": "Workstation",
"type": "boot",
"volume_id": "Fedora-WS-dvd-x86_64-rawh"
}
]
}
}
}
}
}
/rest_api/v1/releases
-----------------------
/rest_api/v1/releases
---------------------
::
.. code-block::
{
"active": false,
"allow_buildroot_push": false,
"allowed_debuginfo_services": [],
"allowed_push_targets": [],
"base_product": null,
"bugzilla": null,
"compose_set": [],
"dist_git": {
"branch": "f25"
{
"active": false,
"allow_buildroot_push": false,
"allowed_debuginfo_services": [],
"allowed_push_targets": [],
"base_product": null,
"bugzilla": null,
"compose_set": [],
"dist_git": {
"branch": "f25"
},
"integrated_with": null,
"name": "Fedora Updates",
"product_version": "fedora-25",
"release_id": "fedora-25-updates",
"release_type": "updates",
"short": "fedora",
"sigkey": null,
"version": "25"
},
"integrated_with": null,
"name": "Fedora Updates",
"product_version": "fedora-25",
"release_id": "fedora-25-updates",
"release_type": "updates",
"short": "fedora",
"sigkey": null,
"version": "25"
},
rest_api/v1/rpms
---------------
::
{
"count": 12473109,
"next": "https://pdc.fedoraproject.org/rest_api/v1/rpms/?page=2",
"previous": null,
"results": [
{
"arch": "x86_64",
"built_for_release": null,
"dependencies": {
"conflicts": [],
"obsoletes": [],
"provides": [],
"recommends": [],
"requires": [],
"suggests": []
},
"epoch": 0,
"filename": "readline-6.3-7.fc24.x86_64.rpm",
"id": 20,
"linked_composes": [
"Fedora-24-20160202.n.0",
"Fedora-24-20160203.n.0",
"Fedora-24-20160203.n.1",
"Fedora-24-20160204.n.0"
],
"linked_releases": [],
"name": "readline",
"release": "7.fc24",
"srpm_commit_branch": null,
"srpm_commit_hash": null,
"srpm_name": "readline",
"srpm_nevra": "readline-0:6.3-7.fc24.src",
"version": "6.3"
} ] }
/rest_api/v1/product-versions
----------------
::
{
"active": true,
"allowed_push_targets": [],
"name": "fedora 34",
"product": "fedora",
"product_version_id": "fedora-34",
"releases": [
"fedora-34"
],
"short": "fedora",
"version": "34"
},
{
"active": true,
"allowed_push_targets": [],
"name": "fedora rawhide",
"product": "fedora",
"product_version_id": "fedora-rawhide",
"releases": [
"fedora-Rawhide",
"fedora-modular-Rawhide"
],
"short": "fedora",
"version": "rawhide"
}
.. code-block::
{
"count": 12473109,
"next": "https://pdc.fedoraproject.org/rest_api/v1/rpms/?page=2",
"previous": null,
"results": [
{
"arch": "x86_64",
"built_for_release": null,
"dependencies": {
"conflicts": [],
"obsoletes": [],
"provides": [],
"recommends": [],
"requires": [],
"suggests": []
},
"epoch": 0,
"filename": "readline-6.3-7.fc24.x86_64.rpm",
"id": 20,
"linked_composes": [
"Fedora-24-20160202.n.0",
"Fedora-24-20160203.n.0",
"Fedora-24-20160203.n.1",
"Fedora-24-20160204.n.0"
],
"linked_releases": [],
"name": "readline",
"release": "7.fc24",
"srpm_commit_branch": null,
"srpm_commit_hash": null,
"srpm_name": "readline",
"srpm_nevra": "readline-0:6.3-7.fc24.src",
"version": "6.3"
} ] }
/rest_api/v1/product-versions
-----------------------------
.. code-block::
{
"active": true,
"allowed_push_targets": [],
"name": "fedora 34",
"product": "fedora",
"product_version_id": "fedora-34",
"releases": [
"fedora-34"
],
"short": "fedora",
"version": "34"
},
{
"active": true,
"allowed_push_targets": [],
"name": "fedora rawhide",
"product": "fedora",
"product_version_id": "fedora-rawhide",
"releases": [
"fedora-Rawhide",
"fedora-modular-Rawhide"
],
"short": "fedora",
"version": "rawhide"
}

83
docs/pdc/index.rst
View file

@ -1,23 +1,26 @@
PDC replacement research
========================
`PDC <https://pdc.fedoraproject.org/>`_ is repository and API for storing and querying metadata related to packages and product releases.
We rely on information in it to produce Fedora releases, manage retirement of packages and more.
`PDC <https://pdc.fedoraproject.org/>`_ is repository and API for storing and querying
metadata related to packages and product releases. We rely on information in it to
produce Fedora releases, manage retirement of packages and more.
Our current deployment running on https://pdc.fedoraproject.org/.
It uses https://github.com/product-definition-center/product-definition-center/tree/python-pdc-1.9.0-2
It uses
https://github.com/product-definition-center/product-definition-center/tree/python-pdc-1.9.0-2
The software is orphaned by it's original maintainers with version requiring EOL version of python and EOL version of Django Framework,
this means we need to upgrade it or replace it.
The software is orphaned by it's original maintainers with version requiring EOL version
of python and EOL version of Django Framework, this means we need to upgrade it or
replace it.
Abstract take-away
------------------
There is no silver bullet, it will require development effort
Responsibility for retirement and sla's of packages could be moved to pagure-dist_git
Tesponsibility for critical-path and releases can be moved to Bodhi
Nightly compose metadata requires new application, that would be much simpler than current PDC
There is no silver bullet, it will require development effort Responsibility for
retirement and sla's of packages could be moved to pagure-dist_git Tesponsibility for
critical-path and releases can be moved to Bodhi Nightly compose metadata requires new
application, that would be much simpler than current PDC
.. toctree::
:maxdepth: 1
@ -29,28 +32,31 @@ Nightly compose metadata requires new application, that would be much simpler th
Previous work
-------------
There has been `a proposal by Clement Verna <https://docs.google.com/document/d/1agV5YRtDvqE2UxLl9f6E_l0iukRkv7F4qCXAx4PfFng/edit#>`_
with much of the usecases already mapped out, and `a POC in fedora-infra/fpdc repo <https://github.com/fedora-infra/fpdc/tree/master/fpdc>`_ ,
but it was left for more pressing work.
There has been `a proposal by Clement Verna
<https://docs.google.com/document/d/1agV5YRtDvqE2UxLl9f6E_l0iukRkv7F4qCXAx4PfFng/edit#>`_
with much of the usecases already mapped out, and `a POC in fedora-infra/fpdc repo
<https://github.com/fedora-infra/fpdc/tree/master/fpdc>`_ , but it was left for more
pressing work.
Current users of PDC
--------------------
Based on conversations with coleagues from Releng and Fedora QA,
and cursory investigation of our repositories we use pdc API or through CLI client in:
Based on conversations with coleagues from Releng and Fedora QA, and cursory
investigation of our repositories we use pdc API or through CLI client in:
- releng scripts
- fedpkg
- fedfind
- bodhi
- pagure
- modulebuildservice
- modulebuildservice
- mirrormanager scripts in ansible-repo
- new hotness
- fedora messaging
- fedora messaging
- osbs client
As a part of this investigation we want to create exhaustive list, with analysis of the actual PDC use-case in each application.
As a part of this investigation we want to create exhaustive list, with analysis of the
actual PDC use-case in each application.
Solutions to be explored
------------------------
@ -60,15 +66,16 @@ We have sevral options:
- Upgrade PDC to supported version of Django and take over the maintenance
- Use a database with a simple api gateway like Postgrest or Prest in front of it
- Create a new bespoke application that better suits our needs
- Incorporate the functionality we need to other applications we already have (Pagure/Bodhi/Datagrepper)
- Incorporate the functionality we need to other applications we already have
(Pagure/Bodhi/Datagrepper)
Currently we are proposing primarily the last option.
Preliminary notes on maintianing current PDC
--------------------------------
--------------------------------------------
This would require significant investment, as it would require several upgrades of underlying software
and probably even a Python 2.x to 3.x migration as well.
This would require significant investment, as it would require several upgrades of
underlying software and probably even a Python 2.x to 3.x migration as well.
Moreover, based on the discussions within CPE:
@ -78,32 +85,30 @@ Moreover, based on the discussions within CPE:
Because of this, we won't focus on this avenue of exploration.
Preliminary notes on using Postgrest
-------------------------------------
------------------------------------
Based on our discussions, we mostly use simple CRUD API,
which means that a new database-model that would better suit our needs
could be all we need to migrate.
Based on our discussions, we mostly use simple CRUD API, which means that a new
database-model that would better suit our needs could be all we need to migrate.
The biggest advantage of this approach is using off the shelf service,
where only thing we need to maintain is the database-model, clients,
and keeping the software up-to-date.
The biggest advantage of this approach is using off the shelf service, where only thing
we need to maintain is the database-model, clients, and keeping the software up-to-date.
We have chosen two candidates to investigate based on the fact,
we probably want to utilize Postgresql on the backend:
We have chosen two candidates to investigate based on the fact, we probably want to
utilize Postgresql on the backend:
- https://postgrest.org/en/stable/
Postgrest as a simple api on top of a database model sounded promissing,
but after investigating how to integrate it with ou currentaccount system,
it turned out we would need to maintain a separate service that would serve as a bridge.
- https://postgrest.org/en/stable/auth.html
- https://samkhawase.com/blog/postgrest/postgrest_auth0_service/
Postgrest as a simple api on top of a database model sounded promissing, but after
investigating how to integrate it with ou currentaccount system, it turned out we would
need to maintain a separate service that would serve as a bridge. -
https://postgrest.org/en/stable/auth.html -
https://samkhawase.com/blog/postgrest/postgrest_auth0_service/
This speaks against using Postgrest.
Create a new bespoke application that better suits our needs
----------------------------------------------------
------------------------------------------------------------
We investigated using fastapi and fas, and the previously started fpdc proof of concept by Clement.
Based on discussions with others, we decided to investigate way to merge the existing functionality to the services we already maintain.
We investigated using fastapi and fas, and the previously started fpdc proof of concept
by Clement. Based on discussions with others, we decided to investigate way to merge the
existing functionality to the services we already maintain.

57
docs/pdc/split.rst
View file

@ -1,5 +1,5 @@
Proposal to retire PDC by splitting it among existing services
====================
==============================================================
Based on our research, currently we are using PDC for:
@ -10,10 +10,12 @@ Based on our research, currently we are using PDC for:
The endpoints we use are:
- /rest_api/v1/global-components - for storing package names
- /rest_api/v1/component-branches/ - stores SLA's, active/retired and critical-path flag
- /rest_api/v1/component-branches/ - stores SLA's, active/retired and critical-path flag
- /rest_api/v1/component-branch-slas/ - auxiliary endpoint for SLA's
- /rest_api/v1/compose-images//rest_api/v1/compose-images/ - stores composeinfo.json and image-manifest.json
- /rest_api/v1/releases/ - metadata on type (ga, updates, eus, aus, els, fast, updates-testing) of a release of a product version and if it is active
- /rest_api/v1/compose-images//rest_api/v1/compose-images/ - stores composeinfo.json and
image-manifest.json
- /rest_api/v1/releases/ - metadata on type (ga, updates, eus, aus, els, fast,
updates-testing) of a release of a product version and if it is active
- /rest_api/v1/rpms/ - liks rpms to composes
- /rest_api/v1/product-versions - stores major versions of fedora
@ -21,47 +23,50 @@ Bodhi
-----
Bodhi would take over responsibilities of
- /rest_api/v1/releases/
- /rest_api/v1/component-branches/ - partially, just the critical-path flag
- /rest_api/v1/releases/
- /rest_api/v1/component-branches/ - partially, just the critical-path flag
Bodhi already has concept that maps to releases and components,
we should be able to create auxiliary table that will pair these with additional metadata,
mostyl the critical-path flag, that previously had to be queried from PDC.
Bodhi already has concept that maps to releases and components, we should be able to
create auxiliary table that will pair these with additional metadata, mostyl the
critical-path flag, that previously had to be queried from PDC.
PDC actually isn't the source-of-truth for packages on critical path.
The source of truth, according to https://fedoraproject.org/wiki/Critical_path_package is
in https://pagure.io/fedora-comps/tree/main when you search for groups with critical-path-*
PDC actually isn't the source-of-truth for packages on critical path. The source of
truth, according to https://fedoraproject.org/wiki/Critical_path_package is in
https://pagure.io/fedora-comps/tree/main when you search for groups with critical-path-*
prefix. For example: https://pagure.io/fedora-comps/blob/main/f/comps-f35.xml.in#_622
The data is available through dnf, but generating it can be really slow, PDC is then used as
a sort of a pre-computed cache.
The data is available through dnf, but generating it can be really slow, PDC is then
used as a sort of a pre-computed cache.
To update it, we used to use https://docs.pagure.org/releng/sop_update_critpath.html
but now we have https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/plugins/pdc_update_critpath.py
To update it, we used to use https://docs.pagure.org/releng/sop_update_critpath.html but
now we have
https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/plugins/pdc_update_critpath.py
Only place we actually use the information during the fedora release is in Bodhi,
to have stricter requirements on `critpath` package updates in Bodhi:
Only place we actually use the information during the fedora release is in Bodhi, to
have stricter requirements on `critpath` package updates in Bodhi:
https://github.com/fedora-infra/bodhi/blob/master/bodhi/server/util.py#L192
Pagure-dist-git
---------------
Pagure-dist-git would take over responsibilities for
Pagure-dist-git would take over responsibilities for
- /rest_api/v1/product-versions
- /rest_api/v1/global-components
- /rest_api/v1/component-branches/
- /rest_api/v1/component-branch-slas/
Pagure already de-facto has a database of global-components (the repositories), and product-versions (repository branches)
and it uses PDC api to query component-branches if the package was retired,
with auxiliary table in pagure-dist-git storing reason for orphanning.
Pagure already de-facto has a database of global-components (the repositories), and
product-versions (repository branches) and it uses PDC api to query component-branches
if the package was retired, with auxiliary table in pagure-dist-git storing reason for
orphanning.
Remaining endpoints
-------------
-------------------
/rest_api/v1/compose-images/ - as we are mostly storing json blobs, based on discussions, we should just store the json in network-accessible file-server.
/rest_api/v1/compose-images/ - as we are mostly storing json blobs, based on
discussions, we should just store the json in network-accessible file-server.
/rest_api/v1/rpms/ - this is only information that so far doesn't fit into any existing application. QA is using it to query what packages are in particular composes.
/rest_api/v1/rpms/ - this is only information that so far doesn't fit into any existing
application. QA is using it to query what packages are in particular composes.

92
docs/pdc/users.rst
View file

@ -2,32 +2,32 @@ Current users of PDC
====================
release engineering - releng scripts
-----------------------------
------------------------------------
Repository: https://pagure.io/releng
Retired packages
- scripts/block_retired.py
- /rest_api/v1/component-branches/?name={1}&type={2}&active=false&page_size=100'
- {1} - branch name, i.e. rawhide
- {2} - type, i.e. rpm
- scripts/block_retired.py
- /rest_api/v1/component-branches/?name={1}&type={2}&active=false&page_size=100'
- {1} - branch name, i.e. rawhide
- {2} - type, i.e. rpm
Package unretirement
- scripts/check-unretirement.py
- https://pdc.fedoraproject.org/rest_api/v1/component-branches/?global_component={args.pck}&name={args.brc}&type={args.nms[:-1]}
- scripts/check-unretirement.py
- https://pdc.fedoraproject.org/rest_api/v1/component-branches/?global_component={args.pck}&name={args.brc}&type={args.nms[:-1]}
Package retirement
- scripts/fedretire
- results = pdc.get_paged(pdc['component-branches'], global_component=name, name=branch, type=NAMESPACES[namespace])
- uses fedpkg for actual retirement
- scripts/fedretire
- results = pdc.get_paged(pdc['component-branches'], global_component=name,
name=branch, type=NAMESPACES[namespace])
- uses fedpkg for actual retirement
Critical path
- scripts/get-critpath
- pdc.get_paged(endpoint, critical_path=True, **kwargs)
- scripts/get-critpath
- ``pdc.get_paged(endpoint, critical_path=True, **kwargs)``
Mass rebuild modules
- /rest_api/v1/component-branch-slas/?page_size=100&branch_type=module&branch_active=1'
- /rest_api/v1/component-branch-slas/?page_size=100&branch_type=module&branch_active=1'
fedpkg
------
@ -35,26 +35,26 @@ fedpkg
Repository: https://pagure.io/fedpkg
get_release_branches
- fedpkg/utils.py
- query_args = {'fields': ['short', 'version'], 'active': True}
- endpoint = 'product-versions'
- fedpkg/utils.py
- query_args = {'fields': ['short', 'version'], 'active': True}
- endpoint = 'product-versions'
get_stream_branches
- fedpkg/utils.py
- query_args = {'global_component': package_name,'fields': ['name', 'active']}
- endpoint = 'component-branches'
- fedpkg/utils.py
- query_args = {'global_component': package_name,'fields': ['name', 'active']}
- endpoint = 'component-branches'
query_pdc
- fedpkg/utils.py
- uses requests
- fedpkg/utils.py
- uses requests
pungi
----------
-----
According to lubomir.sedlar@gmail.com:
"Pungi used to integrate with PDC in the past, but this support has been
deprecated and dropped."
"Pungi used to integrate with PDC in the past, but this support has been deprecated and
dropped."
fedoraqa - fedfind
------------------
@ -78,7 +78,7 @@ Pungi4Release.get_package_nevras_pdc
- pdc_query('rpms', [('compose', self.cid), ('arch', 'src')])
Bodhi
------------------
-----
Repository: https://github.com/fedora-infra/bodhi
@ -91,14 +91,15 @@ get_critpath_components_from_pdc
fedscm-admin
------------
We have dropped fedscm-admin in favor of toddlers.
The endpoints used by it should be the same since it solves the same problem.
We have dropped fedscm-admin in favor of toddlers. The endpoints used by it should be
the same since it solves the same problem.
Repository: https://pagure.io/fedora-infra/toddlers/
fedscm_admin/pdc.py
fedscm_admin/pdc.py
- get_global_component
- new_global_component
- new_global_component
- get_branch
- new_branch
- get_sla
@ -106,23 +107,26 @@ fedscm_admin/pdc.py
pagure
------
- /rest_api/v1/component-branches/
- to get retired packages
- /rest_api/v1/component-branches/
- to get retired packages
modulebuildservice
------------------
As of Fedora 39 we are dropping modules support. With Fedora 38 EOL we will drop MBS.
- /rest_api/v1/component-branches/
- to get end-of-life packages
- based on ansible roles/mbs/common/files/fedora.json.productbion
- /rest_api/v1/component-branches/
- to get end-of-life packages
- based on ansible roles/mbs/common/files/fedora.json.productbion
mirrormanager scripts
---------------------
- /rest_api/v1/releases/
- checks for currently active releases
- roles/mirrormanager/backend/templates/handle_propagation.sh
- roles/mirrormanager/crawler/files/check_propagation.sh
- /rest_api/v1/releases/
- checks for currently active releases
- roles/mirrormanager/backend/templates/handle_propagation.sh
- roles/mirrormanager/crawler/files/check_propagation.sh
ODCS
----
@ -131,16 +135,16 @@ no longer uses pdc
new hotness
-----------
- /rest_api/v1/component-branches/
- to get retired packages
- /rest_api/v1/component-branches/
- to get retired packages
fedora messaging
----------------
- no integrations found
osbs client
-----------
- no integrations found