Add SOP that deals with OpenH264
Signed-off-by: Patrik Polakovič <patrik@alphamail.org>
This commit is contained in:
Patrik Polakovič
parent
b52b32f2d9
commit
c5e22908dd
2 changed files with 206 additions and 185 deletions
|
|
@ -1,185 +0,0 @@
|
||||||
== Generating Openh264 Composes
|
|
||||||
|
|
||||||
=== Description
|
|
||||||
|
|
||||||
Openh264 repos are a special case and we need to generate the composes
|
|
||||||
for it in a different way. We use ODCS to generate the private compose
|
|
||||||
and send the rpms to Cisco to publish them on their CDN. We publish the
|
|
||||||
repodata on our side.
|
|
||||||
|
|
||||||
[WARNING]
|
|
||||||
.Warning
|
|
||||||
====
|
|
||||||
We do not have all the appropriate legal rights to distribute these
|
|
||||||
packages, so we need to be extra carefull to make sure they are never
|
|
||||||
distributed via our build system or websites
|
|
||||||
====
|
|
||||||
|
|
||||||
=== Action
|
|
||||||
|
|
||||||
==== Permissions needed
|
|
||||||
|
|
||||||
You will need some ODCS permissions in order to request private composes
|
|
||||||
and composes from tags. You can set this in infra/ansible in
|
|
||||||
inventory/group_vars/odcs in the odcs_allowed_clients_users variable.
|
|
||||||
See other releng users entries for format.
|
|
||||||
|
|
||||||
==== Get the odcs token
|
|
||||||
|
|
||||||
In order to generate an odcs compose, you need a openidc token.
|
|
||||||
|
|
||||||
Run the odcs-token.py under `scripts/odcs/` from pagure releng
|
|
||||||
repository to generate the token.
|
|
||||||
|
|
||||||
....
|
|
||||||
$ ./odcs-token.py
|
|
||||||
....
|
|
||||||
|
|
||||||
==== Make sure rpms are written out with the right signature
|
|
||||||
|
|
||||||
....
|
|
||||||
$ koji write-signed-rpm eb10b464 openh264-2.2.0-1.fc38
|
|
||||||
....
|
|
||||||
|
|
||||||
Where the key for that branch is listed, then the open264 package and
|
|
||||||
version.
|
|
||||||
|
|
||||||
==== Generate a private odcs compose
|
|
||||||
|
|
||||||
With the token generated above, generate the odcs private compose
|
|
||||||
|
|
||||||
....
|
|
||||||
$ python odcs-private-compose.py <token> <koji_tag> <signingkeyid>
|
|
||||||
....
|
|
||||||
|
|
||||||
`koji_tag`: fxx-openh264 (Openh264 builds are tagged to fxx-openh264
|
|
||||||
tags where [.title-ref]#xx# represents the fedora release)
|
|
||||||
|
|
||||||
`signingkeyid`: The short hash of the key for this Fedora branch.
|
|
||||||
|
|
||||||
The composes are stored under `/srv/odcs/private/` dir on
|
|
||||||
`odcs-backend-releng01.iad2.fedoraproject.org`
|
|
||||||
|
|
||||||
==== Pull the compose to your local machine
|
|
||||||
|
|
||||||
We need to extract the rpms and tar them to send them to Cisco. In order
|
|
||||||
to that, first of all we need to pull the compose to our local machine.
|
|
||||||
|
|
||||||
===== Move the compose to your home dir on odcs-backend-releng01.iad2.fedoraproject.org
|
|
||||||
|
|
||||||
Since the compose is owned by [.title-ref]#odcs-server# pull it into
|
|
||||||
your home dir
|
|
||||||
|
|
||||||
....
|
|
||||||
$ mkdir ~/32-openh264
|
|
||||||
$ sudo rsync -avhHP /srv/odcs/private/odcs-3835/ ~/32-openh264/
|
|
||||||
$ sudo chown -R mohanboddu:mohanboddu ~/32-openh264/
|
|
||||||
....
|
|
||||||
|
|
||||||
===== Sync the compose to your local machine
|
|
||||||
|
|
||||||
Pull in the compose from your home dir on odcs releng backend to your
|
|
||||||
local machine into a temp working dir
|
|
||||||
|
|
||||||
....
|
|
||||||
$ mkdir openh264-20200813
|
|
||||||
$ scp -rv odcs-backend-releng01.iad2.fedoraproject.org:/home/fedora/mohanboddu/32-openh264/ openh264-20200813/
|
|
||||||
....
|
|
||||||
|
|
||||||
===== Make the changes needed
|
|
||||||
|
|
||||||
Please follow the following commands to make the necessary tar files to
|
|
||||||
send to Cisco
|
|
||||||
|
|
||||||
....
|
|
||||||
$ cd openh264-20200813
|
|
||||||
$ mkdir 32-rpms
|
|
||||||
# Copy rpms including devel rpms
|
|
||||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*rpm 32-rpms/
|
|
||||||
# Copy debuginfo rpms
|
|
||||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*/*rpm 32-rpms/
|
|
||||||
# copy the src.rpm
|
|
||||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*src.rpm 32-rpms/
|
|
||||||
$ cd 32-rpms
|
|
||||||
# Create the tar file with the rpms
|
|
||||||
$ tar -cJvf ../fedora-32-openh264-rpms.tar.xz *rpm
|
|
||||||
....
|
|
||||||
|
|
||||||
We need to send this tar file to Cisco along with the list of rpms in
|
|
||||||
each tarball.
|
|
||||||
|
|
||||||
===== Syncing the compose to sundries01
|
|
||||||
|
|
||||||
Once we get a confirmation from Cisco that the rpms are updated on their
|
|
||||||
CDN, verify them by using curl. For example:
|
|
||||||
|
|
||||||
....
|
|
||||||
$ curl -I http://ciscobinary.openh264.org/openh264-2.1.1-1.fc32.x86_64.rpm
|
|
||||||
....
|
|
||||||
|
|
||||||
Now push these composes to *sundries01.iad2.fedoraproject.org* and
|
|
||||||
*mm-backend01.iad2.fedoraproject.org*
|
|
||||||
|
|
||||||
On sundries01 we need to sync to a directory that is owned by _apache_,
|
|
||||||
so first we sync to the home directory on sundries01. Same with
|
|
||||||
mm-backend01 as the directory is owned by _root_.
|
|
||||||
|
|
||||||
Create a temp working directory on sundries01
|
|
||||||
|
|
||||||
....
|
|
||||||
$ ssh sundries01.iad2.fedoraproject.org
|
|
||||||
$ mkdir openh264-20200825
|
|
||||||
....
|
|
||||||
|
|
||||||
Create a temp working directory on mm-backend01
|
|
||||||
|
|
||||||
....
|
|
||||||
$ ssh mm-backend01.iad2.fedoraproject.org
|
|
||||||
$ mkdir openh264-20200825
|
|
||||||
....
|
|
||||||
|
|
||||||
Then from your local machine, sync the compose
|
|
||||||
|
|
||||||
....
|
|
||||||
$ cd openh264-20200825
|
|
||||||
$ rsync -avhHP 32-openh264 sundries01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825
|
|
||||||
$ rsync -avhHP 32-openh264 mm-backend01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825
|
|
||||||
....
|
|
||||||
|
|
||||||
On sundries01
|
|
||||||
|
|
||||||
....
|
|
||||||
$ cd openh264-20200825
|
|
||||||
$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/web/codecs.fedoraproject.org/openh264/32/
|
|
||||||
....
|
|
||||||
|
|
||||||
On mm-backend01
|
|
||||||
|
|
||||||
....
|
|
||||||
$ cd openh264-20200825
|
|
||||||
$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/codecs.fedoraproject.org/openh264/32/
|
|
||||||
....
|
|
||||||
|
|
||||||
===== Extra info
|
|
||||||
|
|
||||||
Normally that should be it, but in some cases you may want to push
|
|
||||||
things out faster than normal, and here's a few things you can do to do
|
|
||||||
that:
|
|
||||||
|
|
||||||
On mm-backend01.iad2.fedoraproject.org you can run:
|
|
||||||
|
|
||||||
....
|
|
||||||
# sudo -u mirrormanager /usr/local/bin/umdl-required codecs /var/log/mirrormanager/umdl-required.log
|
|
||||||
....
|
|
||||||
|
|
||||||
This will have mirrormanager scan the codecs dir and update it if it's
|
|
||||||
changed.
|
|
||||||
|
|
||||||
On batcave01.iad2.fedoraproject.org you can use ansible to force all the
|
|
||||||
proxies to sync the codec content from sundries01:
|
|
||||||
|
|
||||||
....
|
|
||||||
# ansible -a '/usr/bin/rsync --delete -a --no-owner --no-group sundries01::codecs.fedoraproject.org/ /srv/web/codecs.fedoraproject.org/' proxies
|
|
||||||
....
|
|
||||||
|
|
||||||
Mirrorlist servers should update every 15min.
|
|
||||||
206
modules/releng_misc_guide/pages/sop_openh264.adoc
Normal file
206
modules/releng_misc_guide/pages/sop_openh264.adoc
Normal file
|
|
@ -0,0 +1,206 @@
|
||||||
|
== Generating OpenH264 RPMs and sending them to Cisco
|
||||||
|
|
||||||
|
=== Description
|
||||||
|
|
||||||
|
OpenH264 RPMs are a special case and we need to handle it with caution.
|
||||||
|
|
||||||
|
The process is as follows:
|
||||||
|
|
||||||
|
. We generate and sign the RPMs using `koji dist-repo`.
|
||||||
|
. We send the RPMs to Cisco for hosting.
|
||||||
|
. Once we get confirmation from Cisco that they published the RPMs on their CDN
|
||||||
|
we publish the repodata on our side.
|
||||||
|
|
||||||
|
=== Legal background
|
||||||
|
|
||||||
|
OpenH264 is a free software library for real-time encoding and decoding video
|
||||||
|
streams in the H.264/MPEG-4 AVC format.
|
||||||
|
|
||||||
|
In 2013 Cisco released both binaries and source code, and pay all royalties for
|
||||||
|
its use to MPEG LA themselves for any software projects that use Cisco's
|
||||||
|
precompiled binaries (thus making Cisco's OpenH264 binaries free to use);
|
||||||
|
any software projects that use Cisco's source code instead of its binaries
|
||||||
|
would be legally responsible for paying all royalties to MPEG LA themselves,
|
||||||
|
however.
|
||||||
|
|
||||||
|
The workaround is that the Cisco binaries are built in Fedora's infrastructure
|
||||||
|
but *distributed* by Cisco.
|
||||||
|
|
||||||
|
[WARNING]
|
||||||
|
.Warning
|
||||||
|
====
|
||||||
|
The RPMs *must never* be distributed via our build system or websites.
|
||||||
|
====
|
||||||
|
|
||||||
|
=== The process
|
||||||
|
|
||||||
|
==== Permissions needed
|
||||||
|
|
||||||
|
. Make sure to have a valid Kerberos token (`kinit`).
|
||||||
|
. Relevant FAS group membership.
|
||||||
|
. *TODO*
|
||||||
|
|
||||||
|
==== Generating the repository
|
||||||
|
|
||||||
|
Once a request comes in through the RelEng issue tracker to update OpenH264 to
|
||||||
|
a new version we will first need to generate the RPMs and repodata. We do this
|
||||||
|
using the `koji dist-repo` utility.
|
||||||
|
|
||||||
|
[NOTE]
|
||||||
|
.Note
|
||||||
|
====
|
||||||
|
The arguments that should be passed to the commands will certainly vary from the
|
||||||
|
examples listed below.
|
||||||
|
|
||||||
|
Practice common sense and double check everything.
|
||||||
|
====
|
||||||
|
|
||||||
|
Run this command:
|
||||||
|
|
||||||
|
....
|
||||||
|
$ koji dist-repo f43-openh264 31645531 --noinherit --with-src -a aarch -a
|
||||||
|
ppc64le -a s390x -a x86_64
|
||||||
|
....
|
||||||
|
|
||||||
|
In the case of the above command:
|
||||||
|
|
||||||
|
. `f43-openh264` is the tag the builds are tagged into.
|
||||||
|
. `31645531` is the signing key for the RPMs. The keys for various versions
|
||||||
|
can be found https://fedoraproject.org/security[here].
|
||||||
|
. `--noinherit` causes only packages tagged into `f43-openh264` to be included,
|
||||||
|
without inheritance from the parent tag (e.g. `f43`).
|
||||||
|
. `--with-src` causes the source RPM to be included.
|
||||||
|
. `-a <ARG>` options are the respective architectures.
|
||||||
|
|
||||||
|
After the command finishes the directory should appear
|
||||||
|
https://kojipkgs.fedoraproject.org/repos-dist[here].
|
||||||
|
|
||||||
|
The RPMs are visible through the web frontend, but we have a rewrite rule that
|
||||||
|
redirects any attempt to download them to a wiki page saying why that is not
|
||||||
|
possible.
|
||||||
|
|
||||||
|
In order to get to the RPMs themselves we must do it from a host on which Koji
|
||||||
|
is mounted where they will be in the `/mnt/koji/repos-dist/f43-openh264` directory.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
....
|
||||||
|
$ ssh compose-branched01.iad2.fedoraproject.org
|
||||||
|
....
|
||||||
|
|
||||||
|
After you verify that everything needed is there you may clone the directory to
|
||||||
|
your local machine.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ rsync -avhHP username@compose-branched01.iad2.fedoraproject.org:/mnt/koji/repos-dist/f43-openh264/6555489 /home/username/openh264/2.6.0/f43
|
||||||
|
....
|
||||||
|
|
||||||
|
==== Sending the RPMs to Cisco for hosting
|
||||||
|
|
||||||
|
===== Preparing the tarball
|
||||||
|
|
||||||
|
We now have the necessary directory on our local machine. Now we need to package
|
||||||
|
it into a tarball and send it to Cisco.
|
||||||
|
|
||||||
|
Create a new directory into which we can copy the RPMs and out of which we will
|
||||||
|
create the tarball.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ mkdir /home/username/openh264/2.6.0/f43/f43_2.6.0_rpms
|
||||||
|
....
|
||||||
|
|
||||||
|
Copy the RPMs (including the source RPM) from the original directory to the
|
||||||
|
new directory.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ cd /home/username/openh264/2.6.0/f43/
|
||||||
|
$ cp -rv 6553481/*/*/*/*rpm f43_2.6.0_rpms
|
||||||
|
....
|
||||||
|
|
||||||
|
Create a list of the RPMs that will be sent to Cisco along with the tarball.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ ls f43_2.6.0_rpms > f43_2.6.0_rpms_list.txt
|
||||||
|
....
|
||||||
|
|
||||||
|
Create the tarball.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ tar -cJvf f43_2.6.0_rpms.tar.xz f43_2.6.0_rpms
|
||||||
|
....
|
||||||
|
|
||||||
|
===== Sending the tarball to Cisco
|
||||||
|
|
||||||
|
*TODO*
|
||||||
|
|
||||||
|
[NOTE]
|
||||||
|
.Note
|
||||||
|
====
|
||||||
|
We used to do this through e-mail. However, lately we have started experiencing issues
|
||||||
|
with e-mail filters (can't send tarballs). We are trying to reach Cisco about
|
||||||
|
possible solutions to this issue.
|
||||||
|
====
|
||||||
|
|
||||||
|
==== Publishing the repodata
|
||||||
|
|
||||||
|
Once we receive confirmation from Cisco that their CDN was updated we can
|
||||||
|
verify it using `curl`.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ curl -I http://ciscobinary.openh264.org/openh264-2.6.0-1.fc43.x86_64.rpm
|
||||||
|
....
|
||||||
|
|
||||||
|
Now it is time to publish the repodata on our infrastructure.
|
||||||
|
Navigate to the relevant directory on your local machine and sync the
|
||||||
|
`dist-repo` directory to your home directory on `sundries01`.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ rsync -avhHP 6556232 sundries01.iad2.fedoraproject.org/home/fedora/username/openh264/2.6.0/43
|
||||||
|
....
|
||||||
|
|
||||||
|
Connect to `sundries01`.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ ssh sundries01.iad2.fedoraproject.org
|
||||||
|
....
|
||||||
|
|
||||||
|
Navigate to the relevant directory.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ pwd
|
||||||
|
/home/fedora/username/openh264/2.6.0/43
|
||||||
|
....
|
||||||
|
|
||||||
|
Sync the contents of the directory to `/srv/web/codecs.fedoraproject.org`.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ sudo rsync -avhHP 6556232/. /srv/web/codecs.fedoraproject.org/openh264/43
|
||||||
|
....
|
||||||
|
|
||||||
|
Make sure the directory is owned by root.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ sudo chown -R root:root /srv/web/codecs.fedoraproject.org/openh264/43
|
||||||
|
....
|
||||||
|
|
||||||
|
*TODO*
|
||||||
|
|
||||||
|
[NOTE]
|
||||||
|
.Note
|
||||||
|
====
|
||||||
|
This section will have to be improved with more details.
|
||||||
|
====
|
||||||
|
|
||||||
|
In the database:
|
||||||
|
....
|
||||||
|
update directory set ctime=0 where name like 'ct.org/openh264/43%codecs.fedoraproject.org/openh264/43%';
|
||||||
|
delete from repository where prefix like '%cisco%43%';
|
||||||
|
....
|
||||||
|
|
||||||
|
In a MirrorManager OpenShift debug container:
|
||||||
|
....
|
||||||
|
(app-root) sh-5.2$ scan-primary-mirror --category "Fedora Codecs" -d
|
||||||
|
....
|
||||||
|
|
||||||
|
=== Resources
|
||||||
|
. https://src.fedoraproject.org/rpms/openh264[OpenH264 dist-git repository]
|
||||||
Loading…
Add table
Add a link
Reference in a new issue