Add SOP that deals with OpenH264
Signed-off-by: Patrik Polakovič <patrik@alphamail.org>
This commit is contained in:
Patrik Polakovič
parent
b52b32f2d9
commit
c5e22908dd
2 changed files with 206 additions and 185 deletions
|
|
@ -1,185 +0,0 @@
|
|||
== Generating Openh264 Composes
|
||||
|
||||
=== Description
|
||||
|
||||
Openh264 repos are a special case and we need to generate the composes
|
||||
for it in a different way. We use ODCS to generate the private compose
|
||||
and send the rpms to Cisco to publish them on their CDN. We publish the
|
||||
repodata on our side.
|
||||
|
||||
[WARNING]
|
||||
.Warning
|
||||
====
|
||||
We do not have all the appropriate legal rights to distribute these
|
||||
packages, so we need to be extra carefull to make sure they are never
|
||||
distributed via our build system or websites
|
||||
====
|
||||
|
||||
=== Action
|
||||
|
||||
==== Permissions needed
|
||||
|
||||
You will need some ODCS permissions in order to request private composes
|
||||
and composes from tags. You can set this in infra/ansible in
|
||||
inventory/group_vars/odcs in the odcs_allowed_clients_users variable.
|
||||
See other releng users entries for format.
|
||||
|
||||
==== Get the odcs token
|
||||
|
||||
In order to generate an odcs compose, you need a openidc token.
|
||||
|
||||
Run the odcs-token.py under `scripts/odcs/` from pagure releng
|
||||
repository to generate the token.
|
||||
|
||||
....
|
||||
$ ./odcs-token.py
|
||||
....
|
||||
|
||||
==== Make sure rpms are written out with the right signature
|
||||
|
||||
....
|
||||
$ koji write-signed-rpm eb10b464 openh264-2.2.0-1.fc38
|
||||
....
|
||||
|
||||
Where the key for that branch is listed, then the open264 package and
|
||||
version.
|
||||
|
||||
==== Generate a private odcs compose
|
||||
|
||||
With the token generated above, generate the odcs private compose
|
||||
|
||||
....
|
||||
$ python odcs-private-compose.py <token> <koji_tag> <signingkeyid>
|
||||
....
|
||||
|
||||
`koji_tag`: fxx-openh264 (Openh264 builds are tagged to fxx-openh264
|
||||
tags where [.title-ref]#xx# represents the fedora release)
|
||||
|
||||
`signingkeyid`: The short hash of the key for this Fedora branch.
|
||||
|
||||
The composes are stored under `/srv/odcs/private/` dir on
|
||||
`odcs-backend-releng01.iad2.fedoraproject.org`
|
||||
|
||||
==== Pull the compose to your local machine
|
||||
|
||||
We need to extract the rpms and tar them to send them to Cisco. In order
|
||||
to that, first of all we need to pull the compose to our local machine.
|
||||
|
||||
===== Move the compose to your home dir on odcs-backend-releng01.iad2.fedoraproject.org
|
||||
|
||||
Since the compose is owned by [.title-ref]#odcs-server# pull it into
|
||||
your home dir
|
||||
|
||||
....
|
||||
$ mkdir ~/32-openh264
|
||||
$ sudo rsync -avhHP /srv/odcs/private/odcs-3835/ ~/32-openh264/
|
||||
$ sudo chown -R mohanboddu:mohanboddu ~/32-openh264/
|
||||
....
|
||||
|
||||
===== Sync the compose to your local machine
|
||||
|
||||
Pull in the compose from your home dir on odcs releng backend to your
|
||||
local machine into a temp working dir
|
||||
|
||||
....
|
||||
$ mkdir openh264-20200813
|
||||
$ scp -rv odcs-backend-releng01.iad2.fedoraproject.org:/home/fedora/mohanboddu/32-openh264/ openh264-20200813/
|
||||
....
|
||||
|
||||
===== Make the changes needed
|
||||
|
||||
Please follow the following commands to make the necessary tar files to
|
||||
send to Cisco
|
||||
|
||||
....
|
||||
$ cd openh264-20200813
|
||||
$ mkdir 32-rpms
|
||||
# Copy rpms including devel rpms
|
||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*rpm 32-rpms/
|
||||
# Copy debuginfo rpms
|
||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*/*rpm 32-rpms/
|
||||
# copy the src.rpm
|
||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*src.rpm 32-rpms/
|
||||
$ cd 32-rpms
|
||||
# Create the tar file with the rpms
|
||||
$ tar -cJvf ../fedora-32-openh264-rpms.tar.xz *rpm
|
||||
....
|
||||
|
||||
We need to send this tar file to Cisco along with the list of rpms in
|
||||
each tarball.
|
||||
|
||||
===== Syncing the compose to sundries01
|
||||
|
||||
Once we get a confirmation from Cisco that the rpms are updated on their
|
||||
CDN, verify them by using curl. For example:
|
||||
|
||||
....
|
||||
$ curl -I http://ciscobinary.openh264.org/openh264-2.1.1-1.fc32.x86_64.rpm
|
||||
....
|
||||
|
||||
Now push these composes to *sundries01.iad2.fedoraproject.org* and
|
||||
*mm-backend01.iad2.fedoraproject.org*
|
||||
|
||||
On sundries01 we need to sync to a directory that is owned by _apache_,
|
||||
so first we sync to the home directory on sundries01. Same with
|
||||
mm-backend01 as the directory is owned by _root_.
|
||||
|
||||
Create a temp working directory on sundries01
|
||||
|
||||
....
|
||||
$ ssh sundries01.iad2.fedoraproject.org
|
||||
$ mkdir openh264-20200825
|
||||
....
|
||||
|
||||
Create a temp working directory on mm-backend01
|
||||
|
||||
....
|
||||
$ ssh mm-backend01.iad2.fedoraproject.org
|
||||
$ mkdir openh264-20200825
|
||||
....
|
||||
|
||||
Then from your local machine, sync the compose
|
||||
|
||||
....
|
||||
$ cd openh264-20200825
|
||||
$ rsync -avhHP 32-openh264 sundries01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825
|
||||
$ rsync -avhHP 32-openh264 mm-backend01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825
|
||||
....
|
||||
|
||||
On sundries01
|
||||
|
||||
....
|
||||
$ cd openh264-20200825
|
||||
$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/web/codecs.fedoraproject.org/openh264/32/
|
||||
....
|
||||
|
||||
On mm-backend01
|
||||
|
||||
....
|
||||
$ cd openh264-20200825
|
||||
$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/codecs.fedoraproject.org/openh264/32/
|
||||
....
|
||||
|
||||
===== Extra info
|
||||
|
||||
Normally that should be it, but in some cases you may want to push
|
||||
things out faster than normal, and here's a few things you can do to do
|
||||
that:
|
||||
|
||||
On mm-backend01.iad2.fedoraproject.org you can run:
|
||||
|
||||
....
|
||||
# sudo -u mirrormanager /usr/local/bin/umdl-required codecs /var/log/mirrormanager/umdl-required.log
|
||||
....
|
||||
|
||||
This will have mirrormanager scan the codecs dir and update it if it's
|
||||
changed.
|
||||
|
||||
On batcave01.iad2.fedoraproject.org you can use ansible to force all the
|
||||
proxies to sync the codec content from sundries01:
|
||||
|
||||
....
|
||||
# ansible -a '/usr/bin/rsync --delete -a --no-owner --no-group sundries01::codecs.fedoraproject.org/ /srv/web/codecs.fedoraproject.org/' proxies
|
||||
....
|
||||
|
||||
Mirrorlist servers should update every 15min.
|
||||
206
modules/releng_misc_guide/pages/sop_openh264.adoc
Normal file
206
modules/releng_misc_guide/pages/sop_openh264.adoc
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
== Generating OpenH264 RPMs and sending them to Cisco
|
||||
|
||||
=== Description
|
||||
|
||||
OpenH264 RPMs are a special case and we need to handle it with caution.
|
||||
|
||||
The process is as follows:
|
||||
|
||||
. We generate and sign the RPMs using `koji dist-repo`.
|
||||
. We send the RPMs to Cisco for hosting.
|
||||
. Once we get confirmation from Cisco that they published the RPMs on their CDN
|
||||
we publish the repodata on our side.
|
||||
|
||||
=== Legal background
|
||||
|
||||
OpenH264 is a free software library for real-time encoding and decoding video
|
||||
streams in the H.264/MPEG-4 AVC format.
|
||||
|
||||
In 2013 Cisco released both binaries and source code, and pay all royalties for
|
||||
its use to MPEG LA themselves for any software projects that use Cisco's
|
||||
precompiled binaries (thus making Cisco's OpenH264 binaries free to use);
|
||||
any software projects that use Cisco's source code instead of its binaries
|
||||
would be legally responsible for paying all royalties to MPEG LA themselves,
|
||||
however.
|
||||
|
||||
The workaround is that the Cisco binaries are built in Fedora's infrastructure
|
||||
but *distributed* by Cisco.
|
||||
|
||||
[WARNING]
|
||||
.Warning
|
||||
====
|
||||
The RPMs *must never* be distributed via our build system or websites.
|
||||
====
|
||||
|
||||
=== The process
|
||||
|
||||
==== Permissions needed
|
||||
|
||||
. Make sure to have a valid Kerberos token (`kinit`).
|
||||
. Relevant FAS group membership.
|
||||
. *TODO*
|
||||
|
||||
==== Generating the repository
|
||||
|
||||
Once a request comes in through the RelEng issue tracker to update OpenH264 to
|
||||
a new version we will first need to generate the RPMs and repodata. We do this
|
||||
using the `koji dist-repo` utility.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The arguments that should be passed to the commands will certainly vary from the
|
||||
examples listed below.
|
||||
|
||||
Practice common sense and double check everything.
|
||||
====
|
||||
|
||||
Run this command:
|
||||
|
||||
....
|
||||
$ koji dist-repo f43-openh264 31645531 --noinherit --with-src -a aarch -a
|
||||
ppc64le -a s390x -a x86_64
|
||||
....
|
||||
|
||||
In the case of the above command:
|
||||
|
||||
. `f43-openh264` is the tag the builds are tagged into.
|
||||
. `31645531` is the signing key for the RPMs. The keys for various versions
|
||||
can be found https://fedoraproject.org/security[here].
|
||||
. `--noinherit` causes only packages tagged into `f43-openh264` to be included,
|
||||
without inheritance from the parent tag (e.g. `f43`).
|
||||
. `--with-src` causes the source RPM to be included.
|
||||
. `-a <ARG>` options are the respective architectures.
|
||||
|
||||
After the command finishes the directory should appear
|
||||
https://kojipkgs.fedoraproject.org/repos-dist[here].
|
||||
|
||||
The RPMs are visible through the web frontend, but we have a rewrite rule that
|
||||
redirects any attempt to download them to a wiki page saying why that is not
|
||||
possible.
|
||||
|
||||
In order to get to the RPMs themselves we must do it from a host on which Koji
|
||||
is mounted where they will be in the `/mnt/koji/repos-dist/f43-openh264` directory.
|
||||
|
||||
For example:
|
||||
|
||||
....
|
||||
$ ssh compose-branched01.iad2.fedoraproject.org
|
||||
....
|
||||
|
||||
After you verify that everything needed is there you may clone the directory to
|
||||
your local machine.
|
||||
|
||||
....
|
||||
$ rsync -avhHP username@compose-branched01.iad2.fedoraproject.org:/mnt/koji/repos-dist/f43-openh264/6555489 /home/username/openh264/2.6.0/f43
|
||||
....
|
||||
|
||||
==== Sending the RPMs to Cisco for hosting
|
||||
|
||||
===== Preparing the tarball
|
||||
|
||||
We now have the necessary directory on our local machine. Now we need to package
|
||||
it into a tarball and send it to Cisco.
|
||||
|
||||
Create a new directory into which we can copy the RPMs and out of which we will
|
||||
create the tarball.
|
||||
|
||||
....
|
||||
$ mkdir /home/username/openh264/2.6.0/f43/f43_2.6.0_rpms
|
||||
....
|
||||
|
||||
Copy the RPMs (including the source RPM) from the original directory to the
|
||||
new directory.
|
||||
|
||||
....
|
||||
$ cd /home/username/openh264/2.6.0/f43/
|
||||
$ cp -rv 6553481/*/*/*/*rpm f43_2.6.0_rpms
|
||||
....
|
||||
|
||||
Create a list of the RPMs that will be sent to Cisco along with the tarball.
|
||||
|
||||
....
|
||||
$ ls f43_2.6.0_rpms > f43_2.6.0_rpms_list.txt
|
||||
....
|
||||
|
||||
Create the tarball.
|
||||
|
||||
....
|
||||
$ tar -cJvf f43_2.6.0_rpms.tar.xz f43_2.6.0_rpms
|
||||
....
|
||||
|
||||
===== Sending the tarball to Cisco
|
||||
|
||||
*TODO*
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
We used to do this through e-mail. However, lately we have started experiencing issues
|
||||
with e-mail filters (can't send tarballs). We are trying to reach Cisco about
|
||||
possible solutions to this issue.
|
||||
====
|
||||
|
||||
==== Publishing the repodata
|
||||
|
||||
Once we receive confirmation from Cisco that their CDN was updated we can
|
||||
verify it using `curl`.
|
||||
|
||||
....
|
||||
$ curl -I http://ciscobinary.openh264.org/openh264-2.6.0-1.fc43.x86_64.rpm
|
||||
....
|
||||
|
||||
Now it is time to publish the repodata on our infrastructure.
|
||||
Navigate to the relevant directory on your local machine and sync the
|
||||
`dist-repo` directory to your home directory on `sundries01`.
|
||||
|
||||
....
|
||||
$ rsync -avhHP 6556232 sundries01.iad2.fedoraproject.org/home/fedora/username/openh264/2.6.0/43
|
||||
....
|
||||
|
||||
Connect to `sundries01`.
|
||||
|
||||
....
|
||||
$ ssh sundries01.iad2.fedoraproject.org
|
||||
....
|
||||
|
||||
Navigate to the relevant directory.
|
||||
|
||||
....
|
||||
$ pwd
|
||||
/home/fedora/username/openh264/2.6.0/43
|
||||
....
|
||||
|
||||
Sync the contents of the directory to `/srv/web/codecs.fedoraproject.org`.
|
||||
|
||||
....
|
||||
$ sudo rsync -avhHP 6556232/. /srv/web/codecs.fedoraproject.org/openh264/43
|
||||
....
|
||||
|
||||
Make sure the directory is owned by root.
|
||||
|
||||
....
|
||||
$ sudo chown -R root:root /srv/web/codecs.fedoraproject.org/openh264/43
|
||||
....
|
||||
|
||||
*TODO*
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
This section will have to be improved with more details.
|
||||
====
|
||||
|
||||
In the database:
|
||||
....
|
||||
update directory set ctime=0 where name like 'ct.org/openh264/43%codecs.fedoraproject.org/openh264/43%';
|
||||
delete from repository where prefix like '%cisco%43%';
|
||||
....
|
||||
|
||||
In a MirrorManager OpenShift debug container:
|
||||
....
|
||||
(app-root) sh-5.2$ scan-primary-mirror --category "Fedora Codecs" -d
|
||||
....
|
||||
|
||||
=== Resources
|
||||
. https://src.fedoraproject.org/rpms/openh264[OpenH264 dist-git repository]
|
||||
Loading…
Add table
Add a link
Reference in a new issue