diff --git a/modules/releng_misc_guide/pages/sop_generating_openh264_composes.adoc b/modules/releng_misc_guide/pages/sop_generating_openh264_composes.adoc deleted file mode 100644 index 2cc9fae..0000000 --- a/modules/releng_misc_guide/pages/sop_generating_openh264_composes.adoc +++ /dev/null @@ -1,185 +0,0 @@ -== Generating Openh264 Composes - -=== Description - -Openh264 repos are a special case and we need to generate the composes -for it in a different way. We use ODCS to generate the private compose -and send the rpms to Cisco to publish them on their CDN. We publish the -repodata on our side. - -[WARNING] -.Warning -==== -We do not have all the appropriate legal rights to distribute these -packages, so we need to be extra carefull to make sure they are never -distributed via our build system or websites -==== - -=== Action - -==== Permissions needed - -You will need some ODCS permissions in order to request private composes -and composes from tags. You can set this in infra/ansible in -inventory/group_vars/odcs in the odcs_allowed_clients_users variable. -See other releng users entries for format. - -==== Get the odcs token - -In order to generate an odcs compose, you need a openidc token. - -Run the odcs-token.py under `scripts/odcs/` from pagure releng -repository to generate the token. - -.... -$ ./odcs-token.py -.... - -==== Make sure rpms are written out with the right signature - -.... -$ koji write-signed-rpm eb10b464 openh264-2.2.0-1.fc38 -.... - -Where the key for that branch is listed, then the open264 package and -version. - -==== Generate a private odcs compose - -With the token generated above, generate the odcs private compose - -.... -$ python odcs-private-compose.py -.... - -`koji_tag`: fxx-openh264 (Openh264 builds are tagged to fxx-openh264 -tags where [.title-ref]#xx# represents the fedora release) - -`signingkeyid`: The short hash of the key for this Fedora branch. - -The composes are stored under `/srv/odcs/private/` dir on -`odcs-backend-releng01.iad2.fedoraproject.org` - -==== Pull the compose to your local machine - -We need to extract the rpms and tar them to send them to Cisco. In order -to that, first of all we need to pull the compose to our local machine. - -===== Move the compose to your home dir on odcs-backend-releng01.iad2.fedoraproject.org - -Since the compose is owned by [.title-ref]#odcs-server# pull it into -your home dir - -.... -$ mkdir ~/32-openh264 -$ sudo rsync -avhHP /srv/odcs/private/odcs-3835/ ~/32-openh264/ -$ sudo chown -R mohanboddu:mohanboddu ~/32-openh264/ -.... - -===== Sync the compose to your local machine - -Pull in the compose from your home dir on odcs releng backend to your -local machine into a temp working dir - -.... -$ mkdir openh264-20200813 -$ scp -rv odcs-backend-releng01.iad2.fedoraproject.org:/home/fedora/mohanboddu/32-openh264/ openh264-20200813/ -.... - -===== Make the changes needed - -Please follow the following commands to make the necessary tar files to -send to Cisco - -.... -$ cd openh264-20200813 -$ mkdir 32-rpms -# Copy rpms including devel rpms -$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*rpm 32-rpms/ -# Copy debuginfo rpms -$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*/*rpm 32-rpms/ -# copy the src.rpm -$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*src.rpm 32-rpms/ -$ cd 32-rpms -# Create the tar file with the rpms -$ tar -cJvf ../fedora-32-openh264-rpms.tar.xz *rpm -.... - -We need to send this tar file to Cisco along with the list of rpms in -each tarball. - -===== Syncing the compose to sundries01 - -Once we get a confirmation from Cisco that the rpms are updated on their -CDN, verify them by using curl. For example: - -.... -$ curl -I http://ciscobinary.openh264.org/openh264-2.1.1-1.fc32.x86_64.rpm -.... - -Now push these composes to *sundries01.iad2.fedoraproject.org* and -*mm-backend01.iad2.fedoraproject.org* - -On sundries01 we need to sync to a directory that is owned by _apache_, -so first we sync to the home directory on sundries01. Same with -mm-backend01 as the directory is owned by _root_. - -Create a temp working directory on sundries01 - -.... -$ ssh sundries01.iad2.fedoraproject.org -$ mkdir openh264-20200825 -.... - -Create a temp working directory on mm-backend01 - -.... -$ ssh mm-backend01.iad2.fedoraproject.org -$ mkdir openh264-20200825 -.... - -Then from your local machine, sync the compose - -.... -$ cd openh264-20200825 -$ rsync -avhHP 32-openh264 sundries01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825 -$ rsync -avhHP 32-openh264 mm-backend01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825 -.... - -On sundries01 - -.... -$ cd openh264-20200825 -$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/web/codecs.fedoraproject.org/openh264/32/ -.... - -On mm-backend01 - -.... -$ cd openh264-20200825 -$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/codecs.fedoraproject.org/openh264/32/ -.... - -===== Extra info - -Normally that should be it, but in some cases you may want to push -things out faster than normal, and here's a few things you can do to do -that: - -On mm-backend01.iad2.fedoraproject.org you can run: - -.... -# sudo -u mirrormanager /usr/local/bin/umdl-required codecs /var/log/mirrormanager/umdl-required.log -.... - -This will have mirrormanager scan the codecs dir and update it if it's -changed. - -On batcave01.iad2.fedoraproject.org you can use ansible to force all the -proxies to sync the codec content from sundries01: - -.... -# ansible -a '/usr/bin/rsync --delete -a --no-owner --no-group sundries01::codecs.fedoraproject.org/ /srv/web/codecs.fedoraproject.org/' proxies -.... - -Mirrorlist servers should update every 15min. diff --git a/modules/releng_misc_guide/pages/sop_openh264.adoc b/modules/releng_misc_guide/pages/sop_openh264.adoc new file mode 100644 index 0000000..a0f3263 --- /dev/null +++ b/modules/releng_misc_guide/pages/sop_openh264.adoc @@ -0,0 +1,206 @@ +== Generating OpenH264 RPMs and sending them to Cisco + +=== Description + +OpenH264 RPMs are a special case and we need to handle it with caution. + +The process is as follows: + +. We generate and sign the RPMs using `koji dist-repo`. +. We send the RPMs to Cisco for hosting. +. Once we get confirmation from Cisco that they published the RPMs on their CDN + we publish the repodata on our side. + +=== Legal background + +OpenH264 is a free software library for real-time encoding and decoding video +streams in the H.264/MPEG-4 AVC format. + +In 2013 Cisco released both binaries and source code, and pay all royalties for +its use to MPEG LA themselves for any software projects that use Cisco's +precompiled binaries (thus making Cisco's OpenH264 binaries free to use); +any software projects that use Cisco's source code instead of its binaries +would be legally responsible for paying all royalties to MPEG LA themselves, +however. + +The workaround is that the Cisco binaries are built in Fedora's infrastructure +but *distributed* by Cisco. + +[WARNING] +.Warning +==== +The RPMs *must never* be distributed via our build system or websites. +==== + +=== The process + +==== Permissions needed + +. Make sure to have a valid Kerberos token (`kinit`). +. Relevant FAS group membership. +. *TODO* + +==== Generating the repository + +Once a request comes in through the RelEng issue tracker to update OpenH264 to +a new version we will first need to generate the RPMs and repodata. We do this +using the `koji dist-repo` utility. + +[NOTE] +.Note +==== +The arguments that should be passed to the commands will certainly vary from the +examples listed below. + +Practice common sense and double check everything. +==== + +Run this command: + +.... +$ koji dist-repo f43-openh264 31645531 --noinherit --with-src -a aarch -a +ppc64le -a s390x -a x86_64 +.... + +In the case of the above command: + +. `f43-openh264` is the tag the builds are tagged into. +. `31645531` is the signing key for the RPMs. The keys for various versions + can be found https://fedoraproject.org/security[here]. +. `--noinherit` causes only packages tagged into `f43-openh264` to be included, + without inheritance from the parent tag (e.g. `f43`). +. `--with-src` causes the source RPM to be included. +. `-a ` options are the respective architectures. + +After the command finishes the directory should appear +https://kojipkgs.fedoraproject.org/repos-dist[here]. + +The RPMs are visible through the web frontend, but we have a rewrite rule that +redirects any attempt to download them to a wiki page saying why that is not +possible. + +In order to get to the RPMs themselves we must do it from a host on which Koji +is mounted where they will be in the `/mnt/koji/repos-dist/f43-openh264` directory. + +For example: + +.... +$ ssh compose-branched01.iad2.fedoraproject.org +.... + +After you verify that everything needed is there you may clone the directory to +your local machine. + +.... +$ rsync -avhHP username@compose-branched01.iad2.fedoraproject.org:/mnt/koji/repos-dist/f43-openh264/6555489 /home/username/openh264/2.6.0/f43 +.... + +==== Sending the RPMs to Cisco for hosting + +===== Preparing the tarball + +We now have the necessary directory on our local machine. Now we need to package +it into a tarball and send it to Cisco. + +Create a new directory into which we can copy the RPMs and out of which we will +create the tarball. + +.... +$ mkdir /home/username/openh264/2.6.0/f43/f43_2.6.0_rpms +.... + +Copy the RPMs (including the source RPM) from the original directory to the +new directory. + +.... +$ cd /home/username/openh264/2.6.0/f43/ +$ cp -rv 6553481/*/*/*/*rpm f43_2.6.0_rpms +.... + +Create a list of the RPMs that will be sent to Cisco along with the tarball. + +.... +$ ls f43_2.6.0_rpms > f43_2.6.0_rpms_list.txt +.... + +Create the tarball. + +.... +$ tar -cJvf f43_2.6.0_rpms.tar.xz f43_2.6.0_rpms +.... + +===== Sending the tarball to Cisco + +*TODO* + +[NOTE] +.Note +==== +We used to do this through e-mail. However, lately we have started experiencing issues +with e-mail filters (can't send tarballs). We are trying to reach Cisco about +possible solutions to this issue. +==== + +==== Publishing the repodata + +Once we receive confirmation from Cisco that their CDN was updated we can +verify it using `curl`. + +.... +$ curl -I http://ciscobinary.openh264.org/openh264-2.6.0-1.fc43.x86_64.rpm +.... + +Now it is time to publish the repodata on our infrastructure. +Navigate to the relevant directory on your local machine and sync the +`dist-repo` directory to your home directory on `sundries01`. + +.... +$ rsync -avhHP 6556232 sundries01.iad2.fedoraproject.org/home/fedora/username/openh264/2.6.0/43 +.... + +Connect to `sundries01`. + +.... +$ ssh sundries01.iad2.fedoraproject.org +.... + +Navigate to the relevant directory. + +.... +$ pwd +/home/fedora/username/openh264/2.6.0/43 +.... + +Sync the contents of the directory to `/srv/web/codecs.fedoraproject.org`. + +.... +$ sudo rsync -avhHP 6556232/. /srv/web/codecs.fedoraproject.org/openh264/43 +.... + +Make sure the directory is owned by root. + +.... +$ sudo chown -R root:root /srv/web/codecs.fedoraproject.org/openh264/43 +.... + +*TODO* + +[NOTE] +.Note +==== +This section will have to be improved with more details. +==== + +In the database: +.... +update directory set ctime=0 where name like 'ct.org/openh264/43%codecs.fedoraproject.org/openh264/43%'; +delete from repository where prefix like '%cisco%43%'; +.... + +In a MirrorManager OpenShift debug container: +.... +(app-root) sh-5.2$ scan-primary-mirror --category "Fedora Codecs" -d +.... + +=== Resources +. https://src.fedoraproject.org/rpms/openh264[OpenH264 dist-git repository]