infra-docs-fpo/modules/sysadmin_guide/pages/ipsilon.adoc

= Ipsilon Infrastructure SOP

== Contents

* <<_contact_information>>
* <<_description>>
* <<_known_issues>>
* <<_restarting>>
* <<_configuration>>
* <<_common_actions>>
** <<_openid_connect_scope_registration>>
** <<_generate_an_openid_connect_token>>
** <<_create_openid_connect_secrets_for_apps>>
* <<_troubleshooting>>

== Contact Information

Owner::
  Fedora Infrastructure Team
Contact::
  #fedora-admin
Primary upstream contact::
  Patrick Uiterwijk - FAS: puiterwijk
Backup upstream contact::
  Simo Sorce - FAS: simo (irc: simo) Howard Johnson - FAS: merlinthp
  (irc: MerlinTHP) Rob Crittenden - FAS: rcritten (irc: rcrit)
Servers::
* ipsilon01.iad2.fedoraproject.org
* ipsilon02.iad2.fedoraproject.org
* ipsilion01.stg.iad2.fedoraproject.org
Purpose::
  Ipsilon is our central authentication service that is used to
  authenticate users agains FAS. It is seperate from FAS.

== Description

Ipsilon is our central authentication agent that is used to authenticate
users agains FAS. It is seperate from FAS. The only service that is not
using this currently is the wiki. It is a web service that is presented
via httpd and is load balanced by our standard haproxy setup.

== Known issues

No known issues at this time. There is not currently a logout option for
ipsilon, but it is not considered an issue. If group memberships are
updated in ipsilon the user will need to wait a few minutes for them to
replicate to the all the systems.

== Restarting

To restart the application you simply need to ssh to the servers for the
problematic region and issue an `service httpd restart`. This should
rarely be required.

== Configuration

Configuration is handled by the ipsilon.yaml playbook in Ansible. This
can also be used to reconfigure application, if that becomes necessary.

== Common actions

This section describes some common configuration actions.

=== OpenID Connect Scope Registration

As documented on
https://fedoraproject.org/wiki/Infrastructure/Authentication,
application developers can request their own scopes. When a request for
this comes in, look in _ansible/roles/ipsilon/files/oidc_scopes/_ and copy
an example module. Copy this to a new file, so we have a file per scope
set. Fill in the information:

* _name_ is an Ipsilon-internal name. This should not include any spaces
* _display_name_ is the name that is displayed to the category of scopes
to the user
* _scopes_ is a dictionary with the full scope identifier (with namespace)
as keys. The values are dicts with the following keys:
** _display_name_: The complete display name for this scope. This is what
the user gets shown to accept/reject
** _claims_: A list of additional "claims" (pieces of user information) an
application will get when the user consents to this scope. For most scopes, this will be the empty list.

In _ansible/roles/ipsilon/tasks/main.yml_, add the name of the new file
(without .py) to the _with_items_ of "Copy OpenID Connect scope
registrations"). To enable, open
_ansible/roles/ipsilon/templates/configuration.conf_, and look for the
lines starting with "openidc enabled extensions". Add the name of the
plugin (in the "name" field of the file) to the environment this
scopeset has been requested for. Run the ansible _ipsilon.yml_ playbook.

=== Generate an OpenID Connect token

There is a handy script in the Ansible project under
`scripts/generate-oidc-token` that can help you generate an OIDC token.
It has a self-explanatory `--help` argument, and it will print out some
SQL that you can run against Ipsilon's database, as well as the token
that you seek.

The `SERVICE_NAME` (the required positional argument) is the name of the
application that wants to use the token to perform actions against
another service.

To generate the scopes, you can visit our
https://fedoraproject.org/wiki/Infrastructure/Authentication[authentication] docs and
find the service you want the token to be used for. Each service has a
base namespace (a URL) and one or more scopes for that namespace. To
form a scope for this script, you concatenate the namespace of the
service with the scope you want to grant the service. You can provide
the script the `-s` flag multiple times if you want to grant more than one
scope to the same token.

As an example, to give Bodhi access to create waivers in WaiverDB, you
can see that the base namespace is
`https://waiverdb.fedoraproject.org/oidc/` and that there is a
`create-waiver` scope. You can run this to generate Ipsilon SQL and a
token with that scope:

....
[bowlofeggs@batcave01 ansible][PROD]$ ./scripts/generate-oidc-token bodhi -e 365 -s https://waiverdb.fedoraproject.org/oidc/create-waiver

Run this SQL against Ipsilon's database:

--------START CUTTING HERE--------
BEGIN;
insert into token values ('2a5f2dff-4e93-4a8d-8482-e62f40dce046','username','bodhi@service');
insert into token values ('2a5f2dff-4e93-4a8d-8482-e62f40dce046','security_check','-ptBqVLId-kUJquqkVyhvR0DbDULIiKp1eqbXqG_dfVK9qACU6WwRBN3-7TRfoOn');
insert into token values ('2a5f2dff-4e93-4a8d-8482-e62f40dce046','client_id','bodhi');
insert into token values ('2a5f2dff-4e93-4a8d-8482-e62f40dce046','expires_at','1557259744');
insert into token values ('2a5f2dff-4e93-4a8d-8482-e62f40dce046','type','Bearer');
insert into token values ('2a5f2dff-4e93-4a8d-8482-e62f40dce046','issued_at','1525723744');
insert into token values ('2a5f2dff-4e93-4a8d-8482-e62f40dce046','scope','["openid", "https://someapp.fedoraproject.org/"]');
COMMIT;
-------- END CUTTING HERE --------


Token: 2a5f2dff-4e93-4a8d-8482-e62f40dce046_-ptBqVLId-kUJquqkVyhvR0DbDULIiKp1eqbXqG_dfVK9qACU6WwRBN3-7TRfoOn
....

Once you have the SQL, you can run it against Ipsilon's database, and
you can provide the token to the application through some secure means
(such as putting into Ansible's secrets and telling the requestor the
Ansible variable they can use to access it.)

=== Create OpenID Connect secrets for apps

Application wanting to use OpenID Connect need to register against our
OpenID Connect server (Ipsilon). Since we do not allow self-registration
(except on _iddev.fedorainfracloud.org_) for obvious reasons, the secrets
need to be created and configured per application and environment
(production vs staging).

To do so: - Go to the private ansible repository. - Edit the file:
`files/ipsilon/openidc.{{env}}.static` - At the bottom of this file, add
the information concerning the application you are adding. This will
look something like:

____
....
fedocal client_name="fedocal"
fedocal client_secret="<long random string>"
fedocal redirect_uris=["https://calendar.stg.fedoraproject.org/oidc_callback"]
fedocal client_uri="https://calendar.stg.fedoraproject.org/"
fedocal ipsilon_internal={"type":"static","client_id":"fedocal","trusted":true}
fedocal contacts=["admin@fedoraproject.org"]
fedocal client_id=null
fedocal policy_uri="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
fedocal grant_types="authorization_code"
fedocal response_types="code"
fedocal application_type="web"
fedocal subject_type="pairwise"
fedocal logo_uri=null
fedocal tos_uri=null
fedocal jwks_uri=null
fedocal jwks=null
fedocal sector_identifier_uri=null
fedocal request_uris=[]
fedocal require_auth_time=null
fedocal token_endpoint_auth_method="client_secret_post"
fedocal id_token_signed_response_alg="RS256"
fedocal request_object_signing_alg="none"
fedocal initiate_login_uri=null
fedocal default_max_age=null
fedocal default_acr_values=null
fedocal client_secret_expires_at=0
....
____

In most of situation, only the first 5 lines (up to `ipsilon_internal`)
will change. If the application is not using flask-oidc or is not
maintained by the Fedora Infrastructure the first 11 lines (up to
`application_type`) may change. The remaining lines require a deeper
understanding of OpenID Connect and Ipsilon.

[NOTE]
====
`client_id` in `ipsilon_internal` must match the begining of the line,
and the `client_id` field must either match the begining of the line or
be `null` as in the example here.
====
[NOTE]
====
In our OpenID connect server, `OIDC.user_getfield('nickname')` will return
the FAS username, which we know from FAS is unique. However, not all
OpenID Connect servers enforce this constraint, so the application code
may rely on the `sub` which is the only key that is sure to be unique.
If the application relies on `sub` and wants `sub` to return the FAS
username, then the configuration should be adjusted with:
`subject_type="public"`.
====
After adjusting this file, you will need to make the `client_secret`
available to the application via ansible, for this simply add it to
`vars.yml` as we do for the other private variables and provide the
variable name to the person who requested it.

Finally, commit and push the changes to both files and run the
`ipsilon.yml` playbook.

== Troubleshooting
Here is a guide one can follow when trying to troubleshoot authentication
issues within Fedora Infrastructure for users.

[NOTE]
====
This guide is meant for admins. If you don't have access to the machines bellow,
this guide will not help you.
====

. SSH to ipsilon machine (ipsilon01 or ipsilon02)
+
----
ssh ipsilon01
----
+
. Find the entry in logs
+
The logs are located in `/var/log/httpd/error_log`. You can find the entry either by
looking at the timestamps or looking for the `ipsilon_transaction_id`.
+
If you don't see the error in the ipsilon logs you can continue to next step.
+
. SSH to IPA machine (usually ipa01, there is also ipa02 and ipa03 that can be checked)
+
----
ssh ipa01
----
+
. Look into the logs
+
On ipa machine the logs could be found in `/var/log/httpd/error_log`, but it's hard to map
the ipsilon transaction to exact entry in the logs, so you can either map them together by time
or just looking for any error that seems related.