From 1e071502590515b31464ad897ac44823e1930471 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michal=20Kone=C4=8Dn=C3=BD?= Date: Wed, 18 Aug 2021 13:29:22 +0200 Subject: [PATCH] Review database SOP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Michal Konečný --- modules/sysadmin_guide/nav.adoc | 2 +- modules/sysadmin_guide/pages/database.adoc | 96 +++++++++------------- 2 files changed, 41 insertions(+), 57 deletions(-) diff --git a/modules/sysadmin_guide/nav.adoc b/modules/sysadmin_guide/nav.adoc index 4a49993..a4428d8 100644 --- a/modules/sysadmin_guide/nav.adoc +++ b/modules/sysadmin_guide/nav.adoc @@ -17,7 +17,7 @@ ** xref:compose-tracker.adoc[Compose Tracker - SOP] ** xref:contenthosting.adoc[Content Hosting Infrastructure - SOP] ** xref:copr.adoc[Copr - SOP] -** xref:database.adoc[database - SOP in review ] +** xref:database.adoc[Database Infrastructure - SOP] ** xref:datanommer.adoc[datanommer - SOP in review ] ** xref:debuginfod.adoc[debuginfod - SOP in review ] ** xref:departing-admin.adoc[departing-admin - SOP in review ] diff --git a/modules/sysadmin_guide/pages/database.adoc b/modules/sysadmin_guide/pages/database.adoc index 446b1a5..4fb8e8a 100644 --- a/modules/sysadmin_guide/pages/database.adoc +++ b/modules/sysadmin_guide/pages/database.adoc @@ -2,35 +2,22 @@ Our database servers provide database storage for many of our apps. -Contents +== Contents -[arabic] -. Contact Information -. Description -. Creating a New Postgresql Database -. Troubleshooting and Resolution -+ -____ -[arabic] -.. Connection issues -.. Some useful queries -+ -____ -[arabic] -... What queries are running -... Seeing how "dirty" a table is -... XID Wraparound -____ -.. Restart Procedure -+ -____ -[arabic] -... Koji -... Bodhi -____ -____ -. Note about TurboGears and MySQL -. Restoring from backups or specific dbs +* <<_contact_information>> +* <<_description>> +* <<_creating_a_new_postgresql_database>> +* <<_troubleshooting_and_resolution>> +** <<_connection_issues>> +** <<_some_useful_queries>> +*** <<_what_queries_are_running>> +*** <<_seeing_how_dirty_a_table_is>> +*** <<_xid_wraparound>> +** <<_restart_procedure>> +*** <<_koji>> +*** <<_bodhi>> +* <<_turbogears_and_mysql>> +* <<_restoring_from_backups_or_specific_dbs>> == Contact Information @@ -39,7 +26,7 @@ Owner:: Contact:: #fedora-admin, sysadmin-main, sysadmin-dba group Location:: - Phoenix + iad2 Servers:: sb01, db03, db-fas01, db-datanommer02, db-koji01, db-s390-koji01, db-arm-koji01, db-ppc-koji01, db-qa01, dbqastg01 @@ -48,11 +35,11 @@ Purpose:: == Description -db01, db03 and db-fas01 are our primmary servers. db01 and db-fas01 run -PostgreSQL. db03 contain mariadb. db-koji01, db-s390-koji01, -db-arm-koji01, db-ppc-koji01 contain secondary kojis. db-qa01 and -db-qastg01 contain resultsdb. db-datanommer02 contains all storage -messages from postgresql database. +_db01_, _db03_ and _db-fas01_ are our primary servers. _db01_ and _db-fas01_ run +_PostgreSQL_. _db03_ contain _mariadb_. _db-koji01_, _db-s390-koji01_, +_db-arm-koji01_, _db-ppc-koji01_ contain secondary _kojis_. _db-qa01_ and +_db-qastg01_ contain _resultsdb_. _db-datanommer02_ contains all storage +messages from _postgresql_ database. == Creating a New Postgresql Database @@ -71,12 +58,14 @@ Say we have an app called "raffle". We'd have three users: database. It should not be used in day to day but only for things like updating the database schema when an update occurs. We could very likely disable this account in the db whenever we are not using it. + * raffleapp: the database user that the web application uses. This will likely need to be able to insert and select from all tables. It will probably need to update most tables as well. There may be some tables that it does _not_ need delete on. It should almost certainly not need schema modifying permissions. (With postgres, it likely also needs permission to insert/select on sequences as well). + * rafflereadonly: Only able to read data from tables, not able to modify anything. Sadly, we aren't using this often but it can be useful for scripts that need to talk directly to the database without modifying it. @@ -110,7 +99,7 @@ to incorporate it into the config file. See fas.pp for an example. There are no known outstanding issues with the database itself. Remember that every time either database is restarted, services will have to be -restarted (see below). +restarted (see <<_restart_procedure>>). === Some useful queries @@ -170,8 +159,6 @@ select datname, age(datfrozenxid), pow(2, 31) - age(datfrozenxid) as xids_remain from pg_database order by xids_remaining; .... -Information on [61]wraparound - == Restart Procedure If the database server needs to be restarted it should come back on it's @@ -185,7 +172,7 @@ service postgresql restart === Koji Any time postgreql is restarted, koji needs to be restarted. Please also -see [62]Restarting Koji +see <>. === Bodhi @@ -195,9 +182,8 @@ currently exists for this. == TurboGears and MySQL [NOTE] -.Note ==== -about TurboGears and MySQL +About TurboGears and MySQL There's a known bug in TurboGears that causes MySQL clients not to automatically reconnect when lost. Typically a restart of the TurboGears @@ -206,30 +192,28 @@ application will correct this issue. == Restoring from backups or specific dbs. -Our backups store the latest copy in /backups/ on each db server. These +Our backups store the latest copy in `/backups/` on each db server. These backups are created automatically by the db-backup script run fron cron. -Look in /usr/local/bin for the backup script. +Look in `/usr/local/bin` for the backup script. To restore partially or completely you need to: [arabic] . setup postgres on a system -. {blank} + -start postgres/run initdb:: - * {blank} - + - if this new system running postgres has already run ansible then it - will;; - have wrong config files in /var/lib/pgsql/data - clear them out - before you start postgres so initdb can work. -. {blank} +. start postgres/run initdb ++ +if this new system running postgres has already run ansible then it +will have wrong config files in `/var/lib/pgsql/data` - clear them out +before you start postgres so initdb can work. ++ +. grab the backups you need from `/backups` - also grab _global.sql_ ++ +edit up _global.sql_ to only create/alter the dbs you care about + -grab the backups you need from /backups - also grab global.sql:: - edit up global.sql to only create/alter the dbs you care about . as postgres run: `psql -U postgres -f global.sql` -. {blank} +. when this completes you can restore each db with (as postgres user) ++ +`createdb $dbname pg_restore -d dbname dbname_backup_file.db` + -when this completes you can restore each db with (as postgres user):::: - createdb $dbname pg_restore -d dbname dbname_backup_file.db . restart postgres and check your data.