Infrastructure/fedora-infrastructure

Fork 0

Don't drop user related tables from Anitya's db dump #7298

New issue

Closed

opened 2018-10-09 09:18:18 +00:00 by zlopez · 12 comments

zlopez commented

2018-10-09 09:18:18 +00:00

Contributor

Describe what you need us to do:
I found out, that Anitya's dump is missing these tables:
"users"
"tokens"
"social_auth_association"
"social_auth_code"
"social_auth_nonce"
"social_auth_partial"
"social_auth_usersocialauth"

It will be better if you can just leave these tables empty instead of dropping them entirely.

When do you need this? (YYYY/MM/DD)
Asap
When is this no longer needed or useful? (YYYY/MM/DD)
If we cannot complete your request, what is the impact?
Current development environment for Anitya is using production dump and without these tables you are getting error every time application works with user (login, check if admin etc.)

* Describe what you need us to do: I found out, that Anitya's dump is missing these tables: "users" "tokens" "social_auth_association" "social_auth_code" "social_auth_nonce" "social_auth_partial" "social_auth_usersocialauth" It will be better if you can just leave these tables empty instead of dropping them entirely. * When do you need this? (YYYY/MM/DD) Asap * When is this no longer needed or useful? (YYYY/MM/DD) * If we cannot complete your request, what is the impact? Current development environment for Anitya is using production dump and without these tables you are getting error every time application works with user (login, check if admin etc.)

bowlofeggs commented

2018-10-09 13:14:42 +00:00

Metadata Update from @bowlofeggs:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: database, release-monitoring

**Metadata Update from @bowlofeggs**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: database, release-monitoring

bowlofeggs commented

2018-10-09 13:18:33 +00:00

@zlopez This is the commit that made the change:

https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=f0c596c427cc591148634d31cbe521579fc462ff

You may be able to adjust it to insert the empty tables you desire, or you may be able to adjust the development environment to do so.

@zlopez This is the commit that made the change: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=f0c596c427cc591148634d31cbe521579fc462ff You may be able to adjust it to insert the empty tables you desire, or you may be able to adjust the development environment to do so.

zlopez commented

2018-10-09 14:24:52 +00:00

Author

Contributor

@bowlofeggs
I got advice from @mizdebsk on IRC, that this could be done by running sed '/COPY table_name /,/\./d' on dump. Is this possible, when the tables were excluded from actual dump?

@bowlofeggs I got advice from @mizdebsk on IRC, that this could be done by running `sed '/COPY table_name /,/\./d'` on dump. Is this possible, when the tables were excluded from actual dump?

bowlofeggs commented

2018-10-09 14:36:05 +00:00

On Tue, 2018-10-09 at 14:24 +0000, Michal Kone=C4=8Dn=C3=BD wrote:

I got advice from @mizdebsk on IRC, that this could be done by
running sed '/COPY table_name /,/\./d' on dump. Is this possible,
when the tables were excluded from actual dump?

The database dump is a text file, so there are a variety of ways you
could edit it to add an empty version of the table to it.

On Tue, 2018-10-09 at 14:24 +0000, Michal Kone=C4=8Dn=C3=BD wrote: > I got advice from @mizdebsk on IRC, that this could be done by > running `sed '/COPY table_name /,/\./d'` on dump. Is this possible, > when the tables were excluded from actual dump? The database dump is a text file, so there are a variety of ways you could edit it to add an empty version of the table to it.

zlopez commented

2018-10-09 14:38:17 +00:00

Author

Contributor

Ok, I will then just play with it.

mizdebsk commented

2018-10-09 14:43:44 +00:00

My idea was not to exclude these tables from pg_dump, but instead truncate them on-the-fly, eg with sed, i.e. replace pg_dump -T users $DB | ... with pg_dump $DB | sed '/COPY users /,/\./d' | ...

My idea was not to exclude these tables from pg_dump, but instead truncate them on-the-fly, eg with sed, i.e. replace `pg_dump -T users $DB | ...` with `pg_dump $DB | sed '/COPY users /,/\./d' | ...`

kevin commented

2018-10-09 18:02:31 +00:00

Note that these were dropped for security reasons, so make sure sensitive data isn't being published.

I think we can switch -T to --exclude-table-data for these tho and have the table exist, but have no data in it. Would that work?

Note that these were dropped for security reasons, so make sure sensitive data isn't being published. I think we can switch -T to --exclude-table-data for these tho and have the table exist, but have no data in it. Would that work?

zlopez commented

2018-10-09 18:44:52 +00:00

Author

Contributor

@kevin This will be ideal