Add more spiders which do not seem to honour robots.txt

smooge commented

2024-07-08 14:16:13 +00:00

Contributor

I went through the last couple of logs afer the first round of 'turn
off the spiders' went out. I looked at the areas which the /robots.txt
disregard and then looked for the bots which ignored it and still
looked up stuff in 'accounts'. This may cut down CPU spikes as these
are looking at dynamic data which can 'blow' things up.

It might be good to add similar tooling to pagure and src since they
seem to be hit a lot in the logs also.

Signed-off-by: Stephen Smoogen ssmoogen@redhat.com

I went through the last couple of logs afer the first round of 'turn off the spiders' went out. I looked at the areas which the /robots.txt disregard and then looked for the bots which ignored it and still looked up stuff in 'accounts'. This may cut down CPU spikes as these are looking at dynamic data which can 'blow' things up. It might be good to add similar tooling to pagure and src since they seem to be hit a lot in the logs also. Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>

zuul commented

2024-07-08 14:20:49 +00:00

First-time contributor

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/6cbcf10e569b4d75b369dac91ef6beea

fi-ansible-lint-diff : SUCCESS in 4m 13s
fi-yamllint-diff : SUCCESS in 2m 39s

Build succeeded. https://fedora.softwarefactory-project.io/zuul/buildset/6cbcf10e569b4d75b369dac91ef6beea - [fi-ansible-lint-diff ](https://fedora.softwarefactory-project.io/zuul/build/47daba603cc3405a997add99eceef4dc) : SUCCESS in 4m 13s - [fi-yamllint-diff ](https://fedora.softwarefactory-project.io/zuul/build/a76eb4b8efb94f1aaa75f903c9165579) : SUCCESS in 2m 39s

smooge commented

2024-07-08 16:23:12 +00:00

Author

Contributor

2 new commits added

Add blocks to nagios.conf httpd
Add blockers to dl.fedoraproject.org

**2 new commits added** * ``Add blocks to nagios.conf httpd`` * ``Add blockers to dl.fedoraproject.org``

zuul commented

2024-07-08 16:27:52 +00:00

First-time contributor

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/dcb195cb44d54fae8de7fdb7babfe217

fi-ansible-lint-diff : SUCCESS in 4m 24s
fi-yamllint-diff : SUCCESS in 2m 33s

Build succeeded. https://fedora.softwarefactory-project.io/zuul/buildset/dcb195cb44d54fae8de7fdb7babfe217 - [fi-ansible-lint-diff ](https://fedora.softwarefactory-project.io/zuul/build/e587c6af272d4c919c2031aa3bb647c7) : SUCCESS in 4m 24s - [fi-yamllint-diff ](https://fedora.softwarefactory-project.io/zuul/build/35fd4cd429d5440d9f839f83eab5ed56) : SUCCESS in 2m 33s

zlopez commented

2024-07-09 09:19:05 +00:00

Contributor

rebased onto 377e83fdd1

rebased onto 377e83fdd1c854c4ed4645707aa7f09588f48203

zlopez commented

2024-07-09 09:19:13 +00:00

Contributor

rebased onto 377e83fdd1

rebased onto 377e83fdd1c854c4ed4645707aa7f09588f48203

zlopez commented

2024-07-09 09:20:32 +00:00

Contributor

Pull-Request has been merged by zlopez

zlopez commented

2024-07-09 10:18:26 +00:00

Contributor

Merged and deployed by running:

ansible-playbook /srv/web/infra/ansible/playbooks/groups/mailman.yml -t config,robots
ansible-playbook /srv/web/infra/ansible/playbooks/groups/download.yml -t config
ansible-playbook /srv/web/infra/ansible/playbooks/groups/noc.yml -t nagios_server
ansible-playbook /srv/web/infra/ansible/playbooks/groups/pagure.yml -t config
ansible-playbook /srv/web/infra/ansible/playbooks/groups/pkgs.yml -t config

Merged and deployed by running: ``` ansible-playbook /srv/web/infra/ansible/playbooks/groups/mailman.yml -t config,robots ansible-playbook /srv/web/infra/ansible/playbooks/groups/download.yml -t config ansible-playbook /srv/web/infra/ansible/playbooks/groups/noc.yml -t nagios_server ansible-playbook /srv/web/infra/ansible/playbooks/groups/pagure.yml -t config ansible-playbook /srv/web/infra/ansible/playbooks/groups/pkgs.yml -t config ```

Rows
Columns

Add more spiders which do not seem to honour robots.txt #2135