redeploy aws proxies #12469

New issue

Open

opened 2025-04-01 02:14:58 +00:00 by kevin · 6 comments

kevin commented

2025-04-01 02:14:58 +00:00

We have a number of proxies in aws:

inventory/host_vars/proxy09.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy30.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy31.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy32.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy33.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy34.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy35.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy36.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy37.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy38.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy39.fedoraproject.org:sponsor: amazon
inventory/host_vars/proxy40.fedoraproject.org:sponsor: amazon

Many/most of these are c5d.xlarge, but thats an older type. Thats 8GB memory and 4 cpus and it's not handling things too well these days.

So, we need to:

Find a better instance type. I think at least 16GB/8 cpus, but perhaps there's a better mix. We also want to make sure and have one disk with ~100GB of space.
Once we decide that, we need to take one out of dns and setup downtime in nagios one instance (probibly starting with proxy35 as it's been having the most problem lately)
Spin up the new one, add to dns, ansiblize.

We are entering final freeze, so we would need a freeze break to do these, but I think we can do so carefully, so we will need to request a freeze break...

@phsmoura was gonna work on this.

We have a number of proxies in aws: ``` inventory/host_vars/proxy09.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy30.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy31.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy32.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy33.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy34.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy35.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy36.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy37.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy38.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy39.fedoraproject.org:sponsor: amazon inventory/host_vars/proxy40.fedoraproject.org:sponsor: amazon ``` Many/most of these are c5d.xlarge, but thats an older type. Thats 8GB memory and 4 cpus and it's not handling things too well these days. So, we need to: 1. Find a better instance type. I think at least 16GB/8 cpus, but perhaps there's a better mix. We also want to make sure and have one disk with ~100GB of space. 2. Once we decide that, we need to take one out of dns and setup downtime in nagios one instance (probibly starting with proxy35 as it's been having the most problem lately) 3. Spin up the new one, add to dns, ansiblize. We are entering final freeze, so we would need a freeze break to do these, but I think we can do so carefully, so we will need to request a freeze break... @phsmoura was gonna work on this.

zlopez commented

2025-04-01 12:20:21 +00:00

Contributor

Metadata Update from @zlopez:

Issue assigned to phsmoura
Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: aws, high-gain, medium-trouble

**Metadata Update from @zlopez**: - Issue assigned to phsmoura - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: aws, high-gain, medium-trouble

phsmoura commented

2025-04-04 21:08:32 +00:00

I was reviewing AWS instance types, and the c5d.2xlarge stands out for having similar specifications. It offers 8 CPUs, 16 GiB of memory, and 200 GB of SSD storage. Some proxies are already utilizing it

kevin commented

2025-04-05 00:00:57 +00:00

Author

Looks like proxy09 is a c5n.2xlarge... thats the same cpus and disk, but 20GiB memory?

Perhaps one will work? Thats used by:

proxy09.fedoraproject.org: "ansible_product_name": "c5n.2xlarge",
proxy30.fedoraproject.org: "ansible_product_name": "c5n.2xlarge",
proxy31.fedoraproject.org: "ansible_product_name": "c5n.2xlarge",
proxy32.fedoraproject.org: "ansible_product_name": "c5n.2xlarge",
proxy38.fedoraproject.org: "ansible_product_name": "c5n.2xlarge",
proxy40.fedoraproject.org: "ansible_product_name": "c5n.2xlarge",

These are the ones that probibly we want to replace:

proxy33.fedoraproject.org: "ansible_product_name": "c5d.xlarge",
proxy34.fedoraproject.org: "ansible_product_name": "c5.2xlarge",
proxy35.fedoraproject.org: "ansible_product_name": "c5d.xlarge",
proxy36.fedoraproject.org: "ansible_product_name": "c5d.xlarge",
proxy37.fedoraproject.org: "ansible_product_name": "c5d.xlarge",
proxy39.fedoraproject.org: "ansible_product_name": "c5d.xlarge",

Thoughts?

Looks like proxy09 is a c5n.2xlarge... thats the same cpus and disk, but 20GiB memory? Perhaps one will work? Thats used by: proxy09.fedoraproject.org: "ansible_product_name": "c5n.2xlarge", proxy30.fedoraproject.org: "ansible_product_name": "c5n.2xlarge", proxy31.fedoraproject.org: "ansible_product_name": "c5n.2xlarge", proxy32.fedoraproject.org: "ansible_product_name": "c5n.2xlarge", proxy38.fedoraproject.org: "ansible_product_name": "c5n.2xlarge", proxy40.fedoraproject.org: "ansible_product_name": "c5n.2xlarge", These are the ones that probibly we want to replace: proxy33.fedoraproject.org: "ansible_product_name": "c5d.xlarge", proxy34.fedoraproject.org: "ansible_product_name": "c5.2xlarge", proxy35.fedoraproject.org: "ansible_product_name": "c5d.xlarge", proxy36.fedoraproject.org: "ansible_product_name": "c5d.xlarge", proxy37.fedoraproject.org: "ansible_product_name": "c5d.xlarge", proxy39.fedoraproject.org: "ansible_product_name": "c5d.xlarge", Thoughts?

phsmoura commented

2025-05-10 05:54:32 +00:00

done so far: proxy 35, 36 and 37

phsmoura commented

2025-06-05 16:26:00 +00:00

Updating, those are the proxies reinstalled so far:
proxy33.fedoraproject.org
proxy34.fedoraproject.org
proxy35.fedoraproject.org
proxy36.fedoraproject.org
proxy37.fedoraproject.org
proxy39.fedoraproject.org

Updating, those are the proxies reinstalled so far: proxy33.fedoraproject.org proxy34.fedoraproject.org proxy35.fedoraproject.org proxy36.fedoraproject.org proxy37.fedoraproject.org proxy39.fedoraproject.org

phsmoura commented

2025-06-13 21:39:29 +00:00

Finished all of them