2021-07-26 10:39:47 +02:00
|
|
|
= Fedora ARM Infrastructure
|
|
|
|
|
|
|
|
|
|
== Contact Information
|
|
|
|
|
|
|
|
|
|
Owner::
|
|
|
|
|
Fedora Infrastructure Team
|
|
|
|
|
Contact::
|
|
|
|
|
#fedora-admin, sysadmin-main, sysadmin-releng
|
|
|
|
|
Location::
|
|
|
|
|
Phoenix
|
|
|
|
|
Servers::
|
|
|
|
|
arm01, arm02, arm03, arm04
|
|
|
|
|
Purpose::
|
|
|
|
|
Information on working with the arm SOCs
|
|
|
|
|
|
|
|
|
|
== Description
|
|
|
|
|
|
|
|
|
|
We have 4 arm chassis in phx2, each containing 24 SOCs (System On Chip).
|
|
|
|
|
|
|
|
|
|
Each chassis has 2 physical network connections going out from it. The
|
|
|
|
|
first one is used for the management interface on each SOC. The second
|
|
|
|
|
one is used for eth0 for each SOC.
|
|
|
|
|
|
|
|
|
|
Current allocations (2016-03-11):
|
|
|
|
|
|
|
|
|
|
arm01::
|
|
|
|
|
primary builders attached to koji.fedoraproject.org
|
|
|
|
|
arm02::
|
|
|
|
|
primary arch builders attached to koji.fedoraproject.org
|
|
|
|
|
arm03::
|
|
|
|
|
In cloud network, public qa/packager and copr instances
|
|
|
|
|
arm04::
|
|
|
|
|
primary arch builders attached to koji.fedoraproject.org
|
|
|
|
|
|
|
|
|
|
== Hardware Configuration
|
|
|
|
|
|
|
|
|
|
Each SOC has:
|
|
|
|
|
|
|
|
|
|
* eth0 and eth1 (unused) and a management interface.
|
|
|
|
|
* 4 cores
|
|
|
|
|
* 4GB ram
|
|
|
|
|
* a 300GB disk
|
|
|
|
|
|
|
|
|
|
SOCs are addressed by:
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
arm{chassisnumber}-builder{number}.arm.fedoraproject.org
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
Where chassisnumber is 01 to 04 and number is 00-23
|
|
|
|
|
|
|
|
|
|
== PXE installs
|
|
|
|
|
|
|
|
|
|
Kickstarts for the machines are in the kickstarts repo.
|
|
|
|
|
|
|
|
|
|
PXE config is on noc01. (or cloud-noc01.cloud.fedoraproject.org for
|
|
|
|
|
arm03)
|
|
|
|
|
|
|
|
|
|
The kickstart installs the latests Fedora and sets them up with a base
|
|
|
|
|
package set.
|
|
|
|
|
|
|
|
|
|
== IPMI tool Management
|
|
|
|
|
|
|
|
|
|
The SOCs are managed via their mgmt interfaces using a custom ipmitool
|
|
|
|
|
as well as a custom python script called 'cxmanage'. The ipmitool
|
|
|
|
|
changes have been submitted upstream and cxmanage is under review in
|
|
|
|
|
Fedora.
|
|
|
|
|
|
|
|
|
|
The ipmitool is currently installed on noc01 and it has ability to talk
|
|
|
|
|
to them on their management interface. noc01 also serves dhcp and is a
|
|
|
|
|
pxeboot server for the SOCs.
|
|
|
|
|
|
|
|
|
|
However you will need to add it to your path:
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
export PATH=$PATH:/opt/calxeda/bin/
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
Some common commands:
|
|
|
|
|
|
|
|
|
|
To set the SOC to boot the next time only with pxe:
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org chassis bootdev pxe
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
To set the SOC power off:
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org power off
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
To set the SOC power on:
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org power on
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
To get a serial over lan console from the SOC:
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org -I lanplus sol activate
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
== DISK mapping
|
|
|
|
|
|
|
|
|
|
Each SOC has a disk. They are however mapped to the internal 00-23 in a
|
|
|
|
|
non direct manner:
|
|
|
|
|
|
2021-08-17 12:58:48 +02:00
|
|
|
[cols="1,1,1,1"]
|
|
|
|
|
|===
|
|
|
|
|
|HDD Bay |EnergyCard |SOC (Port 1) |SOC Num
|
|
|
|
|
|0 |0 | 3 | 03
|
|
|
|
|
|1 |0 | 0 | 00
|
|
|
|
|
|2 |0 | 1 | 01
|
|
|
|
|
|3 |0 | 2 | 02
|
|
|
|
|
|4 |1 | 3 | 07
|
|
|
|
|
|5 |1 | 0 | 04
|
|
|
|
|
|6 |1 | 1 | 05
|
|
|
|
|
|7 |1 | 2 | 06
|
|
|
|
|
|8 |2 | 3 | 11
|
|
|
|
|
|9 |2 | 0 | 08
|
|
|
|
|
|10 |2 | 1 | 09
|
|
|
|
|
|11 |2 | 2 | 10
|
|
|
|
|
|12 |3 | 3 | 15
|
|
|
|
|
|13 |3 | 0 | 12
|
|
|
|
|
|14 |3 | 1 | 13
|
|
|
|
|
|15 |3 | 2 | 14
|
|
|
|
|
|16 |4 | 3 | 19
|
|
|
|
|
|17 |4 | 0 | 16
|
|
|
|
|
|18 |4 | 1 | 17
|
|
|
|
|
|19 |4 | 2 | 18
|
|
|
|
|
|20 |5 | 3 | 23
|
|
|
|
|
|21 |5 | 0 | 20
|
|
|
|
|
|22 |5 | 1 | 21
|
|
|
|
|
|23 |5 | 2 | 22
|
|
|
|
|
|===
|
2021-07-26 10:39:47 +02:00
|
|
|
|
|
|
|
|
Looking at the system from the front, the bay numbering starts from left
|
|
|
|
|
to right.
|
|
|
|
|
|
|
|
|
|
== cxmanage
|
|
|
|
|
|
|
|
|
|
The cxmanage tool can be used to update firmware or gather diag info.
|
|
|
|
|
|
|
|
|
|
Until cxmanage is packaged, you can use it from a python virtualenv:
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
virtualenv --system-site-packages cxmanage
|
|
|
|
|
cd cxmanage
|
|
|
|
|
source bin/activate
|
|
|
|
|
pip install --extra-index-url=http://sources.calxeda.com/python/packages/ cxmanage
|
|
|
|
|
<use cxmanage>
|
|
|
|
|
deactivate
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
Some cxmanage commands
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
cxmanage sensor arm03-builder00-mgmt.arm.fedoraproject.org
|
|
|
|
|
Getting sensor readings...
|
|
|
|
|
1 successes | 0 errors | 0 nodes left | .
|
|
|
|
|
|
|
|
|
|
MP Temp 0
|
|
|
|
|
arm03-builder00-mgmt.arm.fedoraproject.org: 34.00 degrees C
|
|
|
|
|
Minimum : 34.00 degrees C
|
|
|
|
|
Maximum : 34.00 degrees C
|
|
|
|
|
Average : 34.00 degrees C
|
|
|
|
|
... (and about 20 more sensors)...
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
cxmanage info arm03-builder00-mgmt.arm.fedoraproject.org
|
|
|
|
|
Getting info...
|
|
|
|
|
1 successes | 0 errors | 0 nodes left | .
|
|
|
|
|
|
|
|
|
|
[ Info from arm03-builder00-mgmt.arm.fedoraproject.org ]
|
|
|
|
|
Hardware version : EnergyCard X04
|
|
|
|
|
Firmware version : ECX-1000-v2.1.5
|
|
|
|
|
ECME version : v0.10.2
|
|
|
|
|
CDB version : v0.10.2
|
|
|
|
|
Stage2boot version : v1.1.3
|
|
|
|
|
Bootlog version : v0.10.2
|
|
|
|
|
A9boot version : v2012.10.16-3-g66a3bf3
|
|
|
|
|
Uboot version : v2013.01-rc1_cx_2013.01.17
|
|
|
|
|
Ubootenv version : v2013.01-rc1_cx_2013.01.17
|
|
|
|
|
DTB version : v3.7-4114-g34da2e2
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
firmware update:
|
|
|
|
|
|
|
|
|
|
....
|
|
|
|
|
cxmanage --internal-tftp 10.5.126.41:6969 --all-nodes fwupdate package ECX-1000_update-v2.1.5.tar.gz arm03-builder00-mgmt.arm.fedoraproject.org
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
(note that this runs against the 00 management interface for the chassis
|
|
|
|
|
and updates all the nodes), and that we must run a tftpserver on port
|
|
|
|
|
6969 for firewall handling.
|
|
|
|
|
|
|
|
|
|
== Links
|
|
|
|
|
|
|
|
|
|
http://sources.calxeda.com/python/packages/cxmanage/
|
|
|
|
|
|
|
|
|
|
== Contacts
|
|
|
|
|
|
|
|
|
|
help.desk@boston.co.uk is the contact to send repair requests to.
|
