= Fedora ARM Infrastructure == Contact Information Owner:: Fedora Infrastructure Team Contact:: #fedora-admin, sysadmin-main, sysadmin-releng Location:: Phoenix Servers:: arm01, arm02, arm03, arm04 Purpose:: Information on working with the arm SOCs == Description We have 4 arm chassis in phx2, each containing 24 SOCs (System On Chip). Each chassis has 2 physical network connections going out from it. The first one is used for the management interface on each SOC. The second one is used for eth0 for each SOC. Current allocations (2016-03-11): arm01:: primary builders attached to koji.fedoraproject.org arm02:: primary arch builders attached to koji.fedoraproject.org arm03:: In cloud network, public qa/packager and copr instances arm04:: primary arch builders attached to koji.fedoraproject.org == Hardware Configuration Each SOC has: * eth0 and eth1 (unused) and a management interface. * 4 cores * 4GB ram * a 300GB disk SOCs are addressed by: .... arm{chassisnumber}-builder{number}.arm.fedoraproject.org .... Where chassisnumber is 01 to 04 and number is 00-23 == PXE installs Kickstarts for the machines are in the kickstarts repo. PXE config is on noc01. (or cloud-noc01.cloud.fedoraproject.org for arm03) The kickstart installs the latests Fedora and sets them up with a base package set. == IPMI tool Management The SOCs are managed via their mgmt interfaces using a custom ipmitool as well as a custom python script called 'cxmanage'. The ipmitool changes have been submitted upstream and cxmanage is under review in Fedora. The ipmitool is currently installed on noc01 and it has ability to talk to them on their management interface. noc01 also serves dhcp and is a pxeboot server for the SOCs. However you will need to add it to your path: .... export PATH=$PATH:/opt/calxeda/bin/ .... Some common commands: To set the SOC to boot the next time only with pxe: .... ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org chassis bootdev pxe .... To set the SOC power off: .... ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org power off .... To set the SOC power on: .... ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org power on .... To get a serial over lan console from the SOC: .... ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org -I lanplus sol activate .... == DISK mapping Each SOC has a disk. They are however mapped to the internal 00-23 in a non direct manner: [cols="1,1,1,1"] |=== |HDD Bay |EnergyCard |SOC (Port 1) |SOC Num |0 |0 | 3 | 03 |1 |0 | 0 | 00 |2 |0 | 1 | 01 |3 |0 | 2 | 02 |4 |1 | 3 | 07 |5 |1 | 0 | 04 |6 |1 | 1 | 05 |7 |1 | 2 | 06 |8 |2 | 3 | 11 |9 |2 | 0 | 08 |10 |2 | 1 | 09 |11 |2 | 2 | 10 |12 |3 | 3 | 15 |13 |3 | 0 | 12 |14 |3 | 1 | 13 |15 |3 | 2 | 14 |16 |4 | 3 | 19 |17 |4 | 0 | 16 |18 |4 | 1 | 17 |19 |4 | 2 | 18 |20 |5 | 3 | 23 |21 |5 | 0 | 20 |22 |5 | 1 | 21 |23 |5 | 2 | 22 |=== Looking at the system from the front, the bay numbering starts from left to right. == cxmanage The cxmanage tool can be used to update firmware or gather diag info. Until cxmanage is packaged, you can use it from a python virtualenv: .... virtualenv --system-site-packages cxmanage cd cxmanage source bin/activate pip install --extra-index-url=http://sources.calxeda.com/python/packages/ cxmanage deactivate .... Some cxmanage commands .... cxmanage sensor arm03-builder00-mgmt.arm.fedoraproject.org Getting sensor readings... 1 successes | 0 errors | 0 nodes left | . MP Temp 0 arm03-builder00-mgmt.arm.fedoraproject.org: 34.00 degrees C Minimum : 34.00 degrees C Maximum : 34.00 degrees C Average : 34.00 degrees C ... (and about 20 more sensors)... .... .... cxmanage info arm03-builder00-mgmt.arm.fedoraproject.org Getting info... 1 successes | 0 errors | 0 nodes left | . [ Info from arm03-builder00-mgmt.arm.fedoraproject.org ] Hardware version : EnergyCard X04 Firmware version : ECX-1000-v2.1.5 ECME version : v0.10.2 CDB version : v0.10.2 Stage2boot version : v1.1.3 Bootlog version : v0.10.2 A9boot version : v2012.10.16-3-g66a3bf3 Uboot version : v2013.01-rc1_cx_2013.01.17 Ubootenv version : v2013.01-rc1_cx_2013.01.17 DTB version : v3.7-4114-g34da2e2 .... firmware update: .... cxmanage --internal-tftp 10.5.126.41:6969 --all-nodes fwupdate package ECX-1000_update-v2.1.5.tar.gz arm03-builder00-mgmt.arm.fedoraproject.org .... (note that this runs against the 00 management interface for the chassis and updates all the nodes), and that we must run a tftpserver on port 6969 for firewall handling. == Links http://sources.calxeda.com/python/packages/cxmanage/ == Contacts help.desk@boston.co.uk is the contact to send repair requests to.