= Fedora ARM Infrastructure

== Contact Information

Owner::
  Fedora Infrastructure Team
Contact::
  #fedora-admin, sysadmin-main, sysadmin-releng
Location::
  Phoenix
Servers::
  arm01, arm02, arm03, arm04
Purpose::
  Information on working with the arm SOCs

== Description

We have 4 arm chassis in phx2, each containing 24 SOCs (System On Chip).

Each chassis has 2 physical network connections going out from it. The
first one is used for the management interface on each SOC. The second
one is used for eth0 for each SOC.

Current allocations (2016-03-11):

arm01::
  primary builders attached to koji.fedoraproject.org
arm02::
  primary arch builders attached to koji.fedoraproject.org
arm03::
  In cloud network, public qa/packager and copr instances
arm04::
  primary arch builders attached to koji.fedoraproject.org

== Hardware Configuration

Each SOC has:

* eth0 and eth1 (unused) and a management interface.
* 4 cores
* 4GB ram
* a 300GB disk

SOCs are addressed by:

....
arm{chassisnumber}-builder{number}.arm.fedoraproject.org
....

Where chassisnumber is 01 to 04 and number is 00-23

== PXE installs

Kickstarts for the machines are in the kickstarts repo.

PXE config is on noc01. (or cloud-noc01.cloud.fedoraproject.org for
arm03)

The kickstart installs the latests Fedora and sets them up with a base
package set.

== IPMI tool Management

The SOCs are managed via their mgmt interfaces using a custom ipmitool
as well as a custom python script called 'cxmanage'. The ipmitool
changes have been submitted upstream and cxmanage is under review in
Fedora.

The ipmitool is currently installed on noc01 and it has ability to talk
to them on their management interface. noc01 also serves dhcp and is a
pxeboot server for the SOCs.

However you will need to add it to your path:

....
export PATH=$PATH:/opt/calxeda/bin/
....

Some common commands:

To set the SOC to boot the next time only with pxe:

....
ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org chassis bootdev pxe
....

To set the SOC power off:

....
ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org power off
....

To set the SOC power on:

....
ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org power on
....

To get a serial over lan console from the SOC:

....
ipmitool -U admin -P thepassword -H arm03-builder11-mgmt.arm.fedoraproject.org -I lanplus sol activate
....

== DISK mapping

Each SOC has a disk. They are however mapped to the internal 00-23 in a
non direct manner:

....
HDD Bay   EnergyCard  SOC (Port 1) SOC Num
0     0       3    03
1     0       0    00
2     0       1    01
3     0       2    02
4     1       3    07
5     1       0    04
6     1       1    05
7     1       2    06
8     2       3    11
9     2       0    08
10        2       1    09
11        2       2    10
12        3       3    15
13        3       0    12
14        3       1    13
15        3       2    14
16        4       3    19
17        4       0    16
18        4       1    17
19        4       2    18
20        5       3    23
21        5       0    20
22        5       1    21
23        5       2    22
....

Looking at the system from the front, the bay numbering starts from left
to right.

== cxmanage

The cxmanage tool can be used to update firmware or gather diag info.

Until cxmanage is packaged, you can use it from a python virtualenv:

....
virtualenv --system-site-packages cxmanage
cd cxmanage
source bin/activate
pip install --extra-index-url=http://sources.calxeda.com/python/packages/ cxmanage
<use cxmanage>
deactivate
....

Some cxmanage commands

....
cxmanage sensor arm03-builder00-mgmt.arm.fedoraproject.org 
Getting sensor readings...
1 successes  |  0 errors  |  0 nodes left  |  .  

MP Temp 0
arm03-builder00-mgmt.arm.fedoraproject.org: 34.00 degrees C
Minimum         : 34.00 degrees C
Maximum         : 34.00 degrees C
Average         : 34.00 degrees C
... (and about 20 more sensors)...
....

....
cxmanage info arm03-builder00-mgmt.arm.fedoraproject.org 
Getting info...
1 successes  |  0 errors  |  0 nodes left  |  .  

[ Info from arm03-builder00-mgmt.arm.fedoraproject.org ]
Hardware version   : EnergyCard X04
Firmware version   : ECX-1000-v2.1.5
ECME version       : v0.10.2
CDB version        : v0.10.2
Stage2boot version : v1.1.3
Bootlog version    : v0.10.2
A9boot version     : v2012.10.16-3-g66a3bf3
Uboot version      : v2013.01-rc1_cx_2013.01.17
Ubootenv version   : v2013.01-rc1_cx_2013.01.17
DTB version        : v3.7-4114-g34da2e2
....

firmware update:

....
cxmanage --internal-tftp 10.5.126.41:6969 --all-nodes fwupdate package ECX-1000_update-v2.1.5.tar.gz arm03-builder00-mgmt.arm.fedoraproject.org
....

(note that this runs against the 00 management interface for the chassis
and updates all the nodes), and that we must run a tftpserver on port
6969 for firewall handling.

== Links

http://sources.calxeda.com/python/packages/cxmanage/

== Contacts

help.desk@boston.co.uk is the contact to send repair requests to.
