Uppsala Multidisciplinary Center for Advanced Computational Science

The Rackham Cluster

The Rackham cluster was introduced in February 2017. Rackham is a SNIC resource and is estimated to be in production until first of January 2021. The major features of Rackham and its storage system Crex is found below. For more technical data please see the end of this article.

  • Rackham comprises 6080 cores in the form of 334 nodes with two 10-core Intel Xeon V4 CPU's each. 32 "fat" nodes will have 256 GB of memory, with the rest having 128 GB.
  • Rackham's storage system (named Crex) uses the Lustre file system and provides 1PB of storage.
  • The interconnect is Infiniband FDR which supports a theoretical bandwidth of 56Gb/s and a latency of 0.7 microseconds [1].

Research groups interested in acquiring dedicated resources are invited to contact UPPMAX to discuss potential extensions of Rackham, which may prove cost efficient, before the end of 2017. You may contact us at support@uppmax.uu.se

Technical data

Compute and system node hardware

Each compute node consists of two (2) Intel Xeon E5 2630 v4 at 2.20 GHz/core (10 cores, 20 threads, 25 MB LLC, and a bandwidth of 68.3 GB/s) [2]. Each compute node has either 8x16384MB (128GB) or 16x16384MB (256GB) of ECC 2400MHz DIMM DRAM memory. For local storage each compute node has a 2TB disk at 7200RPM .

Each system node also has two CPUs identical to a compute node. System nodes has 16x16384 (256GB) of ECC 2400MHz DIMM DRAM. For local storage each system node contains 4x600TB 10 000 RPM SAS disks for a total of 2.4 TB.

Crex

Rackham's storage system (named Crex) is a DDN (DataDirect Networks) EXAScaler filesystem based on the ES14KX platform. Crex uses 420 6TB NL-SAS drives and 24 300GB SAS drives for metadata storage. Total storage reserved for SNIC projects is 1PB. The filesystem is Lustre, a highly scalable filesystem common in HPC.

Network

The nodes and network uses Infiniband FDR with a maximum bandwidth of 56 Gb/s and a latency of 0.7 microseconds. The topology is fat tree with 14 leaf switches and 6 core switches. Each leaf switch connects 24 servers and reserves 12 ports for uplinks. The subscription rate is 2:1.

Operating system and software

All Rackham nodes uses CentOS 7. We provide most compilers and interpreters and provide requested software from our module system, where you will find build tools, bioinformatics tools and software such as MATLAB, GROMACS, VASP, OpenFOAM, COMSOL Multiphysics, RStudio, and more.

[1] http://www.mellanox.com/page/performance_infiniband

[2] https://ark.intel.com/sv/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz