The IBM eServer p575

Introduction
HPC Architecture
  1. Shared-memory SIMD machines
  2. Distributed-memory SIMD machines
  3. Shared-memory MIMD machines
  4. Distributed-memory MIMD machines
  5. ccNUMA machines
  6. Clusters
  7. Processors
    1. AMD Opteron
    2. IBM POWER5+
    3. IBM BlueGene processors
    4. Intel Itanium 2
    5. Intel Xeon
    6. The MIPS processor
    7. The SPARC processors
  8. Networks
    1. Infiniband
    2. InfiniPath
    3. Myrinet
    4. QsNet
    5. SCI
Available systems
  1. The Bull NovaScale
  2. The C-DAC PARAM Padma
  3. The Cray X1E
  4. The Cray XT3
  5. The Cray XT4
  6. The Cray XMT
  7. The Fujitsu/Siemens M9000
  8. The Fujitsu/Siemens PRIMEQUEST
  9. The Hitachi BladeSymphony
  10. The Hitachi SR11000
  11. The HP Integrity Superdome
  12. The IBM eServer p575
  13. The IBM BlueGene/L&P
  14. The Liquid Computing LiquidIQ
  15. The NEC Express5800/1000
  16. The NEC SX-8
  17. The SGI Altix 4000
  18. The SiCortex SC series
  19. The Sun M9000
Systems disappeared from the list
Systems under development
Glossary
Acknowledgments
References

Machine type RISC-based distributed-memory multi-processor cluster
Models IBM eServer p575
Operating system AIX (IBMs Unix variant), Linux.
Connection structure Ω-switch
Compilers XL Fortran (Fortran 90), (HPF), XL C, C++
Vendors information Web page http://www-03.ibm.com/systems/p/hardware/enterprise.html
Year of introduction 2005 (8/16-CPU POWER5+ SMP)

System parameters:

Model eServer p575 eServer p575
Model Single Core Dual Core
Clock cycle 2.2 GHz 1.9 GHz
Theor. peak performance    
Per Proc. (64-bits) 8.8 Gflop/s 7.6 Gflop/s
Per node 70.4 Gflop/s 121.6 Gflop/s
Per 12-node frame 844.8 Gflop/s 1.46 Tflop/s
Maximal (512-node system) 36 Tflop/s 62 Tflop/s
Main memory    
Memory/node ≤ 256 GB ≤ 256 GB
Memory/maximal 131 TB 131 TB
No. of processors/frame 8–96 8–192
Communication bandwidth    
Node-to-node (bidirectional) 2 GB/s 2 GB/s

Remarks:

There is a multitude of high end servers in the eServer p-series. However, IBM singles out the POWER5+ based p575 model specifically for HPC. The eServer p575 is the successor of the RS/6000 SP. It retains much of the macro structure of this system: multi-CPU nodes are connected within a frame either by a dedicated switch or by other means, like switched Ethernet. The structure of the nodes, however, has changed considerably, see POWER5+. Up to 8 Dual Chip Modules (DCMs) are housed in a node totaling 8 or 16 cores in a node depending on whether the dual or single core version of the chip is used. For High-Performance Computing IBM recommends to employ the 8 CPU, single core nodes because a higher effective bandwidth from the L2 cache can be expected in this case. For less data intensive work that primarily uses the L1 cache the difference would be small while there is a large cost advantage using the 16-CPU nodes. The dual-core chips run at a lower clock frequency because of power dissipation considerations. The p575 is accessed through a front-end control workstation that also monitors system failures. Failing nodes can be taken off line and exchanged without interrupting service.

The so-called Federation switch is the fourth generation of the high-performance interconnects made for the p575 series. The Federation switch is, like its predecessors, an Ω-switch as described in the section on SM-MIMD systems. It has a bi-directional link speed of 2 GB/s and an MPI latency of 5–7 µs. Although we mentioned only the highest speed option for the communication, the high-performance switch, there is a wide range of other options that could be chosen instead, e.g., Infiniband or Gbit Ethernet is also possible.

Applications can be run using PVM or MPI. IBM used to support High Performance Fortran, both a proprietary version and a compiler from the Portland Group. It is not clear whether this is still the case. IBM uses its own PVM version from which the data format converter XDR has been stripped. This results in a lower overhead at the cost of generality. Also the MPI implementation, MPI-F, is optimised for the p575-based systems. As the nodes are in effect shared-memory SMP systems, within the nodes OpenMP can be employed for shared-memory parallelism and it can be freely mixed with MPI if needed. In addition to its own AIX OS IBM also supports some Linux distributions: both the professional versions of RedHat and SuSe Linux are available for the p575 series.

The standard commercial models that are marketed contain up to 128 nodes. However, on special request systems with up to 512 nodes can be built. This largest configuration is used in the table.

Measured Performances:
In [49] a performance of 75.8 Tflop/s for a 12,208 processor system, the ASC Purple, is reported for solving a dense linear system of order N = 1,383,600, yielding an efficiency of 82%. Note that this machine is a 763-node dual-core system that falls even outside the range of the commercially offered maximum of 512 nodes.