The IBM/Numascale system, installed at the University of Oslo in 2012, consists of 72 IBM x3755 2U servers connected in a 3D torus with NumaConnect, using four cabinets with 18 servers apiece in a 3x6x4 topology. Each server has 24 cores and 64 GBytes, providing a single system image to all 1728 cores and 4.6 TBytes. The system was designed to meet user demand for "very large memory" hardware solutions running a standard single image Linux OS on commodity x86 based servers.
"Our new system with NumaConnect can be used as one single system or partitioned in smaller systems", stated Dr. Ole W. Saastad, Senior Analyst and HPC expert at USIT, the University of Oslo's central IT resource department. "With proper Numa-awareness, applications with high bandwidth requirements will be able to utilize the combined bandwidth of all of the memory controllers and still be able to share data with low latency access through the coherent shared memory."
"Eliminating the difficulty of MPI coding for large data problems has increased the productivity of our scientists who are not trained in MPI programming", Dr. Saastad continued. "Systems with NumaConnect now provide shared memory and MPI capabilities with the same cost structure as a cluster. This alternative represents a compelling solution for scientists who are used to working with their shared memory codes, who can now scale up their data sets without any extra effort."
The PRACE system is a prototype system used for theoretical studies and testing, for users at both USIT centre and at the PRACE partner entities. The single memory image cluster provides both shared memory - including threads and OpenMP - and MPI programming options. The scalable system takes advantage of COTS and NumaConnect to offer significant savings compared to conventional shared memory systems. In addition, system administration is identical to a single server because there are no separate node images to maintain and distribute.