18 Jun 2013 Leipzig - Paul Arts from Eurotech presented the DEEP project at the first BoF session on European exascale research at the ISC'13 event in Leipzig. The EC-funded DEEP project is addressing exascale architecture and monitoring. Paul Arts described how the new hardware architecture is paving way to exascale computing using Xeon Phi processors and the EXTOLL interconnect. He explained how the project tackles the problem of the vast amount of data while monitoring the architecture and he also showed how the DEEP project is using hot-water cooling for its solutions.
The DEEP project is involved in the design of a novel architecture to pave the way to exascale computing. The developers are constructing a Booster based on Intel's Xeon Phi processor technology and the EXTOLL interconnnect.
The project is guided by six Exascale applications in a joint resource-management and is trying to create a
transparent programming environment and performance analysis tools.
Paul Arts explained that the hardware is split into two diffentent components based on Infiniband fat tree and IBM BlueGene technology. The project is relying on both a highly scalable architecture but also Infiniband for a low-medium scalable architecture.
The DEEP project partners are developing a CPU cluster. Instead of adding accelerators to a cluster of CPUs, they are building a cluster of accelerators called BOOSTER. BOOSTER is a network of accelerators consisting of 256 nodes with 2 Xeon Phi processors connected in an Extoll 3D-Torus network.
In the BOOSTER architecture, each Xeon Phi is connected to 1 Extoll 3D-Torus lattice point. For the BOOSTER I/O node, there are on 8 nodes 2 links bridge between Inifiniband and Extoll.
Paul Arts told the audience that the developers need a few boards. Software and hardware development go in parallel. The hardware design has to be very efficient. Xeon Phi processors have to be cooled with water. The 2 lattice points have to process a massive amount of bandwidth.
The 3D-Torus is implemented, using the Extoll communication hardware. FPGA cards offer low latency.
The developers have set up an experiment with I/O links. They try to plug additional cables to change the algorithm.
The interface between the 3D Torus domain and the Infiniband domain is less scalable, as Paul Arts explained. The developers are using Mellanox switches.
The processor can also be used for collecting data to monitor the system. The focus is on measuring. Paul Arts said that the developers do not know how to measure the system to predict the behaviour. The measuring communication is taking place between the cluster and the BOOSTER.
Paul Arts explained that cooling the DEEP cluster is taking place at Juelich with hot and cold water. The system has a massive power density. The BOOSTER consumes up to 200 kW on 1.5 m2.
The hot-water cooling is done with Aurora direct liquid cooling. It is hot pluggable and has up to 50 degrees of water inlet.
The rack is built with each booster node over the backplane including 12 links x 12 lanes x 10 Gbps, up to 1.5 Tbps. The twin backplane is made out of 16 Booster nodes and 2 Booster interface nodes.
The distribution of electric power and the distribution of water is being monitored. The Booster monitoring is a safety monitoring with a real-time update frequency. Immediate local action is taken on failure. There is also standard monitoring with low update frequency. The selected sensors and events are only available at the central monitor.
Paul Arts said that the data is stored locally and the data access is handled globally
More information is available at the DEEP project website.