The new supercomputer of the Joint Institute for Nuclear Research has been developed and deployed for the Theoretical Physics Laboratory (TPL) named after N.N. Bogolyubov and the Information Technology Laboratory (ITL) with support of JINR management. The project was completed by RSC Group with Intel specialists' support.
The project involves plans of radical acceleration of comprehensive theoretical and experimental research in the fields of particle physics, nuclear physics and condensed matter physics. This includes deployment of the NICA accelerator complex based at JINR to recreate quark-gluon plasma - a special state of matter of the Universe in the first moments after the Big Bang. The NICA mega-project started in 2013 and it is planned to commission this accelerator facility in 2020.
The new supercomputer of the Joint Institute for Nuclear Research will enable resource-intensive massive parallel calculations to solve problems of lattice quantum chromodynamics for studying the properties of hadronic matter with high energy density and baryon charge and in presence of strong electromagnetic fields. It will greatly accelerate simulation of collision dynamics of relativist heavy ions and provide new resources to study the properties of highly correlated systems in the physics of new materials. It will also be used to develop and adapt software for the NICA mega-project for new computing architectures, create an innovative hardware and software environment for high performance computing and simulating complex scientific problems and to train IT specialists in all relevant areas.
"It's great to note that we launch the new heterogeneous supercomputer named after professor Govorun at JINR's Information Technology Laboratory in the year of 60th anniversary of commissioning of the first Ural-1 supercomputer at our institute in 1958. Our scientists and research groups now have a powerful and modern tool that will greatly accelerate theoretical and experimental research of nuclear physics and condensed matter physics", stated Vladimir Vasilyevich Korenkov, Director of the Information Technology Laboratory of the Joint Institute for Nuclear Research.
The new JINR supercomputer is a natural development of the HybriLIT heterogeneous platform with significant increase of performance. Theoretical peak performance of the new supercomputer is about 500 Teraflops (in double precision).
The new supercomputer includes the high dense and energy-efficient RSC Tornado solution based on Intel server products with 100% 'hot water' direct liquid cooling. The Joint Institute for Nuclear Research deployed new universal RSC Tornado computing cabinets with record-breaking power density and precision liquid cooling balanced for continuous usage of high temperature cooling agent (up to +63°C at cabinet entry point). JINR's specifications required an optimal cabinet temperature mode with +45°C constant cooling agent temperature at node entries (+57°C peak temperature).
'Hot water' mode enabled deployment of free cooling solution (24x7x365) using only dry coolers running at up to +50°C ambient temperature and complete elimination of freon circuit and chillers. As the result, average power efficiency factor (PUE) is less than 1.027. The system uses less than 3% of consumed power for cooling, which is an outstanding result for HPC industry.
The computing nodes of the JINR system are based on Intel server products: the most powerful 72-core Intel Xeon Phi 7290 processors, Intel Xeon Scalable family processors (Intel Xeon Gold 6154), Intel Server Board S7200AP and Intel Server Board S2600BP, Intel SSD DC S3520 with SATA interface in M.2 form-factor and the newest high-speed Intel SSD DC P45xx drives NVMe interface and 1TB capacity.
High-speed data transfer between JINR's supercomputer nodes uses innovative Intel Omni-Path Architecture communication technology providing up to 100 Gbps non-blocking switching rate. It is based on 48-port Intel Omni-Path Edge Switch 100 Series with 100% liquid cooling to provide high efficiency of the cooling system in 'hot water' mode and the lowest TCO of the system. Intel Omni-Path Architecture meets current requirements of resource-intensive user applications and provides sufficient throughput reserve for the future.
The innovative monitoring and management system based on RSC BasIS software stack guarantees high availability, redundancy and simplicity of usage of computing systems based on RSC solutions. It can be used to manage single nodes and the entire solution, including infrastructure components. All elements of the system - computing nodes, power supplies, hydraulic regulation modules, etc. - have an integrated management module providing broad capabilities for detailed telemetry and flexible management. Cabinet design supports replacement of computing nodes, power supplies and hydraulic regulation modules - with redundancy - in hot-swap mode without interruption of system operation. Most components of the system - such as computing nodes, power supplies, network and infrastructure components, etc. - are software- defined, and this significantly simplifies and speeds up initial deployment, maintenance and future upgrades of the system. 100% liquid cooling of all components ensures their longevity.
"We are glad that RSC's innovative HPC solutions will help to advance the Russian science and to improve the efficiency of international scientific research cooperation that has a long history in JINR", stated Alexander Moskovsky, CEO of RSC Group.
RSC has unique experience of developing, creating and operating high-density energy-efficient supercomputer facilities with direct liquid cooling. The company is a de-facto global technological expert in this field. RSC's key customers are higher education institutions (leading Russian universities), scientific institutions, research centres, laboratories and design agencies. Besides the new JINR project, RSC solutions are installed and actively used for simulations and computing of a wide range of research and industrial problems in St. Petersburg State Polytechnic University named after Peter the Great (SPbPU), the Joint Supercomputer Center of the Russian Academy of Sciences (JSCC RAS), the Siberian Supercomputer Center (SSCC) of the Institute of Computing Mathematics and Mathematical Geophysics of the Siberian Branch of RAS, South Urals State University (SUSU), the Oceanology Institute of RAS named after P. P. Shirshov, the Moscow Institute of Physics and Technology (MIPT), the Russian Weather Forecast Agency (Roshydromet) and other customers from various industries and fields including civil aircraft engine production, energy industry, computer graphics, oil and gas industry, etc.