The TOP500 list was announced on November 14, at the International Conference for High Performance Computing, Networking, Storage and Analysis, SC16, in Salt Lake City. It's announced semi-annually and is comprised of systems that are ranked according to their computation speeds as determined by the LINPACK Benchmark, which is a calculation of a linear system represented by a large double precision dense matrix.
SciPhi-XVI was delivered to Jefferson Lab and installed in mid-August and was deployed on September 30 with 200 nodes. An additional 64 nodes were added to the machine in late October, and the High Performance Computing group then benchmarked the machine. The SciPhi-XVI machine benchmarked at 425.8 Teraflops using 256 of its 264 nodes.
The cluster's 264 nodes are comprised of the second-generation Intel Xeon Phi chips, which were released in 2016 and are codenamed Knights Landing. Each node has 64 computing cores, and individual nodes within the machine are connected via the Intel Omni-Path Architecture, which supports a 100 Gigabit/second data transfer rate between the nodes.
"A special Intel distributed LINPACK benchmark software, HPL (High Performance Linpack), was used to rank the Jefferson Lab cluster, and it is based on an open source distribution with performance enhancement for Intel Xeon Phi processors", stated Jie Chen, a member of Jefferson Lab's High Performance Computing group. "To achieve the performance number for the SciPhi-XVI cluster, a set of HPL running parameters has to be determined through many trials. In particular, the size of the matrix representing the linear equations solved by the HPL was chosen to be 2,457,600."
Jefferson Lab conducts discovery-caliber nuclear physics research in exploring the atomic nucleus and its fundamental constituents, such as protons and neutrons and these particles' building blocks: quarks and gluons. The theory of how these particles interact - Quantum Chromodynamics, or QCD, is calculated in the form of lattice QCD, or LQCD.
Scientists will use the SciPhi-XVI machine for LQCD calculations, as well as a test system for the use of the Xeon Phi processors for analyzing data that is generated in Jefferson Lab's experimental physics programme. The Chroma code, which is used to solve LQCD on the Xeon Phi processor platform, will be used with an open source library of equation solvers for LQCD. The High Performance Computing group has been modernizing the freely available Chroma code, so that it will run faster and provide better results, more quickly on the new generations of hardware.
"There are always challenges to bringing online state-of-the-art hardware systems. We are very proud of our staff for their hard work and dedication that made this achievement possible", stated Jefferson Lab Chief Information Office, Amber Boehnlein. "This hardware achievement, together with our efforts to continually optimize the performance of our software for the newest systems, positions the laboratory to continue performing great science in high-performance computation."
Jefferson Lab work on Chroma is funded through the U.S. Department of Energy Office of Science. The research is carried out in collaboration with partner institutions, such as DOE's National Energy Research Scientific Computing Center (NERSC) Exascale Science Application Programme (NESAP) and the Scientific Discovery through Advanced Computing (SciDAC) programme. NERSC is a DOE Office of Science User Facility at Lawrence Berkeley National Laboratory.
Supercomputer computation time on SciPhi-XVI is allocated based on the recommendations of the United States Lattice Gauge Theory Computational Programme (USQCD), a consortium of top LQCD theorists in the United States that spans both high-energy physics and nuclear physics. The USQCD consortium developed the QUDA code library that is used with the Chroma code.