Originally founded in 1614, the University of Paderborn is one of the oldest academic institutions in Germany. It was re-established in 1972 as Universität-Gesamthochschule (Institute for Higher Education), and subsequently transformed into the Universität Paderborn in its own right in 2002. Today the University is home to 18,800 students offering over 60 under-graduate and post-graduate degrees in the faculties of Arts, Humanities, Business and Sciences.
The new HPC cluster is located at the Paderborn Center for Parallel Computing - known as PC². PC² is an integral interdisciplinary institute of the University of Paderborn, specialising in distributed and parallel high performance computing. Staff and students at PC² work alongside collaboration partners to investigate a range of high performance computing topics in research, development and emerging practical applications.
"This system is a powerful compute resource for all researchers in the region of East Westphalia and Lippe, and our partners in Germany and Europe", stated Prof. Dr. Holger Karl, head of the PC² board.
Since its foundation in 1991 the PC² facility has housed several generations of the University's HPC technology, including several Top500 ranked systems. PC² currently has 5 HPC systems many of which are used to develop methods and principles for the future construction and efficient use of distributed and parallel computer systems. The new HPC cluster system will offer approximately 10,000 Intel Xeon processor cores arranged in 614 compute nodes, giving an anticipated performance in the range of 200 Tflops. A combination of 32 NVIDIA K20 GPUs and 8 Intel Xeon Phi co-processors provide 40 Tflops, giving it a high top-half ranking by the standards of the current Top500 listing.
The Paderborn HPC cluster will run a range of commercial and open source scientific applications. Manufacturing and engineering applications include the popular multi-physics Finite Element Analysis (FEA) codes ANSYS, Abaqus, and COMSOL, and the Computational Fluid Dynamics (CFD) applications Fluent, StarCD and OpenFOAM. Technical applications also include the commercial code MATLAB and its open source equivalent Octave, both used for performing and displaying numerical computations in engineering and mathematics. The application tools from Xilinx and Altera are used in integrated circuit design to program the hardware description language (HDL) for field programmable gate arrays (FPGA). The Paderborn cluster will also host a variety of popular applications for computational Physics and Chemistry. These include the molecular modelling codes Gromacs, Gaussian, MOE, ORCA, TurboMole, VASP, VMD and Siesta.
In addition to running HPC applications PC² works with the European Commission and other partners to conduct fundamental research and development in a number of HPC related areas. These include performance acceleration, fault tolerant middleware and system software, and network virtualisation.
Performance acceleration using custom machines and many-cores architecture is one of the forefront research projects at PC². Custom computing machines use massively parallel, programmable hardware, for example field-programmable gate arrays (FPGA), to build processing units that are highly tailored to specific applications. FPGA based systems have been shown to be able to accelerate computationally intensive applications by orders of magnitude over traditional architectures. Alongside FPGA's, researchers at PC² are also exploring the use of a number of other acceleration technologies. These include many-core architectures, GP-GPU coprocessors and floating point arrays. In designing the new Paderborn system, ClusterVision therefore worked closely with staff at PC² to incorporate some of the very latest acceleration technologies available.
As primary contractor for the design and build of the Paderborn cluster, ClusterVision brought together the technology and expertise of a large number of HPC partners, including ASUS, Intel, Dell, Supermicro, NVIDIA, Knürr Emerson, Bright Computing, and the Fraunhofer Institute.
The Paderborn system configuration comprises 614 compute nodes. Of these 572 are ASUS E7 rack-mount compute servers, each with 2 Intel Xeon E5-2670 (16 cores), 2.6GHz processors. The 552 smaller compute nodes have 64 GByte of main memory per node, whilst 20 larger nodes are enhanced with 256 GByte of main memory.
Compute power is significantly enhanced with 40 additional GPU nodes, arranged in 2 different configurations. Type-1 enhances the Intel Xeon E5-2670 processors with 32 NVIDIA Tesla K20 GPUs. Type-2 takes the same arrangement but uses 8 of Intel's own Xeon Phi co-processors to achieve the required acceleration. Running alongside other PC² systems this novel arrangement allows researchers to optimally run a range of applications, and specifically to investigate the relative performance of the leading HPC acceleration technologies which are available today.
The system also has 2 SMP nodes, using Dell PowerEdge R820 servers with four 8-Core Xeon E5-4650 processors, and 6 front-end and management and administration nodes in arranged in Supermicro Superserver 825TQ-R740WB and 1027GR-TRFchassis. Dell also provide the storage components for the 45 TByte capacity in its PowerVault MD3200 series units, along with much of the cabling and interconnect switchgear. The interconnect system itself is a fast QDR InfiniBand, 40 Gbit/s system from Mellanox Technologies.
All of the systems components are housed in 14 42U Emerson/Knürr server racks, 12 of which incorporate Knürr's backdoor chilled water cooling technology.
The software components of the cluster include Bright Cluster Manager from Bright Computing, which is used to provide the provisioning and administrative management of the system, and FraunhoferFS (FhGFS) the parallel file-system from the Fraunhofer Institutefor Technological and Industrial Mathematics (ITWM).
The size and complexity of the new Paderborn system required a high level of expertise in engineering assembly and on-site integration. ClusterVision began off-site engineering the assembly components at their headquarters in Amsterdam in November 2012, followed by an intensive 4-man on-site build at the end of the year. The build and software installation process culminated in a programme of system provisioning, burn-in testing, and benchmark tuning, with acceptance and handover being successfully completed in early February 2013. Post-delivery services include user support and maintenance for the Bright Cluster Manager and Fraunhofer FhGFS software installations and multi-year warranty and repair provisions for the critical hardware components.
"It is always exciting for our company to work on Top500 class systems like the new HPC cluster at Paderborn. Large scale, complex systems like this understandably represent a showcase of possibility to the HPC community, both in academia and commercial enterprise, and enable our team to draw upon and demonstrate all of their experience in system design and their expertise in build and configuration", stated Christopher Huggins, Commercial Director at ClusterVision.
ClusterVision were proud guests at the official opening of the new cluster which was held after the meeting of the ZKI Supercomputing Working Group at the University of Paderborn on 14/15 March 2013. The inauguration ceremony was attended by staff and students from Paderborn University and members of the PC² Management and Advisory boards. The proceedings included an opening address by Prof. Dr. Holger Karl, head of the board of PC², and greetings from Prof. Dr. Nikolaus Risch, president of the University of Paderborn, Dr. Oliver Hermann, president of the East Westphalia-Lippe University of Applied Sciences, Prof. Dr. Friedrich Biegler-King, Vice President for Planning and Infrastructure of the Bielefeld University of Applied Sciences and Prof. Dr. Gudrun Oevel, CIO of the University of Paderborn. The opening ceremony also showcased some of the scientific applications in system design and nano-engineering simulation through presentations from Prof. Dr. Volker Lindenstruth of the Goethe University Frankfurt and Prof. Dr.-Ing. Jadran Vrabec of the University of Paderborn.
"The engineering sciences must take full advantage of exponentially increasing computing power. The new HPC cluster in Paderborn offers excellent opportunities for research and development", stated Prof.Dr.-Ing. Jadran Vrabec, member of the PC² board.