Back to Table of contents

Primeur weekly 2018-02-19

Exascale supercomputing

Ready for Exascale: researchers find algorithm for large-scale brain simulations on next-generation supercomputers ...

Quantum computing

Researchers demonstrate promising method for improving quantum information processing ...

Fingerprints of quantum entanglement ...

Focus on Europe

Karel Luyben appointed Dutch National Coordinator for Open Science ...

Middleware

An OLCF-developed visualization tool offers customization and faster rendering ...

Hardware

Dell EMC next generation converged infrastructure makes data centre modernization even simpler ...

Cray reports 2017 full year and fourth quarter financial results ...

Michael Levine and Ralph Roskies Day proclaimed in Pittsburgh and Allegheny County ...

MACH-2 put into operation at 77 trillion operations/second ...

Applications

CENIC recognizes technology projects to combat California wildfires ...

3D-e-Chem team develops building blocks and recipes for computer-aided drug discovery ...

Researchers find blood pressure drug holds promise for preventing onset of Type 1 diabetes ...

NCSA allocates over $2.4 million in new Blue Waters supercomputer awards to Illinois researchers ...

NCSA researchers create one of the most reliable tools for long-term crop prediction in the U.S. Corn Belt ...

Embracing complexity in biological systems ...

Particle interactions calculated on Titan support the search for new physics discoveries ...

GM revs up diesel combustion modeling on Titan supercomputer ...

NEC and Tohoku University succeed in AI-based new material development ...

Physics data processing at NERSC dramatically cuts reconstruction time ...

Advanced computing and water management at the AAAS Meeting 2018 ...

Neural networks everywhere ...

Supermassive black hole model predicts characteristic light signals at cusp of collision ...

New turbulent transport modelling shows multiscale fluctuations in heated plasma ...

TACC and DOD engage in four-year transformational design project ...

The Cloud

Atos signs key contract with the European Space Agency to enable new services with satellite data ...

Oracle buys Zenedge ...

NEC contributes to ecosystem for enhancing virtualized network orchestration ...

Physics data processing at NERSC dramatically cuts reconstruction time


From left: Berkeley Lab's Jan Balewski, Mustafa Mustafa, Jeff Porter.
6 Feb 2018 Berkeley - In a recent demonstration project, physicists from Brookhaven National Laboratory (BNL) and Lawrence Berkeley National Laboratory used the Cori supercomputer at the National Energy Research Scientific Computing Center (NERSC) to reconstruct data collected from a nuclear physics experiment, an advance that could dramatically reduce the time it takes to make detailed data available for scientific discoveries.

The researchers reconstructed multiple datasets collected by the STAR - Solenoidal Tracker At RHIC - detector during particle collisions at the Relativistic Heavy Ion Collider (RHIC), a nuclear physics research facility at BNL. By running multiple computing jobs simultaneously on the allotted supercomputing cores, the team transformed raw data into "physics-ready" data at the petabyte scale in a fraction of the time it would have taken using in-house high-throughput computing resources - even with a two-way transcontinental journey via ESnet, the Department of Energy's high-speed, high-performance data-sharing network that is managed by Berkeley Lab.

Preparing raw data for analysis typically takes many months, making it nearly impossible to provide such short-term responsiveness, according to Jérôme Lauret, a senior scientist at BNL and co-author on a paper outlining this work that was published in theJournal of Physics.

"This is a key usage model of high performance computing (HPC) for experimental data, demonstrating that researchers can get their raw data processing or simulation campaigns done in a few days or weeks at a critical time instead of spreading out over months on their own dedicated resources", stated Jeff Porter, a member of the data and analytics services team at NERSC and co-author on theJournal of Physicspaper.

The STAR experiment is a leader in the study of strongly interacting QCD matter that is generated in energetic heavy ion collisions. STAR consists of a large, complex set of detector systems that measure the thousands of particles produced in each collision event. Detailed analyses of billions of such collisions have enabled STAR scientists to make fundamental discoveries and measure the properties of the quark-gluon plasma. Since RHIC started running in the year 2000, this raw data processing, or reconstruction, has been carried out on dedicated computing resources at the RHIC and ATLAS Computing Facility (RACF) at BNL. High-throughput computing clusters crunch the data event by event and write out the coded details of each collision to a centralized mass storage space accessible to STAR physicists around the world.

In recent years, however, STAR datasets have reached billions of events, with data volumes at the multi-petabyte scale. The raw data signals collected by the detector electronics are processed using sophisticated pattern recognition algorithms to generate the higher-level datasets that are used for physics analysis. So the STAR computing team investigated the use of external resources to meet the demand for timely access to physics-ready data, ultimately turning to NERSC. Among other things, NERSC operates the PDSF cluster for the HEP/NP experiment community, which represents the second largest compute cluster available to the STAR collaboration.

Unlike the high-throughput computers at the RACF and PDSF, which analyze events one by one, HPC resources like those at NERSC break large problems into smaller tasks that can run in parallel. So the challenge was to parallelize the processing of STAR event data in a way that can scale out to run on large amounts of data with reproducible results.

The processing framework run at NERSC was built upon several core features. Shifter, a Linux container system developed at NERSC, provided a simple solution to the difficult problem of porting complex software to new computing systems and keep its expected behaviour. Scalability was achieved by eliminating bottlenecks in accessing both the event data and experiment databases that record environmental changes - voltage, temperature, pressure and other detector conditions—during data taking. To do this, the workload was broken up into data chunks, sized to run on a single node onto which a snapshot of the STAR database could also be stored. Each node was then self-sufficient, allowing the work to automatically expand out to as many nodes as available without any direct intervention.

"Several technologies developed in-house at NERSC allowed us to build a highly fault-tolerant, multi-step, data-processing pipeline that could scale to practically unlimited number of nodes with the potential to dramatically fold the time it takes to process data for many experiments", noted Mustafa Mustafa, a Berkeley Lab physicist who helped design the system.

Another challenge in migrating the task of raw data reconstruction to an HPC environment was getting the data from BNL in New York to NERSC in California and back. Both the input and output datasets are huge. The team started small with a proof-of-principle experiment - just a few hundred jobs - to see how their new workflow programs would perform. Colleagues at RACF, NERSC and ESnet - including Damian Hazen of NERSC and Eli Dart of ESnet - helped identify hardware issues and optimize the data transfer and the end-to-end workflow.

After fine-tuning their methods based on the initial tests, the team started scaling up, initially using 6,400 computing cores on Cori; in their most recent test they utilized 25,600 cores. The end-to-end efficiency of the entire process - the time the program was running (not sitting idle, waiting for computing resources) multiplied by the efficiency of using the allotted supercomputing slots and getting useful output all the way back to BNL - was 98 percent.

"This was a very successful large-scale data processing run on NERSC HPC", stated Jan Balewski, a member of the data science engagement group at NERSC who worked on this project. "One that we can look to as a reference as we actively test alternative approaches to support scaling up the computing campaigns at NERSC by multiple physics experiments."
Source: National Energy Research Scientific Computing Center - NERSC

Back to Table of contents

Primeur weekly 2018-02-19

Exascale supercomputing

Ready for Exascale: researchers find algorithm for large-scale brain simulations on next-generation supercomputers ...

Quantum computing

Researchers demonstrate promising method for improving quantum information processing ...

Fingerprints of quantum entanglement ...

Focus on Europe

Karel Luyben appointed Dutch National Coordinator for Open Science ...

Middleware

An OLCF-developed visualization tool offers customization and faster rendering ...

Hardware

Dell EMC next generation converged infrastructure makes data centre modernization even simpler ...

Cray reports 2017 full year and fourth quarter financial results ...

Michael Levine and Ralph Roskies Day proclaimed in Pittsburgh and Allegheny County ...

MACH-2 put into operation at 77 trillion operations/second ...

Applications

CENIC recognizes technology projects to combat California wildfires ...

3D-e-Chem team develops building blocks and recipes for computer-aided drug discovery ...

Researchers find blood pressure drug holds promise for preventing onset of Type 1 diabetes ...

NCSA allocates over $2.4 million in new Blue Waters supercomputer awards to Illinois researchers ...

NCSA researchers create one of the most reliable tools for long-term crop prediction in the U.S. Corn Belt ...

Embracing complexity in biological systems ...

Particle interactions calculated on Titan support the search for new physics discoveries ...

GM revs up diesel combustion modeling on Titan supercomputer ...

NEC and Tohoku University succeed in AI-based new material development ...

Physics data processing at NERSC dramatically cuts reconstruction time ...

Advanced computing and water management at the AAAS Meeting 2018 ...

Neural networks everywhere ...

Supermassive black hole model predicts characteristic light signals at cusp of collision ...

New turbulent transport modelling shows multiscale fluctuations in heated plasma ...

TACC and DOD engage in four-year transformational design project ...

The Cloud

Atos signs key contract with the European Space Agency to enable new services with satellite data ...

Oracle buys Zenedge ...

NEC contributes to ecosystem for enhancing virtualized network orchestration ...