Back to Table of contents

Primeur weekly 2018-09-10

Quantum computing

Quantum computing in any language ...

Defiance ETFs launches QTUM, the Quantum Computing ETF, offering investors exposure to leaders in the field ...

Light exchange - A quantum gate between atoms and photons may help in scaling up quantum computers ...

Focus on Europe

cOAlition S: making Open Access a reality by 2020 ...

National Centre for Supercomputing Applications in Sofia, Bulgaria hosts PRACE Winter School on Computational Chemistry, Biochemistry and Medicinal chemistry - Methods and Tools ...

Middleware

DataDirect Networks completes $60 million Tintri acquisition and enters enterprise virtualization market ...

Bright Computing announces partnership with Japanese firm Towa Electric ...

Introduction of Altair Inspire and Altair 365 poised to transform product development ...

Hardware

DDN storage supporting breakthroughs in automomous vehicles and data-intensive research at the University of Michigan ...

Supermicro ships world's best performance, capacity and end-to-end throughput 1U all-flash storage server optimized with Samsung NF1 NVMe storage ...

Tech industry mentor Marian Gazdik joins Tachyum Board of Advisors ...

GigaSpaces brings enterprise-class real-time analytics, AI and extreme data processing to Kubernetes ...

Fujitsu launches liquid immersion cooling system that lowers total cost of ownership ...

Applications

NERSC, Intel and Cray harness the power of deep learning to better understand the Universe ...

Topology, physics and machine learning take on climate research data challenges ...

NCSA Faculty Fellow part of $25 million software institute to enable discoveries in high-energy physics ...

Miguel Holgado uses Blue Waters supercomputer to understand how binary compact-object mergers form ...

Veiled supernovae provide clue to stellar evolution ...

Experiment obtains entanglement of six light waves with a single laser ...

The Cloud

Baidu Cloud collaborates with Intel AI to advance financial services, shipping and video processing ...

NERSC, Intel and Cray harness the power of deep learning to better understand the Universe

Example simulation of dark matter in the universe, used as input to the CosmoFlow network.5 Sep 2018 Berkeley - A Big Data Center collaboration between computational scientists at Lawrence Berkeley National Laboratory's National Energy Research Scientific Computing Center (NERSC) and engineers at Intel and Cray has yielded another first in the quest to apply deep learning to data-intensive science: CosmoFlow, the first large-scale science application to use the TensorFlow framework on a CPU-based high performance computing platform with synchronous training. It is also the first to process three-dimensional (3D) spatial data volumes at this scale, giving scientists an entirely new platform for gaining a deeper understanding of the universe.

Cosmological "Big Data" problems go beyond the simple volume of data stored on disk. Observations of the universe are necessarily finite, and the challenge that researchers face is how to extract the most information from the observations and simulations available. Compounding the issue is that cosmologists typically characterize the distribution of matter in the universe using statistical measures of the structure of matter in the form of two- or three-point functions or other reduced statistics. Methods such as deep learning that can capture all features in the distribution of matter would provide greater insight into the nature of dark energy. First to realize that deep learning could be applied to this problem were Siamak Ravanbakhsh and his colleagues, as referenced in proceedings of the 33rd International Conference on Machine Learning . However, computational bottlenecks when scaling up the network and dataset limited the scope of the problem that could be tackled.

Motivated to address these challenges, CosmoFlow was designed to be highly scalable; to process large, 3D cosmology datasets; and to improve deep learning training performance on modern HPC supercomputers such as the Intel processor-based Cray XC40 Cori supercomputer at NERSC. CosmoFlow is built on top of the popular TensorFlow machine learning framework and uses Python as the front end. The application leverages the Cray PE Machine Learning Plugin to achieve unprecedented scaling of the TensorFlow Deep Learning framework to more than 8,000 nodes. It also benefits from Cray's DataWarp I/O accelerator technology, which provides the I/O throughput required to reach this level of scalability.

In a technical paper to be presented at SC18 in November, the CosmoFlow team describes the application and initial experiments using dark matter N-body simulations produced using the MUSIC and pycola packages on the Cori supercomputer at NERSC. In a series of single-node and multi-node scaling experiments, the team was able to demonstrate fully synchronous data-parallel training on 8,192 of Cori with 77% parallel efficiency and 3.5 Pflop/s sustained performance.

"Our goal was to demonstrate that TensorFlow can run at scale on multiple nodes efficiently", stated Deborah Bard, a Big Data architect at NERSC and a co-author of the technical paper. "As far as we are aware, this is the largest ever deployment of TensorFlow on CPUs, and we think it is the largest attempt to run TensorFlow on the largest number of CPU nodes."

Early on, the CosmoFlow team laid out three primary goals for this project: science, single-node optimization and scaling. The science goal was to demonstrate that deep learning can be used on 3D volumes to learn the physics of the universe. The team also wanted to ensure that TensorFlow ran efficiently and effectively on a single Intel Xeon Phi processor node with 3D volumes, which are common in science but not so much in industry, where most deep learning applications deal with 2D image data sets. And finally, ensure high efficiency and performance when scaled across 1000's of nodes on the Cori supercomputer system.

As Joe Curley, Sr. Director of the Code Modernization Organization in Intel's Data Center Group, noted: "The Big Data Center collaboration has produced amazing results in computer science through the combination of Intel technology and dedicated software optimization efforts. During the CosmoFlow project, we identified framework, kernel and communication optimization that led to more than 750x performance increase for a single node. Equally as impressive, the team solved problems that limited scaling of deep learning techniques to 128 to 256 nodes - to now allow the CosmoFlow application to scale efficiently to the 8,192 nodes of the Cori supercomputer at NERSC."

"We're excited by the results and the breakthroughs in artificial intelligence applications from this collaborative project with NERSC and Intel", stated Per Nyberg, vice president of market development, artificial intelligence and cloud at Cray. "It is exciting to see the CosmoFlow team take advantage of unique Cray technology and leverage the power of the a Cray supercomputer to effectively scale deep learning models. It is a great example of what many of our customers are striving for in converging traditional modelling and simulation with new deep learning and analytics algorithms, all on a single, scalable platform."

Prabhat, Group Leader of Data & Analytics Services at NERSC, added: "From my perspective, CosmoFlow is an exemplar project for the Big Data Center collaboration. We've truly leveraged competencies from various institutions to solve a hard scientific problem and enhance our production stack, which can benefit the broader NERSC user community."

In addition to Deborah Bard and Prabhat, co-authors on the SC18 paper include Amrita Mathuriya, Lawrence Meadows, Lei Shao, Tuomas Karna, John Pennycook, Jason Sewall, Nalini Kumar and Victor Lee from Intel; Peter Mendygral, Diana Moise, Kristyn Maschhoff and Michael Ringenburg from Cray; Siyu He and Shirley Ho from the Flatiron Institute; and James Arnemann from UC Berkeley.

Source: National Energy Research Scientific Computing Center - NERSC

Back to Table of contents

Primeur weekly 2018-09-10

Quantum computing

Quantum computing in any language ...

Defiance ETFs launches QTUM, the Quantum Computing ETF, offering investors exposure to leaders in the field ...

Light exchange - A quantum gate between atoms and photons may help in scaling up quantum computers ...

Focus on Europe

cOAlition S: making Open Access a reality by 2020 ...

National Centre for Supercomputing Applications in Sofia, Bulgaria hosts PRACE Winter School on Computational Chemistry, Biochemistry and Medicinal chemistry - Methods and Tools ...

Middleware

DataDirect Networks completes $60 million Tintri acquisition and enters enterprise virtualization market ...

Bright Computing announces partnership with Japanese firm Towa Electric ...

Introduction of Altair Inspire and Altair 365 poised to transform product development ...

Hardware

DDN storage supporting breakthroughs in automomous vehicles and data-intensive research at the University of Michigan ...

Supermicro ships world's best performance, capacity and end-to-end throughput 1U all-flash storage server optimized with Samsung NF1 NVMe storage ...

Tech industry mentor Marian Gazdik joins Tachyum Board of Advisors ...

GigaSpaces brings enterprise-class real-time analytics, AI and extreme data processing to Kubernetes ...

Fujitsu launches liquid immersion cooling system that lowers total cost of ownership ...

Applications

NERSC, Intel and Cray harness the power of deep learning to better understand the Universe ...

Topology, physics and machine learning take on climate research data challenges ...

NCSA Faculty Fellow part of $25 million software institute to enable discoveries in high-energy physics ...

Miguel Holgado uses Blue Waters supercomputer to understand how binary compact-object mergers form ...

Veiled supernovae provide clue to stellar evolution ...

Experiment obtains entanglement of six light waves with a single laser ...

The Cloud

Baidu Cloud collaborates with Intel AI to advance financial services, shipping and video processing ...