Back to Table of contents

Primeur weekly 2016-11-14

Exascale supercomputing

Berkeley Lab to lead AMR co-design centre for DOE's Exascale Computing Project ...

Exascale Computing Project announces $48 million to establish Exascale co-design centres ...

US Exascale Computing Project awards $34 million for software development ...

Quantum computing

Breakthrough in the quantum transfer of information between matter and light ...

Focus on Europe

European Commission reveals its forthcoming call for energy efficient, high performance processors ...

World-leading HPC centres partner to form accelerated computing institute ...

Atos Bull to boost Dutch research at SURFsara with first Bull sequana supercomputer installed ...

Middleware

Allinea tools yield a 50% speed up for genome applications at the Earlham Institute ...

NERSC's 'Shifter' scales up to more than 9,000 Cori KNL processors ...

DDN's Big Data storage provides Aalto University ample capacity and fast access to vital research data ...

DDN unveils industry's fastest multi-level security Lustre solution ...

DDN delivers new burst buffer appliance and updates block and file appliances, completing total product line refresh ...

Atos Bull tackles storage bottlenecks for High Performance Computing ...

Cycle Computing debuts the newest version of its groundbreaking CycleCloud ...

Hardware

University of Toronto selects CoolIT Systems to liquid cool signal processor for CHIME project ...

CoolIT Systems optimizes Trade and Match solution with custom closed-loop liquid cooling ...

SDSC to host high-speed, large data transfer experiment at SC16 Show ...

Cray XC40 "Theta" supercomputer accepted at Argonne National Laboratory ...

Cray launches next-generation supercomputer: the Cray XC50 ...

Cray reports third quarter 2016 financial results ...

Mellanox drives Virtual Reality to new levels with breakthrough performance ...

Mellanox announces 200Gb/s HDR InfiniBand solutions enabling record levels of performance and scalability ...

Computers made of genetic material? ...

CoolIT Systems to showcase best-in-class HPC liquid cooling offering at SC16 ...

Applications

BoschDoc, AHCODA-DB and OpenML winners of the Data Prize 2016 ...

Blue Waters simulates largest membrane channel made of DNA origami ...

Cray joins iEnergy, the oil and gas industry's foremost community for exploration and production ...

Large-scale computer simulations reveal biological growth processes ...

NASA science and technology advancements demonstrated at Supercomputing Conference ...

Unlocking big genetic datasets ...

Accelerating cancer research with deep learning ...

System opens up high-performance programming to non-experts ...

Studying structure to understand function within 'material families' ...

Chury is much younger than previously thought ...

TOP500

Global supercomputing capacity creeps up as Petascale systems blanket Top 100 ...

InfiniBand chosen by nearly 4x more end-users versus proprietary offerings in 2016 as shown on the TOP500 supercomputers list ...

The Cloud

SURFnet selects eight Cloud providers for Dutch education and research ...

NERSC's 'Shifter' scales up to more than 9,000 Cori KNL processors


7 Nov 2016 Berkeley - Shifter - the scalable software toolkit that leverages container-based computing to help supercomputer users run a wider range of software more easily and securely - is now better than ever. Developed at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) to improve the flexibility and usability of HPC systems for data-intensive workloads by leveraging container-based computing tools such as Docker, Shifter was originally released in 2015. Development of this open-source tool has been ongoing, and in October NERSC released Shifter 16.08.3.

The new version, which improves Shifter's functionality and also fixes some security bugs, is a recommended replacement for all previous releases, according to Doug Jacobsen, an HPC consultant at NERSC and one of the primary developers of Shifter. Doug Jacobsen recently tested Shifter's scalability using the new version and said the results were better than expected.

"We demonstrated that we can run complex, scientific, Python-based codes on more than 9,000 KNL cores", he stated. "This is significant because this is a tremendously challenging workload to run on large-scale platforms. Shifter's scalability brings into reach the ability for a large quantity of modern scientific codes to run at scale at NERSC."

From the get-go, Shifter was designed to allow a user to supply supercomputing centers such as NERSC with a Docker image that can then be converted to a form that can efficiently be distributed and accessed on the compute nodes at scale. Shifter leverages the user interface that Docker provides to allow users to define and create their software "images" - a snap shot of an operating environment for an application or a service that typically contains the required operating system software, libraries, and binaries in addition to anything specific to the application.

"Shifter is a productivity enhancer", stated Shane Canon, a project engineer at NERSC who has also been instrumental in the software package's development. "It makes it easier for users to develop something locally on their laptop and push to another place. It also enhances scientific reproducibility; being able to take that image and know that you can reliably instantiate it over and over or share it with others is really powerful."

Shifter leverages parts of the Docker ecosystem, such as Docker registries and Docker Hub, but does not directly use their software internally, Shane Canon emphasized. It is not a replacement for Docker functionality; it is specifically focused on HPC use cases.

"Shifter is strongly focused on the needs and requirements of the HPC workload, which means it can deliver the functionality we are looking for while still meeting the overall performance requirements and constraints that a large-scale HPC platform imposes", Doug Jacobsen stated.

The update also includes the ability to create per-node writable temporary spaces, even if the system lacks a local disk. "The new per-node write cache is particularly useful for data analysis stacks like Spark which often assume a local disk", Shane Canon explained. Using this capability, NERSC was able to run a Spark job across more than 1,600 nodes on its Cori system.

Over the past year, numerous experimental facilities and academic institutions have found that Shifter makes it much easier for them to run their data-intensive workloads in an HPC environment, Shane Canon noted.

"We continue to see adoption of Shifter not just at NERSC but at other sites", he stated, citing deployments at the Swiss National Supercomputing Centre (CSCS), the Victorian Life Sciences Computation Initiative in Australia and Argonne National Laboratory. Shifter has been particularly popular among the high-energy physics and genomics communities, which often have complex software stacks or use a large range of tools, he added.
Source: National Energy Research Scientific Computing Center - NERSC

Back to Table of contents

Primeur weekly 2016-11-14

Exascale supercomputing

Berkeley Lab to lead AMR co-design centre for DOE's Exascale Computing Project ...

Exascale Computing Project announces $48 million to establish Exascale co-design centres ...

US Exascale Computing Project awards $34 million for software development ...

Quantum computing

Breakthrough in the quantum transfer of information between matter and light ...

Focus on Europe

European Commission reveals its forthcoming call for energy efficient, high performance processors ...

World-leading HPC centres partner to form accelerated computing institute ...

Atos Bull to boost Dutch research at SURFsara with first Bull sequana supercomputer installed ...

Middleware

Allinea tools yield a 50% speed up for genome applications at the Earlham Institute ...

NERSC's 'Shifter' scales up to more than 9,000 Cori KNL processors ...

DDN's Big Data storage provides Aalto University ample capacity and fast access to vital research data ...

DDN unveils industry's fastest multi-level security Lustre solution ...

DDN delivers new burst buffer appliance and updates block and file appliances, completing total product line refresh ...

Atos Bull tackles storage bottlenecks for High Performance Computing ...

Cycle Computing debuts the newest version of its groundbreaking CycleCloud ...

Hardware

University of Toronto selects CoolIT Systems to liquid cool signal processor for CHIME project ...

CoolIT Systems optimizes Trade and Match solution with custom closed-loop liquid cooling ...

SDSC to host high-speed, large data transfer experiment at SC16 Show ...

Cray XC40 "Theta" supercomputer accepted at Argonne National Laboratory ...

Cray launches next-generation supercomputer: the Cray XC50 ...

Cray reports third quarter 2016 financial results ...

Mellanox drives Virtual Reality to new levels with breakthrough performance ...

Mellanox announces 200Gb/s HDR InfiniBand solutions enabling record levels of performance and scalability ...

Computers made of genetic material? ...

CoolIT Systems to showcase best-in-class HPC liquid cooling offering at SC16 ...

Applications

BoschDoc, AHCODA-DB and OpenML winners of the Data Prize 2016 ...

Blue Waters simulates largest membrane channel made of DNA origami ...

Cray joins iEnergy, the oil and gas industry's foremost community for exploration and production ...

Large-scale computer simulations reveal biological growth processes ...

NASA science and technology advancements demonstrated at Supercomputing Conference ...

Unlocking big genetic datasets ...

Accelerating cancer research with deep learning ...

System opens up high-performance programming to non-experts ...

Studying structure to understand function within 'material families' ...

Chury is much younger than previously thought ...

TOP500

Global supercomputing capacity creeps up as Petascale systems blanket Top 100 ...

InfiniBand chosen by nearly 4x more end-users versus proprietary offerings in 2016 as shown on the TOP500 supercomputers list ...

The Cloud

SURFnet selects eight Cloud providers for Dutch education and research ...