Back to Table of contents

Primeur weekly 2017-12-11

Quantum computing

NUS scientist develops 'toolboxes' for quantum cybersecurity ...

Focus on Europe

EUDAT CDI Council appoints Per Öster as Project Director of the EOSC-hub project ...

Verne Global delivers hpcDIRECT, a purpose-built, industrial scale HPC-as-a-Service platform ...

European researchers publish Quantum Software Manifesto ...

Commit2Data grant for better, faster and real-time adaptable Big Data analysis ...

Hartree Centre contributes essential computing resource to new UK National Biofilms Innovation Centre ...

Research for more efficient self-learning systems ...

Board members of the Finnish Union of University Professors visited the Netherlands eScience Center ...

Towards data storage at the single molecule level ...

Middleware

System Fabric Works and ThinkParQ become partners for highest speed Parallel File System ...

Hardware

IBM unveils industry's most advanced server designed for artificial intelligence ...

DDN's 5th annual High Performance Computing Trends survey cites complex I/O workloads as nr. 1 challenge ...

NVIDIA TITAN V transforms the PC into AI supercomputer ...

NVIDIA GPU Cloud now available to hundreds of thousands of AI researchers using NVIDIA desktop GPUs ...

New core facility boosts the University of Oregon's research computing power ...

New University of Queensland supercomputer to aid big research discoveries ...

Mellanox and NEC partner to deliver innovative high-performance and artificial intelligence platforms ...

Supermicro expands enterprise solutions portfolio with new scale-up SuperServer certified for SAP HANA ...

Applications

Cray and NERSC partnership to drive advanced AI development at scale ...

Computer simulations reveal roots of drug resistance ...

CMU receives $7.5 million in federal BRAIN initiative funding ...

New algorithm repairs corrupted digital images in one step ...

Neurons have the right shape for deep learning ...

The Cloud

Oracle open sources Kubernetes tools for serverless deployment and intelligent multi-Cloud management ...

Microsoft Azure becomes first global Cloud provider to deploy AMD EPYC ...

DDN's 5th annual High Performance Computing Trends survey cites complex I/O workloads as nr. 1 challenge


5 Dec 2017 Santa Clara - DataDirect Networks (DDN) has issued the results of its annual High Performance Computing (HPC) Trends survey, which reflects the continued adoption of flash-based storage as essential to respondent's overall data centre strategy. While flash is deemed essential, respondents anticipate needing additional technology innovations to unlock the full performance of their HPC applications. Managing complex I/O workload performance remains far and away the largest challenge to survey respondents, with 60 percent of end-users citing this as their number one challenge.

Run for the fifth year in a row by DDN, the survey results included input from more than 100 global end-users across a wide number of data-intensive industries. Respondents included individuals responsible for high performance computing as well as networking and storage systems from financial services, government, higher education, life sciences, manufacturing, national labs, and oil and gas organisations.

As expected, the amount of data under management in these organisations continues to grow. Of organisations surveyed:

  • 85 percent manage or use more than one petabyte of data storage (up 12-percentage points from last year).

Survey respondents continue to take a nuanced approach to Cloud adoption. Respondents planning to leverage Cloud-based storage - encompassing both private and public Clouds - for at least part of their data in 2017 jumped to 48 percent, an 11-percentage point increase from 2016 survey results. Despite a more positive disposition toward Cloud storage, only 5 percent of respondents anticipated more than 30 percent of their data residing in the Cloud. Maybe because of the limited use, as well as the ever-improving economics of public Cloud services, a full 40 percent of respondents anticipated using public Cloud in some way as a solution in the coming year even if for a limited amount of data. This response compares with only 20 percent of respondents last year who said they anticipated using public Cloud storage options.

While the basic application of flash storage in HPC data centres remains relatively flat, at approximately 90 percent of respondents using flash storage at some level within their data centres today, the main shift is in how much data is being retained in flash. While the vast majority of respondents (76 percent) store less than 20 percent of their data on flash media, many respondents anticipate an increase in 2018, with a quarter of respondents expecting 20-to-30 percent of their data to be flash based, and another 10 percent expecting 20-to-40 percent of their storage to be on a flash tier.

How customers are applying flash to their workflows is also particularly interesting. A majority of survey respondents (54 percent) are primarily using flash to accelerate file system metadata. There is a growing interest in using flash for application-specific data as well, with 45 percent of respondents indicating that they are using at least some of their flash storage this way. On the other end of the flash usage spectrum few customers are using flash user data, which is logical given current cost deltas between flash and spinning disk storage.

"Once again, DDN's annual HPC Trends Survey reflects the developments we see in the wider HPC community. I/O performance is a huge bottleneck to unlocking the power of HPC applications in use today, and customers are beginning to realize that simply adding flash to the storage infrastructure isn't delivering the anticipated application level improvements", stated Kurt Kuckein, director of marketing for DDN. "Suppliers are starting to offer architectures that include flash tiers optimized specifically for NVM and customers are actively pursuing implementations utilizing these technologies. Technologies like DDN's IME are specifically targeted to have the most impact on accelerating I/O all the way to the application."

I/O bottlenecks continue to be the main concern for HPC storage administrators. Especially in intensive I/O workflows like analytics, where 76 percent of customers running analytics workloads consider I/O their top challenge. Given this, it is not surprising that only 19 percent of survey participates consider existing storage technologies sufficient to scale to exascale requirements.

A majority of respondents (68 percent) view flash-native caches as the most likely technology to resolve the I/O challenge and to push HPC storage to the next level, reflecting an eight-percentage point increase versus last year's survey. HPC storage administrators have already, or are beginning to evaluate flash-native cache technologies at a greater rate than before, with more than 60 percent of the responses indicating that they have implemented, are evaluating now, or plan to evaluate flash-native cache solutions such as NVM. Evidence of the impact of these technologies can be seen in the recent io500.org results where JCAHPC utilized a flash-based cache to achieve stellar performance and the top spot in the first annual storage I/O benchmark ranking.

As an increasing number of HPC sites move to real-world implementation of multi-site HPC collaboration, concerns about security remain at the forefront. Perhaps somewhat surprisingly, the second highest barrier to multi-site collaboration has nothing to do with technology or security. Organisational bureaucracy was identified by 43 percent of respondents as a major impediment to data sharing. This means that even though data sharing has become technically possible as well as cost effective, there are still limiting perceptions that stand in the way of wider collaboration.

Source: DataDirect Networks - DDN

Back to Table of contents

Primeur weekly 2017-12-11

Quantum computing

NUS scientist develops 'toolboxes' for quantum cybersecurity ...

Focus on Europe

EUDAT CDI Council appoints Per Öster as Project Director of the EOSC-hub project ...

Verne Global delivers hpcDIRECT, a purpose-built, industrial scale HPC-as-a-Service platform ...

European researchers publish Quantum Software Manifesto ...

Commit2Data grant for better, faster and real-time adaptable Big Data analysis ...

Hartree Centre contributes essential computing resource to new UK National Biofilms Innovation Centre ...

Research for more efficient self-learning systems ...

Board members of the Finnish Union of University Professors visited the Netherlands eScience Center ...

Towards data storage at the single molecule level ...

Middleware

System Fabric Works and ThinkParQ become partners for highest speed Parallel File System ...

Hardware

IBM unveils industry's most advanced server designed for artificial intelligence ...

DDN's 5th annual High Performance Computing Trends survey cites complex I/O workloads as nr. 1 challenge ...

NVIDIA TITAN V transforms the PC into AI supercomputer ...

NVIDIA GPU Cloud now available to hundreds of thousands of AI researchers using NVIDIA desktop GPUs ...

New core facility boosts the University of Oregon's research computing power ...

New University of Queensland supercomputer to aid big research discoveries ...

Mellanox and NEC partner to deliver innovative high-performance and artificial intelligence platforms ...

Supermicro expands enterprise solutions portfolio with new scale-up SuperServer certified for SAP HANA ...

Applications

Cray and NERSC partnership to drive advanced AI development at scale ...

Computer simulations reveal roots of drug resistance ...

CMU receives $7.5 million in federal BRAIN initiative funding ...

New algorithm repairs corrupted digital images in one step ...

Neurons have the right shape for deep learning ...

The Cloud

Oracle open sources Kubernetes tools for serverless deployment and intelligent multi-Cloud management ...

Microsoft Azure becomes first global Cloud provider to deploy AMD EPYC ...