Conducted by DDN for the third consecutive year, the survey polled a cross-section of end-users managing data intensive infrastructures worldwide representing hundreds of petabytes of storage investment. Respondents included individuals responsible for high performance compute, networking and storage systems from Financial Services, Government, Higher Education, Life Sciences, Manufacturing, National Labs and Oil & Gas organisations. The data under management in each of these organisations is staggering.
Of organisations surveyed:
Storage is the fastest growing segment in IT spending in HPC, and according to an overwhelming majority of survey respondents (77 percent) data and data storage has now become the most strategic part of the HPC data centre as end-users seek to solve data access, workflow and analytics challenges to accelerate time to results.
The diverse set of applications driving this ever growing data in large scale environment and analytical workflows places rigorous demands on storage infrastructures and creates unique challenges for HPC users.
As the graph highlights:
More than half of respondents (56 percent) identify storage I/O as the main bottleneck in analytics workflows.
These responses illustrate how performance issues have surged as Big Data environments grapple with a proliferation of diverse applications creating mixed IO patterns and stifling the performance of their storage infrastructure.
Only a small percent of respondents believe today's file systems and data management technologies will be able to scale to Exascale levels, while almost 2/3 of respondents believe new innovation will be required. This belief is exemplified in respondents views on addressing performance issues:
Given the intense performance requirements of HPC environments as well as security, data access, workflow management and cost considerations, it is not surprising that according to 75 percent of those surveyed, security and data sharing complexity remain the largest impediments to increased collaboration across sites. This concern is underscored by respondents overwhelming preference for private Clouds:
With storage performance a critical requirement for today's large scale and petascale-level data centres, site-wide file systems like those at Oakridge National Lab (ORNL), National Energy Research Scientific Computing Center (NERSC) and Texas Advanced Computing Center (TACC) continue to be a significant infrastructure trend in HPC environments, according to more than two-thirds (67 percent) of HPC customers polled. Site-wide file systems allow architects to either consolidate multiple computers on the same system and/or provide the flexibility to upgrade storage and servers as needed.
"The results of DDNs annual HPC Trends Survey reaffirms the key issues and requisite requirements we hear from HPC users regularly. Handling mixed I/O workloads and resolving I/O bottlenecks in data intensive workloads are critical issues DDN has worked fervently to address within our suite of high performance end-to-end solutions", stated Molly Rector, CMO, executive vice president product management and worldwide marketing, DDN. "Survey respondents were clear that data storage has become the most strategic part of the data centre, and users rely on DDN to deliver continued technology innovation that enables research and business organisations to accelerate strategic insights and to speed time to results. DDN remains committed to the relentless innovation that raises the bar to meet the surging I/O performance, scale, content distribution and collaboration demands of high performance computing - in ways that are faster, smarter, more efficient and more cost effective than anyone else in the market."