Steve Conway introduced the topic of High Performance Data Analysis by defining the different items. Performance needs HPC resources. It is often near-real time and involves intelligent questions that are algorithmically complex. There is a huge variety in data types: some are partitionable and are handled by clusters; others are non-partitionable and are addressed by tightly coupled systems. As far as the analysis is concerned, this is a process of search and discovery by means of modelling, simulation, analytics, and visualization. Analysis happens in all domains, in governmental, industrial, commercial and academic environments.
HPDA addresses tasks involving sufficient data volumes and algorithmic complexity to require HPC resources with established simulation or newer analytics methods. The data can be structured, unstructured or both showing regular or irregular patterns. IDC also observes upward extensions of commercial business problems. HPDA presents the accumulated results of iterative problem-solving methods, including stochastic and parametric modelling.
The different parameters are velocity, variety, volume and value. Steve Conway stated that HPC architectures today are compute-centric (FLOPS vs. IOPS).
In partitionable Big Data work, most jobs are present. The goal is to search by means of regular access patterns, e.g. locality. The global memory is not important. Standard clusters, Hadoop, and Cassandra are being used. In non-partitionable work, the toughest jobs, such as graphing, are being handled. The goal is to discover by means of irregular access patters. The global memory is very important. The systems are being turbo-charged for data movement and graphing.
If we take a look at the IDC HPDA server forecast, we can observe a fast growth from a small starting point: $1.2 billion or 900 million euro by 2016. The total HPDA ecosystem will represent more than $2 billion or 1.5 billion euro in 2016.
Steve Conway ended his introductory presentation by giving some examples of HPDA use cases, both in science and industry. HPDA is paying big services in fraud and error detection, so it seems.