The Maestro project has been set up to tackle one of the most important and difficult problems in HPC, namely the orchestration of data across multiple levels of the memory and storage hardware as well as the software stack. Although data movement is now recognized as the primary obstacle to performance efficiency, much of the software stack is not well suited to optimizing data movement, and was instead designed in an age where optimizing arithmetic operations was the priority. The Maestro project aims to capture the data- and memory-aware aspects of applications and the software stack into a new middleware layer which will perform basic data movement and optimisation on behalf of the application, also making use of modern memory systems.
Prof. Dirk Pleiter, Coordinator of the Maestro project, stated: "The Maestro project will provide a unique opportunity to challenge traditional approaches for handling data objects and data movements in complex HPC applications and workflows, which will be key for efficient exploitation of future exascale level supercomputers."
Maestro will build a data-aware and memory-aware middleware framework that addresses ubiquitous problems of data movement in complex memory hierarchies and at many levels of the HPC software stack. The Maestro consortium consists of 7 expert partners, each bringing specialist knowledge and expertise to the technical challenge. Forschungszentrum Juelich will provide overall coordination and leadership of the project, as well as being responsible for one example use-case. Cray will provide technical coordination and will lead the core middleware design efforts.
ECMWF will provide their production and research numerical weather prediction workflows as an example use-case to drive the direction of the project, and will co-design many aspects of the middleware. Seagate Technologies will develop an Object storage back-end to the middleware and interface Maestro to it. CEA will develop an in-situ framework based on the Maestro middleware. Appentra Solutions will enhance their Parallelware software for data awareness to support the Maestro middleware. CSCS operated by ETH Zurich will develop a dynamic provisioning system targeting data-intensive applications.