30 Sep 2015 Frankfurt - At the recent ISC Cloud and Big Data Conference in Frankfurt,Primeur Magazinehad the opportunity to talk with Alexander Wolf, professor at Imperial College London and coordinator of the HARNESS project which has been exploiting heterogeneous resources in Cloud data centres during the past three years. Alexander Wolf is very proud of and happy with the project results.
The goal of HARNESS was to develop a set of principles for how heterogeneous resources such as GPUs, FPGAs, and ASIC base routers, could be integrated into the Cloud platform. What Alexander Wolf and his partners have been able to do is demonstrate through advanced architecture that's been integrated with OpenStack how they can hide the peculiarities of these resources, at the same time as leveraging their advanced features. The result is a much better performance for a variety of difficult applications, including scientific computing, streamed data processing, and even MapReduce traditional sorts of Cloud processing systems.
Primeur Magazine:How many partners were there involved in this European project?
Alexander Wolf:There were six partners involved, including, aside from Imperial College London, the University of Rennes, École Polytechnique Fédérale de Lausanne in Switzerland, the industrial partners SAP and an FPGA company in London, called Maxeler. Maxeler sells a very interesting data flow engine device for accelerating applications. The last partner was the Konrad Zuse Institute which is a supercomputer centre in Berlin.
Primeur Magazine:If you have an accelerator, it is not sufficient to put your programme on it to make it faster. You need to do something more, right?
Alexander Wolf:One of the interesting things that we found is that accelerators do not always accelerate a programme. In fact, sometimes they can run slower, so you have to be very careful about how you manage and decide whether to schedule a particular job for an accelerator.
Primeur Magazine:If you schedule it for a certain accelerator, you can do some parts of the programme on the accelerator, and some parts on the GPU.
Alexander Wolf:Exactly. Being sensitive to which portions of an applications are appropriate for acceleration was a very fundamental part of the project. We built a system which can do automatic performance modelling to allow you to show which portions of the programme would benefit from acceleration. That led into an automatic scheduling mechanism to schedule the use of the resources when an application came to the Cloud.
Primeur Magazine:So people don't have to do the programming and scheduling themselves. You provide the system where this can automatically be performed?
Alexander Wolf:We did assume in the project that different implementations of components would be provided for the different heterogeneous devices. This is reasonable because there is a lot of work going on in the programming of these devices but the actual act of scheduling these tests for the devices and doing it automatically was the callenge for the project.
Primeur Magazine:If people want to use your system, can they do it?
Alexander Wolf:The goal of HARNESS was to demonstrate the ideas. We have a working system, integrated with OpenStack, and you certainly can find more about it by going to the HARNESS website. What is important to understand is that the world is becoming more heterogeneous, that our computing platforms are going to be more and more diverse, and until we have systems that can handle that diversity, we won't be able to really take good advantage of that wonderful technology.
Primeur Magazine:Thank you very much for that message.