Primeur Magazine:What are the new developments during the past year for Tesla?
Keith Morris:The big news of the show is that NVIDIA is now powering the number 1 and number 3 supercomputers in the United States with the V100 Tensor Core GPU and not only that. These GPUs are now powering the most powerful supercomputers in the United States, Europe and Japan.
Primeur Magazine:This is all done with the Teslas?
Primeur Magazine:Do you know the biggest machines in Europe that you have with Tesla?
Keith Morris:Yes. It is the Piz Daint at the Swiss National Supercomputing Centre (CSCS) in Europe and the ABCI system in Japan.
Keith Morris:Tensor Core and GPUs support multiple precision. This means that they're great for HPC because one can do double precision, single precision and, now also with Tensor Cores, one can do mixed precision which is great for Artificial Intelligence (AI). This is really important as we see HPC and AI come together. On the AI front, one is borrowing techniques from the high-performance computing world and in high-performance computing people are finding ways to use AI to solve problems in different ways.
Primeur Magazine:Somewhere, there were also machines that that were said to have the biggest GPU with having one petaflop. Were they also built from Teslas or not?
Keith Morris:Just to give an example. This DGX-2 machine is actually two petaflops of AI performance and two petaflops of mixed precision. It has 16 Tensor Core GPUs interconnected by some new technology which is called NVSwitch. This allows any two GPUs to talk to each other at full bandwidth enabling you to treat it like it was one massive GPU. This is why one likes to call it the world's biggest GPU.
Primeur Magazine:There are of course different modules?
Keith Morris:Yes, but they all work together as one and they can work together on one problem.
Primeur Magazine:Is it correct that it is more than one petaflop?
Keith Morris:This is two petaflops of FP16, meaning FP16 input and FP32 output.
Primeur Magazine:And if one would do this with 64 bits?
Keith Morris:Well, 2025 flops with double precision.
Primeur Magazine:This is still very fast and impressive. What are the next developments of the technology?
Keith Morris:Volta GPUs are everywhere. The next development is enabling people with the software. One of the other announcements of the show was that NVIDIA introduced another nine containers for high-performance computing applications on its NVIDIA GPU Cloud (NGC) service. This is the way people can get the software and run the GPUs very easily, even if they are not experts in packaging software. If they are domain experts in a certain area, they can go straight on and do their science, rather than worrying about how to put all the software together to do that.
Primeur Magazine:Do you set up special collaborations with them to get the applications running?
Keith Morris:Yes, NVIDIA has a really strong programme in terms of working with developers and software vendors to get the software take advantage of GPUs. The company has over 550 applications accelerated for GPUs. The NVIDIA customers see tremendous value when they run their applications on a GPU-basis.
Primeur Magazine:You work together with universities on these topics too?
Keith Morris:Yes, NVIDIA has a programme called NVAIL, which stands for NVIDIA AI Labs, to sponsor universities around the world for doing research in AI. NVIDIA also has ambassadors around the world who work on accelerating their codes.
Primeur Magazine:If you have a lot of GPUs that you need in your application, how do you program that? How can you do that easily?
Keith Morris:There are different levels of getting access to the power of GPU. One way is to just use the NVIDIA libraries. NVIDIA develops libraries like cuBLAS, cuDNN, libraries that are supported by popular applications or frameworks. That is one level of leveraging GPU. The next thing is you can use something called open HCC which is a performance programming language that allows easy access to porting your code for acceleration. With that type of technology, you can very quickly, perhaps within one or two weeks, get a large performance boost on your application on GPUs. If you want to go further with the application in terms of finetuning it, then you can use NVIDIA's CUDA programming language to further optimize the code for your application.
Some of the things that NVIDIA does, is to make it easy for people to use multiple GPUs together. NVIDIA has a library called NCCL. If you have a machine like the DGX-2 with 16 GPUs inside, NVIDIA handles all of the communication between the GPUs and makes it look like one large GPU to the programmer. Programmers with this machine have 512 gigabytes of memory in one GPU. All of these GPUs are tightly coupled together with the technology NVSwitch, so it looks like one giant GPU.
The next thing is once we have accelerated the applications, how do you get those out to the users and make them easy to install on accelerated machines? For that, NVIDIA has a service called NVIDIA GPU Cloud. If somebody wants to get one of these accelerated applications, you can sign up for NVIDIA GPU Cloud which is free. NVIDIA provides ready-packaged HPC and Deep Learning applications for people that are ready to use. The company has gone through the trouble of putting all of the right software together, all the correct versions, the libraries and the applications together into a container. People can simply download that and get going with their applications straight away without having to worry about building that from scratch which could save them days or weeks of effort.
Primeur Magazine:You mentioned that the big machine, the Summit, will be used for HPC but also for AI. Is that a typical usage or are still most of the Teslas by NVIDIA used in AI?
Keith Morris:NVIDA has been doing HPC for over ten years and has a large installed base of HPC users but what the company is seeing, with Summit as an example, is that this is the new way people are going to build supercomputers. Mixed precision is going to open up lots of opportunities for people to solve problems in different ways. It is very important to have double precision for model simulation and single precision for model simulation but AI is going to be an important new tool to accelerate applications. Where researchers can take advantage of AI, they can see speed ups versus regular model simulation of thousands, if not hundreds of thousands of times. That makes problems that were thought once intractable now possible. That is really exciting.
Primeur Magazine:Thank you very much for this interview.