Back to Table of contents

Primeur weekly 2020-05-18

Focus

SiPearl's chip will be a platform open to other start-ups to develop accelerators for - An interview with Philippe Notton from SiPearl ...

Exascale supercomputing

Atos launches first supercomputer equipped with NVIDIA A100 GPU ...

Quantum computing

Start of the Fraunhofer Competence Center "Quantum Computing Baden-Württemberg" ...

ADVA brings post-quantum security to packet networks ...

Atos delivers its Quantum Learning Machine to Japan ...

Machine learning cracks quantum chemistry conundrum ...

Making quantum 'waves' in ultrathin materials ...

Seeqc UK awarded GBP 1,8 million in grants to advance quantum computing initiatives ...

Rising investments in quantum technology to boost market growth ...

Total is exploring quantum algorithms to improve CO2 capture ...

Middleware

Mentor Graphics and the University of Basel join the OpenMP effort ...

Hardware

NVIDIA's new Ampere data centre GPU in full production ...

Karlsruhe Institute of Technology procures new supercomputer ...

DDN announces A3I suppport for NVIDIA DGX A100 ...

Yvonne Walker recognized as one of CRN's 2020 Women of the Channel ...

Supermicro expands portfolio with fully integrated NVIDIA A100 GPU-powered systems delivering 5 Petaflop/s of AI performance in a single 4U server ...

Artificial synapses on design ...

Light, fantastic: the path ahead for faster, smaller computer processors ...

OpenFabrics Alliance (OFA) and Gen-Z Consortium announce MoU agreement ...

Tachyum achieves 90 percent of silicon laid for its Prodigy Universal Processor ...

Intel Capital invests $132 million in 11 disruptive technology start-ups ...

Inspur releases five new AI servers powered by NVIDIA A100 Tensor Core GPUs ...

Research and Markets issues Global High-Performance Computing as a Service Market Analysis and Industry Forecasts to 2026 ...

Asperitas and maincubes partner to deliver immersion cooling solutions in dedicated colocation suites ...

Delivery of supercomputer Fugaku completed ...

Applications

Supercomputer simulations help advance electrochemical reaction research ...

3D-printed nuclear reactor promises faster, more economical path to nuclear energy ...

GCS centres support research to mitigate impact of COVID-19 pandemic ...

NIST scientists create new recipe for single-atom transistors ...

ACM service awards recognize leaders who strengthen the computing community ...

New research launched on airborne virus transmission in buildings ...

Seeing the universe through new lenses ...

Liqid delivers industry's fastest single-socket server ...

National Science Foundation grant backs funcX: A smart, automated delegator for computational research ...

Educating the next generation of supercomputer users with Blue Waters ...

The Cloud

Innovium introduces TERALYNX 8, world's highest performance programmable switch for data centre networks with 25,6 Tbps throughput and support for 112 Gbps SerDes I/O ...

Wiwynn unveils standalone rack-level liquid cooling solution for OCP ORV3 at 2020 OCP Virtual Summit ...

Crowd Machine Platform is powered by Oracle Cloud and now available in the Oracle Cloud Marketplace ...

National Science Foundation grant backs funcX: A smart, automated delegator for computational research


14 May 2020 Chicago - Computational scientific research is no longer one-size-fits-all. The massive datasets created by today's cutting-edge instruments and experiments - telescopes, particle accelerators, sensor networks and molecular simulations - aren't best processed and analyzed by a single type of machine. For faster and more efficient discovery, data can be chopped up and shipped to specialized resources, including supercomputers, campus clusters, Cloud data centres, and "accelerators" optimized for specific tasks such as machine learning or visualization.

But delegating chunks of data and analysis functions to their ideal destination isn't trivial. A team led by University of Chicago CS researchers Ian Foster and Kyle Chard and Daniel S. Katz of the National Center for Supercomputing Applications (NCSA) at the University of Illinois seeks to streamline the process with funcX, a new distributed "function-as-a-service" (FaaS) platform that makes it easier for researchers to easily and automatically delegate their computational workload. With a pair of grants to the Universities of Chicago and Illinois from the National Science Foundation (NSF) totalling $3,14 million, the team will work with several large science projects and cyberinfrastructure partners to build and test this new system for "computationally fluid" research.

"Exploding data volumes and velocities, new computational methods and platforms, and ubiquitous connectivity demand new approaches to computation in the sciences", stated Ian Foster, Director of Argonne's Data Science and Learning Division, Argonne Senior Scientist and Distinguished Fellow and the Arthur Holly Compton Distinguished Service Professor of Computer Science at the University of Chicago. "funcX makes it easy for scientists to run computations wherever it makes the most sense and move computations between resources. It will enable new science, reduce barriers and democratize access to advanced cyberinfrastructure, and enable researchers to compute wherever is most efficient."

The scientific software used to process and extract discoveries from experimental data is typically made up of tens to thousands of smaller functions, blocks of code that handle individual jobs in the long pipeline of data analysis. These programmes can be run in their entirety on a single system - be it a laptop, a campus cluster, or a supercomputer - but that uniform approach may not be optimal. Some complex tasks may need to run on high-performance computing resources, but some specialized functions may be better served by GPU accelerators, and the more routine jobs could be tackled by small, energy-efficient computers.

Many obstacles prevent scientists from splitting their applications in this manner. Software is usually designed to run on a single type of machine or system, even if in parallel, and it is difficult to write code that splits off functions and sends them to different destinations. Beyond the programming, additional barriers exist for moving data to the desired computational endpoint, obtaining the proper security authorizations, and scheduling usage between different remote resources based on availability.

funcX solves this problem by adapting the serverless "function-as-a-service" (FaaS) model for the specialized needs and resources of the science and engineering research community. Many of today's mobile apps and Internet of Things devices use this approach for their sporadic computational demand - offloading more intensive tasks that can’t be handled on their smaller, local computers.

"The research ecosystem comprises a range of existing systems: from HPC clusters to Clouds and supercomputers", stated Kyle Chard, Research Assistant Professor at the University of Chicago CS. "Researchers have varied workloads, large amounts of distributed data, and dynamic collaborations, so there isn't a one-size-fits-all mapping from workloads to systems. funcX will make it easy to distribute these functions to different computing systems without needing to think about their differences, and will integrate with the scientific ecosystem for authentication, data management, and heterogeneous computing infrastructure."

The funcX platform builds upon two existing research technologies: Globus, a research data management platform created at the University of Chicago and Argonne National Laboratory, and Parsl, a Python library for executing parallel workflows created at the University of Chicago, Argonne National Laboratory, and the University of Illinois. funcX will also use Amazon Web Services for hosting management services, and integrate cyberinfrastructure from campuses and national laboratories, such as Blue Waters, Stampede2, and XSEDE.

The project will work with a wide range of scientific partners studying urban science, materials science, quantum chemistry, neuro-anatomy, cosmology, and other data-intensive subjects. These partners will provide use cases and testbeds for the funcX platform, working with the designers to create powerful, flexible features and an ecosystem of users and computational endpoints.

"We've got an amazing group of collaborators from different science domains, research computing centres and national cyberinfrastructure providers, and NSF software institutes that will help shape this project", Kyle Chard stated. "We see this engagement as vital for not only understanding needs across a range of domains but also for piloting the system and demonstrating value in their domain. This will help us ensure that funcX is not only valuable for our partners but also more broadly to all researchers."

This project is a collaboration between the Universities of Chicago and Illinois, following their successful partnership in developing Parsl.

"Since its creation by NSF and the state of Illinois in 1986, NCSA has been a leader in working with and supporting scientific and engineering communities through the development, deployment, and use of new computing and software technologies", stated Daniel S. Katz, Assistant Director for Scientific Software and Applications at the National Center for Supercomputing Applications at the University of Illinois. "This project exemplifies our effort to develop leading-edge software that enables new research on our systems."

You can follow the project online at the funcx website.
Source: University of Chicago - Department of Computer Science

Back to Table of contents

Primeur weekly 2020-05-18

Focus

SiPearl's chip will be a platform open to other start-ups to develop accelerators for - An interview with Philippe Notton from SiPearl ...

Exascale supercomputing

Atos launches first supercomputer equipped with NVIDIA A100 GPU ...

Quantum computing

Start of the Fraunhofer Competence Center "Quantum Computing Baden-Württemberg" ...

ADVA brings post-quantum security to packet networks ...

Atos delivers its Quantum Learning Machine to Japan ...

Machine learning cracks quantum chemistry conundrum ...

Making quantum 'waves' in ultrathin materials ...

Seeqc UK awarded GBP 1,8 million in grants to advance quantum computing initiatives ...

Rising investments in quantum technology to boost market growth ...

Total is exploring quantum algorithms to improve CO2 capture ...

Middleware

Mentor Graphics and the University of Basel join the OpenMP effort ...

Hardware

NVIDIA's new Ampere data centre GPU in full production ...

Karlsruhe Institute of Technology procures new supercomputer ...

DDN announces A3I suppport for NVIDIA DGX A100 ...

Yvonne Walker recognized as one of CRN's 2020 Women of the Channel ...

Supermicro expands portfolio with fully integrated NVIDIA A100 GPU-powered systems delivering 5 Petaflop/s of AI performance in a single 4U server ...

Artificial synapses on design ...

Light, fantastic: the path ahead for faster, smaller computer processors ...

OpenFabrics Alliance (OFA) and Gen-Z Consortium announce MoU agreement ...

Tachyum achieves 90 percent of silicon laid for its Prodigy Universal Processor ...

Intel Capital invests $132 million in 11 disruptive technology start-ups ...

Inspur releases five new AI servers powered by NVIDIA A100 Tensor Core GPUs ...

Research and Markets issues Global High-Performance Computing as a Service Market Analysis and Industry Forecasts to 2026 ...

Asperitas and maincubes partner to deliver immersion cooling solutions in dedicated colocation suites ...

Delivery of supercomputer Fugaku completed ...

Applications

Supercomputer simulations help advance electrochemical reaction research ...

3D-printed nuclear reactor promises faster, more economical path to nuclear energy ...

GCS centres support research to mitigate impact of COVID-19 pandemic ...

NIST scientists create new recipe for single-atom transistors ...

ACM service awards recognize leaders who strengthen the computing community ...

New research launched on airborne virus transmission in buildings ...

Seeing the universe through new lenses ...

Liqid delivers industry's fastest single-socket server ...

National Science Foundation grant backs funcX: A smart, automated delegator for computational research ...

Educating the next generation of supercomputer users with Blue Waters ...

The Cloud

Innovium introduces TERALYNX 8, world's highest performance programmable switch for data centre networks with 25,6 Tbps throughput and support for 112 Gbps SerDes I/O ...

Wiwynn unveils standalone rack-level liquid cooling solution for OCP ORV3 at 2020 OCP Virtual Summit ...

Crowd Machine Platform is powered by Oracle Cloud and now available in the Oracle Cloud Marketplace ...