12 Nov 2018 Barcelona - The Workflows and Distributed Computing team at the Barcelona Supercomputing Center (BSC) has issued a new release, version 2.4 - codename Elderflower, of the programming environment COMPSs. The latest COMPSs release includes a new Autoparallel module that enables automatic parallelization of loops, an extension of the Jupyter notebooks support to enable execution in supercomputers, and a new Spark-like syntax based on the Distributed Data Set (DDS) data-structure.
This version of COMPSs, available from today, updates the result of the team's work in the last years on the provision of a set of tools that helps developers to program and execute their applications efficiently on distributed computational infrastructures such as clusters, clouds and container managed platforms. COMPSs is a task based programming model known for notably improving the performance of large scale applications by automatically parallelizing their execution.
COMPSs has been available for the last years for the MareNostrum supercomputer users and for the Spanish Supercomputing Network users, and has been adopted in several research projects such as OPTIMIS, VENUS-C, EUBrazil OpenBio, EUBrazil CloudConnect, EUBra-BIGSEA, transplant, EGI and ASCETIC. In these projects, COMPSs has been applied to implement use cases provided by different communities across diverse disciplines as biomedicine, engineering, biodiversity, chemistry, astrophysics and earth sciences. Currently it is also under extension and adoption in applications in the projects TANGO, NEXTGenIO, MUG, mf2C, CLASS, ExaQUte, LANDSUPPORT, the BioExcel CoE, and the EXPERTISE ETN, as well as in a research contract with FUJITSU.
The new release comes with a new Autoparallel module that enables the automatic taskification of affine loop nests. This feature enables to parallelize the loops with a single Python decorator and frees the user from manually taskifying the original code. Moreover, to avoid fine-grain tasks, AutoParallel also includes an optional feature to increase the tasks' granularity by automatically building data blocks. This contribution was presented at the PyHPC 2018 workshop @SC18, on November 12th.
The support for the Jupyter notebook has been extended to enable the applications' execution in supercomputers. This extension is based on the execution of the Jupyter server in a supercomputer allocation. Applications can be started from the Jupyter client, and will be offloaded to the supercomputer allocation, instantiating the COMPSs runtime and executing the application in parallel.
Version 2.4 comes also with a new interface for Python on top of the Distributed Data Set (DDS) class. The DDS emulates the Spark RDDs, and provides a new syntax to PyCOMPSs which enables the execution of Spark-like codes on top of the COMPSs runtime.
Other relevant features are: support for new architectures, such as Power9 and ARM ThunderX, and the support in C to multi-architectures and cross-compiling build support.
COMPSs 2.4 comes with other minor new features, extensions and bug fixes.
At SC18, PyCOMPSs/COMPSs will have a presence at the BSC booth, with presentations and demos. A live demo on the use of PyCOMPSs with the Jupyter notebook will be delivered at the BSC booth #2038 on November 13th at 4.40 pm, on November 14th at 2.00 pm and on November 15th at 11.20 am.
COMPSs has had around 1000 downloads last year and is used by around 20 groups in real applications. COMPSs has recently attracted interest from areas such as image recognition, genomics and biodiversity, where specific courses and dissemination actions have been performed.
The packages and the complete list of features are available in the Downloads page. A virtual appliance is also available to test the functionalities of COMPSs through a step-by-step tutorial that guides the user to develop and execute a set of example applications.
Additionally, a user guide and papers published in relevant conferences and journals are available.
For more information on COMPSs, you can visit the COMPSs web page.