16 Jul 2015 Brussels - IDC has delivered a report to the European Commission about the HPC developments in Europe. It contains an assessment of PC Action Plan Objectives set out a few years ago. The report also provides some recommendations. The report is now at the European Commission, who will study it, draw conclusions from it and use it in further development of the HPC actions. In general, IDC concludes Europe has made good progress in implementing HPC and exascale plans. Europe is now on par with other major HPC players in the world. In the recommendations IDC advices Europe to expand funding for HPC, concentrate on the whole software stack, and also look at the newly developing High Performance Data Analysis (HPDA) area.
According to the IDC report, Europe has made impressive progress in areas that are crucial for the goals of the HPC Action Plan (High Performance Computing (HPC): Europe's Place in a Global Race (February 2012)), especially organizing the European HPC community to pursue HPC leadership on a unified basis, expanding the scientific and industrial access to and use of supercomputers, and launching initiatives to strengthen the European HPC supply chain.
The IDC assessment of progress in implementing the European HPC strategy, and the related recommendations that IDC made in earlier versions of the report 3 years ago, are based on the four Action Plan Objectives and the six related Strategic Actions. The ultimate aim is to make Europe stronger in using HPC to advance science and industry.
According to IDC, Europe has significantly narrowed the wide gap separating the most capable U.S., Chinese and Japanese supercomputers from their European counterparts. In November 2010, 9 of the world's 50 most powerful supercomputers were located in Europe. Four years later, Europe hosted 19 of the top 50, including the PRACE tier-0 supercomputers.
According to an IDC study on European projects, European HPC investments are producing excellent returns-on-investment (ROI) for science and industry. IDC captured detailed ROI information on 143 European HPC projects. For projects that generated financial returns, IDC sees each euro invested in HPC on average returned 867 euro in increased revenue/income and 69 euro in profits.
IDC concludes Europe achieved healthy HPC funding growth in 2010, 2011 and 2012, but Europe-wide funding declined heavily in 2013 and 2014. On average, however, the net funding increase over the last five-year period was extremely good for pursuing HPC leadership. Significant investments will be needed, however, for pre-exascale systems in the period 2019-2020 and exascale systems in 2022, says IDC. IDC estimates that the overall European-wide HPC R&D investments, supercomputer purchases, and related services in 2010 amounted to approximately 1,8 billion euro: 715 million euro in purchases of supercomputers, plus 90 million euro (in supporting investments, integration services and professional services); plus about 850 million euro (storage, software and repair services purchased with the computers); plus about 30 million euro (from the EC); and about 90 million euro (R&D by vendors and users). IDC says these are IDC estimates, as there is no European tracking of these numbers at this time.
According to IDC, overall, the main actors - the European Commission, PRACE and ETP4HPC, have done an admirable job of coordinating with each other to advance the European HPC strategy. But more needs to be done says IDC. To amass the 1 billion-plus euro in funding needed to acquire pre-exascale and exascale supercomputers in globally competitive time frames, the European Member States will need to find a way to pool resources and the EC will need to find a way to contribute a significant portion of the funding. IDC recommends Europe should contribute 50%.
IDC stresses that adequate investments in software will be one of the most important determinants of future HPC leadership. In addition, the Commission may need to contribute to the escalating operating expenses associated with PRACE tier-0 supercomputers.
The PRACE peer-review system has enabled fair access to leading supercomputers (infrastructure) by scientists, and more recently industrial engineers, from throughout Europe. Greater outreach is needed to industry, especially SMEs.
IDC says the PRACE 2.0 financing scheme - which has not yet been defined, will likely aim to spread the burden more equitably among PRACE members but may risk curtailing access for scientists from Member States that are unable or unwilling to make their assigned contributions.
IDC also believes HPC procurements in Europe should make greater use of PPI mechanisms, and ETP4HPC and Member States within PRACE should coordinate closely to ensure that the ETP4HPC roadmap reflects the innovations PRACE members and others would like to procure. The aims here are to accelerate HPC innovations by European suppliers (as ETP4HPC is already doing), and also to increase the likelihood that these innovations will be incorporated into supercomputer procurements that enable the European technologies to mature in real-world customer environments.
At the very high end of the supercomputer segment, Europe has significantly narrowed the former wide gap separating the most capable U.S. and Japanese supercomputers from their European counterparts. Spending increased substantially in the EU/EU+ for large supercomputers from 2009 to 2012, says IDC, but then declined. European overall spending on large supercomputers grew from 112 million euro in 2009, to 658 million in 2012, then down to 362 million euro in 2014. In November 2010, shortly after the founding of PRACE, 9 of the world's 50 most powerful supercomputers were located in Europe. Four years later, in November 2014, Europe hosted 19 of the top 50, including the PRACE tier-0 supercomputers. The aggregate peak performance of the Europe-based supercomputers rose more than ten-fold during this period, from 4,3 petaflops in November 2010 to 48,9 petaflops four years later. Clearly, according to IDC, Europe's standing as a provider of high-end supercomputing resources advanced in both absolute and relative (worldwide) terms during this period.
IDC sees that from a technical standpoint, Europe's HPC community, building on existing and planned EU- wide HPC development initiatives, is well positioned to exploit a strong base of indigenous and foreign technologies across its commercial, academic and government sectors to assemble exascale HPC capability that could, in some critical application sectors, achieve world-class stature or even global leadership. A clearer path is needed, however, for driving innovation into supercomputer procurements and pooling enough money to deploy pre-exascale and exascale supercomputers in globally competitive time frames.
The overall European HPC strategy is built on a foundation of key objectives that are the source and inspiration for specific strategy actions. IDC has examined progress against these key objectives in detail, and after lengthy discussions with many EU and other HPC technology and policy experts, offers the following assessment.
Assessment of Specific Action Plan Objectives:
1. Provide a world-class European HPC infrastructure, benefitting a broad range of academic and industry users, and especially SMEs, including a workforce well trained in HPC.
IDC assessment: Strong progress was made in deploying large HPC systems across Europe, primarily funded by the Member States, with important secondary funding support by the European Commission through PRACE. Although the overall level of funding was considerably lower than recommended in IDC's 2010 report (as shown in Section 5, Table 5.G), Europe nevertheless made outstanding progress in getting Europe "back in the pack" of top global research areas over the last four years.
2. Ensure independent access to HPC technologies, systems and services
IDC assessment: Hardware technologies from European suppliers today have a very small share and presence in HPC across Europe. Some European software companies are highly successful in Europe and across the world. ETP4HPC collaborations between European and non-European suppliers will benefit European scientists and engineers. A large majority of the European HPC stakeholders IDC interviewed agreed that European scientists and engineers require access to best-in-class HPC technologies and systems, no matter where in the world they come from.
3. Establish a pan-European HPC governance scheme to pool enlarged resources and increase efficiency, including through the strategic use of joint and pre-commercial procurement
IDC assessment: Pre-commercial procurement (PCP) and public procurement of innovative solutions
(PPI) are underutilized in Europe today. PCP and PPI are key mechanisms used by the governments of the United States, China and Japan (Europe's main rivals for HPC leadership) to drive commercial competitive advantage and to advance suppliers of indigenous technologies. European HPC stakeholders greatly prefer PPI (and related mechanisms) over PCP as a way to advance innovation. There is no central procurement agent in Europe with the financial ability and motivation to exploit PPI or related mechanisms on behalf of Europe, as needed to compete successfully with the U.S., China and Japan.
4. Ensure the EU's position as a global actor
IDC assessment: Advances in highly parallel software will be one of the most important determinants of future global HPC leadership. Europe has world-class strengths in highly parallel software. The de-emphasizing of funding for exascale software development by the U.S. government has created an opportunity for Europe to gain an important advantage. EESI's exceptional work has set the stage for this. The cPPP on HPC, established with the ETP4HPC with 700 million euro in EC funding, provides a framework for making it happen, especially through the centers of excellence and continued EESI work.
Assessment of the progress and success of the six Strategy Actions
Based on the overarching objectives defined in the European HPC plan, specific policy-related strategy actions were created. For those six strategy actions, IDC offers the following further assessment and recommendations.
1) Governance at EU level: seeks adequacy, openness, and efficiency of the current organisations (e.g., PRACE and ETP4HPC) to structure the industrial and scientific stakeholders, to steer the high level objectives and policies on HPC, to pool available HPC resources across the Member States, and to efficiently implement the HPC strategy.
IDC assessment: The key organisations contributing to the implementation of the European HPC strategy - the European Commission, PRACE and EPT4HPC - have done an admirable job of advancing Europe's position in the few short years since the 2012 Communication. No single European Member State has the financial and related means to compete effectively with the U.S., China and Japan for HPC leadership. If Europe is to be an HPC leader, it will therefore be necessary for Member States to coordinate their HPC strategies more closely, including the pooling of funding, and for the EC and the Member States to coordinate even more closely. This tighter coordination will require some adjustments to existing governance rules and practices by all parties. Tensions already present within Europe's loosely coupled, collaborative HPC governance model will likely grow as the exascale computing era approaches, unless these issues are addressed. Governments of some Member States hosting PRACE tier-0 supercomputers are reluctant to continue funding 100% of the substantial, rising operating expenses associated with these computers. They would like some relief from the Commission and/or from other PRACE members. Access to more of the tier-0 supercomputers' cycles may need to be secured for European, as opposed to national, use.
In carrying out this study, IDC said it found that even some of the most prominent members of Europe's HPC community did not adequately understand the impact of governance models on the European HPC strategy - or how all the pieces of the strategy fit together. Stronger communications outreach is needed to convey this understanding.
2) Financial envelope for HPC spans current investment levels for the acquisition of high-end HPC resources in Europe, and analysis of required levels to meet the Action Plan objectives (including investments for system acquisition, training, HPC software and applications, etc.).
IDC assessment: The European Commission's HPC investment levels have grown substantially, through a range of initiatives including the 700 million euro, multi-year investment to support the future-oriented cPPP being carried out with the ETP4HPC. But Commission contributions have fallen short of the amounts recommended in IDC's 2010 report, and spending within PRACE has not yet reached the level of commitments. The Commission has contributed to support PRACE, and plans to start helping to support some procurements for large supercomputers in 2016.
3) The implementation of funding mechanisms, such as pre-commercial procurement in the public sector (the major buyer of high-end HPC) and pooling of research resources, to support HPC suppliers for developing a leadership-class HPC system about every 2 years.
IDC assessment: According to IDC, PPI and related mechanisms are crucial for attaining and maintaining HPC leadership. Europe's rivals for HPC leadership - the U.S., China and Japan - regularly employ these mechanisms. These mechanisms are used today only occasionally in Europe. No clear strategy exists for pooling resources to finance the acquisition of pre-exascale supercomputers in 2020 and exascale supercomputers in 2022, in order to Europe to remain competitive with the U.S., China and Japan.
4) Development of European state-of-the-art supply capabilities needed for European independent access to key HPC technologies, systems, services and tools for Europe (including level of pre-commercial procurement and other R&D investments, support to European HPC suppliers, jobs in European HPC supply industry, etc.).
ETP4HPC is dedicated to expanding and strengthening Europe's HPC supply chain. The European Commission is contributing 700 million euro to advance the cPPP on HPC in partnership with ETP4HPC. A large majority of the European HPC stakeholders IDC interviewed agreed that European scientists and engineers require access to best-in-class HPC technologies and systems, no matter where in the world they come from. It is especially important that European suppliers not only participate in exascale technology innovation, but also gain experience and feedback in real-world HPC customer environments. For this reason, ETP4HPC roadmaps for indigenous technology development need to be linked as closely as possible to the requirements that will drive supercomputer procurements.
5) Industrial exploitation of HPC including regional/national centers for the access of industry (including SMEs) to HPC (HPC Competence Centers), industrial HPC-based development and innovation, education and training in HPC, HPC trained workforce in Europe, and more.
Europe already has some of the world's leading HPC centers for collaborations with industry, including SMEs. Many of the European HPC stakeholders IDC interviewed for this study agreed that European programmes supporting industrial access and collaboration, such as PRACE, SHAPE and Fortissimo, have been successful but need to do more. Only a small percentage of European SMEs that could be helped by HPC seem aware of these opportunities, for example. The EC hasn't done an optimal job of communicating the HPC strategy, resources, plans and contact persons to the broader HPC community, and in particular to industry.
6) Ensuring a level playing field, in particular regarding inequalities in HPC market access and exploitation obligations regarding intellectual property rights of HPC results generated in Horizon 2020.
IDC assessment: Europe has long been the world's most open HPC market. Government HPC markets in the U.S., Japan and China all present barriers to non-domestic HPC suppliers, although the private-sector markets in these countries are more open and both government and private-sector markets are generally open to non-domestic commercial software. A large majority of European HPC stakeholders IDC interviewed for this study agreed that European scientists and engineers should continue to have access to the world's best supercomputer systems, no matter where in the world they come from. Specific market asymmetries should be addressed at a government (EC)-to-government level and not made part of Europe's HPC strategy.
Based on an examination of the European HPC objectives and related strategy actions, IDC offers the following recommendations, each keyed to one or more of the strategy actions.
Based on the progress to-date of the EU's Strategy Actions, IDC makes the following targeted recommendations:
1) Expand Funding for HPC (Financial Envelope & Funding Mechanism, and Needed for Making Europe World Class in HPC). PRACE members and the EC should agree to provide significant funding support to acquire two pre-exascale supercomputers in 2019-2020 and two additional exascale supercomputers in 2022. One path should stress European pre-commercial technology. The total recommended HPC cumulative funding increase (for all parties) from 2016 to 2020 is just over 1 billion euro. The European Commission should extend the end date of the Action Plan from 2020 to 2022, to match the expected exascale time frames of the U.S., China and Japan and make it easier to amass the funding levels recommended in this study. The Member States and the EC may need to adapt their practices in order to pool the money needed to fund exascale systems.
2) Improve Communication of the Strategy (Supply Ecosystem and Needed for Making Europe World Class in HPC). The European Commission should create a single website portal enabling access to comprehensive information on the European HPC strategy. There needs to include a single person in charge and as a direct contact for information and for answering questions. This is key for getting more users, ISVs and SMEs involved.
3) Develop the HPC Ecosystem (European Supply Capabilities and Governance). ETP4HPC should continue to support collaborations involving European suppliers and (often much larger) non-European suppliers. This will accelerate the learning curve of some European suppliers who are less experienced. European suppliers will benefit from competing on equal terms with suppliers not based in Europe. Software will be one of the main determinants of future HPC leadership and Europe is very strong in parallel software development. ETP4HPC or another organisation in Europe should create a clearinghouse (online storefront) to help disseminate, and in some cases commercialize, innovative European HPC software that is now in limited use.
4) Improve Support for Industry (Industrial Use of HPC). PRACE should consider promoting SME and industry adoption of HPC with a SHAPE initiative that lets SMEs and other industrial firms that are new to HPC try it out without cost, and without needing to show scientific merit. ETP4HPC should ensure that software advances meant to benefit industry have strong, continuous input from industry representatives. Europe already has some of the world's leading HPC centers for collaborations with industry, including SMEs. Centers with strong industrial experience are well positioned to mentor centers with less experience with industry.
5) Improve Skills and Talent (To Ensure Europe as a Global Actor and Needed for Making Europe World Class in HPC). The European Commission should undertake a communications campaign to update the image of HPC as a career choice. The Commission should also establish a task force to develop practical strategies for integrating HPC and related computational science education and training more fully into the scientific and engineering curricula of European universities.
High Performance Computing in the EU: Progress on the Implementation of the European HPC Strategy, Final Report, A study prepared for the European Commission DG Communications Networks, Content & Technology by IDC
ISBN number 978-92-79-49475-8