Back to Table of contents

Primeur weekly 2015-08-03

Special

So much more than meets the eye with Moab, Nitro, Elastic Compute, Docker at Adaptive Computing ...

IDC reports strong growth of HPC in financial markets and assesses European HPC deployment towards exascale in new report for EC ...

DDN presents three game-changing products: the ES7K storage solution, the Wolfcreek Platform and the Infinite Memory Engine ...

Focus

IDC assessment of European HPC strategy: remarkable progress, but more needs to be done to reach exascale ...

Focus on Europe

BSC and CEA will coordinate efforts to reinforce European HPC research and technology innovation ...

Middleware

General availability of Intel Enterprise Edition for Lustre software v2.3 ...

Hardware

Investments in cyberinfrastructure to enhance Canada's competitive edge in research and innovation ...

DDN financial service storage solutions set world records in STAC performance benchmarks ...

Cray reports second quarter financial results ...

Inova Translational Medicine Institute chooses SGI to power one of the largest genomic databases in the world ...

SGI Japan installs five-screen virtual reality system at Komatsu, Oyama ...

Intel and Micron produce breakthrough memory technology ...

Applications

China's Skyeye-1 supercomputer to support world's largest radio telescope ...

CVS Health and IBM tap Watson to develop care management solutions for chronic disease ...

Illinois receives $1.8 million to create data platform for Big Data in plant breeding ...

Simulations lead to design of near-frictionless material ...

Superfast fluorescence sets new speed record ...

SGI UV 300H 20-socket appliance certified by SAP to run SAP HANA under controlled availability ...

TOP500

NASA's Pleiades supercomputer ranks among world's most powerful systems ...

The Cloud

HP to acquire ActiveState's Stackato business to help customers transition to hybrid Cloud ...

IDC assessment of European HPC strategy: remarkable progress, but more needs to be done to reach exascale

16 Jul 2015 Brussels - IDC has delivered a report to the European Commission about the HPC developments in Europe. It contains an assessment of PC Action Plan Objectives set out a few years ago. The report also provides some recommendations. The report is now at the European Commission, who will study it, draw conclusions from it and use it in further development of the HPC actions. In general, IDC concludes Europe has made good progress in implementing HPC and exascale plans. Europe is now on par with other major HPC players in the world. In the recommendations IDC advices Europe to expand funding for HPC, concentrate on the whole software stack, and also look at the newly developing High Performance Data Analysis (HPDA) area.

According to the IDC report, Europe has made impressive progress in areas that are crucial for the goals of the HPC Action Plan (High Performance Computing (HPC): Europe's Place in a Global Race (February 2012)), especially organizing the European HPC community to pursue HPC leadership on a unified basis, expanding the scientific and industrial access to and use of supercomputers, and launching initiatives to strengthen the European HPC supply chain.

The IDC assessment of progress in implementing the European HPC strategy, and the related recommendations that IDC made in earlier versions of the report 3 years ago, are based on the four Action Plan Objectives and the six related Strategic Actions. The ultimate aim is to make Europe stronger in using HPC to advance science and industry.

According to IDC, Europe has significantly narrowed the wide gap separating the most capable U.S., Chinese and Japanese supercomputers from their European counterparts. In November 2010, 9 of the world's 50 most powerful supercomputers were located in Europe. Four years later, Europe hosted 19 of the top 50, including the PRACE tier-0 supercomputers.

According to an IDC study on European projects, European HPC investments are producing excellent returns-on-investment (ROI) for science and industry. IDC captured detailed ROI information on 143 European HPC projects. For projects that generated financial returns, IDC sees each euro invested in HPC on average returned 867 euro in increased revenue/income and 69 euro in profits.

IDC concludes Europe achieved healthy HPC funding growth in 2010, 2011 and 2012, but Europe-wide funding declined heavily in 2013 and 2014. On average, however, the net funding increase over the last five-year period was extremely good for pursuing HPC leadership. Significant investments will be needed, however, for pre-exascale systems in the period 2019-2020 and exascale systems in 2022, says IDC. IDC estimates that the overall European-wide HPC R&D investments, supercomputer purchases, and related services in 2010 amounted to approximately 1,8 billion euro: 715 million euro in purchases of supercomputers, plus 90 million euro (in supporting investments, integration services and professional services); plus about 850 million euro (storage, software and repair services purchased with the computers); plus about 30 million euro (from the EC); and about 90 million euro (R&D by vendors and users). IDC says these are IDC estimates, as there is no European tracking of these numbers at this time.

According to IDC, overall, the main actors - the European Commission, PRACE and ETP4HPC, have done an admirable job of coordinating with each other to advance the European HPC strategy. But more needs to be done says IDC. To amass the 1 billion-plus euro in funding needed to acquire pre-exascale and exascale supercomputers in globally competitive time frames, the European Member States will need to find a way to pool resources and the EC will need to find a way to contribute a significant portion of the funding. IDC recommends Europe should contribute 50%.

IDC stresses that adequate investments in software will be one of the most important determinants of future HPC leadership. In addition, the Commission may need to contribute to the escalating operating expenses associated with PRACE tier-0 supercomputers.

The PRACE peer-review system has enabled fair access to leading supercomputers (infrastructure) by scientists, and more recently industrial engineers, from throughout Europe. Greater outreach is needed to industry, especially SMEs.

IDC says the PRACE 2.0 financing scheme - which has not yet been defined, will likely aim to spread the burden more equitably among PRACE members but may risk curtailing access for scientists from Member States that are unable or unwilling to make their assigned contributions.

IDC also believes HPC procurements in Europe should make greater use of PPI mechanisms, and ETP4HPC and Member States within PRACE should coordinate closely to ensure that the ETP4HPC roadmap reflects the innovations PRACE members and others would like to procure. The aims here are to accelerate HPC innovations by European suppliers (as ETP4HPC is already doing), and also to increase the likelihood that these innovations will be incorporated into supercomputer procurements that enable the European technologies to mature in real-world customer environments.

At the very high end of the supercomputer segment, Europe has significantly narrowed the former wide gap separating the most capable U.S. and Japanese supercomputers from their European counterparts. Spending increased substantially in the EU/EU+ for large supercomputers from 2009 to 2012, says IDC, but then declined. European overall spending on large supercomputers grew from 112 million euro in 2009, to 658 million in 2012, then down to 362 million euro in 2014. In November 2010, shortly after the founding of PRACE, 9 of the world's 50 most powerful supercomputers were located in Europe. Four years later, in November 2014, Europe hosted 19 of the top 50, including the PRACE tier-0 supercomputers. The aggregate peak performance of the Europe-based supercomputers rose more than ten-fold during this period, from 4,3 petaflops in November 2010 to 48,9 petaflops four years later. Clearly, according to IDC, Europe's standing as a provider of high-end supercomputing resources advanced in both absolute and relative (worldwide) terms during this period.

IDC sees that from a technical standpoint, Europe's HPC community, building on existing and planned EU- wide HPC development initiatives, is well positioned to exploit a strong base of indigenous and foreign technologies across its commercial, academic and government sectors to assemble exascale HPC capability that could, in some critical application sectors, achieve world-class stature or even global leadership. A clearer path is needed, however, for driving innovation into supercomputer procurements and pooling enough money to deploy pre-exascale and exascale supercomputers in globally competitive time frames.

Progress on the Four EU HPC Action Plan Objectives

The overall European HPC strategy is built on a foundation of key objectives that are the source and inspiration for specific strategy actions. IDC has examined progress against these key objectives in detail, and after lengthy discussions with many EU and other HPC technology and policy experts, offers the following assessment.

Assessment of Specific Action Plan Objectives:

1. Provide a world-class European HPC infrastructure, benefitting a broad range of academic and industry users, and especially SMEs, including a workforce well trained in HPC.

IDC assessment: Strong progress was made in deploying large HPC systems across Europe, primarily funded by the Member States, with important secondary funding support by the European Commission through PRACE. Although the overall level of funding was considerably lower than recommended in IDC's 2010 report (as shown in Section 5, Table 5.G), Europe nevertheless made outstanding progress in getting Europe "back in the pack" of top global research areas over the last four years.

2. Ensure independent access to HPC technologies, systems and services

IDC assessment: Hardware technologies from European suppliers today have a very small share and presence in HPC across Europe. Some European software companies are highly successful in Europe and across the world. ETP4HPC collaborations between European and non-European suppliers will benefit European scientists and engineers. A large majority of the European HPC stakeholders IDC interviewed agreed that European scientists and engineers require access to best-in-class HPC technologies and systems, no matter where in the world they come from.

3. Establish a pan-European HPC governance scheme to pool enlarged resources and increase efficiency, including through the strategic use of joint and pre-commercial procurement

IDC assessment: Pre-commercial procurement (PCP) and public procurement of innovative solutions

(PPI) are underutilized in Europe today. PCP and PPI are key mechanisms used by the governments of the United States, China and Japan (Europe's main rivals for HPC leadership) to drive commercial competitive advantage and to advance suppliers of indigenous technologies. European HPC stakeholders greatly prefer PPI (and related mechanisms) over PCP as a way to advance innovation. There is no central procurement agent in Europe with the financial ability and motivation to exploit PPI or related mechanisms on behalf of Europe, as needed to compete successfully with the U.S., China and Japan.

4. Ensure the EU's position as a global actor

IDC assessment: Advances in highly parallel software will be one of the most important determinants of future global HPC leadership. Europe has world-class strengths in highly parallel software. The de-emphasizing of funding for exascale software development by the U.S. government has created an opportunity for Europe to gain an important advantage. EESI's exceptional work has set the stage for this. The cPPP on HPC, established with the ETP4HPC with 700 million euro in EC funding, provides a framework for making it happen, especially through the centers of excellence and continued EESI work.

Assessment of the progress and success of the six Strategy Actions

Based on the overarching objectives defined in the European HPC plan, specific policy-related strategy actions were created. For those six strategy actions, IDC offers the following further assessment and recommendations.

1) Governance at EU level: seeks adequacy, openness, and efficiency of the current organisations (e.g., PRACE and ETP4HPC) to structure the industrial and scientific stakeholders, to steer the high level objectives and policies on HPC, to pool available HPC resources across the Member States, and to efficiently implement the HPC strategy.

IDC assessment: The key organisations contributing to the implementation of the European HPC strategy - the European Commission, PRACE and EPT4HPC - have done an admirable job of advancing Europe's position in the few short years since the 2012 Communication. No single European Member State has the financial and related means to compete effectively with the U.S., China and Japan for HPC leadership. If Europe is to be an HPC leader, it will therefore be necessary for Member States to coordinate their HPC strategies more closely, including the pooling of funding, and for the EC and the Member States to coordinate even more closely. This tighter coordination will require some adjustments to existing governance rules and practices by all parties. Tensions already present within Europe's loosely coupled, collaborative HPC governance model will likely grow as the exascale computing era approaches, unless these issues are addressed. Governments of some Member States hosting PRACE tier-0 supercomputers are reluctant to continue funding 100% of the substantial, rising operating expenses associated with these computers. They would like some relief from the Commission and/or from other PRACE members. Access to more of the tier-0 supercomputers' cycles may need to be secured for European, as opposed to national, use.

In carrying out this study, IDC said it found that even some of the most prominent members of Europe's HPC community did not adequately understand the impact of governance models on the European HPC strategy - or how all the pieces of the strategy fit together. Stronger communications outreach is needed to convey this understanding.

2) Financial envelope for HPC spans current investment levels for the acquisition of high-end HPC resources in Europe, and analysis of required levels to meet the Action Plan objectives (including investments for system acquisition, training, HPC software and applications, etc.).

IDC assessment: The European Commission's HPC investment levels have grown substantially, through a range of initiatives including the 700 million euro, multi-year investment to support the future-oriented cPPP being carried out with the ETP4HPC. But Commission contributions have fallen short of the amounts recommended in IDC's 2010 report, and spending within PRACE has not yet reached the level of commitments. The Commission has contributed to support PRACE, and plans to start helping to support some procurements for large supercomputers in 2016.

3) The implementation of funding mechanisms, such as pre-commercial procurement in the public sector (the major buyer of high-end HPC) and pooling of research resources, to support HPC suppliers for developing a leadership-class HPC system about every 2 years.

IDC assessment: According to IDC, PPI and related mechanisms are crucial for attaining and maintaining HPC leadership. Europe's rivals for HPC leadership - the U.S., China and Japan - regularly employ these mechanisms. These mechanisms are used today only occasionally in Europe. No clear strategy exists for pooling resources to finance the acquisition of pre-exascale supercomputers in 2020 and exascale supercomputers in 2022, in order to Europe to remain competitive with the U.S., China and Japan.

4) Development of European state-of-the-art supply capabilities needed for European independent access to key HPC technologies, systems, services and tools for Europe (including level of pre-commercial procurement and other R&D investments, support to European HPC suppliers, jobs in European HPC supply industry, etc.).

ETP4HPC is dedicated to expanding and strengthening Europe's HPC supply chain. The European Commission is contributing 700 million euro to advance the cPPP on HPC in partnership with ETP4HPC. A large majority of the European HPC stakeholders IDC interviewed agreed that European scientists and engineers require access to best-in-class HPC technologies and systems, no matter where in the world they come from. It is especially important that European suppliers not only participate in exascale technology innovation, but also gain experience and feedback in real-world HPC customer environments. For this reason, ETP4HPC roadmaps for indigenous technology development need to be linked as closely as possible to the requirements that will drive supercomputer procurements.

5) Industrial exploitation of HPC including regional/national centers for the access of industry (including SMEs) to HPC (HPC Competence Centers), industrial HPC-based development and innovation, education and training in HPC, HPC trained workforce in Europe, and more.

Europe already has some of the world's leading HPC centers for collaborations with industry, including SMEs. Many of the European HPC stakeholders IDC interviewed for this study agreed that European programmes supporting industrial access and collaboration, such as PRACE, SHAPE and Fortissimo, have been successful but need to do more. Only a small percentage of European SMEs that could be helped by HPC seem aware of these opportunities, for example. The EC hasn't done an optimal job of communicating the HPC strategy, resources, plans and contact persons to the broader HPC community, and in particular to industry.

6) Ensuring a level playing field, in particular regarding inequalities in HPC market access and exploitation obligations regarding intellectual property rights of HPC results generated in Horizon 2020.

IDC assessment: Europe has long been the world's most open HPC market. Government HPC markets in the U.S., Japan and China all present barriers to non-domestic HPC suppliers, although the private-sector markets in these countries are more open and both government and private-sector markets are generally open to non-domestic commercial software. A large majority of European HPC stakeholders IDC interviewed for this study agreed that European scientists and engineers should continue to have access to the world's best supercomputer systems, no matter where in the world they come from. Specific market asymmetries should be addressed at a government (EC)-to-government level and not made part of Europe's HPC strategy.

Overall Recommendations

Based on an examination of the European HPC objectives and related strategy actions, IDC offers the following recommendations, each keyed to one or more of the strategy actions.

Recommendations on the Strategic Actions

Based on the progress to-date of the EU's Strategy Actions, IDC makes the following targeted recommendations:

1) Expand Funding for HPC (Financial Envelope & Funding Mechanism, and Needed for Making Europe World Class in HPC). PRACE members and the EC should agree to provide significant funding support to acquire two pre-exascale supercomputers in 2019-2020 and two additional exascale supercomputers in 2022. One path should stress European pre-commercial technology. The total recommended HPC cumulative funding increase (for all parties) from 2016 to 2020 is just over 1 billion euro. The European Commission should extend the end date of the Action Plan from 2020 to 2022, to match the expected exascale time frames of the U.S., China and Japan and make it easier to amass the funding levels recommended in this study. The Member States and the EC may need to adapt their practices in order to pool the money needed to fund exascale systems.

2) Improve Communication of the Strategy (Supply Ecosystem and Needed for Making Europe World Class in HPC). The European Commission should create a single website portal enabling access to comprehensive information on the European HPC strategy. There needs to include a single person in charge and as a direct contact for information and for answering questions. This is key for getting more users, ISVs and SMEs involved.

3) Develop the HPC Ecosystem (European Supply Capabilities and Governance). ETP4HPC should continue to support collaborations involving European suppliers and (often much larger) non-European suppliers. This will accelerate the learning curve of some European suppliers who are less experienced. European suppliers will benefit from competing on equal terms with suppliers not based in Europe. Software will be one of the main determinants of future HPC leadership and Europe is very strong in parallel software development. ETP4HPC or another organisation in Europe should create a clearinghouse (online storefront) to help disseminate, and in some cases commercialize, innovative European HPC software that is now in limited use.

4) Improve Support for Industry (Industrial Use of HPC). PRACE should consider promoting SME and industry adoption of HPC with a SHAPE initiative that lets SMEs and other industrial firms that are new to HPC try it out without cost, and without needing to show scientific merit. ETP4HPC should ensure that software advances meant to benefit industry have strong, continuous input from industry representatives. Europe already has some of the world's leading HPC centers for collaborations with industry, including SMEs. Centers with strong industrial experience are well positioned to mentor centers with less experience with industry.

5) Improve Skills and Talent (To Ensure Europe as a Global Actor and Needed for Making Europe World Class in HPC). The European Commission should undertake a communications campaign to update the image of HPC as a career choice. The Commission should also establish a task force to develop practical strategies for integrating HPC and related computational science education and training more fully into the scientific and engineering curricula of European universities.

Additional Recommendations from IDC

  • High Performance Data Analysis (HPDA). This is the market for Big Data workloads that are complex or time-critical enough to require HPC resources. No dramatic shift in the European HPC strategy is needed for HPDA, because most HPDA, today and in the near term, will be data-intensive modelling and simulation (M&S) that HPC sites have been running for decades, but there are many opportunities to expand in this key area.
  • Cloud Computing. In IDC's worldwide studies of high performance computing (HPC) end-user sites, the proportion of sites employing Cloud computing - public or private - has steadily grown from 13.8% in 2011, to 23.5% in 2013, to 34.1% in 2015. As with HPDA, IDC believes that no major change to Europe's HPC strategy is needed to accommodate the growing importance and technical evolution of public and private Cloud computing. The challenge is less about educating users about Cloud computing and more about the ability of Clouds to handle more types of HPC jobs over time.
  • Co-Design. The EU should initiate an EU-wide exascale test bed programme to provide strong support for exascale co-design capabilities. The test bed programme would be a natural extension of ETP4HPC's role. It would require ETP4HPC to collaborate closely with PRACE tier-0 supercomputer hosting members to test hardware and software technologies at large scale.
  • Centers-of-Excellence. These fall under the cPPP on HPC. The centers should focus heavily on software - not just applications but also the whole HPC software stack (EESI has done an excellent job of laying the groundwork for this). At a minimum, one center should focus on the software requirements of industry, and one center be devoted to HPDA.

Reference:

High Performance Computing in the EU: Progress on the Implementation of the European HPC Strategy, Final Report, A study prepared for the European Commission DG Communications Networks, Content & Technology by IDC

ISBN number 978-92-79-49475-8

Ad Emmen

Back to Table of contents

Primeur weekly 2015-08-03

Special

So much more than meets the eye with Moab, Nitro, Elastic Compute, Docker at Adaptive Computing ...

IDC reports strong growth of HPC in financial markets and assesses European HPC deployment towards exascale in new report for EC ...

DDN presents three game-changing products: the ES7K storage solution, the Wolfcreek Platform and the Infinite Memory Engine ...

Focus

IDC assessment of European HPC strategy: remarkable progress, but more needs to be done to reach exascale ...

Focus on Europe

BSC and CEA will coordinate efforts to reinforce European HPC research and technology innovation ...

Middleware

General availability of Intel Enterprise Edition for Lustre software v2.3 ...

Hardware

Investments in cyberinfrastructure to enhance Canada's competitive edge in research and innovation ...

DDN financial service storage solutions set world records in STAC performance benchmarks ...

Cray reports second quarter financial results ...

Inova Translational Medicine Institute chooses SGI to power one of the largest genomic databases in the world ...

SGI Japan installs five-screen virtual reality system at Komatsu, Oyama ...

Intel and Micron produce breakthrough memory technology ...

Applications

China's Skyeye-1 supercomputer to support world's largest radio telescope ...

CVS Health and IBM tap Watson to develop care management solutions for chronic disease ...

Illinois receives $1.8 million to create data platform for Big Data in plant breeding ...

Simulations lead to design of near-frictionless material ...

Superfast fluorescence sets new speed record ...

SGI UV 300H 20-socket appliance certified by SAP to run SAP HANA under controlled availability ...

TOP500

NASA's Pleiades supercomputer ranks among world's most powerful systems ...

The Cloud

HP to acquire ActiveState's Stackato business to help customers transition to hybrid Cloud ...