The interim evaluation considers several aspects of the flagships to assess their usefulness: relevance, effectiveness, coherency, and EU added value.
According to the report, both existing flagships are delivering world-leading results, and are reporting achievements beyond those defined by their Key Performance Indicators (KPIs), for example, in terms of the number of scientific research publications. They are also raising the profile of Europe's leading edge research, while also moving towards innovation outcomes in the longer term.
The report notes that the general objectives of the Flagships are unique to this particular instrument. Such objectives continue to be highly relevant as part of Europe's overall Research and Innovation Strategy. There is a strong justification therefore to continue funding the instrument at EU level. FET Flagships also represent value for money as a Research and Innovation funding instrument.
The report is positive about the effectiveness on the research level, however "their future effectiveness in supporting innovation still needs to be demonstrated. Additional work should be undertaken on refining this aspect". The two year funding cycle of the flagships also puts a burden on the management of the projects.
The evaluators also see possible improvements in the Key Performance Indicators (KPIs) used by the Flagships which are considered to be very traditional in the sense that they are too oriented to describing typical research outcomes. "Further development of the KPIs is needed. KPIs can help emphasise and clarify differences with other research and innovation instruments of Horizon 2020", the report says.
Whether funding through flagship projects is more efficient than through other project funding mechanisms is not yet clear to the evaluators: "It is still too early in the history of the Flagships to be assessing this aspect in detail. Measuring efficiency in terms of share of management cost compared to overall cost, and showing better indicators in the Flagships in comparison to smaller-scale programmes, may not be fully satisfactory. This matter needs to be considered in more detail as the Flagships develop, particularly in respect to factors that may affect efficiency in the longer term. The most important matters are those that relate to the efficiency of strategic and operational management, and the efficiency of the mechanisms that link the Flagships to national initiatives."
EU added value of Flagship programme
European research programmes, by nature, should add to what is done on the national level in the countries. However, the report concludes on this aspect: "Linking research investments made through private and public funding across Europe with the two current Flagships is proving to be more difficult than expected. The relationship between the Flagships and national initiatives must be seen in the framework of a global view of the interaction between European and National programmes. Thus far the EU added value has yet to be fully demonstrated. To improve this situation two issues are crucial; (i) Flagship selection process and (ii) the mechanisms used to link to national initiatives."
All these aspects could influence new flagships, such as the Quantum computing and HPC/Supercomputing one.
The review panel also provides some recommendations:
1. Strategic relevance of the Flagship instrument in setting and implementing the European strategy for research and innovation
The continuing strategic relevance of the Flagship instrument for Europe's research and innovation is confirmed, with a strong endorsement of the thinking underlying the Flagship concept. The funding of the Flagships instrument represents good value for money in terms of the quality of the research and its potential for innovation. It is thus recommended that the Flagship initiative be continued, and new Flagships launched in fields where the concept is relevant.
2. Increase clarity of purpose and differentiation between the Flagships and other research instruments
The nature of FET Flagships and how they differ from other research instruments needs to be further articulated if the value commensurate with the scale of the investment being made is to be achieved. The concept of Technology Readiness Levels (TRLs) should be used to differentiate Flagships from other research instruments. In particular it is important to demonstrate how the focus of the Flagships shifts across TRLs with time. Further reflection on the design of the Flagship instrument is needed to reach a situation where science is driving innovation, and in turn, this innovation is driving new science. The Flagships need to demonstrate that they have strategic research and innovation agendas aligned with industrial interests. This should be based on understandings of existing industries, as well as opportunities for new entrants such as via start-up companies able to boost disruptive innovation. More effort should be devoted to involving SMEs in the Flagships.
3. Establish a standard means of assessing the Flagships based on Key Performance Indicators (KPIs) that fully reflect purpose
The KPIs used by the Flagships should be more sophisticated. They should include the means of assessing future potential. KPIs should be developed to measure the meaningful engagement of industry. Such KPIs should be grounded in realistic potential reflecting the existing industrial landscape and its potential for future developments. KPIs measuring the differences between Flagships and traditional Research and Innovation projects should be developed. A common system of assessing the Flagships should be created so that comparisons of KPIs across Flagships can be made.
4. Improve operational management to enhance the budget flexibility and reduce administrative overhead
Funding models and funding time-scales should be changed to reflect the special nature of the Flagships. A longer funding cycle should be implemented to improve the flexibility needed to respond to changing circumstances and opportunities. This will also help further increase administrative efficiency. National and EU level schemes should be examined to see which good practices could be transferred across to the Flagships. A relevant example at national level is the Innovate UK's funding for Catapult Centres.
5. Improve strategic management to enhance openness of the Flagships towards adopting new directions
The Flagships should be more open to external inputs that can challenge assumptions and direction. Today, the implementation of separated strategic and advisory activities from day-to-day management help Flagships enhance their capabilities to develop in new directions and towards innovation and exploitation of results. Strategic advisory boards that bring in international perspectives should be implemented in all Flagships to create a global context. These boards should also review the Flagships by defining benchmarks so that an understanding of European leadership can be established.
6. Improve coherence with other Horizon 2020 activities
A higher degree of interaction is recommended between initiatives under Horizon 2020 and beyond. It is essential that communication within the Commission is improved to ensure that stakeholders have a clear understanding of the opportunities on offer and the relationship between them.
7. Improve the process of selecting Flagships
It is recommended that commitment and buy-in from stakeholders including national authorities is obtained early on, before Flagships topics are finally selected. The process should be grounded in science and technology potential, the potential for impact and EU added value, and should attentively take into account the priorities recommended by Member States.
8. Improve engagement with national initiatives
The operation of the model of using national Partnering Projects to engage national level activities with the Flagships should be monitored with a view to finding improvements and simplifications. Additional ways for public and private initiatives to engage with the Flagships should be sought. Outreach activities to all interested parties, including relevant research communities that have not yet been engaged by the Flagships, should be increased with the aim of ensuring a greater structuring effect and avoiding duplication of effort.
The report provides more details on each of them.