---

---

Evaluating the Impact of Europlanet 2024 RI 

Evaluation Officer for Europlanet 2024 RI, Jen DeWitt, and Communications Manager, Anita Heward, report on the outcomes of the Europlanet 2024 Research Infrastructure (RI) project.  

Read article in the fully formatted PDF of the Europlanet Magazine.

T

ver two days in February 2020, 69 members of the planetary community met in the historic town of Windsor, UK, for the kick-off of the Europlanet 2024 Research Infrastructure (RI). Largely unaware of the global disruption and challenges that were to come, we discussed our plans for the project and the outcomes that we hoped we would achieve. 

An important part of the conversation was how we could measure success. Europlanet 2024 RI was the third in a series of projects to provide access to research facilities and field sites, virtual services, tools and networking to support the planetary community. To continue these programmes beyond the end of the €10 million grant from the European Commission in July 2024, we would need to be able to clearly show the impact we had achieved, demonstrate a wide-reaching user-base, and make a solid case for sustainability. 

Building on the consultations in Windsor, we put in place an evaluation framework through which we could examine the extent to which the project’s activities reached their intended goals, and understand what caused barriers to their success. Our framework aligned with the Organisation for Economic Co-operation and Development (OECD) strategic objectives for assessing research infrastructures, which include a range of core indicators for scientific, technological, education and training, economic, and social and societal impacts. The framework also incorporated categories of information requested by the European Commission’s reporting portal. 

The project management and activity coordination teams identified metrics and indicators of impact, such as publications and conference presentations, new collaborations, numbers of users or participants in workshops and training sessions, and media coverage. We also defined the user-groups that we wanted to engage, such as researchers, industry, policy makers, early careers and under-represented communities. Through the Europlanet website and online tools, we set up a series of forms and databases to track this information on a regular basis, and designed feedback surveys for users and visitors to facilities to provide input on their experiences. To help understand the experience of participants in more depth, we included questions in the feedback forms that asked respondents if they would be willing to be contacted for a short interview to assist with evaluation. 

Our plans were in place – and then everything changed. A global pandemic was declared on 11 March 2020, bringing months of deadly illness, lockdowns, travel restrictions and uncertainty. Partners were later affected by wars in Ethiopia and then in Ukraine. Many of the activities planned in Windsor were delayed or implemented in a radically different way. A six-month project extension was granted by the European Commission to allow more time for activities to take place. However, four-and-a-half years later, and despite all the challenges faced by the community on personal, professional and societal levels, the project evaluation still shows significant impact emerging from Europlanet 2024 RI. While evidence has been found in all five of the OECD impact areas, it is particularly strong in the scientific, technological, education and training areas. 

Transnational Impact 

There can be no doubt that the flagship Transnational Access (TA) programme, which has supported 292 researchers to visit facilities and field sites and carry out 197 projects, has led to substantial scientific impact. This impact will be ongoing beyond July 2024 as the new lines of research and collaborations catalysed by the visits continue (page 8). 

In terms of metrics, over 300 publications are expected from the TA visits and more than 400 conference presentations have been given or are planned, over half of which are led by early career researchers. Evaluation data collected, which includes a post-visit form from every trip, 62 interviews and 84 long-term follow-up forms submitted 6-12 months after visits, has reflected very positive experiences. The visitors highlight that the trips have been extremely productive scientifically, with two thirds of visitors considering new scientific activities that they would not have envisaged without Europlanet support. Over 90% are planning future collaborations with their host facility. In particular, early career researchers and students report that the TA programme provided them with opportunities – including collaboration and network-building – that would not otherwise have been available to them, thus accelerating their career development. 

As one TA visitor to Botswana said in an evaluation interview, “People don’t collaborate with strangers, they collaborate with people that they know. I feel like [being] part of Europlanet is one of these great things… it helps gets the ball rolling with meeting people and building networks.” 

The TA programme also shows activity related to OECD strategic objectives of scientific integration and collaboration. Thirteen additional facilities in Korea were included in a successful pilot for a sustainable TA programme on a global scale. In an innovation driven by the pandemic, a number of TA visits also happened online. Although not ideal from both a scientific and collaborative standpoint, in some circumstances virtual TA visits have proved their potential as a cost-effective way of getting data, widening participation and reducing the carbon footprint of TAs. In particular, researchers building on existing work and established collaborations reported successful virtual visits. 

The TA programme has also led to impact in areas beyond planetary science. Upgrades to facilities have provided increasing support for local infrastructure and associated employment (page 15), and there has been at least one SME company that was developed as a result of the TA programme. Facilities have been used in other disciplines, e.g. archaeological studies, and wider society has been engaged with the TAs through outreach by visitors, social media, press coverage, and a set of teaching resources linking planetary analogues to schools’ curricula. 

Overall, the TA visits have enabled impacts, particularly on a scientific basis, that would simply not be possible without access to the Europlanet 2024 RI’s facilities and expertise. 

Impacts on Data Services 

Measurable impacts have also emerged from Europlanet’s four Virtual Access (VA) services. The VESPA Virtual Observatory (VO) for planetary data has been in development since 2015 (page 28). EPN-TAP, the protocol developed by VESPA to describe and access data, was approved by the International Virtual Observatory Alliance in August 2022 as the international standard for Solar System data in the VO. By July 2024, there were 290 services declared through EPN-TAP in the IVOA registry. EPN-TAP has also been adopted as a standard by the International Planetary Data Alliance of national and international space agencies. ESA has implemented EPN-TAP on its Planetary Science Archive, which includes over 33 million files from 40 years of space missions, and NASA’s Planetary Data Service Planetary Plasma Interaction node is also developing EPN-TAP access for the entirety of its resources. 

By the end of the Europlanet 2024 RI project, the number of planetary and heliophysical data services installed in VESPA had grown from 52 to 88, with around 15 more in the final stages of testing for inclusion. 

The SPIDER VA tools support modelling of planetary environments and solar wind interactions through a dedicated run-on-request infrastructure and associated databases (page 30). Applications of SPIDER’s tools have led to several high-impact publications, and have been used to support planning of the BepiColombo and Juice missions. Opportunities identified through SPIDER for synergistic observations by BepiColombo during its cruise phase with the Solar Orbiter and Solar Parker Probe missions have also been implemented by ESA and JAXA. 

Two new VA services were introduced in Europlanet 2024 RI. The Geological Mapping (GMAP) activity has developed infrastructure for Europe in geological mapping of planetary bodies (page 34). The establishment of the annual Geology & Planetary Mapping Winter School, which attracted 600 online participants in the most recent edition, has been a standout success of the Europlanet 2024 RI project. As well as receiving very positive feedback from attendees, the Winter Schools have played an important role in building a young, global community of planetary mappers, with the numbers of participants from under-represented states increasing year on year, and early career scientists representing the majority of participants. 

Europlanet 2024 RI has also aimed to foster wider use of Machine Learning (ML) technologies in planetary research (page 23). A portal has been developed to provide access to open-source ML tools for seven science cases, with codes and scripts available through a GitHub. Much of the development work for the ML tools has been performed by early career researchers, enabling them to build both their skills and experience and their professional profile within the scientific community. While dissemination and training for the tools continues, Europlanet ML activities have already led to new projects and a successful case study in collaboration with industry. 

Networking and Collaboration 

Restrictions on travel and face-to-face contact meant that Europlanet 2024 RI’s networking activities were focused online for the first half of the project. In particular Europe’s main annual gathering of planetary scientists, the Europlanet Science Congress (EPSC), was redefined as a fully-virtual meeting in 2020 and 2021, using a combination of asynchronous scientific presentations and a programme of live events. The success of these virtual meetings was critical to the sustainability of Europlanet, as EPSC is a core source of income and a major financial loss in either year would undoubtedly have led to the failure of Europlanet as a viable not-for-profit association. 

Since 2022, face-to-face meetings have resumed, bringing lessons learned from virtual meetings to support hybrid access. In particular, the first Europlanet Research Infrastructure Meeting (ERIM) in 2023, which co-hosted 59 hybrid workshops related to Europlanet activities and sustainability roadmapping, was a unique opportunity to train the RI’s userbase and consult the community on Europlanet’s future (page 52). 

One distinguishing feature of the 2024 RI has been its extensive support for students and early career researchers. The wider societal pressures of the pandemic, from financial restrictions to mental health, have meant that early career researchers have been particularly vulnerable over the past few years. Europlanet has offered a range of programmes to support early careers through bursaries, training schools, internships and mentoring services. A mentorship programme, launched in August 2020, now involves 71 mentor-mentee pairs (page 46). Evaluation forms from 20 mentors and 19 mentees, supported by interviews, report positive experiences of the programme, which has achieved a broad geographical reach of over 20 countries. Thus, this type of initiative can be key in addressing some of the diversity and inclusion issues facing planetary science as a field, as well as continuing to build and support the planetary science community. 

The 30 expert exchanges supported through the project have also led to impacts across all five OECD areas (page 43). All seven participants interviewed for the evaluation of the programme reported very positive experiences of the exchanges. Because individuals could apply easily and for such a wide range of activity, funds often supported very targeted high-impact activities, for relatively small costs. 

Europlanet 2024 RI’s other main innovation, a network of small-scale telescopes in Europe and beyond to support planetary research, was established in June 2020 (page 40). Out of 44 successful applicants who applied for observation time, 13 provided feedback, with respondents citing the distinctiveness of the telescope network in the ease of application, the training provided, and the location and number of telescopes. Many respondents, particularly amateur astronomers, said they would not have been able to collect their data in any other way. 

More widely, collaborations stimulated through Europlanet 2024 RI, and involving its beneficiaries, have led to several successful proposals for new projects funded through the European Commission and national agencies. From the development of ML tools to mobilising researchers in Africa, these external projects act as multipliers for the impact of Europlanet 2024 RI in many different communities. The project has also organised and participated in briefings, conference sessions and other events to engage policy makers and industry with planetary science. Input coordinated through Europlanet into policy consultations has led to changes in strategic documents at a European scale that gives more prominence to planetary science and related fields. 

Informing Sustainability 

The ultimate aim of this evaluation is to help in considering where sustainability efforts should be focused beyond the EC-funding. The VAs will largely be supported through their host institutions, but continuing the support for TA visits would seem to be key. Even a small TA programme is likely to deliver scientific and technological impact for the planetary community, particularly if it continues to prioritise accessibility for early careers and under-represented sectors. Likewise, the Telescope Network, the Winter School, training, mentoring and expert exchange programmes offer value for money with demonstrable impacts for the planetary community. 

In summary, collaboration and networking has emerged as key to the impact of the RI, both as a pathway to impact and an indicator of its attainment. The core access is supported by a host of interlocking activities that clearly impact on individuals and the science that they engage in. The network element provides support for researchers at all levels and strengthens the community, which in turn contributes to sustainability and improving the science. The outward facing nature of much of the RI activity means the impacts are broader than just on the scientists participating directly and their science. Overall, the RI vitally strengthens the planetary science community – and wider society. 

Links

1. Reference framework for assessing the scientific and socio-economic impact of research infrastructures. OECD. 2019. DOI: doi.org/10.1787/3ffee43b-en 

2. Horizon 2020 Periodic Report Template. European Commission: https://bit.ly/3xjD9oo 

Acknowledgements

Europlanet 2024 RI Special Edition of Europlanet Magazine