The evaluation process
This section provides an introduction to evaluation, reasons for evaluating and an overview of the evaluation process. We would advise reading through this section before you start planning your evaluation.
What is evaluation?
Evaluation is a systematic process designed to help you better understand – and improve – your participants’ experience of your outreach activities. It involves collecting information (for example feedback, observations or quiz results) and reflecting on what worked well, what could be improved, and what changed for the people involved as a result of their participation.
Evaluation is an ongoing process and can include both formative and summative evaluation techniques:
- Formative evaluation occurs whilst a project is still happening, particularly towards the start, and is designed to identify actions that can be taken to further improve the project before it ends.
- Summative evaluation can occur at any stage of a project, but is normally focused towards the end. It is designed to provide a summary of the achievements (results) of the project, based on the evaluation data collected.
The data collection tools we’ve included are well suited for summative evaluation (e.g. at the end of an activity or event), but can also be useful in formative evaluation (e.g. informing the development of future activities).
Why evaluate?
As part of the development of this toolkit we spoke to people in the planetary science community about the reasons that they evaluated
their outreach activities. The main reasons identified fell within four key areas:
- Improving – Gathering feedback on participants’ experiences to help identify what could be changed within the activity in order to improve it.
- Providing evidence of knowledge gained – Many outreach activities are specifically designed to improve the audience’s factual understanding of a specific topic (for example space exploration or exoplanets or astronomy more generally). As a result, there is a lot of interest in using evaluation techniques to find out exactly what participants learn from an activity.
- Identifying other successes – Some outreach providers are interested in other types of changes that occur for participants, for example whether they are inspired to follow a career in space science. We recommend the Generic Learning Outcomes framework from the Arts Council, to help people clarify what sorts of wider outcomes they are hoping to achieve. The final rows in the tools overview also explicitly indicate what sorts of tools best suit the main categories of outcomes.
- Celebrating – Providing an opportunity for both participants and outreach providers to creatively share the joy and inspiration, and for such excitement to be shared more widely (for example with funders and managers).
These identified purposes have provided the basis for the choice of tools showcased within the Europlanet Evaluation Toolkit.
We have also heard arguments that evaluations are pointless or ‘only tell me what I already know’. In those cases it’s often the funder that requires the evaluation evidence – so one important improvement could be to identify the most efficient / cost effective way to gather such data. Alternatively, you might be interested in a different approach that digs a little deeper and explores complementary aspects that you’re not so sure about. For this reason we’ve tried to include a wide range of tools and techniques to evaluate outreach activities, to suit different needs.
The evaluation process
The evaluation process can be an ongoing cycle: you plan, collect and analyse your data and report back to funders and stakeholders but the information you collect may also inform ongoing and future events and help you plan further evaluations. To plan a successful evaluation, you will need to consider all the steps in the process before you start or you may find you have, for example, inadvertently chosen an inappropriate tool for the kind of data analysis you wanted to carry out.
– Define
The first step in the process is to define exactly what you want to find out from the evaluation. This may depend on who the evaluation is for – is it just for your own purposes or is there information you need to collect and feed back to stakeholders or funders? You might also consider the budgetary and time resources available and whether there are any ethical considerations you need to take into account.
– Collect
Once you have defined the information you are looking for, you can choose an appropriate tool to collect your data. Here, you need to think about the context of your event, for example, the age of the audience and the time and resources you have available.
– Analyse
Part of the process of choosing the tool will be thinking about how you will analyse the data you collect. This might depend on your expertise in data analysis and could determine the number of people you need to involve in your evaluation.
– Report and Share
Finally, once you have analysed your data, you will need to report your findings in a meaningful way for the target audience. It is also worth thinking whether you could share your results with the wider community in some way. You may also want to use the information you collect to adjust or improve existing or future events.
