Evaluation tool – Pre / post quizzes
What is this tool?
Pre / post quizzes are brief surveys that are used before and after an event. They are ideal for helping you understand whether or not your audiences have learnt key aspects of the content you are trying to convey. Their purpose is to focus specifically on your content (not on other things your audience might or might not know).
What kind of activities can I evaluate with it?
Pre / post quizzes are particularly effective for use with school groups or interested adults participating in interactive workshops, courses, or online content, but can also be used with general public and lectures / presentations.
Pre / post quizzes at a glance…
When should I use it?
This tool is best suited to use at the beginning or end of an event.
What do I need?
- Pre-post quizzes can be delivered on paper or online.
- If you decide to use paper versions, you’ll need to prepare plenty of copies in advance and have pencils or pens available at your event, as well as flat surfaces to write on (e.g. clipboards or tables).
- Time set aside at the beginning and end when people can fill in the quiz!
Let’s get started…
Prior to the event, you need to create the quiz – see our Top Tips section for some essential advice to get you started. The pre- and post-quiz questions need to be the same, so they can be compared. They also need to relate very directly to the content you are covering – this is not the place to find out what else they might know about the topic. For example, if you are giving a talk on exoplanets, you might have three questions related to different aspects, such as the numbers of exoplanets discovered so far, the kinds of stars they have been found near and whether or not scientists have techniques they can use to learn about their atmospheres.
There are many different question types that could be used, but three of the most common types are multiple choice, True/False and multiple response (see the post-event surveys section for some other question types):
Multiple choice involves a question with different response choices (usually three or four) to select from. There should only be one correct answer. For instance:
What is the name for a region of space that has a gravitational field so intense that no matter or radiation can escape?
a. A green hole
b. A black hole
c. A purple hole
d. A cucumber
A True/False question consists of a statement which respondents reply is true or false.
Please select one answer:
The moon is made of cheese. True / False
In a multiple response question, individuals can tick more than one response (i.e. there is more than a single right answer). For instance:
Which of the following are planets in our solar system? (Tick all that apply)
Note: short-answer questions – or any questions where participants have to write in their answers – are not usually suitable for a pre-post quiz as they will take too much time for respondents to complete and it’s usually very complicated to accurately compare before and after answers for open questions.
The questions on your post-quiz need to be identical to those on the pre-quiz, for ease of comparison. The post-quiz can be distributed at the end of the event (either on paper or as an online link participants can access via their smartphones, tablets or computers). For higher response rates be sure people have time to fill them in before leaving the event! You can also choose to run your post-quiz at a later date in order to explore longer term impacts, this is known as a delayed post-quiz.
Ideally you want to be able to match individuals’ responses to the pre-quiz directly with their responses to the post-quiz. However, this means gathering personal information – most likely names – which may be considered unethical (it’s better to keep evaluation data anonymous if possible), or may worry the participants as their answers could be traced back to them individually. One alternative is to have participants create a code that is unique to them, but not known by anyone else, that they write at the bottom of their pre- and post-quizzes. For example you could add the following to the start or end of the quiz:
We would like to compare your responses before and after the event. To do this anonymously, please create a code as follows and enter it on the line below.
- First 3 letters of your father’s first name
- First 3 letters of your mother’s first name
- Last two numbers of the year of your birth
- For example, if my father’s name is Marco and my mother’s name is Sara and I was born in 1983 my code would be MARSAR83.
Or, if your audience is fairly small in size and you think it’s unlikely that they’ll choose the same code, you can ask them to each come up with their own random suggestion (as Julie did in our case study example).
OK, what do I do with my data now?
You’ll need to start by entering your data into a programme for analysis. A standard spreadsheet package such as Excel is generally fine but you can also use a dedicated statistics programme such as SPSS if you are familiar with it. Allow one row per respondent, with each question having its own column.
Score responses to the questions (i.e. ‘1’ for correct and ‘0’ for incorrect) and enter each person’s score into the column corresponding to the question. If there are some questions that can be ‘proportionally correct’, you may need to develop a scale, e.g. from 0-2 (0 if got it wrong, 1 for partially correct, 2 for fully correct; or +1 for each correct response on a multiple response question and -1 for each incorrect response). For example, drawing on the questions above, for a respondent who got the question about black holes correct, the true/false question incorrect and then correctly selected three of the planets but also got one wrong, the table might look like the following:
|Respondent||Black hole||Moon cheese||Planets|
Then, you can compare the percentage correct (or the scores) for each question, by averaging across all the respondents. If you do have matched data from pre/post quizzes, you can come up with a scoring schema that reflects whether your respondents moved from incorrect to correct (or correct to incorrect) or whether their scores increased or decreased on each question and overall. For instance, a person could receive 1 point per question if they have moved from incorrect to correct – you are basically trying to describe whether their performance improved, decreased or stayed the same. (See our case study example for further details on how such scoring can be done successfully).
WARNING! DON’T OVERCLAIM!
In the likely event that you aren’t able to fully match the respondents in your pre- and post-quizzes, you need to be cautious in how you interpret the data. You can’t assume that the same sorts of people completed the before – and after – surveys. For example, maybe only people who particularly felt that they learned something took the time to complete the post-event survey.
However, collecting pre- and post-data can still be useful! If you see evidence of improved understanding of your key items across the board, it is likely that at least some of your audience have learnt something – especially if you have good reason to believe that those who responded before and after aren’t two entirely different groups of people (if so, then you need to be really careful!)
In addition, if your pre- and post-quiz happened fairly closely together in time (especially if the post-quiz was immediately after your event), then you cannot claim a long-term impact. On the other hand, if individuals’ performance on the quiz improved from pre to post and the post-quiz was immediately after the event, it is unlikely that the improvement is due to some other cause!
Finally, although the pre-quiz may have drawn people’s attention to certain parts of the event (or the content covered), that does not mean that other content was not learnt. Just use your common sense and focus on what data you DO have and what you CAN say about that data when writing up your results…
Got it! How can I take this further?
If you would like, you can incorporate more question types into the mix. For instance, to get a sense of what people may have learnt, and how confident they feel, you could expand on the True/False question type to include the following response choices: Definitely true, Probably true, Not sure, Probably false, Definitely false.
You might also be interested in seeing whether people seem to have retained some of the content learned for a period of time after the event. This is known as a delayed post-quiz. To do so, you could try sending a link to the post-quiz after a period of time has passed (if you have been able to collect email addresses from participants). This would be most suitable for use with an adult audience, though if your audience consists of school children, you can also work with their teacher to distribute the questionnaires.
Be warned, however – if you use a delayed post-quiz and see an improvement in responses, this could be because of something people learned later, outside of your event. They could also have looked up the answers when completing the later survey at home! Finally, when using delayed post-quizzes, it is likely that you will receive far fewer responses – you may need an attractive prize or other incentive to encourage people to submit their delayed post-quiz responses.
If you have data from enough respondents (at least 20 and ideally closer to 50) and would like to delve further into your data, you could utilise some basic statistical analyses. Given the short nature of the quiz, either t-tests or non-parametric versions (Mann–Whitney U test or Wilcoxon signed-rank test) are likely to be most appropriate. Based on feedback from members of the Europlanet community when developing this toolkit, we haven’t gone into a lot of detail on statistical techniques here. However there are plenty of good statistical advice resources available; one of the most readable authors is Andy Field.
Quirky questionnaires – Questionnaires conducted in a quirky way related to the topic of the activity, e.g. folded into paper aeroplanes and flown into the ‘hanger’ (collection box) or ‘slam dunked’ through a basketball hoop.
Kahoots – Quizzes become more fun with “Kahoots”: free editable online games. These are perfect for school students of all ages, and can be run on many different types of devices, making them a very flexible addition to a workshop or classroom environment. For older audiences with their own technology you can also run them using their own mobile phones.
Download pdf of the Pre / post quizzes tool worksheet
In this Case study, Tool 7 – Pre / post quizzes were used at a series of astrobiology seminars and science cafes to help evaluate whether the audience had learned new things there and whether everything in the talk was clear.