Evaluation Toolkit Case Study (tool 7 – Pre / Post Quizzes)
With thanks to Julie Nováková for this contribution
The events
Pre-post quizzes were used at three events:
Event 1 (13/3/2018): Astrobiology seminar “Polar sciences in astrobiology” (speaker: Jana Kvíderová)
- Total attendance: 12 people (all adults). Quiz responses: 7
- Time: 16:30-18:00
- Place: Faculty of Science, Charles University in Prague
- Find this talk on YouTube
Event 2 (15/3/2018): Café Nobel talk “The diverse world of exoplanets” (speaker: Julie Nováková)
- Total attendance: 35 people (approximately 2/3 adults and 1/3 high school students, a few children with parents). Quiz responses: 15 (some only partial; 8 complete)
- Time: 18:00-19:30
- Place: Teplice Planetarium
Event 3 (27/3/2018): Astrobiology seminar “Geophysics of planetary habitability” (speaker: Lena Noack)
- Total attendance: 25 people (all adults). Quiz responses: 11
- Time: 16:30-18:00
- Place: Faculty of Science, Charles University in Prague
- Find this talk on YouTube
Further details about the events
The astrobiology seminars are open both to university students (especially attendees of the Introduction to Astrobiology class in the winter semester) and to the general public. Each talk has a Facebook event, and if the speaker agrees, we record the talk and make it public on YouTube. The videos from the previous two years usually have between 300 and 500 views. Julie Nováková is the main organizer, with colleagues Jan Toman and Tomáš Petrásek as co-organizers. The talks take place at the Faculty of Science, Charles University, in late afternoon.
Café Nobel is an activity similar to Science Café and located in northern Bohemia (Ústí nad Labem, Teplice, sometimes other cities). It’s organized by a nonprofit of the same name and includes popular talks on both science and humanities (although more frequently science). They’re usually attended mostly by adults, partly by older pupils, and sometimes children with their parents.
What we did
Data collection
A few minutes before the beginning of each event, I kindly asked the audience whether they would be willing to assist in evaluation of the event by filling a short anonymous pre- and post-questionnaire that would help us see whether they have learned new things there and whether everything in the talk was clear enough or we should improve it the next time. Most people agreed (the response numbers are lower than attendance numbers mostly because some people arrived late or had to leave earlier). I suggested using anything unique as the identifier code – a random number, a favourite character, their initials plus some numbers from their birth date, etc. (I did not suggest only the last option, since using initials and birth date may not seem anonymous to some people, and I was afraid that it would make them think of it more as a test than just a completely voluntary quiz). I emphasized that we’re not testing them, just want to improve the event for them. After each talk, I reminded them of the quiz and asked them to please fill in the post-quiz (and leave the pre-quiz as it was). Afterwards, they handed both papers to me. I waited until the end to collect the papers, since it would delay and disturb the talk at the beginning.
Data analysis
An Excel spreadsheet was created for analysis, with one sheet for each event (we have provided a copy of this spreadsheet at pre-post data). Within the sheet for each event, there is a summary and extra information at the top, followed by scoring criteria (‘Quiz overview’). Finally, there is a line for each respondent, with their pre and post scores for each question, which were entered from the paper quizzes. The total points pre- and post- are found at the end of each row.
What we found out
In all the events, the number of points scored in the post-quiz was generally higher than in the pre-quiz. Sometimes the number stayed the same (especially if the person knew the correct answers already in the pre-quiz), and in only two cases in total, the number dropped (by one and two points, respectively). These drops were related to questions about plate tectonics (twice) and the generation of Earth’s magnetic field. Perhaps the discussion of whether Mars could have had plate tectonics in the past or not confused the two people (while the rest answered correctly) and should have been clearer. I am not sure why there was the one wrong response in the other question (saying that Earth’s magnetic field is generated by convection in the mantle, while in reality, it’s convection in the core). Although there were only 26 complete responses (both pre- and post-quiz fully answered and paired), so any statistical exploration may not be meaningful, I ran a t-test just to see whether the difference between the means (pre: 6.35, post: 8.08) is statistically significant, and it is (two sided p=0.0024). Based on that, we can carefully conclude that the attendees indeed learned something new during the talks, and that our approach is most probably a good start. As I mentioned above, we also identified a couple of areas of content that we could perhaps be a little more careful about explaining in future, to try to avoid confusing our audience.
Back to Tool 7 – Pre / Post Quizzes page
