EvalFest Training Videos: A How-To for Data Collection, Management, and Analysis
Karen Peterman, president of Karen Peterman Consulting, Co., co-wrote this blog. Along with Todd Boyette, Katherine and Karen lead EvalFest, a community of practice that develops, tests, and shares evaluation approaches for science festivals.
In the mid-2000s, just a handful of American science festivals were in existence. Since then, the number of festivals has grown significantly. The growth in these events presented a wonderful and unique opportunity to advance evaluation practice with the community.
EvalFest, established with National Science Foundation funding in 2014 (award nos. 1423004 and 1423050), sought to bring this nascent community together around the topic of science festival evaluation.
So far, 25 U.S. science festivals have participated in EvalFest. They span the country, range in size from a few hundred to several thousand attendees, last from one day to a month, and have anything from an all-volunteer team to a multi-person staff. We came together in a community of practice to develop, test, and share evaluation approaches for science festivals. Ultimately, we hope that our efforts will help the science festival community make more informed decisions about festival evaluation and that the lessons we learn will be of use to others in the informal STEM education (ISE) community.
Pictured above: Volunteer field researchers for the Bay Area Science Festival are trained on administering surveys. Photo courtesy of Katherine Nielsen.
Building a Suite of Evaluation Resources
One thing we learned early on in EvalFest is that we needed to help our partners learn about and use accepted data collection procedures. We started producing one-page guides that evolved into a set of training videos and grew to cover topics in data management and analysis.
We also realized that these topics are tricky for lots of ISE institutions, not just science festivals. The videos and resources available on EvalFest.org have been framed with the hope that they will be of use to all kinds of ISE practitioners. All of the resources are free to download.
Below, we describe some of the things you’ll find on the website.
Data Collection
This section covers some basic rules of data collection (like gathering data from one and only one person), procedures for collecting surveys at public events, understanding consent forms, and working with the Institutional Review Board (IRB). In total, there are the six videos and 12 data collection resources. For example, in the video below, we explain the “5th Person Rule,” which is a systematic strategy for approaching people and having them complete a survey at events. This technique could be used for a variety of event types in any ISE setting; a science center could use it during a weekend family event, or a zoo could employ this method during special evening hours.
Data Management and Analysis
Like the ISE community writ large, our EvalFest partners have a wide range of evaluation expertise. Some partners are professional evaluators, while others are festival team members who conducted their first evaluation as part of our project. Once we figured out how to help partners collect data using best practices, the next step was to help our community clean and analyze their data.
The videos and resources in our Data Management and Data Analysis sections were designed to help ISE practitioners manage their data and do basic analyses like chi square, t-test, and Analysis of Variance (ANOVA). For a quick overview of what we have to offer, watch the video below.
The EvalFest website also features several evaluation instruments designed and used within the festival community, as well as five short videos that explain how to clean and organize data in Excel. These videos include the kinds of data that are typically generated by outreach surveys, and they show step-by-step how to do things like filter your data, recode your data, and create pivot tables. View them here: evalfest.org/data-management
Next Steps
Our tagline on the EvalFest website is “A Celebration of Evaluation.” We think evaluation is important, useful and worth celebrating! We hope that these resources will help support the ISE community in conducting high-quality evaluation work for many years to come. We will continue to add resources throughout 2019. So check out the website, let us know what you think, and feel free to suggest resources that you’d like us to create next! You can reach us at katherine.nielsen@ucsf.edu and karenptermanphd@gmail.com.
Photo courtesy of Lawrence Luk.