In fall 2019, the Bell Museum received funding via a NASA TEAM II grant to create Mars: The Ultimate Voyage, a full-dome planetarium show and accompanying hands-on activities that focus on the interdisciplinary roles that will be needed to send humans to Mars. This report from Catalyst Consulting Group presents the findings from the summative evaluation completed in March–May 2023.
Education Development Center (EDC) conducted the external evaluation of this second phase of NASA@ My Library. Library staff from partner libraries increased their confidence and ability to facilitate library programming related to Earth, space, and engineering.
Making as a term has gained attention in the educational field. It signals many different meanings to many different groups, yet is not clearly defined. This project’s researchers refer to making as a term that bears social and cultural impact but with a broader more sociocultural association than definitions that center making in STEM learning. Using the theoretical lenses of critical relationality and embodiment, our research team position curriculum as a set of locally situated activities that are culturally, linguistically, socially, and politically influenced. We argue that curriculum
DATE:
TEAM MEMBERS:
Veronica OguilveWen WenEm BowenYousra AbourehabAmanda BermudezElizabeth GaxiolaJill Castek
Framing: Broadening participation and achieving equitable outcomes has been a core goal of the science museum field for over two decades. However, how to make progress has proven an intractable problem.
Methods: Focusing on five organizations who officially committed to diversity, equity, access, and inclusion (DEAI) by participating in a national professional development program, the researchers investigate how science museums attempt to enact internally-focused change via a mixed methods case study.
Findings: While these organizations considered a variety of structurally focused change
Described by Wohlwend, Peppler, Keune and Thompson (2017) as “a range of activities that blend design and technology, including textile crafts, robotics, electronics, digital fabrication, mechanical repair or creation, tinkering with everyday appliances, digital storytelling, arts and crafts—in short, fabricating with new technologies to create almost anything” (p. 445), making can open new possibilities for applied, interdisciplinary learning in science, technology, engineering and mathematics (Martin, 2015), in ways that decenter and democratize access to ideas, and promote the construction
DATE:
TEAM MEMBERS:
Jill CastekMichelle Schira HagermanRebecca Woodland
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. The project aims to understand ways to empower Latinx families (adult caregivers) to feel confident in their ability to support their middle school-aged girls in science and engineering activities. The project involves seven weeks of family programming around rockets or urban farming, as well as separate conversation groups for adult family members and girls. The project is relevant for several reasons: females and Latinx individuals are both underrepresented in science, technology, engineering, and math (STEM) coursework and careers; girls tend to lose interest in STEM by middle school age; and adult family members may have an impact on their children's attitudes and interests. The project partners with school districts and nonprofit organizations in Arizona and California.
This multidisciplinary project's priority is broadening participation, with a focus on increasing Latina girls' science and engineering interests through Family Project-Based Learning Activities, Conversation Groups, and a cultivated Community of Learners. It is based on the frameworks of Social Cognitive Career Theory and Community Cultural Wealth. The project aims to empower families (adult caregivers) to feel confident in their ability to support their daughters in science and engineering activities, which is often low especially among Latinx parents. The project will develop and evaluate two out-of-school enrichment methods for aiding families in encouraging and supporting their daughters in science: Family Problem-Based Learning Activities, which focus on rockets and urban farming, and Conversation Groups, which provide information and discussion for separate groups of parents and girls. A series of pilot studies will be conducted with 80 families to iteratively evaluate and improve the materials and procedure prior to the main study with 180 families, featuring a factorial design with a control group.
The materials developed and research findings may inform similar projects, especially those for students from culturally and linguistically diverse backgrounds and projects seeking to enhance the role of families in learning. The hypothesis guiding the project is that the greatest gains will be produced with the synergistic combination of enrichment methods. Another component that can potentially have broad impact is working to create environments where Community Cultural Wealth is recognized and enhanced through interactions of different families, creating Communities of Learners. This can inform projects that recognize the importance of community and/or that seek to use culture as an asset. The proposed study will engage three geographically distributed universities and several community partners. It will also provide university students and community leaders opportunities for work on instructional design, implementation, and research. The team will disseminate their findings and methods through multiple avenues to reach researchers, parents, leaders, curators, and educators in informal and K-12 settings.
This Research in Service to Practice award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
Katherine Short-MeyersonPeter RilleroPeter MeyersonMargarita Jimenez-SilvaChristopher Edwards
Informal STEM field trip programming is a large, yet under-researched area of the education landscape. Informal STEM education providers are often serving a more privileged section of society, leading to a risk of perpetuating inequalities seen throughout the education landscape. In an attempt to address the lack of research, this thesis explores the relationship between educational equity and informal STEM field trips. The intention was to collect data using a critical ethnography approach to the methods of qualitative questionnaire and interviews of informal STEM educators. A change in
The independent evaluators at Knight Williams Inc. developed a front-end survey to gather background and baseline information about the 16 partner organizations selected to conduct outreach programs as part of SciGirls CONNECT2. The goal was for two people from each partner organization to complete the online survey about their background and prior use of the SciGirls Seven and related strategies. A total of 30 partner representatives completed the survey by the requested deadline, resulting in a response rate of 94%. The majority identified as program leaders, with smaller groups saying they
This report presents findings of the Latina SciGirls mixed methods study, investigating the experiences of young Latinas participating in informal STEM programs across the U.S. that utilized the SciGirls educational model (including the SciGirls Seven strategies) and augmented with materials and practices intended to better serve Hispanic girls. The project was led by Twin Cities Public Television with funding from the National Science Foundation as an AISL Innovations in Development project. The STEM-related identity framework and research model used to guide this investigation is presented
Knight Williams, Inc. completed a summative evaluation report that addresses: (i) the reach and breadth of the Latina SciGirls broadcast program and online components compared to project expectations; (ii) the impact of the Family Fiesta events that incorporated use of SciGirls videos, in-person role models, and hands-on activities as experienced by the girls, family members, and role models that participated in the events; and (iii) the partners’ Latina SciGirls programs and how they used and reflected on the value of the SciGirls resources.
Science television shows are an important source of informal learning and enrichment for preschool-aged children. However, one limitation of television programming is that it is largely a one-way, non-interactive medium. Research suggests that children learn best through active engagement with content, and that parents can make TV watching more interactive by co-viewing and talking with their children. However, many parents and other adults may lack the time or experience and comfort with science language and content to provide critcial just-in-time support for their children. This study seeks to take advantage of recent advances in artificial intelligence that now allow children to enjoyably interact with automated conversational agents. The research team will explore whether such conversational agents, embedded as an on-screen character in a science video, can meaningfully interact with children about the science content of the show by simulating the benefits of co-viewing with an adult. If successful, the project could lay the foundation for a new genre of science shows, helping transform video watching into more interactive and engaging learning experiences. This project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This project will develop interactive videos incorporating a conversational agent in three 11-minute episodes of a future children's animated television program. The videos will enable children to speak with the main character of the show as the character solves everyday science mysteries, thus priming children to engage in observation, prediction, pattern finding, and problem solving through scaffolded conversation. This study will be carried out in two iterative cycles with the goal of developing and testing the embedded conversational function for each episode. In each cycle, the project team, which includes experts in children's TV production, as well as educational and HCI researchers will develop the storyboard and conversation prompts and follow-ups, create animated videos based on the revised script, and create a mobile application of the interactive video integrated with the conversational agent. Field testing with 10 children will be conducted to iteratively improve the embedded conversational function. In the pilot testing stage, a controlled study will be conducted with 30 children in each group (N=120): 1) watching the episode with the embedded conversational function; 2) watching the episode with a human partner carrying out the dialogue in the script rather than the virtual character; 3) watching the episode with pseudo-interaction, in which the animated character asks questions but does not attempt to understand or personally respond to children's answers; and 4) watching the episode with no dialogue. Data collected from the experiments will be used to examine whether and in what ways use of a conversational agent affects children's engagement, attention, communication strategies, perceptions, and science learning, and whether these effects vary by children's age, gender, socioeconomic status, language background, and oral language proficiency in English. The project will provide a comprehensive evaluation of the feasibility and potential of incorporating conversational agents into screen media to foster young children's STEM learning and engagement.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
Mark WarschauerDaniel WhitesonSara DeWittAndres BustamanteAbby Jenkins
Informal STEM education institutions seek to engage broader cross sections of their communities to address inequities in STEM participation and remain relevant in a multicultural society. In this chapter, we advance the role that evaluation can play in helping the field adopt more inclusive practices and achieve greater equity than at present through evaluation that addresses sociopolitical contexts and reflects the perspectives and values of non-dominant communities. To do this for specific projects, we argue that evaluation should privilege the voices and lived experiences of non-dominant