The summative evaluation documents and articulates what SCIENCES has improved or changed, and in what ways. The final design of the summative evaluation was based on findings from the front-end and formative evaluations, including using participatory evaluation techniques to engage community members in discussing their experience with the programs and assessment of community needs and assets at the close of the project.
The goal of the summative evaluation was to address discrete program impacts in the context of the project, as well as the cross-program impact of providing a thematically
In The Nature of Community: SCIENCES, we share the lessons learned from an innovative partnership designed to leverage the strengths of two nonprofit organizations—a large cultural institution and a smaller, deeply-rooted community-based organization, both of which offer informal science education expertise.
You’ll read first-hand reflections of how staff members, community leaders and members, children, and adults experienced this partnership: the expectations, surprises, challenges, successes, and lessons learned. We hope the description of this partnership inspires other organizations to
The employment demands in STEM fields grew twice as fast as employment in non-STEM fields in the last decade, making it a matter of national importance to educate the next generation about science, engineering and the scientific process. The need to educate students about STEM is particularly pronounced in low-income, rural communities where: i) students may perceive that STEM learning has little relevance to their lives; ii) there are little, if any, STEM-related resources and infrastructure available at their schools or in their immediate areas; and iii) STEM teachers, usually one per school, often teach out of their area expertise, and lack a network from which they can learn and with which they can share experiences. Through the proposed project, middle school teachers in low-income, rural communities will partner with Dartmouth faculty and graduate students and professional science educators at the Montshire Museum of Science to develop sustainable STEM curricular units for their schools. These crosscutting units will include a series of hands-on, investigative, active learning, and standards-aligned lessons based in part on engineering design principles that may be used annually for the betterment of student learning. Once developed and tested in a classroom setting in our four pilot schools, the units will be made available to other partner schools in NH and VT and finally to any school wishing to adopt them. In addition, A STEM rural educator network, through which crosscutting units may be disseminated and teachers may share and support each other, will be created to enhance the teachers’ ability to network, seek advice, share information, etc.
DATE:
-
TEAM MEMBERS:
Roger Sloboda
resourceresearchProfessional Development, Conferences, and Networks
In this final chapter, we reflect on four themes that cut across the preceding chapters on evaluating outcomes in informal STEM education (ISE): validity, context, technology, and evaluation capacity building. We also note several topics that we hope will be addressed more extensively in future conversations about evaluation in ISE. These include the ethical responsibilities of evaluators, theory‐based approaches to evaluation, formal training and professional development pathways for evaluators, and funding opportunities and the structural landscape of the field.
The NIH Science Education Partnership Award (SEPA) program of Emory University endeavors to use an over-arching theme of citizen science principles to:
develop an innovative curriculum based on citizen science and experiential learning to evaluate the efficacy of informal science education in after-school settings;
promote biomedical scientific careers in under-represented groups targeting females for Girls for Science summer research experiences;
train teachers in Title I schools to implement this citizen science based curriculum; and
disseminate the citizen science principles through outreach.
This novel, experiential science and engineering program, termed Experiential Citizen Science Training for the Next Generation (ExCiTNG), encompasses community-identified topics reflecting NIH research priorities. The curriculum is mapped to Next Generation Science Standards.
A comprehensive evaluation plan accompanies each program component, composed of short- and/or longer-term outcome measures. We will use our existing outreach program (Students for Science) along with scientific community partnerships (Atlanta Science Festival) to implement key aspects of the program throughout the state of Georgia. These efforts will be overseen by a central Steering Committee composed of leadership of the Community Education Research Program of the Emory/Morehouse/Georgia Institute of Technology Atlanta Clinical Translational Science Institute (NIH CTSA), the Principal Investigators, representatives of each program component, and an independent K–12 STEM evaluator from the Georgia Department of Education.
The Community Advisory Board, including educators, parents, and community members, will help guide the program’s implementation and monitor progress. A committee of NIH-funded investigators, representing multiple NIH institutes along with experienced science writers, will lead the effort for dissemination and assure that on-going and new NIH research priorities are integrated into the program’s curriculum over time.
Fostering greater inclusion in science creates benefits for both science and society. In this Innovations and Development project, the University of Utah will investigate how to sustain and scale the STEM Ambassador Program (STEMAP), begun in 2016 with AISL funding. STEMAP developed an innovative process to train scientists to engage members of the public, who cannot or do not gain access to science via conventional science education venues (such as museums, schools, zoos), by implementing activities in non-traditional settings. The 65 scientists trained by the initial STEMAP effort effectively engaged in over 45 settings including an affordable housing development, a youth residential treatment center, a state prison, a cooking class, a daycare facility, and several senior centers. The number of scientists applying to the program quickly exceeded STEMAP's capacity. Other institutions expressed interest in replicating the training. This project will explore strategies for scaling and sustaining public engagement training to support more scientists who can engage more people in more venues. Outcomes will serve to inform the broader implementation of STEMAP and the efforts of other public engagement programs, many of which face similar scaling and sustainability challenges.
Scaling and sustaining public engagement of science (PES) programs is a central challenge for many in the informal science learning community. This project will explore strategies to scale and sustain the STEM Ambassador Program. Research questions include: (1) How do different program formats increase or restrict program capacity and engagement outcomes? (2) What benefits accrue to scientists and their institutions by participating in public engagement in science activities that might serve as motivators to continue these activities? (3) Are funding and organizational models developed in business and other professional settings applicable to sustaining these programs? To address scalability, this project will explore the effectiveness of three dissemination formats: (1) the creation of a mentorship program for in-person trainees, (2) a train-the-trainer approach, and (3) online training with in-person mentorship. The project team will create an evaluation toolkit with participant surveys, rubrics for observers, and "on-the-spot" assessment tools developed under AISL Award 1811022 to assess the effectiveness of engagement activities delivered by trainees in each of the three formats. To address sustainability, the project team will document the values of public engagement training to both the participating scientists and their institutions via surveys and interviews. Consultants from the business sectors will create a PES Campus Council to explore possible financial, organizational, and leadership plans that will help sustain engagement efforts. Outcomes will be published in peer-reviewed journals and compiled into a dissemination framework to inform actions to scale and sustain STEMAP and other public engagement of science programs to engage more hard-to-reach audiences. Inverness Research will serve as the project's external evaluator.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
Research on how museum staff are trained continues to emerge. Training varies considerably across institutions and typically includes observations, shadowing, and trial and error. While museum educators put high value on increasing visitor-centered participatory experiences, engagement based on acquisition-based theories of learning is still common among floor staff, even after training. Facilitating learning about science, technology, engineering, and mathematics (STEM) topics in ways that support visitors in constructing their own understanding is difficult, especially since floor staff/facilitators may be working simultaneously with children and adults of a range of ages, backgrounds, and goals. This project will advance understanding of how to facilitate open-ended learning experiences in ways that engage visitors in practices that align with the STEM disciplines. The project will result in an evidence-based facilitation framework and training modules for training informal science educators. The work is grounded in constructivist theories of learning and identity work and focuses on visitors constructing understanding of STEM topics through active engagement in the practices of STEM. This model also results in learning experiences in informal settings that are mutually reinforcing with the goals of schools. This research is being conducted through an established researcher-practitioner partnership between MOXI, the Wolf Museum of Exploration + Innovation and the University of California at Santa Barbara (UCSB).
The two primary goals of the work are to (1) enable visitors to better engage in STEM practices (practice-based learning) and (2) investigate the role of training in helping facilitators develop the practice-based facilitation strategies needed to support visitors' learning. STEM content in this study is physical science. Prior work resulted in two tools that constitute part of a facilitation framework (a practices-by-engagement matrix and three facilitation pathways) which help educators identify appropriate goals based on how the visitor is engaging with exhibits. The development of the final tool in the framework, facilitation strategies, and the refinement of the first two tools will be done using a design-based implementation research (DBIR) approach. Data collection and analysis will be directed and completed by research-practitioner teams of UCSB graduate students (researchers) and MOXI educators (practitioners); MOXI educators will be both participants and researchers. Data collection activities include: video data using point-of-view cameras worn by visitors and educators; interviews of educators and visitors; observations of the training program; and educator reflections. In the final year, a small field test will be done at six sites, representing different types of museums. Interviews and reflections comprise the data collection at the field sites.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
BioEYES is a K-12 science outreach program that develops self-sustaining teachers as a replication strategy to address high demand for the program while promoting long-term school partnerships. This paper explores the practices of “model teachers” from multiple grades, who are empowered over a three-year period to deliver BioEYES’ hands-on science content autonomously, as compared to the program’s standard co-teaching model (BioEYES educator + classroom teacher). The authors found that BioEYES’ professional development (PD) workshop, classroom co-teaching experience, and refresher trainings
Ruff Family Science is a project funded by the National Science Foundation (NSF) that aims to foster joint media engagement and hands-on science exploration among diverse, low-income parents and their 4- to 8-year-old children. Building on the success of the PBS series FETCH! with Ruff Ruffman, the project leverages FETCH’s funny and charismatic animated host, along with its proven approach to teaching science, to inspire educationally disadvantaged families to explore science together. The project is utilizing a research and design process to create resources that meet the needs of families
Informal STEM education (ISE) organizations, especially museums, have used evaluation productively but unevenly. We argue that advancing evaluation in ISE requires that evaluation capacity building (ECB) broadens to include not only professional evaluators but also other professionals such as educators, exhibit developers, activity facilitators, and institutional leaders. We identify four categories of evaluation capacity: evaluation skill and knowledge, use of evaluation, organizational systems related to conducting or integrating evaluation, and values related to evaluation. We studied a
The Collaboration for Ongoing Visitor Experience Studies (COVES) aims to help science centers gather and share data to better understand visitors’ experiences. For the summative evaluation of COVES, the Museum of Science’s Research and Evaluation Department studied the program’s impacts on participating museums and museum professionals. Specifically, this evaluation was designed to:
Examine participants’ overall satisfaction with the collaboration and their likelihood to recommend it to others;
Understand the impact that COVES is having on Participating Institutions;
Assess whether
The National Federation of the Blind (NFB), in partnership with scholars from Utah State University and educators from the Science Museum of Minnesota (SMM), has developed the Spatial Ability and Blind Engineering Research (SABER) project to assess and improve the spatial ability of blind teens in order to broaden the participation of blind students in STEM fields.
Activities began this summer (2018) with a week-long, residential engineering design program for thirty blind high school students at NFB headquarters in Baltimore. The evaluation focused on perceptions of process and measures of