In this final chapter, we reflect on four themes that cut across the preceding chapters on evaluating outcomes in informal STEM education (ISE): validity, context, technology, and evaluation capacity building. We also note several topics that we hope will be addressed more extensively in future conversations about evaluation in ISE. These include the ethical responsibilities of evaluators, theory‐based approaches to evaluation, formal training and professional development pathways for evaluators, and funding opportunities and the structural landscape of the field.
The NIH Science Education Partnership Award (SEPA) program of Emory University endeavors to use an over-arching theme of citizen science principles to:
develop an innovative curriculum based on citizen science and experiential learning to evaluate the efficacy of informal science education in after-school settings;
promote biomedical scientific careers in under-represented groups targeting females for Girls for Science summer research experiences;
train teachers in Title I schools to implement this citizen science based curriculum; and
disseminate the citizen science principles through outreach.
This novel, experiential science and engineering program, termed Experiential Citizen Science Training for the Next Generation (ExCiTNG), encompasses community-identified topics reflecting NIH research priorities. The curriculum is mapped to Next Generation Science Standards.
A comprehensive evaluation plan accompanies each program component, composed of short- and/or longer-term outcome measures. We will use our existing outreach program (Students for Science) along with scientific community partnerships (Atlanta Science Festival) to implement key aspects of the program throughout the state of Georgia. These efforts will be overseen by a central Steering Committee composed of leadership of the Community Education Research Program of the Emory/Morehouse/Georgia Institute of Technology Atlanta Clinical Translational Science Institute (NIH CTSA), the Principal Investigators, representatives of each program component, and an independent K–12 STEM evaluator from the Georgia Department of Education.
The Community Advisory Board, including educators, parents, and community members, will help guide the program’s implementation and monitor progress. A committee of NIH-funded investigators, representing multiple NIH institutes along with experienced science writers, will lead the effort for dissemination and assure that on-going and new NIH research priorities are integrated into the program’s curriculum over time.
RUFF FAMILY SCIENCE is a project funded by the National Science Foundation (NSF) that aims to foster joint media engagement and hands-on science exploration among diverse, low-income parents and their 4- to 8-year-old children. The project is using a research and design process to create an implementation model and prototype resources (digital media, hands-on activities, and supports for educators) to build new knowledge about the potential for digital media to inspire and support intergenerational science learning among vulnerable families.
WGBH and Education Development Center, Inc. (EDC)
DATE:
TEAM MEMBERS:
Mary HaggertyHeather LavigneJessica AndrewsAlexia RaynalMarion GoldsteinJaime Gutierrez
Tinkering creates a bridging point between a learner’s personal interests and experiences and a broad range of possible learning outcomes. It offers valuable opportunities to engage all students in STEM and fosters a more inclusive STEM education. In this way, it is very much aligned with a Science Capital Teaching Approach: fundamentally, it is a highly personalised pedagogy, which allows the learner to follow their own interests and set their own goals.
This resource has been designed to help teachers integrate the Tinkering approach and the Science Capital framework in their practice
In 2018, the Croucher Foundation conducted its third annual mapping exercise for the out-of-school STEM learning ecosystem in Hong Kong.
The study reveals a rich and vibrant ecosystem for out-of-school STEM in Hong Kong with over 3,000 discrete activities covering a very wide range of science disciplines. This third report indicates extremely rapid growth in available out-of-school STEM activities compared to 2016 and an even larger increase in the number of organisations offering out-of-school STEM activities in Hong Kong.
STEM educators are eager to foster long term collaboration with each other, and with schools. At the same time, good working practice by schools, teachers, STEM educators and institutions that involves and engages local communities was discovered, showing the diversified modes of connection which could enhance the sustainability of STEM ecosystem.
We trust that this three-year study with its associated digital maps, provides a useful resource for schools, teachers, students, parents, STEM educators and education policy makers in Hong Kong.
In 2018, the Croucher Foundation conducted its third annual mapping exercise for the out-of-school STEM learning ecosystem in Hong Kong.
The study reveals a rich and vibrant ecosystem for out-of-school STEM in Hong Kong with over 3,000 discrete activities covering a very wide range of science disciplines. This third report indicates extremely rapid growth in available out-of-school STEM activities compared to 2016 and an even larger increase in the number of organisations offering out-of-school STEM activities in Hong Kong.
STEM educators are eager to foster long term collaboration with
DATE:
TEAM MEMBERS:
Siu Po LeeDavid FosterThe Croucher Foundation
This paper describes a follow‐up focus group study for the larger Exhibit Designs for Girls' Engagement (EDGE) project. Grounded in Culturally Responsive Pedagogical theory (CRP), the project aimed to understand the relationship between female responsive designs and girls' engagement at STEM exhibits. After developing a Female‐Responsive Design (FRD) Framework and conducting a large‐scale study to determine the most important design attributes for engaging girls at exhibits, the final step involved a qualitative investigation into those design attributes. Four focus groups with 22 girls aged 8
This three-year research and implementation project empowers middle school LatinX youth to employ their own assets and funds of knowledge to solve community problems through engineering. Only 7% of adults in the STEM job cluster are of Hispanic/Latino origin. There is a continuing need for filling engineering jobs in our current and future economy. This project will significantly broaden participation of LatinX youth in engineering activities at a critical point as they make career decisions. Design Squad Global LatinX expands on a tested model previously funded by NSF and shown to be successful. It will enable LatinX youth to view themselves as designers and engineers and to build from their strengths to expand their skills and participation in science and engineering. The project goals are to: 1) develop an innovative inclusive approach to informal engineering education for LatinX students that can broaden their engineering participation and that of other underrepresented groups, (2) to galvanize collaborations across diverse local, national, and international stakeholders to create a STEM learning ecosystem and (3) to advance knowledge about a STEM pedagogy that bridges personal-cultural identity and experience with engineering knowledge and skills. Project deliverables include a conceptual framework for a strength-based approach to engineering education for LatinX youth, a program model that is asset based, a collection of educational resources including a club guide for how to scaffold culturally responsive engineering challenge activities, an online training course for club leaders, and a mentoring strategy for university engineering students working with middle school youth. Project partners include the global education organization, iEARN, the Society of Women Engineers, and various University engineering programs.
The research study will employ an experimental study design to evaluate the impact on youth participating in the Design Squad LatinX programs. The key research questions are (1) Does participation increase students' positive perceptions of themselves and understanding of engineering and global perspectives? (2) To what extent do changes in understanding engineering vary by community (site) and by student characteristics (age, gender, ethnicity)? (3) Do educators and club leaders increase their positive perceptions of youths' funds of knowledge and their own understanding of engineering? and (4) Do university mentors increase their ability to lead informal engineering/STEM education with middle school youth? A sample from 72 local Design Squad LatinX clubs with an enrollment of 10-15 students will be drawn with half randomly assigned to the participant condition and half to the control condition. Methods used include pre and post surveys, implementation logs for checks on program implementation, site visits to carry out observations, focus groups with students and interviews with adult leaders. Data will be analyzed by estimating hierarchical linear models with observations. In addition, in-situ ethnographically-oriented observations as well as interviews at two sites will be used to develop qualitative case studies.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
DATE:
-
TEAM MEMBERS:
Mary Haggerty
resourceprojectProfessional Development, Conferences, and Networks
This project will advance evidence-based efforts to broaden informal STEM engagement via the 2019 Inclusive Science Communication (Inclusive SciComm) Symposium, to be held September 27-29, 2019, at the University of Rhode Island in Kingston, Rhode Island. Science communication, defined as any information exchange designed to engage targeted audiences in conversations or activities related to STEM topics, is a rapidly expanding area of research and practice with the potential to significantly increase public participation and sense of belonging in STEM fields. That said, there are few opportunities for its practitioners and scholars to convene around how to make their work both inclusive and equitable, which collectively acknowledge identity, cultural differences, and epistemologies as part of broadening participation. The 2019 symposium will address this gap through panels, workshops, and posters focused on three themes that represent critical and difficult aspects of inclusive science communication: (1) New Languages, Practices, Knowledge, and Research; (2) Changing Systems and Structures through Science Communication; and (3) Social Responsibility and Ethics. Within these themes, sessions will be organized to address major barriers of absence identified by participants in the 2018 Inclusive SciComm Symposium: skills, lessons learned, and knowledge gaps, especially with regard to facilitating difficult conversations across difference (critical dialogue). The symposium also will emphasize the need to integrate research and practice to advance inclusive, equitable, and intersectional approaches to science communication.
There is an urgent need to question assumptions and examine evidence regarding how science communicators and scholars approach efforts to broaden participation, but insufficient data exist on the inputs and outputs of inclusive and equitable practice. Critical dialogue about potentially uncomfortable topics such as privilege, power, or marginalization is an essential tool for inclusive practice and pedagogy. Finding from the 2018 Inclusive SciComm Symposium indicated that many educators and practitioners lack the language, skills, or confidence to initiate this type of dialogue. This project supports the knowledge-building component of the 2019 Inclusive SciComm Symposium to inform future science communication training, practice, and scholarship, by building on preliminary data collected during the 2018 symposium and responding to the need for more robust evaluation of science communication activities. Applying the Theory of Planned Behavior, the project will employ pre/post symposium surveys to investigate how 2019 symposium activities affected knowledge, attitudes, subjective norms, and efficacy (the variables of the Theory of Planned Behavior) of attendees with regard to critical dialogue. Focus groups at the symposium will be used to identify priority research areas related to inclusion, generally, and critical dialogue, specifically, that could advance inclusive science communication practice and beneficial outcomes. The project also will evaluate symposium impacts with regard to 1) attendees' opinions on utility of symposium components for advancing inclusive science communication and 2) how attendees' experience and response orientations inform their approaches to difficult science communication conversations. Qualitative data from the surveys and focus groups will be thematically coded using constant comparison.
This project will have strategic impact for inclusive science communication practice and, therefore, for informal learning and public engagement with STEM topics. Increasing awareness and effective implementation of critical dialogue by science communicators and trainers should enhance both for ethical engagement of traditionally under-represented and marginalized groups and should foster diverse types of public participation in societal debates about scientific issues. The outcomes of this research will benefit and link the complementary, but often siloed, fields of informal science learning and science communication. A final report will summarize research findings and offer specific next steps to advance inclusive science communication practice and research, especially with regard to fostering critical dialogue. The report will be posted on inclusivescicomm.org and distributed via a national network of partners working in informal science education, science communication, and public engagement.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
STEAM, the use of art as a context and tool for science education, is currently a hot topic in the science education field. In almost all instances of study and practice, it involves the use of science-themed or science-informed art in science education. As such, it does not take advantage of the majority of artistic output that does not have an obvious connection to science. The National Academies of Sciences, Engineering and Medicine recently called for more research to expand the "limited but promising" evidence that integrating arts and humanities with science education leads to better learning. The goal of this 2.5-day conference is to bring together representatives of both art and science groups to have a shared discussion around how non-scientific art can influence science education in theory, and how we can apply empirical results to the theory. For purposes of this conference, "non-science art" is defined as art that was not inspired by science. Conference attendees will include researchers (art and science education researchers) and practitioners (artists, art museum interpreters, and science educators). The conference will take place during the 2020 Black Creativity exhibition at the Museum of Science and Industry, Chicago. It is anticipated that by holding the conference at that time the audience for the conference and its impact will be informed by more diverse attendance.
The conference will be implemented starting with a pre-conference reading. Attendees will be sent a copy of the white paper from the Art as a Way of Knowing report for background reading and also asked to contribute to a Google Document that describes their various contexts. Each day of the conference will focus on a theme -- state of the field and possibilities and research -- and be comprised of large and small group interactions. Attendees will be invited from the ranks of practitioners, researchers and educators in the art and science education fields; several slots will be available for open (non-invited) participants. Key outcomes include: (a) a summary of all the research that has been conducted on using non-science art in science education, (b) starting points for building a theory on why non-science art can be used in science education; and (c) a list of specific research topics that would help inform, advance, and test the theory. In addition to assessing satisfaction with the conference, evaluation will also include a one-year post conference survey to investigate impact of participation in the conference.
This conference will generate products that will give guidance to both researchers and practitioners who want to use art in science education. These products include a white paper synthesizing the discussion and appendices that include raw transcripts and a bibliography of resources. Another product is a roadmap to create interventions that can be studied, which should lead to a stronger, more rigorous theory of practice about how art can be integrated into science education.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
There are several critical reasons to understand and support interest development in early childhood: (a) as a primary motivator of engagement and learning; (b) interest development in preschool predicts important learning outcomes and behaviors in early elementary school; and (c) early childhood interests motivate ongoing interest development. Thus, there is growing recognition that interest is not just important but fundamental to education and learning. Head Start on Engineering (HSE) is a multi-component, bilingual (Spanish/English), family-focused program designed to (1) foster long-term interest in the engineering design process for families with preschool children from low-income backgrounds and (2) support family development and kindergarten readiness goals. The HSE program, co-developed with the Head Start community, provides families with developmentally appropriate, story-based engineering design challenges for the home and then connects these to a system of strategically aligned Informal STEM Education (ISE) experiences and resources. This current project, HSE Systems, builds on a previous HSE Pathways project which (a) established that participating families develop persistent engineering-related interests; (b) highlighted the value that the Head Start community has for the program and partnership; and (c) generated a novel, systems perspective on early childhood interest development. The aim of HSE Systems is to develop and test a model of early childhood STEM engagement and advance knowledge of how the family as a system develops interest in STEM from preschool into kindergarten.
Through the Design Based Implementation Research (DBIR) process, the team will iteratively refine and improve the HSE program and theory of change using ongoing feedback and data from staff, families, and partners. It is also designed to explore program impacts on family interest development over a longer period, as children enter kindergarten. The DBIR work will focus primarily on the program model questions, while the case study research will focus on the family interest questions, with both strands informing each other. The initial work is organized around a series of feedback and design-testing cycles to gather input from families and other stakeholders, update the program components and activities in collaboration with families and staff, and prepare for full implementation. During the next phase, the team will implement the full program model with six Head Start classrooms and track family experiences and interest development into kindergarten. During final implementation phase, the team will finish data collection, conduct retrospective analysis with all the data, and update the program model and theory of change.
This project will directly address the AISL program goals by broadening access to early childhood informal STEM education for low-income communities, with a focus on Spanish-speaking families, and building long-term skills and learning dispositions to support STEM learning inside and outside of school. Beyond the topic of engineering, HSE supports Head Start school readiness and child and family development goals, which are the foundation of lifelong success.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
Fostering greater inclusion in science creates benefits for both science and society. In this Innovations and Development project, the University of Utah will investigate how to sustain and scale the STEM Ambassador Program (STEMAP), begun in 2016 with AISL funding. STEMAP developed an innovative process to train scientists to engage members of the public, who cannot or do not gain access to science via conventional science education venues (such as museums, schools, zoos), by implementing activities in non-traditional settings. The 65 scientists trained by the initial STEMAP effort effectively engaged in over 45 settings including an affordable housing development, a youth residential treatment center, a state prison, a cooking class, a daycare facility, and several senior centers. The number of scientists applying to the program quickly exceeded STEMAP's capacity. Other institutions expressed interest in replicating the training. This project will explore strategies for scaling and sustaining public engagement training to support more scientists who can engage more people in more venues. Outcomes will serve to inform the broader implementation of STEMAP and the efforts of other public engagement programs, many of which face similar scaling and sustainability challenges.
Scaling and sustaining public engagement of science (PES) programs is a central challenge for many in the informal science learning community. This project will explore strategies to scale and sustain the STEM Ambassador Program. Research questions include: (1) How do different program formats increase or restrict program capacity and engagement outcomes? (2) What benefits accrue to scientists and their institutions by participating in public engagement in science activities that might serve as motivators to continue these activities? (3) Are funding and organizational models developed in business and other professional settings applicable to sustaining these programs? To address scalability, this project will explore the effectiveness of three dissemination formats: (1) the creation of a mentorship program for in-person trainees, (2) a train-the-trainer approach, and (3) online training with in-person mentorship. The project team will create an evaluation toolkit with participant surveys, rubrics for observers, and "on-the-spot" assessment tools developed under AISL Award 1811022 to assess the effectiveness of engagement activities delivered by trainees in each of the three formats. To address sustainability, the project team will document the values of public engagement training to both the participating scientists and their institutions via surveys and interviews. Consultants from the business sectors will create a PES Campus Council to explore possible financial, organizational, and leadership plans that will help sustain engagement efforts. Outcomes will be published in peer-reviewed journals and compiled into a dissemination framework to inform actions to scale and sustain STEMAP and other public engagement of science programs to engage more hard-to-reach audiences. Inverness Research will serve as the project's external evaluator.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.