This is the final evaluation report from RMC Research Corp. for the PES@LTERs project. Appendix includes instruments.
RMC Research designed evaluation activities to provide formative and summative feedback to Harvard Forest and the Hubbard Brook Research Foundation (Hubbard Brook) on their plan to embed public engagement with science (PES) into the cultures and practices of Long-Term Ecological Research Sites (LTERs) in the northeastern US. The purpose of this project was to build PES mechanisms into long-term ecosystem studies that create on-going, open exchanges between scientists and
Milwaukee has established itself as a leader in water management and technology, hosting a widely recognized cluster of industrial, governmental, nonprofit, and academic activity focused on freshwater. At the same time, Milwaukee faces a wide range of challenges with freshwater, some unique to the region and others common to cities throughout the country. These challenges include vulnerability to flooding and combined sewer overflows after heavy rainfall, biological and pharmaceutical contamination in surface water, lead in drinking water infrastructure, and inequity in access to beaches and other recreational water amenities. Like other cities, Milwaukee grapples with the challenges global climate change imposes on urban water systems, including changing patterns of precipitation and drought.
These problems are further complicated by Milwaukee's acute racial and economic residential segregation. With a population of approximately 595,000, embedded within a metropolitan area of over 1.5 million, Milwaukee remains one of the country's most segregated cities. There is increasing urgency to engage the public--and especially those who are most vulnerable to environmental impacts--more deeply in the stewardship of urban water and in the task of creating sustainable urban futures. The primary goal of this four-year project is to foster community-engaged learning and environmental stewardship by developing a framework that integrates art with Science, Technology, Engineering, and Mathematics (STEM) experiences along with geography, water management, and social science. Synergies between STEM learning and the arts suggest that collaborations among artists, scientists, and communities can open ways to bring informal learning about the science of sustainability to communities.
WaterMarks provides an artist generated conceptual framework developed by Mary Miss / City as Living Laboratory (CALL) to help people better understand their relationship to the water systems and infrastructure that support their lives. Project activities include artist/scientist/community member-led Walks, which are designed to engage intergenerational participants both from the neighborhoods and from across the city, in considering the conditions, characteristics, histories, and ecosystems of neighborhoods. Walks are expanded upon in Workshops with residents, local scientists/experts, and other stakeholders, and include exploring current water-related environmental challenges and proposing solutions. The Workshops draw on diverse perspectives, including lived experience, scientific knowledge, and policy expertise. Art projects created by local artists amplify community engagement with the topics, including programming for teens and young adults. Free Wi-Fi will be integrated into various Marker sites around the city providing access to online, self-guided learning opportunities exploring the water systems and issues facing surrounding neighborhoods. Current programming focuses primarily on Milwaukee's predominantly African American near North Side and the predominantly Latinx/Hispanic near South Side. Many neighborhoods in these sections are vulnerable to such problems as frequent flooding, lead contamination in drinking water, inequities in safety and maintenance of green space, and less access to Lake Michigan, the city's primary natural resource and recreational amenity.
The WaterMarks project advances informal STEM learning in at least two ways. First, while the WaterMarks project is designed to fit Milwaukee, the project includes the development of an Adaptable Model Guide. The Guide is designed so that other cities can modify and employ its inclusive structure, programming, and process of collaboration among artists, scientists, partner organizations, and residents to promote citywide civic engagement in urban sustainability through the combination of informal STEM learning and public art. The Guide will be developed by a Community-University Working Group (CULab) hosted by UW-Milwaukee's Center for Community-Based Learning, Leadership, and Research and made up of diverse community and campus-wide stakeholders. In addition to overseeing the Guide’s creation, CULab will conceptualize onboarding and mentorship strategies for new participants as well as a framework for the program’s expansion and sustainability.
Second, through evaluation and research, the project will build a theoretical model for the relationships among science learning, engagement with the arts, and the distinctive contexts of different neighborhoods within an urban social-ecological system. The evaluation team, COSI’s Center for Research and Evaluation, and led by Co-PI Donnelly Hayde, aims to conduct formative, summative, and process evaluation of the Watermarks project, with the additional goal of producing evaluative research findings that can contribute to the broader field of informal learning. Evaluation foci include: How does the implementation of WaterMarks support positive outcomes for the project’s communities and the development of an adaptable model for city-scale informal science learning about urban environments? 2. To what extent do the type and degree of outcome-related change experienced by participating community residents vary across and/or between project sites? What factors, if any, appear to be linked to these changes? 3. To what extent and in what ways do the activities of the WaterMarks projects appear to have in situ effects related to the experience of place at project sites?
The project’s research team led by PI Ryan Holifield and Co-PI Woonsup Choi, will investigate how visual artistic activities introduced by the programming team as part of the Walks (and potentially other engagement activities) interact with personal, sociocultural, and physical contexts to produce distinctive experiences and outcomes of informal science learning about urban water systems. The aim of the research will be to synthesize the results from the different WaterMarks sites into an analysis generalizable beyond specific neighborhoods and applicable to other cities. The project's research questions include: 1. How does participation in Walks focused on visual artistic activities affect outcomes and experiences of informal STEM learning about urban water systems? 2. How do outcomes and experiences of informal STEM learning vary across different urban water topics, participants from different demographic groups, and contrasting sociocultural and biophysical contexts?
This Innovations in Development project is led by the University of Wisconsin-Milwaukee (UWM), in collaboration with City as Living Laboratory (CALL) and the COSI Center for Research and Evaluation.
The call for more science, technology, engineering, and mathematics (STEM) education taking place in informal settings has the potential to shape future generations, drive new innovations and expand opportunities. Yet, its power remains to be fully realized in many communities of color. However, research has shown that using creative embodied activities to explore science phenomena is a promising approach to supporting understanding and engagement, particularly for youth who have experienced marginalization. Prior pilot work by the principal investigator found that authentic inquiries into science through embodied learning approaches can provide rich opportunities for sense-making through kinesthetic experience, embodied imagining, and the representation of physics concepts for Black and Latinx teens when learning approaches focused on dance and dance-making. This Research in Service to Practice project builds on prior work to better understand the unique opportunities for learning, engagement, and identity development for these youth when physics is explored in the context of the Embodied Physics Learning Lab Model. The model is conceptualized as a set of components that (1) allow youth to experience and utilize their intersectional identities; (2) impact engagement with physics ideas, concepts and phenomena; and (3) lead to the development of physics knowledge and other skills. The project aims to contribute to more expansive definitions of physics and physics learning in informal spaces. While the study focuses primarily on Black and Latinx youth, the methods and discoveries have the potential to impact the teaching of physics for a much broader audience including middle- and high-school children, adults who may have been turned off to physics at an earlier age, and undergraduate physical science majors who are struggling with difficult concepts. This project is funded by the Advancing Informal STEM Learning (AISL) program which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
The research is grounded in sociocultural perspectives on learning and identity, embodied interaction and enactive cognition, and responsive design. The design is also informed by the notion of “ArtScience” which highlights commonalities between the thinking and making practices used by artists and by scientists and builds on the theoretical philosophy that all things can be understood through art or through science but integrating the two lenses allows for more complete understandings. Research will investigate the relationship between embodied learning approaches, design principles, and structures of the Embodied Physics Learning Lab model using the lenses of physics, dance, and integrated ArtScience to better understand the model. The project employs design-based research to address two overarching research questions: (1) What unique opportunities for learning, engagement, and identity development for Black and Latinx youth occur when physics is explored in the context of the Embodied Physics Learning Lab Model? and (2) How do variations in site demographics and site implementation influence the impact and scalability of the Learning Lab model? Further, the inquiry will consider (a) how youth experience and utilize their intersectional various identities in the context of the activities, structures, and essential elements of the embodied physics learning lab; (b) how youth's level of physics engagement changes depending on which embodied learning approaches and essential element structures are used; (c) the physics knowledge and other skills youth attain through the set of activities; and (d) how, if at all, the embodied learning approaches engage youth in thinking about their own agency as STEM doers. An interdisciplinary team of researchers, choreographers, and youth along with community organizations will co-design and implement project activities across four sites. Approximately 200 high school youth will be engaged; 24 will have the role of Teen Thought Partner. Through three iterative design cycles of implementation, the project will refine the model to investigate which elements most affect successful implementation and to identify the conditions necessary for scale-up. Data will be collected in the form of video, field notes, pre- and post- interviews, pre- and post- surveys, and artifacts created by the youth. Analyses will include a combination of interaction analysis, descriptive data analysis, and movement analysis. In addition to the research findings and explication of the affordances and constraints of the model, the project will also create a curricular resource, including narrative text and video demonstrations of physics concepts led by the teen thought partners, video case training modules, and assessment tools.
DATE:
-
TEAM MEMBERS:
Folashade Cromwell SolomonDionne Champion
Creating science education that can contribute to cultivating just, culturally thriving, and sustainable worlds is an important issue of our time. Indigenous peoples have persistently been under-represented in science reproducing inequalities in a myriad of ways from educational attainment, participation in and contributing to innovations in foundational knowledge, to effective policy making that upholds and respects Indigenous sovereignty. The development of models of science education that attend to intersections of knowledge and development, socio-scientific decision-making and civic leadership, and the complexities and contradictions of these realities, is imperative. This five-year Innovations in Development project broadens participation and strengthens infrastructure and capacity for Indigenous learners to meet, adapt to, and lead change in relation to the socio-ecological challenges of the 21st century. The project engages multi-sited community-based design studies to develop and research the impacts of Indigenous informal field-based science education with three Indigenous leadership communities from the Pacific Northwest and the Great Lakes. This project will have broader impacts through model development, building infrastructure to transform the capacity of informal field-based science education, and will produce cutting edge foundational knowledge about pressing 21st century issues with a particular focus on Indigenous communities. The project increases Indigenous participation in research through 1) engagement of Indigenous community members as research assistants, 2) training of Indigenous graduate fellows, and post-doctoral fellows, and 3) supporting the careers of more junior Indigenous scholars.
This research seeks to identify key design features of an Indigenous field (land/water) based model of science education and to understand how learners’ and educators’ reasoning, deliberation, decision-making, and leadership about complex socio-ecological systems and community change evolve in such learning environments. The project also examines key aspects of co-design and partnership with Tribal communities and how these methods of co-production of new science enable new capacities for systems transformation. This multi-layered project is organized through 3 panels of studies including: Panel 1) community-based design experiments to develop and refine a model of Indigenous informal science education; Panel 2) co-design and implementation of professional learning programs for Indigenous informal science education; and Panel 3) foundational studies in cognition and learning with respect to socio-ecological systems thinking and the impact on learning and instructional practices. Of particular importance in this research is the rigorous development and articulation of effective pedagogical practices and orientations. More broadly, findings will have clear implications for theories of cognitive development, deliberation and environmental decision making and especially those pertaining to how knowledge is shaped by culture and experience.
This project is funded by the Advancing Informal STEM Learning (AISL) program.
DATE:
-
TEAM MEMBERS:
Filiberto Barajas-LopezAnna LeesMegan BangAnna LeesFiliberto Barajas-Lopez
This brief focuses on a participatory study with the high school program of the Kitty Andersen Youth Science Center (KAYSC) at the Science Museum of Minnesota (SMM). Young people are organized into teams of up to 20 youth with an adult practitioner who delivers programming based on a STEM content area. Their activities and project-based learning are based in both STEM and social justice, coined in the KAYSC as “STEM Justice.”
As part of our study, we wanted to understand youth and adult needs that exist in an informal STEM education program that weaves equity into its core. This brief
This brief shares youth development insights from a museum-based, informal science learning program that uses STEM as a tool for social justice. Key to the success of this program were young people and adults feeling at home in a welcoming, diverse, and inclusive space; activities that focused on connecting and relationships; a holistically supportive space that attended to family and personal needs; shared norms for conversation and expectations; and science content grounded in young people’s lives, experiences, and communities as well as work with community members.
These needs were
Astronomy has been an inherently visual area of science for millenia, yet a majority of its significant discoveries take place in wavelengths beyond human vision. There are many people, including those with low or no vision, who cannot participate fully in such discoveries if visual media is the primary communication mechanism. Numerous efforts have worked to address equity of accessibility to such knowledge sharing, such as through the creation of three-dimensional (3D) printed data sets. This paper describes progress made through technological and programmatic developments in tactile 3D
STEMroller events disrupt stereotypes surrounding STEM professionals within a neglected space in science communication; a sports hall. Roller derby inspired STEMroller, both the do-it-yourself culture and creating a space for women and genderqueer people to be themselves. Over 100 female and non-binary STEM professionals volunteered to put this event together for students aged 11–19. STEMroller includes networking with people from over 30 science, technology engineering and mathematics industries, watching roller derby and trying it out — albeit in socks not on wheels. STEMroller uses a pool
The U.S. Fish and Wildlife Service estimates that over 41 million people connect to nature through birding. Learning about birds in their natural environments offers opportunities for informal engagement in STEM by a broad range of individuals and groups. Birders often engage in scientific data gathering and analyses, geolocation and remote sensing, and phenology. They also become aware of ecological changes in bird habitats and migratory patterns due to rising temperatures and climate-related events like sea level rise, droughts, fires, and extreme weather. As such, the birding community is an ideal network to better understand and communicate the impacts of climatological changes on bird populations to the public. With this Innovations in Development project, the National Audubon Society will develop a new avian-focused, conservation and climate science community science curriculum for its Nature Centers, and test the effectiveness of the curriculum in educating the public about avian-focused conservation and climatological changes through guided nature experiences. Birding can serve as a pivotal entrée for young people into STEM fields and careers. Through its programs and partnerships, Audubon will leverage its national network to ensure that through this project a more diverse group of voices, particularly young adults and young adults of color, become involved in asking critical questions and developing solutions to address important environmental issues of the future. If successful, the broader impacts of this project on capacity building and public engagement could be far-reaching and long-lasting.
Over the three-year project duration, Audubon will bring educators from its nationwide network of thirty-four Nature Centers (including urban, suburban, and rural sites), together with over 510 young adults (ages 18-25) from its network of college campus chapters. An evidence-based curriculum and community science activities will be created and tested, relying heavily on a team of experts in ornithology, climate science research, STEM curriculum design, diversity, and informal science education. College students will advise on the design of content and activities to effectively interest and engage young adults. These students will be recruited from the new Audubon Campus Chapters Program, which includes 111 college and university campuses, among them, 19 Historically Black Colleges and Universities (HBCUs) and other Minority Serving Institutions (MSIs). The target population will be surveyed to also understand their current and likely participation in guided nature experiences and knowledge base in climate science. Current best practices in guided nature experiences will be gathered from across the Audubon network. The implementation efforts will result in a national STEM model, with train-the-trainer guides and workshops for informal science educators and public engagement opportunities focused on improving the state and condition of avian habitats and communities through climate science research. An external evaluation will be conducted and will include data collection methods such as retrospective pre and post surveys, semi-structured interviews, focus groups, and an embedded assessment to determine impact. The findings will be used to iteratively refine the evidence-based curriculum and measure STEM learning outcomes for the guided nature experience participants. The evaluation will address four areas: (1) fidelity of program implementation to promote accountability; (2) formative evaluation to understand needs and interests of young adults (ages 18-25), and subsequently inform program design; (3) outcomes for Center educators, to inform iterative improvement; and (4) outcomes for program participants, to contribute to the growing knowledge base on effective practices for STEM learning in informal settings.
This Innovations in Development project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
Loren SmithMark ScallionHeather Starck
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. This two-year Pilots and Feasibility project seeks to build knowledge and strategic impact in the informal STEM learning field by studying how and why science-education-art collaborations yield impactful informal STEM learning experiences. By design and implementing interactive and participatory experiences for adult audiences, this project will build knowledge about how to help communities learn about environmental science and apply scientific knowledge to environmental decision-making in their lives.
The project's two overarching research questions are: (1) What are the essential elements of collaboration among scientists, educators, and artists that support learning about adaptation in a changing environment? (2) In what ways do designed, participatory, informal science learning experiences support participant learning? This pilot project will: a) develop methods for facilitating and assessing collaboration among scientists, educators, and artists; b) pilot and refine approaches for engaging scientists, educators, and artists with community members for high quality participatory experiences focus on learning about adaptation to environmental change in informal learning settings; c) pilot and refine methods to measure the outcomes for community participants on knowledge about environmental change and its application to problems in their everyday lives. This project is innovative in bridging a diverse body of scholarship in order to study the process of collaboration and the specific ways interdisciplinary collaborations foster learning. Because informal STEM learning settings often combine the work of multiple disciplines, examining the process and outcomes of collaborative, participatory STEM learning has the potential to deliver widely applicable guidance for achieving more impactful educational outcomes.
The proposed project will broaden participation by engaging adult members of environmentally vulnerable communities in participatory STEM activities and will improving individual and community well-being by delivering tools for future decision-making. The collaborative project will build valuable partnerships and capacity between disparate sectors of society, allowing co-learning and co-production of knowledge. Results will be published in scholarly journals as well as shared with community participants.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
Amy LesenSameer HonwadAma RoganCalvin Mackie
Over the past two decades the prevalence of Autism Spectrum Disorder (ASD) has nearly tripled and yet there is much to learn about serving this audience well. After high school exit, most are left to navigate the world without appropriate support or the requisite skills necessary for success. Educators working in informal science institutions (ISI) can better promote both social interaction and engagement in STEM education for individuals with ASD. A learning environment in which the learner chooses content aligned with their personal interests and where learning can be multifaceted (verbal, hands-on, fast or slow, social or solitary, directed or inquiry based, physical, etc.) is consistent with the central tenets of an evidence-based, outcome-driven approach for autism intervention. ISI educators have the desire but may not have sufficient and timely knowledge and skills to engage and support this audience. Currently, many are working at the local level to develop new programs and approaches for patrons with ASD, with little evaluation or research and not building on each other's work. The project will develop a rigorous customized professional learning experience designed to enhance capacity of ISIs broadly in ASD support techniques and strategies. The goal is to enable more inclusive opportunities for people with ASD based on current and emerging promising practices. The project's theory of action is that the ability of people with ASD to participate in traditional, mainstream experiences will improve their motivation to seek other similar opportunities, build interpersonal skills critical to successful interaction in society, formal education, and careers. This, in turn, will help individuals with ASD gain the skills and confidence needed to pursue STEM academically and professionally. The project is a collaboration between the Institute for Learning Innovation, the SciTech Institute, and the Southwest Autism Research & Resource Center (SARRC). This project is funded by the Advanced Informal STEM Learning program which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
The pilot's main research questions are: (1) To what degree does the professional learning program support the learning outcomes in knowledge, awareness, interest, skills, attitude, and behavior change in informal science education providers? and (2) What features of the program do educators consider most effective for improving their ability to serve this audience? Four Arizona ISIs will participate in a research-based design study; their staff will also comprise the founding members of a Community of Practice aimed at sharing promising practices and promoting broader engagement among the informal science education community. The professional development (PD) will be provided by SARRC. New formative evaluation skills will support ongoing innovation and build participant capacity. Leveraging this training, the ISIs will create and test new approaches and programs, apply new skills in formative evaluation, and develop internal workplace programs to create cultures of ASD understanding. A pilot research study will recruit 20 diverse individuals with ASD who will visit each institution prior to and after the PD for staff. The research will measure the degree to which the PD impacts attendee experience as well as assess the science learning that occurred because of their visit. This project will advance collaboration between ASD experts and ISI educators to iteratively develop effective museum learning strategies. Other goals of this work are to provide important insights into (a) the current state of accessibility programs in ISI venues nationally, (b) how PD can be leveraged to help institutions reach true inclusion, and (c) initial evidenced-based approaches for inclusion of individuals on the Autism spectrum in mainstream informal environments. In addition to the research findings, deliverables include an ASD PD model, national inventory of current practices and programs that support ASD learning and participation, and the establishment of a Community of Practice.
This Innovations in Development award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
Free-choice and interest-driven learning activities are a highly significant source of STEM (science, technology, engineering, and mathematics) learning for adults through their lifespans. Gardening is one such activity that is widespread across communities with one in every fifty American adults reporting an interest in gardening/plants and who associate that interest/hobby with science. While the terms interest and hobby are related, the latter refers to something one actively does, not just thinks about doing. Adults who seek out learning and participation opportunities in highly visible community spaces (e.g., gardening clubs, science centers, botanical gardens) are likely to be White and well-educated. Further understanding is needed of when and how community members from other demographic groups access different resources (people, organizations, and places) for information and opportunities, and what influences them to do so. This Pilot and Feasibility Study will explore informal learning networks in Alameda County, California, specifically around gardeners and gardening. Researchers will use surveys, focus groups, and program observations to gather data on how those who pursue self-directed scientific learning about gardening access information. Of interest is how the differential access to and pursuit of information occurs among diverse community members, especially those outside of more established Master Gardener and other organized gardening programs. This research will: 1) contribute to understanding of the resources that interest-driven adult STEM learners access, describing the barriers they perceive and how/if the accessed resources differ by gender, race, or socioeconomics; 2) determine the feasibility of a sampling approach to gather data from individuals in demographic groups who may not have been reached in prior research efforts; and 3) generate insights for informal science education practitioners and researchers about how to better support diverse interest-driven STEM learners. This project is funded by the Advancing Informal STEM Learning (AISL) program which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
It is hypothesized that there is a large percentage of individuals from non-dominant populations engaged in free-choice STEM learning; past efforts have likely under-counted the participation of individuals from these communities. To achieve the research aims the research team will utilize respondent-driven sampling, engaging gardening hobbyists' social contacts to recruit participants, and collect data from diverse (gender, race, socioeconomics) urban gardeners who may be differently connected to STEM learning ecosystem resources/organizations than their highly visible peers. This approach will be used to investigate the behaviors, perceptions, and outcomes related to STEM learning such as development of self-efficacy and science identity. Focus groups will provide context for themes that arise in the survey data and clarify hobbyists' participation preferences and experiences. In situ observations of learning environments comprise the third mechanism for collecting data. All three data sources will support triangulation of results and contribute to the findings. Key outcomes of this project will be to determine if the target population has been reached through the sampling approach, to identify methodological guidelines for sampling with an intent to reach those from populations under-represented in STEM-related free-choice activities, and to clarify which network variables are most useful to study. This research lays the foundation for future work. It is anticipated that the approach developed and tested in this research may be adapted by others in the future and will have the potential to serve as a model for community-based organizations and researchers interested in studying the learning ecosystems of previously hidden populations of participants, including how these individuals perceive and access resources to support their STEM learning.
This Pilots and Feasibility Studies award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.