This is the final evaluation report from RMC Research Corp. for the PES@LTERs project. Appendix includes instruments.
RMC Research designed evaluation activities to provide formative and summative feedback to Harvard Forest and the Hubbard Brook Research Foundation (Hubbard Brook) on their plan to embed public engagement with science (PES) into the cultures and practices of Long-Term Ecological Research Sites (LTERs) in the northeastern US. The purpose of this project was to build PES mechanisms into long-term ecosystem studies that create on-going, open exchanges between scientists and
Milwaukee has established itself as a leader in water management and technology, hosting a widely recognized cluster of industrial, governmental, nonprofit, and academic activity focused on freshwater. At the same time, Milwaukee faces a wide range of challenges with freshwater, some unique to the region and others common to cities throughout the country. These challenges include vulnerability to flooding and combined sewer overflows after heavy rainfall, biological and pharmaceutical contamination in surface water, lead in drinking water infrastructure, and inequity in access to beaches and other recreational water amenities. Like other cities, Milwaukee grapples with the challenges global climate change imposes on urban water systems, including changing patterns of precipitation and drought.
These problems are further complicated by Milwaukee's acute racial and economic residential segregation. With a population of approximately 595,000, embedded within a metropolitan area of over 1.5 million, Milwaukee remains one of the country's most segregated cities. There is increasing urgency to engage the public--and especially those who are most vulnerable to environmental impacts--more deeply in the stewardship of urban water and in the task of creating sustainable urban futures. The primary goal of this four-year project is to foster community-engaged learning and environmental stewardship by developing a framework that integrates art with Science, Technology, Engineering, and Mathematics (STEM) experiences along with geography, water management, and social science. Synergies between STEM learning and the arts suggest that collaborations among artists, scientists, and communities can open ways to bring informal learning about the science of sustainability to communities.
WaterMarks provides an artist generated conceptual framework developed by Mary Miss / City as Living Laboratory (CALL) to help people better understand their relationship to the water systems and infrastructure that support their lives. Project activities include artist/scientist/community member-led Walks, which are designed to engage intergenerational participants both from the neighborhoods and from across the city, in considering the conditions, characteristics, histories, and ecosystems of neighborhoods. Walks are expanded upon in Workshops with residents, local scientists/experts, and other stakeholders, and include exploring current water-related environmental challenges and proposing solutions. The Workshops draw on diverse perspectives, including lived experience, scientific knowledge, and policy expertise. Art projects created by local artists amplify community engagement with the topics, including programming for teens and young adults. Free Wi-Fi will be integrated into various Marker sites around the city providing access to online, self-guided learning opportunities exploring the water systems and issues facing surrounding neighborhoods. Current programming focuses primarily on Milwaukee's predominantly African American near North Side and the predominantly Latinx/Hispanic near South Side. Many neighborhoods in these sections are vulnerable to such problems as frequent flooding, lead contamination in drinking water, inequities in safety and maintenance of green space, and less access to Lake Michigan, the city's primary natural resource and recreational amenity.
The WaterMarks project advances informal STEM learning in at least two ways. First, while the WaterMarks project is designed to fit Milwaukee, the project includes the development of an Adaptable Model Guide. The Guide is designed so that other cities can modify and employ its inclusive structure, programming, and process of collaboration among artists, scientists, partner organizations, and residents to promote citywide civic engagement in urban sustainability through the combination of informal STEM learning and public art. The Guide will be developed by a Community-University Working Group (CULab) hosted by UW-Milwaukee's Center for Community-Based Learning, Leadership, and Research and made up of diverse community and campus-wide stakeholders. In addition to overseeing the Guide’s creation, CULab will conceptualize onboarding and mentorship strategies for new participants as well as a framework for the program’s expansion and sustainability.
Second, through evaluation and research, the project will build a theoretical model for the relationships among science learning, engagement with the arts, and the distinctive contexts of different neighborhoods within an urban social-ecological system. The evaluation team, COSI’s Center for Research and Evaluation, and led by Co-PI Donnelly Hayde, aims to conduct formative, summative, and process evaluation of the Watermarks project, with the additional goal of producing evaluative research findings that can contribute to the broader field of informal learning. Evaluation foci include: How does the implementation of WaterMarks support positive outcomes for the project’s communities and the development of an adaptable model for city-scale informal science learning about urban environments? 2. To what extent do the type and degree of outcome-related change experienced by participating community residents vary across and/or between project sites? What factors, if any, appear to be linked to these changes? 3. To what extent and in what ways do the activities of the WaterMarks projects appear to have in situ effects related to the experience of place at project sites?
The project’s research team led by PI Ryan Holifield and Co-PI Woonsup Choi, will investigate how visual artistic activities introduced by the programming team as part of the Walks (and potentially other engagement activities) interact with personal, sociocultural, and physical contexts to produce distinctive experiences and outcomes of informal science learning about urban water systems. The aim of the research will be to synthesize the results from the different WaterMarks sites into an analysis generalizable beyond specific neighborhoods and applicable to other cities. The project's research questions include: 1. How does participation in Walks focused on visual artistic activities affect outcomes and experiences of informal STEM learning about urban water systems? 2. How do outcomes and experiences of informal STEM learning vary across different urban water topics, participants from different demographic groups, and contrasting sociocultural and biophysical contexts?
This Innovations in Development project is led by the University of Wisconsin-Milwaukee (UWM), in collaboration with City as Living Laboratory (CALL) and the COSI Center for Research and Evaluation.
Creating science education that can contribute to cultivating just, culturally thriving, and sustainable worlds is an important issue of our time. Indigenous peoples have persistently been under-represented in science reproducing inequalities in a myriad of ways from educational attainment, participation in and contributing to innovations in foundational knowledge, to effective policy making that upholds and respects Indigenous sovereignty. The development of models of science education that attend to intersections of knowledge and development, socio-scientific decision-making and civic leadership, and the complexities and contradictions of these realities, is imperative. This five-year Innovations in Development project broadens participation and strengthens infrastructure and capacity for Indigenous learners to meet, adapt to, and lead change in relation to the socio-ecological challenges of the 21st century. The project engages multi-sited community-based design studies to develop and research the impacts of Indigenous informal field-based science education with three Indigenous leadership communities from the Pacific Northwest and the Great Lakes. This project will have broader impacts through model development, building infrastructure to transform the capacity of informal field-based science education, and will produce cutting edge foundational knowledge about pressing 21st century issues with a particular focus on Indigenous communities. The project increases Indigenous participation in research through 1) engagement of Indigenous community members as research assistants, 2) training of Indigenous graduate fellows, and post-doctoral fellows, and 3) supporting the careers of more junior Indigenous scholars.
This research seeks to identify key design features of an Indigenous field (land/water) based model of science education and to understand how learners’ and educators’ reasoning, deliberation, decision-making, and leadership about complex socio-ecological systems and community change evolve in such learning environments. The project also examines key aspects of co-design and partnership with Tribal communities and how these methods of co-production of new science enable new capacities for systems transformation. This multi-layered project is organized through 3 panels of studies including: Panel 1) community-based design experiments to develop and refine a model of Indigenous informal science education; Panel 2) co-design and implementation of professional learning programs for Indigenous informal science education; and Panel 3) foundational studies in cognition and learning with respect to socio-ecological systems thinking and the impact on learning and instructional practices. Of particular importance in this research is the rigorous development and articulation of effective pedagogical practices and orientations. More broadly, findings will have clear implications for theories of cognitive development, deliberation and environmental decision making and especially those pertaining to how knowledge is shaped by culture and experience.
This project is funded by the Advancing Informal STEM Learning (AISL) program.
DATE:
-
TEAM MEMBERS:
Filiberto Barajas-LopezAnna LeesMegan BangAnna LeesFiliberto Barajas-Lopez
HBCUs are critical to producing a diverse and inclusive workforce as they graduate a disproportionate number of African American future STEM workers and STEM leaders. Although the National Science Foundation is fully committed to diversity and inclusion, there has been little research to determine why Historically Black Colleges and Universities are not fully participating in the NSF STEM educational research opportunities. The project will investigate the challenges, needs and support for Historically Black Colleges and Universities (HBCUs) to succeed in applying for educational research support from the National Science Foundation (NSF). Participants will be recruited from 96 HBCUs that are eligible to apply for such funding and will include the wide range of college and university administration and faculty that are involved in the preparation of research projects and related applications for research funding. The investigation will focus primarily on the Division of Research on Learning in Informal and Formal Settings (DRL) within NSF. The investigation will: 1) determine the submission rate and funding success rate of HBCUs within the DRL funding mechanisms; 2) determine why a greater proportion of HBCUs are not successful in their applications of research or do not apply; and 3) determine what factors, such as institutional support, research expertise, and professional development, could lead to a larger number of research proposals from HBCUs and greater success in obtaining funding. The project has the potential to have significant influence on the national educational and research agenda by providing empirical findings on the best approach to support and encourage HBCU participation in DRL educational research funding programs.
This exploratory research project will investigate what changes and/or supports would contribute to significantly increasing the number of applications and successful grant awards for STEM educational research project proposed by HBCUs. The project has the following research questions: (1) What factors discourage participation of HBCUs in the DRL funding mechanisms and what are the best practices to encourage participation? (2) What approaches have been successful for HBCUs to obtain DRL funding? (3) What dynamic capabilities are necessary for HBCU researchers to successfully submit STEM proposals to NSF? (4) What changes would be helpful to reduce or eliminate any barriers for HBCU applications for DRL educational research funding and what supports, such as professional development, would contribute to greater success in obtaining funding? Participants will be recruited from the 96 eligible HBCUs and will include both individuals from within the administration (e.g., Office Sponsored Programs, Deans, VP, etc.) as well as from within the faculty. The research will collect variety of quantitative and qualitative data designed to support a comprehensive analysis of factors addressing the research questions. The project will develop research findings and recommendations that are relevant to faculty, administrators, and policymakers for improving HBCU participation in research funding opportunities. Results of project research will be widely disseminated to HBCUs and other Minority Serving Institutions (MSIs) through a project website, peer reviewed journals, newsletters, and conference presentations.
This project is funded by the Innovative Technology Experiences for Students and Teachers (ITEST), the Advancing Informal STEM Learning (AISL), and the Discovery Research PreK-12 (DRK-12) programs. These programs which supports projects that build understandings of practices, program elements, contexts and processes contributing to increasing students' and general public knowledge and interest in science, technology, engineering, and mathematics (STEM).
DATE:
-
TEAM MEMBERS:
Cynthia TrawickJohn HaynesTriscia HendricksonTerry Mills
This Innovations in Development project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to (a) advance new approaches to and evidence-based understanding of the design and development of STEM learning in informal environments; (b) provide multiple pathways for broadening access to and engagement in STEM learning experiences; (c) advance innovative research on and assessment of STEM learning in informal environments; and (d) engage the public of all ages in learning STEM in informal environments.
Increasing greater diversity, equity, and inclusion in science not only presents a social justice goal, but is also vital to the financial and social success of the nation. The stereotype of the older white male scientist has obscured the contributions of women and people of color. This project seeks to remedy these perceptions which are barriers to entry into STEM fields. The project will create a large-scale hub for STEM themed video content on YouTube and other social media platforms, featuring 100+ original STEM videos produced by PBS partners. This hub and accompanying research seeks to identify the characteristics of online STEM content that attract (or fail to attract) underrepresented groups, specifically Black and Hispanic communities as well as women of all races. The objectives of this project are to 1) provide a unified online science-themed hub, PBS Terra, on YouTube and other platforms for hosting, sharing, and distributing digital STEM series from diverse producers from across the PBS system; 2) conduct surveys and focus groups to examine and understand the needs and expectations of women, Black and Hispanic communities and their consumption of STEM video content online and 3) test hypotheses about the communicative strategies of STEM videos that feature Black and Hispanic female scientists. Project collaborators include PBS, researchers at the University of Utah and the University of Georgia, and consultants and advisors with expertise in broadening participation and inclusion in STEM.
Little is known about how or why adult Americans seek science content on YouTube, especially the motivations of adults from underrepresented minorities and females. The key research questions in this project are: 1) Why do Black and Hispanic audiences and women of all races seek science video content online? 2) How does showing Black and Hispanic female scientists in science video content on YouTube impact viewers’ identification with and sense of belonging in STEM? 3) How does the use of humor by Black and Hispanic scientists in YouTube science content affect viewers’ perceptions of the communicator and their engagement with STEM content? 4) How does the appearance and manner of dress of Black and Hispanic scientists in YouTube science content affect viewers’ perceptions in the aforementioned areas? A nationally representative baseline survey will be conducted. A probability sample of 2000 respondents will be obtained including oversampling of Black and Hispanic audiences. To complement findings from the survey, focus groups will be conducted in eight different regions of the country to learn why these targeted audiences do or do not seek science content on YouTube and what motivates them to share the content with their social media network. In addition, an experiment embedded in an online survey will test the hypothesis that greater on-screen representation of women and scientists of color will broaden existing perceptions about scientists. The experiment will consist of a 3 (scientist’s race: Black/Hispanic/White) × 2 (science issue: controversial/non-controversial) × 2 (style: casual/professional) between-subjects design. Survey participants will be randomly assigned to the experimental conditions. These factors (science issue and host appearance) can be altered by content producers to better reach and engage the targeted audiences. The project not only investigates theoretical questions at the intersection of STEM stereotypes and race, but findings related to these experimental conditions will offer practical insight into strategies that can be used by science communication practitioners.
DATE:
-
TEAM MEMBERS:
Adam DylewskiSara YeoMichael Cacciatore
This collaborative research project between KQED, a public media organization serving the San Francisco Bay Area, Texas Tech University and Rockman et al conducted research to study how best to provide effective COVID-19 science news and social media content for young adult audiences.
To start the work, four “Knowledge Gap” studies – Twitter Misinformation, Mask Wearing Messaging, Germ Knowledge and Conceptual Mapping – as well as social media testing were conducted to address our research question: How could COVID-19 coverage be designed to best inform, engage and educate millennials and
On October 1, 2015, Oregon Public Broadcasting (OPB) was awarded a 3‑year grant of 2.7 million dollars from the National Science Foundation (NSF) to fund the project Hacking Your Mind (award number 1515520). A major public and social media project, Hacking Your Mind (HYM) planned to engage Americans with the new discoveries being made in the social, behavioral, and economic sciences and the remarkable insights these discoveries offer into how individuals make numerous daily decisions and judgments, as well as the broader impact of this highly personal phenomenon on nearly every aspect of
This NOVA multiplatform media initiative consisted of a 2-hour nationally broadcast PBS documentary, Polar Extremes; a 10-part original digital series, Antarctic Extremes; an interactive game, Polar Lab; accompanying polar-themed digital shorts, radio stories, text reporting, and social media content; a collection of educational resources on PBS LearningMedia; and community screening events and virtual field trips for science classrooms. Across multiple media platforms the project’s video content had nearly 13 million views.
The research explored the potential for informal STEM learning
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. This two-year Pilots and Feasibility project seeks to build knowledge and strategic impact in the informal STEM learning field by studying how and why science-education-art collaborations yield impactful informal STEM learning experiences. By design and implementing interactive and participatory experiences for adult audiences, this project will build knowledge about how to help communities learn about environmental science and apply scientific knowledge to environmental decision-making in their lives.
The project's two overarching research questions are: (1) What are the essential elements of collaboration among scientists, educators, and artists that support learning about adaptation in a changing environment? (2) In what ways do designed, participatory, informal science learning experiences support participant learning? This pilot project will: a) develop methods for facilitating and assessing collaboration among scientists, educators, and artists; b) pilot and refine approaches for engaging scientists, educators, and artists with community members for high quality participatory experiences focus on learning about adaptation to environmental change in informal learning settings; c) pilot and refine methods to measure the outcomes for community participants on knowledge about environmental change and its application to problems in their everyday lives. This project is innovative in bridging a diverse body of scholarship in order to study the process of collaboration and the specific ways interdisciplinary collaborations foster learning. Because informal STEM learning settings often combine the work of multiple disciplines, examining the process and outcomes of collaborative, participatory STEM learning has the potential to deliver widely applicable guidance for achieving more impactful educational outcomes.
The proposed project will broaden participation by engaging adult members of environmentally vulnerable communities in participatory STEM activities and will improving individual and community well-being by delivering tools for future decision-making. The collaborative project will build valuable partnerships and capacity between disparate sectors of society, allowing co-learning and co-production of knowledge. Results will be published in scholarly journals as well as shared with community participants.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
Amy LesenSameer HonwadAma RoganCalvin Mackie
Across the country, educators and mentors in informal settings have provided youth with opportunities to persist and thrive along STEM (science, technology, engineering, and mathematics) trajectories. Unfortunately, the current COVID-19 pandemic may limit youths' opportunities to continue pursuing STEM pathways by removing their access to important resources, such as mentors in science labs and other STEM learning spaces. The impact of the pandemic may be especially harmful for youth from underrepresented groups whose families have been disproportionately affected by this crisis. The purpose of this project is to study the impact of COVID-19 on the STEM trajectories of teenagers from underrepresented groups who had previously worked with scientists as mentors before the pandemic. A survey, which will be administered to hundreds of teenagers, will identify the supports that they used to successfully navigate disruptions as they continued to pursue college and career pathways in STEM. For teenagers who decided not to actively pursue STEM pathways during the pandemic, this survey will illuminate how loss of supports and new challenges discouraged their active pursuit of STEM. By identifying supports and challenges that encouraged or discouraged STEM pursuits, this project will advance knowledge on how informal learning programs can support underrepresented teenagers in persisting along STEM pathways during national crises and emergencies.
The American Museum of Natural History will conduct mixed-methods research to explore the impact of the pandemic on the STEM trajectories of underrepresented teenagers. They will administer surveys to 560 teenagers from underrepresented groups who previously participated in the New York City Science Research Mentoring Consortium. The survey will ascertain whether and how the youth found access to new supports or lost access to former supports; whether they face new challenges and how they have responded to those challenges; and whether and how their feelings about pursuing STEM in college and beyond have changed. Teenagers who complete the survey will identify adults in their lives who have been important in supporting their pursuits in STEM, and two adults per teenager will also be invited to complete surveys on perceived disruptions. Latent class analyses, cognitive interviews, and consultation with youth and survey experts will be used to establish survey validity, while the survey results will be analyzed via two-mode social network analysis. Additionally, 16 teenagers will participate in in-depth interviews regarding the impact of the pandemic on their STEM trajectories. Findings and implications for practice will be disseminated widely to researchers, educators, and mentors in informal science education. These findings will help stakeholders to provide better supports for underrepresented youth during the current pandemic and during future national emergencies. This project is funded by the Advanced Informal STEM Learning (AISL) program. As part of its overall strategy to enhance learning in informal environments, the AISL program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This RAPID award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
Preeti GuptaTimothy PodkulKaren Hammerness