This is the final evaluation report from RMC Research Corp. for the PES@LTERs project. Appendix includes instruments.
RMC Research designed evaluation activities to provide formative and summative feedback to Harvard Forest and the Hubbard Brook Research Foundation (Hubbard Brook) on their plan to embed public engagement with science (PES) into the cultures and practices of Long-Term Ecological Research Sites (LTERs) in the northeastern US. The purpose of this project was to build PES mechanisms into long-term ecosystem studies that create on-going, open exchanges between scientists and
Milwaukee has established itself as a leader in water management and technology, hosting a widely recognized cluster of industrial, governmental, nonprofit, and academic activity focused on freshwater. At the same time, Milwaukee faces a wide range of challenges with freshwater, some unique to the region and others common to cities throughout the country. These challenges include vulnerability to flooding and combined sewer overflows after heavy rainfall, biological and pharmaceutical contamination in surface water, lead in drinking water infrastructure, and inequity in access to beaches and other recreational water amenities. Like other cities, Milwaukee grapples with the challenges global climate change imposes on urban water systems, including changing patterns of precipitation and drought.
These problems are further complicated by Milwaukee's acute racial and economic residential segregation. With a population of approximately 595,000, embedded within a metropolitan area of over 1.5 million, Milwaukee remains one of the country's most segregated cities. There is increasing urgency to engage the public--and especially those who are most vulnerable to environmental impacts--more deeply in the stewardship of urban water and in the task of creating sustainable urban futures. The primary goal of this four-year project is to foster community-engaged learning and environmental stewardship by developing a framework that integrates art with Science, Technology, Engineering, and Mathematics (STEM) experiences along with geography, water management, and social science. Synergies between STEM learning and the arts suggest that collaborations among artists, scientists, and communities can open ways to bring informal learning about the science of sustainability to communities.
WaterMarks provides an artist generated conceptual framework developed by Mary Miss / City as Living Laboratory (CALL) to help people better understand their relationship to the water systems and infrastructure that support their lives. Project activities include artist/scientist/community member-led Walks, which are designed to engage intergenerational participants both from the neighborhoods and from across the city, in considering the conditions, characteristics, histories, and ecosystems of neighborhoods. Walks are expanded upon in Workshops with residents, local scientists/experts, and other stakeholders, and include exploring current water-related environmental challenges and proposing solutions. The Workshops draw on diverse perspectives, including lived experience, scientific knowledge, and policy expertise. Art projects created by local artists amplify community engagement with the topics, including programming for teens and young adults. Free Wi-Fi will be integrated into various Marker sites around the city providing access to online, self-guided learning opportunities exploring the water systems and issues facing surrounding neighborhoods. Current programming focuses primarily on Milwaukee's predominantly African American near North Side and the predominantly Latinx/Hispanic near South Side. Many neighborhoods in these sections are vulnerable to such problems as frequent flooding, lead contamination in drinking water, inequities in safety and maintenance of green space, and less access to Lake Michigan, the city's primary natural resource and recreational amenity.
The WaterMarks project advances informal STEM learning in at least two ways. First, while the WaterMarks project is designed to fit Milwaukee, the project includes the development of an Adaptable Model Guide. The Guide is designed so that other cities can modify and employ its inclusive structure, programming, and process of collaboration among artists, scientists, partner organizations, and residents to promote citywide civic engagement in urban sustainability through the combination of informal STEM learning and public art. The Guide will be developed by a Community-University Working Group (CULab) hosted by UW-Milwaukee's Center for Community-Based Learning, Leadership, and Research and made up of diverse community and campus-wide stakeholders. In addition to overseeing the Guide’s creation, CULab will conceptualize onboarding and mentorship strategies for new participants as well as a framework for the program’s expansion and sustainability.
Second, through evaluation and research, the project will build a theoretical model for the relationships among science learning, engagement with the arts, and the distinctive contexts of different neighborhoods within an urban social-ecological system. The evaluation team, COSI’s Center for Research and Evaluation, and led by Co-PI Donnelly Hayde, aims to conduct formative, summative, and process evaluation of the Watermarks project, with the additional goal of producing evaluative research findings that can contribute to the broader field of informal learning. Evaluation foci include: How does the implementation of WaterMarks support positive outcomes for the project’s communities and the development of an adaptable model for city-scale informal science learning about urban environments? 2. To what extent do the type and degree of outcome-related change experienced by participating community residents vary across and/or between project sites? What factors, if any, appear to be linked to these changes? 3. To what extent and in what ways do the activities of the WaterMarks projects appear to have in situ effects related to the experience of place at project sites?
The project’s research team led by PI Ryan Holifield and Co-PI Woonsup Choi, will investigate how visual artistic activities introduced by the programming team as part of the Walks (and potentially other engagement activities) interact with personal, sociocultural, and physical contexts to produce distinctive experiences and outcomes of informal science learning about urban water systems. The aim of the research will be to synthesize the results from the different WaterMarks sites into an analysis generalizable beyond specific neighborhoods and applicable to other cities. The project's research questions include: 1. How does participation in Walks focused on visual artistic activities affect outcomes and experiences of informal STEM learning about urban water systems? 2. How do outcomes and experiences of informal STEM learning vary across different urban water topics, participants from different demographic groups, and contrasting sociocultural and biophysical contexts?
This Innovations in Development project is led by the University of Wisconsin-Milwaukee (UWM), in collaboration with City as Living Laboratory (CALL) and the COSI Center for Research and Evaluation.
The call for more science, technology, engineering, and mathematics (STEM) education taking place in informal settings has the potential to shape future generations, drive new innovations and expand opportunities. Yet, its power remains to be fully realized in many communities of color. However, research has shown that using creative embodied activities to explore science phenomena is a promising approach to supporting understanding and engagement, particularly for youth who have experienced marginalization. Prior pilot work by the principal investigator found that authentic inquiries into science through embodied learning approaches can provide rich opportunities for sense-making through kinesthetic experience, embodied imagining, and the representation of physics concepts for Black and Latinx teens when learning approaches focused on dance and dance-making. This Research in Service to Practice project builds on prior work to better understand the unique opportunities for learning, engagement, and identity development for these youth when physics is explored in the context of the Embodied Physics Learning Lab Model. The model is conceptualized as a set of components that (1) allow youth to experience and utilize their intersectional identities; (2) impact engagement with physics ideas, concepts and phenomena; and (3) lead to the development of physics knowledge and other skills. The project aims to contribute to more expansive definitions of physics and physics learning in informal spaces. While the study focuses primarily on Black and Latinx youth, the methods and discoveries have the potential to impact the teaching of physics for a much broader audience including middle- and high-school children, adults who may have been turned off to physics at an earlier age, and undergraduate physical science majors who are struggling with difficult concepts. This project is funded by the Advancing Informal STEM Learning (AISL) program which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
The research is grounded in sociocultural perspectives on learning and identity, embodied interaction and enactive cognition, and responsive design. The design is also informed by the notion of “ArtScience” which highlights commonalities between the thinking and making practices used by artists and by scientists and builds on the theoretical philosophy that all things can be understood through art or through science but integrating the two lenses allows for more complete understandings. Research will investigate the relationship between embodied learning approaches, design principles, and structures of the Embodied Physics Learning Lab model using the lenses of physics, dance, and integrated ArtScience to better understand the model. The project employs design-based research to address two overarching research questions: (1) What unique opportunities for learning, engagement, and identity development for Black and Latinx youth occur when physics is explored in the context of the Embodied Physics Learning Lab Model? and (2) How do variations in site demographics and site implementation influence the impact and scalability of the Learning Lab model? Further, the inquiry will consider (a) how youth experience and utilize their intersectional various identities in the context of the activities, structures, and essential elements of the embodied physics learning lab; (b) how youth's level of physics engagement changes depending on which embodied learning approaches and essential element structures are used; (c) the physics knowledge and other skills youth attain through the set of activities; and (d) how, if at all, the embodied learning approaches engage youth in thinking about their own agency as STEM doers. An interdisciplinary team of researchers, choreographers, and youth along with community organizations will co-design and implement project activities across four sites. Approximately 200 high school youth will be engaged; 24 will have the role of Teen Thought Partner. Through three iterative design cycles of implementation, the project will refine the model to investigate which elements most affect successful implementation and to identify the conditions necessary for scale-up. Data will be collected in the form of video, field notes, pre- and post- interviews, pre- and post- surveys, and artifacts created by the youth. Analyses will include a combination of interaction analysis, descriptive data analysis, and movement analysis. In addition to the research findings and explication of the affordances and constraints of the model, the project will also create a curricular resource, including narrative text and video demonstrations of physics concepts led by the teen thought partners, video case training modules, and assessment tools.
DATE:
-
TEAM MEMBERS:
Folashade Cromwell SolomonDionne Champion
This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).
The Accessible Oceans study will design auditory displays that support learning and understanding of ocean data in informal learning environments like museums, science centers, and aquariums. Most data presentations in these spaces use visual display techniques such as graphs, charts, and computer-generated visualizations, resulting in inequitable access for learners with vision impairment or other print-related disabilities. While music, sound effects, and environmental sounds are sometimes used, these audio methods are inadequate for conveying quantitative information. The project will use sonification (turning data into sound) to convey meaningful aspects of ocean science data to increase access to ocean data and ocean literacy. The project will advance knowledge on the design of auditory displays for all learners, with and without disabilities, as well as advance the use of technology for STEM formal and informal education. The study will include 425 participants but will reach tens of thousands through the development of education materials, public reporting, and social media. The study will partner with the Smithsonian National Museum of Natural History, Woods Hole Oceanographic Institution Ocean Discovery Center, the Georgia Aquarium, the Eugene Science Center, the Atlanta Center for the Visually Impaired, and Perkins School for the Blind.
The project will leverage existing educational ocean datasets from the NSF-funded Ocean Observatories Initiative to produce and evaluate the feasibility of using integrated auditory displays to communicate tiered learning objectives of oceanographic principles. Integrated auditory displays will each be comprised of a data sonification and a context-setting audio introduction that will help to make sure all users start with the same basic information about the phenomenon. The displays will be developed through a user-centered design process that will engage ocean science experts, visually impaired students and adults (and their teachers), and design-oriented undergraduate and graduate students. The project will support advocacy skills for inclusive design and will provide valuable training opportunities for graduate and undergraduate students in human-centered design and accessibility. The project will have foundational utility in auditory display, STEM education, human-computer interaction, and other disciplines, contributing new strategies for representing quantitative information that can be applied across STEM disciplines that use similar visual data displays. The project will generate publicly accessible resources to advance studies of inclusive approaches on motivating learners with and without disabilities to learn more about and consider careers in STEM.
This Pilots and Feasibility Studies project is supported by the Advancing Informal STEM Learning program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
Creating science education that can contribute to cultivating just, culturally thriving, and sustainable worlds is an important issue of our time. Indigenous peoples have persistently been under-represented in science reproducing inequalities in a myriad of ways from educational attainment, participation in and contributing to innovations in foundational knowledge, to effective policy making that upholds and respects Indigenous sovereignty. The development of models of science education that attend to intersections of knowledge and development, socio-scientific decision-making and civic leadership, and the complexities and contradictions of these realities, is imperative. This five-year Innovations in Development project broadens participation and strengthens infrastructure and capacity for Indigenous learners to meet, adapt to, and lead change in relation to the socio-ecological challenges of the 21st century. The project engages multi-sited community-based design studies to develop and research the impacts of Indigenous informal field-based science education with three Indigenous leadership communities from the Pacific Northwest and the Great Lakes. This project will have broader impacts through model development, building infrastructure to transform the capacity of informal field-based science education, and will produce cutting edge foundational knowledge about pressing 21st century issues with a particular focus on Indigenous communities. The project increases Indigenous participation in research through 1) engagement of Indigenous community members as research assistants, 2) training of Indigenous graduate fellows, and post-doctoral fellows, and 3) supporting the careers of more junior Indigenous scholars.
This research seeks to identify key design features of an Indigenous field (land/water) based model of science education and to understand how learners’ and educators’ reasoning, deliberation, decision-making, and leadership about complex socio-ecological systems and community change evolve in such learning environments. The project also examines key aspects of co-design and partnership with Tribal communities and how these methods of co-production of new science enable new capacities for systems transformation. This multi-layered project is organized through 3 panels of studies including: Panel 1) community-based design experiments to develop and refine a model of Indigenous informal science education; Panel 2) co-design and implementation of professional learning programs for Indigenous informal science education; and Panel 3) foundational studies in cognition and learning with respect to socio-ecological systems thinking and the impact on learning and instructional practices. Of particular importance in this research is the rigorous development and articulation of effective pedagogical practices and orientations. More broadly, findings will have clear implications for theories of cognitive development, deliberation and environmental decision making and especially those pertaining to how knowledge is shaped by culture and experience.
This project is funded by the Advancing Informal STEM Learning (AISL) program.
DATE:
-
TEAM MEMBERS:
Filiberto Barajas-LopezAnna LeesMegan BangAnna LeesFiliberto Barajas-Lopez
HBCUs are critical to producing a diverse and inclusive workforce as they graduate a disproportionate number of African American future STEM workers and STEM leaders. Although the National Science Foundation is fully committed to diversity and inclusion, there has been little research to determine why Historically Black Colleges and Universities are not fully participating in the NSF STEM educational research opportunities. The project will investigate the challenges, needs and support for Historically Black Colleges and Universities (HBCUs) to succeed in applying for educational research support from the National Science Foundation (NSF). Participants will be recruited from 96 HBCUs that are eligible to apply for such funding and will include the wide range of college and university administration and faculty that are involved in the preparation of research projects and related applications for research funding. The investigation will focus primarily on the Division of Research on Learning in Informal and Formal Settings (DRL) within NSF. The investigation will: 1) determine the submission rate and funding success rate of HBCUs within the DRL funding mechanisms; 2) determine why a greater proportion of HBCUs are not successful in their applications of research or do not apply; and 3) determine what factors, such as institutional support, research expertise, and professional development, could lead to a larger number of research proposals from HBCUs and greater success in obtaining funding. The project has the potential to have significant influence on the national educational and research agenda by providing empirical findings on the best approach to support and encourage HBCU participation in DRL educational research funding programs.
This exploratory research project will investigate what changes and/or supports would contribute to significantly increasing the number of applications and successful grant awards for STEM educational research project proposed by HBCUs. The project has the following research questions: (1) What factors discourage participation of HBCUs in the DRL funding mechanisms and what are the best practices to encourage participation? (2) What approaches have been successful for HBCUs to obtain DRL funding? (3) What dynamic capabilities are necessary for HBCU researchers to successfully submit STEM proposals to NSF? (4) What changes would be helpful to reduce or eliminate any barriers for HBCU applications for DRL educational research funding and what supports, such as professional development, would contribute to greater success in obtaining funding? Participants will be recruited from the 96 eligible HBCUs and will include both individuals from within the administration (e.g., Office Sponsored Programs, Deans, VP, etc.) as well as from within the faculty. The research will collect variety of quantitative and qualitative data designed to support a comprehensive analysis of factors addressing the research questions. The project will develop research findings and recommendations that are relevant to faculty, administrators, and policymakers for improving HBCU participation in research funding opportunities. Results of project research will be widely disseminated to HBCUs and other Minority Serving Institutions (MSIs) through a project website, peer reviewed journals, newsletters, and conference presentations.
This project is funded by the Innovative Technology Experiences for Students and Teachers (ITEST), the Advancing Informal STEM Learning (AISL), and the Discovery Research PreK-12 (DRK-12) programs. These programs which supports projects that build understandings of practices, program elements, contexts and processes contributing to increasing students' and general public knowledge and interest in science, technology, engineering, and mathematics (STEM).
DATE:
-
TEAM MEMBERS:
Cynthia TrawickJohn HaynesTriscia HendricksonTerry Mills
This Innovations in Development project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to (a) advance new approaches to and evidence-based understanding of the design and development of STEM learning in informal environments; (b) provide multiple pathways for broadening access to and engagement in STEM learning experiences; (c) advance innovative research on and assessment of STEM learning in informal environments; and (d) engage the public of all ages in learning STEM in informal environments.
Increasing greater diversity, equity, and inclusion in science not only presents a social justice goal, but is also vital to the financial and social success of the nation. The stereotype of the older white male scientist has obscured the contributions of women and people of color. This project seeks to remedy these perceptions which are barriers to entry into STEM fields. The project will create a large-scale hub for STEM themed video content on YouTube and other social media platforms, featuring 100+ original STEM videos produced by PBS partners. This hub and accompanying research seeks to identify the characteristics of online STEM content that attract (or fail to attract) underrepresented groups, specifically Black and Hispanic communities as well as women of all races. The objectives of this project are to 1) provide a unified online science-themed hub, PBS Terra, on YouTube and other platforms for hosting, sharing, and distributing digital STEM series from diverse producers from across the PBS system; 2) conduct surveys and focus groups to examine and understand the needs and expectations of women, Black and Hispanic communities and their consumption of STEM video content online and 3) test hypotheses about the communicative strategies of STEM videos that feature Black and Hispanic female scientists. Project collaborators include PBS, researchers at the University of Utah and the University of Georgia, and consultants and advisors with expertise in broadening participation and inclusion in STEM.
Little is known about how or why adult Americans seek science content on YouTube, especially the motivations of adults from underrepresented minorities and females. The key research questions in this project are: 1) Why do Black and Hispanic audiences and women of all races seek science video content online? 2) How does showing Black and Hispanic female scientists in science video content on YouTube impact viewers’ identification with and sense of belonging in STEM? 3) How does the use of humor by Black and Hispanic scientists in YouTube science content affect viewers’ perceptions of the communicator and their engagement with STEM content? 4) How does the appearance and manner of dress of Black and Hispanic scientists in YouTube science content affect viewers’ perceptions in the aforementioned areas? A nationally representative baseline survey will be conducted. A probability sample of 2000 respondents will be obtained including oversampling of Black and Hispanic audiences. To complement findings from the survey, focus groups will be conducted in eight different regions of the country to learn why these targeted audiences do or do not seek science content on YouTube and what motivates them to share the content with their social media network. In addition, an experiment embedded in an online survey will test the hypothesis that greater on-screen representation of women and scientists of color will broaden existing perceptions about scientists. The experiment will consist of a 3 (scientist’s race: Black/Hispanic/White) × 2 (science issue: controversial/non-controversial) × 2 (style: casual/professional) between-subjects design. Survey participants will be randomly assigned to the experimental conditions. These factors (science issue and host appearance) can be altered by content producers to better reach and engage the targeted audiences. The project not only investigates theoretical questions at the intersection of STEM stereotypes and race, but findings related to these experimental conditions will offer practical insight into strategies that can be used by science communication practitioners.
DATE:
-
TEAM MEMBERS:
Adam DylewskiSara YeoMichael Cacciatore
This collaborative research project between KQED, a public media organization serving the San Francisco Bay Area, Texas Tech University and Rockman et al conducted research to study how best to provide effective COVID-19 science news and social media content for young adult audiences.
To start the work, four “Knowledge Gap” studies – Twitter Misinformation, Mask Wearing Messaging, Germ Knowledge and Conceptual Mapping – as well as social media testing were conducted to address our first research question: How could COVID-19 coverage be designed to best inform, engage and educate millennials
KQED and Texas Tech advanced professional knowledge in the journalism and science communication fields around crisis reporting and building a media practitioner and academic researcher collaboration for audience research through a study conducted by Scott Burg of Rockman et al. Rockman gathered data between October 2020 - May 2021, interviewed KQED Science staff and participated in virtual observations of KQED project and related staff meetings to answer our second research question:
Can KQED develop a more efficient process of disaster reporting that responds to both constantly updating
This collaborative research project between KQED, a public media organization serving the San Francisco Bay Area, Texas Tech University and Rockman et al conducted research to study how best to provide effective COVID-19 science news and social media content for young adult audiences.
To start the work, four “Knowledge Gap” studies – Twitter Misinformation, Mask Wearing Messaging, Germ Knowledge (A&B) and Conceptual Mapping – as well as social media testing were conducted to address our research question: How could COVID-19 coverage be designed to best inform, engage and educate millennials
This collaborative research project between KQED, a public media organization serving the San Francisco Bay Area, Texas Tech University and Rockman et al conducted research to study how best to provide effective COVID-19 science news and social media content for young adult audiences.
To start the work, four “Knowledge Gap” studies – Twitter Misinformation, Mask Wearing Messaging, Germ Knowledge (A&B) and Conceptual Mapping – as well as social media testing were conducted to address our research question: How could COVID-19 coverage be designed to best inform, engage and educate millennials