This summary brief captures highlights from the evaluation report for the first year of the NSF-funded WaterMarks project (also available on this page). The purpose of this document is to communicate key updates from evaluation in a less technical way with the many different audiences who have an interest in keeping up with WaterMarks.
This is the evaluation report for the first year of the NSF-funded WaterMarks project. It reflects an initial summary of available evidence about the intended outcomes of program activities to date, as well as commentary on how the project is using (or could use) this information moving forward. This report contains descriptions of embedded measures (i.e. anonymized drawings and reflections captured on a thematic postcard) included in community walks and analyses of secondary data (i.e., interviews conducted by other members of hte project team), as well as reflections emerging from the
How to determine a sample size for studying your informal learning program
What’s important to consider when selecting a sample size, or participant group size, for studying your informal (science) education program? What value does a small sample size have? If program managers have ample resources for collecting a lot of data, what factors are important to consider?
Here is a collection of guidelines and examples for use in making an informed decision about determining the sample size for studying your informal learning program. To design a study whose findings are more authentic
Diversity, Equity, Access and Inclusion (DEAI) work in museums is multifaceted, but typically approached from the perspective of external audiences and outcomes rather than a change in internal organizational culture. This article discusses findings from a research study examining what happened in five US science museums that were making a concerted, officially recognized effort towards internal change, and explores what those findings reveal about field-wide barriers to appreciable systemic change along with the impacts of the current status quo on marginalized staff. This study focused
This pilot and feasibility project will explore whether participation in informal science initiatives like citizen science, which is a form of Public Participation in Scientific Research, can foster or enhance participant attachment to the natural places participants investigate via these programs. The project also examines if participant attachment to place influences the development or application of critical thinking skills among adult learners. Critical thinking skills and the factors that enhance critical thinking skills are important areas of inquiry within the informal STEM learning community. Existing scholarship suggests that three components may be linked: (1) feelings of connection to specific places, (2) intentional exploration and investigation of those places (in this case via citizen science), and (3) understanding of complex socio-ecological systems, which is predicated on critical thinking skills ability. However, the degree to which these aspects are related to each other, the scale at which they occur (local to global), and the specific dimensions of place connection or informal science experiences implicated is not known. Working with the Coastal Observation and Seabird Survey Team (COASST) citizen science program, this project advances collaboration among experts from disparate fields to examine if and how citizen science contributes to increases in connection to place and higher-order critical thinking skills among participants and the potential links between those hypothesized outcomes. The ultimate goal of the project is to inform design of Public Participation in Scientific Research programming that optimizes participant learning, interest, and retention; produces societal outcomes like critical thinking in support of science literacy; and creates high quality data of the scale and grain needed to address questions and issues across the basic-applied science continuum.
This research focuses on the degree to which Public Participation in Scientific Research, specifically citizen science, may foster the presence or intensity of place attachment felt by participants for the sites and settings investigated through these programs and to what extent place attachment may be linked to higher order critical thinking skills among adult learners. A three-pronged mixed-methods research strategy will include: (1) a re-analysis of existing survey and interview data for markers of three-dimensional (personal, social, natural) place attachment as well as critical thinking skills and dispositions; (2) an assessment survey to test for the presence and intensity of place attachment and critical thinking skills; and (3) in-depth interviews to better understand the qualitative nature and development of place attachment and critical thinking skills in a citizen science context. The survey and interview sample will be drawn from participants in the COASST citizen science program and will be stratified into four groups as a function of time engaged in the program, including new, novice, and long-term participants. An independent external advisory board and a committee of visitors comprised of experts in informal science, education, and sense of place will critique and help guide this work. Results are expected to reveal important factors that impact the learning and behavioral outcomes of informal STEM initiatives by probing questions about the essential experiences, exposures, and COASST program components that facilitate deeper critical thinking skills and place attachment. Synthesizing theoretical frameworks from the fields of geography, science education, and educational psychology while testing a unique methodological approach to best measure critical thinking skills and place attachment in an informal citizen science setting will enhance knowledge-building among research and practitioner communities.
In a globalized and increasingly technologically complex world, the ability of citizens to interrogate and interpret scientific evidence, views, and values is critical. That is, scientific literacy is essential for the maintenance of robust and healthy economic, social, and environmental systems in the twenty-first century. Informal science learning fills an important gap in national educational efforts to cultivate a scientifically literate populace as research suggests that formal science training is not always capable of fostering the type of higher order critical thinking skills that undergird such scientific competency. This project aims to strengthen infrastructure and build capacity among informal science practitioners by clarifying whether specific aspects and forms of Public Participation in Scientific Research, especially those relating to people-place connections, are implicated in the development and/or application of critical thinking skills in STEM settings. This effort may expand opportunities to strengthen informal science learning program outcomes, including the cultivation of numerous 21st century skills like information literacy and social skills like conflict management. Through a greater understanding of the individual components that shape informal learning experiences and outcomes, this project also has the potential to support the broadening of participation in STEM fields by providing the groundwork for further research on whether or not underrepresented or traditionally marginalized groups of people experience and/or relate differently to both the "places" most common in citizen science and the practice of informal science programming itself.
Milwaukee has established itself as a leader in water management and technology, hosting a widely recognized cluster of industrial, governmental, nonprofit, and academic activity focused on freshwater. At the same time, Milwaukee faces a wide range of challenges with freshwater, some unique to the region and others common to cities throughout the country. These challenges include vulnerability to flooding and combined sewer overflows after heavy rainfall, biological and pharmaceutical contamination in surface water, lead in drinking water infrastructure, and inequity in access to beaches and other recreational water amenities. Like other cities, Milwaukee grapples with the challenges global climate change imposes on urban water systems, including changing patterns of precipitation and drought.
These problems are further complicated by Milwaukee's acute racial and economic residential segregation. With a population of approximately 595,000, embedded within a metropolitan area of over 1.5 million, Milwaukee remains one of the country's most segregated cities. There is increasing urgency to engage the public--and especially those who are most vulnerable to environmental impacts--more deeply in the stewardship of urban water and in the task of creating sustainable urban futures. The primary goal of this four-year project is to foster community-engaged learning and environmental stewardship by developing a framework that integrates art with Science, Technology, Engineering, and Mathematics (STEM) experiences along with geography, water management, and social science. Synergies between STEM learning and the arts suggest that collaborations among artists, scientists, and communities can open ways to bring informal learning about the science of sustainability to communities.
WaterMarks provides an artist generated conceptual framework developed by Mary Miss / City as Living Laboratory (CALL) to help people better understand their relationship to the water systems and infrastructure that support their lives. Project activities include artist/scientist/community member-led Walks, which are designed to engage intergenerational participants both from the neighborhoods and from across the city, in considering the conditions, characteristics, histories, and ecosystems of neighborhoods. Walks are expanded upon in Workshops with residents, local scientists/experts, and other stakeholders, and include exploring current water-related environmental challenges and proposing solutions. The Workshops draw on diverse perspectives, including lived experience, scientific knowledge, and policy expertise. Art projects created by local artists amplify community engagement with the topics, including programming for teens and young adults. Free Wi-Fi will be integrated into various Marker sites around the city providing access to online, self-guided learning opportunities exploring the water systems and issues facing surrounding neighborhoods. Current programming focuses primarily on Milwaukee's predominantly African American near North Side and the predominantly Latinx/Hispanic near South Side. Many neighborhoods in these sections are vulnerable to such problems as frequent flooding, lead contamination in drinking water, inequities in safety and maintenance of green space, and less access to Lake Michigan, the city's primary natural resource and recreational amenity.
The WaterMarks project advances informal STEM learning in at least two ways. First, while the WaterMarks project is designed to fit Milwaukee, the project includes the development of an Adaptable Model Guide. The Guide is designed so that other cities can modify and employ its inclusive structure, programming, and process of collaboration among artists, scientists, partner organizations, and residents to promote citywide civic engagement in urban sustainability through the combination of informal STEM learning and public art. The Guide will be developed by a Community-University Working Group (CULab) hosted by UW-Milwaukee's Center for Community-Based Learning, Leadership, and Research and made up of diverse community and campus-wide stakeholders. In addition to overseeing the Guide’s creation, CULab will conceptualize onboarding and mentorship strategies for new participants as well as a framework for the program’s expansion and sustainability.
Second, through evaluation and research, the project will build a theoretical model for the relationships among science learning, engagement with the arts, and the distinctive contexts of different neighborhoods within an urban social-ecological system. The evaluation team, COSI’s Center for Research and Evaluation, and led by Co-PI Donnelly Hayde, aims to conduct formative, summative, and process evaluation of the Watermarks project, with the additional goal of producing evaluative research findings that can contribute to the broader field of informal learning. Evaluation foci include: How does the implementation of WaterMarks support positive outcomes for the project’s communities and the development of an adaptable model for city-scale informal science learning about urban environments? 2. To what extent do the type and degree of outcome-related change experienced by participating community residents vary across and/or between project sites? What factors, if any, appear to be linked to these changes? 3. To what extent and in what ways do the activities of the WaterMarks projects appear to have in situ effects related to the experience of place at project sites?
The project’s research team led by PI Ryan Holifield and Co-PI Woonsup Choi, will investigate how visual artistic activities introduced by the programming team as part of the Walks (and potentially other engagement activities) interact with personal, sociocultural, and physical contexts to produce distinctive experiences and outcomes of informal science learning about urban water systems. The aim of the research will be to synthesize the results from the different WaterMarks sites into an analysis generalizable beyond specific neighborhoods and applicable to other cities. The project's research questions include: 1. How does participation in Walks focused on visual artistic activities affect outcomes and experiences of informal STEM learning about urban water systems? 2. How do outcomes and experiences of informal STEM learning vary across different urban water topics, participants from different demographic groups, and contrasting sociocultural and biophysical contexts?
This Innovations in Development project is led by the University of Wisconsin-Milwaukee (UWM), in collaboration with City as Living Laboratory (CALL) and the COSI Center for Research and Evaluation.
This award is funded with support from NSF's program for Advancing Informal STEM Education.
This project develops a partnership between language researchers and Planet Word, a new museum devoted to language in Washington D.C., to engage museum visitors in scientific research and outreach. Interested museum visitors from all ages and backgrounds are invited to participate in behavioral research studies on a range of language-related topics. This "living language laboratory" of interactive studies includes accompanying educational demonstrations. These activities will lead to the development of infrastructure and best practices that will allow future language researchers to engage with the public at Planet Word and other similar sites.
The project enhances scientific understanding by engaging visitors in activities that expose them to active science about language as a part of their visit to the museum. For example, the research examines topics from understanding what makes certain American Sign Language signs more learnable, to why it is easier to understand people we know rather than strangers, to whether we think differently when we are reading a text message compared to reading more formal writing. In doing so, the project raises the profile of linguistics among the general public and promotes scientifically informed attitudes about language. The project also provides key opportunities to disseminate research findings of interest to the public and to promote greater interest in STEM topics among museum visitors, as well as student trainees and museum staff. The project creates educational and research opportunities for students, who will be trained in a hands-on course, and will gain first-hand experience with research and outreach in a museum setting. Through the collaborative partnership of researchers from University of Maryland, Howard University, and Gallaudet University, the project broadens participation of underrepresented minority students in the language sciences, seeking to diversify the pipeline of scholars continuing in careers in the language sciences and related STEM fields.
DATE:
-
TEAM MEMBERS:
Charlotte VaughnYi Ting HuangDeanna GagnePatrick Plummer
This project investigates long-term human-robot interaction outside of controlled laboratory settings to better understand how the introduction of robots and the development of socially-aware behaviors work to transform the spaces of everyday life, including how spaces are planned and managed, used, and experienced. Focusing on tour-guiding robots in two museums, the research will produce nuanced insights into the challenges and opportunities that arise as social robots are integrated into new spaces to better inform future design, planning, and decision-making. It brings together researchers from human geography, robotics, and art to think beyond disciplinary boundaries about the possible futures of human-robot co-existence, sociality, and collaboration. Broader impacts of the project will include increased accessibility and engagement at two partner museums, interdisciplinary research opportunities for both undergraduate and graduate students, a short video series about the current state of robotic technology to be offered as a free educational resource, and public art exhibitions reflecting on human-robot interactions. This project will be of interest to scholars of Science and Technology Studies, Human Robotics Interaction (HRI), and human geography as well as museum administrators, educators and the general public.
This interdisciplinary project brings together Science and Technology Studies, Human Robotics Interaction (HRI), and human geography to explore the production of social space through emerging forms of HRI. The project broadly asks: How does the deployment of social robots influence the production of social space—including the functions, meanings, practices, and experiences of particular spaces? The project is based on long-term ethnographic observation of the development and deployment of tour-guiding robots in an art museum and an earth science museum. A social roboticist will develop a socially-aware navigation system to add nuance to the robots’ socio-spatial behavior. A digital artist will produce digital representations of the interactions that take place in the museum, using the robot’s own sensor data and other forms of motion capture. A human geographer will conduct interviews with museum visitors and staff as well as ethnographic observation of the tour-guiding robots and of the roboticists as they develop the navigation system. They will produce an ethnographic analysis of the robots’ roles in the organization of the museums, everyday practices of museum staff and visitors, and the differential experiences of the museum space. The intellectual merits of the project consist of contributions at the intersections of STS, robotics, and human geography examining the value of ethnographic research for HRI, the development of socially-aware navigation systems, the value of a socio-spatial analytic for understanding emerging forms of robotics, and the role of robots within evolving digital geographies.
This project is jointly funded by the Science and Technology Studies program in SBE and Advancing Informal STEM Learning (AISL) Program in EHR.
An ecosystems model of learning suggests there are critical partners within and across a community that support learning across the lifespan. These school-community partnerships, developed with shared accountability and goals, are essential to rural students given the lack of economic and geographic access to such services. Youth in rural areas may have limited opportunities to engage with professionals. The team proposes to overcome this gap by capitalizing on the wide-spread interest in archaeology to teach critical thinking using STEM concepts and testing components of a partnership program. This project will advance knowledge on multidisciplinary STEM education by iteratively developing and researching an after-school program in which youth engage in multidisciplinary inquiry in the context of archeology. Mentored by archaeologists, rural youth and citizen scientists will use concepts and tools drawn from biology, ecology, geospatial science, mathematics, physics, and data science to identify and answer questions related to the history of their local region. An outcome of this project will be a road map for moving from a feasibility project to a larger implementation project locally and an understanding of community partnerships engaging more broadly.
Researchers at SUNY Binghamton will conduct a mixed-methods research study that examines the ways in which participation in a multidisciplinary after-school archaeology program supports the development of STEM identities among rural youth in sixth through eighth grades. The research team will use content analysis to analyze field notes from observations, as well as transcripts from focus groups and interviews with the youth. They will use inferential statistics to explore changes in the youths' STEM identity using an identity survey, which will be administered to the youth before and after participation in the program. Additionally, the research team will conduct qualitative research that explores shifts in the afterschool program providers' perceptions about supporting middle school youth as STEM learners. The program providers are comprised of graduate and undergraduate archaeology students, citizen scientists, and professional archaeologists. The course modules developed for the after-school program will be disseminated through professional networks and organizations dedicated to archaeologists and informal educators, and empirical findings will be shared widely via peer-reviewed publications. This project is funded by the Advanced Informal STEM Learning (AISL) program. As part of its overall strategy to enhance learning in informal environments, the AISL program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This Pilots and Feasability Studies award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
The impacts of changes in the climate at local and global levels threaten how people live. Some frontline communities, especially in historically disenfranchised and under-resourced areas, are particularly vulnerable to the devastating effects of climatological events such as wildfires, flooding, and urban heat islands. As such, there is an urgent need for collective, evidence-based understanding and engagement to prevent and prepare for these potentially fatal events. Led by the Oregon Museum of Science and Industry (OMSI) in Portland, Oregon, in collaboration with local and national partners, Youth Lead the Way is an early-stage Innovations in Development project that offers a theory-based approach for youth in climatologically vulnerable communities to work in climate science research alongside field researchers, develop leadership skills, and engage in timely conversations that impact their own communities. The project will develop and evaluate a Youth Advisory Research Board model to equip and support youth and informal STEM education institutions to conduct evidence-based research on local climate impacts and communicate the findings of their research to their communities. Youth Lead the Way advances the work of several previous NSF-funded projects on climate education, youth advisory boards, and collaborative networks to engage the public in informal STEM learning. Findings from this project will support ongoing efforts in the informal STEM education field to meaningfully engage youth and to more effectively communicate science related to climate and its impacts to the public.
During this initial two-year early-stage project, youth predominantly from racial and ethnic groups underrepresented in STEM will engage in a year-long extended STEM experience. These youth will work collaboratively with scientists and museum professionals to enhance their skills as climate researchers, science communicators, and educational leaders, while reaching an estimated 4,000 or more public audience members through research and events at OMSI, in their schools, and in their communities. Using a cohort model, the youth will conduct scientifically based research studies on various local climate impact topics while concurrently serving in an advisory role at the Oregon Museum of Science and Industry, where they will participate in shaping relevant museum programs and practices. The youth will also develop and present climate stories, a communication approach based on storytelling, to raise public understanding and awareness about local climatological changes and impacts. In addition to the youth component, a companion workshop will be held at the Sciencenter in Ithaca, New York, a partner organization, to train staff and formatively assess the feasibility of scaling the model in other museums. At the program level, an exploratory qualitative research study will be conducted to identify the factors of the overall model that contribute to desired outcomes of youth engagement, climate impact education, and informal science education professional development. Interviews, surveys, focus groups, group chats among youth cohort members, and reviews of artifacts generated by the youth will inform this exploratory study. A theory-based guide outlining key findings, considerations, and recommendations will also be produced. The dissemination of this work will be multi-tiered, reaching thousands within the target communities through public programs, professional networks, at conferences, and a live virtual professional development event hosted by the Association for Science-Technology Centers. If successful, Youth Lead the Way will lay the groundwork for a model that promotes youth and public engagement in STEM through climate science research and identifies promising pathways for future research and similar efforts well beyond this project.
This early-stage Innovations in Development project is funded by the NSF Advancing Informal STEM Learning program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.