As science communication programs grow worldwide, effective evaluation and assessment metrics lag. While there is no consensus on evaluation protocols specifically for science communication training, there is agreement on elements of effective training: listening, empathy, and knowing your audience — core tenets of improvisation. We designed an evaluation protocol, tested over three years, based on validated and newly developed scales for an improvisation-based communication training at the Alan Alda Center for Communicating Science. Initial results suggest that ‘knowing your audience’ should
DATE:
TEAM MEMBERS:
Christine O’ConnellMerryn McKinnonJordan Labouff
This poster describes the Addressing Societal Challenges through STEM (ASCS) project. The project's research goal is to identify and describe the range of ways that informal STEM learning (ISL) institutions are addressing societal challenges and how STEM knowledge and scientific reasoning are situated in that engagement.
The poster was presented at the American Association of Museums (AAM) 2020 Virtual Conference.
Final External Evaluation Report for Informal STEM Learning at Biological Field Stations, an NSF AISL Exploratory Pathways project, which studied the pedagogical and andragogical characteristics of informal educational outreach activities at field stations. This report summarizes the project team’s major research activities and the contextual factors that supported that work.
Appendix includes interview protocol.
In order to engage visitors, guests, participants or audiences in positive STEM learning experiences, informal learning institutions need professionals who understand how to design for and facilitate engaging activities. Initial professional training for informal STEM educators, and subsequent ongoing professional learning create considerable challenges. There is a need for providing informal STEM educators with pathways to professionalization that guarantee high quality educators who can support successful informal STEM education. In this symposium, we propose to share research on key aspects
Addressing Societal Challenges through STEM (ASCs) received NSF AISL funding to conduct a Literature Review and Synthesis to answer the question: How are informal learning institutions advancing the use of STEM knowledge and scientific reasoning in the ways that individuals, families, and communities understand what they can do, and apply their learning to solving the societal challenges of our time? Using a definition of societal challenges based on research around the public understanding of social problems, this systematic literature review will identify, analyze, and synthesize three bodies of peer and field-reviewed literature (peer-reviewed journals, graduate theses, and evaluation reports of nationally-funded project).
Over the past decade, Informal STEM learning organizations have increasingly engaged in innovative ways to present STEM knowledge within the context of societal challenges such as climate change, energy sources, cyber-security, Nanotechnologies, coastal resilience, and other topics. These efforts significantly expand the traditional work of Informal STEM Learning (ISL) organizations, often leading to new types of interventions, partnerships, impacts, and assessment tools. Analyzing and interpreting the aggregate of this work will advance theoretical and practical knowledge about the potential of ISL’s in advancing the place of STEM in addressing societal challenges.
Demonstrating and articulating the characteristics of how ISL organizations are addressing societal challenges, encourages and informs the ways institutions can address the NSF strategic goal to “Advance the capability of the Nation to meet current and future challenges.” The project outputs aim to Enhance Knowledge-building, Build Capacity of the Field, and Maximize Strategic Impact by informing the strategies used by organizations and individuals. The results also aim to Broaden Participation by articulating the ways STEM knowledge is embedded and linked to personal experiences and choices.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
Consideration of the needs of individuals with a wide range of disabilities is not always considered in the early design stages of an informal STEM learning (ISL) activity or program. The primary access approach for people with disabilities becomes the provision of accommodations once the ISL product or environment is created. In contrast, the Universal Design approach considers users with a wide range of characteristics throughout the design process and works to create products and environments that are accessible, usable, and inclusive. This project, called AccessISL, led by the University of Washington's DO-IT (Disabilities, Opportunities, Internetworking and Technology) Center and Museology Program, includes an academic museology program and local ISL sites, representing museums, zoos, aquariums, makerspaces, science centers, and other sites of informal STEM learning. Insights will be gained through the engagement of people with disabilities, museology graduate students and faculty, and ISL practitioners. The AccessISL project model, composed of a set of approaches and interventions, builds on existing research and theory in the fields of education science, change management, effective ISL practices, and inclusive design processes. The project will collect evidence of policies and practices (or lack thereof) that improve the inclusiveness of ISL with respect to a wide range of disabilities and considers approaches for the design and development of new strategies; explores what stakeholders need to make change happen; uncovers challenges to the adoption of inclusive practices in public ISL settings and explores ways to overcome them; and proposes relevant content that might be included in museology curriculum. This project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This project addresses the following two objectives:
For ISL personnel and museology faculty: to increase knowledge, skills, and actions to make ISL programs, facilities, courses, and resources more welcoming and accessible to participants with disabilities and embed relevant practices within their work.
For postsecondary STEM students with disabilities and museology students: to increase knowledge and skills in advocating for ISL offerings that are welcoming and accessible to everyone, including those with a wide variety of disabilities, and to encourage individuals with disabilities to pursue careers in ISL.
The project employs a student-centered approach and a set of practices that embrace the social model of disability, social justice education, disability as a diversity issue, intersectionality, and Universal Design. A leadership team of interns--each member a STEM student with a disability or a museology graduate student--along with project staff will engage with the University of Washington's Museology Program to identify and implement strategies for making ISL activities and courses more welcoming and accessible to individuals with disabilities. An online community of practice will be developed from project partners and others nationwide. A one-day capacity building institute will be held to include presentations, student/personnel panels for sharing project and related experiences, and group discussions to explore issues and further identify systemic changes to make ISL programs more welcoming and accessible to individuals with disabilities. As prototypes of the AccessISL Model are developed, evaluation activities will primarily be formative (looking for strengths and weaknesses) and remedial (identifying/implementing changes that could be made to improve the model). The model will continue to be fine-tuned through formative evaluation. Evaluation of the model components will focus on the experience of a range of stakeholders in the project. Specifically, quantitative data collected will include levels and quality of engagement, accessibility recommendations and products developed, and delivery of ISL services. Qualitative data will be collected through observations, surveys, focus groups, interviews, and case studies.
AccessISL project products will include proceedings of an end-of-project capacity building institute, promising practices, case studies, a video, and other online resources to help ISL practitioners and museology faculty that will result in making future ISL opportunities more inclusive of people with disabilities. AccessISL will advance knowledge and ensure long-term impact using multiple strategies:
broadening the STEM participation of people with disabilities as well as women, racial/ethnic minorities, and other underrepresented groups through the application of universal design
strengthening associations and creating synergy and durable relationships among stakeholders,
encouraging teaching about disability, accessibility, and universal design in museology courses,
empowering students with disabilities and current and future ISL practitioners to advocate for accessible ISL and develops an infrastructure to promote accessible ISL programs nationwide, and
contributing to the body of promising practices with products that will (a) enhance understanding of issues related to the inclusion of people with disabilities in ISL programs and (b) promote inclusive practices.
Outcomes will benefit society by making STEM opportunities available to more people and enhancing STEM fields with the talents and perspectives of people with disabilities.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. This Innovations in Development project will develop and test a model for audience assessment that STEM professionals can use during their public engagement efforts. These on-the-spot assessments will allow scientists to monitor audience understanding and use this to make immediate improvements to their activity. This project fills a critical gap in the field of public science communication as expressed by outreach scientists and by the professional and academic organizations that train them. Project partners include The Astronomical Society of the Pacific, Oregon State University, Pacific Science Center, and the National Radio Astronomy Observatory. Research questions include: (1) How and under what conditions can scientists, regardless of discipline, learn to build assessment into outreach activities? (2) To what degree are scientists willing and able to change their outreach activities to include assessment to ascertain audience attentiveness or understanding? (3) To what degree will scientists be able to adjust their outreach and engagement efforts based on audience feedback, and what support do they need to do so? (4) With constraints on their time and resources, how can the model help scientists conduct audience assessment on their own? (5) Are audience members finding events more enjoyable and fruitful that use assessment techniques? Several iterative rounds of R&D will be used in the spiral design-based approach. A group of Design Testers will start each of the iterative cycles for the initial development and testing techniques, associated training, and support. Later phases will use Field Testers using a rubric for usability, validity, and reliability of various strategies. The summative evaluation of the project will begin early and add data over the life of the project. A mixed methods approach will be used including surveys of scientists once each year after they have participated in the Prototype training. Descriptive and inferential statistics will be used to document the extent to which the project has achieved its intended outcomes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. The Museum of Science, Boston (MOS) and Boston University (BU) will conduct a Pilot and Feasibility Study project that leverages the current Living Laboratory (LL) model and expand it to engage high school students (teens) in experimental psychology research, science communication and science education activities. In LL, which is now an extensive network of museums and university researchers across the country, scientists and museum staff collaborate to engage children in studies on the museum floor and educate caregivers about the research. Multi-site implementation and evaluation of LL has also documented positive impacts for undergraduate researchers. Many sites are eager to extend these benefits to high school students by engaging them as practitioners within the model and by providing them with opportunities to engage in current research, education and communication, thereby helping to foster stronger youth identities with science and its applications in society. This project expands a ten-year LL partnership between MOS and BU to: 1) pilot a program in which high school students both conduct scientific research and engage the public in learning about science; 2) explore strategies for museums and universities to collaboratively engage, support and mentor high school students in science research, communication and education activities; 3) document curricular, other programmatic, and evaluation materials; and 4) convene professional participants to provide feedback on pilot materials, and assess the viability of implementing similar programs at additional sites. Guided by developmental evaluation, these activities will generate knowledge for the field, and act to increase professional capacity to integrate experiences for teens at multiple LL sites in future projects. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
As higher education institutions (HEIs) work to enhance Broader Impacts (BI) efforts, collaborations with informal science education institutions (ISEs) (e.g. science centers, aquaria, zoos) can help them strengthen their impact and reach broader audiences. This project builds on the successful Portal to the Public (PoP) framework, bringing together the expertise and resources of HEIs and ISEs around the shared mission of engaging public audiences in current STEM research. The project is designed to address several critical needs: (1) Public outreach BI activities are relatively uncommon compared to BI that is focused within the infrastructure of academia; (2) Because collaborations with ISEs are frequently tied to individual Principal Investigators (PIs), there is limited opportunity to build a body of knowledge around the practice of partnering for BI work; and (3) Collaborations are often transient, making it more difficult for universities to view BI on an institutional level in ways that leverage particular institutional assets or strategies and even link investigators from multiple projects. The specific areas of study are: a. Develop and test a structure for education/outreach BI experience design that addresses a public audience need and meets NSF's BI criterion: The project will create disseminatable tools around the activity design process (including evaluation of learning impacts). By convening cross-disciplinary teams, the project will ensure that experiences will reflect a wide range of expertise and will help meet the needs of multiple stakeholders. These established structures will lower the barrier to entry for PIs who want to do public outreach BI. b. Design, test, and study structures for long-term, mutually beneficial HEI-ISE partnerships: The project will build on the proven PoP model to create flexible, disseminatable tools around the development of institutional partnerships at three collaborating HEI-ISE site pairings that consider each institution's resources, constraints and strategic goals, including a cross-institutional and cross-disciplinary Broader Impacts Design (BID) Team structure. Sustained partnerships will support ongoing public engagement with current STEM research. c. Anchor the partnership at the HEI with a representative from an office of research support: Research support professionals will be a core part of the BID Team and will help support institutional strategies for aligning BI activities with broader goals around community engagement. d. Study the culture of HEI-ISE partnerships, building knowledge about how these institutions can form effective, sustained and mutually beneficial collaborations. Project partners include Pacific Science Center with the University of Washington, Bothell, WA; University of Wisconsin-Madison with the Wisconsin Institute for Discovery; and the Sciencenter with Cornell University, Ithaca, NY. In addition, the Center for Research in Lifelong Learning, Oregon State University will oversee the research aspects of the project. The project's primary benefit is the development of more effective mechanisms for HEIs and ISEs to collaborate, that will better enable them to engage their communities in experiences and conversations about current STEM research and innovation. This project is being funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
As higher education institutions (HEIs) work to enhance Broader Impacts (BI) efforts, collaborations with informal science education institutions (ISEs) (e.g. science centers, aquaria, zoos) can help them strengthen their impact and reach broader audiences. This project builds on the successful Portal to the Public (PoP) framework, bringing together the expertise and resources of HEIs and ISEs around the shared mission of engaging public audiences in current STEM research. The project is designed to address several critical needs: (1) Public outreach BI activities are relatively uncommon compared to BI that is focused within the infrastructure of academia; (2) Because collaborations with ISEs are frequently tied to individual Principal Investigators (PIs), there is limited opportunity to build a body of knowledge around the practice of partnering for BI work; and (3) Collaborations are often transient, making it more difficult for universities to view BI on an institutional level in ways that leverage particular institutional assets or strategies and even link investigators from multiple projects.
The specific areas of study are:
a. Develop and test a structure for education/outreach BI experience design that addresses a public audience need and meets NSF's BI criterion: The project will create disseminatable tools around the activity design process (including evaluation of learning impacts). By convening cross-disciplinary teams, the project will ensure that experiences will reflect a wide range of expertise and will help meet the needs of multiple stakeholders. These established structures will lower the barrier to entry for PIs who want to do public outreach BI.
b. Design, test, and study structures for long-term, mutually beneficial HEI-ISE partnerships: The project will build on the proven PoP model to create flexible, disseminatable tools around the development of institutional partnerships at three collaborating HEI-ISE site pairings that consider each institution's resources, constraints and strategic goals, including a cross-institutional and cross-disciplinary Broader Impacts Design (BID) Team structure. Sustained partnerships will support ongoing public engagement with current STEM research.
c. Anchor the partnership at the HEI with a representative from an office of research support: Research support professionals will be a core part of the BID Team and will help support institutional strategies for aligning BI activities with broader goals around community engagement.
d. Study the culture of HEI-ISE partnerships, building knowledge about how these institutions can form effective, sustained and mutually beneficial collaborations.
Project partners include Pacific Science Center with the University of Washington, Bothell, WA; University of Wisconsin-Madison with the Wisconsin Institute for Discovery; and the Sciencenter with Cornell University, Ithaca, NY. In addition, the Center for Research in Lifelong Learning, Oregon State University will oversee the research aspects of the project. The project's primary benefit is the development of more effective mechanisms for HEIs and ISEs to collaborate, that will better enable them to engage their communities in experiences and conversations about current STEM research and innovation.
This project is being funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. The goal of this project is to make 21st century quantum science comprehensible and engaging to non-expert informal adult learners. This project has strong potential to add new knowledge about the public's perception and understanding of quantum physics. This scientific content is often difficult for informal audiences to grasp, and there are relatively few accessible learning resources for a non- professional audience. The development of this online, interactive resource with short animations, graphics, and simulations has strong potential to fill this gap. It will develop a visually driven online resource to engage non-expert audiences in understanding the basics of quantum physics. The web design will be modular, incorporating many multimedia elements and the structure will be flexible allowing for future expansion. All content would be freely available for educational use. There is potential for extensive reach and use of the resources by informal adult learners online as well as learners in museums, science centers, and schools. Project partners are the Joint Quantum Institute at the University of Maryland and the National Institute of Standards and Technology, College Park. An independent evaluation of the project will add new knowledge about informal learners' perceptions and/or knowledge about quantum science and technology. An initial needs assessment via focus groups with the general public will be designed to find out more about what they already know about quantum physics topics and terminology, as well as what they want to know and what formats they prefer (games, simulations, podcasts, etc.). In person user testing will be used with early versions of the project online resource using a structured think-aloud protocol. Later in year 1 and 2, online focus groups with the general public will be conducted to learn what they find engaging and what they learned from the content. Iterative feedback from participants during the formative stage will guide the development of the content and format of the online resources. The Summative Evaluation will gather data using a retrospective post-survey embedded with a pop-up link on the Atlas followed by interviews with a subset of online users. Google Analytics will be used to determine the breadth and depth of their online navigation, what resources they download, and what websites they visit afterward. A post-only survey of undergraduate and graduate students who participated in resource development will focus on changes in students' confidence around their science communication skills and level of quantum physics understanding.