This set of appendices represent all research instruments related to study presented in the I/CaLL Art Experiences and Advancing Science Literacy report (NewKnowledge Publication #NSF.097.115.07).
Appendix A: Installation Site Intercept Interview Instrument
Appendix B: Artists-Scientists Walk & Talks Instrument
Appendix C: Post-Performance Event Survey Instrument
Note that researchers did not use an instrument for the fourth aspect of the study, the post-performance event reflection sessions. Instead, they allowed the discussions to be directed by the reflection participants.
DATE:
TEAM MEMBERS:
John FraserRebecca Joy NorlanderSophie SwansonNezam ArdalanKate FlinnerJoanna Laursen BruckerNicole LaMarca
Engagement is the cornerstone of learning in informal science education. During free-choice learning in museums and science centers, visitor engagement shapes how learners interact with exhibits, navigate through exhibit spaces, and form attitudes, interests, and understanding of science. Recent advances in multimodal learning analytics are creating novel opportunities for expanding the range and richness of measures of visitor engagement in free-choice settings. In particular, multimodal learning analytics offer significant potential for integrating multiple data sources to devise a composite picture of visitors' cognitive, affective, and behavioral engagement. The project will center on providing a rich empirical account of meaningful visitor engagement with interactive tabletop science exhibits among individual visitors and small groups, as well as uncovering broader tidal patterns in visitor engagement that unfold across exhibit spaces. A key objective of the project is creating models and practitioner-focused learning analytic tools that will inform the best practices of exhibit designers and museum educators. This project is funded by the Advancing Informal STEM Learning (AISL) program. As part of its overall strategy to enhance learning in informal environments, AISL funds research and innovative approaches and resources for use in a variety of settings. The research team will conduct data-rich investigations of visitors' learning experiences with multimodal learning analytics that fuse the rich multichannel data streams produced by fully-instrumented exhibit spaces with the data-driven modeling functionalities afforded by recent advances in machine learning and educational data mining. The research team will conduct a series of visitor studies of naturalistic engagement in solo, dyad, and group interactions as visitors explore interactive tabletop science exhibits. The studies will utilize eye trackers to capture visitors' moment-to-moment attention, facial expression analysis and quantitative field observations to track visitors' emotional states, trace logs generated by exhibit software, as well as motion-tracking sensors and coded video recordings to capture visitors' behavioral interactions. The studies will also use conversation recordings and pre-post assessment measures to capture visitors' science understanding and inquiry processes. With these multimodal data streams as training data, the research team will use probabilistic and neural machine learning techniques to devise learning analytic models of visitor engagement. The project will be conducted by a partnership between North Carolina State University and the North Carolina Museum of Natural Sciences. The research team will 1) design a data-rich multimodal visitor study methodology, 2) create the Visitor Informatics Platform, a suite of open source software tools for multimodal visitor analytics, and 3) launch the Multimodal Visitor Data Warehouse, a curated visitor experience data archive. Together, the multimodal visitor study methodology, the Visitor Informatics Platform, and the Multimodal Visitor Data Warehouse will enable researchers and practitioners in the informal science education community to utilize multimodal learning analytics in their own informal learning environments. It is anticipated that the project will advance the field of informal STEM learning by extending and enriching measures of meaningful visitor engagement, expanding the evidence base for visitor experience design principles, and providing learning analytic tools to support museum educators. By enhancing understanding of the cognitive, affective, and behavioral dynamics underlying visitor experiences in science museums, informal science educators will be well-positioned to design learning experiences that are more effective and engaging. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
James LesterJames MinogueJonathan RoweNorth Carolina Museum of Natural Sciences
The San Diego Natural History Museum (The Nat) contracted RK&A to conduct a summative evaluation of the temporary exhibition, Unshelved: Cool Stuff from Storage. The priorities for the study focused on exploring (1) whether visitors understand that the museum has vast collections (and why); (2) how the low level of interpretation effects visitors’ experiences and understanding of the exhibition; and (3) visitors’ personal connections and emotional response to the exhibition.
How did we approach this study?
RK&A employed two methodologies for this study. First, RK&A conducted open
The Nelson-Atkins Museum of Art recently completed a renovation of its 19th- and early-20th-century European art galleries (the Bloch Galleries). The renovation includes the incorporation of the Marion and Henry Bloch Collection, a major gift of 29 masterworks by leading Impressionist and Post-Impressionist artists. As part of this renovation project, the museum took the opportunity to experiment with new visitor engagement and interpretation strategies, including both digital and print approaches.
The museum identified three desired visitor outcomes for the galleries: 1) Visitors will
Purpose: The Museum of Pop Culture (MoPOP) in Seattle, WA, changed their name from Experience Music Project in 2016. The institution has undergone several changes since its inception and has not conducted a comprehensive visitor study since 2010. As such, MoPOP partnered with the University of Washington Museology Graduate Program to conduct a large-scale visitor study to better understand who MoPOP visitors are, explore their perceptions of pop culture and MoPOP prior to their visit, and the extent to which these perceptions changed due to their museum experience.
Methods: Pre-visit
The Space and Earth Informal STEM Education (SEISE) project, led by the Arizona State University with partners Science Museum of Minnesota, Museum of Science, Boston, and the University of California Berkeley’s Lawrence Hall of Science and Space Sciences Laboratory, is raising the capacity of museums and informal science educators to engage the public in Heliophysics, Earth Science, Planetary Science, and Astrophysics, and their social dimensions through the National Informal STEM Education Network (NISE Net). SEISE will also partner on a network-to-network basis with other existing coalitions and professional associations dedicated to informal and lifelong STEM learning, including the Afterschool Alliance, National Girls Collaborative Project, NASA Museum Alliance, STAR_Net, and members of the Association of Children’s Museums and Association of Science-Technology Centers. The goals for this project include engaging multiple and diverse public audiences in STEM, improving the knowledge and skills of informal educators, and encouraging local partnerships.
In collaboration with the NASA Science Mission Directorate (SMD), SEISE is leveraging NASA subject matter experts (SMEs), SMD assets and data, and existing educational products and online portals to create compelling learning experiences that will be widely use to share the story, science, and adventure of NASA’s scientific explorations of planet Earth, our solar system, and the universe beyond. Collaborative goals include enabling STEM education, improving U.S. scientific literacy, advancing national educational goals, and leveraging science activities through partnerships. Efforts will focus on providing opportunities for learners explore and build skills in the core science and engineering content, skills, and processes related to Earth and space sciences. SEISE is creating hands-on activity toolkits (250-350 toolkits per year over four years), small footprint exhibitions (50 identical copies), and professional development opportunities (including online workshops).
Evaluation for the project will include front-end and formative data to inform the development of products and help with project decision gates, as well as summative data that will allow stakeholders to understand the project’s reach and outcomes.
This position paper, co-authored Center for Childhood Creativity's Director Elizabeth Rood and Director of Research Helen Hadani, details the importance of exposing children ages 0-8 to science, technology, engineering, and math (STEM) experiences. The review of more than 150 empirical studies led Rood and Hadani to conclude that, despite what has been previously thought, modern research supports the understanding that children are capable of abstract thinking and STEM-learning from infancy, beginning before their first birthday.
The Roots of STEM Success, authored in support of classroom
DATE:
TEAM MEMBERS:
Helen Shwe HadaniElizabeth RoodAmy EisenmannRuthe FousheeGarrett JaegerGina JaegerJoanna KauffmannKatie KennedyLisa Regalla
Slides from the January 30, 2018 Webinar present information for preparing proposals for the NSF INCLUDES Alliance Solicitation (NSF 18-529). Includes a brief description of NSF INCLUDES, an explanation of Collaborative Change strategies and the NSF INCLUDES 5 elements of collaborative change, proposal recommendations, details on the NSF cooperative agreements and the NSF Merit Review criteria, and provides useful resources.
This is the solicitation for proposals to the NSF Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to and evidence based understanding of the design and development of STEM learning opportunities for the public in informal environments; provide multiple pathways for broadening access to and engagement in STEM learning experiences; advance innovative research on and assessment of STEM learning in informal environments; and engage the public of all ages in learning STEM in informal environments.
Charles Darwin is largely unknown and poorly understood as a historical figure. Similarly, the fundamental principles of evolution are often miss-stated, misunderstood, or entirely rejected by large numbers of Americans. Simply trying to communicate more facts about Darwin, or facts supporting the principles of evolution is inadequate; neither students nor members of the public will care or retain the information. On the contrary, building facts into a one-on-one conversational narrative creates an memorable opportunity to learn. Here, we create a digital-media, self-guided question and answer
DATE:
TEAM MEMBERS:
David J. LampeBrinley KantorskiJohn Pollock
This article explores science communication from the perspective of those most at risk of exclusion, drawing on ethnographic fieldwork. I conducted five focus groups and 32 interviews with participants from low-income, minority ethnic backgrounds. Using theories of social reproduction and social justice, I argue that participation in science communication is marked by structural inequalities (particularly ethnicity and class) in two ways. First, participants’ involvement in science communication practices was narrow (limited to science media consumption). Second, their experiences of exclusion
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program funds innovative research, approaches, and resources for use in a variety of settings. The proposed project broadens the utility of Public Participation in Scientific Research (PPSR) approaches, which include citizen science, to support new angles in informal learning. It also extends previous work on interactive data visualizations in museums to encompass an element of active contribution to scientific data. To achieve these goals, this project will develop and research U!Scientist (pronounced `You, Scientist!')--a novel approach to using citizen science and learning research-based technology to engage museum visitors in learning about the process of science, shaping attitudes towards science, and science identity development. Through the U!Scientist multi-touch tabletop exhibit, visitors will: (1) interact with scientific data, (2) provide interpretations of data for direct use by scientists, (3) make statements based on evidence, and (4) visualize how their data classifications contribute to globe-spanning research projects. Visitors will also get to experience the process of science, gaining efficacy and confidence through these carefully designed interactions. This project brings together Zooniverse, experts in interactive design and learning based on large data visualizations in museums, and leaders in visitor experience and learning in science museums. Over fifty thousand museum visitors are expected to interact annually with U!Scientist through this effort. This impact will be multiplied by packaging the open-source platform so that others can easily instantiate U!Scientist at their institution.
The U!Scientist exhibit development process will follow rapid iterations of design, implementation, and revision driven by evaluation of experiences with museum visitors. It will involve close collaboration between specialists in computer science, human-computer interaction and educational design, informal science learning experts, and museum practitioners. The summative evaluation will be based on shadowing observations, U!Scientist and Zooniverse.org logfiles (i.e., automated collection of user behavior metrics), and surveys. Three key questions will be addressed through this effort: Q1) Will visitors participate in PPSR activities (via the U!Scientist touch table exhibit) on the museum floor, despite all the distractions and other learning opportunities competing for their attention? If so, who engages, for how long, and in what group configurations? Q2) If visitors do participate, will they re-engage with the content after the museum visit (i.e., continue on to Zooniverse.org)? Q3) Does engaging in PPSR via the touch table exhibit--with or without continued engagement in Zooniverse.org after the museum visit--lead to learning gains, improved understanding of the nature of science, improved attitudes towards science, and/or science identity development?