This is an NSF Postdoctoral Research Fellowship in Biology, under the program Broadening Participation of Groups Under-represented in Biology. The fellow, Robert Habig, is conducting research and receiving training that is increasing the participation of groups underrepresented in biology. The fellow is being mentored by David Lahti at Queens College, City University of New York. The goal of the fellow's project is to perform a comparative evolutionary analysis of nest construction in the weaverbirds (Ploceus spp.). The evolutionary history of behavior can be nearly intractable and resistant to quantitative analysis. One strategy for illuminating our understanding of behavioral evolution is to conduct comparative studies of animal architectures, such as nests. Unlike behaviors themselves, nests persist through time, and have structures that can be disassociated into several quantitative features, which permits easy and comparable measurements and allows scientists to address questions about evolutionary history and functional relevance. The fellow's research addresses two major questions: (1) How do patterns of nest construction vary within and between species? (2) How do interrelated evolutionary processes shape variation in nest structure? This project is important for advancing foundational scientific knowledge, and will be the first study of weavers incorporating both molecular data and nest morphology to better understand the evolutionary underpinnings of a complex behavioral process. The fellow is also broadening participation in science by mentoring students underrepresented in biology.
The Fellow will reconstruct the evolution of nest construction in Ploceus weaverbirds incorporating advanced phylogenetic and morphological techniques including bioinformatics, computer modeling, X-ray computed tomography, and image processing. The Fellow will also conduct fieldwork in two hotspots of weaverbird diversity, the Awash Valley in Ethiopia and the Limpopo Province of South Africa, and collect behavioral data (e.g. rates of predation and brood parasitism; mating and parental behavior) and morphological data (e.g. nest structure) to test hypotheses of how distinct types of evolutionary selection shape the evolution of nest construction. The proposed comparative study can thus address questions such as how rapidly certain nest structural features evolve, which features are ancestral versus derived, which tend to exhibit phylogenetic signal, and which evolve in response to environmental features. The Fellow is receiving training in three-dimensional morphological analyses, phylogenetic tree construction, bioinformatics, computer modeling, and mentoring skills. The plan to broaden participation includes (1) recruitment, training, and mentoring of Queens College students from underrepresented groups in biology; (2) designing an evolutionary biology curriculum that ties in the research of the fellowship; (3) teaching an evolutionary biology class to underrepresented middle and high school students at the American Museum of Natural History; and (4) facilitating a research team for middle school and high school students.
IMPACT NC is a collaboration between the North Carolina science centers and museums and NC State University (NCSU) to build and foster a Community of Practice (CoP) for collective evaluation among the 54 partner organizations across the state of North Carolina. Funded by IMLS Museum Leadership Grant (MG-70-19-0019-19).
The goals of IMPACT NC are:
Identification of a set of shared goals for informal science education across the state.
Development of metrics to assess these goals.
Enhanced capacity of the Community of Practice of science museums to conduct evaluation centered on these collective evaluation goals and metrics.
Improved cohesion among science museums and other partners in NC (e.g. university collaborators, non-profit organizations) as they collectively work toward shared goals.
Development of a system for reporting program outcomes using shared metrics that is integrated into annual reporting or grant proposal processes across NC, thereby informing decision making.
The Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. An ongoing challenge to the design of effective STEM learning exhibits for diverse young children is the absence of reliable and evidence-based resources that designers can apply to the design of STEM exhibits that draw upon play as a child's primary pedagogy, while simultaneously engaging children with STEM content and processes that support development of STEM skills such as observation. To address these challenges, the project team will use a collaborative process in which learning researchers and informal STEM practitioners iteratively develop, design, and test the STEM for Play Framework that could then be applied to the design of STEM-focused exhibits that support play and STEM skill use among early learners.
This Research in Service to Practice project will address these questions: 1) What is a framework for play in early STEM learning that is inclusive of children's cultural influences?; 2) To what extent do interactions between early learners (ages 3-8) and caregivers or peers at exhibits influence the structure and effectiveness of play for supporting STEM skill development?; 3) How do practitioners link play to STEM skill development, and to what extent does a framework for play in early STEM learning assist in identifying types of play that supports early STEM skill development?; and 4) What do practitioners identify as best practices in exhibit design that support the development of STEM skills for early childhood audiences, and conversely, to what extent do practitioners perceive specific aspects of the design as influential to play? The project team will address these questions across four phases of study that will include (a) development of a critical research synthesis to inform the initial STEM for Play framework; (b) the use of surveys, focus groups, and interviews to solicit feedback from practitioners; (c) testing and revising the framework by conducting structured observations of STEM exhibits at multiple museums. The project team will use multiple analytic approaches including qualitative thematic analyses as well as inferential statistics. Results will be disseminated to children?s museums, science centers, and research communities.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
While museums strive to be as inclusive and welcoming as possible to all visitors, data from many institutions shows that audiences are still disproportionately white, well-educated, and more affluent than the average local population. One contributing factor to the lack of progress is that staff often create programs that work to create inclusivity from their own perspective, rather than grounding the work in a broader vision of the museum experience. This project will allow for a deeper exploration of how visitors, particularly those from groups that visit less frequently, experience a museum visit, and how their sense of belonging is supported or eroded during their visit. The team believes this sense is built up or taken away through specific moments of engagement or alienation and will explore these moments that matter through the work. Through intensive work at one museum, and additional work at three other museums, the project will look for themes and insights that can help all museums to create more positive moments that matter for all audiences. Specifically, the project will result in a) insights for museums in supporting a visitor-based sense of belonging, b) shared methods for working with visitors that could be applied by other researchers to explore specifics in a particular setting, and c) grounding work to develop survey questions for use across the field. This award is funded by the Advanced Informal STEM Learning program which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
Building on existing work around exclusion and inclusion in everyday science learning, this project aims to formally define what a sense of belonging means in the science and natural history museum context as a construct for understanding inclusivity. The research team hypothesizes that the majority of experiences in an entire museum visit have a relatively neutral effect on visitor sense of belonging; however, at times, visitors may experience positive or negative moments, and these moments that matter may influence a visitor's STEM engagement, interest, and/or identity. This exploratory work will help to develop and ground the construct of sense of belonging within the museum visitor's experiences, to identify visitor moments that matter using an equity approach that intentionally centers the experiences of visitors from underrepresented groups, and to form the basis for future research that would support the development of a fieldwide measure of sense of belonging. The research study will focus on defining the construct of sense of belonging so it 1) aligns with the research literature and 2) is grounded in the experiences of science/natural history museum visitors. Photovoice data collection method and interviews will be used with visitors ages six and above to identify moments that matter for them during a visit to a science/natural history museum. This project will create new understanding of this construct for not only science/natural history museums and the larger informal science education (ISE) field, but fill a gap in the overall literature around the construct of sense of belonging. The project will also provide new learnings for the ISE field on how to adapt and use the photovoice method to study complex constructs, such as sense of belonging, in science/natural history museums.
This Pilots and Feasibility Studies award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
There is a national need to expand opportunities to learn coding and computational thinking in informal science, technology, engineering, and mathematics (STEM) education. These skills are increasingly needed in STEM disciplines. As young people learn to code, they engage in computational thinking concepts and practices which are problem solving strategies that include repeated process (iterative) design skills. This project promotes innovation by designing and developing activities for tinkering spaces (a space filled with materials for hands-on exploration of STEM) combined with coding in informal learning organizations such as museums, and community centers. The project supports both tinkering and making as methods to meaningfully incorporate computational thinking in STEM learning experiences. The tinkering approach to learning is characterized by hands-on, trial and error engagement. Making is similar to tinkering with additional attention to learning with peer groups. The long-term goal of the project is to enable informal educators to engage in STEM programming with youth and families from underrepresented groups. The project brings together interdisciplinary teams from the Department of Information Science at the University of Colorado Boulder (CU Boulder), the Tinkering Studio at the Exploratorium, and the Lifelong Kindergarten research group at the Massachusetts Institution of Technology. In collaboration with local partner sites, the project team will design and disseminate a collection of six computational tinkering activity areas that engage learners in creative explorations using a combination of physical objects and computational code. The team will develop visual coding "microworlds" for each of the activity areas, specialized sets of coding blocks designed to provide scaffolding. Additionally, the project team will design and develop facilitation guides to document these activities and facilitation strategies, as well as workshops to better support facilitators in making and tinkering spaces.
The project enhances knowledge building through investigations of what instructional supports informal educators need to develop effective facilitation practices that engage underrepresented youth and families in STEM computational learning experiences. Study participants will include informal educators in museum, library, and community-based settings with varying backgrounds and experiences facilitating computing activities. The project team will also engage youth and families from underrepresented groups through collaborative efforts with community-based partners. Research questions include: 1) What challenges and barriers do informal learning educator, face to engage their learners in design-based activities with computing? 2) What supports informal learning educators to take on key facilitation practices that support children and families in computational tinkering activities? 3) In jointly engaging in these computational tinkering activities, how do the activities and informal learning educators? facilitation of these activities impact children's and families' development of computational tinkering and identities as creators and learners with computing? To answer these research questions the project will use qualitative ethnographic methods to study the developing interactions between learners and facilitators at three local sites. Comparative case studies of facilitators across the local partner sites will also be used to examine what supports facilitators to take on key facilitation practices. Data sources will include participant observation of facilitators and families, documentation in the form of photos, videos, and audio recordings, project artifacts, bi-monthly short surveys with reflective prompts, and interviews with facilitators and families.
This award is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
The impacts of changes in the climate at local and global levels threaten how people live. Some frontline communities, especially in historically disenfranchised and under-resourced areas, are particularly vulnerable to the devastating effects of climatological events such as wildfires, flooding, and urban heat islands. As such, there is an urgent need for collective, evidence-based understanding and engagement to prevent and prepare for these potentially fatal events. Led by the Oregon Museum of Science and Industry (OMSI) in Portland, Oregon, in collaboration with local and national partners, Youth Lead the Way is an early-stage Innovations in Development project that offers a theory-based approach for youth in climatologically vulnerable communities to work in climate science research alongside field researchers, develop leadership skills, and engage in timely conversations that impact their own communities. The project will develop and evaluate a Youth Advisory Research Board model to equip and support youth and informal STEM education institutions to conduct evidence-based research on local climate impacts and communicate the findings of their research to their communities. Youth Lead the Way advances the work of several previous NSF-funded projects on climate education, youth advisory boards, and collaborative networks to engage the public in informal STEM learning. Findings from this project will support ongoing efforts in the informal STEM education field to meaningfully engage youth and to more effectively communicate science related to climate and its impacts to the public.
During this initial two-year early-stage project, youth predominantly from racial and ethnic groups underrepresented in STEM will engage in a year-long extended STEM experience. These youth will work collaboratively with scientists and museum professionals to enhance their skills as climate researchers, science communicators, and educational leaders, while reaching an estimated 4,000 or more public audience members through research and events at OMSI, in their schools, and in their communities. Using a cohort model, the youth will conduct scientifically based research studies on various local climate impact topics while concurrently serving in an advisory role at the Oregon Museum of Science and Industry, where they will participate in shaping relevant museum programs and practices. The youth will also develop and present climate stories, a communication approach based on storytelling, to raise public understanding and awareness about local climatological changes and impacts. In addition to the youth component, a companion workshop will be held at the Sciencenter in Ithaca, New York, a partner organization, to train staff and formatively assess the feasibility of scaling the model in other museums. At the program level, an exploratory qualitative research study will be conducted to identify the factors of the overall model that contribute to desired outcomes of youth engagement, climate impact education, and informal science education professional development. Interviews, surveys, focus groups, group chats among youth cohort members, and reviews of artifacts generated by the youth will inform this exploratory study. A theory-based guide outlining key findings, considerations, and recommendations will also be produced. The dissemination of this work will be multi-tiered, reaching thousands within the target communities through public programs, professional networks, at conferences, and a live virtual professional development event hosted by the Association for Science-Technology Centers. If successful, Youth Lead the Way will lay the groundwork for a model that promotes youth and public engagement in STEM through climate science research and identifies promising pathways for future research and similar efforts well beyond this project.
This early-stage Innovations in Development project is funded by the NSF Advancing Informal STEM Learning program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
Parents and adult caregivers play a significant role in young children's understanding of (and participation in) science, technology, engineering, and mathematics (STEM). Research suggests that early engagement with STEM can have a profound impact on children's use of STEM process skills such as exploration, observation, and problem-solving, as well as future academic success. An immediate yet ongoing challenge facing informal STEM learning providers is to understand how limited resources can be used to support effective STEM learning opportunities and experiences for all children and families. Through a collaboration between researchers, Head Start, two science centers (one rural, one urban), and educators, this project aims to foster STEM access and engagement with specific attention to young children and their caregivers. This project is funded by the Advancing Informal STEM Learning (AISL) program which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
This Pilot and Feasibility study will apply an experimental, mixed-methods design to examine parent/caregiver and child (ages 4-5) interactions before, during, and after informal STEM experiences to identify which factors influence children's transfer of learning STEM process skills across multiple informal contexts. Research results will lay the foundation for a future longitudinal study. The project team will ask: (1) What types of parent/caregiver-child engagement at the science center are most predictive of children's application of STEM process skills in subsequent problem-solving tasks and school readiness? (2) How do variations in parent/caregiver-child conversational strategies during the science center visit influence children's memory and learning? and (3) How can informal educators best support Head Start family engagement and children's emerging STEM knowledge? This study will collect data on 240, 4-5-year-old children, with their caregivers, in two different science centers that serve a largely rural and largely urban population. Data sources will include video/audio of caregiver-child interactions at the science centers and at home, as well as children's recall, engagement with a problem-solving task, and school readiness scores. Coding and analysis of the tasks during and after the science center visit will detail mechanisms underlying children's memory, learning, and application of STEM process skills that transfer to the problem-solving task. The project will be implemented by a research-practice partnership, leveraging the expertise of project partners and communities to ensure the use of culturally responsive research practices. This research has the potential to strategically impact how science centers and Head Start grantees work together on Family Engagement programming to achieve equitable STEM learning opportunities, broadening participation for low-income young children and their families.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
For nearly 20 years, the UAB Center for Community OutReach Development (CORD) has conducted SEPA funded research that has greatly enhanced the number of minority students entering the pipeline to college and biomedical careers, e.g., nearly all of CORD’s Summer Research Interns since 1998 (>300) have completed/are completing college and most of them are continuing on to graduate biomedical research and/or clinical training and careers. CORD’s programs that focused on high and middle school students have drawn many minority students into biomedical careers, but a low percentage of minority students benefit from these programs because far too many are already left behind academically in grades 4-6, due, at least in part, to a significant drop in science grades between grades 4 and 6, a drop from which most students never recover. A major contributor to this effect is that most grade 4-6 teachers in predominantly minority schools lack significant formal training in science and often are not fully aware of the great opportunities offered by biomedical careers.
In SEEC II, CORD will deliver intensive inquiry-based science training to grade 4-6 teachers, providing them with science content and hands-on science experiences that will afford their student both content and skills that will make them excited about, and competitive for, the advanced courses needed to move into biomedical research careers. SEEC II will also link teachers together across the elementary/middle school divide and bring the teachers together with administrators and parents, who will experience firsthand the excitement that inquiry learning brings and the significant advancement it provides in science and in reading and math. At monthly meetings and large annual celebrations, the parents, teachers and administrators will learn about the opportunities that biomedical careers can provide for the student who is well prepared. They will also consider the financial and educational steps required to ensure that students have the ability to reach these professions.
SEEC II will also expand CORD’s middle school LabWorks and Summer Science Camps to include grade 4-5 students and provide the teachers with professional learning in informal settings. During summer training, in small groups, the teachers will expand one of the inquiry-based science activities that they complete in the training, and they will use these in their classrooms and communicate with the others in their group to perfect these experiences in the school year. Finally, the teachers and grade 4-5 students will develop science and engineering fair-type research projects with which they will compete both on the school level and at the annual meeting. Thus, the students will share with their parents the excitement that science brings. The Intellectual Merit of SEEC II will be to test a model to enhance grade 4-6 teacher development and vertical alignment, providing science content, exposure to biomedical scientists and training in participatory science experiments, thus positioning teachers to succeed. The Broader Impacts will include the translation and testing of a science education model to assist minority students to avoid the middle school plunge and reach biomedical careers.
The employment demands in STEM fields grew twice as fast as employment in non-STEM fields in the last decade, making it a matter of national importance to educate the next generation about science, engineering and the scientific process. The need to educate students about STEM is particularly pronounced in low-income, rural communities where: i) students may perceive that STEM learning has little relevance to their lives; ii) there are little, if any, STEM-related resources and infrastructure available at their schools or in their immediate areas; and iii) STEM teachers, usually one per school, often teach out of their area expertise, and lack a network from which they can learn and with which they can share experiences. Through the proposed project, middle school teachers in low-income, rural communities will partner with Dartmouth faculty and graduate students and professional science educators at the Montshire Museum of Science to develop sustainable STEM curricular units for their schools. These crosscutting units will include a series of hands-on, investigative, active learning, and standards-aligned lessons based in part on engineering design principles that may be used annually for the betterment of student learning. Once developed and tested in a classroom setting in our four pilot schools, the units will be made available to other partner schools in NH and VT and finally to any school wishing to adopt them. In addition, A STEM rural educator network, through which crosscutting units may be disseminated and teachers may share and support each other, will be created to enhance the teachers’ ability to network, seek advice, share information, etc.
Biology has become a powerful and revolutionary technology, uniquely poised to transform and propel innovation in the near future. The skills, tools, and implications of using living systems to engineer innovative solutions to human health and global challenges, however, are still largely foreign and inaccessible to the general public. The life sciences need new ways of effectively engaging diverse audiences in these complex and powerful fields. Bio-Tinkering Playground will leverage a longtime partnership between the Stanford University Department of Genetics and The Tech Museum of Innovation to explore and develop one such powerful new approach.
The objective of Bio-Tinkering Playground is to create and test a groundbreaking type of museum space: a DIY community biology lab and bio-makerspace, complete with a unique repertoire of hands-on experiences. We will tackle the challenge of developing both open-ended bio-making activities and more scaffolded ones that, together, start to do for biology, biotech, and living systems what today’s makerspaces have done for engineering.
A combined Design Challenge Learning, making, and tinkering approach was chosen because of its demonstrated effectiveness at fostering confidence, creative capacity, and problem solving skills as well as engaging participants of diverse backgrounds. This educational model can potentially better keep pace with the emerging and quickly evolving landscape of biotech to better prepare young people for STEM careers and build the next generation of biotech and biomedical innovators.
Experience development will be conducted using an iterative design process that incorporates prototyping and formative evaluation to land on a final cohort of novel, highly-vetted Bio-Tinkering Playground experience. In the end, the project will generate a wealth of resources and learnings to share with the broader science education field. Thus, the impacts of our foundational work can extend well beyond the walls of The Tech as we enable other educators and public institutions around the world to replicate our model for engagement with biology.
Citizen science is a form of Public Participation in Scientific Research (PPSR) in which the participants are engaged in the scientific process to support research that results in scientifically valid data. Opportunities for participation in real and authentic scientific research have never been larger or broader than they are today. The growing popularity and refinement of PPSR efforts (such as birding and species counting studies orchestrated by the Cornell Lab of Ornithology) have created both an opportunity for science engagement and a need for more research to better implement such projects in order to maximize both benefits to and contributions from the public.
Towards this end, Shirk et al. have posted a design framework for PPSR projects that delineates distinct levels of citizen scientist participation; from the least to the highest level of participation, these categories are contract, contribute, collaborate, co-create, and colleagues. The distinctions among these levels are important to practitioners seeking to design effective citizen science programs as each increase in citizen science participation in the scientific process is hypothesized to have both benefits and obstacles. The literature on citizen science models of PPSR calls for more research on the role that this degree of participation plays in the quality of that participation and related learning outcomes (e.g., Shirk et al., 2012; Bonney et al., 2009). With an unprecedented interest in thoughtfully incorporating citizen science into health-based studies, citizen science practitioners and health researchers first need a better understanding of the role of culture in how different communities approach and perceive participation in health-related studies, the true impact of intended educational efforts from participation, and the role participation in general has on the scientific process and the science outcome.
Project goal to address critical barrier in the field: Establish best practices for use of citizen science in the content area of human health-based research, and better inform the design of future projects in PPSR, both in the Denver Museum of Nature & Science’s Genetics of Taste Lab (Lab), and importantly, in various research and educational settings across the field.
Aims
Understand who currently engages in citizen science projects in order to design strategies to overcome the barriers to participation that occur at each level of the PPSR framework, particularly among audiences underrepresented in STEM.
Significantly advance the current knowledge regarding how citizen scientists engage in, and learn from, and participate in the different levels of the PPSR framework.
Determine the impact that each stage of citizen science participation has on the scientific process.
Informal learning institutions, such as science centers and museums, are well-positioned to broaden participation in engineering pathways by providing children from underrepresented groups with motivational, self-directed engineering design experiences. Though many informal learning institutions offer opportunities for young visitors to engage in engineering activities, little is known about the specific features of these activities that support children's motivation in engineering design processes such as problem scoping, testing, and iteration. This project will address this gap and advance foundational knowledge by identifying features of engineering design activities, as implemented within an informal setting, which support underrepresented children's engineering motivation and persistence in engineering tasks. Researchers at New York Hall of Science (NYSCI) will observe children interacting with families and museum educators as they engage in different engineering design activities in NYSCI's Design Lab, an exhibition space devoted to hands-on exploration of engineering design. They will also survey and interview the children and their caregivers about these experiences. Analyses of these data sources will result in a description of features of design activities foster motivation and task persistence in engineering design. Findings will be disseminated nationally to other informal learning institutions, which in turn can use the knowledge generated from this project to create motivational, research-based, field-tested engineering design experiences for young visitors, especially for children from underrepresented groups. The experiences may encourage children to further pursue engineering pathways, resulting in a diversified engineering workforce with the potential to drive and sustain national innovation and global technological leadership.
This project uses the framework of goal orientation, defined as learners' self-reflection of why and how they engage in tasks, to understand whether, how, and why underrepresented 7-12-year-olds engage in engineering design activities in an informal learning institution. Though previous research has suggested that goal orientation is strongly, positively related to learning and motivation in formal settings such as schools, research in informal settings has not robustly accounted for the role of goal orientation in participants' engagement with learning tasks in these unique learning environments. To better understand how children's goal orientations contribute to their motivation in engineering in informal learning institutions, researchers will answer the following research questions: (1) What are underrepresented children's goals and goal orientations while participating in engineering design activities in an informal setting? (2) What contextual factors--including facilitation strategies, materials, task relevance, and social interactions with family members--may support or discourage the adoption of different goal orientations? (3) How do goal orientations relate to children's learning experience in the engineering design activities and the likelihood that they will test and iterate their solutions? These questions will be answered through a mixed-method research study conducted with approximately 200 families, with children aged 7-12, recruited from underrepresented groups. Semi-structured clinical interviews, conducted with 20% of the children and their caregivers, as well as observations and surveys gathered from all families, will provide information on the children's goal orientation and engagement as they relate to specific engineering design activities. Qualitative content analyses and multilevel structural equation modeling will result in findings that will be disseminated widely to other institutions of informal learning. Ultimately, this project will generate new empirical knowledge regarding the features of engineering design activities in informal learning environments that increase engineering engagement and motivation among underrepresented children, thereby broadening participation in engineering pathways.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.