This is the final evaluation report from RMC Research Corp. for the PES@LTERs project. Appendix includes instruments.
RMC Research designed evaluation activities to provide formative and summative feedback to Harvard Forest and the Hubbard Brook Research Foundation (Hubbard Brook) on their plan to embed public engagement with science (PES) into the cultures and practices of Long-Term Ecological Research Sites (LTERs) in the northeastern US. The purpose of this project was to build PES mechanisms into long-term ecosystem studies that create on-going, open exchanges between scientists and
The University of Washington, the Exploratorium, the Education Development Center, Inverness Research, and the University of Colorado - Boulder have come together to form a Research+Practice (R+P) Collaboratory. The Collaboratory seeks to address and reframe the gap between research and practice in K-12 STEM education. This gap persists despite decades of work by many leading organizations, associations, and individuals. Attempts to close the gap have generally focused on creating resources and mechanisms that first explain or illustrate "what research says" and then invite educators to access and integrate findings into practice. Recently, however, attention has turned to the ways in which the medical sciences are addressing the gap between research and clinical practice through the developing field of "translational research." In medicine, the strategy has been to shift the focus from adoption to adaptation of research into practice. Implicit in the notion of adaptation is a bi-directional process of cultural exchange in which both researchers and practitioners come to understand how the knowledge products of each field can strengthen the professional activities in the other. Along these lines, the R+P Collaboratory is working with leading professional associations and STEM improvement efforts to leverage their existing knowledge and experience and to build sustainable strategies for closing the gap. Activities include:
Collecting, creating and synthesizing translational research resources to expand STEM educators' and educational leaders' access and awareness to current relevant research.
Supporting multiple opportunities for cross-sector (research and practice; education and social sciences; formal and informal) meetings to foster critical engagement and cultural exchange.
Testing, documenting and innovating new resources and mechanisms at Adaptation Sites and disseminating both products and results through the R+P Resource Center.
The R+P Collaboratory is developing an online 'Go-To' Resource Center website that houses the resources collected, created, and curated by the Collaboratory. The Resource Center also has significant 'Take-Out' features, with all materials meta-tagged so that they can be automatically uploaded, reformatted, and integrated into the existing communication and professional development mechanisms (e.g., newsletters, digests, conferences, and websites) of a dozen leading professional associations within a Professional Association Partner Network.
In light of new and emerging standards in the STEM disciplines, the Collaboratory is focusing its work on four salient and timely bodies of research: (a) STEM Practices, (b) Formative Assessment, (c) Cyberlearning, and (d) Learning as a Cross-Setting Phenomenon. Special emphasis is being placed on research and practice that focuses on the learning of children and youth from communities historically underrepresented in STEM fields.
The work of the R+P Collaboratory includes research and evaluation of its own efforts through studies aimed at answering the following questions:
How are Collaboratory resources and engagement activities accessed, experienced and leveraged by participants?
What resources, mechanisms and learning contexts support cultural exchange among STEM education researchers and practitioners?
What new kinds of practices result when research-based evidence is adapted into evidence-based practices, and how does it change learning opportunities for K-12 aged children?
How can effective strategies, mechanisms and resources of the Collaboratory be scaled and adapted to new contexts?
This project supports the Broader Impacts and Outreach Network for Institutional Collaboration (BIONIC), a national Research Coordination Network of Broader Impacts to support professionals who assist researchers to design, implement, and evaluate the Broader Impacts activities for NSF proposals and awards. All NSF proposals are evaluated not only on the Intellectual Merit of the proposed research, but also on the Broader Impacts of the proposed work, such as societal relevance, educational outreach, and community engagement. Many institutions have begun employing Broader Impacts support professionals, but in most cases, these individuals have not worked as a group to identify and share best practices. As a consequence, there has been much duplication of effort. Through coordination, BIONIC is expected to improve efficiency, reduce redundancy, and have significant impact in several areas: 1) Researchers will benefit from an increased understanding of the Broader Impacts merit review criterion and increased access to collaborators who can help them design, implement, and evaluate their Broader Impacts activities; 2) Institutions and research centers will increase their capacity to support Broader Impacts via mentoring for Broader Impacts professionals and consulting on how to build Broader Impacts support infrastructure, with attention to inclusion of non-research-intensive universities, Historically Black Colleges and Universities, and Hispanic- and Minority-Serving Institutions that may not have the resources to support an institutional Broader Impacts office; and 3) NSF, itself, will benefit from a systematic and consistent approach to Broader Impacts that will lead to better fulfillment of the Broader Impacts criterion by researchers, better evaluation of Broader Impacts activities by reviewers and program officers, and a system for evaluating the effectiveness of Broader Impacts activities in the aggregate, as mandated by Congress and the National Science Board. Through its many planned activities, BIONIC will ultimately help advance the societal aims that the Broader Impacts merit review criterion was meant to achieve.
The main goals of the project will be accomplished through the four specific objectives: 1) Identify and curate promising models, practices, and evaluation methods for the Broader Impacts community; 2) Expand engagement in, and support the development of, high-quality Broader Impacts activities by educating current and future faculty and researchers on effective practices; 3) Develop the human resources necessary for sustained growth and increased diversity of the Broader Impacts community; and 4) Promote cross-institutional collaboration and dissemination for Broader Impacts programs, practices, models, materials, and resources. BIONIC will facilitate collaborative Broader Impacts work across institutions, help leverage previously developed resources, support professional development, and train new colleagues to enter into the Broader Impacts field. This project will improve the quality and sustainability of Broader Impacts investments, as researchers continue to create unique and effective activities that are curated and broadly disseminated. BIONIC will create a network designed to assist NSF-funded researchers at their institutions in achieving the goals of the Broader Impacts Review Criterion. In so doing, BIONIC will promote Broader Impacts activities locally, nationally, and internationally and help to advance the Broader Impacts field.
This award is co-funded by the Divisions of Molecular and Cellular Biosciences and Emerging Frontiers in the Directorate for Biological Sciences and by the Division of Chemistry in the Directorate for Mathematics and Physical Sciences.
Brokering Youth Pathways was created to share tools and techniques around the youth development practice of “brokering” or connecting youth to future learning opportunities and resources.
This toolkit shares ways in which various out-of-school educators and professionals have approached the challenge of brokering. It provides a framework, practice briefs and reports that focus on a particular issue or challenge and provide concrete examples, as well as illustrate how project partners partners worked through designing new brokering routines in partnership with a research team.
This is a report from Workshop II: Overview of Organizational Support for Public Engagement among Scientific Societies, part of the Support Systems for Scientists' Communication and Engagement workshop series. Workshop II was held Feb. 28 - Mar. 1, 2018 at the Howard Hughes Medical Institute in Chevy Chase, MD. The report provides an overview of a project aiming to help reveal how societies view the concept of “public engagement” and to provide an empirical understanding of the availability and prevalence of tools that societies currently use to support engagement efforts.
DATE:
TEAM MEMBERS:
Shupei YuanAnthony DudoJohn BesleyBrooke Smith
resourceprojectWebsites, Mobile Apps, and Online Media
The American Association for the Advancement of Science (AAAS) is creating, implementing and evaluating a forum for the NSF INCLUDES broadening participation community of practice and for engaging the NSF INCLUDES awardees and science, technology, engineering and mathematics (STEM) researchers across the nation to expand the NSF INCLUDES broadening participation network. The NSF INCLUDES program is a comprehensive national initiative designed to enhance U.S. leadership in STEM discoveries and innovations focused on NSF's commitment to diversity, inclusion, and broadening participation in these fields.
The NSF INCLUDES Open Forum will use the AAAS Trellis networking platform and the organization's experience engaging communities of practice focused on broadening participation, STEM education and STEM research. The project builds on the success of a prior NSF INCLUDES Conference award (HRD-1650509) that was addressing goals to define networking needs of the first round of NSF INCLUDES Design and Development Launch Pilots (DDLP); to develop design specifications for NSF INCLUDES networking, curating of resources, and supporting communities of practice; and to propose tools, techniques, capacities and functionalities for an NSF INCLUDES national network.
The NSF INCLUDES Open Forum project includes advisory board members with expertise in networking platforms and others with broadening participation knowledge and experience. A yearly conference for NSF INCLUDES awardees will offer participants an opportunity to learn about how Trellis platform upgrades, functionality and technology options (e.g., a smartphone application) can be used in new ways to engage a broader community of partners interested in broadening participation in STEM research and education contexts. An external evaluator will assess the activities and outcomes of the NSF INCLUDES Open Forum both during implementation and at project end. The PIs will also communicate the outcomes of the project to broader audiences, both academic and non-academic, and encourage a dialogue within the NSF INCLUDES community about the use of technology for organization and communication within a network.
Underrepresented minorities (URMs) are less than 10% of engineering faculty, despite comprising nearly a third of the nation's population. A common explanation for their disproportionate representation, at the engineering faculty level, is related to a lack of access to effective mentorship from other faculty. This NSF INCLUDES Design and Development Launch Pilot project will expand a new mentoring and advocacy-networking paradigm to bring together two stakeholder groups: (1) underrepresented minorities (URMs) who are engineering faculty and (2) well-regarded (primarily non-URM) emeriti/retired engineering faculty. A previously-funded NSF project found that this mentor-mentee pairing was viewed favorable by both parties and beneficial, particularly by the URM engineering faculty. Because of these results, the investigators proposed to scale, test, and evaluate the approach on a broader scale by creating national infrastructural network partners to help increase capacity to serve a greater number of URM engineering faculty and to introduce tele-mentoring and training models to serve URM faculty who work in remote geographical locations with very little access to mentors.
The project will use a multi-phased phenomenological, mixed method research design to gain greater understanding of the ways in which the URM faculty and emeriti faculty experience the opportunities afforded by the project. Further, the investigators plan to collect data to examine how project participants perceive and experience conventional, direct communications (e.g., telephone calls, e-mail, and in-person meetings)through the mentoring process versus the use of Embodied Conversational Agents (ECAs), anthropomorphic interface agents that engage a user in real-time dialogue by using verbal-nonverbal channels to emulate the in-person experience. This project has the potential to broaden participation in the engineering professoriate and opens up new possibilities for supporting URM engineering faculty.
DATE:
-
TEAM MEMBERS:
Comas HaynesValerie ConleySylvia MendezKinnis GoshaRosario Gerhardt
resourceprojectProfessional Development, Conferences, and Networks
The National Alliance for Partnerships in Equity Education Foundation (NAPE) will partner with a diverse group of organizations from six states (CO, ID, NM, NV, UT, and WY) to form the Intermountain STEM (IM STEM) project, an NSF INCLUDES Design and Development Launch Pilot project focused on the goal of increasing the participation and closing achievement gaps in STEM education, including career and technical education. These organizations whose programs impact the formal STEM education system include: Departments of Education; Higher Education agencies; governor supported STEM Action Centers; universities; secondary school districts; community colleges; Department of Energy National Labs; businesses, non-profits and others. The partners in this effort will identify effective practices focused on the common set of objectives and create a model to bring them to scale by employing a collective impact approach. Through the project, the participating organizations will create a common agenda, identify shared metrics, implement mutually reinforcing activities, and maintain continuous communication. This effort addresses directly the lack of diversity of the STEM workforce; a societal challenge of significant magnitude because of its impact on innovation, national security, environmental safety, and income inequality in the US. The IM STEM?s mission to increase the diversity of students who are successful in STEM education will create a more STEM literate society, ensure the contributions of a diverse STEM workforce, and level the playing field for entrance into high wage STEM careers.
The capacity of IM STEM to bring large well-resourced organizations to bear on the broadening participation challenges in STEM will advance the knowledge of how creative social innovations, like collective impact, can create transformative institutional and cultural change. The collection, evaluation and scaling of effective research-based solutions to close equity gaps in STEM will advance inclusion in STEM. Initially, the IM STEM project will pilot the scaling of NAPE's professional development (PD) programs - Program Improvement Process for Equity and Micromessaging to Reach and Teach Every Student - that have proven to impact equity gaps in STEM and career and technical education (CTE). The six participating states are interested in scaling their current small scale implementation of NAPE's PD programs and will also incorporate selected emerging practices. This design and development launch pilot will provide the vehicle for identifying support mechanisms for scaling of the PD and the identification of additional scaling opportunities with other effective practices of the participating partners. These efforts have the potential to develop a model for expansion to other states wanting to scale effective practices.
DATE:
-
TEAM MEMBERS:
Mimi LufkinAlexander CarterAngela HemingwayAnne JakleSusan Thackeray
In this NSF INCLUDES Design and Development Launch Pilot the institutions of "Building on Strengths" propose to build and pilot the infrastructure, induction process, and early implementation of the Mathematician Affiliates of Color network. This network will consist of mathematicians of color from across academia and industry who want to invest time in, share their expertise with, and learn from students of color and their teachers. Building on Strengths will draw on basic needs cognitive theory to support these interactions and will focus narrowly on short and moderate term collaborations (from one month to a semester) between visiting mathematicians, students, and collaborating teachers that will involve three specific types of interactions: doing mathematics together as a habits-of-mind practice, talking about the discipline of mathematics and the experiences of mathematicians of color in that discipline, and relationship-building activities. The foundational infrastructure developed in the project will include systems for recruitment, selection and induction, a process for pairing affiliate mathematicians with classrooms, and support structures for the collaborations. To support the goals of the network a prototype virtual space will be developed in which real-time artifacts can be collected and shared from the classroom interactions. While Building on Strengths will pilot this program in the secondary context, once a viable model is established, scaling to K-16, as well as to other STEM fields, will be possible.
The research study in the project uses an exploratory sequential mixed-methods design and will be conducted in two phases. In the first, quantitative, phase of the study the following questions will be addressed: (1) Is the teacher-mathematician collaboration associated with a change for students in perception of basic human needs being met, mathematical or racial identities, or beliefs about mathematics or who can do mathematics? (2) Is the teacher-mathematician collaboration associated with a change for adults in perceptions of the role of basic needs or in adults' identities or beliefs about mathematics or who can do mathematics? In the second, qualitative, phase of the study, two types of interactions will be selected for in-depth qualitative study, identifying cases where groups of students experienced changes in their needs, identity, and beliefs. In this qualitative case-centered phase, the following questions will be explored: (1) What is the nature of the mentor-student interaction? (2) What aspects of the intervention do students feel are most relevant to them? (3) How did the implementation of the intervention differ from the anticipated intervention? The results of the study will help improve the infrastructure for, and better support the interactions between, mathematicians of color, students of color and their mathematics teachers; the outcomes will also shed light on how students experience their interactions.
DATE:
-
TEAM MEMBERS:
Michael YoungMaisha MosesAlbert CuocoEden Badertscher
Physical science and engineering remain the least diverse of all STEM fields---with regard to women, underrepresented minorities, and persons with disabilities---across all levels of STEM education and training. SCI-STEPS is an NSF INCLUDES Design and Development Launch Pilot that will address this persistent challenge by developing a complete end-to-end pipeline (or system of pathways) from the beginning of college to the PhD, and then into the workforce. Many isolated efforts to broaden participation have shown promise, but they have not produced big enough impact. SCI-STEPS represents a concerted set of coordinated interventions---consciously facilitated, systemically linked, and purposefully disseminated. SCI-STEPS represents a broad regional network among major research universities, Historically Black Colleges and Universities, comprehensive universities, community colleges, national labs, and major scientific organizations. The goal of the network is to ensure that underrepresented individuals in the physical sciences and engineering can get from their starting point in STEM higher education---freshmen at 2-year or 4-year college---through the higher education pathways leading to an appropriate terminal degree and employment in the STEM workforce.
Women, underrepresented minorities, and persons with disabilities collectively represent the majority of college-age individuals entering higher education with an expressed interest in physical science and engineering. A growing body of research indicates that academic and social integration may be even more influential than academic abilities for retention of students. Thus, interventions aimed at stemming the losses of these individuals must ultimately be aimed at changing the system---including unwelcoming institutional climates, racial/ethnic/gender stereotyping, a lack of mentors with whom to identify, and evaluation methods that emphasize conformity over individual capabilities---rather than changing the individual. The SCI-STEPS pilot focuses effort on institutional readiness for implementation of best practice interventions at four key junctures: (i) college freshman to sophomore; (ii) undergraduate to graduate; (iii) PhD to postdoc; and (iv) postdoc to workforce.The pilot will proceed in three steps: (1) a planning phase, (2) development of an initial end-to-end pathways model with four Juncture Transition teams, and (3) scale-up of the SCI-STEPS "network of networks" with all initial partners. By addressing these objectives through a collective impact framework and embedded research, this pilot will demonstrate how best-practice interventions at each pathway juncture can be dovetailed and scaled up across a broad range of institutional types and across a large but distinct geographical area. Addressing these objectives will thus also serve to advance Broadening Participation efforts at a national scale, by suggesting the forms of institutional partnerships and best-practices that may inform other alliances in other STEM disciplines and/or different regional areas.
DATE:
-
TEAM MEMBERS:
Keivan StassunNicole JosephKelly Holley-BockelmannWilliam RobinsonRoger Chalkley
resourceprojectProfessional Development, Conferences, and Networks
Aligning for Impact: Computer Science Pathways Across Contexts [CS-PAC] is an NSF INCLUDES Design and Development Launch Pilot. It broadens participation of students who are underrepresented in computer science by using the convening and policy-making power of the Georgia State Department of Education to coalesce school district leaders to implement K-12 computer science education. The project provides a national model for how to work toward systemic change. With the State Department of Education's coordination, several school districts will collaboratively seek improvements in their own student participation rates. The coordination of data reporting and analysis, resources, communications, and policy promote more equitable participation in computer science education. Research emerging from this project informs other states about how to collaboratively shape computer science education policy and policy implementation.
Using a Collective Impact approach to systemic change, the project creates sustainable institutional change at the community, state, and national levels. Qualitative and quantitative data provide descriptions about how to utilize alignment strategies within Collective Impact in three different contexts: rural, suburban, and urban. Outcomes utilize a regression discontinuity analysis to justify successful implementation as well as qualitative analysis of implementation efforts that were deemed most effective by all stakeholders. The project outputs directly affect over 88,000 students across five districts and indirectly affect over 1.7 million in Georgia alone. The culminating project goal is the development of a coherent framework for aligning K-12 computer science education pathways.
DATE:
-
TEAM MEMBERS:
Caitlin DooleyBryan CoxShawn Utley
resourceprojectProfessional Development, Conferences, and Networks
EvaluATE is a national resource center dedicated to supporting and improving the evaluation practices of approximately 250 ATE grantees across the country. EvaluATE conducts webinars and workshops, publishes a quarterly newsletter, maintains a website with a digital resource library, develops materials to guide evaluation work, and conducts an annual survey of ATE grantees. EvaluATE's mission is to promote the goals of the ATE program by partnering with projects and centers to strengthen the program's evaluation knowledge base, expand the use of exemplary evaluation practices, and support the continuous improvement of technician education throughout the nation. EvaluATE's goals associated with this proposal are to: (1) Ensure that all ATE Principal Investigators and evaluators know the essential elements of a credible and useful evaluation; (2) Maintain a comprehensive collection of online resources for ATE evaluation; (3) Strengthen and expand the network of ATE evaluation stakeholders; and (4) Gather, synthesize, and disseminate data about the ATE program activities to advance knowledge about ATE/technician education. The Center plans to produce a comprehensive set of evaluation resources to complement other services, engaging several community college-based Principal Investigators and evaluators in that process.
EvaluATE's products are informed by current research on evaluation, the National Science Foundation's priorities for the evaluation of ATE grants, and the needs of ATE PIs and evaluators for sound guidance that is immediately relevant and usable in their contexts. The fundamental nature of EvaluATE's work is geared toward supporting ATE grantees to use evaluation regularly to improve their work and demonstrate their impacts. All of EvaluATE's products are available to the public. EvaluATE's findings from the annual survey of ATE grantees aid in advancing understanding of the status of technician education and illuminate areas for additional research. The new survey investigates ATE grantees' work to serve underrepresented and special populations, including women, people of color, and veterans. Survey data are available upon request for research and evaluation purposes.
DATE:
-
TEAM MEMBERS:
Lori WingateArlen GullicksonEmma PerkKelly RobertsonLyssa Becho