This is the final evaluation report from RMC Research Corp. for the PES@LTERs project. Appendix includes instruments.
RMC Research designed evaluation activities to provide formative and summative feedback to Harvard Forest and the Hubbard Brook Research Foundation (Hubbard Brook) on their plan to embed public engagement with science (PES) into the cultures and practices of Long-Term Ecological Research Sites (LTERs) in the northeastern US. The purpose of this project was to build PES mechanisms into long-term ecosystem studies that create on-going, open exchanges between scientists and
NASA@ My Library is made possible through the support of the National Aeronautics and Space Administration (NASA) Science Mission Directorate as part of its Science Activation program. The project is led by the National Center for Interactive Learning (NCIL) at the Space Science Institute (SSI) in partnership with the American Library Association (ALA) Public Programs Office, Lunar and Planetary Institute (LPI), and Education Development Center (EDC). From 2016-2020, 78 public libraries (75 partner libraries and 3 pilot libraries), 18 State Library Agencies, 6 Portal to the Public Network sites, and 30 NASA-funded scientists participated in the project. More than 225,000 library patrons were reached through their efforts.
In 2021-2022, public libraries, universities, and state library agencies will participate in the project to increase and enhance STEAM learning opportunities in their communities, with an emphasis on reaching audiences underrepresented in STEM education and professions.
The University of Washington, the Exploratorium, the Education Development Center, Inverness Research, and the University of Colorado - Boulder have come together to form a Research+Practice (R+P) Collaboratory. The Collaboratory seeks to address and reframe the gap between research and practice in K-12 STEM education. This gap persists despite decades of work by many leading organizations, associations, and individuals. Attempts to close the gap have generally focused on creating resources and mechanisms that first explain or illustrate "what research says" and then invite educators to access and integrate findings into practice. Recently, however, attention has turned to the ways in which the medical sciences are addressing the gap between research and clinical practice through the developing field of "translational research." In medicine, the strategy has been to shift the focus from adoption to adaptation of research into practice. Implicit in the notion of adaptation is a bi-directional process of cultural exchange in which both researchers and practitioners come to understand how the knowledge products of each field can strengthen the professional activities in the other. Along these lines, the R+P Collaboratory is working with leading professional associations and STEM improvement efforts to leverage their existing knowledge and experience and to build sustainable strategies for closing the gap. Activities include:
Collecting, creating and synthesizing translational research resources to expand STEM educators' and educational leaders' access and awareness to current relevant research.
Supporting multiple opportunities for cross-sector (research and practice; education and social sciences; formal and informal) meetings to foster critical engagement and cultural exchange.
Testing, documenting and innovating new resources and mechanisms at Adaptation Sites and disseminating both products and results through the R+P Resource Center.
The R+P Collaboratory is developing an online 'Go-To' Resource Center website that houses the resources collected, created, and curated by the Collaboratory. The Resource Center also has significant 'Take-Out' features, with all materials meta-tagged so that they can be automatically uploaded, reformatted, and integrated into the existing communication and professional development mechanisms (e.g., newsletters, digests, conferences, and websites) of a dozen leading professional associations within a Professional Association Partner Network.
In light of new and emerging standards in the STEM disciplines, the Collaboratory is focusing its work on four salient and timely bodies of research: (a) STEM Practices, (b) Formative Assessment, (c) Cyberlearning, and (d) Learning as a Cross-Setting Phenomenon. Special emphasis is being placed on research and practice that focuses on the learning of children and youth from communities historically underrepresented in STEM fields.
The work of the R+P Collaboratory includes research and evaluation of its own efforts through studies aimed at answering the following questions:
How are Collaboratory resources and engagement activities accessed, experienced and leveraged by participants?
What resources, mechanisms and learning contexts support cultural exchange among STEM education researchers and practitioners?
What new kinds of practices result when research-based evidence is adapted into evidence-based practices, and how does it change learning opportunities for K-12 aged children?
How can effective strategies, mechanisms and resources of the Collaboratory be scaled and adapted to new contexts?
This study researched whether and how affiliation with the Nanoscale Informal Science Education Network (NISE Net) led to change in informal science education organizations’ (ISEs) practices. The NISE Net provided an opportunity to look at how participation in a large but loosely-structured network of museums, science centers, educators, and scientists can influence museums to experience organizational change and adopt new practices.
By conducting qualitative case studies of a few selected partners, this research aimed to understand the conditions that facilitate or impede the influence of
Brokering Youth Pathways was created to share tools and techniques around the youth development practice of “brokering” or connecting youth to future learning opportunities and resources.
This toolkit shares ways in which various out-of-school educators and professionals have approached the challenge of brokering. It provides a framework, practice briefs and reports that focus on a particular issue or challenge and provide concrete examples, as well as illustrate how project partners partners worked through designing new brokering routines in partnership with a research team.
This is a report from Workshop II: Overview of Organizational Support for Public Engagement among Scientific Societies, part of the Support Systems for Scientists' Communication and Engagement workshop series. Workshop II was held Feb. 28 - Mar. 1, 2018 at the Howard Hughes Medical Institute in Chevy Chase, MD. The report provides an overview of a project aiming to help reveal how societies view the concept of “public engagement” and to provide an empirical understanding of the availability and prevalence of tools that societies currently use to support engagement efforts.
DATE:
TEAM MEMBERS:
Shupei YuanAnthony DudoJohn BesleyBrooke Smith
resourceprojectWebsites, Mobile Apps, and Online Media
The American Association for the Advancement of Science (AAAS) is creating, implementing and evaluating a forum for the NSF INCLUDES broadening participation community of practice and for engaging the NSF INCLUDES awardees and science, technology, engineering and mathematics (STEM) researchers across the nation to expand the NSF INCLUDES broadening participation network. The NSF INCLUDES program is a comprehensive national initiative designed to enhance U.S. leadership in STEM discoveries and innovations focused on NSF's commitment to diversity, inclusion, and broadening participation in these fields.
The NSF INCLUDES Open Forum will use the AAAS Trellis networking platform and the organization's experience engaging communities of practice focused on broadening participation, STEM education and STEM research. The project builds on the success of a prior NSF INCLUDES Conference award (HRD-1650509) that was addressing goals to define networking needs of the first round of NSF INCLUDES Design and Development Launch Pilots (DDLP); to develop design specifications for NSF INCLUDES networking, curating of resources, and supporting communities of practice; and to propose tools, techniques, capacities and functionalities for an NSF INCLUDES national network.
The NSF INCLUDES Open Forum project includes advisory board members with expertise in networking platforms and others with broadening participation knowledge and experience. A yearly conference for NSF INCLUDES awardees will offer participants an opportunity to learn about how Trellis platform upgrades, functionality and technology options (e.g., a smartphone application) can be used in new ways to engage a broader community of partners interested in broadening participation in STEM research and education contexts. An external evaluator will assess the activities and outcomes of the NSF INCLUDES Open Forum both during implementation and at project end. The PIs will also communicate the outcomes of the project to broader audiences, both academic and non-academic, and encourage a dialogue within the NSF INCLUDES community about the use of technology for organization and communication within a network.
Underrepresented minorities (URMs) are less than 10% of engineering faculty, despite comprising nearly a third of the nation's population. A common explanation for their disproportionate representation, at the engineering faculty level, is related to a lack of access to effective mentorship from other faculty. This NSF INCLUDES Design and Development Launch Pilot project will expand a new mentoring and advocacy-networking paradigm to bring together two stakeholder groups: (1) underrepresented minorities (URMs) who are engineering faculty and (2) well-regarded (primarily non-URM) emeriti/retired engineering faculty. A previously-funded NSF project found that this mentor-mentee pairing was viewed favorable by both parties and beneficial, particularly by the URM engineering faculty. Because of these results, the investigators proposed to scale, test, and evaluate the approach on a broader scale by creating national infrastructural network partners to help increase capacity to serve a greater number of URM engineering faculty and to introduce tele-mentoring and training models to serve URM faculty who work in remote geographical locations with very little access to mentors.
The project will use a multi-phased phenomenological, mixed method research design to gain greater understanding of the ways in which the URM faculty and emeriti faculty experience the opportunities afforded by the project. Further, the investigators plan to collect data to examine how project participants perceive and experience conventional, direct communications (e.g., telephone calls, e-mail, and in-person meetings)through the mentoring process versus the use of Embodied Conversational Agents (ECAs), anthropomorphic interface agents that engage a user in real-time dialogue by using verbal-nonverbal channels to emulate the in-person experience. This project has the potential to broaden participation in the engineering professoriate and opens up new possibilities for supporting URM engineering faculty.
DATE:
-
TEAM MEMBERS:
Comas HaynesValerie ConleySylvia MendezKinnis GoshaRosario Gerhardt
In this NSF INCLUDES Design and Development Launch Pilot the institutions of "Building on Strengths" propose to build and pilot the infrastructure, induction process, and early implementation of the Mathematician Affiliates of Color network. This network will consist of mathematicians of color from across academia and industry who want to invest time in, share their expertise with, and learn from students of color and their teachers. Building on Strengths will draw on basic needs cognitive theory to support these interactions and will focus narrowly on short and moderate term collaborations (from one month to a semester) between visiting mathematicians, students, and collaborating teachers that will involve three specific types of interactions: doing mathematics together as a habits-of-mind practice, talking about the discipline of mathematics and the experiences of mathematicians of color in that discipline, and relationship-building activities. The foundational infrastructure developed in the project will include systems for recruitment, selection and induction, a process for pairing affiliate mathematicians with classrooms, and support structures for the collaborations. To support the goals of the network a prototype virtual space will be developed in which real-time artifacts can be collected and shared from the classroom interactions. While Building on Strengths will pilot this program in the secondary context, once a viable model is established, scaling to K-16, as well as to other STEM fields, will be possible.
The research study in the project uses an exploratory sequential mixed-methods design and will be conducted in two phases. In the first, quantitative, phase of the study the following questions will be addressed: (1) Is the teacher-mathematician collaboration associated with a change for students in perception of basic human needs being met, mathematical or racial identities, or beliefs about mathematics or who can do mathematics? (2) Is the teacher-mathematician collaboration associated with a change for adults in perceptions of the role of basic needs or in adults' identities or beliefs about mathematics or who can do mathematics? In the second, qualitative, phase of the study, two types of interactions will be selected for in-depth qualitative study, identifying cases where groups of students experienced changes in their needs, identity, and beliefs. In this qualitative case-centered phase, the following questions will be explored: (1) What is the nature of the mentor-student interaction? (2) What aspects of the intervention do students feel are most relevant to them? (3) How did the implementation of the intervention differ from the anticipated intervention? The results of the study will help improve the infrastructure for, and better support the interactions between, mathematicians of color, students of color and their mathematics teachers; the outcomes will also shed light on how students experience their interactions.
DATE:
-
TEAM MEMBERS:
Michael YoungMaisha MosesAlbert CuocoEden Badertscher
resourceprojectProfessional Development and Workshops
This is an "Inclusion across the Nation of Communities of Learners of Underrepresented Discoverers in Engineering and Science" (INCLUDES) Design and Development Launch Pilot that will implement a plan to assess the feasibility of a strategy designed to ensure high levels of improvement in K-12 grade students' mathematics achievement. The plan will focus on an often-neglected group of students--those who have been performing at the lowest quartile on state tests of mathematics, including African American, Hispanic, Native American, students with disabilities, and those segregated in urban and rural communities across the country. The project will draw on lessons learned from the nation's Civil Rights Movement and a community-organizing strategy learned during the struggle to achieve voting rights for African Americans. The Algebra Project (AP) is a national, nonprofit organization that uses mathematics as an organizing tool to ensure quality public school education for every child in America; it believes that every child has a right to a quality education to succeed in this technology-based society. AP's unique approach to school reform intentionally develops sustainable, student-centered models by building coalitions of stakeholders within the local communities, particularly the historically underserved populations. The AP works to change the deeply rooted social attitudes that encourage the disenfranchisement of a third of the nation's population. It delivers a multi-pronged approach to build demand for and support of quality public schools, including research and development, school development, and community development education reform efforts through K-12 initiatives.
The Algebra Project and the Young People's Project (YPP) will join efforts to bring together over 70 individuals and organizations, including 17 universities of which 8 are Historical Black Colleges and Universities, school districts, mathematics educators, and researchers to examine their experiences, and use collective learning to refine and hone strategies that they have piloted and tested to promote mathematics inclusion. The role of YPP in the proposed project will be to organize and facilitate the youth component, such that project activities reflect the language and culture of students, continuously leveraging and building upon their voice, creative input, and ongoing feedback. YPP will conduct workshops for students organized around math-based games that provide collective experiences in which student learning requires individual reflection, small group work, teamwork and discussion. The proposed work will comprise the design of effective learning opportunities; building and supporting a cadre of teachers who can effectively work with students learning under the proposed approach; using technologies to enhance teaching and learning; and utilizing evaluation and research to drive continuous improvement. Because bringing together an effective network with diverse expertise to collaborate towards national impact requires expert facilitation processes, the project will establish working groups around three major principles: (1) Organizing from the bottom up through students, their teachers, and others in local communities committed to their education, allied with individuals and organizations who have expertise and dedication for achieving the stated goals, can produce significant progress and the conditions for collective impact; (2) Effective learning materials and formal and informal learning opportunities in mathematics can be designed and implemented for students performing in the bottom academic quartile; and (3) Teachers and other educators can become more proficient and more confident in their capacity to produce students who are successful in learning the level of mathematics required for full participation in STEM. The working groups will also be tasked to consider two cross-cutting topics: (a) the communication structures and technologies needed to operate and expand the present network, and to create the "backbone" and other structures needed to operate and expand the network; and (b) the measurements and metrics for major needs, such as assessing students' mathematics literacy, socio-emotional development in specified areas; teachers' competencies; as well as the work of the network. The final product of this plan will be a "Theory of Collective Action and Strategic Plan". The plan will contain recommendations for collective actions needed in order for the current network to coordinate, add appropriate partners, develop the needed backbone structures, and become an NSF Alliance for national impact on the broadening participation challenge of improving the mathematics achievement. An external evaluator will conduct both formative and summative aspects of this process.
DATE:
-
TEAM MEMBERS:
Robert MosesNell CobbGregory BudzbanMaisha MosesWilliam Crombie
resourceprojectProfessional Development, Conferences, and Networks
EvaluATE is a national resource center dedicated to supporting and improving the evaluation practices of approximately 250 ATE grantees across the country. EvaluATE conducts webinars and workshops, publishes a quarterly newsletter, maintains a website with a digital resource library, develops materials to guide evaluation work, and conducts an annual survey of ATE grantees. EvaluATE's mission is to promote the goals of the ATE program by partnering with projects and centers to strengthen the program's evaluation knowledge base, expand the use of exemplary evaluation practices, and support the continuous improvement of technician education throughout the nation. EvaluATE's goals associated with this proposal are to: (1) Ensure that all ATE Principal Investigators and evaluators know the essential elements of a credible and useful evaluation; (2) Maintain a comprehensive collection of online resources for ATE evaluation; (3) Strengthen and expand the network of ATE evaluation stakeholders; and (4) Gather, synthesize, and disseminate data about the ATE program activities to advance knowledge about ATE/technician education. The Center plans to produce a comprehensive set of evaluation resources to complement other services, engaging several community college-based Principal Investigators and evaluators in that process.
EvaluATE's products are informed by current research on evaluation, the National Science Foundation's priorities for the evaluation of ATE grants, and the needs of ATE PIs and evaluators for sound guidance that is immediately relevant and usable in their contexts. The fundamental nature of EvaluATE's work is geared toward supporting ATE grantees to use evaluation regularly to improve their work and demonstrate their impacts. All of EvaluATE's products are available to the public. EvaluATE's findings from the annual survey of ATE grantees aid in advancing understanding of the status of technician education and illuminate areas for additional research. The new survey investigates ATE grantees' work to serve underrepresented and special populations, including women, people of color, and veterans. Survey data are available upon request for research and evaluation purposes.
DATE:
-
TEAM MEMBERS:
Lori WingateArlen GullicksonEmma PerkKelly RobertsonLyssa Becho
resourceprojectProfessional Development, Conferences, and Networks
As higher education institutions (HEIs) work to enhance Broader Impacts (BI) efforts, collaborations with informal science education institutions (ISEs) (e.g. science centers, aquaria, zoos) can help them strengthen their impact and reach broader audiences. This project builds on the successful Portal to the Public (PoP) framework, bringing together the expertise and resources of HEIs and ISEs around the shared mission of engaging public audiences in current STEM research. The project is designed to address several critical needs: (1) Public outreach BI activities are relatively uncommon compared to BI that is focused within the infrastructure of academia; (2) Because collaborations with ISEs are frequently tied to individual Principal Investigators (PIs), there is limited opportunity to build a body of knowledge around the practice of partnering for BI work; and (3) Collaborations are often transient, making it more difficult for universities to view BI on an institutional level in ways that leverage particular institutional assets or strategies and even link investigators from multiple projects. The specific areas of study are: a. Develop and test a structure for education/outreach BI experience design that addresses a public audience need and meets NSF's BI criterion: The project will create disseminatable tools around the activity design process (including evaluation of learning impacts). By convening cross-disciplinary teams, the project will ensure that experiences will reflect a wide range of expertise and will help meet the needs of multiple stakeholders. These established structures will lower the barrier to entry for PIs who want to do public outreach BI. b. Design, test, and study structures for long-term, mutually beneficial HEI-ISE partnerships: The project will build on the proven PoP model to create flexible, disseminatable tools around the development of institutional partnerships at three collaborating HEI-ISE site pairings that consider each institution's resources, constraints and strategic goals, including a cross-institutional and cross-disciplinary Broader Impacts Design (BID) Team structure. Sustained partnerships will support ongoing public engagement with current STEM research. c. Anchor the partnership at the HEI with a representative from an office of research support: Research support professionals will be a core part of the BID Team and will help support institutional strategies for aligning BI activities with broader goals around community engagement. d. Study the culture of HEI-ISE partnerships, building knowledge about how these institutions can form effective, sustained and mutually beneficial collaborations. Project partners include Pacific Science Center with the University of Washington, Bothell, WA; University of Wisconsin-Madison with the Wisconsin Institute for Discovery; and the Sciencenter with Cornell University, Ithaca, NY. In addition, the Center for Research in Lifelong Learning, Oregon State University will oversee the research aspects of the project. The project's primary benefit is the development of more effective mechanisms for HEIs and ISEs to collaborate, that will better enable them to engage their communities in experiences and conversations about current STEM research and innovation. This project is being funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.