Measurement and Assessment in Making: Considerations, Constructs, and Cautions

Compiled by Jamie Bell and Kevin Crowley

Just prior to this year’s National Week of Making (6/17- 6/23), a group of researchers and practitioners who are interested in the study of making as learning met at the Children’s Museum of Pittsburgh on May 23 & 24 2016. The convening was funded by the American Educational Research Association (AERA), with the understanding that the meeting would explore various aspects of making and learning, and ultimately produce an edited research volume that would describe the state of the art in making and its potential future directions. The work of the meeting, which included representatives from museums, science centers, universities, libraries and other makerspaces built directly on discussions from a 2015 Institute of Museum and Library Services (IMLS)-funded convening designed to identify areas of traction in research and learning in making and tinkering settings and activities.

In preparation for the convening participants were polled about what issues they thought are at the center of the research on making as learning, and in which directions they saw their future work and that of others heading. Using that data and findings from the earlier convening in Pittsburgh, participants divided into working groups to explore specific issues. The groups were:  measurement and assessment; equity; cultural connections; multiple settings and pathways; and facilitation and professional development. Each group did a deep dive into the current progress on the challenges faced and issues raised when investigating research questions in maker spaces and and experiences.

This blog focuses on the presentation and discussions of the measurement and assessment working group on May 23. The group included those who have developed assessments for making, those who have used others’ assessments and those who were looking for practical tools that could be used immediately across a range of contexts.

Reflecting on their work as researchers and developers of maker settings and experiences, the group noted that assessment and measurement are important for making at this point in the movement’s trajectory for its protection, sense of possibility and potential to facilitate learning. They identified issues to consider when designing assessments. Among these were attending to the proper time required for learning, avoiding measures that confine what counts as making, assessing what audiences and stakeholders value about making, measuring what travels across contexts (both products and processes), designing with fidelity to the setting and acknowledging making and tinkering potential to be transformative.

Making is an activity rich with a wide variety of outcomes and outputs. Hence there are many constructs that can potentially be assessed or measured. The group generated a list and where possible below CAISE has added links to related resources and projects that address them.

• “Designerly things” that would include design skills, design thinking, design processes and tinkering.
• Participation and engagement, using multiple markers of participation, and high/low involvement- all using observation protocols, e.g.
• STEM practices and skills like engineering practices, the Next Generation Science Standards practices, STEM identity and interest, STEM fascination scale, STEM self- efficacy, nature of science/STEM, STEM competency, beliefs, values, STEM value for self and society- all via surveys
• Computational thinking and related STEM areas like computer science interest, data analytics, circuits knowledge
• Maker identity through motivation task analysis approach, acknowledging that it is dispositional but identity can and needs to be characterized
• Educational psychology measures like self-efficacy, instrumentality, self-regulated learning, many of which are validated but also need to be attentive to context
• “Agency” and related constructs like perceived autonomy (in STEM investigations, e.g.), ownership, decision-making and goal-setting
• Innovation and creativity, including innovation “stance,” novelty, variety, quality, quantity
• Linguistic measures through structural and functional linguistic analysis and collaborative linguistic assessments
• Literacy practices such as maker and hacker literacies and “studio habits of mind”
• Resourcefulness through constructs like grit, choice assessment, interest, etc.

The group then turned to “how” to assess these areas. Strategies included various formats of portfolios such as design journals and digital badges as well as internal/external evaluations and portfolio assessments. Internal approaches like internal peer reviews and critique and feedback practices we are also mentioned. They identified methodologies like multi modal analytics, interaction analysis and level and unit analysis as being ones that could be more broadly applied. Overall the measurement and assessment group stressed the importance of setting as the primary consideration for ensuring useful assessments of making. They asked  “Can we assess the setting in which making is happening? Is it a community of practice? Is it [a maker space] a place where participation is seen as learning? Is the setting an asset-based space?, i.e. is it a place that reproduces oppression or structural racism or sexism? Is it a place that fosters diversity on multiple dimensions?” All of these are important questions because assessments are statements of values.

The ensuing discussion among the convening participants were rich, collaborative, and collegially contentious. Everyone agreed that making and tinkering can have a range of outcomes from incremental to transformative and that although many educators and policy stakeholders are interested in making because it is a potentially transformative learning experience, much of what actually occurs in making is incremental change as makers learn about materials, processes, and their own interests and goals. So there is sometimes a tension in the gap between what’s easiest to measure and the reasons why we value making.

There was also recognition that assessment, especially assessments adopted from formal educational settings, might not fit well with the ethos of making and makerspaces. Because making is an interest-driven activity stopping a maker while they are in process can interrupt and inhibit learning and other potential outcomes. Hence, making seems particularly appropriate for embedded assessments, self assessments, portfolios, etc.

With regard to digital badging as a strategy for recognizing learning milestones in making, one participant pointed out that often these types of mechanisms do not adequately acknowledge cross-setting learning or promote cross setting outcomes.

As the convening participants considered this range of methodological approaches, it became clear that there were some shared and fundamental beliefs about what the criteria of good making assessment - regardless of the particular method employed.

First, as making is multifaceted and complex, we should be careful that assessments do not oversimplify the process just to make it easier to measure. Because learner, community and environment are interconnected, assessments need to take into account the products and processes that travel from one making/learning context to others. Hence, assessment tools used must have fidelity to context as well as implementation. The group agreed that assessing making and tinkering requires a broad toolkit of situated assessments, which would also contain cautions about the limitations and tradeoffs of particular tools and approaches.

Second, there are a host of constructs and dispositions that can be measured in making and tinkering settings as indicated in the list above. Convening participants agreed that constructs like engagement, for example, need to be defined from multiple perspectives (i.e.beyond the white middle class conception) and that many of these have established or emerging theoretical foundations that are worth considering, especially when an institution, organization or program is intending to design maker spaces or activities for the first time.

Third, questions and considerations raised in the process of choosing methodologies include ones related to cultural sensitivity and who the stakeholders or audiences are for the assessment. If parents or caregivers send their children to a maker summer camp, e.g. they may value the extent to which they had a good time or made friends, not whether they mastered looping and recursion. What should be measured is dependent on the (learning) construct goal for the making or tinkering activity. For example, if a sense of ownership, or design skills are the goals of an experience, then not taking the construction of an actual material object through to completion may be the right outcome for a learner who has embraced ownership or is developing a more sophisticated design sense in the process of making.

The bottom line? We’re in a period of possibility, with some need for caution. Although the assessment and measurement discussions began from different perspectives, by the end the convening participants established common ground, exchanged knowledge and made progress on characterizing the current state of assessment in making and indicating potential future directions. Going forward, the group put forward at least three cautions:

• Don’t narrow what counts as making. The definition of learning in making and tinkering should be broad enough to include participation itself (don’t limit to the construction of material 3-D objects, e g.)
• Be aware of and attentive to the institutional politics of assessment.The expectation that projects can’t be anything less than transformational can be counterproductive. Learning is hard, takes time and incremental progress is important.
• Don’t push particular constructs or approaches or try to reify new assessment techniques at this stage in the maker movement’s development. As some things are still ill-defined we have the rare opportunity to potentially create a new assessment language for making

At the end of the discussion one participant observed: “Making is a heterogeneous activity that is turning learning on its head in many settings. Maybe we need to turn stakeholders’ expectations on their heads as well!”