What if the struggle isn't productive?

In which I step firmly and with open eyes into a pile of dog-doo

In my previous post I promised to engage with the sceptics of productive struggle, those in favor of direct or explicit instruction. A widely cited article for this position is Kirschner et al, and I’ll start by naming a couple of its stronger points.

The good

The authors talk about working memory, defined in this article as follows.

Working memory is the cognitive structure in which conscious processing occurs. We are only conscious of the information currently being processed in working memory and are more or less oblivious to the far larger amount of information stored in long-term memory.

A body of research shows that working memory is quite limited in capacity and duration, and that fact has important implications for instruction. When working memory gets overloaded, struggle becomes unproductive.

The article also cites a number of research studies that demonstrate the usefulness of worked examples in certain contexts, particularly with novice students and with routine tasks. I think this is a valuable finding, and I will return to it, and to more recent research on the topic, at the end of this article. That said, the discussion of the worked-example effect contains a significant concession: “the worked-example effect first disappears and then reverses as the learners’ expertise increases.” And even in the novice domain, the original 1985 study by Sweller et al found that the worked-example effect did not extend to even modestly varied problems. A 1987 follow-up (paywalled) did achieve some transfer, but only with extended practice periods and on narrowly similar tasks—and transfer depended on rule automation, which develops slowly. Recent research on worked examples goes in a significantly different direction from what Kirschner et al advocate, as I will discuss later.

The bad

The article targets “minimally guided instruction,” a term which expands and contracts to suit the argument. The title names the expanded version: “Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching.” Spoiler alert: the article does not live up to the implicit claim here. When it comes to citing research, it retreats to a much narrower definition:

Studies conducted from 1950 to the late 1980s [compare] pure discovery learning, defined as unguided, problem-based instruction, with guided forms of instruction.

So now we are talking about “pure discovery learning” and “unguided” instruction. The word “minimal,” initially wide enough to include a variety of approaches with very different levels of guidance, has now contracted to zero. I think most people would agree that zero guidance is not appropriate. However, let’s look at the evidence cited. It’s narrower than you would expect from the sweeping claims of failure. On the areas most relevant to mathematics education, it is mostly about novices doing procedural tasks in algebraic manipulation, database software, statistics problems, and LISP programming.

The ugly

Perhaps aware of the narrowness of the research base, the authors bring in the big guns:

A series of reviews by the U.S. National Academy of Sciences has recently described the results of experiments that provide evidence for the negative consequences of unguided science instruction at all age levels and across a variety of science and math content. McCray, DeHaan, and Schuck (2003) reviewed studies and practical experience in the education of college undergraduates in engineering, technology, science, and mathematics. Gollub, Berthanthal, Labov, and Curtis (2003) reviewed studies and experience teaching science and mathematics in high school. Kilpatrick, Swafford, and Findell (2001) reported studies and made suggestions for elementary and middle school teaching of mathematics. Each of these and other publications by the U.S. National Academy of Sciences amply document the lack of evidence for unguided approaches and the benefits of more strongly guided instruction. Most provide a set of instructional principles for educators that are based on solid research.

I was surprised to see the Kilpatrick et al study in there, Adding it Up, because I remembered them saying this in the chapter on Teaching for Mathematical Proficiency:

Much debate centers on forms and approaches to teaching: “direct instruction” versus “inquiry,” “teacher centered” versus “student centered,” “traditional” versus “reform.” These labels make rhetorical distinctions that often miss the point regarding the quality of instruction. Our review of the research makes plain that the effectiveness of mathematics teaching and learning does not rest in simple labels. Rather, the quality of instruction is a function of teachers’ knowledge and use of mathematical content, teachers’ attention to and handling of students, and students’ engagement in and use of mathematical tasks. Moreover, effective teaching—teaching that fosters the development of mathematical proficiency over time—can take a variety of forms.

This passage is followed by four classroom vignettes, with discussion pointing out flaws in both directions and positive aspects of practices that fall between direct instruction and pure discovery.

I wasn’t familiar with the other two reports, but I took a quick look and was astonished to find that they directly contradict the claim of “negative consequences of unguided science instruction at all age levels and across a variety of science and math content.” I’ve placed the quotes in a footnote to spare you.¹

So, to sum up, in support of their thesis, the authors cite three NAS reports that directly contradict their thesis. I tried to find a polite way of describing this citation strategy. The best I could come up with was that maybe, buried somewhere deep in these reports, is evidence that completely unguided instruction doesn’t work. Yup, I believe you, zero guidance is bad. But that is not the headline claim in this article.

The article concludes with a dazzling display of terminological pyrotechnics.

After a half-century of advocacy associated with instruction using minimal guidance, it appears that there is no body of research supporting the technique. In so far as there is any evidence from controlled studies, it almost uniformly supports direct, strong instructional guidance rather than constructivist-based minimal guidance during the instruction of novice to intermediate learners. Even for students with considerable prior knowledge, strong guidance while learning is most often found to be equally effective as unguided approaches. Not only is unguided instruction normally less effective; there is also evidence that it may have negative results when students acquire misconceptions or incomplete or disorganized knowledge.

Count the labels: “minimal guidance,” “constructivist-based minimal guidance,” “unguided approaches,” and “unguided instruction.” Four different formulations in one concluding paragraph.

And notice how the qualifications accumulate even as the conclusion tries to sound definitive. The opening claims “no body of research supporting the technique”—a sweeping dismissal. But by the third sentence they’ve already walked it back to “novice to intermediate learners.” Then the next sentence concedes that for students with considerable prior knowledge, strong guidance is “most often found to be equally effective”—which is an admission that their preferred approach doesn’t actually win for knowledgeable learners, it merely ties. That’s a long way from the opening salvo. And for all the labels deployed, the article never describes what “direct, strong instructional guidance” actually looks like in the classroom.

The implications of limited working memory

The authors suggest that when working memory is overwhelmed by the search for solution methods, actual learning does not occur. I thought this was the single strongest point in the article, and it deserves to be reiterated. Advocates of the reform approach to teaching sometimes make the same mistake I am attributing to the authors of this article: starting with an approach that works some of the time and raising it to the level of a universal mantra—never tell, always ask—that discourages teachers from providing important information as students solve problems.

Their conclusion from the limitations of working memory is that “when dealing with novel information, learners should be explicitly shown what to do and how to do it.” For genuine novices encountering truly novel procedures, that makes sense. You can’t teach a child to count without modeling the counting procedure. Completing the square is a very clever but subtle trick; it makes sense to me just to show algebra students that trick. But the authors’ description of the interaction between working memory and long term memory gives me other ideas. Think of an explorer making their way through unfamiliar territory. Long term memory is full of knowledge about trees and rocks and rivers and bogs that the explorer can draw on. The explorer does not need to be “explicitly shown what to do.” They just need to be in a place that is interesting and exciting enough to make them want to go on, but not so alien as to frighten them.

What are the implications of the interaction between working memory and long term memory for curriculum design? First, a student solving a problem needs to have firmly held prior knowledge to draw on as they explore a solution space. This suggests that problems should be designed at the edge of that knowledge and lessons should be designed to activate it.² Second, because different learners have different reserves of firmly held knowledge in long term memory, they need different strategies. Problems should be designed with multiple entry points so that students with different reserves can all engage. And there is an important role for the teacher in monitoring student work and providing appropriate guidance. Rather than “never tell, always ask” the teacher makes judgements about when to tell and when to ask. And the teacher can help students with different reserves learn from each other, like a band of explorers, sharing knowledge.³

Worked examples

Now let me return to the second strong point of the Kirschner et al article, the worked-example effect. As I mentioned, the evidence for this is drawn from a narrow set of procedural tasks with limited transfer.⁴ But the finding that novices can learn procedures more efficiently by studying worked examples than by flailing through problems on their own strikes me as sound. The question is what to do with it.

Kirschner et al see worked examples as a way to reduce cognitive load—show students the solution so they don’t waste working memory searching for one. But more recent research by Rittle-Johnson, Star, and Durkin takes worked examples in a very different direction. Instead of showing a single solution to reduce the burden on the student, they present two different solutions to the same problem side by side and ask students to compare them. Which method is more efficient? Why does each one work? When would you choose one over the other?

This is not cognitive load reduction. This is using worked examples as objects for analysis and discussion. And the evidence is strong: students who learned through contrasting worked examples made greater gains in procedural flexibility—the ability to choose adaptively among strategies—and sometimes in conceptual understanding as well. That’s precisely the kind of transfer that the original Sweller et al studies failed to produce.

I was involved in the development of the IES Practice Guide on algebra, which rated the recommendation to “use solved problems to engage students in analyzing algebraic reasoning and strategies” as having strong evidence. Notice the language: not “use solved problems so students don’t have to think,” but “use solved problems to engage students in analyzing.” The worked examples are not replacing student thinking; they are giving students something rich to think about.

Here is the irony. The strongest evidence-based use of worked examples—carefully designed, presented in contrasting pairs, with structured opportunities for analysis and discussion—looks a lot like the kind of instruction that Kirschner et al would dismiss as constructivist-based minimal guidance. It manages cognitive load, yes, but through thoughtful task design, not by eliminating the need for student reasoning. It is, in fact, a form of productive struggle.

1. From Improving Undergraduate Instruction in Science, Technology, Engineering, and Mathematics

To be effective, undergraduate teaching faculty must also have at their command an aggregate of instructional strategies and be prepared to use combinations of inquiry-based, problem-solving, information-gathering, and didactic forms of instruction . . .

and

When implemented properly, the evidence suggests that inquiry-based instruction and problem-solving strategies engage the learner in developing the mental models required for conceptual understanding . . .

From Learning and Understanding: Improving Advanced Study of Mathematics and Science in U.S. High Schools:

A mathematics or science curriculum for advanced study that promotes learning with understanding:

• Structures the concepts, factual content, and procedures that constitute the knowledge base of the discipline around the organizing principles (big ideas) of the domain.

• Links new knowledge to what is already known by presenting concepts in a conceptually and logically sequenced order that builds upon previous learning within and across grade levels.

• Focuses on depth of understanding rather than breadth of content coverage by providing students with multiple opportunities to practice and demonstrate what they learn in a variety of contexts.

• Includes structured learning activities that, in a real or simulated fashion, allow students to experience problem solving and inquiry in situations that are drawn from their personal experiences and real-world applications.

• Develops students’ abilities to make meaningful applications and generalization to new problems and contexts.

• Incorporates language, procedures, and models of inquiry and truth verification that are consistent with the accepted practice of experts in the domain.

• Emphasizes interdisciplinary connections and integration and helps students connect learning in school with the issues, problems, and experiences that figure prominently in their lives outside of the classroom.

(Emphasis mine.)

2. The IM curriculum does this with warm-ups and carefully sequenced activities.

3. The instructional routines and development of mathematical community built into the IM curriculum are designed to achieve this.

4. After the citational debacle with the NAS reports I decided to look at the 7 studies that are cited beyond the 1985 and 1987 Sweller et al studies in support of the claim that worked examples are “invariably superior” to discovery for novices. They turn out to be a mixed bag: several are from Sweller’s own research group and show effects conditional on example design; one (Carroll, 1994) compared worked examples to standard instruction rather than to discovery learning; one (Quilici & Mayer, 1996) is about categorization of problem types rather than the worked-example effect; one (Miller, Lehman, & Koedinger, 1999) studied goal structures in a physics simulation rather than worked examples per se; and one (Trafton & Reiser, 1993) found that the benefit depended on interleaving examples with practice. The effect is real but far from invariable.

Disclaimer: These are my personal views and do not represent an official position of Illustrative Mathematics.