Today I’m going to talk about two experiments that give different answers to a question that, for some reason, is at the core of the dispute between explicit and problem-based approaches to teaching mathematics: should problem-solving come before or after telling? Each experiment is part of a much larger literature; I’ve picked these two because they are clean and simple.
Productive failure
In Kapur’s 2014 paper, “Productive Failure in Learning Math” (Cognitive Science 38(5)), ninth-graders were given the following problem, before they had been taught anything about standard deviation:
Who is the most consistent Basketball player?
Mike and Dave are the top two players in a Basketball league. The table shows the number of points scored by Mike and Dave over the course of 20 games in the league.
An award has to be given to the more consistent player of the two. The decision has to be made mathematically.
Design as many measures of consistency as you can to determine the more consistent player.
Show all working.
Note that the mean1 for both players is 24, forcing attention to the spread.
One group spent an hour on the problem and was then taught how to calculate a standard deviation. The other was taught the method first and then spent the hour working on the problem. Both groups had the same teacher and materials and the same time on each phase; only the order differed. On a later test the two groups were equally good at calculating a standard deviation, but the group that had worked on the problem first understood the concept better and did better on unfamiliar problems that went beyond the lesson.
Explicit instruction
In Ashman, Kalyuga, and Sweller’s 2020 paper, “Problem-solving or Explicit Instruction: Which Should Go First When Element Interactivity Is High?” (Educational Psychology Review 32), Year 5 students (fifth grade) were given a booklet of problems, with these instructions:
This booklet contains some problems to try to solve. They are set in everyday situations so think how you would solve the problem in real life. You are not expected to solve all of the problems. Just do what you can.
A typical question asked them to determine which light globe was the most efficient (and/or which was the least efficient):
Both groups sat through the same lesson on how to work out efficiency. The only difference was whether they worked the problems before the lesson or after it. This time it was the group that received instruction first that came out ahead. In a first experiment, with 64 Year 5 students, they scored higher on questions like those in the lesson, with no difference on transfer questions (questions either different or more difficult than the originals). A second experiment, with 71 students, used problems with more connected quantities to hold in your head at once, and there the group that received instruction first was ahead on both the similar and the transfer questions.
This study was particularly clean, with an ingenious method for ensuring that both groups received the instruction portion together in one room; there was a third period of reading unrelated science material that occurred before instruction for one group and after for the other.
What would you do?
The thing that struck me about each of the studies was that there was a natural instructional strategy for each of these problems, and it was the one that the study ended up favoring. This is not to suggest there is anything wrong with the studies, nor do I mean to provide an explanation of the mechanism that led to the results; each paper has its own proposed explanation, and I’ll get to that below. But approaching this purely as a teacher, here is what I’d be inclined to do. I’ll be interested if people have different ideas in the comments.
For the standard deviation problem, I’m facing the fact that the formula for standard deviation is a beast. The basic idea is to average deviations from the mean, but for reasons that can’t be explained at this level you square the deviations before summing and then take the square root of the average. I’d want to separate out the basic idea from this twist. And the concept I’m asking students to work with, the notion of what it means for a player to be consistent, is natural; most students will understand that consistency entails getting pretty much the same number of points each time. They will be led to think about deviations, and some of them may come up with the idea of averaging the deviations (and this is indeed what happened with the experiment). Having them work on the problem first cordons off the mysterious part of the formula (square first then square root) from the basic intuition (average the deviations).
The energy efficiency problem is different. There I would worry that the concept I’m asking students to work with will be the issue, not the calculation. The idea of energy efficiency as a rate is, if not a beast, at least a fairly tricky animal. Rates are difficult. Students might think the bulb with the greatest output was the most efficient (this was a failure mode observed in the study); or maybe the one with least output because you aren’t using as much energy. Rather than have the students flounder around, I’d be inclined to at least give them a push in the right direction, if not just tell them outright that efficiency is measured by the ratio of output to input.
Each group of researchers has a hypothesis: in the productive failure research the idea is that the problem solving period primes students to receive the explanation; in the cognitive load literature the idea is that the number of elements students have to hold in their heads, and the interactivity between them, gets in the way of schema formation. These both seem like reasonable explanations to me for the problems studied; neither seems like a reasonable explanation for the problem the other group studied.
I’m not suggesting that therefore the experiments are rigged, or useless. Each provides a valuable existence proof; there are some problems where it is natural to go one way, and some where it is natural to go the other way. An accumulation of many existence proofs with different types of problems can amount to some general design principles for which types of problems are best for which strategies. And indeed, the literature on each side brings such an accumulation.
What I don’t see here is evidence that simply rules out one approach or the other on the narrow question of order. As I explained in this post, I do believe cognitive load theory places constraints on the problem-solving approach, but I don’t think it rules it out. Nor do I think that studies like the one I described here rule out explicit teaching. I have my preferences, for reasons I explained here. So far, from the research I have read, I do not see a reason either to abandon them nor to impose them on someone else.
Changed from “average” thanks to astute reader Kathleen Smith.
Thank you for a great article. I strongly agree with your point about overgeneralization being the problem. As you pointed out, the PF study picked a topic that favors the PF approach -- a difficult formula that the students do not have enough mathematical maturity to rigorously understand. The PF phase gave the students some intuitive understanding of what a desired measurement of consistency should have, and should help with conceptual understanding. I would like to add a few more points after a brief read of the studies.
I came away with the feeling that the Standard Deviation (SD) study did DI wrong, and the Energy Efficiency (EE) study did PF wrong, almost setting up the other approach for failure. For the SD study, I think from the DI point of view, if conceptual understanding is one of the goals, then the instruction shouldn't only focus on the procedural part. There should be direct instruction related to why this particular formula is used to measure consistency. In the EE study, it looks like the PF part is solving a problem set of multiple choice questions on unfamiliar concepts. Contrast it to the PF study where the problem is an open ended "generate as many answers as you can" type. An open ended question prompts the students to find conceptual understanding, while a multiple choice problem does not. The main point of PF is that it needs to be "productive", and I would classify the failure the students are experiencing in the EE study as unproductive.
I also want to point out that neither study really is about learning math -- they are more about mathematical modeling or how math applies to real world or physics. While these subjects are part of the math curriculum, they are very different from say quadratic equations or fractions. I am not claiming that these studies are bad, but rather on your point of generalizations -- they don't necessarily generalize to teaching of traditional math concepts.
I am sympathetic to the claim that education research is hard to do, and appreciate that these studies are done. But with a subject as complex as education, we must accept the reality that even a best effort study still only capture a small fraction of what's going on. If a stock market investor made a trade on every piece of news, they would be bankrupt in no time. New information should be studied critically and within context. The same attitude should be applied to education.
Thank you so much for saying this. There are times when teaching the formula first is helpful, and other times when letting students work through the concept first is helpful. Nobody who actually teaches (or very very few, I'd surmise) argues SOLELY for one over the other. We're having a philosophical debate about something that most experienced teachers already know how to approach.