Imagine a health study claiming that patients exposed to a new, controversial therapy had outcomes slightly better than those who got the tried-and-true therapy — except for the 94% of patients in the experimental group who died.
That’s more or less what a just-released study of a massive open online course (MOOC) in physics at MIT shows. Only the abstract for the study claims that “The pre- and posttesting showed substantial learning: The students had a normalized gain slightly higher than typical values for a traditional course.” And the study is being promoted in a BostInno article with the subheading, “Skeptics beware: Online courses work.” These claims are misleading.
Indeed, the students who took both the pretests and posttests last summer in MIT’s open online course, 8.MReV, Mechanics ReView, fared at least as well as students in the in-person version of the course. Trouble is, only a very small percentage of the students who enrolled in the MOOC actually completed it. Here’s the MOOC-World candidate for Caveat of the Year:
Most of those completing less than 50% of the homework and quiz problems dropped out during the course and did not take the posttest, so their learning could not be measured.
Alternative interpretation: the students who dropped out did not learn anything. Again, it would be disingenuous to say in a health study, “We cannot comment on the health outcomes of the 94% of patients who died during the study.”
Of the 17,000 students who initially enrolled in the MOOC, 3,899 (about one-fourth) took the pretest. Of those, 1,117 (29%) took the posttest. And altogether, 1,080 successfully completed the course — that’s 28% of the pretest-takers and 6% of the initial enrollees.
A professor teaching an in-person course with a dropout rate between 72% and 94% would be called in to have a frank conversation with the dean. Something must be wrong with that class, if so few students stick around to learn anything in it. MOOC proponents keep telling us not to look at the courses’ completion rates. But if the goal of a MOOC is for students to learn, then students who give up must be counted in the course’s measure of success (or failure).
The problem with MOOCs is not that the pedagogy is inherently bad. In my experience, the modules are often well thought-out and clear. The problem is that the massive online environment is one that enables only very few students to learn. Mostly those students are ones who are already very experienced learners — in other words, not typical undergraduates. It takes a highly disciplined, autonomous student to keep up with the work in a MOOC, where you don’t have the social pressure of the traditional classroom to keep you going. Sure, if you can keep up with the work, you will learn a great deal. It’s just that not many people can keep up.
MOOCs have been touted as potential substitutes for in-person education. The MIT study shows, in its deeply-buried lede, that 8.MReV is not an adequate substitute, because it is set up for most of those who take it to fail. MOOCs may one day attain their promise, but for that to happen, we’d have to see big changes not in technology, but in human nature.
Update (9/26/2014): Betsy Barre of the Center for Teaching Excellence at Rice University noted in a Facebook thread another big problem with this study:
I think this is the most important takeaway from the study, though: “The students had a normalized gain slightly higher than typical values for a traditional course, but significantly lower than typical values for courses using interactive engagement pedagogy.”
So even the 6% only did better relative to poorly taught courses; that’s not really a fair comparison, either.