Results from the Randomized Field Experiment in Economics 1001

How important is class time to student performance in a large lecture course when students have access to a rich source of material online? How much learning is actually achieved when students passively watch even a talented lecturer for 75 minutes twice a week? What would happen if we dropped one of the weekly 75 minute lectures but we provided students with computer graded tests each week, many supplemental problems, and videos that solved problems similar to those on the test?

In the fall of 2013 my colleagues and I conducted a randomized field experiment to address these questions in the large introductory lectures of microeconomics (ECO 1001). Two professors taught one of each format. Students, after formally consenting to participate, were randomly assigned to one of the two formats. Both professors used the same power point slides, gave the same quizzes, midterm and final, and students in both formats had access to the same online material. All tests were multiple choice and computer graded. Here are the results.

1. Students in the traditional class scored 3.2 percentage points higher on a scale of 0 to 100 on the midterm but only 1.6 percentage points better on the final. Differences on the final were not statistically significant.

2. Class size mattered. Students in the smaller lecture class of 100 (as compared to the class of 270) had scores 3 percentage points higher regardless of the format.

3. Students in all formats spent 44 hours on average, roughly 3 hours per week, doing work online.

For those interested in a more detailed description of the study and the results, please see the attached manuscript  (Does Classroom Time Matter?). We also welcome your comments.

What are some of the implications from the study? Class size appears to matter. Classes of 100 students in which faculty can still ask questions or break into groups more easily than in a crowded lecture hall of 275 may be an optimal scale. Second, online material and technology are improving rapidly and they will make “chalk and talk” lectures more obsolete.

What does this mean for faculty? Will technology replace us? The short answer is not all of us. Technology will change how we teach and how many students we will be expected to carry. In short, pedagogy and productivity will improve which means the demand for adjuncts will fall and faculty who do not embrace the new pedagogy will be evaluated less favorably.

What are we doing at ZOLE? Our goal is help faculty use technology in classes more effectively. We also want to evaluate what works and what doesn’t. Randomized designs are challenging to execute but they are not always needed. Simple pre and post analyses with credible comparison groups can be very insightful. But doing evaluations requires a commitment to experiment and evaluation. Our experience is that making your teaching the basis of your research can be highly stimulating.

Leave a Reply

Your email address will not be published. Required fields are marked *