Imagine your classic vision of an automobile factory. I’m picturing Henry Ford’s assembly line in the first decades of the 20th century. The managers have a goal for the number of cars coming out the end of the line. They can work backwards to figure out all the parts they need, the workers required, the speed of the machinery. In the view of organizational theorists, this a “tightly coupled system.” The people running the factory have a clear picture of the outputs and what it takes to get there.
Now imagine a big public bureaucracy like, say, a university. What exactly are the technical outputs? How should they be measured? And what do the managers even know?
Christina Ciocca Eller
, a recent Columbia University graduate and now a new assistant professor at Harvard University, published a working paper
earlier this year that wrestles with those questions and her answer is a little depressing: “We haven’t been terribly thoughtful about what it means to have a ‘good’ college.”
The measures we tend to use to judge a college’s outputs don’t always do a very good job, she says, of telling us much about the direct role colleges play in producing them.
Unlike that tightly coupled assembly line, Eller argues, the university is a “superficially coupled system.” In this understanding, the college’s “technical output” is not a new automobile but its graduation rate. And our focus on accountability means college leaders pay a lot of attention to that rate. But that metric alone, she argues, may often reflect more the student’s attributes and experiences before college than anything the specific college did.
Eller’s research digs into a treasure trove of student data from a large public higher-ed system. She uses several statistical strategies to account for student differences and comes to the conclusion that colleges that appear to be doing a better job because they have a better graduation rate may actually be having little or no positive effect.
And that’s why she maintains that the accountability metrics so far have “backfired.” Colleges end up spending time and attention on broad, general measures, Eller says, rather than on those capable of conveying college effectiveness.
But she also sounds bullish on the broader role of college in society. She just wants to make sure we’re capturing that:
“What else should we think about measuring for the sake of actually improving what ends up happening to students when they walk onto the campus for the first time? That’s the bigger issue,” she told me. “This is not just an issue about how we want to improve BA completion rates. This is an issue about thinking more broadly about what college means and what it does for people in this country. What kind of health impacts does it have? What kind of impact does it have on marital rates and marital stability? On community participation? On voting?”
Some additional Reading: