View profile

What Do College Accountability Metrics Really Tell Us Anyway?

We're not being very thoughtful about how we judge how much good a college is doing — for students or
August 16 · Issue #3 · View online
The Weekly Dispatch
We’re not being very thoughtful about how we judge how much good a college is doing — for students or for society. Then, two community-college leaders join our advisory board. And we tell you why we believe in this.

Graduation Rates Tell Us What Exactly?
Imagine your classic vision of an automobile factory. I’m picturing Henry Ford’s assembly line in the first decades of the 20th century. The managers have a goal for the number of cars coming out the end of the line. They can work backwards to figure out all the parts they need, the workers required, the speed of the machinery. In the view of organizational theorists, this a “tightly coupled system.” The people running the factory have a clear picture of the outputs and what it takes to get there.
Now imagine a big public bureaucracy like, say, a university. What exactly are the technical outputs? How should they be measured? And what do the managers even know?
Christina Ciocca Eller, a recent Columbia University graduate and now a new assistant professor at Harvard University, published a working paper earlier this year that wrestles with those questions and her answer is a little depressing: “We haven’t been terribly thoughtful about what it means to have a ‘good’ college.”
The measures we tend to use to judge a college’s outputs don’t always do a very good job, she says, of telling us much about the direct role colleges play in producing them.
Unlike that tightly coupled assembly line, Eller argues, the university is a “superficially coupled system.” In this understanding, the college’s “technical output” is not a new automobile but its graduation rate. And our focus on accountability means college leaders pay a lot of attention to that rate. But that metric alone, she argues, may often reflect more the student’s attributes and experiences before college than anything the specific college did.
Eller’s research digs into a treasure trove of student data from a large public higher-ed system. She uses several statistical strategies to account for student differences and comes to the conclusion that colleges that appear to be doing a better job because they have a better graduation rate may actually be having little or no positive effect.
And that’s why she maintains that the accountability metrics so far have “backfired.” Colleges end up spending time and attention on broad, general measures, Eller says, rather than on those capable of conveying college effectiveness.
But she also sounds bullish on the broader role of college in society. She just wants to make sure we’re capturing that:
“What else should we think about measuring for the sake of actually improving what ends up happening to students when they walk onto the campus for the first time? That’s the bigger issue,” she told me. “This is not just an issue about how we want to improve BA completion rates. This is an issue about thinking more broadly about what college means and what it does for people in this country. What kind of health impacts does it have? What kind of impact does it have on marital rates and marital stability? On community participation? On voting?”
There’s a ton more to unpack in Eller’s paper (it just won an award at this month’s American Sociological Association meeting). Here’s the link again: Superficially Coupled Systems: The Organizational Production of Inequality in Higher Education
Some additional Reading:
Community Colleges and New Advisors
We’re excited to have the leaders of two prominent community colleges join us as Open Campus advisors. Eduardo Padrón, the retiring president of Miami Dade College, and Maria Harper-Marinick, the chancellor of Maricopa Community Colleges, are new additions to our advisory board.
Community colleges, of course, are central to conversations about how higher education can shape neighborhoods, towns, and cities. And how well those institutions perform can have big effects on social mobility and economic opportunity in an entire region. More than two in five of the nation’s undergraduates are community-college students, and most of them stay very close to home. The median community-college student travels just eight miles to get to campus.
Farewell, Madam President: Diana Natalicio retires after 31 years as UT-El Paso's leader Farewell, Madam President: Diana Natalicio retires after 31 years as UT-El Paso's leader
Articles overstate millennials' loss of interest in going to college
Op-Ed: Why California needs to institute a 'student borrower bill of rights'
Small colleges hope university status will boost their fortunes Small colleges hope university status will boost their fortunes
Why We're Doing This
When we first got going on this project, we wrote about why we believe more local reporting on higher ed matters. If you missed our post, you can check it out here.
We see public confidence in colleges falling and local journalism crumbling. We want to help strengthen both.
The bottom line is, we believe more thoughtful, aggressive coverage of all parts of the American higher-education system — on the ground, in cities and states — can help decrease public cynicism. College, like journalism, is a public good that needs to be fostered and held up to high standards. Its best chance to reclaim the confidence of more Americans is through greater transparency and accountability.
Follow Along and Stay in Touch
Our Open Campus Back-to-School Road Trip starts next Wednesday. (We’d be lying if we said we’ve already packed our bags; neither of us tends to plan that far ahead.) We’ll be on the road from August 21 to August 28. You can follow our journey here.
Keep the feedback coming! We love hearing your stories, tips, and ideas. You can reach us both at
One more request: please forward this email to other people you think would be interested. They can sign up here.
Did you enjoy this issue?
If you don't want these updates anymore, please unsubscribe here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Powered by Revue
Open Campus | 1 Thomas Circle NW, Suite 700, Washington, DC 20005