Study of the Week: Trade Schools Are No Panacea

Freddie

You will likely have encountered the common assertion that we need to send people into trade schools to address problems like college dropout rates and soft labor markets for certain categories of workers. As The Atlantic recently pointed out, the idea that we need to be sending more people to trade and tech schools has broad bipartisan, cross-ideological appeal. This argument has a lot of different flavors, but it tends to come down to the claim that we shouldn’t be sending everyone to college (I agree!) and that instead we should be pushing more people into skilled trades. Oftentimes this is encouraged as an apprenticeship model over a schooling model.

I find there’s far more in the way of narrative force behind these claims than actual proof. It just sounds good – we need to get back to making things, to helping people learn how to build and repair! But… where’s the evidence? I’ve often looked at brute-force numbers like unemployment numbers for particular professions, but it’s hard to make responsible conclusions with that kind of analysis. Well, there’s a big new study out that looks in a much more rigorous way – and the results aren’t particularly encouraging.

Today’s Study of the Week, written by Eric A. Hanushek, Guido Schwerdt, Ludger Woessmann, and Lei Zhang, looks at how workers who attend vocational schools perform relative to those who attend general education schools. Like the recent Study of the Week on the impact of universal free school breakfast, this study uses a difference-in-difference approach to explore causation, again because it’s impossible to do an experiment with this type of question – you can’t exactly tell people that your randomization has sorted them into a particular type of schooling and potentially life-long career path, after all. The primary data they use is the International Adult Literacy Survey, a very large, metadata-robust survey with demographic, education, and employment data from 18 countries, gather from 1994 to 1998. (The authors restrict their analysis to the 11 countries that have robust vocational education systems in place.) The age of the data is unfortunate, but there’s little reason to believe that the analysis here would have changed dramatically, and the data set is so rich with variables (and thus the potential to do extensive checks for robustness and bias) that it’s a good resource. What do they find? In broad strokes, vocational/tech training helps you get a job right out of school, but hurts you as you go along later in life:

(don’t be too offended by excluding women – their overall change in workforce participation made it necessary)

Most important to our purpose, while individuals with a general education are initially (normalized to an age of 16 years) 6.9 percentage points less likely to be employed than those with a vocational education, the gap in employment rates narrows by 2.1 percentage points every ten years. This implies that by age 49, on average, individuals completing a general education are more likely to be employed than individuals completing a vocational education. Individuals completing a secondary-school equivalency or other program (the “other” category) have a virtually identical employment trajectory as those completing a vocational education.

Now, they go on to do a lot of quality controls and checks for robustness and confounds. As much of a slog as that stuff is, I recommend you check some of that out and start to pick some of it apart. Becoming a skilled reader of academic research literature really requires that you get used to picking apart the quality controls, because this is often where the juicy stuff can be found. Still, in this study the various checks and controls all support the same basic analysis: those who attend vocational schools or programs enjoy initial higher employability but go on to suffer from higher unemployment later in life.

What’s going on with these trends? The suggestion of the authors seems correct to me: vocational training is likely more specific and job-focused than general ed, which means that its students are more ready to jump right into work. But over time, technological and economic changes change which skills and competencies are valued by employers, and the general education students have been “taught to learn,” meaning that they are more adaptable and can acquire new and valuable skills.

I’m not 100% convinced that counseling more people into the trades is a bad idea. After all, the world needs people who can do these things, and early-career employability is nothing to dismiss. But the affirmative case that more trade school is a solution to long-term unemployment problems seems clearly wrong. And in fact this type of education seems to deepen one of our bigger problems in the current economy: the speed of technological change moves so fast these days that it’s hard for older workers to adapt, and they often find themselves in truly unfortunate positions. Even in trades that are less susceptible to technological change, there’s uncertainty; a lot of the traditional construction trades, for example, are very exposed to the housing market, as we learned the hard way in 2009. Do we want to use public policy to deepen these risks?

In a broader sense: it’s unclear if it’s ever a good idea to push people into a particular narrow range of occupations, because then people rush into them and… there stops being any shortage and advantage for labor. For a little while there, petrochemical engineering seemed huge. But it takes a lot of schooling to do those jobs, and then the oil market crashed. Pharmacy was the safe haven, and then word got out, a ton of people went into the field, and the labor market advantage was eroded. Also, there are limits to our understanding of how many workers we need in a given field. Some people argue there’s a teacher shortage; some insist there isn’t. Some people believe there’s a shortage of nurses; some claim there’s a glut. If you were a young student, would you want to bet your future on this uncertainty? It seems far more useful to me to try and train students into being nimble, adaptable learners than to train them for particular jobs. That has the bonus advantage of restoring the “practical” value of the humanities and arts, which have always been key aspects of learning to be well-rounded intellects.

My desires are twofold. First, that we be very careful when making claims about the labor market of the future, given the certainty that trends change. (One of my Purdue University students once told me, with a smirk, that he had intended to study Search Engine Optimization when he was in school, only to find that Facebook had eaten Google as the primary driver of many kinds of web traffic.) Second, that we stop saying “the problem is you went into X field” altogether. Individual workers are not responsible for labor market conditions. Those are the product of macroeconomic conditions – inadequate aggregate demand, outsourcing, and the merciless march of automation. What’s needed is not to try and read the tea leaves and guess which fields might reward some slice of our workforce now, but to redefine our attitude towards work and material security through the institution of some sort of guaranteed minimum income. Then, we can train students in the fields in which they have interest and talent, contribute to their human flourishing in doing so, and help shelter them from the fickleness of the economy. The labor market is not a morality play.

why universities can’t be the primary site of political organizing

Freddie1 Comment

This is not a political publication, but I am definitely interested in discussing campus issues in this space, and I would like to take a second and lay out some reasons why Amber A’Lee Frost is correct that the university can’t be the key site of left-wing (or any other) organizing. (If you think that idea’s a strawman, I invite you to read the Port Huron Statement.)

Please note that this is a series of empirical claims, not normative ones. I’m not saying it would be good or bad for campus to be the key site of a given movement’s organizing strategy. I’m saying that it’s not going to work, for good or bad.

There’s not a lot of people on campus. There’s a lot of universities out there, and you could be forgiven for overestimating the size of the student population. But NCES says there’s only about 20 million students, grad and undergrad, enrolled in degree-granting post-secondary institutions. There’s also about 4 million people who work in those institutions. Back of the envelope that means that there’s about 7.5% of the American population regularly on campus in one capacity or another, setting aside questions of online-only education. Is 7.5% nothing? Not at all. It’s a meaningful chunk of people. But even if all of them were capable of being politically organized – which of course is far from the truth – you’re still leaving out the vast majority of the adult population.

Campus activism is seasonal. You aren’t going to hear a lot about campus protests for a few months. Why? Because of summer break. Vacation is notoriously hard on student protest groups. Why did the “campus uprising” of a few years ago fizzle out? In large measure because of Christmas break – the spring semester wasn’t nearly as active as the fall – and then summer break. Activism requires momentum and continuity of practice, and the regularity of vacation makes that quite difficult. Organizations that are careful and have strong leadership in place can take steps to adjust for this seasonal nature, but there’s just always going to be major lulls in campus organizing according to the calendar. And politics happens year-round.

College students are an itinerant population. Speaking of continuity of practice, campus political groups constantly have to replace membership and leadership because students (we hope) will eventually graduate. Again, that problem can be ameliorated with hard work and forethought by these groups, but it’s very difficult to have consistent strength of numbers and a coherent political vision when you’re seeing 100% turnover in a 5-6 year span.

Town and gown conflicts can make local organizing difficult. Sadly, many university towns are sites of tension and mutual distrust between the campus community and the locals. The degree of these tensions varies widely from campus to campus, and they can be ameliorated. In fact, making attempts to heal those divides can be the best form of campus activism. But it’s the case that the complex conflicts between colleges and the towns in which they’re housed will often make it difficult to build meaningful solidarity across the campus borders, which often serve as an invisible wall of attention and community.

Students are too busy to devote too much time to organizing. 70% of college students work. A quarter have dependent children. These students must also do all of the necessary work of being students. We should be realistic and fair with their time and recognize that a majority of students will not be able to engage politically for many hours out of the week.

College students have a natural and justifiable first-order priority of getting employed. Everyone who works is of course at risk of having professional repercussions for their political engagement, but college students perhaps have a unique set of worries about being publicly politically active, particularly in the era of the internet. Nowadays, we’re all constantly building an easily-searchable, publicly-accessible archive of the things we once thought and did. This is particularly troublesome for those who have not yet gotten their first jobs and have yet to build the kind of social capital necessary to feel secure in their ability to get work with a controversial political past. It’s my impression that a lot of college students are inclined to be political but who feel that they simply can’t risk it, and that’s a fear that we should respect given the modern job market.

College activism can either be a low-stakes place where students learn and grow safely, or an essential site of organizing – but it can’t be both. Oftentimes, when campus activists make mistakes (such as forcing a free yoga class for disabled students to be shut down because yoga is “cultural appropriation”), defenders will say, hey, they’re just college kids – they need a chance to screw up, to make mistakes, to be free to fail. And there’s some real truth to that. The problem is that this attitude cannot coexist with the idea that campus has to be a central site or the central site of left-wing political organizing. If what happens on campus is crucial to the broader left movement, it can’t then be called not worth worrying about; if campus organizing is a space that is largely free of consequences for young activists, then it can’t be a space where essential political work gets done. These ideas are not compatible.

Organize the campus’s workforce according to labor principles. None of this means that organizing shouldn’t take place on campus; it absolutely should. But like Frost I think that the left is far too fixated on what happens in campus spaces, likely because these spaces are some of the only areas where the left appears to hold any meaningful power. Student activists should be encouraged to engage politically in order to learn and grow, but we should not imagine that they are the necessary vanguard of the young left, given that only a third of Americans ever gets a college degree. Meanwhile, we absolutely must continue to organize the campus as a workplace. (For the record, Frost is a member of a campus union, as am I.) But that organization takes place according to labor principles, not according to any special dictates of academic culture. And this returns to Frost’s basic thesis: it is the organization of labor, not of students, that must be the primary focus and goal of the American left.

correlation: neither everything nor nothing

Freddie
via Overthinking

One thing that everyone on the internet knows, about statistics, is this: correlation does not imply causation. It’s a stock phrase, a bauble constantly polished and passed off in internet debate. And it’s not wrong, at least not on its face. But I worry that the denial of the importance of correlation is a bigger impediment to human knowledge and understanding than belief in specious relationships between correlation and causation.

First, you should read two pieces on the “correlation does not imply causation” phenomenon, which has gone from a somewhat arcane notion common to research methods classes to a full-fledged meme. This piece by Greg Laden is absolute required reading on correlation and causation and how to think about both. Second, this piece by Daniel Engber does good work talking about how “correlation does not imply causation” became an overused and unhelpful piece of internet lingo.

As Laden points out, the question is really this: what does “imply” mean? The people who employ “correlation does not imply causation” as a kind of argumentative trump card are typically using “imply” in a way that nobody actually means, which is as synonymous with “prove.” That’s pretty far from what we usually mean by “implies”! In fact, using the typical meaning of implication, correlation sometimes implies causation, in the sense that it provides evidence for a causal relationship. In careful, rigorously conducted research, a strong correlation can offer some evidence of causation, if that correlation is embedded in a theoretical argument for how that causative relationship works. If nothing else, correlation is often the first stage in identifying relationships of interest that we might then investigate in more rigorous ways, if we can.

A few things I’d like people to think about.

There are specific reasons that an assertion of causation from correlation data might be incorrect. There is a vast literature of research methodology, across just about every research field you can imagine. Correlation-causation fallacies have been investigated and understood for a long time. Among the potential dangers is the confounding variable, where an unknown variable is driving the change in two other variables, making them appear to influence one another. This gives us the famous drownings-and-ice cream correlation – as drownings go up, so do ice cream sales. The confounding variable, of course, is temperature.1 There are all sorts of nasty little interpretation problems in the literature. These dangers are real. But in order to have understanding, we have to actually investigate why a particular relationship is spurious. Just saying “correlation does not imply causation” doesn’t do anything to actually improve our understanding. Explore why, if you want to be useful. Use the phrase as the beginning of a conversation, not a talisman.

Correlation evidence can be essential when it is difficult or impossible to investigate a causative mechanism. Cigarette smoking causes cancer. We know that. We know it because of many, many rigorous and careful studies have established that connection. It might surprise you to know that the large majority of our evidence demonstrating that relationship comes from correlation studies, rather than experiments. Why? Well, as my statistics instructor used to say – here, let’s prove cigarette smoking causes cancer. We’ll round up some infants, and we’ll divide them into experimental and control groups, and we’ll expose the experimental group to tobacco smoke, and in a few years, we’ll have proven a causal relationship. Sound like a good idea to you? Me neither. We knew that cigarettes were contributing to lung cancer long before we identified what was actually happening in the human body, and we have correlational studies to thank for that. Blinded randomized controlled experimental studies are the gold standard, but they are rare precisely because they are hard, sometimes impossible. To refuse to take anything else as meaningful evidence is nihilism, not skepticism.

Sometimes what we care about is association. Consider relationships which we believe to be strong but in which we are unlikely to ever identify a specific causal mechanism. I have on my desk a raft of research showing a strong correlation between parental income and student performance on various educational metrics. It’s a relationship we find in a variety of locations, across a variety of ages, and through a variety of different research contexts. This is important research, it has stakes; it helps us to understand the power of structural advantage and contributes to political critique of our supposedly meritocratic social systems.

Suppose I was prohibited from asserting that this correlation proved anything because I couldn’t prove causation. My question is this: how could I find a specific causal mechanism? The relationship is likely very complex, and in some cases, not subject to external observation by researchers at all. To refuse to consider this relationship in our knowledge making or our policy decisions because of an overly skeptical attitude towards correlational data would be profoundly misguided. Of course there’s limitations and restrictions we need to keep in mind – the relationship is consistent but not universal, its effect is different for different parts of the income scale, it varies with a variety of factors. It’s not a complete or simple story. But I’m still perfectly willing to say that poverty is associated with poor educational performance. That’s the only reasonable conclusion from the data. That association matters, even if we can’t find a specific causal mechanism.

Correlation is a statistical relationship. Causation is a judgement call. I frequently find that people seem to believe that there is some sort of mathematical proof of causation that a high correlation does not merit, some number that can be spit out by statistical packages that says “here’s causation.” But causation is always a matter of the informed judgment of the research community. Controlled experiments are the gold standard in that regard, but there are controlled experiments that can’t prove causation and other research methods that have established causation to the satisfaction of most members of a discipline.

Human beings have the benefit of human reasoning. One of my frustrations with the “correlation does not imply causation” line is that it’s often deployed in instances where no one is asserting that we’ve adequately proved causation. I sometimes feel as though people are trying to protect us from mistakes of reasoning that no one would actually fall victim to. In an (overall excellent) piece for the Times, Gary Marcus and Ernest Davis write, “A big data analysis might reveal, for instance, that from 2006 to 2011 the United States murder rate was well correlated with the market share of Internet Explorer: Both went down sharply. But it’s hard to imagine there is any causal relationship between the two.” That’s true – it is hard to imagine! So hard to imagine that I don’t think anyone would have that problem. I get the point that it’s a deliberately exaggerated example, and I also fully recognize that there are some correlation-causation assumptions that are tempting but wrong. But I think that, when people state the dangers of drawing specious relationships, they sometimes act as if we’re all dummies. No one will look at these correlations and think they’re describing real causal relationships because no one is that senseless. So why are we so afraid of that potential bad reasoning?

Those disagreeing with conclusions drawn from correlational data have a burden of proof too. This is the thing, for me, more than anything. It’s fine to dispute a suggestion of causation drawn from correlation data. Just recognize that you have to actually make the case. Different people can have responsible, reasonable disagreements about statistical inferences. Both sides have to present evidence and make a rational argument drawn from theory. “Correlation does not imply causation” is the beginning of discussion, not the end.

I consider myself on the skeptical side when it comes to Big Data, at least in certain applications. As someone who is frequently frustrated by hype and woowoo, I’m firmly in the camp that says we need skepticism ingrained in how we think and write about statistical inquiry. I personally do think that many of the claims about Big Data applications are overblown, and I also think that the notion that we’ll ever be post-theory or purely empirical are dangerously misguided. But there’s no need to throw the baby out with the bathwater. While we should maintain a healthy criticism of them, new ventures dedicated to researched, data-driven writing should be greeted as a welcome development. What we need, I think, is to contribute to a communal understanding of research methods and statistics, including healthy skepticism, and there’s reason for optimism in that regard. Reasonable skepticism, not unthinking rejection; a critical utilization, not a thoughtless embrace.

 

you learn by being taught

Freddie

Forgive the relative quiet lately; I’ve been enjoying my birthday weekend and then catching up on a ton of work. There’s a bunch of good things coming this week, including the return of book reviews after a brief (and unplanned) break.

This morning I spoke to an entire public high school, where I was invited to discuss being a product of public schools, higher ed, and success. It was very funny for me to be asked, though flattering – as I told the kids today, I would never think of myself casually as a success. Who ever thinks that way, beyond the wealthy and the deluded? But it was flattering and fun. I told them that there was no great wisdom in life, just a series of decisions before you, and hopefully with time the perspective to be able to choose better from worse. And, because I think this is important, I told them that they needed to cultivate a sense of “good enough” in their lives. At that age, they are being told constantly that they should pursue their dreams. But very few of us get what we’ve dreamed of, and those who have often find it’s far less grand than they’d imagined. So I told them to learn and experience and enjoy and to figure out how to live in the essential disappointment of human life.

It wasn’t as much of a bummer as it sounds!

I have been reflecting on the value of teachers. I have been accused a lot, lately, of not believing that teachers matter. That’s the opposite of the truth, really. I just think that this notion of casting the value of teachers in purely quantitative terms is a mistake, and a very recent one. The entire history of the Western canon, from Socrates to Aquinas to Locke to Dewey to Baldwin, contains arguments against this reduction. But this fight, to define what I mean and what I don’t against the tide, is a fight I suspect I will always have to keep fighting, and I intend to.

Our culture celebrates autodidacts. It talks constantly of “disrupting” education. It insists always that we need to radically reshape how we teach and learn. It treats as heroic the rejection of teachers and traditional mentorship. The self-help aisle of the bookstore abounds with writers who insist that they truly learned by rejecting the typical method of education and became, instead, self-taught, self-made. It’s an unavoidable trope.

What amazes me about my own education is just how far that is from the truth for me personally. I’ve learned, over decades, how I learn. It’s pretty simple: teachers teach me. That was true in kindergarten and it’s true now that I have my doctorate. I can’t tell you how often I have found myself feeling lost and ignorant, only to have patient, kind teachers take me through the familiar processes of modeling and repetition that are cornerstones of education. I think back to my graduate statistics classes, where I often feel like the slowest person in class, but where I always ended up getting there, thanks to steady and reassuring teaching. When I don’t get what I need from class, I’d go to office hours, or I’d go to the statistics help room, where brilliant graduate students eagerly shared knowledge and experience with me. None of this is fundamentally any different than when Mrs. Gebhardt taught me to cut shapes out of paper or when Mr. Shearer taught me simple algebra or when Mr. Tucci taught me to read poetry or when Dr. Nunn taught me to write a real research paper. The process is always the same, and in every case, I have succeeded not through rejecting the authority of teachers but by accepting their help, by recognizing their superior knowledge and letting them use it to enrich my life.

Is that a contradiction of what I’ve said about the limited ability of teachers to control the outcomes of their students? I don’t think so. The question is, do you want us to have a fuller and more humane vision of what it means to learn? I do.

They say that great men see farther than others by standing on the shoulders of giants. I think most of us are enabled to see as far as others because others have collectively reached their hands down and pulled us up.

another notch in the belt

Freddie

It’s my birthday today. Wasn’t that long ago that I was part of a vanguard of young writer types. What the hell happened?

This project’s about three months old now, and I gotta tell you guys: I haven’t had this much fun writing in ages. It’s been better than I could have hoped. Thanks for coming along.

I woke up one day to find that my life had gotten pretty damn good. My job’s not perfect, but it’s still pretty great. I miss teaching, and I’d love to be in a position where I had some motivation to get peer reviewed stuff published. But I’m working at a great college with a gorgeous campus in a system I admire immensely. It’s part of my job to stay on top of the research literature, so I’m reading books and articles at a good clip. Polyani said that a scholar is someone who lives with the questions, and I do, and that’s enough. Very few people get that opportunity. It’s a privilege.

It’s also a privilege to live in this city. The other day I was walking home, cutting through Prospect Park right after dusk. I came to the Long Meadow, which a few hours before had been absolutely packed with people picnicking and jogging and flying kites and walking dogs. For a brief moment I found it utterly empty, not another soul in sight, alone in one of the most popular parks in the city. And I knew in that moment that it was all for me.

Study of the Week: Feed Kids to Feed Them

Freddie

Today’s Study of the Week is about subsidized meal programs for public school students, particularly breakfast. School breakfast programs have been targeted by policymakers for awhile, in part because of discouraging participation levels. Even many students who are eligible for subsidized lunches often don’t take advantage of school breakfast. The reasons for this are multiple. Price is certainly a factor. As you’d expect, price is inversely related to participation rates for school breakfast. Also, in order to take advantage of breakfast programs, you need to arrive at school early enough to eat before school formally begins, and it’s often hard enough to get teenagers to school on time just for class. Finally, there’s a stigma component, particularly associated with subsidized breakfast programs. It was certainly the case at my public high school, where 44% of students were eligible for federal school lunch subsidies, that school breakfast carried class associations. At lunch, everybody’s eating together, but students at breakfast tended to be poorer kids – which in turn likely makes it less likely that students will want to be seen getting school breakfast.

The study, written by Jacob Leos-Urbel, Amy Ellen Schwartz, Meryle Weinstein, and Sean Corcoran (all of NYU), takes advantage of a policy change in New York public schools in 2003. Previously, school breakfast had been free only to those who were eligible for federal lunch subsidies, which remains the case in most school districts. New York made breakfast free for all students, defraying the costs by raising the price of unsubsidized lunch from $1.00 to $1.50. They then went looking to see if the switch to free breakfast for all changed participation in the breakfast program, looking for differences between the three tiers – free lunch students, reduced lunch students, and students who pay full price. They also compared outcomes from traditional schools to Universal Free Meal (UFM) schools, where the percentage of eligible students is so high that everyone in the school gets meals for free already. This helped them tease out possible differences in participation based on moving to a universal free breakfast model. They were able to use a robust data set comprising results from 723,843 students from 667 schools, grades 3–8. They also investigated whether breakfast participation rates were associated with performance in quantitative educational metrics.

It’s important to say that it’s hard to really get at causality here because we’re not doing a randomized experiment. Such an experiment would be flatly unethical – “sorry, kid, you got sorted into the no-free-breakfast group, good luck.” So we have to do observational studies and use what techniques we can to adjust for their weaknesses. In this study, the authors used what’s called a difference in difference design. These techniques are often used when analyzing natural experiments. In the current case, we have schools where the change in policy has no impact on who receives free breakfast (the UFM schools) and schools where there is an impact (the traditional schools). Therefore the UFM schools can function as a kind of natural control group, since they did not receive the “treatment.” You then use a statistical model to compare the change in the variables of interest for the “control” group to the change for the “treatment” group. Make sense?

What did the authors find? The results of the policy change were modest, in almost every measurable way, and consistent across a number of models that the authors go into in great detail in the paper. Students did take advantage of school breakfast more after breakfast became universally free. On the one hand, students who paid full price increased breakfast participation by 55%, which is a large number; but on the other hand, their initial baseline participation rates were so low (again because breakfast participation is class-influenced) that they only ate on average 6 additional breakfasts a year. Reduced price and free were increased by 33% and 15%, respectively – the latter particularly interesting given that those students did not pay for breakfast to begin with. Still, that too only represents about 6 meals over the course of a year, not nothing but perhaps less than we’d hope for a program with low participation rates. The only meaningful difference in models seems to be when they restrict their analysis to the small number (91) of schools where less than a third of students are eligible for lunch subsidies, in which case breakfast participation grew by a substantially larger amount. The purchase of lunches, for what it’s worth, remained static despite the price increase.

There’s a lot of picking apart the data and attempting to determine to what degree these findings are related to stigma. I confess I find the discussion a bit muddled but your money may vary. The educational impacts, also, were slight. They found a small increase in attendance, but this result was not significant, and no impact on reading and math outcomes.

These findings are somewhat discouraging. Certainly we would hope that moving to a universal program would help to spur participation rates to a greater degree than we’re seeing here. But it’s important to note that the authors largely restricted their analysis to the years immediately before and after the policy change, thanks to the needs of their model. When broadening the time frame by a couple years, they find an accelerating trend in participation rates, though the model is somewhat less robust. What’s more, as the authors note, decreasing stigma is the kind of thing that takes time. If it is in fact the case that stigma keeps students from taking part in school breakfast, it may well take a longer time period for universal free breakfast to erode that disincentive.

I’m also inclined to suspect that the need to get kids to school early to eat represents a serious challenge to the pragmatic success of this program. There’s perhaps good news on the way:

Even when free for all, school breakfast is voluntary. Further, unlike school lunch, breakfast traditionally is not fully incorporated into the school day and students must arrive at school early in order to participate. Importantly, in the time period since the introduction of the universal free breakfast policy considered in this paper, New York City and other large cities have begun to explore other avenues to increase participation. Most notably, some schools now provide breakfast in the classroom.

Ultimately, I believe that making school breakfast universally free is a great change even in light of relatively modest impacts on participation rate. We should embrace providing free breakfast to all students regardless of income level out of the principle of doing so, particularly considering that fluctuations in parental income might make kids who are technically ineligible unable to pay for breakfast. In time, if we set up this universal program as an embedded part of the school day, and work diligently to erase the stigma of using it, I believe more and more kids will begin their days with a full stomach.

As for the lack of impacts on quantitative metrics, well – I think that’s no real objection at all. We should feed kids to feed them, not to improve their numbers. This all dovetails with my earlier point about after school programs: if we insist on viewing every question through the lens of test scores, we’re missing out on opportunities to improve the lives of children and parents that are real and important. Again, I will say that I recognize the value of quantitative academic outcome in certain policy situations. But the relentless focus on quantitative outcomes leads to scenarios where we have to ask questions like whether giving kids free breakfast improves test scores. If it does, great – but the reason to feed children is to feed children. When it comes to test scores and education policy, the tail too often wags the dog, and it has to stop.

two economists ask teachers to behave as irrational actors

Freddie2 Comments

I was considering doing a front-to-back fisking of this interview of Raj Chetty, Professor of Economics at Stanford University, conducted by the libertarian economist Tyler Cowen. Despite Chetty’s obviously impressive credentials, he says several things in the interview that simply don’t hold up to scrutiny, in particular regarding the simultaneity problem1 and the impact of the shared environment2 I’ve decided to just focus on one key point, though.

The standard neoliberal ed reform argument goes like this: the major entrenched socioeconomic and racial inequalities in this country are no excuse for poor quantitative outcomes for groups of students; teachers and schools, despite all of the evidence to the contrary, control most of the variation in educational outcomes; therefore our perceived education problems are the result of lazy, untalented teachers; introducing a market for schooling will force schools to get rid of those teachers and metrics will improve. Now this story has failed to play out this way again and again in places like Detroit and Washington DC, but we’ll let that slide for now. If we accept this argument on its own terms, we need to get many talented people into teaching and replace the hundreds of thousands of “bad” teachers we’d be getting rid of.

Ed reform types are typically cagey about the scale of teacher dismissals – they hate to actually come out and say “I’d like to get hundreds of thousands of teachers fired” – but based on their own numbers, their own claims about the size and extent of the problem, that’s what needs to happen. You can’t simultaneously say that there’s a nationwide education crisis that needs to be solved by firing teachers and avoid the conclusion that huge numbers need to be fired. If reformers claim that even one out of every ten public teachers needs to be let go (a low number in reform rhetoric), we’re talking about more than 300,000 fired teachers.

I’ve argued before that the idea that market economics are effective means to solve educational problems falls apart once you recognize that, unlike a factory building a widget, educators don’t control most of what contributes to a child’s learning outcomes. But suppose you do believe in the standard conservative economics take on school reform: how can Chetty’s ideas make sense, if we trust young workers in a labor market to act in their own rational best interest? Chetty believes that we need, at scale, to “either retrain or dismiss the teachers who are less effective, [to] substantially increase productivity without significantly increasing cost.” Without increasing costs, in other words, by raising teacher salaries. The median teacher in this country makes ~$57,000 a year; the 75th percentile makes ~$73k, and the 25th percentile, ~$45k. Compare with median lawyer salaries well above $100,000 a year and median doctor salaries close to $200,000, or an average of $125,000+ for MBA graduates. So we’re not going to pay teachers more, and we’re going to sufficiently erode labor protections, if we’re going to dismiss those less effective teachers. This doesn’t sound like a good deal already.

Of course, teachers don’t just suffer from low median wages compared to people with similar levels of schooling. They also suffer from far lower social status than they are typically afforded in other countries, as Dr. Chetty acknowledges:

Yeah, I think status seems incredibly important. My sense of the K–12 education system in the US is, unfortunately for many kids graduating from top colleges, teaching is not near the top of the list of professions that they’d consider. It’s partly because, in a sense, they can’t afford to be teachers because it entails such a pay cut. But also because they feel that it’s not the most prestigious career to pursue.

Why yes, Dr. Chetty, it’s true! Teachers don’t get a lot of prestige in this country! Maybe that’s because well-paid celebrity academics who make several times the median teacher salary – people like you – talk casually about firing them en masse and insist that they are the source of poor metrics! The ed reform movement has insulted the profession of public school teacher for years. Popular expressions of that philosophy, like the execrable documentary Waiting for “Superman,“ have contributed to widespread assumptions that students are failing because their teachers are lazy and corrupt. How can a political movement that has relentlessly insulted the teaching profession not contribute to declining interest in being part of that profession?

Here in New York, the numbers are clear: we’re already facing a serious teacher shortage.

What Chetty and Cowen are asking for makes no sense according to their own manner of thinking. Dr. Chetty, Dr. Cowen: there is no bullpen. Even if I thought that teachers controlled far more of the variance in quantitative education metrics than I do, and even if I didn’t have objections about fair labor practices against removing hundreds of thousands of teachers, we would be stuck with this simple fact. We do not have hundreds of thousands of talented young professionals, eager to forego the far greater rewards available in the private sector, ready to jump in and start teaching. And we certainly won’t have such a thing if we share Chetty’s resistance to paying teachers more and his commitment to making it easier to fire them.

So: no higher salaries for a relatively low-paying profession, eroding the job security that is the most treasured benefit of the job, continuing to degrade and insult the current workforce as lazy and undeserving, getting rid of hundreds of thousands of them, and yet somehow attracting hundreds of thousands of more talented, more committed young workers to become teachers.

According to what school of economics, exactly, is such a thing possible?

 

Study of the Week: Better and Worse Ways to Attack Entrance Exams

Freddie1 Comment

For this week’s Study of the Week I want to look at standardized tests, the concept of validity, and how best – and worst – to criticize exams like the SAT and ACT. To begin, let’s consider what exactly it means to call such exams valid.

What is validity?

Validity is a multi-faceted concept that’s seen as a core aspect of test development. Like many subjects in psychometrics and stats, it tends to be used casually and referred to as something fairly simple, when in fact the concept is notoriously complex. Accepting that any one-sentence definition of validity is thus a distortion, generally we say that validity refers to the degree that a test measures that which it purports to measure. A test is more valid or less depending on its ability to actually capture the underlying traits we are interested in investigating through its mechanism. No test can ever be fully or perfectly validated; rather we can say that it is more or less valid. Validity is a vector, not a destination.

Validity is so complex, and so interesting, in part because it sits at the nexus of both quantitative and philosophical concerns. Concepts that we want to test may appear superficially simple but are often filled with hidden complexity. As I wrote in a past Study of the Week, talking about the related issues of construct and operationalization,

If we want to test reading ability, how would we go about doing that? A simple way might be to have a a test subject read a book out loud. We might then decide if the subject can be put into the CAN READ or CAN’T READ pile. But of course that’s quite lacking in granularity and leaves us with a lot of questions. If a reader mispronounces a word but understands its meaning, does that mean they can’t read that word? How many words can a reader fail to read correctly in a given text before we sort them into the CAN’T READ pile? Clearly, reading isn’t really a binary activity. Some people are better or worse readers and some people can reader harder or easier texts. What we need is a scale and a test to assign readers to it. What form should that scale take? How many questions is best? Should the test involve reading passages or reading sentences? Fill in the blank or multiple choice? Is the ability to spot grammatical errors in a text an aspect of reading, or is that a different construct? Is vocabulary knowledge a part of the construct of reading ability or a separate construct?

Questions such as these are endemic to test development, and frequently we are forced to make subjective decisions about how best to measure complex constructs of interest. Common to the quantitative social sciences, this subjective, theoretical side of validity is often written out of our conception of the topic, as we want to speak with the certainty of numbers and the authority of the “harder” sciences. But theory is inextricable from empiricism, and the more that we wish to hide it, the more subject we are to distortions that arise from failing to fully think through our theories and what they mean. Good empiricists know theory comes first; without it, the numbers are meaningless.

Validity has been subdivided into a large number of types, which reflect different goals and values within the test development process. Some examples include:

  • Predictive Validity: The ability of a test’s results to predict that which it should be able to predict if the test is in fact valid. If a reading test predicts whether students can in fact read texts of a given complexity or reading level, that would provide evidence of predictive validity. The SAT’s ability to predict the grades of college freshmen is a classic example.
  • Concurrent Validity: If a test’s results are strongly correlated with that of a test that measures similar constructs and which has itself been sufficiently validated, that provides evidence of concurrent validity. Of course, you have to be careful – two invalid tests might provide similar results but not tell us much of actual worth. Still, a test of quantitative reasoning and a test of math would be expected to be imperfectly yet moderately-to-strongly correlated if each is itself a valid test of the given construct.
  • Curricular Validity: As the name implies, curricular validity reflects the degree to which a test matches with a given curriculum. If a test of biology closely matches the content in the syllabus of that biology course, we would argue for high curricular validity. This is important because we can easily imagine a scenario where general ability in biology could be measured effectively by a test that lacked curricular validity – students who are strong in biology might score well on a test, and students who are poor would likely score poorly, even if that test didn’t closely match the curriculum. But that test would still not be a particularly valid measure of biology as learned in that class, so curricular validity would be low. This is often expressed as a matter of ethics.
  • Ecological Validity: Heading in a “softer” direction, ecological validity is often discussed to refer to the degree to which a test or similar assessment instrument matches the real-life contexts in which its consequences will be enacted. Take writing assessment. In previous generations, it was common for student writing ability to be tested through multiple choice tests on grammar and sentence combining. These tests were argued to be valid because their results tend to be highly correlated with the scores that students receive on written essay exams. But writing teachers objected, quite reasonably, that we should test student writing by having them write, even if those correlations are strong. This is an invocation of ecological validity and reflects a broader (and to me positive) effort to not reduce validity to narrowly numerical terms.

I could go on!

When we talk about entrance examinations like the SAT or GRE, we often fixate on predictive validity, for obvious reasons. If we’re using test scores as criteria for entry into selective institutions, we are making a set of claims about the relationship between those scores and the eventual performance of those students. Most importantly, we’re saying that the tests help us to know that students can complete a given college curriculum, that we’re not setting them up to fail by admitting them to a school where they are not academically prepared to thrive. This is, ostensibly, the first responsibility of the college admissions process. Ostensibly.

Of course, there are ceiling effects here, and a whole host of social and ethical concerns that predictive validity can’t address. I can’t find a link now but awhile back a Harvard admissions officer admitted that something like 90% of the applicants have the academic ability to succeed at the school, and that much of the screening process had little to do with actual academic preparedness. This is a big subject that’s outside of the bounds of this week’s study.

The ACT: Still Predictively Valid

Today’s study, by Paul A. Westrick, Huy Le, Steven B. Robbins, Justine M. R. Radunzel, and Frank L. Schmidt1, is a large-n (189,612) study about the predictive validity of the ACT, with analysis of the role of socioeconomic status (SES) and high school grades in retention and college grades. The researchers examined the outcomes of students who took the ACT and went on to enroll in 4-year institutions from 2000 to 2006.

The nut:

After corrections for range restriction, the estimated mean correlation between ACT scores and 1st-year GPA was .51, and the estimated mean correlation between high school GPA and 1st-year GPA was .58. In addition, the validity coefficients for ACT Composite score and high school GPA were found to be somewhat variable across institutions, with 90% of the coefficients estimated to fall between .43 and .60, and between .49 and .68, respectively (as indicated by the 90% credibility intervals). In contrast, after correcting for artifacts, the estimated mean correlation between SES and 1st-year GPA was only .24 and did not vary across institutions….

…1st-year GPA, the most proximal predictor of 2nd-year retention, had the strongest relationship (.41). ACT Composite scores (.19) and high school GPA (.21) were similar in the strength of their relationships with 2nd-year retention, and SES had the weakest relationship with 2nd-year retention (.10).

The results should be familiar to anyone who has taken a good look at the literature on these tests, and to anyone who has been a regular reader of this blog. The ACT is in fact a pretty strong predictor of GPA, though far from a perfect one at .51. Context is key here; in the world of social sciences and education, .51 is an impressive degree of predictive validity for the criterion of interest. But there’s lots of wiggle! And I think that’s ultimately a good thing; it permits us to recognize that there are a variety of ways to effectively navigate the challenges of the college experience… and to fail to do so. (As the Study of the Week post linked to above notes, GPA is strongly influenced by Conscientiousness, the part of the Five Factor Model associated with persistence and delaying gratification.) We live in a world of variability, and no test can ever make perfectly accurate predictions about who will succeed or fail. Exceptions abound. Proponents of these tests will say, though, that they are probably much more valid predictors of college grades and dropout rates than more subjective criteria like essays and extracurricular activities. And they have a point.

Does the fact that SES correlates “only” at .24 with college GPA mean SES doesn’t matter? Of course not. That level of correlation for a variable that is truly construct-irrelevant and which has such obvious social justice dimensions is notable even if its less powerful than some would suspect. It simply means that we should take care not to exaggerate that relationship, or the relationship between SES and performance on tests like the ACT and SAT, which is similar at about .25 in the best data known to me. Again: clearly that is a relevant relationship, and clearly it does not support the notion that these tests only reflect differences in SES.

Ultimately, every read I have of the extant evidence demonstrates that tests like the SAT and ACT are moderately to highly effective at predicting which students will succeed in terms of college GPA and retention rates. They are not perfect and should not be treated as such, so we should use other types of evidence such as high school grades and other, “soft” factors in our college admissions procedures – in other words, what we already do – if we’re primarily concerned with screening for prerequisite ability. Does that mean I have no objections to these tests or their use? Not at all. It just means that I want to make the right kinds of criticisms.

Don’t Criticize Strength, Criticize Weakness

A theme that I will return to again and again in this space is that we need to consider education and its place in society from a high enough level to think coherently. Critics of the SAT and ACT tend to pitch their criticisms at a level that does them no good.

So take this piece in Slate from a couple enthusiastic SAT (and IQ) proponent. In it, they take several liberal academics to task for making inaccurate claims about the SAT, in particular the idea that the SAT only measures how well you take the SAT. As the authors say, the evidence against this is overwhelming; the SAT, like the ACT, is and has always been an effective predictor of college grades and retention rates, which is precisely what the test is mean to predict. The big testing companies invest a great deal of money and effort in making them predictively valid. (And a great deal of test taker time and effort, too, given that one section out of each given exam is “experimental,” unscored and used for the production of future tests.) When you attack the predictive validity of these tests – their ability to make meaningful predictions about who will succeed and who will fail at college – you are attacking them at their strongest point. It’s like their critics are deliberately making the weakest critique possible.

“These tests are only proxies for socioeconomic status” is a factually incorrect attempt to make a criticism of how our educational system replicates received advantage. It fails because it does not operate at the right level of perspective. Here’s a better version, my version: “these tests are part of an educational system that reflects a narrow definition of student success that is based on the needs of capitalism, rather than a fuller, more humanistic definition of what it means to be a good student.”

These tests do indeed tell us how well students are likely to do in college and in turn provide some evidence of how well they will do in the working world. But college, like our educational system as a whole, has been tuned to attend to the needs of the market rather than to the broader needs of humanity. The former privileges the kind of abstract processing and brute reasoning skills that tests are good at measuring and which makes one a good Facebook or Boeing employee. The latter would include things like ethical responsibility, aesthetic appreciation, elegance of expression, and dedication to equality, among other things, which tests are not well suited to measuring. A more egalitarian society would of course also have need for, and value, the raw processing power that we can test for effectively, but that strength would be correctly seen as just one value among many. To get there, though, we have to make much broader critiques and reforms of contemporary society than the “SAT just measures how well you take the SAT” crowd tend to engage in.

What I am asking for, in other words, is that we focus on telling the whole story rather than distorting what we know about part of the story. There is so much to criticize in our system and how it doles out rewards, so let’s attack weakness, not strength.

notes

Freddie
  • For some odd reason my last post, on public subsidies for wealthy Ivies in an era of austerity, did not get pushed out to RSS readers. Apparently that’s happened before. It’s frustrating and I’m not sure what’s happening. You can always follow the ANOVA’s Twitter account for new posts.
  • That post has been republished at Jacobin.
  • I was on the left-leaning military affairs podcast What a Hell of a Way to Die, talking about the GI Bill, recently.
  • I will be appearing on the Katie Halper Show on June 14th at Brooklyn Commons from 7 PM to 10 PM, with the brilliant Angela Nagle. It’s a fundraiser for WBAI which is well-worth supporting.
  • This past week’s book review, on “Rebekah Nathan”‘s My Freshman Year, has been delayed and will be pushed out to Patreon patrons tomorrow afternoon. Archival content for patrons is coming later today.
  • Sometimes I write about non-education stuff on Medium. Here’s me on podcasts.
  • A couple people have asked about my Academia.edu and ResearchGate profiles, so I’ll just note that I often forget those exist and they are rarely if ever updated, though I’m going to make an effort to get them up to speed this week.
  • Coming soon: posts on teacher observations, corpus linguistics, and regression.

two sets of universities, two countries, two futures

Freddie2 Comments
click for creator's Flickr page
image by Flickr user John Walker used under CC License

Today, Yale University’s 316th commencement will take place. Beaming young people and their proud parents will flock to the immaculate New Haven campus, eager to start their climb further up the ladder of American success. They know, as they surely knew the day they arrived, that their passage through such an august institution prepares them for a life of financial security and high social standing. They know, in other words, that as much as any young people, they are positioned to advance to the rarefied world of elite America.

Meanwhile, elsewhere in Connecticut, twelve community colleges and four public universities – including one found in the very same city – are starved to death by austerity and neoliberalism, as the Democrat governor and a Democratic state legislature in a rich blue state enact brutal cuts to education, social services, and mental health care, while fighting to cut taxes on corporations. The cuts to the Connecticut State University system are particularly devastating. They risk killing majors, shuttering departments, and destroying tenure. Programs that help shepherd a student body that comes disproportionately from non-traditional backgrounds, and thus needs help the most, are under threat. Classes may be cut from course schedules, making it even harder for working students and students who are parents to fit school into the schedules. In every way, a university system that already struggles to serve its students and its state thanks to resource constraints will be hurt even more.

These cuts are personal, for me, as I am a graduate of Central Connecticut State University in the CSU system. I will risk self-aggrandizement in saying that I am an example of the kind of success story that is routinely produced by the CSU system and systems like it. In my early 20s I was lost – orphaned, broke, alcoholic, struggling from then-undiagnosed mental illness, and completely without direction or a sense of purpose. But I took classes at the local community college for a year, then transferred to Central, where I met warm, engaging, committed educators who shepherded me through my education and showed me that I had skills and knowledge that had value – that my life had value. Today, I have a PhD, live in New York City, work at a wonderful public college myself, and have been published by some of the most prominent newspapers and magazines in the world. I owe all of that, without exception, to my time in the CSU system. It was there that I put my life back together, thanks to the dedication of the professionals who worked there and the relatively low tuition costs that enabled me to attend. I say with no exaggeration: the Connecticut State University system saved my life. And now, for shortfalls of less than $100 million a year, that system risks being permanently crippled.

To make all of this worse, down I-91 from my old university, Yale sits on a mountain of money, and yet receives more and more from public funds. The degree to which our government subsidizes the immensely wealthy Ivy League schools defies belief. A report from Open the Books, an organization that works for transparency in government spending, estimates that the federal and state governments spent over $40 billion on the Ivy League schools in tax exemptions, contracts, grants, and direct gifts from 2010 to 2015. The eight Ivy League universities – small, elite institutions from one region of the country that serve a tiny fraction of our college students and who could scarcely need government support less – receive more money annually from the federal government, on average, than 16 states. Four in ten students from the top 0.1% of families by income attend the Ivy League or similarly elite institutions; in 2012, 70% of Yale’s incoming freshmen came from families making more than $120,000; the median family income for Harvard students is triple the national average. The overwhelming majority of these students go on to lives of economic security, and many to the upper echelons of our economy.

Yet we continue to pour in government money to these rich institutions, and their wealthy alumni pour in hundreds of millions of dollars to their endowments untaxed, often invoking the spirit of giving and the need for equal opportunity while they do so. Meanwhile, we know empirically that systems like the CSU system, or the City University of New York system (where I now work), or the California State University system – America’s Great Working Class Colleges – do a far better job of creating social mobility than their elite counterparts. Yet each of these systems struggles under brutal cuts to their funding even though our country has never been richer.

What political philosophy, exactly, could possibly justify this condition? What ideology would conclude that this is a good use of resources, either public or philanthropic?

And yet the condition endures, even accelerates, year after year. No one seems to ask why those institutions who are objectively fabulously wealthy should receive such outlandish public subsidy, nor does anyone provide an answer as to why so many of our wealthiest continue to cut large checks for these institutions while our working class colleges, who need the money so desperately, starve. I am absolutely committed to the idea that higher education should be funded with public moneys, but I am also perplexed at the tendency of charitable donations to go where they are needed least of all. Where is Bill Gates to subsidize our working class colleges? Where is Mark Zuckerberg? Why does the philanthropic impulse, when it comes to higher education, always result in the rich getting richer? Connecticut is home to a small army of hedge fund managers and other incredibly wealthy types. I would love it if we could take their money by force for the good of all of society. But barring that, why don’t they use Connecticut’s starving public system for tax avoidance, rather than elite universities that are already filthy rich? Unless the entire point of such gifts is not to create equality of opportunity but to destroy it, to ensure that only those who start out at the top get to end up there. Our elite universities do many good things, but there is no question that they perpetuate and deepen inequality. That is in fact their most basic function: the replication of the ruling class.

I have no doubt that Yale’s class of 2017 is full of smart, talented, and passionate young people. I wish them the best. I also have no doubt that those among them who may not be talented or hardworking will be wholly inoculated from that condition thanks to the accidents of birth and privilege that helped them reach their rarefied station in the first place. As a socialist, I am not interested in making them more susceptible to material hardship and the vagaries of chance, but rather of giving everyone that same level of protection – and that means raiding the coffers of their school, their parents, and their future employers for the betterment of all. I also don’t doubt that, on balance, graduates of the Connecticut State system will succeed as well. College graduates writ large enjoy a substantial premium in income and unemployment rates over those without degrees, after all. But how hard will they have to struggle, as their instructors are stretched thinner and thinner by these brutal cuts? How many of them will sink deeper into debt as they are forced to take additional semesters of classes to complete their degrees? How many of them will drop out, thanks to these cuts, and suffer under the burden of student loan debt with no degree to help them secure a better life? How many people who could have been saved, as I was saved, now won’t be because of these cuts?

Today’s Yale commencement ceremony, of course, will be stocked with liberals, decent progressive folk who will tell you they believe in equality and social justice. The parents will mostly be liberal Democrats. The student ranks will be filled, no doubt, by genuine radicals, and the faculty with Marxists and socialists. They do good deeds at these places, such as how Yale’s community recently forced the school to change the name of Calhoun College, thanks to John C Calhoun’s history as a slave owner. I celebrate the activist zeal of all involved in such actions. Yet what Yale’s community can’t do – and perhaps wouldn’t, if it could – is to dismantle its place in the engine of American inequality. For all of the decent people involved in that institution, there is no chance that it will ever voluntarily abandon its role as an incubator of the ruling class. To do so would be unthinkable. That’s the reality of higher education: ostensible leftists preside over the ever-accelerating accumulation of power, money, and privilege. A better way is possible, but it cannot be achieved from within campus.

Until we reach that better world, we’ll be left with these ugly divides. In a sea of political ugliness it’s hard for me to imagine a more stark statement of America’s grand failures than this, a starving public university system that serves the poor and the brown and the needy, while next door a school for the 1% sits on $25 billion dollars, untaxed. CSU students, like Yale students, will walk on campus lawns with caps and gowns, eager to begin their new lives. Like Yale students, CSU students will seek a better life. But how many of them will be stuck here in this other America, inequality America, austerity America, while those who’ve already been given so much are given even more?

Correction: Fixed some inaccurate wording in the fourth paragraph.