All hat, no cattle? Mismatch and Fisher v. University of Texas at Austin

In a recent SCOTUSblog interview, Richard Sander and Stuart Taylor argue that consideration of diversity as currently practiced in admissions harms “many or even most of the black and Hispanic supposed beneficiaries by subjecting them to the mismatch effect.” Because Sander and Taylor write so confidently, many readers will be unaware that the key studies on which their claims are based have never passed the test of peer review, and several remain unpublished. Take, for example, their claims about mismatch and African-American law students. Numerous attempts to replicate Sander’s (and Taylor’s) claimed results have been unsuccessful. The research consensus that emerges – not just from what two of us have written, but also from independent analyses and reviews by Ayres & Brooks; Dauber; Wilkins; Rothstein & Yoon; Ho; Holzer & Neumark; Barnes; Camilli & Jackson and Camilli & Welner – is that Sander has never provided reliable evidence of systemic mismatch effects in U.S. legal education, particularly at the most selective schools.

Someone who is not familiar with the relevant science, the tenets of transparent and well-warranted research, and well-earned scientific reputations may not readily appreciate how devastating  several amicus briefs submitted in Fisher v. University of Texas at Austin are to Sander and Taylor’s claim.  One brief, filed by leading individual scientists, cautions the Court against giving any credence to the principal studies Sander and Taylor cite in support of their mismatch claim. It has as its lead signers Donald Rubin, Gary King, Richard Berk, and Guido Imbens – a foursome who have never staked out a position on affirmative action and may or may not be supporters. Rubin and King are leading scholars in statistics and political science, respectively, and both are elected members of the National Academy of Science. Berk, a sociologist/criminologist, and Imbens, an econometrician, are similarly among the leading statistical experts in their fields. Their brief identifies fatal design flaws in Sander’s work on mismatch and explains why his work and work by Doug Williams (whom we will discuss below) fails to meet basic standards of good science.

In the SCOTUSblog interview, Sander and Taylor’s main response to the methodologists’ brief is to claim that Sander’s “second choice” analysis shows that minority law students do better in less competitive environments. Putting aside the fact that other more rigorous analyses of the issue report very different results, the finding even if uncontroversial would still be hugely misleading. It ignores the fact that frequently the difference in selectivity between first- and second-choice schools (for example, the University of Michigan versus Columbia University) is often  small, meaning that the degree of mismatch is pretty much the same regardless of school attended.  It also ignores evidence suggesting that students often attend a second-choice school for financial reasons, including generous aid packages that are a tribute to their academic strength.

Sander and Taylor now claim that Sander’s earlier work is “confirmed” by economist Doug Williams, who has been a frequent co-author of Sander’s for decades. Williams’s works on law school mismatch are in unpublished working papers and have been critiqued as untethered to the real world and methodologically suspect (Camilli & Welner 2011). In the paper most prominently mentioned, half his sample’s black students are dropped from the analysis, and three-quarters of those he treats as presumably not “mismatched” attended historically black law schools, which in important ways are unlike the nearly two hundred other U.S. law schools. Relatedly, Professor Williams acknowledged at a recent conference that he cannot differentiate his findings from the positive effects that would result from being at a school with high numbers of minority students.

A more subtle claim that Sander and Taylor raise in their SCOTUSblog interview and amicus brief is that African-American and Latino students receive a huge advantage in the admissions process, as judged by the differences in average SAT scores. Such comparisons of mean SAT scores are again misleading, because test score gaps are not a proxy for the extent to which race is considered in admissions.  From 2005 until 2009, for UT-Austin freshmen outside the Top Ten Percent Plan, the black-white difference in mean SAT scores was 210 points (on the 1600 point scale).  But over the same span the size of black-white gaps in SAT scores is even larger at Sander’s home institution of UCLA (224 points) and at UC Berkeley (254 points), and that is after African-American enrollments had already plunged when affirmative action was prohibited. At selective universities and colleges, virtually any race-blind admissions system will yield racial/ethnic SAT score gaps similar to those highlighted by Sander and Taylor, because these gaps reflect the underlying distribution of test scores in the different ethnic populations.  Any racially neutral measure of merit, apart from the test scores themselves, will reproduce these population differences.

Sander and Taylor also assert that “mismatch is bound to be a serious problem for the racially preferred at UT.” However, UT-Austin’s latest report on student performance indicates that, for freshman entering between 2005 and 2009, the first-year persistence rates of African Americans are the same whether they were admitted under the formally race-neutral Top Ten Percent Plan (88.2%) or outside the Top Ten Percent, where race can serve as one of many plus factors (88.8%). The same goes for the persistence rates for Latinos at UT-Austin (86.7% versus 86.0%). Detailed and multi-year data on six-year graduation rates are not yet available for the cohorts at UT-Austin of interest, but federally reported six-year graduation rates are available for the 2005 entering class. UT-Austin’s closest comparison institution in many respects is Texas A&M University at College Station, which does not consider race in admissions. The Latino graduation rates are the same at these two Texas flagship universities (72%), and for African Americans the graduation rate is six points higher at UT-Austin (66% versus 60%).

Sander and Taylor’s thirty-six-page Fisher amicus brief on mismatch devotes one lonely paragraph to university-level graduation rates, although one might think that these are the best test of whether mismatch is harming students who benefited from consideration of diversity. Sander and Taylor cite two studies to support their criticism (Loury & Dratcher; Light & Strayer), but both use data that are now thirty to forty years old. By contrast, a substantial body of recent scholarship confirms that the opportunity to attend a selective institution is associated with African-American and Latino undergraduates achieving higher graduation rates. Relevant to Fisher, Cortes (2010) looked at many of Texas’s public universities and found that, after the Fifth Circuit’s decision in Hopwood barred race-conscious affirmative action, the graduation rates of African-American and Latino students ineligible for admission under the Top Ten Percent Plan fell further behind whites in Texas even though the mismatch theory would predict improvement.

Using multiple data sets and a range of analytical methods, a large number of peer-reviewed studies – including Alon & Tienda (2005), Small & Winship (2007), Fischer & Massey (2007), Melguizo (2008), and Bowen, Chingos & McPherson (2009) – all reach the conclusion that African-American and/or Latino graduation rates are higher at selective institutions, other things being equal.  Given the state of the overall social science regarding college graduation rates and beyond, the American Education Research Association (AERA) and several other top research associations including the American Association for the Advancement of Science and the National Academy of Engineering filed an amicus brief in Fisher concluding that the “educational harms resulting from the so-called ‘mismatch’ of minority students at selective institutions have not been established by the studies said to prove them and indeed are regularly contradicted by sounder and more widely accepted research.”

Until a few years ago, Sander argued that the most reliable way of measuring mismatch effects and overcoming the problem of “unobserved differences” (highlighted in the SCOTUSblog interview) was a 2002 labor market study by economists Dale and Krueger that used statistical matching. But as Dale and Krueger recognized all along, their study had too few African Americans to allow conclusions about minority mismatch. With better data now available, Dale and Krueger have examined this issue more adeptly. Their conclusion:  For African-American and Latino students, the labor market benefits of choosing a more selective college “remain large, even in models that adjust for unobserved student characteristics.” This is consistent with other labor market evidence that weighs against the mismatch hypothesis (Long 2010).

In Texas, the phrase “All hat, no cattle” is used to describe folks who have plenty to say but lack substantiation and follow-through. The phrase nicely captures the jarring contrast in Fisher between the carefully orchestrated presence of Sander and Taylor in the public discourse – a new book plus pieces in The Washington Post, Wall Street Journal, Los Angeles Times, Atlantic Monthly, SCOTUSblog, and so on— and the cumulative peer-reviewed findings and conclusions of the scholarly community writ large. Because of the absence of peer review, failures of replication, and lack of general acceptance in relevant scientific communities, the “evidence” that Sander and Taylor have offered for the Court’s guidance in Fisher would be rejected by any trial court that took Daubert v. Merrell Dow Pharmaceutical’s standards for the admission of scientific evidence seriously. If the Supreme Court thinks that empirical data are relevant to their decision in Fisher, the Justices should study closely all of the social science amicus briefs.  Sander and Taylor’s success in advancing their views through an uncritical media is wildly disproportionate to the cumulative and compelling research evidence.

Richard Lempert is the Eric Stein Distinguished University Professor Emeritus in Law and Sociology at the University of Michigan; William Kidder is the Assistant Provost at the University of California, Riverside; Felice J. Levine is Executive Director of the American Educational Research Association