The “full” method of measuring the Court’s review of decisions by the courts of appeals: October Term 2011
on Oct 23, 2012 at 10:59 am
This blog occasionally publishes longer posts commenting on trends in the study of the Supreme Court. In this post, John S. Summers and Michael J. Newman – who are shareholder and associate, respectively, in the litigation department of Hangley Aronchick Segal Pudlin & Schiller – examine one common Supreme Court metric: circuit reversal rates. The authors very much appreciate the statistical assistance of Michael T. Cliff, Ph.D., Vice President, Analysis Group, Inc.
The traditional measure of the Supreme Court’s review of the courts of appeals, for any given Term, is the “circuit reversal rate” – that is, the ratio of the number of cases in which the Court reversed a particular circuit divided by the total number of cases from that circuit that the Court decided on the merits. We have criticized this method, used by observers such as the Harvard Law Review and SCOTUSblog, on the ground that it provides a less complete picture of the Court’s review than our alternative “full” measure of reversals. (For more criticism of the traditional method and more on our full measure, see our article Towards a Better Measure and Understanding of U.S. Supreme Court Review of Courts of Appeals Decisions.) For cases in which there is a circuit split, our measure looks not only at whether the Court affirmed or reversed the actual case that is the subject of an appeal, but also at the other circuits’ “shadow decisions” – that is, those circuit decisions identified by the Court as part of a circuit split. The full measure also focuses on whether the Court agrees or disagrees with the approach taken by the courts of appeals, rather than simply on whether it reverses or affirms those courts.
The traditional measure has two significant shortcomings: it is incomplete and outcome driven. By contrast, the full method suffers from neither of these failings and, in our view, is more accurate in measuring the frequency with which the Supreme Court agrees with the approach of a particular circuit.
It is incomplete because it fails to pick up the wealth of information, beyond that contained in the decision under direct review, included in the Court’s review of decisions by the courts of appeals that are the subject of circuit splits. For instance, when the Court reviews the decision of the First Circuit on an issue on which the Second and Third Circuits have also ruled, the traditional measure only captures the reversal (or affirmance) of the First Circuit, while the full measure also includes the Court’s implicit reversal (or affirmance) of the Second and Third Circuits.
The traditional measure also suffers from being outcome-driven because it looks only at whether the Court affirms or reverses, not at whether it accepts the reasoning or approach of the court of appeals. The Court may, for example, affirm the ruling of the court of appeals, but on alternative legal grounds. The traditional approach would consider this an affirmance, even though the Court rejected the reasoning of the court below. The full measure would count such a ruling as a reversal.
The incompleteness of the traditional measure is quantitatively striking. Over the Roberts Court’s first seven years (October Term 2005 through October Term 2011), the Court has heard 457 merits decisions from the federal courts of appeals; if the shadow decisions are included, that number increases to 1219. Last Term, for example, the Court heard 71 merits cases from the courts of appeals, but the inclusion of shadow decisions more than doubles that number to 150.
Table One below displays the traditional and full reversal rates, by circuit, for the 2005 to 2010 Terms and the 2011 Term. (For those readers interested in a discussion of our methodology and more disaggregated statistics, see our project website.) These data permit us to examine (1) whether in the October Term 2011 the Court reversed the circuits more or less frequently than the past; (2) which courts of appeals were most and least reversed in OT2011; and (3) how specific courts of appeals performed in 2011, compared to prior years. In addition, from the data, we constructed concordance tables that permit us to determine the extent to which each circuit agreed (or disagreed) with each other circuit on the cases that comprised the circuit splits.
TABLE 1: SUPREME COURT REVERSALS OF LOWER COURT DECISIONS BY TERM AND CIRCUIT (2005 – 2011)
1. Did the Supreme Court reverse the courts of appeals less frequently in OT2011 than the Roberts Court’s historic average?
According to the conventional wisdom of the traditional reversal measure, the Roberts Court reversed the courts of appeals less in OT2011 (61.6%) than during the previous six years (73.1%). The full measure demonstrates that this conclusion is misleading. In fact, under the full measure, the Roberts Court reversal rate held to a steady 56% (56.4% over OT2005-2010 v. 57.1% for OT2011.)
2. Which courts of appeals were the most and least reversed in OT2011?
The most reversed circuits, as measured by the full measure, were the Federal (75%), Second (72.7%), and Eighth (71.4%) Circuits; the least reversed were the Fourth (22.2%), Seventh (33.3%), and D.C. (33.3%).
The traditional reversal measure failed to pick up consistently these results. While it correctly identifies the Second (100%) as among the three most reversed, it (mis)identifies the Sixth (100%) and Eleventh (75%) as most reversed and does not pick up that the Federal Circuit (66.7%) was so frequently reversed. The traditional measure fares worse in identifying the least reversed circuits; it fails to identify the Seventh (66.7%) Circuit as a rarely reversed circuit and mistakenly identifies the Fifth (0%) as a low-reversal circuit.
3. Which courts of appeals were reversed less in OT2011 than prior years?
Let’s start with the circuits that did best and worst in OT2011 compared to their Roberts Court track record. The Fourth Circuit stands out as performing relatively best, halving its historic reversal rate (52.7%) to 22.2% in OT2011. Which circuit’s reversal rate has increased dramatically? The Second Circuit, whose historic reversal rate of 51.9% increased to 72.7%.
Now let’s look at the performance of the three most- and least-reversed circuits. From OT2005 to OT2010, the Ninth, Sixth, and Federal Circuits were the three most-reversed circuits. In OT2011, the Ninth Circuit improved slightly (68.9% v. 64.7% and dropping from the most-reversed to the sixth most-reversed circuits), the Sixth Circuit remained unchanged at about 66% (66.3% v. 69.2%), and the Federal Circuit was reversed substantially more (63.3% v. 75.0%), becoming the most reversed circuit. The least-reversed Circuits historically were the Third, Tenth, and Seventh Circuits; in OT2011, the Third Circuit did marginally worse (42.7% v. 46.7%), the Tenth Circuit substantially worse (44.4% v. 58.3%) and the Seventh Circuit substantially better (48.0% v. 33.3%).
Our emphasis on reversal rates invites a few comments and caveats about the significance and normative content of such rates. What does it mean to say that a circuit enjoys a lower reversal rate than another circuit (or a lower rate than in past years)? The most apparent meaning is the following: the job of the courts of appeals is to apply and interpret the law as the Supreme Court would apply it, to the extent such guidance exists. Thus, the Supreme Court’s ex post reversal (or affirmance) of a circuit’s approach on an issue is a good measure of the circuit’s ex ante prediction of how the Court would resolve the issue. The reversal rate, in this way, is a reasonable normative measure of how well a court is doing its job. To the extent that is true, all the more reason for commentators to focus on the better, full reversal rate measure.
One could say, however, in the absence of a Supreme Court opinion on a given subject – say the interpretation of specific federal statute – the job of each court of appeals is to do its own best to interpret the statute. A range of approaches across the circuits, moreover, may be arguably more helpful to the Supreme Court than if the circuits are unified because the Supreme Court can then better select the best interpretation among the developed diversity of approaches. Thus, even if a circuit’s approach is not ultimately accepted, its work can be important in contributing to the development of the law. While we agree, it also seems difficult to disagree with the notion that the circuit that contributed to that development and accurately predicted the Supreme Court’s ruling was, at the margin, even more helpful to the Court. Further, in comparing the reversal rate of two courts of appeals or comparing a given circuit’s reversal rate over time, it seems that, all else equal, a circuit with a lower reversal rate in a given year is doing a better job than one with a higher one.
A final observation comes from the growing body of important work by legal empiricists. These scholars properly note that the cases that appellate courts hear are not random but rather a formation of the cases that litigants choose to appeal. When a party loses, it decides whether to drop the case, appeal it, or – with the agreement of its opponent – settle it. An appeal is only one of three potential outcomes. Take the simple case of a defendant’s decision whether to appeal a trial court verdict in favor of a plaintiff. If the trial judge made a serious and relatively obvious error (say, incorrectly instructing the jury on an important matter of law), the likelihood of reversal should be high. In a world where, among other things, the parties and their counsel share similar and good information and have comparable resources, the parties and their counsel should both know that the case will likely be reversed on appeal and therefore settle the case without the benefit of an appeal. The settlement amount should reflect the parties’ likelihood of success on appeal as they are negotiating in the shadow of the outcome of their potential appeal. The parties, therefore, don’t need the appellate court to correct the error. They just need the right to appeal to that court. As a consequence, these legal empiricists argue that cases with relatively obvious and significant errors may not be appealed and never included in a court’s reversal rate.
Conversely, the cases that are appealed are the ones where the likelihood of success on appeal is unclear. This could be because, for example, the trial judge’s instruction could turn on an important and unsettled issue or the trial judge’s alleged error may or may not have been significant.
A key insight from this approach is that clients and their counsel, all other things being equal, self-select cases to appeal that have less clear answers. This suggests that one’s intuition about what percentage reversal rate is “good” or “bad” is less reliable than you might think. Is 30% (or 70%) low (or high)? In the abstract, we submit, it is not at all clear because the rate is importantly driven by the parties’ self-selection of cases to appeal, not just the correctness (or incorrectness) of the reviewed courts’ decisions. Importantly, however, this insight does not change the importance of what we are focusing on, namely a circuit’s relative reversal rates (i.e., one circuit relative to another or one circuit compared to itself over time). Unless there is some demonstrable difference in the mix of cases appealed from the various circuits (or over time), the differences in reversal rates across the circuits or changes over time remain instructive. And to do that, the full measure is clearly superior for the reasons discussed above.
4. Concordance among the courts of appeals
We can also use our data set to examine the frequency with which the courts of appeals agree or disagree with each other by constructing circuit court concordance tables. This is analogous to the Justice concordance tables published by SCOTUSblog and others displaying the voting patterns of the Justices of the Supreme Court, identifying the frequencies with which each Justice votes with every other Justice.
Tables 2 and 3 below show the degree to which the circuits agreed with each other in circuit split cases, for the period OT2005-10 and then for OT2011. The agreement percentages are calculated as the number of agreements (both circuits affirmed or both reversed) divided by the sum of agreements and disagreements between the two circuits.
TABLE 2: PERCENTAGE AGREEMENTS IN RULINGS OF THE COURTS OF APPEALS (OT2005-2010)
TABLE 3: PERCENTAGE AGREEMENTS IN RULINGS OF THE COURTS OF APPEALS (OT2011)
These tables permit us to answer the question whether the most (and least) aligned circuits historically stayed true to that pattern in OT2011. From OT 2005 to 2010, the most aligned circuits were the Fifth and Tenth (80.6%), First and Second (73.5%), and Seventh and D.C. (73.0%). In OT2011, however, the most aligned circuits — each with 100% agreement — were the Fourth and Fifth, Second and Eighth, First and Tenth, and Second and Tenth. The First and Second Circuits agreed significantly less often in OT2011 (60.0%) while the Fifth and Tenth Circuits agreed about as often as in the past (80.0%).
The circuits that agreed the least often, historically, were the Ninth and Fourth (40.0%) and Ninth and Seventh (38.8%). This result is consistent with the conventional wisdom that the more conservative Fourth and Seventh Circuits are at odds with the most liberal Ninth Circuit. This pattern continued in OT2011 for both the Seventh (33.8%) and Fourth (40.0%) Circuits. Finally, there was a five-way tie for the circuits that agreed the least (First-Seventh, First-Ninth, Seventh-Ninth, Ninth-Eleventh, and Tenth-Eleventh), with each agreeing in only one-third of the cases.
