Empirical SCOTUS: Follow the experts in framing petitions for cert

The age of big data is upon us. Data in law is ubiquitous, informative of past decisions and helpful in prognosticating future results. How we make use of this data is often a topic of debate. One view holds that large amounts of processed information can inform legal decision-making (Here are examples of how this looks in practice.). Lawyers taking this approach try to maximize the use of this abundant information. A particular area of Supreme Court practice in which big data can improve lawyers’ decisions is in the case-selection stage. With well over 7,000 annual petitions for certiorari, the justices and their clerks must wade through what may seem like an infinite number of pages in order to pare down to the 70 or so cases they hear in a term.

There are several signals that lawyers can send the justices and clerks to enhance their chances of a cert grant. One factor that plays a large role and is beyond a lawyer’s control in any given instance is a lawyer’s experience before the Supreme Court. Expert Supreme Court counsel are much more successful in persuading the justices to hear their cases than lawyers with less experience. There are multiple explanations for these results, some of which have been better analyzed than others.

Decades ago scholars recognized that amicus briefs filed at the cert stage affect the likelihood of a cert grant. Researchers recently posited a theory that lawyers’ ideologies affect the chances of a cert grant as well. While helping to explain factors in the justices’ decision making, these works focus on aspects outside of the cert petitions themselves. This short piece brings language into the fray with text analyses of a sample of cert petitions. The goal was to see if experts are using different and unique writing techniques to get the attention of the clerks reviewing cert petitions. Through these analyses, various differences between experts’ and non-experts’ petitions became quite evident.

Signals

With so many annual cert petitions, the justices’ clerks require mechanisms to help them move through each individual petition quickly. These mechanisms are often in the fashion of signals. Possible signals include the lawyer who authored the petition, amici curiae at the cert stage and the language of the petition. In particular, the overall language of the petition as well as specific word choice may affect a clerk’s focus on a given petition.

The “bible” on Supreme Court practice, Stern and Gressman’s (now with various additional co-authors) Supreme Court Practice, describes case dimensions that might lead to a greater likelihood of cert. Lawyers at the firm Mayer Brown have done a good job of digesting material from this tome among others. These pieces in turn provide examples of language that might catch clerks’ eyes when reviewing cert petitions.

Sample

With so many annual cert petitions, there is no shortage of primary-source research material. This analysis is based on 103 cert petitions. The initial plan was to review 100 petitions, but the results contained three cases that were decided by a grant, vacate and remand (GVR). These are not as informative as clear cert grants or denials because many of the decisions in these instances are predicated on similar cases that the court already decided.

The 103 petitions were drawn from random stratified samples from the beginning of 2017 to the present. Data from this set, including the cases and dimensions of each petition, can be found here (login required). Because petitions from non-experts far outnumber those from experts, the original goal was for an overall sample composed of about three quarters non-expert petitions and one quarter expert petitions (As it turns out, the actual numbers were 27 from experts and 76 from non-experts.). The petitions were chosen from a series of sequences of docket numbers. Some petitions were discarded and others retained to balance out the proportion of expert to non-expert petitions. Because the docket numbers are in groups and proximate docket numbers within the groups indicate petitions that were filed on similar dates, there may be some impact on a petition’s success resulting from when a petition was filed, although the diversity of dates within the overall period should minimize this. I also excluded petitions written by a party not represented by counsel (“pro se” petitions).

Supreme Court experts were defined from a series of sources including Chambers & Partners’ top-ranked appellate attorneys and Reuters’ Echo Chamber, and from a review of repeat advocates before the justices over the past several terms. Lawyers from firms with successful Supreme Court practices who were not recognized Supreme Court experts themselves were included in the non-expert group.

Macro view

When the three GVRs were discounted, six of the 100 or 6 percent of the total set of petitions were granted. All six as well as the GVRs came from the experts’ pool of 27 petitions. This means that experts were successful in 25 percent of their petitions excluding the GVRs and 33 percent of their petitions when the GVRs were included.

The expert petitions were from attorneys including Carter Phillips, Jeffrey Fisher, Tom Goldstein and other veterans of this terrain. Counsel of record on the six granted petitions were David Boies in Animal Science Products v. Hebei Welcome Pharmaceutical Co., Theodore Boutrous in New Prime Inc. v. Oliveira, Adam Unikowsky in Marinello v. United States, Kannon Shanmugam in Dahda v. United States, Pam Karlan in Lozman v. The City of Riviera Beach and Jeffrey Wall in United States v. Microsoft.

Because the experts in the overall sample fared so much better than the non-experts, one obvious question is whether the experts did anything differently. Expert lawyers had much more support from external organizations, in the form of amici, than non-experts: Experts’ petitions in the sample had 1.7 associated amicus briefs on average while non-experts averaged only 0.3 amicus briefs per petition.

Another area of difference that was well within the purview of petitioning attorneys was in the text of the petitions themselves. Lawyers, however, have very little information from the court on what the justices are looking for when reviewing petitions. The entirety of the court’s contribution to this subject is in Supreme Court Rule 10, which describes the various types “compelling reasons” for cert grants.

The factors described in Rule 10 center around conflicts between various federal and state courts. Although these conflicts play a significant role in the justices’ decisions about which cases to take, they are not the only considerations. Texts like “Supreme Court Practice” help supplement Rule 10 by providing other factors that the justices and clerks look for when reviewing petitions.

The first part of this analysis looks at three parts of the petitions individually – the question presented in the introduction, the section headings and the central petition text. Each of these parts of the petition serves a unique purpose in the clerks’ (and justices’) review.

A simple area to observe for differences is the length of each of these petition components. The box-plot distribution graphs show the median word length in the middle of each box between the two shades of gray, with the distributions extending outwards. Each blue dot represents the length of an individual petition and dots outside of the lines reflect outliers that are far from the median. The combined median of the experts’ and non-experts’ length is depicted with the gray line running across the graph.

The first graph is of the lengths of the questions presented.

Both experts’ and non-experts’ question-section distributions center around the overall median of 117 words. The non-experts’ distribution has a much greater spread about the median, with two question sections exceeding 400 words. While some of the non-experts’ question sections exceed those for the experts, the differences here are not of a great magnitude.

Next, looking at the section headings for the reasons to grant the petition, the differences between experts’ and non-experts’ headings are more noticeable.

The experts’ section headings were somewhat densely packed between 40 and 170 words. Although the median length for non-experts was similar to that for experts, their spreads were quite divergent. Several non-experts’ petitions lacked section headings, which can be quite harmful to the chances of a petition’s success. A handful of others exceeded the 170-word-length upper limit for experts’ headings, with several well over 200 words.

The biggest difference between the sets of petitions was in the area with the most abundant substance – the petition text.

As with the section headings, spread of the non-experts’ petition text was much greater than that for the experts. The experts’ median was almost 2,000 words greater than the non-experts’, at close to 7,500 words compared to approximately 5,500 for non-experts. All of the experts’ petition-text length was in the range of just under 4,000 words to nearly 9,000 words, but the lengths of many non-experts’ petitions were well under 4,000 words, with some approaching as few as 1,000 words. Such differences and especially the tightly packed nature of the experts’ petition lengths are emphasized in histograms that compare the two distributions (These figures count the number of petitions within buckets of 500 words.).

All of these disparities, especially those related to the petition-text length, should provide some insight for petition drafters into the approximate lengths to strive for. Although a good petition might be shorter or longer than the experts’ petitions in this sample, the lack of non-experts’ success in this sample of petitions intimates that the guidelines followed by the experts may be good general standards.

The petition length was not the only area in which experts’ and non-experts’ petitions diverged. Another general area of divergence was in overall word choice. To gauge the word choice in these petitions the text was first pre-processed through a series of steps, including eliminating stop-words, generating two word n-grams (bigrams), and stemming.

Once pre-processed, the frequency of single- and two-word n-grams was calculated. The most frequent n-grams in the three sections – question presented, section headings and overall text — in terms of appearing in the overall number of documents are analyzed below. These analyses convey the percentage of documents in which the terms appear in the experts’ set and non-experts’ set.

Many similar top terms appear in both the experts’ and non-experts’ question-presented sections, although the relative usage of these terms differs between the two.

These terms are not particularly telling, because they mostly regurgitate the main aspects of this section with “question,” “presented” and “question_present,” although many petitions lack some or all of these n-grams. Two terms that only appear in the experts’ top terms for questions are “held” and “feder” (the stem for terms including “federal” and “federally”). This distinction is notable because both the lower court’s holding and the federal versus state court distinction may affect the justices’ ultimate decision on cert.

The differences in the section headings within the samples are particularly illuminating.

At first blush both sets of section headings have a high frequency of n-grams, including the term “conflict,” which follows the guidance from Rule 10. The most interesting difference between the two sets has to do with n-grams that include the stem “vehic.” A proper or good vehicle for the court involves a case that will resolve the primary issue and will not lead to resolution or break down based on less important, subsidiary issues. The non-experts’ functional equivalent for a good vehicle may come in the form of a “reason” or “reason_grant,” although these n-grams do not convey the same essential point. The quality of the case in the form of the “vehicle” may very well play a significant role in clerks’ and justices’ assessment of petitions.

In the figure for the top n-grams in the petitions’ texts, the two sets have partially similar and partially different focal n-grams.

The substance of the differences between the two sets of petitions is not clear at the outset. Several n-grams including “court,” “grant” and “reason” appear in 100 percent of petitions. All of the experts’ petitions included the stemmed n-grams “appli,” “requir,” and “reason_grant.” Although the inclusion of these n-grams and the exclusion of others very likely do not make a significant difference in a petition’s chances of success, the differences in experts’ and non-experts’ patterns in petition drafting may reflect overall differences in approach that do appear to affect a petition’s likelihood of success.

Particular phrases

Digging into particular n-gram usage exposes greater differences between the two sets of petitions. Each n-gram analysis includes three graphs comparing the n-grams used in the text of experts’ and non-experts’ petitions. The first graph looks at the percentage of documents in which a given n-gram appears in each set. The second looks at the overall document count in which the n-gram appears in each set. The third looks at the count of times the n-gram appears in either set, which may include multiple times within the same petitions.

The text of “Supreme Court Practice” helped define terms of interest. As the section-headings figure above conveyed, some of the greatest differences were in the use of n-grams that included the stem “vehic.”

Experts appeared much more cognizant of the importance of a good vehicle for the court, as multiple n-grams including “vehic” appear in experts’ petitions in higher frequency and several were not used by non-experts at all.

The head of Orrick’s appellate practice, Josh Rosenkranz, presented an example of the importance of a good vehicle in his petition in EchoStar Satellite, LLC v. Florida Department of Revenue, which was ultimately denied.

Rosenkranz highlighted the vehicle concern in a section heading directing a reader’s attention to the importance of the case within this particular area. Although the petitioning attorney needs to support this point with sufficient evidence, a clerk reading the petition may initially recognize that the petitioner’s case is sufficiently important from framing that includes these key terms or n-grams.

Experts appeared more invested in good vehicles, but both experts and non-experts seemed clear on the importance of a conflict between courts.

Many of the n-grams above appear in a high-percentage of both sets of petitions. A few of the differences in n-gram use are still worth highlighting. A significant disparity exists in the usage of the n-gram “acknowledge_conflict.” The additional facet that a conflict is evident to others outside of a given case may be a helpful indicator of the impact of a case’s decision.

The one “conflict” n-gram used in multiple experts’ petitions and in no non-experts’ petitions was “intract_conflict.” A conflict that is truly intractable is appropriate fodder for the court’s adjudication, because without a resolution of such conflicts, disparities in laws would exist across the nation. Stanford Supreme Court Litigation Clinic Co-Director Jeffrey Fisher presented an apt example of the n-gram “intract_conflict” in the petition in Evans v. Georgia Regional Hospital, which was eventually denied by the court.

As Fisher described, the difference in courts’ interpretations of Title VII presented “uncertainty for employees and employers” as well as “compounding pervasive discrimination.” These differences in treatment are strong indicia that a case is suitable for the court’s adjudication.

A third area assessed by the court and described in the “Supreme Court Practice” text is the relative importance of a case. “Importance” has multiple connotations and so the most-used n-grams are helpful in disentangling particular aspects of importance.

Although both experts’ and non-experts’ petitions tended to highlight the importance of cases, they did so in different ways. Only experts’ petitions brought up the n-grams “import_circuit” and “import_issu,” indicating the importance of the geographic region and of the substance of the petition. Deepak Gupta’s petition in Donziger v. Chevron Corp. (later denied) provides a good example of how to describe the case both as an “ideal vehicle” and as one of “exceptional importance.”

This snippet from the Donziger petition presents multiple n-grams that underscore the case’s importance to the court. Beyond the importance of the issue conveyed in the petition and the vehicle for resolution, this small section describes other elements often relevant to the court, such as a “deepened … circuit split.” Within this section heading and three lines of text, Gupta provided at least three reasons for why the justices should hear this case.

This idea of the generalized importance of a case that Gupta conveys in the petition above was notable in both sets of petitions. Both sets focused on when the issue in the petition was of “nation_import,” or national importance, although a greater portion of the experts’ petitions than non-experts’ petitions noted this distinction. Finally, only experts highlighted the “practic_import,” or practical importance, of the issues in a case, which may have led to significant distinctions between the two sets of petitions. The fact that an issue is of practical importance brings it beyond the theoretical into a zone that directly affects portions of the public, and the court would much rather adjudicate disputes surrounding actual rather than hypothetical concerns.

Beyond n-grams including “importance,” petitioners use a variety of other terms and phrases to draw out the unique significance of cases. Several of these techniques are provided in the set of n-grams below.

All of the n-grams above relate to aspects of petitions described in “Supreme Court Practice” and all are used in greater frequency in experts’ petitions. Several of the n-grams relate to “splits,” which are another way to convey conflicts between courts’ interpretations. Another way that experts convey the relevance of a case is through the confusion (“confus”) created by disparate decisions across the nation. The differences between courts’ interpretations may be especially significant when they surround cases that appear similar on their face (“circuit_similari”).

Just as the justices want a good vehicle for a case, they also want to make sure the issue has been thoroughly resolved by the courts below. Experts described this “percolation” in a much greater portion of petitions than non-experts. Josh Rosenkranz’s petition in EchoStar provides context for how sufficient percolation can be effectively conveyed.

Here, Rosenkranz lets the court know that there are stark differences in courts’ interpretations of the same issue and that without the court’s resolution there is little hope for any geographic consistency in these rules.

Another set of n-grams that appeared more frequently in experts’ petitions involve the term “square.” “Square” is another way of intimating that something is on point: A square conflict may present an “ideal vehicle” for the court to resolve. Below is an example of this usage from Kannon Shanmugam’s granted petition in Dahda v. United States.

Many of the other n-grams in the set above go to the similar point of the unique qualities of a case that make it well-suited for the court’s resolution. Even though all petitions are capable of including these n-grams, the differences between experts’ and non-experts’ uses are stark.

None of the differences between experts’ and non-experts’ petitions highlighted above are likely to change the outcome of a petition in isolation. But with a finite amount of time for review, clerks and justices need shorthand ways to make their way through large numbers of petitions. The lack of artful terminology may hinder a petition’s likelihood of success, especially if the reader is skimming to get the gist of a petition in a quick initial read. Conversely, well-scripted petitions may attract a reader’s eye, helping a petition make its way to the top of a set of petitions that get reviewed in greater depth. The difference in the success of experts’ petitions and non-experts’ petitions from this set alone is clear. The cause of the difference breaks down to multiple factors. Although experience in the form of reputation almost assuredly plays a role in a petition’s likelihood of success, so does experience in the form of clearly articulating the importance substance of a case. Reputational differences take time, energy and luck to overcome, but differences in petition-drafting language may be more readily surmounted.

This post was originally published at Empirical SCOTUS.