The Supreme Court Forecasting Project: Legal and Political Science Approaches to Predicting Supreme Court Decisionmaking

Citation

Ruger, T. W., Kim, P. T., Martin, A. D., & Quinn, K. M. (2004). The Supreme Court Forecasting Project: Legal and political science approaches to predicting Supreme Court decision-making. Retrieved from https://scholarship.law.upenn.edu/cgi/viewcontent.cgi?referer=&httpsredir=1&article=1671&context=faculty_scholarship

Key Takeaways

A simple classification‑tree model using only general case characteristics can outperform leading legal experts at predicting Supreme Court case outcomes; The model’s edge stems largely from better forecasting of swing Justices, particularly O’Connor and Kennedy, whose votes often determine results; Experts retain an advantage in highly technical judicial‑power and procedural cases where fine doctrinal detail drives decisions; In broad economic and business disputes, systematic behavioral patterns make data‑driven forecasting especially powerful relative to intuition; Practicing Supreme Court advocates predict more accurately than academics, while clerkship experience adds limited incremental value; Empirical judicial forecasting is a practical complement to doctrinal analysis for litigation strategy, risk assessment, and appellate decisionmaking.

Dataset Description

Training data consisted of all 628 argued cases decided by the stable nine‑Justice Rehnquist Court from the 1994 through 2001 Terms (“natural court”). For prospective testing, the authors forecast every case argued in the Supreme Court’s 2002 Term (76 cases; 68 used for case‑outcome analysis and 67 for individual‑vote analysis after exclusions). Ex ante predictor variables, drawn primarily from the U.S. Supreme Court Judicial Database (Spaeth) and coded from lower‑court opinions and merits briefs, included circuit of origin, issue area, party configuration, ideological direction of the lower‑court decision, and whether the petitioner challenged a law or practice as unconstitutional. The comparison sample comprised 83 legal experts (71 academics, 12 practicing appellate advocates, and 38 former Supreme Court clerks, with some overlap), each of whom made pre‑argument predictions for assigned 2002 Term cases and later completed surveys on the factors they believed influenced case outcomes and their own forecasts. The jurisdiction is the U.S. Supreme Court; the primary time period studied is the Rehnquist Court from 1994–2002.

Key Findings

Using only six coarse, pre‑argument variables in a classification‑tree model, the authors correctly predicted 75% of the Supreme Court’s affirm/reverse outcomes in argued cases from the 2002 Term, while individual legal experts achieved 59.1% accuracy, and aggregated expert majorities reached 65.6%. At the level of individual Justice votes, overall accuracy between the model and the experts was comparable, but performance differed by type of Justice: experts were better at predicting the ideologically extreme Justices, whereas the model performed relatively well in forecasting the pivotal centrist Justices, particularly Justices O’Connor and Kennedy, whose votes were most often decisive in close cases. This advantage of swing Justices largely explains the model’s superior case‑level accuracy. Predictive performance also varied by legal issue area. Experts significantly outperformed the model in “judicial power” cases involving jurisdiction, justiciability, and institutional rules, where detailed doctrinal knowledge and procedural subtleties matter most. In contrast, the model substantially outperformed the experts in the broad “economic activity” category, including business, commercial, regulatory, and some statutory disputes, where general ideological and institutional patterns appear to dominate over fine‑grained doctrinal reasoning. Within the expert pool, active Supreme Court practitioners predicted outcomes more accurately than academics, suggesting that repeated exposure to real cases, client pressures, and post‑hoc outcomes produces better calibrated intuitions than purely scholarly doctrinal expertise; prior clerkship experience with the Court did not yield a strong systematic advantage. The study concludes that Supreme Court decisionmaking is structured and patterned enough that a relatively simple, transparent empirical model, without access to briefs, oral arguments, or detailed doctrinal analysis, can outperform seasoned experts on bottom‑line results, especially in ideologically charged or policy‑salient disputes, although it does not substitute for doctrinal analysis in highly technical domains or in understanding the content and reasoning of opinions.

Summary

This article presents the Supreme Court Forecasting Project, a rare, fully prospective test of whether a statistical model can rival or exceed expert legal judgment in predicting U.S. Supreme Court decisions. The authors construct a classification‑tree model trained on all 628 argued cases decided by the stable nine‑Justice Rehnquist Court from the 1994–2001 Terms. Using only six ex ante variables, such as issue area, party configuration, circuit of origin, and ideological direction of the lower‑court ruling, they then forecast the outcome and individual Justice votes for every argued case in the 2002 Term before decisions were announced.

To benchmark the model, the project recruits 83 prominent legal experts, including constitutional law scholars, former Supreme Court clerks, and elite appellate advocates. These experts make case‑by‑case predictions, also ex ante and without coordination, and later complete surveys describing the factors they believe matter most for forecasting the Court. This design allows a direct, head‑to‑head comparison between algorithmic inference from historical data and human judgment grounded in doctrinal analysis, professional experience, and perceived familiarity with particular Justices.

The results are striking. On bottom‑line affirm/reverse outcomes for 2002 Term argued cases, the model achieves 75% accuracy, while individual experts average 59.1%, and the majority vote of experts reaches 65.6%. At the individual‑Justice level, overall accuracy is similar for humans and machines, but their strengths diverge: experts are better at predicting the ideologically predictable Justices, while the model performs relatively well on the pivotal centrists, especially Justices O’Connor and Kennedy, whose votes often control close cases. This difference at the ideological center largely explains why the model wins on case outcomes. Performance is also issue‑dependent: doctrinal experts shine in “judicial power” and other technical procedural cases, whereas the model dominates in the diverse but policy‑laden “economic activity” category, revealing stable patterns that lawyers may overlook because they appear doctrinally heterogeneous.

The study further disaggregates the expert pool. Active Supreme Court practitioners do significantly better than academics, suggesting that repeated exposure to real‑world case outcomes and client‑driven risk assessment improves calibration. By contrast, prior Supreme Court clerkship, even with sitting Justices, shows little systematic predictive payoff. The authors argue that these findings challenge the common assumption that high‑level appellate judging is idiosyncratic and resistant to quantification. Instead, they show that the Justices’ voting behavior is structured, partially stable over time, and amenable to relatively simple, interpretable modeling.

At the same time, the authors emphasize important limits. A model that is correct three‑quarters of the time is useful but far from infallible, and it is silent on the content, reasoning, and doctrinal evolution of opinions, matters central to the development of the law and lower‑court behavior. The approach also presupposes a relatively stable Court and a sufficiently large history of prior cases for training. Nonetheless, the project demonstrates that empirical models can provide valuable, decision‑relevant information for litigators and clients: informing certiorari strategies, settlement decisions, and risk assessments, while complementing, rather than replacing, traditional doctrinal expertise and advocacy.

How the Study Advances Empirical Understanding of Legal Outcomes

The study finds that Supreme Court decisions exhibit stable, observable structure rooted in institutional context and general case characteristics, as demonstrated by a statistical model that consistently matched or exceeded expert accuracy using only pre-argument variables. The results indicate that aggregate outcome regularities arise from how cases are categorized, appealed, and positioned within the Court’s decision environment, rather than from randomness or purely case-specific doctrinal nuance. By relying on transparent, historically grounded data drawn from a complete set of decided cases, the study’s methodology aligns with Pre Dicta’s emphasis on empirical, case-based analysis as a necessary foundation for understanding legal outcomes and decision contexts at the highest levels of litigation.