How Quantitative Methods Explain Judge Behavior

Key Takeaways

Judicial voting patterns toward specific litigant classes and civil-liberties claims are stable over time and statistically detectable; Simple quantitative tools applied to coded facts can achieve surprisingly high accuracy in predicting case outcomes within narrowly defined issue areas; Justices occupy durable positions on an ideological continuum, which are reflected in consistent voting in civil-liberties and economic-regulation cases; Institutional design choices such as jury size and required majority have large, quantifiable effects on the probability of wrongful conviction or acquittal; Treating courts and juries as patterned decision-makers opens space for empirically informed litigation strategy rather than relying solely on doctrinal reasoning

Dataset Description

The article synthesizes several empirical lines of research rather than building a single unified dataset. It draws primarily on: (1) U.S. Supreme Court decisions from roughly 1921–1962, including focused subsets from the 1955–1960 Terms involving aliens, alleged communists, and Black defendants; (2) issue-specific samples such as 59 Fair Labor Standards Act cases (1941–1959), 60 Federal Employers’ Liability Act cases (1938–1958), and 19 Sherman Act monopoly cases (1949–1959), along with civil liberties dockets across seven Supreme Court Terms (1955–1961); and (3) a small, intensively coded set of 19 federal search-and-seizure cases (circa 1942–1962) involving warrantless searches, for which 20 factual variables were coded and a subset of 4 was used to construct a discriminant-function example. For the jury-reliability analysis, Ulmer relies on abstract probabilistic models rather than case-level data, using hypothetical parameters for juror accuracy, jury size, and voting rules to derive error probabilities.

Key Findings

Ulmer demonstrates that judicial decision-making, particularly on the U.S. Supreme Court, exhibits stable, measurable patterns rather than random or purely doctrinal variation. Across multiple decades and issue domains, certain classes of litigants (such as the federal government and labor) and certain types of claims (especially civil liberties claims) systematically fare better or worse before particular Justices. Voting tables for cases involving aliens, alleged communists, and Black defendants show that identifiable “liberal” Justices regularly support such claimants at high rates, while more conservative Justices consistently oppose them, and these differences are statistically significant under simple binomial tests. By arraying Justices along a civil-liberties continuum across seven Terms, Ulmer finds a remarkably stable rank ordering of ideological positions, reinforcing the idea that “liberal” and “conservative” tendencies are structured and persistent. Moving from attitudes to prediction, he shows that even small, carefully defined issue areas (such as federal warrantless search-and-seizure cases) can be modeled using a limited set of coded facts: a discriminant-function analysis using four factual variables correctly classifies 18 of 19 decisions, illustrating that combinations of recurring fact patterns are strongly associated with outcomes. Finally, using classical probabilistic models of jury decision-making, he shows that the reliability of collegial bodies depends systematically on jury size, required majority, and underlying guilt rates among defendants tried. Larger juries combined with unanimity rules sharply reduce the theoretical probability of erroneous decisions compared with small juries or bare-majority rules, and shifts in the underlying base rate of guilt change the relative incidence of wrongful convictions versus wrongful acquittals. Collectively, these results suggest that judicial behavior, case outcomes, and institutional error rates can be meaningfully quantified and partially predicted.

Summary

Ulmer’s article is a landmark in the early quantitative study of courts, arguing that judicial and jury decisions can be systematically modeled and, to a meaningful extent, predicted. He reframes appellate opinions and votes as data points, showing that when decisions are aggregated across time and coded in simple ways, regular patterns emerge in how judges treat classes of litigants, issues, and fact configurations. This challenges the view of judging as opaque, idiosyncratic, or driven solely by formal doctrine.

In the first part of the paper, Ulmer compiles empirical findings on the U.S. Supreme Court across multiple decades and terms. He shows that particular litigant types, such as the federal government, labor, and certain criminal defendants, have systematically different success rates depending on which Justice and panel they face. By examining cases involving aliens, alleged communists, and Black defendants from the 1955–1960 Terms, he documents that nominally “liberal” Justices reliably support these claimants at high levels, while more conservative Justices are markedly less supportive. These patterns are not only visible but also statistically significant under binomial testing, and when civil-liberties decisions across seven Terms are scaled, Justices fall into a stable rank ordering from most to least pro–civil-liberties, underscoring that ideology is a persistent structural feature of judicial behavior.

Ulmer then moves from broad attitude patterns to concrete outcome prediction. Drawing on prior work in quantitative content analysis and adding his own example, he argues that carefully chosen factual variables can be used to construct classification rules for specific issue niches. In his illustration, 19 federal cases involving warrantless searches are coded on 20 potential facts; a discriminant function based on only four of these variables, such as whether the search involved living quarters, whether an arrest warrant was present, whether state officers participated, and whether forcible entry occurred, correctly predicts the Court’s decision in all but one case. The example is deliberately modest, but it shows that recurring fact patterns cluster with particular outcomes in ways that can be captured by simple statistical tools, providing a template for more systematic predictive modeling of judicial decisions.

The second part of the article turns from individual and panel voting patterns to the reliability of collective decision structures, especially juries. Using probabilistic models rooted in Condorcet’s jury theorem and related work, Ulmer treats each juror as having a fixed probability of reaching a correct judgment, then derives the group’s error rates under different jury sizes and voting rules. He shows that, holding individual accuracy constant, larger juries with unanimity requirements yield much lower probabilities of wrongful conviction or acquittal than small juries that require only a bare majority. He also highlights that the underlying proportion of truly guilty defendants among those tried materially affects the trade-off between the two types of error, providing a quantitative framework for policy debates over six-versus-twelve-person juries, supermajority verdicts, and the design of appellate panels.

Across these strands, Ulmer’s central message is that courts can be studied as empirical systems. Judges have stable preferences, litigant status and identity matter in patterned ways, factual constellations map onto outcomes with exploitable regularity, and institutional rules systematically shape error probabilities. The article thus anticipates later work in judicial behavior and legal analytics by insisting that predictive and probabilistic thinking is not merely academic: it is essential to understanding, evaluating, and strategically engaging with the judicial process.

How the Study Advances Empirical Understanding of Legal Outcomes

The study finds that judicial and jury decisions exhibit durable, statistically observable regularities across time, issue areas, and institutional settings, demonstrating that legal outcomes are shaped by structured decision environments rather than random variation. By systematically analyzing voting patterns, fact configurations, and probabilistic models of collegial decision making, the analysis shows how institutional design and recurring factual and procedural contexts generate consistent outcome distributions that can be examined analytically. This empirical, case-based methodology aligns with Pre/Dicta’s emphasis on grounding high-level litigation analysis in observable decision structures and rigorously measured outcome regularities, rather than intuition or purely doctrinal abstraction.