What Predicts Law Student Success? A Longitudinal Study Correlating Law Student Applicant Data and Law School Outcomes

University of Colorado Law School University of Colorado Law School

Colorado Law Scholarly Commons Colorado Law Scholarly Commons

Publications Colorado Law Faculty Scholarship

2016

What Predicts Law Student Success? A Longitudinal Study What Predicts Law Student Success? A Longitudinal Study

Correlating Law Student Applicant Data and Law School Correlating Law Student Applicant Data and Law School

Outcomes Outcomes

Alexia Brunet Marks

University of Colorado at Boulder

Scott A. Moss

University of Colorado Law School

Follow this and additional works at: https://scholar.law.colorado.edu/faculty-articles

Part of the Legal Education Commons

Citation Information Citation Information

Alexia Brunet Marks & Scott A. Moss,

What Predicts Law Student Success? A Longitudinal Study

Correlating Law Student Applicant Data and Law School Outcomes

, 13 J. Empirical Legal Stud. 205

(2016), available at http://onlinelibrary.wiley.com/doi/10.1111/jels.12114/pdf.

Educational Use clauses of the U.S. Copyright Law may violate federal law. Permission to publish or reproduce is

required.

This Article is brought to you for free and open access by the Colorado Law Faculty Scholarship at Colorado Law

Scholarly Commons. It has been accepted for inclusion in Publications by an authorized administrator of Colorado

Law Scholarly Commons. For more information, please contact [email protected].

What Makes a Law Student Succeed or Fail? A Longitudinal Study

Correlating Law Student Applicant Data and Law School Outcomes

Alexia Brunet Marks and Scott A. Moss

Despite the rise of "big data" empiricism, law school admission remains heavily

impressionistic; admission decisions based on anecdotes about recent students,

idiosyncratic preferences for certain majors or jobs, or mainly the Law School Admission

Test (LSAT). Yet no predictors are well-validated; studies of the LSAT or other factors

fail to control for college quality, major, work experience, etc. The lack of evidence of

what actually predicts law school success is especially surprising after the 2010s downturn

left schools competing for fewer applicants and left potential students less sure of law

school as a path to future success. We aim to fill this gap with a two-school, 1400-student,

2005-2012 longitudinal study. After coding non-digitized applicant data, we used

multivariate regression analysis to predict law school grades ("LGPA") from many

variables: LSAT; college grades ("UGPA"), quality, and major; UGPA trajectory;

employment duration and type (legal, scientific, military, teaching, etc.); college

leadership; prior graduate degree; criminal or discipline record; and variable interactions

(e.g., high-LSAT/low-UGPA or vice-versa).

Our results include not only new findings about how to balance LSAT and UGPA, but

the first findings that college quality, major, work experience, and other traits are

significant predictors: (1) controlling for other variables, LSAT predicts more weakly, and

UGPA more powerfully, than commonly assumed – and a high-LSAT/low-UGPA profile

may predict worse than the opposite; (2) a STEM (science, technology, engineering, math)

or EAF (economics, accounting, finance) major is a significant plus, akin to 3½-4 extra

LSAT points; (3) several years' work experience is a significant plus, with teaching

especially positive and military the weakest; (4) a criminal or disciplinary record is a

significant minus, akin to 7½ fewer LSAT points; and (5) long-noted gender disparities

seem to have abated, but racial disparities persist. Some predictors were interestingly

nonlinear: college quality has decreasing returns; UGPA has increasing returns; a rising

UGPA is a plus only for law students right out of college; and 4-9 years of work is a

"sweet spot," with neither 1-3 or 10+ years’ work experience significant. Some, such as

those with military or science work, have high LGPA variance, indicating a mix of high

and low performers requiring close scrutiny. Many traditionally valued traits had no

predictive value: typical pre-law majors (political science, history, etc.); legal or public

sector work; or college leadership.

These findings can help identify who can outperform overvalued predictors like the

LSAT. A key caveat is that statistical models cannot capture certain difficult-to-code key

traits: some who project to have weak grades retain appealing lawyering or leadership

Alexia Brunet Marks, J.D., Ph.D. ([email protected]), is an Associate Professor, and

Scott A. Moss, J.D., M.A. ([email protected]), a Professor, at the University of

Colorado Law School. For valuable feedback on drafts and presentations, we thank Professors

Jeff Stake, Royce de rohan Barondes, Daniel Ho, James Lindgren, Harry Surden, and Susan

Yeh. For comments on presentations of this research, we thank those at the Midwestern Law

and Economics Association (MLEA) and Colloquium on Scholarship in Employment and

Labor Law (COSELL) annual meetings, and workshops at the law schools of Marquette

University, St. Louis University, Ohio Northern University, Case Western Reserve University,

and the University of Colorado. We especially thank the Deans, Registrars, and Admissions

Officers at Case Western and the University of Colorado for letting us access confidential data

and helping us sift through dusty old files in various closets and file cabinets in their buildings.

The authors express their gratitude to this journal’s anonymous reviewers for very helpful

comments on a previous draft.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 2

potential; and many will over- or under-perform any projection. Thus, admissions will

always be both art and science – but perhaps with a bit more science.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 3

TABLE OF CONTENTS

I. INTRODUCTION: THE NEED FOR BETTER LAW SCHOOL DECISION-MAKING ........... 4

II. BACKGROUND: PRIOR STUDY OF DESIRABLE STUDENT TRAITS AND SUCCESS

PREDICTORS ............................................................................................................. 8

A. The Value of Academic and Numerical Qualities: LSAT, UGPA, and Factors

Moderating UGPA ............................................................................................ 8

B. Learning Strategies, from Reading Styles to Professional Orientation .......... 13

C. Emotional Intelligence .................................................................................... 15

III. METHODOLOGY ...................................................................................................... 16

A. The Data Set .................................................................................................... 16

B. Regression Analysis of Admissions Criteria on Law School Grades .............. 18

1. Hypotheses ............................................................................................... 18

2. Models ...................................................................................................... 19

a. The Primary Regressions: Models 1 (LGPA) and 2 (1L GPA) .... 19

b. The Quarter Regressions: Model 3 and Model 4 .......................... 22

c. The Splitters Regression: Model 5 ............................................... 23

d. The Variance Analysis .................................................................. 23

IV. KEY RESULTS AND INTERPRETATIONS ................................................................... 23

A. Caveats: Limitations on Modeling Law Student Performance ........................ 23

B. The Primary Regressions: Predicting Cumulative LGPA (Model 1) and 1L

GPA (Model 2) ................................................................................................ 26

1. LSAT: 1 LSAT Point ≈ 0.016 LGPA ....................................................... 30

2. UGPA: Increasing Returns; 0.03-0.06 UGPA ≈ 1.0 LSAT Point ............ 32

3. LCM: Modest, Decreasing Returns; 1 LCM ≈ 0.2 LSAT Pt., But with

LCM<152 Amounting to an Extra -1 LSAT Point ................................... 33

4. College Majors: STEM/EAF ≈ 3.5-4 LSAT Pts.; No Negative Majors ... 35

5. Work Duration: 4-9 Years ≈ 6.5 LSAT Points ......................................... 38

6. Work Type: Teaching ≈ 5 LSAT Pts.; Military ≈ -7

; Sci/Tech ≈ -3..... 39

7. Negative Criminal/Disciplinary Record ≈ -7

LSAT Points .................. 40

8. Rising UGPA (If in Law School Right after College) ≈ 2 LSAT Points.. 41

9. Demographics: Person of Color Self-ID, -9 to -9½ LSAT Pts. ................ 42

C. The Quarter Regressions (Models 3 and 4): What Predicts Especially Strong

or Weak Law School Performance? ................................................................ 44

D. The "Splitters" Regression (Model 5): Which Is Better, High-UGPA/Low-

LSAT or the Reverse? ...................................................................................... 46

E. The Variance Analysis: Examining LGPA Variance Based on Membership in

Various Groups ............................................................................................... 48

F. Notable Non-Findings: Variables with Little or No Relationship to LGPA,

Contrary to Our Hypotheses or Common Assumptions .................................. 50

1. Nontraditional Pre-Law Majors: Not a Negative ..................................... 50

2. Traditional Pre-Law and Reading-Heavy Majors: Not a Positive ............ 51

3. Traditional Pre-Law Work (Legal and Public Sector): Not a Positive ..... 51

4. Prior Graduate Degrees: Not a Positive ................................................... 52

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 4

5. Major Leadership Roles in College: Not a Positive ................................. 52

V. PRESCRIPTIONS: BRIEF NOTES ON POSSIBLE REFORMS TO HOW SCHOOLS ADMIT

AND PREPARE STUDENTS ....................................................................................... 52

A. Holistic Review, Given that No One Score, Credential, or Experience Possibly

Can Predict Success or Failure by Itself ......................................................... 53

B. The Heterogeneity of Candidates with Similar Backgrounds: The Need to

Distinguish Apples from Slightly Different Apples.......................................... 53

C. Helping Students Adjust – and Expanding the Talent Base by Doing So........ 55

VI. CONCLUSION .......................................................................................................... 56

Appendix ...................................................................................................................... 58

I. INTRODUCTION: THE NEED FOR BETTER LAW SCHOOL DECISION-

MAKING

The modern legal education crisis – years of rising tuition and legal sector

retrenchment

yielding declining law school applications

– put a premium on a

question that always should have mattered to law schools and their students: What

qualities predict law student success? This concern has grown as the downturn has

left schools competing for far fewer applicants: applications are at a 30-year low,

down 38% over two years alone,

forcing schools to shrink, decrease selectivity, or

both.

Part of the decline may be cyclical, but there also are core long-term,

structural causes: the obsolescence of the large-firm model, especially as clients

National Association for Law Placement (hereinafter "NALP") statistics show that only 86%

of 2011 graduates obtained paying jobs, with less than 66% of those requiring a law license. Joe

Palazzolo & Chelsea Phipps, With Profession Under Stress, Law Schools Cut Admissions,

WALL ST. J., June 11, 2012, http://online.wsj.com/article/SB10001424052702303444204577

458411514818378.html. Many of the latter job category, moreover, were mere contract work,

which is by definition non-permanent and only pays around $25/hour. Jordan Weissmann, Law

School Applications Are Collapsing (as They Should Be), THE ATLANTIC (Jan. 2013),

http://www.theatlantic.com/business/archive/2013/01/law-school-applications-are-collapsing-

as-they-should-be/272729/.

See Richard Susskind, Tomorrow’s Lawyers, Vol. 39 No. 4. A.B.A. L. PRAC. MAG., July/Aug.

2013, available at http://www.americanbar.org/publications/law_practice_magazine/2013/july-

august/tomorrows-lawyers.html (noting that law schools are “under fire” for admitting more

students than the likely number of law jobs); BRIAN Z. TAMANAHA, FAILING LAW SCHOOLS

(2012) (arguing that modern law schools lack sustainable business models due to increased

tuition and decreased employment rates); STEPHEN HARPER, THE LAWYER BUBBLE: A

PROFESSION IN CRISIS 124 (2013) (detailing layoffs and closures at previously large, successful

law firms).

Ethan Bronner, Law Schools’ Applications Fall as Costs Rise and Jobs Are Cut, N.Y. TIMES

(Jan. 30, 2013), http://www.nytimes.com/2013/01/31/education/law-schools-applications-fall-

as-costs-rise-and-jobs-are-cut.html.

Paul Lippe, D-Day for Law School Deans, A.B.A. J. (May 1, 2013),

http://www.abajournal.com/legalrebels/article/d-day_for_law_school_deans (noting clients' new

unwillingness to subsidize associate training by paying hourly rates for inexperienced lawyers).

Palazzolo & Phipps, supra note 1.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 5

began demanding experienced lawyers, not higher-profit-margin junior lawyers;

the rise of a legal process outsourcing industry as digitization allows offsite work;

and cheaper competition, as technology streamlines high-markup labor-intensive

tasks, from simple software for creating simple documents

to replacing multi-

lawyer document review with "predictive coding" in which "machine algorithms

partially replac[e] humans altogether in the search for relevant information."

With schools seeing fewer applicants, all schools have been forced to admit

students with lower numerical predictors. Especially in a diminished pool,

discerning who likely can outperform their numbers is an imperative. Elite schools

want to keep admitting those who pass bar exams at high rates and display the

talent to land elite jobs; non-elite schools want those who, despite low grades or

LSAT scores, still can perform competent legal work and pass a bar exam. The

interests are similar from applicants' perspective. Those with strong LSAT/grade

profiles do not always win admission to top schools, and ideally those who are

truly stronger should win those coveted seats; those with weak LSAT/grade

profiles may not win admission to a reputable (or any) school, yet it is a loss for

society and the profession if the stronger low-numbers candidates lack good (or

any) admission offers. More broadly, the value of students getting admission offers

they deserve goes beyond this era of fewer in law applications. Even if applications

rise, schools and students still should want to know who projects to succeed or fail

based on factors other than the obvious, such as LSAT, and factors of unclear

import, such as college major. Even if the tide rises or some schools can stand pat,

the innovative gain advantage from better projecting which prospects are more (or

less) promising than they first appear.

Yet law school admission decisions are less data-driven than impressionistic,

often basing on anecdotes (e.g., admitting those resembling recent stars, not those

like recent underachievers), on idiosyncratic preferences (e.g., for certain majors or

jobs), or heavily numerical criteria (e.g., a high LSAT nearly guaranteeing

admission).

The studies on law school success control for few or no other

Marc Galanter & William Henderson, The Elastic Tournament: A Second Transformation of

the Big Law Firm, 60 STAN. L. REV. 1867 (detailing evolution of associate-heavy large firms

from a classic (inverted funnel) pyramid with a standard tournament model to “core and mantle”

pyramids with “elastic” tournaments); Lippe, supra note 4.

See SUSSKIND, THE END OF LAWYERS? 27-57; Law Firms Are Losing Work to LPO Providers,

MANAGING PARTNER (Sept. 3, 2012), http://www.managingpartner.com/news/business-

strategy/law-firms-are-losing-work-lpo-providers [hereinafter Law Firms Losing Work] (noting

overseas LPO alone now exceeds $1 billion).

Deborah L. Jacobs, The Case Against Law School, FORBES (Jan. 31, 2013),

http://www.forbes.com/sites/deborahljacobs/2011/10/11/the-case-against-law-school

(recounting how a group of venture capitalists, including Google, invested $18.5 million in

Rocket Lawyer, while LegalZoom raised $66 million in venture capital the month before).

William D. Henderson, A Blueprint for Change, 40 PEPP. L. REV. 461, 487 (2013).

The authors have served for years as Chair (Moss) and Vice-Chair (Marks) of the University

of Colorado Law School Faculty Admissions Committee, casting votes on thousands of

applicants. So their critique of law school admissions is not a criticism of others; it is an effort

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 6

variables in finding that LSAT correlates with first-year law grades, or that a

certain interpersonal quality is a plus. Studies with one or only a few variables

leave unclear whether a seemingly significant variable is a true predictor, or is

simply correlated with another predictor, or is a weaker predictor when other

variables are evaluated simultaneously. For example, do high-LSAT students

really do better, or does a high LSAT just correlate with other predictors, such as

attending a strong college? Do any majors, like traditional pre-law majors such as

political science or history, predict success or failure, or is there no difference

among majors? And what of key interactive mixes of variables – for example,

which kind of "splitter" does better, the high-LSAT/low-UGPA college student or

the reverse? No prior study has examined who succeeds with a broad range of

actual data allowing testing of the individual impact of as many measurable

metrics as possible – a gap this Article aims to fill.

This Article details the methodology and findings of a longitudinal study based

on data spanning 2005 to 2012, from over 1400 students, at two law schools, Case

Western Reserve University and the University of Colorado Law Schools. The

study examines how data in the students' 2005-2008 law school applications

correlate with their 2006-2011 grades - an effort requiring a substantial

undertaking to code data from paper files and to merge separate admissions and

registrar databases. The study attempts to predict law school grade-point average

("LGPA") as a function of numerous independent variables: LSAT score;

undergraduate grade-point average ("UGPA"); college quality, as measured by a

metric available for virtually all colleges, the mean LSAT of students at the college

("LCM"); college major; years, and type, of full-time work; significant

extracurricular leadership; having another graduate degree; having a substantially

rising UGPA; negative criminal or academic misconduct records; and various

interactions of these variables (e.g., having a high LSAT but low UGPA, or vice-

versa; or only those who just graduated college having a rising UGPA, on the

theory that UGPA trajectory matters more for those right out of college). Most of

this data did not exist in digital form and therefore had to be manually entered; for

example, college majors are listed on transcripts, years and type of work

experience are listed on applicants' résumés, and criminal/disciplinary records are

submitted with law applications. Other data were digitized but required manual

review to enter the relevant variables; for example, UGPAs are digitized, but not

whether UGPAs rose during college, requiring review of year-to-year grades.

Our results include not only new findings about how to balance LSAT and

UGPA, but also the first statistical findings that college quality, major, work

experience, and other variables are significant predictors: (1) controlling for other

variables, LSAT predicts more weakly, and UGPA more powerfully, than

commonly assumed – and a high-LSAT/low-UGPA profile predicts worse than a

high-UGPA/low-LSAT profile; (2) a STEM (science, technology, engineering,

to improve their own and others' admissions work alike.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 7

math) or EAF (economics, accounting, finance) major is a significant plus, akin to

having 3½-4 extra LSAT points; (3) several years' work experience is a significant

plus, with teaching especially positive, and military the weakest; (4) a criminal or

disciplinary record is a significant minus, akin to 7½ fewer LSAT points; and (5)

long-noted gender disparities appear to have abated, but racial disparities persist.

Some predictors were interestingly nonlinear: college quality has decreasing

returns; UGPA has increasing returns; a rising UGPA is a plus for only those right

out of college; and 4-9 years of work is the "sweet spot," with 1-3 and over 10 not

significant. Some students display high LGPA variance, indicating a mix of high

and low performers requiring close scrutiny – e.g., those with military or science

work. Finally, many traits traditionally seen as plusses had no predictive value:

common pre-law majors like political science or history; legal or public sector

work; and college leadership. Most findings proved robust across various

specifications.

These findings have key caveats. First, law grades are incomplete predictors of

contribution to society, career fulfillment, or even long-term job prospects, given

that law grades predict lawyers' earnings for only their first several years;

many

applicants predicted to have middling grades are appealing for reasons, such as

leadership, diversity, and intangible qualities. Second, no statistical model captures

all human qualities, and many traits are not readily reducible to data; many will

over- or under-perform even the best predictions, so talent assessment is more art

than science. Third, negative predictors are not consistent across individuals: some

groups that project poorly are a heterogeneous mix that individualized scrutiny can

distinguish; and certain predictors are not consistent over time, such as predictors

that are negative just because some people need more time to adjust to law study.

Given the above three caveats, we in no way suggest that simply including

enough variables makes admissions reducible to a formula. Even with these

caveats, law grades are useful as predictors – of the bar passage that is necessary to

most lawyer jobs, of gaining employment in the first several years after law school,

and of at least some aspects of legal acumen. Our findings thus should inform law

schools tasked with difficult decisions: who among numerically similar applicants

is most promising; who can outperform their LSAT and UGPA enough to warrant

admission or scholarship offers; and which traditionally valued or under-valued

qualities truly are, or are not, provable predictors of success. And later work on the

same data set will explore further the extent to which the law school applicant

qualities predict post-law school bar passage and employment, not just law grades.

This Article proceeds as follows. Part II analyzes the literature on what

qualities affect student success and on the limited, mainly univariate, empirical

Jeffrey E. Stake et al., Income and Career Satisfaction in the Legal Profession: Survey Data

from Indiana Law Graduates, 4 J. EMPIRICAL LEG. STUD. 939, 970, 973 (2007) (finding that five

years after law school, "each additional 0.1 on the graduate's [L]GPA yields $3,449 in

additional annual income," but by fifteen years after law school, LGPA has no effect on

income).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 8

analyses of student traits. Part III details our methodology – how and what data

was procured, and our statistical models. Part IV, the core of the Article, details

our findings: which variables proved significant positive or negative LGPA

predictors; the relative magnitudes of the variables' effects, e.g., how much in

UGPA, college quality, or work experience is akin to an extra LSAT point; and our

interpretations of what these findings show about various students' law school

prospects. Part IV notes that while the vast literature on law school reform is

beyond this Article's scope, our findings do provide new evidence supporting some

reforms and undercutting others. A brief Conclusion previews future work

predicting employment and bar exam outcomes based on this Article's data set, and

other similarly obtainable data, if law schools devote resources to similar analytics

in the future – as we hope they do.

II. BACKGROUND: PRIOR STUDY OF DESIRABLE STUDENT TRAITS AND

SUCCESS PREDICTORS

This Part divides the literature on factors predicting success into three

categories: (A) the impact of academic factors, including LSAT, UGPA, and other

college record information; (B) the impact of varied learning strategies, from

reading styles to professional orientation; and (C) the impact of personal qualities,

such as emotional intelligence, resilience, and maturity. We discuss three ways this

Article aims to fill gaps in that literature. First, various factors that may predict

success or failure have drawn little or no prior analysis because they are not coded

in statistics-friendly digital form – such as college major, duration and kind of

work experience, and criminal record. Second, where no clear data exist on a

potentially important quality, such as interpersonal skills, resilience, or maturity,

we propose certain variables as proxies – for example, leadership role as a proxy

for interpersonal skill, rising UGPA after a weak college start as a proxy for

resilience, or disciplinary or criminal record as a proxy for lack of maturity. Third,

most studies are univariate, simply finding correlations between success and one

factor without controlling for, or examining interactions with, other factors.

A. The Value of Academic and Numerical Qualities: LSAT, UGPA, and Factors

Moderating UGPA

Law schools strongly eye a few numerical indicators. In particular, median

LSAT is a top driver of a school's reputation: among innumerable qualities

students possess, LSAT alone is worth 12.5% of the U.S. News & World Report

law school rankings.

But William Henderson found that this linear weight

understates the impact of LSAT on school rank, in a study aiming to “identify the

relative winners and losers over time in the competition for the finite number of

high-LSAT students, and examine … factors that can explain the underlying

pattern in the movements of LSAT scores at law schools.”

Henderson found that

William D. Henderson & Andrew P. Morriss, Student Quality as Measured by LSAT Scores:

Migration Patterns in the U.S. News Rankings Era, 81 INDIANA L. J. 163 (2006).

Id. at 169.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 9

90% of differences in schools' ranks can be explained solely by median LSAT,

which both varies greatly among schools and is more readily "gamed" by schools,

at all rankings levels,

than other major rank components, such as school

reputation.

Partly because it drives school rank, LSAT is by far the dominant admissions

factor, even compared to UGPA, the main other numerical predictor. The "Law

School Probability Calculator," which estimates admission odds by LSAT and

UGPA from thousands of data points,

shows a vast gap between the fates of the

two "splitter" applicant types: high LSAT with a low UGPA; and low LSAT with a

high UGPA. Illustrating schools' preference for high-LSAT over high-UGPA

splitters is anecdotal evidence from two examples of mid-tier schools, Santa Clara

University and St. John’s University (which have very similar LSAT and UGPA

medians)

and two highly-ranked schools, Georgetown University Law Center

and University of Michigan Law School (also with very similar LSAT and UGPA

medians).

Id. at 191 (noting that statistics for transfer students and, until recently, entering part-time

students were not included in rankings, so a school could raise median LSAT by shrinking the

full-time program and expanding transfer and part-time admissions, and top-tier schools are

better-positioned to stay selective and admit transfer students to make up for revenue losses).

Id at 165. Henderson and Morriss specifically found as follows: (1) the legal education market

is segmented into a national market, roughly the current top quarter (“Tier 1”) of law schools,

and a regional market encompassing the rest of the law school hierarchy; (2) within each

segment, a higher initial starting position was associated with increases in median LSAT; (3) in

quarter 2-4, lower-cost schools have a better yield of high-LSAT students, but in quarter 1,

prestige is more important than price; (4) in quarters 2-4, law schools in major Am Law 200

markets have a significant advantage in attracting high-LSAT students; and (5) in quarters 2-4,

changes in lawyer/judge and academic reputations are unrelated to changes in median LSAT

whereas in quarter 1, an increase in academic reputation is associated with higher LSAT. Id. at

182-88 (further noting that the median LSAT of top-16 schools has increased an average of 1.69

points, while schools that began in quarter 2 had a 0.45 increase in their median LSAT scores,

and schools in quartiers 3 and 4 experienced declines of -1.56 and -1.34, id. at 186).

Law School Probability Calculator, http://www.hourumd.com (last visited Feb. 26, 2015)

(explaining that it "uses data gathered from Law School Numbers to calculate probability of

admission at various law schools. All data is self-reported, but with over 143,000 data points, it

should be somewhat accurate").

These schools were chosen simply because they are law schools on opposite coasts but close

to the middle of the rankings, with similar median LSAT and UGPA statistics: 3.21/157 for

Santa Clara, 2013 Class Profile, SANTA CLARA L., http://law.scu.edu/admissions/2013-class-

profile (last visited Feb. 26, 2015); 3.39/156 for St. John's, FAQs, ST. JOHN’S UNIV. SCH. OF L.

http://www.stjohns.edu/law/admissions/faqs (last visited Feb. 26, 2015).

Georgetown's medians are a 3.75 GPA and 168 LSAT. Stats, Facts & More, GEO. UNIV. L.

CTR., http://www.law.georgetown.edu/admissions-financial-aid/jd-admissions/full-time-part-

time-program/faqs/General.cfm (last visited Feb. 26, 2015). Michigan's are a 3.71 GPA and 168

LSAT. Class Statistics, UNIV. MICH. L. SCH.,

http://www.law.umich.edu/prospectivestudents/pages/classstatistics.aspx (last visited Feb. 26,

2015).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 10

• Santa Clara University Law School admitted 94% of those with an

above-median 158-160 LSAT and a below-median 3.0-3.2 UGPA,

but only 40% of those with a reverse LSAT/UGPA profile that is

roughly equivalent in distance from the school's medians

– an above-

median 3.7-3.9 UGPA, and a below-median 151-153 LSAT.

• St. John’s University Law School was almost exactly the same as

Santa Clara, admitting 100% with the same above-median LSAT and

below-median UGPA (158-160/3.0-3.2),

but 37.5% with the same

above-median UGPA and below-median LSAT (3.7-3.9/151-153).

• Georgetown University Law Center admitted 83.02% with an above-

median LSAT and below-median UGPA (170-172/3.2-3.4),

but

38.3% of those with a reverse LSAT/UGPA profile that is roughly

equivalent in distance from the school's medians – an above-median

3.8-4.0 UGPA, and a below-median 164-166 LSAT.

• University of Michigan Law School was almost exactly the same as

Georgetown, admitting 75.51% with the same above-median LSAT

and below-median UGPA (170-172/3.2-3.4),

but 34.45% with the

same above-median UGPA and below-median LSAT (3.8-4.0/164-

166).

Law School Probability Calculator, query: http://www.hourumd.com/?lsat=160-

162&gpa=3.0-3.2&money=no&urm=yes&waitlist=yes&range=no (last visited Feb. 26, 2015).

As detailed below, one LSAT point is, roughly, equivalent to 0.03-0.06 in UGPA, a shorthand

useful for comparing high-UGPA/low-LSAT "splitters" to the reverse splitter type. We cannot

know whether each of these four schools (St. John's, Santa Clara, Georgetown, and Michigan)

would agree that these opposite profiles are equivalent in distance from their medians, so

possibly they believed the low-UGPA/high-LSAT group to be weaker than the opposite high-

UGPA/low-LSAT group. Still, our findings indicate that these opposite-profile groups are

roughly in par with each other, so the difference is striking, and strikingly consistent, between

the fate of the high-UGPA/low-LSAT splitters (34-40% admitted at each of the four schools)

and the high-LSAT/low-UGPA splitters (75-100% admitted at each of the four schools).

Law School Probability Calculator, query: http://www.hourumd.com/?lsat=151-

153&gpa=3.7-3.9&money=no&urm=yes&waitlist=yes&range=no (last visited Feb. 26, 2015).

Law School Probability Calculator, query: http://www.hourumd.com/?lsat=160-

162&gpa=3.0-3.2&money=no&urm=yes&waitlist=yes&range=no (last visited Feb. 26, 2015).

Law School Probability Calculator, query: http://www.hourumd.com/?lsat=151-

153&gpa=3.7-3.9&money=no&urm=yes&waitlist=yes&range=no (last visited Feb. 26, 2015).

Law School Probability Calculator, query: http://www.hourumd.com/?lsat=170-

172&gpa=3.2-3.4&money=no&urm=yes&waitlist=yes&range=no (last visited Feb. 26, 2015).

Law School Probability Calculator, query: http://www.hourumd.com/?lsat=164-

166&gpa=3.8-4.0&money=no&urm=yes&waitlist=yes&range=no (last visited Feb. 26, 2015).

Law School Probability Calculator, query: http://www.hourumd.com/?lsat=170-

172&gpa=3.2-3.4&money=no&urm=yes&waitlist=yes&range=no (last visited Feb. 26, 2015).

Law School Probability Calculator, query: http://www.hourumd.com/?lsat=164-

166&gpa=3.8-4.0&money=no&urm=yes&waitlist=yes&range=no (last visited Feb. 26, 2015).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 11

Though schools clearly weight LSAT over UGPA, evidence the LSAT truly

predicts law grades is underwhelming. The few findings on LSAT predictive

power are mixed and fail to control for other key variables. The most prominent

studies are by LSAC, the Law School Admission Council – a hardly unbiased

source, because it is the entity that is "best known for administering the …

LSAT[], with about 100,000 tests administered annually," and that "publishes

LSAT preparation books and law school guides, among many other services" it

sells.

LSAC reports that "LSAT scores help to predict which students will do

well in law school."

But it also admits that its studies show only that LSAT

correlates with first-year grades:

[M]ost law schools have participated in studies that have compared

students’ LSAT scores with their first-year grades. … [T]hese studies

show that LSAT scores help to predict which students will do well in law

school. … [T]he combination of … LSAT score and undergraduate grade-

point average yields a better prediction … than either measure used alone.

… [C]orrelations between average LSAT score and first-year law school

grades ranged [among schools] from .16 to .54, with a median … of .36.

… [C]orrelations between UGPA and first-year law school grades ranged

from .09 to .45, with a median … of .28. … [C]orrelations between the

combination of average LSAT score and undergraduate grades with first-

year … grades ranged from .27 to .63, with a median … of .46.

Similar studies found that LSAT better predicted first-year law grades

, while

UGPA predicted overall grades

, and a combined LSAT/UGPA index was better

LSAC describes itself as follows:

LSAC[] is a nonprofit corporation … best known for … [the] LSAT …. LSAC also

processes academic credentials for an average of 60,000 law school applicants

annually, provides essential software and information for admission offices and

applicants, conducts educational conferences … , sponsors and publishes research,

funds diversity and other outreach … , and publishes LSAT preparation books and law

school guides.

About LSAC, LAW SCHOOL ADMISSIONS COUNCIL, http://www.lsac.org/aboutlsac/about-lsac

(last visited July 28, 2014).

LAW SCHOOL ADMISSION COUNCIL, 2012–2013 LAW SCHOOL ADMISSION REFERENCE

MANUAL 11 (2012).

Id. (emphases added).

Marjorie M. Shultz & Sheldon Zedeck, Predicting Law Effectiveness: Broadening the Basis

for Law School Admission Decisions, 36 LAW & SOC. INQUIRY 620, 622 (2011).

David A. Thomas, Predicting Law School Academic Performance From LSAT Scores and

Undergraduate Grade Point Averages: A Comprehensive Study, 35 ARIZ. ST. L.J. 1007, 1021

(2003). See also Neal Schmitt, Jessica Keeney and Fredrick L Oswald, (2009), Prediction of 4-

year College Student Performance Using Cognitive and Noncognitive Predictors and the

Impact on Demographic Status of Admitted Students, Journal of Applied Psychology, vol. 94,

no. 6, 1479-1497 (this study also uses graduation as a measure of success and shows that the

most important predictor of college graduation status was high school grades).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 12

than either alone at predicting both first-year and overall law school grades.

These studies indicate that while both LSAT and UGPA have predictive power, the

LSAT perhaps should not be given disproportionate weight. These studies also

raise further questions about how predictive each of LSAT and UGPA would be in

a study that controls for other variables about students' personal and college

backgrounds.

A study of the similar Master's in Business Administration ("MBA")

admissions process, which typically bases heavily on UGPA and the LSAT-like

Graduate Management Admission Test (“GMAT”), similarly found UGPA more

important than the standardized test: GMAT did predict MBA grades, but to a

limited degree;

UGPA predicted grades better than GMAT verbal and

quantitative scores;

and a combination of all predictors (UGPA and GMAT

verbal and quantitative scores) predicted better than any factor alone.

The study

noted that schools should not rely on GMAT and UGPA to the exclusion of other

factors, such as motivation and work experience, yet did not control for such

difficult-to-quantify factors.

Even if the LSAT helps predict LGPA, it may do so for a less substantive

reason: test-taking speed helps determine performance on the LSAT and traditional

in-class law exams that produce most law grades.

William Henderson notes that

the LSAT is a stronger predictor of timed, in-class exam grades than of take-home

exam or research paper grades:

"on take-home exams and papers, … it appears

that the LSAT is actually a weaker predictor of law school performance than

UGPA," which measures a composite of reasoning, writing, motivation, and

persistence.

Thus, a school's emphasis on timed in-class exams increases the

predictive power of a timed in-room exam like the LSAT. Yet test-taking speed is

not a meaningful intelligence measure, Henderson notes: "[w]ithin the field of

psychometrics, test-taking speed and reasoning ability are viewed as distinct,

separate abilities with little or no correlation."

And while the old model of legal

education consisted mainly of timed, in-class tests, schools have shifted to a

Id. at 1011 (summarizing aggregate correlation scores for students in all twenty-seven classes:

LSAT and 1L rank, 0.744; UGPA and 1L rank, 0.740; index and 1L rank, 0.759; LSAT and 3L

rank, 0.730; UGPA and 3L rank, 0.733; index and 3L rank, 0.744).

Baiyin Yang & Diaopin Rosa Lu, Predicting Academic Performance in Management

Education: An Empirical Investigation of MBA Success, 77 J. EDUC. FOR BUS. 15, 16 (2001).

Id. at 18.

Id. at 19.

Id.

William D. Henderson, The LSAT, Law School Exams and Meritocracy: The Surprising and

Undertheorized Role of Test-Taking Speed, 82 TEX. L. REV. 975 (2004).

Id. at 1030 ("[Law school] reliance on time-pressured exams exerts a significant … effect on

the relative importance of the LSAT [over UGPA] …. [D]ifferences in test-taking speed rather

than reasoning ability may account for why the LSAT … emerges as a stronger predictor.").

Id. at 1044.

Id. at 979 (surveying literature and collecting citations).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 13

broader mix of take-home exams, papers, and clinical-and-simulation

performances as "arguably more reflective of the systemic time pressure found in

the actual practice of law" than traditional in-class tests.

Most critically, no studies control for data on many other important traits, such

as college quality or major, work experience type or duration, or criminal or

disciplinary records. A more rigorous major or college might predict law school

success, whether because grades in a more rigorous curriculum are more reliable

predictors, because the same 3.3 UGPA (for example) is a more impressive

accomplishment in a more rigorous curriculum, or both. One study a legal writing

professor conducted, of her 538 students over 16 years, found that students' majors

do make a difference: economics majors earned the best legal writing grades, with

double-majors and those with M.B.A.s also performing above-average.

However,

that study was unpublished, did not did not control for other factors, and featured

modest subgroup sizes (e.g., 16 economics majors);

thus, possibly the higher-

performing economics majors just had higher LSATs, UGPAs, or college quality.

In sum, by not controlling for other predictors, LSAC's and other studies leave

unknown the predictive validity of their findings on LSAT and UGPA. To be sure,

no study can control for all influences on LGPA: some data are unavailable; other

factors (e.g., motivation) are not reducible to the sort of binary or continuous

variables susceptible to regression analysis; still other factors that affect law

student performance, such as major events in the life of a student, are too

individualized to be a part of any statistical model. Thus, no regression can control

for all factors that predict LGPA; the best any study can do is to include reasonably

available data that measures, or serves as a proxy for, as many of the truly critical

student qualities as possible – an effort detailed, as to this study, in the

methodology section below.

B. Learning Strategies, from Reading Styles to Professional Orientation

Law schools frequently do assess students' personal and professional qualities,

not just their numbers – yet almost no studies examine how personal or

professional qualities actually predict law school success. Two helpful studies by

Leah Christensen document the importance of a few key factors and argue more

broadly to take personal and professional qualities seriously in assessing student

potential.

In arguing for the importance of legal skills training, Christensen found that

law school class rank was statistically significantly correlated with not only high

lawyering skills class grades, but with being a "mastery-oriented" learner focused

Id. at 1044.

Karin Mika, Do Undergraduate Majors Correlate Highly with Success in Legal Writing

Classes?, at 27-28, 35 (2010) (unpublished study) (on file with authors) (summarizing that the

sole categories in which students had above-average grades were "those with economics majors,

those with double majors, and those with advanced degrees, and, more specifically MBAs").

Id. at 32.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 14

on learning something valuable,

and in contrast was not significantly correlated

with being a "performance-oriented" learner focused on academic success for its

own sake.

Correlating 157 law student responses to a learning goals survey with

academic variables, including class rank, LSAT score, UGPA, and lawyering skills

grades, the study found as follows: class rank positively correlated with lawyering

skills grades (r=0.57), but less so with UGPA (r=0.46), and even more weakly with

LSAT (r=0.23).

The study also found class rank was positively correlated with

being a "mastery-oriented" learner

but not with being a "performance-oriented"

learner.

Another Christensen study found different legal reading strategies correlate

with high first semester grades.

Among 24 students, high-performance and low-

performance groups did not significantly differ in average LSAT or UGPA,

but

different reading styles dominated each group. The latter spent the most time on

basic “default” reading strategies: paraphrasing, re-reading, noting certain

structural elements of text, underlining text, and making margin notes.

The

former made heavier use of two more critical reading strategies: “problematizing”

strategies of purposefully asking themselves questions, making predictions, and

hypothesizing about meaning; and “rhetorical” strategies of moving through the

text in an evaluative manner or by synthesizing with the reader’s experiences.

Christensen's findings evidence the value of positivity, emotional intelligence,

work ethic, and learning styles – theories that abound but have not been proven as

to law school grades. Yet Christensen's and other studies do not control for other

variables, leaving a real possibility that the key variables are just proxies for other

qualities. Perhaps older students with real-world experience are more "mastery-

oriented" than those just out of college, whose recent focus on grades makes them

"performance oriented"; if so, then the key predictor is work experience, not

"orientation." Perhaps those with better reading strategies just did more recent

reading due to majoring in (for example) history or starting law school right after

college; if so, the key predictor is less “strategy” than quantity of recent reading.

And the finding that lawyering skill grades correlate with LGPA may show not

Leah M. Christensen, The Power of Skills: An Empirical Study of Lawyering Skills Grades as

the Strongest Predictor of Law School Success, 83 ST. JOHN'S L. REV. 795, 799, 806 (2009).

Id. at 800, 804.

Id. at 805. Where “r” is the correlation coefficient.

Id. at 799, 806.

Id. at 800, 804.

Leah M. Christensen, Legal Reading and Success in Law School: An Empirical Study, 30

SEATTLE U. L. REV. 603, 604 (2007).

Id. at 615.

Id. (LP students spent a mean time of 77.48% engaged in default strategies, 12.54% in

problematizing strategies, and 9.56% in rhetorical strategies).

Id. at 609-610, 625 (HP students spent a mean time of 21.43% engaged in default strategies,

45.70% in problematizing strategies, and 32.87% in rhetorical strategies).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 15

that particular student types do well; it may show just that good students perform

equally well in skills and other classes. Multivariate analyses simultaneously

examining all available data could distinguish between factors Christensen notes

and other factors.

C. Emotional Intelligence

Research outside of law indicates that IQ-like raw intelligence may predict

academic success, yet poorly predict job or relationship success.

The reverse may

be true of emotional intelligence (“EQ”), or "social intelligence": ability to

recognize and manage emotions, as well as see and care about impacts on others.

One study on MBA graduates found that businesses look less for IQ and more for

EQ traits, such as initiative, communication ability, and interpersonal skills.

Another study found that roughly half of job performance relates to EQ.

And yet

another study examined showed that student’s background, interests, hobbies and

typical behaviors in a wide variety of academic and life situations positively affect

performance.

Notably, EQ can improve,

making it not a purely endogenous

predictor, but a trait learnable from training or experience in roles requiring

emotional awareness. These studies support Kenneth Kleppel's argument that

lawyer intellectual and professional skills are overvalued compared to EQ.

Lawyers have enough intellect to pass law school and bar exams, and most gain

needed skills early in their careers – but they vary widely in EQ,

which can help

them in several ways: dealing with emotions like anxiety and anger; making them

leaders; and improving how clients or juries view them.

While there is solid theory and data on the importance to work success of EQ,

and of related traits such as leadership, maturity, and discipline, there is less solid

data on the importance of these traits to academic success.

Work, especially

Carl A. Leonard, Chapter 3. Leading the Law Firm, in HILDEBRANDT HANDBOOK OF LAW

FIRM MANAGEMENT (2012).

Gretchen Neels, The EQ Difference, 28 LEGAL MGMT. 44, 46 (2009).

Id. at 46.

ADELE B. LYNN, THE EQ INTERVIEW: FINDING EMPLOYEE WITH HIGH EMOTIONAL

INTELLIGENCE (2008).

Neal Schmitt et al., supra note 32 (showing that biographical data positively predicts

undergraduate performance).

Id.

Kenneth Kleppel, Emotional Intelligence is Key to Success, 2007 OHIO LAWYER 1, 1 (2007).

Id. at 1.

Id. at 2-3.

Marjorie M. Shultz & Sheldon Zedeck, supra note 31; For a discussion of non-cognitive

factors explaining academic performance in an undergraduate context, see Neal Schmitt et al.,

supra note 32 (concludes that Results indicate that the primary predictors of cumulative college

grade point average (GPA) were Scholastic Assessment Test/American College Testing

Assessment (SAT/ACT) scores and high school GPA (HSGPA) though biographical data and

situational judgment measures added incrementally to this prediction); and For a discussion of

non-cognitive factors explaining academic performance in an medical context, see Lievens and

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 16

lawyer roles requiring client contact, ability to persuade, and resilience under

stress, likely places a premium on EQ and related traits. While students likely do

better by managing emotions and understanding others as well, little evidence

proves so.

In sum, the broad theoretical, and limited empirical, work on beyond-the-

numbers soft skills and traits is valuable – but further study, especially multivariate

analysis, is needed to assess their impact on law student grades. No study can code

thousands of students' personal traits, of course; this study attempts to code for

various experiences viewable as proxies for personal traits, such as having work

experience versus attending law school right after college (a possible proxy for

maturity), college leadership roles (a proxy for EQ), a criminal or disciplinary

record (also a proxy for maturity, as well as for impulse control), and an improving

GPA during college after a lower starting GPA (a proxy for resilience, in the sense

of ability to improve after suffering a setback in an important endeavor).

III. METHODOLOGY

A. The Data Set

Following is how the authors procured and coded their data – a lengthy process

that made this Article's empirical analyses possible. The working hypothesis was

that information in students' law school applications and academic records can help

predict their future success as law students. For each of the over 1400 students in

the University of Colorado Law School and Case Western University Law School

graduating classes of 2008-2011, we collected the following: (1) data from the

original 2005-2008 law school applications on their college, employment,

extracurricular, and criminal/disciplinary records; (2) data from law school and

university registrars on their law school courses, grades, and activities; and (3) data

from law school career services offices on their bar passage and post-graduation

employment. Most of the data in categories (2) and (3) are for future study of

employment and bar outcomes, so the focus below is category (1): applicant data.

We collected data from the 2005 to 2008 applications received by the

University of Colorado Law School or Case Western Law School from those

matriculating to join the graduating classes of 2008 to 2011: the basic application

LSAC collects and distributes to each law school; the transcript and semester-by-

semester UGPA report that LSAC compiles and distributes to each law school; the

resume that nearly every applicant submits; and any other materials that flesh out

details in the application.

Because reviewing and entering this data required reviewing each individual

Sackett (2012), The Validity of Interpersonal Skills Assessment via Situational Judgment Tests

for Predicting Academic Success and Job Performance, Journal of Applied Psychology, vol.

97, 460-468.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 17

application, the authors, and those they employed to assist, spent several hundred

hours on that review and data entry: opening each applicant's folder; reviewing the

information; discussing any ambiguous or unclear data so the authors could decide

how to code such data; and entering the data into a spreadsheet. All such data

review and entry was either conducted by, or supervised on-premises by, one of

the authors; i.e., no data was evaluated or entered without one author present for

resolving any ambiguities. The admissions data entry was on-site at each law

school,

because the paper files were voluminous and contain sensitive data that

had to remain secure.

We created our database by entering the following information from each

application: (1) LSAT score (the highest if there were multiple); (2) UGPA; (3) the

median LSAT score of those at the college from which the student graduated

("LCM"), as a measure of college quality; (3) college major; (4) college graduation

date; (5) whether UGPA rose materially during the final undergraduate semesters

(yes=1, no=0); (6) significant college leadership roles (yes=1, no=0); (7)

attainment of a graduate degree (yes=1, no=0); (8) a significant criminal or college

disciplinary record, i.e., more serious than an "open alcohol container" infraction

(yes=1, no=0); (9) number of years between college and law school; (10) total

number of years employed before law school; (11)-(16) number of years employed

in each of six categories of employment (each is listed and defined below); (17)

number years of substantive work experience, i.e., more substantial than temporary

or part-time work; (18) a written summary of the employment experience;

(19)

state of residency as of the application date; (20) year of birth; (21) whether the

student identified as having any nonwhite ethnicity (yes=1, no=0); (22)-(25)

whether the student identified any nonwhite ethnicity (African American;

Hispanic/ Latino; Asian / Pacific Islander; or Native American / Native Alaskan)

(yes=1, no=0); (26) gender (male=1, female=0); and (27)-(33) whether the student

had one of seven categories of college majors (each is listed and defined below)

(yes=1, no=0).

Regarding the six categories of employment and seven categories of college

majors: because there are too many particular jobs or majors to code each

individually with a useful sample size, we grouped similar job types, and similar

majors, into several broad categories – and the data entered were whether the

student had each specified major or job category, as well as the number of years

worked in each job category. We had the following categories of majors and jobs:

Moss traveled twice to Case Western, personally entering nearly half the data at that school

and supervising Case Western staff who helped him enter the rest. Marks and Moss, combined,

entered the vast majority of the Colorado data, with help from staff with whom they worked.

Institutional Review Board ("IRB") review and each law school dean's consent were procured

to access all data; the authors also signed a confidentiality agreement allowing reporting of the

aggregated findings in this Article, just not disclosure of information on individual students.

We did not create a separate variable based on this written summary; we just entered and

maintained this data to document what kinds of work we classified in (11)-(16), the dummy

variables for each of six categories of employment types.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 18

• Majors: (1) psychology, sociology, anthropology, or religious studies; (2)

economics, finance, or accounting; (3) political science, public policy, or

government; (4) science, technology, engineering, or math; (5) fine arts,

music, drama, or performing arts; (6) environmental studies, forestry, or

ecology; and (7) liberal arts, history, any language, or philosophy.

• Jobs: (1) teaching (any level, preschool to college); (2) legal (e.g., paralegal,

investigator, or law-related job such as child services); (3) business or

management (financial work like accounting, investing, or banking, as well

as sales work above that of a retail salesperson, such as securities work or

managing an entire retail store); (4) science, technology, or medical (e.g.,

scientist, lab technician, nurse, programmer, or engineer); (5) military (any

branch); or (6) public service (e.g., government, non-profit, or political

work).

B. Regression Analysis of Admissions Criteria on Law School Grades

1. Hypotheses

By including as many variables as we could code, we set out to test various

hypotheses that law student success can be predicted by (a) traits law schools value

highly for applicant selection, (b) traits law schools appear to value less (if at all),

and (c) traits the literature depicts as positive predictors of success. Specifically,

we tested the hypotheses that high LGPA can be predicted by variables serving as

metrics of the following personal qualities – with certain variables serving as

possible proxies for more than one personal quality (e.g., having work experience

may be a proxy for maturity, but having no work experience, may be a proxy for

being more able to acclimate to law school quickly). Table 1 outlines traits we

hypothesized to predict law school success, followed by variables selected to test

these hypotheses in the empirical analysis that follows. To be clear, some

hypotheses included in Table 1 were exploratory, rather than testing a clear

hypothesis or taking a particular side. For example, it is beyond the scope of this

paper to review the literature on the effect of demographic factors on law school

success, such as whether female students are more successful than male students.

Where a major did not fit cleanly into one category, either (a) no "1" was entered in any

category (e.g., for the few "recreation management" or "equestrian" majors), or (b) a judgment

call was made about which category a particular major fit into (e.g., "forestry" could be more a

science major or more an environmental major, depending on the particular student's

coursework). We coded 103 students with no major. When a student had a double major, we

counted that major as well. There were 239 double-majors and six triple-majors.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 19

Table 1: Hypotheses, and Variables Selected to Test Those Hypotheses

Traits Hypothesized to

Predict Success

Variables Selected to Test the Hypotheses

1. Academic ability

• LSAT (& increasing/decreasing return variants)

• UGPA (& increasing/decreasing return variants)

• Certain majors (e.g., STEM)

2. Rigorousness of prior

Academics

• Having another graduate degree (& interactive term of graduate

degree & being right out of college)

• LCM (& increasing/decreasing return variants, as well as variant

interacting LCM & UGPA)

3. Familiarity with the

Educational setting

• Certain majors (e.g., reading- or law-related)

• Work experience as binary dummy variable (i.e., no work equals

attending law school right after college)

• Certain work types (e.g., law or public service)

4. Work ethic and

Resilience

• Rising UGPA (generally, or only if right out of college)

• High-UGPA/Low-LSAT profile

5. Maturity and emotional

Intelligence

• Leadership experience (generally, or only if right out of college)

• Lack of criminal/disciplinary record

• Certain work types (e.g., military or teaching)

• Work experience length (i.e., 1-4, 5-9, or 10+ years)

6. Demographic traits

• Gender

• Various race/ethnicity self-identifications

NOTE: This table describes the hypotheses and variables used to test those hypotheses

2. Models

a. The Primary Regressions: Models 1 (LGPA) and 2 (1L GPA)

We specified two ordinary least squares ("OLS") regression models to test the

above hypotheses. Our two primary models included the same independent

variables as predictors, but with different dependent variables: cumulative law

GPA ("LGPA") in Model 1, and first-year law GPA ("1L GPA") in Model 2. We

explored both on the theory that some students may adjust more or less quickly to

law school, so some variables may more strongly predict 1L GPA than cumulative

LGPA. For example, consider law students with less, or less-recent, reading and

writing exposure, such as science or finance majors (compared to history, political

science, or English majors), or those several years removed from college. Such

students may under-perform 1L year, being unfamiliar or rusty with heavy reading

and writing – yielding subpar 1L GPA. But as they adjust to law school, or

specialize in their chosen upper-level curriculum (e.g., intellectual property or

corporate transactions), their performances may disproportionately improve –

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 20

yielding improved LGPAs. This is just one example of how some talented students

may need more time to adjust to law school – yielding subtle differences in

predicting 1L GPA and cumulative LGPA.

We ran these two regressions, Model 1 and Model 2, using the entire data set,

with 1419 observations and 28 independent variables; Table 2 in Section III

displays the results. Among the independent variables, three are continuous

variables and 25 are dichotomous (0/1) “dummy,” variables. Table 4 in the

Appendix provides the summary statistics for the variables in the dataset, while

Table 9 provides means and variances for selected dummy variables. The means

and standard deviations of the continuous variables in our study are as follows:

LSAT (mean=159, std. dev.=5.30, range=133 to 178), UGPA (mean=3.43, std.

dev.=0.35, range=2 to 4.11), LCM (mean=154, std. dev.=4.15, range=132 to 168),

LGPA (mean=3.18, std. dev.=0.34, range=2.03 to 3.99), and 1L GPA (mean=3.08,

std. dev.=0.41, range=1.87 to 4.0).

We were interested in the incremental effects of adding variables to the model

instead of entering them all simultaneously. We ran six versions of each model to

measure the effect of adding certain pre-determined groups of variables. For each

set of regressions, we began by running a simple "base" regression model

mentioned in the previous studies, with only the most obviously relevant

predictors (e.g., UGPA, LSAT, and LCM). While LSAT was used in its simplest

form, we adjusted two variables, UGPA and LCM, after conducting robustness

checks for nonlinear effects of LSAT, UGPA, and LCM.

We also checked for

interactions between variables, such as whether UGPA mattered more at a

stronger college,

but ultimately did not use most interaction terms because they

We performed several tests to determine whether the effect of each continuous variable was

linear or nonlinear. First, we tested whether LSAT, UGPA, and LCM had consistently

increasing or decreasing, rather than linear, returns, by raising each to various powers above

1.0 (increasing returns) or below 1.0 (decreasing returns). For example, we replaced the LSAT

variable with LSAT raised to various powers from 0.25 to 3.0, to see which was a stronger

predictor. (We subtracted 130 from LSAT before raising it to any power, because 132 was the

lowest LSAT in the data, and raising values from 132 to 178 to various powers would

understate any nonlinearity, compared to a score starting just above 0.) Second, we tested for

discontinuities or sudden jumps at particular levels, such as (a) that LCMs below a certain level

may be especially bad (i.e., that weak colleges may be not just incrementally worse, but worse

by some nonlinear quantum, than average to strong colleges), (b) that UGPAs above a certain

level (e.g., some B+/A- level) might be especially strong plusses, or (c) that UGPAs below a

certain level (e.g., C+/B-) might be especially negative predictors. Third, as a catch-all test of

any nonlinear effects we might not suspect, we used the Stata fracpoly command to obtain an

estimate of any other nonlinear models that might fit the data better than the specific ones we

hypothesized; ultimately, the fracpoly results yielded no other nonlinear model better than the

models we ultimately chose on our own.

To test whether college grades are better predictors when adjusted for college quality, we

interacted UGPA with LCM (i.e., replacing UGPA and LCM with UGPA multiplied by LCM);

to test whether pre-law school academic traits – rising UGPA, college leadership, and having

another graduate degree – are better predictors when limited to those attending law school right

after college we replaced those three variables with an interaction between each and whether the

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 21

did not add any predictive power. Appendix Table 5 displays the simple LGPA

regression under column 1a; Appendix Table 6 displays the simple 1L GPA

regression under column 2a. Of note, the LGPA regression is based upon 1419

observations while the 1L GPA regression is based upon 1317 observations,

because it excludes those who transferred to the school after spending their first

year at another law school.

Not surprising, our results predicting 1L GPA, found in Table 6 column 1a,

are typical of other results found by the LSAC in their analysis of the usefulness

of LSAT as a predictor of 1L GPA. In a series of regressions using data from 152

unnamed schools over 2011 and 2012, LSAC estimated first year GPA from a

combination of LSAT and UGPA.

The LSAC study shows that our two schools

are “typical” in that the correlation coefficients between first year grades and the

LSAT, UGPA, and a combination of LSAT and UGPA, respectively, in our study,

are nearly identical to the LSAC study averages. The LSAC study reported these

median correlations: First Year Average (“FYA”) (a variable equivalent to our 1L

GPA) and LSAT (r=0.35), FYA and UGPA (r=0.29), and LSAT and UGPA

combined (r=0.47). Comparable to the LSAC study findings, our study found

these median correlations: FYA and LSAT (r=0.37), FYA and UGPA (r=0.28),

and LSAT and UGPA combined (r=0.39). Our results track the LSAC results,

making our two schools “typical” for comparison purposes.

As far as 1L GPA is concerned, our correlations and R-square results

generally track the LSAC findings. While the correlation coefficient gives us the

strength of the linear relationship between the coefficients, squaring the

correlation coefficient yields the coefficient of determination (“R-square”), which

gives us the variation that can be explained by the linear relationship between the

two variables. Their highest FYA and LSAT correlation (r=0.54), translates in an

R-square of 0.29 while their lowest FYA and LSAT correlation (r=0.16),

translates into an R-square of 0.03. The R-square values that we report in our

study are not the highest R-square values that the LSAC study reports – but they

are also not the lowest. They are closer to the averages that the LSAC study finds,

making our schools ‘typical’.

student had any work experience before law school. The sole interactive term that proved more

powerful was rising UGPA for those with no work experience, i.e., the interactive variable

testing whether rising UGPA had a greater effect for those attending law school right after

college.

See Anthony, Lisa A., Dalessandro, Susan P., and Reese, Lynda M., Predictive Validity of the

LSAT: A National Summary of the 2011 and 2012 LSAT Correlation Studies, Law School

Admissions Council, LSAT Technical Report No. 13-03 (Nov. 2013), available at

http://www.lsac.org/docs/default-source/research-%28lsac-resources%29/tr-13-03.pdf.

Id. at 17. Two further points reveal why, perhaps, our R-square for regressions 1a are within

the range of LSAC findings yet not on the high range of their findings. First, the LSAC study

cautions that r-square values can vary greatly among schools due to wider distributions which

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 22

After running the initial “base” regression model using a combination of

LSAT, UGPA and LCM, we successively re-ran the regression adding additional

variables parsimoniously (e.g., first adding ethnicity, then years of work

experience, work experience type, college majors and other control variables, in

that order). We inserted variables in groups because those variables had something

intrinsically in common, we inserted them when we did because we had a sequence

in mind. Admittedly, we expected the R-squared to grow as those variables

reduced the overall variance; we expected the ‘base’ variables to remain strong and

significant; and we expected that a variable that became significant would not lose

its significance in subsequent models. Table 5 (Appendix) displays the additional

LGPA regressions under columns 1b-1f; Table 6 (Appendix) displays the

additional 1L GPA regressions under columns 2b-2f. (Columns 1f and 2f in Tables

5 and 6, respectively are the full models, reproduced and interpreted in Table 2,

Section III, infra). In the LGPA regressions, we were surprised to find that ‘1-3

years of work experience’ variable was significant in regression 2c but lost

significance to ‘4-9 years of work experience’ in the final model, 2f. Both

variables ‘tech employment’ and ‘art and music major’ were negative and

significant (albeit at the 10% level) in the final regression only. In the 1L GPA

regressions, we were surprised to see that the variable ‘10+ years of work

experience’ was later replaced in significance by the variable ‘4-9 years of work

experience”. The ‘teaching work experience’ variable decreased in significance

from the 1% level in regressions 2c-e, to 5% in the final regression, 2f.

In addition to the primary models noted above, we specify 3 additional models

to explore additional questions. First, are there subtle differences between what

predicts especially high and especially low grades? Second, who is the better bet,

the high-UGPA candidate with a low LSAT, or the high-LSAT candidate with the

low UGPA? We tackle each inquiry below.

b. The Quarter Regressions: Model 3 and Model 4

While the primary regressions examine what predicts LGPA and 1L GPA,

Models 3 and 4 (“The Quarter Models”) test for subtle differences between what

predicts success and what predicts failure. Our hypothesis was that perhaps a

certain negative trait predicts a very low LGPA, but its absence does not predict

any difference between high and mid-range LGPAs, and the reverse could be true

for a positive trait. To examine what predicts top-quarter (“Q1”) or bottom-quarter

(“Q4”) LGPAs, we specified two logistic regression models.

Logistic regression

will lead to lower R-squares, individual schools’ variability of LSAT scores and UGPAs, the

correlation between LSAT score and UGPA, and the amount of variability in the first year

grades. Another factor to consider is that our study reports adjusted r-square, a value which is a

lower (adjusted for the parameters) value than the r-square.

The top-quarter subset included the top quarter of students at both law schools; the bottom-

quarter subset included the bottom quarter of students at both law schools.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 23

techniques are used when the dependent variable is dichotomous; in our case, the

dependent variable was coded “1” if the student was in the specified quarter, else

“0.” Thus, in Model 3, the dependent variable is membership in the top quarter; in

Model 4, membership in the bottom quarter. We ran these regressions using the

same independent variables used in Model 1; the results are in Appendix Table 7.

c. The Splitters Regression: Model 5

There is a recurring debate in the admissions world: if forced to choose

between the two major numerical criteria, LSAT and UGPA, who is the better bet,

the high-UGPA candidate with a low LSAT, or the high-LSAT candidate with the

low UGPA? We specified a model to test whether students with either "splitter" –

high-UGPA/low-LSAT or low-UGPA/high-LSAT – performs differently from the

other type, or from non-splitters. Using only a dataset of splitters (733

observations) Model 5 uses OLS regression techniques to predict LGPA using all

independent variables in the previous models, replacing the UGPA and LSAT

variables with (a) an index combining LSAT and UGPA and (b) including an

indicator variable for "mild splitters", students with a top-50% LSAT but bottom-

50% UGPA and vice versa, coded "1" if the applicant fit into that profile, else "0."

Since the dataset only contained splitters, the default category is the high-

UGPA/low-LSAT profile. The Model 5 results are found in the Appendix Table 8.

For robustness, we ran two additional OLS models. In the first regression we

used a dataset of "extreme splitters", students with a top-25% LSAT but bottom-

25% UGPA, and vice-versa to test whether the high-LSAT but low-UGPA

performs differently than the high-UGPA but low-LSAT profile. Next, we ran a

second model including all 1435 observations, the index again in place of LSAT

and UGPA, and a dummy variable for the high-UGPA/low-LSAT splitters to test

whether the high-UGPA/low-LSAT splitters did worse or better than non-splitters.

Table 4 in the Appendix also details the sample sizes in these groups. A more

lengthy discussion of the splitter regressions is found infra, Section IV.D.

d. The Variance Analysis

Finally, following the five regression models, we examined whether LGPA

had greater variance for any group represented by one of the dichotomous dummy

variables, e.g., each cluster of majors, and each cluster of job types. A finding that

a group had higher variance than other similarly-sized groups could hint that the

group contains high-risk/high-reward candidates, or that the group is a

heterogeneous mix requiring closer individual scrutiny of individual members.

IV. KEY RESULTS AND INTERPRETATIONS

A. Caveats: Limitations on Modeling Law Student Performance

This Article's core findings are from the Model 1 regressions exploring what

predicts LGPA. The results of the “Quarter Regressions” and “Splitter Models”

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 24

further refine those findings.

Before detailing the results, three key caveats and

limitations of our regression models warrant mention, to avoid overstating the

findings and to note possible biases in the results.

First, we could not code for many variables that may be valuable as predictors

of law school performance. Writing ability is likely an important predictor, but one

that was not feasible to enter as coded data. Reading and grading the writing in

over 1400 applications with sufficient consistency would have been a possibly

insurmountable challenge, but more importantly, true writing samples were not

consistently available. Applicants' personal statements are commonly edited by

others, as evidenced by how (in the authors' experience reading thousands of law

school applications) the unedited handwritten LSAT essays are far less strong,

grammatically and stylistically. Yet a sizeable minority of the handwritten LSAT

essays are illegible, either because of bad handwriting or because they are written

in often-smudged pencil. We similarly could not code directly for personal

qualities and backgrounds that could bear on law school success, such as family

educational and socioeconomic background and personal qualities such as

resilience, optimism, etc. Even if we could code hints of such factors reliably from

subjective indicia in personal statements, many applicants do not mention or hint at

such factors (e.g., some mention family economic and educational background, but

many do not, and some mention obstacles they overcame, while others do not), so

the data would be too incomplete to be entered into a regression for most or all of

the population. However, we tried to keep these possibly important but uncoded

qualities in mind in interpreting our results, because – as detailed below – the

findings hint that certain variables may be proxies for uncoded qualities such as

work ethic, resilience, etc.

Second, though the populations of the two law schools vary, they do not cover

the entire range of law students. For example, the population in our data set

contains a wide range of LSAT scores: the bottom 5% (i.e., about 72 students) are

at or below 150, while the top 5% (also about 72 students) are at or above 168. Yet

there are law schools at which many more students have LSAT scores in the 140s

or in the 170s. Thus, while we chose our two schools to maximize representation

of the low 150s to mid-160s LSAT range that is most common, our results may be

less generalizable to the very top and bottom of the law student population.

Third, there is possibly a bias in favor of the high-LSAT/Low-UGPA splitters

over the high-UGPA/Low-LSAT splitters. There is some evidence of this in that

our data on “mild splitters” – students with top-50% LSAT but bottom-50%

UGPA or vice versa contained more high/LSAT splitters. Law schools may bias

admission toward one splitter category to improve their LSAT and UGPA

medians.

Finally, we face an inherent limit in statistically modeling a population that is

not a random sample. Law students are not a random sample of law school

All models were run using the Stata, version 12, statistical software.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 25

applicants, but the subset deemed worthy of admission – which biases our findings

mainly toward understating the effect of certain traits.

For example:

• those with the worst negative discipline or criminal records are denied

admission, so our population includes only less negative records –

biasing our study toward finding a record has less (or no) effect; and

• among applicants with low UGPA or LSAT scores, only those with

enough other positive qualities are admitted, so our population

includes only the subset of low-scorers with other positives – biasing

our study in favor of finding less (or no) effect of a lower score.

Formally, our data set features Berkson's bias, a form of selection bias: by

analyzing only the subset of applicants who matriculated, we obtain only

conditional estimates (of the subset who met the condition of being admitted), not

unconditional estimates (of how the entire applicant population would perform).

This form of selection bias is common in many fields, such as criminal or civil

litigation, where analyses of trial outcomes consider only a conditional subset –

cases not resolved before trial (by plea, settlement, dismissal, etc.).

Because the

problem is a bias due to an omitted variable (the odds of being selected into the

population being examined), the Heckman model can sometimes correct for the

bias, by adding a second step to the regression: first, a "selection function"

estimates the odds an individual becomes part of the population (here, the odds of

admission); then, that estimate is inserted into the "response function" analyzing

the effect of each variable (here, LGPA), to correct for the fact that some

individuals were more likely to be selected than others.

Yet the Heckman model

proved not to be a feasible corrective here, because it requires fuller data than we

could procure on all potential population members, and because it requires strict

conditions which cannot be met in this study.

Ultimately, lacking counter-factual data on how non-admitted students would

have performed if admitted (e.g., those with especially negative records, or low

scores, not mitigated by other positives), we simply must note that our study, like

other studies on matriculants

, is biased toward under-stating the effect of most

variables. Ultimately, the bias may not be substantial for two reasons.

LSAC also acknowledges this bias in their studies of law student performance. See Anthony,

Lisa A. et al., supra note 70 at 12-13.

See, e.g., Shawn Bushway, Brian D. Johnson, and Lee Ann Slocum, Is the Magic Still There?

The Use of the Heckman Two-Step Correction for Selection Bias in Criminology, 23 J. OF

QUANTITATIVE CRIMINOLOGY 151 (2007).

James Heckman, Sample Selection Bias as a Specification Error, 47 ECONOMETRICA 153

(1979); James Heckman, Dummy Endogenous Variables in a Simultaneous Equation Model, 46

ECONOMETRICA 931 (1978).

See Anthony, Lisa A. et al., supra note 70 at 17 (In their study on LSAT validity, LSAC

notes, “Correlations obtained from matriculated students tend to underestimate the true validity

of the test. Even so, they are the best information we have available, and even as underestimates

they are quite reliable”).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 26

First, we did find many variables to be highly significant predictors of 1L GPA

and LGPA – likely because the two key predictors, LSAT and UGPA were not

negatively correlated. The worst-case scenario for bias would have been if LSAT

and UGPA had been negatively correlated. If, among those admitted with a high

LSAT, those with a low UGPA were more likely to matriculate (because those

with a high LSAT and UGPA receive more and better admission offers), then the

matriculants with a high LSAT (a positive predictor) would have a

disproportionately low UGPA (a negative predictor); to the extent that a high

LSAT is usually accompanied by a low UGPA, then LSAT would not appear to be

as positive a predictor as it truly is. And vice-versa: if those who matriculated with

a high UGPA tended to have lower LSAT scores, then UGPA would not appear to

be as positive a predictor as it truly is. Yet in our data set, LSAT and UGPA were

not negatively correlated.

Thus, the data do not support a key feared source of

bias: that those who matriculated with one positive predictor probably were worse

in other ways, leaving the effect of that positive predictor understated.

Second, the relative predictive power of LSAT and UGPA that we found made

intuitive sense, was consistent with findings in other studies, and should not be

affected by selection bias. LSAT is stronger at predicting first-year grades (the

correlation between 1L GPA and LSAT, and 1L GPA and UGPA, are 0.36 and

0.27, respectively); UGPA is slightly better at predicting cumulative grades (the

correlation between LGPA and LSAT, and LGPA and UGPA, are 0.28 and 0.29,

respectively). While these correlations might be higher if it were feasible to

examine how the full applicant pool (including rejected applicants) would have

performed, their relative values would not likely change. Corroborating this

interpretation is an LSAC study of 152 law schools, in which correlations for a full

applicant pool did prove higher than those for a matriculant pool, but the relative

predictive power of LSAT and UGPA as to first-year grades remained the same.

B. The Primary Regressions: Predicting Cumulative LGPA (Model 1) and 1L

GPA (Model 2)

What variables predict higher law school grades? Below, Table 2 is the full set

of results detailing each variable's OLS coefficient and significance; Table 3

summarizes the magnitude of each significant variable's correlation with LGPA;

and Table 9 provides variances and standard deviations for selected dummy

variables.

LSAT and UGPA had a positive and modest correlation of 0.187. See also Anthony, Lisa A.

et al., supra note 70 at 18 (in a study of 152 law schools between 2011 and 2012, finding the

average correlations between LSAT and UGPA are close to zero and range from -0.45 to 0.24,

suggesting that a number of law schools employ a compensatory admissions model in which a

high LSAT score compensates for a low UGPA, or vice-versa).

Id. at 18 (to estimate the correlation coefficients with first year law school grades for the

entire applicant group, a statistical adjustment for restriction of range was applied to the data

that are available for the group of students who matriculate; the applicant pool correlations are

adjusted based on Pearson-Lawley formulas).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 27

Unsurprisingly, factors predicting 1L GPA (Model 2) were much the same: 1L

GPA is a subset of LGPA, so variables predicting 1L GPA likely impact LGPA,

and qualities predicting 1L grades also likely predict 2L-3L grades. We

hypothesized and found only subtle differences between the 1L GPA and LGPA

predictors: some factors predict slower acclimation to the reading, writing, and

legal analysis demands of law school (i.e., worse 1L than cumulative LGPA);

others predict faster acclimation (i.e., better 1L than cumulative LGPA). The

adjusted R-squared is 0.263 for Model 1 and 0.279 for Model 2, meaning the

predictor variables explained 26.3% of all variation in LGPA, and 27.9% of all

variation in 1L GPA, among law students.

Of note, we ran an OLS specification identical to those used above, only this

time we included 1L GPA to predict LGPA. Now, because 1L GPA is part of

LGPA, those two variables are highly correlated (r=0.88) and we expect that 1L

will be a strong and significant predictor of LGPA. Using 1315 variables to

predict LGPA, as expected, the adjusted R-squared in that regression is 0.7914 and

the coefficient for 1L GPA is 0.688, positive and significant at the 1% level.

Among our three highest Model 1 and Model 2 predictors, LSAT, UGPA and

LCM, only two are significant in this regression. The coefficient for UGPA is

0.083, positive and significant at the 1% level and the coefficient for LCM is

0.001, positive and significant at the 5% level. LSAT is negative but not

significant. Among all other variables, the data suggests that Asian Americans are

less likely to get higher LGPA (coefficient was -0.0544, significant at the 1%

level), and those with STEM or EAF backgrounds are more likely to get higher

LGPA (coefficients were 0.0340 and 0.0290, respectively, both at the 5%

significance level). While the goal of this study is not to predict LGPA using a

component of LGPA, this specification does reveal one interesting point about the

relationship between first year grades and third year grades. While the data

supports the finding that students who do well in their first year do well overall,

the same can be said for the bottom of the class – students who do not do well in

their first year do not do well overall. However, the 1L GPA predictor is not

perfect. It may explain 79% of the variance but it does not explain 100% of the

variance, revealing that interventions after the first year can potentially make a

difference in increasing LGPA.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 28

Table 2: OLS Regression Results for Model 1 (Dependent Variable: Cumulative LGPA) and Model 2

(Dependent Variable: First-Year LGPA)

Variables

Model 1: Cumulative Law

School GPA (LGPA)

Model 2: First Year Law

School GPA (1L GPA)

Traditional factors

Law School Admissions Test (LSAT)

0.016*** (9.31)

0.030*** (12.63)

Adjusted LSAT College Median (LCM)

0.003*** (3.55)

0.004** (2.98)

Adjusted Undergraduate GPA (UGPA)

0.272*** (12.44)

0.328*** (11.22)

Ethnicity

African American

-0.155*** (3.77)

-0.170** (3.35)

Latino/a

-0.148*** (3.29)

-0.148** (2.52)

Asian American

-0.154*** (5.81)

-0.130*** (3.77)

Native American

-0.173** (2.28)

-0.188** (1.97)

Employment duration

1-3 years

0.032 (1.47)

0.032 (1.16)

4-9 years

0.109** (2.88)

0.110** (2.49)

10+ years

0.014 (0.25)

0.081 (1.11)

Employment type

Teaching

0.082+ (2.20)

0.086+ (1.80)

Legal

0.022 (0.69)

0.015 (0.35)

Business

-0.023 (0.75)

-0.025 (0.61)

Technology

-0.05 (1.55)

-0.077+ (1.85)

Military

-0.119+ (2.25)

-0.231** (3.43)

Public Service

0.043 (1.17)

0.068 (1.44)

College major

Science, Tech., Engineering, Math (STEM)

0.066** (2.65)

0.061+ (1.90)

Economics, Accounting, Finance

0.058** (2.30)

0.032 (0.97)

Psychology, Sociology, Anthropology

-0.006 (0.30)

0.011 (0.38)

Art, Music, Drama

-0.038 (0.80)

-0.084+ (1.33)

Environmental Sciences

0.022 (0.42)

0.012 (0.17)

Liberal Arts, History

-0.001 (0.08)

0.016 (0.70)

Other factors

No work experience & rising college GPA

0.033 (1.45)

0.053+ (1.82)

Criminal History

-0.119** (3.39)

-0.137** (2.99)

Graduate Degree

0.030 (1.22)

0.037 (1.16)

University of Colorado Law Student

-0.209*** (10.12)

-0.225*** (8.33)

College leadership

0.018 (0.67)

0.019 (0.51)

Gender male

0.014 (0.89)

0.015 (0.72)

Constant

-0.821** (2.70)

-3.470*** (8.21)

Adjusted R

0.26

0.28

Observations

1419

1317

NOTES: Absolute value of z-statistics in parentheses. +p<0.10; ** p<0.05; ***p<0.01.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 29

Table 3: Summary of Magnitudes of Variable Correlations with LGPA (Model 1)

NOTES: +p<0.10; ** p<0.05; ***p<0.01.

Interpreting the Model 1-2 OLS regression coefficients is straightforward. The

coefficient for each independent variable reflects both the strength and type of

relationship the explanatory variable has to the dependent variable. When the sign

associated with the coefficient is negative, the relationship is negative; conversely,

when the sign associated with the coefficient is positive, the relationship is

positive. The more positive or negative the coefficient, the more it predicts LGPA.

The interpretations of the coefficients vary depending on the type of variables

in the study. Some variables are continuous, comprised of numbers along a

spectrum (e.g., UGPA, LSAT, LCM, and number of work years), while others are

dichotomous (e.g., a "yes" or "no" – coded for each work type, major, or

criminal/disciplinary record). For a dichotomous variable, "1" means having the

trait and "0" means not having it, so the coefficient reveals how much LGPA rises

or falls when that trait is present. For a continuous variable, the coefficient

Positive Predictors

Negative Predictors

Non-Predictive (No

Correlation w/ LGPA)

LSAT*** (best fit: linear)

• +1 LSAT pt. ≈ +0.02 LGPA

UGPA*** (best fit: increasing returns)

• if UGPA<3.4: +.08 UGPA ≈ +1 LSAT

• if UGPA≥3.4: +.04 UGPA ≈ +1 LSAT

(consistent across all college qualities)

LCM*** (best fit: decreasing returns)

• +1 LCM pt. ≈ +0.2 LSAT

• LCM<152 ≈ additional -1 LSAT

Major: STEM**; EAF**

• STEM major ≈ +4 LSAT

• EAF major ≈ +3½ LSAT

Work duration: 4-9 yr.**

• 4-9 yrs.' work ≈ +6½ LSAT

Work type: Teaching*

• Teaching ≈ +5 LSAT

UGPA rising ≥ 0.3, if enter law school

right after college (not sig.: p=0.126)

• Rising GPA ≈ +2 LSAT

Negative Disciplinary or

Criminal Record**

• Neg. Rec. ≈ -7

LSAT

Work Type: Military+;

Sci/Tech (not sig.: p=.110)

• Military ≈ -7

LSAT

• Sci/Tech. ≈ -3 LSAT

Demographics: Person of

Color Self-ID (** to ***)

• Person of Color Self-ID

≈ -9 to -10 LSAT

(but partly b/c a portion

enter w/ lower scores)

Work Duration: 10 or

more years.

Work Type: All other

than teaching & military

(i.e., law, sci./tech.,

business, public service)

Majors: All other than

STEM/EAF (i.e., social or

political sciences; history;

liberal arts; fine arts;

environment)

Demographics:

Gender (No discernible

M/F difference)

Prior Graduate Degree

(Any)

Major College

Leadership Role (Any)

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 30

represents the expected change in the dependent variable for a one-unit increase or

decrease in the associated independent variable, holding all other variables

constant. So the coefficient is the LGPA difference predicted by a one-unit

difference in the variable, holding all other variables constant – e.g., the LSAT

coefficient shows how much LGPA rises with each one-point LSAT rise. A few

continuous variables are nonlinear, to test for increasing or decreasing effects as

the variable rises, or for interactions with other variables; interpreting those

coefficients is less intuitive and will be discussed below as needed. The statistical

significance of each variable's correlation with LGPA is noted in Tables 2-3 by

asterisks: three asterisks (***) is the strongest statistical relationship, a 1% or

lower chance the relationship resulted from chance variation; two (**) means a 5%

or lower chance; a (+) means a 10% or lower chance (typically considered barely

significant); no asterisk means a variable is not significantly correlated with

LGPA. In Table 2, the results in bold highlight statistically significant results.

One novelty of this study is the way that it presents the key results in two

ways. Like most traditional empirical studies, it presents the results based on

coefficients and relative magnitudes. To explain the results more intuitively, we

also present results in comparison to LSAT points. Because Model 1 uses a linear

regression, the coefficient on each variable is the effect on LGPA of a one-unit

change in the that variable (e.g., the .016 coefficient on LSAT means each extra

LSAT point predicts an extra 0.016 in LGPA, holding other factors constant). That

also means each variable's effect can be compared – and here, comparison to

LSAT points is an intuitive way to illustrate the relative power of each variable

that proved significant (e.g., the coefficient on teaching experience, 0.082, is just

over five times the LSAT coefficient, 0.016, so it is roughly equivalent to five

LSAT points). Table 3 lists of the number of LSAT points to which each other

significant variable is equivalent.

The following nine subparts of this section detail the key results.

1. LSAT: 1 LSAT Point ≈ 0.016 LGPA

LSAT is, as in all prior studies, a significant LGPA predictor. The coefficient

is 0.016, positive and significant at the 1% level. Roughly, each additional LSAT

point predicts a 0.016 LGPA rise (the coefficient on LSAT, measuring the effect

on LGPA of each LSAT point). This magnitude is large enough to make a real

difference, because candidates typically vary by many points; a 6-point LSAT gap

between two candidates predicts a 0.1 LGPA gap – a material difference in class

standing.

Though LSAT is a significant predictor, for three reasons its validity as an

admissions criterion is more modest than is implied by how heavily schools weight

it in admission and scholarship decisions.

First, the magnitude of the predictive

power of LSAT is modest compared to how heavily schools weight LSAT scores.

See supra Part III(B)(2)(c) (noting evidence various law schools weight LSAT far more than

UGPA).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 31

A 6-point LSAT difference is enough to make a dispositive difference in where

one attends law school and whether one receives a six-figure scholarship – but

even that large an LSAT gap really predicts only a modest 0.1 difference in LGPA.

Further, LSAT is just one valid predictor among many: as detailed below, many

other valid predictors each are the equivalent of a 2-7 point LSAT difference.

Second, changes in LSAT do not appear to have increasing or decreasing

returns; an X-point difference between a low and very low LSAT predicts the same

as an X-point difference between a high and very high LSAT.

Thus, contrary to

some common assumptions, a "cutoff" driven by fear of an especially low LSAT is

unsound: the difference between a 147 and a 152 is the same as the difference

between a 157 and a 162; and as noted below, various positive predictors each are

akin to having several additional LSAT points, so even an LSAT score 12-15

points below a school's median can easily be counteracted by enough other

positives.

Third, roughly half the LSAT's predictive power may be for the non-

substantive reason William Henderson hypothesized: most law school exams and

the LSAT are roughly three-hour, timed, in-class tests, so the LSAT is predictive

partly as a mere measure of comfort and experience taking such exams. Henderson

so concluded in finding that the LSAT predicts in-class test grades better than

other grades (research papers, etc.), and our regressions provide further support for

that conclusion: the LSAT is nearly twice as predictive of 1L GPA as it is of

cumulative LGPA. Table 3 illustrates that each additional LSAT point predicts a

rise in 1L GPA of 0.030 (significant at the 1% level). If the LSAT purely tested

brainpower, it would not lose half its predictive power after 1L year. Because 1L

year amounts to an in-class exam boot camp, students' test-taking skills converge

by their 2L and 3L years – when the LSAT loses about half its predictive power.

Thus, while the LSAT helps predict LGPA, as much as half its predictive value is

not an aptitude test, but a non-substantive measure of ephemeral differences in

test-taking comfort and experience.

In sum, these findings – on the modest magnitude of LSAT's predictive power,

and on how half of that predictive power may be for a non-substantive reason –

call into question the heavy reliance on LSAT in law school admissions, law

school scholarship decisions, and law school rankings. To be sure, it is

understandable that law schools feel compelled to rely heavily on LSAT: as Part

This linear LSAT-LCM model was a better fit for our data than other models we explored,

including (a) a consistently increasing-returns LSAT-LGPA relationship (e.g., an exponent

above 1.0 on LSAT), (b) a consistently decreasing-returns LSAT-LGPA relationship (e.g., an

exponent below 1.0 on LSAT), (c) models hypothesizing a discontinuous effect at especially

high or low levels of LSAT (e.g., that a drop below a certain level such as 150 or 152, or a rise

above a certain level, such as 165 or 167, has a disproportionate impact), or (d) models allowing

a different coefficient on bottom-quarter and top-quarter LSAT scores (i.e., replacing LSAT

with an interactive terms of LSAT multiplied by whether LSAT was in each quarter), to test

whether to the effect of additional LSAT points was different in the mid-range than at the

extremes (and we found no material difference in the LSAT coefficient for any quarter).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 32

II(A) details, LSAT is a dominant driver of changes in law schools' rankings, to a

far greater extent than UGPA (which the rankings consider, but to a lesser degree)

and other factors wholly ignored by rankings' limited set of variables for student

quality (e.g., students' college quality, majors, and work experience). This Article's

findings simply indicate that the goal of accurately assessing applicant potential

does not support the substantial weight on LSAT that rankings incentivize law

schools to accord.

2. UGPA: Increasing Returns; 0.03-0.06 UGPA ≈ 1.0 LSAT Point

UGPA significantly predicts LGPA, but increases in UGPA have greater effect

at higher levels of UGPA. The coefficient is 0.272, positive and significant at the

1% level. The 0.272 coefficient on UGPA means that each full-point UGPA rise

(e.g., 2.0 to 3.0) predicts a 0.27-point LGPA rise, or (identically) each extra

hundredth of a point of UGPA predicts a 0.0027 LGPA rise. But the UGPA

variable that best fit the data was a doubling of that effect when UGPA is above

3.4 (i.e., just over the B+ level, the mean at most colleges); above 3.4, each extra

hundredth of a point of UGPA predicts a 0.0054 LGPA rise.

The most intuitive understanding of this magnitude may be to compare it to the

effect of LSAT: each 0.06 rise in UGPA is akin to 1 extra LSAT point, but above

3.4, the effect doubles, so each 0.03 rise in UGPA is akin to 1 extra LSAT point.

Thus, the difference average and weak UGPA is material (e.g., 3.0 versus 3.3 is

akin to 5 LSAT points), but not as powerful as the difference between good and

elite UGPA (e.g., 3.5 versus 3.8 is akin to 10 LSAT points).

Compared to prevailing models deeming LSAT a better predictor than UGPA,

we find that UGPA is more powerful – at least when, as here, the analysis controls

for factors that moderate the effect of UGPA, such as college quality and college

majors. For example, the U.S. News & World Report Law School rankings formula

assumes that one LSAT point is roughly equal to 0.084 of a point of UGPA.

That

would appear to over-weight LSAT substantially, compared to our finding that one

LSAT point is actually worth from 0.03 of a point of UGPA (for UGPA levels

above 3.4) to 0.06 of a point of UGPA (for UGPA levels below 3.4).

This inflection point at 3.4 was surprising but has a plausible explanation: a

This increasing-returns UGPA model was a better fit for the data than other models we

explored, including (a) a linear UGPA-LGPA relationship, (b) an increasing-returns UGPA-

LGPA relationship (e.g., an exponent above 1.0 on UGPA), (c) a decreasing-returns UGPA-

LGPA relationship (e.g., an exponent above 1.0 on UGPA), or (d) other sizes or locations for a

discontinuity in the slope of the UGPA-LGPA relationship, such as placing the discontinuity at

other levels from 2.7 to 3.8.

Each school is ranked by U.S. News based on a score that is 12.5% its median LSAT score

and 10% its median UGPA. See Sam Flanigan & Robert Morse, Methodology: 2016 Best Law

Schools Rankings, U.S. NEWS & WORLD REPORT, http://www.usnews.com/education/best-

graduate-schools/articles/law-schools-methodology (last visited Feb. 26, 2015). One additional

LSAT point therefore adds 0.21% to a school's score; the quantum of additional UGPA that

adds an equal 0.21% is 0.084.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 33

higher UGPA is better, but the difference between "weak to average UGPA" (e.g.,

2.9 to 3.3) is less impactful than the difference between "good to great UGPA"

(e.g., 3.5 to 3.9)." The typical college has a roughly 3.3 mean, so 3.4 may be

serving as a rough threshold for having a better-than-average UGPA.

Despite the plausibility of this finding, this sort of sudden jump in the effect of

UGPA at 3.4 is probably an oversimplification, reflecting only that an inflection

point was the curve of best fit for modeling what appears to be a reality that while

rises in UGPA are always better, they are more significant for above-average than

for weak UGPAs. Furthermore, we cannot be sure of the exact magnitude of the

over-weighting – there likely are more subtle gradations from 0.03 to 0.06 than our

model can estimate – but U.S. News likely has not run any similar study, so it’s far

greater LSAT-to-UGPA ratio seems to over-weight LSAT substantially as a

measure of a school's student quality. A final disclaimer is that a law school with

an unusually strong student body (e.g., Yale, Harvard, or Stanford) or an unusually

weak one (e.g., schools with nearly open admissions that admit many students with

UGPAs in the C grade range) might experience no such inflection point, or a

different one than 3.4.

3. LCM: Modest, Decreasing Returns; 1 LCM ≈ 0.2 LSAT Pt., But with

LCM<152 Amounting to an Extra -1 LSAT Point

A college's LCM, the average LSAT of its students, may be an unintuitive

college quality measure. But a universal college quality metric is hard to find.

Published college rankings are no viable option because they do not place all

colleges on one continuum, instead ranking only the best colleges (others are listed

as "unranked") and separately ranking "National Universities," "National Liberal

Arts Colleges," "Regional Universities," and "Regional Colleges."

Similarly,

rankings of colleges' research quality, even if a valid measure of college quality, do

not help distinguish the quite varied quality of the many non-research-focused

colleges (e.g., local commuter-based public colleges).

Unlike rankings, LCM is data available for virtually all colleges that law

students attended – and it does significantly predict LGPA. In the Model 1

Regression on Table 2, the coefficient for LCM is 0.003, positive and significant at

the 1% level. A 1-point LCM rise is akin to a 0.215 LSAT rise, so 4.7 LCM points

are akin to 1 LSAT

– a common difference between a flagship state school and a

solid yet weaker satellite campus. But the LCM variable that best fit the data had a

Best College Rankings and Lists, U.S. NEWS & WORLD REP.,

http://colleges.usnews.rankingsandreviews.com/best-colleges/rankings?int=a8f209 (last visited

Feb. 26, 2015) (separately listing four school categories and leaving several unranked in each).

The coefficient on LCM, noting the effect of each LCM point, was 0.0034795; the number of

LCM points necessary to equal the 0.0163022 effect of one LSAT point thus is 4.68. In addition

to the relationship between LCM and LSAT, we also examined the relationship between LCM

and college majors and found no evidence that college quality matters for one major versus

another. Regardless of major, we found a 0.15 LGPA difference between the students in a top-

quarter LCM college and students in a bottom-quarter LCM college.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 34

discontinuity: a sub-152 LCM is akin to almost a full-point drop in an individual

student's LSAT.

Thus, college quality matters, but (a) not as much as individual

student qualities, and (b) the difference between weak and middling schools

matters more than between average and strong schools.

Any discontinuity this striking could reflect quirks in the data – but we find it

plausible: while college quality matters, subtle differences matter only modestly;

what is most important is whether a student attended a particularly weak college –

e.g., those with a sub-152 LCM. Take the state of Colorado, the source of many

Colorado Law students: the flagship state college, the University of Colorado at

Boulder, typically has a 156 LCM (depending on the year), while the other

prominent state college, Colorado State University, typically has a 153 LCM; both

draw students from across and outside the state. In contrast, other public colleges

in Colorado have mainly local, commuter draw: the University of Colorado

campuses in Denver and Colorado Springs typically have 151 LCMs; Metro State

University in Denver has a 149. The four-point discontinuity between 151 and 152

plausibly reflects that the three-point difference between the top state schools (with

LCMs of 153 and 156) matters less than the difference between those two and the

weaker local public colleges (with LCMs of 149-151). Admittedly, this strong a

discontinuity is suspect as a literal statement; it surely is not true that all colleges

with a 151 LCM are barely different from all those with a 150 yet very different

from those with a 152. But an LCM-LGPA relationship with this discontinuity

appears to be the curve of best fit to model a valid point: a difference between solid

and strong colleges matters less than a difference between weak and solid colleges

that is plausibly marked by having a sub-152 LCM.

Once we found that college quality matters, we examined whether, in addition,

the predictive power of UGPA depends on college quality. Specifically, while a

stronger college is better, is a higher UGPA also more of a positive predictor at a

stronger rather than a weaker college? To answer this question, we ran a variant of

Model 1 that estimated the difference, if any, between the effect of UGPA at (a)

top-quarter LCM colleges (LCM≥158 in our sample), (b) bottom-quarter LCM

colleges (LCM≤151), and (c) colleges with an LCM in the middle half

(152≤LCM≤157).

Ultimately, we found no difference between the predictive

The transformed LCM variable that best fit the data was linear, but with a discontinuity: when

LCM dropped below 152, an extra four LCM points were subtracted, making the drop from 152

to 151 the equivalent of a 5-point drop. This decreasing-returns LCM model was a better fit for

the data than other models we explored, including (a) a linear LCM-LGPA relationship, (b) an

increasing-returns LCM-LGPA relationship (e.g., an exponent above 1.0 on LCM), (c) a

decreasing-returns UGPA-LGPA relationship (e.g., an exponent above 1.0 on LCM), or (d)

other sizes or locations for a discontinuity in the LCM-LGPA relationship, such as a smaller

jump at 152, or a jump at other levels from 150 to 160.

We first created dummy variables for top-quarter LCM (dQ1LCM), bottom-quarter LCM

(dQ4LCM), and middle-half LCM (dQ2-3LCM). We then replaced UGPA with the following

three interactive variables: (a) GPA x dQ1LCM; (b) GPA x dQ2-3LCM; and (c) GPA x

dQ4LCM. This simply allowed the regression results to estimate a different coefficient for

UGPA depending on whether the student's college was high-, mid-, or low-LCM.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 35

power of UGPA at colleges with different LCMs: the coefficient on each of the

three UGPA interactive terms was similar (0.157 to 0.175). Thus, college quality

matters, but does not change whether UGPA matters; the difference between high

and low UGPA is just as important at weaker and stronger colleges.

4. College Majors: STEM/EAF ≈ 3.5-4 LSAT Pts.; No Negative Majors

We tested seven categories of majors, with the number of observations for

each group in parentheses: science, technology, engineering, or math (231);

economics, finance, or accounting (160); fine arts, music, drama, or performing

arts (38); environmental studies, forestry, or ecology (32); liberal arts, history, any

language, or philosophy (471); psychology, sociology, anthropology, or religious

studies (233); and political science, public policy, or government (428).

Among all college majors tested, only the Science, Technology, Engineering

and Math (STEM) and Economics, Accounting and Finance (EAF) majors proved

to have a significant effect on LGPA, and the effect was positive for both.

The

coefficients on STEM and EAF variables, 0.066 and 0.058 respectively, were

positive, similar in magnitude, and highly significant (at the 5% level). These

majors were akin to having an extra 4 and 3.5 LSAT points, respectively.

major predicted LGPA negatively: the closest was an Art, Music, or Drama major,

which was a negative, but only borderline-significant (at the 10% level), and only

for 1L GPA (Model 2) – and it was not at all significant as to cumulative LGPA

(Model 1).

The positive STEM result was especially surprising, because we had

hypothesized that while many STEM majors are more talented than their UGPAs

indicate, they tend to be less experienced or inclined toward reading and writing.

And we did find evidence these students may need time to grow along a "learning

curve" during 1L year. Comparing the Model 1 and Model 2 results, the STEM

and EAF coefficients are positive in Model 2 (1L GPA), but even more positive

and significant in Model 1 (LGPA). Thus, takes time for those with STEM and

EAF majors to reach their potential, but the finding remains that they outperform

others.

The reason STEM majors did not suffer due to lesser reading and writing

experience may be selection bias: we examined the performance of not a random

sample of STEM majors, but the modest subset who chose law school – likely

We coded seven categories of majors. The “Political Science/Government” major is excluded

from the statistical analysis, because running regressions requires excluding one "reference

group," and this group was large (428 students), and performed very close to average. We ran

two OLS regressions similar to Table 2, Models 1-2, this time using liberal arts as a reference

category, and the variable political science was again, positive and not significant.

The coefficient on STEM, noting the effect of having a STEM major, was 0.066; the number

of LSAT points (each of which has an effect of 0.0163) necessary to equal the effect of a STEM

major thus is 4.09. Similarly, the coefficient on EAF, noting the effect of having an EAF major,

was 0.0581; the number of LSAT points (each with an effect of 0.016) necessary to equal the

effect of an EAF major thus is 3.57.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 36

those most comfortable with reading and writing. Confirming that our STEM

population was no random sample is its gender breakdown: roughly 75% of STEM

majors are male,

yet our population's gender-STEM correlation was essentially

zero.

There are several possible explanations for the positive, significant effect of

STEM and EAF majors. First, such majors might either train or select for technical

or mathematical thinking that translates well to law study. For a major to be an

LGPA predictor, not all those with the major must be the same; it suffices if a

higher percent of such majors are suited to law than others are. However,

undercutting the theory that STEM and EAF thinking inherently translate well to

law school is the finding that STEM and EAF majors do not do as well 1L year as

they do later in law school: the coefficients on STEM and EAF majors were still

positive as predictors of 1L grades, but 10-45% lower in magnitude and not as

significant.

Thus, contrary to the view that STEM and EAF majors have

cognitive styles favorable for legal study, the evidence is that such majors face

some adjustment difficulty – implying that law school requires different skills,

such as more written and verbal work, and more disputed interpretations than the

sometimes black-and-white conclusions of science, engineering, accounting,

finance, and to a lesser extent economics.

A second reason STEM (but not EAF) may be a positive predictor is that

STEM courses often feature a lower grading curve, making a STEM major's 3.3

UGPA more impressive than a 3.3 in history; STEM courses typically give out

fewer As and more C (or lower) grades. Thus, among students with identical

UGPAs, the STEM majors show more potential – which may explain why STEM

is a somewhat larger plus than EAF, in which the grading curves typically are not

unusually tough.

A third reason STEM and EAF majors may be plusses is that they may have a

smaller percentage of students looking for an easy ("gut") major than, say, political

science or psychology.

This does not mean that STEM or EAF majors actually

are harder than any others: some political science departments, and especially their

top students, focus on statistical analysis as much as many economics majors do;

some psychology and environmental studies majors focus on not only statistical

Kelsey Sheehy, Colleges Work to Retain Women in STEM Majors, U.S. NEWS & WORLD REP.

(July 1, 2013), http://www.usnews.com/education/best-colleges/articles/2013/07/01/colleges-

work-to-retain-women-in-stem-majors ("Only about 25 percent of STEM degree holders are

women, due largely to a lack of female college students studying engineering, computer science

and physical sciences such as physics and chemistry.").

Specifically, the correlation coefficient between gender (male) and STEM major was 0.003.

The coefficients on STEM were 0.067 for cumulative LGPA (significant at the 1% level,

p=0.008) but 0.061 for 1L GPA (with far more marginal significance, only the 10% level,

p=0.057). The coefficients on EAF were 0.058 for cumulative LGPA (significant at the 5%

level, p=0.022) but 0.032 for 1L GPA (not significant, p=0.330).

We thank Jonathan Adler for this interpretation of the predictive value of various majors.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 37

analysis, but also biological science; and non-scientific/non-statistical academic

fields like history and English are in no way inherently easier. But some fraction of

college students look for easy majors because they are low on motivation, and such

students may be less likely to choose to major in physics, math, or perhaps

economics or finance. Even if such students are wrong in thinking courses in

another field will be easier: if non-STEM/EAF majors have a higher share of low-

motivation students that could explain why STEM/EAF majors eventually perform

better academically.

The second and third reasons – that STEM may feature tougher grading and

STEM and EAF may have a smaller share of low-motivation students – actually

support a broader point than a plus factor for STEM/EAF majors: (a) extra caution

may be warranted for applicants in any major with an unusually easy curriculum;

and (b) extra consideration may be warranted for applicants in any major with an

unusually rigorous curriculum.

A history or English major who took a heavy load

of upper-level courses and wrote a rigorous honor thesis may be every bit as

promising as a STEM major with a similar UGPA. More specifically, as noted

above, many non-STEM/EAF majors do scientific or statistical work nearly

indistinguishable from what STEM and EAF majors do. Yet far from all political

science, psychology, and environmental studies major so focus, and it is a

limitation of this study that we could not scrutinize students' transcripts to

distinguish which did so; transcripts feature far too little detail in course titles to

spot which courses are actually STEM/EAF-like.

Consequently, our results do

not indicate that a mathematical, statistical, or science-focused non-STEM/EAF

major is worse than a STEM/EAF major; to the contrary, the STEM/EAF plus

factor seems applicable to any other major with a similarly intensive mathematical,

statistical, or science focus.

One final caveat is that selecting a major is an important decision, and our

findings are not prescriptive advice that aspiring lawyers should choose STEM or

EAF majors. STEM, for example, may cease to be a positive predictor if liberal

arts students, en masse, switched to STEM majors. Choosing a major ill-suited to

one's interests or aptitude would seem a recipe for learning less, enjoying less

motivation, earning lower grades, and harming the academic confidence that

contributes to success. A material difference in UGPA, moreover, is a stronger

predictor than any major (the 0.3 UGPA difference between B and B+ is more

powerful than a STEM or EAF major), so choosing a major less suited to one's

interests or talents seems a poor strategic choice, in addition to a poor educational

choice.

We thank Professor Jennifer Hendricks, who (despite being a math major herself) provided

this point that the imprecise match between major and curricular difficulty requires a close look

at the undergraduate courses applicants choose, whatever their majors.

For example, the most statistics-heavy political science college courses one of the authors

(Moss) took was a seminar in "American Political Institutions"; that course name on his

transcript would not indicate that the course was as heavily quantitative as his economics major

courses.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 38

5. Work Duration: 4-9 Years ≈ 6.5 LSAT Points

Work duration was measured three different ways, only one of which was

positive and highly significant. The coefficient for 4-9 years of work was 0.109

(positive and significant at the 1% level), akin to 6.5 extra LSAT points.

Working

1-3 years and working 10+ years were both positive but not significant. It was

surprising that a "sweet spot" of 4-9 years' work proved better than having more or

fewer years. We lacked a firm ex ante hypothesis as to the optimal quantity of

work experience, the conventional wisdom in the admissions world was that work

experience has roughly the sort of nonlinear relationship with LGPA that we

found: while work experience is a plus, and more is better, too much is a negative.

To test whether increasing years of work experience had this sort of initially

increasing, but then decreasing, effect on LGPA, we ran a correlation matrix of

LGPA and each number of years of work experience (1, 2, 3, etc.). The

correlations showed a fairly clear break between 1-3 years, 4-9 years, and 10+

years: 4 years through 9 years each showed a fairly consistent positive correlation

with LGPA; yet there was no clear relationship (positive or negative) for 1-3 years

or for 10 or more years.

We offer a two-part likely explanation for 4-9 years' work experience being an

apparent sweet spot for law students. First, the difference between 1-3 and 4+

years likely reflects a maturity difference. Having work experience (compared to

starting law school right after college) either provides or selects for maturity, but 1-

3 years may not truly provide real-world experience. Someone in law school after

only one year of work was applying to law school that entire year; with 2-3 years

of work, the student still was applying or studying for the LSAT halfway through

those years, and probably planning to apply to law school from the start. Thus, 1-3

years of work is not enough to provide the experience of making one's way in the

world before law school; that length serves only as a waystation between college

and law school.

Second, the difference between up to 9 years and 10+ years likely reflects the

difficulty some longtime workers have readjusting to school. Those with 10+ years

include many with the best experience and maturity, but also many with trouble

readjusting to student life, which could explain why having that much work is, on

average, neither a positive nor a negative; it includes a mix of plusses and minuses.

To be sure, as with other nonlinear relationships we found, the bright lines in

our work experience dummy variables should not be relied upon too literally: some

people mature greatly with 2-3 years' work, while others do not mature with 4-5;

some with 7-8 years have trouble readjusting while some with 12-13 readjust

easily. The idea of a 4-9 year sweet spot is thus an oversimplification, but one that

we think reflects a reality, and one that comports with some conventional wisdom

in the law admissions world: work experience is a material plus factor, a proxy

The coefficient on having 4-9 years' work was 0.109; the number of LSAT points (each with

an effect of 0.0163) necessary to equal the effect of 4-9 years' work thus is 6.66.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 39

either for maturity or for having made an informed decision to take the plunge

back into student life; but just a few years of work is too little to make a difference,

and too many years risks making it difficult to readjust back to student life.

6. Work Type: Teaching ≈ 5 LSAT Pts.; Military ≈ -7

; Sci/Tech ≈ -3

Of the six categories of employment, two proved significant LGPA predictors:

teaching experience had a coefficient of 0.082, positive and significant only at the

10% level; military experience had a coefficient of -0.119, negative and significant

at the 5% level. Science and technology experience had a coefficient of -0.077, but

was significant only at the 10% level, and only in Model 2, the 1L GPA regression.

No other category was significant.

Teaching experience is akin to 5 extra LSAT points,

which likely reflects

personal qualities. Among jobs held in the early- to mid-twenties age of most

entering law students, teaching may be the one that most selects for – or develops –

the ability to be a responsible adult wielding authority and urging others to take

work seriously. Also, choosing a teaching career in one's early twenties likely

indicates comfort in a learning environment, which bodes well for meeting the

demands of law school. Thus, while teaching work may confer some benefit, more

likely it is that having selected a teaching job reveals a student to be of a type –

responsible and comfortable with classroom learning – likely to do well in law

school.

Military experience is akin to -7

LSAT points.

However, most law students

from the military had several years of service, placing them in the 4-9 years' work

category that is a countervailing plus of similar magnitude. The plus of lengthy

work and the minus of military work therefore roughly cancel out; i.e., 4-9 years in

the military is not materially better or worse than having no work experience at all.

The reason military work is essentially the opposite of teaching as a predictor

is likely because they select for different traits and backgrounds. As noted above,

those choosing teaching may adjust to three years of classroom lectures and

textbook reading easily. In contrast, more of those choosing the military may find

law school a difficult adjustment for two reasons. First, whereas teaching selects

for those comfortable with classroom learning, the military may select for

kinesthetic learners, providing learn-by-doing experience that makes the more

passive experience of law school a major adjustment. Second, military experience

may be a negative predictor as a proxy for low socioeconomic status. Pentagon

data show that the military "lean[s] heavily for recruits on economically depressed,

rural areas … , with nearly half coming from lower-middle-class to poor

households."

Those from less privileged socioeconomic backgrounds not only

The coefficient on teaching experience was 0.082; the number of LSAT points (each with an

effect of 0.016) necessary to equal the effect of 4-9 years' work thus is 5.02.

The coefficient on military experience was -0.119; the number of LSAT points (each with an

effect of 0.016) necessary to equal the effect of military experience thus is -7.32.

Ann Scott Tyson, Youths in Rural U.S. Are Drawn To Military, WASH. POST (Nov. 4, 2005),

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 40

may face a tougher adjustment to the culture and expectations of law school,

100

but

– especially following recent decades of rising tuition – are more likely to need to

divert time to paid work during law school, further negatively impacting their

grades.

101

Comparison of the 1L and cumulative LGPA results corroborates the

adjustment-difficulty theory of why military work predicts negatively. Military

work predicts a 0.118 lower cumulative LGPA, but a 0.231 lower 1L GPA; thus,

the effect on 1L LGPA is nearly double the effect on cumulative LGPA. Similarly,

scientific or technical work experience – which also might make for a difficult

adjustment to law school – is not a significant predictor of cumulative LGPA (it is

akin to -3 LSAT, but the correlation is not statistically significant

102

), yet is a

mildly significant negative predictor of 1L GPA. This corroborates that some jobs

may be negative predictors because they are so different from law study that law

school requires a major adjustment that many can make eventually (as shown by

the cumulative GPAs being better than the 1L GPAs), but many do not make (as

shown by the continued negative effect of military work after 1L year).

7. Negative Criminal/Disciplinary Record ≈ -7

LSAT Points

The coefficient on the variable for having a significant negative or criminal

record was -0.119, negative and significant at the 5% level; it was also negative

and significant in this magnitude in the 1L GPA regression. A negative record thus

http://www.washingtonpost.com/wp-dyn/content/article/2005/11/03/AR2005110302528.html

("[T]he military is leaning heavily for recruits on economically depressed, rural areas where

youths' need for jobs may outweigh the risks of going to war. … Many of today's recruits are

financially strapped, with nearly half coming from lower-middle-class to poor households,

according to new Pentagon data … . Nearly two-thirds of [2004] Army recruits … came from

counties in which … income is below the U.S. median").

100

Eli Wald et al., Looking Beyond Gender: Women’s Experiences at Law School, 48 TULSA L.

REV. 27, 45-49 (2012) (describing, from first-hand student account, how and why low-

socioeconomic status background led to poor grades and overall performance in law school).

101

Eli Wald, The Visibility of Socioeconomic Status and Class-Based Affirmative Action: A

Reply to Professor Sander, 88 DENV. U. L. REV. 861, 866-67 (2011) (noting that law school,

especially the first year, "involves reading significant volumes of case law. Sixty-, seventy-, and

even eighty-hour weeks are not unheard of, and a part-time or full-time job may put one at a

significant disadvantage," and thus, "the possible need of some students of lower socioeconomic

status to work either part-time or full-time while enrolled … may also constitute a significant

hurdle to one's academic success"). Cf. NALP FOUNDATION FOR LAW CAREER RESEARCH AND

EDUCATION (NALP) AND AMERICAN BAR FOUNDATION (ABF), AFTER THE JD: FIRST RESULTS

OF A NATIONAL STUDY OF LEGAL CAREERS (2004) (corroborating Wald's hypothesis that

students from lower incomes have more need to work, by reporting that students from more

affluent backgrounds graduate with less debt: "Individuals with no educational debt leaving law

school were more likely … to be white or Asian, and of higher socioeconomic status.").

102

The coefficient on scientific or technical experience was -0.0504446; the number of LSAT

points (each with an effect of 0.0163022) necessary to equal the effect of scientific or technical

experience thus is -3.09. But the coefficient was not statistically significant (p=0.121).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 41

appears to be is a significant negative, akin to almost -7

LSAT points.

103

This finding was somewhat surprising because the pool of law students with

negative records is a biased subsample of the population with such records. Law

schools reject those with the worst records, or those with the weakest explanations

of their records. Yet even this positive-biased sample of those with records

performed worse on average. Likely, the population with negative records is a

heterogeneous mix of some who are fine and some who lack necessary personal

qualities (discipline, self-control, drive, etc.) to succeed.

A notable caveat to this finding is that although most variables in this study

were objective numbers or binary conditions, two were highly subjective: deciding

what was a significant criminal or disciplinary record; and deciding what was a

major leadership role. A great many students have a modest negative record

(particularly common are drinking alcohol underage or marijuana possession), just

as a great many have some modest leadership experience (e.g., being an officer in

a small college club). Thus, we noted only major negative records or major

leadership roles, to avoid lumping into one yes-or-no binary variable all negative

records from public drinking to major felonies, or all leadership roles from

president of a bridge club to president of a student government. This need to

impose a threshold added subjectivity, however. We tried to limit that subjectivity

by giving guidance and on-site supervision to those entering data: (a) that "major

criminal or disciplinary record" means anything more than merely using a

controlled substance underage, or privately without any violence or selling of the

controlled substance; (b) that "major leadership role" means a high officer position

in a major organization (e.g., Treasurer of an entire college student government) or

being the top leader of multiple smaller organizations (e.g., president or captain of

a bridge club and a mock trial team); and (c) that one of the authors was in the

room for all data entry and should be consulted about any borderline cases -- to

maximize the extent to which the threshold of "major" was applied consistently,

even if with unavoidable subjectivity

8. Rising UGPA (If in Law School Right after College) ≈ 2 LSAT Points

The coefficient on a rising undergraduate UGPA was 0.053, positive and

significant at the 10% level in the 1L GPA regression only. This supports the

calculation that a UGPA rising by at least 0.3 by the end of college was a positive

predictor, akin to 2 LSAT points,

104

but with two caveats. First, rising UGPA did

not correlate with LGPA for those with work experience.

105

Second, rising UGPA

103

The coefficient on negative criminal or disciplinary record was -0.1190152; the number of

LSAT points (each with an effect of 0.0163022) necessary to equal the effect of scientific or

technical experience thus is -7.30.

104

The coefficient on having a rising UGPA, for those right out of college, was 0.032; the

number of LSAT points (each with an effect of 0.016) necessary to equal that effect thus is 2.01.

But, as noted below, the coefficient was not statistically significant (p=0.146).

105

More precisely, the dummy variable was the product of two other dummy variables: rising

UGPA (1=yes, 0=no) multiplies by no work experience (1=yes, 0=no). This result makes sense:

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 42

was not a statistically significant predictor of cumulative LGPA.

106

Like LSAT, a

rising UGPA predicts a higher 1L GPA more strongly than it predicted a higher

cumulative LGPA. Thus, having a rising GPA may be a plus, but an ephemeral

one, reflecting that those who did well late in college, then attended law school

right after, are performing above par to an extent not likely to persist.

9. Demographics: Person of Color Self-ID, -9 to -9½ LSAT Pts.

Any self-identification as a person of color – African-American, Latino/a,

Asian-American, or Native American – was a statistically significant negative

predictor of both LGPA and 1L GPA. The coefficients for African American,

Latino/a, Asian American and Native American categories were -0.155, -0.148,

-0.154, -0.173 respectively; all but Native American are significant at the 1% level,

and Native American is significant at the 5% level. However, even with a

combined dataset from two schools, the number of observations in the categories –

African American (59), Latino/a (45), Asian-American (142), and Native

American (15) – is relatively low.

107

A group of 15 is too small from which to

draw conclusions, and even 45 is relatively low.

Still, the magnitude of the racial disparity was substantial and relatively

consistent: each category of person of color self-identification was akin to -9 to -

9½ LSAT points.

108

In contrast, gender had no effect. This racial disparity is our

most challenging to interpret: we have only modest space to devote to each of our

many findings, yet racial disparity is an extraordinarily complex social

phenomenon. A full analysis of racial disparities – including relevant sub-issues

such as bias, affirmative action, alienation, stereotype threat, etc. – is far beyond

the scope of this paper; whole articles or books exist to analyze such topics. Still,

our findings hint that some explanations have more persuasive power than others.

Our finding provides evidence that racial disparities in law school performance

cannot be entirely the result of members of racial minorities being "mismatched" to

their schools due to affirmative action helping them gain admission with lesser

UGPA trajectory is recent information for those starting law school right after college, but not

for those whose college work was years ago. Thus, the only rising UGPA trait that correlated

with LGPA was an interactive term of those who had a rising GPA and were attending law

school right after college.

106

The coefficient was 0.034 with a p-value (0.126) near but not reaching the 10% level

marking modest significance.

107

See Appendix, Table 4 (listing all variables and summary statistics).

108

The coefficients on African-American, Latino/a, Asian-American, and Native American

were -0.155, -0.148, -0.154, and -0.172, respectively; the number of LSAT points (each with an

effect of 0.016) necessary to equal those effects thus are -9.53, -9.09, -9.47, and -10.61,

respectively. However, we do not place much weight on the coefficient for being Native

American because, as noted above, the sample size of that group was too low to allow any valid

conclusions, leaving us reporting mainly the other groups that predicted as akin to -9 to -9.5

LSAT points.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 43

credentials, as Richard Sander hypothesized.

109

We find racial disparities despite

controlling, better than prior studies do, for not only academic ability on

standardized tests (i.e., LSAT) and prior academic performance (i.e., UGPA), but

also a number of other variables relevant to academic credentials, such as college

quality, college major, and UGPA trajectory (all factors helping distinguish

between the predictive power of similar UGPAs), as well as various nonlinear

relationships LGPA has with college quality and UGPA.

110

To more closely examine whether a correlation between race and entering

credentials could explain the disparity, we re-ran the Model 1 regressions on two

subsets of the data: (a) just those with a bottom-quarter "index" (i.e., a linear

combination of LSAT and UGPA into one number); and (b) those with an LSAT-

UGPA in the first to third quarter. We found that, among African-Americans (but

not other people of color), having an index not in the bottom quarter more than

halved the disparity: the predicted LGPA impact was -0.207 for those with a

bottom-quarter index, but -0.093 for others. Thus, controlling as carefully as

possible for academic credentials lessens the disparity, but does not eliminate it.

Given that controlling as much as possible for low entering academic

credentials lessens the disparity only for African-Americans, and only by about

half, it seems likely that the racial disparity reflects something not merely about the

students, but about legal education itself – which may be unsurprising, given the

substantial literature on how people of color, and those with less privileged

socioeconomic backgrounds, can find law school alienating or a challenging

adjustment, to the detriment of their performance.

111

A full survey of the literature

on alienation, stereotype threat, and other similar phenomena is beyond the scope

of this paper – but such phenomena are well-documented and long-known. Lani

Guinier noted two decades ago, from survey and academic performance data, that

women, then a minority of law students, found law school a source of

"alienat[ion]" and "distress" – and performed worse in law school despite

credentials on par with those of men:

[W]e find strong academic differences between graduating men and

women. Despite identical entry-level credentials, this performance

differential … is created in the first year of law school and maintained

109

Richard H. Sander, A Systemic Analysis of Affirmative Action in American Law Schools, 57

STAN. L. REV. 367, 453-54 (2004) (arguing as to law school admission, and reviewing prior

literature so arguing as to undergraduate admission, that due to "large racial preferences,"

African-Americans often "go[] to a school where one’s academic credentials are well below

average[, which] has powerful effects on performance. … [S]uch a student is learning less than

she would have learned at a school where her credentials were closer to average.").

110

As with our other variables, we do not believe there is anything unique about the two schools

we studied. The racial disparity was significant at both, even though each features a national,

but relatively different, geographic population; each draws the majority of its students from

outside its own state, and both have many east coasters, but Colorado draws more heavily from

the west and Texas, while Case Western draws more from the Midwest and parts of the South.

111

See supra Part IV(B)(9).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 44

over the next three years. By the end of their first year … , men are three

times more likely than women to be in the top 10% of their law school

class.

112

If anything, it is surprising that we found only racial disparities, not the gender

disparities Guinier documented. Our findings thus evidence progress in eliminating

law school gender disparities, but not racial disparities – warranting further support

for struggling or alienated students, as we later discuss.

113

Unconscious bias is another possible explanation for the racial disparity. We

do not assume, and know no evidence of, systemic bias by many or most law

professors. Implicit bias has been shown to be pervasive in human cognition,

114

however, so it is always a possible explanation worth exploring for any racial

disparities. While most law school examinations are graded anonymously, bias still

can infect (a) the non-anonymous class participation plus-minus factors that can

make course grades differ from exam grades, and (b) the many classes are not

anonymously graded, such as seminars, clinics, and most skills courses. Because of

the modest size of the racial disparities we found – averaging about 0.15 in LGPA

– even episodic, limited bias could be enough to explain a material portion of the

disparities.

C. The Quarter Regressions (Models 3 and 4): What Predicts Especially Strong

or Weak Law School Performance?

Models 3 and 4 attempt to predict who lands in the top quarter ("Q1") or

bottom quarter ("Q4") of their law school classes. Since presence in a quarter is a

dichotomous variable, Models 3 and 4 use logistic regression to predict the odds

each student will be in the top or bottom quarter.

115

Table 7 in the Appendix

reports the findings of the Quarter Regressions as odds ratios.

116

Odds ratios are

used to compare the relative odds of the occurrence of a particular outcome. The

112

Lani Guinier et al., Becoming Gentlemen: Women's Experiences at One Ivy League Law

School, 143 U. PENN. L. REV. 1, 2-3 (1994) (finding the female minority at the University of

Pennsylvania Law School experienced "alienat[ion]" and "distress," based on academic

performance data from 981 students and self-reported survey data from 366 students).

113

See infra Part V (noting possible prescriptions for admissions reform).

114

Anthony G. Greenwald & Linda Hamilton Krieger, Implicit Bias: Scientific Foundations, 94

CAL. L. REV. 945, 955-56 (2006) (reporting various findings, such as that only 20% of survey

respondents displayed "explicit" bias but 64% displayed "implicit bias," and concluding that the

data "strongly suggest that any non-African American subgroup … will reveal high proportions

of persons showing statistically noticeable implicit race bias" against African-Americans).

115

Specifically, on the full data set, we regressed dichotomous dependent variables Q1 and Q4

(top- and bottom-quarter LGPA) on all independent variables; each independent variables'

coefficient thus estimates its effect on the logarithm of the odds of the dependent variable (i.e.,

presence in the quarter), adjusting for all other variables included in the model. In Stata, the

logistic command produces results in terms of odds ratios while logit produces results in terms

of coefficients scales in log odds.

116

Logistic results can be interpreted in one of two ways. A variable's coefficient is the "log

odds of the dependent variable," or the exponentiated coefficient is the "odds ratio.”

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 45

results can be interpreted as in the following example from the Table 7 (Q1)

regression: the odds ratio for having 4-9 years of work experience is 2.78, so the

odds of this student being in the top quarter are 178% greater when the student has

this work experience; in contrast, the odds ratio for having 10+ years’ work

experience is 0.69, meaning that the likelihood of this student being in the top

quarter decreases 31%, or 1-0.69. The odds ratios indicate the increased likelihood

(or decreased likelihood in the case of values under 1.00) of a certain effect; an

odds ratio of (or close to) 1.00 indicates no effect.

Most results were similar to the Model 1 LGPA results, as expected: if a factor

predicts law grades generally (Model 1), it also predicts whose grades are the best

(Model 3) or worst (Model 4). We lacked strong ex ante hypotheses as to what

predictors would differ from Model 1 to Models 3-4. We nevertheless thought it

important to examine whether any factors, apart from predicting grades generally

in Model 1, further predict who becomes (a) a Q1 high achiever likely to land top

jobs (e.g., clerkships, large firms, or elite public interest jobs), or (b) a Q4 low

achiever less likely to land quality jobs or pass a bar exam.

117

What is notable about Models 3-4 is where they either (a) found significant

predictive power in variables that were not significant in Model 1, or (b) helped

pinpoint whether a significant predictor in Model 1 (e.g., STEM) more strongly

predicted high success odds (i.e., Q1) or low odds of failure (i.e., Q4).

• Higher Odds of Q4, But Not Lower Odds of Q1: Military and

Science/Technology Work. We expected military work, a negative

Model 1 LGPA predictor, to predict being in the top or bottom quarter

of the class in terms of LGPA. Students with military work experience

are 209% more likely to be in the bottom quarter of the class (Q4). We

did not expect science/technology work (not a significant Model 1

LGPA predictor) to be positive and significant in the quarter

regressions. Yet students with science/technology work experience are

83% more likely to be in Q4. This supports the view that the reason

military work, and to an extent science/technology work, predicts

negatively is not that most have lower aptitude, but that some fraction

have difficulty adjusting – which is why the impact is higher odds of

Q4, not lower odds of Q1.

• Higher Odds of Q1: STEM, and EAF to lesser extent. Both majors

are similar-sized positive predictors of LGPA, yet STEM has a much

larger effect in predicting higher Q1 odds. STEM majors are 71%

more likely to be the Q1 compared to EAF majors who are 30% more

likely to be in the Q1. This partially supports the "hard curve" theory

117

LINDA F. WIGHTMAN, LAW SCHOOL ADMISSIONS COUNCIL, INC., LSAC NATIONAL

LONGITUDINAL BAR PASSAGE STUDY 23-24 (1998), http://www.unc.edu/edp/pdf/NLBPS.pdf

(concluding from empirical study that LGPA and LSAT were the two most significant

predictors of the odds of passing a bar examination, and in particular that LGPA correlated

more strongly than LSAT did with bar outcome).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 46

of why STEM predicts well: both STEM and EAF majors arguably

contain fewer weak students, but perhaps STEM has the tougher

grading curve, which may be why STEM majors have the higher

likelihood of being in the Q1.

• STEM Predicts Q1 While Sci/Tech Work Predicts Q4. There is

some inconsistency between STEM majors predicting higher Q1 odds

and science/technology work (which correlates with having a STEM

major) predicting higher Q4 odds. This supports the theory that certain

groups, like scientists, are high-variance populations: some are high

performers whose talents outstrip their LSAT/UGPA predictors; others

are low performers who never adjust to the differences between

science and law.

• Graduate Degrees and Rising GPA Predicts Lower Odds of Q4.

This relationship is similar for rising UGPA and graduate degrees

(both significant at the 10% level), but this is the only notable finding

as to graduate degrees. A graduate degree makes a student 32% less

likely to be in the Q4; a rising UGPA makes one 34% less likely to be

in the Q4. This hints that the import of rising UGPA is not that it

shows greater intellect, i.e., not that the student who rose from 3.3 to

3.7 is smarter than the one with a consistent 3.5. Rather, rising UGPA

shows a student learned to succeed academically; it may be on the

same logic that completing another graduate program indicates lower

odds a student will fail to perform in law school.

• Lower Odds of Q4: Male and Asian-American Students. These

results were contrary to the Model 1: while male students do not do

better overall (Model 1), they are 28% less likely to be in the Q4; and

while all nonwhite ethnicities do worse overall (Model 1), Asian-

Americans are 62% less likely to be in the Q4. The gender finding

may be evidence that while long-noted gender disparities have abated,

they are not fully gone; e.g., perhaps some professors are more likely

to "save" a weaker student from a low grade if he is male. The

inconsistent ethnicity findings, though, may be a mere statistical quirk,

given that the low sample sizes for these groups becomes even lower

when only a quarter of the dataset is in the regression (as in Models 3

and 4).

D. The "Splitters" Regression (Model 5): Which Is Better, High-UGPA/Low-LSAT

or the Reverse?

Because LSAT and UGPA both are powerful predictors of LGPA, a tradeoff of

one versus another, theoretically, could be a wash.

118

But law schools do not

118

The tradeoff between LSAT and UGPA with respect to first year law 1L GPA has been

studied extensively by LSAC, who find in their studies a correlation coefficient between LSAT

and first year law GPA to be 0.36 and between UGPA and first year law GPA to be 0.27. See

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 47

behave as if that were the case; high-LSAT/low-UGPA candidates are far more

likely to win admission and scholarship offers than low-LSAT/high-UGPA

candidates, as documented above.

119

Model 5 thus explores whether this strong law

school preference for high-LSAT over high-UGPA students is (a) a valid

preference reflecting the superiority of the former, or (b) a preference that is

misguided and/or a mere effort to boost the LSAT median that U.S. News over-

weights.

Like Model 1, Model 5 aims to predict LGPA from all independent variables,

adding two "splitter" profiles: high-LSAT/low-UGPA and high UGPA/low-LSAT.

The "mild splitters" regression examines students from both schools who had a

top-50% LSAT but bottom-50% UGPA and vice-versa.

120

The model includes a

dummy variable for students who fit the high-LSAT/low-UGPA profile, a

shortened list of predictor variables

121

, and an "index" variable combining LSAT

and UGPA, used to control for whether the splitter type has a higher LSAT-and-

UGPA average.

122

The key finding is that in predicting LGPA, high-LSAT and high-UGPA

splitter profiles are not equal. High-LSAT/low-UGPA splitters perform subpar,

controlling for all other variables, including the LSAT-UGPA index. The

coefficient for the high-LSAT splitters was -0.052, negative and significant at the

5% level. This means that high-LSAT/low-UGPA profile predicts lower LGPA,

compared to high-UGPA/low-LSAT splitters. Appendix Table 8 presents the

Anthony, Lisa A. et al., supra note 70. Our findings are identical to those in the 2013 LSAC

study, showing LSAT to be the stronger predictor of 1L GPA. We find that over time, the

LSAT loses its relative strength over UGPA as a predictor of LGPA. In our study, the

correlations between LSAT and LGPA, and UGPA and LGPA were 0.28 and 0.29, respectively

-- nearly the same.

119

Supra Part III(B)(2)(c).

120

The ‘mild splitters’ subset contains 733 students from both schools: 396 students had a top-

50% LSAT but bottom-50% UGPA, and 337 students had a top-50% UGPA but bottom-50%

LSAT. In the regression, a dummy variable was used for the top-50% LSAT but bottom-50%

UGPA profile (coded “1”). For robustness, we also ran an "extreme splitters" regression which

contained 192 students from both schools: 142 students had a top-25% LSAT but bottom-25%

GPA, and 80 students had a top-25% UGPA but bottom-25% LSAT. Again, a dummy variable

was used for the high-LSAT/low-UGPA profile. The low number of observations of extreme

splitters were too few to test many variables; nonetheless, we ran this OLS regression and did

not find any significance indicating a preference toward any extreme splitter category.

121

This regression with 733 variables does not include these predictors with fewer than 40

observations: African American, Latin American, Native American, 10+ years of work

experience, military work history, Art major, environmental sciences major.

122

The index variable equals LSAT+ (UGPA*10). We used the index in the splitter regressions

(instead of UGPA and LSAT) because the index was not highly correlated with the splitter

variable. The correlation between LSAT and the splitter variable was mildly high (r=0.40); the

correlation between UGPA and the splitter variable was very high (r=-0.73); the correlation

between the index and the splitter variable was low (r=-0.04).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 48

Model 5 OLS regression testing for the significance of the high-LSAT/low-UGPA

profile. Using 733 observations -- containing mild splitters (of both types) – this

regression tested the significance of the dummy variable for the high-LSAT/low-

UGPA profile.

If high-LSAT/low-UGPA splitters perform subpar compared to high-

UGPA/low-LSAT splitters using a subset of only mild splitters (733 observations),

a follow-up question to ask is how do high-UGPA/low-LSAT splitters perform

compared to non-splitters (1435 observations) as a whole? For robustness, we ran a

second OLS regression this time including all variables, the index in place of

LSAT and UGPA, and a dummy variable for the high-UGPA/low-LSAT splitters

group. The coefficient on the high-UGPA/low-LSAT splitter was 0.23, positive

and not significant, indicating that the high-UGPA/low-LSAT splitters did no

worse or better than non-splitters. To conserve space, we report these results here

and do not present them in a table format.

One caveat to this finding is that a high-LSAT/low-UGPA profile may still be

equal or superior to other profiles, because the result may trace to selection bias

discussed earlier in Section IV. As noted above, schools admit the vast majority of

high-LSAT/low-UGPA candidates, but a minority of low-LSAT/high-UGPA

candidates. By so liberally admitting high-LSAT splitters, schools may be

admitting some who are less likely to succeed – whereas by hand-picking among

high-UGPA splitters, schools are choosing more solid students. If schools admitted

high-UGPA splitters as liberally as they admit high-LSAT splitters, then the

former might suffer the lower average LGPA we see from the more

indiscriminately admitted high-LSAT splitters.

Even with this caveat, two notable findings remain. First, high-UGPA/low-

LSAT splitters, when chosen as carefully as is current practice, are no less

promising than those with a more balanced profile or a higher LSAT, so schools

need not fear dipping too low in LSAT for a candidate with a high UGPA or other

plusses. Second, the worse performance of high-LSAT/low-UGPA splitters

indicates that schools may too indiscriminately admit those with a high LSAT but

few other plusses.

E. The Variance Analysis: Examining LGPA Variance Based on Membership in

Various Groups

Finally, we examine the absolute variance of LGPA for each group defined by

a binary dummy variable, e.g., each group of majors, jobs, and splitters, and also

relative variances. Variances are reported on Table 9. If group X has higher

variance than group Y, then group X is a more heterogeneous mix of high and low

performers. That would indicate that group X is a high-risk/high-reward mix

warranting more individualized scrutiny of its members – both to try to spot the

extreme high-performers to admit eagerly, and the extreme low-performers to

avoid. Comparison of LGPA variance is most meaningful among groups of similar

sizes, because variance tends to decrease as sample size increases, so the following

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 49

are summaries of which groups have higher LGPA variance than others of similar

sizes.

• Military Experience. This was the one work group that was a negative

predictor, but the high variance (0.0046, compared to 0.0005-0.0029 for

other groups of similar size) shows it includes a wide mix of high and low

performers. This adds nuance to interpreting the negative coefficient: the

group does not predict uniformly negatively; it sees more bad than good

outcomes, but so much variance that good outcomes remain for a subset.

• Criminal/Disciplinary Record. This was the most negative predictor, but

its high variance (0.0018, compared to 0.0007-0.0013 for other groups of

similar size) supports interpreting this group, too, as a heterogeneous mix.

As with military experience: given a significant negative coefficient and

high variance on a binary dummy variable, the effect is not that all with a

negative record perform worse; rather, it is that some fraction do much

worse.

• Public Sector Experience. This group also had high variance (0.0017,

compared to 0.0007-0.0013 for other groups of similar size, and higher than

all other work categories),

123

corroborating a "gunners and meanderers"

interpretation: those with traditional pre-law backgrounds do average

overall, but feature a mix of (a) a few very high-performing "gunners"

unusually motivated to be lawyers, and (b) many "meanderers" with weak

motivation who attended law school as a path of least resistance for those

with their majors and work experience. On this view, those with traditional

law backgrounds perform average overall, but are a heterogeneous mix of

high- and low-motivation students deserving careful scrutiny.

Overall, the above high-variance groups (high relative to other groups

similarly sized) mark populations that may or may not successfully adjust to law

school: those with (a) military experience that may be especially different from law

study, (b) criminal/disciplinary records that may or may not hint at serious

problems, or (c) traditional pre-law backgrounds that include a mix of high

motivation for law study and low-motivation students who applied as a path of

least resistance. The heterogeneity of applicants from high-variance groups means

that, rather than paint with a broad brush in predicting their success or failure,

schools should carefully scrutinize such applicants for other indications that they

are more likely or less likely to succeed in law school, e.g.: a personal statement or

resume items making a persuasive case for high motivation for law study; for

splitters, high or low writing quality, or unusually strong academic

recommendations, could break the tie between dueling academic predictors such as

a high UGPA and a low LSAT (or vice-versa).

123

LGPA variance was fairly consistently at or near 0.0010 for all other work categories:

business (0.0010); teaching (0.0013); science, technology, or medicine (0.0009); and legal

(0.0009).

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 50

Finally, and in contrast, following are groups that we hypothesized might be

high-variance mixes of high and low achievers – but that ultimately did not feature

higher LGPA variance than other similarly sized subsamples.

• Splitters. We hypothesized that high-LSAT splitters are risky holders of

unfulfilled potential, or that both splitter types might show high variance,

because an LSAT-UGPA gap hints at a wide range of outcomes. But both

splitter types had LGPA variances on par with other similar-sized groups

(work types, majors, etc.): the splitters' variances were 0.0007-0.0011,

compared to 0.0009-0.0013 for other groups. Thus, there is no reason to be

more skeptical of a splitter than a candidate with more UGPA-LSAT

balance; a higher UGPA balances a low LSAT, and vice-versa, without any

penalty or extra unpredictability for an unbalanced splitter profile.

• Longer Work Experience. We hypothesized that those with especially

long work experience, even if not worse overall, are a riskier mix of mature

second-career aspirants and those who might find it too difficult to re-enter

academia. But those with 4-9 years or 10+ years of work experience had no

greater variance than other similar-sized subgroups (work types, majors,

etc.). Accordingly, there is no evidence supporting extra skepticism of

those long removed from college due to lengthy work experience.

F. Notable Non-Findings: Variables with Little or No Relationship to LGPA,

Contrary to Our Hypotheses or Common Assumptions

Earlier sections detailed all findings as to all variables that proved significant

predictors, positive or negative, of LGPA. This subpart, in contrast, details

variables that did not prove significant LGPA predictors. We report these non-

findings for the same reason tested these variables in the first place: we had

hypothesized, and/or prevailing admissions practices have assumed, that they

might help predict LGPA.

1. Nontraditional Pre-Law Majors: Not a Negative

One hypothesis was a negative effect on LGPA of various nontraditional pre-

law majors: performing arts (e.g., art, music, and drama); environmental studies

(which included related, more specific majors, such as forestry); and STEM

majors. These three groups cover all majors other than the more traditional pre-law

majors: political science, any other social sciences, and any liberal arts subjects.

STEM was a subject of dueling hypotheses – perhaps they are elite majors, or

perhaps they are too foreign to law study – and the findings in Table 2, Models 1-

2, show that the coefficient for STEM is 0.061, positive and significant at the 10%

level for 1L GPA, and 0.066, positive and significant at the 5% level, for LGPA. It

is slightly larger and more significant coefficient in the LGPA regression

presumably because STEM majors need time to adjust. The other two groups of

nontraditional pre-law majors – performing arts and environmental studies – were

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 51

hypothesized to be negative predictors.

Yet neither arts- nor environment-related majors had any significant

relationship with LGPA. Either students with such majors are just as prepared as

others for law study, or there is a selection bias: relatively few such majors attend

law school (there were 70 arts-related and environmental-subject majors, roughly

5% of the sample), so perhaps the few performing arts or environmental majors

who choose law school are those with more preparation or aptitude for legal study.

Whatever the explanation, there appears to be no basis for extra skepticism for

nontraditional pre-law majors – though difficulty of curriculum may remain

relevant, because it may be one explanation of why STEM majors perform above-

par.

2. Traditional Pre-Law and Reading-Heavy Majors: Not a Positive

Law school classes are reading-intensive, and most grading is of prose essay-

and paper-writing, so we hypothesized that LGPA would positively correlate with

majors that do more reading and writing, such as political science, liberal arts (e.g.,

history or English), or social sciences (e.g., psychology, sociology, or

anthropology). Yet no such majors correlated significantly with LGPA.

124

Modest support for the reading-as-preparation hypothesis did, however, appear

in how some variables more negatively predict 1L than cumulative LGPA: military

or technical work; and STEM or EAF major. That such students needed time to

reach their potential hints that the absence of recent reading or writing experience

(e.g., working in technical or military jobs less likely to entail reading and writing)

is more important than subtle differences among majors in reading and writing

content.

3. Traditional Pre-Law Work (Legal and Public Sector): Not a Positive

We hypothesized, and it is a common assumption in law admissions, that the

sort of quasi-legal work available before law school (paralegal, caseworker, etc.) is

a positive predictor of law school success, for various reasons: it could provide

training in legal study that gives a leg up, at least during 1L year; it could be a

proxy for high motivation to be a lawyer; or it could provide exposure to the

unglamorous side of legal work, making those who still forge ahead with law

school less likely to get disillusioned later (e.g., a former paralegal is not going to

be shocked that law study is more about paperwork than about being a spellbinding

courtroom orator).

Legal work was not a significant predictor of LGPA in any model. This

undercuts the above hypotheses; perhaps it also indicates that, thanks to bans on

124

The Political Science/Government major is the reference category and dropped in the Model

1 regression. If we re-run Model 1 and intentionally drop a different major (environmental

science), the Political Science/Government major has positive coefficient but it is far from

significant, therefore it does not demonstrate a statistical and reportable relationship with

LGPA.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 52

unauthorized practice of law, legal work before law school is likely low in

responsibility and substance, and thus a less impressive experience, than many

teaching, engineering, computer programming, or other jobs.

4. Prior Graduate Degrees: Not a Positive

The one modest predictive effect of a prior graduate degree is lower odds of a

Q4 LGPA – but this was a modest effect (significant at only the 10% level), and

overall, prior graduate degree had no overall correlation with LGPA. We were

surprised prior graduate degrees were not predictors of LGPA, as markers of either

higher academic ability, success at graduate-level work, or passion for academics.

There are three possible reasons for this lack of a provable relationship

between prior graduate work and LGPA. First, the vast majority of other graduate

degrees held by law students are master's degrees, so our finding is mainly that

master’s-level work is non-predictive; PhDs may well be predictive but are too rare

for a useful sample size, even in a two-school, four-year sample.

Second, master's degrees are quite heterogeneous; perhaps an M.B.A., an

engineering master's, a teaching master's, and a social work master's predict

differently. But, again, the sample sizes were not high enough to divide master's

degrees into multiple categories.

Third, even if a subset of graduate degrees may be a plus, that subset may

correlate with other positive variables. For example, scientific graduate degrees

may be a positive, but those with such degrees typically had STEM majors as well,

which itself is a positive significant predictor.

In sum, it remains possible that a subset of graduate degrees may be a positive,

but graduate degrees are too heterogeneous to so prove. Still, our findings undercut

any conventional wisdom that simply having a master's degree is a plus by itself.

5. Major Leadership Roles in College: Not a Positive

The leadership roles students often pursue, and view as resume-builders, were

not a significant predictor of LGPA. This finding comes with two major caveats.

First, the definition of a "major" leadership role is subjective. That subjectivity was

unavoidable and, as discussed in Part IV(B)(7) above (the section on the similarly

subjective variable for major criminal/disciplinary record) was mitigated by

various efforts to define the term and provide consistent review by the authors.

The second caveat to this finding – and potentially to other of the above

findings – is that leadership and other qualities not predicting academic success

might, nevertheless, predict later success, in either getting a job or performing well

as a lawyer. Future work based on this Article's data set will explore this

possibility.

V. PRESCRIPTIONS: BRIEF NOTES ON POSSIBLE REFORMS TO HOW

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 53

SCHOOLS ADMIT AND PREPARE STUDENTS

How to reform law schools – both who should go to law school and what law

schools should do differently – is a vast literature far beyond the scope of this one

section of a primarily empirical Article. This Article's findings, though, do provide

new evidence supporting some reforms and undercutting others. Readers likely

will draw their own conclusions as to what prescriptions they might support or

oppose more based on these findings, which is as it should be: empirical studies do

not produce prescriptions by themselves; they simply provide evidence that,

ideally, helps inform decisions about prescriptions. Because this section cannot do

justice to the complex topic of assessing and reforming legal education, following

is simply a brief discussion of three implications of this Article's findings that, in

the authors' views, support or undercut various practices and proposed reforms of

law schools.

A. Holistic Review, Given that No One Score, Credential, or Experience Possibly

Can Predict Success or Failure by Itself

A key overall lesson of all the above findings is the need for a broadly holistic

review of all applications – because no one variable, alone, is powerful enough to

justify admitting or denying a particular applicant. Thus LSAT or UGPA "cutoffs"

are ill-advised, even though those are two of the more powerful significant

predictors of LGPA. Our dataset includes students who vary widely in LSAT and

UGPA, because it combines four years of students from two schools with different

LSAT and UGPA profiles. Even within that dataset, however, the seemingly large

13-point difference between 10th and 90th percentile LSAT (153 to 166) predicts

only a 0.21 difference in LGPA. Among the binary group-membership variables

(majors, work experiences, ethnicities, negative records, etc.), the largest plus and

minus factors were akin to 6-10 LSAT points, meaning only a 0.10 to 0.16

difference in LGPA.

With almost no variable capable of predicting much more than one or two

tenths of a point of difference in LGPA, treating any one applicant credential as

dispositive is clearly a mistake. An applicant can make up for even a dozen fewer

LSAT points with a high UGPA alone, or with some mix of other plusses, such as

a positive-predicting major, work type, and duration of work experience.

B. The Heterogeneity of Candidates with Similar Backgrounds: The Need to

Distinguish Apples from Slightly Different Apples

While no one factor is dispositive, law schools do have to make their best

guesses as to who will and will not thrive in law school, and several factors are

material plusses or minuses. But other findings show real heterogeneity among

even high-performing groups: military experience predicts negatively, but with

unusually high variance; STEM predicts positively but science or technology work

experience predicts heightened risk of bottom-quarter LGPA. The hypothesized

explanations for these positive and negative predictors hint at how to distinguish

among high-variance population, such as military and science candidates.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 54

As to military: because military experience predicts worst for 1L year, and

likely derives in part from the difficulty some have adjusting to the more sedentary

law student life, law schools could favor those military veterans (a) who already

have shown academic success, e.g., favor those with high-UGPA/low-LSAT over

the reverse, or (b) who, unintuitively, held more sedate "desk jobs" in the military,

such as intelligence analysts, paralegals in the Judge Advocate General's ("JAG")

Corps, or those who worked on matters such as budgets and legal regulations.

As to those with science backgrounds: STEM majors' strengths (succeeding in

courses with hard curves, etc.) are not discernibly counteracted by weaknesses

from what such majors lack (e.g., less reading-and-writing experience, and less of

the pro-and-con dueling interpretations work that liberal arts or social science

majors do), likely because the subset of STEM majors applying to law school is

skewed (as shown by its nearly 50/50 gender split) toward those most comfortable

with verbal work and grey-area interpretations. On the other hand, those with

science work experience overpopulate the bottom quarter of LGPA, and while

STEM majors do well in both their first year and cumulative LGPAs, our results

suggest that they take time to develop their legal skills. According to Table 2,

Models 1-2, while STEM is significant and positive for both 1L GPA and LGPA

results, in the 1L GPA regression, the coefficient for STEM is 0.061, positive and

significant at the 10% level, and in the LGPA regression, it is stronger and more

significant -- 0.066 and significant at the 5% level. In evaluating those with science

or technology backgrounds, law schools should scrutinize for skills useful to legal

study that science training might under-provide: writing ability (as shown by the

personal statement and LSAT unedited essay); performance in classes entailing

reading and writing; and recommenders' statements, if any, about the applicant's

verbal or writing skills.

More generally, the various positive or negative predictors should not be

overinterpreted, because many are proxies for personal qualities, like maturity, that

a particular candidate may or may not actually have. Teaching experience (a

positive predictor) is best interpreted as a proxy for maturity and/or comfort with

classroom learning, while negative criminal or disciplinary record (a negative

predictor) is best interpreted as a proxy for immaturity or inability to handle

institutional rules. But some with teaching experience show other signs of

immaturity (e.g., a shallow or self-aggrandizing personal statement) or discomfort

with learning (e.g., a middling-to-weak UGPA), while some with negative records

show other signs of maturity and ability to play by the rules (e.g., the passage of

years since the negative record, or earning promotions in jobs they held for years

and from which they received strong recommendation letters attesting to their

maturity and responsibility).

In short, the significance of variables implies that certain qualities are plusses

and minuses only on average, not for everyone; we examined the data in other

ways (e.g., for variance, or for top- and bottom-quarter odds) for hints of how each

predictor might be a proxy for more fundamental qualities (maturity, etc.) that

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 55

careful scrutiny of applications can assess more fully.

C. Helping Students Adjust – and Expanding the Talent Base by Doing So

This Article's findings support reform beyond simply making better admission

decisions – such as reforms aimed at improving incoming students' adjustments to

law school. As noted above, many of the positive and negative predictors reflect

not pure talent level, but also (or instead) how well and how quickly various

student types adjust to law school: some, like STEM or EAF majors, perform

above-par but not as well 1L year; others, like those with military experience or

people of color, perform well below par 1L year, which could yield

discouragement that explains their less negative, but still below-par, cumulative

LGPAs; still others, like those with teaching experience, perform above-par due

possibly to their greater recent familiarity or comfort with the classroom setting.

To the extent that some students do worse not simply because of lesser talent,

but because they have more of an adjustment to make, that supports improved

early interventions to speed students' adjustment to the demands and culture of law

school. Improved interventions would increase the fairness and accuracy of law

school grades: if two students are equally talented, then the one with an academic,

work, or cultural background less on-point for law school might fall behind 1L

year; that falling behind would then leave LGPA inaccurately implying that this

student is inferior in talent or lawyering potential to the equally talented student

who simply had a more on-point background. Improved interventions therefore

could help a law school admit students who project less positively, but could

perform better if the school adopts effective interventions to speed their

adjustment.

In this light, improved interventions could help a school find more talent, by

letting it admit those who have weaker predictors, but who also have potential to

improve with the right adjustment help. Some schools do have various such

programs: spring semester 1L remedial courses for those who under-performed in

their 1L year or fall semester, taught by legal writing faculty or by a professor with

a dedicated role of providing additional support for student writing and legal

analysis;

125

and/or pre-1L summer courses that either offer remediation for

incoming students with low numerical predictors, or offer an opportunity for

waitlisted candidates with low predictors to show their ability to perform in law

125

See, e.g., Legal Writing Faculty – Amy Griffin, UNIV. OF COLO. L. SCH.,

http://lawweb.colorado.edu/profiles/profile.jsp?id=504 (last visited Feb. 26, 2015) ("Amy

Griffin … [is] the law school's first Student Legal Writing Engagement Coordinator. Colorado

Law added this new position to ensure that second- and third- year students continue to have

access to comprehensive one-on-one legal writing support. Thus, in addition to teaching an

advanced legal writing course, Amy works individually with students to continue the

development of their legal writing skills throughout law school[,] … [on] law journal notes,

seminar papers, independent research projects, externship assignments, and writing in the

clinics.").

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 56

classes.

126

This Article provides evidence that such programs hold promise not

only to increase the fairness of law school grading, but also to increase law

schools' strategic ability to admit those who have lower predictors yet display

potential – based on their work ethic, positivity, growth mindset, etc. – to

overcome obstacles like facing a difficult adjustment, if given proper support.

VI. CONCLUSION

This Article's findings confirm certain longstanding law school admissions

criteria, but call others into question, and support enhanced consideration of other

criteria not traditionally given as much (or any) weight. While data-driven

decision-making has entered the mainstream, it also faces pushback, raising

concerns about treating people as numbers rather than holistically. This Article's

findings, however, provide strong support for a more rather than less holistic

approach, and a less rather than more numbers-driven approach, to law admissions.

For example: LSAT is over-weighted compared to other, less univariate academic

metrics such as a broad view of not only UGPA but college quality and college

major; work experience truly is the positive that many believe it to be, with work

in teaching especially positive; certain backgrounds make for quicker or slower

adjustment to law study; and various markers of personal qualities – maturity,

work ethic, and motivation – truly are significant positives or negatives. One novel

aspect of this study is the way that it presents the key results in two ways. Like

most traditional empirical studies, the results are presented using regression

coefficients and degrees of significance; but also, the results are presented in

comparison to LSAT points, to provide more intuitive explanations to non-

empirical audiences.

That significant findings and take-home lessons for law student selection

resulted from this Article's data-gathering supports further such studies. Further

work can assess, for example, what qualities, both preceding and during law

school, predict which law students will earn full-time jobs, higher-paying jobs, and

bar passage. The increased maintenance in electronic form of law applicant data,

law school grades, and law student employment data can facilitate such work, but

with effort still required to code the data not maintained in any electronic form

126

"Some law schools offer programs where admission is contingent upon the successful

completion of a pre-enrollment program" just before 1L year starts. Law School Admission

Council, Conditional Admission Programs, LAW SCHOOL ADMISSION COUNCIL (June 12, 2014),

http://www.lsac.org/jd/diversity-in-law-school/racial-ethnic-minority-applicants/conditional-

admission-programs (listing 23 such programs); e.g., NSU Law Professor Receives Patent for

an Alternative Admission Model Program for Legal Education, Nova Southeastern Univ. L. Ctr.

(May 27, 2014), http://nsunews.nova.edu/nsu-law-professor-receives-patent-for-an-alternative-

admission-model-program-for-legal-education ("AAMPLE®, the Alternative Admissions

Model Program[,] … [is] an additional method of identifying candidates for admission ….

[A]pplicants are enrolled in two [courses] …. replicat[ing] an appropriate portion of an

equivalent regular J.D. offering …. The primary purpose … [is] evaluating the capabilities of

prospective students.").

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 57

(e.g., items on students' resumes), to code data maintained electronically in textual

form (e.g., law students' courses and activities), and to merge disparate databases

(e.g., in admissions, registrar, and career services offices). Law schools may be

understandably reluctant to devote substantial staffing resources to such efforts, to

let researchers who are strangers to the school access confidential data

(applications, grades, disciplinary problems, etc.), or both. Such entirely valid

concerns are why, to obtain a dataset of two schools, the authors had to ask eleven

schools to join this study; nine schools other than Colorado and Case Western

declined. Given that this Article offers findings law schools may find useful, the

data-gathering, coding, and statistical analysis effort seems a worthwhile use of

school staffing resources and researcher effort. Thankfully, the data-gathering and

coding effort required for this Article produced a data set that will allow further

analyses and publications as to employment and bar examination outcomes in the

future.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 58

Appendix

Table 4: Summary Statistics for Indicator Variables

Indicator Variables

As Percent of

Dataset

Ethnicity

African American

Latino/a

Asian American

142

10%

Native American

Employment duration

1-3 years

409

28%

4-9 years

112

10+ years

Employment type

Teaching

Legal

100

Business

111

Technology

124

Military

Public Service

College major

Science, Tech., Engineering, Math (STEM)

237

16%

Economics, Accounting, Finance (EAF)

166

12%

Psychology, Sociology, Anthropology

233

16%

Art, Music, Drama

Environmental Sciences

Liberal Arts, History

472

33%

Other factors

No work experience & rising college GPA

252

18%

Criminal history

Graduate degree

185

13%

University of Colorado Law Student

571

40%

College leadership

118

Gender male

797

55%

NOTE: Summary statistics of indicator variables – the number of observations in each sample

and the relative percent in the dataset.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 59

Table 5: OLS Regression: Dependent Variable is LGPA

NOTES: Absolute value of z-statistics in parentheses. +p<0.10; ** p<0.05; ***p<0.01.

(1a) (1b) (1c) (1d) (1e) (1f)

Traditional Factors

Law School Admissions Test (LSAT)

0.014*** (8.51) 0.011*** (6.54) 0.012*** (6.77) 0.011***(6.53) 0.010*** (6.10) 0.016*** (9.31)

Adjusted LSAT College Median (LCM)

0.003*** (3.39) 0.004*** (3.58) 0.003*** (3.09) 0.003** (3.11) 0.003** (3.05) 0.003*** (3.55)

Adjusted Undergraduate GPA (UGPA)

0.215*** (10.34) 0.191*** (9.36) 0.191*** (9.33) 0.191*** (9.30) 0.199*** (9.64) 0.272*** (12.44)

Ethnicity

African American

-0.216*** (5.12) -0.208*** (4.92) -0.204*** (4.81) -0.204*** (4.83) -0.155*** (3.77)

Latino/a

-0.251*** (5.48) -0.251*** (5.48) -0.248*** (5.40) -0.244*** (5.33) -0.148*** (3.29)

Asian American

-0.161*** (5.89) -0.162*** (5.90) -0157*** (5.71) -0.161*** (5.86) -0.154*** (5.81)

Native American

-0.295*** (3.78) -0.289*** (3.70) -0.288*** (3.69) -0.290*** (3.71) -0.173** (2.28)

Employment duration

1-3 years

-0.026 (1.40) -0.030 (1.43) -0.020 (1.35) 0.032 (1.47)

4-9 years

-011 (0.36) -0.010 (0.26) 0.004 (0.11) 0.109** (2.88)

10+ years

-0.128** (2.39) -0.142** (2.45) -0.136** (2.34) 0.014 (0.25)

Employment type

Teaching

0.086** (2.26) 0.084** (2.22) 0.082+ (2.20)

Legal

-0.004 (0.12) -0.001 (0.03) 0.022 (0.69)

Business

-0.023 (0.69) -0.034 (1.04) -0.023 (0.75)

Technology

0.009 (0.31) -0.027 (0.81) -0.05 (1.55)

Military

-0.091+ (1.66) -0.097+ (1.78) -0.119+ (2.25)

Public Service

0.037 (0.96) 0.038 (1.00) 0.043 (1.17)

College major

Science, Tech., Engineering, Math (STEM)

0.081** (3.15) 0.066** (2.65)

Economics, Accounting, Finance (EAF)

0.062** (2.36) 0.058** (2.30)

Psychology, Sociology, Anthropology

0.003 (0.14) -0.006 (0.30)

Art, Music, Drama

-0.015 (0.32) -0.038 (0.80)

Environmental Sciences

-0.043 (0.78) 0.022 (0.42)

Liberal Arts, History

0.018 (1.00) -0.001 (0.08)

Other factors

No work experience & rising college GPA

0.033 (1.45)

Criminal history

-0.119** (3.39)

Graduate degree

0.030 (1.22)

University of Colorado Law student

-0.209*** (10.12)

College leadership

0.018 (0.67)

Gender male

0.014 (0.89)

Constant -0.317 (1.30) 0.302 (1.06) 0.259 (0.90) 0.313 (1.08) 0.380** (1.31) -0.821** (2.70)

Adjusted R

0.15 0.19 0.20 0.20 0.20 0.26

Observations 1419 1419 1419 1419 1419 1419

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 60

Table 6: OLS Regression: Dependent Variable is 1L GPA

NOTES: Absolute value of z-statistics in parentheses. +p<0.10; ** p<0.05; ***p<0.01.

(2a) (2b) (2c) (2d) (2e) (2f)

Traditional Factors

Law School Admissions Test (LSAT)

0.028*** (12.73) 0.024*** (10.62) 0.025*** (10.77) 0.024*** (10.48) 0.024*** (10.18) 0.030*** (12.63)

Adjusted LSAT College Median (LCM)

0.004** (2.78) 0.004** (2.90) 0.003*** (2.57) 0.003** (2.58) 0.003*** (2.52) 0.004** (2.98)

Adjusted Undergraduate GPA (UGPA)

0.262*** (9.66) 0.235*** (8.68) 0.233*** (8.62) 0.233** (8.60) 0.240*** (8.78) 0.328*** (11.22)

Ethnicity

African American

-0.254*** (4.70) -0.244*** (4.48) -0.240*** (4.42) -0.241*** (4.43) -0.170** (3.35)

Latino/a

-0.267*** (4.54) -0.263*** (4.47) -0.260*** (4.42) -0.258*** (4.39) -0.148** (2.52)

Asian American

-0.137*** (3.88) -0.138*** (3.91) -0.134*** (3.77) -0.137*** (3.85) -0.130*** (3.77)

Native American

-0.308*** (3.20) -0.302** (3.13) -0.310** (3.21) -0.318** (3.28) -0.188** (1.97)

Employment duration

1-3 years

-0.040+ (1.73) -0.039 (1.46) -0.035 (1.30) 0.032 (1.16)

4-9 years

-0.036 (0.94) 0.013 (0.28) 0.007 (0.16) 0.110** (2.49)

10+ years

-0.096 (1.44) -0.090 (1.25) -0.085 (1.18) 0.081 (1.11)

Employment type

Teaching

0.090+ (1.88) 0.084+ (1.74) 0.086+ (1.80)

Legal

-0.012 (0.29) -0.01 (0.23) 0.015 (0.35)

Business

-0.030 (0.69) -0.036 (0.86) -0.025 (0.61)

Technology

-0.019 (0.49) -0.05 (1.18) -0.077+ (1.85)

Military

-0.198** (2.88) -0.206** (2.99) -0.231** (3.43)

Public Service

0.065 (1.33) 0.062 (1.27) 0.068 (1.44)

College major

Science, Tech., Engineering, Math (STEM)

0.076** (2.30) 0.061+ (1.90)

Economics, Accounting, Finance (EAF)

0.036 (1.07) 0.032 (0.97)

Psychology, Sociology, Anthropology

0.019 (0.66) 0.011 (0.38)

Art, Music, Drama

-0.051 (0.77) -0.084+ (1.33)

Environmental Sciences

-0.054 (0.76) 0.012 (0.17)

Liberal Arts, History

0.037 (1.55) 0.016 (0.70)

Other factors

No work experience & rising college GPA

0.053+ (1.82)

Criminal history

-0.137** (2.99)

Graduate degree

0.037 (1.16)

University of Colorado Law student

-0.225*** (8.33)

College leadership

0.019 (0.51)

Gender male

0.015 (0.72)

Constant -2.885*** (7.57) -2.090*** (5.34) -2.190*** (5.52) -2.080*** (5.26) -2.041*** (5.12) -3.470*** (8.21)

Adjusted R

0.20 0.23 0.23 0.23 0.23 0.28

Observations 1317

1317

1317 1317 1317 1317

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 61

Table 7: Model 3 and 4 Results, Logistic Regression, Dependent Variable is Having an LGPA in either the

Top (Q1) or Bottom (Q4) Quarter of the Class

Model 3

Model 4

Odds Ratio of Being in

Top Quarter (Q1)

Odds Ratio of Being in the

Bottom Quarter (Q4)

Traditional factors

Adjusted LSAT College Median (LCM)

1.04*** (3.31)

0.96*** (3.48)

Adjusted Undergraduate GPA (UGPA)

6.80*** (8.86)

0.17*** (8.86)

LSAT

1.12*** (6.62)

0.921*** (5.28)

Ethnicity

African American

0.45 (1.39)

3.62*** (3.83)

Latino/a

0.23+ (1.99)

1.86+ (1.83)

Asian American

0.38*** (3.31)

Native American

1.34 (0.41)

2.76+ (1.76)

Employment duration

1-3 years

1.30 (1.40)

0.673** (2.02)

4-9 years

2.78*** (3.23)

0.395*** (2.66)

10+ years

0.69 (0.63)

0.85 (0.32)

Employment type

Teaching

1.27 (0.78)

0.58 (1.42)

Legal

0.78 (0.82)

0.78 (0.86)

Business

0.77 (0.89)

0.90 (0.36)

Technology

0.71 (1.20)

1.83** (2.13)

Military

0.67 (0.79)

3.09** (2.52)

Public Service

1.48 (1.31)

1.27 (0.72)

College major

Science, Tech., Engineering, Math (STEM)

1.71** (2.53)

0.761 (1.22)

Economics, Accounting, Finance (EAF)

1.30** (1.21)

0.84 (0.75)

Psychology, Sociology, Anthropology

1.19 (0.93)

1.20 (0.97)

Art, Music, Drama

1.08 (0.21)

Environmental Sciences

1.27 (0.59)

1.24 (0.47)

Liberal Arts, History

1.26 (1.48)

1.29 (1.61)

Other factors

No work experience & rising college GPA

1.33 (1.47)

0.66+ (1.67)

Criminal history

0.44** (2.15)

1.96** (2.46)

Graduate degree

1.16 (0.69)

0.68+(1.67)

University of Colorado Law Student

0.27*** (6.97)

3.47*** (6.67)

College leadership

1.07 (0.30)

0.85 (0.66)

Gender male

1.13 (0.91)

0.72** (2.3)

Adjusted R

0.15

0.16

Observations

1419

NOTES: Absolute value of z-statistics in parentheses. +p<0.10; ** p<0.05; ***p<0.01.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 62

Table 8: Model 5 Results, OLS Regression using only "Splitters" (High-LSAT

and Low-GPA or Vice-Versa). Dependent Variable is LGPA

Model 5

Traditional factors

Adjusted LSAT College Median (LCM)

0.005*** (3.32)

Index

0.016*** (7.00)

Splitter category

Top 50% LSAT, bottom 50% GPA

-0.052** (2.21)

Ethnicity

Asian American

-0.176*** (5.35)

Employment duration

1-3 years

-0.017***(5.35)

4-9 years

0.106** (2.27)

Employment type

Teaching

0.079+ (1.65)

Legal

-0.005 (0.12)

Business

-0.077 + (1.81)

Technology

-0.116** (2.71)

Public Service

0.052 (1.05)

College major

Science, Tech., Engineering, Math (STEM)

0.072** (2.18)

Economics, Accounting, Finance

0.037 (1.07)

Psychology, Sociology, Anthropology

0.030 (0.97)

Liberal Arts, History

-0.011 (0.48)

Other factors

No work Experience & rising college GPA

-0.003 (0.11)

Criminal history

-0.066 (1.47)

Graduate degree

0.067+ (1.88)

University of Colorado Law Student

-0.173*** (5.82)

College leadership

0.026 (0.67)

Gender – male

0.002 (0.12)

Constant

-0.607 (1.25)

Adjusted R

0.13

Observations

732

NOTES: Absolute value of z-statistics in parentheses. +p<0.10; ** p<0.05; ***p<0.01.

MARKS & MOSS, WHAT MAKES A LAW STUDENT SUCCEED? 63

Table 9: Summary Statistics for Entire Sample and for Selected Dichotomous Variables

Mean

LGPA

Observations

UGPA

LSAT

Index

Mean

Std. Dev.

Variance

Entire dataset

1419

3.43

159

194

3.18

0.009

0.0001

Selected dichotomous

Variables:

Top 25% GPA/bottom

25% LSAT

3.81

192

3.17

0.033

0.0011

Top 25% LSAT/bottom

25% GPA

114

3.01

165

195

3.18

0.027

0.0007

Majored in STEM

3.34

161

194

3.22

0.022

0.0005

Majored in EAF

166

3.42

160

194

3.23

0.024

0.0006

No work experience

814

3.44

159

193

3.19

0.011

0.0001

Work: 1-3 years

400

3.43

160

195

3.18

0.017

0.0003

Work: 4-9 years

111

3.41

162

196

3.21

0.036

0.0013

Work: 10+ years

3.49

162

196

3.07

0.055

0.003

Work: in Teaching

3.47

162

197

3.30

0.036

0.0013

Work: in Tech Field

120

3.36

161

194

3.18

0.030

0.0009

Work: in Military

3.47

160

194

3.08

0.068

0.0046

Graduate degree

175

3.40

160

194

3.24

0.027

0.0007

Criminal history

3.34

159

193

3.05

0.042

0.0018

No work experience &

Rising GPA

246

3.25

158

191

3.15

0.021

0.0004

NOTES: This table provides the number of observations, in addition to the mean and LGPA summary

statistics for the entire dataset of two schools combined and for selected dichotomous variables.

Table 10: Summary Statistics for Law Schools

LSAT

UGPA

Median

Top 25%

Bottom 25%

Median

Top 25%

Bottom 25%

University of Colorado

Law School

163

164

160

3.64

3.74

3.43

Case Western

University Law school

158

157

3.39

3.54

3.29

Combined law schools

159

158

3.48

3.62

3.35

NOTES: This table presents LSAT and UGPA summary statistics for the two individual law schools

and for the two law schools combined.