An AI Program May Soon Be The One With The Best Grades In Your Study Group

The “AI Can’t Replace X!” gambit is a key component of assessing how effective AI is at any given time. AI can’t replace artists quickly progressed from recreations that look like stroke symptoms to Balenciaga Pope Francis and seemingly self-aware Voe 3 short films in the span of four years. “But those are artistic endeavors”, the naysayers say. “AI could never break down chaotic fact patterns into bite-sized bits of information and apply formulaic processes to arrive at legal conclusions like we do!” At first, it seemed like they were right. Earlier AI models were given the sort of answers you’d find in a law school exam and scored 6.4 out of 10. But that was February of this year. We’re four months down the line from back then, which was more than enough time for a new generation of AI to emerge and be submitted to law school exam-like conditions. According to Reuters, the scores are looking a lot better than a 64%:

The latest generation of generative artificial intelligence can ace most law school final exams, a new study has found.

OpenAI’s newest model, called o3, earned grades ranging from A+ to B on eight spring finals given by faculty at the University of Maryland Francis King Carey School of Law, researchers found in a new paper published on SSRN.
…
Generative AI looks to be catching up to actual high-performing law students, based on the latest study. Unlike ChatGPT, which immediately generates text in response to a user’s query, o3 is what is known as a reasoning model. This means that it generates tentative answers and multiple approaches to questions after internally evaluating and revising those responses, after which it produces the final text for the user.

AHT AHT AHT, come back here! These results do not mean that you have free license to go use o3 to cite controlling cases in your work product without checking it first! Nor does it mean that you shouldn’t study as hard in the coming semesters — your professors and bar exam adjudicators presumably won’t allow you to use o3 or some reasoning model, besides your own brain, to try and get top marks on your exams. Does this mean that law professors should be a little extra cautious with assigning take-home finals? Hell yeah, it does! The study authors are already considering having the program throw in a couple of spelling and grammar errors just to make it harder for the grading professors to differentiate AI answers from that one kid who constantly shows up to office hours. I never would have guessed that the Turing test would be carried out by law professors grading Torts finals.

What’s next for o3 and its progeny? My money is on it writing the next bar exam, but I doubt it will stop there. How big is the skill gap between performing at a level of an ahead-of-the-curve law student and doing paralegal work? Between that and summer associate assignments? Jobs are safe for now, but who knows what developments will be made in another four months.

Artificial Intelligence Is Now An A+ Law Student, Study Finds [Reuters]

Earlier: Linklaters Put AI Through Law Exams

Chris Williams became a social media manager and assistant editor for Above the Law in June 2021. Prior to joining the staff, he moonlighted as a minor Memelord™ in the Facebook group Law School Memes for Edgy T14s .  He endured Missouri long enough to graduate from Washington University in St. Louis School of Law. He is a former boatbuilder who is learning to swim, is interested in critical race theory, philosophy, and humor, and has a love for cycling that occasionally annoys his peers. You can reach him by email at cwilliams@abovethelaw.com and by tweet at @WritesForRent .

The post An AI Program May Soon Be The One With The Best Grades In Your Study Group appeared first on Above the Law.