Generative AI may have passed the bar exam, but an LLM still can't think like a lawyer.
See also: Prosecutors warn about tackling harmful AI behavior
When the Generative AI boom began in 2022, Rudi Miller and her law school classmates were suddenly gripped by anxiety. “Before we graduated, there was talk about what the job market would be like for us if artificial intelligence was adopted,” she recalls.
So when it came time to choose a career, Miller decided to become a lawyer who represents clients in court. She hopes that court will be the last human stage. “Judges haven’t yet allowed ChatGPT-enabled bots to argue in court,” she says.
He had reason to worry. The apocalyptic era of artificial intelligence seemed to be upon us for lawyers. In March 2023, researchers reported that GPT-4 had crushed the Uniform Bar Exam. That same month, an industry report predicted that 44% of legal work could be automated.
The legal tech industry has boomed as law firms have begun to adopt genetic AI to sift through mountains of documents and draft contracts, work typically done by junior associates. Last month, law firm Clifford Chance laid off 10% of its London staff, citing the increased use of AI as the reason.
But despite all the hype, LLMs are still far from thinking like lawyers—let alone replacing them. Models continue to create bogus case reports, struggle to navigate the gray areas of the law and think through new questions, and stumble when trying to synthesize information scattered across statutes, regulations, and court cases.
See also: Ignoring AI in the threat chain could be a costly mistake
And there are deeper institutional reasons to believe that models could struggle to replace legal jobs. While artificial intelligence reshapes the profession's tedious work, the end of lawyers may not be coming anytime soon.
The legal industry has long been defined by long hours and grueling workloads, so the promise of superhuman efficiency is enticing. Law firms are experimenting with general-purpose tools like ChatGPT and Microsoft Copilot and specialized legal tools like Harvey and Thomson Reuters’ CoCounsel, with some building their own in-house tools on top of cutting-edge models.
They host AI boot camps and allow associates to bill for hundreds of hours of AI experimentation. By 2024, 47.8% of lawyers at law firms with 500 or more lawyers will be using AI, according to the American Bar Association.
But lawyers say LLMs are far from reasoning well enough to replace them.
Allison Douglis, a senior associate at Jenner & Block, uses an LLM to kickstart her legal research. But the tools only get her so far. “When it comes to actually developing and crafting an argument as a lawyer, I don’t think it’s there,” she says. She’s seen models create false case reports and stumble into questionable areas of the law.
See also: Evolutionary artificial intelligence – Technology learns to invent itself
“Right now, I would much rather work with a young collaborator than an AI tool,” he says. “Unless they get really good very quickly, I can’t imagine that changing anytime soon.”
