Are you continue to on the fence about whether or not generative synthetic intelligence can do the work of human attorneys? If that’s the case, I urge you to learn this new examine.
Revealed yesterday, this first-of-its-kind examine evaluated the efficiency of 4 authorized AI instruments throughout seven core authorized duties. In lots of circumstances, it discovered, AI instruments can carry out at or above the extent of human attorneys, whereas providing considerably sooner response instances.
The Vals Legal AI Report (VLAIR) represents the primary systematic try and independently benchmark authorized AI instruments towards a lawyer management group, utilizing real-world duties derived from Am Regulation 100 companies.
It evaluated AI instruments from 4 distributors — Harvey, Thomson Reuters (CoCounsel), vLex (Vincent AI), and Vecflow (Oliver) — on duties together with doc extraction, doc Q&A, summarization, redlining, transcript evaluation, chronology era, and EDGAR analysis.
LexisNexis initially participated within the benchmarking however, after the report was written, it selected to withdraw from all of the duties through which it participated besides authorized analysis. The outcomes of the authorized analysis benchmarking shall be printed in a separate report.
Key Findings
Harvey Assistant emerged because the standout performer, reaching the very best scores in 5 of the six duties it participated in, together with a powerful 94.8% accuracy fee for doc Q&A. Harvey exceeded lawyer efficiency in 4 duties and matched the baseline in chronology era.
(Every vendor may select which of the evaluated abilities they wished to decide into.)
“Harvey’s platform leverages fashions to supply high-quality, dependable help for authorized professionals,” the report stated. “Harvey attracts upon a number of LLMs and different fashions, together with customized fine-tuned fashions skilled on authorized processes and information in partnership with OpenAI, with every question of the system involving between 30 and 1,500 mannequin calls.”
CoCounsel from Thomson Reuters was the one different vendor whose AI device acquired a prime rating — 77.2% for doc summarization — and constantly ranked amongst top-performing instruments throughout all 4 duties it participated in, with scores starting from 73.2% to 89.6%.
The Lawyer Baseline (the outcomes produced by a lawyer management group) outperformed the AI instruments on two duties — EDGAR analysis (70.1%) and redlining (79.7%), suggesting these areas could stay, for now a minimum of, higher suited to be achieved by people. AI instruments collectively surpassed the Lawyer Baseline on doc evaluation, info retrieval and information extraction duties.
Maybe not surprisingly, the examine discovered a dramatic distinction in response instances between AI and people. The report discovered that AI instruments had been “six instances sooner than the attorneys on the lowest finish, and 80 instances sooner on the highest finish,” making a robust case for AI instruments as effectivity drivers in authorized workflows.
“The generative AI-based programs present solutions so rapidly that they are often helpful beginning factors for attorneys to start their work extra effectively,” the report concluded.
Doc Q&A produced the very best scores out of any process within the examine, main the report back to conclude that it’s a process for which attorneys ought to discover worth in utilizing generative AI.
The report discovered that Harvey Assistant was constantly the quickest, with CoCounsel additionally being “terribly fast,” each offering responses in lower than a minute.
Nevertheless it additionally stated that Vincent AI “gave responses exceptionally rapidly as typically one of many quickest merchandise we evaluated.”
Oliver was discovered to be the slowest, usually taking 5 minutes or extra per question. The report stated that is doubtless as a consequence of Oliver’s agentic workflow, which breaks duties into a number of steps.
Vendor-Particular Efficiency
Harvey, the fastest-growing authorized expertise startup within the house (having raised over $200 million and achieved unicorn standing since its founding in 2022), opted into extra duties than another vendor and acquired the very best scores in doc Q&A, doc extraction, redlining, transcript evaluation, and chronology era.
“Harvey Assistant both matched or outperformed the Lawyer Baseline in 5 duties and it outperformed the opposite AI instruments in 4 duties evaluated,” the report stated. “Harvey Assistant additionally acquired two of the three highest scores throughout all duties evaluated within the examine, for Doc Q&A (94.8%) and Chronology Technology (80.2% — matching the Lawyer Baseline).”
CoCounsel 2.0 from Thomson Reuters was submitted for 4 of the duties and constantly carried out nicely, the examine discovered, reaching a mean rating of 79.5% throughout its 4 evaluated duties — the very best common rating within the examine. It notably excelled at doc Q&A (89.6%) and doc summarization (77.2%).
“CoCounsel surpassed the Lawyer Baseline in these 4 duties alone by greater than 10 factors,” the examine stated.
For doc summarization, all of the gen AI instruments carried out higher than the Lawyer Baseline.
Vincent AI from vLex participated in six duties — second solely to Harvey in variety of duties — with scores starting from 53.6% to 72.7%, outperforming the Lawyer Baseline on doc Q&A, doc summarization, and transcript evaluation.
The report stated that Vincent AI’s design is especially noteworthy for its potential to deduce the suitable subskill to execute primarily based on the person’s query, and that the solutions it offered had been “impressively thorough.”
Oddly (I assumed), the report praised Vincent AI for refusing to reply questions when it didn’t have adequate information to reply, relatively than give a hallucinated reply. However the report stated these refusals to reply additionally negatively affected its scores.
Oliver, launched final September from the startup Vecflow, was described within the report as “the best-performing AI device” on the difficult EDGAR analysis process. That would appear a given, because it was the one AI device to take part in that process. It scored 55.2% towards the Lawyer Baseline’s 70.1%.
The report highlighted Oliver’s “agentic workflow” method as probably worthwhile for complicated analysis duties requiring a number of steps and iterative decision-making, and stated it excels at explaining its reasoning and actions as it really works.
“Oliver bested a minimum of one different product for each process it opted into,” the report stated. “Oliver additionally outperformed the Lawyer Baseline for Doc Q&A and Doc Summarization.”
Methodology
The examine was developed in partnership with Legaltech Hub and a consortium of legislation companies together with Reed Smith, Fisher Phillips, McDermott Will & Emery, and Ogletree Deakins, together with 4 nameless companies. The consortium created a dataset of over 500 samples reflecting real-world authorized duties.
Vals AI developed an automatic analysis framework to supply constant evaluation throughout duties. The examine notes that the lawyer management group was “blind” — taking part attorneys had been unaware they had been a part of a benchmarking examine and acquired assignments formatted as typical consumer requests.
Tara Waters was Vals AI’s challenge lead for the examine.
Future Instructions
The report signifies this benchmark is the primary iteration of what its says shall be a daily analysis of authorized business AI instruments, with plans to repeat this examine yearly and add others. Future iterations could increase to incorporate extra distributors, further duties, and protection of worldwide jurisdictions past the present U.S. focus.
“There may be rising momentum throughout the authorized business for standardized methodologies, benchmarking, and a shared language for evaluating AI instruments,” the report notes.
Nicola Shaver and Jeroen Plink of Legaltech Hub had been credited for his or her “partnership in conceptualizing and designing the examine and bringing collectively a high-quality cohort of distributors and legislation companies.”
“Total, this examine’s outcomes help the conclusion that these authorized AI instruments have worth for attorneys and legislation companies,” the examine concludes, “though there stays room for enchancment in each how we consider these instruments and their efficiency.”
