(Sep 9) What happened in LegalTech last week
Last week in legal tech, Harvey introduced a framework for assessing AI in legal tasks, while discussions on AI reliability and significant VC investments dominated the landscape.
Welcome to your Law Tech weekly briefing covering what happened in legal tech last week.
This briefing and podcast is 99% created and produced by AI š¤ and 100% useful to humans. Please share it with your colleagues.
TLDR; Listen instead:
Key Stories
Harvey's new benchmarking framework has set a precedent for evaluating large language models (LLMs) on legal tasks ("The Gen AI LLM benchmarking war starts here" - Legal IT Insider). By using rubrics that assess tone, length, and relevance, this framework aims to standardize and measure the effectiveness of LLMs. Prominent legal experts like Neil Cameron praise the move but highlight the challenges in standardizing time entry data. This coincides with recent debates spurred by Stanford University's paper on AI hallucination, which underscores the importance of understanding limitations and ensuring proper training for Gen AI tools. Experts argue that, despite their imperfections, these AI tools can be valuable when deployed correctly and benchmarked against human standards ("Real world reflections on Gen AI hallucination and risk" - Legal IT Insider).
Furthermore, the legal tech landscape continues to attract substantial investment. The first half of 2024 saw a surge in VC funding and M&A deals, driven by mounting interest in GenAI technology ("Legal Tech Investment + M&A Deals Up On H1 2023" - Artificial Lawyer). Although investment remains below the highs of 2021, the upward trend signifies confidence in the sector's growth. Raymond James forecasts continued expansion but stresses the importance of addressing the ethical and operational challenges posed by GenAI. This burgeoning investment volume reflects a broad market consensus that while GenAI holds transformative potential, its deployment must be cautious and well-regulated.
All Other Stories
The broader landscape of legal innovation presents a mixed bag of advancements and cautionary tales. Centerbase's enhanced Automated Time Capture (ATC) feature now seamlessly tracks time spent on tasks within Microsoft Word and Outlook, promising to reduce time leakage and boost revenues for law firms ("Law Practice Management Platform Centerbase Now Automatically Captures Time Spent Working in Microsoft Word and Outlook" - LawSites).
On the strategic front, balancing innovation with human-centric service remains key for law firms. Joey Seeber emphasizes embracing a startup mentality and transparent communication as essential for staying ahead ("Balancing Technology and Human Connection in Legal Services with Joey Seeber" - Fringe Legal). A more radical approach suggests that law firms bypass traditional recruitment processes and build their own legal talent pools, potentially enhancing diversity and practical readiness ("Forget law school recruitment. Build your own lawyers." - Jordan Furlong).
Despite advancements, the ILTA 2024 Technology Survey reveals a significant gap in GenAI adoption between large and small law firms, highlighting the need for better training initiatives ("ILTA 2024 Survey suggests chasm between small and large law firmsā Gen AI plans & use" - Legal IT Insider).
Links and summaries
Want to explore a story more deeply? Here are the original links to the stories, with summaries to aid review.
Key stories
Harvey's release of a benchmarking framework for evaluating large language models (LLMs) on real-world legal tasks marks a significant advancement in legal technology. By transforming time entries into model-based tasks and categorizing them by practice area and type of work, Harvey addresses the complexity inherent in legal evaluations, which cannot rely on simplified, one-size-fits-all criteria. The framework employs rubrics to assess accuracy, relevance, and tone, among other factors, and benchmarks the viability of a modelās sourcing, assigning scores to measure overall performance. Neil Cameron of Legal IT Insider lauds the initiative, noting its foundational criteria of time entry comparison and source confirmation, though he flags potential subjectivity issues in time data normalization. Cameron also emphasizes the need for ongoing development toward industry-standard benchmarks for Gen AI in law, indicating that Harvey's approach is a solid step in that direction but not the final solution.
Real world reflections on Gen AI hallucination and risk (Legal IT Insider)
The rapid advancement of legal generative AI tools is undeniably promising, yet the shadow of Stanford Universityās study on AI hallucinations looms large. The controversy surrounding the paper, which revealed a 17% hallucination rate in top AI legal research tools, underscores a critical industry challenge: contextualizing the risks of AI inaccuracies. Critics argue that Stanfordās methodology was flawed, yet the study highlights the need for better benchmarks and standardized testing. As industry experts note, education on the strengths and limitations of AI tools is crucial. While these tools arenāt perfect, they can enhance efficiency and accuracy if used and evaluated correctly, avoiding binary judgments on their utility.
Legal Tech Investment + M&A Deals Up On H1 2023 (Artificial Lawyer)
The latest data from Raymond James indicates a notable upswing in venture capital funding for the legal tech sector in H1 2024, with investments reaching Ā£430m compared to Ā£244m in H1 2023. The number of fundraises also surged from 68 to 118 in the same period. M&A activity saw a slight boost with 35 deals in H1 2024 versus 28 the previous year. GenAI technologies are prominently driving these investments, signifying their growing importance in the legal industry. However, despite improvements, funding levels remain lower than the peak seen in 2021, largely due to high interest rates and economic uncertainties. Overall, the outlook appears promising, but tempered with caution due to ongoing macroeconomic challenges and debates on GenAI's long-term impact on the sector.
All the rest
Harvey Launches Legal GenAI Evaluation System: BigLaw Bench (Artificial Lawyer)
Harvey launched BigLaw Bench to evaluate the accuracy of generative AI tools in legal tasks. Their model scored 74%, outperforming GPT-4o's 61%, particularly in sourcing accuracy. This initiative aims to set benchmarks and encourage improvements in legal AI.
Centerbase has upgraded its Automated Time Capture (ATC) feature to track time spent on emails and document editing in Microsoft Word and Outlook. This enhancement helps lawyers accurately log billable hours without altering their workflow. The technology aims to reduce time leakage and increase law firm revenue by capturing more billable time effortlessly.
Balancing Technology and Human Connection in Legal Services with Joey Seeber (Fringe Legal)
Law firms must balance tech innovation with human-centric service to stay competitive. Embracing a startup mindset, continuous improvement, and transparent communication is crucial. The future lies in blending technology with human expertise and empathy.
Forget law school recruitment. Build your own lawyers. (Jordan Furlong)
Law firms are now recruiting law students earlier, skipping traditional on-campus interviews. The article proposes that firms should expand this by recruiting from the general population and creating their own legal education programs. This strategy could diversify talent and better equip new lawyers for practical work.
Balancing Innovation + Risk in Corporate Legal Leadership (Artificial Lawyer)
Emmanuel Caradec, General Counsel at Natixis, discusses the adoption of Gen AI in legal tech, focusing on risk management and compliance. He emphasizes evaluating tools based on security, effectiveness, and integration, considering both financial and non-financial returns. Caradec predicts Gen AI will transform legal departments by enhancing efficiency and enabling data-driven decision-making.
Lawyers Will Need To Be Capitalists In The Age Of AI (Artificial Lawyer)
Despite heavy investment in legal tech, the legal industry remains largely unchanged because lawyers are central to legal work. AI advancements could enable lawyers to control automated tools, shifting their role. This change may prompt lawyers to adopt a more industrial capitalist mindset, transforming the legal market.
Lawyers using Venmo risk exposing client information due to the app's default public setting. This exposure violates American Bar Association confidentiality rules. Attorneys should set their Venmo transactions to private to protect client privacy.
ILTAās 2024 Technology Survey highlights a significant disparity in Generative AI adoption between small and large law firms. Larger firms are ahead in using tools like Microsoft Copilot for eDiscovery and Marketing, while smaller firms struggle. Key challenges include user adoption and skill gaps, emphasizing the need for training.
ILTA Survey: āGeneral GenAI Tools Most Popular With [Smaller] Law Firmsā (Artificial Lawyer)
The ILTA survey reveals that generative AI tools like Microsoft Copilot and ChatGPT are popular but not very effective for legal work, especially in small to medium-sized firms. Larger firms are more advanced in AI adoption, but the broader legal market remains cautious and primarily in the investigation phase. Overall, AI integration in the legal sector is slow and cautious, far from revolutionary.
UK law firm Roythornes has selected the cloud-based 3E financial and practice management platform from Elite. The firm aims to boost efficiency, access critical management data, and ensure future scalability. Implementation will be supported by Helm360, reflecting Roythornes' dedication to innovation and collaboration.
āThe Disruptive Power of AIā ā Legal Mastermind Podcast (Artificial Lawyer)
Richard Tromans, founder of Artificial Lawyer, discussed the evolution of legal AI from 2015 to the present on the Legal Mastermind podcast. He highlighted the significant impact of generative AI on law firms. The podcast, hosted by Eric Bersano and produced by Paul Raphel, focuses on law firm growth and management.
The Fable of Starbucks (Adam Smith, Esq.)
Starbucks is facing challenges from changes in its business model that hurt customer experience. The company hired new CEO Brian Niccol to tackle these issues. The article warns the legal industry about the dangers of straying from core business principles.
