Key Moments
Building Legal Agents When You Can't See the Data — Jacob Lauritzen, CTO of Legora
Latent Space PodcastKey Moments
Legora's AI agents can't see client data, forcing them to build and test models through indirect feedback loops, making direct performance analysis nearly impossible.
Key Insights
Legora started with ChatGPT, Swedish law, and a search tool, focusing initially on legal research and expanding to drafting, editing, and complex analysis.
The company's 'tabular view' feature, designed for bulk contract extraction and review, has become an industry-recognized term.
Legora uses a mix of engineers and lawyers, including 'legal engineers' with big law experience and CS degrees, for evaluation and product development.
Despite AI advancements, complex legal analysis still requires more intelligence, and the expansion of AI capabilities may also expand the scope of legal work.
A significant challenge for Legora is the inability to directly observe agent performance on client data due to strict confidentiality, necessitating a reliance on indirect feedback and synthetic data for evaluation.
The legal industry's adoption of AI is seen as a necessary adaptation due to competitive pressure, potentially shifting billing models from hourly to project-based fees.
The challenge of building AI without seeing the data
Jacob Lauritzen, CTO of Legora, details the unique hurdles his company faces in developing AI agents for the legal sector. Because client data is highly confidential and belongs to the law firms, Legora cannot directly access or analyze the work their agents perform. This fundamental constraint forces the company to rely on indirect feedback, approximations, and synthetic data for evaluating and improving their AI models. This 'black box' problem, where the agents' actions are not directly observable, is a core operational challenge that Legora constantly works to mitigate through sophisticated evaluation teams and indirect validation methods. The inability to 'look at traces' of data means they must approximate user behavior and model performance, which is a significant deviation from how many AI products are developed and tested.
Legora's product evolution from assistant to agent
Legora's journey began with a simple premise: combining ChatGPT with Swedish legal databases and a search tool to assist lawyers with legal research. This initial 'assistant' model has since evolved into sophisticated 'agents' capable of much more. The product now handles drafting, editing Word files, organizing documents, and performing complex analyses. A key innovation is the 'tabular view,' a feature that allows for bulk extraction and review of numerous agreements, a crucial process in due diligence and fundraising. Beyond performing tasks, Legora is also focusing on orchestrating legal workflows, introducing features like 'lists' for task management and 'intake' and 'portal' for enhanced client and lawyer collaboration. This evolution reflects a shift from simply aiding lawyers to fundamentally changing how legal work is managed and executed.
The role of legal engineers and AI evaluation
To navigate the complexities of legal AI, Legora employs a specialized team comprising engineers and lawyers. This team includes 'legal engineers,' individuals with significant experience in big law and a technical background (often with CS degrees), who act as a crucial bridge between legal practice and AI development. They are instrumental in product development and, importantly, in the evaluation process. Legora tracks model performance across various legal practice areas and tasks using a 'heatmap' approach. This rigorous evaluation is essential because even minor model updates can sometimes lead to regressions in specific areas, a phenomenon that occurs more frequently than one might expect. The legal engineers' deep understanding of legal work allows them to build realistic evaluation environments, such as simulating a due diligence process on thousands of files to assess drafting quality and identify red flags.
Adapting to client needs and privacy concerns
Legora's infrastructure is primarily hosted on Microsoft Azure, a strategic choice driven by the legal industry's historical hesitations towards cloud services like AWS or Google Cloud. Many law firms already use Microsoft 365, making Azure an approved vendor and easing the transition to cloud-based solutions. This migration reflects a broader industry shift, moving away from on-premises solutions towards the perceived security and scalability of the cloud. Even with cloud adoption, the strict data privacy requirements remain paramount. Legora cannot view client data traces, leading to innovative evaluation strategies, such as creating synthetic data or using paraphrased examples to approximate real-world scenarios. This ensures that while they are developing powerful AI tools, client confidentiality is never compromised.
The evolving legal profession and partnership models
Lauritzen predicts that AI will significantly increase lawyer productivity, enabling them to focus more on strategic judgment, risk assessment, and client relationships, while automating menial tasks like document review. This shift may also impact the traditional partnership model in law firms. Historically, partnerships were structured to retain human capital, the firms' primary asset. However, as AI-powered systems can codify expertise and create intangible assets, the incentive structure for partners may need to evolve. The focus might shift from merely bringing in clients to also contributing to the creation of durable, codified IP within the firm. This could lead to new business models where the firm's collective knowledge, enhanced by AI, becomes a significant asset.
The future of legal work and AI integration
The adoption of AI is not just about efficiency but also about fundamentally reshaping legal practice. While AI can handle a significant portion of legal work, complex analyses and strategic decision-making will still require human expertise. The future may see lawyers acting more as orchestrators and verifiers of AI-generated work, spending less time on tedious tasks and more on high-level judgment. This could lead to a polarization of roles, with highly skilled junior lawyers becoming even more valuable due to their ability to leverage AI effectively. The partnership model itself is also in question, with the potential for firms to build 'intangible capital' through codified expertise, creating unique IP.
Core primitives and strategic product development
Legora focuses on developing core 'primitives'—reusable components that can be configured to build various products. Key primitives include 'lists' (a flexible database-like system), 'projects' (containers for files), and the 'agent' itself. Features like 'tabular view' are seen as instantiations of these primitives, leveraging the spreadsheet format for efficient data extraction and analysis. The company's product strategy involves making strategic bets on where the industry is headed, such as the development of more autonomous agents and enhanced collaboration within the platform, even if these features aren't directly requested by all clients initially. This proactive approach ensures Legora remains at the forefront of legal tech innovation.
The critical role of search and embeddings in legal AI
In the realm of legal AI, search and retrieval are paramount. Lauritzen emphasizes that the accuracy of information pulled is critical, as flawed data leads to flawed analysis and outputs. Legora invests heavily in customizing search and retrieval pipelines, including the use of specialized embedding models. They advocate for 'customizing embeddings' rather than relying on default or generic options, especially for multilingual applications, highlighting that this offers a significant competitive advantage. The ability to accurately find and present relevant information from vast document sets is foundational to effective legal work, and continuously improving these search capabilities is a top priority.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●People Referenced
Common Questions
Legora is a legal tech company that builds AI agents and tools to help lawyers work more efficiently and orchestrate their tasks better. They aim to automate legal research, drafting, and analysis, making the legal profession more productive.
Topics
Mentioned in this video
A model mentioned as being available in Legora's agent, noted for being chatty and verbose, but potentially improving.
A leading grid component that Legora might be using or considering for their tabular view feature.
A key feature of Legora's product that allows for bulk extraction and review of agreements, likened to a spreadsheet for legal data.
An early version of Gemini is highlighted as being surprisingly good for transcription and speaker attribution.
Mentioned as the initial inspiration and foundation for Legora's legal research tool, and as a general comparison for their product.
A feature within Legora's platform, described as similar to Linear or Notion databases, used for orchestrating work and tasks.
Microsoft's cloud platform where Legora hosts its infrastructure, chosen for its acceptance by law firms.
A model available in Legora's agent, mentioned in the context of model selection and experimentation.
Used as an analogy for Legora's 'Lists' feature, emphasizing its flexibility and configurability like a database.
Mentioned as a competitor and as a platform that is offering legal services, but Legora believes they go deeper for complex transactions.
A model mentioned as being available in Legora's agent, noted for its concise writing style compared to Fable.
A transcription model that Legora finds less accurate for specific jargon compared to their custom solutions.
A company offering custom embeddings, recommended for workloads that are heavily reliant on Retrieval Augmented Generation (RAG).
Initially, law firms were hesitant to trust AWS for their data, leading Legora to choose Microsoft Azure instead.
Mentioned as a comparison for Legora's 'Lists' feature, highlighting its task management capabilities.
A fast-growing legal tech company focused on building legal agents and tools for lawyers, originating from Europe.
Legora hosts its infrastructure on Azure (Microsoft's cloud platform) due to law firms' trust in Microsoft's existing security protocols.
Mentioned as a provider of models and services that offer a comprehensive suite of tools, which is valuable for Legora's clients.
Not used by Legora for transcription due to Data Processing Agreement (DPA) reasons, prioritizing data security.
A company acquired by Legora, specializing in fact intelligence for litigation, which involves extracting and comparing facts from documents.
More from Latent Space
View all 271 summaries
27 minUsing Agents for Long-Horizon Financial Work — Aman Singh, Arca
28 minBuilding ChatGPT for Finance — Joseph Kim, OpenAI's Head of Financial Services
107 minHow Anthropic ships the #1 LLM in the World — with Katelyn Lesse and Angela Jiang
86 minWhy AI Compute Comes in Rectangles — Erik Bernhardsson, Modal
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free