Key Moments
The Watchdogs of AGI — Rune Kvist of AI Underwriting Company
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI development is outpacing regulation, so AIUC is building trust infrastructure with standards and insurance to make advanced AI deployable, even as the AI risks themselves evolve.
Key Insights
AIUC has raised a $40 million Series A, indicating strong investor confidence in the need for trust infrastructure in frontier AI adoption.
The binding constraint on AI adoption is shifting from capability to risk and liability, as demonstrated by the delayed rollout of models like Fable.
AIUC's approach combines standards, which set the rules and tests for AI safety, with insurance, which incentivizes risk quantification and reduction.
The AIUC certification process involves quarterly technical testing against standards developed with a consortium of Fortune 1000 risk leaders, covering areas like agent security, safety, and reliability.
AIUC plans to expand its certification to models and robotics, recognizing that the risk surface increases with AI sophistication and physical embodiment.
The insurance market for AI is nascent, with a 'lemons problem' where demand for copyright infringement insurance signals higher risk, but insurers like Lloyd's of London are partnering with AIUC to underwrite AI risks based on their standards.
The accelerating race between AI capability and trust infrastructure
The rapid advancement of AI, particularly exemplified by the scaling laws that predict larger models are smarter models, has created a significant gap between AI capabilities and the infrastructure needed for trustworthy deployment. Rune Kvist, co-founder of AI Underwriting Company (AIUC), highlights that the primary bottleneck for AI adoption is no longer its capability but rather the associated risks and liabilities. This is evidenced by the restricted access to powerful models like Fable, not due to their performance, but because their unpredictable behaviors make it difficult to make promises about their actions. AIUC's $40 million Series A funding round, led by Ribbit Capital and Frost Money, underscores the market's recognition of this critical need for trust-building mechanisms in the face of increasingly autonomous and capable AI systems.
Establishing trust through standards and insurance
AIUC's core mission is to build confidence infrastructure for frontier AI through a combination of standards and insurance. Drawing parallels to historical technological waves like electricity and automobiles, Kvist explains that new technologies always face challenges related to safety, liability, and public trust. In each instance, the market developed common blueprints involving standards and insurance. Standards define the rules of the road and the necessary tests to assess risk, while insurers are incentivized to quantify and reduce risk to protect their capital. AIUC's AIUC1 agent certification standard is designed to address concerns of security leaders in large enterprises by operationalizing comprehensive risk management frameworks. The standard requires rigorous, quarterly technical testing, ensuring that AI agents meet defined benchmarks for security, safety, and reliability. This dual approach of setting clear technical benchmarks and providing financial backing through insurance aims to bridge the trust gap between AI developers and users, enabling wider adoption.
The AIUC1 certification process: Technical rigor and evolving standards
The AIUC1 standard is a comprehensive framework for agent security, safety, and reliability, designed to address the fears and questions that slow down AI adoption. It's grounded in technical testing, requiring companies to undergo rigorous simulations every quarter to assess risks like jailbreaking, hallucinations, and data leaks. The standard is intentionally updated quarterly to keep pace with AI's rapid evolution, a stark contrast to the decade-long cycles of traditional standards. This dynamic update process is informed by a consortium of risk leaders from Fortune 1000 companies in critical sectors like banking and healthcare, who provide direct input on emerging concerns. The standard itself breaks down high-level requirements into specific technical controls, evidence needed, and test controls, aiming to move beyond abstract principles to actionable, auditable criteria. This detailed approach helps risk leaders ask the right questions and evaluate AI systems effectively.
From agents to models and robotics: Expanding the trust framework
AIUC's roadmap includes expanding its certification efforts beyond agents to encompass models and robotics. The recent high-profile concerns surrounding models like Mythos and Fable highlight the urgent need for model-level certification, particularly concerning national security risks. Kvist envisions a future where a neutral third party, akin to financial rating agencies like Moody's, will sit between AI labs and governments to provide objective assessments of model capabilities and risks. This expansion acknowledges that as AI becomes more powerful and capable of autonomous interaction and physical embodiment (like robots), the risk surface grows exponentially. The challenges with physical AI, such as robots interacting with the physical world, will demand even higher levels of stringency and accountability, potentially necessitating entirely new legal frameworks.
The evolving landscape of AI risk and insurance
The AIUC framework is adaptable to new modalities and emerging risks. While current concerns often revolve around cyber threats and user safety, Kvist points to future risks such as those related to biology (e.g., aiding adversaries in producing biological weapons) and the increasing complexity of agent-to-agent interactions. The 'lemons problem' is particularly evident in areas like copyright, where the very act of seeking insurance might signal higher risk. However, partnerships with established insurers like Lloyd's of London are crucial for developing AI insurance policies. These insurers, while lacking deep technical AI expertise, bring crucial capital and a long history of managing risk. AIUC's standards serve as the underwriting framework, allowing insurers to price policies and providing a signal of manageable risk to enterprises adopting AI solutions.
The role of standards in shaping liability and fostering trust
Standards play a vital role in clarifying liability, which is foundational for both insurance and broader AI adoption. When an AI system causes harm, courts often look to established standards to determine negligence or duty of care. By creating and promoting widely adopted standards, AIUC helps to define best practices and make it harder for companies to claim ignorance of potential risks. This clarity benefits not only those seeking insurance but also legal systems and regulators grappling with novel AI-related incidents. The legal precedent set by cases like Air Canada's hallucinating chatbot, where the company was held liable for the AI's promises, underscores the importance of AI providers taking responsibility for their systems. Standards provide a measurable benchmark for this responsibility.
Building a 'universal red teamer' for consistent AI risk assessment
A core engineering challenge for AIUC is building a 'universal red teamer'—a consistent methodology and taxonomy for assessing AI risks across diverse applications, from coding agents to customer support bots. This approach aims to provide a unified language for risk assessment that can be understood by decision-makers in large enterprises. While specialized certifications for specific AI use cases might exist, AIUC's focus on a single, adaptable framework ensures that regardless of an enterprise's primary risk area, the standard can offer relevant insights. This comprehensive approach is essential for building trust, as it allows for consistent evaluation and promises about AI safety and reliability, enabling businesses to confidently integrate AI into critical infrastructure.
The future of AI underwriting and the necessity of independent oversight
Even in a future where AGI is achieved, the need for independent oversight and trustworthy AI underwriting will persist. Kvist argues that AI labs, despite their intelligence and stated good intentions, will always face incentives to cut corners or withhold information in a competitive race. Therefore, an independent third party, like AIUC, is essential to inspect data, share information transparently, and build confidence. This role is distinct from the labs' internal efforts, as it provides an external, objective assessment. The 'watchdog' function is crucial, and the market dynamics suggest that only entities directly bearing financial risk, such as insurers, can provide the necessary balance against the potential for standards to become race-to-the-bottom mechanisms. The combination of robust standards and insurance is seen as the enduring infrastructure for trustworthy AI deployment.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Studies Cited
●Concepts
●People Referenced
Common Questions
AI Underwriting Company provides confidence infrastructure for frontier AI through standards and insurance. They aim to solve the problem of risk and liability that hinders AI adoption by developing comprehensive frameworks and facilitating insurance.
Topics
Mentioned in this video
Mentioned as an example of a physical AI with potential for strict liability issues, similar to how Waymo faces challenges.
An AI company that is a customer of AI Underwriting Company, and has purchased a first-of-its-kind AI agent insurance policy. They are also mentioned as a provider of voice technology.
The AI lab founded by former OpenAI researchers, where the speaker initially worked. They were wrestling with deployment and revenue questions, and had detailed vision documents for the future of AI.
The AI research company from which the founders of Anthropic had broken off. Concerns about OpenAI's handling of deployment were a factor in Anthropic's formation.
An autonomous vehicle company whose driving capabilities, despite being superhuman, are hindered by liability and trust issues, illustrating the problem AI Underwriting Company aims to solve.
An auditing firm that AI Underwriting Company partners with to conduct technical testing for their certification standards.
A company that went bankrupt due to fraud, mentioned in the context of prediction markets and whether they could have predicted the bankruptcy.
A company whose chatbot hallucinated a refund policy, leading to a court case that set a precedent for AI-generated legally binding promises.
Mentioned again as an example of a trusted financial auditor that governments rely on, reinforcing the idea of independent third-party verification.
A credit rating agency that provides ratings for bonds, serving as a neutral third party that aggregates public information and provides a common information layer for investors. It is also used as an example of a watchdog that can be scrutinized.
Mentioned for its Luma agent, which still requires significant human in the loop for video generation.
Mentioned in the context of Waymo's autonomous vehicle efforts and their potential need for insurance, and also in relation to their agent studio for building managed agents.
A for-profit company that provides credit scores, used as an example of a for-profit standard that serves the world well.
Mentioned as a current AI product where risk is the binding constraint on adoption, alongside Fable.
Mentioned as a current AI product where risk is the binding constraint on adoption, alongside Mythos. It's currently not open for access due to the difficulty in making promises about its behavior.
An AI company that is a customer of AI Underwriting Company, highlighted as a frontier company that has had an easy time selling pilots but needs to go through risk processes for wider adoption.
An AI company that is a customer of AI Underwriting Company, highlighted as a frontier company that has had an easy time selling pilots but needs to go through risk processes for wider adoption.
An AI company that is a customer of AI Underwriting Company.
The most popular AI product at the time of the founder's early involvement in AI (early 2022), indicating the early stage of the market.
AI models developed by the same team that founded Anthropic, demonstrating their capability and experience in scaling large AI models.
Mentioned as a recent development raising concerns for security leaders, related to agent-to-agent interactions.
A tool mentioned in the context of Mechinurb activation signals.
A technology that shows promising scientific potential for risk reduction in AI, though not yet commercially available on demand, but could become an optional control for credit.
The paper on scaling laws in AI, which states that larger models are smarter, struck the founder like lightning and made the potential of AI clear to him, especially with the understanding that capital would drive returns.
The founder's academic background, which provided a different lens for interpreting the scaling laws paper compared to machine learning experts. It's also described as a program where prime ministers are born.
Mentioned in the context of electricity's emergence and the initial risks and accidents that followed.
Mentioned as an example of Lloyd's of London's creative insurance policies, like insuring his right foot.
Mentioned as an example of Lloyd's of London's creative insurance policies, like insuring her butt.
Mentioned humorously as a legal expert who could potentially extract copyrighted material, but is in short supply.
An auditing firm that AI Underwriting Company partners with to conduct technical testing for their certification standards.
A sub-body of NIST that typically sets standards, mentioned as a government body that could potentially build its own third-party auditing capabilities.
An open-source community of security practitioners that builds frameworks for addressing security concerns, considered a partner and a source of intelligence by AI Underwriting Company.
The National Institute of Standards and Technology, under which the Center for AI Standards and Innovation operates.
Mentioned as a company that offers Mechinurb, which is considered commercially available.
The world's oldest insurer, partnered with AI Underwriting Company, that has never failed to pay a claim and provides trusted underwriting for AI insurance policies.
Underwriters Laboratories, a historical example of a standard-setting body that started as a nonprofit and later spun out a for-profit entity to better serve customers, serving as an inspiration for AI Underwriting Company.
More from Latent Space
View all 250 summaries
94 minRecursive Self-Improvement: from Auto Research to Superintelligence — Richard Socher, Recursive
41 minInside the Team That Killed Mandatory Code Review — Quinn Slack, AMP
28 minFaster Chips Designs That Don't Melt — Anima Anandkumar & Benedikt Jenik, Accelerated Understanding
45 minThe Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free