Key Moments
How to kill your SaaS: what we learned replacing a $43K/yr tool with AI agents
Latent Space PodcastKey Moments
AI agents can clone existing SaaS tools for a fraction of the cost, but human oversight is still crucial for true quality and maintenance.
Key Insights
Replacing a $43,000/year SaaS tool with AI agents resulted in a functional clone that cost approximately $1,000 in API tokens and $10,000 in AI engineering expertise.
Human event teams evaluated the AI-generated clones, finding that six out of the top 10 AI-rated applications were considered weak by human experts.
The top three finalists, while successful, spent significant time on pre-coding research and specification writing, indicating AI's role as an execution assistant rather than a fully autonomous developer.
A significant portion of the AI-generated code, around 91,000 lines on average per submission, was built using a hybrid approach, with 40 out of 66 submissions using both Cloud Code and Codex.
The winning submission, built by Valentin de Matos, created a 'software factory' capable of launching multiple portals and fixing bugs, which the company plans to integrate into future conferences.
While AI agents can create functional software, the crucial 'last mile' of maintenance, error reporting, and human-centric quality assurance still requires human intervention and accountability.
The $43,000 SaaS problem and the AI solution
The AI Engineer conference organizers faced a significant cost for event management software, with a specific tool, Sessionboard, quoted at $43,000 annually. This expense, coupled with the slow turnaround time for feature requests (3-6 months or never), prompted a desire for a more agile and cost-effective solution. Historically, the company policy prohibited building in-house software due to the 'not invented here' syndrome and the risk of diverting resources from core business. However, the advent of capable AI agents has shifted this perspective, leading to a bold experiment: 'Kill My SaaS.' The goal was to clone a mid-tier, routine business software using AI agents and have it evaluated by actual users – the events team – to assess its viability. This challenge was designed to test the limits of AI in replicating complex, yet common, business functionalities. The provided specifications included detailed tasks like speaker submission and review processes, speaker portals for uploads, automated reminders, and dynamic agenda management. The complexity arises from accommodating multiple user roles (speakers, sponsors, attendees, organizers) and the need for flexibility, as event requirements can change rapidly. The risk of software failure, leading to event delays or cancellations, necessitates robust and reliable solutions, which traditional SaaS often struggles to provide affordably or adaptively.
The 'Kill My SaaS' challenge and its participants
To facilitate the cloning process, developers were provided with a comprehensive brief, including a product demo video and 42 screenshots. A $10,000 prize was offered for the winner, with a $500 'serious attempt' compensation for incomplete but earnest submissions. Crucially, the final decision rested with the company's actual events team, ensuring a real-world, user-centric evaluation. Seventy-two teams submitted their solutions, many of which remained open-source. The challenge highlighted common pitfalls encountered by AI clones, including inadequate multi-portal support, ineffective drag-and-drop functionality for scheduling, and difficulties in seamlessly adding presentations without a formal submission process. These were areas not always well-documented in the original software but quickly identified by real users.
Evaluating AI clones: The AI judge versus human experts
An AI judge was employed as a preliminary check, simulating user interactions by having 'Claude' log in as various roles and execute 20 pre-defined user scenarios. This process, involving over 56 applications and 40 sufficiently covered submissions, aimed to provide a scalable evaluation framework. The AI judge's scoring showed high competition among the top seven teams, with scores within 2.3 points of each other, indicating a saturation point. This progressive refinement approach, guided by automated evaluation, was deemed a useful first step for SaaS software development. However, a critical divergence emerged when comparing the AI judge's rankings with the human events team's assessments. Six of the top 10 AI-rated applications were deemed weak by the human experts, and the overall top-ranked AI submission placed 19th among humans. This discrepancy underscored the limitations of purely automated evaluation, especially when nuanced user experience and qualitative aspects are paramount.
Common technical challenges in AI-generated clones
The AI agents frequently struggled with three specific areas. Firstly, building flexible and distinct portals for speakers, invited speakers, and sponsors proved challenging; most submissions offered only one type of generic portal. Secondly, implementing effective drag-and-drop functionality for scheduling, a complex UI element, was difficult for AI to replicate accurately, impacting usability and preventing conflicts like double-booking sessions or rooms. Thirdly, allowing speakers to add presentations directly without the need for a formal Call for Papers (CFP) process, essential for VIPs, was another hurdle. While tooltips were sometimes provided, they often felt like an overlay rather than integrated guidance, failing to simplify the user experience.
How developers built the AI clones: Tooling and strategies
The process of cloning involved significant effort, with an average repository size of 91,000 lines of code and agents running for days. A common practice was the hybridization of AI models, with 'Cloud Code' and 'Codex' being nearly tied in usage, and 40 out of 66 submissions utilizing both. GPT-3 was also a frequent choice among finalists and other submissions. Developers often employed role-switching strategies, where one AI model planned and reviewed (e.g., Cloud), and another executed the coding (e.g., Codex). Other patterns included a build-and-verify approach, where a browser-testing AI confirmed the output of another, or separate parallel workflows where one AI's work was validated by another to prevent self-deception. The raw code generation was often the last step, preceded by extensive research, specification writing, and planning.
The human element: 'Taste' and user experience validation
The experiment revealed that 'taste' – an intuitive understanding of good user experience – is a critical factor that AI agents currently lack. The human evaluators prioritized seamless navigation, intuitive interfaces, and a holistic user journey. For instance, one submission offered a single-page view of all outputs, which was highly praised by the operations lead, Kelsey. Another winner, Yazin, made procedural emails engaging, a difficult feat. The finalists emphasized pre-coding research, like scanning entire help sites and converting them into markdown, before even prompting the AI. They also conducted rigorous manual testing, dedicating time to ensure the user experience was smooth, and iterating on the AI's output based on these real-world interactions. Valentin's winning approach involved building a 'software factory' with multiple portals and a focus on robust error handling and maintenance, directly addressing the need for accountability beyond automated code generation.
The verdict: AI can clone, but humans must maintain
The experiment concluded that while AI agents can successfully clone existing SaaS tools for a fraction of the cost (estimated at $1,000 in tokens and $10,000 in AI engineering expertise), they cannot entirely replace human oversight and maintenance. The ability to clone a functional application is achievable, as demonstrated by numerous submissions. However, the crucial 'last mile' – keeping the software running, handling bug reports, and ensuring ongoing quality – still requires human intervention and accountability. The event organizers found that reliance solely on AI agents is insufficient when issues arise; a human is needed to troubleshoot and fix problems, especially under pressure. The chosen solutions, a blend of 'Open Rostrum' and 'Conference,' along with ideas from other participants, will be integrated for future events, reflecting a hybrid approach where AI accelerates development, but humans own the final product and its upkeep. This ownership, while bringing responsibility, also allows for continuous improvement based on actual user feedback, a significant advantage over third-party SaaS.
Mentioned in This Episode
●Software & Apps
●Companies
●People Referenced
Killing Your SaaS with AI Agents: Key Takeaways
Practical takeaways from this episode
Do This
Avoid This
Common Questions
While AI agents can effectively clone existing software functionalities for a fraction of the cost, they currently require significant human oversight for maintenance, bug fixing, and ensuring accountability. Completely replacing SaaS without human involvement is not yet feasible.
Topics
Mentioned in this video
One of the applications that benefited from the AI judge's feedback for incremental improvement.
The language model used by the top three finalists and many other participants in the challenge.
A large language model used by developers in the challenge, often in combination with other models.
Mentioned as a knowledge base that Marco Kramer's 'Cortex' project is a clone of.
Yazeen's submission, praised for its seamless navigation, single-page output, and enjoyable procedural emails.
Valentin de Matos's winning submission, described as a software factory with multiple portals.
A large language model used by developers in the challenge, often in combination with other models.
A large language model used by the AI judge to test applications and by one of the finalists for development.
One of the applications that benefited from the AI judge's feedback for incremental improvement.
A team that used separate Cloud and Codex workflows, with each requiring verification from the other.
Marco Kramer's core function, described as a clone of ChatGPT's knowledge base.
The existing event management software that the company was considering replacing, costing $43K/year.
An example of an application where Cloud planned and reviewed, while Codex wrote the code.
Mentioned as an example of high-end enterprise software, contrasting with the mid-level software the challenge addressed.
Marco Kramer's company or project, associated with his winning submission.
A company that organizes AI industry conferences and was looking for event management software.
More from Latent Space
View all 263 summaries
82 minRobots That Predict the Future: Dyna-2 & Physical AI — Jason Ma, Dyna Robotics
33 minWhy AlphaFold Didn't Solve Protein Folding — Pushmeet Kohli, Google DeepMind & Sal Candido, Biohub
58 minWhich GPU Clouds Are Actually Good? | ClusterMAX 3.0
104 minRecursive Language Models — Alex Zhang, MIT PhD
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free