robots.txt
Software / AppMentioned in 2 videos
A file used to instruct web crawlers, proposed as an analogy for an 'llm.txt' file to guide LLM behavior on a domain.
Videos Mentioning robots.txt
E125: SpaceX launch, Fox News settlement, "Zombie-corn" exodus to AI, late-stage implosion
All-In Podcast
A file used by websites to instruct web robots about which areas of the site should not be processed or crawled, mentioned in context of AI data scraping.
Andrej Karpathy: Software Is Changing (Again)
Y Combinator
A file used to instruct web crawlers, proposed as an analogy for an 'llm.txt' file to guide LLM behavior on a domain.