r

robots.txt

Tool / ProductMentioned in 1 video

A file used to instruct web crawlers, proposed as an analogy for an 'llm.txt' file to guide LLM behavior on a domain.

Videos Mentioning robots.txt

Andrej Karpathy: Software Is Changing (Again)

Andrej Karpathy: Software Is Changing (Again)

Y Combinator

A file used to instruct web crawlers, proposed as an analogy for an 'llm.txt' file to guide LLM behavior on a domain.