Published:
robots.txt and AI crawlers: what to watch out for
AI providers run different crawlers for indexing, training data, and live fetches. robots.txt determines which of these crawlers can access a site — access is necessary, but it's no guarantee of a later mention.
AI providers run different crawlers for different purposes: indexing for classic search, collecting training data, and user-initiated fetches to answer a specific query.
A robots.txt configuration should deliberately decide which of these crawler categories get access. Always check current provider documentation before making changes, since crawler names and behavior can change.
Allowing a crawler alone doesn't guarantee a mention in generated answers — it's a necessary but not sufficient condition.