can AI assistants read your site?
check whether ChatGPT, Claude, Perplexity, Gemini, Copilot and Apple Intelligence are allowed to read your site, straight from your robots.txt, plus whether you have an llms.txt and a sitemap.
free · no sign-up · nothing stored
why it matters now
more and more people ask an AI assistant instead of searching. when someone asks "what's a good tool for…", the assistant can only recommend what its crawler was allowed to read. a single line in robots.txt, often added by a template or a security plugin, can quietly keep you out of every answer.
there are two kinds of AI crawlers. training crawlers like GPTBot and ClaudeBot collect pages to teach models. search and user crawlers like OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User fetch pages to answer questions right now, with a link back to you. you can block the first kind and still allow the second.
how to allow a crawler
in robots.txt, a crawler follows the group that names it, or the "*" group if none does. to make sure ChatGPT search can read everything:
User-agent: OAI-SearchBot Allow: /
then give assistants a summary of your site with an llms.txt, and something worth quoting: dated, specific pages about what your product does. commiToday turns every change you ship into one.
questions
+which crawlers does it check?
GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User, Google-Extended and Googlebot, Bingbot, Applebot-Extended, Meta-ExternalAgent and CCBot.
+does allowed in robots.txt mean they'll show my site?
it means they may read it. whether they quote it depends on whether your pages answer what people ask. robots.txt is the door; your content is the reason to walk in.
+my site has no robots.txt. is that bad?
no. without one, every crawler is allowed. adding one is still worth it, to point crawlers to your sitemap.
+does it check Cloudflare's AI bot blocking?
no. it reads robots.txt. if you turned on a firewall setting that blocks AI bots, they're blocked even when robots.txt allows them; check that in your hosting settings.