AI Workflows

See If ChatGPT, Claude and Perplexity Crawl Your Site (Log Filter)

One User-Agent filter for your server or CDN logs that shows which pages ChatGPT, Claude and Perplexity bots visit, plus what to fix.

Platform rules change often. Always confirm on the linked source before applying.

See which pages ChatGPT, Claude and Perplexity actually visit, straight from your logs.

sh
OAI-SearchBot|ChatGPT-User|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User
OAI-SearchBotChatGPTChatGPT-UserChatGPTClaude-SearchBotClaudeClaude-UserClaudePerplexityBotPerplexityPerplexity-UserPerplexity
1Open logs
Your server or CDN access logs.
2Filter User-Agent
Paste the filter below.
3Sort
By URL, then request count.
4Fix gaps
Key page at 0 visits? Check access before writing more.
What to look for
  • Which pages AI engines crawl, and which they barely touch.
  • Whether product and comparison pages get reached.
  • Old pages getting unexpected AI attention.
  • An important page with 0 visits while weaker pages get crawled a lot: check robots.txt, internal links, rendering and accessibility first, before writing more articles.

These six User-Agent tokens match Cloudflare's AI Crawl Control bot reference.

Source: tip by @wespreadjam (Jam, spreadjam.com, an AEO tool for founders).

Check which platforms you can join today →

The Starbelo 5 · weekly

Subscribe to the Starbelo newsletter

5 AI money wins in your inbox every week. Free, unsubscribe anytime.

Weekly AI wins. Unsubscribe anytime. See Privacy.

Top creator toolsAffiliate links

Keep Exploring

Advertise hereReach creators starting from zero.
Top creator toolsAffiliate links
vidIQResearchTryOpus ClipClipsTryStarbelo 50Playbook$9ElevenLabsVoiceTry