Cloudflare crawlers

Cloudflare may crawl or make HTTP requests to your site to make sure its protected and performing properly.

Crawling situations

Specific products

Cloudflare will crawl your site when you have specific products enabled:

• Always Online
  • User-Agent: Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
• Health checks
  • User-Agent: Mozilla/5.0 (compatible; Cloudflare-Healthchecks/1.0; +https://www.cloudflare.com/; healthcheck-id: <HEALTHCHECK_ID>)
  • HEALTHCHECK_ID is a 16-character string associated with the health check ID.
• Load balancing monitors
  • User-Agent: Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: <POOL_ID>)
  • POOL_ID is a 16-character string associated with the load balancing pool ID being monitored.
• Prefetch URLs
  • User-Agent: Mozilla/5.0 (compatible; CloudFlare-Prefetch/0.1; +http://www.cloudflare.com/)
• SSL/TLS recommender
  • User-Agent: Cloudflare-SSLDetector
  • This crawler ignores your robots.txt file unless there are rules explicitly targeting the user agent.
• Security Insights
  • User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/)

Other situations

Cloudflare will also crawl your site in other, specific situations:

• Speed tests
  • User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 PTST/190628.140653
  • Triggered when: You launch a speed test from within the Cloudflare dashboard.
• Support diagnostics:
  • User-Agent: Cloudflare-diagnostics
  • Triggered when: Cloudflare Support Engineers perform error checks and by continuous monitoring used to raise intelligent alerts in the Cloudflare dashboard.
• Custom Hostname validation:
  • User-Agent: Cloudflare Custom Hostname Verification
  • Triggered when: You choose to validate a custom hostname with an HTTP ownership token.