# AI search and training crawlers — allow public content User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: Meta-ExternalAgent Allow: / # Ensure ads crawlers can always fetch the seller declaration. User-agent: AdsBot-Google Allow: /ads.txt User-agent: Googlebot Allow: /ads.txt # All crawlers — block private / auth flows User-agent: * Allow: / Disallow: /account Disallow: /account/ Disallow: /onboarding Disallow: /onboarding/ Disallow: /auth Disallow: /auth/ Disallow: /signin Disallow: /signup Disallow: /password-forgotten Disallow: /password-reset Disallow: /create Disallow: /create/ Disallow: /edit Disallow: /edit/ Disallow: /subscription Disallow: /subscription/ Disallow: /notification-preferences Disallow: /notification-analytics Disallow: /notification-metrics-admin Disallow: /dev/ Sitemap: https://hivefolio.gg/sitemap.xml