User-agent: * Allow: / Disallow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: Googlebot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: Yeti Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: NaverBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: Bingbot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md # AI answer/search and user-requested retrieval bots. Public pages and the # machine-readable market data are available; internal operating files are not. User-agent: OAI-SearchBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: ChatGPT-User Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: PerplexityBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: Perplexity-User Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: Claude-User Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: Claude-SearchBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md # Gemini grounding and Google AI features. This does not change Google Search # inclusion or ranking; the Googlebot rules above remain the search controls. User-agent: Google-Extended Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md # xAI Grok fetchers — allowed. Unnamed agents use the default group, where the # root AI feed stays public while raw runtime data and internal paths stay blocked. User-agent: GrokBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: xAI-Crawler Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md # Social link-preview crawlers — ALLOWED so Threads / Facebook / Instagram render # rich link cards when we share the site. User-agent: FacebookBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: Meta-ExternalAgent Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md # GPTBot — allowed (2026-08-27). Its crawl also feeds ChatGPT search; # OAI-SearchBot is the link-display fetcher. User-agent: GPTBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md # ClaudeBot — allowed (2026-08-27), same reason as GPTBot (Claude search). # Training-only anthropic-ai stays blocked below. User-agent: ClaudeBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md # Other answer/search agents — ALLOWED explicitly (2026-09-08): Apple (Siri / Apple Intelligence search), DuckDuckGo AI, # You.com and Mistral Le Chat all fetch pages to answer. The default group already allows them; naming them also # opens the raw /data/ path the same way as the other answer engines. User-agent: Applebot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: DuckAssistBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: YouBot Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md User-agent: MistralAI-User Allow: / Allow: /data/ Disallow: /docs/ Disallow: /tools/ Disallow: /social/ Disallow: /HANDOFF.md Disallow: /CLAUDE.md Disallow: /AGENTS.md Disallow: /SECURITY.md # Training-only scrapers — BLOCKED (bulk-harvest for model training, no exposure benefit). User-agent: anthropic-ai Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Amazonbot Disallow: / Sitemap: https://opboxindex.com/sitemap.xml