# robots.txt for istcabinets.com (In Stock Today Cabinets)
# Draft 2026-09-30. Every AI token below was checked that day against the
# vendor's own crawler documentation. Dropped (no official page): Claude-Web,
# anthropic-ai, cohere-ai, YouBot. Bytespider is blocked by Omer's choice.
#
# Policy (same as usacabinetstore.com): search engines and AI crawlers are all
# allowed, Bytespider is blocked, no Content-Signal lines.
#
# No WooCommerce on this site (/cart/, /checkout/, /shop/ return 404), so no shop rules.
# Attachment pages (about 1,790 URLs in the attachment sitemaps) are NOT blocked here.
# Fix them in Rank Math (redirect attachments to parent + drop the attachment sitemap);
# a robots block would stop Google from seeing that redirect.
#
# How the groups work (RFC 9309 / Google):
# - A bot that matches a named group IGNORES the "*" group, so every group
#   repeats the same rules. When you change a rule, change it in all five groups.
# - Blank lines only between groups. Some parsers end a group at a blank line.
# - To opt out of one category, replace that group's rules with "Disallow: /".
# - Do not use robots.txt to hide pages you want de-indexed: Google must be able
#   to crawl a page to see its redirect, 410 or noindex.

# --- AI training ---
# Crawlers, plus control tokens that crawl nothing themselves
# (Google-Extended, Applebot-Extended).
# Blocking this group does not affect Google Search, Bing or ChatGPT Search.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: AI2Bot
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /search/
Disallow: /*?s=
Disallow: /*&s=
Disallow: /feed/
Disallow: /*feed/

# --- AI search and citation ---
# These bots build the indexes that AI answers cite: ChatGPT Search, Claude,
# Perplexity, Alexa, Meta AI, DuckAssist and Mistral.
# Copilot uses Bingbot, so the "*" group covers it.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /search/
Disallow: /*?s=
Disallow: /*&s=
Disallow: /feed/
Disallow: /*feed/

# --- User-triggered fetches ---
# A person asked an assistant to open a page. Anthropic says Claude-User obeys
# robots.txt. OpenAI, Perplexity, Meta and Amazon say their user agents may not.
# Google-Agent and Google-GeminiNotebook have no robots.txt token at all.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: meta-externalfetcher
User-agent: Amzn-User
User-agent: MistralAI-User
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /search/
Disallow: /*?s=
Disallow: /*&s=
Disallow: /feed/
Disallow: /*feed/

# --- Ad landing-page checks ---
# OAI-AdsBot checks pages submitted as ChatGPT ads (not used for training).
# meta-externalads crawls for Meta's ads and business products.
# AdsBot-Google and Mediapartners-Google are left out on purpose: they ignore
# the "*" group and would only start obeying these Disallow lines if named here.
User-agent: OAI-AdsBot
User-agent: meta-externalads
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /search/
Disallow: /*?s=
Disallow: /*&s=
Disallow: /feed/
Disallow: /*feed/

# --- Blocked ---
# Heavy crawler that brings no visibility in return. ByteDance publishes no
# official crawler page.
User-agent: Bytespider
Disallow: /

# --- Everyone else ---
# Googlebot (plus -Image/-Video/-News), Storebot-Google, GoogleOther,
# Google-CloudVertexBot, Bingbot (which also feeds Copilot), Applebot,
# facebookexternalhit and every other compliant crawler.
# AdsBot-Google and Mediapartners-Google ignore this group by design.
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /search/
Disallow: /*?s=
Disallow: /*&s=
Disallow: /feed/
Disallow: /*feed/

Sitemap: https://istcabinets.com/sitemap_index.xml