Are Your Product Pages Accessible to AI Search? Audit the Right Controls

A product page cannot compete for a discovery opportunity if the relevant system cannot use it. But finding a blocked bot does not prove lost sales, and opening access does not put a product into an answer. For an ecommerce leader, the useful question is narrower: are the public pages you want shoppers to find accessible to each destination you care about, under the controls you intended?

Treat this as an access and measurement audit, not an instruction to let every automated request through. Product information, search discovery and traffic performance are connected, but each requires separate evidence.

Start with the commercial decision

Choose a small set of public product and category pages tied to a real priority, such as a category receiving investment or products with reliable availability and pricing. Include the corresponding mobile pages or regional URLs where those are the shopper destinations. Record their current search traffic and downstream activity before changing anything. This makes the audit useful to a VP or director deciding whether a technical fix deserves engineering time.

Keep checkout, account, staging, confidential pricing and other intentionally restricted paths out of scope. Preserve valid exclusions and security controls. The objective is to identify accidental barriers on selected public pages, not to make the whole site crawlable.

Identify the destination before changing a rule

There is no universal split between one training bot and one answer bot. Each provider documents several agents with different jobs, and an allow rule written for one does not carry over to another. Review these destinations separately rather than copying an allow rule from one platform to another.

• OAI-SearchBot is the crawler for ChatGPT search, per OpenAI's crawler documentation. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, but may still appear as navigational links. That exception matters when interpreting a spot check. Allowing OAI-SearchBot does not establish that a product will be selected for an answer.
• GPTBot covers potential foundation-model training. OpenAI says its search and training settings are independent, although it may reuse one crawl when both are allowed. A GPTBot decision is therefore not a substitute for checking OAI-SearchBot access.
• ChatGPT-User handles certain user-initiated actions. It is not OpenAI's automatic search crawler, and robots.txt rules may not apply to user-initiated visits.
• Googlebot and normal Search controls govern crawling for Google Search, including AI Overviews and AI Mode, per Google's guidance for AI features in Search. A supporting link must come from a page indexed and eligible to appear in Google Search with a snippet.
• Google-Extended is a robots.txt control for training future Gemini models and for grounding in specified Gemini and Vertex AI experiences, per Google's crawler reference. It does not control Google Search inclusion and has no separate HTTP request user-agent string. Calling it a training-only crawler would mislead the team reviewing logs.
• PerplexityBot surfaces and links sites in Perplexity's search results, per Perplexity's crawler reference.
• Perplexity-User supports user-requested visits and generally ignores robots.txt.

Audit the pages and the full access path

For each selected URL and destination, inspect the live robots.txt file, relevant user-agent group and path rules. Check the actual product or category URL, not only the home page. If you find a broad restriction, establish which agent it applies to before editing it. Google's robots.txt specification explains how Google's crawlers select and combine matching groups. Do not assume every provider's parser behaves identically.

Next check what reaches the origin. Review CDN and WAF rules, bot challenges, response codes and server logs for the selected paths. Compare observed requests against the provider's published IP information when validating an identity. A request carrying a bot's user-agent text alone is not proof it came from that provider. Perplexity specifically recommends combining IP and user-agent checks for WAF configuration. Avoid a broad bypass that would admit unrelated traffic or expose pages you meant to protect.

For Google, use Search Console's URL Inspection tool to examine indexing and the HTML Googlebot received. Check noindex and snippet controls as well as robots and network access. Google says no special AI file or markup is required for its AI Search features, so adding an llms.txt file is not a replacement for fixing a verified access problem. For another destination, use its documented controls and your own logs rather than projecting Google's eligibility rules onto it.

Approve a scoped change, then measure the right result

Document the finding as a specific URL group, destination, blocking layer and intended policy. If a public page is blocked accidentally, propose the smallest rule or security change that admits verified legitimate requests while retaining training preferences, rate limits and exclusions where appropriate. Have the security and ecommerce owners approve it. A recommendation is not an approved configuration change, and an approved change is not a measured outcome.

After deployment, verify the exact URLs and the relevant bot traffic again. Record whether requests reach the page and receive the intended response. Then look separately for indexing or observable inclusion where the destination offers evidence, and track whether the pages you opened actually show up in AI answers for the shopper questions that matter to you.

For Google, AI-feature traffic is included in Search Console's Web search performance data, not presented here as a guaranteed standalone AI-placement measure. Track relevant landing-page traffic and conversions in your analytics, with appropriate attribution caveats. Do not label an increase as incremental revenue from the rule change without a credible comparison.

Allow time for systems to process changes, but do not promise a recovery date. OpenAI notes that its search systems may take about 24 hours to adjust to a robots.txt update. Google explicitly does not guarantee crawling, indexing or serving. The executive takeaway is a sequence: verify access, decide whether a narrow correction is justified, and evaluate observed discovery and commercial performance afterward. Each step answers a different question.

If your team does not have the time to run this audit alongside everything else, this is work Paz handles for lean ecommerce teams. A Paz expert finds the gaps in your product information and access controls, you approve the fixes, and the platform publishes them and tracks what shows up in AI answers afterwards. Talk to us.