# Audit AI crawler access through robots, logs and WAF rules. - working worksheet

Reviewed alongside the Eli guide: https://hireeli.io/resources/audit-ai-crawler-access-logs-waf

## Decision to make

Verify that search and answer crawlers can reach public commercial pages

Owner: ____________________
Review date: ____________________
Canonical page or destination: ____________________

## Buyer questions

- [ ] How do you verify AI crawler access?
- [ ] Is an allowed robots.txt rule enough?
- [ ] Which bots should be tested separately?

## Evidence-led work plan

### 1. Check declared access

Review bot-specific and wildcard rules for public routes.

Evidence or artifact: ____________________
Status: [ ] not started  [ ] working  [ ] verified  [ ] blocked

### 2. Probe the delivery layer

A CDN or WAF can block documented crawlers even when robots allows them.

Evidence or artifact: ____________________
Status: [ ] not started  [ ] working  [ ] verified  [ ] blocked

### 3. Read the server evidence

Store the user agent, path, response, time and verified IP state when available.

Evidence or artifact: ____________________
Status: [ ] not started  [ ] working  [ ] verified  [ ] blocked

### 4. Keep training and search controls distinct

Crawler names and purposes differ by provider.

Evidence or artifact: ____________________
Status: [ ] not started  [ ] working  [ ] verified  [ ] blocked

## Source ledger

- [ ] OpenAI: publishers and developers FAQ: https://help.openai.com/en/articles/12627856-publishers-and-developers-faq
- [ ] Perplexity: crawler documentation: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- [ ] Google Search Central: AI features and your website: https://developers.google.com/search/docs/appearance/ai-features

## Measurement boundary

Baseline date and exact context: ____________________
7-day observation: ____________________
14-day observation: ____________________
30-day observation: ____________________
Known limitations: ____________________

Keep technical eligibility, citations, recommendations, visits, leads and revenue as separate evidence. This worksheet does not guarantee an external ranking or commercial result.