Know when AI assistants read your pages, and which ones.

AI crawlers fetch your HTML without running JavaScript, so a browser script never sees them. A few lines of middleware send ProductLense the path, the status code and the user agent of each request, and ProductLense keeps only the requests from known AI crawlers, dropping everything else without storing it.

User agents from OpenAI, Anthropic, Google, Perplexity, Microsoft, Apple, Meta, Amazon, ByteDance, Common Crawl and others. Each is sorted into answering a user, such as ChatGPT-User fetching a page for a question, indexing for AI search, or training. Unknown user agents are dropped rather than guessed, so the counts stay honest.

No. The middleware sends each report after your response has gone out, gives up after 250 milliseconds at most, and never throws, so a slow or unreachable ProductLense cannot affect your pages. The package has no runtime dependencies, so it adds almost nothing to your build or your deploy size.

Every plan includes 100,000 crawler hits a month, counted separately from your events, so heavy crawling never uses up your analytics allowance. The AI crawlers tab in settings shows this month’s usage against the allowance, next to the crawler token and the install snippets for your framework.

The @productlense/ai-crawl package has adapters for Next.js middleware, Cloudflare Workers and Pages, Express and Hono. For anything else, a single nginx mirror line or a plain HTTP request from your server does the same job. Each site gets its own crawler token, which you can rotate from settings at any time.