# AI crawler tracking

Count visits from ChatGPT, Claude, Perplexity and other AI crawlers in three steps: create a bot token, add a few lines to your server, check a hit arrives.

AI assistants read your pages without running JavaScript, so the ProductLense script never sees them. To count them, your server tells ProductLense each time a crawler fetches a page. ProductLense sorts every visit into one of three kinds:

| Kind | Meaning | Examples |
|---|---|---|
| Answers | A live fetch made for one person's question. The closest thing to a reader. | `ChatGPT-User`, `Claude-User`, `Perplexity-User` |
| Indexing | A crawl that builds the search index AI answers cite | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot` |
| Training | A crawl that collects model training data | `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended` |

Crawler hits are stored without IP addresses and count against a separate allowance of 100,000 hits a month on every plan, never against your events.

Setting it up takes three steps and about five minutes:

1. [Create a bot token](#step-1-create-a-bot-token) in the dashboard.
2. [Add a few lines to your server](#step-2-add-it-to-your-server), for whatever your site runs on.
3. [Check the first hit arrives](#step-3-check-it-works).

Your pages are served exactly as before. Only crawler requests are reported, after or beside the response, and a slow or unavailable ProductLense never slows your site down.

## Step 1: Create a bot token

1. In the ProductLense dashboard, open your site and go to **Settings → AI crawlers**.
2. Click **Create token**. You need to be the site's owner or an admin.
3. Copy the token from the window that opens. It starts with `plbot_`.
4. Store it as an environment variable named `PL_BOT_TOKEN` where your site runs. See [where to put the token](#where-to-put-the-token).

The full token is shown only once; afterwards the dashboard shows just its first characters. If you lose it, click **Rotate token**: you get a new one, and the old one stops working at once, so update your server too. The token belongs to one site and only lets its holder add crawler hits, but keep it out of your code and out of git.

## Step 2: Add it to your server

Pick the row that matches how your site is served:

| Your site runs on | Use |
|---|---|
| Next.js (Vercel or anywhere) | [The npm package](#javascript-and-typescript) |
| Express, Hono, Cloudflare Workers, Cloudflare Pages | [The npm package](#javascript-and-typescript) |
| Another JavaScript server: SvelteKit, Nuxt, Astro, Remix, Fastify, Koa | [The npm package, one call](#any-other-javascript-server) |
| Django, Flask or FastAPI | [Python](#python-django-flask-fastapi) |
| Ruby on Rails, or any Rack app | [Ruby](#ruby-on-rails-and-rack) |
| WordPress or Laravel | [PHP](#php-wordpress-laravel) |
| Netlify | [Netlify Edge Function](#netlify) |
| Your own server behind Nginx | [Nginx](#nginx) |
| Webflow, Framer, Wix, Squarespace, Shopify, Ghost(Pro), GitHub Pages or another builder | [Put Cloudflare in front](#site-builders-and-static-hosts) |

Every option checks the user agent on your server first, so human visitors never cause a request to ProductLense. ProductLense checks every hit again on its side and drops anything it does not recognise.

### JavaScript and TypeScript

Install the package:

```bash tab="npm"
npm install @productlense/ai-crawl
```

```bash tab="pnpm"
pnpm add @productlense/ai-crawl
```

```bash tab="yarn"
yarn add @productlense/ai-crawl
```

Then add the lines for your framework:

```ts tab="Next.js" title="middleware.ts"
import { withProductLenseCrawlers } from '@productlense/ai-crawl/next';

export const middleware = withProductLenseCrawlers({ token: process.env.PL_BOT_TOKEN });

// Already have a middleware? Pass it as the second argument:
// export const middleware = withProductLenseCrawlers({ token: process.env.PL_BOT_TOKEN }, yourMiddleware);
```

```js tab="Express" title="server.js"
import { productLenseCrawlers } from '@productlense/ai-crawl/express';

// before your routes
app.use(productLenseCrawlers({ token: process.env.PL_BOT_TOKEN }));
```

```ts tab="Hono" title="src/index.ts"
import { productLenseCrawlers } from '@productlense/ai-crawl/hono';

app.use('*', productLenseCrawlers((env) => ({ token: env.PL_BOT_TOKEN })));
```

```ts tab="Cloudflare Workers" title="src/index.ts"
import { withProductLenseCrawlers } from '@productlense/ai-crawl/cloudflare';

export default withProductLenseCrawlers(
  { fetch: (request) => fetch(request) }, // or your existing handler
  (env) => ({ token: env.PL_BOT_TOKEN }),
);
```

```ts tab="Cloudflare Pages" title="functions/_middleware.ts"
import { productLensePagesMiddleware } from '@productlense/ai-crawl/cloudflare';

export const onRequest = productLensePagesMiddleware((env) => ({ token: env.PL_BOT_TOKEN }));

// Already have a _middleware.ts? Run both, the crawler one first:
// export const onRequest = [productLensePagesMiddleware((env) => ({ token: env.PL_BOT_TOKEN })), yourMiddleware];
```

The same lines are under **Settings → AI crawlers → Install**, with a copy button. Next.js middleware runs before the page renders, so Next.js hits arrive without an HTTP status; the other adapters send the status your server answered with.

### Any other JavaScript server

Call `reportCrawl` once per request, after you have the response. It checks the user agent itself, so it does nothing for people, and it never throws.

```ts title="SvelteKit: src/hooks.server.ts"
import { reportCrawl } from '@productlense/ai-crawl';
import { env } from '$env/dynamic/private';

export async function handle({ event, resolve }) {
  const response = await resolve(event);
  const report = reportCrawl(
    { token: env.PL_BOT_TOKEN },
    { path: event.url.pathname, userAgent: event.request.headers.get('user-agent'), status: response.status },
  );
  // On Cloudflare, keep the report alive after the response is sent (ctx, or context on older adapters)
  const ctx = event.platform?.ctx ?? event.platform?.context;
  ctx?.waitUntil?.(report);
  return response;
}
```

The report runs in the background, so the page goes out without waiting for it. On a long-running Node server that is all you need. On serverless hosts that stop work once the response is sent, hand the promise to the platform's `waitUntil`, as above for Cloudflare. Where there is none, `await` it before returning: that delays only crawlers, never people, and by at most 3 seconds.

### Python (Django, Flask, FastAPI)

Save this file in your project. It uses only the standard library and sends in a background thread, so the response is never delayed.

```python title="productlense_crawl.py"
"""Reports AI crawler visits to ProductLense. Needs only the standard library."""
import json
import os
import re
import threading
import urllib.request
from urllib.parse import urlsplit

ENDPOINT = "https://api.productlense.app/api/crawl"
AI_BOTS = re.compile(r"ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User|DuckAssistBot|Meta-ExternalFetcher|Gemini-Deep-Research|cohere-ai|OAI-SearchBot|Claude-SearchBot|PerplexityBot|Googlebot|Bingbot|Applebot|DuckDuckBot|YouBot|Bravebot|GPTBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|Google-Extended|GoogleOther|Bytespider|Meta-ExternalAgent|FacebookBot|Amazonbot|cohere-training-data-crawler|Diffbot|AI2Bot|ImagesiftBot|Omgilibot|PanguBot|Timpibot", re.IGNORECASE)


def _send(body, token):
    request = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(body).encode(),
        method="POST",
        headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
    )
    try:
        urllib.request.urlopen(request, timeout=2).close()
    except Exception:
        pass  # never let ProductLense affect your site


def report_crawl(path, user_agent, status=None):
    """Call once per request. Does nothing for people; for crawlers it sends in the background."""
    token = os.environ.get("PL_BOT_TOKEN")
    if not token or not user_agent or not AI_BOTS.search(user_agent[:1024]):
        return
    body = {"path": urlsplit(path).path or "/", "ua": user_agent}
    if status:
        body["status"] = status
    threading.Thread(target=_send, args=(body, token), daemon=True).start()
```

Then call it from your framework. In Django, also add `"yourapp.middleware.ProductLenseCrawlers"` to `MIDDLEWARE` in `settings.py`.

```python tab="Django" title="yourapp/middleware.py"
from .productlense_crawl import report_crawl


class ProductLenseCrawlers:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        report_crawl(request.path, request.META.get("HTTP_USER_AGENT", ""), response.status_code)
        return response
```

```python tab="Flask" title="app.py"
from flask import request
from productlense_crawl import report_crawl


@app.after_request
def productlense_crawlers(response):
    report_crawl(request.path, request.headers.get("User-Agent", ""), response.status_code)
    return response
```

```python tab="FastAPI" title="main.py"
from fastapi import Request
from productlense_crawl import report_crawl


@app.middleware("http")
async def productlense_crawlers(request: Request, call_next):
    response = await call_next(request)
    report_crawl(request.url.path, request.headers.get("user-agent", ""), response.status_code)
    return response
```

### Ruby on Rails and Rack

A Rack middleware, so it works in Rails, Sinatra and any other Rack app:

```ruby title="lib/productlense_crawlers.rb"
require "json"
require "net/http"

class ProductLenseCrawlers # Rack middleware: reports AI crawlers after your app answers
  ENDPOINT = URI("https://api.productlense.app/api/crawl")
  AI_BOTS = /ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User|DuckAssistBot|Meta-ExternalFetcher|Gemini-Deep-Research|cohere-ai|OAI-SearchBot|Claude-SearchBot|PerplexityBot|Googlebot|Bingbot|Applebot|DuckDuckBot|YouBot|Bravebot|GPTBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|Google-Extended|GoogleOther|Bytespider|Meta-ExternalAgent|FacebookBot|Amazonbot|cohere-training-data-crawler|Diffbot|AI2Bot|ImagesiftBot|Omgilibot|PanguBot|Timpibot/i

  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, body = @app.call(env)
    report(env, status)
    [status, headers, body]
  end

  private

  def report(env, status)
    token = ENV["PL_BOT_TOKEN"]
    ua = env["HTTP_USER_AGENT"].to_s
    return if token.to_s.empty? || !AI_BOTS.match?(ua[0, 1024])

    payload = { path: env["PATH_INFO"].to_s.empty? ? "/" : env["PATH_INFO"], ua: ua, status: status.to_i }.to_json
    Thread.new do
      Net::HTTP.start(ENDPOINT.host, ENDPOINT.port, use_ssl: true, open_timeout: 2, read_timeout: 2) do |http|
        http.post(ENDPOINT.path, payload, "Authorization" => "Bearer #{token}", "Content-Type" => "application/json")
      end
    rescue StandardError
      nil # never let ProductLense affect your site
    end
  rescue StandardError
    nil
  end
end
```

In Rails, load it and add it to the middleware stack in `config/application.rb`:

```ruby title="config/application.rb"
require_relative "../lib/productlense_crawlers"

module YourApp
  class Application < Rails::Application
    config.middleware.use ProductLenseCrawlers
  end
end
```

In any other Rack app, add `use ProductLenseCrawlers` to `config.ru`.

### PHP (WordPress, Laravel)

```php tab="WordPress" title="wp-content/mu-plugins/productlense-crawlers.php"
<?php
/**
 * Plugin Name: ProductLense AI crawlers
 * Description: Reports AI crawler visits to ProductLense.
 */
add_action('shutdown', function () {
    $token = defined('PL_BOT_TOKEN') ? PL_BOT_TOKEN : getenv('PL_BOT_TOKEN');
    $ua = isset($_SERVER['HTTP_USER_AGENT']) ? (string) $_SERVER['HTTP_USER_AGENT'] : '';
    if (!$token || !preg_match('/ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User|DuckAssistBot|Meta-ExternalFetcher|Gemini-Deep-Research|cohere-ai|OAI-SearchBot|Claude-SearchBot|PerplexityBot|Googlebot|Bingbot|Applebot|DuckDuckBot|YouBot|Bravebot|GPTBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|Google-Extended|GoogleOther|Bytespider|Meta-ExternalAgent|FacebookBot|Amazonbot|cohere-training-data-crawler|Diffbot|AI2Bot|ImagesiftBot|Omgilibot|PanguBot|Timpibot/i', substr($ua, 0, 1024))) {
        return;
    }
    $path = parse_url(isset($_SERVER['REQUEST_URI']) ? $_SERVER['REQUEST_URI'] : '/', PHP_URL_PATH);
    $status = http_response_code();
    if (function_exists('fastcgi_finish_request')) {
        fastcgi_finish_request(); // the crawler gets its page before the report is sent
    }
    wp_remote_post('https://api.productlense.app/api/crawl', array(
        'timeout'  => 2,
        'blocking' => false,
        'headers'  => array('Authorization' => 'Bearer ' . $token, 'Content-Type' => 'application/json'),
        'body'     => wp_json_encode(array('path' => $path ? $path : '/', 'ua' => $ua, 'status' => $status ? $status : 200)),
    ));
});
```

```php tab="Laravel" title="app/Http/Middleware/ProductLenseCrawlers.php"
<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Http;
use Symfony\Component\HttpFoundation\Response;

class ProductLenseCrawlers
{
    private const AI_BOTS = '/ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User|DuckAssistBot|Meta-ExternalFetcher|Gemini-Deep-Research|cohere-ai|OAI-SearchBot|Claude-SearchBot|PerplexityBot|Googlebot|Bingbot|Applebot|DuckDuckBot|YouBot|Bravebot|GPTBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|Google-Extended|GoogleOther|Bytespider|Meta-ExternalAgent|FacebookBot|Amazonbot|cohere-training-data-crawler|Diffbot|AI2Bot|ImagesiftBot|Omgilibot|PanguBot|Timpibot/i';

    public function handle(Request $request, Closure $next): Response
    {
        return $next($request);
    }

    // Laravel calls this after the response has been sent.
    public function terminate(Request $request, Response $response): void
    {
        $token = config('services.productlense.bot_token');
        $ua = (string) $request->userAgent();
        if (! $token || ! preg_match(self::AI_BOTS, substr($ua, 0, 1024))) {
            return;
        }
        try {
            Http::timeout(2)->withToken($token)->post('https://api.productlense.app/api/crawl', [
                'path' => '/'.ltrim($request->path(), '/'),
                'ua' => $ua,
                'status' => $response->getStatusCode(),
            ]);
        } catch (\Throwable) {
            // never let ProductLense affect your site
        }
    }
}
```

**WordPress:** create the `mu-plugins` folder inside `wp-content` if it is not there; files in it load automatically, with nothing to activate. Put the token in `wp-config.php`, above the "That's all, stop editing!" line: `define('PL_BOT_TOKEN', 'plbot_…');`. Page caches that answer without running PHP (WP Rocket, LiteSpeed Cache, Cloudflare APO, Varnish) also skip this code, so crawler visits to cached pages are missed. If you use one, add the report at the cache instead: [Nginx](#nginx) or a [Cloudflare Worker](#site-builders-and-static-hosts).

**Laravel:** add `'productlense' => ['bot_token' => env('PL_BOT_TOKEN')]` to `config/services.php`, then register the middleware globally: on Laravel 11 and later, `$middleware->append(\App\Http\Middleware\ProductLenseCrawlers::class);` inside `withMiddleware` in `bootstrap/app.php`; on Laravel 10, add it to `$middleware` in `app/Http/Kernel.php`.

### Netlify

An Edge Function runs in front of every page, static or not:

```ts title="netlify/edge-functions/productlense-crawlers.ts"
import type { Config, Context } from '@netlify/edge-functions';

const AI_BOTS = /ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User|DuckAssistBot|Meta-ExternalFetcher|Gemini-Deep-Research|cohere-ai|OAI-SearchBot|Claude-SearchBot|PerplexityBot|Googlebot|Bingbot|Applebot|DuckDuckBot|YouBot|Bravebot|GPTBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|Google-Extended|GoogleOther|Bytespider|Meta-ExternalAgent|FacebookBot|Amazonbot|cohere-training-data-crawler|Diffbot|AI2Bot|ImagesiftBot|Omgilibot|PanguBot|Timpibot/i;

export default async (request: Request, context: Context) => {
  const response = await context.next();
  const ua = request.headers.get('user-agent') ?? '';
  const token = Netlify.env.get('PL_BOT_TOKEN');
  if (token && AI_BOTS.test(ua.slice(0, 1024))) {
    const report = fetch('https://api.productlense.app/api/crawl', {
      method: 'POST',
      headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
      body: JSON.stringify({ path: new URL(request.url).pathname, ua, status: response.status }),
    }).catch(() => {});
    context.waitUntil?.(report); // keeps the report alive after the response
  }
  return response;
};

export const config: Config = {
  path: '/*',
  excludedPath: ['/*.css', '/*.js', '/*.png', '/*.jpg', '/*.svg', '/*.webp', '/*.woff2', '/*.ico'],
};
```

### Nginx

If Nginx serves your site, or sits in front of your app, it can report crawlers on its own, for any language behind it:

The `map` goes in the `http` block, and both `location` blocks in your `server` block:

```nginx title="nginx.conf"
map $http_user_agent $pl_ai_bot {
    default 0;
    "~*(ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User|DuckAssistBot|Meta-ExternalFetcher|Gemini-Deep-Research|cohere-ai|OAI-SearchBot|Claude-SearchBot|PerplexityBot|Googlebot|Bingbot|Applebot|DuckDuckBot|YouBot|Bravebot|GPTBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|Google-Extended|GoogleOther|Bytespider|Meta-ExternalAgent|FacebookBot|Amazonbot|cohere-training-data-crawler|Diffbot|AI2Bot|ImagesiftBot|Omgilibot|PanguBot|Timpibot)" 1;
}

location / {
    mirror /_pl_crawl;
    mirror_request_body off;
    # ...your existing configuration
}

location = /_pl_crawl {
    internal;
    if ($pl_ai_bot = 0) { return 204; }
    proxy_method POST;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header Authorization "Bearer plbot_YOUR_TOKEN";
    proxy_set_header X-PL-Path $request_uri;
    proxy_set_header User-Agent $http_user_agent;
    proxy_connect_timeout 1s;
    proxy_read_timeout 2s;
    proxy_ssl_server_name on;
    proxy_pass https://api.productlense.app/api/crawl;
}
```

Replace `plbot_YOUR_TOKEN` with your token, then run `nginx -t` and reload. Nginx cannot read environment variables here, so keep this file out of git.

### Site builders and static hosts

Webflow, Framer, Wix, Squarespace, Shopify, Ghost(Pro), Carrd, GitHub Pages and similar hosts do not let you run code on their servers, so they cannot report crawlers themselves. The way to count crawlers on them is to put Cloudflare's free plan in front of your domain and add a small Worker:

1. Add your domain to Cloudflare and switch its DNS to Cloudflare. Some builders do not support a proxied domain, so check your builder's help pages first.
2. Make sure the DNS record for your site is **Proxied** (orange cloud).
3. In Cloudflare, go to **Workers & Pages → Create → Worker**, paste the code below and deploy.
4. In the Worker's **Settings → Variables and Secrets**, add `PL_BOT_TOKEN` as a **Secret**.
5. In **Settings → Domains & Routes**, add a route for `yourdomain.com/*` (and `www.yourdomain.com/*` if you use it).

```js title="worker.js"
const AI_BOTS = /ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User|DuckAssistBot|Meta-ExternalFetcher|Gemini-Deep-Research|cohere-ai|OAI-SearchBot|Claude-SearchBot|PerplexityBot|Googlebot|Bingbot|Applebot|DuckDuckBot|YouBot|Bravebot|GPTBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|Google-Extended|GoogleOther|Bytespider|Meta-ExternalAgent|FacebookBot|Amazonbot|cohere-training-data-crawler|Diffbot|AI2Bot|ImagesiftBot|Omgilibot|PanguBot|Timpibot/i;

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request); // your site, unchanged
    const ua = request.headers.get('user-agent') || '';
    if (env.PL_BOT_TOKEN && AI_BOTS.test(ua.slice(0, 1024))) {
      ctx.waitUntil(
        fetch('https://api.productlense.app/api/crawl', {
          method: 'POST',
          headers: { Authorization: `Bearer ${env.PL_BOT_TOKEN}`, 'Content-Type': 'application/json' },
          body: JSON.stringify({ path: new URL(request.url).pathname, ua, status: response.status }),
        }).catch(() => {}),
      );
    }
    return response;
  },
};
```

The Worker passes every request through to your site as before. Cloudflare's free plan includes 100,000 Worker requests a day.

### Where to put the token

| Where your site runs | Where `PL_BOT_TOKEN` goes |
|---|---|
| Vercel | Project → **Settings → Environment Variables**, then redeploy |
| Netlify | Site → **Site configuration → Environment variables**, then redeploy |
| Cloudflare Workers or Pages | **Settings → Variables and Secrets**, type **Secret**; or `npx wrangler secret put PL_BOT_TOKEN` (Workers) or `npx wrangler pages secret put PL_BOT_TOKEN` (Pages) |
| Render, Railway, Heroku, Fly.io | The service's environment variables or config vars (`fly secrets set PL_BOT_TOKEN=…` on Fly.io) |
| Docker or your own server | An environment variable for the process, from your `.env` file, `docker run -e`, or a systemd `Environment=` line |
| WordPress | `define('PL_BOT_TOKEN', 'plbot_…');` in `wp-config.php` |
| Nginx | In the config file itself, as in the example |

Restart or redeploy after adding it: every option above sends nothing while the token is missing.

## Step 3: Check it works

1. Request one of your pages as a crawler, from any terminal:
   ```bash
   curl -A "ChatGPT-User/1.0" https://yourdomain.com/
   ```
2. Open the **Overview** in the ProductLense dashboard and scroll to the crawler card at the bottom. Within a minute the hit appears there under **AI answers**.

If nothing shows up after a few minutes:

| What you see | What to do |
|---|---|
| Nothing at all | Check `PL_BOT_TOKEN` is set where the site actually runs, then redeploy or restart. Every snippet stays silent without it. |
| Still nothing | Test the token on its own (this adds one test hit): `curl -X POST https://api.productlense.app/api/crawl -H "Authorization: Bearer $PL_BOT_TOKEN" -H "Content-Type: application/json" -d '{"path":"/test","ua":"GPTBot/1.2"}'`. `{"accepted":1,...}` means the token works and the problem is in your server code; `401` means the token is wrong or was rotated. |
| `"dropped": 1` | ProductLense did not recognise the user agent. Use one from the table at the top. |
| Hits on some pages only | A cache or CDN answered the others without running your code. Report at the cache: [Nginx](#nginx) or a [Cloudflare Worker](#site-builders-and-static-hosts). |

## What is sent

For each crawler request: the path (never the query string), the crawler's user agent and, where your server knows it, the HTTP status. Never cookies, other headers, or visitor IP addresses. Requests from people are never sent at all.

## The request

To build your own integration, `POST https://api.productlense.app/api/crawl` with `Authorization: Bearer plbot_…` and a JSON body:

| Field | Required | Meaning |
|---|---|---|
| `path` | no | The path served, or a full `http(s)` URL. Query and fragment are dropped; empty means `/`. At most 512 bytes are kept. |
| `ua` | yes | The crawler's user agent, unchanged |
| `ts` | no | RFC 3339 time of the request. Times more than a day old or a minute ahead are replaced by the time of receipt. |
| `status` | no | The HTTP status your server answered with |

Send one hit, `{"hits": [...]}` or a JSON array of up to 50 hits; the body is limited to 16 KB. Without a body, the path is read from the `X-PL-Path` header and the user agent from `User-Agent`, as in the Nginx example.

| Status | Meaning |
|---|---|
| `202` | `{"accepted": n, "dropped": n}`. Hits from browsers and unknown agents, bad paths and hits over the allowance are dropped, not refused, so never retry them. |
| `401` | Missing, malformed or replaced token |
| `429` | Too many requests from one server address; wait for `Retry-After` |
| `503` | ProductLense is briefly busy; retry after `Retry-After` seconds, or skip |

Check the user agent before you send, as every example above does, so people's visits never leave your server. The pattern only keeps human traffic away from ProductLense; an out-of-date pattern costs nothing but a missed crawler.