MenuBots and filtering

Track

Bots and filtering

Which automated traffic ProductLense drops before it is stored, why it never counts toward your plan, and what still gets through.

ProductLense filters automated traffic twice: the script does not run its sends in automated browsers, and the server drops requests whose user agent looks automated. Dropped requests still answer 204, so a bot learns nothing, and they never count toward your monthly events.

In the browser

The script sends nothing when:

  • navigator.webdriver is true (Selenium, Playwright, Puppeteer and most headless browsers set it).
  • The hostname is localhost, unless the script tag has data-dev="true".
  • window.__pl_disable is set before the script runs.

On the server

An event is treated as a bot when the user agent is empty, when the user-agent parser recognises it as a bot, or when its first 512 characters contain any of these markers, ignoring case:

KindMarkers
Genericbot, crawl, spider, slurp, preview, fetch, monitor
Headless and automationheadless, phantomjs, puppeteer, playwright, selenium, webdriver
HTTP clientscurl/, wget/, python-requests, python-urllib, python/, go-http-client, java/, axios/, node-fetch, libwww, httpclient, http_request, scrapy, aiohttp
App clients (see below)electron/, okhttp
Audits and uptimelighthouse, pagespeed, gtmetrix, pingdom, uptimerobot, statuscake, site24x7
SEO crawlersahrefs, semrush, mj12, dotbot, petalbot, bytespider, dataforseo, screaming frog
Link unfurlersfacebookexternalhit, whatsapp, telegram, discord, skypeuripreview, embedly
Search enginesgooglebot, bingbot, yandex, baiduspider, duckduckbot, applebot, ia_archiver

App clients

Electron desktop apps and okhttp (the HTTP client under React Native and most Android apps) are used by real people and by scripts. An event from one of them is counted when it carries the screen width (w) that the ProductLense script reads in the browser, and dropped as a bot without it. A user agent that also matches another marker, such as HeadlessChrome … Electron/28, is always dropped. Counted app visits show as browser Electron (desktop) or Android app (mobile).

Other checks on every request

  • The site id must exist, and the page URL's hostname must be one of the site's hostnames. When a browser request carries an Origin or Referer header, its hostname must be on the list as well.
  • The body must be valid JSON of at most 8 KB with at most 50 events, and every event must pass the field rules. A request with an invalid event is rejected as a whole with 400.

What still gets through

User-agent rules cannot catch a bot that runs a real browser and hides navigator.webdriver. Such traffic usually shows as one-page sessions from a single country and device in a short window. ProductLense has no IP blocklist, because it never stores IP addresses.

    ↑ ↓ to move, Enter to open