Advertising Related Bots

Another bot from Yandex which downloads information about the content of Yandex Advertising network partner sites to identify their topic categories to match relevant advertising.

Yandex bot which downloads the ads' landing pages to check their availability and topic. This is necessary for ad placement in the search results and on the partner sites. When crawling a site, the robot does not use the robots.txt file and ignores the directives set for it.

Quantcast Bot is the name of a web crawler used by Quantcast for advertisement quality assurance and to understand page content for Interest-Based Audiences. Quantcast’s QA technology will not access your website unless Quantcast is trafficking advertisements that click through to your website. Advertisements are checked regularly, so you may see several visits from Quantcastbot while advertisements are being delivered. The bot will only visit the landing page of an advertisement; the bot does not crawl your website content. As the bot follows a click on an advertisement, it doesn’t obey the rules specified in your website’s robots.txt file. This is to ensure that the advertisement landing page meets our guidelines, in addition to the originating advertisement. It will also visit your ads.txt file, if you have one.

PubMatics products provide clients with a full advertising technology stack, allowing clients to optimize Ad performance. Supporting the ads.txt initiative PubMatics crawler reads the file on customers domains.If you subscribe to PubMatics Ad platform then you should allow this bot.

Ad fraud checker, checks for incongruous browsers, data centre generate traffic, bot traffic, and anomalous clicks.

Internet Ad Fraud platform for publishers and social media platforms. Helps prevent click fraud and protect content brand reputation. The IAS Wombles robot captures additional information allowing IAS to analyze their invalid traffic models.

Checks ad quality using NLP. Does not respect Robots.txt

Ads.txt offers a flexible and secure method for publishers and distributors to declare the companies they authorize to sell their digital inventory. To do this, publishers add a lightweight text file to their domain that includes a list of all companies authorized to sell their inventory. Buyers can then crawl the web for a publisher’s ads.txt file and verify that the seller is authorized by the publisher.

The Google Ads system also visits your landing page to evaluate your mobile site (as viewed by mobile devices with full browsers, like Android devices and iPhones). If you have a distinct, mobile-optimized version of your site, we recommend you configure your server to show the mobile-optimized site when the Google Ads mobile User-Agent is detected crawling your site.

This is one of the AdSense crawlers which is used by Google to process and index the content of webpages. The AdSense crawler visits your site to determine its content in order to provide relevant ads. This crawler should only be visiting once a week where it will make a series of requests to index your site. The AdSense crawler is different from the Google crawler, Google state that the crawler indexes by URL. The crawler won't access pages or directories prohibited by a robots.txt file.

The Google Ads mobile bot visits landing pages to evaluate mobile sites (as viewed by mobile devices with full browsers, e,g Android / iPhones). If you have a distinct, mobile-optimized version of your site, it's best to configure your server to show the mobile-optimized site when the Google Ads mobile User-Agent is detected crawling your site.

Facebook Ad bot.

checks page contextuality to increase Ad analytics and performance

Criteo provides a suite of advertising solutions for Retailers, Brands and Publishers. Core technology includes Ad Campaign and Media management, along with AI powered Bidding and Research tools. If your business uses Criteo to monitor your own site, or your parters do then allow this bot. The Criteo Crawler is a software that visits web pages and analyzes its content to serve relevant ads on them, it does this by making one or more GET requests on your website.

AdIdxBot is the crawler used by Bing Ads. AdIdxBot is responsible for crawling ads and following through to websites from those ads for quality control purposes

Ad stats tool to enable customers to gain insight into competitor ad spending