International Search Engines

This Yandex bot collects information in order to be able to generate previews with the aim of helping their users answer questions from within the search results.

Yandex Vertical search robot.

The Yandex sitelinks “fetcher” used for checking the availability of the pages detected as sitelinks.

The Yandex.Video indexing robot, searches for videos across your site that it can index. This bot needs to be specifically disallowed by robots.txt by referencing the user-agent.

The Yandex.Video robot with the sole purpose of indexing video clips for the Yandex video site.

Yandex bot which makes a snapshot of a page. To avoid being unintentionally blocked by the site owners, this bot may igore the robots.txt directives designed for random robots (User-agent: *. You can stop it by disallowing the agent YandexScreenshotBot.

The Yandex mobile devices robot deployed to support the needs of mobile users.

The Yandex Multimedia data indexer bot.

The Yandex Resource Renderer loads resources to render the page with JavaScript. It will ignore instructions in robots.txt if the HTML page on which these resources are located is accessible to the Yandex robot.The robot does not access resources if HTML pages where these resources are used are prohibited in robots.txt.

The Yandex News robot.

The Yandex robot that validates markup submitted through the Structured data validator form.

The Yandex robot detects site mirrors.

The Yandex.Images indexing robot.

Yandex bot YandexDirectFetcher downloads ad landing pages to check their availability and content, so that ads can be placed in the Yandex search results and on partner sites. To avoid being unintentionally blocked by the site owners, this bot may ignore the robots.txt directives designed for random robots (User-agent: *.) You can stop it by specifically disallowing the agent YandexDirectDyn

This is the blog indexing bot from Yandex. If you run a blog and want your blog comments indexing by Yandex, select this one.

Crawler of Yahoo! in Japan

Chinese Search Engine

Seekport is an internet search engine. This search engine is operated by SISTRIX, a platform intelligence provider from Germany.The search engine is a public, free and independent alternative to Google. Seekport does not store user data and does not profile users. Seekport is also operated without advertising and has no conflicts of interest in the display of search results.The Seekport Bot crawls the public internet to find and update content for Seekport's search index.

So.com is a major Search Engine in China, this is another of its crawlers.

This newer spider will not only crawl the HTML of the web pages but also render the page with other elements including CSS, JS, and images to help Baidu better understand the content of the page and provide more meaningful results.

This newer spider will not only crawl the HTML of the web pages but also render the page with other elements including CSS, JS, and images to help Baidu better understand the content of the page and provide more meaningful results.