Bot Database

Specialist Search Engine

Specialised search engines have crawlers that scrape data for a particular vertical market or sector. Typical applications are for company data monitoring, for example amongst SME companies, government sectors, and other vertical markets. Typically, these are very niche players, and won't add much if any traffic, unless in a very niche sector, where they can add value.

Vendor

Bot Service

Recommendation

Description

logotypelogotype

You

You

Recommended

Not recommended

AI Search Engine - powered by You. AI/ Chat search

logotypelogotype

Wikimedia Foundation

Wikipedia Crawler

Recommended

Not recommended

It's Wikipedia! The COIBot is a bot that tries to track edits that are made by users who may have a conflict of interest.

logotypelogotype

Pipl

PIPL

Recommended

Not recommended

Just when you thought recruitment agents couldn't be more annoying, how about recruiters that use bots to crawl your domains for personal data including location and telephone numbers as well as names and job-titles? PiplBot crawler collects documents from the Web to build a searchable index which powers its People Search database. As part of the crawling, PiplBot does respect robots.txt so you can choose to exclude pages whose content you do not want included in Pipl Search.

logotypelogotype

Majestic

Majestic

Recommended

Not recommended

Majestic is a UK based specialist link intelligence whose platform spreads across 13 languages and over 60 countries to paint a map of the Internet independent of the consumer based search engines. MJ12Bot does not currently cache web content or personal data. Instead it maps the link relationships between websites to build a search engine. Used by many SEO companies to see the underlying patterns of linking strategies. This data is available to technologies and the public, either by searching for a keyword or a website at Majestic. MJ12Bot obeys Robots.txt and Crawl Delay.Because of the way the Majestic Search Engine builds its data the crawler is launched from a varied and distributed locations, indeed users of the service can invoke the crawler locally. Choose this one with care.

logotypelogotype

Hiya

Mr. Number.

Recommended

Not recommended

Searches for telephone numbers and has android & iphone apps to block unwanted calls. Uses a bot to search for the originating telephone numbers.

logotypelogotype

DueDil

Electricmonk

Recommended

Not recommended

Due Diligence tool aimed at tracking SME company records and the people that run them. Has an API and web platform for bulk searches.

logotypelogotype

Dassault Systèmes (3DS)

Exalead Thumbnails

Recommended

Not recommended

Exalead® is a leading search-based application platform provider to business and government. Exalead's worldwide client base includes leading companies such as PricewaterhouseCooper, ViaMichelin, GEFCO, WorldBank and Sanofi Pasteur, and more than 100 million unique users a month use Exalead's technology for search.

logotypelogotype

Dassault Systèmes (3DS)

Exalead

Recommended

Not recommended

Exalead® is the leading search-based application platform provider to business and government. Exalead's worldwide client base includes leading companies such as PricewaterhouseCooper, ViaMichelin, GEFCO, WorldBank and Sanofi Pasteur, and more than 100 million unique users a month use Exalead's technology for search.

logotypelogotype

Dassault Systèmes (3DS)

CloudView

Recommended

Not recommended

Exalead is reshaping the digital content landscape with its platform, Exalead CloudView™, which uses advanced semantic technologies to bring structure, meaning and accessibility to unused or under-used data in the new hybrid enterprise and Web information cloud. CloudView collects data from virtually any source, in any format, and transforms it into structured, pervasive, contextualized building blocks of business information.It will crawl your site to gather and add content to the Exalead business and government search engine.

logotypelogotype

Cliqz

Cliqz

Recommended

Not recommended

Cliqz is a European based, independent, provider of browser extensions allowing users to perform web searches. Like traditional search engines they crawl sites to index content. The Cliqzbot crawler respects Robots.txt

logotypelogotype

Business Seek

BusinessSeek.biz_Spider

Recommended

Not recommended

Business Seeker is a business directory site that scrapes business contacts from other sites to host on its directory.

logotypelogotype

aiHitdata

Company Data Search

Recommended

Not recommended

Specialist search engine for company data monitoring. Hitdata claims to use artificial intelligence/machine learning, to automated the gathering and building up of detailed company information from the web using this bot. Respects robots.txt.

logotypelogotype

AddSearch

AddSearch

Recommended

Not recommended

AddSearch is a specialist search engine designed to be embedded in your own website, to allow relevant and tailored search results to be surfaced to your user base. If you use the AddSearch product you should allow this bot to prevent issues.