Dev Tools for WebMasters

The Yandex robot that validates markup submitted through the Structured data validator form.

BlogVault is a solution for automating WordPress site backups, allowing their customers to recover their website easily from any sort of site failure. If you are a Wordpress user and have this plugin, allow this crawler.

The Markup Validator is a free service by W3C that helps check the validity of Web documents, it is part of a suite of services from W3C to help web designers and developers build their sites.Most Web documents are written using markup languages, such as HTML or XHTML. These languages are defined by technical specifications, which usually include a machine-readable formal grammar (and vocabulary). The Markup Validator automatically checks the website against the technical standardsThis validator can process documents written in most markup languages. Supported document types include the HTML (through HTML 4.01) and XHTML (1.0 and 1.1) family, MathML, SMIL and SVG (1.0 and 1.1, including the mobile profiles). The Markup Validator can also validate Web documents written with an SGML or XML DTD, provided they use a proper document type declaration.This validator is also An HTML validating system conforming to International Standard ISO/IEC 15445—HyperText Markup Language, and International Standard ISO 8879—Standard Generalized Markup Language (SGML) – which basically means that in addition to W3C recommendations, it can validate according to these ISO standards.If you or anyone in your web development team use these tools, you should allow this bot.

Suite of internet mapping databases for domain, IP and MX records, available as an API or by download for integrating into security and analytics products or data enrichments

Useful set of general purpose API for web developers - includes bulk API based email, tel phone and SMS verification, geocoding, profanity filters, URL and html checkers

A Robots.txt file checker - ensures the robots.txt files syntax is correct on your website

Monsido offer an all in one web analytics, monitoring and automation platform to help their customers deliver a flawless website UX across Web Accessibility, Content Quality Assurance, Data Privacy, Performance and more. If you see this crawler its either because your team, or a partner business who links to your website uses Monsido.

Bulk image resize and caching service hosted with CloudFlare's CDN

IPv6 and IPv4 speed testing and validator. Only uses the bot if the widget is added to your site.

Simple web based WhoIs IP checker

This is the page checker for Google Chrome Lighthouse, as launched from inside the Google ecosystem. Lighthouse is an open-source, automated tool for improving the quality of web pages. You can run it against any web page, public or requiring authentication. It has audits for performance, accessibility, progressive web apps, SEO and more.You can run Lighthouse in Chrome DevTools, from the command line, or as a Node module. You give Lighthouse a URL to audit, it runs a series of audits against the page, and then it generates a report on how well the page did. If you are seeing this, someone is specifically auditing pages on your site.

This bot is part of Facebooks Developer programme tools and will likely operate if you need content fetching for example as part of your developed solution. If you use Facebooks Developer solutions to connect to your website, you should consider allowing this bot.

Suite of DNS tools, does port scanning, and reverse IP lookups. Based in Switzerland

Jetpack is a common and powerful tool for managing your WordPress website, indeed its a pre-installed component. Very much a jack-of-all-trades it supports security, analytics and content management activities for your site. This makes a very small number of POST requests to a specific php page in your WordPress website. If you have a WordPress website you will likely want this bot, if your site is not built on the WordPress platform do not allow it.