Link & URL checkers & tools

The BLEXBot crawler is an automated robot that visits pages to examine and analyse the content and check backlinks, in this sense it is similar to the robots used by the major search engine companies.

An experimental service provided by the W3C organisation to help in better HTML checking. Its behavior remains subject to change. In particular, because new types of error checks continue to be actively added to the checker.Seeing this means someone is interested in the HTML structure of your site.

Sur.ly alters the outbound links on your site so that visitors can get to external target pages without leaving your domain.

This is the backlink audit bot from SEMrush which aims to find and check all links pointing back to a SEMrush users domain. It can also be used by them to audit other sites to see its backlink profile.

The Site Audit tool that analyzes on-page SEO, technical and usability issues

Ryte offer solutions to help web teams and SEO teams understand the on-page performance of their website. If you see crawls from this bot it should be because your team uses it themselves, or your website it linked from a company that does and the bot is trying to ascertain if the linked pages are valid. Crawls are initiated by the user so can happen at any time, but by default are limited to simulating 10 user sessions.

SortSite scans entire websites for broken links and other quality problems

Dotbot is Moz's web crawler, it gathers web data for the Moz Link Index. It's utilised for both the free and Pro product offerings. The data collected through Dotbot is available in the Links section of your Moz Pro campaign, Link Explorer, and the Moz Links API. Dotbot is different from Rogerbot, which is the site audit crawler for Moz Pro Campaigns.

The best monitoring tool for backlinks and keywords. Check the good and bad backlinks for you and the competition.

Link checker and SiteMaps

With LINKTIGER’S dashboard you can quickly get an overview of the link integrity of your website. Drill down on any of the charts to view detailed reports where individual links as well as their status are listed.

Google-Structured-Data-Testing-Tool allows users to enter a URL and Google will fetch and test the URL to ensure that it's working. This bot scrapes the URL searches the web for the URL.

This bot collects images for Google Docs, so if your business uses GSuite, or you are happy for GSuite docs containing links to your site to be able to gather image assets, then allow this bot. When accessing your site this bot makes a small number of requests to get the image assets it needs. It does not check for Robots.txt.

This is the mobile version of Google-InspectionTool used by Search testing tools such as the Rich Result Test and URL inspection in Search Console. Apart from the user agent and user agent token, it mimics Googlebot.The URL Inspection tool provides information about Google's indexed version of a specific page, and also allows you to test whether a URL might be indexable. Information includes details about structured data, video, linked AMP, and indexing/indexability.

search performance

Ghost Inspector is an automated browser and application test solution which mimics user journey through your web site, similar to Selenium IDE. Allows you to record the journey and replay via the cloud.

Web based linkchecker to verify URLs before visting - preview any URL shortening e.g. tiny, or t.co etc. Powered by data from Google safe browsing

Free broken link checker

Global Backlink Database

Censys is a platform that helps information security practitioners discover, monitor, and analyze devices that are accessible from the Internet. It regularly probes every public IP address and popular domain names, curate and enrich the resulting data, and make it intelligible through an interactive search engine and API.One way that Censys finds publicly-reachable devices is by using Internet-wide scanning. You will see a small number of connection attempts to every IPv4 address worldwide each day. When they discover that a computer or device is configured to accept connections, you will see follow up requests by completing protocol handshakes to learn more about the running services.This service should never attempt to bypass any technical barriers, exploit security problems, or otherwise access non-public-facing services.

Bitlys url shortening technology is supported by the bitlybot, their linkchecker.

Linkdexbot v2.1 is the Linkdex general site crawler that helps marketers detect technical issues with a website’s pages, and works in 2 versions: a standard Web Search and here a Mobile Search configuration. This bot crawls relatively slowly, often waiting up to 5 seconds between pages.

The linkdexbot v2.1 is a more general site crawler that helps marketers detect technical issues with a website’s pages, and works in 2 versions: a standard Web Search and a Mobile Search configuration.

The linkdexbot v2.0 checks for links between Web pages. This helps them maintain an accurate picture of links between websites.

Free broken link checker.

This is a new version of the crawler for the linkchecker Ahrefs. This is a multi-purpose crawler which supports crawling and data gathering for all their SEO tools. Ahrefs offers a suite of SEO and search traffic monitoring & competitor research. AhrefsBot is a Web Crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. It constantly crawls web to fill their database with new links and check the status of the previously found ones.Ahrefs is used by digital marketers around the world to plan, execute, and monitor their online marketing campaigns.