Verify fake bots automatically

What is the scale of the Masquerading Bot Problem?

In general, over 50% of the total internet traffic is bot generated. Although your mileage may vary, we find that of the 50% of bot traffic that is likely to be hitting your site. Hiding in that bot traffic are fake bots that are masquerading as legitimate bot services using a fake agent string and IP spoofing. The good news is that once you implement our Verified Allowlist we ensure the services you actually add to your allow list are legitimate.

Protecting yourself from the Pretenders

Today most bots are identified primarily from the user agent string. It's important to realize that the user agent string is self-reported by the author of the bot. Effectively, it’s up to the bot writer to accurately tell you who they are.

Cybercriminals regularly take advantage of this and routinely disguise bad bots as legitimate bots. Webmasters are going to be cautious about blocking a major service provider e.g. Google. The cybercriminals take advantage of this situation and come up with clever ways to disguise their malicious bots. If you can't trust the agent string - webmasters can just run a reverse DNS lookup. However, the malicious bot authors also use IP ranges from legitimate cloud service providers, further disguising their origins. Imagine a bot that is taking up a lot of your web resources and is reporting to be Bing. You check it and it resolves to a Microsoft IP range, but is it really Bing? It may well be, but at the very least the checking is going to cause doubt and waste your time.

These impostors provide a real problem for anyone looking to whitelist bot agents. If you can’t trust the agent string or the IP range - how can you tell if the bot is genuine?

What happens if you whitelist an agent that is hijacked by a malicious bot?

Without verification, the danger of white-listing by agent string is that the malicious bot is specifically given access to your domain and excluded from further scrutiny. Adding the rogue agent to the whitelist is exactly what the cybercriminals want you to do. You’ve created the ideal breeding ground to create false negatives; bots that should be blocked are trusted instead. Likewise, if a legitimate bot changes its agent string, which happens all the time, it won’t be on your whitelist, and you could end up blocking a legitimate service that the marketing team actually needs.

The dangers of Blacklisting

If whitelisting can lead to high false negatives, blacklisting can cause high levels of false positives; bots that can be trusted are blocked. Blocking is often done while under attack, and the blacklists typically just grow and grow over the years, with little to no maintenance. Often the original reason for blocking is lost, and the blacklist quickly becomes a stale library of rules that are out of date almost from the moment they are written. Over time, today’s botnets return their IP address to the legitimate owner which can create strings of false positives that are difficult to track and trace.

Multi-factor bot Authentication

We have to move from ‘good’ or ‘bad’ bots to 'verified' or 'unverified' bots, and that’s what we do, really well at VerifiedVisitors.

VerifiedVisitors provides a simple bot management service that constantly performs a range of multi-factor authentications to verify the agent you are whitelisting is legitimate. The internet changes constantly, so our service is supplied as an API and changes are pushed across in near real-time. This means that you can rest assured, that we will dynamically verify each and every agent in your whitelist. We proactively ping each verified agent in our database to ensure that the service is legitimate, as well as performing a range of other tests, including examining the actual usage pattern of the agent itself.

Dynamically updated VerifiedAllowList

While the range of legitimate bot agents is very large, it is a knowable universe. We’ve built our own network to monitor this legitimate bot activity so we can offer you the very best in bot detection and verification to determine if the bot agent is from a reliable supplier, or is merely masquerading. Updates are pushed to the API services almost immediately, so you can benefit from the latest verified allowlist data available.