Why do I need another platform to monitor bots?

What is the scale of the Masquerading Bot Problem?

In general, over 50% of the total internet traffic is bot generated. Although your mileage may vary, we find that of the 50% of bot traffic that is likely to be hitting your site, as much as 20% of those bots are masquerading as legitimate bots using a fake agent string. The good news is that once you implement VerifiedVisitors we ensure the services you actually add to your access list are legitimate.

Protecting yourself from the Pretenders

Today most bots are identified primarily from the user agent string. It's important to realize that the user agent string is self-reported by the author of the bot. Effectively, it’s up to the bot writer to accurately tell you who they are.

Cybercriminals regularly take advantage of this and routinely disguise malicious bots as legitimate bots. Webmasters are going to be cautious about blocking a major service provider e.g. Google. The cybercriminals take advantage of this situation and come up with clever ways to disguise their malicious bots. If you can't trust the agent string - webmasters can just run a reverse DNS lookup. However, the malicious bot authors also use IP ranges from legitimate cloud service providers, further disguising their origins. Imagine a bot that is taking up a lot of your web resources and is reporting to be Bing. You check it and it resolves to a Microsoft IP range, but is it really Bing? It may well be, but at the very least the checking is going to cause doubt and waste your time.

These impostors provide a real problem for anyone looking to allow bot user agents. If you can’t trust the agent string or the IP range - how can you tell if the bot is genuine?

What happens if you allow a hijacked agent?

Without verification, the danger of allowing by agent string is that the malicious bot is specifically given access to your domain and excluded from further scrutiny. Adding the rogue agent to the access list is exactly what the cybercriminals want you to do. You’ve created the ideal breeding ground to create false negatives; bots that should be blocked are trusted instead. Likewise, if a legitimate bot changes its agent string, which happens all the time, it won’t be on your access list, and you could end up blocking a legitimate service that the marketing team actually needs.

The dangers of using block lists

If access lists can lead to high false negatives, block lists can cause high levels of false positives; bots that can be trusted are blocked. Blocking is often done while under attack, and these lists typically just grow and grow over the years, with little to no maintenance. Often the original reason for blocking is lost, and the list quickly becomes a stale library of rules that are out of date almost from the moment they are written. Over time, today’s botnets return their IP address to the legitimate owner which can create strings of false positives that are difficult to track and trace.

Multi-factor bot Authentication

We have to move from ‘good’ or ‘bad’ bots to 'verified' or 'unverified' bots, and that’s what we do, really well at VerifiedVisitors.

VerifiedVisitors provides a simple API based subscription service that constantly performs a range of multi-factor authentications to verify the agent you are allowing is legitimate. The internet changes constantly, so our service is supplied as API and changes are pushed across in near real-time. This means that you can rest assured, that we will dynamically verify each and every agent in your list. *We proactively ping each verified agent in our database to ensure that the service is legitimate, as well as performing a range of other tests, including examining the actual usage pattern of the agent itself.*

Dynamically updated VerifiedVisitors

While the range of legitimate bot agents is very large, it is a knowable universe. We’ve built our own network to monitor this legitimate bot activity so we can offer you the very best in verification to determine if the bot agent is from a reliable supplier, or is merely masquerading. Updates are pushed to the API services almost immediately, so you can benefit from the latest verified data available.

‍