Comprehensive Analysis of Good Bots, and how to Verify and Manage good bots with a comprehensive Bot Automated Traffic Policy

VerifiedVisitors allows you to take control over all bot traffic, and decide exactly who gets access to your valuable data. VerifiedVisitors provides the discovery to allow you to find out exactly which bots are hitting your site, and more importantly, why. We examine your existing robots.txt file and match up with our Bot database to automate the entire process so you don’t have to dig into weblogs. Once the business understands how their own data is being used, they often are absolutely not OK with it, and actively want to block access.

‍

Once you know the nature of the automated traffic, we provide a recommendation engine that guides you through the best policy per class of bot. You decide on your security policies once. 

VerifiedVisitors allows you to apply a security policy that get’s applied according to the actual threats and risks on each of the sites that policy is tied too. As the risk changes, the policy adapts to cover the risk. This is all automated for you - all you need to do is to set the policy once in the command and control console, and VerifiedVisitors does the rest.

VerifiedVisitors has 42 categories of bots in our recommendation engine, to ensure you can manage your good bots, just as you manage your actual human visitors today. We verify and authenticate the bots, match them up to our database, and then give you recommendations on allowing them or not.

Now you have one set of clear policies it’s much easier to manage your security at the policy layer. You have one simple set of clear security standards, that you can update centrally as the risks change over time.

VerifiedVisitors dynamically applies the actual rules automatically at  each endpoint. Policy applied.  Problem solved.

Bot Discovery

VerifiedVisitors gives you detailed information on each category of bot as you can see in the search engine panel above. The filters at the top allow you to sync with your existing Robots.txt file, so you can pick up all the bots you want to allow or disallow. Instead of relying on robots.txt, VerifiedVisitors will now enforce your instructions, and validate the bots to eliminate fakes or imposters. Additional filters allow you to match the entire database up with bots that have crawled, so you can see at a glance the actual bots you need to take care of. The bot panel also shows the latest crawling activity. You can use the search engine panel to see the last crawled dates, requests made and crawl volume, which can be helpful to see how often your site is indexed. 

Bot Recommendation Engine

While customers can fine tune their bot policy to be as granular as they like, most choose to just apply the recommended setttings. This allows us to apply the entire bot verified allow list instantly. Customers can then fine tune the allow list, or indeed at their own custom bots, for services they know are crawling or have developed themselves for internal reasons.

‍

Bot Categories Included in the VerifiedVisitors Bot Database