Clean Data By Excluding the Bots
“If we have data, let's look at the data. If all we have are opinions, let's go with mine". This famous quote by Jim Barksdale, the former CEO of Netscape, needs a bit of refresh. "If we have clean data, let's look at the clean data. If all we have is bots, let's not bother".
Bots now make up over 50% of the traffic on the net, so if you're making product, marketing conversion decisions based on traffic data that includes non-human visitors, you may as well go to back to your opinion, as your data will likely be wrong. Sometimes very wrong.
We worked with a team that had spent almost a year optimising the best possible signup form for their financial services products. These were complex forms, such as mortgage applications, and their business model was to generate a sales bounty per completed application. Conversion rates were critical. The team engaged with the best possible UX designers, and worked hard to optimise the UX over many months. Tests showed that the new forms were much easier to fill in, and the test users liked them. All good.
The company then performed A/B testing to verify the results on the live site. To their surprise the old form performed much, much better, despite all the hard work. Intuitively, that couldn't be right. When they drilled down on the underlying data and looked for bot traffic, it was very clear what was happening. The old form had been optimised for bot traffic, faking real humans applying for a mortgage along with a mobile phone contact and verified email. The new form had been architected as a Single Page Architecture (SPA) and the bots hadn't yet been re-tooled to hit it up.
The sheer scale of bot traffic constantly amazes client.In fact so much traffic especially on the large social networking sites like YouTube is bots pretending to be people, that the very detection systems used to detect the fakes on YouTube could potentially be compromised. This hypothetical event the YouTube employees called "the Inversion" - where the 'normal' data of the vast majority of 'users' that the security system uses to determine the baseline of actual behaviour was actually fake. The detection systems then became very confused. The new normal was a proper full on fake.
Five-hundred million fake bots can't be wrong can they?
VerifiedVisitors can be used easily and rapidly filter out this noise, allowing just the good bots you need crawling your website. Removing the rest of the bots allows you to see the real underlying human data patterns and take proper decisions based on data driven analytics.
