Comprehensive Guide to Bot Detection in 2023: Protecting Your API, mobile and Web endpoints.

Recognizing bot traffic is a crucial aspect of effective bot detection.

‍

Here are some indicators that can help you identify suspicious bot activities on your websites, apps, and APIs:

‍

1. Sudden Changes in Login Pass / Fail -Ratios

Look out for abnormal login pass to fail ratios, or accessibility logins that are easier to pass e.g. audio.

2. Sudden Spikes in user Registrations

Bots create sleeper accounts that are then activated for a sale or other time bound event. Sudden increases in registrations, particularly if they have junk data, numeric values in the email, or other tell-tale indicators of a likely automated process for bulk registration..

3. Spikes in Pageviews

Certain bot attacks aim to overwhelm your servers, resulting in an inexplicable spike in pageviews on your analytics software. This is usually the first tell-tale sign of a bot attack.

4. Unusually High Bounce Rates

Bots often have specific objectives and leave a page as soon as their task is complete or unattainable. This behavior leads to an unusually high and rapid bounce rate.

5. Changes in Session Length

Suspiciously short or excessively long session durations are tell tale signs to look out for. Humans tend to spend a few seconds on a page, while unusually lengthy sessions can also indicate bot activity. Bots are often written specifically to avoid WAF rate-limiting, and will engage in long sessions over time to fly under the radar. 

6. Traffic Spikes from Unusual Locations or Platforms

If your business doesn't operate in certain regions, a sudden influx of requests from those locations is a clear sign of a bot attack. Similarly, if your site typically doesn’t get for example Linux based or Mobile traffic, and it’s pattern changes, it may be indicative of a bot attack.

7. Automated looking data submissions

Look out for meaningless contact form submissions, users constantly adding items to shopping carts without making purchases, or a surge in bouncebacks from your free newsletter. Bots often may subscribe to a newsletter, but won’t then verify their email address. These are signs of bot-induced chaos.

‍

Common Bot Detection Techniques

Bot detection tools have evolved to counter increasingly sophisticated bots. However, some older techniques have become ineffective. Let's explore a few common bot detection methods:

‍

1. CAPTCHAs

Traditional CAPTCHAs, like reCAPTCHA, were once effective in deterring bots. However, bots now routinely pass CAPTCHA and also use CAPTCHA farms to get around even the most complex of CAPTCH by using cheap labour in remote areas.  Extensive use of CAPTCHA also raises concerns about accessibility and data privacy compliance,which can negatively  impact user experience. Many sites have also tried to solve the problem by introducing CAPTCHAs that are progressively harder to solve, if failed the first time. This is often a great source of frustration for legitimate users, who may have accessibility issues, or are just trying to quickly access webpages on the go on a mobile device.

‍

2. Web Application Firewalls (WAFs)

WAFs are designed to protect against known attacks but are less effective against modern, advanced bots. They can reliably pick up known bad bot signatures - but the problem is they will fail with bots that are either new or purpose built that are attacking your site. New intelligent bots know the signature based methods, and take steps to disguise their origins and hide as legitimate domestic traffic using regular machines or mobile devices. 

‍

WAFs also rely heavily on IP reputation, which can be easily circumvented by bot operators using residential or domestic IPs in botnets or in commercially available residential proxy services. Many of these services contain millions of domestic IPs, which renders the traditional IP reputation services almost obsolete. Worse, the reputation based services need constant updates, and can lead to a whole host of false positives. Infected botnet machines that have a blacklisted IP are often upgraded or patched to remove the botnet, but the IP address is still blacklisted. 

‍

3. Multi-Factor Authentication (MFA)

While MFA can enhance user account security, it doesn't protect against various bot attacks, leaving businesses vulnerable to scrapers, scalpers, and DDoS attacks.

‍

The Challenge of Bot Detection

Identifying bots has become increasingly challenging for several reasons:

• Generative AI linked to intelligent bot agents is a game-changer as anyone without programming skills can now build sophisticated attacks using bots.
• Bots target a wide range of endpoints, not just websites. API data abuse is particularly widespread and hard to stop with traditional fingerprinting techniques that seek to distinguish humans from bots.
• Bots employ technologies that closely mimic human behavior - including mouse trails and triggering a mobile accelerometer.
• Bots frequently rotate through numerous clean, residential IPs, rendering IP-based detection ineffective. Fingerprint detection often fails, as the machines and platforms are  legitimate and pass the fingerprint tests.
• The rise of "Cybercrime as a Service (CaaS) allows anyone with zero programming knowledge to launch bot attacks with advanced capabilities, using residential IP, user agent rotation, and real fingerprints, all  just using simple prompts or templates. 
• Bots can distribute attacks across different times and locations with ease - and often go low and slow to avoid simple rate limiting WAF rules.