Protect API from Data Mining

It was the run-up to the incredibly busy Black Friday season in November 2021 and Your Store Wizards’ developer Brett Bittke realized something strange was happening to his company’s popular search enhancement application, Search Magic.

For ecommerce sites, a tool like Search Magic makes a huge difference to sales by accelerating the speed at which consumers can find products. Suggested products appear automagically as users type in the search box, spelling errors are corrected, while the application can even resolve unusual synonyms or words to the correct product.

Now, after many years of flawless performance, the application had slowed to a crawl for its 200 customers as it experienced what looked like a denial-of-service (DoS) attack on the tool’s API. 

“We were getting traffic coming in from large numbers of mobile phones in different locations with all sorts of IP addresses. It was a constant pain that at times was knocking down our servers,” says Your Store Wizards developer, Brett Bittke.

The company responded by increasing the number of servers but throwing horsepower at the problem made no difference. The rogue traffic simply scaled to consume that too. 

It looked like a DoS but the fact that the traffic was emanating from what appeared to be legitimate mobile phone user agents was a clue they’d met a new enemy that has grown in recent times from occasional nuisance to major business hazard – price scraping and product surveillance bots. 

Today’s ecommerce sites are afflicted by all manner of bots with different purposes, but price scrapers are among the most troublesome. Their aim is to monitor a competitor’s prices on a 24x7 basis with a view to understanding their economic model in detail.

Normally, price scrapers can be blocked by a few tweaks to a Web Application Firewall (WAF) which is why the best bots have started using large numbers of residential IPs - genuine home PCs and mobiles - to make blocking difficult or impossible without risking false positives.

“We were using Cloudflare’s general bot protection, but this wasn’t working,” comments Bittke. “We couldn’t risk blocking users, or we might end up blocking real customers.” The alternative was to subscribe to Cloudflare’s enterprise bot service, but this way out of their price range. 

The traffic slowing Search Magic was to an API, which because it is always automated makes distinguishing legitimate traffic from rogue especially difficult. The traditional WAF approach struggles to defend against this type of threat, while user CAPTCHAS won’t work at all. 

The VerifiedVisitors platform offered the company a way out. Integrating with Cloudflare and AWS Cloudfront, VerifiedVisitors detects and blocks bot scraping traffic at the network edge – before it reaches the API – using algorithms that compare normal traffic patterns with suspicious ones. 

“It was a simple automated setup that stopped 98% of the problem immediately. For the last 2%, we ask VerifiedVisitors and they make a tweak to their automation detection algorithms,” says Bittke.

The Search Magic API quickly became available to the company’s customers again and the developers were able to return expensive server capacity to its normal level. 

Inevitably, price scraping bots evolve, which is why constant vigilance and support from VerifiedVisitors is essential to counter new techniques. But if there’s a problem now, Bittke and his team can visit the VerifiedVisitors dashboard and get real-time status on any unusual traffic hitting their API.

“Unlike in 2021, we can see what we’re up against.”