More than a quarter of all website traffic comes from malicious bots that are used by cybercriminals to perform various types of cyberattacks: account takeover, DDoS, data theft, and much more.
When left unmanaged (or worse, undetected), these malicious bots can not only cause huge financial repercussions but can also lead to legal disputes and long-term damage to your brand’s reputation.
This is why bot protection, the practice of identifying bad bots and managing their activities is now a necessity for all kinds of businesses. Cybersecurity is now not only a concern exclusive for big enterprises and tech companies. In fact, the latest data has suggested that cybercriminals nowadays are increasingly targeting smaller businesses.
The thing is, threats from malicious bots have not only increased in quantity but also quality. That is, today’s malicious bots are now more advanced and sophisticated at bypassing standard bot protection measures. This is why an appropriate bot protection strategy supported with the right tools is very important.
Bot Protection: The Two Conundrums
A bot protection or bot manager solution is a software product that is designed to detect and manage malicious bots. So, there are two “phases” of bot protection: the detection phase, and the management phase.
The thing is, both of these phases have their own conundrums.
The Bot Detection Conundrum: Good Bots VS Bad Bots VS Human Users
At first glance, bot detection might seem like a relatively simple concept: identify the presence of bots, and block these bots from accessing our websites or online resources.
In practice, however, it is not that simple due to two main challenges:
- Bots aren’t inherently “bad”, and there are good bots owned by reputable companies that will benefit your website. Googlebot, for example, is responsible for indexing our websites so they can get ranked by Google, and we wouldn’t want to block it. The thing is, discerning these good bots from malicious bots isn’t always straightforward.
- Today’s bot programmers are very sophisticated, and they are adopting the latest technologies (including AI) to mask the malicious bot’s identity. These bots can, for example, rotate between thousands of IP addresses and use AI technologies to mimic human behavior. So, discerning them from legitimate users can also be very challenging.
So, in order to be effective, a capable bot protection solution should reliably perform two phases of detection:
- Discerning bots from human users, and
- Differentiating malicious bots from good bots
With the objective that we can manage as many malicious bots as possible (preventing false negatives) while ensuring all good bots and legitimate users can access our resources (preventing false positives).
Effective Bot Detection Method
Although there are various bot detection and management techniques available, generally we can categorize bot detection techniques into just three main groups:
1. Challenge-based detection
In this method, we challenge the incoming traffic with a test designed to differentiate between human users and bots. CAPTCHA is a common example of this type of detection.
The main downside of the challenge-based method is that it will also challenge real users, which will ruin their user experience. Also, it’s not very effective against very advanced bots, and there is also the case of CAPTCHA farm services.
2. Signature-based detection
In this method, we analyze a user’s action and compare it against a known list of bad bot signatures (i.e. blacklisted IP address, signs of headless browsers, mouse movement patterns, etc. ).
The downside of this method is that we can’t use it to identify unknown bots with unknown signatures.
3. Behavior-based detection
The most advanced detection method at the moment. In this method, the bot protection solution uses advanced statistical techniques and AI technologies (machine learning) to analyze a massive amount of user activity data to identify anomalies.
An advanced behavior-based bot mitigation software like Datadome uses AI and machine learning technology to monitor and analyze the network traffic in real-time and is considered the best line of defense against malicious bot attacks at the moment.
The Bot Management Conundrum: To Block or Not o Block
Let’s assume we’ve successfully identified the presence of a bot and we are 100% sure that it is malicious in nature. What should we do about it?
At first glance, the answer is obvious: block its activities. By blocking the bot’s activities, we don’t need to serve any of our resources, so technically it’s the most efficient and effective way to manage the malicious bot.
However, in practice, blocking is not always the most ideal solution.
Blocking simply won’t stop persistent attackers. They will simply modify the bot or use another bot altogether to launch another attack. When they know they are blocked, it’s just information for them that the attack is not working at the moment so they have to do more.
In fact, if you are not careful, you may “help” them in modifying these bots to bypass your bot protection. For example, if you provide an error message saying “headless browser detected” when blocking the bot, then they’ll know that they should hide the headless browser presence in the next iteration of the bot.
So, blocking is not always effective, and this is why there are various bot management methods available, for example:
- Rate limiting
Also known as “throttling”, is to keep serving the bot’s requests but at a slower bandwidth. Bots run on resources, which can be expensive for the attackers, so they’d like to finish their operations as fast as possible.
By slowing down their operations, the hope is that the attacker will give up and move to another target.
- Honeypot
The idea of a “honeypot” is to trap the bot with something attractive that will reveal their identity as a bot. For example, adding invisible content (i.e. similar in color with the site’s background) that can only be found by bots.
- Serving fake content
Similar in principle to rate limiting. Here we serve the bot with fake or thinner content to poison its data and let it waste its resources.
Closing Thoughts
While blocking malicious bots isn’t always ideal, it is the most effective and efficient approach provided your bot detection method is 100% reliable. This is why using an advanced bot protection solution like Datadome with close to 100% accuracy is crucial to ensure effective protection against all kinds of bot attacks to your website, mobile app, and API.