A recent study by Imperva showed that more than half of internet traffic in 2016 was made up by bots. Luckily not all that bot traffic is malicious, there’s a lot of good bots working the internet. The problem of removing the good bots from the bad and more importantly, the bad from human users, is getting increasingly difficult as all bots improve.
Internet bots, some are good, some are bad
In the big pot of bot-created internet traffic, most of it comes from feed fetching bots that ferry data between applications, the biggest belonging to a certain Social network and helps you have ‘refreshed’ content in the feed on your mobile app. Another common bot-type is the search engine bots that uses search algorithms in order to rank websites in web-searches. Other common bots include crawlers that, like the name suggest crawls the internet collecting data most commonly used for marketing purposes, as well as monitoring bots that keeps an ‘eye’ on websites and makes sure that they work the way they should.
Bad bots, that made up more than half of bot traffic in 2016, range from scrapers that unauthorized extracts data, spammers that clogs up discussions sites to scavengers that searches for weak systems and websites they can infect and steal from. However, the most common of the bad bots are the (human) impersonating bots. These bots attack security solutions, create new accounts on websites, conduct unauthorized searches on websites as well as being the main line of DDoS-attacks. As technology improves so does the malicious bots and the impersonation ones are getting better and better at passing off as humans which makes them harder and harder for security systems to detect. That brings the current dilemma with security, as bots gets better and better at browsing the web in a human-like fashion it becomes more challenging to build safe websites without turning away real users with annoying security features.
Bot and human behavior
Separating the bad bots, especially the human impersonating ones, from humans have been a long-standing problem for website owners. Whilst bots can be programmed to act much better than humans in some tasks it was quickly discovered that bots had trouble conducting some simple things that’s very easy for a human user. This is the foundation of the old practice of captcha-images as a human verification tool. Whilst captcha served a purpose, it is quite user intrusive and clearly not a bullet proof method of separating bots (if you don’t believe me just check out this video).
Don’t judge a bot by its cover
Modern behavioral technology allows us to find the bot through its actions instead of just trying to separate bots from humans with user-unfriendly challenges. It’s a much harder task for bad bot creator to completely mimic human behavior throughout a session compared to making the bot capable of completing one specific task, like captcha.
The mechanic nature of bots tends to make them slightly more predictable than humans. Looking at a real user versus a bot in mouse movement alone we can observe quite a few differences. Even when a bot is programmed to behave very close to a human, at least to the naked eye, they can’t get away from the fact that their core is programmed to complete a task. Humans take more time, behave more erratically and shifts behavior much more than even the best programmed impersonating bots. If you read (and remember) the piece I wrote about our first DARPA-participation we could flag a human fraudster with 86 mouse-only micro-interactions, roughly 3-4 minutes of use, over six years ago. Using our much-improved behavioral technology to eliminate bots from humans is a much easier feat.
Analyzing typing rhythm is an art that goes back to Morse code and the technology involved has just gotten better by the year. Bots here face the same issues as they do with mouse-movement, they are simply put just too bad at behaving imperfectly. Human typing ranges from 0 to over 150+ words per minute and even the fastest, most experienced people, types with imperfections that separates them from bots when the typing is broken down with behavioral biometrics. Six years ago, it took us 3-4 minutes to eliminate human fraudsters with mouse data alone we’d see the fraudsters in 10 (!) seconds if they used the keyboard today. Roughly 3 keystrokes were all it took for us then to stop the wrong human and, same as with mouse, we are much stronger now.
If you want to know more on how it can look when we stop a bot in it tracks using BehavioSec technology you should check out the bot-detection video my colleagues made for more information.