In a nutshell: A recent report by Akamai estimates that 42 percent of overall web traffic is generated by bots, with 65 percent of these bots having clearly malicious intent. Akamai's study primarily focuses on "scraping bots," which are designed to harvest data and content for use in cyber-crime operations. These bots can significantly impact web-based businesses, and AI algorithms are making life easier for hackers and criminals.
As one of the largest content delivery networks on the market alongside the likes of Cloudflare and Amazon AWS, Akamai knows pretty well what's happening on the internet right now. And the internet as we know it is mostly bots.
That is one of Akamai's conclusions according to a 2024 report on web scraping that is further supported by similar numbers seen in 2022 and 2023 reports by different sources. Akamai reports that the e-commerce sector has been most affected by scraping and "high-risk" bot traffic. While some types of bots can benefit businesses, web scraper bots typically have a negative impact on both the bottom line and the overall customer experience. Currently, web scraping bots are used for competitive intelligence, espionage, inventory hoarding, scam site creation, and other criminal activities.
Patrick Sullivan, Akamai's CTO, explains that bots pose significant challenges, creating "multiple pain points" for web apps, services, and API owners. Scraping bots can easily obtain product images, descriptions, pricing information, and other data. Cyber-criminal gangs can then exploit this data to create fake websites that impersonate well-known brands or retail services.
It happens to content websites as well like TechSpot and many others, when bots are used to scrape content, spruce some slightly changed text (made easier today using Gen AI) and replicate our content without permission for the purpose of ranking on Google, stealing traffic, and making money through ads.
The scraper landscape is evolving due to AI. Bots using artificial intelligence algorithms are becoming much harder to detect. AI botnets can work well even with unstructured data, Akamai says, and they can employ actual business intelligence to provide scammers with a much refined "decision-making process." Thanks to AI, criminals can collect, extract, and process data more efficiently than ever before.
Bots are also useful to create new fake accounts, which can then be used to target real people and their finances. Even non-malicious scraping bots can degrade a website's performance, impact search engine metrics, and increase computing and hosting costs.
Companies now face increasingly sophisticated bots that use AI algorithms, headless browser technology, and other advanced solutions. These new threats require novel, more complex mitigation approaches beyond traditional methods. A robust firewall is now only the beginning of the numerous security measures needed by website owners today.