What I learn from this week's emerging technology class - week nine
This week, I am officially a Cisco Certified Network Associate! Congratulations to me!β
This week, let's talk about AI crawlers.
Lately, many AI companies have been using web crawlers to grab content from websites to train their models. To help out, Cloudflare made a free tool that blocks these crawlers, which can also improve website performance.
This tool is available to free users and improves by learning crawler patterns over time. This makes it easier for website owners to stop their content from being scraped.
Stats show that many crawlers bypass traditional defences, forcing stricter filters that can affect regular visitors, traffic, and search rankings. ByteSpider from ByteDance and GPTBot from OpenAI are the top crawlers, making up most of the traffic on Cloudflare-protected sites.
Even with these tools, some AI companies still find ways to sneak past and grab data.
So, who's' stronger: the firewall or the crawlers? There's' a saying, ""Virtue is one foot tall, the devil ten foot."" AI will always find a way around limits, which is something to worry about.
Comments
Post a Comment