Cloudflare's 2025 data reveals Google's structural advantage in AI training: Googlebot crawled 11.6% of web pages vs OpenAI's 3.6%. Publishers face an impossible choice - they can't block Google's AI crawling without losing search visibility entirely, since the same bot handles both functions. #AI #WebCrawling #DigitalRights
#WebCrawling
Search Engine Roundtable: OpenAI Scales Up Crawling & Bots For The Holidays. “OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Merj noticed OpenAI adding a lot of new IP ranges for its bots and crawlers.”
Google updates crawling infrastructure documentation with new technical details: Google publishes updated crawling infrastructure documentation on November 20, 2025, adding HTTP caching support details and transfer protocol specifications. https://ppc.land/google-updates-crawling-infrastructure-documentation-with-new-technical-details/ #GoogleUpdates #WebCrawling #HTTPCaching #SEO #TechnicalSEO
Released scrapy-contrib-bigexporter 1.0.0 (https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters) - additional export formats for the webscraping framework Scrapy.
Migrated parquet export from fastparquet to pyarrow as fastparquet is deprecated (https://docs.dask.org/en/stable/changelog.html#fastparquet-engine-deprecated)
Migrated orc export from pyorc to pyarrow to reduce the number of dependencies
#scrapy #crawling #python #parquet #orc #pyarrow #webcrawling #scraping
Why does Katana stand out as a web crawler? 🤔✨
Katana blends **speed** and **flexibility**, supporting standard and headless crawling. It handles JavaScript, automatic form filling, and advanced scope control features like regex-based filtering. Perfect for modern web exploration. #WebCrawling #OpenSource
🔗 Project link on #GitHub 👉 https://github.com/projectdiscovery/katana
#Infosec #Cybersecurity #Software #Technology #News #CTF #Cybersecuritycareer #hacking #redteam #blueteam #purpleteam #tips #opensource #cloudsecurity
— ✨
🔐 P.S. Found this helpful? Tap Follow for more cybersecurity tips and insights! I share weekly content for professionals and people who want to get into cyber. Happy hacking 💻🏴☠️
How to Recrawl a Website in Ahrefs Console | https://techygeekshome.info/how-to-recrawl-a-website-in-ahrefs-console/?fsp_sid=11609 | #ahrefs #Guide #refresh #SEO#AhrefsConsole
#RecrawlWebsite
#SEOTools
#DigitalMarketing
#WebsiteOptimization
#AhrefsGuide
#MastodonSEO
#WebCrawling
#SearchEngineOptimization
#SEOStrategy
https://techygeekshome.info/how-to-recrawl-a-website-in-ahrefs-console/?fsp_sid=11609
Remember when 'robots.txt' was supposed to solve all our crawling problems? Online media brands are trying a new protocol to deter 'unwanted' AI crawlers. Because clearly, we need more digital fences. What's your bet on how long it takes for a savvy AI to find a workaround?
Search Engine Land: Google fixes reduced crawling issue impacting some websites. “Google has confirmed it fixed an issue with its crawlers impacting ‘some sites.’ The issue was ‘reduced / fluctuating crawling’ from Google’s end with Googlebot. It is now resolved and Google said the crawling should pick back up in the near future.”
Robots.txt Is a Suicide Note
https://wiki.archiveteam.org/index.php/Robots.txt
#HackerNews #RobotsTxt #SuicideNote #WebCrawling #InternetArchive #TechEthics
"Perplexity’s accusations aren’t exactly fair, either. One argument that Prince and Cloudflare used for calling out Perplexity’s methods was that OpenAI doesn’t behave in the same way.
“OpenAI is an example of a leading AI company that follows these best practices,” Cloudflare wrote. “They respect robots.txt and do not try to evade either a robots.txt directive or a network level block. And ChatGPT Agent is signing http requests using the newly proposed open standard Web Bot Auth.”
Web Bot Auth is a Cloudflare-supported standard being developed by the Internet Engineering Task Force that hopes to create a cryptographic method for identifying AI agent web requests.
The debate comes as bot activity reshapes the internet. As TechCrunch has previously reported, bots seeking to scrape massive amounts of content to train AI models have become a menace, especially to smaller sites.
For the first time in the internet’s history, bot activity is currently outstripping human activity online, with AI traffic accounting for over 50%, according to Imperva’s Bad Bot report released last month. Most of that activity is coming from LLMs. But the report also found that malicious bots now make up 37% of all internet traffic. That’s activity that includes everything from persistent scraping to unauthorized login attempts."
#AI #GenerativeAI #AITraining #Perplexity #Cloudflare #AIAgents #WebCrawling #Chatbots #LLMs
Perplexity Fires Back at Cloudflare, Denying ‘Stealth Crawler’ Accusations
#AI #Cloudflare #Perplexity #WebCrawling #AIethics #DataScraping #SearchEngines #Web #AISearch
KIMissbrauch
Cloudflare wirft dem KI-Anbieter ##Perplexity vor, sich mit undeklarierten Crawlern Zugang zu gesperrten Websites zu verschaffen.
Trotz robots.txt-Verboten und IP-Blockaden soll Perplexity mit wechselnden User-Agents und IPs Inhalte verdeckt auslesen.
Das wäre eine Verletzung etablierter Webstandards und Missachtung von Website-Präferenzen.
#WebCrawling #BotTraffic #Cloudflare #WebSecurity #PerplexityAI #Chatbots
ICYMI: Google outlines pathway for robots.txt protocol to evolve: How the 30-year-old web crawler control standard could adopt new functionalities while maintaining its simplicity. https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/ #Google #RobotsTxt #WebCrawling #SEO #DigitalMarketing
Crawl Order and Disorder
https://www.marginalia.nu/log/a_117_crawl_order/
#HackerNews #Crawl #Order #Disorder #WebCrawling #TechInsights #Marginalia
Ars Technica: Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries. “Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures—adjusting robots.txt, blocking known […]
Perishable Press: Ultimate Block List to Stop AI Bots. “The focus of this post is aimed at website owners who want to stop AI bots from crawling their web pages, as much as possible. To help people with this, I’ve been collecting data and researching AI bots for many months now, and have put together a ‘Mega Block List’ to help stop AI bots from devouring your content.”
https://rbfirehose.com/2025/02/12/perishable-press-ultimate-block-list-to-stop-ai-bots/
- 📊 Optional: Markov-generated nonsense content to distort data
- 💻 Developed by programmer Aaron B. out of frustration with #Webcrawling practices
- ⚠️ Challenges: Server load, scalability, effectiveness questioned
#Nepenthes: #Tool against #AI webcrawlers 🕷️
Generates self-referencing links, extends load times. Goal: Trap crawlers in endless loop. Developer warns against casual use. #AI #Webcrawling #DataPrivacy
🧵 ↓
https://www.heise.de/en/news/Nepenthes-a-tarpit-for-AI-web-crawlers-10256257.html
Hackaday: Trap Naughty Web Crawlers In Digestive Juices With Nepenthes. “More commonly known as ‘pitcher plants’, nepenthes is a genus of carnivorous plants that use a fluid-filled cup to trap insects and small critters unfortunate enough to slip & slide down into it. In the case of this Lua-based project the idea is roughly the same. Configured as a trap behind a web server (e.g. […]
Client Info
Server: https://mastodon.social
Version: 2025.07
Repository: https://github.com/cyevgeniy/lmst