Lmst

#Business #Analyses
Are 73% of e-commerce visitors fake? · What’s broken in measuring online success https://ilo.im/167nz4

_____
#Visitors #Crawlers #Conversion #Ecommerce #Website #Metrics #WebAnalytics #Advertising #AI

Just added 280 #Crawlers incoming via #AWS to my personal #Blocklist which is distributed to all #PfSense I maintain.

Crazy, they produce 90% of the traffic of one on my sites......

"Optimize4AI" là công cụ mới giúp bạn xem metadata trang web của mình trông như thế nào đối với các trình thu thập thông tin và AI. Nó phân tích các khoảng trống metadata và cách AI hiểu nội dung, cung cấp chế độ xem song song (trình duyệt vs AI), phân tích JSON-LD và đánh giá E-E-A-T. Miễn phí 20 lần quét, không cần đăng nhập. Hữu ích cho các nhà phát triển và chủ trang web!

#Optimize4AI #Metadata #SEO #WebDevelopment #AI #Crawlers #SideProject #WebDev #CôngCụWeb #PhânTíchMetadata

https://www

#Crawlers #KissMe

Bloquez les #crawlers de l'#IA: https://github.com/ai-robots-txt/ai.robots.txt 🤖 🙅

https://plutocrat.diet/robots.txt

Yay, I implemented a way to keep #AI #Crawlers from hammering our punny community Server w/o relying on javascript/anubis yet:

* hashlimit https to few connections/min only. (browsers will use pipelining anyway and make only one/few connections)
* Anything about that hashlimit becomes hard dropped.
* Configure the Webserver to drop a connection when the reply code is >=400
* make a hidden page with thousands of nonsense links that end in 404
* link that hidden page with hidden links from almost any other page.
* put that hidden page (and some non existing links) into robots.txt as 'Disallow'

🍿🎉

#Business #Initiatives
AI’s free web scraping days may be over · Say hello to RSS’s younger, tougher brother https://ilo.im/166s9q

_____
#Web #Publishing #Website #Blog #Content #AI #Crawlers #Payments #RSL #RSS #RobotsTxt

#Business #Reports
The web has a new AI payment system · The RSL Standard sets rules for AI scraping fees https://ilo.im/166ryy

_____
#Web #Publishing #Website #Blog #Content #AI #Crawlers #Payments #RSL #RobotsTxt

#Development #Releases
Introducing AI Crawl Control · Cloudflare boosts content creators’ control over AI crawlers https://ilo.im/166gqk

_____
#AI #Crawlers #Business #Paywall #Monetization #Content #Website #Development #WebDev #Backend

Some sites (not all) are seeing massive declines in crawl rates in Google Search Console - not super widespread but wide enough... https://www.seroundtable.com/google-crawl-rate-decline-40013.html via @glenngabe @dhruvpandyadp @senormunoz and @vercel CTO @cramforce and @aleyda @pedrodias and more

#google #bing #googlesearchconsole #crawlers #spiders #crawling #seo #search #bug

#Perplexity is using #stealth, undeclared #crawlers to evade website no-crawl directives

https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/

I've had it with the aggressive #AI #crawlers now. Some bot has been hitting #MacPorts with a legitimate enough user agent that I can't block it without also blocking users.

Yesterday, it sent 377k requests (62 % of the total), 369k to URLs forbidden in robots.txt from 274k unique IPs. Most of it for content that could be analyzed quicker using `svn checkout` or `git clone`.

Dynamic content on the #web is broken. There's just no way to do that anymore. What a waste of energy.

I have a recent project to stop (LLM training) crawlers from copyright-thefting my website. I find these bots with either a hidden-link tarpit or by looking for single access events (no css), which I then ban if they come from a cloud server. So far I have learned:

• Amazon AWS, Google Cloud, Microsoft Azure, and Chinese telecom companies are pretty easy to block. These were the early heavy hitters.
• Huawei has little cloud server farms all over the world. I seem to still find about one a day.
• Some mysterious entity rents servers all over the world and crawls by sniping one page at a time. The snipes come in clusters, so all these bots are running the same crawler, with some but not complete inter-communication. Popular cloud companies are OVH Cloud, EGI Hosting, Web2Objects, Host Royale, Digital Ocean, Cloud Innovation, ....

#apache #litespeed #htaccess #crawlers #botfarms

https://codeberg.org/skewray/htaccess

🤖🎉 Wow, #AI #crawlers are now the Indiana Jones of #Codeberg, fearlessly solving #Anubis #challenges while we mere mortals fumble with #JavaScript on Mastodon. 🙄 Clearly, the robots are one step closer to world domination, and we're still struggling to open our native apps. 📱💥
https://social.anoxinon.de/@Codeberg/115033790447125787 #WorldDomination #HackerNews #ngated