→ Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/

“We observed that #Perplexity [an AI-powered answer engine] uses not only their declared #user_agent, but also a generic browser intended to #impersonate Google Chrome on macOS when their declared crawler was blocked.”

“This activity was observed across tens of thousands of domains and millions of requests per day.”

#AI #evade #stealth #website #browser #crawler #blocked

CyberChef from GCHQ: Cyber Swiss Army Knife
https://gchq.github.io/CyberChef/
#ycombinator #base64 #hex #decode #encode #encrypt #decrypt #compress #decompress #regex #regular_expressions #hash #crypt #hexadecimal #user_agent #url #certificate #x_509 #parser #JSON #gzip #md5 #sha1 #aes #des #blowfish #xor

CyberChef from GCHQ: The Cyber Swiss Army Knife
https://gchq.github.io/CyberChef/
#ycombinator #base64 #hex #decode #encode #encrypt #decrypt #compress #decompress #regex #regular_expressions #hash #crypt #hexadecimal #user_agent #url #certificate #x_509 #parser #JSON #gzip #md5 #sha1 #aes #des #blowfish #xor

@timtilberg ooooh I now see the issue with `this` and multiple header/cookie sets.

Currently I have `#user_agent=` write it's value directly into `#headers`. However, this pattern doesn't exactly work neatly with cookies, if you override `#headers=` after setting `#cookie=`.