I’ve made a little something, so I thought I'd share.

Gort is a robots.txt parser and evaluator. It implements RFC 9309.

More details in the ReadMe: https://github.com/pointlessone/gort

#Ruby #rubygem #release #robotstxt #robots_txt #rfc9309

Setting up /robots.txt, not because it helps, but because being crabby in compliance with an RFC is satisfying.

Who has some unsavory ones besides ChatGPT and Twitterbot?

https://rossabaker.com/configs/website/webcrawlers/

#RFC9309 #RobotsTxt

Released #CrawlerCommons 1.4: Java 11, #RobotsTxt compliant with #rfc9309 - https://github.com/crawler-commons/crawler-commons#18th-july-2023----crawler-commons-14-released