The older, the simpler: robotstxt.org. robots.txt in a nutshell
Posted: Mon Apr 05, 2021 7:35 pm
I fall on this today, after i know it since his creation: has been a surprise to see that, in all these years, it changed nothing at all!
http://www.robotstxt.org
http://www.robotstxt.org