[tahoe-dev] Modifying the robots.txt file on allmydata.org

Peter Secor secorp at allmydata.com
Wed Feb 24 02:52:10 UTC 2010

Hi everyone (sorry for the slightly operational message),

   There is currently a robots.txt[1] file which blocks crawlers from a 
few of the projects on the site, specifically everything under /trac. In 
the interest of getting the information from allmydata.org present in 
searches for it, I propose we change this to allow crawly spiders to be 
able to index all of our projects.

   Please let me know any issues or suggestions you may have with this, 
I'm planning to make the change within the next few days barring 
compelling reasons not to.


[1] http://allmydata.org/robots.txt

More information about the tahoe-dev mailing list