[tahoe-dev] Modifying the robots.txt file on allmydata.org

[Peter Secor]

Hi everyone (sorry for the slightly operational message),

   There is currently a robots.txt[1] file which blocks crawlers from a 
few of the projects on the site, specifically everything under /trac. In 
the interest of getting the information from allmydata.org present in 
searches for it, I propose we change this to allow crawly spiders to be 
able to index all of our projects.

   Please let me know any issues or suggestions you may have with this, 
I'm planning to make the change within the next few days barring 
compelling reasons not to.

Thanks,
Peter

[1] http://allmydata.org/robots.txt