[tahoe-dev] Modifying the robots.txt file on allmydata.org
secorp at allmydata.com
Wed Feb 24 02:52:10 UTC 2010
Hi everyone (sorry for the slightly operational message),
There is currently a robots.txt file which blocks crawlers from a
few of the projects on the site, specifically everything under /trac. In
the interest of getting the information from allmydata.org present in
searches for it, I propose we change this to allow crawly spiders to be
able to index all of our projects.
Please let me know any issues or suggestions you may have with this,
I'm planning to make the change within the next few days barring
compelling reasons not to.
More information about the tahoe-dev