[tahoe-dev] Modifying the robots.txt file on allmydata.org
zandr at mozilla.com
Wed Feb 24 07:33:10 UTC 2010
On 2/23/10 6:52 PM, Peter Secor wrote:
> Please let me know any issues or suggestions you may have with this,
> I'm planning to make the change within the next few days barring
> compelling reasons not to.
Keep that Crawl-delay line in there, though, or MSNbot will render your
Trac is actually pretty expensive to serve, so if you don't have
something slowing down the bots, they'll crater your machines.
BTDT. This week, in fact.
More information about the tahoe-dev