[tahoe-dev] Modifying the robots.txt file on allmydata.org

Aleksandr Milewski zandr at mozilla.com
Wed Feb 24 07:33:10 UTC 2010


On 2/23/10 6:52 PM, Peter Secor wrote:

>     Please let me know any issues or suggestions you may have with this,
> I'm planning to make the change within the next few days barring
> compelling reasons not to.

Keep that Crawl-delay line in there, though, or MSNbot will render your 
server unusable.

Trac is actually pretty expensive to serve, so if you don't have 
something slowing down the bots, they'll crater your machines.

BTDT. This week, in fact.

-Zandr



More information about the tahoe-dev mailing list