[tahoe-dev] [tahoe-lafs] #752: speed up directories more

[tahoe-lafs]

#752: speed up directories more
---------------------------+------------------------------------------------
 Reporter:  zooko          |           Owner:           
     Type:  enhancement    |          Status:  new      
 Priority:  major          |       Milestone:  undecided
Component:  code-dirnodes  |         Version:  1.4.1    
 Keywords:  performance    |   Launchpad_bug:           
---------------------------+------------------------------------------------
 Even after [3969], [3971], [3972], ''and'' the not-yet-committed patch to
 generate IVs with hashes of writecaps (#750) ''and'' a not-yet-committed
 patch to run with {{{-OO}}} optimization mode, directories are still too
 slow.  This set of five patches speeds directories up significantly:

 on my amd64 workstation yukyuk:

 before these five optimization patches: unpack and repack a 64-entry
 directory: 0.10 seconds, a 1024-entry directory: 1.75s, a 4096-entry
 directory: 7.75s, a 65536-entry directory: 441.65s.

 after these five optimization patches: unpack and repack a 64-entry
 directory: 0.08s, a 1024-entry directory: 1.3s, a 4096-entry directory:
 4.95s, a 65536-entry directory: 115.4s

 on Zandr's 266 MHz armel box:

 before these five optimization patches: unpack and repack a 64-entry
 directory:  2.19 seconds, a 1024-entry directory: 35.48s, a 4096-entry
 directory: 183.05s (65536-entry directory not measured).

 after these five optimization patches: unpack and repack a 64-entry
 directory: 2.16s, a 1024-entry directory: 32.55s, a 4096-entry directory:
 124.28s (a 65536-entry directory not measured).

 So this is a significant speed-up.  Except wait -- 5s to open a directory
 with 4096 entries on an amd64 workstation, or 2 minutes to open it on a
 266 MHz armel NAS?  We can do better than that!

-- 
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/752>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid