[tahoe-dev] [tahoe-lafs] #962: Performance problems testing on a 45 disk Dual core 3.3Ghz 4G memory Box

tahoe-lafs trac at allmydata.org
Tue Feb 23 01:08:26 UTC 2010


#962: Performance problems testing on a 45 disk Dual core 3.3Ghz 4G memory Box
-------------------------+--------------------------------------------------
 Reporter:  stott        |           Owner:  somebody 
     Type:  enhancement  |          Status:  new      
 Priority:  major        |       Milestone:  undecided
Component:  website      |         Version:  1.5.0    
 Keywords:  Performance  |   Launchpad_bug:           
-------------------------+--------------------------------------------------

Comment(by stott):

 System Configuration  (http://blog.backblaze.com/2009/09/01/petabytes-on-a
 -budget-how-to-build-cheap-cloud-storage)

 1  Intel(R) Core(TM)2 Duo CPU     E8600  @ 3.33GHz
 1  80G boot drive mounted as /
 45 Seagate 1.5 T drives mounted as /tahoe/node1-45
 4G memory

 eth0 is at 100MB/FULL

 Default tahoe configuration.

 SMEM :
 root at firetruck1:~/app/smem-0.9# ./smem |egrep "PID|python"
   PID User     Command                         Swap      USS      PSS
 RSS
 13025 root     egrep --color=auto PID|pyth        0      260      276
 860
 13024 root     /usr/bin/python ./smem             0     4560     4609
 6224
 12402 www-data /usr/bin/python /usr/bin/tw    23028    33356    33391
 35504
 12546 www-data /usr/bin/python /usr/bin/tw    14796    47700    47754
 50764
 12537 www-data /usr/bin/python /usr/bin/tw     9412    53300    53351
 56228
 12590 www-data /usr/bin/python /usr/bin/tw       12    60732    60786
 63796
 12634 www-data /usr/bin/python /usr/bin/tw        0    60792    60846
 63856
 12645 www-data /usr/bin/python /usr/bin/tw        0    60796    60854
 63868
 12612 www-data /usr/bin/python /usr/bin/tw        8    60904    60958
 63968
 12601 www-data /usr/bin/python /usr/bin/tw        4    61192    61246
 64256
 12425 www-data /usr/bin/python /usr/bin/tw      668    61860    61911
 64788
 12447 www-data /usr/bin/python /usr/bin/tw      728    62240    62291
 65168
 12557 www-data /usr/bin/python /usr/bin/tw       12    62368    62422
 65432
 12480 www-data /usr/bin/python /usr/bin/tw        0    62384    62438
 65448
 12524 www-data /usr/bin/python /usr/bin/tw      156    62500    62552
 65492
 12623 www-data /usr/bin/python /usr/bin/tw        4    62500    62554
 65564
 12579 www-data /usr/bin/python /usr/bin/tw        8    62564    62616
 65556
 12513 www-data /usr/bin/python /usr/bin/tw       52    62632    62686
 65696
 12700 www-data /usr/bin/python /usr/bin/tw        0    62644    62698
 65708
 12491 www-data /usr/bin/python /usr/bin/tw       40    62772    62824
 65764
 12469 www-data /usr/bin/python /usr/bin/tw      264    62796    62850
 65860
 12656 www-data /usr/bin/python /usr/bin/tw        0    63004    63057
 66060
 12502 www-data /usr/bin/python /usr/bin/tw        0    63184    63238
 66248
 12667 www-data /usr/bin/python /usr/bin/tw        0    63264    63318
 66328
 12436 www-data /usr/bin/python /usr/bin/tw       92    63392    63446
 66456
 12458 www-data /usr/bin/python /usr/bin/tw      164    63780    63834
 66844
 12744 www-data /usr/bin/python /usr/bin/tw        0    64016    64070
 67080
 12788 www-data /usr/bin/python /usr/bin/tw        0    65400    65451
 68328
 12722 www-data /usr/bin/python /usr/bin/tw        0    66472    66526
 69536
 12678 www-data /usr/bin/python /usr/bin/tw        0    67036    67090
 70100
 12689 www-data /usr/bin/python /usr/bin/tw        0    67112    67166
 70176
 12568 www-data /usr/bin/python /usr/bin/tw        0    70172    70224
 73164
 12832 www-data /usr/bin/python /usr/bin/tw        0    71800    71854
 74864
 12898 www-data /usr/bin/python /usr/bin/tw        0    71760    71949
 75168
 12777 www-data /usr/bin/python /usr/bin/tw        0    72432    72486
 75496
 12854 www-data /usr/bin/python /usr/bin/tw        0    73000    73050
 75920
 12735 www-data /usr/bin/python /usr/bin/tw        0    73096    73150
 76160
 12711 www-data /usr/bin/python /usr/bin/tw        0    73308    73362
 76372
 12414 www-data /usr/bin/python /usr/bin/tw      908    73256    73449
 76680
 12799 www-data /usr/bin/python /usr/bin/tw        0    73652    73706
 76716
 12843 www-data /usr/bin/python /usr/bin/tw        0    74660    74714
 77724
 12865 www-data /usr/bin/python /usr/bin/tw        0    74888    74946
 77960
 12821 www-data /usr/bin/python /usr/bin/tw        0    76804    76858
 79868
 12887 www-data /usr/bin/python /usr/bin/tw        0    79180    79233
 82236
 12755 www-data /usr/bin/python /usr/bin/tw        0    79892    79946
 82956
 12766 www-data /usr/bin/python /usr/bin/tw        0    80548    80599
 83532
 12810 www-data /usr/bin/python /usr/bin/tw        0    86512    86563
 89440
 12876 www-data /usr/bin/python /usr/bin/tw        0    87592    87645
 90648


 ================

 Note,
   After switching to

 shares.needed = 32
 shares.happy = 35
 shares.total = 40


 Performance has greatly increased. The 3.3 Blowup may have been the
 largest contributing factor.

 I'm still concerned about memory utilization per node as the nodes
 increase per grid, but that may be a separate ticket.

 Performace is still much less than I would like to see.

 bash-3.2$ time curl -T 524stottlemyer.zip
 http://10.20.0.151:3470/uri/URI:DIR2:whzhc6kxh5mfem4basdoioewji:4qpwpea2s46tg3lnd5znwsyqqenwjoddjxpsvr733sk7n4on6uua/
 URI:CHK:fsmf5rn6me7mrueq2ri4zvboay:sd5lnugn2gnisrvodxtzp7zvkx4f2skyk6ri5ncs65rmq6qkdoqa:32:40:106554122
 real    1m17.962s

 bash-3.2$ ls -larth 524stottlemyer.zip
 -rw-r--r--  1 justin  staff   102M Nov 20 15:44 524stottlemyer.zip

 I'll have to collect more stats during an upload to present what is going
 on.  A snapshot indicated heavy CPU, nearly no swap in use at this time,
 and disks were much better able to handle the load w/ great distribution.

 Here is a brief network snapshot though time spent on internal operations
 was much greater.

  eth0:  0.003 In,  0.000 Out Mbps |    4 In,    0 Out pps
  eth0: 40.860 In,  0.963 Out Mbps | 3709 In, 1912 Out pps
  eth0: 92.249 In,  2.226 Out Mbps | 8409 In, 4354 Out pps
  eth0: 37.314 In,  0.893 Out Mbps | 3430 In, 1761 Out pps
  eth0:  0.006 In,  0.015 Out Mbps |    8 In,    3 Out pps

-- 
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/962#comment:3>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid


More information about the tahoe-dev mailing list