[tahoe-dev] [tahoe-lafs] #598: add 'tahoe backup' command: fast versioned readonly backups

tahoe-lafs trac at allmydata.org
Fri Feb 20 23:46:42 UTC 2009


#598: add 'tahoe backup' command: fast versioned readonly backups
-----------------------------------+----------------------------------------
     Reporter:  warner             |       Owner:        
         Type:  enhancement        |      Status:  closed
     Priority:  major              |   Milestone:  1.3.0 
    Component:  code-frontend-cli  |     Version:  1.2.0 
   Resolution:  fixed              |    Keywords:        
Launchpad_bug:                     |  
-----------------------------------+----------------------------------------

Comment(by azazel):

 I've done some little benchmarks of uploading one of my darcs repos to the
 production grid. I've uploaded it first using "tahoe cp -r -v" and then i
 uploaded a tar (not zipped) of the same data. The repo is composed of 67
 dirs and 4098 files, the tar size is 27 MB. The "cp -r -v" took roughly
 3.5 hours, the "cp repo.tar" took 760 seconds.
 The client is configured to use an helper.

 Here are the stats for one of the files involved in the first upload:
 {{{
     * Timings:
           o File Size: 3424 bytes
           o Total: 3.88s (882Bps)
                 + Storage Index: 194us (17.64MBps)
                 + [Contacting Helper]: 723ms
                       # [Helper Already-In-Grid Check]: 228ms
                 + [Upload Ciphertext To Helper]: 352ms (9.7kBps)
                 + Peer Selection: 879ms
                 + Encode And Push: 1.05s (69.9kBps)
                       # Cumulative Encoding: 705us (4.86MBps)
                       # Cumulative Pushing: 48ms (71.0kBps)
                       # Send Hashes And Close: 881ms
                 + [Helper Total]: 3.37s
 }}}
 Next, the stats for the tar upload:
 {{{
     * Timings:
           o File Size: 27176960 bytes
           o Total: 760.13s (35.8kBps)
                 + Storage Index: 261us (104099.02MBps)
                 + [Contacting Helper]: 702ms
                       # [Helper Already-In-Grid Check]: 454ms
                 + [Upload Ciphertext To Helper]: 723.25s (37.6kBps)
                 + Peer Selection: 461ms
                 + Encode And Push: 35.01s (807.5kBps)
                       # Cumulative Encoding: 1.50s (18.13MBps)
                       # Cumulative Pushing: 32.16s (845.1kBps)
                       # Send Hashes And Close: 996ms
                 + [Helper Total]: 759.47s
 }}}

 This small test demostrated an overhead of 1.5 ~ 2 seconds for every
 upload operation.
 Lastly i post the results of a "du --si $repo; find $repo -type f |wc -l;
 find $repo -type d |wc-l" command:
 {{{
 33k     wip/cute/lib/python2.5/site-packages/fixture-1.1.1-py2.5.egg/EGG-
 INFO
 136k    wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/command/generate
 144k    wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/command
 25k     wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/examples/db
 29k     wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/examples
 115k    wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/loadable
 29k     wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/setup_cmd
 78k     wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/test/test_command/test_generate
 82k     wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/test/test_command
 213k    wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/test/test_loadable
 426k    wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture/test
 922k    wip/cute/lib/python2.5/site-
 packages/fixture-1.1.1-py2.5.egg/fixture
 955k    wip/cute/lib/python2.5/site-packages/fixture-1.1.1-py2.5.egg
 353k    wip/cute/lib/python2.5/site-
 packages/zope.schema-3.5.0a2-py2.5.egg/zope/schema/tests
 627k    wip/cute/lib/python2.5/site-
 packages/zope.schema-3.5.0a2-py2.5.egg/zope/schema
 635k    wip/cute/lib/python2.5/site-
 packages/zope.schema-3.5.0a2-py2.5.egg/zope
 54k     wip/cute/lib/python2.5/site-packages/zope.schema-3.5.0a2-py2.5.egg
 /EGG-INFO
 689k    wip/cute/lib/python2.5/site-packages/zope.schema-3.5.0a2-py2.5.egg
 46k     wip/cute/lib/python2.5/site-
 packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope/interface/common/tests
 168k    wip/cute/lib/python2.5/site-
 packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope/interface/common
 267k    wip/cute/lib/python2.5/site-
 packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope/interface/tests
 1,0M    wip/cute/lib/python2.5/site-
 packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope/interface
 1,1M    wip/cute/lib/python2.5/site-
 packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope
 87k     wip/cute/lib/python2.5/site-
 packages/zope.interface-3.5.0-py2.5-linux-i686.egg/EGG-INFO
 1,1M    wip/cute/lib/python2.5/site-
 packages/zope.interface-3.5.0-py2.5-linux-i686.egg
 21k     wip/cute/lib/python2.5/site-
 packages/zope.event-3.4.0-py2.5.egg/zope/event
 29k     wip/cute/lib/python2.5/site-
 packages/zope.event-3.4.0-py2.5.egg/zope
 29k     wip/cute/lib/python2.5/site-packages/zope.event-3.4.0-py2.5.egg
 /EGG-INFO
 58k     wip/cute/lib/python2.5/site-packages/zope.event-3.4.0-py2.5.egg
 41k     wip/cute/lib/python2.5/site-
 packages/zope.component-3.5.1-py2.5.egg/zope/component/bbb
 29k     wip/cute/lib/python2.5/site-
 packages/zope.component-3.5.1-py2.5.egg/zope/component/testfiles
 672k    wip/cute/lib/python2.5/site-
 packages/zope.component-3.5.1-py2.5.egg/zope/component
 680k    wip/cute/lib/python2.5/site-
 packages/zope.component-3.5.1-py2.5.egg/zope
 95k     wip/cute/lib/python2.5/site-
 packages/zope.component-3.5.1-py2.5.egg/EGG-INFO
 775k    wip/cute/lib/python2.5/site-
 packages/zope.component-3.5.1-py2.5.egg
 4,0M    wip/cute/lib/python2.5/site-packages
 13k     wip/cute/lib/python2.5/distutils
 4,0M    wip/cute/lib/python2.5
 4,0M    wip/cute/lib
 0       wip/cute/include
 1,2M    wip/cute/bin
 5,3M    wip/cute/cute/_darcs/pristine.hashed
 13M     wip/cute/cute/_darcs/patches
 21k     wip/cute/cute/_darcs/prefs
 435k    wip/cute/cute/_darcs/inventories
 19M     wip/cute/cute/_darcs
 463k    wip/cute/cute/docs/tutorial/images
 517k    wip/cute/cute/docs/tutorial
 0       wip/cute/cute/docs/experiments
 517k    wip/cute/cute/docs
 29k     wip/cute/cute/lib/cute/app
 25k     wip/cute/cute/lib/cute/ui/widgets
 91k     wip/cute/cute/lib/cute/ui/resources
 13k     wip/cute/cute/lib/cute/ui/designer_plugins
 13k     wip/cute/cute/lib/cute/ui/ui
 8,2k    wip/cute/cute/lib/cute/ui/test
 304k    wip/cute/cute/lib/cute/ui
 8,2k    wip/cute/cute/lib/cute/db/search
 17k     wip/cute/cute/lib/cute/db/source
 91k     wip/cute/cute/lib/cute/db
 3,6M    wip/cute/cute/lib/cute/tests/sample_data/birt/images/logos
 263k    wip/cute/cute/lib/cute/tests/sample_data/birt/images/productlines
 4,1M    wip/cute/cute/lib/cute/tests/sample_data/birt/images
 4,3M    wip/cute/cute/lib/cute/tests/sample_data/birt
 4,3M    wip/cute/cute/lib/cute/tests/sample_data
 4,3M    wip/cute/cute/lib/cute/tests
 4,8M    wip/cute/cute/lib/cute
 4,8M    wip/cute/cute/lib
 24M     wip/cute/cute
 33M     wip/cute
 3647
 70
 }}}

-- 
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/598#comment:9>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid


More information about the tahoe-dev mailing list