[tahoe-dev] exonerating crypto as a culprit for our performance problems

[Zooko O'Whielacronx]

Folks:

Just a quick note on the speed of crypto algorithms. We've been
benchmarking and profiling Tahoe-LAFS v1.8.0c2 in search of a
performance regression. Last night I ran a big download overnight (a
1.4 GB movie file [1]), under the Python profiler, and this morning I
looked at the profiling results [2].

I noticed this detail: it took 9159 wall-clock seconds to download the
file, and 32 seconds were spent in the SHA-256 hashing. I'm going to
remember this for the next time someone mentions the crypto as being a
potential performance problem in Tahoe-LAFS. In this run on my Macbook
Pro (2.66 GHz Intel Core 2 Duo), hashing was 0.35% of the CPU time.
AES encryption was about half as much again: 14 CPU seconds from last
night's run. Adding the two crypto algorithms together means they
accounted for about half a percent of the time spent in the download.

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   227571   31.557    0.000   31.557    0.000 {method 'update' of
'_sha256.SHA256' objects}
    11375   14.141    0.001   14.141    0.001 {method 'process' of
'_aes.AES' objects}

So, if you are a user who is using Tahoe-LAFS v1.8 on a modern laptop,
desktop, or server, you should really think of the cryptography built
into Tahoe-LAFS as having no significant cost.

However if you are a Tahoe-LAFS developer helping to invent new
versions of Tahoe-LAFS, there are two caveats to this comforting
story:

1. The CPUs in my laptop are monsters! Hugely powerful, expensive, and
power-hungry. If you want to run Tahoe-LAFS on your wireless router,
phone, wall wart, or NAS with a 32-bit ARM CPU running at a few
hundred MHz then it might be a different story. François has offered
us the use of his ARM box but nobody has really done careful
benchmarks on it yet.

François wrote:
"""
It's an Intel SS4000-E NAS, one of the worse NAS box that you can find
on the market ;) Its performance is way below what you'd expected from
a RAID-5 NAS with a 4-disk enclosure.

BTW, I just got Tahoe-LAFS running on a PogoPlug¹ device, but
unfortunately I had to install 'Plugbox Linux' on it instead of using
the standard firmware. I have a tutorial waiting to be sent on
tahoe-dev.
"""

2. Tahoe-LAFS doesn't fill up the network pipes with data, it allows a
lot of idle pipe in the network. In fact, Tahoe-LAFS v1.7.1 takes
about 5000 seconds to download that file that Tahoe-LAFS v1.8.0c2 took
about 10,000 seconds, in which case the cryptography would cost about
1% of the time. Hopefully 1.8.0-final, 1.8.1 or 1.9.0 will really take
full advantage of the available networking, at which point we should
look at profiling results again to see how much different things
contribute to the overall time.

Regards,

Zooko

[1] http://localhost:3456/file/URI%3ACHK%3A4klgnafrwsm2nx3bqy24ygac5a%3Acrz7nhthi4bevzrug6xwgif2vhiacp7wk2cfmjutoz2ns3w45qza%3A3%3A10%3A1490710513/@@named=/bbb-360p24.i420.lossless.drc.ogg.fixed.ogg%2Bbbb-24fps.flac.via-ffmpeg.ogg
[2] http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170#comment:64