Poor performance in test grid

[Lukas Pirl]

Hello Zooko,

thanks for your reply.

On 12/30/2013 07:09 PM, Zooko O'Whielacronx wrote:
> Dear Lukas Pirl:
>
> Thanks for the data!
>
> On Sat, Dec 28, 2013 at 3:39 PM, Lukas Pirl <mail at lukas-pirl.de>
> wrote:
>>
>> * Timings: * File Size: 1073741824 bytes * Total: 7 minutes
>> (2.32MBps) * Storage Index: 47 seconds (22.84MBps) * Peer
>> Selection: 6.12s * Encode And Push: 6 minutes (2.65MBps) *
>> Cumulative Encoding: 55 seconds (19.47MBps)
>
> This means that the time spent in encryption and erasure-coding was
> 55 seconds.
>
>> * Cumulative Pushing: 5 minutes (3.07MBps)
>
> This means that the time spent in sending blocks of ciphertext
> over the network and waiting for the server to ACK each block was
> 5 minutes.

That sounds pretty much for localhost…!?

>
> One thing that I'm not sure of is that you also reported that a
> CPU core was maxed out in the gateway during upload. That is
> inconsistent with my model of the uploader as spending most of its
> time idle, waiting for an ACK to come back over the network from
> the storage server that the block it sent has been received.

Hm - with the timings above - I agree. Maybe my observation was
inaccurate…

>
> I have a suggestion for a configuration parameter you could tweak
> that might change this performance.
>
> But before I do, I have a question for you: why are you running
> the gateway on the device that stores the data? That means you're
> transferring the cleartext of the file (by means of Nautilus,
> apparently) over HTTP (or hopefully HTTPS if you configured that!)
> to the gateway, and the gateway is storing the file (in cleartext)
> on the SSD in a tempfile, and then once it is done transferring the
> gateway then encrypts and "uploads" the file to the storage server
> process that is running on the same device. (Then the gateway
> deletes the tempfile.)
>
> Instead, if you ran the gateway on the computer where the original
> file-to-be-backed-up lives, and left Nautilus out of it, and ran
> "tahoe backup $DIRECTORY" on the directory that contained the
> original file-to-be-backed-up, then this might improve things
> significantly.

Yes, totally true.
The reason for the structure of my test setup is, that I use to hide
details of a service within the server.
So, my machines in productive use have very few and proven interfaces
to the outside world (mainly, SSH). Also, because I don't want to
install and maintain an exotic backup software on every backup client.
On the storage server side, I want to use tahoe to replace/manage JBOD
and provide sftp access for the backup clients.

Best,

Lukas

>
> Regards,
>
> Zooko _______________________________________________ tahoe-dev
> mailing list tahoe-dev at tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
>

-- 
+49 174 940 74 71