[tahoe-dev] Debugging memory leaks with guppy

[David-Sarah Hopwood]

On 2010-10-24 16:18, Francois Deppierraz wrote:
> The same problem could also be reproduced by repeatedly listing a single
> directory. Your suspicion that the ResponseCache object is living longer
> than expected sounds correct.
> 
> Each read operation on a mutable file adds a new object into the
> ResponseCache even when there's already an entry for this file. This
> makes me think that this ResponseCache is not actually being used.
> 
> Would it be possible that some field of the verinfo tuple (such as
> offset) are different between calls to mutable.filenode._add_to_cache()
> and mutable.filenode._read_from_cache()?
> 
> On 10/22/2010 09:23 AM, Brian Warner wrote:
>> Or, maybe we should get rid of ResponseCache
>> altogether: I don't remember what performance improvement it tries to
>> provide.

If there were no cache at all for mutable files, each operation on a
mutable directory would redownload shares from the storage servers (note
that DirectoryNode._read in dirnode.py appears to be called for each
directory operation).

> I gave a try to this approach with the following change and the memory
> leak indeed disappeared!
> 
> http://github.com/ctrlaltdel/tahoe-lafs/commit/3a4dbdee1a3ea8a30559dd49f51dba40ba1661fb

Even if it were working as intended, the ResponseCache stores shares as
retrieved from storage servers. Wouldn't it be more efficient, in both memory
and compute cycles, to cache the reconstructed file rather than the shares?

Ah, I see that is ticket #465.

Since I think the main performance issue here is for directories, another
possibility is to cache the decoded directory contents in DirectoryNode,
rather than trying to cache mutable file shares in general.

(nodemaker.py create_from_cap will ensure that a NodeMaker creates only
one node object for a given cap. However, there will be separate
DirectoryNode objects and separate underlying MutableFileNode objects
for read and write caps to a directory -- see also ticket #1105. But this
would not cause a performance regression since the two MutableFileNode
objects would currently have separate ResponseCaches.)

There is no additional security problem created by cacheing plaintext in
memory, since the corresponding filecap is already in memory.

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 292 bytes
Desc: OpenPGP digital signature
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20101024/49fcbdfa/attachment.asc>