[tahoe-dev] Tahoe as glue filesystem

[Duncan McGreggor]

On Mon, Jul 7, 2008 at 2:23 PM, Valentino Volonghi <dialtone at gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On Jul 7, 2008, at 11:24 AM, Brian Warner wrote:
>
>>> Man, it was great to hear you ask this question. The possibility of
>>> this for use in small, distributed, diskless devices is what most
>>> recently prompted me to take another look at tahoe. Of course I'm
>>> staying for the party anyway, but I am *deeply* interested in a
>>> memory- only option for tahoe.
>>
>> Hm, I suppose it depends upon how much storage you have to work
>> with. RAM or
>> flash? Swapping out the allmydata.storage module for something that
>> keeps all
>> shares in a dictionary instead of in disk would be pretty easy.. we
>> actually
>> have some test cases that do just that (to test upload/download code
>> in
>> isolation from the storage code). But I think it would be hard to
>> achieve
>> good reliability if your share integrity depends upon continuous
>> power,
>> especially if the nodes are small (and likely to be portable or
>> battery-powered).
>
> Well the basic idea is to avoid going to disk when data is temporary or
> short lived. And reusing the fail-over attributes of tahoe would make
> the
> users of tahoe careless of the way you store data.
>
> Basically, since my usecase is a mapreduce, I'm pretty happy with
> uploading
> tons of logfiles to tahoe on a cluster of servers, since it's mostly a
> write-once-read-all-the-other-times kind of thing.

You wounldn't, by any chance, be working with Tommi on this, would you?

In particular, I'm referring to his article on an adjusted mapreduce:
  http://eagain.net/articles/incremental-mapreduce/

d