[tahoe-dev] timestamps on mutable files

Jody Harris imhavoc at gmail.com
Mon Jan 4 18:57:05 UTC 2010


IF the mutable file update problem is due to memory use problems with
downloading the file, keeping it in memory then re-uploading it, then (for
this use case) it is worth the overhead (to me) for now.

IF there are other issues that I need to be aware of, please let me know.

Question: Why was the decision made to download the file before uploading
the new version?

thanks,
jody
----
- Think carefully.
- Contra mundum - "Against the world" (St. Athanasius)
- Credo ut intelliga - "I believe that I may know" (St. Augustin of Hippo)


On Mon, Jan 4, 2010 at 11:33 AM, Peter Secor <secorp at allmydata.com> wrote:

> You are correct Kevan, the hard limit has been removed, sorry I didn't
> realize that had been fixed already!
>
> So, the limit is removed, but Jody you may want to read the ticket #694
> notes, there is a good discussion of why we originally had a hard-limit.
>
> Ps
>
> On 1/4/10 10:24 AM, Kevan Carstensen wrote:
> > I'm sure someone will correct me if I'm wrong, but I think #694 removed
> > the hard limit on mutable files. So your data should be backed up.
> >
> > (apologies for formatting - I'm on a phone)
> > --
> > Kevan Carstensen | <kevan at isnotajoke.com <mailto:kevan at isnotajoke.com>>
> >
> > On Jan 4, 2010, at 10:10 AM, Jody Harris <imhavoc at gmail.com
> > <mailto:imhavoc at gmail.com>> wrote:
> >
> >> Ack!
> >>
> >> So, I'm NOT backing up my data!? (I'm just backing up the first 2MB of
> >> each compressed table?)
> >>
> >> Okay, if immutable files are my only choice, how do I do garbage
> >> collection to insure that a minimum of dead space is taken up by
> >> "yesterday's" files?
> >>
> >> jody
> >> ----
> >> - Think carefully.
> >> - Contra mundum - "Against the world" (St. Athanasius)
> >> - Credo ut intelliga - "I believe that I may know" (St. Augustin of
> Hippo)
> >>
> >>
> >> On Mon, Jan 4, 2010 at 11:05 AM, Peter Secor <
> >> <mailto:secorp at allmydata.com>secorp at allmydata.com
> >> <mailto:secorp at allmydata.com>> wrote:
> >>
> >>     Mutable files currently have a small (2MB?) limit and cannot be
> >>     used for
> >>     larger files. Immutable files (the default) have a much higher limit
> >>     (12GB now?) and are much better suited for the purpose you
> >>     described of
> >>     large snapshots of data that won't change later.
> >>
> >>     There are ideas behind larger mutable files (#393) but they are not
> >>     implemented yet.
> >>
> >>     Ps
> >>
> >>     On 1/4/10 7:37 AM, Jody Harris wrote:
> >>     > Oh, I forgot to answer the "how?"
> >>     >
> >>     > ... put --mutable ...
> >>     >
> >>     > jody
> >>     > ----
> >>     > - Think carefully.
> >>     > - Contra mundum - "Against the world" (St. Athanasius)
> >>     > - Credo ut intelliga - "I believe that I may know" (St. Augustin
> >>     of Hippo)
> >>     >
> >>     >
> >>     > On Sun, Jan 3, 2010 at 10:59 PM, Jody Harris <
> >>     <mailto:imhavoc at gmail.com>imhavoc at gmail.com <mailto:
> imhavoc at gmail.com>
> >>     > <mailto: <mailto:imhavoc at gmail.com>imhavoc at gmail.com
> >>     <mailto:imhavoc at gmail.com>>> wrote:
> >>     >
> >>     > I have a script that:
> >>     > - iterates though my MySQL tables
> >>     > - dumps each table
> >>     > - pipes the dump through gzip
> >>     > - write the file to a temp file
> >>     > - copies the file to [remote storage] (now Tahoe grid)
> >>     > - deletes temp file
> >>     >
> >>     > This gives me a daily snapshot of all of the active databases on
> my
> >>     > server every night.
> >>     >
> >>     > The current backup is ~ 300 MB/night, which isn't bad, but after
> one
> >>     > year, that's 110+GB x 3.3 and counting of consumed tahoe space.
> >>     > Unless tahoe has a better garbage collection system than I have
> come
> >>     > to believe, that's going to be a problem on a 500 GB grid.
> >>     >
> >>     > In my reading of the tahoe docs, I have not come across any alarms
> >>     > concerning mutable files. Is there something I should be made
> >>     aware of?
> >>     >
> >>     > jody
> >>     > ----
> >>     > - Think carefully.
> >>     > - Contra mundum - "Against the world" (St. Athanasius)
> >>     > - Credo ut intelliga - "I believe that I may know" (St. Augustin
> of
> >>     > Hippo)
> >>     >
> >>     >
> >>     > On Sun, Jan 3, 2010 at 10:50 PM, Zooko O'Whielacronx
> >>     > < <mailto:zookog at gmail.com>zookog at gmail.com
> >>     <mailto:zookog at gmail.com> <mailto:
> >>     <mailto:zookog at gmail.com>zookog at gmail.com
> >>     <mailto:zookog at gmail.com>>> wrote:
> >>     >
> >>     > On Sun, Jan 3, 2010 at 10:33 PM, Jody Harris
> >>     > < <mailto:havoc at harrisdev.com>havoc at harrisdev.com
> >>     <mailto:havoc at harrisdev.com> <mailto:
> >>     <mailto:havoc at harrisdev.com>havoc at harrisdev.com
> >>     <mailto:havoc at harrisdev.com>>> wrote:
> >>     > > I have set up my web server to store backup files on my tahoe
> >>     > grid, shifting
> >>     > > from the use of a server at my house. One thing that
> >>     > initially alarmed me
> >>     > > was that the timestamps of the backed up files is not updated
> >>     > in the
> >>     > > tahoe-lafs interface. I did confirm that the shares
> >>     > timestamps are updated.
> >>     >
> >>     > Tahoe-LAFS doesn't store timestamps or other metadata on files,
> only
> >>     > on links to files, which links are stored in directories. So if
> you
> >>     > have a directory which contains a link named "README.txt" to
> >>     > version 1
> >>     > of your file, and then you upload version 2 of your file and you
> >>     > change that directory to link to version 2, then Tahoe-LAFS will
> >>     > update the timestamp on that link named "README.txt" when it
> updates
> >>     > the link to point to the new file.
> >>     >
> >>     > Did you say "mutable files"? You should probably not be using
> >>     > mutable
> >>     > files for anything. Immutable files (plus directories) probably
> fit
> >>     > all your needs for backups, and mutable files have significant
> >>     > performance problems.
> >>     >
> >>     > How do you trigger these backups -- are you using "tahoe cp" or
> >>     > "tahoe
> >>     > backup" or sending HTTP requests to the WAPI or something else?
> >>     >
> >>     > Regards,
> >>     >
> >>     > Zooko
> >>     > _______________________________________________
> >>     > tahoe-dev mailing list
> >>     > <mailto:tahoe-dev at allmydata.org>tahoe-dev at allmydata.org
> >>     <mailto:tahoe-dev at allmydata.org> <mailto:
> >>     <mailto:tahoe-dev at allmydata.org>tahoe-dev at allmydata.org
> >>     <mailto:tahoe-dev at allmydata.org>>
> >>     >
> >>     <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev>
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> >>     >
> >>     >
> >>     >
> >>     >
> >>     >
> >>     > _______________________________________________
> >>     > tahoe-dev mailing list
> >>     > <mailto:tahoe-dev at allmydata.org>tahoe-dev at allmydata.org
> >>     <mailto:tahoe-dev at allmydata.org>
> >>     >
> >>     <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev>
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> >>     _______________________________________________
> >>     tahoe-dev mailing list
> >>     <mailto:tahoe-dev at allmydata.org>tahoe-dev at allmydata.org
> >>     <mailto:tahoe-dev at allmydata.org>
> >>     <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev>
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> >>
> >>
> >> _______________________________________________
> >> tahoe-dev mailing list
> >> tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>
> >> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> >
> >
> > _______________________________________________
> > tahoe-dev mailing list
> > tahoe-dev at allmydata.org
> > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at allmydata.org
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20100104/390070b8/attachment.html>


More information about the tahoe-dev mailing list