[tahoe-dev] on backward-compatibility and new features
zooko at zooko.com
Fri Nov 20 20:01:57 UTC 2009
[N.B. I pressed Brian about this in conversation on IRC yesterday,
arguing that it wouldn't hurt to make users explicitly enable
immutable-dirs-in-backups before they could use them.]
On Thursday, 2009-11-19, at 22:01 , Brian Warner wrote:
> * "tahoe backup" is incredibly faster with immutable directories
> * the directories it creates are repairable
> * The only directory that has a DIR2-CHK objects placed into it is
> the top-level target of the "tahoe backup" command
> * I don't believe that "tahoe backup" is the tool of choice for
> sharing files
> So yeah, if we were talking about changing "tahoe mkdir" to create
> new caps, then I'd agree with a conservative approach and add a
> config knob to slowly introduce the new feature over time. But for
> "tahoe backup", I think the benefits outweigh the downsides.
Okay, good arguments! Let's do it your way!
Now about the more general principle for future reference:
>> I would like people to get used to the idea that they can always
>> upgrade to the new version of Tahoe-LAFS and continue sharing with
>> their more conservative friends without having to even think
>> about these issues.
> Ok, but then how is it possible to ever introduce new features?
By an explicit action on the part of the user. This could be setting
a configuration option or (better) just clicking on a different
button in the WUI or invoking a different command in the CLI or
adding a "--newfangled-awesomeness" option in the CLI. Also, with
sufficient passage of time we -- the Tahoe-LAFS developers -- can
observe that people have in practice upgraded to a sufficiently new
version that we can make the new feature default on in the next
release of Tahoe-LAFS after that. (See below about that.)
> A good concrete example would be Elk Point..
N.B. Elk Point won't necessarly be the next cap format. There is a
project to invent a New Cap format , and Elk Point is one
candidate for that design. I really like Elk Point -- a lot! -- but
we might end up implementing a different candidate (at least at
first), or we might end up implementing a subset of Elk Point at
first for reasons of simplicity -- ease of implementation,
verification, testing, deployment.
The next step on New Cap Design project is, I think, to document the
alternatives in a comparable level of detail to the current Elk Point
docs -- svg diagrams, What Could Go Wrong , etc.
But yes, a good concrete example would be Elk Point or some other New
> This could mean that some snazzy new features might be implemented
> and shipped but go effectively unused for a year or more, depending
> upon the release schedule and our overall patience with old versions.
Be careful about this assumption that seems to be implicit in this
statement -- that new features go unused if they are not turned on by
default. If the feature isn't desirable enough that the user learns
about it and explicitly indicates the willingness to use it (at the
cost of sharing files with her stick-in-the-mud friends), then
perhaps that means it's just not that important. They can have it by
default a few years later. :-) But I suspect that good new features
*would* get used before they are turned on by default.
For example, suppose New Caps come out in Tahoe-LAFS v1.7, are faster
to create and use, offer better security, and generate smaller,
prettier, and more usable caps. Suppose that the "create a new
directory" button continues to produce old-caps just like it did in
Tahoe-LAFS v1.6. But, suppose that there is a checkbox next to it
that says "new cap format". Users can easily (without even reading
the NEWS.txt) figure out how the button behaves differently with and
without the checkbox, figure out that their old-version-using friends
can't read the new caps, and then decide for themselves whether to
keep using old-caps or to persuade their friends to upgrade.
Let me restate my use case now that you've persuaded me that
immutable dirs in backupdb is okay. It's not that new features have
to be disabled by default, it's that:
A user wants to know if she can upgrade to the new release of Tahoe-
LAFS, briefly glance at the NEWS.txt file, and continue to
interoperate with her old-version-using friends, without
significantly altering her workflow.
I still think this use case is important, because if we miss it by
too much then our user base splits into two camps and we have to
either support both or abandon the stick-in-the-muds. I think it is
possible to really screw up here by overestimating the amount of work
users are willing to undertake in order to upgrade. For example, the
darcs project made this mistake and now they have two repository
formats to support and so far nobody has figured out how to get the
stick-in-the-muds (which includes us) to make the jump to the new
format. For another example, the Python project made a move like
this (the backwards-incompatible Python 3), and I'm still waiting to
find out how many years, if ever, before stick-in-the-muds like us
make the jump to Python 3.
But, I think you're right that there are some cases, such as
immutable-dirs in backupdb, where the current user base and their
current workflows don't run afoul of the backwards-compatibility issue.
By the way, I checked and allmydata.com customers who use the
allmydata.com Windows client are still using Tahoe-LAFS v1.1.0, which
came out in June of 2008, plus a few critical bugfixes to the WUI
which were backported from newer releases of Tahoe-LAFS. (A few of
the "power user" customers of allmydata.com use Tahoe-LAFS v1.4.1 or
v1.5.0 with the allmydata.com grid, on Linux and on Mac OS X.)
I'm fairly sure that none of those allmydata.com-Windows-client-using
users wants to interoperate with modern Tahoe-LAFS users, so I'm not
worried about that, but it is interesting to note. I wouldn't be
surprised if the addition of immutable directories is the final straw
that prompts allmydata.com to make the jump to a modern version of
Tahoe-LAFS, bringing them up to Tahoe-LAFS v1.6. (It would improve
client performance and reduce load on the allmydata.com storage grid
The other pool of potential stick-in-the-muds is Ubuntu users. The
current version of Ubuntu -- 9.10 (Karmic Koala) -- comes with Tahoe-
LAFS v1.5. I think we'll be able to get Tahoe-LAFS v1.6 into Ubuntu
10.04 if we pay attention to their schedule . If that happens
then the people who use Tahoe-LAFS as packaged by Ubuntu will
probably upgrade to Tahoe-LAFS v1.6 because Ubuntu 10.04 is an LTS
(Long-Term-Support) release. After that, a lot of them will not
upgrade to a newer Ubuntu for many years. Whatever version of Tahoe-
LAFS ships in Ubuntu 10.04 will remain in widespread use for at least
>> This is also why "forward-compatibility" features are so important
>> to me. They can sometimes ease these transitions.
> Yes! I'm very glad you pushed for #708 (tolerate unknown nodes),
> because having it implemented in v1.5 made DIR2-CHK much easier to
> deploy in v1.6, and makes me feel more confident about things like
> Elk Point.
Yes! That worked out well! Now think about what features you would
like to add to Tahoe-LAFS v2.0, and then think about what we can do
in v1.6 or v1.7 to provide forward-compatibility with those
More information about the tahoe-dev