[Tahoe-dev] announcing Allmydata Tahoe

[zooko at zooko.com]

Dear p2p-hackers:

Allmydata, Inc. [1], provider of the "Allmydata" consumer backup product, is
pleased to announce the first public release of "Tahoe", a secure,
distributed storage grid with a free-software licence.

The source code that we are releasing is the current working prototype for
Allmydata's next-generation product.  This release is targeted at hackers who
are willing to use a minimal, text-oriented web user interface.

This prototype is not recommended for storage of confidential data nor for
data which is not otherwise backed up, but it already implements a functional
distributed storage grid and is useful for experimentation, prototyping, and
extension.


LICENCE

Tahoe is offered under the GNU General Public License (v2 or later), with the
added permission that, if you become obligated to release a derived work
under this licence (as per section 2.b), you may delay the fulfillment of
this obligation for up to 12 months.


INSTALLATION

This release of Tahoe works and passes all unit tests on Linux/x86,
Linux/amd64, Mac/Intel, Mac/PPC, Windows-native, and Cygwin.

To install, download the tarball [2], untar it, go into the resulting
directory, and follow the directions in the README [3].


USAGE

Once installed, create a "client node".  Instruct this client node to connect
to a specific "introducer node" by means of config files in the client node's
working directory.  To join a public grid, copy in the .furl files for that
grid.  To create a private grid, run your own introducer, and copy its .furl
files.  See the README for step-by-step instructions.

Each client node runs a local webserver (enabled by writing the desired port
number into a file called 'webport').  The front page of this webserver shows
the node's status, including which introducer is being used and which other
nodes are connected.  Links from the status page lead to others that give
access to a shared virtual filesystem, in which each directory is represented
by a separate page.  Each directory page shows a list of the files available
there, with download links, and forms to upload new files.

Other ways to access the filesystem are planned, as well as other structures
than the single globally-shared namespace implemented by this release: please
see the roadmap.txt [5] for some rough details.


HACKING AND COMMUNITY

Please join the mailing list [4] to discuss the ideas behind Tahoe and
extensions of and uses of Tahoe.  Patches that extend and improve Tahoe are
gratefully accepted -- roadmap.txt shows the next improvements that we plan
to make.  You can browse the revision control history, source code, and issue
tracking at the Trac instance [6].  Please see the buildbot [7], which shows
how Tahoe builds and passes unit tests on each checkin, and the code coverage
results [8] and percentage-covered graph [9], which show how much of the
Tahoe source code is currently exercised by the test suite.


NETWORK ARCHITECTURE

Each peer maintains a connection to each other peer.  A single distinct
server called an "introducer" is used to discover other peers with which to
connect.

To store a file, the file is encrypted and erasure coded, and each resulting
share is uploaded to a different peer.  The secure hash of the encrypted file
and the encryption key are packed into a URI, knowledge of which is necessary
and sufficient to recover the file.

To fetch a file, starting with the URI, a subset of shares is downloaded from
peers, the file is reconstructed from the shares, and then decrypted.

A single distinct server called a "vdrive server" maintains a global mapping
from pathnames/filenames to URIs.

We are well aware of the limitations of decentralization and scalability
inherent in this prototype.  In particular, the completely-connected property
of the grid and the requirement of a single distinct introducer and vdrive
server limits the possible size of the grid.  We have plans to loosen these
limitations (see roadmap.txt).  Currently it should be noted that the grid
already depends as little as possible on the accessibility and correctness of
the introduction server and the vdrive server.  Also note that the choice of
which servers to use is easily configured -- you should be able to set up a
private grid for you and your friends almost as easily as to connect to our
public test grid.


SOFTWARE ARCHITECTURE

Tahoe is a "from the ground-up" rewrite, inspired by Allmydata's existing
consumer backup service.  It is primarily written in Python.

Tahoe is based on the Foolscap library [10] which provides a remote object
protocol inspired by the capability-secure "E" programming language [11].
Foolscap allows us to express the intended behavior of the distributed grid
directly in object-oriented terms while relying on a well-engineered, secure
transport layer.

The underlying networking is provided by the Twisted library [12].
Computationally intensive operations are performed in native compiled code,
such as the "zfec" library for fast erasure coding (also available
separately: [13]).


[1]  http://allmydata.com
[2]  http://allmydata.org/source/tahoe/tahoe-0.2.0-0-UNSTABLE.tar.gz
[3]  http://allmydata.org/trac/tahoe/browser/README
[4]  http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
[5]  http://allmydata.org/trac/tahoe/browser/roadmap.txt
[6]  http://allmydata.org/trac/tahoe
[7]  http://allmydata.org/buildbot
[8]  http://allmydata.org/tahoe-figleaf/figleaf/
[9]  http://allmydata.org/tahoe-figleaf-graph/hanford.allmydata.com-tahoe_figleaf.html
[10] http://twistedmatrix.com/trac/wiki/FoolsCap
[11] http://erights.org/
[12] http://twistedmatrix.com/
[13] http://allmydata.org/trac/tahoe/browser/src/zfec