update on Tahoe/Tor/Foolscap integration

[Brian Warner]

We had a couple of meetings this week to talk about Tor. The current
plan is to switch to a more "static" node configuration process, meaning
that any (port, IP address, onion address) allocation will happen during
"tahoe create-node", so everything will be nailed down by the time
"tahoe start" happens and the Node object is created. This should
simplify Tahoe's startup code, and will allow some configuration errors
to be signalled (to stdout) before "tahoe start" exits (rather than only
appearing in the logfile).

The biggest consequences of this cleanup are:

* "tahoe create-client" won't listen on a socket at all[1]
* When you run "tahoe create-node" to make a server, you must choose a
  port to listen on, and must provide the hostname+port to advertise to
  other clients. Previously these were computed automatically.

We'll maintain backwards-compatibility for nodes that currently use
automatic IP-address detection (where tahoe.cfg's "tub.location" has
"AUTO" or is omitted), but for new nodes, "tahoe create-node" will
complain if you omit the new --listen and --location arguments.

The general plan is:

* "tahoe create-client/create-node" acquires --listen and --location,
  which get written into tahoe.cfg directly.
* They also acquire some new Tor-related arguments. We're not sure what
  they should be just yet.. these are placeholder names.
  * --tor-only: don't use plain TCP to talk to anybody
  * --listen-on-tor: please allocate a hidden service, and advertise an
    .onion address
  * --tor-socksport=: here is a SOCKS port you can use to talk to
    regular IP addresses, as well as .onion hidden-services
  * --tor-controlport=: use this to control a pre-existing Tor daemon
  * --tor-executable=: here is a /usr/bin/tor, please launch it
* Any hidden services or listening ports or whatever will be figured out
  while create-client/create-node runs, and the results will be written
  into tahoe.cfg (and other state files underneath ~/.tahoe/). At
  runtime, the Tahoe node might launch Tor, but it doesn't need to
  reconfigure it.

There'll be one set of arguments that make it possible to use tor:
either instructions to launch a new executable, or a way to reach an
existing executable. If these arguments are missing, that means we don't
have Tor, and won't be able to use Tor-only servers.

There'll be another argument that says we care about the anonymity of
our node, which means the storage server should listen on a hidden
service instead of a regular TCP port, and we should strictly use Tor
for all outbound connections.

We need to write up the main use cases and make sure we've got a
combination of arguments for each.


## Foolscap tasks:

* specify client plugin API
* write "default" (TCP) plugin
* deprecate/remove setLocationAutomatically
* add allocate_port() function

## Tahoe tasks:

* move controlport/logport to a separate tub, with a separate key, add
  config knob to control its listen/location, maybe use 127.0.0.1 and
  allocate_port()

* replace keyserver/stats-gatherer/tests use of
  Tub.setLocationAutomatically with something else

* tub.location=AUTO: replace async (twisted.internet.subprocess) with
  sync (stdlib.subprocess), move call from Node._setup_tub to
  Node.create_tub. Move Tub.setLocation to create_tub. Replace
  when_tub_ready() with sync calls.

* (maybe) change "tahoe create-client/node" to run reactor, if it needs
  to allocate a port (maybe for the logport/controlport?)

* change "tahoe create-client" to make a tahoe.cfg that doesn't listen
  on anything

* change "tahoe create-node" to require --listen and --location

* change "tahoe create-node" to handle the tor arguments

* add Tor support to the main tahoe code: launch Tor if necessary,
  configure and install the Foolscap plugins if necessary

* (after create-node changes are done): remove write to
  Node._portnumfile

## Other tasks:

* figure out overrides for specific servers: either build a foolscap
  client plugin that knows how to look up the override rules to decide
  whether a given connection hint should use TCP or Tor, or use multiple
  Tubs (one with the Tor plugin, and one without). We need a syntax for
  the rules file, and some sane behaviors to follow when merging these
  rules with data we get from the introducer.

## More Info

* Foolscap #236 covers the foolscap-side items.

* Tahoe #517 covers the Tahoe-side items.

* Tahoe #1010 is specifically about the "protect me" flag, which will
  probably not be spelled with "tor" in the name, since e.g. i2p would
  qualify.


cheers,
 -Brian


[1]: It *will* still listen on a "localhost" socket for the
     logport and controlport, but these aren't advertised to the
     introducer, and remote systems can't connect to them.
[foolscap#236]: http://foolscap.lothar.com/trac/ticket/236
[tahoe#517]: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517
[tahoe#1010]: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1010