[tahoe-dev] [tahoe-lafs] #905: gather information about historical server performance

tahoe-lafs trac at allmydata.org
Fri Jan 15 06:09:23 UTC 2010

#905: gather information about historical server performance
 Reporter:  warner        |           Owner:           
     Type:  enhancement   |          Status:  new      
 Priority:  major         |       Milestone:  undecided
Component:  code-network  |         Version:  1.5.0    
 Keywords:  performance   |   Launchpad_bug:           
 While patiently uploading some relatively small (3MB) files to the
 I found myself wishing for more specific information about how long each
 server was taking to respond. Some part of me wanted to blame specific
 servers for being slow, but I realize that my upstream bandwidth is
 and I'd like to compare the upload time of each segment against the
 possible upload time given my available bandwidth. So I want more
 about server performance.

 The download staus web page (via "recent uploads and downloads") reports
 per-server per-segment fetch response times as a big list of numbers. It's
 possible to eyeball this and look for trends, for example in the following

 * Per-Server Segment Fetch Response Times:
     * {{{hk475awa}}}: 1.56s, 876ms, 714ms, 705ms, 545ms, 545ms, 539ms,
 579ms, 392ms, 386ms, 377ms, 371ms, 372ms, 387ms, 555ms, 388ms, 373ms,
 373ms, 372ms, 372ms, 370ms, 373ms, 370ms, 370ms
     * {{{lwkv6cji}}}: 160ms, 103ms, 105ms, 106ms, 100ms, 119ms, 117ms,
 105ms, 106ms, 109ms, 102ms, 107ms, 103ms, 1.02s, 91ms, 93ms, 90ms, 93ms,
 90ms, 92ms, 89ms, 93ms, 89ms, 80ms
     * {{{2gn6njsm}}}: 763ms, 536ms, 520ms, 537ms, 537ms, 548ms, 537ms,
 534ms, 527ms, 535ms, 528ms, 525ms, 628ms, 532ms, 541ms, 528ms, 530ms,
 528ms, 534ms, 527ms, 528ms, 527ms, 529ms, 448ms

 you can tell that {{{lwkv6cji}}} was pretty quick, and that {{{2gn6njsm}}}
 was 5-6x slower.

 I'd like three improvements on this:

  * collect data on uploads, not just downloads
  * display data in some sort of graphical form, to make it easier to spot
    trends. Maybe segnum should be the X axis, reponse time is the Y axis,
    serverid is color, and each sample should be drawn as a dot. Servers
    gave consistent service would show up as horizontal stripes. Common-
    delays would appear as spikes.
  * connect+display data across multiple uploads and downloads. This would
    involve storing some history about each server, maybe in a sqlite
    or something.

 I've also been thinking about this in the context of a new Downloader
 in which I want to sort potential servers according to their likely
 speeds (favoring fast servers, but experimenting with slower ones to
 the load and learn about alternatives). Mainly I want the Downloader to be
 able to tell the difference between a server that's running at normal
 one that's running unusually slowly, and one that's disconnected, and I
 historical data will help here.

 Also, with historical data, we might be able to deduce each server's local
 upload/download speed limits by looking for a minimum reponse time for
 given-sized messages. We might use this to build up a list of download
 servers with an aggregate bandwidth that matches our own download
 (exactly filling all the pipes), or to influence share placement during

 Another challenge is that currently the only data we get is response time
 each request, which combines outbound request size (and contention for the
 wire), server response turnaround time (i.e. CPU load, disk IO latency,
 and response size (and wire contention). In some ways, we want to be able
 distinguish between those components. In other ways, we really only care
 about the sum.

 But mainly, distinguishing between "slow" and "disconnected" may work
 if we had finer-grained information, like how many bytes have arrived at
 socket since the last complete message was received, and how long ago the
 last byte arrived. Likewise, when sending data, it would be useful to know
 that the transmit buffer still has pending data, and how long it's been
 the last socket write was allowed. This would help tell the difference
 between a connection that is alive but only trickling data through slowly,
 versus one that has been stopped for the last 10 seconds. With only
 per-message data (which, for 3-of-10 and 128KiB segments, means about 40kB
 per message), it might be hard to confidently declare a disconnect within
 reasonable multiple of the expected reponse time.

 I don't know exactly what sorts of data I'd want or how much of it to
 This ticket is to collect ideas on what to collect and what to do with it.
 The first concrete improvement would be to record per-request response
 for upload just like we do for download. The second would be to graph

Ticket URL: <http://allmydata.org/trac/tahoe/ticket/905>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid

More information about the tahoe-dev mailing list