[tahoe-dev] Go package for Zfec

[Simon Liu]

Hi,

I've updated go-zfec with parallel encoding on a valid io.Reader.  The
underlying assumption is that the C function fec_encode() is thread-safe.

The go-zfec-cmdline tool has been updated with a --numcpus option.
Otherwise, the go version should be fully compatible with the Python
version.

Some benchmarks from the go-zfec README are copied below.

======================================

EncodeReaderParallel() lets you take advantage of multiple cores. From
running some test benchmarks using the Python and Go versions of the
zfec command line tool, we find that on a multi-core processor the Go
zfec can run twice as fast as the Python zfec.

Linux 3.8.0-32-generic (x86_64), Core i7-2640M
- Python 2.7.4
- Go version devel +f4d1cb8d9a91
- Data file is xubuntu-13.10-desktop-amd64.iso (882,900,992 bytes)

Encoding time with k=3, m=8 (times are average of 3 runs, lower is better)
Python = 7.49733 secs
Go 1 core = 5.51867
Go 2 cores = 4.92233
Go 4 cores = 3.635

Encoding time with k=10, m=30
Python = 15.05467 secs
Go 1 core = 14.96633
Go 2 cores = 9.86467
Go 4 cores = 8.26233

======================================

Cheers,
Simon

> 
> There is also a go port (as opposed to go bindings for the C implementation)
> here: https://github.com/korvus81/jfec
> 
> (I have not used it; I just discovered it the other day while looking for a
> link to zfec to send to this list.)
> 
> There is also a JavaScript implementation here:
> https://github.com/richardkiss/js_zfec
> 
> It would be nice if there was a single place on the web with information about
> all of these, perhaps https://tahoe-lafs.org/trac/zfec ?
> 
> Right now that page has an error about trac not having the darcs plugin.
> 
> ~leif
>