[tahoe-dev] building and pre-existing dependencies
gdt at ir.bbn.com
Fri Jul 23 18:30:56 UTC 2010
> I am trying to package tahoe for pkgsrc, the native packaging system on
> NetBSD and Dragonfly, also portable to Darwin, Linux, Solaris, OpenBSD,
> FreeBSD, IRIX, AIX, and a few others.
Sweet! I would be happy to help.
Thanks very much - your egg hint is I think what I needed.
From a packager viewpoint, it would be nice if tahoe's setup.py had an
option --no-recurse (name not important) to tell it to fail instead of
trying to download and build other packages. I can see why the
automatic behavior is useful for users but it works against bringing the
whole build process under CM (sources identified, repeatable builds, no
net use during build, etc.) I'd rather have it fail and then go package
the other python dependency and depend on it.
> I don't understand how setup.py decides if a particular python package
> is already present or needs to be downloaded and built.
In theory, it looks for a file or directory named
$DISTRIBUTIONNAME-$VERSION-py$PYTHONVERSION.egg-info in any directory
which is on the Python sys.path. In practice there are a few known
bugs in different versions of different tools which might cause it to
do this re-download behavior that you observed even if the .egg-info
file is present.
Yes, that is exactly the issue.
An import thing to keep in mind is that "Crypto" is the name of a
Python module (which is sometimes confusingly called a Python
"package") i.e., something that can be imported. "pycrypto" is the
name of the Python package (which is always confusingly called a
Python "distribution") i.e. something that can be installed onto a
system and that another distribution (package) can depend on.
So, to test whether the Python "distribution" is installed under the
expected name and version number it doesn't help to import Crypto,
which only tells you whether the Python module is importable. To see
whether the Python "distribution" is installed you have to do this:
Wow - that makes sense but is not obvious from anything I'd managed to
read so far.
python -c 'import pkg_resources;print pkg_resources.require("pycrypto")'
That failed, with my security/py-crypto package. py-asn1 uses egg.mk
and seems ok
(http://www.netbsdpkgsrc.info/packages/security/py-asn1.html). I had
used extension.mk instead of egg.mk because of trouble with
Now you could if you want patch the Tahoe-LAFS setup.py to not use
this mechanism at all and just rely on the pkgsrc dependency
mechanism. That would be okay, but it would be one more patch you'd
have to maintain, and other Python tools besides the Tahoe-LAFS
setup.py might likewise be depending on the .egg-info. So it might be
good to get to the bottom of why there isn't a
pycrypto-$VER-py$PYVER.egg-info on your sys.path.
Agreed; the missing egg-info is the real bug, and other such files have
been placed there by pkgsrc. Apparently there is something odd about
gdt 670 /usr/pkg/lib/python2.6/site-packages > l -d *egg*
drwxr-xr-x 2 root wheel 512 Jul 20 19:59 Twisted-10.0.0-py2.6.egg-info
drwxr-xr-x 2 root wheel 512 Jul 16 15:43 bsddb3-5.0.0-py2.6.egg-info
drwxr-xr-x 2 root wheel 512 Jul 20 19:59 pyasn1-0.0.11a-py2.6.egg-info
-rw-r--r-- 1 root wheel 623 Jul 23 13:45 pycrypto-2.1.0-py2.6.egg-info
drwxr-xr-x 2 root wheel 512 Jul 22 16:24 pycryptopp-0.5.19-py2.6.egg-info
drwxr-xr-x 2 root wheel 512 Jul 16 15:43 setuptools-0.6c11-py2.6.egg-info
drwxr-xr-x 2 root wheel 512 Jul 20 19:58 zope.interface-3.6.1-py2.6.egg-info
I now think there are two kinds of egg-aware packages: 1st-class ones
that take --single-version-externally-managed, and older ones that just
install an egg file, and that pkgsrc's egg.mk is only about the first
Here's my question to the pkgsrc technical list:
From: Greg Troxel <gdt at ir.bbn.com>
Subject: python egg info
To: tech-pkg at netbsd.org
Date: Fri, 23 Jul 2010 14:07:24 -0400
I am packaging some python modules/packages in order to have
dependencies for tahoe-lafs, and finding egg metadata file usage in
pkgsrc confusing. The particular package in question is
After including lang/python/egg.mk, 'make package' (DESTDIR mode) fails
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: option --single-version-externally-managed not recognized
If I comment out that argument in egg.mk, the package builds, but then I have
no installed egg info.
In python26/patches/patch-av, installation of egg metadata files is
disabled unless PY_NO_EGG is defined to other than yes. Setting it to
NO in the package causes egg metadata to be installed. It seems the
python way is to install these files so that the python requirement
system works. There are other egg metadata files installed.
With removing --single-version-externally-managed and setting PY_NO_EGG
to NO, I get an egg metadata file installed and require works:
> python2.6 -c 'import pkg_resources;print pkg_resources.require("pycrypto")'
[pycrypto 2.1.0 (/usr/pkg/lib/python2.6/site-packages)]
1) Why does PY_NO_EGG default to yes? Should it be set to NO in egg.mk?
Or is that about not installing egg metadata in the package directory
2) Is a python module that has setup.py and doesn't implement
My tentative conclusions are:
A) that --single-version-externally-managed is the python way for an
egg-aware package to be told
1) there will only be one version of this package installed, and
2) the egg data should be in .../site-package so that it will be found
B) py-crypto is not really 'egg aware' in the modern mainstream sense,
but in some more limited sense that still allows require to find it.
Do we need to all EGG_OLD_STYLE=YES to cause egg.mk to
not add --single-version-externally-managed
not set PY_NO_EGG to YES
Or perhaps there's another approach - I think the odds my understanding
is correct is only about 50%.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 194 bytes
Desc: not available
More information about the tahoe-dev