buildout.recipe.isolation 0.2.0

Recipe deliverables

• A directory that contains a specified distribution(s) package and its dependency package(s).

• A directory that contains a .pth file, which lists the absolute path for each package in the isolation context.

• A directory that contains the scripts that have been generated from the distribution(s) package and its dependency packages.

Dependency exclusion

Let’s now try a buildout with a slightly larger example that we can use to illustrate the exclude dependencies from a certain isolation.

Let’s create a new buildout configuration based on the previous one. This configuration is setup to isolate the bigdemo distribution and its dependencies, but exclude the demoneeded dependency.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts =
...     demoneeded
...     demo
... find-links = %(server)s
... index = %(server)s/index
...
... [demoneeded]
... recipe = buildout.recipe.isolation
... dists = demoneeded
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = bigdemo
... exclude-dists = ${demoneeded:dists}
... """ % dict(server=link_server))

>>> print system(buildout), #doctest: +ELLIPSIS
Uninstalling demo.
Installing demoneeded.
demoneeded: Copying demoneeded to the destination directory.
Installing demo.
Getting distribution for 'bigdemo'.
Got bigdemo 0.1.
Getting distribution for 'demo'.
Got demo 0.4c1.
demo: Copying demo to the destination directory.
demo: Copying bigdemo to the destination directory.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.

Check the isolated results:

>>> ls(sample_buildout, 'parts/demo')
-  bigdemo-0.1-py2.6.egg
-  demo-0.4c1-py2.6.egg
>>> ls(sample_buildout, 'parts/demoneeded')
d  demoneeded-1.2c1-py2.6.egg

Script generation

Some distributions supply command-line scripts with there packages. Buildout typically generates these scripts for us, because it needs to supply the built packages to to script. It does this by injecting the distribution locations into the Python system path. In some cases we do not want to inject anything into the Python system path, because we may have deposited the generated .pth file in a virtual environment’s site-packages directory. While in other cases, we might want to supply our .pth file as a mean for import resolution. Let’s take a closer look at both cases.

For the general case, we will likely want to supply our .pth file to the script. Additionally, we will probably want to supply any .pth files that dependent isolations may have generated. Here is an example.

>>> import sys
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts =
...     demoneeded
...     demo
... find-links = %(server)s
... index = %(server)s/index
...
... [demoneeded]
... recipe = buildout.recipe.isolation
... dists = demoneeded
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = bigdemo
... exclude-dists = ${demoneeded:dists}
... extra-pth = ${demoneeded:pth-file-location}
... executable = %(python)s
... """ % dict(server=link_server, python=sys.executable))
>>> print system(buildout), #doctest: +ELLIPSIS
Uninstalling demo.
Uninstalling demoneeded.
Installing demoneeded.
demoneeded: Copying demoneeded to the destination directory.
Installing demo.
demo: Copying demo to the destination directory.
demo: Copying bigdemo to the destination directory.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.

The resulting script should have two .pth files in it. The demo.pth file has been defined and generated from the recipe in context. The demoneeded.pth file was generated by the demoneeded section and pulled in using the extra-pth recipe option.

>>> if sys.platform == 'win32':
...    script_name = 'demo-script.py'
... else:
...    script_name = 'demo'
>>> script_dir = 'parts/demo-scripts'
>>> f = open(os.path.join(sample_buildout, script_dir, script_name))
>>> shebang = f.readline().strip()
>>> if shebang[:3] == '#!"' and shebang[-1] == '"':
...     shebang = '#!'+shebang[3:-1]
>>> shebang == '#!' + os.path.realpath(sys.executable)
True
>>> print f.read(), # doctest: +NORMALIZE_WHITESPACE
<BLANKLINE>
import sys
def pth_injector(pth_file):
    path_file = open(pth_file, 'r')
    sys.path[0:0] = [line
        for line in path_file.read().split('\n')
        if line is not None]
<BLANKLINE>
pth_files = ['/sample-buildout/parts/demo-pth/demo.pth', '/sample-buildout/parts/demoneeded-pth/demoneeded.pth']
for pth in pth_files:
    pth_injector(pth)
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
    eggrecipedemo.main()
>>> f.close()

The second case is where we have deposited the .pth files into a virtual environment. Let’s setup a fake virtual environment structure inside the buildout structure for demonstration sake.

>>> virtenv = os.path.join(sample_buildout, 'virtenv')
>>> mkdir(virtenv)
>>> mkdir(virtenv, 'bin')
>>> mkdir(virtenv, 'lib')
>>> mkdir(virtenv, 'lib', 'python2.6')
>>> mkdir(virtenv, 'lib', 'python2.6', 'site-packages')
>>> site_pkgs = os.path.join(virtenv, 'lib', 'python2.6', 'site-packages')

All we really need for the purpose of this demonstration is the site-packages directory.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts =
...     demoneeded
...     demo
... find-links = %(server)s
... index = %(server)s/index
...
... [demoneeded]
... recipe = buildout.recipe.isolation
... dists = demoneeded
... pth-file-location = %(site_pkgs)s
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = bigdemo
... exclude-dists = ${demoneeded:dists}
... pth-file-location = %(site_pkgs)s
... exclude-own-pth = trUE
... python = %(python)s
... """ % dict(server=link_server, python=sys.executable,
...     site_pkgs=site_pkgs))
>>> print system(buildout), #doctest: +ELLIPSIS
Uninstalling demo.
Uninstalling demoneeded.
Installing demoneeded.
demoneeded: Copying demoneeded to the destination directory.
Installing demo.
demo: Copying demo to the destination directory.
demo: Copying bigdemo to the destination directory.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.

Now if we print out the demo script, we’ll find no mention of the .pth files.

>>> f = open(os.path.join(sample_buildout, script_dir, script_name))
>>> shebang = f.readline().strip()
>>> if shebang[:3] == '#!"' and shebang[-1] == '"':
...     shebang = '#!'+shebang[3:-1]
>>> shebang == '#!' + os.path.realpath(sys.executable)
True
>>> print f.read(), # doctest: +NORMALIZE_WHITESPACE
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
    eggrecipedemo.main()
>>> f.close()

Why does this work? If we were to use the virtual environments Python executable, it would load the site-packages directory and any .pth files in it. This would in turn load the modules we built using the buildout.

Staging the isolation

In some situations it is handy to build the packages locally before transfering these resources to a final destination. To do this we stage the isolation process with the stage-locally option.

This option will allow you to set the dists-location, scripts-location and pth-file-location as final destinations, but place the results in their default build location. The default build location, if you recall, is in the buildout’s parts directory.

Let’s have a look at how this works by creating similar buildout to those about execept now we are setting the stage-locally option to true:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts =
...     demo
... find-links = %(server)s
... index = %(server)s/index
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = bigdemo
... dists-location = %(site_pkgs)s
... scripts-location = %(bin_dir)s
... pth-location = %(site_pkgs)s
... executable = %(python)s
... stage-locally = true
... """ % dict(server=link_server,
...            bin_dir=os.path.join(virtenv, 'bin'),
...            python=os.path.join(virtenv, 'bin', 'python'),
...            site_pkgs=site_pkgs))
>>> print system(buildout), #doctest: +ELLIPSIS
Uninstalling demo.
Uninstalling demoneeded.
Installing demo.
demo: Copying demo to the staging directory.
demo: Copying demoneeded to the staging directory.
demo: Copying bigdemo to the staging directory.
demo: Can't find the executable on the filesystem. Perhaps this setup is not destine to be used on this machine. So we are using the given executable value /sample-buildout/virtenv/bin/python as is.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.

To verify that things have been staged, let’s have a closer look at the demo script to verify everything went as planned. For one, we expect the script to be in the parts directory:

>>> parts_dir = os.path.join(sample_buildout, 'parts')
>>> demo_script = os.path.join(parts_dir, 'demo-scripts', 'demo')
>>> os.path.exists(demo_script)
True
>>> cat(demo_script)
#!/sample-buildout/virtenv/bin/python
<BLANKLINE>
import sys
def pth_injector(pth_file):
    path_file = open(pth_file, 'r')
    sys.path[0:0] = [line
        for line in path_file.read().split('\n')
        if line is not None]
<BLANKLINE>
pth_files = ['/sample-buildout/virtenv/lib/python2.6/site-packages/demo.pth']
for pth in pth_files:
    pth_injector(pth)
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
    eggrecipedemo.main()

And also we want to check that the pth locations are correct and that the pth itself is in the staging area with parts:

>>> demo_pth = os.path.join(parts_dir, 'demo-pth', 'demo.pth')
>>> cat(demo_pth)
/sample-buildout/virtenv/lib/python2.6/site-packages/demo-0.4c1-py2.1.egg
/sample-buildout/virtenv/lib/python2.6/site-packages/demoneeded-1.2c1-py2.1.egg
/sample-buildout/virtenv/lib/python2.6/site-packages/bigdemo-0.1-py2.1.egg

Issues and help

If you have issues or need assistance, file a bug report on WebLion’s project site or contact us via IRC or email.