skip to navigation
skip to content

Not Logged In

zope.kgs 1.2.0

Known-Good-Set (KGS) Support

===============================
Zope 3 Controlled Package Index
===============================

This package has been developed to support the maintenance of a stable set of
Zope project distributions. It manages the controlled packages configuration
file and supports the generation of buildout configuration files that can be
used by developers.

Another use of this package is to use it for testing new distributions against
the index. Here is the workflow for testing a new package against stable set:

1. Install the correct version of this package.

   (a) Download the version of this package that manages the stable set that
       you are interested in. Currently only the trunk exists, which manages
       the Zope 3.4 release::

         $ svn co svn://svn.zope.org/repos/main/zope.release/trunk zope3.4
         $ cd zope3.4

   (b) Bootstrap the checkout::

         $ python ./bootstrap.py

   (c) Run buildout to create the scripts::

         $ ./bin/buildout

   (d) Run the ``buildout.cfg`` generation script to build a configuration
       file that can be used for testing:

         $ ./bin/generate-buildout

2. From the generated configuration file, you can now build a testing
   environment.

   (a) Enter the test directory and create a buildout:

         $ cd test
         $ python ../bootstrap.py
         $ ./bin/buildout

   (b) Run all the tests to verify that all tests are initially passing:

         $ ./bin/test -vpc1

3. Modify the ``buildout.cfg`` to look for your the new distribution to be
   tested:

   (a) Comment out the "index" option. This needs to be done, so that the new
       package is going to be picked up.

   (b) Change the version number of the package of interest in the "versions"
       section.

   Alternative:

   (a) Check out the new distribution from SVN.

   (b) Add a "develop path/to/my/package" line in the "buildout" section of
       ``buildout.cfg``.

4. Run the tests, making sure that they all pass.

5. Modify ``controlled-packages.cfg`` by adding the new version of the package
   to the package's version list.

6. Generate all files again and upload them:

     $ cd ..
     $ ./bin/generate-buildout
     $ ./bin/generate-versions
     $ ./bin/upload

   Once the files are uploaded, a crontab-job, running every minute, will
   detect the changes in ``controlled-pages.cfg`` and will generate the new
   controlled package pages.

Note: I think the process is still a tiny bit too long. I probably write a
script that makes testing a new version of a package easier, but let's see
whether this process is workable first.


===============
Known Good Sets
===============

This package provides a set of scripts and tools to manage Good-Known-Sets, or
short KGSs. A KGS is a set of package distributions that are known to work
well together. You can verify this, for example, by running all the tests of
all the packages at once.

Let me show you how a typical controlled packages configuration file looks
like:

  >>> import tempfile
  >>> cfgFile = tempfile.mktemp('-cp.cfg')
  >>> open(cfgFile, 'w').write('''\
  ... [DEFAULT]
  ... tested = true
  ...
  ... [KGS]
  ... name = zope-dev
  ... version = 1.2.0
  ... date = 2009-01-01
  ... changelog = CHANGES.txt
  ... announcement = ANNOUNCEMENT.txt
  ... files =
  ...     zope-dev-1.2.0.tgz
  ...     zope-dev-1.2.0.zip
  ...     zope-dev-1.2.0.exe
  ...
  ... [packageA]
  ... versions = 1.0.0
  ...            1.0.1
  ...
  ... [packageB]
  ... versions = 1.2.3
  ... test-extras = test
  ...
  ... [packageC]
  ... # Do not test this package.
  ... tested = false
  ... versions = 4.3.1
  ... ''')

As you can see, this file uses an INI-style format. The "DEFAULT" section is
special, as it will insert the specified options into all other sections as
default. The "KGS" section specifies some global information about the KGS,
such as the name of the KGS. Since this section references several external
files, we should quickly create those.

  >>> import os
  >>> dir = os.path.dirname(cfgFile)

  >>> open(os.path.join(dir, 'CHANGES.txt'), 'w').write('''\
  ... =======
  ... Changes
  ... =======
  ...
  ... packageA
  ... ========
  ...
  ... Version 1.0.0
  ... -------------
  ...
  ... * Initial Release
  ... ''')

  >>> open(os.path.join(dir, 'ANNOUNCEMENT.txt'), 'w').write('''\
  ... =======================
  ... zope-dev 1.2.0 Released
  ... =======================
  ...
  ... The announcement text!
  ... ''')

  >>> open(os.path.join(dir, 'zope-dev-1.2.0.tgz'), 'w').write('tgz')
  >>> open(os.path.join(dir, 'zope-dev-1.2.0.exe'), 'w').write('exe')

All other sections refer to package names. Currently each package section
supports two options. The "versions" option lists all versions that are known
to work in the KGS. Those versions should *always* only be bug fixes to the
first listed version. The second option, "tested", specifies whether the
package should be part of the KGS test suite. By default, we want all packages
to be tested, but some packages require very specific test setups that cannot
be easily reproduced _[1], so we turn off those tests.

You can also stack controlled package configurations on top of each
other. Base configurations can be specified using the `extends` option:

  >>> import tempfile
  >>> cfgFile2 = tempfile.mktemp('-cp.cfg')
  >>> open(cfgFile2, 'w').write('''\
  ... [DEFAULT]
  ... tested = true
  ...
  ... [KGS]
  ... name = grok-dev
  ... version = 0.1.0
  ... extends = %s
  ...
  ... [packageA]
  ... versions = 1.0.2
  ...
  ... [packageD]
  ... versions = 2.2.3
  ...            2.2.4
  ... ''' %cfgFile)

As you can see, you can completely override another package's version
specification as well.

Generating the configuration file and managing it is actually the hard
part. Let's now see what we can do with it.

.. [1]: This is usually due to bugs in setuptools or buildout, such as PYC
files not containing the correct reference to their PY file.


Generate Versions
-----------------

One of the easiest scripts, is the version generation. This script will
generate a "versions" section that is compatible with buildout.

  >>> versionsFile = tempfile.mktemp('-versions.cfg')

  >>> from zope.kgs import version
  >>> version.main((cfgFile, versionsFile))

  >>> print open(versionsFile, 'r').read()
  [versions]
  packageA = 1.0.1
  packageB = 1.2.3
  packageC = 4.3.1

Let's now ensure that the versions also work for the extended configuration:

  >>> versionsFile2 = tempfile.mktemp('-versions.cfg')

  >>> version.main((cfgFile2, versionsFile2))

  >>> print open(versionsFile2, 'r').read()
  [versions]
  packageA = 1.0.2
  packageB = 1.2.3
  packageC = 4.3.1
  packageD = 2.2.4


Generate Buildout
-----------------

In order to be able to test the KGS, you can also generate a full buildout
file that will create and install a testrunner over all packages for you:

  >>> buildoutFile = tempfile.mktemp('-buildout.cfg')

  >>> from zope.kgs import buildout
  >>> buildout.main((cfgFile, buildoutFile))

  >>> print open(buildoutFile, 'r').read()
  [buildout]
  parts = test
  versions = versions
  
  [test]
  recipe = zc.recipe.testrunner
  eggs = packageA
      packageB [test]
  
  [versions]
  packageA = 1.0.1
  packageB = 1.2.3
  packageC = 4.3.1
  

Let's make sure that the buildout generation also honors the extensions:

  >>> buildoutFile2 = tempfile.mktemp('-buildout.cfg')

  >>> buildout.main((cfgFile2, buildoutFile2))

  >>> print open(buildoutFile2, 'r').read()
  [buildout]
  parts = test
  versions = versions
  
  [test]
  recipe = zc.recipe.testrunner
  eggs = packageA
      packageB [test]
      packageD
  
  [versions]
  packageA = 1.0.2
  packageB = 1.2.3
  packageC = 4.3.1
  packageD = 2.2.4
  


Flat Links Pages
----------------

We can also create a flat links page that can be used in the
`dependency_links` argument in your `setup.py` file. Since this module
accesses the original PyPI to ask for the download locations and filenames, we
have to create a controlled packages configuration file that contains real
packages with real version numbers:

  >>> cfgFileReal = tempfile.mktemp('-cp.cfg')
  >>> open(cfgFileReal, 'w').write('''\
  ... [DEFAULT]
  ... tested = true
  ...
  ... [KGS]
  ... name = zope-dev
  ... version = 3.4.0b2
  ...
  ... [PIL]
  ... versions = 1.1.6
  ...
  ... [zope.component]
  ... versions = 3.4.0
  ...
  ... [zope.interface]
  ... versions = 3.4.0
  ...            3.4.1
  ...
  ... [z3c.formdemo]
  ... versions = 1.1.0
  ... ''')

Let's now create the links page:

  >>> linksFile = tempfile.mktemp('-links.html')

  >>> from zope.kgs import link
  >>> link.main((cfgFileReal, linksFile))

  >>> print open(linksFile, 'r').read()
  
  
  Links for the "zope-dev" KGS (version 3.4.0b2)
  
  
  

Links for the "zope-dev" KGS (version 3.4.0b2)

z3c.formdemo-1.1.0-py2.4.egg
z3c.formdemo-1.1.0.tar.gz
zope.component-3.4.0-py2.4.egg
zope.component-3.4.0.tar.gz
zope.interface-3.4.0.tar.gz
zope.interface-3.4.0-py2.4-win32.egg
zope.interface-3.4.1.tar.gz
PPIX Support ------------ You can also use the KGS to limit the available packages in a package index generated ``zc.mirrorcheeseshopslashsimple``. This script also uses PyPI to look up distribution file, so wave to use the real configuration file again. Let's create the pages: >>> indexDir = tempfile.mkdtemp('-ppix') >>> from zope.kgs import ppix >>> ppix.main((cfgFileReal, indexDir)) The index contains one directory per package. So let's have a look: >>> import os >>> sorted(os.listdir(indexDir)) ['PIL', 'z3c.formdemo', 'zope.component', 'zope.interface'] Each directory contains a single "index.html" file with the download links: >>> pkgDir = os.path.join(indexDir, 'zope.component') >>> sorted(os.listdir(pkgDir)) ['index.html'] >>> pkgIndex = os.path.join(pkgDir, 'index.html') >>> print open(pkgIndex, 'r').read() Links for "zope.component"

Links for "zope.component"

zope.component-3.4.0-py2.4.egg
zope.component-3.4.0.tar.gz
PIL is an interesting case, because it does not upload its distribution files yet, at least not for version 1.1.6: >>> pkgIndex = os.path.join(indexDir, 'PIL', 'index.html') >>> print open(pkgIndex, 'r').read() Links for PIL

Links for PIL

1.1.5 home_page
1.1.5 download_url
1.1.5a2 home_page
1.1.5a2 download_url
1.1.5a1 home_page
1.1.5a1 download_url
1.1.4 home_page
1.1.3 home_page
1.1.3 download_url
1.1.6 home_page
1.1.6 download_url
Optionally, you can also specify the `-i` option to generate an overview: >>> ppix.main(('-i', cfgFileReal, indexDir)) >>> sorted(os.listdir(indexDir)) ['PIL', 'index.html', 'z3c.formdemo', 'zope.component', 'zope.interface'] Let's now look at the file: >>> indexPage = os.path.join(indexDir, 'index.html') >>> print open(indexPage, 'r').read() Simple Index for the "zope-dev" KGS (version 3.4.0b2)

Simple Index for the "zope-dev" KGS (version 3.4.0b2)

PIL
z3c.formdemo
zope.component
zope.interface
Allowing exisitng package pages to be overwritten and making the main index page an optional feature makes it possible to use this script for two use cases: (1) Merge the constraints into a PPIX index created by ``zc.mirrorcheeseshopslashsimple``, and (2) create a standalone index which only provides the packages of the KGS. Getting the Latest Versions --------------------------- When updating the KGS, it is often useful to know for which packages have new releases. >>> from zope.kgs import latest >>> latest.main((cfgFileReal,)) z3c.formdemo: 1.1.1, 1.1.2, 1.2.0, 1.3.0, 1.3.0b1, 1.4.0, ... However, it is often desired only to show new minor versions; in this case, we can pass an option to exclude all versions that have a different major version: >>> latest.main(('-m', cfgFileReal)) z3c.formdemo: 1.1.1, 1.1.2 Sometimes you're only interested in changes that apply to a single package, and you won't want to wait for the script to query all of the others >>> latest.main(('-m', cfgFileReal, 'zope.app.server')) >>> latest.main(('-m', cfgFileReal, 'z3c.formdemo')) z3c.formdemo: 1.1.1, 1.1.2 Extracting Change Information ----------------------------- When releasing a version of the KGS, it is desirable to produce a list of changes since the last release. Changes are commonly compared to an older version. >>> cfgFileRealOrig = tempfile.mktemp('-cp.cfg') >>> open(cfgFileRealOrig, 'w').write('''\ ... [DEFAULT] ... tested = true ... ... [KGS] ... name = zope-dev ... version = 3.4.0b1 ... ... [PIL] ... versions = 1.1.6 ... ... [zope.component] ... versions = 3.4.0 ... ... [zope.interface] ... versions = 3.4.0 ... ''') Let's now produce the changes: >>> from zope.kgs import change >>> change.main((cfgFileReal, cfgFileRealOrig)) Processing ('PIL', '1.1.6') Processing ('z3c.formdemo', '1.1.0') Processing ('zope.component', '3.4.0') Processing ('zope.interface', '3.4.1') === PIL === No changes or information not found. ============ z3c.formdemo ============ 1.1.0 (unknown) --------------- - Feature: New "SQL Message" demo shows how ``z3c.form`` can be used with non-object data. Specificically, this small application demonstrates using a Gadfly database using pure SQL calls without any ORM. - Feature: New "Address Book" demo that demonstrates more complex use cases, such as subforms, composite widgets, and mappings/lists ============== zope.component ============== 3.4.0 (2007-09-29) ------------------ No further changes since 3.4.0a1. ============== zope.interface ============== 3.4.1 (unknown) --------------- Fixed a setup bug that prevented installation from source on systems without setuptools. 3.4.0 (unknown) --------------- Final release for 3.4.0. You can also create the changes without an original file, in which case only the versions listed in the current KGS are considered. >>> change.main((cfgFileReal,)) Processing ('PIL', '1.1.6') Processing ('z3c.formdemo', '1.1.0') Processing ('zope.component', '3.4.0') Processing ('zope.interface', '3.4.1') === PIL === No changes or information not found. ============ z3c.formdemo ============ 1.1.0 (unknown) --------------- - Feature: New "SQL Message" demo shows how ``z3c.form`` can be used with non-object data. Specificically, this small application demonstrates using a Gadfly database using pure SQL calls without any ORM. - Feature: New "Address Book" demo that demonstrates more complex use cases, such as subforms, composite widgets, and mappings/lists ============== zope.component ============== 3.4.0 (2007-09-29) ------------------ No further changes since 3.4.0a1. ============== zope.interface ============== 3.4.1 (unknown) --------------- Fixed a setup bug that prevented installation from source on systems without setuptools. 3.4.0 (unknown) --------------- Final release for 3.4.0. The Site Generator ------------------ The easiest way to publish the KGS is via a directory published by a Web server. Whenever a new `controlled-packages.cfg` file is uploaded, a script is run that generates all the files. I usually set up a crontab job to do this. The site generator script acts upon a directory, in which it assumes a `controlled-packages.cfg` file was placed: >>> siteDir = tempfile.mkdtemp() >>> cfgFileSite = os.path.join(siteDir, 'controlled-packages.cfg') >>> import shutil >>> shutil.copy(cfgFileReal, cfgFileSite) >>> from zope.kgs import site >>> site.main(['-s', siteDir]) Let's have a look at the generated files: >>> from pprint import pprint >>> pprint(sorted(os.listdir(siteDir))) ['3.4.0b2', 'index.html', 'intro.html', 'resources'] >>> sorted(os.listdir(os.path.join(siteDir, '3.4.0b2'))) ['ANNOUNCEMENT.html', 'CHANGES.html', 'buildout.cfg', 'controlled-packages.cfg', 'index', 'index.html', 'links.html', 'minimal', 'versions.cfg'] >>> sorted(os.listdir(os.path.join(siteDir, '3.4.0b2', 'minimal'))) ['PIL', 'index.html', 'z3c.formdemo', 'zope.component', 'zope.interface'] If you try to generate the site again without adding the controlled packages config file to the site directory again, it will simply return: >>> site.main(['-s', siteDir]) Basic Parser API ---------------- The ``kgs.py`` module provides a simple class that parses the KGS configuration file and provides all data in an object-oriented manner. >>> from zope.kgs import kgs The class is simply instnatiated using the path to the config file: >>> myKGS = kgs.KGS(cfgFile) >>> myKGS The name, version and date of the KGS is available via: >>> myKGS.name 'zope-dev' >>> myKGS.version '1.2.0' >>> myKGS.date datetime.date(2009, 1, 1) When the changelog and/or announcement files are available, the KGS references the absolute path: >>> myKGS.changelog '.../CHANGES.txt' >>> myKGS.announcement '.../ANNOUNCEMENT.txt' The same is true for other release-related files: >>> myKGS.files ('.../zope-dev-1.2.0.tgz', '.../zope-dev-1.2.0.exe') The packages are available under `packages`: >>> myKGS.packages [, , ] Each package is also an object: >>> pkgA = myKGS.packages[0] >>> pkgA >>> pkgA.name 'packageA' >>> pkgA.versions ['1.0.0', '1.0.1'] >>> pkgA.tested True As we have seen in the scripts above, the KGS class also supports the `entends` option. Thus, let's load the KGS for the config file 2: >>> myKGS2 = kgs.KGS(cfgFile2) >>> myKGS2 >>> myKGS2.name 'grok-dev' >>> myKGS2.packages [, , , ] ======= CHANGES ======= 1.2.0 (2009-07-24) ------------------ - Add ability to specify the extra-includes for running tests. 1.1.0 (2009-02-01) ------------------ - Added '--no-links', '--no-index', and '--no-minimal-index' options to the site generation sctipt to make it faster. - Generating list of latest versions does not fail anymore, if downloading and parsing of the index page fails. - Update links to PyPI web sites. 1.0.1 (2009-01-29) ------------------ - Fix documentation in all scripts, fixing missing imports and incorrect wording. - The package should depend on `python-dateutl` and not `datetutil`, since the latter is not available in PyPI anymore. 1.0.0 (2009-01-29) ------------------ - Initial version as ``zope.kgs``. * A script that extracts the relevant part of the changelog of each package in the KGS. * A script that lists all versions of a package released after the latest version listed in the KGS. * A script that manages the generation of the entire KGS site. + Generates generic and version-specific pages. + Page generation is template-based for easy customization. * Generate `links.html` file which lists all controlled packages files. * Features copied from ``zope.release``: + Parser for KGS configuration files. + Generate `versions.cfg` and `buildout.cfg` script. * Features copied from ``zc.mirrorcheeseshopslashsimple``: + Generate new index pages for the controlled packages.
 
File Type Py Version Uploaded on Size
zope.kgs-1.2.0.tar.gz (md5) Source 2009-07-24 62KB
  • Downloads (All Versions):
  • 8 downloads in the last day
  • 65 downloads in the last week
  • 350 downloads in the last month