skip to navigation
skip to content

Not Logged In

plone.recipe.varnish 1.3

Buildout recipe to install varnish

Varnish recipe for buildout

plone.recipe.varnish is a zc.buildout recipe to install Varnish. Even though the name contains the name Plone, there is nothing Plone specific about this recipe: it works for non-Zope sites just as well.

Configuring it is very simple. For example:

[varnish-build]
recipe = zc.recipe.cmmi
url = ${varnish:download-url}

[varnish]
recipe = plone.recipe.varnish
daemon = ${buildout:parts-directory}/varnish-build/sbin/varnishd
bind = 127.0.0.1:8000
backends = 127.0.0.1:8080
cache-size = 256M

This configures two buildout parts: varnish-build which will download, compile and install varnish, and varnish which runs Varnish, configured to listen on 127.0.0.1:8000 for requests, using a 256 megabyte cache and sending requests to a backend at 127.0.0.1:8080.

Wrappers for all the varnish commands are created in the bin directory of your buildout.

Please note that the configuration generated by this recipe requires Varnish 2.0.3 or later. The download-url provided by the recipe points to the latest tested Varnish release compatible with the generated configuration.

Virtual hosting

Varnish supports virtual hosting by selecting a different backend server based on headers on the incoming request. You can configure the backends through the backends option:

[varnish-instance]
backends =
   plone.org:127.0.0.1:8000
   plone.net:127.0.0.1:9000

This will generate a configuration which sends all traffic for the plone.org host to a backend server running on port 8000 while all traffic for the plone.net host is send to port 9000.

Zope 2 hosting (with Virtual Host Monster)

If you are using Zope 2 as backend server you will need to rewrite the URL so the Zope Virtual Host Monster (VHM) can generate correct links for links in your pages. This can be done either by a web server such as Apache or nginx (placed either in front or behind Varnish) but can also be done by Varnish itself.

The three options are described below.

Option 1 (rewrites after Varnish)

If generating these VHM-style URLs in a proxy behind Varnish (or if using VHM's 'mapping' feature), no extra Varnish configuration is needed. Just make sure the "backends" option directs the traffic to the proxy.

Option 2 (rewrites before Varnish)

If generating these VHM-style URLs in a proxy in-front of Varnish, no extra Varnish configuration is needed as long as the original hostname is still retained in the URL. If the hostname is not retained, you can tell Varnish to direct requests based on the "path" instead of the hostname. For example:

[varnish-instance]
backends =
  /VirtualHostBase/http/plone.org:80/Plone:127.0.0.1:8000
  /VirtualHostBase/http/plone.net:80/Plone:127.0.0.1:9000

This will generate a configuration which sends all traffic for any request whose path starts with "/VirtualHostBase/http/plone.org:80/Plone" to a backend server running at 127.0.0.1 on port 8000, while request paths starting with "/VirtualHostBase/http/plone.net:80/Plone" are sent to port 9000.

Option 3 (rewrites within Varnish)

To have Varnish generate these VHM-style URLs, you can use the zope2_vhm_map option. Here is an example:

[varnish-instance]
zope2_vhm_map =
    plone.org:/plone
    plone.net:/plone

This tells us that the domain plone.org should be mapped to the location /plone in the backend. By combining this with the information from the backends option a varnish configuration will be generated that maps URLs correctly.

Load Balancing

Varnish supports load balancing by configuring a director for a pool of backends. This director sends the incoming requests that cannot be fulfulled by varnish to backends in the pool in either random or round robin fashion. You can configure the director via the balancer option:

[varnish-instance]
balancer = random

This will generate a configuration which sends all traffic to the director, which will choose a 'random' backend server to fulfill the request if the content requested is not cached by varnish itself.

plone.recipe.varnish reference

The plone.recipe.varnish recipe create a Varnish configuration file and creates a wrapper script inside your buildout that will start Varnish with the correct configuration.

Please note that the configuration generated by this recipe requires Varnish 2.0.3 or later.

Please note that in order to use this recipe with varnish >= 3.0, varnish_version option must be set to 3

It can be configured with any of these options:

varnish_version
Varnish target version. Default is 2.
daemon
The path of the varnish daemon to use. Set this to the sbin/varnishd executable created by the zc.recipe.cmmi build.
mode
Specify whether the varnish daemon should run in 'daemon' or 'foreground' mode. The latter is useful when varnish is run by service supervision tools like daemontools or runit. Defaults to 'daemon'.

cache-type

Specify the type of cache storage to use with Varnish. Possible values: file (storage for each object is allocated from an arena backed by a file), malloc (storage for each object is allocated with malloc; in memory), or persistent (experimental as at Varnish 2.1.4). Defaults to 'file'.

cache-location

Customise the location for the Varnish file storage. Option only applicable when used with file or persistent cache-type options. Defaults to using a file named storage inside the relevant parts directory (eg parts/varnish/storage). Changing the default location can be useful in putting the storage somewhere with quicker read speeds (RAM disk etc).
cache-size
The size of the cache (limited to 2G on 32bit systems). Defaults to 256M.
bind
Hostname and port on which Varnish will listen for requests. Defaults to 127.0.0.1:8000.
config
Path for a Varnish VCL configuration to use. If you use this option you can not use the backends, zope2_vhm_map or verbose-headers options .
backends

Specifies the backend or backends which will process the (uncached) requests. The syntax for backends:

[<hostname>][/<path>]:<ip address>:<port>

The optional 'hostname' and 'path' allows you to do virtual hosting. If multiple backends are specified then each backend must include either a hostname or path (or both) so that Varnish can direct the matching request to the appropriate backend. Defaults to 127.0.0.1:8080.

purge-hosts
Specifies hostnames or ip addresses for purge acl. By default localhost and the backends are allowed to purge. Additional allowed hosts are listed here.
name

If specified this sets the name of the varnish instance (defaults to the host name).

From varnishd's manpage:

Amongst other things, this name is used to construct the name of the directory in which varnishd keeps temporary files and persistent state. If the specified name begins with a forward slash, it is interpreted as the absolute path to the directory which should be used for this purpose.
zope2_vhm_map
Defines a virtual host mapping for Zope servers. This is a list of hostname:ZODB location entries which specify the location inside Zope where the website for a virtual host lives.
verbose-headers
Varnish VCL configuration: a http-response header line X-Varnish-Action is set for debugging purposes. It shows a hit, why it bypass/fetch from backend and if if the object was inserted into cache. Possible values: on or off (default).
telnet
If specified sets the hostname and port on which Varnish will listen for commands using its telnet interface.
user
The name of the user varnish should switch to before accepting any requests. Defaults to nobody.
group
The name of the group varnish should switch to before accepting any request. This defaults to the main group for the specified user.
balancer
If included and set to either 'random' or 'round-robin', this option configures varnish to load balance the servers specified by the 'backends' directive. Possible values: none (default), round-robin or random.
connect-timeout
If specified, this option configures the connection timeout (in seconds) for Varnish connecting to a backend server. Varnish will only try to connect to a given backend for this many seconds before giving up. Defaults to 0.4s, as per Varnish's default settings.
first-byte-timeout
If specified, this option configures the timeout (in seconds) for Varnish receiving the first byte from a backend. Varnish will only wait for this many seconds before giving up. A value of 0s means Varnish will never time out. Defaults to 300s, see http://vincentfretin.ecreall.com/articles/varnish-guru-meditation-on-timeout
between-bytes-timeout
If specified, this option configures the timeout (in seconds) for Varnish waiting between bytes when receiving data from a backend. Varnish will only wait this many seconds between bytes before giving up. A value of 0s means this will never time out. Defaults to 60s, as per Varnish's default settings.
runtime-parameters
Runtime parameter configuration options. The full list of available options can be found in the manpage varnishd(1) for your version of varnish. Examples include 'thread_pool_max', 'thread_pool_min', 'sess_timeout'.
vcl_recv, vcl_hit, vcl_miss, vcl_fetch, vcl_deliver, vcl_pipe
Insert arbitrary vcl into the generated config.

Changelog

1.3 (2013-08-21)

  • Add varnish_version option in order to control vcl generation for varnish version >= 3 [rnix]

1.2.2 (2012-10-14)

1.2.1 (2011-05-13)

  • Update known good Varnish to 2.1.5. [elro]
  • Add vcl_recv, vcl_hit, vcl_miss, vcl_fetch, vcl_deliver, vcl_pipe options to insert arbitrary vcl. [elro]

1.2 (2011-01-11)

  • Added new options cache-type, cache-location for specifying type of Varnish storage (such as using malloc for alternative storage) and setting a custom location for said storage [davidjb]
  • Added additional unit tests to check Varnish initialisation script [davidjb]
  • Added new option 'purge-hosts'. Enables additional addresses allowed to purge. [jensens]
  • Added the name option to be able to define the directory varnishd puts temporary files to and identify the instance when using varnishlog or varnishstat. [fRiSi]
  • fixed configuration for verbose-headers=on (context in vlc_fetch is bresp instead of obj in newer varnish versions) [fRiSi]

1.1 (2010-08-05)

  • Changed the default cache size to 256M from 1G. [hannosch]
  • Updated Varnish to 2.1.3. [hannosch]

1.1b1 (2010-04-25)

  • Updated advertised Varnish version to 2.1 and adjusted config. [hannosch]
  • Correct documentation for the daemon setting and remove the default. [hannosch]
  • Removed the deprecated build recipe. [hannosch]
  • Added basic test infrastructure and a test for the simple buildout. [hannosch]
  • Use the built-in set type instead of the deprecated sets module. This recipe now requires at least Python 2.4. [hannosch]
  • Added the ability to configure runtime parameters in the varnish runner configuration and added information to the documentation for it. [benliles]
  • Improve readability of the generated config. [ldr]

1.0.2 (2010-01-18)

  • Update proposed Varnish to 2.0.6. [hannosch]
  • Further documentation cleanup. [hannosch, vincentfretin]

1.0.1 (2009-11-27)

  • Expose the download-url of a known-good Varnish release that works with the configuration produced by the instance recipe. [hannosch]
  • Consistently use tabs in the generated vcl file. [hannosch]
  • Whitespace and documentation cleanup. [hannosch]

1.0 (2009-08-27)

  • Made the vcl template build its acl purge section. At present, the vcl will only allow purges coming from the local host. If we have multiple hosts that are separate from localhost, any PURGE requests will be denied without this. See http://varnish.projects.linpro.no/wiki/VCLExamplePurging [rockdj]
  • Added ability to set various Varnish timeouts (connect_timeout, first_byte_timeout, and between_bytes_timeout) from each option in the buildout. Default values are set at Varnish defaults of 0.4s for connect_timeout, and 60s for between_bytes_timeout. Time for first_byte_timeout is set at 300s as per plone.recipe.varnish 1.0rc9. [rockdj]
  • Set req.http.host for incoming virtual hosted URLs. Without setting this, purge requests sent from hosts other than localhost (the only host in the acl purge list) will result in a 404 message. See http://davidjb.com/blog/2009/01/plone-varnish-configuration-cache-hits-purge-fails [rockdj]

1.0rc11 (2009-06-27)

  • Reintroduced grace options. What the varnish documentation say about grace: "varnish serves stale (but cacheable) objects while retrieving object from backend". The problem is "default_ttl" value is 120s (see bin/varnishd/mgt_param.c in varnish 2.0.4). Added a special rule for createObject url to not look up in the cache. [vincentfretin]

1.0rc10 (2009-06-26)

  • 1.0rc9 generated broken configuration with balancer=none [vincentfretin]

1.0rc9 (2009-06-25)

1.0rc8 (2008-02-12)

  • Remove the custom vcl_hash from the template. Adding the Accept-Encoding header to the cache break effectively breaks purging since nobody will ever include those headers in a PURGE request. To make this safe we just remove the Accept-Encoding header from all incoming requests as well. [wichert]

1.0rc7 (2008-11-26)

  • Be more explicit about deprecating the :build entry point. [wichert]
  • Make the :instance specifier optional: after :build has been removed we can deprecate :instance as well. [wichert]

1.0rc6 (2008-09-22)

  • Deprecate plone.recipe.varnish:build in favour of zc.recipe.cmmi: it does not make sense to duplicate its logic here. [wichert]
  • Add feature to enable verbose headers in varnish.vcl. This is primary interesting for debugging of cache-settings. See README.txt. [jensens]
  • Deal better with sources which do not have executable-bits set or are svn exports. [wichert]
  • The 1.0rc5 release was broken and has been retracted. Currently the trunk is only usable with the Varnish 2.0-beta1 and later. [hannosch]

1.0rc5 (2008-04-27)

  • Pipe is evil: it pipes the whole connection to the backend which means varnish will no longer process any further requests if HTTP pipelining is used. Switch to using pass instead. [wichert]
  • Add a default_ttl of zero seconds to the Varnish runner to avoid a Varnish bug with the handling of an Expires header with a date in the past. [newbery]
  • Merged branches/newbery-hostnamepath. [newbery]
  • We don't need to include Accept-Encoding in the hash. Varnish takes care of Vary negotiation already. [newbery]

1.0rc4 (2008-03-18)

  • Fixed typos / whitespace. [hannosch]
  • Varnish 1.1.2 is out. [wichert]
  • Merged witsch-foreground-support back to trunk. [witsch]
  • Use a pidfile. [wichert]

1.0rc3 (2007-09-02)

  • Fixed a bug where options["location"] was being used before it was being set. [rocky]
  • Made the module name determination a little more robust during createVarnishConfig so that recipes that specify version deps still work. [rocky]
  • Do not use defaults for user and group. [wichert]
  • We do need the parts: we use it for the file storage. [wichert]

1.0rc2 (2007-08-29)

  • Add an option to use an existing configuration file. [wichert]
  • Remove hardcoded caching for images, binaries, CSS and javascript. This should be done by the backend server or a custom varnish configuration. [wichert]
  • Add Accept-Encoding to the cache key so we can handle compressed content. [wichert]
  • Test if a bin-directory exists. This allows us to compile varnish 1.0 which does not have an sbin directory. [wichert]

1.0rc1 (2007-08-27)

  • Document the OSX bugfix we apply when building varnish. [wichert]
  • Add a dummy update method to prevent needless recompiles. [wichert]
  • Update for Varnish 1.1.1. [wichert]

1.0b2 (2007-08-25)

  • When building from svn, we need to run autogen.sh. [optilude]
  • Refactor the recipe: there are now separate recipes to build and configure Varnish. This makes it possible to reconfigure varnish without having to recompile with as well as using an already installed varnish. [wichert]
  • Move the OSX patching code into a separate method. [wichert]
  • Use pass for non-GET/HEAD requests. This makes a bit more sense and fixes a login problem for Plone sites. [wichert]
  • Reorganize a bit for readability. [wichert]
  • Support Python 2.3 as well. [wichert]
  • Make it possible to specify the user and group as well. [wichert]
  • Do not create the source directory - we move the extracted source in its place later. [wichert]
  • If running on OS X, patch libtool as described in http://varnish.projects.linpro.no/ticket/118 and http://thread.gmane.org/gmane.comp.web.varnish.misc/668/focus=669. [optilude]
  • VCL is not C. You need the curlies even on single-line if statements. [optilude]
  • This rewriting style only works on Zope 3 - Zope 3 reinvented that wheel. [wichert]
  • Add support for If-Modified-Since and If-None-Match requests. Thanks to newbery for the suggstions. [wichert]
  • Explicitly mention that there is nothing Plone or Zope specific about this recipe. [wichert]

1.0b1 (2007-08-04)

  • More documentation. [wichert]
  • Ignore the port information in the host header. [wichert]
  • Use the port varnish is bound to in the VHM mapping. [wichert]
  • Define all default values centrally. [wichert]
  • Add support for Zope virtual hosts. [wichert]
  • Add support for virtual hosting. [wichert]
  • Initial import of Varnish recipe. [wichert]
 
File Type Py Version Uploaded on Size
plone.recipe.varnish-1.3.tar.gz (md5) Source 2013-08-21 26KB
  • Downloads (All Versions):
  • 36 downloads in the last day
  • 519 downloads in the last week
  • 2296 downloads in the last month