slapos.cookbook 1.0.59

slapos.cookbook
===============

Cookbook of SlapOS recipes.


testing
=======

Unit tests for recipes can be found under ``slapos/test/recipe``. To run the
tests use provided unittest.defaultTestLoader inside ``slapos/test/test_recipe``
by invoking

python setup.py test --test-suite slapos.test.test_recipe.additional_tests


Changes
=======

1.0.59 (2018-03-15)
-------------------
* librecipe.execute: fix convert process arguments to string formatting.

1.0.58 (2018-03-14)
-------------------

* generic.mysql: unregister UDFs before (re)adding UDFs
* Remove obsolete/unused recipes.
* neoppod: add support for new --dedup storage option.
* Use inotify-simple instead of inotifyx.
* erp5.test: remove duplicated code.
* librecipe: bugfixes found by pylint, performance improvements, and major
refactoring of executable wrappers.
* GenericBaseRecipe.createWrapper: remove 'comments' parameter.
* Drop the 'parameters-extra' option and always forward extra parameters.
* wrapper: new 'private-dev-shm' option (useful for wendelin.core).
* generic.cloudooo: OnlyOffice converter support odf.
* erp5testnode: don't tell git to ignore SSL errors.

1.0.53 (2017-09-13)
-------------------

* check_port_listening: workaround for shebang limitation, reduce to a single file
* erp5.test: pass new --conversion_server_url option to runUnitTest

1.0.52 (2017-07-04)
-------------------

* wrapper: Add option to reserve CPU core
* slapconfiguration: Recipe reads partitions resource file
* neoppod: add support for new --disable-drop-partitions storage option
* random: Fix the monkeypatch in random.py to incorporate the recent changes in buildout 'get' function
* random: Add Integer recipe.
* librecipe.execute: Notify on file moved
* zero_knowledge: allow to set destination folder of configuration file


1.0.50 (2017-04-18)
-------------------

* pbs: Do not parallelize calculus when the heaviest task is IO
* re6st-registry: Refactor integration with re6st registry
* erp5testnode: make shellinabox reusing password file of pwgen

1.0.48 (2017-01-31)
-------------------

* random-recipe: add option create-once to prevent storage file deletion by buildout

1.0.45 (2017-01-09)
-------------------

* recipe: set default timeout of check url promise to 20 seconds

1.0.44 (2016-12-30)
-------------------

* pbs: handles the fact that some parameters are not present when slaves are down
* recipe: allow usage of pidfile in wrapper recipe
* sshd: fix generation of authorized_keys

1.0.43 (2016-11-24)
-------------------

* pbs: fixes trap command for dash intepreter
* pbs: remove infinite loops from pbs scripts.
* random.py: new file containing recipes generating random values.
* testnode: disallow frontend access to all folders, avoiding publishing private repositories

1.0.41 (2016-10-26)
-------------------

* dcron: new parameter to get a random time, with a frequency of once a day
* softwaretype: fix parse error on '+ =' when using buildout 2
* pbs: General Improvement and fixes.

1.0.35 (2016-09-19)
-------------------

* pbs: fix/accelerates deployment of resilient instances
* recipe: new recipe to get a free network port
* Remove url-list parameter to download fonts from fontconfig instance

1.0.31 (2016-05-30)
-------------------

* Implement cross recipe cache for registerComputerPartition
* Fix workaround for long shebang (place script on bin)

1.0.30 (2016-05-23)
-------------------

* Implement a workarround for long shebang
* Implement Validation for user inputs ssl certificates

1.0.25 (2016-04-15)
-------------------

* fixup slap configuration: provide instance and root instance title

1.0.22 (2016-04-01)
-------------------

* slap configuration: provide instance and root instance title

1.0.16 (2015-10.27)
-------------------

* kvm recipe: fix bugs dowload image and disk creation

1.0.14 (2015-10.26)
-------------------

* kvm recipe: Allow to set keyboard layout language used by qemu and VNC
* simplehttpserver-recipe: fix encoding error

For older entries, see https://lab.nexedi.com/nexedi/slapos/blob/a662db75cc840df9d4664a9d048ef28ebfff4d50/CHANGES.rst


generic_cloudooo
================

The generic_cloudooo recipe helps you to deploy cloudooo services with their configuration files.


How to use?
-----------

Here is an example of a section to add in your software.cfg :

.. code-block::

[cloudooo-configuration]
recipe = slapos.cookbook:generic_cloudooo
configuration-file = ${directory:etc}/cloudooo.cfg
wrapper = ${directory:services}/cloudooo
data-directory = ${directory:srv}/cloudooo
ip = 0.0.0.0
port = 1234
ooo-paster = ${directory:bin}/cloudooo_paster
mimetype_entry_addition =
text/html application/pdf wkhtmltopdf
openoffice-port = 1235
ooo-binary-path = ${directory:libreoffice-bin}/program
environment =
FONTCONFIG_FILE = ${fontconfig-instance:conf-path}
PATH = ${binary-link:target-directory}
ooo-uno-path = ${directory:libreoffice-bin}/basis-link/program


Where :

- `configuration-file` is the path where the put the configuration file;
- `wrapper` is the path where the put the final executable file;
- `data-directory` is the folder where cloudooo would put it's temporary files;
- `ip` and `port` is where cloudooo will listen to;
- `ooo-paster` is the path of the program that will load cloudooo configuration
and start the application;
- `mimetype_entry_addition` is additional entries to give to the default
mimetype registry. (see section below.) The mimetype entry list is sorted in
order to make the global mimetype at the bottom of the list.
(i.e. `* * ooo` > `text/* * ooo`)

.. code-block::

mimetype_entry_addition =
<input_format> <output_format> <handler>

- `openoffice-port` is the port where the internal OpenOffice.org service will
listen to;
- `ooo-binary-path` is the path of the openoffice service executable file;
- `environment` are environment vars to use with the openoffice binary;
- `ooo-uno-path` is the path where UNO library is installed.


Default mimetype registry
-------------------------

.. code-block::

application/vnd.oasis.opendocument* * ooo
application/vnd.sun.xml* * ooo
application/pdf text/* pdf
application/pdf * ooo
video/* * ffmpeg
audio/* * ffmpeg
application/x-shockwave-flash * ffmpeg
application/ogg * ffmpeg
application/ogv * ffmpeg
image/png image/jpeg imagemagick
image/png * ooo
image/* image/* imagemagick
text/* * ooo
application/zip * ooo
application/msword * ooo
application/vnd* * ooo
application/x-vnd* * ooo
application/postscript * ooo
application/wmf * ooo
application/csv * ooo
application/x-openoffice-gdimetafile * ooo
application/x-emf * ooo
application/emf * ooo
application/octet* * ooo
* application/vnd.oasis.opendocument* ooo


generic_varnish
===============

This recipe creates a varnish instance dedicated for ERP5 with a web checker[1]
set up.

How to Use
==========

On slap console, you can instanciate varnish like this:

instance = request(
software_type='varnish',
partition_parameter_kw={
'backend-url':'https://[your_backend_address]:your_backend_port',
'web-checker-frontend-url':'http://www.example.com',
'web-checker-mail-address':'web-checker-result@example.com',
'web-checker-smtp-host':'mail.example.com',
}
)

backend-url is the backend url that varnish will cache.

web-checker-frontend-url is the entry-point-url that web checker will check
the HTTP headers of all the pages in the web site.

web-checker-mail-address is the email address where web checker will send
the HTTP Cache cheking result.

web-checker-smtp-host is the smtp server to be used to send the web checker
result.

[Note]
When web-checker-* parameters are not given, web_checker will be disabled.

TODO
====

We need to merge this and apache_frontend recipe.

References
==========

[1] web_checker (it is a part of erp5.util)
http://pypi.python.org/pypi/erp5.util
web_checker: Web site HTTP Cache header checking tool



kvm
===

Introduction
------------

The erp5.recipe.kvm aims to integrate KVM setups and buildout. This recipe is
able to download one remote image and setup a KVM environment to use it.

This recipe is also capable to reuse images or partitions already present on
disk to create the setup.

Examples
--------

The follow examples lists different kind of configurations.


KVM with Remote and gzipped Image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

[kvm-testing-with-remote-gzip-image]
image = http://URL/public.mdv2010.0_x86_64.qcow2.img.gz

# md5 checks are optional
md5_download = adcff8adcff8adcff8adcff8
md5_image = 1a4e371a4e371a4e371a4e371a4e37

gzip = true

# Use -hda instead -drive arg
# Default is drive (see Options below)
image_type = hda

### Common Configuration below. ###

# VNC is optional
kvm_vnc = <SOME-IP>:<VNC-DISPLAY>

# Graphic is optional
kvm_graphic = std


# Define list of redirections.
kvm_redir =
tcp:13480::80

kvm_net =
user,hostname=publicmandriva
nic,model=ne2k_pci

# This automatically create a redirection for 13456 -> 22
ssh_port = 13456
ssh_key_path = /path/to/mykey.key

KVM with Remote and raw Image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

[kvm-testing-with-remote-raw-image]
image = http://URL/public.mdv2010.0_x86_64.qcow2.img
md5_download = 1a4e371a4e371a4e371a4e371a4e37
md5_image = 1a4e371a4e371a4e371a4e371a4e37

gzip = false

### The Rest Same configuration as previous ###

KVM with direct local Image file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This does not copy and/or download the image::

[kvm-testing-with-local-image]

file = /home/you/public.mdv2010.0_x86_64.qcow2.img
md5_image = 1a4e371a4e371a4e371a4e371a4e37

### The Rest Same configuration as previous ###

KVM with a linux partition
~~~~~~~~~~~~~~~~~~~~~~~~~~

This does not copy and/or download the image::

[kvm-testing-a-linux-partition]

file = /dev/sdb

### The Rest Same configuration as previous ###


Options
-------


location

When define, it does not use buildout parts directory to allocate the image.

image

URL to a remote image.

file

Use file makes recipe ignore image option. You can define a path to a image
or partion to use.

image_type

You can define how the KVM will use the image with "-hdx" or "-drive". By
default it uses drive and the result is:

"kvm -drive file=IMAGE,if=virtio,cache=none,boot=on ..."

if you use image_type=hda:

"kvm -hda IMAGE ..."

gzip

If true means the image is compressed (gzipped), and the recipe will
uncompress the image before copy it.

md5_download

When defined, this values is used to verify the file downloaded.

md5_image

When defined, this values is used to verify the image generated, don't use it
when a partition is used in file parameter.

kvm_vnc

Define the ip-address:display used by KVM to start the VNC server. If not
defined, no VNC port is created.

kvm_redir

Define port redirections to the buildout.

kvm_graphic

If defined it adds the "-vga value" at the KVM command.

kvm_net

Define the net definitions, each value defines one "-net" in kvm command.
Example:

kvm_net =
user,hostname=publicmandriva
nic,model=ne2k_pci

It generates:

"kvm -net user,hostname=publicmandriva -net nic,model=ne2k_pci ..."

kvm_snapshot

Use "-snapshot" when run a KVM. This not write the changes direct into the
image. Default value is False.

ssh_port

If defined creates a new redirection for port 22 and creates few script to
connect to the instance.

ssh_hostname

By default it uses localhost. You don't need to define this.

ssh_key_path

Path to the ssh key used to connect without password to the image running.

ssh_user

Define the server that will be used to connect to the instance.

kvm_bin_directory

Place where the scripts will be created. By default it uses bin-directory from
buildout.

kvm_run_directory

Place where the pid file will be created, by default it uses var-directory
from buildout.


Generated Commands
------------------

Few scripts are generated to you manage your KVM instance. The scripts names are
created with the followed standard:

KVM-PARTS-NAME-ctl

Commands usage
~~~~~~~~~~~~~~

KVM-PARTS-NAME-ctl (start|stop|status|restart)

This script is used to manage the KVM instance.


KVM-PARTS-NAME-sendfile REMOTEFILE LOCALFILE

Copy the local file to a remote place.

KVM-PARTS-NAME-getfile REMOTEFILE LOCALFILE

Copy the remote file to a local place.

KVM-PARTS-NAME-runscript COMMAND

Run a command into remote KVM computer.


kvm_frontend
=============

Introduction
------------

The ``slapos.recipe.kvm_frontend`` aims to provide proxy server to KVM instances.

It allows HTTPS IPv4/IPv6 proxying (with or without domain name), and supports
the WebSocket technology needed for VNC-in-webapplication noVNC.

It works following the master/slave instances system. A master instance is
created, containing all what is needed to run the proxy. Slave instances
are later created, adding one line in the master instance's proxy configuration
that specify the IP/port to proxy to the KVM.

The slave instance (kvm) is then accessible from
http://[masterinstanceIPorhostname]/[randomgeneratednumber]


Instance parameters
------------

Incoming master instance parameters
+++++++

``port`` - Port of server, optional, defaults to 4443.
``domain`` - domain name to use, optional, default to
"host.vifib.net".
``redirect_plain_http`` - if value is one of ['y', 'yes', '1', 'true'], instance
will try to create a simple http server on port 80
redirecting to the proxy. Optional.

Incoming slave instance parameters
+++++++

``host`` - KVM instance IP or hostname. Mandatory.
``port`` - KVM instance port, Mandatory.
``https`` - if value is one of ['n', 'no', '0', 'false'], will try to connect
to target in plain http. Optional.

Connection parameters
-------------

Outgoing master connection parameters
+++++++

``domain_ipv6_address`` - Proxy IP
``site_url`` - Proxy URL

Outgoing slave connection parameters are :
+++++++

``site_url`` - URL of instance
``domain_name`` - Domain name of proxy
``port`` - Port of proxy


lamp
=====

The lamp recipe helps you to deploy simply a php based application on slapos. This recipe is
able to setup mariadb, apache and apache-php for your php application, and is also capable of
configuring your software during installation to ensure a full compatibility.


How to use?
-----------

just add this part in your software.cfg to use the lamp.simple module

.. code-block::

[instance-recipe]
egg = slapos.cookbook
module = lamp.simple

you also need to extend lamp.cfg

.. code-block::

extends =
https://lab.nexedi.com/nexedi/slapos/raw/slapos-0.50/stack/lamp.cfg


lamp.runner
===========

When you install some software (such as prestashop) you need to remove or rename folder, with slapos you can not
access to the www-data directory. to do this, you need to tell to lamp recipe to remove or/and it when software
will be instantiated. Some software requires more than rename or delete a folder (manualy create database etc...)
in this case you need to write a python script and lamp recipe must run it when installing your software.



How to use?
-----------

this part of lamp recipe work with slapos.toolbox, Therefore you must add it to your recipe.
in software.cfg, replace instance-recipe-egg part by

.. code-block::

[instance-recipe-egg]
recipe = zc.recipe.egg
eggs =
${python-mysqlclient:egg}
${instance-recipe:egg}
slapos.toolbox[lampconfigure]

and add into your instance.cfg

.. code-block::

lampconfigure_directory = ${buildout:bin-directory}/lampconfigure


CONDITION
----------

the action (delete, rename, script, chmod) only starts when the condition is filled.
in instance.cfg, add::

file_token = path_of_file

and the action will begin when path_of_www-data/path_of_file will be created
you can also use database to check condition. add ::

table_name = name_of_table
constraint = sql_where_condition

name_of_table is the full or partial name(in some cases we can not know the prefix used to create tables) of table
into mariadb databse for example table_name = admin. if you use
name_of_table = **, the action will begin when database is ready.
constraint is the sql_condition to use when search entry into name_of_table for example constraint = `admin_id`=1

you can't use file_token and table_name at the same time, otherwise file_token will be used in priority. Beware of conditions that will never be satisfied.



ACTION
-------

The action starts when condition is true

1- delete file or folder into instance.cfg, use::

delete = file_or_folder1, file_or_folder2, file_or_folder3 ...

for example delete = admin

2- rename file or folder
into instance.cfg, use

rename_chmod = mode (optional)
rename = old_name1 => new_name1, old_name2 => new_name2, ...

you can also use
rename = old_name1, old_name2 => new_name2, ... in this case old_name1 will be rename and the new name will be chose
by joining old_name1 and mysql_user: this should give
rename = old_name1 => old_name1-mysql_user, old_name2 => new_name2, ...
use rename_chmod to apply a mode to new_name1, new_name2, ...

3- Change mode of file or directory

this is usefull if you want to change the mode of files into your www-data, use

chmod = file1, file2, ....
mode = mode_to_apply (ex= 0644)

4- Launch python script

use script = ${configure-script:location}/${configure-script:filename} into instance.cfg, add part configure-script
into software.cfg

.. code-block::

parts = configure-script

[configure-script]
recipe = hexagonit.recipe.download
location = ${buildout:parts-directory}/${:_buildout_section_name_}
url = url_of_script_name.py
filename = script_name.py
download-only = True

The script_name.py should contain a main function, sys.argv is given to the main. you can write script_name.py like this::

...
def setup(args):
mysql_port, mysql_host, mysql_user, mysql_password, mysql_database, base_url, htdocs = args
.......

if __name__ == '__main__':
setup(sys.argv[1:])


- base_url: is the url of php software
- htdocs: is the path of www-data directory
- mysql_user, mysql_password, mysql_database, mysql_host: is the mariadb parameters

you can also use "import MySQLdb" if you want to access to database via your python script

libcloud
========

Slapified recipe to interact with any libcloud supported IaaS system


librecipe
=========

Thanks to using slapos.cookbook:librecipe it is easier to create zc.buildout recipes in SlapOS environment.

How to use?
-----------

In setup.py of recipe add only one install requires to slap.lib.recipe.

In code itself subclass from slap.lib.recipe.BaseSlapRecipe.BaseSlapRecipe.

Use _install hook:

::

from slap.lib.recipe.BaseSlapRecipe import BaseSlapRecipe

class Recipe(BaseSlapRecipe):
...
def _install(self):
# refer below for list of available objects
specific code
of recipe

Available variables self.:

* name and options passed by zc.buildout during init
* work_directory -- buildout's directory
* bin_directory -- places for generated binaries
* running_wrapper_location -- filename of wrapper to create
* data_root_directory -- directory container for data -- inside this
directory it is advised to create named directories for provided servers
which needs data
* backup_directory -- directory container for backups -- inside this
directory it is advised o created named directories for backups, with same
structure as in data_root_directory
* var_directory -- container for various, unix following things:

* log_directory -- container for logs
* run_directory -- container for pidfiles and sockets

* etc_directory -- place to put named files and directories of configuration
for provided servers
* computer_id -- id of computer
* computer_partition_id -- if of computer partition
* server_url - url of Vifib server
* software_release_url -- url of software release being instantiated
* slap -- initialised connection to Vifib server
* computer_partition -- initialised connection to computer partition
* request -- shortcut to computer partition request method

By default all directories are created before calling _install hook.

_install method shall return list of paths which are safe to be removed by
buildout during part uninstallation.

Important assumptions
---------------------

Because in SlapOS environment zc.buildout does not know when data are changed,
recipes shall be always uninstalled/installed. This is done during constructing
recipe instance which subclasses from BaseSlapRecipe.


mkdirectory
===========

mkdirectory loops on its options and create the directory joined

.. Note::

Use a slash ``/`` as directory separator. Don't use system dependent separator.
The slash will be parsed and replace by the operating system right separator.

Only use relative directory to the buildout root directory.

The created directory won't be added to path list.


seleniumrunner
==============

Allows to run selenium tests through browser and xvfb. Posts the results on
Nexedi ERP5.

Parameters
==========

* project : name of the project inside of ERP5 test result instance
* user : username to use in ERP5 instance to test
* password : password to use in ERP5 instance to test
* suite_name : name of test suite to launch
* url : url to portal_test of ERP5 isntance to test
* test_report_instance_url : url of test_result_module to put results

* Example::

<?xml version="1.0" encoding="utf-8"?>
<instance>
<parameter id="project">Vifib</parameter>
<parameter id="user">myuser</parameter>
<parameter id="password">mypassword</parameter>
<parameter id="suite_name">my_zuite</parameter>
<parameter id="url">https://myerp5totest/erp5/portal_tests</parameter>
<parameter id="test_report_instance_url">https://user:passwordwww.myerp5withtestresults.com/test_result_module/</parameter>
</instance>


These recipes provide the ability to save some buildout parameters and their value in a custom file, inside the instance folder.

In both recipes, you HAVE TO give a filename, which will be stored at the root of the instance folder


WriteRecipe :
-------------

* Is used to create a section (named according to the buildout section_name).
* You can give then as much parameters you wish, with their default values.
* Whenever you run buildout, if the parameter has yet been saved in the config file, it will do nothing.
* If the parameter's value has changed in the config file, it won't be overwritten
* /!\ If you decide to change the default value of one parameter, ALL other parameters will be reseted in the config file, even if you changed it manually. Explanation : The default values aren't expected to change, except while development purposes.

ReadRecipe :

* It fills its own section with all the options in all the sections of the config file.