Manage Galaxy servers
Project description
========================================
gravity - Galaxy Server Administration
========================================
A process manager (`supervisor`_) and management tools for `Galaxy`_ servers.
Installing this will give you two executables, ``galaxycfg`` and ``galaxyadm``.
As their names imply, ``galaxycfg`` does things related to configuration and
``galaxyadm`` does things related to administration.
A virtualenv will automatically be created for your Galaxy server. It's a good
thing.
Installation
============
Python 2.7 is required. Sadly, Python 3 won't work, because supervisor doesn't
support it, `although work is in progress <supervisor issue 491_>`_.
To install:
``pip install gravity``
To make your life easier, you are encourged to install into a `virtualenv`_,
and to make that trivial, you are encouraged to use `virtualenv-burrito`_.
TODO
====
- Write tests
- Write documentation
- Sort out whether the AttributeDict stuff is really a good idea or
not, and either access things as items or attributes but not both.
- Check for hash collisions when generating instance names
- Move the TODO and ideas to issues in Github.
Ideas
=====
- Enable arbitrary environment configuration for supervisor processes
- ``galaxycfg set``? e.g.
``galaxycfg set instance_name autostart=true``. But then should all
of ``[galaxy:server]`` become options directly controlled by
``galaxycfg``? The nice thing about ``[galaxy:server]`` is that
gravity is updated from it, which may be more convenient than
command-line driven configuration.
Notes
=====
``[galaxy:server]``
Add this section to your ``galaxy.ini``, ``reports_wsgi.ini``,
``tool_shed_wsgi.ini`` to set:
::
instance_name = string # override the default auto-generated instance name
galaxy_root = /path # if galaxy is not at ../ or ./ from the config file
virtualenv = /path # override default auto-generated virtualenv path (it
# will be created if it does not exist)
log_dir = /path # where to log galaxy server output. for uwsgi you
# probably want to use uwsgi's logto option though
uwsgi_path = /path # explicit path to uwsgi, otherwise it will be found on
# $PATH. Or, the special value `install`, which will
# cause it to be installed into service's virtualenv
Potentially useful information, tricks, etc.:
- Unless you set different state dirs with ``--state-dir`` or
``$GALAXYADM_STATE_DIR``, there will only be one supervisord for all
of your galaxy instances. But they will be separated out by a
generated ``instance_name``. You can override this with
``instance_name`` in ``[galaxy:server]``.
- To put configs (galaxy configs, galaxy + reports, whatever) into the
same instance, set their ``instance_name`` to the same string in each
config's ``[galaxy:server]``. This puts them into a single supervisor
group, which may be what you want, although note that any
start/stop/restart/etc. is performed on the entire group, which may
not be what you want.
- The config manager generally views things in terms of config files.
If you change the virtualenv or ``galaxy_root`` in a config file, it
will not change that value for all services in the instance
(supervisor group), it will only change it for the services started
from that config.
- Anything you drop in to
``$GALAXYADM_STATE_DIR/supervisor/supervisord.conf.d`` will be picked
up by supervisord on a ``galaxyadm supervisorctl update`` or just
``galaxyadm update``
- The ``job_conf.xml`` parsed corresponds to the galaxy config, it'll
check the path in ``job_config_file`` in ``[app:main]`` or default to
``galaxy_root/config/job_conf.xml`` if that file exists. If handlers
in ``job_conf.xml`` have a corresponding ``[server:]`` in
``galaxy.ini``, they will be started using Paste. If there is not a
corresponding ``[server:]`` they will be started as a "standalone"
server with ``galaxy_root/lib/galaxy/main.py``
Commands
========
galaxycfg
---------
gravity needs to know where your Galaxy config file(s) are. ``galaxcfg``
is how you manage them.
Use ``galaxycfg -h`` for details. Subcommands also support ``-h``, e.g.
``galaxycfg add -h``.
galaxycfg subcommands
~~~~~~~~~~~~~~~~~~~~~
``add``
Register a Galaxy, Reports, or Tool Shed server config with the process
manager, create a virtualenv, create supervisor configs, and update.
Does not start.
``list``
List config files registered with the process manager.
``instances``
List known instances and services.
``get /path/to/galaxy.ini``
Show stored configuration details for the named config file.
``rename /path/to/old.ini /path/to/new.ini``
Use this if you move your config.
``remove /path/to/galaxy.ini`` ``remove instance_name``
Deregister a Galaxy et. al. server config., or all configs referencing
the supplied ``instance_name``.
galaxyadm
---------
``galaxyadm`` controls your Galaxy server processes. Use this after
you've registered a config file with ``galaxycfg add``.
Use ``galaxyadm -h`` for details.
galaxyadm subcommands
~~~~~~~~~~~~~~~~~~~~~
| ``start [instance_name]``
| ``stop [instance_name]``
| ``restart [instance_name]``
Roughly what you'd expect. If ``instance_name`` isn't provided, perform
the operation on all known instances.
If you call ``start`` from the root (or from 1 subdirectory deep) of a
Galaxy source tree, ``config/galaxy.ini`` if it exists, or else
``config/galaxy.ini.sample`` will automatically be registered with
``galaxycfg`` add and then ``galaxyadm start`` will start the newly
added server.
``reload [instance_name]``
The same as restart but uWSGI master processes will only receive a
``SIGHUP`` so the workers restart but the master stays up.
``graceful [instance_name]``
The same as reload but Paste servers will be restarted sequentially, and
the next one will not be restarted until the previous one is up and
accepting requests.
``update``
Figure out what has changed in configs, which could be:
- changes to ``[galaxy:server]``
- adding or removing ``[server:]`` sections
- adding or removing a ``[uwsgi]`` section
- adding or removing handlers in ``job_conf.xml``
This will perform the operation for all registered configs, which may
cause unintended service restarts.
Any needed changes to supervisor configs will be performed and then
``supervisorctl update`` will be called. You will need to do a
``galaxy start`` after this to start any newly added instances (or
possibly even old instances, since adding new programs to a group in
supervisor causes the entire group to be stopped).
Update is called automatically for the ``start``, ``stop``, ``restart``,
``reload``, and ``graceful`` subcommands.
``supervisorctl [subcommand]``
Pass through directly to supervisor
``shutdown``
Stop supervisord
.. _supervisor: http://supervisord.org/
.. _Galaxy: http://galaxyproject.org/
.. _supervisor issue 491: https://github.com/Supervisor/supervisor/issues/491
.. _virtualenv: https://virtualenv.pypa.io/
.. _virtualenv-burrito: https://github.com/brainsik/virtualenv-burrito
=========
History
=========
0.8
===
- Add auto-register to ``galaxy start`` if it's called from the root (or
subdirectory) of a Galaxy root directory.
- Make ``galaxycfg remove`` accept instance names as params in addition to
config file paths.
- Use the same hash generated for an instance name as the hash for a generated
virtualenv name, so virtualenvs are more easily identified as belonging to a
config.
- Renamed from ``galaxyadmin`` to ``gravity`` (thanks John Chilton).
0.7
===
- Added the ``galaxyadm`` subcommand ``graceful`` on a suggestion from Nicola
Soranzo.
- Install uWSGI into the config's virtualenv if requested.
- Removed any dependence on Galaxy and eggs.
- Moved project to its own repository from the Galaxy clone I'd been working
from.
Older
=====
- Works in progress as part of the Galaxy code.
gravity - Galaxy Server Administration
========================================
A process manager (`supervisor`_) and management tools for `Galaxy`_ servers.
Installing this will give you two executables, ``galaxycfg`` and ``galaxyadm``.
As their names imply, ``galaxycfg`` does things related to configuration and
``galaxyadm`` does things related to administration.
A virtualenv will automatically be created for your Galaxy server. It's a good
thing.
Installation
============
Python 2.7 is required. Sadly, Python 3 won't work, because supervisor doesn't
support it, `although work is in progress <supervisor issue 491_>`_.
To install:
``pip install gravity``
To make your life easier, you are encourged to install into a `virtualenv`_,
and to make that trivial, you are encouraged to use `virtualenv-burrito`_.
TODO
====
- Write tests
- Write documentation
- Sort out whether the AttributeDict stuff is really a good idea or
not, and either access things as items or attributes but not both.
- Check for hash collisions when generating instance names
- Move the TODO and ideas to issues in Github.
Ideas
=====
- Enable arbitrary environment configuration for supervisor processes
- ``galaxycfg set``? e.g.
``galaxycfg set instance_name autostart=true``. But then should all
of ``[galaxy:server]`` become options directly controlled by
``galaxycfg``? The nice thing about ``[galaxy:server]`` is that
gravity is updated from it, which may be more convenient than
command-line driven configuration.
Notes
=====
``[galaxy:server]``
Add this section to your ``galaxy.ini``, ``reports_wsgi.ini``,
``tool_shed_wsgi.ini`` to set:
::
instance_name = string # override the default auto-generated instance name
galaxy_root = /path # if galaxy is not at ../ or ./ from the config file
virtualenv = /path # override default auto-generated virtualenv path (it
# will be created if it does not exist)
log_dir = /path # where to log galaxy server output. for uwsgi you
# probably want to use uwsgi's logto option though
uwsgi_path = /path # explicit path to uwsgi, otherwise it will be found on
# $PATH. Or, the special value `install`, which will
# cause it to be installed into service's virtualenv
Potentially useful information, tricks, etc.:
- Unless you set different state dirs with ``--state-dir`` or
``$GALAXYADM_STATE_DIR``, there will only be one supervisord for all
of your galaxy instances. But they will be separated out by a
generated ``instance_name``. You can override this with
``instance_name`` in ``[galaxy:server]``.
- To put configs (galaxy configs, galaxy + reports, whatever) into the
same instance, set their ``instance_name`` to the same string in each
config's ``[galaxy:server]``. This puts them into a single supervisor
group, which may be what you want, although note that any
start/stop/restart/etc. is performed on the entire group, which may
not be what you want.
- The config manager generally views things in terms of config files.
If you change the virtualenv or ``galaxy_root`` in a config file, it
will not change that value for all services in the instance
(supervisor group), it will only change it for the services started
from that config.
- Anything you drop in to
``$GALAXYADM_STATE_DIR/supervisor/supervisord.conf.d`` will be picked
up by supervisord on a ``galaxyadm supervisorctl update`` or just
``galaxyadm update``
- The ``job_conf.xml`` parsed corresponds to the galaxy config, it'll
check the path in ``job_config_file`` in ``[app:main]`` or default to
``galaxy_root/config/job_conf.xml`` if that file exists. If handlers
in ``job_conf.xml`` have a corresponding ``[server:]`` in
``galaxy.ini``, they will be started using Paste. If there is not a
corresponding ``[server:]`` they will be started as a "standalone"
server with ``galaxy_root/lib/galaxy/main.py``
Commands
========
galaxycfg
---------
gravity needs to know where your Galaxy config file(s) are. ``galaxcfg``
is how you manage them.
Use ``galaxycfg -h`` for details. Subcommands also support ``-h``, e.g.
``galaxycfg add -h``.
galaxycfg subcommands
~~~~~~~~~~~~~~~~~~~~~
``add``
Register a Galaxy, Reports, or Tool Shed server config with the process
manager, create a virtualenv, create supervisor configs, and update.
Does not start.
``list``
List config files registered with the process manager.
``instances``
List known instances and services.
``get /path/to/galaxy.ini``
Show stored configuration details for the named config file.
``rename /path/to/old.ini /path/to/new.ini``
Use this if you move your config.
``remove /path/to/galaxy.ini`` ``remove instance_name``
Deregister a Galaxy et. al. server config., or all configs referencing
the supplied ``instance_name``.
galaxyadm
---------
``galaxyadm`` controls your Galaxy server processes. Use this after
you've registered a config file with ``galaxycfg add``.
Use ``galaxyadm -h`` for details.
galaxyadm subcommands
~~~~~~~~~~~~~~~~~~~~~
| ``start [instance_name]``
| ``stop [instance_name]``
| ``restart [instance_name]``
Roughly what you'd expect. If ``instance_name`` isn't provided, perform
the operation on all known instances.
If you call ``start`` from the root (or from 1 subdirectory deep) of a
Galaxy source tree, ``config/galaxy.ini`` if it exists, or else
``config/galaxy.ini.sample`` will automatically be registered with
``galaxycfg`` add and then ``galaxyadm start`` will start the newly
added server.
``reload [instance_name]``
The same as restart but uWSGI master processes will only receive a
``SIGHUP`` so the workers restart but the master stays up.
``graceful [instance_name]``
The same as reload but Paste servers will be restarted sequentially, and
the next one will not be restarted until the previous one is up and
accepting requests.
``update``
Figure out what has changed in configs, which could be:
- changes to ``[galaxy:server]``
- adding or removing ``[server:]`` sections
- adding or removing a ``[uwsgi]`` section
- adding or removing handlers in ``job_conf.xml``
This will perform the operation for all registered configs, which may
cause unintended service restarts.
Any needed changes to supervisor configs will be performed and then
``supervisorctl update`` will be called. You will need to do a
``galaxy start`` after this to start any newly added instances (or
possibly even old instances, since adding new programs to a group in
supervisor causes the entire group to be stopped).
Update is called automatically for the ``start``, ``stop``, ``restart``,
``reload``, and ``graceful`` subcommands.
``supervisorctl [subcommand]``
Pass through directly to supervisor
``shutdown``
Stop supervisord
.. _supervisor: http://supervisord.org/
.. _Galaxy: http://galaxyproject.org/
.. _supervisor issue 491: https://github.com/Supervisor/supervisor/issues/491
.. _virtualenv: https://virtualenv.pypa.io/
.. _virtualenv-burrito: https://github.com/brainsik/virtualenv-burrito
=========
History
=========
0.8
===
- Add auto-register to ``galaxy start`` if it's called from the root (or
subdirectory) of a Galaxy root directory.
- Make ``galaxycfg remove`` accept instance names as params in addition to
config file paths.
- Use the same hash generated for an instance name as the hash for a generated
virtualenv name, so virtualenvs are more easily identified as belonging to a
config.
- Renamed from ``galaxyadmin`` to ``gravity`` (thanks John Chilton).
0.7
===
- Added the ``galaxyadm`` subcommand ``graceful`` on a suggestion from Nicola
Soranzo.
- Install uWSGI into the config's virtualenv if requested.
- Removed any dependence on Galaxy and eggs.
- Moved project to its own repository from the Galaxy clone I'd been working
from.
Older
=====
- Works in progress as part of the Galaxy code.
