jug-schedule 0.2.5

Requirements

This project depends on drmaa and obviously jug.

Installation

..note ::

Until jug 1.5.0 is released you will need to use jug and jug_schedule from git. Alternatively you can use the 1.4.0+git version of jug from my anaconda channel with conda install -c Unode jug-schedule.

Install jug_schedule with:

pip install jug-schedule

and then simply add the following to your ~/.jug/config/jug_user_commands.py:

try:
    from jug_schedule.schedule import ScheduleCommand
    schedule = ScheduleCommand()
except Exception as e:
    sys.stderr.write("Couldn't import schedule, error was {0}\n".format(e))

If you are running directly from git you can instead use:

import os
import sys

sys.path.insert(0, "/path/to/clone/of/jug_schedule/")

try:
    from jug_schedule.schedule import ScheduleCommand
    schedule = ScheduleCommand()
except Exception as e:
    sys.stderr.write("Couldn't import schedule, error was {0}\n".format(e))

Usage

If installed properly, running jug should now include a schedule subcommand.

Running it will try to detect a queueing system and submit jobs to it. jug schedule will only produce warning and errors. Use --verbose debug for detailed messages.

jug status will behave as usual and is the recommended way to check progress of execution.

Configuration

jug_schedule relies on DRMAA for interaction with the queueing system.

In order to use jug_schedule your environment needs to define DRMAA_LIBRARY_PATH. If running env | grep DRMAA_LIBRARY_PATH returns no match, ask your system administrators for the location of this library.

Then use:

export DRMAA_LIBRARY_PATH=/path/to/libdrmaa.so

You only need to set this option on the environment that runs jug schedule.

Resources

An additional feature of jug_schedule is the ability to define job resources.

If you already know jug’s TaskGenerator decorator you can simply replace it with the following where applicable:

from jug_schedule.resources import ResourcesTaskGenerator

@ResourcesTaskGenerator(cpu=10, mem=10, queue="default")
def func(...):
    ...

Supported arguments include: cpu, mem (in MB) and queue.

Command-line options

The following options are available:

--script            - command used to run jug on the cluster. Point this to a shell script if you need to setup jug's environment prior to execution
--max-jobs          - how big is the pool of jug jobs (max number of simultaneous jobs)
--logs              - where to write job logs. Defaults to a directory 'jug_logs' in the current directory.
--cycle-time        - how many seconds to wait between every interaction with the queue system. Defaults to 60
--continue-on-error - jug_schedule will continue until all jobs fail. Default is to stop queueing jobs when a job fails.