Work In Progress

Features

• CLI for downloading model from s3 and pulling docker image of triton server
• Support different container engines
  • Docker
  • Apptainer
• Support different compute environments
  • Local
  • Slurm

Installation

pip install hypha-launcher

Usage

$ hypha-launcher
NAME
    hypha-launcher

SYNOPSIS
    hypha-launcher - GROUP | COMMAND | VALUE

GROUPS
    GROUP is one of the following:

     container_engine
       Container engine abstraction. Provides a common interface to container engines, such as docker, apptainer, podman, etc.

COMMANDS
    COMMAND is one of the following:

     download_models_from_s3
       Download models from S3

     pull_image

     run_launcher_server
       Start a launcher server, run in the login node of HPC.

     run_worker
       Run a worker server, run in the compute node of HPC.

VALUES
    VALUE is one of the following:

     debug

     store_dir

Download model from s3

$ hypha-launcher --store-dir="./triton_store" - download_models_from_s3 bioengine-model-runner.* --n_parallel=5

Pull docker image of triton server

$ hypha-launcher --store-dir="./triton_store" - pull_image

Start launcher

On local machine:

$ hypha-launcher --store-dir="./triton_store/" - run_launcher_server 

On HPC(Slurm):

$ hypha-launcher --store-dir="./triton_store/" - run_launcher_server --slurm-settings='{"account": "your-account", "gpus_per_node": "V100:1", "time": "01:00:00"}'

TODO

• Download model from s3
• Pull docker image of triton server
• Run triton server
• Register service on hypha
• Conmmunicate with triton server
• Test on HPC
• Job management(Auto stop and restart)
• Load balancing
• Documentation

Development

Install the package in editable mode with the following command:

pip install -e .
pip install -r requirements-dev.txt