Heron

Heron is an open source software conceived as an experiment (and general data) pipeline implementation tool. It allows users to realise their conceptual experimental designs such that the final result, i.e. the implementation running the experiment, visually and structurally bears a significant resemblance to its mental schema, i.e. the idea of the experiment in the experimenter’s imagination.

Heron combines a Node based experiment designer with a Python API allowing users to quicly construct and add their own Nodes. It fully obfuscates the Node communication from the user and offers seamless communication between separate machines running different operating systems.

A more thorough description of both why Heron exists in the first place and some info on how to use it can be found in this document.

The manual is here.

Videos and Tutorials

A summer 2022 talk at the Sainsbury Wellcome Centre introducing Heron: https://www.youtube.com/watch?v=o6Pz-1uaJjY

A quick overview of using Heron through an example of putting together a pipeline involving Single Board Computers, Deep Convolutional Networks and a game of Snake. https://www.youtube.com/watch?v=dRXeFBYG50Q

Introduction

Heron is a Python only graphical editor designed to initiate, connect and orchestrate multiple processes, passing data between each other, potentially running on multiple computers. Each process is graphically represented as a node with inputs and outputs. Connecting an output of one process to the input of another will result in the data generated by the output process to be passed to the input one, subsequently triggering the input processes functionality.

Heron has originally been conceptualised as a framework to allow modular and fast experimental setup. It is designed to connect multiple pieces of hardware with analysis and data collection code in a versatile / experiment specific matter. Its main goal is to abstract away the need to pass data through the different parts of a pipeline with specific data collection and on the fly analysis requirements, an effort which more often than not, results in single use, spaghetti code.

Heron is not another visual programming language. It is just a graphical abstraction of the way individual pieces of code, each running in its own process, communicate with each other in an experimental pipeline. The code that does the actual work is Python code and the user is expected to construct their own pieces of code (that can be trivially wrapped into the code necessary to have them represented as Heron nodes).

Here is an example of the Heron editor running a simple pipeline

More information on the parts of the Heron editor and its use see the Use paragraph.

Installation

One way is pip:

python -m pip install heron-42ad

Yet, Heron does not require any type of installation. One can just download the repo, put it somewhere and then run the editor.py script found in the Heron/Heron/gui folder.

Important note. The whole of the Heron repo must be found in all the computers that run nodes in a pipeline. The editor will only run in one of the computers but the whole repo needs to be present in all computers (since this defines Heron's communication protocol). Any code that runs in a machine separate to the computer running the editor must be part of the Heron directory structure wrapped appropriately in the code that creates the Heron nodes.

The Heron repository comes installed with a small number of Nodes that are generic enough to be useful to most users. This set will over time increase. Yet, for more specialised Nodes Heron allows users to create in Heron's folder structure symbolic links to any directory on their hard disks that has a set of Heron Nodes. For a user to do this they will either have to create their own Nodes or download from an online repository hub (e.g. GitHub) a set of Nodes. To begin with look into https://github.com/Heron-Repositories. Once the code for one or more Nodes is on a user's hard disk then one should access the Operations button on the toolbar, press Add a new Operations Folder (as Symbolic Link from Existing Repo) button and navigate to the folder that carries the Sources, Transforms and/or Sinks folders inside of which the new Nodes exist. That will create a symbolic link of the correct directory strucre inside Heron's folder structure and Heron will from that point on be able to use the new Nodes. If one wants to remove the Nodes all they have to do is delete the symbolic link from Heron's Operations/XXX directory (where XXX = Sources or Transforms or Sinks). That will not affect the actual repository of the new Nodes.

Requirements.

The library requirements of Heron are:

python >=3.11

pyzmq >= 22.x

numpy

pandas

dearpygui >= 1.2

paramiko

tornado

psutil

pyinput

pyserial

h5py

Unfortunately the in-Node visualisation capabilities and a good few Nodes (Visualiser, Cameras, OpenCV filters) repy on opencv = 4.x (3.x should work but not tested). Sorry about that! In the future there will be an effort to remove basic dependency on opencv but for now, it is what it is. At the time of writting (Feb. 2022) conda will not install opencv for python > 3.8, so if you are using python 3.11 or newer you have to install opencv through pip.

If you use conda then dearpygui has to also be installed by pip (no conda packages yet).

So once you have a conda environment up and running (and you have added the conda-forge channel to your config as mentioned here https://ostechnix.com/enable-conda-forge-channel-for-conda-package-manager/) and you are in the environment in your command prompt you need to do the following:

conda install numpy
conda install pandas
conda install tornado
conda install pyzmq
conda install paramiko
conda install psutil

pip install dearpygui
pip install opencv-python

To test that everything works start Heron. That is done by running python folder_of_Heron/Heron/gui/editor.py

Other libraries, which are used in specific Nodes only, can be installed as needs be. One can check the import statements of the Worker scripts of the specific Operations to see what libraries they import.

For optional libraries that are called by often used Nodes do

conda install scipy
conda install h5py
conda install pynput
conda install pyserial

Use

The left hand side table in the editor is a list of all available nodes. The user clicks on the ones they need and then proceeds to connect the outputs to the inputs and sets up the parameters values. Once this is done, the user starts the Graph that has been generated. The result is the initialisation of a series of processes (double the number of nodes plus one) which then pass data between them as the connectivity of their nodes prescribes.

There are three types of nodes:

The Source nodes which are meant to bring data from the outside world into the pipeline. Examples of these are cameras, microphones, data acquisition devices and in general any type of hardware that will generate data. For example, in the pipeline shown above, there is a single Source node live capturing camera frames. Source nodes have only outputs.

Then there are the Transform nodes which grab data from one of their inputs, transform it and then pass it to their output. Heron supports both one output to one input and one output to many inputs connectivity. It also supports Nodes with multiple inputs and multiple outputs. In the above example, each captured frame is passed through a canny filter.

Finally there are Sink nodes which only have inputs and which are meant as final saving points for the data. In the above example this is the Save FFMPEG video which takes the canny transformed frame and adds it to a video file (through running ffmpeg).

In the future there will be Kernel nodes with no inputs or outputs (self sufficient processes that do stuff without requiring input or generating output).

Heron protocol

The below picture gives the overview of the communication protocol Heron uses.