FTIO captures periodic I/O using frequency techniques. Many high-performance computing (HPC) applications perform their I/O in bursts following a periodic pattern. Predicting such patterns can be very efficient for I/O contention avoidance strategies, including burst buffer management, for example. FTIO allows offline detection and online prediction of periodic I/O phases. FTIO uses the discrete Fourier transform (DFT), combined with outlier detection methods to extract the dominant frequency in the signal. Additional metrics gauge the confidence in the output and tell how far from being periodic the signal is. A complete description of the approach is provided here.

This repository provides two main Python-based tools:

• ftio: uses frequency techniques and outlier detection methods to find the period of I/O phases
• predictor: implements the online version of FTIO. It reinvokes FTIO whenever new traces are appended to the monitored file. See online prediction for more details. We recommend using TMIO to generate the file with the I/O traces.

Other tools:

• ioplot generates interactive plots in HTML
• ioparse parses and merges several traces to an Extra-P supported format. This allows one to examine the scaling behavior of the monitored metrics. Traces generated by FTIO (frequency modls), TMIO (msgpack, json and jsonl) and other tools (Darshan, Recorder, and TAU Metric Proxy) are supported.

Join the Slack channel or see the latest updates here: Latest News

Installation

FTIO is available on PYPI and can be easily installed via pip:

pip install ftio-hpc

For the latest GitHub version, FTIO can be installed either automatically or manually. As a prerequisite, for the virtual environment, python3.11-venv is needed, which can be installed on Ubuntu, for example, with:

apt install python3.11-venv

Note there are currently issues with pyDarshan on Mac, that can be solved as mentioned here

Automated installation

Simply call the make command:

make install

This generates a virtual environment in the current directory, sources .venv/bin/activate, and installs FTIO as a module.

If you don't need a dedicated environment, just call:

make ftio PYTHON=python3

Manual installation

Create a virtual environment if needed and activate it:

python3 -m venv .venv
source .venv/bin/activate

Install all tools provided in this repository simply by using pip:

pip install .

Note: you need to activate the environment to use ftio and the other tools using:

source path/to/venv/bin/activate

⬆

Usage

For installation instructions see installation.

To call ftio on a single file, use:

ftio filename.extension

Supported extensions are json, jsonLines, msgpack, and darshan. For recorder, you provide the path to the folder instead of filename.extension. For more on the input format including a custom format see supported file formats.

FTIO provides various options and extensions. To see all available command line arguments, call:

ftio -h

  
usage: ftio [-h] [-m MODE] [-r RENDER] [-f FREQ] [-ts TS] [-te TE] [-tr TRANSFORMATION] [-e ENGINE]
            [-o OUTLIER] [-le LEVEL] [-t TOL] [-d] [-nd] [-re] [--no-reconstruction] [-p] [-np] [-c] [-w]
            [-fh FREQUENCY_HITS] [-v] [-s] [-ns] [-a] [-na] [-i] [-ni] [-x DXT_MODE] [-l LIMIT]
            files [files ...]

ftio generates frequency predictions. There are several options available to enhance the predictions. In the standard mode, the DFT is used in combination with an outlier detection method. Additionally, autocorrelation can be used to further increase the confidence in the results:

1. DFT + outlier detection (Z-score, DB-Scan, Isolation forest, peak detection, or LOF)​
2. Optionally: Autocorrelation + Peak detection (-c flag)
3. If step 2. is performed, the results from both predictions aer merged automatically

Several flags can be specified. The most relevant settings are:

predictor has the same syntax as ftio. All arguments that are available for ftio are also available for predictor.

⬆

Testing

There is a 8.jsonl file provided for testing under examples. On your system, navigate to the folder and call:

ftio 8.jsonl

⬆

Contributing

If you have a suggestion that would make this better, please fork the repository and create a pull request.

1. Fork the project
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a pull request

⬆

Contact

• Ahmad Tarraf: ahmad.tarraf@tu-darmstadt.de

⬆

License

Distributed under the BSD 3-Clause License. See LISCENCE for more information.

⬆

Acknowledgments

Authors:

• Ahmad Tarraf

This work is a result of cooperation between the Technical University of Darmstadt and INRIA in scope of the EuroHPC ADMIRE project.

⬆

Citation

 @inproceedings{Tarraf_Bandet_Boito_Pallez_Wolf_2024, 
  author={Tarraf, Ahmad and Bandet, Alexis and Boito, Francieli and Pallez, Guillaume and Wolf, Felix},
  title={Capturing Periodic I/O Using Frequency Techniques}, 
  booktitle={2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)}, 
  address={San Francisco, CA, USA}, 
  year={2024},
  month=may, 
  pages={1–14}, 
  notes = {(accepted)}
 }

⬆

Publications

1. A. Tarraf, A. Bandet, F. Boito, G. Pallez, and F. Wolf, “Capturing Periodic I/O Using Frequency Techniques,” in 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS), San Francisco, CA, USA, May 2024, pp. 1–14.

2. A. Tarraf, A. Bandet, F. Boito, G. Pallez, and F. Wolf, “FTIO: Detecting I/O periodicity using frequency techniques.” arXiv preprint arXiv:2306.08601 (2023).

⬆