Sports2D

Sports2D lets you compute 2D joint and segment angles from a video.

Warning: Angle estimation is only as good as the pose estimation algorithm, i.e., it is not perfect.
Warning: Results are acceptable only if the persons move in the 2D plane, from right to left or from left to right.
If you need research-grade markerless joint kinematics, consider using several cameras, and constraining angles to a biomechanically accurate model. See Pose2Sim for example.

Know issue: Results won't be good with some iPhone videos in portrait mode. This is solved by priorly converting them with ffmpeg -i video_input.mov video_output.mp4, or even more simply with any random online video converter such as https://video-converter.com.

Announcement: Apps with GUI will soon be released for Windows, Linux, MacOS, as well as Android and iOS. Mobile versions will only support simple exploratory analysis. This involves single-person angle computation, in a potentially less accurate and tunable way.

Contents

1. Installation and Demonstration
   1. Installation
   2. Demonstration: Detect pose and compute 2D angles
2. Go further
   1. Use on your own videos
   2. Use OpenPose for multi-person, more accurate analysis
   3. Advanced-settings
   4. How it works
3. How to cite and how to contribute

Installation and Demonstration

Installation

• OPTION 1: Quick install
  Open a terminal. Type python -V to make sure python '>=3.7 <=3.10' is installed, and then:

  pip install sports2d
  

• OPTION 2: Safer install with Anaconda
  Install Miniconda:
  Open an Anaconda prompt and create a virtual environment by typing:

  conda create -n Sports2D python=3.10 -y
  conda activate Sports2D
  pip install sports2d
  

• OPTION 3: Build from source and test the last changes
  Open a terminal in the directory of your choice and clone the Sports2D repository.

  git clone https://github.com/davidpagnon/sports2d.git
  cd sports2d
  pip install .
  

Demonstration: Detect pose and compute 2D angles

Open a terminal, enter pip show sports2d, check sports2d package location.
Copy this path and go to the Demo folder by typing cd <path>\Sports2D\Demo.
Type ipython, and test the following code:

from Sports2D import Sports2D
Sports2D.detect_pose('Config_demo.toml')
Sports2D.compute_angles('Config_demo.toml')

You should obtain a video with the overlaid 2D joint positions, and angle text values. This output is also provided as an image folder.
You should additionally obtain the same information as time series, stored in .csv files. These files can be opened with any spreadsheet software, or with the Pandas Python library for example.
In addition, you will get the original OpenPose-like json coordinates files.

Go further

Use on your own videos

1. Copy-paste Config_demo.toml in the directory of your video.

2. Open it with any text editor.
   Replace the video_file value with the name of your video.

3. Open a terminal or an Anaconda prompt, type:

   cd <Path/to/video/directory>
   conda activate Sports2D (skip if you did not install with Anaconda)
   ipython
   

   from Sports2D import Sports2D
   Sports2D.detect_pose('Config_demo.toml')
   Sports2D.compute_angles('Config_demo.toml')
   

Optionally: If your video is not in the same folder as Config_demo.toml, specify its location in video_dir.
Similarly, if you launch Sports2D in an other directory, specify the location of the config file this way: Sports2D.detect_pose(<path_to_Config_demo.toml>)
In pose, specify the use of BlazePose or OpenPose as joint detectors: this will be detailed in the next section.
In compute_angles, select your angles of interest.

Use OpenPose for multi-person, more accurate analysis

OpenPose is slower than BlazePose, but usually more accurate. It also allows for the detection of multiple persons, whose indices are consistent across frames.

1. Install OpenPose (instructions there).
   Windows portable demo works fine.

2. If you want even more accurate results, use the BODY_25B experimental model instead of the standard BODY_25 one. This requires manually downloading the model. You can optionally download from there the BODY_135 model which will let you compute wrist flexion and hand segment angle, however this will be much slower.

3. In Config_demo.toml → pose.OPENPOSE, specify your OpenPose model, and the path where you downloaded OpenPose.

N.B.: If you want to benefit from the capabilities of OpenPose but do not manage to install it, you can use the Colab notebook version. Note that your data will be sent to the Google servers, which do not follow the European GDPR requirements regarding privacy.

Advanced settings

1. Config_demo.toml → pose_advanced: These settings are only taken into account if OpenPose is used.

   1. load_pose: If you need to change some settings but have already run a pose estimation, you can set this to true in order to not regenerate the json pose files.

   2. save_vid and save_img: You can choose whether you want to save the resulting video and images. If set to false, only pose and angle .csv files will be generated.

   3. interp_gap_smaller_than: Gaps are interpolated only if they are not too wide.

   4. filter: true or false. If true, you can choose among Butterworth, Gaussian, LOESS, or Median, and specify their parameters.
      Beware that the appearance of the unfiltered skeleton may not match the filtered coordinates and angles.

   5. show_plots: Displays a window with tabs corresponding to the coordinates of each detected point. This may cause Python to crash.

2. Config_demo.toml → compute_angles_advanced: These settings are taken into account both with BlazePose and OpenPose.

   1. save_vid and save_img: Cf pose_advanced.

   2. filter: Cf pose_advanced.

   3. show_plots: Cf pose_advanced.

   

N.B.: The settings and results of all analyses are stored int the logs.txt file, which can be found in in the sports2d installation folder (pip show sports2d to find the path).

How it works

Pose detection:

BlazePose or OpenPose are used to detect joint centers from a video.

• If BlazePose is used, only one person can be detected.
  No interpolation nor filtering options are available. Not plotting available.

• If OpenPose is used, multiple persons can be consistently detected across frames. A person is matched to another in the next frame when their average point clouds are at a small euclidian distance.
  Sequences of missing data are interpolated if they are less than N frames long.
  Resulting coordinates can be filtered with Butterworth, gaussian, median, or loess filter. They can also be plotted.

Angle computation:

Joint and segment angles are computed from csv position files.
If a person suddently faces the other way, this change of direction is taken into account.
Resulting angles can be filtered in the same way as point coordinates, and they can also be plotted.

Joint angle conventions:

• Ankle dorsiflexion: Between heel and big toe, and ankle and knee
• Knee flexion: Between hip, knee, and ankle
• Hip flexion: Between knee, hip, and shoulder
• Shoulder flexion: Between hip, shoulder, and elbow
• Elbow flexion: Between wrist, elbow, and shoulder

Segment angle conventions:
Angles are measured anticlockwise between the horizontal and the segment.

• Foot: Between heel and big toe
• Shank: Between knee and ankle
• Thigh: Between hip and knee
• Arm: Between shoulder and elbow
• Forearm: Between elbow and wrist
• Trunk: Between shoulder midpoint and hip midpoint

How to cite and how to contribute

How to cite

If you use this code or data, please cite [Pagnon, 2023].

 @misc{Pagnon2023,
   author = {Pagnon, David},
   title = {Sports2D - Angles from video},
   year = {2023},
   doi= {10.5281/zenodo.7903963},
   publisher = {GitHub},
   journal = {GitHub repository},
   howpublished = {\url{https://github.com/davidpagnon/Sports2D}},
 }

How to contribute

I would happily welcome any proposal for new features, code improvement, and more!
If you want to contribute to Sports2D, please follow this guide on how to fork, modify and push code, and submit a pull request. I would appreciate it if you provided as much useful information as possible about how you modified the code, and a rationale for why you're making this pull request. Please also specify on which operating system and on which python version you have tested the code.

Here is a to-do list: feel free to complete it:

• Compute segment angles.
• Multi-person detection, consistent over time.
• Only interpolate small gaps.
• Filtering and plotting tools.
• Handle sudden changes of direction.
• Colab version for those who cannot install OpenPose.
• Full test on MacOS.
• GUI applications for Windows, Mac, and Linux, as well as for Android and iOS (minimal version on mobile device, with only BlazePose). Code with Kivy.
• Include OpenPose in Sports2D executable files. Dockerize it?
• Track other points and angles with classic tracking methods (cf. Kinovea), or by training a model (cf. DeepLabCut).
• Pose refinement. Click and move badly estimated 2D points. See DeepLabCut for inspiration.
• Constrain points to OpenSim skeletal model for better angle estimation (like with Pose2Sim, but in 2D.
• Add tools for annotating images, undistort them, take perspective into account, etc. (cf. Kinovea).
• Check other methods for sorting people across frames (STAF or LivePoseTracker.

BSD 3-Clause License

Copyright (c) 2022, perfanalytics All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.