Pose2Sim

This Python repository offers a way to perform a markerless kinematic analysis from multiple calibrated views as a unified workflow from an OpenPose input to an OpenSim result.
It has been tested on Windows 10 but should work similarly on Linux or MacOS.
Pull requests and suggestions are very welcome.

Contents

1. Installation and Demonstration
2. Use on your own data
   1. Prepare for running on your own data
   2. 2D pose estimation
   3. Cameras calibration
   4. 2D Tracking of person
   5. 3D triangulation
   6. 3D filtering
   7. OpenSim kinematics
3. Utilities
4. How to cite and others

Installation and Demonstration

1. Install OpenPose (instructions there).
   Install OpenSim (there).
   Install Anaconda or Miniconda.
2. Clone the Pose2Sim repository.
3. Install requirements: Open an Anaconda terminal in the cloned folder.

   conda create --name Pose2Sim -c conda-forge --file requirements.txt
   conda activate Pose2Sim
   

4. Add the directory where Pose2Sim is installed to your PYTHONPATH environment variable (or append it to the sys.path).
5. This Demo provides an example of the processing of a set of OpenPose input data for a 4 cameras experiment of a person balancing on a beam. Open an Anaconda terminal in the Demo folder of the cloned directory and try the following code in a Python session :

   from Pose2Sim import Pose2Sim
   Pose2Sim.calibrateCams()
   Pose2Sim.track2D()
   Pose2Sim.triangulate3D()
   Pose2Sim.filter3D()
   

   You should obtain a plot of all the trajectories of 3D coordinates. Results are stored as .trc files in the Demo/pose-3d directory. You can check the logs inDemo\Users\logs.txt.
   Default parameters have been provided in Demo\Users\Config.toml but can be edited.
   Section Opensim kinematics describes how to obtain 3D joint angles.

Use on your own data

Prepare for running on your own data

Get ready.

1. Edit the Config.toml file as needed, especially regarding the path to your project. The file is located by default in the User folder of your cloned repository, but you can copy-paste it anywhere (e.g. in your data directory).

2. Set up project hierarchy (in your data directory).

   Project
   │
   ├──opensim
   │    ├──Model_Pose2Sim_Body25b.osim
   │    ├──Scaling_Setup_Pose2Sim_Body25b.xml
   │    └──IK_Setup_Pose2Sim_Body25b.xml
   │        
   └── raw-2d
       ├──raw_cam1_img
       ├──...
       └──raw_camN_img
   

2D pose estimation

Estimate 2D pose from images with Openpose.

Open a command prompt in your OpenPose directory.
Launch OpenPose for each raw image folder:

bin\OpenPoseDemo.exe --model_pose BODY_25B --image_dir <PATH_TO_PROJECT_DIR>\raw-2d\raw_cam1_img --write_json <PATH_TO_PROJECT_DIR>\pose-2d\pose_cam1_json

• N.B.: The BODY_25B model has more accurate results; however, feel free to use any OpenPose model (BODY_25B, BODY_25, COCO, with face and/or hands, etc), and to work with videos instead of image files.
• N.B.: You can also use DeepLabCut, or other 2D pose estimators instead.
  If you decide to do so, you'll have to (1) translate the format to json files (with DLC_to_OpenPose.py script, see Utilities); (2) report the model keypoints in the 'skeleton.py' file; (3) create an OpenSim model if you need 3D joint angles.

N.B.: Markers are not needed and are used only for validation

   Project
   │
   ├──opensim
   │    ├──Model_Pose2Sim_Body25b.osim
   │    ├──Scaling_Setup_Pose2Sim_Body25b.xml
   │    └──IK_Setup_Pose2Sim_Body25b.xml
   │
   ├──pose-2d
   │    ├──pose_cam1_json
   │    ├──...
   │    └──pose_camN_json
   │        
   └── raw-2d
       ├──raw_cam1_img
       ├──...
       └──raw_camN_img
   

Cameras calibration

Calibrate your cameras.

1. If you already have a calibration file (.qca.txt from Qualisys for example):

• copy it in the calib-2d folder
• set [calibration] type to 'qca' in your Config.toml file.

or

2. If you have taken pictures or videos of a checkerboard with your cameras:

• create a folder for each camera in your calib-2d folder,
• copy there the images or videos of the checkerboard
• set [calibration] type to 'checkerboard' in your Config.toml file, and adjust other parameters.

Open an Anaconda prompt or a terminal.
By default, calibrateCams() will look for Config.toml in the User folder of your current directory. If you want to store it somewhere else (e.g. in your data directory), specify this path as an argument: Pose2Sim.calibrateCams(r'path_to_config.toml').

from Pose2Sim import Pose2Sim
Pose2Sim.calibrateCams()

Output:

   Project
   │
   ├──calib-2d
   │   └──Calib.toml
   │
   ├──opensim
   │    ├──Model_Pose2Sim_Body25b.osim
   │    ├──Scaling_Setup_Pose2Sim_Body25b.xml
   │    └──IK_Setup_Pose2Sim_Body25b.xml
   │
   ├──pose-2d
   │    ├──pose_cam1_json
   │    ├──...
   │    └──pose_camN_json
   │        
   └── raw-2d
       ├──raw_cam1_img
       ├──...
       └──raw_camN_img
   

2D tracking of person

Track the person viewed by the most cameras, in case of several detections by OpenPose.
N.B.: Skip this step if only one person is in the field of view.

Open an Anaconda terminal By default, track2D() will look for Config.toml in the User folder of your current directory. If you want to store it somewhere else (e.g. in your data directory), specify this path as an argument: Pose2Sim.track2D(r'path_to_config.toml').

from Pose2Sim import Pose2Sim
Pose2Sim.track2D()

Check printed output. If results are not satisfying, try and release the constraints in the Config.toml file.

Output:

   Project
   │
   ├──calib-2d
   │   ├──calib_cam1_img
   │   ├──...
   │   ├──calib_camN_img
   │   └──Calib.toml
   │
   ├──opensim
   │    ├──Model_Pose2Sim_Body25b.osim
   │    ├──Scaling_Setup_Pose2Sim_Body25b.xml
   │    └──IK_Setup_Pose2Sim_Body25b.xml
   │
   ├──pose-2d
   │   ├──pose_cam1_json
   │   ├──...
   │   └──pose_camN_json
   │
   ├──pose-2d-tracked
   │   ├──tracked_cam1_json
   │   ├──...
   │   └──tracked_camN_json
   │        
   └── raw-2d
       ├──raw_cam1_img
       ├──...
       └──raw_camN_img
   

3D triangulation

Triangulate your 2D coordinates in a robust way.

Open an Anaconda terminal. By default, triangulate3D() will look for Config.toml in the User folder of your current directory. If you want to store it somewhere else (e.g. in your data directory), specify this path as an argument: Pose2Sim.triangulate3D(r'path_to_config.toml').

from Pose2Sim import Pose2Sim
Pose2Sim.triangulate3D()

Check printed output, and vizualise your trc in OpenSim.
If your triangulation is not satisfying, try and release the constraints in the Config.toml file.

Output:

   Project
   │
   ├──calib-2d
   │   ├──calib_cam1_img
   │   ├──...
   │   ├──calib_camN_img
   │   └──Calib.toml
   │
   ├──opensim
   │    ├──Model_Pose2Sim_Body25b.osim
   │    ├──Scaling_Setup_Pose2Sim_Body25b.xml
   │    └──IK_Setup_Pose2Sim_Body25b.xml
   │
   ├──pose-2d
   │   ├──pose_cam1_json
   │   ├──...
   │   └──pose_camN_json
   │
   ├──pose-2d-tracked
   │   ├──tracked_cam1_json
   │   ├──...
   │   └──tracked_camN_json
   │
   ├──pose-3d
       └──Pose-3d.trc>
   │        
   └── raw-2d
       ├──raw_cam1_img
       ├──...
       └──raw_camN_img
   

3D Filtering

Filter your 3D coordinates.

Open an Anaconda terminal. By default, filter3D() will look for Config.toml in the User folder of your current directory. If you want to store it somewhere else (e.g. in your data directory), specify this path as an argument: Pose2Sim.filter3D(r'path_to_config.toml').

from Pose2Sim import Pose2Sim
Pose2Sim.filter3D()

Check your filtration with the displayed figures, and vizualise your trc in OpenSim. If your filtering is not satisfying, try and change the parameters in the Config.toml file.

Output:

   Project
   │
   ├──calib-2d
   │   ├──calib_cam1_img
   │   ├──...
   │   ├──calib_camN_img
   │   └──Calib.toml
   │
   ├──opensim
   │    ├──Model_Pose2Sim_Body25b.osim
   │    ├──Scaling_Setup_Pose2Sim_Body25b.xml
   │    └──IK_Setup_Pose2Sim_Body25b.xml
   │
   ├──pose-2d
   │   ├──pose_cam1_json
   │   ├──...
   │   └──pose_camN_json
   │
   ├──pose-2d-tracked
   │   ├──tracked_cam1_json
   │   ├──...
   │   └──tracked_camN_json
   │
   ├──pose-3d
   │   ├──Pose-3d.trc
   │   └──Pose-3d-filtered.trc
   │        
   └── raw-2d
       ├──raw_cam1_img
       ├──...
       └──raw_camN_img
   

OpenSim kinematics

Obtain 3D joint angles.

Scaling

In the same vein as you would do with marker-based kinematics, start with scaling your model.

1. Use the previous steps to capture a static pose, typically a T-pose (example file: Standing_for_Scaling.trc.)
2. Open the provided Model_Pose2Sim_Body25b.osim model from Pose2Sim/Demo/opensim. (File -> Open Model)
3. Load the provided Scaling_Setup_Pose2Sim_Body25b.xml scaling file from Pose2Sim/Demo/opensim. (Tools -> Scale model -> Load)
4. Replace the example static .trc file with your own data if needed.
5. Run
6. Save the new scaled OpenSim model.

Inverse kinematics

1. Use Pose2Sim to generate 3D trajectories (example file: Balancing_for_IK.trc.)
2. Load the provided IK_Setup_Pose2Sim_Body25b.xml scaling file from Pose2Sim/Demo/opensim. (Tools -> Inverse kinematics -> Load)
3. Replace the example .trc file with your own data if needed, and specify the path to your angle kinematics output file.
4. Run
5. Motion results will appear as .mot file in the Demo/opensim directory (automatically saved).

Batch processing

Alternatively, you can use command-line tools: Open an Anaconda terminal in your OpenSim/bin directory, typically C:\OpenSim <Version>\bin.
You'll need to adjust the time_range, output_motion_file and marker_file in your setup file.

opensim-cmd run-tool <PATH_TO_POSE2SIM>/OpenSim/Setup/<YOUR SCALING OR IK SETUP FILE>.xml

   Project
   │
   ├──calib-2d
   │   ├──calib_cam1_img
   │   ├──...
   │   ├──calib_camN_img
   │   └──Calib.toml
   │
   ├──opensim  
   │    ├──Model_Pose2Sim_Body25b.osim
   │    ├──Scaling_Setup_Pose2Sim_Body25b.xml
   │    ├──Model_Pose2Sim_Body25b_Scaled.osim  
   │    ├──IK_Setup_Pose2Sim_Body25b.xml
   │    └──IK_result.mot   
   │
   ├──pose-2d
   │   ├──pose_cam1_json
   │   ├──...
   │   └──pose_camN_json
   │
   ├──pose-2d-tracked
   │   ├──tracked_cam1_json
   │   ├──...
   │   └──tracked_camN_json
   │
   ├──pose-3d
   │   ├──Pose-3d.trc
   │   └──Pose-3d-filtered.trc
   │        
   └── raw-2d
       ├──raw_cam1_img
       ├──...
       └──raw_camN_img
   

Utilities

A list of standalone tools, which can be both run as scripts or imported as functions. Check usage in the docstrings of each python file.
Optional: If you want to execute them systemwide, add the path to your environment variables:
Windows -> Environment Variables -> New -> Name: PYTHONPATH, Value: <PATH_TO_POSE2SIM>/Utilities \

DLC_to_OpenPose.py
Converts a DeepLabCut (h5) 2D pose estimation file into OpenPose (json) files.
c3d_to_trc.py
Converts 3D point data of a .c3d file to a .trc file compatible with OpenSim. No analog data (force plates, emg) nor computed data (angles, powers, etc) are retrieved.
calib_from_checkerboard.py
Calibrates cameras with images or a video of a checkerboard, saves calibration in a Pose2Sim .toml calibration file.
calib_qca_to_toml.py
Converts a Qualisys .qca.txt calibration file to the Pose2Sim .toml calibration file (similar to what is used in AniPose).
calib_toml_to_qca.py
Converts a Pose2Sim .toml calibration file (e.g., from a checkerboard) to a Qualisys .qca.txt calibration file.
calib_yml_to_toml.py
Converts OpenCV intrinsic and extrinsic .yml calibration files to an OpenCV .toml calibration file.
calib_toml_to_yml.py
Converts an OpenCV .toml calibration file to OpenCV intrinsic and extrinsic .yml calibration files.

json_display_with_img.py
Overlays 2D detected json coordinates on original raw images. High confidence keypoints are green, low confidence ones are red.
json_display_without_img.py
Plots an animation of 2D detected json coordinates.
trc_plot.py
Displays X, Y, Z coordinates of each 3D keypoint of a TRC file in a different matplotlib tab.

trc_desample.py
Undersamples a trc file.
trc_Zup_to_Yup.py
Changes Z-up system coordinates to Y-up system coordinates.
trc_filter.py
Filters trc files. Available filters: Butterworth, Butterworth on speed, Gaussian, LOESS, Median.
trc_gaitevents.py
Detects gait events from point coordinates according to Zeni et al. (2008).

How to cite and others

How to cite

If you use this code or data, please cite Pagnon et al., 2022 or Pagnon et al., 2021.

@Article{Pagnon_2022_Pose2Sim,
  AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
  TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy},
  JOURNAL = {Sensors},
  YEAR = {2022},
  PUBLISHER = {Multidisciplinary Digital Publishing Institute},
  URL = {https://www.mdpi.com/1424-8220/22/7/2712}
}

@Article{Pagnon_2021_Pose2Sim,
  AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
  TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 1: Robustness},
  JOURNAL = {Sensors},
  YEAR = {2021},
  PUBLISHER = {Multidisciplinary Digital Publishing Institute},
  URL = {https://www.mdpi.com/1424-8220/21/19/6530},
}

Contributers

David Pagnon (maintainer, developer), contact@david-pagnon.com
Lionel Reveret, lionel.reveret@inria.fr
Mathieu Domalain, mathieu.domalain@univ-poitiers.fr

To do list

• Pip or conda install
• Make Docker image
• Multiple persons kinematics (triangulating multiple persons, and sorting them in time)
• Solve limb swapping
• Implement normalized DLT and RANSAC triangulation
• Implement optimal fixed-interval Kalman smoothing
• Implement OpenSim London Lumbar Spine Model (validated for a wide range of daily tasks)
• Calibrate with Aruco, Charuco, etc
• Utilities: convert Vicon xcp calibration file to toml
• Run from command line via click or typer
• Make GUI interface
• Catch errors

Pull requests and suggestions are always welcome!

BSD 3-Clause License

Copyright (c) 2021, David PAGNON All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.