Skip to main content

A stall tracker for Python's GIL and Trio tasks

Project description

Perpetuo

perpetuo, verb: To cause to continue uninterruptedly, to proceed with continually

Perpetuo is a stall tracker for Python. Specifically, it can detect when:

  • One thread is holding the GIL for too long, blocking other threads from having a chance to run (requires an instrumented version of CPython)

  • One Trio task is running too long without checkpointing, blocking other tasks from having a chance to run

The actual monitoring is done from a separate process, using a customized version of py-spy. So the monitoring is very low overhead and should not interfere with the monitored process at all. The goal is to be able to use this in production.

Quickstart

  1. pip install perpetuo

  2. Optional: patch CPython (see below)

  3. import perpetuo

  4. If you're using Trio: call perpetuo.dwim() inside trio.run

    If you're not using Trio: call perpetuo.dwim() anywhere

  5. Optional: log perpetuo.dwim()'s return value to see what it did

Available API

perpetuo.start_watcher(): Spawns the monitoring process in the background.

perpetuo.instrument_gil(): Enables GIL instrumentation, or raises RuntimeError if you don't have the patched version of CPython.

perpetuo.instrument_trio(): Enables Trio instrumentation. Must be called inside trio.run.

perpetuo.dwim(): Attempts to call all the above functions as appropriate, and returns a list of strings describing which operations it actually performed. If you're using Trio, make sure to call it inside trio.run.

StallTracker: Low-level class that allows you to add custom instrumentation to other things. See source for details.

Patching CPython to instrument the GIL

Currently the patch is only available for CPython version 3.10.*. It can be downloaded here:

https://github.com/python/cpython/compare/3.10...njsmith:cpython:njs/perpetuo-gil.diff

Bonus

Niccolò Paganini's Moto Perpetuo, performed by Antal Zalai

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

perpetuo-0.3.2.tar.gz (30.7 kB view hashes)

Uploaded Source

Built Distributions

perpetuo-0.3.2-cp39-abi3-win_amd64.whl (1.2 MB view hashes)

Uploaded CPython 3.9+ Windows x86-64

perpetuo-0.3.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB view hashes)

Uploaded CPython 3.9+ manylinux: glibc 2.17+ x86-64

perpetuo-0.3.2-cp39-abi3-macosx_10_9_universal2.whl (2.7 MB view hashes)

Uploaded CPython 3.9+ macOS 10.9+ universal2 (ARM64, x86-64)

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page