Threading and Multiprocessing for every project.
Project description
Threading and multiprocessing made easy.
Free software: Apache-2.0 license
Documentation: https://lox.readthedocs.io.
Python >=3.6
Lox provides decorators and synchronization primitives to quickly add concurrency to your projects.
Installation
pip3 install –user lox
Features
Multithreading: Powerful, intuitive multithreading in just 2 additional lines of code.
Multiprocessing: Truly parallel function execution with the same interface as multithreading.
Synchronization: Advanced thread synchronization, communication, and resource management tools.
Todos
All objects except lox.process are for threads. These will eventually be multiprocess friendly.
Usage
Easy Multithreading
>>> import lox >>> >>> @lox.thread(4) # Will operate with a maximum of 4 threads ... def foo(x,y): ... return x*y >>> foo(3,4) # normal function calls still work 12 >>> for i in range(5): ... foo.scatter(i, i+1) -ignore- >>> # foo is currently being executed in 4 threads >>> results = foo.gather() # block until results are ready >>> print(results) # Results are in the same order as scatter() calls [0, 2, 6, 12, 20]
Or, for example, if you aren’t allowed to directly decorate the function you would like multithreaded/multiprocessed, you can just directly invoke the decorator:
>>> # Lets say we don't have direct access to this function
... def foo(x, y):
... return x * y
...
>>>
>>> def my_func():
... foo_threaded = lox.thread(foo)
... for i in range(5):
... foo_threaded.scatter(i, i + 1)
... results = foo_threaded.gather()
... # foo is currently being executed in default 50 thread executor pool
... return results
...This also makes it easier to dynamically control the number of thread/processes in the executor pool. The syntax is a little weird, but this is just explicitly invoking a decorator that has optional arguments:
>>> # Set the number of executer threads to 10
>>> foo_threaded = lox.thread(10)(foo)Easy Multiprocessing
>>> import lox
>>>
>>> @lox.process(4) # Will operate with a pool of 4 processes
... def foo(x, y):
... return x * y
...
>>> foo(3, 4) # normal function calls still work
12
>>> for i in range(5):
... foo.scatter(i, i + 1)
...
-ignore-
>>> # foo is currently being executed in 4 processes
>>> results = foo.gather() # block until results are ready
>>> print(results) # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]Progress Bar Support (tqdm)
>>> import lox
>>> from random import random
>>> from time import sleep
>>>
>>> @lox.thread(2)
... def foo(multiplier):
... sleep(multiplier * random())
...
>>> for i in range(10):
>>> foo.scatter(i)
>>> results = foo.gather(tqdm=True)
90%|████████████████████████████████▌ | 9/10 [00:03<00:00, 1.32it/s]
100%|███████████████████████████████████████| 10/10 [00:06<00:00, 1.46s/it]History
0.11.0 (2022-04-07)
Set number of workers to 0 (in thread execution) if the environment variable LOX_DEBUG is set to a true-like value (true, 1, etc.). Makes it easier to set breakpoints in multithreaded code without having to manually edit the decorator.
0.10.0 (2021-12-18)
Remove dependency pinning.
Allow @lox.thread(0). This will execute scatter calls in parent thread. Useful for debugging breakpoints in parallelized code.
0.9.0 (2020-11-25)
tqdm support on lox.process.gather. See v0.8.0 release notes for usage.
0.8.0 (2020-11-25)
tqdm support on lox.thread.gather * Can be a bool:
>>> my_func.gather(tqdm=True)
Can be a tqdm object:
>>> from tqdm import tqdm >>> pbar = tqdm(total=100) >>> for _ in range(100): >>> my_func.scatter() >>> my_func.gather(tqdm=pbar)
0.7.0 (2020-07-20)
Complete rework of workers + Fix memory leaks
Drop support for python3.5
Drop support for chaining in favor of simpler codebase
0.6.3 (2019-07-30)
Alternative fix for 0.6.2.
0.6.2 (2019-07-21)
Update dependencies
Fix garbage-collecting exclusiviity
0.6.1 (2019-07-21)
Fix memory leak in lox.process.
0.6.0 (2019-07-21)
lox.Announcement subscribe() calls now return another Announcement object that behaves like a queue instead of an actual queue. Allows for many-queue-to-many-queue communications.
New Object: lox.Funnel. allows for waiting on many queues for a complete set of inputs indicated by a job ID.
0.5.0 (2019-07-01)
New Object: lox.Announcement. Allows a one-to-many thread queue with backlog support so that late subscribers can still get all (or most recent) announcements before they subscribed.
New Feature: lox.thread scatter calls can now be chained together. scatter now returns an int subclass that contains metadata to allow chaining. Each scatter call can have a maximum of 1 previous scatter result.
Documentation updates, theming, and logos
0.4.3 (2019-06-24)
Garbage collect cached decorated object methods
0.4.2 (2019-06-23)
Fixed multiple instances and successive scatter and gather calls to wrapped methods
0.4.1 (2019-06-23)
Fixed broken workers and unit tests for workers
0.4.0 (2019-06-22)
Semi-breaking change: lox.thread and lox.process now automatically pass the object instance when decorating a method.
0.3.4 (2019-06-20)
Print traceback in red when a thread crashes
0.3.3 (2019-06-19)
Fix bug where thread in scatter of lox.thread double releases on empty queue
0.3.2 (2019-06-17)
Fix manifest for installation from wheel
0.3.1 (2019-06-17)
Fix package on pypi
0.3.0 (2019-06-01)
Multiprocessing decorator. lox.pool renamed to lox.thread
Substantial pytest bug fixes
Documentation examples
timeout for RWLock
0.2.1 (2019-05-25)
Fix IndexSemaphore context manager
0.2.0 (2019-05-24)
Added QLock
Documentation syntax fixes
0.1.1 (2019-05-24)
CICD test
0.1.0 (2019-05-24)
First release on PyPI.
