Multithreading TF-IDF vectorization for similarity search using sparse matrices for computations.
Project description
Threaded-Sparse-TFIDF
Creating a repository for multithreading TF-IDF vectorization for similarity search using sparse matrices for computations.
Usage:
from TF_IDF import TF_IDF_Vectorizer
tf_idf = TF_IDF_Vectorizer(use_cached=True, print_output=False)
_, ranking = tf_idf.get_similarity_score("science fiction super hero movie", num_workers=k)
Performance:
Image:
Table:
num_workers | time | partition_size |
---|---|---|
1.0 | 1.1117637634277344 | 6.778499999999999 |
2.0 | 0.8195240020751953 | 3.4149000000000003 |
3.0 | 0.7357232332229614 | 2.2773 |
4.0 | 0.7232689380645752 | 1.7081 |
5.0 | 0.7375946760177612 | 1.3555999999999997 |
6.0 | 0.7682486534118652 | 1.1307000000000003 |
7.0 | 0.7640876531600952 | 0.9618 |
8.0 | 0.7513441801071167 | 0.8506 |
9.0 | 0.7795052766799927 | 0.7587 |
10.0 | 0.8141436100006103 | 0.6807 |
11.0 | 0.8003325223922729 | 0.6195000000000002 |
12.0 | 0.8441393852233887 | 0.5697 |
13.0 | 0.8490614175796509 | 0.5258000000000002 |
14.0 | 0.9322290658950806 | 0.48739999999999994 |
15.0 | 0.8824400186538697 | 0.45729999999999993 |
Data
A subset of the Information Retrieval Dataset - Internet Movie Database (IMDB) specifically movies after the year 2007.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for Threaded_Sparse_TFIDF-0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 37b897491abffa2008f25fb048f50902d9c0b54981097b4002eda689499f76f4 |
|
MD5 | b29d15d128a3cb40fd7099bf14c4d01b |
|
BLAKE2b-256 | 80a768dac6d72b076bd99e8eb5751b5f5a826d9c29954118dd310516365d4f14 |
Close
Hashes for Threaded_Sparse_TFIDF-0.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c94e23a9c986dab845e7dfbd9e888f6014250ec87c8099f99d7621c1fb9fad3f |
|
MD5 | 405f22fd3513ce8947b1f6a483531d09 |
|
BLAKE2b-256 | 6d625a9fa985546b5d55bebc9b96124dfe9b008f469ec28d7b20ba69f2ffea19 |