Python libraries for crowdsourcing
Project description
Crowd-Kit: Computational Quality Control for Crowdsourcing
Crowd-Kit is a powerful Python library that implements commonly-used aggregation methods for crowdsourced annotation and offers the relevant metrics and datasets. We strive to implement functionality that simplifies working with crowdsourced data.
Currently, Crowd-Kit contains:
- implementations of commonly-used aggregation methods for categorical, pairwise, textual, and segmentation responses
- metrics of uncertainty, consistency, and agreement with aggregate
- loaders for popular crowdsourced datasets
The library is currently in a heavy development state, and interfaces are subject to change.
Installing
Installing Crowd-Kit is as easy as pip install crowd-kit
Getting Started
This example shows how to use Crowd-Kit for categorical aggregation using the classical Dawid-Skene algorithm.
First, let us do all the necessary imports.
from crowdkit.aggregation import DawidSkene
from crowdkit.datasets import load_dataset
import pandas as pd
Then, you need to read your annotations into Pandas DataFrame with columns task, performer, label. Alternatively, you can download an example dataset.
df = pd.read_csv('results.csv') # should contain columns: task, performer, label
# df, ground_truth = load_dataset('relevance-2') # or download an example dataset
Then you can aggregate the performer responses as easily as in scikit-learn:
aggregated_labels = DawidSkene(n_iter=100).fit_predict(df)
Implemented Aggregation Methods
Categorical Responses
| Method | Status |
|---|---|
| Majority Vote | ✅ |
| Dawid-Skene | ✅ |
| Gold Majority Vote | ✅ |
| M-MSR | ✅ |
| Wawa | ✅ |
| Zero-Based Skill | ✅ |
| GLAD | 🟡 |
| BCC | 🟡 |
Textual Responses
| Method | Status |
|---|---|
| RASA | ✅ |
| HRRASA | ✅ |
| ROVER | ✅ |
Image Segmentation
| Method | Status |
|---|---|
| Segmentation MV | ✅ |
| Segmentation RASA | ✅ |
| Segmentation EM | ✅ |
Pairwise Comparisons
| Method | Status |
|---|---|
| Bradley-Terry | ✅ |
| Noisy Bradley-Terry | ✅ |
Questions and Bug Reports
For reporting bugs please use the Toloka/bugreport page.
License
© YANDEX LLC, 2020-2021. Licensed under the Apache License, Version 2.0. See LICENSE file for more details.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for crowd_kit-0.0.7-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | c07781250dd644ac1b4b0686f772e987138365d7028270eccd2c1150fdf692df |
|
| MD5 | 58116f9a5ae926f4647f3864cc577ac3 |
|
| BLAKE2b-256 | 06fdf2db7e5621bdf537357be8680e9d86b217a14d2567a1ee6c16ccb8c81a6b |
