Skip to main content

Turn StackExchange dumps into ZIM files for offline usage

Project description

Sotoki

Sotoki (Stack Overflow to Kiwix) is an openZIM scraper to create offline versions of Stack Exchange websites such as Stack Overflow.

It is based on Stack Exchange's Data Dumps hosted by The Internet Archive.

CodeFactor Docker License: GPL v3 PyPI version shields.io

Usage

Sotoki works off a domain that you must provide. That is the domain-name of the stackexchange website you want to scrape. Run sotoki --list-all to get a list of those

Docker

docker run -v my_dir:/output openzim/sotoki sotoki --help

Installation

sotoki is a Python3 software. If you are not using the Docker image, you are advised to use it in a virtual environment to avoid installing software dependencies on your system.

python3 -m venv ./env  # creates a virtual python environment in ./env folder
./env/bin/pip install -U pip  # upgrade pip (package manager). recommended
./env/bin/pip install -U sotoki  # install/upgrade sotoki inside virtualenv

# direct access to in-virtualenv sotoki binary, without shell-attachment
./env/bin/sotoki --help
# alias or link it for convenience
sudo ln -s $(pwd)/env/bin/sotoki /usr/local/bin/

# alternatively, attach virtualenv to shell
source env/bin/activate
sotoki --help
deactivate  # unloads virtualenv from shell

Developers

Anybody is welcome to improve the Sotoki.

To run Sotoki off the git repository, you'll need to download a few external dependencies that we pack in Python releases. Just run python src/sotoki/dependencies.py.

See requirements.txt for the list of python dependencies.

Users

You don't have to make your own ZIM files of Stack Exchange's Web sites. Updated ZIM files are built on a regular basis for all of them. Look at https://library.kiwix.org/?category=stack_exchange to download them.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sotoki-2.0.2.tar.gz (1.6 MB view hashes)

Uploaded Source

Built Distribution

sotoki-2.0.2-py3-none-any.whl (1.8 MB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page