Makes a complete archive of imageboard threads including images, HTML, and JSON.
Project description
BASC Archiver
The BASC Archiver is a Python library used to archive imageboard threads. It uses the 4chan API with the py4chan wrapper. Developers are free to use the BASC-Archiver library for some interesting third-party applications, as it is licensed under the LGPLv3.
It comes with a CLI interface for archiving threads, the thread-archiver. (A GUI interface, the BASC-Archiver, is under development.)
The thread-archiver is designed to archive all content from a 4chan thread:
Download all images and/or thumbnails in given threads.
Download a JSON dump of thread comments using the 4chan API.
Download the HTML page
Convert links in HTML to use the downloaded images
Download CSS/JS and convert HTML to use them
Keep downloading until 404 (with a user-set delay)
Can be restarted at any time
The thread-archiver replaces the typical “Right-click Save As, Web Page Complete” action, which does not save full-sized images or JSON. It works as a guerilla, static HTML alternative to Fuuka.
Usage
Usage: thread-archiver <url>... [--path=<string>] [--runonce] [--silent] [--verbose] [--delay=<int>] [--nothumbs] [--thumbsonly] [--ssl] thread-archiver -h | --help thread-archiver -v | --version Options: --path=<string> Path to folder where archives will be saved [default: ./archive] --runonce Downloads the thread as it is presently, then exits --delay=<int> Delay between thread checks [default: 20] --nothumbs Don't download thumbnails --thumbsonly Download thumbnails, no images --ssl Download using HTTPS --silent Suppresses mundane printouts, prints what's important --verbose Printout more information than normal -h --help Show help -v --version Show version
Example
thread-archiver http://boards.4chan.org/b/res/423861837 --delay 5 --thumbsonly
Installation
The BASC-Archiver works on both Python 2.x and 3.x, and can be installed on Windows, Linux, or Mac OS X.
Windows
Install ActivePython, Either version 2.x and 3.x will work. Make sure to enable the PyPM option, or else pip will not be installed!
After installation, go to the Start Menu and under the ActiveState ActivePython programs folder, click Python Package Manager (PyPM).
A command prompt will appear. Type in the command below and press enter:
pip install BASC-Archiver
Open up a command prompt and navigate to the folder where the python scripts are stored. (In this example we assume it is Python33. Check the root of your C drive if it isn’t)
cd C:\Python33\Scripts
Now we can use the script. Replace the URL in the example below with the one you want to save, and replace the path (keep the quotations) with the folder you want to save to.
python thread-archiver http://boards.4chan.org/b/res/423861837 --path="C:\Users\Danny\4chan-threads"
Sometimes, 4chan will make changes to it’s API. We aim to update the script as soon as any change occurs, so if something’s not working right, use the command below to upgrade to the latest version. If you’re still having problems, raise an issue on our Github.
pip install BASC-Archiver --upgrade
Linux/Mac
Install Python on your computer. On Linux, Python is almost always preinstalled.
We also need to install the Pip package manager, to download all the dependencies, and Requests, a needed library. For an Ubuntu/Debian system, use the following commands:
sudo apt-get install python3-pip python3-requests sudo pip3 install BASC-Archiver
Navigate your terminal to the folder you wish to save the threads into, and run the command below to download an entire thread. (you may also manually specify a path, using the –path= argument)
thread-archiver http://boards.4chan.org/b/res/423861837
Sometimes, 4chan will make changes to it’s API. We aim to update the script as soon as any change occurs, so if something’s not working right, use the command below to upgrade to the latest version. If you’re still having problems, raise an issue on our Github.
sudo pip3 install BASC-Archiver --upgrade
Where to Post Archived threads
Your archived threads can be viewed from any web browser. Just enter the thread’s folder, and open the HTML file.
Alternatively, you can upload the entire archive folder to any static HTML host, no PHP required. We strongly recommend that you share them with the world on some kind of Static HTML host, such as the following:
Github Pages
Gitorious Pages
Google Pages
000webhost
And more!
Please make sure that your content follows the chosen host’s Terms of Service policies (keep your pr0n to yourself, etc.).
Wishlist
Migrate to BA-py4chan, the improved fork of Edgeworth’s original py-4chan wrapper.
.chan.arc - Standard archival format definition for imageboards.
Create a pyFuuka, for archiving from Fuuka’s API.
index.html list - In the future we will make an extension that records links to currently downloaded threads in an index.html file…
License
The 4chan Archiver Class is jointly written and maintained by by antonizoon and Daniel Oaks.
It is based on, and supersedes the Bibliotheca Anonoma’s BA-4chan-thread-archiver tool, written by antonizoon..
The BA-4chan-thread-archiver was originally forked from Socketub’s 4chan-thread-archiver, originally licensed under the GNU Affero General Public License v3 or later.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
