A python interface to archive.org.
Project description
A python interface to archive.org
This package installs a CLI tool named ia for using archive.org from the command-line. It also installs the internetarchive python module for programatic access to archive.org. Please report all bugs and issues on Github.
Installation
You can install this module via pip:
pip install internetarchive
Alternatively, you can install a few extra dependencies to help speed things up a bit:
pip install "internetarchive[speedups]"
This will install ujson for faster JSON parsing, and gevent for concurrent downloads.
If you want to install this module globally on your system instead of inside a virtualenv, use sudo:
sudo pip install internetarchive
Configuring
You can configure both the ia command-line tool and the Python interface from the command-line:
$ ia configureYou will be prompted to enter your Archive.org login credentials. If authorization is successful a config file will be saved on your computer that contains your Archive.org S3 keys for uploading and modifying metadata.
Command-Line Usage
Help is available by typing ia --help. You can also get help on a command: ia <command> --help. Available subcommands are configure, metadata, upload, download, search, delete, list, and catalog.
Downloading
To download the entire TripDown1905 item:
$ ia download TripDown1905ia download usage examples:
#download just the mp4 files using ``--glob``
$ ia download TripDown1905 --glob='*.mp4'
#download all the mp4 files using ``--formats``:
$ ia download TripDown1905 --format='512Kb MPEG4'
#download multiple formats from an item:
$ ia download TripDown1905 --format='512Kb MPEG4' --format='Ogg Video'
#list all the formats in an item:
$ ia metadata --formats TripDown1905
#download a single file from an item:
$ ia download TripDown1905 TripDown1905_512kb.mp4
#download multiple files from an item:
$ ia download TripDown1905 TripDown1905_512kb.mp4 TripDown1905.ogvUploading
You can use the provided ia command-line tool to upload items. After configuring ia, you can upload files like so:
#upload files:
$ ia upload <identifier> file1 file2 --metadata="title:foo" --metadata="blah:arg"
#upload from `stdin`:
$ curl http://dumps.wikimedia.org/kywiki/20130927/kywiki-20130927-pages-logging.xml.gz |
ia upload <identifier> - --remote-name=kywiki-20130927-pages-logging.xml.gz --metadata="title:Uploaded from stdin."Metadata
You can use the ia command-line tool to download item metadata in JSON format:
$ ia metadata TripDown1905You can also modify metadata after configuring ia.
$ ia metadata <identifier> --modify="foo:bar" --modify="baz:foooo"Data Mining
IA Mine can be used for data mining Archive.org metadata and search results: https://github.com/jjjake/iamine.
Searching
You can search using the provided ia command-line script:
$ ia search 'subject:"market street" collection:prelinger'Parallel Downloading
If you have the GNU parallel tool intalled, then you can combine ia search and ia metadata to quickly retrieve data for many items in parallel:
$ia search 'subject:"market street" collection:prelinger' | parallel -j40 'ia metadata {} > {}_meta.json'Python module usage
Below is brief overview of the internetarchive Python library. Please refer to the API documentation for more specific details.
Downloading from Python
The Internet Archive stores data in items. You can query the archive using an item identifier:
>>> from internetarchive import get_item
>>> item = get_item('stairs')
>>> print(item.metadata)Items contains files. You can download the entire item:
>>> item.download()or you can download just a particular file:
>>> f = item.get_file('glogo.png')
>>> f.download() #writes to disk
>>> f.download('/foo/bar/some_other_name.png')You can iterate over files:
>>> for f in item.iter_files():
... print(f.name, f.sha1)Uploading from Python
You can use the IA’s S3-like interface to upload files to an item after configuring the internetarchive library.
>>> from internetarchive import get_item
>>> item = get_item('new_identifier')
>>> md = dict(mediatype='image', creator='Jake Johnson')
>>> item.upload('/path/to/image.jpg', metadata=md)Item-level metadata must be supplied with the first file uploaded to an item.
You can upload additional files to an existing item:
>>> item = internetarchive.Item('existing_identifier')
>>> item.upload(['/path/to/image2.jpg', '/path/to/image3.jpg'])You can also upload file-like objects:
>>> import StringIO
>>> fh = StringIO.StringIO('hello world')
>>> fh.name = 'hello_world.txt'
>>> item.upload(fh)Modifying Metadata from Python
You can modify metadata for existing items, using the item.modify_metadata() function. This uses the IA Metadata API under the hood and requires your IAS3 credentials. So, once again make sure you have the internetarchive library configured.
>>> from internetarchive import get_item
>>> item = get_item('my_identifier')
>>> md = dict(blah='one', foo=['two', 'three'])
>>> item.modify_metadata(md)Searching from Python
You can search for items using the archive.org advanced search engine:
>>> from internetarchive import search_items
>>> search = search_items('collection:nasa')
>>> print(search.num_found)
186911You can iterate over your results:
>>> for result in search:
... print(result['identifier'])Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
