arXiv Loader

This tool is a wrapper of the arXiv API that allows you to retrieve metadata of articles published on arXiv as pandas.DataFrame.
Please abide by the Terms of Usage of the arXiv API.

Installation

pip install arxivloader

Usage

Please consult the arXiv API documentation for help in constructing a valid query string.

Searching by keyword

To search for a keyword the query needs to start with search_query= followed by a prefix and the search word.
Possible prefixes are

Please have a look at the arXiv API documentation for details.

import arxivloader

keyword = "DustPy"
prefix = "all"
query = "search_query={pf}:{kw}".format(pf=prefix, kw=keyword)
columns = ["id", "title", "authors"]

df = arxivloader.load(query, columns=columns)
print(df)

Searching by id

To search for a specific arXiv ID the query needs to start with id_list= followed by a comma-separated list of arXiv IDs:

import arxivloader

IDs = ["1909.04674", "1909.10526"]
query = "id_list={}".format(",".join(IDs))
columns = ["id", "title", "authors"]

df = arxivloader.load(query, columns=columns)

print(df)

Filtering specific articles by keywords

If both, search_query= and id_list= are present, the given arXiv articles are filtered by the give key word.

import arxivloader

keyword = "DSHARP"
prefix = "ti"
IDs = ["1909.04674", "1909.10526"]
query = "search_query={pf}:{kw}&id_list={ids}".format(pf=prefix, kw=keyword, ids=",".join(IDs))
columns = ["id", "title", "authors"]

df = arxivloader.load(query, columns=columns)

print(df)

Searching by date

It is possible to only retrieve entries in a specified date window.
This query selects all publications that have been submitted to astro-ph.EP on July 1st 2022 between 8am and 1pm.

import arxivloader

prefix = "cat"
cat = "astro-ph.EP"
submittedDate = "[20220701080000+TO+20220701130000]"
query = "search_query={pf}:{cat}+AND+submittedDate:{sd}".format(pf=prefix, cat=cat, sd=submittedDate)
columns = ["id", "title", "authors", "published"]

df = arxivloader.load(query, columns=columns, sortBy="submittedDate", sortOrder="ascending")
print(df)

Searching by category

It is possible to search large number of articles by category. Please be responsible with the traffic this query causes.

import arxivloader

keyword = "astro-ph.EP"
prefix = "cat"
query = "search_query={pf}:{kw}".format(pf=prefix, kw=keyword)
columns = ["id", "title", "primary_category", "categories", "published"]

df = arxivloader.load(query, columns=columns, sortBy="submittedDate", sortOrder="descending", num=1000, page_size=100)

print(df.head(5))

Options

arxivloader.load() has several keyword arguments:

The default options are usually good enough.
The delay has to be at least three seconds to be fair with the load on the arXiv API.
It can happen that the arxiv API does not respond for a query. timeout will set the time after which arxivloader assumes a failed attempt and will retry at most five times. Please note, that timeout needs to be larger than the arXiv API takes to process the query, which depends on page_size. Consider two minutes for ten thousand entries in a page.
If verbosity is 0, arxivloader will not display anything on screen. If verbosity is 1, arxivloader will print out the number of retrieved entries at the end of execution. If verbosity is 2, arxivloader will additionally show a progess bar.

Acknowledgements

Thank you to arXiv for use of its open access interoperability.