A web scraper for ticker symbols from yahoo finance
Project description
Produces .csv, .xlsx, .json and .yaml files (All files contain same data but in a different format) for stocks, ETF, futures, indexes, mutual funds, currency, warrants and bonds. The ticker symbol, company name and exchange are saved for all symbols. The stock symbols also have a category name.
It gets its data from https://finance.yahoo.com/lookup/. Please note: it is not possible to get all the symbols due to limitations set by Yahoo. About 75%-90% of all symbols are gathered using this script depending on type.
Requirements
Python 2.7 or Python 3.5+
Install
From python package manager (preferred):
pip install Yahoo-ticker-downloader
From source:
python setup.py install
Example Usage
usage: YahooTickerDownloader.py [-h] [-i] [-e] [-E EXCHANGE] [-s SLEEP] [-p]
[-m MARKET]
type
positional arguments:
type The type to download, this can be: future mutualfund
etf stocks index currency
optional arguments:
-h, --help show this help message and exit
-i, --insecure use HTTP instead of HTTPS
-e, --export export immediately without downloading (Only useful if
you already downloaded something to the .pickle file)
-E EXCHANGE, --Exchange EXCHANGE
Only export ticker symbols from this exchange (the
filtering is done during the export phase)
-s SLEEP, --sleep SLEEP
The time to sleep in seconds between requests
-p, --pandantic Stop and warn the user if some rare assertion fails
-m MARKET, --market MARKET
Specify the Region of queried exchanges (us =
USA+Canada, dr=Germany, fr=France, hk=Hongkong,
gb=United Kingdom, default= all)
The first positional argument must be one of the following: stocks etf future index mutualfund currency warrant bond
For example to download all stock symbols you run it like:
YahooTickerDownloader.py stocks
The program takes several hours before it is finished so if you don’t need all symbols it might be a good idea to specify the Region of the exchanges you’re interested in. The program supports suspending and resuming a download. Press CTRL+C to suspend download. Restart the program in the same working directory to resume downloading. It is possible to export a partially downloaded results using the -e flag.
Example of CSV output:
Ticker,Name,Exchange,categoryName,categoryNr
ENZ,"Enzo Biochem Inc.",NYQ,"Medical Laboratories & Research",525
ENZN,"Enzon Pharmaceuticals Inc.",NMS,Biotechnology,515
ENZR,"Energizer Resources Inc.",PNK,,0
EOAA.DE,"E.ON AG",GER,"Diversified Utilities",913
KMX,"CarMax Inc.",NYQ,"Auto Dealerships",744
KMY.MU,KIMBERLY-CLARK,MUN,"Personal Products",323
KN1.DU,ECOUNION,DUS,"Business Software & Services",826
KNCAY,"Konica Minolta Holdings Inc.",PNK,,0
KND,"Kindred Healthcare Inc.",NYQ,"Long-Term Care Facilities",523
KNDI,"Kandi Technologies, Corp",NGM,"Auto Manufacturers - Major",330
...ect
Depending on the type you are downloading, you will get between 3.000 and 100.000+ entries.
Further resources
Download history for symbols: ystockquote
Changelog
Version 2.2.0 (2018-01-31)
Continue exporting to different formats if one export fails ( #41 )
Check robots.txt ( anti-feature )
Version 2.1.1 (2017-08-02)
A too old requests dependency was listed ( #35 )
Version 2.1.0 (2017-05-10)
Added market parameter ( pull request #33 )
Version 2.0.1 (2017-05-07)
Fixed issue where all downloads except stock and currency stopped working.
Version 2.0.0 (2017-05-05)
Switched over to JSON api
Version 1.0.0 (2017-04-04)
Reverted some changes from 0.10.0. Bond is back. Reverted back to English site instead of German.
Resolved CSV issue again. Closes #23 and #16.
Merged #26 Workaround Y! b>2000 limit
Scraper now scrapes a lot more at the expense of runtime.
Support for python2 is back. Latest python 2 & 3 are supported.
Removed xls support
Added xlsx support (#29)
Version 0.10.1 (2017-02-04)
More descriptive help message
Version 0.10.0 (2017-02-02)
Removed bond downloading option.
Uses different yahoo source. Fixes #18
Removed python2 from classifiers. Related to #16
Version 0.9.0 (unreleased)
Added a flag to restrict output to specific stock exchanges.
Version 0.8.1 (2016-08-17)
Workaround for #7 : downloading interruption
Solution for #9 : UnicodeEncodeError
Version 0.7.0 (2016-03-20)
Added –export option. It will transcode the .pickle file immediately to the desired output formats.
Version 0.6.0 (unreleased)
Add 3 retries with an exponential back-off if HTTPError or ChunkedEncodingError is raised when processing _fetchHtml.
Version 0.5.0 (2015-08-16)
Allows downloading using a insecure connection.
The temporarily download file-names now include the ticker type.
Version 0.4.0 (2014-10-28)
Warrant symbols can now be downloaded.
Bond symbols can now be downloaded.
Version 0.3.0 (2014-08-14)
Use HTTPS instead of HTTP
Retry to fetch a page if it contains no symbols (A “fix” for issue #4)
Renamed all ‘Curreny’ to ‘Currency’
Relative imports are used
Fix: .csv file it outputs is encoded in UTF-8 when using python2
Performance: Considerable reduced memory consumption
It now outputs .json, .yaml and .xls files in addition to .csv
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for Yahoo-ticker-downloader-2.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a74de16b4a4f47f588516350e0dc90c0f05498668d0c6e6a59faa7c982f73e46 |
|
MD5 | 98b443cdbf42b88029a354fdbca5bede |
|
BLAKE2b-256 | 89e46e2aa2300f5c03fbc241f9d6c0496aeb821b2ed382d5e05fa1b25d0f950d |