pyhelpers

Author: Qian Fu

A small toolkit of helper functions to facilitate data manipulation.

Installation

pip install --upgrade pyhelpers

Note:

• Only a few frequently-used dependencies are specified as essential requirements in the setup.py for packaging pyhelpers. This is to avoid installing redundant packages. If you need to use some functions to which dependencies are not available with the installation of this package (and if you happen not to have those dependencies installed yet), error warnings will be prompted when you import them and so you will know what they are. You can always install those dependencies yourself.

Quick start - some examples

The current version includes the following modules:

• settings
• dir
• download
• store
• geom
• text
• ops

There are a number of functions included in each of the above-listed modules. For a quick start of pyhelpers, one example is provided for each module to demonstrate what the package may do.

settings

This module can be used to change some common settings with 'pandas', 'numpy', 'matplotlib' and 'gdal'. For example:

from pyhelpers.settings import pd_preferences  

pd_preferences changes a few default 'pandas' settings (when reset=False), such as the display representation and maximum number of columns when viewing a pandas.DataFrame.

pd_preferences(reset=False)

If reset=True, all changed parameters should be reset to their default values.

Note that the preset parameters are for the authors' own preference; however you can always change them in the source code to whatever suits your use.

dir

from pyhelpers.dir import cd

cd() returns the current working directory

path_to_pickle = cd("tests", "dat.pickle")
print(path_to_pickle)

If you would like to save dat to a customised folder, say "data". cd() can also change directory

path_to_test_pickle = cd("tests", "data", "dat.pickle")  # cd("tests\\data\\dat.pickle")
print(path_to_test_pickle)

You should see the difference between path_to_pickle and path_to_test_pickle.

download

from pyhelpers.download import download

Note that this module requires requests and tqdm.

Suppose you would like to download a Python logo from online where URL is as follows:

url = 'https://www.python.org/static/community_logos/python-logo-master-v3-TM.png'

Firstly, specify where the .png file will be saved and what the filename will be. For example, to name the downloaded file as "python-logo.png" and save it to a folder named "picture":

path_to_python_logo = cd("tests", "picture", "python-logo.png")

Then use download()

download(url, path_to_python_logo)

If you happen to have Pillow installed, you may also view the downloaded picture:

import Image
python_logo = Image.open(path_to_python_logo)
python_logo.show()

store

Let's now create a pandas.DataFrame (using the above xy_array) as follows:

import pandas as pd
dat = pd.DataFrame(xy_array, columns=['Easting', 'Northing'])

If you would like to save dat as a "pickle" file and retrieve it later, you may import save_pickle and load_pickle:

from pyhelpers.store import save_pickle, load_pickle

To save dat to path_to_test_pickle (see dir):

save_pickle(dat, path_to_test_pickle, verbose=True)  # default: verbose=False

To retrieve/load dat from path_to_test_pickle:

dat_retrieved = load_pickle(path_to_test_pickle, verbose=True)

dat_retrieved and dat should be identical:

print(dat_retrieved.equals(dat))  # should return True

In addition to .pickle, store.py also works with other formats, such as .feather, .csv and .xlsx/.xls.

geom

Note that this module requires pyproj.

If you need to convert coordinates from British national grid (OSGB36) to latitude and longitude (WGS84), you import osgb36_to_wgs84 from geom.py

from pyhelpers.geom import osgb36_to_wgs84

To convert a single coordinate, xy:

xy = np.array((530034, 180381))  # London

easting, northing = xy
lonlat = osgb36_to_wgs84(easting, northing)  # osgb36_to_wgs84(xy[0], xy[1])
print(lonlat)

To convert an array of OSGB36 coordinates, xy_array:

import numpy as np
xy_array = np.array([(530034, 180381),   # London
                     (406689, 286822),   # Birmingham
                     (383819, 398052),   # Manchester
                     (582044, 152953)])  # Leeds

eastings, northings = xy_array.T
lonlat_array = np.array(osgb36_to_wgs84(eastings, northings))
print(lonlat_array.T)

Similarly, if you would like to convert coordinates from latitude and longitude (WGS84) to OSGB36, import wgs84_to_osgb36 instead.

text

Suppose you have a str type variable, named string :

string = 'ang'

If you would like to find the most similar text to one of the following lookup_list:

lookup_list = ['Anglia',
               'East Coast',
               'East Midlands',
               'North and East',
               'London North Western',
               'Scotland',
               'South East',
               'Wales',
               'Wessex',
               'Western']

Let's try find_similar_str included in text.py:

from pyhelpers.text import find_similar_str

find_similar_str relies on two dependencies: fuzzywuzzy (recommended) and nltk. You may choose either one as appropriate.

Use 'fuzzywuzzy' - token_set_ratio

result_1 = find_similar_str(string, lookup_list, processor='fuzzywuzzy')
print(result_1)

Use 'nltk' - edit_distance

result_2 = find_similar_str(string, lookup_list, processor='nltk', substitution_cost=100)
print(result_2)

You may also give find_matched_str a try:

from pyhelpers.text import find_matched_str
result_3 = find_matched_str(string, lookup_list)
print(result_3)

ops

If you would like to request a confirmation before proceeding with some processes, you may use confirmed included in ops.py:

from pyhelpers.ops import confirmed

You may specify, by setting prompt, what you would like to be asked as to the confirmation:

confirmed(prompt="Continue?...", confirmation_required=True)

Continue?... [No]|Yes:
>? # Input something here

If you input Yes (or Y, yes, or something like ye), it should return True; otherwise, False given the input being No (or something like n). When confirmation_required is False, this function would be null, as it would always return True.