geopyspark 0.1.0

A Quick Example

Here is a quick example of GeoPySpark. In the following code, we take NLCD data of the state of Pennslyvania from 2011, and do a polygonal summary of an area of interest to find the min and max classifcations values of that area.

If you wish to follow along with this example, you will need to download the NLCD data and the geojson that represents the area of interest. Running these two commands will download these files for you:

curl -o /tmp/NLCD2011_LC_Pennsylvannia.zip https://s3-us-west-2.amazonaws.com/prd-tnm/StagedProducts/NLCD/2011/landcover/states/NLCD2011_LC_Pennsylvania.zip?ORIG=513_SBDDG
unzip /tmp/NLCD2011_LC_Pennsylvannia.zip
curl -o /tmp/area_of_interest.geojson https://s3.amazonaws.com/geopyspark-test/area_of_interest.json

import json
from functools import partial

from geopyspark.geopycontext import GeoPyContext
from geopyspark.geotrellis.constants import SPATIAL, ZOOM
from geopyspark.geotrellis.geotiff_rdd import get
from geopyspark.geotrellis.catalog import write

from shapely.geometry import Polygon, shape
from shapely.ops import transform
import pyproj


# Create the GeoPyContext
geopysc = GeoPyContext(appName="example", master="local[*]")

# Read in the NLCD tif that has been saved locally.
# This tif represents the state of Pennsylvania.
raster_rdd = get(geopysc=geopysc, rdd_type=SPATIAL,
uri='/tmp/NLCD2011_LC_Pennsylvania.tif',
options={'numPartitions': 100})

tiled_rdd = raster_rdd.to_tiled_layer()

# Reproject the reclassified TiledRasterRDD so that it is in WebMercator
reprojected_rdd = tiled_rdd.reproject(3857, scheme=ZOOM).cache().repartition(150)

# We will do a polygonal summary of the north-west region of Philadelphia.
with open('/tmp/area_of_interest.json') as f:
    txt = json.load(f)

geom = shape(txt['features'][0]['geometry'])

# We need to reporject the geometry to WebMercator so that it will intersect with
# the TiledRasterRDD.
project = partial(
    pyproj.transform,
    pyproj.Proj(init='epsg:4326'),
    pyproj.Proj(init='epsg:3857'))

area_of_interest = transform(project, geom)

# Find the min and max of the values within the area of interest polygon.
min_val = reprojected_rdd.polygonal_min(geometry=area_of_interest, data_type=int)
max_val = reprojected_rdd.polygonal_max(geometry=area_of_interest, data_type=int)

print('The min value of the area of interest is:', min_val)
print('The max value of the area of interest is:', max_val)

# We will now pyramid the relcassified TiledRasterRDD so that we can use it in a TMS server later.
pyramided_rdd = reprojected_rdd.pyramid(start_zoom=1, end_zoom=12)

# Save each layer of the pyramid locally so that it can be accessed at a later time.
for pyramid in pyramided_rdd:
    write('file:///tmp/nld-2011', 'pa', pyramid)

Contact and Support

If you need help, have questions, or like to talk to the developers (let us know what you’re working on!) you contact us at:

• Gitter

• Mailing list

As you may have noticed from the above links, those are links to the GeoTrellis gitter channel and mailing list. This is because this project is currently an offshoot of GeoTrellis, and we will be using their mailing list and gitter channel as a means of contact. However, we will form our own if there is a need for it.

Setup

> echo $SPARK_HOME
/usr/local/bin/spark

pip install geopyspark
geopyspark install-jar -p [path/to/install/jar]

git clone https://github.com/locationtech-labs/geopyspark.git
cd geopyspark

make install

make build
pip install -e .

git clone https://github.com/locationtech-labs/geopyspark.git
cd geopyspark
export PYTHONPATH=$VIRTUAL_ENV/lib/<your python version>/site-packages

make virtual-install

make build
pip install -e .

jupyter notebook

pip uninstall geonotebook
git clone --branch feature/geotrellis https://github.com/geotrellis/geonotebook ~/geonotebook
pip install -r ~/geonotebook/prerequirements.txt
pip install -r ~/geonotebook/requirements.txt
pip install -e ~/geonotebook
jupyter serverextension enable --py geonotebook
jupyter nbextension enable --py geonotebook
make notebook

export PYSPARK_PYTHON="/usr/local/bin/python3"
export SPARK_HOME="/usr/local/apache-spark/2.1.1/libexec"
export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.4-src.zip:${SPARK_HOME}/python/lib/pyspark.zip"

%load_ext autoreload
%autoreload 2

GeoPySpark Script

When GeoPySpark is installed, it comes with a script which can be accessed from anywhere on you computer. These are the commands that can be ran via the script:

geopyspark install-jar -p, --path [download/path] //downloads the jar file
geopyspark jar-path //returns the relative path of the jar file
geopyspark jar-path -a, --absolute //returns the absolute path of the jar file

The first command is only needed when installing GeoPySpark through pip; and it must be ran before using GeoPySpark. If no path is selected, then the jar will be installed wherever GeoPySpark was installed.

The second and third commands are for getting the location of the jar file. These can be used regardless of installation method. However, if installed through pip, then the jar must be downloaded first or these commands will not work.

make docker-build

docker run -it --rm -p 8000:8000 quay.io/geodocker/jupyter-geopyspark:6

Uninstalling

To uninstall GeoPySpark, run the following in the terminal:

pip uninstall geopyspark
rm .local/bin/geopyspark

Contributing

Any kind of feedback and contributions to GeoPySpark is always welcomed. A CLA is required for contribution, see Contributing for more information.