Skip to main content

PXM ingest quality control

Project description

Build Status PyPI

Tilecontrol contains utilities for Satellite basemap quality control. We use it catch Nodata and HTTP errors in our tilesets.

Installation

pip install tilecontrol

Requirements

A scene cache

An update-to-date list of the contents of s3://mapbox-pxm/scenes is required to run accurate QC work. Generating that list can be be quite time consuming, on the order of multiple hours.

Alternatively, use an existing cache.

$ aws s3 ls mapbox/playground/satellite-qc/scene_caches/
2015-11-03 10:13:21 2864329827 2015-10-11.log
2015-10-27 10:33:36 4712506108 2015-10-26.log
2015-11-04 11:51:02 6533519570 2015-11-04.log

An API Token

Your own token should be sufficient for public datasets. For private mosaics (like DG), we use the token at https://github.com/mapbox/dg-utils/issues/140.

Command Line Interface

$ tlc
Usage: tlc [OPTIONS] COMMAND [ARGS]...

  Utilities for Satellite basemap quality control

Options:
  --help  Show this message and exit.

Commands:
  check                  Catch HTTP and Nodata errors.
  edges                  Finds the edges of a set of tiles.
  init                   Bootstrap the QC process.
  parse_checker_results  Legacy - Parse `checker` output.

init

tlc init bootstraps the QC process. With the exception of the s3 scene cache, it creates the files upon which the remainder of the QC process operates on.

It takes the following arguments: - source_id of the mosaic as found in pxm-sources. - layer_id of the mosaic, such as mapbox.satellite-test. - cache refers to the s3 scene list, and is expected to be a local file. - destination refers to where you’d like the output files to be stored.

tlc init takes an important option, --token, which let’s you specify the $MapboxAccessToken you’d like to use for the query. Your personal key may work well for public tilesets, but private tilesets will often require use of a special key. For example, DG QC work requires use of the token found at https://github.com/mapbox/dg-utils/issues/140.

For example:

tlc init dg_vivid_latam_caribbean digitalglobe.vivid-latam-caribbean 2015-10-11.log output --token $token

Would yield the following files:

dg_vivid_latam_caribbean_awsparse_outputs.json
dg_vivid_latam_caribbean_edges_2015_11_09
dg_vivid_latam_caribbean_geojsons_2015_11_09
dg_vivid_latam_caribbean_sourceimgGeojsons_2015_11_09
dg_vivid_latam_caribbean_sourceimgnames_2015_11_09
dg_vivid_latam_caribbean_zxy_2015_11_09

edges

tlc edges is typically called as a part of the init subcommand. For a mosaic of any shape, it will identify those tiles which exist on the periphery of the mosaic.

  • Input is a list of ZXY’s

  • Output is a list of ZXY’s

check

tlc check performs the QC process, which consists of checking for HTTP or Nodata errors for the URL’s we expect to exist.

It takes the following arguments: - layer_id of the mosaic, such as mapbox.satellite-test. - inzxy a list of ZXY’s to check. - edges a list of the ZXY’s of the edges of the mosaic. - geojsons a list of the filename’s we expect to find in the scene cache.

Additionally, it takes a --token option as described for tlc init.

For example:

tlc check dg_vivid_eastern_europe dg_vivid_eastern_europe_zxy_2015_10_27 dg_vivid_eastern_europe_edges_2015_10_27 dg_vivid_eastern_europe_geojsons_2015_10_27 --token $token

Would yield the following files:

dg_vivid_eastern_europe_missing_and_nodata_zxy
dg_vivid_eastern_europe_missing_and_nodata_sourceimgs

Contributing

To develop against tilecontrol:

git clone git@github.com:mapbox/tilecontrol.git
cd tilecontrol
pip install -e ".[test]"

Tests can be run with py.test

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tilecontrol-0.1.6.tar.gz (12.1 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page