AFSC GAP for Python

Microlibrary for pythonic interaction with the public bottom trawl surveys data from the NOAA AFSC GAP.

See project Github and webpage.

Purpose

Unofficial microlibrary for interacting with the API for bottom trawl surveys from the Ground Fish Assessment Program (GAP), a dataset produced by the Resource Assessment and Conservation Engineering (RACE) Division of the Alaska Fisheries Science Center (AFSC) as part of the National Oceanic and Atmospheric Administration's Fisheries organization (NOAA Fisheries).

Need

Scientists and developers working on ocean health have an interest in survey data from organizations like NOAA Fisheries. However, interacting with the GAP API from NOAA AFSC in Python requires understanding a complex schema, how to interact with a proprietary REST data service, forming long query URLs, and navigating pagination. These various elements together may increase the barrier for working with these data, limiting their reach within the Python community.

Goal

This low-dependency library provides a type-annoated and documented Python interface to these data with ability to query with filters and pagination, providing results in various formats compatible with different Python usage modalities (Pandas, pure-Python, etc).

It adapts the Oracle REST Data Service used by the agency with Python type hints for easy query and interface. Furthermore, Python docstrings annotate the data structures provided by the API to help users navigate the various fields avilable, offering contextual documentation when supported by Python IDEs.

Though not intended to be general, this project also provides an example for working with Oracle REST Data Services (ORDS) APIs in Python.

Installation

This open source library is available for install via Pypi / Pip:

$ pip install afscgap

Note that its only dependency is requests and Pandas / numpy are not expected.

Usage

This library provides access to the public API endpoints with query keywords matching the column names described in the official metadata repository. Records are parsed into plain old Python objects with optional access to a dictionary representation.

Basic Usage

For example, this requests all records of Pasiphaea pacifica in 2021 from the Gulf of Alaska to get the median bottom temperature when they were observed:

import statistics

import afscgap

results = afscgap.query(
    year=2021,
    srvy='GOA',
    scientific_name='Pasiphaea pacifica'
)

temperatures = [record.get_bottom_temperature_c() for record in results]
print(statistics.median(temperatures))

Note that afscgap.query is the main entry point which returns Record objects whose fields and methods are defined in the data structure section.

Using an iterator will have the library negotiate pagination behind the scenes. You can do this with list comprehensions, maps, etc or with a good old for loop like in this example which gets a histogram of temperatures:

count_by_temperature_c = {}

results = afscgap.query(
    year=2021,
    srvy='GOA',
    scientific_name='Pasiphaea pacifica'
)

for record in results:
    temp = record.get_bottom_temperature_c()
    temp_rounded = round(temp)
    count = count_by_temperature_c.get(temp_rounded, 0) + 1
    count_by_temperature_c[temp_rounded] = count

print(count_by_temperature_c)

Note that this operation will cause multiple HTTP requests while the iterator runs.

Pagination

By default, the library will iterate through all results and handle pagination behind the scenes. However, one can also request an individual page:

results = afscgap.query(
    year=2021,
    srvy='GOA',
    scientific_name='Pasiphaea pacifica'
)

results_for_page = results.get_page(offset=20, limit=100)
print(len(results_for_page))  # Will print 32 (results contains 52 records)

Client code can also change the pagination behavior used when iterating:

results = afscgap.query(
    year=2021,
    srvy='GOA',
    scientific_name='Pasiphaea pacifica',
    start_offset=10,
    limit=200
)

for record in results:
    print(record.get_bottom_temperature_c())

Note that records are only requested once during iteration and only after the prior page has been returned via the iterator ("lazy" loading).

Serialization

Users may request a dictionary representation:

results = afscgap.query(
    year=2021,
    srvy='GOA',
    scientific_name='Pasiphaea pacifica'
)

# Get dictionary from individual record
for record in results:
    dict_representation = record.to_dict()
    print(dict_representation['bottom_temperature_c'])

results = afscgap.query(
    year=2021,
    srvy='GOA',
    scientific_name='Pasiphaea pacifica'
)

# Get dictionary for all records
results_dicts = results.to_dicts()

for record in results_dicts:
    print(record['bottom_temperature_c'])

Note to_dicts returns an iterator by default, but it can be realized as a full list using the list() command.

Pandas

The dictionary form of the data can be used to create a Pandas dataframe:

import pandas

import afscgap

results = afscgap.query(
    year=2021,
    srvy='GOA',
    scientific_name='Pasiphaea pacifica'
)

pandas.DataFrame(results.to_dicts())

Note that Pandas is not required to use this library.

Advanced Filtering

Finally, users may provide advanced queries using Oracle's REST API query parameters. For example, this queries for 2021 records with haul from the Gulf of Alaska in a specific geographic area:

import afscgap

results = afscgap.query(
    year=2021,
    latitude_dd={'$between': [56, 57]},
    longitude_dd={'$between': [-161, -160]}
)

count_by_common_name = {}

for record in results:
    common_name = record.get_common_name()
    count = count_by_common_name.get(common_name, 0) + 1
    count_by_common_name[common_name] = count

For more info about the options available, consider the Oracle docs or a helpful unaffiliated getting started tutorial from Jeff Smith.

Incomplete or invalid records

Metadata fields such as year are always required to make a Record whereas others such as catch weight cpue_kgkm2 are not present on all records returned by the API and are optional. See the Schema section below for additional details. For fields with optional values:

• A maybe getter (get_cpue_kgkm2_maybe) is provided which will return None without error if the value is not provided or could not be parsed.
• A regular getter (get_cpue_kgkm2) is provided which asserts the value is not None before it is returned.

Record objects also have an is_complete method which returns true if both all optional fields on the Record are non-None and the date_time field on the Record is a valid ISO 8601 string. By default, records for which is_complete are false are returned when iterating or through get_page but this can be overridden by with the filter_incomplete keyword argument like so:

results = afscgap.query(
    year=2021,
    srvy='GOA',
    scientific_name='Pasiphaea pacifica',
    filter_incomplete=True
)

for result in results:
    assert result.is_complete()

Results returned by the API for which non-Optional fields could not be parsed (like missing year) are considered "invalid" and always excluded during iteration when those raw unreadable records are kept in a queue.Queue[dict] that can be accessed via get_invalid like so:

results = afscgap.query(year=2021, srvy='GOA')
valid = list(results)

invalid_queue = results.get_invalid()
percent_invalid = invalid_queue.qsize() / len(valid) * 100
print('Percent invalid: %%%.2f' % percent_invalid)

complete = filter(lambda x: x.is_complete(), valid)
num_complete = sum(map(lambda x: 1, complete))
percent_complete = num_complete / len(valid) * 100
print('Percent complete: %%%.2f' % percent_complete)

Note that this queue is filled during iteration (like for result in results or list(results)) and not get_page whose invalid record handling behavior can be specified via the ignore_invalid keyword.

Debugging

For investigating issues or evaluating the underlying operations, you can also request a full URL for a query:

results = afscgap.query(
    year=2021,
    latitude_dd={'$between': [56, 57]},
    longitude_dd={'$between': [-161, -160]}
)

print(results.get_page_url(limit=10, offset=0))

The query can be executed by making an HTTP GET request at the provided location.

Data structure

The schema drives the getters and filters available on in the library.

Schema

A Python-typed description of the fields is provided below.

For more information on the schema, see the metadata repository but note that the fields may be slightly different in the Python library per what is actually returned by the API.

Filters and getters

These fields are available as getters on afscgap.model.Record (result.get_srvy()) and may be used as optional filters on the query asfcgagp.query(srvy='GOA'). Fields which are Optional have two getters. First, the "regular" getter (result.get_count()) will assert that the field is not None before returning a non-optional. The second "maybe" getter (result.get_count_maybe()) will return None if the value was not provided or could not be parsed.

Record objects also have a is_complete method which returns true if all the fields with an Optional type are non-None and the date_time could be parsed and made into an ISO 8601 string.

License

We are happy to make this library available under the BSD 3-Clause license. See LICENSE for more details. (c) 2023 The Eric and Wendy Schmidt Center for Data Science and the Environment at UC Berkeley.

Community

Thanks for your support! Pull requests and issues very welcome.

Contribution guidelines

We invite contributions via our project Github. We have a few guidelines:

• Please follow the Google Python Style Guide where possible for compatibility with the existing codebase.
• Tests are encouraged and we aim for 80% coverage where feasible.
• Type hints are encouraged and we aim for 80% coverage where feasible.
• Docstrings are encouraged and we aim for 80% coverage.
• Please check that you have no mypy errors when contributing.
• Please check that you have no linting (pycodestyle, pyflakes) errors when contributing.
• As contributors may be periodic, please do not re-write history / squash commits for ease of fast forward.
• Open source is an act of love. Please be kind and respectful of all contributors at all levels.

Note that imports should be in alphabetical order in groups of standard library, third-party, and then first party. It is an explicit goal to provide a class with type hints for all record fields. Getters on an immutable record object are encouraged as to enable use of the type system and docstrings for understanding the data structures. Data structures have been used that could allow for threaded request but everything is currently single threaded.

Contacts

Sam Pottinger is the primary contact. Thanks to Giulia Zarpellon and Carl Boettiger for their contributions. This is a project of the The Eric and Wendy Schmidt Center for Data Science and the Environment at UC Berkeley. Please contact us via dse@berkeley.edu.

Open Source

We are happy to be part of the open source community.

At this time, the only open source dependency used by this microlibrary is Requests which is available under the Apache v2 License from Kenneth Reitz and other contributors.

Our build and documentation systems also use the following but are not distributed with or linked to the project itself:

• mypy under the MIT License from Jukka Lehtosalo, Dropbox, and other contributors.
• nose2 under a BSD license from Jason Pellerin and other contributors.
• pdoc under the Unlicense license from Andrew Gallant and Maximilian Hils.
• pycodestyle under the Expat License from Johann C. Rocholl, Florent Xicluna, and Ian Lee.
• pyflakes under the MIT License from Divmod, Florent Xicluna, and other contributors.

Thank you to all of these projects for their contribution.

Version history

Major updates:

• 0.0.3: Minor updates in documentation.
• 0.0.2: Re-license under BSD.
• 0.0.1: Initial release.