Skip to main content

Fast differentiable resizing and warping of arbitrary grids

Project description

Hugues Hoppe    Aug 2022.

[Open in Colab]   [Kaggle]   [MyBinder]   [DeepNote]   [GitHub source]   [API docs]   [PyPI package]

The notebook resampler_notebook.ipynb demonstrates the resampler library and contains documentation, usage examples, unit tests, and experiments.

Overview

The resampler library enables fast differentiable resizing and warping of arbitrary grids. It supports:

  • grids of any dimension (e.g., 1D, 2D images, 3D video, 4D batches of videos), containing

  • samples of any shape (e.g., scalars, colors, motion vectors, Jacobian matrices) and

  • any numeric type (e.g., uint8, float64, complex128)

  • within several array libraries (numpy, tensorflow, torch, and jax);

  • either 'dual' ("half-integer") or 'primal' grid-type for each dimension;

  • many boundary rules, specified per dimension, extensible via subclassing;

  • an extensible set of filter kernels, selectable per dimension;

  • optional gamma transfer functions for correct linear-space filtering;

  • prefiltering for accurate antialiasing when resize downsampling;

  • efficient backpropagation of gradients for tensorflow, torch, and jax;

  • few dependencies (only numpy and scipy) and no C extension code, yet

  • faster resizing than C++ implementations in tf.image and torch.nn.

A key strategy is to leverage existing sparse matrix representations and operations.

Example usage

!pip install -q mediapy numpy resampler

import mediapy as media
import numpy as np
import resampler
array = np.random.default_rng(1).random((4, 6, 3))  # 4x6 RGB image.
upsampled = resampler.resize(array, (128, 192))  # To 128x192 resolution.
media.show_images({'4x6': array, '128x192': upsampled}, height=128)
image = media.read_image('https://github.com/hhoppe/data/raw/main/image.png')
downsampled = resampler.resize(image, (32, 32))
media.show_images({'128x128': image, '32x32': downsampled}, height=128)
import matplotlib.pyplot as plt
array = [3.0, 5.0, 8.0, 7.0]  # 4 source samples in 1D.
new_dual = resampler.resize(array, (32,))  # (default gridtype='dual') 8x resolution.
new_primal = resampler.resize(array, (25,), gridtype='primal')  # 8x resolution.

_, axs = plt.subplots(1, 2, figsize=(7, 1.5))
axs[0].set_title("gridtype='dual'")
axs[0].plot((np.arange(len(array)) + 0.5) / len(array), array, 'o')
axs[0].plot((np.arange(len(new_dual)) + 0.5) / len(new_dual), new_dual, '.')
axs[1].set_title("gridtype='primal'")
axs[1].plot(np.arange(len(array)) / (len(array) - 1), array, 'o')
axs[1].plot(np.arange(len(new_primal)) / (len(new_primal) - 1), new_primal, '.')
plt.show()
batch_size = 4
batch_of_images = media.moving_circle((16, 16), batch_size)
upsampled = resampler.resize(batch_of_images, (batch_size, 64, 64))
media.show_videos({'original': batch_of_images, 'upsampled': upsampled}, fps=1)

original

upsampled

Most examples above use the default resize() settings:

  • gridtype='dual' for both source and destination arrays,
  • boundary='auto' which uses 'reflect' for upsampling and 'clamp' for downsampling,
  • filter='lanczos3' (a Lanczos kernel with radius 3),
  • gamma=None which by default uses the 'power2' transfer function for the uint8 image in the second example,
  • scale=1.0, translate=0.0 (no domain transformation),
  • default precision and output dtype.

Advanced usage:

Map an image to a wider grid using custom scale and translate vectors, with horizontal 'reflect' and vertical 'natural' boundary rules, providing a constant value for the exterior, using different filters (Lanczos and O-MOMS) in the two dimensions, disabling gamma correction, performing computations in double-precision, and returning an output array in single-precision:

new = resampler.resize(
    image, (128, 512), boundary=('natural', 'reflect'), cval=(0.2, 0.7, 0.3),
    filter=('lanczos3', 'omoms5'), gamma='identity', scale=(0.8, 0.25),
    translate=(0.1, 0.35), precision='float64', dtype='float32')
media.show_images({'image': image, 'new': new})

Warp an image by transforming it using polar coordinates:

shape = image.shape[:2]
yx = ((np.indices(shape).T + 0.5) / shape - 0.5).T  # [-0.5, 0.5]^2
radius, angle = np.linalg.norm(yx, axis=0), np.arctan2(*yx)
angle += (0.8 - radius).clip(0, 1) * 2.0 - 0.6
coords = np.dstack((np.sin(angle) * radius, np.cos(angle) * radius)) + 0.5
resampled = resampler.resample(image, coords, boundary='constant')
media.show_images({'image': image, 'resampled': resampled})

Limitations:

  • Filters are assumed to be separable.
  • Although resize implements prefiltering, resample does not yet have it (and therefore may have aliased results if downsampling).
  • Differentiability is only with respect to the grid values, not wrt the resize shape, scale, translation, or the resampling coordinates.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

resampler-0.8.7.tar.gz (48.1 kB view hashes)

Uploaded Source

Built Distribution

resampler-0.8.7-py3-none-any.whl (45.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page