Skip to main content

Django migrations CI optimization

Project description

django-migrations-ci

Optimizations to run less migrations on CI.

Django migrations are slow because of state recreation for every migration and other internal Django magic.

In the past, I tried to optimize that on Django core, but learnt it's a running issue.

Assumptions

  1. I want to run my migrations on CI. It is a sanity check, even if they are generated by Django in some cases.
  2. I don't have to run migrations all the time. If migrations didn't change, I can reuse them.
  3. The solutions probably are CI-specific and database-specific, but it is possible to write code for a generic workflow and snippets to support each CI.

Install

Install the package with pip:

pip install django-migrations-ci

Add django_migrations_ci to Django settings INSTALLED_APPS.

INSTALLED_APPS = [
    ...,  # other packages
    "django_migrations_ci",
]

How to use

The command migrateci execute all migrations and generate dump files migrateci-* to be cached on CI.

If these files already exist on disk, they are used to prepare the database without running all migrations again.

Configure your CI to cache these migrateci-* files, based on migration files. /

Workflow

This is how the "run test" CI job should work.

Restore migrateci.sql from CI cache

if migrateci.sql exists on cache:
  Restore migrateci.sql to test database
else:
  Setup test database
  Dump test database to migrateci.sql

Clone the test database to run threaded tests

Save migrateci.sql to CI cache

Cache example on GitHub

    steps:
    - uses: actions/cache@v3
      name: Cache migrations
      with:
        path: migrateci-*
        key: ${{ secrets.EXAMPLE_CACHE_PREFIX }}-${{ hashFiles('**/migrations/*.py') }}
    - name: Migrate database
      run: ./manage.py migrateci --parallel $(nproc)
    - name: Test with Django
      run: ./manage.py test --keepdb --parallel $(nproc)

Cache example on GitLab

Still have to abstract psql/pg_dump/pg_restore, but I expect something like that will work:

test_job:
  stage: test
  script:
    - ./manage.py migrateci $(nproc)
    - ./manage.py test --keepdb --parallel $(nproc)
  cache:
    key:
      # GitLab docs say it accepts only two files, but for some reason it works with wildcards too.
      # You can't add more than two lines here.
      files:
        - "requirements.txt"
        - "*/migrations/*.py"
    paths:
      - migrateci-*

How databases for parallel tests are named

Django test framework has a --parallel N flag to test with N parallel processes, naming databases from 1 to N.

  • On sqlite3, a db.sqlite3 generate db_N.sqlite3 files.
  • On PostgreSQL, a db generate test_db_N.

Pytest pytest-django use pytest-xdist for parallel support, naming databases from 0 to N-1.

  • On sqlite3, a db.sqlite3 generate db.sqlite3_gwN files.
  • On PostgreSQL, a db generate test_db_gwN.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django_migrations_ci-0.0.4.tar.gz (10.0 kB view hashes)

Uploaded Source

Built Distribution

django_migrations_ci-0.0.4-py3-none-any.whl (13.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page