Creates and drops temporary databases for testing purposes.
Project description
Creates and drops temporary databases for testing purposes.
This package has been tested with Python 3.8, 3.9, 3.10 and 3.11.
Copyright (c) 2008-2015, 2020-2021 gocept gmbh & co. kg and contributors.
All Rights Reserved.
This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED “AS IS” AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE.
gocept.testdb provides small helper classes that create and drop temporary databases.
>>> import gocept.testdb
>>> import os.path
>>> import shutil
>>> import sqlalchemy
>>> import tempfile
>>> sql_dir = tempfile.mkdtemp()
>>> schema = os.path.join(sql_dir, 'sample.sql')
>>> with open(schema, 'w') as f:
... _ = f.write('CREATE TABLE foo (dummy int);')
We’ll use a custom prefix specific to the current process whenever we create fixed-name databases during this test, in order to allow concurrent test runs on the same machine (such as a CI server):
>>> import os >>> pid_prefix = 'gocept.testdb.tests-PID%s-' % os.getpid() >>> db_template = pid_prefix + 'templatetest'
MySQL
gocept.testdb expects the usage of PyMySQL as database driver.
First, instantiate a test database object and have it create the database on the server. You can specify the name or the prefix of the database:
>>> gocept.testdb.MySQL(schema_path=schema, db_name='mytestdb').dsn 'mysql+pymysql://.../mytestdb' >>> db = gocept.testdb.MySQL(schema_path=schema, prefix='my-tests') >>> db.dsn 'mysql+pymysql://.../my-tests-...' >>> db.create()
This will use the appropriate command-line tools to create a database with a random name.
You can use the following environment variables to customize the DSN:
- MYSQL_HOST
hostname, defaults to localhost
- MYSQL_PORT
port of the database, defaults to use a socket when not set
- MYSQL_USER
username, defaults to None which means to use the name of the user logged into the operating system.
- MYSQL_PASS
password, defaults to None which means no password required.
- MYSQL_COMMAND_POSTFIX
attach this postfix to MySQL commands, defaults to an empty string. You need this variable if your MySQL commands are named like mysql5 instead of mysql.
The dbapi DSN can then be used to connect to the database:
>>> engine = sqlalchemy.create_engine(db.dsn)
The database is marked as a testing database by creating a table called tmp_functest in it:
>>> with engine.connect() as conn:
... with conn.begin():
... ignore = conn.execute(sqlalchemy.text('SELECT * from tmp_functest'))
The database object also offers convenience methods for determining the status of the database:
>>> db.exists True >>> db.is_testing True
If you passed a schema_path to the constructor, the SQL code in this file is executed, e. g. to set up tables:
>>> with engine.connect() as conn:
... with conn.begin():
... ignore = conn.execute(sqlalchemy.text('SELECT * from foo'))
When done, simply drop the database:
>>> engine.dispose() >>> db.drop()
PostgreSQL
General
The same procedure also works for PostgreSQL. You can use the following environment variables to customize the DSN:
- POSTGRES_HOST
hostname, defaults to localhost
- POSTGRES_PORT
port of the database server, not used when not set, aka 5432
- POSTGRES_USER
username, defaults to None which means to use the name of the user logged into the operating system.
- POSTGRES_PASS
password, defaults to None which means no password required. Note: Instead of using POSTGRES_PASS, use the ~/.pgpass mechanism provided by postgres itself.
>>> db = gocept.testdb.PostgreSQL(schema_path=schema)
>>> db.create()
>>> engine = sqlalchemy.create_engine(db.dsn)
>>> conn = engine.connect()
>>> ignore = conn.execute(sqlalchemy.text('SELECT * from tmp_functest'))
>>> ignore = conn.execute(sqlalchemy.text('SELECT * from foo'))
>>> conn.invalidate()
>>> db.drop()
Encoding
For PostgreSQL an optional encoding, database name and database name prefix parameters can be specified in the constructor. They are used when creating the database.
>>> gocept.testdb.PostgreSQL(encoding='UTF8', db_name='mytestdb').dsn 'postgresql://...localhost.../mytestdb' >>> gocept.testdb.PostgreSQL(prefix='my-tests').dsn 'postgresql://...localhost.../my-tests-...'
Templates
For PostgreSQL, an optional template parameter can be passed to the constructor. It specifies the name of a template database which is used for the creation of the test database. If the template database does not exist, it is created with the specified schema.
The first time you create the database with the db_template argument, the template database is created (if it does not exist already) along with the requested database:
>>> db = gocept.testdb.PostgreSQL(schema_path=schema, db_template=db_template)
Now with the template available, the schema is not used any more to create the database (it’s copied from the template database).
When creating the database, we can, however, force the template database to be created afresh from the schema. Doing so now will leave us with both a test database and a template database according to the modified schema:
>>> db = gocept.testdb.PostgreSQL( ... schema_path=schema, db_template=db_template, force_template=True)
The template database (and with it, the test database) is also created anew if the schema file is newer than the existing template database.
If, however, the template database cannot be set up properly, it is removed altogether to avoid a broken template database interfering with subsequent tests.
The drop-all command-line script
The Database classes’ drop_all functionality is available independently through a command-line script named drop-all. The script drops any test databases from both the PostgreSQL and MySQL servers that match the test-database naming convention with any of the prefixes passed as command-line arguments. Usage:
$ bin/drop-all "<prefix>"
Test clean up:
>>> shutil.rmtree(sql_dir)
Development
To run the buildout of this package copy local.cfg.example to local.cfg and edit it to match your needs:
MYSQL_COMMAND_POSTFIX is needed if your MySQL commands look like mysql5 instead of mysql
MySQL has to open a port for the tests to connect to. Configure this in your my.cnf.
Running tests
Install tox and run the tests calling tox.
Using docker containers for the tests
Prerequisites
mysqladmin and mysql have to be on $PATH.
createdb has to be on $PATH.
Run the follwing commands (you might change the passwords used):
docker run --name mysql56 -e MYSQL_ROOT_PASSWORD=84_L224JF0GlTXcL -d -p 3307:3306 mysql:5.6 docker run --name postgres96 -e POSTGRES_PASSWORD=j§V7iJY@1xTG67J@ -d -p 5433:5432 postgres:9.6 echo "localhost:5433:*:postgres:j§V7iJY@1xTG67J@" >> ~/.pgpass chmod 600 ~/.pgpass
Run the tests
MYSQL_PORT=3307 MYSQL_USER=root MYSQL_PASS=84_L224JF0GlTXcL POSTGRES_PORT=5433 POSTGRES_USER=postgres tox
Source code
The source code is available at https://github.com/gocept/gocept.testdb
Bug reports
Please report any bugs you find to https://github.com/gocept/gocept.testdb/issues
Changelog
6.0 (2023-08-28)
Backwards incompatible changes
Drop support for Python 2.7, 3.6.
Drop support for SQLAlchemy < 1.2.
Features
Add support for Python 3.10, 3.11.
Add support for SQLAlchemy 2.x.
5.2.1 (2023-03-16)
Restrict supported SQLALchemy to < 2.
5.2 (2021-04-26)
Ensure compatibility with pytest > 6.0.
Add support for Python 3.9.
Switch CI to GHA.
Fix PostgreSQL drop_db to be able to forcibly drop a data base with open connections, even though there is no database with the same name as the user.
5.1 (2020-07-06)
Forcibly close connections to test database before dropping, so this does not have to be done beforehand. (Ported from 4.x but missing in change log there.)
Calculate a random database name that is random enough for parallel execution. (Ported from 4.2)
5.0 (2020-07-02)
Backwards incompatible changes
Caution: This release is based on version 1.3 ignoring the changes made in the 4.x releases! At least some of those changes will be added later on.
Drop support for Python 2.6 and Python 3.3 up to 3.5.
Remove module gocept.testdb.db containing only imports for backwards compatibility, import now from gocept.testdb.mysql resp. gocept.testdb.postgres.
Features
Officially support Python 3.6 up to 3.8.
Add support to specify the port to connect to the database, thus the environment variables MYSQL_PORT and POSTGRES_PORT are taken into account. This eases using docker containers on custom ports.
Other changes
Migrate code to Github.
1.3 (2015-10-07)
Drop support of Python 3.2.
Streamline documentation.
1.3a1 (2015-09-24)
Officially support Python 3.2 up to Python 3.4.
Switch PyMySQL driver to support Python 3 for MySQL.
Add environment variable MYSQL_COMMAND_POSTFIX to use MySQL commands like mysql5 instead of mysql.
1.2.1 (2013-09-03)
Improve retry logic for dropping databases: ensure that db actually exists before trying to drop it (#12706).
1.2 (2013-06-21)
Provisional compatibility to Python 3 (no compatible mysql driver exists yet, though).
Fixed test code that made implicit assumptions about existing databases on the PostgreSQL server used or depended on timing conditions.
1.1.1 (2013-03-19)
Use template0 when changing LC_COLLATE which was introduced in v1.1.
1.1 (2013-03-19)
Add possibility to set the LC_COLLATE when creating a database.
1.0 (2013-03-12)
Allow a postfix for mysql-commands.
1.0b5 (2011-12-22)
Use timestamp for randomizing database names to avoid collisions.
1.0b4 (2011-09-23)
drop-all entry point now also works, if PostgreSQL or MySQL is not installed/running.
1.0b3 (2011-05-13)
Added connect convenience method.
1.0b2 (2011-05-04)
Added is_testing property as a convenience API.
Added exists property as a convenience API.
1.0b1 (2011-04-12)
Changed the protocol for using test databases: separated instantiating a Database instance from creating the database on the server. Creating Database instances is cheap now so they can be interacted with and passed around, deferring the expensive database creation on the server to the moment db.create() is called explicitly. This is also more symmetric with respect to db.drop().
Added functionality for dropping all databases matching the testdb naming scheme which may have been left on the server by earlier test runs. Optionally also drops the template db.
Added option to specify the name of the database for MySQL and PostgreSQL.
Added an option to specify a template database for PostgreSQL. If it does not exist, it is created from the specified schema. It is also possible to force the creation of the template even if it exists (dropping the current template database). Whenever the schema has changed since the template db was last created, the template db is updated automatically.
0.4 (2010-12-15)
Added option to specify the encoding for the PostgreSQL database.
Updated PostgreSQL protocol from postgres: to postgresql: as the former one is deprecated in SQLAlchemy 0.6, thus requiring atleast version 0.5.6 of SQLAlchemy.
Added documentation how to develop this package further.
Added doumentation about usage of Database prefix.
0.3 (2010-10-15)
PostgreSQL: Don’t call createdb/dropdb with --quite but only psql.
0.2 (2009-02-26)
implemented authentication with password for mysql. Passwords for postgres are still not supported, though.
0.1 (2008-09-26)
first release
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for gocept.testdb-6.0-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 5b0b477e72bcc73e7acebf6ed2a3ff05ce1d352f98f2e071da1df014691a4b4e |
|
| MD5 | 438fada438108e45c286932a9e459819 |
|
| BLAKE2b-256 | 4d2719fc4310007fd80b151da71d68214a76281aa435a6a8b3a899b75557d109 |
