GithubHelp home page GithubHelp logo

testing.mysqld's Introduction

testing.mysqld automatically setups a mysqld instance in a temporary directory, and destroys it after testing

https://travis-ci.org/tk0miya/testing.mysqld.svg?branch=master https://coveralls.io/repos/tk0miya/testing.mysqld/badge.png?branch=master

Install

Use easy_install (or pip):

$ easy_install testing.mysqld

And testing.mysqld requires MySQL server in your PATH.

Usage

Create MySQL instance using testing.mysqld.Mysqld:

import testing.mysqld
from sqlalchemy import create_engine

# Lanuch new MySQL server
with testing.mysqld.Mysqld() as mysqld:
    # connect to MySQL
    engine = create_engine(mysqld.url())

    # if you use mysqldb or other drivers:
    #   import _mysql
    #   db = _mysql.connect(**mysqld.dsn())

    #
    # do any tests using MySQL...
    #

# MySQL server is terminated here

testing.mysqld.Mysqld executes mysql_install_db and mysqld on instantiation. On deleting Mysqld object, it terminates MySQL instance and removes temporary directory.

If you want a database including tables and any fixtures for your apps, use copy_data_from keyword:

# uses a copy of specified data directory of MySQL.
mysqld = testing.mysqld.Mysqld(copy_data_from='/path/to/your/database')

You can specify parameters for MySQL with my_cnf keyword:

# boot MySQL server without socket listener (use unix-domain socket)
mysqld = testing.mysqld.Mysqld(my_cnf={'skip-networking': None})

For example, you can setup new MySQL server for each testcases on setUp() method:

import unittest
import testing.mysqld

class MyTestCase(unittest.TestCase):
    def setUp(self):
        self.mysqld = testing.mysqld.Mysqld(my_cnf={'skip-networking': None})

    def tearDown(self):
        self.mysqld.stop()

To make your tests faster

testing.mysqld.Mysqld invokes initdb command on every instantiation. That is very simple. But, in many cases, it is very waste that generating brandnew database for each testcase.

To optimize the behavior, use testing.mysqld.MysqldFactory. The factory class is able to cache the generated database beyond the testcases, and it reduces the number of invocation of mysql_install_db command:

import unittest
import testing.mysqld

# Generate Mysqld class which shares the generated database
Mysqld = testing.mysqld.MysqldFactory(cache_initialized_db=True)


def tearDownModule(self):
    # clear cached database at end of tests
    Mysqld.clear_cache()


class MyTestCase(unittest.TestCase):
    def setUp(self):
        # Use the generated Mysqld class instead of testing.mysqld.Mysqld
        self.mysqld = Mysqld()

    def tearDown(self):
        self.mysqld.stop()

If you want to insert fixtures to the cached database, use initdb_handler option:

# create initial data on create as fixtures into the database
def handler(mysqld):
    conn = psycopg2.connect(**mysqld.dsn())
    cursor = conn.cursor()
    cursor.execute("CREATE TABLE hello(id int, value varchar(256))")
    cursor.execute("INSERT INTO hello values(1, 'hello'), (2, 'ciao')")
    cursor.close()
    conn.commit()
    conn.close()

# Use `handler()` on initialize database
Mysqld = testing.mysqld.MysqldFactory(cache_initialized_db=True,
                                      on_initialized=handler)

Requirements

  • Python 2.7, 3.3, 3.4, 3.5
  • pymysql

License

Apache License 2.0

History

1.4.0 (2016-08-20)

  • Drop py26, py32 support
  • Allow user and password argument to connect authorized database
  • Depend on testing.common.database >= 2.0.0

1.3.0 (2016-02-03)

  • Add timeout to server invoker
  • Support MySQL-5.7
  • Add testing.mysqld.MysqldFactory
  • Depend on testing.common.database package
  • Assign port if networking not disabled

1.2.8 (2015-04-06)

  • Fix bugs

1.2.7 (2014-12-20)

  • Support for relative mysql_install_db links
  • Use absolute path for which command

1.2.6 (2014-06-19)

  • Add timeout on terminating mysqld
  • Fix bugs

1.2.5 (2014-06-11)

  • Fix ImportError if caught SIGINT on py3

1.2.4 (2014-02-13)

  • Fix testing.mysqld.Mysqld#start() fails if mysql_install_db does not create database named "test"

1.2.3 (2013-12-11)

  • Use pymysql driver as default in Mysqld#url()

1.2.2 (2013-12-06)

  • Change behavior: Mysqld#stop() cleans workdir
  • Fix caught AttributeError on object deletion

1.2.1 (2013-12-05)

  • Add mysqld.skipIfNotInstalled decorator (alias of skipIfNotFound)
  • Suport python 2.6 and 3.2

1.2.0 (2013-12-04)

  • Add @skipIfNotFound decorator

1.1.2 (2013-11-26)

  • Fix it does not cleanup temporary directory if Mysqld object has been deleted

1.1.1 (2013-11-25)

  • Add charset parameter to Mysqld#url()

1.1.0 (2013-11-22)

  • Rename package: test.mysqld -> testing.mysqld
  • Add Mysqld#url() method (for sqlalchemy)

1.0.0 (2013-10-17)

  • First release

testing.mysqld's People

Contributors

chriso avatar kovacsbalu avatar tk0miya avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

testing.mysqld's Issues

command not found: mysql_install_db

I use mysql v8.0 and use the latest tesing.mysqld but failed for "command not found: mysql_install_db".

I found mysql v8.0 has deprecated mysql_install_db command. Is this repository still being updated?

testing.mysql.MysqldFactory cannot instantiate more than one instance if using my_cnf

>>> import testing.mysqld
>>> f = testing.mysqld.MysqldFactory(cache_initialized_db=True, my_cnf={'local--infile': None})
>>> d = f()
>>> d.stop()
>>> d = f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../penv/lib/python2.7/site--packages/testing/common/database.py", line 54, in __call__
    return self.target_class(**self.settings)
  File ".../penv/lib/python2.7/site--packages/testing/common/database.py", line 94, in __init__
    self.start()
  File ".../penv/lib/python2.7/site--packages/testing/common/database.py", line 150, in start
    self.wait_booting()
  File ".../penv/lib/python2.7/site--packages/testing/common/database.py", line 167, in wait_booting
    self.read_bootlog())
RuntimeError: *** failed to launch Mysqld ***
180129 12:22:40 [Warning] One can only use the ---user switch if running as root

/usr/libexec/mysqld: Can't create/write to file '/tmp/tmpY5hH_x/tmp/ibrjY1lf' (Errcode: 2)
180129 12:22:40  InnoDB: Error: unable to create temporary file; errno: 2
180129 12:22:40 [ERROR] Plugin 'InnoDB' init function returned error.
180129 12:22:40 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
180129 12:22:40 [ERROR] Can't start server : Bind on unix socket: No such file or directory
180129 12:22:40 [ERROR] Do you already have another mysqld server running on socket: /tmp/tmpY5hH_x/tmp/mysql.sock ?
180129 12:22:40 [ERROR] Aborting

180129 12:22:40 [Note] /usr/libexec/mysqld: Shutdown complete

Server terminates unexpectedly in Python 3

I am using testing.mysqld in a database library test suite. It works really great (thanks for it!), but in Python 3 (not Python 2), the server terminates unexpectedly at the tearDown point. It also happens for the testing.postgresql module, so it is not strictly a MySQL issue, but I had to choose some place top file it.

You can find the test suite here: https://github.com/Veripeditus/OSMAlchemy/blob/master/test/test_util_db.py

In Python 2, everthing works well, but in Python 3, both mysqld and postgresql terminate unexpectedly somewhere near the end:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy
    fairy._reset(pool)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line 774, in _reset
    self._reset_agent.rollback()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1563, in rollback
    self._do_rollback()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1601, in _do_rollback
    self.connection._rollback_impl()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 670, in _rollback_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 185, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 668, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback
    dbapi_connection.rollback()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 772, in rollback
    self._read_ok_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 746, in _read_ok_packet
    pkt = self._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 961, in _read_packet
    packet_header = self._read_bytes(4)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 998, in _read_bytes
    2013, "Lost connection to MySQL server during query")
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
ERROR

What I should probably add is that the data in the test database is massive - however, it works in Python 2, as I said.

Following quickstart causes noisy errors on setUp

First of all, thank you for wonderful work.

Following the quickstart at https://pypi.org/project/testing.mysqld/, I would get this very noisy error message:

.{base.py:680} ERROR - Exception during reset or similar
Traceback (most recent call last):
  File "/Users/romanhartmann/.virtualenvs/ub-data-product/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 671, in _finalize_fairy
    fairy._reset(pool)
  File "/Users/romanhartmann/.virtualenvs/ub-data-product/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 858, in _reset
    pool._dialect.do_rollback(self)
  File "/Users/romanhartmann/.virtualenvs/ub-data-product/lib/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2227, in do_rollback
    dbapi_connection.rollback()
  File "/Users/romanhartmann/.virtualenvs/ub-data-product/lib/python3.6/site-packages/pymysql/connections.py", line 430, in rollback
    self._read_ok_packet()
  File "/Users/romanhartmann/.virtualenvs/ub-data-product/lib/python3.6/site-packages/pymysql/connections.py", line 394, in _read_ok_packet
    pkt = self._read_packet()
  File "/Users/romanhartmann/.virtualenvs/ub-data-product/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet
    packet_header = self._read_bytes(4)
  File "/Users/romanhartmann/.virtualenvs/ub-data-product/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

My unit test setup was like this:

class TestWhatever(unittest.TestCase):

    def setUp(self):
        self.mysql = testing.mysqld.Mysqld(port=7531)
        self.target_conn = create_engine(self.mysql.url()).connect()
        self.target_conn.execute("""CREATE TABLE `foo` (blah)""")

    def tearDown(self):
        self.target_conn.execute("DROP TABLE foo")
        self.mysql.stop()

    def test_something(self):
        # something useful

This error did not cause any tests to fail, it was just being logged all the time.
I was using the exact same setup for testing.postgresql without the error.

I changed my test class to this to get rid of the error:

MYSQLD_FACTORY = testing.mysqld.MysqldFactory(cache_initialized_db=True, port=7531)


def tearDownModule():
    """Tear down databases after test script has run.

    https://docs.python.org/3/library/unittest.html#setupclass-and-teardownclass
    """
    MYSQLD_FACTORY.clear_cache()


class TestWhatever(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        cls.mysql = MYSQLD_FACTORY()
        cls.target_conn = create_engine(cls.mysql.url()).connect()

    def setUp(self):
        self.mysql.start()
        self.target_conn.execute("""CREATE TABLE `foo` (blah)""")

    def tearDown(self):
        self.target_conn.execute("DROP TABLE foo")

    @classmethod
    def tearDownClass(cls):
        cls.mysql.stop()

    def test_something(self):
        # something useful

Would you please update your documentation to either

  • Outline what I did wrong to prevent other people having to figure this out. I've seen a lot of SO posts about the vague pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') error in relation to this package
  • Document this pattern as a working pattern

Support MariaDB

With MariaDB 10 as mysqld, I only get pymysql.err.InternalError: (1698, "Access denied for user 'root'@'localhost'") with the default config.

I imagine the fix might be trivial.

Invoking stop() on factory instantiated test database shuts down mysqld instance

>>> import testing.mysqld
>>> f = testing.mysqld.MysqldFactory(cache_initialized_db=True)
>>> d = f()
>>> d.child_process.pid
5984
>>> 
[1]+  Stopped                 python
$ ps -eaf | grep 5984 | grep -v grep | sed "s/\<${USER}\>/anon/g"
anon     5984  5551  0 12:26 pts/3    00:00:00 /usr/libexec/mysqld --defaults-file=/tmp/tmpSzRaO5/etc/my.cnf --user=root
$ fg
python
d.stop()
>>> 
[1]+  Stopped                 python
$ ps -eaf | grep 5984 | grep -v grep | sed "s/\<${USER}\>/anon/g"
$ ps -eaf | grep mysqld | grep -v grep | sed "s/\<${USER}\>/anon/g"
$

This makes testing as slow as not using the cached instance, which defeats the purpose.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.