GithubHelp home page GithubHelp logo

esnme / ultramysql Goto Github PK

View Code? Open in Web Editor NEW
546.0 52.0 95.0 404 KB

A fast MySQL driver written in pure C/C++ for Python. Compatible with gevent through monkey patching.

Home Page: http://www.esn.me

License: Other

C++ 30.33% C 47.16% Python 22.51%

ultramysql's Introduction

:: Description ::
A fast MySQL driver written in pure C/C++ for Python. 
Compatible with gevent through monkey patching

:: Requirements ::
Requires Python (http://www.python.org) 

:: Installation ::
python setup.py build install

ultramysql's People

Contributors

4031651 avatar fcicq avatar hongqn avatar leechael avatar leexiaolan avatar marlonyao avatar paulie4 avatar pdh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ultramysql's Issues

Still got "Socket receive buffer full" exception

I have a table which has a blob field, and when i want to query all data from it, i got "Socket receive buffer full".
And when i increase the MYSQL_PACKET_SIZE from (1024 * 1024 * 16) to (1024 * 1024 * 32), it's disappeared.

alldata = list(executeSQL('select * from table_with_big_blob_field', ()).rows)

Why not compatible with PEP 249?

I am very intrest in the greenlet support, but I look into the testcase, found the interface is total different with from now, so It's not very easy to transform from old driver to your driver. I hope you can make it compatible with PEP 249. Then many people would choose this.

Connection obj can't search the right result.

I found some problem when using umysql lib.
i use some db pool to reuse the connection obj.

In case gevent timeout during the connection obj searching data from db,then the connection obj can't get right data after. the testing code below,expecting to your solution.

# -*- coding:utf-8 -*-  
# author:Ewing
# date:2014-2-20

''' sql here
CREATE DATABASE test;
USE test;
CREATE TABLE IF NOT EXISTS `tb_user` (
  `UserId` int(11) NOT NULL,
  `LastLoginTime` int(11) NOT NULL,
  PRIMARY KEY (`UserId`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

INSERT INTO `tb_user` (`UserId`, `LastLoginTime`) VALUES
(1, 1392859212);
'''

from gevent import monkey; monkey.patch_all()
import time
from gevent import Timeout
import umysql
c = umysql.Connection()
c.connect('10.1.1.18', 3306, 'root','think1dfh@RUNpadm', 'test', False, 'utf8')

looptime = 100000 # FOR TEST set  1 or 100000
def doing():
    start = time.time()
    for i in xrange(looptime):
        sql = 'UPDATE tb_user SET LastLoginTime = %s WHERE UserId = 1'%int(time.time())
        c.query(sql,())
    runtime = time.time()-start
    print 'do all runtime: %s'%runtime

def afterdoing():
    sql = 'SELECT * FROM tb_user WHERE UserId = 1'
    result_set = c.query(sql,())
    print 'afterdoing result_set:',result_set  #return (1L, 0L) case timeout then <umysql.ResultSet object> nomal

timeout = Timeout(1)
timeout.start()
try:
    doing()
except Timeout:
    print 'Could not complete'

afterdoing()  
print 'end'

std::bad_alloc on 150 parallel connections

The following code raises std::bad_alloc on 151st iteration (32bit Ubuntu 12.04 with 4Gb RAM). I think 15 is not too many...

tdbs = []
for i in xrange(100000):
        print i
        db = umysql.Connection()
        db.connect(SQL_HOST, SQL_PORT, SQL_USER, SQL_PASSWD, SQL_DB)
        tdbs.append(db)

Python interpreter corruption

The module doesn't seem to play nice with Python:

import umysql
a.b = umysql.Connection()

gives me:

RuntimeError: Exception is set for no error in Connection_Destructor

What I would expect:

NameError: name 'a' is not defined

Even more strangely, there is this phenomenon:

I have also seen exceptions where the innermost entry of the stacktrace is missing.

ultramysql throws not connected exception

ultramysql throws not connected exception

db = umysql.Connection()
db.connect(config.db_host, config.db_port, config.db_user, config.db_password, config.db_database)
...

def get_info():
    rs = db.query(sql)
RuntimeError: Not connected
<Greenlet at 0x1df6870: handle(<socket at 0x1e59390 fileno=[Errno 9] Bad file des, ('127.0.0.1', 64871))> failed with RuntimeError

How to make it KeepAlive forever?

Dict Cursor in ultraMySQL

So here q.rows[0] gives a tuple.

How can I get results from dict cursor as in MySQL.

import umysql
cnn = umysql.Connection()
cnn.connect(DB_HOST, 3306, DB_USER, DB_PASSWD, DB_DB)
cnn.connect(DB_HOST, 3306, DB_USER, DB_PASSWD, DB_DB)
q = cnn.query("select * from test_table")
q.rows[0]

(1,
u'abc',
u'def',
0,
datetime.datetime(2013, 8, 13, 14, 25, 46),
0,
datetime.datetime(2012, 8, 13, 14, 26, 3),
1)

Long queries crashes amysql

Cut-n-paste from other issue:
Tracked down the assertion crash - apparently amysql doesn't like long lines, e.g. those produced by mysqldump. This only happens for amysql built against gevent.
$ wc test.sql
1 5 959312 test.sql

!/usr/bin/env python

import sys, gevent, gevent.queue, amysql, time, logging, random
SQL=file("test.sql").read()

slave = amysql.Connection()
slave.connect("xxx", 3306, "xxx", "xxx", "test", True, 'utf8');
slave.query(SQL)
python: PacketWriter.cpp:93: void PacketWriter::pull(size_t): Assertion `m_writeCursor - m_readCursor <= cbSize' failed.
Aborted

./python/umysql.h:81:3: error: expected specifier-qualifier-list before 'UINT8'

Hi,

I'm unable to build ultramysql. Any idea?

C:\ultramysql>python setup.py build
running build
running build_ext
building 'umysql' extension
C:\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -DWIN32_LEAN_AND_MEAN -I./python -I./lib -IC:\Python27\include -IC:\Python27\PC -c ./python/umysql.c -o build\temp.win32-2.7\Release\.\python\umysql.o
In file included from ./python/umysql.c:60:0:
./python/umysql.h:81:3: error: expected specifier-qualifier-list before 'UINT8'
./python/umysql.c: In function 'API_resultSetField':
./python/umysql.c:150:3: error: 'UMTypeInfo' has no member named 'type'
./python/umysql.c: In function 'DecodeString':
./python/umysql.c:257:13: error: 'UMTypeInfo' has no member named 'charset'
./python/umysql.c:492:103: error: 'UMTypeInfo' has no member named 'charset'
./python/umysql.c: In function 'API_resultRowValue':
./python/umysql.c:511:15: error: 'UMTypeInfo' has no member named 'type'
./python/umysql.c:655:73: error: 'UMTypeInfo' has no member named 'type'
./python/umysql.c: At top level:
./python/umysql.c:679:3: warning: initialization from incompatible pointer type
./python/umysql.c: In function 'EscapeQueryArguments':
./python/umysql.c:1058:5: warning: implicit declaration of function '_alloca'
./python/umysql.c: At top level:
./python/umysql.c:1278:3: warning: initialization from incompatible pointer type
./python/umysql.c:1374:3: warning: initialization from incompatible pointer type
error: command 'gcc' failed with exit status 1

Segmentation fault in uwsgi-1.4+python2.7 using ultramysql

I am using uwsgi in an async ( ugreen) mode, and communicating with mysql using ultramysql, but am getting this trace, mentioning segmentation fault, quite a lot

Mon Nov 24 19:46:55 2014 - !!! uWSGI process 910 got Segmentation Fault !!!
Mon Nov 24 19:46:55 2014 - *** backtrace of 910 ***
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_backtrace+0x25) [0x4401c5]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_segfault+0x21) [0x4402a1]
Mon Nov 24 19:46:55 2014 - /lib64/libc.so.6() [0x3ae0a329a0]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x3a) [0x7f425f2d5dba]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x43) [0x7f425f37d593]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(python_call+0x25) [0x44d205]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_gevent_current_wsgi_req+0x2a) [0x45d6ba]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(warn_pipe+0xa) [0x43cc0a]
Mon Nov 24 19:46:55 2014 - /lib64/libc.so.6() [0x3ae0a329a0]
Mon Nov 24 19:46:55 2014 - /lib64/libpthread.so.0(send+0x6c) [0x3ae0e0ecdc]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/lib-dynload/_socket.so(+0x6df2) [0x7f4252b29df2]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5910) [0x7f425f383da0]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x75c21) [0x7f425f304c21]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x593df) [0x7f425f2e83df]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_CallMethodObjArgs+0x1b2) [0x7f425f2da212]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(API_sendSocket+0x50) [0x7f424f3c4730]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(_ZN10Connection11writeSocketEv+0x59) [0x7f424f3c50e9]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(_ZN10Connection10sendPacketEv+0x18) [0x7f424f3c5158]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(_ZN10Connection5queryEPKcm+0x92) [0x7f424f3c57a2]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(Connection_query+0x12d) [0x7f424f3c3bcd]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5910) [0x7f425f383da0]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x75d28) [0x7f425f304d28]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x412a) [0x7f425f3825ba]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x563a) [0x7f425f383aca]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x638e) [0x7f425f38481e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x638e) [0x7f425f38481e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x563a) [0x7f425f383aca]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x563a) [0x7f425f383aca]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xbc1) [0x7f425f37f051]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xbc1) [0x7f425f37f051]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xbc1) [0x7f425f37f051]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xbc1) [0x7f425f37f051]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyIter_Next+0xb) [0x7f425f2d50db]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_response_subhandler_wsgi+0x5a) [0x45066a]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_request_wsgi+0x3fd) [0x44f91d]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(py_uwsgi_gevent_request+0x92) [0x45c212]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x58ec) [0x7f425f383d7c]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x75c21) [0x7f425f304c21]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x593df) [0x7f425f2e83df]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x43) [0x7f425f37d593]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/greenlet.so(+0x31d5) [0x7f424f9791d5]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/greenlet.so(+0x28cb) [0x7f424f9788cb]
Mon Nov 24 19:46:55 2014 - [0x156f250]
Mon Nov 24 19:46:55 2014 - *** end of backtrace ***

Any clues as to what can be going wrong here??

Non existing db when connecting gives cryptic error

When connecting to a db and specifying a database that does not exist you get the error "SystemError: error return without exception set". Setting the db arg to a empty string is a work around.

Minimal example:

import unittest, umysql

class TestMySQL(unittest.TestCase):
def testUnique(self):
conn = umysql.Connection()
conn.connect ('127.0.0.1', 3306, 'testuser', 'testuser', 'NO_EXISTING_DB')

    conn.close()

if name == 'main':
unittest.main()

umysql can hang forever

I've found the frequently when starting a large number of simultaneous jobs that all hit a MySQL server at once, sometimes umysql will hang indefinitely when trying to connect. Stacktrace is:

(gdb) bt
#0 0x00007f64612d14cc in recv () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004fb139 in ?? ()
#2 0x0000000000510b4b in ?? ()
#3 0x00000000004cbb0a in PyObject_CallMethodObjArgs ()
#4 0x00007f645e2ee69c in API_recvSocket (sock=0x4f03d00, buffer=0x7f644774b010 "", cbBuffer=cbBuffer@entry=65536) at ./python/io_cpython.c:229
#5 0x00007f645e2ef1c0 in Connection::readSocket (this=this@entry=0x5301720) at ./lib/Connection.cpp:153
#6 0x00007f645e2ef264 in Connection::recvPacket (this=this@entry=0x5301720) at ./lib/Connection.cpp:354
#7 0x00007f645e2efd84 in Connection::connect (this=0x5301720, _host=, _port=, _username=,

_password=<optimized out>, _database=<optimized out>, _autoCommit=0x0, _charset=MCS_utf8_general_ci) at ./lib/Connection.cpp:487

#8 0x00007f645e2ee8aa in UMConnection_Connect (conn=, _host=, _port=, _username=,

_password=<optimized out>, _database=<optimized out>, _autoCommit=0x0, _charset=33) at ./lib/capi.cpp:84

#9 0x00007f645e2ed74e in Connection_connect (self=0x510fcd8, args=) at ./python/umysql.c:860
#10 0x00000000004ac5ce in PyEval_EvalFrameEx ()
#11 0x00000000004b3fd8 in PyEval_EvalCodeEx ()
#12 0x00000000004b4b4c in ?? ()
#13 0x0000000000481cc4 in ?? ()
#14 0x00000000004613b4 in ?? ()
#15 0x0000000000463cc2 in ?? ()
#16 0x00000000004acc66 in PyEval_EvalFrameEx ()
#17 0x00000000004acde0 in PyEval_EvalFrameEx ()
#18 0x00000000004b3fd8 in PyEval_EvalCodeEx ()
#19 0x00000000004acb98 in PyEval_EvalFrameEx ()
#20 0x00000000004b3fd8 in PyEval_EvalCodeEx ()
#21 0x0000000000536723 in ?? ()
#22 0x0000000000446bf2 in PyRun_FileExFlags ()
#23 0x00000000004470ec in PyRun_SimpleFileExFlags ()
#24 0x0000000000447cdc in Py_Main ()
#25 0x00007f64606b6ead in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#26 0x00000000004c7f39 in _start ()

Connection reset by peer when receiving

cnn.connect ('a internal ip address which running mysql (some old version)', 3306,'user','pwd', "bugzilla")
cnn.query("show tables")
Traceback (most recent call last):
File "", line 1, in
umysql.Error: (0, 'Connection reset by peer when receiving')

sometimes show :

cnn.query("select count(*) from bugs")
Traceback (most recent call last):
File "", line 1, in
umysql.SQLError: (1043, 'Bad handshake')

Unicode tests failing

Hi, I'm running the tests on the current head (05166cc) and the two unicode tests are failing. Can you reproduce it or am I doing something wrong?

Assert triggers if not all bytes can be sent at once

The number of bytes to send is calculated by:
size_t bytesToSend = m_writer.getWriteCursor() - m_writer.getReadCursor();
(at https://github.com/esnme/ultramysql/blob/master/lib/Connection.cpp#L173)

But, if not all bytes can be sent at once (slow connection and/or large payload), only the number of sent bytes will be passed to m_writer.pull.
(at https://github.com/esnme/ultramysql/blob/master/lib/Connection.cpp#L191)

However, PacketWriter:pull has the following assert:
assert (m_writeCursor - m_readCursor <= cbSize);
(at https://github.com/esnme/ultramysql/blob/master/lib/PacketWriter.cpp#L93)

Can this assert simply be removed? Or is there something else going on?

Parallel queries

I am unable to execute queries to same database in separate connections in parallel.
amysql + pthreads somehow serializes queries (perhaps doesn't release GIL?)
amysql + gevent crashes (python: PacketWriter.cpp:93: void PacketWriter::pull(size_t): Assertion `m_writeCursor - m_readCursor <= cbSize' failed.)

for comparison, mysqldb + pthreads runs same queries in parallel just fine.

version info: x86_64, amysql git, python 2.7.1, gevent 0.13.6

Stored Procedures on amysql

I tried to call some stored procedures that returns recordset. I'm getting errors I think it does not support stored procedure. I currently use python mysqldb for calling stored procedures. Are there some modifications that can be done to support stored procedure returns?

Please tag releases

Would be great if you could tag releases. Much easier to pin and track versions.

valgrind reports possible lost memory

I've used valgrind to check ultramysql and I'm seeing around 700+ bytes of possibly lost bytes. Is there a known memory leak issue? When I am using ultramysql in my app, I am seeing growing memory

Native gevent support

I would love to use ultramysql in my project, except for this one thing: "Compatible with gevent through monkey patching"; unfortunately, gevent's monkey-patching breaks some unsuspecting 3rd-party code. After gevent.monkey.patch_all(), strange failures started to show up in Hbase-Thrift, for example.

Can there be an option in ultramysql to make it natively gevent-aware without monkey-patching?

Many thanks!

Lazy result fetching

Right now ResultSet.rows contains the whole result. That's not a good idea, as I really need to process some large queries which doesn't fit in main memory and, even if it would fit, the latency would kill me. :-/

Ideally, ResultSet should be an iterator which fetches datasets as needed. For compatibility, the .rows would still be an array with the results; you can pre-fill it by way of a property.
Of course, that doesn't work as-is with nested queries, so the connection needs to keep a pointer to the last-returned resultset and tell it to fetch+store the rest of its data when the next query is started.

This is fairly straightforward to implement, I did it in ten minutes in the old gevent-mysql code.
See github.com:smurfix/gevent-MySQL.git.

Looking at the ultramysql code, this would mean breaking up Connection::handleResultPacket(). If you create a new connection::readOneRow() instead, and call that from the resultset, that'd allow you to drop the m_capi.resultRowBegin/End() calls, so the whole thing would actually become (slightly) faster.

I hope to be able to get around to implementing that in a few weeks, I have a vacation coming up โ€ฆ if somebody beats me to it, good. ;-)

confused about conn.is_connected()(a bug ?)

when is_connected() return True, is it said that the connection is established with mysql? In my case, before every conn.query(), I will call ensure_connected() which to check is_connected() is True(will do reconnect if False).But a Exception complain about "Broken Pipe" throwed when I call conn.query(sql).

build with mingw32

There is an error when building,
python setup.py build -c mingw32

./lib/Connection.cpp:603:79: error: 'alloca' was not declared in this scope

To fix this:
Insert a new line to Connection.cpp:69

define alloca __builtin_alloca

May be put this patch in next version?

[storedprocedure branch] Stored Procedure and MultiResult not Fully Working

I'm always greeted by "SQL Error 1312 PROCEDURE can't return a result set in the given context" everytime I attempt to call a procedure. It does fail too on testMultiResult and testProcedure from the tests.py file included. I can call the procedures however by using mysql cli.

Did I do something wrong or this branch does not yet fully support procedures and multiresult?

can add a ping() method?

   I just use gevent frame to make a game socket server.

and use DBUtils+umysqldb(https://github.com/hongqn/umysqldb by hongqn) as you known umysqldb is compatible wrapper around ultramysql.
DBUtils.PooledDB need dbapi umysqldb's con has a ping() method,if not sometimes will appear mysql error "cursor closed", and hongqn as me to pull a request to you to add one,then he can add in umysqldb.
thanks very much!

testConnectWithWrongDB expects wrong error code

It expects 1049 on trying to connect a nonexisting database. 1049 is only returned when a non existing db requested by admin user (root ie). For regular users, it returns 1044 access denied error, most probably due to a security issue.

Tried on mysql v5.5.31, linux mint olivia.

Connection obj can't search the right result.

I found some problem when using umysql lib.
i use some db pool to reuse the connection obj.

In case gevent timeout during the connection obj searching data from db,then the connection obj can't get right data after. the testing code below,expecting to your solution.

-- coding:utf-8 --

author:Ewing

date:2014-2-20

''' sql here
CREATE DATABASE test;
CREATE TABLE IF NOT EXISTS tb_user (
UserId int(11) NOT NULL,
LastLoginTime int(11) NOT NULL,
PRIMARY KEY (UserId)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

INSERT INTO tb_user (UserId, LastLoginTime) VALUES
(1, 1392859212);
'''

from gevent import monkey; monkey.patch_all()
import time
from gevent import Timeout
import umysql
c = umysql.Connection()
c.connect('10.1.1.18', 3306, 'root','think1dfh@RUNpadm', 'test', False, 'utf8')

looptime = 100000 # FOR TEST set 1 or 100000
def doing():
start = time.time()
for i in xrange(looptime):
sql = 'UPDATE tb_user SET LastLoginTime = %s WHERE UserId = 1'%int(time.time())
c.query(sql,())
runtime = time.time()-start
print 'do all runtime: %s'%runtime

def afterdoing():
sql = 'SELECT * FROM tb_user WHERE UserId = 1'
result_set = c.query(sql,())
print 'afterdoing result_set:',result_set #return (1L, 0L) case timeout then <umysql.ResultSet object> nomal

timeout = Timeout(1)
timeout.start()
try:
doing()
except Timeout:
print 'Could not complete'

afterdoing()
print 'end'

Issue on connect timeout

I tried this code

import amysql
cnn = amysql.Connection()
cnn.connect(DB_HOST, DB_PORT, DB_USER, DB_PASS, DB_NAME)

wait for > 10 seconds
cnn.close()

cnn = amysql.Connection()
cnn.connect(DB_HOST, DB_PORT, DB_USER, DB_PASS, DB_NAME)
Traceback (most recent call last):
File "", line 1, in
RuntimeError: Connection timed out (0)

It produces an error I have to do connect two times for it to have a reconnect

Truncated data when escaping a single string argument

If a string to be escaped is passed in as either a single argument or in a tuple it gets truncated to the first character.

This does not occur if the string is passed within a list, or if multiple arguments are given.

Minimal test case:
import umysql
conn = umysql.Connection()
conn.connect('localhost', 3306, 'user', 'pass', 'test')
rs = conn.query("SELECT %s", 'abcdef')
print rs.rows[0][0]
rs = conn.query("SELECT %s", ('abcdef'))
print rs.rows[0][0]
rs = conn.query("SELECT %s", ['abcdef'])
print rs.rows[0][0]

It would be expected that all of these would function the same
abcdef
abcdef
abcdef
However the observed result is:
a
a
abcdef
Only the 3rd list usage works as expected.

The truncation occurs before the query, from the general query log:
4341 Query SELECT 'a'
4341 Query SELECT 'a'
4341 Query SELECT 'abcdef'

Unable to compile

LINK : error LNK2001: unresolved external symbol PyInit_umysql build\temp.win32-3.3\Release./python\umysql.lib : fatal error LNK1120: 1 unresolved externals
error: command '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe"' failed with exit status 1120

I am running Windows 7 x64
I have Visual Studio 9, 10, 11 (2008, 2010, 2012)
I am running under Python 3.3 environment.
MySQL connector for C/C++/Python are installed.

I had minor hiccups along the way, but this one, i really have no idea how to solve. have always been a problem for me to solve 'link' issue when compiling C/C++

ImportError: No module named pymysql

setup.py need installed pymysql.

$ python setup.py install
Traceback (most recent call last):
File "skiped/src/umysqldb/setup.py", line 3, in
from umysqldb import version
File "skiped/src/umysqldb/umysqldb/init.py", line 2, in
import pymysql
ImportError: No module named pymysql

connecting to AWS RDS instance fails

When I try to connect with ultramysql to Amazon's db instance, I get weird exception:

In [1]: import umysql

In [2]: conn = umysql.Connection()

In [3]: conn.connect('xxxxx.us-east-1.rds.amazonaws.com', 3306, 'xxxxx', 'xxxxx', 'xxxxx')
---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
/home/fevral13/<ipython-input-3-213a95b4356f> in <module>()
----> 1 conn.connect('xxxxx.us-east-1.rds.amazonaws.com', 3306, 'xxxxx', 'xxxxx', 'xxxxx')

SystemError: error return without exception set

In [4]: conn.is_connected()
Out[4]: True

In [5]: conn.query('select 1')
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
/home/fevral13/<ipython-input-5-4ec56317c561> in <module>()
----> 1 conn.query('select 1')

Error: (0, 'Connection reset by peer when receiving')

But when I use pymysql, for example, everything works:

In [6]: import pymysql

In [7]: c = pymysql.Connect(host='xxxxx.us-east-1.rds.amazonaws.com', port=3306, user='xxxxx', passwd='xxxxx', db='xxxxx')

In [8]: c.query('select 1')
Out[8]: 1

So the db is up and connectable

query('SELECT...) returns tuple (0L, 0L) instead of rows/fields, sometimes

Versions: umysql-2.61, python-2.7.6, gevent-1.0.1.

query('SELECT...) returns tuple (0L, 0L) sometimes,
as if it is query('INSERT...) or query('UPDATE...).

In almost all cases exactly the same query returns expected "result.rows" and "result.fields".
But then it suddenly returns tuple (0L, 0L) and this behaviour can be cured by reconnecting to DB.
I catch this bug several times a day, so I may add any debug code - please advise.

Current workaround:

    result = db_conn.query(sql, values)

    if sql.lstrip().startswith('SELECT') and isinstance(result, tuple):
        log.error('reconnecting to db on tuple SELECT: {}'.format(result)) # Logs: (0L, 0L)
        try:
            db_conn.close()
        except Exception:
            pass
        db_conn = umysql.Connection()
        db_conn.connect(...)
        return db_conn.query(sql, values) # Normal "result.rows" this time.

    return result

Segmentation fault

The backtrace is:
#0 0x00007fffe02f033c in API_resultRowValue (result=0x920ae8, column=2, ti=0x33405d0, value=0x2 <Address 0x2 out of bounds>, cbValue=1) at ./python/umysql.c:665
#1 0x7d2928657a697365 in ?? ()
#2 0x732e736968742c29 in ?? ()
#3 0x6576655f65706977 in ?? ()
#4 0x003f00002873746e in ?? ()
#5 0x00fd003f50891a0c in ?? ()
#6 0x5009000300215009 in ?? ()
#7 0x00214008fffd003f in ?? ()
#8 0x000c003f40080003 in ?? ()
#9 0x40087ffd003f0080 in ?? ()
#10 0x0021000000fd0021 in ?? ()
#11 0x00fd0021400829fd in ?? ()
#12 0x400800fd00214008 in ?? ()
#13 0x00007fffe02f0021 in API_connectSocket (sock=0x920ae8, host=0x2 <Address 0x2 out of bounds>, port=53741008) at ./python/io_cpython.c:211
#14 0x0000000010250a50 in ?? ()
#15 0x0000000017c6a8a0 in ?? ()
#16 0x0000000000000003 in ?? ()
#17 0x0000000000000009 in ?? ()
#18 0x0000000000000006 in ?? ()
#19 0x0000000000000006 in ?? ()
#20 0x0000000000000010 in ?? ()
#21 0x0000000000000010 in ?? ()
#22 0x0000000092dc29c0 in ?? ()
#23 0x00007fffffff8af0 in ?? ()
#24 0x0000000000000064 in ?? ()
#25 0x00000000fffffffd in ?? ()
#26 0x000000000000000f in ?? ()
#27 0x000000001eef29f0 in ?? ()
#28 0x0000000010250a50 in ?? ()
#29 0x00000000102480d0 in ?? ()
#30 0x00007fffe02f091d in Connection_query (self=0x920ae8, args=0x2) at ./python/umysql.c:1223
#31 0x00000000004495a7 in vgetargs1 (args=0x920ae8, format=0x2 <Address 0x2 out of bounds>) at Objects/methodobject.c:81
#32 PyArg_ParseTuple (args=0x920ae8, format=0x2 <Address 0x2 out of bounds>) at Python/getargs.c:85
#33 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#34 0x00000000005c2d7e in PyEval_EvalCodeEx (co=0x920ae8, globals=0x2, locals=0x33405d0, args=0x2, argcount=1, kws=0x59ba28, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#35 0x00000000005cca1a in function_call (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/funcobject.c:526
#36 0x00000000005c0592 in ext_do_call (func=0x920ae8, pp_stack=0x2, flags=53741008, na=2, nk=1) at Objects/abstract.c:2529
#37 0x00000000004501d5 in normalizestring (encoding=0x920ae8 "\nh") at Python/ceval.c:2705
#38 _PyCodec_Lookup (encoding=0x920ae8 "\nh") at Python/codecs.c:112
#39 0x00000000005c2d7e in PyEval_EvalCodeEx (co=0x920ae8, globals=0x2, locals=0x33405d0, args=0x2, argcount=1, kws=0x59ba28, kwcount=0, defs=0x0, defcount=0, closure=0x1630310) at Python/ceval.c:3253
#40 0x000000000044a02b in fast_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4117
#41 call_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4042
#42 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#43 0x00000000005c2d7e in PyEval_EvalCodeEx (co=0x920ae8, globals=0x2, locals=0x33405d0, args=0x2, argcount=1, kws=0x59ba28, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#44 0x00000000005cca1a in function_call (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/funcobject.c:526
#45 0x00000000005c0592 in ext_do_call (func=0x920ae8, pp_stack=0x2, flags=53741008, na=2, nk=1) at Objects/abstract.c:2529
#46 0x00000000004501d5 in normalizestring (encoding=0x920ae8 "\nh") at Python/ceval.c:2705
#47 _PyCodec_Lookup (encoding=0x920ae8 "\nh") at Python/codecs.c:112
#48 0x0000000000449531 in PyTuple_New (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4107
#49 load_args (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4191
#50 call_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4019
#51 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#52 0x0000000000449531 in PyTuple_New (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4107
#53 load_args (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4191

---Type to continue, or q to quit---
#54 call_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4019
#55 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#56 0x0000000000449531 in PyTuple_New (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4107
#57 load_args (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4191
#58 call_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4019
#59 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#60 0x00000000005c2d7e in PyEval_EvalCodeEx (co=0x920ae8, globals=0x2, locals=0x33405d0, args=0x2, argcount=1, kws=0x59ba28, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#61 0x00000000005cca1a in function_call (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/funcobject.c:526
#62 0x00000000005d028e in PyTuple_New (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/abstract.c:2529
#63 instancemethod_call (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/classobject.c:2589
#64 0x00000000005de33f in PyObject_Call (exc=0x920ae8, val=0x2, tb=0x33405d0) at Objects/abstract.c:2529
#65 PyEval_CallObjectWithKeywords (exc=0x920ae8, val=0x2, tb=0x33405d0) at Python/ceval.c:3890
#66 PyObject_Call (exc=0x920ae8, val=0x2, tb=0x33405d0) at Python/errors.c:192
#67 0x0000000000483380 in slot_tp_call (self=0x920ae8, args=0x2, kwds=0x33405d0) at Objects/typeobject.c:5432
#68 0x00000000005dc987 in PyObject_Call (o=0x920ae8, a=0x2) at Objects/abstract.c:2529
#69 PyEval_CallObjectWithKeywords (o=0x920ae8, a=0x2) at Python/ceval.c:3890
#70 PyObject_CallObject (o=0x920ae8, a=0x2) at Objects/abstract.c:2517
#71 0x00007fffe4415cbb in app_handler (self=0x920ae8, args=0x2) at meinheld/server/server.c:540
#72 0x00000000005e3c51 in PyObject_Call (exc=0x920ae8, val=0x2, tb=0x33405d0) at Objects/abstract.c:2529
#73 PyEval_CallObjectWithKeywords (exc=0x920ae8, val=0x2, tb=0x33405d0) at Python/ceval.c:3890
#74 PyEval_CallObjectWithKeywords (exc=0x920ae8, val=0x2, tb=0x33405d0) at Python/errors.c:192
#75 0x00007fffe19acc92 in g_initialstub (mark=0x920ae8) at greenlet.c:804
#76 0x00007fffe19ac7b5 in g_switch (target=0x920ae8, args=0x2, kwargs=0x33405d0) at greenlet.c:576
#77 0x00007ffff7fff814 in ?? ()

Socket receive buffer full

I have a couple simple loops that calculate a bunch of values then try to insert them into MySQL, something like this:

for i in items:
    # compare i to every other item, returned as a dictionary
    calcedVals = calculateValues(items, i)
    for j in calcedVals:
        dbconn.query('INSERT INTO tableName VALUES (%s, %s, %s)' %
                     (i, j, calcedVals[j]))

where i and j are INTs in MySQL, and each calculated comparison is a DOUBLE in MySQL.

There are 11,396 items so it would be inserting about 11,396*11,395 (129,857,420) rows, but it only inserted 1,525,202 (it was on the 134th i and 9667th j). Then umysql gave this error and essentially quit:

umysql.Error: (0, 'Socket receive buffer full')

Do you know what could have caused that? Do you think I should just put a try..except around it, and just close and reopen the connection if that happens? Although, I'm not sure if that will fix it, since I don't see any references to m_reader in Connection::close().

Thanks,
Paulie

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.