esnme / ultramysql Goto Github PK
View Code? Open in Web Editor NEWA fast MySQL driver written in pure C/C++ for Python. Compatible with gevent through monkey patching.
Home Page: http://www.esn.me
License: Other
A fast MySQL driver written in pure C/C++ for Python. Compatible with gevent through monkey patching.
Home Page: http://www.esn.me
License: Other
:: Description :: A fast MySQL driver written in pure C/C++ for Python. Compatible with gevent through monkey patching :: Requirements :: Requires Python (http://www.python.org) :: Installation :: python setup.py build install
I have a table which has a blob field, and when i want to query all data from it, i got "Socket receive buffer full".
And when i increase the MYSQL_PACKET_SIZE
from (1024 * 1024 * 16)
to (1024 * 1024 * 32)
, it's disappeared.
alldata = list(executeSQL('select * from table_with_big_blob_field', ()).rows)
I would like to point out that identifiers like "__UMCONNECTION_H___
" and "__PACKETREADER_H__
" do not fit to the expected naming conventions of the C/C++ language standard.
Would you like to adjust your selection for unique names?
I am very intrest in the greenlet support, but I look into the testcase, found the interface is total different with from now, so It's not very easy to transform from old driver to your driver. I hope you can make it compatible with PEP 249. Then many people would choose this.
I found some problem when using umysql lib.
i use some db pool to reuse the connection obj.
In case gevent timeout during the connection obj searching data from db,then the connection obj can't get right data after. the testing code below,expecting to your solution.
# -*- coding:utf-8 -*-
# author:Ewing
# date:2014-2-20
''' sql here
CREATE DATABASE test;
USE test;
CREATE TABLE IF NOT EXISTS `tb_user` (
`UserId` int(11) NOT NULL,
`LastLoginTime` int(11) NOT NULL,
PRIMARY KEY (`UserId`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
INSERT INTO `tb_user` (`UserId`, `LastLoginTime`) VALUES
(1, 1392859212);
'''
from gevent import monkey; monkey.patch_all()
import time
from gevent import Timeout
import umysql
c = umysql.Connection()
c.connect('10.1.1.18', 3306, 'root','think1dfh@RUNpadm', 'test', False, 'utf8')
looptime = 100000 # FOR TEST set 1 or 100000
def doing():
start = time.time()
for i in xrange(looptime):
sql = 'UPDATE tb_user SET LastLoginTime = %s WHERE UserId = 1'%int(time.time())
c.query(sql,())
runtime = time.time()-start
print 'do all runtime: %s'%runtime
def afterdoing():
sql = 'SELECT * FROM tb_user WHERE UserId = 1'
result_set = c.query(sql,())
print 'afterdoing result_set:',result_set #return (1L, 0L) case timeout then <umysql.ResultSet object> nomal
timeout = Timeout(1)
timeout.start()
try:
doing()
except Timeout:
print 'Could not complete'
afterdoing()
print 'end'
The following code raises std::bad_alloc on 151st iteration (32bit Ubuntu 12.04 with 4Gb RAM). I think 15 is not too many...
tdbs = []
for i in xrange(100000):
print i
db = umysql.Connection()
db.connect(SQL_HOST, SQL_PORT, SQL_USER, SQL_PASSWD, SQL_DB)
tdbs.append(db)
The module doesn't seem to play nice with Python:
import umysql
a.b = umysql.Connection()
gives me:
RuntimeError: Exception is set for no error in Connection_Destructor
What I would expect:
NameError: name 'a' is not defined
Even more strangely, there is this phenomenon:
I have also seen exceptions where the innermost entry of the stacktrace is missing.
ultramysql throws not connected exception
db = umysql.Connection()
db.connect(config.db_host, config.db_port, config.db_user, config.db_password, config.db_database)
...
def get_info():
rs = db.query(sql)
RuntimeError: Not connected
<Greenlet at 0x1df6870: handle(<socket at 0x1e59390 fileno=[Errno 9] Bad file des, ('127.0.0.1', 64871))> failed with RuntimeError
How to make it KeepAlive forever?
So here q.rows[0] gives a tuple.
How can I get results from dict cursor as in MySQL.
import umysql
cnn = umysql.Connection()
cnn.connect(DB_HOST, 3306, DB_USER, DB_PASSWD, DB_DB)
cnn.connect(DB_HOST, 3306, DB_USER, DB_PASSWD, DB_DB)
q = cnn.query("select * from test_table")
q.rows[0]
(1,
u'abc',
u'def',
0,
datetime.datetime(2013, 8, 13, 14, 25, 46),
0,
datetime.datetime(2012, 8, 13, 14, 26, 3),
1)
For some reason it escapes with double quotes
Hi there,
Is it support micropython?
Regards.
Cut-n-paste from other issue:
Tracked down the assertion crash - apparently amysql doesn't like long lines, e.g. those produced by mysqldump. This only happens for amysql built against gevent.
$ wc test.sql
1 5 959312 test.sql
import sys, gevent, gevent.queue, amysql, time, logging, random
SQL=file("test.sql").read()
slave = amysql.Connection()
slave.connect("xxx", 3306, "xxx", "xxx", "test", True, 'utf8');
slave.query(SQL)
python: PacketWriter.cpp:93: void PacketWriter::pull(size_t): Assertion `m_writeCursor - m_readCursor <= cbSize' failed.
Aborted
Hi,
I'm unable to build ultramysql
. Any idea?
C:\ultramysql>python setup.py build
running build
running build_ext
building 'umysql' extension
C:\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -DWIN32_LEAN_AND_MEAN -I./python -I./lib -IC:\Python27\include -IC:\Python27\PC -c ./python/umysql.c -o build\temp.win32-2.7\Release\.\python\umysql.o
In file included from ./python/umysql.c:60:0:
./python/umysql.h:81:3: error: expected specifier-qualifier-list before 'UINT8'
./python/umysql.c: In function 'API_resultSetField':
./python/umysql.c:150:3: error: 'UMTypeInfo' has no member named 'type'
./python/umysql.c: In function 'DecodeString':
./python/umysql.c:257:13: error: 'UMTypeInfo' has no member named 'charset'
./python/umysql.c:492:103: error: 'UMTypeInfo' has no member named 'charset'
./python/umysql.c: In function 'API_resultRowValue':
./python/umysql.c:511:15: error: 'UMTypeInfo' has no member named 'type'
./python/umysql.c:655:73: error: 'UMTypeInfo' has no member named 'type'
./python/umysql.c: At top level:
./python/umysql.c:679:3: warning: initialization from incompatible pointer type
./python/umysql.c: In function 'EscapeQueryArguments':
./python/umysql.c:1058:5: warning: implicit declaration of function '_alloca'
./python/umysql.c: At top level:
./python/umysql.c:1278:3: warning: initialization from incompatible pointer type
./python/umysql.c:1374:3: warning: initialization from incompatible pointer type
error: command 'gcc' failed with exit status 1
I am using uwsgi in an async ( ugreen) mode, and communicating with mysql using ultramysql, but am getting this trace, mentioning segmentation fault, quite a lot
Mon Nov 24 19:46:55 2014 - !!! uWSGI process 910 got Segmentation Fault !!!
Mon Nov 24 19:46:55 2014 - *** backtrace of 910 ***
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_backtrace+0x25) [0x4401c5]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_segfault+0x21) [0x4402a1]
Mon Nov 24 19:46:55 2014 - /lib64/libc.so.6() [0x3ae0a329a0]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x3a) [0x7f425f2d5dba]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x43) [0x7f425f37d593]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(python_call+0x25) [0x44d205]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_gevent_current_wsgi_req+0x2a) [0x45d6ba]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(warn_pipe+0xa) [0x43cc0a]
Mon Nov 24 19:46:55 2014 - /lib64/libc.so.6() [0x3ae0a329a0]
Mon Nov 24 19:46:55 2014 - /lib64/libpthread.so.0(send+0x6c) [0x3ae0e0ecdc]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/lib-dynload/_socket.so(+0x6df2) [0x7f4252b29df2]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5910) [0x7f425f383da0]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x75c21) [0x7f425f304c21]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x593df) [0x7f425f2e83df]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_CallMethodObjArgs+0x1b2) [0x7f425f2da212]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(API_sendSocket+0x50) [0x7f424f3c4730]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(_ZN10Connection11writeSocketEv+0x59) [0x7f424f3c50e9]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(_ZN10Connection10sendPacketEv+0x18) [0x7f424f3c5158]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(_ZN10Connection5queryEPKcm+0x92) [0x7f424f3c57a2]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/umysql.so(Connection_query+0x12d) [0x7f424f3c3bcd]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5910) [0x7f425f383da0]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x75d28) [0x7f425f304d28]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x412a) [0x7f425f3825ba]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x563a) [0x7f425f383aca]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x638e) [0x7f425f38481e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x638e) [0x7f425f38481e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x563a) [0x7f425f383aca]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x563a) [0x7f425f383aca]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xbc1) [0x7f425f37f051]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xbc1) [0x7f425f37f051]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xbc1) [0x7f425f37f051]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xbc1) [0x7f425f37f051]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x6a8d7) [0x7f425f2f98d7]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyIter_Next+0xb) [0x7f425f2d50db]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_response_subhandler_wsgi+0x5a) [0x45066a]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(uwsgi_request_wsgi+0x3fd) [0x44f91d]
Mon Nov 24 19:46:55 2014 - /usr/local/uwsgi-1.4.4/uwsgi(py_uwsgi_gevent_request+0x92) [0x45c212]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x58ec) [0x7f425f383d7c]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f425f38591e]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x75c21) [0x7f425f304c21]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(+0x593df) [0x7f425f2e83df]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f425f2d5dd3]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x43) [0x7f425f37d593]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/greenlet.so(+0x31d5) [0x7f424f9791d5]
Mon Nov 24 19:46:55 2014 - /usr/local/python2.7/lib/python2.7/site-packages/greenlet.so(+0x28cb) [0x7f424f9788cb]
Mon Nov 24 19:46:55 2014 - [0x156f250]
Mon Nov 24 19:46:55 2014 - *** end of backtrace ***
Any clues as to what can be going wrong here??
When connecting to a db and specifying a database that does not exist you get the error "SystemError: error return without exception set". Setting the db arg to a empty string is a work around.
Minimal example:
import unittest, umysql
class TestMySQL(unittest.TestCase):
def testUnique(self):
conn = umysql.Connection()
conn.connect ('127.0.0.1', 3306, 'testuser', 'testuser', 'NO_EXISTING_DB')
conn.close()
if name == 'main':
unittest.main()
I've found the frequently when starting a large number of simultaneous jobs that all hit a MySQL server at once, sometimes umysql will hang indefinitely when trying to connect. Stacktrace is:
(gdb) bt
#0 0x00007f64612d14cc in recv () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004fb139 in ?? ()
#2 0x0000000000510b4b in ?? ()
#3 0x00000000004cbb0a in PyObject_CallMethodObjArgs ()
#4 0x00007f645e2ee69c in API_recvSocket (sock=0x4f03d00, buffer=0x7f644774b010 "", cbBuffer=cbBuffer@entry=65536) at ./python/io_cpython.c:229
#5 0x00007f645e2ef1c0 in Connection::readSocket (this=this@entry=0x5301720) at ./lib/Connection.cpp:153
#6 0x00007f645e2ef264 in Connection::recvPacket (this=this@entry=0x5301720) at ./lib/Connection.cpp:354
#7 0x00007f645e2efd84 in Connection::connect (this=0x5301720, _host=, _port=, _username=,
_password=<optimized out>, _database=<optimized out>, _autoCommit=0x0, _charset=MCS_utf8_general_ci) at ./lib/Connection.cpp:487
#8 0x00007f645e2ee8aa in UMConnection_Connect (conn=, _host=, _port=, _username=,
_password=<optimized out>, _database=<optimized out>, _autoCommit=0x0, _charset=33) at ./lib/capi.cpp:84
#9 0x00007f645e2ed74e in Connection_connect (self=0x510fcd8, args=) at ./python/umysql.c:860
#10 0x00000000004ac5ce in PyEval_EvalFrameEx ()
#11 0x00000000004b3fd8 in PyEval_EvalCodeEx ()
#12 0x00000000004b4b4c in ?? ()
#13 0x0000000000481cc4 in ?? ()
#14 0x00000000004613b4 in ?? ()
#15 0x0000000000463cc2 in ?? ()
#16 0x00000000004acc66 in PyEval_EvalFrameEx ()
#17 0x00000000004acde0 in PyEval_EvalFrameEx ()
#18 0x00000000004b3fd8 in PyEval_EvalCodeEx ()
#19 0x00000000004acb98 in PyEval_EvalFrameEx ()
#20 0x00000000004b3fd8 in PyEval_EvalCodeEx ()
#21 0x0000000000536723 in ?? ()
#22 0x0000000000446bf2 in PyRun_FileExFlags ()
#23 0x00000000004470ec in PyRun_SimpleFileExFlags ()
#24 0x0000000000447cdc in Py_Main ()
#25 0x00007f64606b6ead in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#26 0x00000000004c7f39 in _start ()
cnn.connect ('a internal ip address which running mysql (some old version)', 3306,'user','pwd', "bugzilla")
cnn.query("show tables")
Traceback (most recent call last):
File "", line 1, in
umysql.Error: (0, 'Connection reset by peer when receiving')
sometimes show :
cnn.query("select count(*) from bugs")
Traceback (most recent call last):
File "", line 1, in
umysql.SQLError: (1043, 'Bad handshake')
Hi, I'm running the tests on the current head (05166cc) and the two unicode tests are failing. Can you reproduce it or am I doing something wrong?
Hi,
Do you have any method on manually commit?
Thanks.
Hello,
It would be interesting to support DBAPI 2.0 http://www.python.org/dev/peps/pep-0249/
That would facilitate ultramysql's integration as a driver into sqlalchemy for instance
like this "mysql+umysql://root:123456@localhost:3306/mysql_drivers_test"
The number of bytes to send is calculated by:
size_t bytesToSend = m_writer.getWriteCursor() - m_writer.getReadCursor();
(at https://github.com/esnme/ultramysql/blob/master/lib/Connection.cpp#L173)
But, if not all bytes can be sent at once (slow connection and/or large payload), only the number of sent bytes will be passed to m_writer.pull
.
(at https://github.com/esnme/ultramysql/blob/master/lib/Connection.cpp#L191)
However, PacketWriter:pull
has the following assert:
assert (m_writeCursor - m_readCursor <= cbSize);
(at https://github.com/esnme/ultramysql/blob/master/lib/PacketWriter.cpp#L93)
Can this assert simply be removed? Or is there something else going on?
I am unable to execute queries to same database in separate connections in parallel.
amysql + pthreads somehow serializes queries (perhaps doesn't release GIL?)
amysql + gevent crashes (python: PacketWriter.cpp:93: void PacketWriter::pull(size_t): Assertion `m_writeCursor - m_readCursor <= cbSize' failed.)
for comparison, mysqldb + pthreads runs same queries in parallel just fine.
version info: x86_64, amysql git, python 2.7.1, gevent 0.13.6
I tried to call some stored procedures that returns recordset. I'm getting errors I think it does not support stored procedure. I currently use python mysqldb for calling stored procedures. Are there some modifications that can be done to support stored procedure returns?
Would be great if you could tag releases. Much easier to pin and track versions.
I've used valgrind to check ultramysql and I'm seeing around 700+ bytes of possibly lost bytes. Is there a known memory leak issue? When I am using ultramysql in my app, I am seeing growing memory
I would love to use ultramysql in my project, except for this one thing: "Compatible with gevent through monkey patching"; unfortunately, gevent's monkey-patching breaks some unsuspecting 3rd-party code. After gevent.monkey.patch_all(), strange failures started to show up in Hbase-Thrift, for example.
Can there be an option in ultramysql to make it natively gevent-aware without monkey-patching?
Many thanks!
Right now ResultSet.rows contains the whole result. That's not a good idea, as I really need to process some large queries which doesn't fit in main memory and, even if it would fit, the latency would kill me. :-/
Ideally, ResultSet should be an iterator which fetches datasets as needed. For compatibility, the .rows would still be an array with the results; you can pre-fill it by way of a property.
Of course, that doesn't work as-is with nested queries, so the connection needs to keep a pointer to the last-returned resultset and tell it to fetch+store the rest of its data when the next query is started.
This is fairly straightforward to implement, I did it in ten minutes in the old gevent-mysql code.
See github.com:smurfix/gevent-MySQL.git.
Looking at the ultramysql code, this would mean breaking up Connection::handleResultPacket(). If you create a new connection::readOneRow() instead, and call that from the resultset, that'd allow you to drop the m_capi.resultRowBegin/End() calls, so the whole thing would actually become (slightly) faster.
I hope to be able to get around to implementing that in a few weeks, I have a vacation coming up โฆ if somebody beats me to it, good. ;-)
when is_connected() return True, is it said that the connection is established with mysql? In my case, before every conn.query(), I will call ensure_connected() which to check is_connected() is True(will do reconnect if False).But a Exception complain about "Broken Pipe" throwed when I call conn.query(sql).
There is an error when building,
python setup.py build -c mingw32
./lib/Connection.cpp:603:79: error: 'alloca' was not declared in this scope
To fix this:
Insert a new line to Connection.cpp:69
May be put this patch in next version?
I'm always greeted by "SQL Error 1312 PROCEDURE can't return a result set in the given context" everytime I attempt to call a procedure. It does fail too on testMultiResult and testProcedure from the tests.py file included. I can call the procedures however by using mysql cli.
Did I do something wrong or this branch does not yet fully support procedures and multiresult?
I just use gevent frame to make a game socket server.
and use DBUtils+umysqldb(https://github.com/hongqn/umysqldb by hongqn) as you known umysqldb is compatible wrapper around ultramysql.
DBUtils.PooledDB need dbapi umysqldb's con has a ping() method,if not sometimes will appear mysql error "cursor closed", and hongqn as me to pull a request to you to add one,then he can add in umysqldb.
thanks very much!
It would be great if the library was compatible with PEP-0249 so we could for exemple create code tests using SQLite in memory and would be easier to make code compatible with other databases. http://www.python.org/dev/peps/pep-0249/
It expects 1049 on trying to connect a nonexisting database. 1049 is only returned when a non existing db requested by admin user (root ie). For regular users, it returns 1044 access denied error, most probably due to a security issue.
Tried on mysql v5.5.31, linux mint olivia.
I found some problem when using umysql lib.
i use some db pool to reuse the connection obj.
''' sql here
CREATE DATABASE test;
CREATE TABLE IF NOT EXISTS tb_user
(
UserId
int(11) NOT NULL,
LastLoginTime
int(11) NOT NULL,
PRIMARY KEY (UserId
)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
INSERT INTO tb_user
(UserId
, LastLoginTime
) VALUES
(1, 1392859212);
'''
from gevent import monkey; monkey.patch_all()
import time
from gevent import Timeout
import umysql
c = umysql.Connection()
c.connect('10.1.1.18', 3306, 'root','think1dfh@RUNpadm', 'test', False, 'utf8')
looptime = 100000 # FOR TEST set 1 or 100000
def doing():
start = time.time()
for i in xrange(looptime):
sql = 'UPDATE tb_user SET LastLoginTime = %s WHERE UserId = 1'%int(time.time())
c.query(sql,())
runtime = time.time()-start
print 'do all runtime: %s'%runtime
def afterdoing():
sql = 'SELECT * FROM tb_user WHERE UserId = 1'
result_set = c.query(sql,())
print 'afterdoing result_set:',result_set #return (1L, 0L) case timeout then <umysql.ResultSet object> nomal
timeout = Timeout(1)
timeout.start()
try:
doing()
except Timeout:
print 'Could not complete'
afterdoing()
print 'end'
I tried this code
import amysql
cnn = amysql.Connection()
cnn.connect(DB_HOST, DB_PORT, DB_USER, DB_PASS, DB_NAME)
wait for > 10 seconds
cnn.close()
cnn = amysql.Connection()
cnn.connect(DB_HOST, DB_PORT, DB_USER, DB_PASS, DB_NAME)
Traceback (most recent call last):
File "", line 1, in
RuntimeError: Connection timed out (0)
It produces an error I have to do connect two times for it to have a reconnect
If a string to be escaped is passed in as either a single argument or in a tuple it gets truncated to the first character.
This does not occur if the string is passed within a list, or if multiple arguments are given.
Minimal test case:
import umysql
conn = umysql.Connection()
conn.connect('localhost', 3306, 'user', 'pass', 'test')
rs = conn.query("SELECT %s", 'abcdef')
print rs.rows[0][0]
rs = conn.query("SELECT %s", ('abcdef'))
print rs.rows[0][0]
rs = conn.query("SELECT %s", ['abcdef'])
print rs.rows[0][0]
It would be expected that all of these would function the same
abcdef
abcdef
abcdef
However the observed result is:
a
a
abcdef
Only the 3rd list usage works as expected.
The truncation occurs before the query, from the general query log:
4341 Query SELECT 'a'
4341 Query SELECT 'a'
4341 Query SELECT 'abcdef'
LINK : error LNK2001: unresolved external symbol PyInit_umysql build\temp.win32-3.3\Release./python\umysql.lib : fatal error LNK1120: 1 unresolved externals
error: command '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe"' failed with exit status 1120
I am running Windows 7 x64
I have Visual Studio 9, 10, 11 (2008, 2010, 2012)
I am running under Python 3.3 environment.
MySQL connector for C/C++/Python are installed.
I had minor hiccups along the way, but this one, i really have no idea how to solve. have always been a problem for me to solve 'link' issue when compiling C/C++
setup.py need installed pymysql.
$ python setup.py install
Traceback (most recent call last):
File "skiped/src/umysqldb/setup.py", line 3, in
from umysqldb import version
File "skiped/src/umysqldb/umysqldb/init.py", line 2, in
import pymysql
ImportError: No module named pymysql
When I try to connect with ultramysql to Amazon's db instance, I get weird exception:
In [1]: import umysql
In [2]: conn = umysql.Connection()
In [3]: conn.connect('xxxxx.us-east-1.rds.amazonaws.com', 3306, 'xxxxx', 'xxxxx', 'xxxxx')
---------------------------------------------------------------------------
SystemError Traceback (most recent call last)
/home/fevral13/<ipython-input-3-213a95b4356f> in <module>()
----> 1 conn.connect('xxxxx.us-east-1.rds.amazonaws.com', 3306, 'xxxxx', 'xxxxx', 'xxxxx')
SystemError: error return without exception set
In [4]: conn.is_connected()
Out[4]: True
In [5]: conn.query('select 1')
---------------------------------------------------------------------------
Error Traceback (most recent call last)
/home/fevral13/<ipython-input-5-4ec56317c561> in <module>()
----> 1 conn.query('select 1')
Error: (0, 'Connection reset by peer when receiving')
But when I use pymysql, for example, everything works:
In [6]: import pymysql
In [7]: c = pymysql.Connect(host='xxxxx.us-east-1.rds.amazonaws.com', port=3306, user='xxxxx', passwd='xxxxx', db='xxxxx')
In [8]: c.query('select 1')
Out[8]: 1
So the db is up and connectable
Versions: umysql-2.61, python-2.7.6, gevent-1.0.1.
query('SELECT...) returns tuple (0L, 0L) sometimes,
as if it is query('INSERT...) or query('UPDATE...).
In almost all cases exactly the same query returns expected "result.rows" and "result.fields".
But then it suddenly returns tuple (0L, 0L) and this behaviour can be cured by reconnecting to DB.
I catch this bug several times a day, so I may add any debug code - please advise.
Current workaround:
result = db_conn.query(sql, values)
if sql.lstrip().startswith('SELECT') and isinstance(result, tuple):
log.error('reconnecting to db on tuple SELECT: {}'.format(result)) # Logs: (0L, 0L)
try:
db_conn.close()
except Exception:
pass
db_conn = umysql.Connection()
db_conn.connect(...)
return db_conn.query(sql, values) # Normal "result.rows" this time.
return result
The backtrace is:
#0 0x00007fffe02f033c in API_resultRowValue (result=0x920ae8, column=2, ti=0x33405d0, value=0x2 <Address 0x2 out of bounds>, cbValue=1) at ./python/umysql.c:665
#1 0x7d2928657a697365 in ?? ()
#2 0x732e736968742c29 in ?? ()
#3 0x6576655f65706977 in ?? ()
#4 0x003f00002873746e in ?? ()
#5 0x00fd003f50891a0c in ?? ()
#6 0x5009000300215009 in ?? ()
#7 0x00214008fffd003f in ?? ()
#8 0x000c003f40080003 in ?? ()
#9 0x40087ffd003f0080 in ?? ()
#10 0x0021000000fd0021 in ?? ()
#11 0x00fd0021400829fd in ?? ()
#12 0x400800fd00214008 in ?? ()
#13 0x00007fffe02f0021 in API_connectSocket (sock=0x920ae8, host=0x2 <Address 0x2 out of bounds>, port=53741008) at ./python/io_cpython.c:211
#14 0x0000000010250a50 in ?? ()
#15 0x0000000017c6a8a0 in ?? ()
#16 0x0000000000000003 in ?? ()
#17 0x0000000000000009 in ?? ()
#18 0x0000000000000006 in ?? ()
#19 0x0000000000000006 in ?? ()
#20 0x0000000000000010 in ?? ()
#21 0x0000000000000010 in ?? ()
#22 0x0000000092dc29c0 in ?? ()
#23 0x00007fffffff8af0 in ?? ()
#24 0x0000000000000064 in ?? ()
#25 0x00000000fffffffd in ?? ()
#26 0x000000000000000f in ?? ()
#27 0x000000001eef29f0 in ?? ()
#28 0x0000000010250a50 in ?? ()
#29 0x00000000102480d0 in ?? ()
#30 0x00007fffe02f091d in Connection_query (self=0x920ae8, args=0x2) at ./python/umysql.c:1223
#31 0x00000000004495a7 in vgetargs1 (args=0x920ae8, format=0x2 <Address 0x2 out of bounds>) at Objects/methodobject.c:81
#32 PyArg_ParseTuple (args=0x920ae8, format=0x2 <Address 0x2 out of bounds>) at Python/getargs.c:85
#33 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#34 0x00000000005c2d7e in PyEval_EvalCodeEx (co=0x920ae8, globals=0x2, locals=0x33405d0, args=0x2, argcount=1, kws=0x59ba28, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#35 0x00000000005cca1a in function_call (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/funcobject.c:526
#36 0x00000000005c0592 in ext_do_call (func=0x920ae8, pp_stack=0x2, flags=53741008, na=2, nk=1) at Objects/abstract.c:2529
#37 0x00000000004501d5 in normalizestring (encoding=0x920ae8 "\nh") at Python/ceval.c:2705
#38 _PyCodec_Lookup (encoding=0x920ae8 "\nh") at Python/codecs.c:112
#39 0x00000000005c2d7e in PyEval_EvalCodeEx (co=0x920ae8, globals=0x2, locals=0x33405d0, args=0x2, argcount=1, kws=0x59ba28, kwcount=0, defs=0x0, defcount=0, closure=0x1630310) at Python/ceval.c:3253
#40 0x000000000044a02b in fast_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4117
#41 call_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4042
#42 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#43 0x00000000005c2d7e in PyEval_EvalCodeEx (co=0x920ae8, globals=0x2, locals=0x33405d0, args=0x2, argcount=1, kws=0x59ba28, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#44 0x00000000005cca1a in function_call (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/funcobject.c:526
#45 0x00000000005c0592 in ext_do_call (func=0x920ae8, pp_stack=0x2, flags=53741008, na=2, nk=1) at Objects/abstract.c:2529
#46 0x00000000004501d5 in normalizestring (encoding=0x920ae8 "\nh") at Python/ceval.c:2705
#47 _PyCodec_Lookup (encoding=0x920ae8 "\nh") at Python/codecs.c:112
#48 0x0000000000449531 in PyTuple_New (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4107
#49 load_args (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4191
#50 call_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4019
#51 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#52 0x0000000000449531 in PyTuple_New (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4107
#53 load_args (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4191
---Type to continue, or q to quit---
#54 call_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4019
#55 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#56 0x0000000000449531 in PyTuple_New (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4107
#57 load_args (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4191
#58 call_function (pp_stack=0x920ae8, oparg=2) at Python/ceval.c:4019
#59 0x000000000044d008 in PyEval_EvalFrameEx (f=0x920ae8, throwflag=2) at Python/ceval.c:2666
#60 0x00000000005c2d7e in PyEval_EvalCodeEx (co=0x920ae8, globals=0x2, locals=0x33405d0, args=0x2, argcount=1, kws=0x59ba28, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#61 0x00000000005cca1a in function_call (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/funcobject.c:526
#62 0x00000000005d028e in PyTuple_New (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/abstract.c:2529
#63 instancemethod_call (func=0x920ae8, arg=0x2, kw=0x33405d0) at Objects/classobject.c:2589
#64 0x00000000005de33f in PyObject_Call (exc=0x920ae8, val=0x2, tb=0x33405d0) at Objects/abstract.c:2529
#65 PyEval_CallObjectWithKeywords (exc=0x920ae8, val=0x2, tb=0x33405d0) at Python/ceval.c:3890
#66 PyObject_Call (exc=0x920ae8, val=0x2, tb=0x33405d0) at Python/errors.c:192
#67 0x0000000000483380 in slot_tp_call (self=0x920ae8, args=0x2, kwds=0x33405d0) at Objects/typeobject.c:5432
#68 0x00000000005dc987 in PyObject_Call (o=0x920ae8, a=0x2) at Objects/abstract.c:2529
#69 PyEval_CallObjectWithKeywords (o=0x920ae8, a=0x2) at Python/ceval.c:3890
#70 PyObject_CallObject (o=0x920ae8, a=0x2) at Objects/abstract.c:2517
#71 0x00007fffe4415cbb in app_handler (self=0x920ae8, args=0x2) at meinheld/server/server.c:540
#72 0x00000000005e3c51 in PyObject_Call (exc=0x920ae8, val=0x2, tb=0x33405d0) at Objects/abstract.c:2529
#73 PyEval_CallObjectWithKeywords (exc=0x920ae8, val=0x2, tb=0x33405d0) at Python/ceval.c:3890
#74 PyEval_CallObjectWithKeywords (exc=0x920ae8, val=0x2, tb=0x33405d0) at Python/errors.c:192
#75 0x00007fffe19acc92 in g_initialstub (mark=0x920ae8) at greenlet.c:804
#76 0x00007fffe19ac7b5 in g_switch (target=0x920ae8, args=0x2, kwargs=0x33405d0) at greenlet.c:576
#77 0x00007ffff7fff814 in ?? ()
I have a couple simple loops that calculate a bunch of values then try to insert them into MySQL, something like this:
for i in items:
# compare i to every other item, returned as a dictionary
calcedVals = calculateValues(items, i)
for j in calcedVals:
dbconn.query('INSERT INTO tableName VALUES (%s, %s, %s)' %
(i, j, calcedVals[j]))
where i and j are INT
s in MySQL, and each calculated comparison is a DOUBLE
in MySQL.
There are 11,396 items so it would be inserting about 11,396*11,395 (129,857,420) rows, but it only inserted 1,525,202 (it was on the 134th i and 9667th j). Then umysql gave this error and essentially quit:
umysql.Error: (0, 'Socket receive buffer full')
Do you know what could have caused that? Do you think I should just put a try..except
around it, and just close and reopen the connection if that happens? Although, I'm not sure if that will fix it, since I don't see any references to m_reader
in Connection::close()
.
Thanks,
Paulie
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.