monetdb / pymonetdb Goto Github PK
View Code? Open in Web Editor NEWThe Python API for MonetDB
Home Page: https://www.monetdb.org/
License: Mozilla Public License 2.0
The Python API for MonetDB
Home Page: https://www.monetdb.org/
License: Mozilla Public License 2.0
Currently, when something (anything?) went wrong during a query execution, the user gets an OperationalError. For instance,
>>> c = pymonetdb.connect("demo", host="localhost", port=50000)
>>> cc = c.cursor()
>>> c.set_autocommit(True)
>>> cc.execute("select * from sys.tables t1, sys.tables t2, sys.tables t3")
# I killed the server during the execution of this query!
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/kutsurak/.cache/pypoetry/virtualenvs/test-pymonetdb-_ivNU08G-py3.8/lib/python3.8/site-packages/pymonetdb/sql/cursors.py", line 165, in execute
block = self.connection.execute(query)
File "/home/kutsurak/.cache/pypoetry/virtualenvs/test-pymonetdb-_ivNU08G-py3.8/lib/python3.8/site-packages/pymonetdb/sql/connections.py", line 140, in execute
return self.command('s' + query + '\n;')
File "/home/kutsurak/.cache/pypoetry/virtualenvs/test-pymonetdb-_ivNU08G-py3.8/lib/python3.8/site-packages/pymonetdb/sql/connections.py", line 145, in command
return self.mapi.cmd(command)
File "/home/kutsurak/.cache/pypoetry/virtualenvs/test-pymonetdb-_ivNU08G-py3.8/lib/python3.8/site-packages/pymonetdb/mapi.py", line 240, in cmd
response = self._getblock()
File "/home/kutsurak/.cache/pypoetry/virtualenvs/test-pymonetdb-_ivNU08G-py3.8/lib/python3.8/site-packages/pymonetdb/mapi.py", line 317, in _getblock
return self._getblock_inet()
File "/home/kutsurak/.cache/pypoetry/virtualenvs/test-pymonetdb-_ivNU08G-py3.8/lib/python3.8/site-packages/pymonetdb/mapi.py", line 323, in _getblock_inet
flag = self._getbytes(2)
File "/home/kutsurak/.cache/pypoetry/virtualenvs/test-pymonetdb-_ivNU08G-py3.8/lib/python3.8/site-packages/pymonetdb/mapi.py", line 347, in _getbytes
raise OperationalError("Server closed connection")
pymonetdb.exceptions.OperationalError: Server closed connection
Would it be possible to separate SQL errors from other errors, in particular, connection errors? In that way, it would be much easier for the users to capture and deal with the query exeuction errors differently than all other types of errors.
In the Oct2020 version of MonetDB, we added day interval as a new SQL type, but pymonetdb doesn't handle it. The format is the same as the second interval. I can try to make a pull request.
import pymonetdb
c1 = pymonetdb.connect(database='demo',port=50000)
cc = c1.cursor()
cc.execute('select interval '1' day')
....
gives
pymonetdb.exceptions.ProgrammingError: type day_interval is not supported
Hi,
I'm trying to connect to a monetdb via sqlalchemy ORM and a given schema "iacm".
When trying to do a simple query I get an error like:
pymonetdb.exceptions.OperationalError: 42000!TODO: column names of level >= 3
which I do not understand really.
The same happens also with a minimal example using pymonetdb 1.3.1 directly (assuming a table users with column id exists in schema iacm.
from pymonetdb.sql.connections import Connection as connect
conn = connect(hostname="localhost",database="demo",port=50000,
username="monetdb",password="monetdb")
c = conn.cursor()
c.execute("SELECT iacm.users.id FROM iacm.users")
r = c.fetchall()
print(r)
conn.close()
Traceback (most recent call last):
File "test.py", line 119, in <module>
c.execute("SELECT iacm.users.id FROM iacm.users")
File "C:\Users\user\Apps\Anaconda3\lib\site-packages\pymonetdb\sql\cursors.py", line 165, in execute
block = self.connection.execute(query)
File "C:\Users\user\Apps\Anaconda3\lib\site-packages\pymonetdb\sql\connections.py", line 140, in execute
return self.command('s' + query + '\n;')
File "C:\Users\user\Apps\Anaconda3\lib\site-packages\pymonetdb\sql\connections.py", line 145, in command
return self.mapi.cmd(command)
File "C:\Users\user\Apps\Anaconda3\lib\site-packages\pymonetdb\mapi.py", line 266, in cmd
raise exception(msg)
pymonetdb.exceptions.OperationalError: 42000!TODO: column names of level >= 3
Does this mean that monetdb cannot be used with sqlalchemy and schemas?
That would be really a pity as it would mean for me that there is no way accessing the given db data...
two issues during running unit tests:
42000!CREATE TABLE: 'dbapi20test_booze' name conflict
for Linuxit is quite python2 specfic
[this is just for the record]
Heard from several users that transferring a large amount of data using pymonetdb is so slow that they all resort to alternatives, such as using mclient.
I have no more detailed information, but when we get around to it, we should benchmark this and try to improve it.
pymonetdb cannot handle all acceptable monetdb dates.
Replicate bug
SELECT DATE '2-01-01'
Or in VOC database execute:
SELECT min(cape_arrival) FROM voyages;
Result
ValueError: time data '2-01-01' does not match format '%Y-%m-%d'
Hi Gijs,
In the 1.3.1 release I noticed a INFO:pymonetdb.mapi:Connection refused
message when everything is fine.
Try this script for example on a clean instance (running on port 53001):
import logging
from pymonetdb.sql.connections import Connection as connect
logging.basicConfig(filename='testconn.log',level=logging.DEBUG)
conn = connect(hostname='localhost',database='testdb',port=53001,username='monetdb',password='monetdb')
c = conn.cursor()
c.execute("create table t1 (i int,d double)")
c.execute("insert into t1 (i,d) values (1,3.14)")
conn.commit()
c.execute("select * from t1")
r = c.fetchall()
print(r)
conn.close()
conn = connect(hostname='localhost',database='testdb',port=53001,username='monetdb',password='monetdb')
and you'll see a log file output like:
INFO:pymonetdb.mapi:Connection refused
DEBUG:pymonetdb.mapi:restarting authentication
DEBUG:pymonetdb.mapi:executing command Xauto_commit 0
DEBUG:pymonetdb.mapi:executing command Xsizeheader 1
DEBUG:pymonetdb.mapi:executing command Xreply_size 100
DEBUG:pymonetdb.mapi:executing command screate table t1 (i int,d double)
;
DEBUG:pymonetdb.mapi:executing command sinsert into t1 (i,d) values (1,3.14)
;
DEBUG:pymonetdb.mapi:executing command sCOMMIT
;
DEBUG:pymonetdb.mapi:executing command sselect * from t1
;
DEBUG:pymonetdb.mapi:executing command sROLLBACK
;
INFO:pymonetdb.mapi:disconnecting from database
INFO:pymonetdb.mapi:Connection refused
DEBUG:pymonetdb.mapi:restarting authentication
DEBUG:pymonetdb.mapi:executing command Xauto_commit 0
DEBUG:pymonetdb.mapi:executing command Xsizeheader 1
DEBUG:pymonetdb.mapi:executing command Xreply_size 100
Logging goes fine with the pymonetdb 1.1.1 version, without Connection refused
messages.
Cheers
The MonetDB server version 11.33.1 and later have support for reading and writing CSV files on the client side. The client side of this has so far only been implemented in mclient. It would be nice if pymonetdb could also do this.
This is related to #74. As a workaround, I try to check for the SQLSTATE error code to distinguish an SQL error from a connection error, since the SQL standard says that every SQL error should have an SQLSTATE error code.
Then I noticed that pymonetdb removes the SQLSTATE error code for several cases including COPY INTO.
(see https://github.com/gijzelaerr/pymonetdb/blob/8e6830d005d0f25e7d4630423e40014057058919/pymonetdb/mapi.py#L46 and until line 74)
What's the retional behind the removal of those error codes? Is it possible to add them back?
It would be nice if the connect() has a variant that can distangle the URI for monetdb, e.g.
pymonetdb.connect(URI=mapi:monetdb://Martins-MacBook-Pro-3.local:50000/sf1)
This can be broken down into host,port, database
Tried to run the following code (as per the example in the doc)
from pymonetdb import mapi
server = mapi.Server()
server.connect(hostname="localhost", port=50000, username="monetdb",
password="monetdb", database="demo", language="sql")
But I got:
AttributeError Traceback (most recent call last)
<ipython-input-1-f3ddc95aff5b> in <module>()
1 from pymonetdb import mapi
----> 2 server = mapi.Server()
3 server.connect(hostname="localhost", port=50000, username="monetdb",
4 password="monetdb", database="demo", language="sql")
AttributeError: module 'pymonetdb.mapi' has no attribute 'Server'
What do I do wrong?
The API always raises a OperationalError
when there is a problem, while for example an IntegrityError
should be raised on a unique or primairy key constrain.
I already added an error handler that parses the undocumented MAPI error format:
OID types in MonetDB are represented as 1235@0, which may cause an error in pymonetdb/sql/pythonize.py", line 144, in convert
A sample line received from the server:
line
'[ 4837,\t"monetdb",\t2019-08-22 12:33:52.000000,\tNULL,\t"running",\t4837@0,\t"select qtag, user, started, estimate, status, tag, query from sys.queue()\n;"\t]'
See if people did any interesting work.
Today I was testing with pymonetdb and I found this:
select interval '1000000000' day;
gives
OverflowError: days=1000000000; must have magnitude <= 999999999
From mclient is fine, I guess the size limit on pymonetdb is not as much.
That would be nice to have an equivalent of cursor.fetchall
, which return results stored as column in numpy masked array.
For example, instead of returning:
[[1, 1.1], [2, 2.2], [3, None]] # Column names are ['id', 'x']
Returning:
{'id': ma.array([1, 2, 3], dtype='int32'),
'x': ma.array([1.1, 2.2, 0], mask=[False, False, True], dtype='float64')}
For a column-store database, it makes sense to have column-store results, and for a scientific Python API, it makes sense to have numpy arrays.
I seems easy to implement, I can submit a pull-request if you are interested.
Using cursor.executemany
to run the same parametric query on many arguments discards all but the last result:
import pymonetdb
DATABASE = "demo"
c = pymonetdb.connect(DATABASE)
cc = c.cursor()
cc.execute("DROP TABLE IF EXISTS foo")
cc.execute("CREATE TABLE foo (i INT, t TEXT)")
cc.execute("INSERT INTO foo VALUES (1, 'a'), (2, 'b'), (3, 'c'), (4, 'd')")
print("Using execute")
for i in range(1, 5):
cc.execute('SELECT * FROM foo WHERE i=%s', [i])
print(cc.fetchall())
print("Using executeall")
cc.executemany('SELECT * FROM foo WHERE i=%s', [(1,), (2,), (3,), (4,)])
print(cc.fetchall())
produces the following output:
Using execute
[(1, 'a')]
[(2, 'b')]
[(3, 'c')]
[(4, 'd')]
Using executeall
[(4, 'd'), (4, 'd'), (4, 'd'), (4, 'd')]
Moreover, passing to the executemany
call an argument than does not exist in the database (cc.executemany('SELECT * FROM foo WHERE i=%s', [(1,), (2,), (3,), (4,), (5,)])
), raises an Exception with fetchone
and seems to block on a read with fetchall
.
This was tested using pymonetdb version 1.4.0 and python 3.9.
Hello Gijs,
Continuing from yesterday's PR I noticed the following:
In mapi.py, when executing a command the response might consist of multiple lines. For instance if autocommit mode is on and one transaction fails because of concurrency conflicts the client gets the following response:
>>&2 1 -1
!40000!COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead
<<
(I have added the >> and << sequence to indicate newlines).
In this case I believe that an exception should be raised, since the client should be notified that the transaction failed. What do you think? I am not an expert in the mapi protocol (not by a long shot), but I will try to create some tests and send a pull request early next week. Would that be ok with you?
Best regards,
Panos.
cursor.fetchall fails if the result is larger than the cursor.arraysize. The code throws an exception while I expect all result rows.
could be related that i'm using the python3 backend.
Traceback (most recent call last):
File "/home/gijs/Work/pymonetdb/tests/test_udf.py", line 33, in test_debug_udf
self.cursor.export('SELECT test_python_udf(1)', 'test_python_udf', filespath=fname)
File "/home/gijs/Work/pymonetdb/pymonetdb/sql/cursors.py", line 193, in export
return export(self, query, fname, sample, filespath)
File "/home/gijs/Work/pymonetdb/pymonetdb/sql/debug.py", line 229, in export
arguments = exportparameters(cursor, ftype, fname, query, quantity_parameters, sample)
File "/home/gijs/Work/pymonetdb/pymonetdb/sql/debug.py", line 174, in exportparameters
cursor.execute(query)
File "/home/gijs/Work/pymonetdb/pymonetdb/sql/cursors.py", line 165, in execute
block = self.connection.execute(query)
File "/home/gijs/Work/pymonetdb/pymonetdb/sql/connections.py", line 140, in execute
return self.command('s' + query + '\n;')
File "/home/gijs/Work/pymonetdb/pymonetdb/sql/connections.py", line 145, in command
return self.mapi.cmd(command)
File "/home/gijs/Work/pymonetdb/pymonetdb/mapi.py", line 249, in cmd
raise exception(msg)
OperationalError: PY000!Invalid string encoding used. Please return a regular ASCII string, or a Numpy_Unicode object.
I'm not clear how to start monetdb programmatically.
I downloaded monetdb with homebrew.
I thought the following ought to work:
import os
import pymonetdb
from pymonetdb.control import Control
MAPIPORT = int(os.environ.get('MAPIPORT', 50000))
TSTHOSTNAME = os.environ.get('TSTHOSTNAME', 'localhost')
TSTPASSPHRASE = os.environ.get('TSTPASSPHRASE', 'testdb')
control = Control(TSTHOSTNAME, MAPIPORT, TSTPASSPHRASE)
But I get:
ConnectionRefusedError Traceback (most recent call last)
<ipython-input-15-9b314d5f4743> in <module>()
3 TSTPASSPHRASE = os.environ.get('TSTPASSPHRASE', 'testdb')
4
----> 5 control = Control(TSTHOSTNAME, MAPIPORT, TSTPASSPHRASE)
/Users/e/anaconda/lib/python3.5/site-packages/pymonetdb/control.py in __init__(self, hostname, port, passphrase, unix_socket)
89 password=passphrase,
90 database='merovingian', language='control',
---> 91 unix_socket=unix_socket)
92 self.server.disconnect()
93
/Users/e/anaconda/lib/python3.5/site-packages/pymonetdb/mapi.py in connect(self, database, username, password, language, hostname, port, unix_socket)
141 # control doesn't require authentication over socket
142 self._login()
--> 143
144 self.state = STATE_READY
145
ConnectionRefusedError: [Errno 61] Connection refused
like postgresql does
http://initd.org/psycopg/docs/cursor.html#cursor.description
is backwards compatible.
This may not be a bug, but I can't understand what I'm doing wrong. (Tested with Python 2.7 and 3.6).
Creating a farm, a database, and populating it works:
export DOTMONETDBFILE="monetdb_config"
cat << EOF > $DOTMONETDBFILE
user=monetdb
password=monetdb
EOF
monetdbd create dbfarm
monetdbd start dbfarm
monetdb create demo
monetdb start demo
echo "CREATE TABLE names (id integer,name varchar(20));" | mclient -d demo
echo "INSERT INTO names VALUES (0, 'Alice');" | mclient -d demo
echo "INSERT INTO names VALUES (1, 'Bob');" | mclient -d demo
echo "SELECT * FROM names;" | mclient -d demo
Querying the database in Python works:
import pymonetdb
connection = pymonetdb.connect(username="monetdb", password="monetdb",
hostname="localhost", database="demo")
cursor = connection.cursor()
cursor.execute('SELECT * FROM names')
cursor.fetchall()
But I always get access denied
when trying to use Control
:
import pymonetdb.control
pymonetdb.control.Control(passphrase='monetdb', hostname='localhost')
No handlers could be found for logger "pymonetdb.mapi"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "(...)/lib/python2.7/site-packages/pymonetdb/control.py", line 91, in __init__
unix_socket=unix_socket)
File "(...)/lib/python2.7/site-packages/pymonetdb/mapi.py", line 142, in connect
self._login()
File "(...)/lib/python2.7/site-packages/pymonetdb/mapi.py", line 165, in _login
raise DatabaseError(prompt[1:])
pymonetdb.exceptions.DatabaseError: access denied
Thanks!
Note: same error with:
pymonetdb.control.Control(passphrase='monetdb', hostname=None)
at the moment we are not thread safe when sharing connections or cursors. This is okay, the DBAPI allowes it, as long as as the .threadsafety
parameter is set correctly:
https://www.python.org/dev/peps/pep-0249/#threadsafety
which we do. But ideally we are also threadsafe when sharing connections and cursors.
Described in this paper: http://www.vldb.org/pvldb/vol10/p1022-muehleisen.pdf
This fork implements the changes master...Mytherin:master
py 2.6 support is broken since @sjoerdmullender his change to the external six module. It looks like six depends on the typing module, which only has Python 2.7(+) support.
Please stop me if 2.6 support is crucial but I guess we are living in the new millennium now.
Hi,
I am working on an application that uses sqlalchemy for translating machine learning models into SQL code. I need to get the database version from the dialect for debugging purposes.
For Monetdb, the database version is always None, it probably needs to be a tuple.
Some script to reproduce:
import sqlalchemy as sa
lDSN1 = "monetdb://monetdb:monetdb@localhost/db"
engine1 = sa.create_engine(lDSN1, encoding='utf-8', echo=True)
print("INITIALIZE_ENGINE_DRIVER_SERVER_VERSION" , engine1.dialect.driver, engine1.dialect.server_version_info)
Sample outputs (with other databases):
INITIALIZE_ENGINE_DRIVER_SERVER_VERSION ibm_db_sa ('DB2/LINUXX8664', '11.01.0000')
INITIALIZE_ENGINE_DRIVER_SERVER_VERSION kinterbasdb (3, 0, 33100, 'firebird')
INITIALIZE_ENGINE_DRIVER_SERVER_VERSION pymonetdb None
INITIALIZE_ENGINE_DRIVER_SERVER_VERSION pymssql (15, 0, 2000, 5)
INITIALIZE_ENGINE_DRIVER_SERVER_VERSION mysqldb (10, 2, 21, 'MariaDB', 10, 2, '21+maria~sid', 'log')
INITIALIZE_ENGINE_DRIVER_SERVER_VERSION cx_oracle (11, 2, 0, 2, 0)
INITIALIZE_ENGINE_DRIVER_SERVER_VERSION psycopg2 (10, 5)
INITIALIZE_ENGINE_DRIVER_SERVER_VERSION pysqlite (3, 30, 1)
Thanks in advance
Antoine
I'm using MonetDB Mar2018 release and the low level MAPI Python interface. When requesting data from MonetDB, only 100 records are returned. Executing the same query from the mclient returns 584 records. My code goes like this:
from pymonetdb import mapi
mapiCon = mapi.Connection()
mapiCon.connect( hostname="localhost", port=50000, username= and so on....)
data = mapiCon.cmd(...myQuery...)
My question is: How does one change this 100 limit using the mapi interface?
socket works.
When using the library and try to build a project using python3.7
, we have recently this error.
ERROR: You need Python 2.7 or 3.4 to install the typing package.
I was able to reproduce this error from the version 1.3.1
until 1.2.1
, works fine on 1.1.1
Since Nov2019 release of MonetDB, both monetdbd (via ipv6 property) and mserver5 (via --set mapi_ipv6=true option) support IPv6 connections. However pymonetdb still lacks this option (via AF_INET6 property on the socket call).
The connection object has a mapi object, which has a password field. The password is plain text.
I was wondering if there is a way to hide that field.
Psycopg for example doesn't return the password, but probably if someone inspects the memory of the program they could find it:
https://www.psycopg.org/docs/connection.html?highlight=password
λ .venv/bin/mypy pymonetdb
pymonetdb/sql/debug.py:103: error: Name 'execfile' is not defined
pymonetdb/sql/debug.py:178: error: Argument 1 to "loads" has incompatible type "str"; expected "bytes"
``
should probably also add stub files for libraries using pymonetdb as 3rd party lib.
Unicode support is hairy, we can make it less hairy by dropping python2 support. Python2 is EOL in 2020.
test suite fails now
The recent commits for issue #74 to use BrokenPipeError instead of OperationalError breaks DPAPI compatiblity.
PEP 249 indicates that "The module should make all error information available through these exceptions or subclasses thereof"
This change impacts getting SQLAlchemy's pool pre-ping functionality to work with pymonetdb. It catches only exceptions that are subclasses of the DBAPI Error class, and won't catch BrokenPipeError.
This could be fixed by using the existing InterfaceError exception, or create a new BrokenPipeError class that subclasses InterfaceError.
I could not find any way to use pymonetdb with a list of ids for where in.
ids = 1,2
cursor.execute('SELECT*FROM raw.bla WHERE "ID" in %s', ids)
I already tried data, list and tuples, but none of them works.
Always "this type is not supported".
This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL > was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
Would it be possible to include the license file in the repo, and add it to MANIFEST.in
? (discussed here)
On types.py file is missing some monetdb types such as UUID and JSON.
seems to be unmaintained, switch to py.test
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.