Python3 compatibility about pypyodbc

[chrisimcevoy]

@sodul I've been looking into this over at https://github.com/chrisimcevoy/purepyodbc/

The code is very alpha and I've taken a different approach to pypyodbc. However, I have a small (but growing) pytest suite which runs tests against different ODBC driver managers on different platforms. In GH Actions there is an Ubuntu agent on which I spin up MSSQL and Postgres in containers and point the test suite at them. The tests are executed in CPython 3.7+ and PyPy3.7. The idea is to eventually include Windows and macOS agents in the CI.

Feel free to take anything you deem worthwhile from there and use it in pypyodbc.

[braian87b]

Just start using it, ignore the unit tests, we will fix those unittest soon, don't worry about it, trust me, it's totally legit working code.

[sodul]

@rolweber thanks for the quick reply. The existing tests are not really unit tests, from a pure definition standpoint. This is unfortunate because unit tests would not have system dependencies and would fast and easy to run. I can probably add some, especially after dropping the python 2.x support.

That said GH Actions might be able to run some of the existing tests. There is an official docker image for MSSQL and as long as the GH Action memory allows it, we should be able to run it as a service container:

• https://dev.to/mihinduranasinghe/using-docker-containers-in-jobs-github-actions-3eof
• https://hub.docker.com/_/microsoft-mssql-server

[braian87b]

Unfortunately I just been using these library sporadically in the recent two years, but I had plans to use it in the near future so I'll have to work to make it work with python3.
please if you have found issues feel free to add to this thread and I will try to address them.

[odidev]

@braian87b Thanks for the response.

While testing this package using the command python setup.py test, I was getting syntax errors.

Here is the log for reference: pypyodbc_testing_result.txt

Corrected the syntax of the print statement according to python3 in test.py and testutils.py file getting below error:

NameError: name 'file' is not defined 

---------------------------------------------------- 

AttributeError: 'TestSuite' object has no attribute 'startswith' 

Here is the log for reference: pypyodbc_testing_result_after_changes.txt

As given in the above error name 'file' is not defined, I replace the file() with open(), as it is not supported in Python 3, currently getting the AttributeError:

AttributeError: 'TestSuite' object has no attribute 'startswith'

Here is the log for reference: pypyodbc_testing_current_result.txt

[odidev]

@braian87b could you please let me know is there any update on this.

[chrisimcevoy]

python setup.py test is not how you run pyodbc unit tests as far as I can see.

For instance, you would run the SQL Server tests with:
python sqlservertests.py <connection string>

FYI, I just tried that with the current sqlservertests.py in pypyodbc using pypy2:

======================================================================
ERROR: test_binary_null (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 316, in test_binary_null
    self._test_strtype('varbinary', None, colsize=100)
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 205, in _test_strtype
    self.cursor.execute("insert into t1 values(?)", (value,))
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1632, in execute
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 976, in ctrl_err
    raise DataError(state,err_text)
DataError: (u'22018', u'[22018] [FreeTDS][SQL Server]Implicit conversion from data type char to varbinary is not allowed. Use the CONVERT function to run this query.')

======================================================================
ERROR: test_different_bindings (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 128, in test_different_bindings
    self.cursor.execute("create table t2(d datetime)")
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1641, in execute
    self.execdirect(query_string)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1667, in execdirect
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 974, in ctrl_err
    raise ProgrammingError(state,err_text)
ProgrammingError: (u'42S01', u"[42S01] [FreeTDS][SQL Server]There is already an object named 't2' in the database.")

======================================================================
ERROR: test_drivers (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1388, in test_drivers
    drivers = pypyodbc.drivers()
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 2742, in drivers
    raise Exception('This function is available for use in Windows only.')
Exception: This function is available for use in Windows only.

======================================================================
ERROR: test_large_binary_null (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 320, in test_large_binary_null
    self._test_strtype('varbinary', None, colsize=4000)
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 205, in _test_strtype
    self.cursor.execute("insert into t1 values(?)", (value,))
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1632, in execute
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 976, in ctrl_err
    raise DataError(state,err_text)
DataError: (u'22018', u'[22018] [FreeTDS][SQL Server]Implicit conversion from data type char to varbinary is not allowed. Use the CONVERT function to run this query.')

======================================================================
ERROR: test_none_param (__main__.SqlServerTestCase)
Ensure None can be used for params other than the first
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1210, in test_none_param
    self.cursor.execute("update t1 set n=?, blob=?", (2, None))
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1632, in execute
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 976, in ctrl_err
    raise DataError(state,err_text)
DataError: (u'22018', u'[22018] [FreeTDS][SQL Server]Implicit conversion from data type char to varbinary(max) is not allowed. Use the CONVERT function to run this query.')

======================================================================
ERROR: test_noscan (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 153, in test_noscan
    self.assertEqual(self.cursor.noscan, False)
AttributeError: Cursor instance has no attribute 'noscan'

======================================================================
ERROR: test_row_execute (__main__.SqlServerTestCase)
Ensure we can use a Row object as a parameter to execute
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1145, in test_row_execute
    self.cursor.execute("create table t2(n int, s varchar(10))")
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1641, in execute
    self.execdirect(query_string)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1667, in execdirect
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 974, in ctrl_err
    raise ProgrammingError(state,err_text)
ProgrammingError: (u'42S01', u"[42S01] [FreeTDS][SQL Server]There is already an object named 't2' in the database.")

======================================================================
ERROR: test_row_executemany (__main__.SqlServerTestCase)
Ensure we can use a Row object as a parameter to executemany
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1158, in test_row_executemany
    self.cursor.execute("create table t2(n int, s varchar(10))")
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1641, in execute
    self.execdirect(query_string)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1667, in execdirect
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 974, in ctrl_err
    raise ProgrammingError(state,err_text)
ProgrammingError: (u'42S01', u"[42S01] [FreeTDS][SQL Server]There is already an object named 't2' in the database.")

======================================================================
ERROR: test_rowcount_reset (__main__.SqlServerTestCase)
Ensure rowcount is reset to -1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 829, in test_rowcount_reset
    self.cursor.execute("create table t2(i int)")
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1641, in execute
    self.execdirect(query_string)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1667, in execdirect
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 974, in ctrl_err
    raise ProgrammingError(state,err_text)
ProgrammingError: (u'42S01', u"[42S01] [FreeTDS][SQL Server]There is already an object named 't2' in the database.")

======================================================================
ERROR: test_sets_execute (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1127, in test_sets_execute
    self.assertRaises(pypyodbc.ProgrammingError, f)
  File "/usr/lib64/pypy-7.3/lib-python/2.7/unittest/case.py", line 473, in assertRaises
    callableObj(*args, **kwargs)
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1125, in f
    self.cursor.execute("insert into t1 (word) VALUES (?)", words)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1488, in execute
    raise TypeError("Params must be in a list, tuple, or Row")
TypeError: Params must be in a list, tuple, or Row

======================================================================
ERROR: test_sp_results (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 685, in test_sp_results
    self.assertEquals(type(rows[0].refdate), datetime)
AttributeError: 'Row' object has no attribute 'refdate'

======================================================================
ERROR: test_sp_results_from_temp (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 711, in test_sp_results_from_temp
    self.assertEquals(type(rows[0].refdate), datetime)
AttributeError: 'Row' object has no attribute 'refdate'

======================================================================
ERROR: test_sp_results_from_vartbl (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 732, in test_sp_results_from_vartbl
    self.assertEquals(type(rows[0].refdate), datetime)
AttributeError: 'Row' object has no attribute 'refdate'

======================================================================
ERROR: test_temp_select (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 909, in test_temp_select
    self.cursor.execute("select s into t2 from t1")
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1641, in execute
    self.execdirect(query_string)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1667, in execdirect
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 974, in ctrl_err
    raise ProgrammingError(state,err_text)
ProgrammingError: (u'42S01', u"[42S01] [FreeTDS][SQL Server]There is already an object named 't2' in the database.")

======================================================================
ERROR: test_too_large (__main__.SqlServerTestCase)
Ensure error raised if insert fails due to truncation
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1238, in test_too_large
    self.assertRaises(pypyodbc.DataError, test)
  File "/usr/lib64/pypy-7.3/lib-python/2.7/unittest/case.py", line 473, in assertRaises
    callableObj(*args, **kwargs)
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1237, in test
    self.cursor.execute("insert into t1 values (?)", ( value,))
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1632, in execute
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 974, in ctrl_err
    raise ProgrammingError(state,err_text)
ProgrammingError: (u'42000', u"[42000] [FreeTDS][SQL Server]String or binary data would be truncated in table 'db.dbo.t1', column 's'. Truncated value: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'.")

======================================================================
ERROR: test_varchar_upperlatin (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 283, in test_varchar_upperlatin
    self._test_strtype('varchar', 'á')
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 205, in _test_strtype
    self.cursor.execute("insert into t1 values(?)", (value,))
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1632, in execute
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 974, in ctrl_err
    raise ProgrammingError(state,err_text)
ProgrammingError: (u'42000', u"[42000] [FreeTDS][SQL Server]String or binary data would be truncated in table 'db.dbo.t1', column 's'. Truncated value: '\xc3'.")

======================================================================
ERROR: test_view_select (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1025, in test_view_select
    self.cursor.execute("create view t2 as select * from t1")
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1641, in execute
    self.execdirect(query_string)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1667, in execdirect
    check_success(self, ret)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 1006, in check_success
    ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
  File "/home/chris/Projects/pypyodbc/pypyodbc.py", line 974, in ctrl_err
    raise ProgrammingError(state,err_text)
ProgrammingError: (u'42S01', u"[42S01] [FreeTDS][SQL Server]There is already an object named 't2' in the database.")

======================================================================
FAIL: test_context_manager_fail (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1315, in test_context_manager_fail
    self.assertEquals(count, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_cursor_context_manager_fail (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 1352, in test_cursor_context_manager_fail
    self.assertEquals(count, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_decimal_1_0_n (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 452, in t
    self._decimal(p, s, n)
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 448, in _decimal
    self.assertEqual(v, value)
AssertionError: Decimal('9') != Decimal('-9')

======================================================================
FAIL: test_decimal_38_0_n (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 452, in t
    self._decimal(p, s, n)
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 448, in _decimal
    self.assertEqual(v, value)
AssertionError: Decimal('99999999999999999999999999999999999999') != Decimal('-99999999999999999999999999999999999999')

======================================================================
FAIL: test_lower_case (__main__.SqlServerTestCase)
Ensure pypyodbc.lowercase forces returned column names to lowercase.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 882, in test_lower_case
    self.assertEquals(names, [ "abc", "def" ])
AssertionError: Lists differ: [u'\u4564f', u'\u6241c'] != ['abc', 'def']

First differing element 0:
u'\u4564f'
'abc'

- [u'\u4564f', u'\u6241c']
+ ['abc', 'def']

======================================================================
FAIL: test_unicode_0 (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 295, in t
    self._test_strtype('nvarchar', value, colsize=len(value))
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 207, in _test_strtype
    self.assertEqual(type(v), resulttype)
AssertionError: <type 'str'> != <type 'unicode'>

======================================================================
FAIL: test_xml_buffer_0 (__main__.SqlServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 408, in t
    self._test_strliketype('xml', value)
  File "/home/chris/Projects/pypyodbc/sqlservertests.py", line 240, in _test_strliketype
    self.assertEqual(type(v), resulttype)
AssertionError: <type 'str'> != <type 'unicode'>

----------------------------------------------------------------------
Ran 217 tests in 9.351s

FAILED (failures=7, errors=17)

As you would expect, the same script fails immediately under pypy3 with SyntaxError:

Traceback (most recent call last):
  File "sqlservertests.py", line 33, in <module>
    from testutils import *
  File "/home/chris/Projects/pypyodbc/testutils.py", line 41
    print 'python:  %s' % sys.version
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print('python:  %s' % sys.version)?

[sodul]

I'm not a direct user of pypyodbc but it has been a challenging one to keep in our internal requirements.txt.

Considering the project is relatively small I'm willing to volunteer some time to work on the following:

• drop support of obsolete versions of python: 2.x and 3.6 or older. They are no longer officially maintained and I feel that keeping backward compatibility is causing more pain than necessary.
• add support for running unittests through GitHub Actions. This will help with accepting pull requests and is free for OSS projects. We should be able to test for linux, windows and macOS as well as several versions of Python.
• help hook the project with SonarCloud. This is free for OSS projects and I found the analysis to be quite good. I already have an analysis performed here: https://sonarcloud.io/summary/overall?id=clumio-oss_pypyodbc
• help push for a style guide with an autoformatter. I recommend Gray https://github.com/dizballanze/gray which combines several tools such as pyuprade, isort, unify and Black to make the source code much more readable.

@braian87b @rolweber let me know if you are interested.

[rolweber]

@sodul Thanks, I think that would be very helpful! Though it won't be me who's going to pick up development of the code. I was in a similar situation as you, having to install pypyodbc as a dependency of some other package. Meanwhile, I got rid of that other package.

I had a brief look at the available tests at the beginning of this year. Looks to me as if they don't use a test framework, and are specifically coded to run against Access and MS SQL Server. On GitHub Actions, the available databases are PostgreSQL and MariaDB, iirc. So I expect a lot of refactoring to be required in this area.