essolutions / django-mssql-backend

Django backend for Microsoft SQL Server

django-mssql-backend

django-mssql-backend is a fork of django-pyodbc-azure

Features

• Supports Django 2.2, 3.0
• Supports Microsoft SQL Server 2008/2008R2, 2012, 2014, 2016, 2017, 2019
• Passes most of the tests of the Django test suite
• Compatible with Micosoft ODBC Driver for SQL Server, SQL Server Native Client, and FreeTDS ODBC drivers

Dependencies

• Django 2.2 or newer
• pyodbc 3.0 or newer

Installation

1. Install pyodbc and Django

2. Install django-mssql-backend

   pip install django-mssql-backend
   

3. Now you can point the ENGINE setting in the settings file used by your Django application or project to the 'sql_server.pyodbc' module path

   'ENGINE': 'sql_server.pyodbc'
   

Regex Support

django-mssql-backend supports regex using a CLR .dll file. To install it, run

python manage.py install_regex_clr {database_name}

Configuration

Standard Django settings

The following entries in a database-level settings dictionary in DATABASES control the behavior of the backend:

• ENGINE

  String. It must be "sql_server.pyodbc".

• NAME

  String. Database name. Required.

• HOST

  String. SQL Server instance in "server\instance" format.

• PORT

  String. Server instance port. An empty string means the default port.

• USER

  String. Database user name in "user" format. If not given then MS Integrated Security will be used.

• PASSWORD

  String. Database user password.

• AUTOCOMMIT

  Boolean. Set this to False if you want to disable Django's transaction management and implement your own.

and the following entries are also available in the TEST dictionary for any given database-level settings dictionary:

• NAME

  String. The name of database to use when running the test suite. If the default value (None) is used, the test database will use the name "test_" + NAME.

• COLLATION

  String. The collation order to use when creating the test database. If the default value (None) is used, the test database is assigned the default collation of the instance of SQL Server.

• DEPENDENCIES

  String. The creation-order dependencies of the database. See the official Django documentation for more details.

• MIRROR

  String. The alias of the database that this database should mirror during testing. Default value is None. See the official Django documentation for more details.

OPTIONS

Dictionary. Current available keys are:

• driver

  String. ODBC Driver to use ("ODBC Driver 13 for SQL Server", "SQL Server Native Client 11.0", "FreeTDS" etc). Default is "ODBC Driver 13 for SQL Server".

• isolation_level

  String. Sets transaction isolation level for each database session. Valid values for this entry are READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, SNAPSHOT, and SERIALIZABLE. Default is None which means no isolation levei is set to a database session and SQL Server default will be used.

• dsn

  String. A named DSN can be used instead of HOST.

• host_is_server

  Boolean. Only relevant if using the FreeTDS ODBC driver under Unix/Linux.

  By default, when using the FreeTDS ODBC driver the value specified in the HOST setting is used in a SERVERNAME ODBC connection string component instead of being used in a SERVER component; this means that this value should be the name of a dataserver definition present in the freetds.conf FreeTDS configuration file instead of a hostname or an IP address.

  But if this option is present and its value is True, this special behavior is turned off. Instead, connections to the database server will be established using HOST and PORT options, without requiring freetds.conf to be configured.

  See https://www.freetds.org/userguide/dsnless.html for more information.

• unicode_results

  Boolean. If it is set to True, pyodbc's unicode_results feature is activated and strings returned from pyodbc are always Unicode. Default value is False.

• extra_params

  String. Additional parameters for the ODBC connection. The format is "param=value;param=value".

• collation

  String. Name of the collation to use when performing text field lookups against the database. Default is None; this means no collation specifier is added to your lookup SQL (the default collation of your database will be used). For Chinese language you can set it to "Chinese_PRC_CI_AS".

• connection_timeout

  Integer. Sets the timeout in seconds for the database connection process. Default value is 0 which disables the timeout.

• connection_retries

  Integer. Sets the times to retry the database connection process. Default value is 5.

• connection_retry_backoff_time

  Integer. Sets the back off time in seconds for reries of the database connection process. Default value is 5.

• query_timeout

  Integer. Sets the timeout in seconds for the database query. Default value is 0 which disables the timeout.

backend-specific settings

The following project-level settings also control the behavior of the backend:

• DATABASE_CONNECTION_POOLING

  Boolean. If it is set to False, pyodbc's connection pooling feature won't be activated.

Example

Here is an example of the database settings:

DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc',
        'NAME': 'mydb',
        'USER': 'user@myserver',
        'PASSWORD': 'password',
        'HOST': 'myserver.database.windows.net',
        'PORT': '',

        'OPTIONS': {
            'driver': 'ODBC Driver 13 for SQL Server',
        },
    },
}

# set this to False if you want to turn off pyodbc's connection pooling
DATABASE_CONNECTION_POOLING = False

Limitations

The following features are currently not supported:

• Altering a model field from or to AutoField at migration

django-mssql-backend's Issues

Invalid query is generated: For field=BitField

Hi all, thanks for your work on this. Noticed the below when testing version 2.8.1 with not yet supported Django 3.1.1. However, it works as expected up to version 3.0.10. Just adding this here fwiw or any that misinterpret "Django 2.2 or newer" :-).

models.py:

class Book(models.Model):
    BitField = models.BooleanField(db_column="BitField", default=0)

query:
queryset = Book.objects.filter(BitField=True)

Generated T-SQL (Django 3.1.1):

SELECT TOP 10 dbo.Book.PK, dbo.Book.BitField
FROM dbo.Book
WHERE NOT dbo.Book.BitField

-- For Django up to 3.0.10 the correct T-SQL is generated:
-- ... WHERE dbo.Book.BitField = 0

Error:
An expression of non-boolean type specified in a context where a condition is expected, near 'BitField'.

Native UUID field

#20 enabled native UUID fields which later was discovered was not fully supported.

As described in 8bf0154, the following chain of migrations does not work as expected:

operations = [
    migrations.CreateModel(
        name='UUIDModel',
        fields=[
            ('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
            ('other',  models.UUIDField(null=True)),
        ],
    ),
    migrations.AlterField(
        model_name='uuidmodel',
        name='other',
        field=models.UUIDField(default=uuid.uuid4),
    ),
]

Before enabling this again more tests will have to be added to ensure correct functionality.

---

@aceofwings Since you are the author of #20, do you have any thoughts on this?

Ubuntu 20.04.1 with python 3.8.2 Django 3.0 SyntaxError: invalid syntax except ImportError, e:

I am having problems deploying my Django app (using Python 3.8.2) on a Ubuntu Server 20.04.1 LTS that uses MS SQL Server as its database. No issues reverting back to 'ENGINE': 'django.db.backends.sqlite3' on Linux but I have to be honest, it is frustrating.

I already had to downgrade to Django==3.0 to get my app working on my local windows pc but the 2 migrations I did proves that I am missing something and I hope someone can give me a hand. Any help is appreciated.

installed components:
pip freeze
asgiref==3.2.10
Django==3.0
django-mssql-backend==2.8.1
pyodbc==4.0.30
pytz==2020.1
sql-server.pyodbc==1.0
sqlparse==0.3.1

when I run the server I get a syntax error:

Watching for file changes with StatReloader
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 109, in inner_run
    autoreload.raise_last_exception()
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/utils/autoreload.py", line 76, in raise_last_exception
    raise _exception[1]
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 357, in execute
    autoreload.check_errors(django.setup)()
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/luca/app/MDM_SQLServer/cpus/models.py", line 6, in <module>
    from sites.models import Site
  File "/home/luca/app/MDM_SQLServer/sites/models.py", line 4, in <module>
    from wifis.models import Wifi
  File "/home/luca/app/MDM_SQLServer/wifis/models.py", line 3, in <module>
    class Wifi(models.Model):
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/db/models/base.py", line 121, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/db/models/base.py", line 325, in add_to_class
    value.contribute_to_class(cls, name)
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/db/models/options.py", line 208, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/db/utils.py", line 207, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/luca/app/venv/lib/python3.8/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  **File "/home/luca/app/venv/lib/python3.8/site-packages/sql_server/pyodbc/base.py", line 7
    except ImportError, e:
                      ^
SyntaxError: invalid syntax**

Wrong value for supports_ignore_conflicts feature support

The defaut for supports_ignore_conflicts is set to True in BaseDatabaseFeatures.
See django/db/backends/base/features.py:

# Does the backend support ignoring constraint or uniqueness errors during
# INSERT?
supports_ignore_conflicts = True

django-mssql-backend doesn't set this to False even though it seems to be unsupported.
Should supports_ignore_conflicts be set to False in features.py?

pyodbc fast_executemany feature doesn't work

Hello!
pyodbc has a fast_executemany feature, which significantly increases inserts speed when using the ODBC Driver 17 for SQL Server driver. It works perfectly when I creating connection and cursor using pyodbc API directly:

conn = pyodbc.connect(conn_str, autocommit=False)
cursor = conn.cursor()
cursor.fast_executemany = True

cursor.executemany(f'INSERT INTO [{table_name}] ({columns_string}) VALUES ({columns_values_placeholders_string})', data)
conn.commit()

However, I get no effect (no speed boost and MSSQL profiler shows the exec sp_prepexec calls instead of the exec sp_execute ones, which appear when fast_executemany really works) when using it with Django API (i.e. via wrapper provided by django-mssql-backend):

with connection.cursor() as cursor:
    cursor.fast_executemany = True
    inserts go here

Am I doing something wrong or this feature simply not implemented?

Thank you.

django.db.utils.InterfaceError: ('28000', "[28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user <user_name>. (18456) (SQLDriverConnect)")

I am getting the above error while trying to connect with django locally. Could there be a problem here?

django-mssql-backend/sql_server/pyodbc/base.py

Line 284 in 7982506

if ms_drivers.match(driver):

    DATABASES = {
        'default':
            {
                'ENGINE': 'sql_server.pyodbc',
                'NAME': os.environ.get('MSSQL_DB'),
                'USER': os.environ.get('MSSQL_USER'),
                'PASSWORD': os.environ.get('MSSQL_PASSWORD'),
                'HOST': os.environ.get('MSSQL_HOST'),
                'PORT': os.environ.get('MSSQL_PORT'),
                'OPTIONS': {
                    'driver': 'ODBC Driver 17 for SQL Server',
                },
            }
    }

I can attest that my database credentials are correct.

'sql_server.pyodbc' isn't an available database backend.

Hi all,

I get the following error when doing runserver:

django.core.exceptions.ImproperlyConfigured: 'sql_server.pyodbc' isn't an available database backend.
Try using 'django.db.backends.XXX', where XXX is one of:
    'mysql', 'oracle', 'postgresql', 'sqlite3'

My configuration in settings is like that:

{'NAME': '***',
  'USER': '***',
  'PASSWORD': '***',
  'HOST': '192.168.10.70',
  'PORT': '',
  'ENGINE': 'sql_server.pyodbc',
  'OPTIONS': {'driver': 'ODBC Driver 17 for SQL Server'}
}

The library seems to be installed, this import works without errors:
import sql_server.pyodbc

Any help appreciated.

Alter field migration fails if field part of unique_together constraint

A simple max_length change on a char field will trigger the failure:-

operations = [
    migrations.CreateModel(
        name='TestAlter',
        fields=[
            ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
            ('key', models.CharField(max_length=10)),
            ('value', models.TextField(blank=True, default='')),
            ('uniqueness', models.CharField(max_length=10)),
        ],
        options={
            'unique_together': {('key', 'uniqueness')},
        },
    ),
    migrations.AlterField(
        model_name='TestAlter',
        name='key',
        field=models.CharField(max_length=20),
    ),
]

File "/opt/test/rest/venv_django_2/lib64/python3.6/site-packages/sql_server/pyodbc/base.py", line 553, in execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: ('42S11', "[42S11] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The operation failed because an index or statistics with name 'test_testalter_key_uniqueness_f43d78f4_uniq' already exists on table 'test_testalter'. (1913) (SQLExecDirectW)")

Django 2.2.13
django-mssql-backend 2.8.1

Gunicorn gets error: "The cursor's connection has been closed."

When using Django with Gunicorn i get the error "The cursor's connection has been closed." when trying to get request from mssql tables.
When i run the server without Gunicorn everything is ok, and even when i try to use sqlalchemy connectio with Gunicorn it worked ok.

this is my config for Gunicorn

$ gunicorn sela.wsgi:application -c gunicorn.conf.py

unicorn.conf.py

import multiprocessing

bind = "unix:api.sock"
reload = True
logconfig = "/opt/api/gunicorn_logging.ini"
workers = multiprocessing.cpu_count() * 2 + 1

Invalid Query is generated: Each GROUP BY expression must contain at least one column that is not an outer reference.

I have the following simple query which runs on SQL Server.

expr = ExpressionWrapper(
     (Value(10) - Value(5)) - Sum(F('colA')), output_field=FloatField())
Table.objects.values('colb').annotate(expr=expr)

this backend generates the following SQL query

SELECT ((%s - %s) - SUM([table].[colA])) AS [expr] FROM [table] GROUP BY [table].[colB], (%s - %s)

the issue here is that the query above seems to be illegal as the code crashes with the exception below

Traceback (most recent call last):
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/.env/lib/python3.6/site-packages/sql_server/pyodbc/base.py", line 553, in execute
    return self.cursor.execute(sql, params)
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Each GROUP BY expression must contain at least one column that is not an outer reference. (164) (SQLExecDirectW)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "driver_mssql.py", line 28, in <module>
    for r in ret2:
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/models/query.py", line 287, in __iter__
    self._fetch_all()
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/models/query.py", line 1316, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/models/query.py", line 111, in __iter__
    for row in compiler.results_iter(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size):
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/models/sql/compiler.py", line 1115, in results_iter
    results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/models/sql/compiler.py", line 1160, in execute_sql
    cursor.execute(sql, params)
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/backends/utils.py", line 99, in execute
    return super().execute(sql, params)
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/.env/lib/python3.6/site-packages/Django-3.2-py3.6.egg/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/.env/lib/python3.6/site-packages/sql_server/pyodbc/base.py", line 553, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Each GROUP BY expression must contain at least one column that is not an outer reference. (164) (SQLExecDirectW)')

Invalid usage of the option FIRST in the FETCH statement.

All of a sudden I recently started having this error occur on any django ORM operation that I use:

[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Invalid usage of the option FIRST in the FETCH statement. (153) (SQLExecDirectW); [42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Statement(s) could not be prepared. (8180)

This is what the query looks like in the django traceback:

QUERY = 'SELECT [Entity].[Audit], [Entity].[Class], [Entity].[FromArrow], [Entity].[FromAssc], [Entity].[ID], [Entity].[Name], [Entity].[SeqNum], [Entity].[Properties], [Entity].[ToArrow], [Entity].[ToAssc], [Entity].[Type], [Entity].[UpdateDate], [Entity].[SAGuid], [Entity].[PropType], [Entity].[ShortProps], [Entity].[Activity], [Entity].[CRID], [Entity].[CRURL], [Entity].[CRROOTURL], [Entity].[Version] FROM [Entity] WHERE [Entity].[ID] = %s FETCH FIRST 21 ROWS ONLY' - PARAMS = (1,)

This is the line of code that is being executed where the error occurs:

Entity.objects.using(encyclopedia).get(pk=pk)

Here is my database connection from settings.py (we are on an internal network, so it authenticates using my built in windows credentials):

{
    'ENGINE': 'sql_server.pyodbc',
    'NAME': 'database',
    'HOST': 'server',
    'OPTIONS': {
        'driver': 'ODBC Driver 13 for SQL Server',
    }
} 

Here are the details of my application installation and versions:

django-mssql-backend==2.8.1
Django==3.0.7
MS SQL Server Version==14.0.3238.1

I've been using this package on these same databases in the past, and I have not seen this issue occur before. This makes me think that its something I must have done, but I can't think of what would be causing it. Any help would be greatly appreciated! Thanks!

FYI: new Stack Overflow tag

Just an FYI, I've created a Stack Overflow tag for the django-mssql-backend package:

https://stackoverflow.com/questions/tagged/django-mssql-backend

I created the django-pyodbc-azure tag years ago, and have updated the description to note that it is no longer maintained, and point people here.

Improving MSSQL backend support with Microsoft

Hello,

My name is Warren, a development manager for several Microsoft-backed MSSQL connectivity teams. I'd like to collaborate with this repo's maintainers on how Microsoft can best improve MSSQL backend support. I'm hoping to discuss this opportunity at your convenience - please reach out to me at [email protected] and we can discuss.

Thank you!

[ODBC Driver 13 for SQL Server][SQL Server]'SYSDATETIME' is not a recognized built-in function name

Environment

OS: Windows 2003 (NT 5.2) SP1 3790
DB: Microsoft SQL Server Standard 2005 x86

Python: 3.7.5 x64
pyodbc: 4.0.27
django-mssql-backend: 2.3.0
OS: Windows 10 x64
DB: MS SQL
driver: ODBC Driver 13 for SQL Server

Issue

I'm trying to connect to the database with following settings_dict:

        {'ENGINE': 'sql_server.pyodbc', 'HOST': '10.11.241.16', 'PORT': 1433, 'USER': 'rsafeadmin', 'PASSWORD': 'Qwerty123', 'OPTIONS': {'driver': 'ODBC Driver 13 for SQL Server'}, 'TIME_ZONE': 'America/Chicago', 'CONN_MAX_AGE': 0, 'AUTOCOMMIT': True, 'NAME': 'master'}

Expected behavior:
Successfull connection
Observed behavior:
[ODBC Driver 13 for SQL Server][SQL Server]'SYSDATETIME' is not a recognized built-in function name. (195) (SQLExecDirectW)

SQL Server 2005 has no function SYSDATETIME. It has GETDATE instead. So to manage this issue i had to change base.py (line 356):

        val = cursor.execute('SELECT SYSDATETIME()').fetchone()[0]

I simply replaced with:

        try:
            val = cursor.execute('SELECT SYSDATETIME()').fetchone()[0]
        except Exception:
            # unable to find SYSDATETIME command, invoke GETDATE
            val = cursor.execute('SELECT GETDATE()').fetchone()[0]

Azure SQL Server Database AD token - Django

I have successfully connected the Azure SQL Server using AccessToken in the pyodbc. Here I didn't use a username or password to connect to the database. Instead of that, I used attrs_before for pass token. Here I am generating the token automatically.

Token Generation:

identity_endpoint = os.environ["IDENTITY_ENDPOINT"]
identity_header = os.environ["IDENTITY_HEADER"]

def get_bearer_token(resource_uri): #Automattically token will generate
    token_auth_uri = f"{identity_endpoint}?resource={resource_uri}&api-version=2019-08-01"
    head_msi = {'X-IDENTITY-HEADER': identity_header}

    resp = requests.get(token_auth_uri, headers=head_msi)
    access_token = resp.json()['access_token']
    return access_token

Pyodbc Connection:

accessToken = bytes(get_bearer_token("https://database.windows.net/"), 'utf-8');
exptoken = b"";
for i in accessToken:
    exptoken += bytes({i});
    exptoken += bytes(1);
tokenstruct = struct.pack("=i", len(exptoken)) + exptoken;

conn = pyodbc.connect(
      "Driver={ODBC Driver 17 for SQL Server};Server=yoursqlserver.database.windows.net,1433;Database=dbName",
      attrs_before={1256: bytearray(tokenstruct)});

Now the problem is how to use the same configuration using this package to connecting DB? We can't have a username or password to connect the database when using the azure token.

GitHub repository's website should point at own PyPI page

The "website" for this GitHub repo is still set to be https://pypi.org/project/django-pyodbc-azure/, which means that the project's front page links to it at the top:

You (@OskarPersson ?) should probably edit that to point to https://pypi.org/project/django-mssql-backend/ instead to reduce / avoid confusion.

Thanks for your work on this 👍

Migration failure making unique column non-nullable: Could not drop constraint

Problem

Modifying a field which is nullable and unique, to no longer be nullable, results in the following error during migration:

...
  File "...\venv.37\lib\site-packages\django\db\backends\base\schema.py", line 535, in alter_field
    old_db_params, new_db_params, strict)
  File "...\venv.37\lib\site-packages\sql_server\pyodbc\schema.py", line 270, in _alter_field
    self._delete_unique_constraints(model, old_field, new_field, strict)
  File "...\venv.37\lib\site-packages\sql_server\pyodbc\schema.py", line 479, in _delete_unique_constraints
    self.execute(self._delete_constraint_sql(self.sql_delete_unique, model, constraint_name))
  File "...\venv.37\lib\site-packages\sql_server\pyodbc\schema.py", line 681, in execute
    cursor.execute(sql, params)
  File "...\venv.37\lib\site-packages\django\db\backends\utils.py", line 99, in execute
    return super().execute(sql, params)
  File "...\venv.37\lib\site-packages\django\db\backends\utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "...\venv.37\lib\site-packages\django\db\backends\utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "...\venv.37\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "...\venv.37\lib\site-packages\django\db\utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "...\venv.37\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "...\venv.37\lib\site-packages\sql_server\pyodbc\base.py", line 559, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]'myapp_mymodel_my_field_a717c0d4_uniq' is not a constraint. (3728) (SQLExecDirectW); [42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Could not drop constraint. See previous errors. (3727)")

Reproduction steps

This is reproducible in a minimal Django project with these dependencies:

Django==2.2.10
django-mssql-backend==2.4.2
pyodbc==4.0.30
pytz==2019.3
sqlparse==0.3.0

and the following single model:

class MyModel(models.Model):
    my_field = models.CharField(max_length=100, null=True, unique=True)

1. makemigrations to get 0001 which creates this model with 1 field that is null=True, unique=True
2. migrate which does the following (note: the uniqueness is enforced by a filtered unique index rather than by a constraint on the table, since this change: #1)

CREATE TABLE [myapp_mymodel] ([id] int IDENTITY (1, 1) NOT NULL PRIMARY KEY, [my_field] nvarchar(100) NULL)
CREATE UNIQUE INDEX [myapp_mymodel_my_field_a717c0d4_uniq] ON [myapp_mymodel]([my_field]) WHERE [my_field] IS NOT NULL

3. then modify my_field in models.py by removing null=True
4. makemigrations to get 0002 which does an AlterField
5. migrate which tries to do the following in preparation (before altering the nullability)

ALTER TABLE [myapp_mymodel] DROP CONSTRAINT [myapp_mymodel_my_field_a717c0d4_uniq]

but blows up with the error & stack trace above, because myapp_mymodel_my_field_a717c0d4_uniq is an index, not a constraint.

Cause

I believe this is a knock-on effect of #1 and this is as far as I got with diagnosing the problem:

• _alter_field calls _delete_unique_constraints because of the 'Nullability change', which calls _constraint_names(..., unique=True)
• _constraint_names returns ['myapp_mymodel_my_field_a717c0d4_uniq'] despite this being an index, and then _delete_unique_constraints tries to delete it using sql_delete_unique (i.e. ALTER TABLE ... DROP CONSTRAINT ...) which fails because it is just an index

Possible solution

Perhaps _delete_unique_constraints should call _constraint_names twice as follows:

• once with unique=True, index=False and DROP those using ALTER TABLE... DROP CONSTRAINT
• once with unique=True, index=True and DROP those using DROP INDEX ...

Does that sound reasonable?

remove limitation: "Altering a model field from or to AutoField at migration"

First, let me say a huge thank you for picking up this project!
I haven't had a chance to look at this fork in any detail yet, so please forgive any duplication. I did search issues and this might overlap with #38 and #39.

I was wondering if you had looked at michiya#189 which I submitted back in November 2018 that removes the "Altering a model field from or to AutoField at migration" limitation.

Large queries cause errors

I have a product where around 300k instances are created in an import.

Before importing (which I fear may fail as well), I perform an existence check against a single field in the form MyModel.objects.filter(my_field__in=new_values).exists() where new_values is a set of strings.

Here is the stack trace of the issue:

Traceback (most recent call last):
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/sql_server/pyodbc/base.py", line 536, in execute
    return self.cursor.execute(sql, params)
pyodbc.ProgrammingError: ('The SQL contains 6623 parameter markers, but 137695 parameters were supplied', 'HY000')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/home/jk/git/my-project/core/management/command_util.py", line 117, in wrapper
    return f(*args, **kwargs)
  File "/home/jk/git/my-project/core/management/command_util.py", line 145, in wrapper
    return f(*args, **options)
  File "/home/jk/git/my-project/core/management/command_util.py", line 117, in wrapper
    return f(*args, **kwargs)
  File "/home/jk/git/my-project/core/management/commands/my_import_command.py", line 159, in handle
    verify_only=verify_only,
  File "/home/jk/git/my-project/core/util.py", line 150, in import_and_verify_terms
    import_and_log_new_terms(user, model, new_terms[search_type], source_description)
  File "/home/jk/git/my-project/core/util.py", line 228, in import_and_log_new_terms
    if model.objects.filter(term__in=new_terms).exists():
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/models/query.py", line 766, in exists
    return self.query.has_results(using=self.db)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
    return compiler.has_results()
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1070, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
    cursor.execute(sql, params)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/backends/utils.py", line 99, in execute
    return super().execute(sql, params)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jk/.virtualenvs/my-project/lib/python3.6/site-packages/sql_server/pyodbc/base.py", line 536, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: ('The SQL contains 6623 parameter markers, but 137695 parameters were supplied', 'HY000')

Contribution

Hi. do you mind add a Contribution file, so people can contribute to this repo?

Migration fails when altering a Nullable OneToOneField to FK field

It works ok if Null=false on the field originally. Looks like it fails to find the unique constraint that was in place, /pyodbc/schema.py->_alter_field, 267-278 (# Has unique been removed?)

I presume a previous operation had already removed it?
Note this would have worked previously, as this is an old migration that is failing, on django-pyodbc-azure. I'm not sure about previous versions of django-mssql-backend

Support for Django 3.1.x?

Just writing to see if there are plans for supporting Django 3.1.x

We were considering moving - but when we attempted to do so we ran into a lot of MSSQL errors (varying in degree and varying across applications).

Downgrading to 3.0.x fixes all issues. Just curious if there is a timeline for 3.1.x versions? I checked online but couldn't find any mention of it anywhere.

Compatibility with django 1.11?

We would like to switch to this fork, as it is actively developed, and also because it has the regex support that was missing in the original repository.
As we still need to support django 1.11 and I couldn't find the information: can you say if the current version is still compatible with django 1.11? And in case it is not, would it be possible to merge the regex support into the 1.11 branch and make a pypi release (I could make a PR if you want - basically a copy of the original PR)? It would be more convenient than patching the released version...

Thanks!

Migrations for auth fails on Django 3.1: KeyError: 'deferrable'

I started a fresh project with Django 3.1. and the latest django-mssql-backend. Running python manage.py migrate auth fails on the "0008" migration:

  Applying auth.0008_alter_user_username_max_length...Traceback (most recent call last):                           File "manage.py", line 22, in <module>
    main()
  File "manage.py", line 18, in main                                                                                 execute_from_command_line(sys.argv)
...
  File "/.../lib/python3.6/site-packages/django/db/backends/base/schema.py", line 572, in alter_field
    old_db_params, new_db_params, strict)
  File "/.../lib/python3.6/site-packages/sql_server/pyodbc/schema.py", line 479, in _alter_field
    self.execute(self._create_unique_sql(model, columns=[old_field.column]))
  File "/.../lib/python3.6/site-packages/sql_server/pyodbc/schema.py", line 863, in execute
    sql = str(sql)
  File "/.../venv/lib/python3.6/site-packages/django/db/backends/ddl_references.py", line 200, in __str__
    return self.template % self.parts
KeyError: 'deferrable'

While experimenting a little, downgrading to Django 2.2 fixed the issue with the problematic migrations, and after upgrading back to 3.1 I was able to run the remaining migrations (those that are new in Django 3). Digging into the code base the problem seems quite straightforward: the django.db.backends.ddl_references.Statement seems indeed to expect a deferrable keyword argument, and after hacking the source code so that an empty string was given as deferrable, the migrations worked just fine. Unfortunately I'm not familiar with mssql or this library, and can't simply attempt a pull request for this.

BooleanFields returning as 1/0 (instead of True/False)

Something I've noticed occurring is that BooleanFields are returning 0/1 instead of False/True. The field definitions haven't changed, and pointing the app at another DB with the same data returns the proper False/True.

Any ideas for better debugging of this?

This may be as simple as bringing this PR over marcogiusti@d6c5c49

Django 3.0 Support?

Just curious if there is a version of this out there (or plans for this) that will be supporting Django 3.0.

Currently looking to tryout the 3.0 features and since this forked worked with 2.2 I wasn't sure if 3.0 support would be added.

on_delete Action is not set

Hi

in my model I have this line:
deliveryorder = models.ForeignKey(DeliveryNote, blank=True, null=True, on_delete=models.SET_NULL,

But in SSMS the Delete Rule is set to No Action. So if I delete the parent object I get a SQL REFERENCE error.
After manuell changing the Delete Rule to Set Null I was able to delete the parent.

django.db.utils.NotSupportedError: This backend does not support window expressions.

When executing a query with
.annotate(rank=Window(...))
I get the error django.db.utils.NotSupportedError: This backend does not support window expressions.

I am running
Django==3.0.3
django-mssql-backend==2.8.1
pyodbc==4.0.30

Exactly the same query used to work with previous version of django-pyodbc-azure and django 2.x

I just gave it a try: If in django-mssql-backend/sql_server/pyodbc/features.py I add the attribute supports_over_clause

class DatabaseFeatures(BaseDatabaseFeatures):
supports_over_clause = True
...

it works.

Was this just missed or do I oversee anything?

Uncaught transient login error on Azure SQL Database serverless: Unable to complete login process due to delay in login response (258)

When trying to connect to an auto-paused Azure SQL Database serverless instance, the following error occurs:

Django Version:	3.0.11
Exception Type:	OperationalError
Exception Value:	('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Timeout error [258]. (258) (SQLDriverConnect); [08001] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 17 for SQL Server]Invalid connection string attribute (0); [08001] [Microsoft][ODBC Driver 17 for SQL Server]Unable to complete login process due to delay in login response (258)')

On retrying the request (presumably once the Azure SQL Database has resumed) the issue resolves itself.

Any idea what the issue could be here? Is "Invalid connection string attribute (0);" relevant?

Here are the value of the locals at sql_server\pyodbc\base.py:314 when the error occurs:

var	value
backoff_time	5
conn	None
conn_params	{'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'ENGINE': 'sql_server.pyodbc', 'HOST': '[REDACTED].database.windows.net', 'NAME': '[REDACTED]', 'OPTIONS': {'driver': 'ODBC Driver 17 for SQL Server'}, 'PASSWORD': '[REDACTED]', 'PORT': '', 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIRROR': None, 'NAME': None}, 'TIME_ZONE': None, 'USER': '[REDACTED]@[REDACTED]'}
connstr	('DRIVER=ODBC Driver 17 for SQL ' 'Server;SERVER=[REDACTED].database.windows.net;UID=[REDACTED]@[REDACTED];PWD=[REDACTED];DATABASE=[REDACTED];MARS_Connection=yes')
cstr_parts	{'DATABASE': '[REDACTED]', 'DRIVER': 'ODBC Driver 17 for SQL Server', 'MARS_Connection': 'yes', 'PWD': '[REDACTED]', 'SERVER': '[REDACTED].database.windows.net', 'UID': '[REDACTED]@[REDACTED]'}
database	'[REDACTED]'
driver	'ODBC Driver 17 for SQL Server'
dsn	None
error_number	'49920'
host	'[REDACTED].database.windows.net'
ms_drivers	re.compile('^ODBC Driver .* for SQL Server$|^SQL Server Native Client')
need_to_retry	False
options	{'driver': 'ODBC Driver 17 for SQL Server'}
password	'[REDACTED]'
port	''
query_timeout	0
retries	5
retry_count	0
self	<sql_server.pyodbc.base.DatabaseWrapper object at 0x0000018572CC6208>
timeout	0
unicode_results	False
user	'[REDACTED]@[REDACTED]'

One possible solution: Should '258' be added to sql_server.pyodbc.base.DatabaseWrapper._transient_error_numbers?

django.core.exceptions.ImproperlyConfigured: The database driver doesn't support modern datatime types.

Hi Guys,

I know this error has been raised in the original repo, and setting the freetds version to 7.3 worked.
But since I switched to use this library after upgrading Django to 2.2, I am getting this error back, and I cannot solve it on an Ubuntu box.

Tried: version 2.5.0 to 2.7.0, same problem.

However, it is working fine on my Mac so I really don't know what the real problem here.

File "/srv/www/tigerpaw_webui/lib/python3.5/site-packages/sql_server/pyodbc/base.py", line 358, in init_connection_state
    "The database driver doesn't support modern datatime types.")
django.core.exceptions.ImproperlyConfigured: The database driver doesn't support modern datatime types.

No release notes

The current release (2.2.0) is not on the release page of this repository.
Furthermore I cannot find release notes anywhere.

sql_flush method doesn't accept the reset_sequences keyword argument

Environment
Django Version: 3.1.2
PyODBC Version: 4.0.30
Django MSSQL Backend Version: Master branch from github

Description

When running tests using django.test.TransactionTestCase as the test case base class, the test runner attempts to use sql_flush to reset the contents of the test database. One of the parameters to this function is reset_sequences which was introduced in Django 3.1 (django/django#12733). The sql_flush method in sql_server.pyodbc.operations.DatabaseOperations does not accept this new syntax and still uses the older method of specifying the sequences to reset (in the sequences parameter). This prevents the test suite from running properly.

Reproducing Bug

Attached is a simple Django project that reproduces the error.

bug_sample.tar.gz

bulk_update fails when a column is to be updated to None (NULL)

Django: 2.2.17

The error I get when attempting to bulk update a column to None is:

...
  File "/.../site-packages/django/db/models/query.py", line 525, in bulk_update
    self.filter(pk__in=pks).update(**update_kwargs)
  File "/.../site-packages/django/db/models/query.py", line 741, in update
    rows = query.get_compiler(self.db).execute_sql(CURSOR)
...
django.db.utils.ProgrammingError: ('42000', '[42000] [FreeTDS][SQL Server]At least one of the result expressions in a CASE specification must be an expression other than the NULL constant. (8133) (SQLExecDirectW)')

Bring forward abandoned PRs?

Hi there, there are a few PRs on the original repo https://github.com/michiya/django-pyodbc-azure/ that would be useful here. Without too deep a dive:
michiya#188 which seems based on one of your (@OskarPersson) previous PRs.
and,
michiya#189

Nullable unique constraints not reinstated correctly

Problem

Unique constraints which are now implemented using a filtered UNIQUE INDEX (to get desired behaviour for multiple NULLs) are not correctly reinstated after an AlterField e.g. if the column type is changed.

This happens in at least these cases:

1. after changing the type of a field which has null=True and unique=True
2. after changing the type of a field involved in a unique_together

The constraint is reinstated using ALTER TABLE ... ADD CONSTRAINT instead of CREATE UNIQUE INDEX ... WHERE x is NOT NULL

This means the unique constraint is no longer getting the ANSI-compliant NULL behaviour which was recently introduced by PR #1 (for single columns) and PR #24 (for unique_together) to allow for multiple NULL values.

Symptoms

If there are already multiple NULL values in the table, the migration will fail because reinstating the over-strict constraint fails with an error like this:

django.db.utils.IntegrityError: ('23000', "[23000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.myapp_foomodel' and the index name 'myapp_foomodel_field_a_d7c02f64_uniq'. The duplicate key value is (<NULL>). (1505) (SQLExecDirectW); [23000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Could not create constraint or index. See previous errors. (1750); [23000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The statement has been terminated. (3621)")

(note: despite the error message saying CREATE UNIQUE INDEX this error is thrown during execution of ALTER TABLE ... ADD CONSTRAINT - presumably because internally in MSSQL this results in running CREATE UNIQUE INDEX without any conditions/filter)

Or if there aren't multiple NULLs then the migration may 'succeed' but:

• have the wrong runtime behaviour with multiple NULLs
• potentially fail on a future migration step - for example in the 2nd case with unique_together, if in future an AlterUniqueTogether tries to add/remove a field, it will first try to delete the existing constraint as if it were a UNIQUE INDEX (which it should be) but actually it's a CONSTRAINT so the later migration will fail like this:

django.db.utils.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]An explicit DROP INDEX is not allowed on index 'bar_baz_deadbeef_uniq'. It is being used for UNIQUE KEY constraint enforcement. (3723) (SQLExecDirectW)")

Reproduction steps

I can reproduce this in a minimal Django project with using:

Django==2.2.10
django-mssql-backend==2.8.0
pyodbc==4.0.30
pytz==2019.3
sqlparse==0.3.0

1. Add the following model and run makemigrations and then migrate

class FooModel(models.Model):
    field_a = models.CharField(max_length=10, null=True, unique=True)

2. Create 2 rows which both have field_a as NULL
3. Edit max_length to say 15, makemigrations again and run migrate
4. This explodes with an error as in the first example above

To repro the 2nd example:

1. Add field_x = models.CharField(max_length=10) which is in a unique_together with at least 1 other field e.g. field_y
2. Then do AlterField on field_x to change it to max_length=15 (this will succeed if there are no rows in the table)
3. Then do AlterUniqueTogether to add a 3rd field field_z to the unique_together
4. This explodes with an error like the second example above

Cause

As far as I can tell the issue is that #24 changed the way such unique indexes were implemented, but did not ensure that _alter_field reinstates them in the same way, after they are temporarily dropped in order to make changes to the field.

Proposed solution

In _alter_field under # Restore an index, SQL Server requires explicit restoration, the handling of unique=True/unique_together needs to reinstate filtered UNIQUE INDEX rather than a CONSTRAINT in the relevant cases.

I am working on a PR for this.

supported django version match

I believe that django-pyodbc-azure major release version was a match with supported django version. I liked it. Could you consider bumping version to match supported django - 3.0 ?

Windows authentication support?

For some reason not able to make this work with Windows authentication.

Tries setting extra params and also using FreeTDS.

Weirdly using pyodbc directly works fine:

cnxn = pyodbc.connect('DRIVER=FreeTDS;SERVER=****;PORT=1433;DATABASE=***;UID=***.local\username;PWD=******;')

Error I keep getting is

OperationalError: ('08S01', '[08S01] [FreeTDS][SQL Server]Unable to connect: Adaptive Server is unavailable or does not exist (20009) (SQLDriverConnect)')

SQL query using nvarchar instead of char

Hello! I have a legacy table that has a column of CHAR(8) datatype. There is an index on this column which makes look-ups very fast even though it is a large table.

Here is a snippet of the model generated through inspectdb:

class OrderBuild(models.Model):
    ord_no = models.CharField(max_length=8)
    id = models.AutoField(db_column='ID', unique=True, primary_key=True)

    class Meta:
        managed = False
        db_table = 'imorders_sql'

I can query this model on the ord_no column like so:

order="12345678"
OrderBuild.objects.filter(ord_no=order)

This appears to work fine, however, I noticed it seemed to execute much slower than it should be. I noticed that when using the SSMS Event Profiler to track that execution of the query, the filter parameter is declared as nvarchar(16).

declare @p1 int set @p1=1 exec sp_prepexec @p1 output,N'@P1 nvarchar(16)',N'SELECT [imorders_sql].[ord_no], [imorders_sql].[item_no], [imorders_sql].[qty], [imorders_sql].[qty_per], [imorders_sql].[ID] FROM [imorders_sql] WHERE [imorders_sql].[ord_no] = @P1',N'12345678' select @p1

Unfortunately, because the parameter is an nvarchar(16) rather than char(8), SQL Server cannot use an index seek to query the data and rather does an index scan which takes significantly longer in my case.

I can work around this problem by doing something like:

OrderBuild.objects.extra(
    where=('ord_no = CAST(%s as CHAR(8))',),
    params=(order,))

This forces the parameter to be converted to the correct datatype and SQL Server is able to use an Index Seek rather than scan making it execute much faster. However, this is not a great long-term solution because extra() may be deprecated by Django in the future.

I am wondering why it is using nvarchar(16) rather than char(8) and if there is any way besides extra() to force the query to parameterize with a char(8) datatype instead. Thanks!

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

install regex does not seem to work

Hello, I tried "install_regex_clr {database_name}" from manage.py but get an "Unknown command: 'install_regex_clr " in manage.py.
Can anyone advise? Do I need to add anything to the INSTALLED_APPS? django-mssql-backend works great otherwise

Regression: index dropped in migration and not recreated

Problem

In the following cases (possibly more) any indexes on the column are dropped, but aren't recreated afterwards:
(a) when a column is made null-able (i.e. adding null=True)
(b) when a column is renamed
(c) when a table is renamed

This is quite a major issue because it is silently leaving the database in the wrong state - the migration doesn't fail, but now certain columns with db_index=True won't actually have an index which could cause very slow queries.

Cause

As far as I can see this is a triple-regression introduced by 2e60754 (for #24).

That commit added new calls to _delete_indexes in 2 places causing the first two cases listed above:

(a) 2e6075456a#diff-e4c1520d8b49ccbf46382bd2eed4e740R400
(b) 2e6075456a#diff-e4c1520d8b49ccbf46382bd2eed4e740R334-R335

and an explicit index deletion here before renaming a table causing the third case:

(c) 2e6075456a#diff-e4c1520d8b49ccbf46382bd2eed4e740R222-R225

but in none of those cases is the index re-instated afterwards, even if the field still has db_index=True. Index restoration only happens in certain specific cases (e.g. type change / removal of nullability) which doesn't include the above 3 cases.

Reproduction

See the 3 tests I added on this branch:
master...sparrowt:issue58-index-reinstatement
which fail due to the bugs described above, but pass if run on django-mssql-backend v2.4.2 (before #24)

Failure with .annotate() and Concat()

Thanks for putting together this update.

I've run across a bug when I run the follow statement:
People.objects.annotate(full_name=Concat('last_name', Value(', '), 'first_name'), other_name=Concat('first_name', Value(' '), 'last_name'))

It generates a ProgrammingError:
ProgrammingError: ('42000', "[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Column 'core_people.last_name' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause. (8120) (SQLExecDirectW); [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Column 'core_people.first_name' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause. (8120);

This is running on SQL Server Express 2012.

Any ideas? The code runs fine on MySQL so it's not just a code issue.

Looking closer, this seems similar to michiya#94 and michiya#183. ichiya#94](michiya#94) points to a possible fix at azure-2.0...rnovacek:fix-concat-group-by-new

MARS_Connection disabled on non-windows servers

Lines 290-291 of sql_server/pyodbc/base.py do not enable MARS_Connection unless the web server is running windows - I think this is an over-sight?

if ms_drivers.match(driver) and os.name == 'nt': cstr_parts['MARS_Connection'] = 'yes'

RenameModel does not bring unique_together constraints to new table

Cursor.execute() with params doesn't seem to work with pyodbc

Environment
To diagnose, we usually need to know the following, including version numbers. On Windows, be
sure to specify 32-bit Python or 64-bit:

Python: 3.7.7
pyodbc: 4.0.30
OS: Debian 9 Linux (container through win64) (python:3.7-slim-stretch)
DB: SQL Server 2016
driver: ODBC Driver 17 for SQL Server

Problem:
Execute without parameters seems to work fine, issue comes when parameters are added in the manner specified in the pyodbc documentation. e.g.:
crsr.execute("{CALL usp_UpdateFirstName (?,?)}", params)

Tried with a single and multiple parameters with params as a list, tuple (as suggested from docs), raw strings.
Also tried with the query inside curly braces and without - same issue.

The line at which this fails implies that the format_sql() method is expecting an sql string with '%s' placeholders rather than '?' however I have also tried using %s placeholders which has resulted in a different error:

('The SQL contains 0 parameter markers, but 1 parameters were supplied', 'HY000')

The pyodbc package specifies '?' as the required placeholder, so I think something needs to be adjusted here.

Original error:
TypeError: not all arguments converted during string formatting
Traceback (most recent call last):
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
backend_dev_vet | response = get_response(request)
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
backend_dev_vet | response = self.process_exception_by_middleware(e, request)
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
backend_dev_vet | response = wrapped_callback(request, *callback_args, **callback_kwargs)
backend_dev_vet | File "/opt/project/backend/iservices/views.py", line 38, in testprocview
backend_dev_vet | }""", [f_id])
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 99, in execute
backend_dev_vet | return super().execute(sql, params)
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
backend_dev_vet | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
backend_dev_vet | return executor(sql, params, many, context)
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
backend_dev_vet | return self.cursor.execute(sql, params)
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/sql_server/pyodbc/base.py", line 549, in execute
backend_dev_vet | sql = self.format_sql(sql, params)
backend_dev_vet | File "/usr/local/lib/python3.7/site-packages/sql_server/pyodbc/base.py", line 517, in format_sql
backend_dev_vet | sql = sql % tuple('?' * len(params))

Can't open lib 'ODBC Driver 17 for SQL Server' : file not found in Heroku

In Heroku, it seems not to find the Driver.

runtime.txt

python-3.7.3

Aptfile (I added the apt build pack to add apt dependencies)

unixodbc
unixodbc-dev
python-pyodbc
libsqliteodbc

requirements.txt

Django==3.0.4
djangorestframework==3.10.3
django-environ==0.4.5
pre-commit==2.1.1
black==19.10b0
flake8==3.7.9
pre-commit-hooks==2.5.0
pyodbc==4.0.30
django-mssql-backend==2.8.0
serpy==0.3.1
gunicorn==20.0.4

settings.py:DATABASES

NAME_DRIVER = "ODBC Driver 17 for SQL Server"
DATABASES = {
    "default": {
        "ENGINE": "sql_server.pyodbc",
        "NAME": "",
        "USER": "",
        "PASSWORD": "",
        "HOST": "",
        "PORT": 9876,
        "OPTIONS": {
            "driver": NAME_DRIVER,
            # "host_is_server": True,
            "unicode_results": True,
        },
    }
}

UPDATE: Similar problem

https://stackoverflow.com/questions/54625929/cannot-connect-to-sql-server-from-django-on-heroku

Specify Authentication option in connection string

First of all: brilliant work. I really appreciate the effort and it cannot be said often enough how great it is that the development of a Django database backend for SQL server continues.

At our company, users have access to the database based on their Azure AD credentials using Multi-Factor-Authentication. Thus, what we need is to be able to set Authentication=ActiveDirectoryInteractive in the connection string.

First, the current configuration will, if given a user, always include the PWD option in the connection string. This is suboptimal as when the connection string specifies Authentication=ActiveDirectoryInteractive it will not allow PWD to be set.

This could be remedied currently by simply not specifying a user or a password. (Which is, however, not as comfortable for the user as it could be because the login site will prefill the username if you specify it in the connection string.)

Now, I use the following configuration:

DATABASES = {
    "default": {
        "ENGINE": "sql_server.pyodbc",
        "NAME": "dbname",
        "HOST": "we.database.net",
        "OPTIONS": {
            "driver": "ODBC Driver 17 for SQL Server",
            "extra_params": "Authentication=ActiveDirectoryInteractive",
        },
    }
}

But then the ODBC driver throws an error:

[FA001] [Microsoft][ODBC Driver 17 for SQL Server]Invalid connection string attribute (0); [FA001] [Microsoft][ODBC Driver 17 for SQL Server]Cannot use Authentication option with Integrated Security option.

This leaves me with a problem as I cannot change the Integrated Security option in the connection string.

I would love to extend the connection string code. I think this is the appropriate part.

Still, I have some quesitons:

• Would you accept a PR that implements the Authentication parameter, probably implemented as an additional setting in the options for the backend?
• When cloning the repository and working with it, is there any document that helps me get started?

I have this problem with the sqlserver connection

File "manage.py", line 22, in
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management_init_.py", line 381, in execute_from_command_line
utility.execute()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management_init_.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management\base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management\base.py", line 364, in execute
output = self.handle(*args, **options)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management\commands\showmigrations.py", line 48, in handle
return self.show_list(connection, options['app_label'])
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management\commands\showmigrations.py", line 67, in show_list
loader = MigrationLoader(connection, ignore_no_migrations=True)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\loader.py", line 49, in init
self.build_graph()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\loader.py", line 212, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\recorder.py", line 73, in applied_migrations
if self.has_table():
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\recorder.py", line 56, in has_table
return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 256, in cursor
return self._cursor()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\sql_server\pyodbc\base.py", line 220, in _cursor
conn = super()._cursor()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 233, in _cursor
self.ensure_connection()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 217, in ensure_connection
self.connect()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 217, in ensure_connection
self.connect()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\sql_server\pyodbc\base.py", line 316, in get_new_connection
timeout=timeout)
django.db.utils.OperationalError: ('08001', '[08001] [Microsoft][ODBC Driver 13 for SQL Server]TCP Provider: Tiempo de espera de la operación de espera agotado.\r\n (258) (SQLDriverConnect); [08001
] [Microsoft][ODBC Driver 13 for SQL Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 13 for SQL Server]Invalid connection string attribute (0); [08001] [Microsoft][ODBC Driver 13
for SQL Server]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and
if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. (258)')

C:\Users\Paolo\OneDrive - SAGA COMERCIAL\Escritorio\Django>py manage.py migrate
Traceback (most recent call last):
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 217, in ensure_connection
self.connect()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\sql_server\pyodbc\base.py", line 316, in get_new_connection
timeout=timeout)
pyodbc.OperationalError: ('08001', '[08001] [Microsoft][ODBC Driver 13 for SQL Server]TCP Provider: Tiempo de espera de la operación de espera agotado.\r\n (258) (SQLDriverConnect); [08001] [Micros
oft][ODBC Driver 13 for SQL Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 13 for SQL Server]Invalid connection string attribute (0); [08001] [Microsoft][ODBC Driver 13 for SQL S
erver]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Se
rver is configured to allow remote connections. For more information see SQL Server Books Online. (258)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "manage.py", line 22, in
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management_init_.py", line 381, in execute_from_command_line
utility.execute()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management_init_.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management\base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management\base.py", line 364, in execute
output = self.handle(*args, **options)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management\base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\core\management\commands\migrate.py", line 87, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\executor.py", line 18, in init
self.loader = MigrationLoader(self.connection)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\loader.py", line 49, in init
self.build_graph()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\loader.py", line 212, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\recorder.py", line 73, in applied_migrations
if self.has_table():
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\migrations\recorder.py", line 56, in has_table
return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 256, in cursor
return self._cursor()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\sql_server\pyodbc\base.py", line 220, in _cursor
conn = super()._cursor()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 233, in _cursor
self.ensure_connection()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 217, in ensure_connection
self.connect()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 217, in ensure_connection
self.connect()
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\django\db\backends\base\base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "C:\Users\Paolo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\sql_server\pyodbc\base.py", line 316, in get_new_connection
timeout=timeout)
django.db.utils.OperationalError: ('08001', '[08001] [Microsoft][ODBC Driver 13 for SQL Server]TCP Provider: Tiempo de espera de la operación de espera agotado.\r\n (258) (SQLDriverConnect); [08001
] [Microsoft][ODBC Driver 13 for SQL Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 13 for SQL Server]Invalid connection string attribute (0); [08001] [Microsoft][ODBC Driver 13
for SQL Server]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and
if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. (258)')

Django 2.2.8 - 'sql_server.pyodbc' isn't an available database backend.

Hello,

I have tried all of the available examples online to get MSSQL support for Django 2.2. It seems that people have had success. Please let me know if I am making an obvious mistake or if django 2.2 is still lacking support.

Thank you in advance.

Current versions
Windows 10
django==2.2.8
django-mssql-backend==2.5.0
pyodbc==4.0.28

Django DB Config

DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc',
        # 'ENGINE': 'sqlsever_ado',
        # 'ENGINE': 'django.db.backends.mssql',
        # 'ENGINE': 'django_pyodbc',
        # 'ENGINE': 'django-pyodbc-azure',
        'NAME': 'db',
        'USER': 'sa',
        'PASSWORD': 'pw',
        'HOST': 'localhost',
        'PORT': '1433',
        'ATOMIC_REQUESTS': True,
        'OPTIONS': {
            'driver': 'ODBC Driver 13 for SQL Server',
        },
    }
}

Error
django.core.exceptions.ImproperlyConfigured: 'sql_server.pyodbc' isn't an available database backend.
Try using 'django.db.backends.XXX', where XXX is one of:
'mysql', 'oracle', 'postgresql', 'sqlite3'

Migration that sets unique nullable field to non-nullable fails

I've discovered this issue in a project where we are using djangorestframework-api-key.
This package has a migration that creates a CharField with unisque=True and null=True, the populates data for that field and afterwards it sets null=False. This is a pretty common scenario when you need to add non-nullable unique field to a pre-existing model.

Migrations of this kind fail with the following error:
42S11] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The operation failed because an index or statistics with name 'testapp_testalternullableinuniquefield_a_67b9d427_uniq' already exists on table 'testapp_testalternullableinuniquefield'. (1913) (SQLExecDirectW)

Oddly enough, this only happens the first time django tries to run the migration, but the second time it passes. It is not yet clear to me whether the indexes and constraints are in place after the migration passes.

I've been able consistently to reproduce the error by creating a migration with the following operations:

    operations = [
        migrations.CreateModel(
            name='TestAlterNullableInUniqueField',
            fields=[
                ('a', models.CharField(max_length=50, null=True, unique=True)),
            ],
        ),
        migrations.AlterField(
            model_name='testalternullableinuniquefield',
            name='a',
            field=models.CharField(max_length=50, unique=True),
        )
    ]

I would like to contribute a PR to fix this issue but at this point it is not clear to me how to generate a unit test for the project as I would like to test the migration itself. Any suggestions?

By the way, this error is preventing us from normally run the tests in our project since test database cannot be created due to this error.