mahmoudimus / sqlalchemy-citext

CITEXT type for SQLAlchemy

sqlalchemy-citext

Creates a SQLAlchemy user defined type to understand PostgreSQL's CIText extension.

Installation

This requires some kind of PostgreSQL compatible db-api driver already installed in order to work.

Make sure you have something like psycopg2 already installed.

pip install sqlalchemy-citext

Usage

from sqlalchemy import create_engine, MetaData, Integer
from sqlalchemy.schema import Column, Table
import sqlalchemy.orm as orm

from citext import CIText


engine = create_engine('postgresql://localhost/test_db')
meta = MetaData()

test_table = Table('test', meta,
    Column('id', Integer(), primary_key=True),
    Column('txt', CIText()))

conn = engine.connect()

meta.bind = conn
meta.drop_all()
meta.create_all()

class TestObj(object):
    def __init__(self, id_, txt):
        self.id = id_
        self.txt = txt

    def __repr__(self):
        return "TestObj(%r, %r)" % (self.id, self.txt)

orm.mapper(TestObj, test_table)
Session = orm.sessionmaker(bind=engine)
ses = Session()

to = TestObj(1, txt='FooFighter')
ses.add(to)
ses.commit()
row = ses.query(TestObj).filter(TestObj.txt == 'foofighter').all()
assert len(row) == 1
print row
ses.close()

License

sqlalchemy-citext is an MIT/BSD dual-Licensed library.

Contribute

• Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
• Fork the repository on GitHub to start making your changes to the master branch (or branch off of it).
• Write a test which shows that the bug was fixed or that the feature works as expected.
• Send a pull request and bug the maintainer until it gets merged and published.
• Make sure to add yourself to the author's file in setup.py and the Contributors section below :)

Contributors

• @mahmoudimus
• @vad
• @dstufft
• @brmzkw
• @graingert
• @cjmayo
• @libre-man

sqlalchemy-citext's Issues

The version specified ('1.5_0') is an invalid version

Running build on the version downloaded from PyPi:

sqlalchemy-citext-1.5-0 $ python setup.py build
/usr/lib/python3.7/site-packages/setuptools/dist.py:485: UserWarning: The version specified ('1.5_0') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
"details." % self.metadata.version

UserDefinedType CIText() will not produce a cache key because the ``cache_ok`` attribute is not set to True

I keep getting this error. Tried to use cache_ok = True in the Table Class mapping where column of type CIText() is defined. But it does not help.

UserDefinedType CIText() will not produce a cache key because the cache_ok attribute is not set to True. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)

Below are the package versions with python 3.10
sqlalchemy-citext 1.8.0
SQLAlchemy 1.4.31

---

The error goes away if I add,
cache_ok = True

in class CIText(types.Concatenable, types.UserDefinedType) defined in init.py file of citext package.
Is this the right way to handle this?

Support psycopg3

This library currently expects psycopg2 to be co-installed, but psycopg3 has since been released, which is currently incompatible:

Dockerfile:

from python

RUN python -m pip install -U pip psycopg sqlalchemy-citext

RUN python -c "import citext"

Result:

$ docker build .
[+] Building 8.3s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                            0.0s
 => => transferring dockerfile: 35B                                             0.0s
 => [internal] load .dockerignore                                               0.0s
 => => transferring context: 2B                                                 0.0s
 => [internal] load metadata for docker.io/library/python:latest                0.0s
 => CACHED [1/3] FROM docker.io/library/python                                  0.0s
 => [2/3] RUN python -m pip install -U pip psycopg sqlalchemy-citext            7.8s
 => ERROR [3/3] RUN python -c "import citext"                                   0.3s
------
 > [3/3] RUN python -c "import citext":
#6 0.317 Traceback (most recent call last):
#6 0.317   File "<string>", line 1, in <module>
#6 0.318   File "/usr/local/lib/python3.9/site-packages/citext/__init__.py", line 3, in <module>
#6 0.319     import psycopg2.extensions
#6 0.320 ModuleNotFoundError: No module named 'psycopg2'
------
executor failed running [/bin/sh -c python -c "import citext"]: exit code: 1

Hard psycopg2 dependency is still breaking pypy

Effectively the issue as commented in #19 (comment) This package is broken for environments that use alternative postgres DB adapters (psycopg2cffi, pg8000).

I'm rolling updates to my requirements for various project, and this is still happening.

FYI: PR in to sqlacodegen

Hello,

Thanks for your work on this project, it's made my life much simpler :>

I wanted to let you know that I submitted a PR to the sqlacodegen tool that incorporates sqlalchemy-citext as an extra. This allows sqlacodegen to produce classes cleanly without any complaints or manual fixups when one of the columns is a CITEXT type. Without this PR, it fails to recognize the column type and emits a NULL column type. This is easy to fix manually, but it also fails to add other metadata for the column such as the default values, etc.

It seems that the sqlacodegen project is looking for a new maintainer however, so the PR has not been accepted or commented on

I thought I would just put this in as a note. There is no action required in sqlalchemy-citext, it's more just to make you and other users aware. I don't know how popular sqlacodegen is, but I use it exclusively because it's boring and annoying manually writing up classes for tables.

Thanks again, please feel free to close this issue

KeyError for the citext field

I wanted a citext field in my API, so I decided to use this package. But it raises an error saying that there is a keyError for the citext field. The code is like so:

class Book(Model):
    __tablename__ = 'books'
    id = db.Column(db.String(32), primary_key=True)
    book_code = db.Column(CIText())

    def __init__(self, bookcode):
        self.id = str(uuid.uuid4().hex)
        self.book_code = bookcode

Any idea what's wrong?

Changes required for sqlalchemy 2.x / sqlalchemy 1.4 with future Engine

I'm trying to bring a project up to SQLAlchemy 2.x conventions and have noticed a handful of (mostly) subtle changes. I just noticed one that breaks sqlalchemy-citext, in register_citext_array()

Using Engine.execute() directly is no longer supported. So register_citext_array() should become something like this:

def register_citext_array(engine):
    """Call once with an engine for citext values to be returned as strings instead of characters"""
    # engine.execute(sqlalchemy.text("SELECT typarray FROM pg_type WHERE typname = 'citext'"))
    with engine.connect() as connection:
        results = connection.execute(sqlalchemy.sql.text("SELECT typarray FROM pg_type WHERE typname = 'citext'"))

    oids = tuple(row[0] for row in results)
    array_type = psycopg2.extensions.new_array_type(oids, 'citext[]', psycopg2.STRING)
    psycopg2.extensions.register_type(array_type, None)

I can send a PR if you want to support SQLAlchemy 2.x / SQLAlchemy 1.4 future Engines but I still need to do some testing

Quering against literal values doesn't always work

When querying a citext column against some literal values I get the error Don't know how to literal-quote value '$LITERAL_VALUE'. It seems that a literal_processor method is missing on the CIText, class. When adding the below method everything works, is there a reason why this isn't defined in the class?

# This is  copied from the `literal_processor` of sqlalchemy's own 
# `String` type.
def literal_processor(self, dialect: t.Any) -> t.Callable[[str], str]:
    def process(value: str) -> str:
        value = value.replace("'", "''")

        if dialect.identifier_preparer._double_percents:
            value = value.replace("%", "%%")

        return "'%s'" % value

    return process

SAWarning in cache key for CIText()

Observed using SQLAlchmey 1.4.28 and sqlalchemy-citext 1.8

SAWarning: UserDefinedType CIText() will not produce a cache key because the cache_ok attribute is not set to True. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)

Introduced in SQLAlchemy sqlalchemy/sqlalchemy@22deafe

I'm not entirely certain how this would affect behaviors, but wanted to surface this.

Don't depend on psycopg2

Depending on psycopg2 makes this module unusable on PyPy. If the dependency was left off PyPy users could use psycopg2cffi (which implements a psycopg2 compatible api ontop of cffi which works with PyPy).

I'd really hate to need to fork this library just to delete a dependency :/

Feature Request: Compatibility with AsyncPG

AsyncPG has been added to sqlalchemy and soon is going to be mainstream, some modificitations should be needed in order to not just work with classical psycopg driver but asyncpg ones.

Create changelog

It'd be great to have a changelog, to know what's going on between two releases!

column.type is NullType

If i ask the type of a CIText() column, i get NullType. From the NullType docstring:

    NullTypes will stand in if :class:`~sqlalchemy.Table` reflection
    encounters a column data type unknown to SQLAlchemy.  The
    resulting columns are nearly fully usable: the DB-API adapter will
    handle all translation to and from the database data type.

Versions:

SQLAlchemy==0.8.1
sqlalchemy-citext==1.0-2

CIText Array

If anyone else wants to use CIText in a postgresql array, this is what I did:

# Allow CIText to be in ARRAY
def patched_array_proc_array(self, arr, itemproc, dim, collection):
    if getattr(self.item_type, 'postgres_type', None) == 'CITEXT':
        if isinstance(arr, basestring):
            assert arr.startswith('{')
            assert arr.endswith('}')
            for row in csv.reader([arr[1:-1]]):
                arr = row
    return original_array_proc_array(self, arr, itemproc, dim, collection)
# noinspection PyProtectedMember
original_array_proc_array = ARRAY._proc_array
ARRAY._proc_array = patched_array_proc_array

How to use with table reflections

Can you provide an example with table reflections, because I am getting the following error with psql 9.4, sqlalchemy 1.0.6: SAWarning: Did not recognize type 'citext' of column 'foo'. The code is as follows:

engine = sqlalchemy.create_engine('postgresql://' + creds + '@' + args.db)
meta = sqlalchemy.MetaData()
meta.reflect(bind=engine)
foo = meta.tables['foo']