gbwey / persistent-odbc

uses persistent connecting via hdbc odbc

persistent-odbc

Uses the Persistent package and HDBC-odbc to access databases via ODBC.

Thanks to Dmitry Olshansky at https://github.com/odr/persistent-odbc/ for doing the initial hookup to hdbc-odbc.

Supports Postgres, MySql, MS Sql Server, Oracle, DB2, and SQLite.

see TestODBC.hs for usage and tests.

How to get started using cabal >= 1.18

git clone https://github.com/gbwey/persistent-odbc
cd persistent-odbc
stack ghci --test --flag persistent-odbc:tester
:load TestODBC.hs  (also need to define the appropriate odbc system dsn below and then run the command)

e.g. for postgresql
1. create a system odbc dsn "pg_test"
2. :main p

DBMS	ODBC System Dsn	Command
Postgresql	dsn=pg_test	:main p
MySQL	dsn=mysql_test	:main m
DB2	dsn=db2_test	:main d
Ms Sql Server >= 2012	dsn=mssql_test	:main s
Ms Sql Server pre 2012	dsn=mssql_test	:main so
Oracle pre 12c	dsn=oracle_test	:main o
Oracle >= 12c	dsn=oracle_test	:main on
Sqlite	dsn=sqlite_test	:main q

Limit and Offset in Ms Sql Server and Oracle

MSSQL True for MS Sql Server 2012 which has limit and offset support (esqueleto as well). MSSQL False for MS Sql Server pre 2012 only supports limit using Persistent's select* operations. (select top n ...).

Oracle True for Oracle >=12c which has limit and offset support (esqueleto as well). Oracle False for Oracle <12c where there is no limit and offset support.

Ms Sql Server and blobs

Blobs are problematic in hdbc-odbc.

Ms Sql Server and deleteCascadeWhere

Can cause segfault in Ms Sql Server.

Oracle and nulls

Treats empty string as a null (oracle issue).

Oracle and sorting blobs

Cannot sort on a blob field (oracle issue).

DB2 and blobs

Blobs don't support nulls (both insert and select) in this version of persistent-odbc (also doesn't work in hdbc-odbc). Select returns the blob values as unpacked strings at the moment (if there is interest in getting this fixed let me know).

persistent-odbc's Issues

Esqueleto bug

My relevant models:

ProductCharacteristic
  productInstance ProductInstanceId sql=productInstanceId
  characteristicInstance CharacteristicInstanceId sql=characteristicInstanceId
  deriving Show

Product
  codePrefix Text
  defaultPrice Double
  defaultBuyingPrice Double
  name Text
  obs Text
  disabled Bool
  deriving Show

ProductInstance
  product ProductId sql=productId
  code Text
  barCode Text
  price Double
  buyingPrice Double
  proportional Bool
  obs Text
  disabled Bool
  deriving Show

Now my Esqueleto expression:

f' :: Handler [(Entity ProductInstance, Entity Product, Entity ProductCharacteristic)]
f' = runDB
   $ E.select
   $ E.from $ \(pi `E.InnerJoin` p `E.InnerJoin` pc) -> do
     E.on $ pi ^. ProductInstanceProduct E.==. p ^. ProductId
     E.on $ pc ^. ProductCharacteristicProductInstance E.==. pi ^. ProductInstanceId
     return ( pi, p, pc)

The code I showed generated the following select instruction (which is wrong):

SELECT `ProductInstance`.`id`, `ProductInstance`.`productId`, `ProductInstance`.`code`, `ProductInstance`.`barCode`, `ProductInstance`.`price`, `ProductInstance`.`buyingPrice`, `ProductInstance`.`proportional`, `ProductInstance`.`obs`, `ProductInstance`.`disabled`, `Product`.`id`, `Product`.`codePrefix`, `Product`.`defaultPrice`, `Product`.`defaultBuyingPrice`, `Product`.`name`, `Product`.`obs`, `Product`.`disabled`, `ProductCharacteristic`.`id`, `ProductCharacteristic`.`productInstanceId`, `ProductCharacteristic`.`characteristicInstanceId`
FROM `ProductInstance` INNER JOIN `Product` ON `ProductCharacteristic`.`productInstanceId` = `ProductInstance`.`id` INNER JOIN `ProductCharacteristic` ON `ProductInstance`.`productId` = `Product`.`id`;

It is flipping the on clause in the JOIN!! Please, help me, I have to deliver this code next week...

The correct select instruction should be:

SELECT `ProductInstance`.`id`, `ProductInstance`.`productId`, `ProductInstance`.`code`, `ProductInstance`.`barCode`, `ProductInstance`.`price`, `ProductInstance`.`buyingPrice`, `ProductInstance`.`proportional`, `ProductInstance`.`obs`, `ProductInstance`.`disabled`, `Product`.`id`, `Product`.`codePrefix`, `Product`.`defaultPrice`, `Product`.`defaultBuyingPrice`, `Product`.`name`, `Product`.`obs`, `Product`.`disabled`, `ProductCharacteristic`.`id`, `ProductCharacteristic`.`productInstanceId`, `ProductCharacteristic`.`characteristicInstanceId`
FROM `ProductInstance` INNER JOIN `Product` ON `ProductInstance`.`productId` = `Product`.`id` INNER JOIN `ProductCharacteristic` ON `ProductCharacteristic`.`productInstanceId` = `ProductInstance`.`id`;

Strange persistent-template dependency

Current github versions requires persistent-template library with the following version:

persistent-template >= 2.1.0.1 && < 3

Both current hackage and github versions of the persistent-template library have a version 2.1, which causes cabal dependency to fail. Is this dependency a mistake or are you using some forked persistent -template?

Also please publish the recent version to the hackage if you don't mind.

mssql blobs are not working

informat/outformat problems with blobs. Can insert "0102" (as long as there are an event number of 0-f pairs) but not eg "zzz"
Also cannot "select" any blobs.

add optional foreign key support for sqlite

tinyint, charChk

tinyint can be used not only for booleans but also for single byte ints, in my example its 1, 2 or 3....

update to newer versions of persistent

Are there plans for adjusting persistent-odbc to newer persistent
versions? persistent-odbc-0.2.1.1 has dependency persistent < 2.11. The persistent package has moved past 2.11 quite a few iterations.

DBName is used in multiple places in persistent-odbc.
Database.Persist.Sql.DBName was removed in persistent-2.12 it seems.
lts-17.2 is the last stackage snapshot with a pre-2.12 version of persistent.

Building against persistent-2.11.0.2 we run into at least two
problems:

1. The type of Database.Persist.Sql.mkColumns has changed,
2. Database.Persist.Sql.Column has one additional constructor.

mkColumns recently requires BackendSpecificOverrides which can be
patched by giving emptyBackendSpecificOverrides as additional argument.

Column got the additional cGenerated field as fifth argument and the
last field cReference changed type from Maybe (DBName,DBName) to
Maybe ColumnReference which in turn contains a crTableName field.
That solves the problems in Database.Persist.MigrateMySQL but for
MigratePostgres the code is more complicated.

I used the compiler errors to start migrating persistent-odbc to
peristent-2.11 but at some point I am stuck because I don't know the
semantics.

fix cabal file so that flags work

compatibility with new persistent and resoucet

Some of the functions from these libraries have changed to the MonadUnliftIOconstraint from the old MonadBaseControl. ExceptT is not an instance of the former, hence the package doesn't compile against the newer dependencies. resourcet should be < 1.2 and persistent < 2.8 for the current master.

ODBC Example

I see that this project allows for ODBC support with persistent, but I can't seem to figure out how to configure it to work with Oracle. Is there any chance we could get some simple example database connection's?

use sqlint for primary key in MSSQL

The fix in commit 2cb5019 changed primary key to bigint, which is wrong, foreign key data type should be fix instead.

Foreign key data type follow primary key data type, not the other way round.

add support for limit and offset for mssql and oracle

foreign key types mismatch

person.id - INT
blog_post.author_id - BIGINT

Migrating: CREATe TABLE person(id int NOT NULL AUTO_INCREMENT PRIMARY KEY,name TEXT CHARACTER SET utf8 NOT NULL,age BIGINT NULL,score BIGINT NULL)
[Debug#SQL] CREATe TABLE person(id int NOT NULL AUTO_INCREMENT PRIMARY KEY,name TEXT CHARACTER SET utf8 NOT NULL,age BIGINT NULL,score BIGINT NULL); []
Migrating: CREATe TABLE blog_post(id int NOT NULL AUTO_INCREMENT PRIMARY KEY,title TEXT CHARACTER SET utf8 NOT NULL,author_id BIGINT NOT NULL REFERENCES person)
[Debug#SQL] CREATe TABLE blog_post(id int NOT NULL AUTO_INCREMENT PRIMARY KEY,title TEXT CHARACTER SET utf8 NOT NULL,author_id BIGINT NOT NULL REFERENCES person); []
Migrating: ALTER TABLE blog_post ADD CONSTRAINT blog_post_author_id_fkey FOREIGN KEY(author_id) REFERENCES person(id)
[Debug#SQL] ALTER TABLE blog_post ADD CONSTRAINT blog_post_author_id_fkey FOREIGN KEY(author_id) REFERENCES person(id); []
persist2-exe.exe: SqlError {seState = "["HY000"]", seNativeError = -1, seErrorMsg = "execute execute: ["1215: [MySQL][ODBC 5.3(w) Driver][mysqld-5.7.13-log]Cannot add foreign key constraint"]"}

ODBC Driver 17 for SQL Server: Invalid transaction state on disconnect

To reproduce:
Pristine MS SQL server 2012.
Installed the driver on Debian using
apt-get install msodbcsql17

Upon running either printMigration or runMigration, the commands seem to execute allright but the disconnect produces an error:

*** Exception: SqlError {seState = "["25000"]", seNativeError = -1, seErrorMsg = "freeDbcIfNotAlready/SQLDisconnect: ["0: [Microsoft][ODBC Driver 17 for SQL Server]Invalid transaction state"]"}

A web search suggests that after the last command the SQL driver implicitly starts another transaction which gets in the way of the disconnect. Putting SQL_AUTOCOMMIT_OFF in the connection string does not help.

The migration to run:

Migrating: CREATe TABLE [meter]([id] BIGINT NOT NULL IDENTITY(1,1) PRIMARY KEY ,[serial] VARCHAR(1000) NOT NULL,[type] VARCHAR(1000) NOT NULL,[obis] VARCHAR(1000) NOT NULL)
Migrating: CREATe TABLE [lastgang]([id] BIGINT NOT NULL IDENTITY(1,1) PRIMARY KEY ,[meter] BIGINT NOT NULL,[end_time] DATETIME2 NOT NULL,[k_w] REAL NOT NULL)
Migrating: ALTER TABLE [meter] ADD CONSTRAINT [unique_meter] UNIQUE([serial],[obis])
Migrating: ALTER TABLE [lastgang] ADD CONSTRAINT [unique_lastgang] UNIQUE([meter],[end_time])
Migrating: ALTER TABLE [lastgang] ADD CONSTRAINT [lastgang_meter_fkey] FOREIGN KEY([meter]) REFERENCES [meter]([id])

add db2 support

Is there a reason for escapeDBName?

Hi,

I'm new to Persistent and to Haskell. Persistent can already annotate table names and columns names using the sql modifier on the Persistent entity. The private function escapeDNName in MigrateMSSQL.hs encloses the table name in square parentheses. However, this is bad behaviour as the only way to specify the fully qualified table name is through the sql attribute on the Entity.

Querying the table sys.all_columns is an impossible query as it exists in the the schema 'sys'. However, persistent-odbc wraps sys.all_columns in square brackets. The database attempts to find [sys.all_columns] in the schema 'dbo' and then throws an exception because the table doesn't exist.

Just wondering if there was a reason to enclose all names of objects in square brackets rather than just letting the user annotate the columns when they need to?

Yesod

How can I easily use it with yesod through stack? (By the way, thank you so much for this repo, it was the only way I've found to use Persistent+Mysql+Windows)

fields of ‘SqlBackend’ not initialised: connInsertManySql, connUpsertSql

Probably a new version of persistent:

src/Database/Persist/ODBC.hs:128:12: warning: [-Wmissing-fields]
    • Fields of ‘SqlBackend’ not initialised: connInsertManySql,
                                              connUpsertSql
    • In the first argument of ‘return’, namely
        ‘SqlBackend
           {connLogFunc = logFunc, connPrepare = prepare' conn,
            connStmtMap = smap, connInsertSql = dbmsInsertSql mig,
            connClose = O.disconnect conn, connMigrateSql = dbmsMigrate mig,
            connBegin = const
                        $ E.catch (O.commit conn) (\ (_ :: E.SomeException) -> return ()),
            connCommit = const $ O.commit conn,
            connRollback = const $ O.rollback conn,
            connEscapeName = dbmsEscape mig, connNoLimit = "",
            connRDBMS = T.pack $ show (dbmsType mig),
            connLimitOffset = dbmsLimitOffset mig}’
      In a stmt of a 'do' block:
        return
          (SqlBackend
             {connLogFunc = logFunc, connPrepare = prepare' conn,
              connStmtMap = smap, connInsertSql = dbmsInsertSql mig,
              connClose = O.disconnect conn, connMigrateSql = dbmsMigrate mig,
              connBegin = const
                          $ E.catch (O.commit conn) (\ (_ :: E.SomeException) -> return ()),
              connCommit = const $ O.commit conn,
              connRollback = const $ O.rollback conn,
              connEscapeName = dbmsEscape mig, connNoLimit = "",
              connRDBMS = T.pack $ show (dbmsType mig),
              connLimitOffset = dbmsLimitOffset mig})
      In the expression:
        do { let mig = ...;
             smap <- newIORef Map.empty;
             return
               (SqlBackend
                  {connLogFunc = logFunc, connPrepare = prepare' conn,
                   connStmtMap = smap, connInsertSql = dbmsInsertSql mig,
                   connClose = O.disconnect conn, connMigrateSql = dbmsMigrate mig,
                   connBegin = const
                               $ E.catch (O.commit conn) (\ (_ :: E.SomeException) -> return ()),
                   connCommit = const $ O.commit conn,
                   connRollback = const $ O.rollback conn,
                   connEscapeName = dbmsEscape mig, connNoLimit = "",
                   connRDBMS = T.pack $ show (dbmsType mig),
                   connLimitOffset = dbmsLimitOffset mig}) }

Issue with SqlChar in Convertible instance of SqlValue

I'm having this issue where I have one table column with type tinyint but when I specify a Word8 type in my persistent models file, I get an error saying that

Failed to parse Haskell type `Word8`; expected integer from database, but received: PersistText \"\\STX\"

After some debugging, I found out that the library actually converts SqlChar to Persistent using this function called charChk where it converts \0 and \1 to boolean and other values to text

charChk :: Char -> PersistValue
charChk '\0' = PersistBool False
charChk '\1' = PersistBool True
charChk c    = PersistText $ T.singleton c

But as far as I can tell from hdbc-odbc, the SqlChar is actually being used to handle BindColBit and BindColTinyInt which are both meant to be used as an 8-bit number https://github.com/hdbc/hdbc-odbc/blob/06833d77799f16634d2038bcdc308c35d4752cdd/Database/HDBC/ODBC/Statement.hsc#L873-L880

I think a good fix here would be to convert the Char to PersistInt64 instead, as persistent handles nicely to convert PersistInt64 to either Word, Int or even Bool type.

I can open a PR if you'd like.

Related to #18

Number fields in Oracle 11gR2

Hi,

I've tried to work with legacy DB in our company with persistent-odbc 0.2.0.1 and I am getting the following marshalling error:
tariff-zone-exe.exe: PersistMarshalError "field treeId: int32 Expected Integer, received: PersistByteString \"123\""

My environment is:

• Windows 7 x64 Enterprise
• GHC 7.10.2 x64 for Windows

The Haskell code is:

runDB = runResourceT . runNoLoggingT . withODBCConn (Just $ Oracle False) "dsn=pc-test5;PWD=<some pwd>" . runSqlConn

share [mkPersist sqlSettings] [persistUpperCase|
TariffZone sql=TARIFF_ZONE
    Id Int32 sql=TARIFF_ZONE_ID
    name  String sql=TARIFF_ZONE_NAME
    name2 String Maybe sql=TARIFF_ZONE_NAME2
    description String Maybe sql=DESCRIPTION
    treeId Int32 sql=TARIFF_ZONE_TREE_ID    
    parentId Int32 sql=PARENT_TARIFF_ZONE_ID
    deriving Show
|]

main :: IO ()
main = do
    tz <- runDB $ do
        select $
            from $ \tz -> do
                where_ (tz ^. TariffZoneName `like` val "Roam_Voice_Out.fromZone1.France%")
                orderBy [asc (tz ^. TariffZoneName)]
                return tz

    mapM_ (putStrLn . show) tz
    return ()

Table definition is:

CREATE TABLE "TARIFF_ZONE" 
(   "TARIFF_ZONE_ID" NUMBER(*,0) NOT NULL, 
    "TARIFF_ZONE_TREE_ID" NUMBER(*,0) NOT NULL, 
    "PARENT_TARIFF_ZONE_ID" NUMBER(*,0), 
    "TARIFF_ZONE_NAME" VARCHAR2(128 BYTE) NOT NULL, 
    "TARIFF_ZONE_NAME2" NVARCHAR2(128), 
    "DESCRIPTION" VARCHAR2(256 BYTE),
         CONSTRAINT "PK_TARIFF_ZONE" PRIMARY KEY ("TARIFF_ZONE_ID")
);