nikita-volkov / hasql Goto Github PK

I need to be able to store "default" connection pool settings in my program. Since the poolSettings function won't give me a PoolSettings until runtime, I can't store a literal representing the connection pooling parameters in my program unless I create my own data type and then translate that into a call to poolSettings, e.g.,

data Pool = Pool Int Int
mkPool :: Pool -> Maybe H.PoolSettings

I'd like Hasql to provide a data type to represent this so I don't have to do it in each program that uses this library. In other words, PoolSettings represents validated settings, but I need a type to store unvalidated settings so I can write literals for these settings in my programs.

My app needs to bulk-load data and postgres' copy mode is the fastest way to do it. Postgresql-simple supports it with this. Would be great if this library did too.

Thanks for making hasql! I'll actually be switching over to it anyway, and the copy command would be the icing on the cake.

When I accidentally used the wrong port number in my hasql session settings it caused my program to lock up. The deadlock or whatever it is happens upon calling tx for the first time. Adding a connection timeout would be great, and throwing an exception.

After upgrading to GHC version 8.0.1. I finally could install Hasql on win7 64 bits.
cabal install hasql => no errors. If I do cabal install again, it reports hasql is already installed.

Then I tried my first program:

module Main
where
import qualified Hasql as H

main :: IO ()
main = putStrLn "hello"
I did ghd --make testHasql.hs => Failed to load interface for Hasql. Use -v to see....
I did ghc --make -v testHasql.hs and got the error message below.

I wonder whether Hasql needs SSL at runtime or can I connect to postgresql without SSL. I got a popup during installation complaining about a missing SSL dll.
But the installation of Hasql didn't abort.

What can I do?
Kees

Glasgow Haskell Compiler, Version 8.0.1, stage 2 booted by GHC version 7.10.2
Using binary package database: D:\Program Files\Haskell Platform\8.0.1\lib\package.conf.d\package.cache
Using binary package database: C:\Users\kees\AppData\Roaming\ghc\x86_64-mingw32-8.0.1\package.conf.d\package.cache
loading package database D:\Program Files\Haskell Platform\8.0.1\lib\package.conf.d
loading package database C:\Users\kees\AppData\Roaming\ghc\x86_64-mingw32-8.0.1\package.conf.d
wired-in package ghc-prim mapped to ghc-prim-0.5.0.0
wired-in package integer-gmp mapped to integer-gmp-1.0.0.1
wired-in package base mapped to base-4.9.0.0
wired-in package rts mapped to rts
wired-in package template-haskell mapped to template-haskell-2.11.0.0
wired-in package ghc mapped to ghc-8.0.1
wired-in package dph-seq not found.
wired-in package dph-par not found.
Hsc static flags:
loading package database D:\Program Files\Haskell Platform\8.0.1\lib\package.conf.d
loading package database C:\Users\kees\AppData\Roaming\ghc\x86_64-mingw32-8.0.1\package.conf.d
wired-in package ghc-prim mapped to ghc-prim-0.5.0.0
wired-in package integer-gmp mapped to integer-gmp-1.0.0.1
wired-in package base mapped to base-4.9.0.0
wired-in package rts mapped to rts-1.0
wired-in package template-haskell mapped to template-haskell-2.11.0.0
wired-in package ghc mapped to ghc-8.0.1
wired-in package dph-seq not found.
wired-in package dph-par not found.
*** Chasing dependencies:
Chasing modules from: testHaSql.hs
!!! Chasing dependencies: finished in 0.00 milliseconds, allocated 0.322 megabytes
Stable obj: []
Stable BCO: []
Ready for upsweep
[NONREC
ModSummary {
ms_hs_date = 2016-06-28 11:51:08.0848841 UTC
ms_mod = Main,
ms_textual_imps = [(Nothing, Prelude), (Nothing, Hasql)]
ms_srcimps = []
}]
** Deleting temp files:
Deleting:
compile: input file testHaSql.hs
*** Checking old interface for Main:
[1 of 1] Compiling Main ( testHaSql.hs, testHaSql.o )
*** Parser [Main]:
!!! Parser [Main]: finished in 0.00 milliseconds, allocated 0.125 megabytes
*** Renamer/typechecker [Main]:
!!! Renamer/typechecker [Main]: finished in 15.60 milliseconds, allocated 5.820 megabytes

testHaSql.hs:4:1: error:
Failed to load interface for `Hasql'
Locations searched:
Hasql.hs
Hasql.lhs
Hasql.hsig
Hasql.lhsig
Upsweep partially successful.
*** Deleting temp files:
Deleting:
link(batch): upsweep (partially) failed OR
Main.main not exported; not linking.
*** Deleting temp files:
Deleting:
*** Deleting temp dirs:
Deleting:

Currently, for any type to be an instance of the Mapping typeclass, you'll need to define both renderValue and parseResult. However, sometimes, I would only want a one way conversion (either to render them to binary, or read them from binary).

If say the Mapping typeclass is split into FromSql and ToSql (or whatever appropriate), like how aeson does it, it provides more flexibility in using the library.

By usual SQL workflow usually involves building up a large SQL statement from several subexpressions, which is currently a bit convoluted to do with the current quasiquoter since there is currently no CxValue instance for the Stmt themselves. Ideally something like the following, I'm curious if this possible in the implementation?

predicate :: H.Stmt HP.Postgres
predicate = [H.stmt|LIMIT 3|]

select :: H.Stmt HP.Postgres
select = [H.stmt|SELECT * FROM table ?|] predicate

My current workaround involves basically just dropping down to the underlying string type and using the following types to manipulate the underlying string.

instance IsString (HB.Stmt a) where
  fromString text = (HB.Stmt (T.pack text) mempty True)

toStmt :: Text -> HB.Stmt c
toStmt x = fromString (T.unpack x)

toText :: HB.Stmt c -> Text
toText (HB.Stmt a _ _) = a

Sometimes it would be useful to manually initiate a rollback in the current transaction monad without throwing an error. For example, when writing specs via Hspec for an application we're building, automatically rolling-back the database after each spec can save much hassle. Currently, we achieve this via:

withQuery :: (((forall s. Tx Postgres s a) -> IO a) -> b) -> b
withQuery spec = spec $ \queries ->
    initDb (config_database config) $
      tx trans $ do
        val <- queries
        _ <- unit [q| ROLLBACK |]
        return val

which can then be used like so:

  describe "User.find query" $
    it "finds a User by its ID" $ withQuery $ \q -> do
      found <- q $ do
        user <- fromJust <$> User.insert mockUserData
        User.find $ user_id user

      user_data (fromJust found) `shouldBe` mockUserData

While the unit [q| ROLLBACK |] works, it causes a warning when Hasql tries to initiate a COMMIT and fails. A proper rollback function might look something like:

  rollback :: Backend b => Tx b s ()
  rollback = Tx $ ReaderT $ finishTransaction False

though I haven't played with it enough to know if this could cause other issues.

If you'd welcome the addition of a rollback function to Hasql, I'm happy to create, test, and submit a pull request. Let me know!

I'm trying to implement streaming support (foldrRows doesn't seem to cut it, since it builds the whole stream in the IORef first). The translation is straightforward except in statement where the interaction with the MVar is difficult to handle in a stream. Rather than reimplementing and running into a wall, I'm hoping you could give some insight on the design choice for using the MVar/IORef throughout.

First, thank you for creating the library!

I'm currently experimenting with hstore and I was wondering how I'd insert/update hstore columns. I couldn't find an encoder for hstore even though the postgresql-binary includes one. Do I have to roll my own or is there already a combinator I overlooked?

data WithAffectedRows a 
  = WithAffectedRows Int64 a

withAffectedRows :: Result a -> Result (WithAffectedRows a)

Useful when using returning in insert/update/delete queries.

UPDATE t SET x = 1 WHERE y = 2 RETURNING id;

Postgres treats an empty array as an array with 0 dimensions, so trying to parse a empty 1-dimensional array fails with the error ResultError (RowError 0 (ValueError "A missing dimension length")).

Is there a recommended way to handle empty arrays?

Hasql is awesome, but different enough from existing Haskell database libraries that using it can be initially puzzling to use. I think an instructive README in the repo (and linked from Hackage) could go a long way towards helping users get situated with Hasql.

If you're cool with adding a README to the project, I'm happy to write up a draft, which you'd of course be free to splice as suits you.

My web app uses text parameters from the outside world to construct fairly freeform queries. As a simplified case consider

select * from foo where id <= ?

I supply the query with [renderValue value] where value :: Text. When the query is created Hasql must be adding an explicit type cast because I get this error

A status error.
Status: "FatalError";
Code: "42883";
Message: "operator does not exist: bigint <= text";
Hint: "No operator matches the given name and argument type(s). You might need to add explicit type casts.".

However postgres is actually fine with a query like select * from foo where id <= '5' (I tried it in pgadmin). It coerces the types and the comparison works. So I'm wondering if you can provide a way to construct a new kind of StatementArgument which signals that I would like to allow postgres to decide the conversion for me.

Another option would be for me to try parsing an Int first, then a Double, etc and build the correct StatementArgument. However there are a lot of types to detect and some are ambiguous for my application with its limited knowledge of the schema (is "$3.99" the postgres currency type or just a string? Depends on the column really).

A final option would be for me to format and escape the unknown parameter as a literal and concatenate it onto the string. Doing this will be less efficient though because of missing the prepared statement cache.

Sometimes I know very little about a query I am building until runtime. I would like to use ? parameters for things other than simple values, like this:

SELECT * FROM ?

The substitution above is for an identifier, not a value. So it should not be single-quoted. Postgres provides a format function that works like printf but supports %I to generate these kinds of strings. However I realize you don't want to make another round-trip to the database to run prepare. So I wrote this function and used quickcheck to verify that it behaves exactly like Postgres' identifier formatter.

pgFmtIdent :: Text -> Text
pgFmtIdent x =
  let escaped = replace "\"" "\"\"" (trimNullChars x) in
  if escaped =~ danger
    then "\"" <> escaped <> "\""
    else escaped

  where
    danger = "^$|^[^a-z_]|[^a-z_0-9]" :: Text
    trimNullChars = Data.Text.takeWhile (/= '\x0')

For a workaround I can use this function and string concatenation to build the queries I need. Just thought it would be nice to allow the ? to work in this situation with a special kind of StatementArgument.

Would a PR adding the classes:

class Encodable a where
  encode :: Params a

class Decodable a where
  decode :: Row a

With generic instances for them be welcome? I found myself wanting this and wrote it as a separate package.

One downside is that the desired behavior for sum types is not obvious, but we could rule out classes with sum types.

This would allow to get rid of the Identity wrapper in single-column results.

After bashing my head against the Contravariant, Divisible, and Decidable interfaces, I have failed to come up with the combinator I want.

I would like a function uncurry :: (a -> Params b) -> Params (a, b). I believe that it is possible to implement its counterpart

curry :: Params (a, b) -> a -> Params b
curry p x = contramap (x,) p

It should be possible to implement uncurry because Params a is just a newtype over a -> Something, but I don't see how to do it using the public interface.

It may also be nice to document how to use the Divisible and Decidable interfaces (e.g. by having examples of their use in the haddocks).

Sorry if this is just me misunderstanding things, I'm still pretty new to Haskell.

I'm using hasql for a small project, and am getting the error library/Hasql.hs:89:40-43: A "hasql" package bug: Unexpected TransactionConflict exception which appears to be coming from here. I'm expecting a few transaction conflicts with what I'm doing, but it appears that they're not being handled properly.

I'm running the following code on multiple threads via forkIO

saveClips :: H.SessionSettings -> [Clip] -> IO ()
saveClips s clips = do                            
        H.session psqlConf s $ do                 
            forM_ clips $ \(Clip (ClipData dn yd qr ds)) -> do  
                if ds >= -100 && dn <= 100        
                    then                          
                        H.tx (Just (H.Serializable, True)) $ do
                            (mrow :: Maybe (Double,I.Int64,I.Int64,I.Int64,I.Int64)) <-
                                        H.single $ [H.q|SELECT *
                                                        FROM clips
                                                        WHERE down=?
                                                          AND yard_line=?
                                                          AND quarter=?|] dn yd qr
                            case mrow of          
                                Just (down,yard,quarter,avgDist,numClips) ->
                                            let newNumClips = numClips + 1 
                                                newAvg = (avgDist * numClips + ds) `div` newNumClips
                                            in H.unit $ [H.q|UPDATE clips
                                                             SET avg_dist=?
                                                               , num_clips=?
                                                             WHERE down=?
                                                               AND yard_line=?
                                                               AND quarter=?|] newAvg newNumClips dn yd qr
                                Nothing ->  H.unit $ [H.q|INSERT INTO clips
                                                              ( down
                                                              , yard_line
                                                              , quarter
                                                              , avg_dist
                                                              , num_clips
                                                              ) VALUES (?,?,?,?,?)|] dn yd qr ds (1 :: I.Int64)
                    else return ()                
        return ()

and here's the function that initializes my database:

initDb :: H.SessionSettings -> IO ()
initDb s = H.session psqlConf s $ do              
            H.tx Nothing $ do                     
              H.unit [H.q|DROP TABLE IF EXISTS clips|]
              H.unit [H.q|CREATE TABLE clips ( 
                              down      DECIMAL NOT NULL,
                              yard_line BIGINT NOT NULL,
                              quarter   BIGINT NOT NULL,
                              avg_dist  BIGINT NOT NULL,
                              num_clips BIGINT NOT NULL,
                              PRIMARY KEY ( down
                                          , yard_line
                                          , quarter
                                          )       
                          )|]

It'd be cool if I could get this fixed, but it's not too big of a deal if I need to keep this not parallel. Just wanted to make sure the devs were aware of this.

OK this isn't really a bug in the library but it did confuse me. Maybe there could be a clarification in the docs.

Both Hasql and Hasql.Backend define an Error type. I was originally importing both modules qualified as H. I use catchJust to handle certain sql errors but for some reason it would never catch any Hasql errors. Turns out I had to import Hasql.Backend as HB and catch HB.Error specifically -- H.Error never got thrown.

I notice that Hasql.PTI exposes

unknown = mkPTI 705  Nothing

But Hasql.Encoders does not provide a function to construct this kind of Value, along the lines of

unknown :: Value Int64
unknown =
  Value (Value.unsafePTI PTI.unknown (const Encoder.unknown))

In fact I don't see an unknown encoder in PostgreSQL.Binary.Encoder. Would it difficult to add the ability to create these values? I'd like to pass a ByteString and have postgresql do its best to coerce it.

The reproduction: https://github.com/begriffs/hasql-stress/tree/18263c1416a77a816c0d2d4d403e4dec47b097af

Thanks to @begriffs and @ruslantalpa.

As I try to integrate hasql with scotty, I encounter the following problem: both scotty and hasql define a Monad transformer to run an arbitrary monad.

In order to integrate hasql and scotty: I would have to do either:

• Integrate the hasql Session inside the ScottyT Monad, however hasql does not expose its session via a createSession/withSession API.
• Integrate the Scotty Monad inside a hasql Session, however I can't find a way to lift the Scotty monad inside a Session.

I would like to integrate hasql into scotty the following way:

main = do
    sess <- Hasql.createSession backend sessionSettings
    scottyT port (id) (Hasql.withSession sess) $ do
       -- Scotty declarations here

which would require a createSession/withSession type API. Is it possible to add such functions, or am I trying the wrong way?

I'm using Hasql in an application that needs a test suite. As part of that test suite, I want to test database code to check that it works properly. In doing this, the pattern I need to use is to run each test in a transaction and then roll it back. The only wan I can see to do this is to put a fail at the end of the Tx sequence for a given test to force a rollback. However, another thing I need is to return values from these tests -- such as Bool values indicating whether the expected comparisons succeeded. But if I use fail, I can't get any such return values.

Would it be possible to provide an explicit rollback mechanism that combines these behaviors? Say,

rollback :: a -> Tx c s a

where rollback triggers a rollback and returns a immediately.

Right now, with GHC 7.10.1:

$ cabal install hasql
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: hasql-0.7.2 (user goal)
next goal: base (dependency of hasql-0.7.2)
rejecting: base-4.8.0.0/installed-901... (conflict: hasql => base>=4.5 &&
<4.8)
rejecting: base-4.8.0.0, 4.7.0.2, 4.7.0.1, 4.7.0.0, 4.6.0.1, 4.6.0.0, 4.5.1.0,
4.5.0.0, 4.4.1.0, 4.4.0.0, 4.3.1.0, 4.3.0.0, 4.2.0.2, 4.2.0.1, 4.2.0.0,
4.1.0.0, 4.0.0.0, 3.0.3.2, 3.0.3.1 (global constraint requires installed
instance)
Dependency tree exhaustively searched.

It does not seem to be possible to pass the name of a table in a quasiquoted query using substituted params like:

[H.stmt| INSERT INTO ? (name, age) VALUES (?, ?) |] tName kName vValue]

This generates a db error:

pg_1       | ERROR:  syntax error at or near "$1" at character 13
pg_1       | STATEMENT:  INSERT INTO $1 (name, age) VALUES ($2, $3)

Is there any other way to achieve this? I want to be able to write a function that can take the table name as a function argument and send queries for that table. I have asked in detail on stackoverflow.com here with more sample code and more error info - http://stackoverflow.com/questions/32309098/how-to-pass-table-name-as-a-variable-in-quasiquoted-hasql-query-code

If UTCTime represents a UTC time, why does it encode/decode with timestamptz? Seems like UTCTime should actually transcode with timestamp since it is the UTC time...

Consequently, timestamptz should convert into a type with time zone information like ZonedTime.

list :: Params [(LibPQ.Oid, Maybe ByteString)]

It is useful when dealing with dynamically generated queries.

E.g.:

\id -> [stmt| SELECT name FROM table WHERE id = $id |]

I currently get the following message when trying to view the hackage documentation for hasql. Old versions still seem to work.

There is no documentation for hasql-0.7.3.2

As part of seeding my database for tests I have code like this

schema <- loadFixture "schema"
H.session pgSettings testSettings $ do
  H.tx (Just (H.ReadCommitted, True)) $ do
    H.unit schema

However when I run it I get a postgres error

A status error.
Status: "FatalError"; Code: "42601";
Message: "cannot insert multiple commands into a prepared statement".

The schema contains many statements and unit tries to prepare them all at once. Can I have it skip the preparation? I notice that the postgres backend contains a way to do this, but I don't know how to trigger it through the general Hasql interface.

https://github.com/nikita-volkov/hasql-postgres/blob/fdcbe35bb5db36fb6f72839468e47303974f1150/library/Hasql/Postgres/Statement.hs#L10-L11

I'm having difficulty using Composite. Could you please provide an example of how to decode something basic like select (1,'a'); in the documentation?

Hi,
I wanted to know if there are plans for a sqlite backend to hasql?

If not, I would be interested in creating a backend. To do that, how much should I base off of the Postgresql backend?

Thanks

Inserting a value into a postgres enum type through hasql gives an error. How can I insert a value into a column with enum type defined in postgres as CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy')?

For a persistent web application, I am keeping a single connection pool to service all requests, and had been using the simple runReaderT on a Session to execute queries. Now, session seems to be the only way the execute a session, which creates a new pool each time. Is it not recommended to reuse backend connections?

For instance, function/trigger definitions such as those found here: http://www.postgresql.org/docs/9.2/static/plpgsql-trigger.html

When I upgraded from the hasql-postgres-0.7 to hasql-postgres-0.8, my code failed to compile (no surprises there). But, documentation on any breaking changes is not readily available (looking at the demo in the repo quickly solved the issue). Maintaining a changelog on hackage would help. I can send a pull request, but I'm only aware of the changes that affected my code.

The executeMany functionality in the postgresql-simple driver can be the difference between inserting a few hundred rows a second and inserting tens of thousands. Are there any plans to implement this functionality in hasql?

We intend to use hasql with yesod. It'll be great if you can include hasql in stackage. It'll avoid unnecessary build conflicts.

I'd like to be able to do bulk insertions and updates with queries like `INSERT INTO a VALUES (?)" where ? :: Mapping a, Mapping b => [(a,b)]

How do I create query with named parameters? In haddock I only see way to created query with positional parameters.

In PostgREST we get the main response body as Data.ByteString.Char8 from postgres/hasql only to convert it to Data.ByteString.Lazy and send it to the user

It would be cool if here
https://github.com/nikita-volkov/hasql/blob/master/library/Hasql/Decoders.hs#L346
we could also have bytea_lazy

It seems to be possible since you already have that here
https://github.com/nikita-volkov/postgresql-binary/blob/master/library/PostgreSQL/Binary/Decoder.hs#L244

I am far from understanding all these types and the format of the data sent by postgresql over the wire but am i right in saying that if we implement this then we would basically stream raw bytes sent by postgresql to the http client?
Any other places we should touch to get postgrest to just forward/stream the data from the database?

Thank you

I was making a simple example program, due to my inability to find example programs for hasql outside of this repo, and ran into what I think might be another bug.

When I run the code on my laptop it prints out the error Main: UnparsableRow "Null result", and when I run it on my desktop it hangs on what appears to be the second select statement for a little while and is then killed.

Laptop output:

"1"
"2"
"going to select"
"selected. Going to insert or update"
"inserted or updated"
"3"
"going to select"
Main: UnparsableRow "Null result"

Desktop output:

"1"
"2"
"going to select"
"selected. Going to insert or update"
"inserted or updated"
"3"
"going to select"
zsh: killed     ./Main

The code I'm running is available here.

As it stands, hasql supports performing transactions that end either in a rollback or a commit. Would you consider adding a third option, namely savepoints? That would allow for sane transaction isolation while letting interleaving IO in between. That is, perform this operation at the end of Hasql.Transaction.run: http://www.postgresql.org/docs/current/static/sql-savepoint.html

Then, when you continue the transaction starting from that savepoint, you'd perform a rollback to savepoint instead of an abort, if need be.

run would need to have a type like

run :: Transaction a -> ... -> IO (Either Query.ResultsError (Either (SavePoint a) a))

and Transaction should have operations for ending the transaction with a savepoint instead of a return, and for rolling back to a specific savepoint instead of just the latest one. Continuing the transaction would involve a version of run that would take the savepoint instead of the isolation, mode and connection parameters.

I'm not certain how all the kinks of an API with savepoints would go. I may try to sketch an implementation myself, yet. I'll need to familiarize myself with the new hasql version first. But I'd love to hear first whether this kind of functionality would be anything you'd care to see in hasql. I like the purity of leaving IO out of SQL transactions but I feel that there's a way to have my cake and eat it too.

Currently the statements are encoded as bytestrings, which limits the user with the ASCII symbols table for use statement templates. We should switch to Text as the internal format for statement templates to fix that.

The Error exceptions contain a text description but it can be irregular and hard to parse. It has unescaped and sometimes redundant quotes:

Status: ""Fatal error""; Code: "42703"; Message: "column "foo" does not exist”.

I wrote a parser for the strings the way they are now, but it would probably be better to standardize the quoting. Also the string inside CantConnect has an entirely different format.

Even better would be to augment Error with separate fields for message, status, code, and hint. I know you use the error code internally to construct different instances such as ConnectionLost vs UnparsableRow but the codes themselves can provide more nuance and it would be nice to have them easily accessible.

A status error.
Status: "FatalError"; Code: "42883";
Message: "operator does not exist: information_schema.sql_identifier = bytea";
Hint: "No operator matches the given name and argument type(s). You might need to add explicit type casts.".

The workaround is to ask for the column as Text. Perhaps ByteString is intended for binary data and Text for textual data and I'm just using it wrong. Thought I'd report the situation though.

If I use fail inside a Tx action, a transaction retry still occurs. This is not what I want in some cases. Is there a way I can indicate that I want to abort the transaction without attempting a retry?

Are there any plans to introduce a rudimentary sql syntax check into the QuasiQuoter? Something like https://github.com/markdrago/pgsanity ?

In particular I have the following questions.

• How do transactions work?
• Is it right, that one has to be explicit when constructing a Hasql.Query.Query? Is there no type inference anymore?

Thanks for your efforts.