facebook / haxl

A Haskell library that simplifies access to remote data, such as databases or web-based services.

Haxl

Haxl is a Haskell library that simplifies access to remote data, such as databases or web-based services. Haxl can automatically

• batch multiple requests to the same data source,
• request data from multiple data sources concurrently,
• cache previous requests,
• memoize computations.

Having all this handled for you behind the scenes means that your data-fetching code can be much cleaner and clearer than it would otherwise be if it had to worry about optimizing data-fetching. We'll give some examples of how this works in the pages linked below.

There are two Haskell packages here:

• haxl: The core Haxl framework
• haxl-facebook (in https://github.com/facebook/Haxl/tree/master/example/facebook): An (incomplete) example data source for accessing the Facebook Graph API

To use Haxl in your own application, you will likely need to build one or more data sources: the thin layer between Haxl and the data that you want to fetch, be it a database, a web API, a cloud service, or whatever.

There is a generic datasource in "Haxl.DataSource.ConcurrentIO" that can be used for performing arbitrary IO operations concurrently, given a bit of boilerplate to define the IO operations you want to perform.

The haxl-facebook package shows how we might build a Haxl data source based on the existing fb package for talking to the Facebook Graph API.

Where to go next?

• The Story of Haxl explains how Haxl came about at Facebook, and discusses our particular use case.

• An example Facebook data source walks through building an example data source that queries the Facebook Graph API concurrently.

• Fun with Haxl (part 1) Walks through using Haxl from scratch for a simple SQLite-backed blog engine.

• The N+1 Selects Problem explains how Haxl can address a common performance problem with SQL queries by automatically batching multiple queries into a single query, without the programmer having to specify this behavior.

• Haxl Documentation on Hackage.

• There is no Fork: An Abstraction for Efficient, Concurrent, and Concise Data Access, our paper on Haxl, accepted for publication at ICFP'14.

Contributing

We welcome contributions! See CONTRIBUTING for details on how to get started, and our Code of Conduct.

License

Haxl uses the BSD 3-clause License, as found in the LICENSE file.

haxl's Issues

New version to Hackage for GHC 8.8 and beyond

Hi,

Sorry to bother, but is there any plans on releasing a new version to Hackage as the current highest version 2.1.2.0 doesn't work on ghc 8.8?

test suite build failure

As seen on the Stackage build server:

Preprocessing test suite 'test' for haxl-0.4.0.0...
[1 of 1] Compiling Main             ( tests/TestMain.hs, dist/build/test/test-tmp/Main.o )

tests/TestMain.hs:6:1: error:
    Failed to load interface for ‘AllTests’
    Use -v to see a list of the files searched for.

Release version compatible with aeson-2 and GHC 9

The latest version on hackage still throws errors for GHC 9, the fix is already in main. Now with the aeson PR merged, could you release a new version?
Thanks

Plans for a monad transformer interface?

Currently, the fetch method of the type class lives in IO. Flaxl showed that it is possible to free it into any monad by relaxing the constraints from IO. Any plans for Haxl to move to or also provide a monad transformer interface?

Haxl 2 release on Hackage

Make instance MonadThrow GenHaxl

There is already throw and catch.

Is it ok to make a PR for this?

bounded caches

Hi there,

I've been using this great library for some time now, thanks for opening it up to the public!

My use case involves large amounts of data, so I had to change the library slightly to be able to use a bounded cache. I have made it configurable so that each type of request can either use a bounded cache of a given size, or an unbounded cache.

In order to configure it, I have added an additional typeclass that has a DataSource has to be an instance of. It has a single function, cacheSize, that determines the size of the cache for each request (which may be infinite).

If this is useful to other people, I'd be glad to do a pull request. Notice however, that this would be a breaking change, since all DataSources have to add an instance of the new typeclass (which is easy to do).

I have not yet added tests that check that the cache stays within the specified bounds, nor done benchmarks (but it's on my agenda).

Best,
Philipp

Inaccurate lower bound on `base`

haxl.cabal claims compatiblity with base == 4.*, however GHC 7.8.4 falsifies this:

Configuring library for haxl-2.0.1.0..
Preprocessing library for haxl-2.0.1.0..
Building library for haxl-2.0.1.0..
[ 1 of 18] Compiling Haxl.Core.Util   ( Haxl/Core/Util.hs, /tmp/matrix-worker/1535645265/dist-newstyle/build/x86_64-linux/ghc-7.6.3/haxl-2.0.1.0/build/Haxl/Core/Util.o )
[ 2 of 18] Compiling Haxl.Core.DataCache ( Haxl/Core/DataCache.hs, /tmp/matrix-worker/1535645265/dist-newstyle/build/x86_64-linux/ghc-7.6.3/haxl-2.0.1.0/build/Haxl/Core/DataCache.o )
[ 3 of 18] Compiling Haxl.Core.StateStore ( Haxl/Core/StateStore.hs, /tmp/matrix-worker/1535645265/dist-newstyle/build/x86_64-linux/ghc-7.6.3/haxl-2.0.1.0/build/Haxl/Core/StateStore.o )

Haxl/Core/StateStore.hs:44:19:
    Not in scope: type constructor or class `Proxy'

Haxl/Core/StateStore.hs:45:18: Not in scope: `typeRep'

Haxl/Core/StateStore.hs:61:41:
    Not in scope: data constructor `Proxy'

Haxl/Core/StateStore.hs:61:50:
    Not in scope: type constructor or class `Proxy'

Haxl/Core/StateStore.hs:69:24:
    Not in scope: data constructor `Proxy'

Haxl/Core/StateStore.hs:69:33:
    Not in scope: type constructor or class `Proxy'

Haxl/Core/StateStore.hs:75:22:
    Not in scope: data constructor `Proxy'

Haxl/Core/StateStore.hs:75:31:
    Not in scope: type constructor or class `Proxy'
<<ghc: 1030926808 bytes, 1905 GCs, 15138471/60948896 avg/max bytes residency (8 samples), 118M in use, 0.00 INIT (0.00 elapsed), 0.58 MUT (0.69 elapsed), 0.59 GC (0.59 elapsed) :ghc>>

I've already revised the affected releases on Hackage,

• https://hackage.haskell.org/package/haxl-2.0.1.0/revisions/
• https://hackage.haskell.org/package/haxl-2.0.0.0/revisions/

so there's no immediate need for action on your part; however please take this into account for future releases of haxl.

2.0.0.0 tag is missing.

GHC-8.0 compatible release

The master branch has lifted bounds. Please make a revision / new release to Hackage. Thanks.

monad law `(<*>) = ap` is ignored?

I have taken look at some talks of @simonmar about haxl:

1. Keynote from Simon Marlow - Fun with Haxl
2. The Haskell Cast #4 - Simon Marlow on Parallelism and Concurrency

And looks like Haxl is not lawful as it's ignoring law (<*>) = ap.
also as (>>) = (*>) is implied by that law it causes issues like one in first link (1.) about sequencing monadic actions in do even when applicative do was off:

run $ do
  cmd "touch" ["foo"]
  cmd "rm" ["foo"]

And there was need to add _ <- in order that to be sequential:

run $ do
  _ <- cmd "touch" ["foo"]
  cmd "rm" ["foo"]

My question are:

• Is it intentional to ignore that law?
• Can Haxl be lawful and also produce same benefits?
• How often is it ignored in haskell libraries?
• What are benefits of that law?

Also I have taken look at the original paper about Applicative and did get reasons why this law was intorduced, so if you know any discussion on that topic you can link would be great!

Also in js community some implementations of Futures also Ignore that law (fantasy-land#179), looks like this law is not that widely known.

Up dependency versions to allow build on GHC 7.10.1

cabal install failed until I upped filepath to 1.4.* and time to 1.5.*

I'm on GHC 7.10.1.

$ cabal install
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: haxl-0.1.0.0 (user goal)
trying: vector-0.10.12.3/installed-cbb... (dependency of haxl-0.1.0.0)
next goal: time (dependency of haxl-0.1.0.0)
rejecting: time-1.5.0.1/installed-f60..., 1.5.0.1, 1.5 (conflict: haxl =>
time==1.4.*)
rejecting: time-1.4.2, 1.4.1, 1.4.0.2, 1.4.0.1, 1.4 (conflict: vector =>
deepseq==1.4.1.1/installed-936..., time => deepseq>=1.1 && <1.4)
rejecting: time-1.3, 1.2.0.5, 1.2.0.4, 1.2.0.3, 1.2.0.2, 1.2.0.1, 1.2, 1.1.4,
1.1.3, 1.1.2.4, 1.1.2.3, 1.1.2.2, 1.1.2.1, 1.1.2.0, 1.0 (conflict: haxl =>
time==1.4.*)
Dependency tree exhaustively searched.

dependent data sources?

perhaps i've missed something - I have some datasources that rely on other datasources to get their data. Is there some way of indicating in the Datasource implementation that fetch should be a Haxl expression, rather than just IO?

dependency version issue when build

:Haxl/ (master✗) $ ./Setup configure [16:14:51]
Configuring haxl-0.3.1.0...
Setup: Encountered missing dependencies:
aeson >=0.6 && <0.12, exceptions ==0.8., text >=1.1.0.1 && <1.3
:Haxl/ (master✗) $ cabal install aeson [16:14:53]
Resolving dependencies...
All the requested packages are already installed:
aeson-0.11.2.0
Use --reinstall if you want to reinstall anyway.
:Haxl/ (master✗) $ cabal install exceptions [16:15:02]
Resolving dependencies...
All the requested packages are already installed:
exceptions-0.8.3
Use --reinstall if you want to reinstall anyway.
:Haxl/ (master✗) $ cabal install text [16:15:10]
Resolving dependencies...
All the requested packages are already installed:
text-1.2.2.1
Use --reinstall if you want to reinstall anyway.
:Haxl/ (master✗) $ ./Setup configure [16:15:17]
Configuring haxl-0.3.1.0...
Setup: Encountered missing dependencies:
aeson >=0.6 && <0.12, exceptions ==0.8., text >=1.1.0.1 && <1.3

Broken Hackage links

I'm not sure if this is the right place to report this, but with the new Haxl release there are a couple of broken links in hackage:

• https://hackage.haskell.org/package/haxl-2.0.0.0/haxl-2.0.0.0/srcexample/sql/readme.md
• https://hackage.haskell.org/package/haxl-2.0.0.0/haxl-2.0.0.0/srcexample/facebook/readme.md

I am happy to raise a PR if needed

using haxl for data sources where dependencies are not statically known

I have a learning algorithm that chooses variables to make decisions on that i'm trying to integrate into my Haxl-based project. The snag here is that the learning alg chooses those variable dynamically, making it hard to write a datasource that doesn't just fetch everything it might look at.

I think I can see a way around this using template haskell and introspecting on the generated decision tree, but if I'm overcomplicating it I'd love to hear about it.

tight upper bounds for ghc-8.6.1

while building with ghc-8.6.1 we get the following warnings with --allow-newer

WARNING: Ignoring out of range dependency (allow-newer enabled): containers-0.6.0.1. haxl requires: ==0.5.*
WARNING: Ignoring out of range dependency (allow-newer enabled): stm-2.5.0.0. haxl requires: ==2.4.*

Why the software history was not kept?

Hi there,

I'm a researcher studying software evolution. As part of my current research, I'm studying the implications of open-sourcing a proprietary software, for instance, if the project succeed in attracting newcomers. However, I observed that some projects, like Haxl, deleted their software history.

74a3874

Knowing that software history is indispensable for developers (e.g., developers need to refer to history several times a day), I would like to ask Haxl developers the following four brief questions:

1. Why did you decide to not keep the software history?
2. Do the core developers faced any kind of problems, when trying to refer to the old history? If so, how did they solve these problems?
3. Do the newcomers faced any kind of problems, when trying to refer to the old history? If so, how did they solve these problems?
4. How does the lack of history impacted on software evolution? Does it placed any burden in understanding and evolving the software?

Thanks in advance for your collaboration,

Gustavo Pinto, PhD
http://www.gustavopinto.org

Missing test file: sigma/haxl/core/tests/LoadCache.txt

Missing test file:

tests/TestExampleDataSource.hs:  loadcache <- readFile "sigma/haxl/core/tests/LoadCache.txt"

Adding MonadMask instance

Is there any obstacle preventing addition of MoandMask instance for GenHaxl u w ? It already has MonadThrow and MonadCatch instances.

My usecase for this is that I would like to dump caches to file at the end of GenHaxl computation, to be able to reproduce any problem that ocurred. I would really like to use finally there, but it requires MonadMask instance.

logging?

Not really an issue, just a question - how do you handle logging? I can see ifTrace in there, but it's going to return something in GenHaxl - is it legit to hook that up to unsafeLiftIO and use something like monad-logger?

Memory consumption depending on type annotation

Hi!

While experimenting with Haxl for our MSc thesis we ran into a weird memory issue that we can only reproduce with a combination of Haxl and stack-run. We cannot reproduce with only Haxl (without using stack-run), or purely with only stack-run (no Haxl involved), but are ot sure exactly where the problem lies.

A strict foldl' eats lots of memory depending on whether there is a type annotation for the result or not. See this gist for a minimal code example. The issue persists over different machines.

No way to update or invalidate cache

I'm using Haxl to talk to a REST API, and for the sake of example, we can pretend like there are two endpoints. One endpoint lets me fetch a Message given an ID, and another endpoint lets me fetch, say, n messages that were sent before a given ID.

data FooRequest a where
    GetMessage :: Id -> FooRequest Message
    GetMessagesBefore :: -> Id -> Int -> FooRequest [Message]

Whenever I perform a GetMessagesBefore request, I would like to be able to cache the results of this request such that performing a GetMessage request with an id fetched previously via GetMessagesBefore hits the cache.

I can do something like:

-- let's pretend we have `env`
getMessagesBefore id_ count = runHaxl env $ do
    res <- dataFetch (GetMessagesBefore id_ count)
    for res $ \message -> cacheRequest (GetMessage (messageId message)) (Right message)

However, cacheRequest throws an exception if the request has already been cached.

1. Is there some fundamental reason why we shouldn't be updating the cache? I can see nothing regarding cache invalidation anywhere in the Haxl codebase. I think I can probably whip something up to be able to update the cache, but I'd rather not if it's going to break any invariants.
2. Having two different requests that fetch the same things where one fetches a single thing and the other multiple things seems like a pretty common occurrence, and having such requests intelligently share cache would be a cool thing to have in the library. Are there any plans to implement something like this?

Thanks for your time!

Can equal Requests be sent in same round of fetch?

From documentation it is clear that fetch requests are cached between rounds of fetch. But It is not clear (at least for me) if DataSource fetch can get more than one equal Requests. For example

data SqlRequest a where
    RequestComments :: PostId -> SqlRequest [Comment]

instance DataSource () SqlRequest where
   fetch state _ _ fetches =  -- can fetches here contain more than one RequestComments to same PostId?

This package is not licensed under BSD3

The Cabal file declares this code to be licensed under the BSD-3-Clause license, but that is actually not the case. The code is licensed under BSD3 plus Facebook's "patent termination clause", which is an entirely different thing.

Non-deterministic failure in collectsdata test

On most builds The collectsdata test passes fine, but occasionally it fails with no error output.

Feature request: extend Haxl to support streaming throughout the tree

If Haxl is implemented to support streaming throughout the tree, from each leaf up to the root. With this feature, Haxl will (?) be usable as a framework to perform operations on a data which does not necessary fit into the memory.

Alternative Typeclass Implementation?

Pardon my ignorance if there's an obvious reason not to do this, I'm really new to functional programming, but I was playing around with porting Haxl to different environments (in particular, Purescript to try to ease in the nodejs community to the idea), and I realized that it's pretty straightforward to implement a <|> operator. This implementation is Purescript so I'm not sure how well it translates to real Haskell

https://github.com/jqyu/purescript-rad/blob/master/src/Rad/Core/Monad.purs#L236-L243

I don't know much about the Haskell runtime but I suppose the fact that it's a recursive implementation may make memory bounds hard to predict, but this allows for a graceful way of recovering from errors, as well as straightforward implementations of retried requests (i.e., chaining a fixed number of <|> calls to catch failures)

In particular, I'm working on a GraphQL implementation using Haxl and it would be really useful for me to be able to catch errors and return a null result for a branch of a query so I figured other people might be able to benefit from this as well.

runHaxl stuck

I am getting stuck right when I call runHaxl. I enabled tracing and the only output is:

schedule: 1

What usually causes this?

Rename Show1 to ShowP

I propose to rename Show1 class to ShowP or ShowU (for "show parametric" or "show universally").

the current Show1 clashes with Show1 class in base, which is a bit un-ergonomic.

Don't rely on `Show` to be valid Haskell code

dumpCacheAsHaskell is a neat thing. Unfortunately this relies on the fact that Show emits valid code. Not always are the instances under control of the developer. Just recently I tried to dumpCacheAsHaskell with types from the time library which resulted in invalid Haskell code being generated.

Could we introduce a new type class, maybe with a default implementation for types that are showable?

haxl failing to compile with ghc-8.2 rc2

While trying to compile with ghc-8.2

    /tmp/stack12680/haxl-0.5.0.0/Haxl/Core/Types.hs:112:1: error:
        Could not find module ‘Data.Typeable.Internal’
        it is a hidden module in the package ‘base-4.10.0.0’
        Use -v to see a list of the files searched for.
        |
    112 | import Data.Typeable.Internal
        | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Replacing the import with Data.Typeable seemed to fix the issue.

compiler : The Glorious Glasgow Haskell Compilation System, version 8.2.0.20170507

Real World SQL API

I'm trying to get Haxl running against a real database from the sql example.

Is this the only function you need to implement?

sql :: SQLResult a => String -> IO a

Browsing this example it seems the user of the library keeps track of the DB connection pool, not Haxl?

DataCacheTest fails to build with executable profiling enabled

Most relevant details:

• version 2.3.0 (identical code present in current main branch)
• Building in nix sandbox with ghc 8.10.4.
• This only happens if executable profiling is enabled, with just library profiling everything compiles normally.

tests/DataCacheTest.hs:33:15: error:
    • Couldn't match type ‘GHC.Ptr.Ptr GHC.Stack.CCS.CostCentreStack
                           -> IVar u0 w0 a’
                     with ‘IVar u w a’
      Expected type: IO (IVar u w a)
        Actual type: IO
                       (GHC.Ptr.Ptr GHC.Stack.CCS.CostCentreStack -> IVar u0 w0 a)
    • In the expression: IVar <$> newIORef (IVarFull (Ok a NilWrites))
      In an equation for ‘newResult’:
          newResult a = IVar <$> newIORef (IVarFull (Ok a NilWrites))
    • Relevant bindings include
        a :: a (bound at tests/DataCacheTest.hs:33:11)
        newResult :: a -> IO (IVar u w a)
          (bound at tests/DataCacheTest.hs:33:1)
   |
33 | newResult a = IVar <$> newIORef (IVarFull (Ok a NilWrites))
   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests/DataCacheTest.hs:36:13: error:
    • The constructor ‘IVar’ should have 2 arguments, but has been given 1
    • In the pattern: IVar ref
      In an equation for ‘takeResult’:
          takeResult (IVar ref)
            = do e <- readIORef ref
                 case e of
                   IVarFull a -> return a
                   _ -> error "takeResult"
   |
36 | takeResult (IVar ref) = do
   |             ^^^^^^^^

Complete build log: haxl-build-failure.log

With a little work I can supply a minimal nix derivation to reproduce the issue.

Current workaround is to pretend that haxl doesn't have any executables so we don't enable executable profiling.

There is haxl-0.3.1.0 on the Hackage

... but this repository has version: 0.3.1.0. Is the development branched privately?

http://hdiff.luite.com/cgit/haxl/diff?id=0.3.1.0&id2=0.3.0.0 (seems that not, but still weird).

2.1.*.0 tags and Hackage release

Just a reminder :D

Building Error

When I was trying to build the Haxl-2.3.0.0 using stack, I got this error message.
The version of GHC for building this is 8.8.3

Haxl\Core\Exception.hs:303:20: error:
haxl                >     Ambiguous occurrence `MonadFail'
haxl                >     It could refer to
haxl                >        either `Prelude.MonadFail',
haxl                >               imported from `Prelude' at Haxl\Core\Exception.hs:31:8-26
haxl                >               (and originally defined in `Control.Monad.Fail')
haxl                >            or `Haxl.Core.Exception.MonadFail',

haxl > defined at Haxl\Core\Exception.hs:300:1