GithubHelp home page GithubHelp logo

haxl's Introduction

Haxl Logo

Haxl

Support Ukraine Build Status

Haxl is a Haskell library that simplifies access to remote data, such as databases or web-based services. Haxl can automatically

  • batch multiple requests to the same data source,
  • request data from multiple data sources concurrently,
  • cache previous requests,
  • memoize computations.

Having all this handled for you behind the scenes means that your data-fetching code can be much cleaner and clearer than it would otherwise be if it had to worry about optimizing data-fetching. We'll give some examples of how this works in the pages linked below.

There are two Haskell packages here:

To use Haxl in your own application, you will likely need to build one or more data sources: the thin layer between Haxl and the data that you want to fetch, be it a database, a web API, a cloud service, or whatever.

There is a generic datasource in "Haxl.DataSource.ConcurrentIO" that can be used for performing arbitrary IO operations concurrently, given a bit of boilerplate to define the IO operations you want to perform.

The haxl-facebook package shows how we might build a Haxl data source based on the existing fb package for talking to the Facebook Graph API.

Where to go next?

Contributing

We welcome contributions! See CONTRIBUTING for details on how to get started, and our Code of Conduct.

License

Haxl uses the BSD 3-clause License, as found in the LICENSE file.

haxl's People

Contributors

adarqui avatar aledista avatar alexbiehl avatar algoriddle avatar buridiaditya avatar christopherbiscardi avatar crdrost avatar dmitryvinn avatar dylanza avatar jchia avatar jda avatar jfischoff avatar jmorag avatar joncoens avatar josefs avatar juhp avatar juzley avatar jvanbruegge avatar karen avatar katiejots avatar niteria avatar phadej avatar rayshih avatar richard-zhang avatar ruippeixotog avatar simonmar avatar watashi avatar xich avatar zilberstein avatar zudov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

haxl's Issues

bounded caches

Hi there,

I've been using this great library for some time now, thanks for opening it up to the public!

My use case involves large amounts of data, so I had to change the library slightly to be able to use a bounded cache. I have made it configurable so that each type of request can either use a bounded cache of a given size, or an unbounded cache.

In order to configure it, I have added an additional typeclass that has a DataSource has to be an instance of. It has a single function, cacheSize, that determines the size of the cache for each request (which may be infinite).

If this is useful to other people, I'd be glad to do a pull request. Notice however, that this would be a breaking change, since all DataSources have to add an instance of the new typeclass (which is easy to do).

I have not yet added tests that check that the cache stays within the specified bounds, nor done benchmarks (but it's on my agenda).

Best,
Philipp

dependent data sources?

perhaps i've missed something - I have some datasources that rely on other datasources to get their data. Is there some way of indicating in the Datasource implementation that fetch should be a Haxl expression, rather than just IO?

monad law `(<*>) = ap` is ignored?

I have taken look at some talks of @simonmar about haxl:

  1. Keynote from Simon Marlow - Fun with Haxl
  2. The Haskell Cast #4 - Simon Marlow on Parallelism and Concurrency

And looks like Haxl is not lawful as it's ignoring law (<*>) = ap.
also as (>>) = (*>) is implied by that law it causes issues like one in first link (1.) about sequencing monadic actions in do even when applicative do was off:

run $ do
  cmd "touch" ["foo"]
  cmd "rm" ["foo"]

And there was need to add _ <- in order that to be sequential:

run $ do
  _ <- cmd "touch" ["foo"]
  cmd "rm" ["foo"]

My question are:

  • Is it intentional to ignore that law?
  • Can Haxl be lawful and also produce same benefits?
  • How often is it ignored in haskell libraries?
  • What are benefits of that law?

Also I have taken look at the original paper about Applicative and did get reasons why this law was intorduced, so if you know any discussion on that topic you can link would be great!

Also in js community some implementations of Futures also Ignore that law (fantasy-land#179), looks like this law is not that widely known.

Up dependency versions to allow build on GHC 7.10.1

cabal install failed until I upped filepath to 1.4.* and time to 1.5.*

I'm on GHC 7.10.1.

$ cabal install
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: haxl-0.1.0.0 (user goal)
trying: vector-0.10.12.3/installed-cbb... (dependency of haxl-0.1.0.0)
next goal: time (dependency of haxl-0.1.0.0)
rejecting: time-1.5.0.1/installed-f60..., 1.5.0.1, 1.5 (conflict: haxl =>
time==1.4.*)
rejecting: time-1.4.2, 1.4.1, 1.4.0.2, 1.4.0.1, 1.4 (conflict: vector =>
deepseq==1.4.1.1/installed-936..., time => deepseq>=1.1 && <1.4)
rejecting: time-1.3, 1.2.0.5, 1.2.0.4, 1.2.0.3, 1.2.0.2, 1.2.0.1, 1.2, 1.1.4,
1.1.3, 1.1.2.4, 1.1.2.3, 1.1.2.2, 1.1.2.1, 1.1.2.0, 1.0 (conflict: haxl =>
time==1.4.*)
Dependency tree exhaustively searched.

Real World SQL API

I'm trying to get Haxl running against a real database from the sql example.

Is this the only function you need to implement?

sql :: SQLResult a => String -> IO a

Browsing this example it seems the user of the library keeps track of the DB connection pool, not Haxl?

Alternative Typeclass Implementation?

Pardon my ignorance if there's an obvious reason not to do this, I'm really new to functional programming, but I was playing around with porting Haxl to different environments (in particular, Purescript to try to ease in the nodejs community to the idea), and I realized that it's pretty straightforward to implement a <|> operator. This implementation is Purescript so I'm not sure how well it translates to real Haskell

https://github.com/jqyu/purescript-rad/blob/master/src/Rad/Core/Monad.purs#L236-L243

I don't know much about the Haskell runtime but I suppose the fact that it's a recursive implementation may make memory bounds hard to predict, but this allows for a graceful way of recovering from errors, as well as straightforward implementations of retried requests (i.e., chaining a fixed number of <|> calls to catch failures)

In particular, I'm working on a GraphQL implementation using Haxl and it would be really useful for me to be able to catch errors and return a null result for a branch of a query so I figured other people might be able to benefit from this as well.

Can equal Requests be sent in same round of fetch?

From documentation it is clear that fetch requests are cached between rounds of fetch. But It is not clear (at least for me) if DataSource fetch can get more than one equal Requests. For example

data SqlRequest a where
    RequestComments :: PostId -> SqlRequest [Comment]

instance DataSource () SqlRequest where
   fetch state _ _ fetches =  -- can fetches here contain more than one RequestComments to same PostId?

Don't rely on `Show` to be valid Haskell code

dumpCacheAsHaskell is a neat thing. Unfortunately this relies on the fact that Show emits valid code. Not always are the instances under control of the developer. Just recently I tried to dumpCacheAsHaskell with types from the time library which resulted in invalid Haskell code being generated.

Could we introduce a new type class, maybe with a default implementation for types that are showable?

haxl failing to compile with ghc-8.2 rc2

While trying to compile with ghc-8.2

    /tmp/stack12680/haxl-0.5.0.0/Haxl/Core/Types.hs:112:1: error:
        Could not find module ‘Data.Typeable.Internal’
        it is a hidden module in the package ‘base-4.10.0.0’
        Use -v to see a list of the files searched for.
        |
    112 | import Data.Typeable.Internal
        | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Replacing the import with Data.Typeable seemed to fix the issue.

compiler : The Glorious Glasgow Haskell Compilation System, version 8.2.0.20170507

Why the software history was not kept?

Hi there,

I'm a researcher studying software evolution. As part of my current research, I'm studying the implications of open-sourcing a proprietary software, for instance, if the project succeed in attracting newcomers. However, I observed that some projects, like Haxl, deleted their software history.

74a3874

Knowing that software history is indispensable for developers (e.g., developers need to refer to history several times a day), I would like to ask Haxl developers the following four brief questions:

  1. Why did you decide to not keep the software history?
  2. Do the core developers faced any kind of problems, when trying to refer to the old history? If so, how did they solve these problems?
  3. Do the newcomers faced any kind of problems, when trying to refer to the old history? If so, how did they solve these problems?
  4. How does the lack of history impacted on software evolution? Does it placed any burden in understanding and evolving the software?

Thanks in advance for your collaboration,

Gustavo Pinto, PhD
http://www.gustavopinto.org

runHaxl stuck

I am getting stuck right when I call runHaxl. I enabled tracing and the only output is:

schedule: 1

What usually causes this?

DataCacheTest fails to build with executable profiling enabled

Most relevant details:

  • version 2.3.0 (identical code present in current main branch)
  • Building in nix sandbox with ghc 8.10.4.
  • This only happens if executable profiling is enabled, with just library profiling everything compiles normally.
tests/DataCacheTest.hs:33:15: error:
    • Couldn't match type ‘GHC.Ptr.Ptr GHC.Stack.CCS.CostCentreStack
                           -> IVar u0 w0 a’
                     with ‘IVar u w a’
      Expected type: IO (IVar u w a)
        Actual type: IO
                       (GHC.Ptr.Ptr GHC.Stack.CCS.CostCentreStack -> IVar u0 w0 a)
    • In the expression: IVar <$> newIORef (IVarFull (Ok a NilWrites))
      In an equation for ‘newResult’:
          newResult a = IVar <$> newIORef (IVarFull (Ok a NilWrites))
    • Relevant bindings include
        a :: a (bound at tests/DataCacheTest.hs:33:11)
        newResult :: a -> IO (IVar u w a)
          (bound at tests/DataCacheTest.hs:33:1)
   |
33 | newResult a = IVar <$> newIORef (IVarFull (Ok a NilWrites))
   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests/DataCacheTest.hs:36:13: error:
    • The constructor ‘IVar’ should have 2 arguments, but has been given 1
    • In the pattern: IVar ref
      In an equation for ‘takeResult’:
          takeResult (IVar ref)
            = do e <- readIORef ref
                 case e of
                   IVarFull a -> return a
                   _ -> error "takeResult"
   |
36 | takeResult (IVar ref) = do
   |             ^^^^^^^^

Complete build log: haxl-build-failure.log

With a little work I can supply a minimal nix derivation to reproduce the issue.

Current workaround is to pretend that haxl doesn't have any executables so we don't enable executable profiling.

No way to update or invalidate cache

I'm using Haxl to talk to a REST API, and for the sake of example, we can pretend like there are two endpoints. One endpoint lets me fetch a Message given an ID, and another endpoint lets me fetch, say, n messages that were sent before a given ID.

data FooRequest a where
    GetMessage :: Id -> FooRequest Message
    GetMessagesBefore :: -> Id -> Int -> FooRequest [Message]

Whenever I perform a GetMessagesBefore request, I would like to be able to cache the results of this request such that performing a GetMessage request with an id fetched previously via GetMessagesBefore hits the cache.

I can do something like:

-- let's pretend we have `env`
getMessagesBefore id_ count = runHaxl env $ do
    res <- dataFetch (GetMessagesBefore id_ count)
    for res $ \message -> cacheRequest (GetMessage (messageId message)) (Right message)

However, cacheRequest throws an exception if the request has already been cached.

  1. Is there some fundamental reason why we shouldn't be updating the cache? I can see nothing regarding cache invalidation anywhere in the Haxl codebase. I think I can probably whip something up to be able to update the cache, but I'd rather not if it's going to break any invariants.
  2. Having two different requests that fetch the same things where one fetches a single thing and the other multiple things seems like a pretty common occurrence, and having such requests intelligently share cache would be a cool thing to have in the library. Are there any plans to implement something like this?

Thanks for your time!

tight upper bounds for ghc-8.6.1

while building with ghc-8.6.1 we get the following warnings with --allow-newer

WARNING: Ignoring out of range dependency (allow-newer enabled): containers-0.6.0.1. haxl requires: ==0.5.*
WARNING: Ignoring out of range dependency (allow-newer enabled): stm-2.5.0.0. haxl requires: ==2.4.*

Memory consumption depending on type annotation

Hi!

While experimenting with Haxl for our MSc thesis we ran into a weird memory issue that we can only reproduce with a combination of Haxl and stack-run. We cannot reproduce with only Haxl (without using stack-run), or purely with only stack-run (no Haxl involved), but are ot sure exactly where the problem lies.

A strict foldl' eats lots of memory depending on whether there is a type annotation for the result or not. See this gist for a minimal code example. The issue persists over different machines.

using haxl for data sources where dependencies are not statically known

I have a learning algorithm that chooses variables to make decisions on that i'm trying to integrate into my Haxl-based project. The snag here is that the learning alg chooses those variable dynamically, making it hard to write a datasource that doesn't just fetch everything it might look at.

I think I can see a way around this using template haskell and introspecting on the generated decision tree, but if I'm overcomplicating it I'd love to hear about it.

Plans for a monad transformer interface?

Currently, the fetch method of the type class lives in IO. Flaxl showed that it is possible to free it into any monad by relaxing the constraints from IO. Any plans for Haxl to move to or also provide a monad transformer interface?

dependency version issue when build

:Haxl/ (master✗) $ ./Setup configure [16:14:51]
Configuring haxl-0.3.1.0...
Setup: Encountered missing dependencies:
aeson >=0.6 && <0.12, exceptions ==0.8., text >=1.1.0.1 && <1.3
:Haxl/ (master✗) $ cabal install aeson [16:14:53]
Resolving dependencies...
All the requested packages are already installed:
aeson-0.11.2.0
Use --reinstall if you want to reinstall anyway.
:Haxl/ (master✗) $ cabal install exceptions [16:15:02]
Resolving dependencies...
All the requested packages are already installed:
exceptions-0.8.3
Use --reinstall if you want to reinstall anyway.
:Haxl/ (master✗) $ cabal install text [16:15:10]
Resolving dependencies...
All the requested packages are already installed:
text-1.2.2.1
Use --reinstall if you want to reinstall anyway.
:Haxl/ (master✗) $ ./Setup configure [16:15:17]
Configuring haxl-0.3.1.0...
Setup: Encountered missing dependencies:
aeson >=0.6 && <0.12, exceptions ==0.8.
, text >=1.1.0.1 && <1.3

Rename Show1 to ShowP

I propose to rename Show1 class to ShowP or ShowU (for "show parametric" or "show universally").

the current Show1 clashes with Show1 class in base, which is a bit un-ergonomic.

Inaccurate lower bound on `base`

haxl.cabal claims compatiblity with base == 4.*, however GHC 7.8.4 falsifies this:

Configuring library for haxl-2.0.1.0..
Preprocessing library for haxl-2.0.1.0..
Building library for haxl-2.0.1.0..
[ 1 of 18] Compiling Haxl.Core.Util   ( Haxl/Core/Util.hs, /tmp/matrix-worker/1535645265/dist-newstyle/build/x86_64-linux/ghc-7.6.3/haxl-2.0.1.0/build/Haxl/Core/Util.o )
[ 2 of 18] Compiling Haxl.Core.DataCache ( Haxl/Core/DataCache.hs, /tmp/matrix-worker/1535645265/dist-newstyle/build/x86_64-linux/ghc-7.6.3/haxl-2.0.1.0/build/Haxl/Core/DataCache.o )
[ 3 of 18] Compiling Haxl.Core.StateStore ( Haxl/Core/StateStore.hs, /tmp/matrix-worker/1535645265/dist-newstyle/build/x86_64-linux/ghc-7.6.3/haxl-2.0.1.0/build/Haxl/Core/StateStore.o )

Haxl/Core/StateStore.hs:44:19:
    Not in scope: type constructor or class `Proxy'

Haxl/Core/StateStore.hs:45:18: Not in scope: `typeRep'

Haxl/Core/StateStore.hs:61:41:
    Not in scope: data constructor `Proxy'

Haxl/Core/StateStore.hs:61:50:
    Not in scope: type constructor or class `Proxy'

Haxl/Core/StateStore.hs:69:24:
    Not in scope: data constructor `Proxy'

Haxl/Core/StateStore.hs:69:33:
    Not in scope: type constructor or class `Proxy'

Haxl/Core/StateStore.hs:75:22:
    Not in scope: data constructor `Proxy'

Haxl/Core/StateStore.hs:75:31:
    Not in scope: type constructor or class `Proxy'
<<ghc: 1030926808 bytes, 1905 GCs, 15138471/60948896 avg/max bytes residency (8 samples), 118M in use, 0.00 INIT (0.00 elapsed), 0.58 MUT (0.69 elapsed), 0.59 GC (0.59 elapsed) :ghc>>

I've already revised the affected releases on Hackage,

so there's no immediate need for action on your part; however please take this into account for future releases of haxl.

test suite build failure

As seen on the Stackage build server:

Preprocessing test suite 'test' for haxl-0.4.0.0...
[1 of 1] Compiling Main             ( tests/TestMain.hs, dist/build/test/test-tmp/Main.o )

tests/TestMain.hs:6:1: error:
    Failed to load interface for ‘AllTests’
    Use -v to see a list of the files searched for.

Building Error

When I was trying to build the Haxl-2.3.0.0 using stack, I got this error message.
The version of GHC for building this is 8.8.3

Haxl\Core\Exception.hs:303:20: error:
haxl                >     Ambiguous occurrence `MonadFail'
haxl                >     It could refer to
haxl                >        either `Prelude.MonadFail',
haxl                >               imported from `Prelude' at Haxl\Core\Exception.hs:31:8-26
haxl                >               (and originally defined in `Control.Monad.Fail')
haxl                >            or `Haxl.Core.Exception.MonadFail',

haxl > defined at Haxl\Core\Exception.hs:300:1

This package is not licensed under BSD3

The Cabal file declares this code to be licensed under the BSD-3-Clause license, but that is actually not the case. The code is licensed under BSD3 plus Facebook's "patent termination clause", which is an entirely different thing.

logging?

Not really an issue, just a question - how do you handle logging? I can see ifTrace in there, but it's going to return something in GenHaxl - is it legit to hook that up to unsafeLiftIO and use something like monad-logger?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.