GithubHelp home page GithubHelp logo

tdbgamer / flink-statefulfun-hs Goto Github PK

View Code? Open in Web Editor NEW
13.0 4.0 3.0 140 KB

Haskell SDK for Flink stateful functions

License: Mozilla Public License 2.0

Dockerfile 4.97% Haskell 62.36% Python 25.45% Nix 6.58% Shell 0.64%

flink-statefulfun-hs's Introduction

Flink Stateful Functions Haskell SDK

Hackage Build Join the chat at https://gitter.im/tdbgamer/flink-statefulfun-hs

Provides a typed API for creating flink stateful functions. Greeter example is in example/greeter/main/Main.hs

How to run example

cd example
docker-compose build
docker-compose up -d
docker-compose logs -f event-generator

How to compile locally

  1. Install nix
  2. Install cachix
  3. Setup nix cache.
cachix use iohk
cachix use static-haskell-nix
cachix use flink-statefulfun
  1. Compile inside a nix shell.
nix-shell
cabal build

Tutorial

Define our protobuf messages

// Example.proto
syntax = "proto3";

package example;

message GreeterRequest {
  string name = 1;
}

message GreeterResponse {
  string greeting = 1;
}

Declare a function

import Network.Flink.Stateful
import qualified Proto.Example as EX
import qualified Proto.Example_Fields as EX

printer :: StatefulFunc () m => EX.GreeterResponse -> m ()
printer msg = liftIO $ print msg

This declares a simple function that takes an GreeterResponse protobuf type as an argument and simply prints it. StatefulFunc makes this a Flink stateful function with a state type of () (meaning it requires no state).

Declaring a function with state

import Data.Aeson (FromJSON, ToJSON)
import Data.Text (Text)
import qualified Data.Text as T
import GHC.Generics

newtype GreeterState = GreeterState
  { greeterStateCount :: Int
  }
  deriving (Generic, Show)

instance ToJSON GreeterState
instance FromJSON GreeterState

counter :: StatefulFunc GreeterState m => EX.GreeterRequest -> m ()
counter msg = do
  newCount <- (+ 1) <$> insideCtx greeterStateCount
  let respMsg = "Saw " <> T.unpack name <> " " <> show newCount <> " time(s)"

  sendMsgProto ("printing", "printer") (response $ T.pack respMsg)
  modifyCtx (\old -> old {greeterStateCount = newCount})
  where
    name = msg ^. EX.name
    response :: Text -> EX.GreeterResponse
    response greeting =
      defMessage
        & EX.greeting .~ greeting

The StatefulFunc typeclass gives us access to the GreeterState that we are sending to and from Flink on every batch of incoming messages our function receives. For every message, this function will calculate its new count, send a message to the printer function we made earlier, then update its state with the new count.

Internally this is batched over many events before sending state back to Flink for efficiency.

NOTE: For JSON (or anything other than protobuf) messages, you must use sendByteMsg instead. When communicating with other SDKs, you'll likely want to use sendMsg and protobuf.

Serve HTTP API

import qualified Data.ByteString.Lazy.Char8 as BSL
import qualified Data.Map as Map
import Network.Wai.Handler.Warp (run)
import Network.Wai.Middleware.RequestLogger

main :: IO ()
main = do
  putStrLn "http://localhost:8000/"
  run 8000 (logStdout $ createApp functionTable)

functionTable :: FunctionTable
functionTable =
  Map.fromList
    [ ((FuncType "greeting" "greeterEntry"), flinkWrapper () (Expiration NONE 0) (greeterEntry . getProto)),
      ((FuncType  "greeting" "counter"), flinkWrapper (JsonSerde (GreeterState 0)) (Expiration AFTER_CALL 5) (jsonState . counter . getProto))
    ]

The FunctionTable is a Map of (namespace, functionType) to wrappedFunction. jsonState is a helper to serialize your function state as JSON for storage in the flink backend. protoState can also be used if that is your preference. flinkWrapper transforms your function into one that takes arbitrary ByteStrings so that every function in the FunctionTable Map is homogenous. protoInput indicates that the input message should be deserialized as protobuf. jsonInput can be used instead to deserialize the messages as JSON.

createApp is used to turn the FunctionTable into a Warp Application which can be served using the run function provided by Warp.

NOTE: JSON messages may not play nice with other SDKs, you'll probably want to stick with protobuf if you're communicating with another SDK without knowing too much Flink Statefun internals.

Create module YAML

To use the functions that are now served from the API, we now need to declare it in the module.yaml.

version: "3.0"
module:
  meta:
    type: remote
  spec:
    endpoints:
      - endpoint:
        meta: 
          kind: http
        spec:
          functions: greeting/*
          urlPathTemplate: http://localhost:8000/statefun

Flink Statefun supports multiple states, but for simplicity the SDK just serializes the whole record and hard codes flink_state as the only state value it uses.

flink-statefulfun-hs's People

Contributors

gitter-badger avatar nathantalewis avatar sigevsky avatar timbess avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

flink-statefulfun-hs's Issues

Building the greeter example with Docker fails

I'm trying to modify the example and to build it locally but it fails with a COPY failed: stat /var/lib/docker/tmp/docker-builder708422819/shell.nix: no such file or directory

The full build log:

Sending build context to Docker daemon  32.26kB
Step 1/19 : FROM nixos/nix
latest: Pulling from nixos/nix
df20fa9351a1: Already exists
8cc8f1d84c41: Pull complete
eea3194069c7: Pull complete
Digest: sha256:2da921898aa6c89e2e60b1fb72d74525b8464b47412482c7f1cf77b8e707a099
Status: Downloaded newer image for nixos/nix:latest
# Executing 1 build trigger
 ---> Running in 241cd4864085
Removing intermediate container 241cd4864085
 ---> a82ef7225748
Step 2/19 : RUN nix-env -iA cachix -f https://cachix.org/api/v1/install
 ---> Running in ffaa84d7180f
unpacking 'https://cachix.org/api/v1/install'...
installing 'cachix-0.3.8'
copying path '/nix/store/sgmi81aggspycw7h7yf547bfrzs8h6g4-busybox-1.31.1-x86_64-unknown-linux-musl' from 'https://cache.nixos.org'...
copying path '/nix/store/sf0gi1lli224jq76lfdbfnfzw2j6a2kr-aws-c-common-0.3.11' from 'https://cache.nixos.org'...
copying path '/nix/store/pgj5vsdly7n4rc8jax3x3sill06l44qp-libunistring-0.9.10' from 'https://cache.nixos.org'...
copying path '/nix/store/8ksw7mj0w4qjszjl47k91snr83zb7n70-nix-2.3.6-man' from 'https://cache.nixos.org'...
copying path '/nix/store/9l6d9k9f0i9pnkfjkvsm7xicpzn4cv2c-libidn2-2.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/jx19wa4xlh9n4324xdl9rjnykd19mmq3-glibc-2.30' from 'https://cache.nixos.org'...
copying path '/nix/store/mis0j0260qp8vykm5qpgjbvkcbd283kx-attr-2.4.48' from 'https://cache.nixos.org'...
copying path '/nix/store/37pgh4zsassk26g60n6m015y98kca6nb-aws-checksums-0.1.5' from 'https://cache.nixos.org'...
copying path '/nix/store/v60xxjdhf192hqm0hms2qgyip1hlbvac-acl-2.2.53' from 'https://cache.nixos.org'...
copying path '/nix/store/lf8nps4p125g6iiiyblxfvw3q86gv1zd-aws-c-event-stream-0.1.1' from 'https://cache.nixos.org'...
copying path '/nix/store/xfbmj7sl2ikicym9x3yq7cms5qx1w39k-bash-4.4-p23' from 'https://cache.nixos.org'...
copying path '/nix/store/wmc6hyssi3smi66v15iviw4d1ph8cpp7-brotli-1.0.7-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/1a9sliyg4bf53ngk4wli9ypii835sh0x-bzip2-1.0.6.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/id23l2prxafywpps8ssh3gq2wn6p84gi-coreutils-8.31' from 'https://cache.nixos.org'...
copying path '/nix/store/r59h4sc1qgivrghi08hkwcsiyf9f8bkl-bzip2-1.0.6.0.1-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/lhygkmgsz3v7fy586yrvr90kn4f4yp28-editline-1.17.0' from 'https://cache.nixos.org'...
copying path '/nix/store/a6z7ighixg7gb6krf9k60ylgmahij63x-gcc-9.3.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/5mp196fls2rkna26wiglnszhnpjj2glz-gnutar-1.32' from 'https://cache.nixos.org'...
copying path '/nix/store/398pmpbdjg0g2dbic40rrskbzpy5khb0-boehm-gc-8.0.4' from 'https://cache.nixos.org'...
copying path '/nix/store/hm5siiq4dh216r8z51ckijd1dhv9zs14-gmp-6.2.0' from 'https://cache.nixos.org'...
copying path '/nix/store/lb7zc6jhqvfnanzdhkbmd0j8cjx4ikyb-gzip-1.10' from 'https://cache.nixos.org'...
copying path '/nix/store/ca88npfrcpiibz7vlwkhj29hb4006pkk-icu4c-64.2' from 'https://cache.nixos.org'...
copying path '/nix/store/cs35zh08x2z95h7qmw86sqjwpyhpb97y-keyutils-1.6.1-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/qnk89y1996sbj1bhsxcia57x0iqwhgiq-libffi-3.3' from 'https://cache.nixos.org'...
copying path '/nix/store/50gghjlsn94j2bi8ckzj9bg4csb84hg4-libkrb5-1.18' from 'https://cache.nixos.org'...
copying path '/nix/store/adbfd78fav8z3xmar2pd95pdxxvzx6sd-libseccomp-2.4.3-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/ira0m9yd83pcqxk6sglz01zk2qb73xh3-libsodium-1.0.18' from 'https://cache.nixos.org'...
copying path '/nix/store/hc00jxav45p2simp6qz44z5djyv1m1fy-ncurses-6.2' from 'https://cache.nixos.org'...
copying path '/nix/store/k8bvpif710gvidqm160bf6ywiiyjy6ky-nghttp2-1.40.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/qnjnjw8q9wzyk2dhx715cmh6ysi72zq1-openssl-1.1.1g' from 'https://cache.nixos.org'...
copying path '/nix/store/p6990f23k9srnnc2mih3f3pjxjsvsh4q-xz-5.2.5' from 'https://cache.nixos.org'...
copying path '/nix/store/ml4ipdnvc7pr07dr6i35831f7ffxny0k-zlib-1.2.11' from 'https://cache.nixos.org'...
copying path '/nix/store/c4855s26ncyplqjn3snjwhpf6ai018vb-xz-5.2.5-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/jijd3hi94w5nwzrd9ihvs34q69hx0vcd-boost-1.69.0' from 'https://cache.nixos.org'...
copying path '/nix/store/7lfkc47n07g7xrhwrz8chhdhm4mdzbnq-libssh2-1.9.0' from 'https://cache.nixos.org'...
copying path '/nix/store/xyzx9y4wyv76r8q0rj4v9rk50kv9bgxr-sqlite-3.31.1' from 'https://cache.nixos.org'...
copying path '/nix/store/y5a5ca9a42nivq54h82vgn1n524nm1hb-curl-7.70.0' from 'https://cache.nixos.org'...
copying path '/nix/store/ir829710sw4isacpsbq6cwv3fcbl76nm-aws-sdk-cpp-1.7.90' from 'https://cache.nixos.org'...
copying path '/nix/store/43822pavpsvlnx6cfph58pxnhndjnmdd-nix-2.3.6' from 'https://cache.nixos.org'...
copying path '/nix/store/spznih45c56kfwygx8qyq1skd1rs4zv1-cachix-0.3.8' from 'https://cache.nixos.org'...
building '/nix/store/q5vnysc9mi1xzax3k58545h4fqxdyjvx-user-environment.drv'...
created 22 symlinks in user environment
Removing intermediate container ffaa84d7180f
 ---> 839e00c094f1
Step 3/19 : RUN cachix use iohk
 ---> Running in 26fbda827cf0
Configured https://iohk.cachix.org binary cache in /etc/nix/nix.conf
Removing intermediate container 26fbda827cf0
 ---> f4b6b461e042
Step 4/19 : RUN cachix use static-haskell-nix
 ---> Running in a0bb560b61dd
Configured https://static-haskell-nix.cachix.org binary cache in /etc/nix/nix.conf
Removing intermediate container a0bb560b61dd
 ---> 79d2c0ccb1bd
Step 5/19 : RUN cachix use flink-statefulfun
 ---> Running in 8ecb96ee4855
Configured https://flink-statefulfun.cachix.org binary cache in /etc/nix/nix.conf
Removing intermediate container 8ecb96ee4855
 ---> 3de5cf1c387f
Step 6/19 : RUN mkdir -p /app/example/greeter
 ---> Running in 030f9291880d
Removing intermediate container 030f9291880d
 ---> 9c23170a6878
Step 7/19 : WORKDIR /app/example/greeter
 ---> Running in 25dd0e8a0c7b
Removing intermediate container 25dd0e8a0c7b
 ---> 371931cc7c47
Step 8/19 : COPY shell.nix /app/example/greeter
COPY failed: stat /var/lib/docker/tmp/docker-builder708422819/shell.nix: no such file or directory```

Document how to get started/Use this library

Hi @tdbgamer,
What do you think about adding some basic information to the README (or any other place that you would find suitable), that describes how to use this SDK?

Perhaps something along the lines of Python SDK

I find the example to be a bit hard to differentiate what parts are relevant only to the example, what parts are SDK and what parts are the HTTP server that accepts the hosts the example+SDK.

Thanks.

Building the example with Docker fails

Hi,
I was trying to lunch the example via Docker, and building the image failed with:

 ---> Running in ba332fa93760
cabal: The program 'git' is required but it could not be found.```

Add Python artifacts to .gitignore

It seems like some compiled Python classes have made it into the repo,
you may want to consider adding __pycahce__ and *pyc to your .gitignore

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.