scalaz / scalaz-analytics

A high-performance, purely-functional library for doing computational analysis and statistics over data in a type-safe way

scalaz-analytics

Goal

Scalaz Analytics provides a high-performance, purely-functional library for doing computational analysis and statistics over data in a type-safe way.

Introduction & Highlights

Scalaz Analytics is a principled functional programming library for data processing and analytics.

• Simple and principled
• First class support for analytics and data science
• Pure type-safe, functional interface that integrates with other Scalaz projects
• Supports batch and streaming
• Efficient on both small and large data sets, single machine and distributed
• Can be used from a REPL for interactive analysis or as a library for applications

Other libraries

Below is a selection of Analytics/Data processing Libraries that we are being used as inspiration. Some of these metrics are somewhat subjective but they give an idea for what we are looking at from each library. Note that these metrics assume native support, so libraries that achieve these things via another library are not considered.

Library	Scales to Big Data	Supports Batch	Supports Streaming	FP	Easy to Debug	Out of the box analytics
Spark	✔	✔	✔ (mini batch)	✘	✘	✘
Flink	✔	✔	✔	✘	✘	✘
Pandas	✘	✔	✘	✘	✔	✘
R	✘	✔	✔	✘	?	✔
Dask	✔	✔	✔	✘	?	✘
Apex	✔	✔	✔	✘	?	✘
Beam	✔	✔	✔	✘	?	✘

Background

• Apache Spark - Internal Data Formats, Tungsten 1/Tungsten 2 and Catalyst optimisations, Mastering Apache Spark, Frameless - a library that adds type-safety to Spark
• Quark - Functional Data Processing DSL - presentation and video
• Flink documentation
• Apache Arrow - High performance in memory data platform
• Icicle - Optimisation of DSLs into efficient code for data processing talk
• Machine Fusion
• An algebra for distributed big data analytics - Leonidas Fegaras
• Compiling to categories - Conal Elliott
• Linear algebra for data processing and other similar work by José Nuno Oliveira
• Data cube as typed linear algebra operator
• Comprehensions syntax for query languages Monoid and others
• Efficient derivation of folds in haskells foldl
• Mining Massive Data Sets free online book
• Scipy stats package
• Concurrent commutative folds in Tesser
• Explanation of zero copy

scalaz-analytics's Issues

Add Datatypes/Data structures to stdlib

We need to provide a set of Type's that can be used in constructing computation descriptions in the StdLib.

As an initial list might be:

• Int
• Long
• Double
• String
• Boolean
• Byte
• Decimal
• Char
• Float
• Null
• Short
• Timestamp
• Date

Data Structures (we need to define the semantics of these since they are not concrete collections at this point)

• Array
• Map
• Set
• Tuple

Add methods to setops

We need to flesh out the setops trait in the design sketch.

This ticket will need to pay close attention to the progress of #4 since the approach we take could impact the types of setops that are possible with our abstraction.

Implement Arrow Ops

We need to add the Arrow Ops to the =>: and StdLib traits as mentioned in the design session.

=>: is a Strong/Choice Profunctor. The best links I've found to describe this are:
https://elvishjerricco.github.io/2017/03/10/profunctors-arrows-and-static-analysis.html
https://github.com/typelevel/cats/tree/master/core/src/main/scala/cats/arrow

General Usage Examples

This ticket is to capture the types of computations we want to be able to express with this library. I think it's probably best to have this as a discussion ticket for now and it will likely help feed the design of #4.

Numeric Syntax

Like the DatasetSyntax example, we will need syntax for Numerics

Think basic operators (+, -, /, *) etc

Consider Removal of Numeric

This refers to a discussion in gitter about Numeric (see https://gitter.im/scalaz/scalaz-analytics/archives/2018/09/26 and https://gitter.im/scalaz/scalaz-analytics/archives/2018/09/27)

Basically the issue is Numeric is not fully defined for very common numerical types e.g. Int not total over division, Double not having a modulo operation.

The consensus of the discussion seems to be that using algebraic abstractions would probably be cleaner. Even better would be to make sure that we can interoperate with scalaz-algebra, spire and algebird as easily as possible.

Right now implementing this would merely mean to remove the code about numeric. Perhaps we should tackle this issue once we have a couple of implementation examples so we can analyze all the implications more easily?

Dataset design

Creating this issue to discuss the detailed design of Dataset[A] (and potentially DataStream[A]).

As discussed in the meeting with John, I think its worth thinking through what this API would look like as both batch and stream. We can go severals ways with this:

The first would be a unified API like Spark's Dataset (and structured streaming).
https://spark.apache.org/docs/2.3.1/structured-streaming-programming-guide.html#programming-model

Another would be Flink's Dataset/DataStream API (which are build on top of their stateful streams abstraction).
https://ci.apache.org/projects/flink/flink-docs-release-1.5/concepts/programming-model.html

I'd like to flesh out what this API should look like and how it should function in more detail here.

Word Count usage example

This should demonstrate how the industry standard example "word count" would look with our scalaz-analytics API.

For reference:
Spark:
https://github.com/databricks/learning-spark/blob/master/mini-complete-example/src/main/scala/com/oreilly/learningsparkexamples/mini/scala/WordCount.scala
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/HdfsWordCount.scala

Flink:
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/wordcount/WordCount.scala