andygrove / how-query-engines-work

This is the companion repository for the book How Query Engines Work.

How Query Engines Work

This is the companion repository for the book How Query Engines Work and contains source code for a simple in-memory query engine implemented in Kotlin.

The query engine is designed to be easy to learn and hack on rather than being optimized for performance, scalability, or robustness.

The query engine contains the following components:

• DataFrame API
• SQL Parser
• SQL Query Planner
• Logical Plan
• Query Optimizer
• Physical Plan
• Server
• JDBC Driver

The following operators are supported:

• Table Scan (Parquet and CSV)
• Projection
• Filter
• Hash Aggregate

The following expressions are supported:

• Literals
• Attributes
• Simple Aggregates (Min, Max, Sum)
• Cast
• Boolean expressions (AND, OR, NOT)
• Simple math expressions (+, -, *, /)

Installing Locally

Install the Google Protocol Buffer compiler

The gradle build script uses the protobuf-gradle-plugin Gradle plugin to generate Java source code from the Ballista protobuf file and this depends on the protobuf compiler being installed.

Use the following instructions to install the protobuf compiler on Ubuntu or similar Linux platforms.

wget https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protobuf-all-3.11.4.tar.gz
tar xzf protobuf-all-3.11.4.tar.gz
cd protobuf-3.11.4/
./configure
make
sudo make install
sudo ldconfig

Build libraries

cd jvm
./gradlew publishToMavenLocal

Sample Data

Some of the examples in the book use the following data set.

wget https://nyc-tlc.s3.amazonaws.com/trip+data/yellow_tripdata_2019-12.csv

how-query-engines-work's Issues

null while reading parquet file

I'm trying to use the engine from Scala.
Pretty simple setup. Using example parquet file from testdata folder.
Code looks like this:

  val ctx = new ExecutionContext(Map.empty[String,String].asJava)
  val pqtSource = new ParquetDataSource("data/alltypes_plain.parquet")

  println(pqtSource.schema().toString)

  ctx.registerDataSource("pdata",pqtSource)
  val df2 = ctx.sql("select id,bool_col from pdata")
  val c2 =  ctx.execute(df2).iterator().asScala.toList.map(r => println(r))

First prinln statement works as expected, gives the structure:

Schema(fields=[Field(name=id, dataType=Int(32, true)), Field(name=bool_col, dataType=Bool), Field(name=tinyint_col, dataType=Int(32, true)), Field(name=smallint_col, dataType=Int(32, true)), Field(name=int_col, dataType=Int(32, true)), Field(name=bigint_col, dataType=Int(64, true)), Field(name=float_col, dataType=FloatingPoint(SINGLE)), Field(name=double_col, dataType=FloatingPoint(DOUBLE)), Field(name=date_string_col, dataType=Binary), Field(name=string_col, dataType=Binary), Field(name=timestamp_col, dataType=Binary)])

Second, only nulls:

Reading 8 rows
null,null
null,null
null,null
null,null
null,null
null,null
null,null
null,null

What am I doing wrong ? Or is might be a Scala incompatibility ?

Typo in the online book example

An example under https://howqueryengineswork.com/05-logical-plan.html#math-expressions has a small typo, it should be name instead of "mult"

override fun toField(input: LogicalPlan): Field {
      return Field("mult", l.toField(input).dataType)
}

Example should be 12 months instead of 2

I think there is a typo ;)

https://github.com/ballista-compute/ballista-jvm/blob/ec55780c9985ddbc3a23d64cce575522e923ac8b/jvm/examples/src/main/kotlin/ParallelQuery.kt#L31

If the sql statement contains (), an error will occur in SqlParser

If the sql where statement contains (), an error will occur in SqlParser , eg : " select a from t where (a=1 or b = 2) and c =3 "

Page 43, `createPhysicalExpr` uses `PhysicalExpr` as return type but updated type name is just `Expression`

how-query-engines-work/jvm/query-planner/src/main/kotlin/QueryPlanner.kt

Line 62 in 9c27d47

fun createPhysicalExpr(expr: LogicalExpr, input: LogicalPlan): Expression =

Small question/clarification about whether something would fit into Logical Plans or Physical Plans

To cement my understanding of the book, I'm working through it a second time, this time changing the implementation from using Arrow vectors + types (columnar) to row-based data (Map<String, Any>) and JDBC types:

data class Field(val name: String, val type: JDBCType)
data class Schema(val fields: List<Field>)

interface DataSource {
    fun schema(): Schema
    fun scan(projection: List<String>): Sequence<Map<String, Any?>>
}

// Physical plans return iterators over rows.
interface PhysicalPlan {
    fun schema(): Schema
    fun children(): List<PhysicalPlan>
    fun execute(): Sequence<Map<String, Any?>>
}

// Physical expression interface
interface Expression {
    fun evaluate(rows: Map<String, Any?>): Any?
}

My question is this:

If I have different "translation" strategies for converting Logical Plans to SQL, which then get executed as a IE a JDBC query -- would these be considered different Physical Plans or would they be implementation details of the same Physical Plan?

IE, translating a Logical Plan to a SQL string with StringBuilder, versus using an ORM/Query-builder like jOOQ: