shelnutt2 / mytile

Mariadb Storage Engine based on TileDB

MyTile

MariaDB storage engine based on TileDB

Requirements

Requires MariaDB 10.2 or newer. It is untested on older versions.

Installation

Inside MariaDB Source Tree (Recommended)

The first is inside a MariaDB source tree (recommended).

git clone [email protected]:MariaDB/server.git -b 10.2
cd server
git submodule add https://github.com/Shelnutt2/MyTile.git storage/mytile
mkdir build && cd build
cmake ..
make -j4

Features

• Based on TileDB's Key-Value store
• No write locking due to TileDB's eventual consistency model

Known Issues

• Geometry is not supported #6

Planned Features

• Transaction support #8
• Secondary Index Support #9
• Spatial Indexing #7
• Consolidation/optimize table for compaction of kv store #10
• Add support for specify tiledb parameter as table options #11

mytile's Issues

Switch TileDB to use ExternalProject

Switch TileDB to use ExternalProject_Add for building if missing from system.

Add support for updates

Add support for updates. In tiledb we just need to call write_row, delete isn't necessary as each write overrides the last. Tiledb always tracks the latest "fragment".

Add support for creating tables

Add support for create_table

Switch from using KV Store to Array

It was mentioned that the Key-Value store is not as optimal as the underlying array. The KV store is newer and less optimized and tested. It would be idea to switch to using the array instead.

A benefit is removing the requirement for primary key support. A downside is now key must be managed independently. This was already the case for when secondary index support (#9 ) is added.

Add secondary index support

Add secondary index support. Right now only primary indexes are support. The foundation is in place for secondary indexes, we just need to produce maps that contain the secondary key -> primary key mapping.

Implement optimize table for consolidation of tiledb map

Implement optimize table for consolidation of tiledb map.

int tile::mytile::optimize_table() {
  tiledb::Map::consolidate(ctx, name);
}

Tiledb parameters should be able to be set as table options

Tiledb parameters should be able to be set as table options. Parameters including, default compression methods, compression level, flushing setting, etc.

Add support for Geometry Columns

Add support for Geometry Columns and datatypes. These will be treated exactly like blobs.

Add support for spatial indexing of geometry columns

Add support for spatial indexing of geometry columns.

Add support for Read/Write/Delete/Drop

Add support for Read/Write/Delete/Drop

Implement Table Discovery based on schema discovery

Implement Table Discovery based on loading TileDB schema

Add support for null values

Add support for null values by having a vector (or map) attribute indicating of the field is null or not.

Add transaction support

Add transaction support. Items should be built and stored until commit, then added to the map. We also should only flush the map on commit.

Add auto increment support

Add auto increment support.

This partially overlaps with #9 in that index_last() is needed.

Use tiledb VFS api for filesystem manipulations

Use tiledb VFS api for filesystem manipulations

Use table->read_set/write_set for smarter columnar access

Mysql sets the bitmap for columns that are requested for a query. When processing a scan or write the table->read_set/write_set bitmap is set with columns that to are be read/written to. We should take advantage of this during read_init to only create buffers for columns to be accessed. This will reduce overhead and speed up performance since tiledb is columnar based.

Add Field Options for Compression settings

Add Field Options for Compression settings. The field_option of a handler can be used to offer specific field options per column. This would be perfect for setting the compression type and compression level.