GithubHelp home page GithubHelp logo

blobcity / db Goto Github PK

View Code? Open in Web Editor NEW
173.0 10.0 21.0 48.12 MB

A blazing fast ACID compliant NoSQL DataLake with support for storing 17 formats of data. Full SQL and DML capabilities along with Java stored procedures for advanced data processing.

Home Page: https://www.blobcity.com

License: GNU Affero General Public License v3.0

Dockerfile 0.04% Java 92.11% Shell 0.10% C 3.23% C++ 4.32% Makefile 0.21%
blobcity-db database dbaas nosql multi-model

db's Introduction

License: AGPL v3 PRs Welcome GitHub issues Generic badge Ask Me Anything !

Description

BlobCity DB is an All-in-One Database. It offers support for natively storing 17 different formats of data, including JSON, XML, CSV, PDF, Word, Excel, Log, GIS, Image amongst others. It run two full feature storage engines. One that stores data in memory and the other that stores data on disk. In-memory storage offers sheer performance for real-time analytics, while the disk storage make BlobCity an excellent alternative for DataLakes.

Supported Data Formats

Push data in any of these 17 formats: JSON, XML, CSV, SQL, Plaintext, PDF, Excel, Word, RTF, ZIP, Log, Powerpoint, syslog, audio files, video files, image files, GIS

Multi-Model Example

JSON Record

{"col1": 1, "col2": 2}

XML Record

<col1>3</col1></col2>4</col2>

Auto created table schema and data

col1 col2
1 2
3 4

Push variety of data into a single collection within BlobCity, and get back a standardised response.

user$ nc localhost 10113
username>root
password>9a371c6445
You are now inside the BlobCity DB console
Type 'help' for assistance and 'exit' to quit
blobcity>create-ds test
Datastore successfully created
blobcity>create-collection test.test
Collection successfully created

blobcity>insert into test.test JSON
In insert mode. Type 1 JSON per line and press enter to insert
{"col1": 1, "col2": 2}
Inserted
exit
Exited insert mode

blobcity>insert into test.test XML
In insert mode. Type 1 XML per line and press enter to insert
<col1>3</col1><col2>4</col2>
Inserted
exit
Exited insert mode

blobcity>sql test: select * from test.test
{"p":[{"_id":"5cb30531-dde1-493c-9c67-86b5f4dce36c","col2":2,"col1":1},{"_id":"57f653e3-de68-4591-9563-af9ad66af56b","col2":4,"col1":3}],"time(ms)":2,"ack":"1","rows":2}

blobcity>sql test: select col1 from test.test
{"p":[{"col1":"1"},{"col1":"3"}],"time(ms)":18,"ack":"1","rows":2}

blobcity>set-column-type test.test col1 integer 
Column type successfully updated in schema

blobcity>sql test: select SUM(col1) from test.test
{"p":[{"SUM(col1)":4}],"time(ms)":27,"ack":"1","rows":1}

The above example shows inserting both JSON and XML recoreds into the same collection. The DB seamlessly creates columns and merges the columns to allow querying of both records using SQL.

Features

  • Full SQL: Run SQL queries over REST, ODBC & JDBC connectivity
  • DataLake: On-disk storage engine optimised for DataLake scale with low latency query response
  • DML Support: Designed like a DataLake, but works like a database. Full support for UPDATE & DELETE queries
  • Realtime: HIgh speed in-memory storage optimised for real-time analytics
  • 17 Data Formats: Stores 17 formats of data such as JSON, XML, PDF, Excel, Word amongst others for collective analytics
  • ACID: Full ACID compliant transactions on individual records
  • Stored Procedures: Run Java & Scala code within the database for complex operations on data without moving the data out of the database
  • Fine-grained Access Control: Control data access across users and departments, with column level control on user access
  • On-Cloud: Fully managed virtually infinte scale, multi-tenant cloud with unlimited free storae and pay only for what you analyse

Get Started

        

On BlobCity Cloud

Store unlimited data for free and pay only for what you analyse. Delivers ultra high speed analytics over multi-tenant infrastructure, starting at $10/month.

Start Now

Using Docker

docker run -i -p 10111:10111 -p 10113:10113 blobcity/db

Once container is started, open a telnet connection on port 10113 to connect to over network CLI.

nc localhost 10113
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
username>root
password>xxxxx
You are now inside the BlobCity DB console
Type 'help' for assistance and 'exit' to quit
blobcity>

A random auto-generated password is placed at /mnt/data/root-pass.txt. This file can be found within the container. It can be fetched from within the container, or by mounting this folder to an external mount point.

docker run -i -v /my-folder:/mnt/data -p 10111:10111 -p 10113:10113 blobcity/db

The password file can now be found at /my-folder/root-pass.txt on your computer.

Using Binary Distribution

Supported only on Unix & MacOS distributions

Download latest tar.gz archive from the releases.

Decompress the download file, then run blobcity.sh from inside the bin folder.

user>tar -xvf blobcity-db-x.x.x.tar.gz
user>cd blobcity-db-x.x.x/bin
user>sh ./blobcity.sh

JAVA_HOME must be set to a JDK / JRE version 8 or higher for the DB to be booted.

The database will create a folder called data at blobcity-db-x.x.x/data. The randomly generated root user password can be found inside a text file at blobcity-db-x.x.x/data/root-pass.txt.

Use this password to connect to the CLI console to start using the DB. It is recommended that the data folder be stored at a difference location than the boot volume, and the volume be XFS formatted.

The location of the data folder can be set by editing the blobcity.sh file and uncommenting the following line and setting a folder path of your choice.

#export BLOBCITY_DATA=

Look at some of the best practices for optimal disk storage performance.

Acceleration

BlobCity is a winner of Economic Times Power of Ideas (Season 2), is funded by CIIE IIM-Ahmedabad and is a graduate from NetApp Excellerator (Cohort #2).

Docs

https://docs.db.blobcity.com

Contribute

Join our Slack community and request to become a contributor. We encourage your contributions :)

Authors

BlobCity DB was created by Sanket Sarang along with notable contributions from Akshay Dewan and Karun Japhet, amongst others. BlobCity DB is sponsored by BlobCity, Inc..

License

GNU Affero General Public License v3.0

Kraken

                       ___
                    .-'   `'.
                   /         \
                   |         ;
                   |         |           ___.--,
          _.._     |0) ~ (0) |    _.---'`__.-( (_.
   __.--'`_.. '.__.\    '--. \_.-' ,.--'`     `""`
  ( ,.--'`   ',__ /./;   ;, '.__.'`    __
  _`) )  .---.__.' / |   |\   \__..--""  """--.,_
 `---' .'.''-._.-'`_./  /\ '.  \ _.-~~~````~~~-._`-.__.'
       | |  .'R_.-' |  |  \K \  '.               `~---`
        \K\/ .'     \  \   '. '-._)
         \/ /        \  \    `=.__`~-.
         / /\         `) )    /E/ `"".`\
   , _.-'.'\ \        /A/    ( (     /N/
    `--~`   ) )    .-'.'      '.'.  | (
           (/`    ( (`          ) )  '-;
            `      '-;         (-'

Kraken was our internal project code name until open sourcing. You may still find some mentions of it in the code docs.

db's People

Contributors

b0n541 avatar dependabot-preview[bot] avatar dependabot-support avatar karishma-k avatar sanketsarang avatar viraptor avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

db's Issues

Last load-code should be memorised across DB reboots

Custom stored procedures can be loaded today using the load-code command. The load stored procedures remain loaded and can be used for as long as the database cluster is running. However on full cluster reboot, the system does not automatically load the stored procedures that were previously loaded. This needs to be changed to memorise previously loaded stored procedures to have them auto load on cluster reboot.

Delete dropping data but not index entry

The delete query properly deletes the data, but retains the index entry for the deleted record. This causes select distinct to give inconsistent results.

delete from ds.collection where code = '123'

Now run a select distinct on the table for code

select distinct code from ds.collection

The code 123 is present in the response.

However the data is actually not present, so the delete has properly executed

select * from ds.collection where code = '123`

The above statement will return 0 rows, thereby making the select distinct operation return inconsistent results.

Add support for MD5 indexes

MD5 indexes will work exactly as BTree indexes. They are better suited than BTree indexes for String columns having large and complex data. The are used for exact searches only and cannot perform approximate matches such as a LIKE query. The index will store the MD5 value instead of the raw String value.

Implement on lines of the BTree index https://github.com/blobcity/db/blob/master/engine/src/main/java/com/blobcity/db/indexing/OnDiskBTreeIndex.java

BTree index stores column value to disk. The MD5 index must instead store the MD5 value to disk, but in exactly the same manner as the BTree.

Null pointer exception in request handling bean

The RequestHandlingBean in unknown scenarios quits with a null pointer exception. Impact of this on the running query is unknown. The error is reproducible when DB is in high load for inserts.

blobcity-db_1  | java.lang.NullPointerException
blobcity-db_1  | 	at com.blobcity.db.requests.RequestHandlingBean.processRequest(RequestHandlingBean.java:285)
blobcity-db_1  | 	at com.blobcity.db.requests.RequestHandlingBean.newRequest(RequestHandlingBean.java:72)
blobcity-db_1  | 	at com.blobcity.db.bquery.BQueryExecutorBean.insert(BQueryExecutorBean.java:539)
blobcity-db_1  | 	at com.blobcity.db.bquery.BQueryExecutorBean.runQuery(BQueryExecutorBean.java:223)
blobcity-db_1  | 	at com.blobcity.lib.database.web.end.point.db.BQueryResource.getJson(BQueryResource.java:88)
blobcity-db_1  | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
blobcity-db_1  | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
blobcity-db_1  | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
blobcity-db_1  | 	at java.lang.reflect.Method.invoke(Method.java:498)
blobcity-db_1  | 	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
blobcity-db_1  | 	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
blobcity-db_1  | 	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
blobcity-db_1  | 	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
blobcity-db_1  | 	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
blobcity-db_1  | 	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
blobcity-db_1  | 	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
blobcity-db_1  | 	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
blobcity-db_1  | 	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
blobcity-db_1  | 	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
blobcity-db_1  | 	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
blobcity-db_1  | 	at com.sun.jersey.server.impl.container.grizzly2.GrizzlyContainer._service(GrizzlyContainer.java:222)
blobcity-db_1  | 	at com.sun.jersey.server.impl.container.grizzly2.GrizzlyContainer.service(GrizzlyContainer.java:192)
blobcity-db_1  | 	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)
blobcity-db_1  | 	at org.glassfish.grizzly.http.server.HttpHandlerChain.service(HttpHandlerChain.java:196)
blobcity-db_1  | 	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)
blobcity-db_1  | 	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:175)
blobcity-db_1  | 	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
blobcity-db_1  | 	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
blobcity-db_1  | 	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
blobcity-db_1  | 	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
blobcity-db_1  | 	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
blobcity-db_1  | 	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
blobcity-db_1  | 	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:815)
blobcity-db_1  | 	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
blobcity-db_1  | 	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
blobcity-db_1  | 	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
blobcity-db_1  | 	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
blobcity-db_1  | 	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)
blobcity-db_1  | 	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)
blobcity-db_1  | 	at java.lang.Thread.run(Thread.java:745)

Inconsistent results when limiting select to specific columns

In version 1.7.10, it is observed that when limiting a select to specific columns, the database occasionally misses a few records in the response.

The following are logs for the same query fired twice on the same table. The query was expected to respond with 2 records on both occasions.

blobcity>sql test: select col1, col2 from test.test
{"p":[{"col2":2,"col1":1}],"time(ms)":24,"ack":"1","rows":1}
blobcity>sql test: select col1, col2 from test.test
{"p":[{"col2":4,"col1":2},{"col2":2,"col1":1}],"time(ms)":3,"ack":"1","rows":2}

However, the following queries deliver consistent results
select col1 from test.test
select * from test.test

null pointer exception on SQL update on a record with an indexed sparce column

If a record has a sparce column, and the specific column is indexed but has a value of null in the current record, as that the column is missing in the record. Then running an update on any other column of this record, results in the following null pointer exception.

java.lang.NullPointerException
	at com.blobcity.db.indexing.OnDiskBTreeIndex.remove(OnDiskBTreeIndex.java:313)
	at com.blobcity.db.bsql.BSqlDataManager.save(BSqlDataManager.java:1072)
	at com.blobcity.db.bsql.BSqlDataManager.updateAsync(BSqlDataManager.java:1097)
	at com.blobcity.db.sql.statements.UpdateExecutor.lambda$bulkUpdate$0(UpdateExecutor.java:200)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
	at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
	at com.blobcity.db.sql.statements.UpdateExecutor.bulkUpdate(UpdateExecutor.java:199)
	at com.blobcity.db.sql.statements.UpdateExecutor.udpate(UpdateExecutor.java:107)
	at com.blobcity.db.sql.statements.UpdateExecutor.execute(UpdateExecutor.java:74)
	at com.blobcity.db.sql.statements.UpdateExecutor.execute(UpdateExecutor.java:69)
	at com.blobcity.db.sql.statements.UpdateExecutor$$FastClassBySpringCGLIB$$a4b8340f.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
	at com.blobcity.db.sql.statements.UpdateExecutor$$EnhancerBySpringCGLIB$$6375d33e.execute(<generated>)
	at com.blobcity.db.bquery.SQLExecutorBean.runQuery(SQLExecutorBean.java:129)
	at com.blobcity.db.bquery.SQLExecutorBean.runQuery(SQLExecutorBean.java:90)
	at com.blobcity.lib.database.web.end.point.db.SqlResource.getPostResponse(SqlResource.java:101)
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
	at com.sun.jersey.server.impl.container.grizzly2.GrizzlyContainer._service(GrizzlyContainer.java:222)
	at com.sun.jersey.server.impl.container.grizzly2.GrizzlyContainer.service(GrizzlyContainer.java:192)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)
	at org.glassfish.grizzly.http.server.HttpHandlerChain.service(HttpHandlerChain.java:196)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:175)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:815)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)
	at java.lang.Thread.run(Thread.java:748)

on-disk btree index column value update problem on empty mapping

The On-Disk Btree index does not delete the column value folder on an update operation, when post operation there are no more records present with the specific column value. This results in incorrect responses for SELECT DISTINCT query, which still assumes that a record with the previous value is present in the DB.

Improve stored procedures performance

Today stored procedures communicate with the database using a TCP connection over the LAN. This connection is typically over localhost and is hence fast, but some time in lost in serialising and deserialising data over the TCP protocol. To get utmost performance, since the stored procedures are loaded as Java code, they should be able to invoke DB functions (like fire a SQL query) using Java reflection. This bypasses any TCP connection requirements and would be the utmost use of available hardware.

Bounty: $300 (Read contributors agreement)

Indexing after data load results in a blank index

Load data into a collection and then index a column. The collection schema shows the column to be indexed, but the index is actually blank thereby operations on the index always returning empty results.

At this stage, it is not know if the problem is file system or OS specific. Only few deployments are reporting this issue.

Sustained inserts cause almost permanent high CPU for unknown reasons

In some environments, as of yet not sure which, performing sustained inserts causes the product to slow down upto the point of it accepting no more inserts. The cause for slow down is a 100% continuous CPU usage for unknown reasons. The problem occurs only in specific environments and it is not yet clear on how to reproduce it, but it is consistently reproducible on systems that present this issue.

Bounty: $750 (Until 30 Nov 2018. Read & Sign Contributors Agreement)

Progress tracking for watch service

Currently watch service progress cannot be monitored. This is a problem when loading large files, as monitoring of upload status of such files is not available. This features aims to report at real-time and store the upload progress of each files.

Intend is to allow admins to confirm upload of certain files, view currently processing files and estimate the time required to upload each file.

Remove license check module for unrestricted access to all features

Enterprise edition of BlobCity required an enterprise license to be purchased. Some features of BlobCity were enabled only if a valid license file was found. Post open sourcing one can get a free enterprise license from BlobCity to unlock all features, but the license check itself must be removed so that all features are readily available to use.

Log level settings should be remembered across database reboots

Currently the database logging level can be changed at runtime. The default level is DEBUG. However the changed setting is not memorised, with the logging level getting set to DEBUG on database reboot. The log settings should be saved in the local datastore and must be auto applied on system reboot.

Provision to optionally disable SelectActivityLog

The SelectActivityLog is used to compute the number of selects performed on a datastore. This information is used in BlobCity hosted cloud deployments for billing purposes. The feature requires an insert operation to be executed, per select operation. This could be an undesirable performance drop for an on-premise user.

New architecture to have this feature disabled by default, with a user having option to enable it through CLI.

Update query to set an inexistent field misses updating index entry for new field

Index not getting updated on running an UPDATE SQL query to set a field that was not originally present on the record being updated.

Existing record
1: {'col1': 1, 'col2': 1}
2: {'col1': 2, 'col3': 2}

Run SQL Update
UPDATE test.test SET col3 = 1 WHERE col1 = 1

Updated Records
1: {'col1': 1, 'col2': 1, 'col3': 1}
2: {'col1': 2, 'col3': 2}

However the index entry for col3 for record #1 does not update. The entire column has to be reindexed after this operation.

CLI unable to operate on .systemdb datastore

CLI queries such as truncate-collection, list-collections that rely on datastore and collection names being provided with a . separator does not work due to the presence . in .systemdb.

Possible resolution includes handling special case if ds.collection starts with .systemdb.

A command view-collection .system.QPS should be processed differently than view-collection MyDs.QPS

ORDER BY not functional when limiting select to 2 or more columns

In version 1.7.10, it is discovered that order by operation has no impact when specifically selecting more than 1 column from a table.

select col1, col2 from ds.collection order by col1

The result of this operation does not provide the data in an ordered fashion. The order by clause in the query seems to be totally ignored by the query processing engine.

However, the following queries work fine
select col1 from ds.collection order by col1
select * from ds.collection order by col1

SELECT DISTINCT on multiple columns gives inconsistent results

Re-firing a SELECT DISTINCT query with multiple columns in the distinct clause gives different rows in response per query for the same data in the table. Possible cause is performance on a non-thread safe operation in executing the DISTINCT clause.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.