spacecloud-io / space-cloud Goto Github PK

@KhushalPShah try to finish this off asap. Will need this to be resolved to merge #85.

Use the context object in every SQL db operation similar to mongo

Currently space cloud doesn't provide any logging. We need a proper logging mechanism with configurable levels.

We need to add support for doing joins. In SQL databases it would be a native join while in Mongo it would be an application side join.

The concern I have here is, how will the where (find) clause be applied on the joints table.

Need to fix those links in the docs template we are using.

Error thrown :-
/d:\codeformer\space-cloud\space-cloud.yaml: The filename, directory name, or volume label syntax is incorrect.

The API for CRUD needs to be updated to make it short and more intuitive.

Here's the current API for javascript:

// Insert one / all:
db.insert(col).one(doc).then(res => ...).catch(ex => ...);
db.insert(col).all(docs).then(res => ...).catch(ex => ...);

// Get one / all:
db.get(col).where(whereClause).one().then(res => ...).catch(ex => ...);
db.get(col).where(whereClause).all().then(res => ...).catch(ex => ...);

// Update one / all:
db.update(col).where(whereClause).set(obj).one().then(res => ...).catch(ex => ...);
db.update(col).where(whereClause).set(obj).all().then(res => ...).catch(ex => ...);

// Delete one / all:
db.delete(col).where(whereClause).one().then(res => ...).catch(ex => ...);
db.delete(col).where(whereClause).all().then(res => ...).catch(ex => ...);

In the new API there should not be explicit one() and all() method. The API should take the default operation type to be all since one is used very rarely and also one is not available on SQL. However, the user should be able to change the op by calling an optional op method. The method apply will actually trigger the request.

This is the new proposal:

// Insert all:
db.insert(col).docs(docs).apply().then(res => ...).catch(ex => ...);
// Insert one:
db.insert(col).doc(doc).apply().then(res => ...).catch(ex => ...);

// Get all:
db.get(col).where(whereClause).apply().then(res => ...).catch(ex => ...);
// Get one:
db.getOne(col).where(whereClause).apply().then(res => ...).catch(ex => ...);
// Get count:
db.count(col).where(whereClause).apply().then(res => ...).catch(ex => ...);
// Get distinct:
db.distinct(col).where(whereClause).apply().then(res => ...).catch(ex => ...);

// Update all:
db.update(col).where(whereClause).set(obj).apply().then(res => ...).catch(ex => ...);
// Update one:
db.updateOne(col).where(whereClause).set(obj).apply().then(res => ...).catch(ex => ...);
// Upsert:
db.upsert(col).where(whereClause).set(obj).apply().then(res => ...).catch(ex => ...);

// Delete all:
db.delete(col).where(whereClause).apply().then(res => ...).catch(ex => ...);
// Delete one:
db.deleteOne(col).where(whereClause).apply().then(res => ...).catch(ex => ...);

Should there be an options struct {upsertFolder: true} or should it be the default behavior?

SSL absent, no provision to transfer data in a secure manner

The main reason Space Cloud was born was to simplify the app / web development process. Right from making simple CURD CRUD operations to syncing data reliably in a distributed environment, everything must be as simple as a function call.

We need to add kafka for the distributed realtime feature. Also we need not rely just on kafka, we can use any message broker with an at least once semantics for this task. Something like the pluggable backend for the file store module.

I noticed some styling issues with the docs specifically for the code tags. Can we make the long code lines wrap automatically?

Also, it would be great if the width of the docs would be larger on the browser.

I've heard quite a few people having troubles with downloading the config file step. Apparently copying and pasting the config directly in a file doesn't work.

@KhushalPShah faced the same issue I guess.

Problem faced right now

There's no way in SC to let users sync data easily without touching the Database module

How can we solve it

We can have a pub sub module built into Space Cloud which lets users to publish and subscribe data.

We need to make sure the security rules are enforced for each object published. Also the consistency level needs to be tunable.

For example, if a client looses the connection due to connectivity issues at the time a data point was published, should he be delivered the message or not?

Loads of things to consider to do this right!

Running npm install -g space-cli installs an incorrect npm package. Let's completely remove the concept of space-cli.

Referring to issue on fixing docs #28

The GRPC endpoints are not Realtime yet.

We should have a mechanism to publish internal events for functions to subscribe on. We can publish events like

• CRUD operation hooks
• File Upload

The docs do not have the motivation section present in the readme. Need to make sure they both are in sync.

It might be too early right not. But should we make a subreddit for Space Cloud?

We should definitely create some use cases. Need ideas on what we can do!

Here are some to get started

• Realtime chat app with "last seen" and "is typing" feature
• An IoT Fleet management system
• a markdown editor where multiple people can collaborate in real time (google docs for markdown
• an e-commerce portal
• a social media platform

Even the most craziest of ideas are welcome!

The main reason Space Cloud was born was to simplify the app / web development process. Right from making simple CURD CRUD operations to syncing data reliably in a distributed environment, everything must be as simple as a function call.

space-cloud doesn't removes the prefix - "Bearer " from the Authorization Header

actual plaintext passwords being sent in user management, really unsafe thing to do. Standard way is to salt the passwords and the compare the hashes, not plain text passwords

I thing the config file provided in the example here should cover all the possible configuration. Maybe we can put the defaults of each module (enabled = false)?

Also it will be of great help if the config file has helpful comments. The config file should be self documenting.

A new transaction endpoint is in the works for http. We need to do the same for grpc.

Can we add support for search in the database we already support.

It would be great if the space-cloud init command could setup the examples and start them as well.

The problem faced right now

There's no way to enforce a schema for schemaless databases(like MongoDB) in SC.

How can it be solved

Allow SC to accept a schema via Mission control for various collections. This schema should have the following field level config optons:

• Data type (int, string, bool etc)
• Default value
• Required
• Optional encryption / hash / bcrypt operation

We need to make a language specific getting started guide as well. It should show the baisc commands which can be used along with the steps to import and initialise the library!

Something similar to what we doing with gRPC

Using NATS on Gcloud and Amazon's internal Service on AWS

The transaction support we have added works more like a batch. Let's make that more explicit. This will require us refactoring:

• The https endpoint (url)
• The gRPC rpc name.
• The CRUD interface in crud.go and operations.go
• the func name in sql and mongo packages

Also, none of the function endpoints support streaming responses yet.

No where in the docs have we mentioned whats the output to expect when space-cloud has started successfully. We should add something in the docs

We are already sending out the where clause as a MongoDB query to the backend. Why not let the user put the raw query directly?

I know this is a request for the client libraries. But lets put it here to keep track. The updates of it being done on individual clients can be done here.

This will be similar to what we have over websockets currently.

It would be really useful if space-cloud allowed the user to host static files as well.
Adding this feature will definitely help space-cloud lift up above the others.

The http endpoint is ready. Lets add support to the js client as well

Currently the user will have to deal with arrays in the security rules for insert many operations. Lets break it down to the document level.

We need to provide a unified transaction api for all the databases that support transactions.

Do we even wanna do this? @Jayesh333 and @KhushalPShah we should select which function's we want in GRPC.

The prefix matching never works properly. @Jayesh26, can you have a look at it and fix.

This is potentially a security bug for MongoDB clusters having multiple databases.

Need to solve this asap!

Currently, server.proto defines BatchRequest in the following way:

message AllRequest {
  string col = 1;
  bytes document = 2;
  string operation = 3;
  bytes find = 4;
  bytes update = 5;
  string type = 6;
}
message BatchRequest {
  repeated AllRequest batchrequest = 1;
  Meta meta = 2;
}

Instead of AllRequest we can directly use the already defined CreateRequest, ReadRequest, etc

message BatchRequest {
  repeated CreateRequest cr = 1;
  repeated ReadRequest rr = 2;
  repeated DeleteRequest dr = 3;
  repeated UpdateRequest ur = 4;
  repeated AggregateRequest ar = 5;
  Meta meta = 6;
}

To be considered:

1. This new method will remove the AllRequest message.
2. The predefined messages will be used. This will help to segregate and identify the specific components in the batch request. This may also make it easier to implement in the client modules.
3. The meta field may be repeated, but we can always omit it in the component CreateRequest, ReadRequest, etc.

Just like Faas (using GRPC) has been implemented inside the FaaS module, with only a function call from the grpc server, we should have the same thing done for CRUD implementation using grpc as well.

The docs are not good enough. People new to the project won't be able to understand its aim and where it fits. Also the quick start seems to be a bit scattered and doesn't paint a clear picture regarding what it does.