idexlabs / idex.nucleus.core

IDEX Nucleus Core is the framework to build flexible, scalable and structured back-end architecture

Nucleus

Nucleus is a flexible library that offers tools to implement a distributed micro-service(-like) architecture in NodeJS.

IDEX Nucleus Core is the framework to build flexible, scalable and structured back-end architecture. It includes:

• Structured dataflow;
• Inter-engine communication protocol;
• Content persistence API;
• Content relationship powered by Graph technology;
• Intuitive workflow;

Getting started

NPM

You can quickly and easily install IDEX Nucleus Core to your project using NPM. Make sure to install NodeJS and NPM before;

$ npm install @idex/nucleus.core

Redis

The communication of Nucleus is heavily based on Redis, first and for all make sure to install Redis. Redis installation guide

For Nucleus to work correctly, you need to make sure that your server can use keyspace notification.
You can test that keyspace notification is enabled using the redis-cli:

$ redis-cli
127.0.0.1:6379> CONFIG GET notify-keyspace-events
1) "notify-keyspace-events"
2) "AKE"

If it isn't enabled you can enable it manually using the CONFIG SET command:

$ redis-cli
127.0.0.1:6379> CONFIG SET notify-keyspace-events AKE
OK

or, you can copy the redis.conf file from Nucleus root directory into your project.

$ redis-server PATH_TO_PROJECT/redis.conf

Nucleus Engine

The Nucleus engine (engine for short) is used to interact with the communication layer. It is task to publish/handle actions and events.

Create a new Engine

You can create an engine by simply instantiating the NucleusEngine class:

const { NucleusEngine } = require('@idex/nucleus.core');

const $engine = new NucleusEngine('Test');

// Here, `$engine` is a proxy: it is both a valid promise and the instantiated Nucleus engine.

$engine
      // The engine is ready to be used.
    .then(() => {
      // Do something with the engine.
    })
    // Something happened during the initialization of the engine.
    .catch(console.error);

Checkout the tutorials for your first steps:

1. Create a basic engine
2. Create a basic API with public Gateway
3. Create a persistent storage API

License

MIT License

Copyright (c) 2018 Sebastien Filion

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Contributing

1. Fork it (https://github.com/idexlabs/idex.nucleus.core/fork)
2. Create your feature branch (git checkout -b feature/foo-bar)
3. Commit your changes (git commit -am 'Add some fooBar')
4. Push to the branch (git push origin feature/food-bar)
5. Create a new Pull Request

Please use the following namespace convention to name you branch:

• feature/[:issue-number/]:feature-name Marks a branch that introduce a new feature. Add the issue number if it is available. ie: feature/#24/emoji-action-name;
• bugfix/[:issue-number/]:bug-name Marks a branch that fixes a bug. Add the issue number if it is available. ie: bugfix/#13/engine-cleanup;
• change/[:issue-number/]:change-name Marks a branch that makes a change to the codebase or documentation. Add the issue number if it is available. ie: change/update-release-note;

idex.nucleus.core's Issues

The retrieve all by type feature is too slow.

The current benchmark is ~200 resource retrieved in 3 seconds.

Allow an engine to send warning events to a specific engine.

Publishing an action should accept a correlation ID.

https://blog.rapid7.com/2016/12/23/the-value-of-correlation-ids/

Implement method hooks

The feature would allow a user to specify code to be executed before and after a specific method.

The engine doesn't throw an error if it has no name

Trying to instantiate an engine without a name doesn't throw an error, but part of the initialization code expects the value to be set.

const $engine = new NucleusEngine();

This throws a Nucleus Datastore error NucleusError: The item must be a string.

The name should be mandatory and throw a clear error when omitted.

Deep hash structure are not expanded properly

When retrieving a deep hash structure, the datastore tries to spread the data using dot notation to allow for the data to be searchable. A structure containing an array should use "bracket notation" and be expanded to an array as expected.

const object = {
  key: [
    'value1',
    'value2'
  ];

const expandedObject = NucleusDatastore.expandDotNotationObject(object);

chai.expect(expandedObject.key[0]).to.equal('value1');

Registered anchor node (indexed node) can't be unregistered.

When creating a relationship with a resource using the is-member-of predicate, the system registers the object as anchor node to speed up queries; this relationship is never unregistered, ie: the anchor node's resource is removed.

Log action execution elapsed time

Implement action's event propagation

Engine should accept a resource datastore and a cache datastore

Implement an action middleware

Implement action's event propagation for distributed system

Part of the mechanism is in place, but not leveraged.
When an event is published, its key is pushed to a sorted list that should be used to make sure only one engine instance handles a given event.

Can’t collapse to dot notation an empty array

The Nucleus Datastore API has a method to collapse an object to dot notation, currently it returns undefined when passed an object with an empty array.

Refactor direct usage of the datastore's direct connection to Redis in the Resource API

Refactor direct usage of the datastore's direct connection to Redis in the Resource API;

Abstract the Datastore API for common persistence task

The Datastore API has to be more generic so it can be used interchangeably with the Resource API.

Implement resource versioning as part of the Resource API

When updating a resource using the Resource API, the changes made to the object and persist it along with resource. As part of this feature, an action to rewind the resource to a specific version should also be implemented.

Share Redis connection for non-hanging Nucleus Datastore

It would be interesting to attempt to reduce the number of Redis connection in a given controlled ecosystem. If one or many engines are instantiated on the same node process, there is no point of having multiple Redis connection since they are both single threaded.
Hanging connection like those sued to subscribe to Redis commands or events should not share that connection for obvious reason.

class NucleusDatastore {
  constructor (datastoreName = 'Untitled', options = {}) {
    const {
      $logger = console,
      $$server,
      index: datastoreIndex = 0,
      port: datastorePort = 6379,
      URL: datastoreURL = 'localhost'
    } = options;

    this.name = datastoreName;
    this.index = datastoreIndex;

    this.$$handlerCallbackListByChannelName = {};
    this.scriptSHAbyScriptName = {};

    this.$$server = $$server || redis.createClient({
      db: datastoreIndex,
      host: datastoreURL,
      port: datastorePort
    });
  // ...
  }
}

A user that is a member of the SYSTEM node can't retrieved all resources by type as expected.

Implement a resource archive API

There are action events stored in the event datastore without a TTL.

Input validation adapter

Like the datastore, an input validator should be interchangeable; right now, the default is the Nucleus validator.

Errors rejected from a publish action request does not contain the meta

When using the #publishActionByNameAndHandleResponse method to publish an action and the action rejects with a Nucleus error containing some meta data, the error that is caught does not contain the meta data.

/**
 * @Nucleus ActionName ThrowErrorWithMetaData
 */
function () {

  throw new NucleusError(`This is a Nucleus error with meta data`, { testID: uuid.v4() }); 
}

...

$engine. publishActionByNameAndHandleResponse('ThrowErrorWithMetaData', {}, uuid.v4())
  .catch((error) => {
    error.meta.testID === undefined
  });

The Nucleus error is correctly store in the action, so the issue probably comes from the NucleusError constructor. library/Error.nucleus.js

The Nucleus error is rebuilt around library/Engine.nucleus.js:740.

Might be related to: #14

Allow file streaming

An action moved to another engine is not removed from the original engine

When using autodiscovery, an engine's action's configuration are stored locally.
If an action was to be moved to another engine, the action's configuration would be stored under the second engine and the first.
The first engine should be able to clean after itself.

The Resource API methods doesnt accept the extended signatures.

Failed action publication error doesn’t log the action ID

The engine datastore and the resource datastore can't be separated

An engine requires a datastore connection which it uses to store engine specific data and passes to the Resource API; if a user chooses to pass a datastore adapter (not Redis), the engine won't be able to perform some low level tasks. The engine should accept, optionally a $resourceDatastore and a $cacheDatastore... Also, the engine should verify that the engine datastore is a Redis adapter.

Nucleus Datastore used as queue handler (blocking connections) are not destroyed properly

When a Nucleus Datastore uses the blocking command BRPOP (BLPOP), the client cannot be destroyed properly.