kanhask / cellstore-1

NoLAP Database Virtualization

CellStore

In traditional data warehousing, it takes months for the IT department to build a single hypercube. With the CellStore, it takes minutes for the business users to build an infinite number of hypercubes.

The CellStore is entirely written in JSONiq and is hosted on 28.io. The deployment can be easily customized to deploy on your own 28.io account.

Demo

A CellStore with all SEC filings since 2010 is available at http://secxbrl.info.

Whitepaper

http://arxiv.org/pdf/1410.0600.pdf

Installation

Prerequisites: NPM, Ruby (needed for SASS).

Windows Users: We advice you to run any commands in the Git Bash. Otherwise, the decryption of config.json won't work and you will have to setup the config.json manually. A working Ruby version for windows can be found at rubyinstaller.org.

$ gem install sass
$ npm install gulp -g
$ npm install && bower install

Configuration

To create a CellStore deployment <name> you need to add a config file config/<name>.json with the following example structure.

Encrypt your config file and add the encrypted config/<name>.json.enc to the repo:

$ export TRAVIS_SECRET_KEY=<secret>
$ gulp encrypt --build-id=test --config=<name>
$ git add config/<name>.json.enc

On github create a branch called test. With the matching config file name you have then created a production branch which automatically deploys to your production s3 bucket and 28.io project as defined in your configuration.

Optionally:

• add reports for your deployment to data/<name>/
• add API tests to queries/public/test/<name>/
• add e2e tests to tests/e2e/<name>

Environment Variables

The TRAVIS_SECRET_KEY environnement variable is mandatory in order to decrypt deployment files.

$ export TRAVIS_SECRET_KEY=<secret> # to decrypt / encrypt config files

You might want to set the following environment variables for convenience:

$ export CELLSTORE_BUILD_ID=<mybuild-id> # default build-id if --build-id=xyz is not provided
$ export CELLSTORE_CONFIG=<name> # default config if --config=xyz is not provided

Deployment

Create a CellStore deployment called test using the encrypted configuration in config/sec.json.enc.

$ gulp 28:setup --build-id=test --config=sec

To resync changes made on hq.28.io into your local branch:

$ gulp download --build-id=test --config=sec

To remove a deployment: Once you are done:

$ gulp teardown --build-id=test --config=sec

Development

To test the development version using the sec.json configuration:

$ gulp server --build-id=mydemo --config=sec

Testing

Run UI tests only:

gulp test --build-id=mydemo --config=sec

Run unit test:

gulp test:unit --build-id=mydemo --config=sec

Configuration Example

{
    "all": {
        "githubToken": "oiifwjro984t834toaihzg9riejloi",

        "s3-region": "us-east-1",
        "s3-bucketPrefix": "csms.example.com",
        "s3-key": "ADUREIGMKODJAEO43SAJ3",
        "s3-secret": "lkhsohqDFhsasdhaAFjaFajajfdWEhDFHAjhud4e4",
        "s3-reportsBucket": "e2eFailureTestReportBucket",

        "28-projectPrefix": "example-project",
        "28-email": "[email protected]",
        "28-password": "<password>",
        "28-datasource-conn": "set-lskjdfaoieurlkajpoidgja/db.example.com:27017,db.example.com:27017",
        "28-datasource-db": "db",
        "28-datasource-user": "dbuser",
        "28-datasource-pass": "<password>",

        "cellstore-profile": "generic",
        "cellstore-filteredAspects": "2",
        "cellstore-allowRegistration": "true",
        "cellstore-adminUser": "[email protected]",
        "cellstore-testUser": "[email protected]",

        "netdna-companyAlias": "mycompanyalias",
        "netdna-consumerKey": "klsjfoaoiqje8ahofdkgjoeqiutjae",
        "netdna-consumerSecret": "lskajforeiwutjgheqoiajejkweasdf",

        "sendmail-host": "smtp.example.com:995/tls/novalidate-cert",
        "sendmail-user": "[email protected]",
        "sendmail-password": "<password>",
        "sendmail-sender-email": "[email protected]",
        "sendmail-sender-name": "CellStore Example Setup"
    },
    "dev": {
        "cellstore-adminPassword": "<password>",
        "cellstore-testPassword": "<password>",
        "cellstore-testToken": "secret",

        "netdna-zone": "none"
    },
    "prod": {
        "cellstore-adminPassword": "<password>",
        "cellstore-testPassword": "<password>",
        "cellstore-testToken": "cdjfalsk-slke-4dkd-73jd-kjsdhfakhjde",

        "netdna-zone": "189474"
    }
}

Git Commit Messages

• Use the present tense ("Add feature" not "Added feature")
• Use the imperative mood ("Move cursor to..." not "Moves cursor to...")
• Limit the first line to 72 characters or less
• Reference issues and pull requests liberally
• Consider starting the commit message with an applicable emoji:
  • 💄 :lipstick: when improving the format/structure of the code
  • 🐎 :racehorse: when improving performance
  • 🚱 :non-potable_water: when plugging memory leaks
  • 📝 :memo: when writing docs
  • 🐧 :penguin: when fixing something on Linux
  • 🍎 :apple: when fixing something on Mac OS
  • 🏁 :checkered_flag: when fixing something on Windows
  • 🐛 :bug: when fixing a bug
  • 🔥 :fire: when removing code or files
  • 💚 :green_heart: when fixing the CI build
  • ✅ :white_check_mark: when adding tests
  • 🔒 :lock: when dealing with security
  • ⬆️ :arrow_up: when upgrading dependencies
  • ⬇️ :arrow_down: when downgrading dependencies