dreamscale-io / gridtime

Idea Flow Network Operating System Kernel

GridTime Server

Generates zoombie tile feeds, and aggregate tile feeds from flow source data streams

IDE Setup

Application stack is Gradle and Springboot. To setup the project in Intellij, first run:

./gradlew idea

Then open the generated Intellij project file in your IDE.

Setup Docker

You'll need to have Docker running to boot app the app, which spins up a postgres instance on docker. You can download here: https://docs.docker.com/docker-for-mac/install/

Type docker ps to confirm docker is running and available in the path.

Running the App

To bootup the application locally, you need to first have docker machine running. You can launch the server with:

./gradlew bootRun

Postgres Refresh

To completely refresh your postgres Database instances, you can run:

./gradlew refreshP (shorthand for target refreshPostgres)

./gradlew createTDB (shorthand for target createTestDB)

Swagger

For the Swagger UI, navigate to /swagger.html on the running server (e.g. for localhost, http://localhost:8080/swagger.html) To retrieve the swagger json, navigate to /v2/api-docs

Heroku

This application is currently deployed on Heroku at http://ds-gridtime.herokuapp.com. For an API-Key on the shared service, please contact [email protected]. We will have account creation setup soon, but while this project is still in early development, accounts are created manually.

Install heroku cli (google)

Log into heroku

heroku login

Initial Heroku Application Creation (Shouldn't need to do again)

Create the heroku application

heroku create ds-gridtime

Create the database (can upgrade to hobby-basic just by associating credit card w/ account)

heroku addons:create heroku-postgresql:hobby-dev

Configure the domain

heroku domains:add gridtime.dreamscale.io

Set up SSL

heroku certs:auto:enable

Ongoing Heroku Application Deployment

If you have not previously deployed to Heroku, initialize the remote

heroku git:remote -a ds-gridtime

Deploy the application on heroku

git push heroku master

If the server crashes:

heroku restart

If you need to access the server's database:

heroku psql

To tail the server logs:

heroku logs --tail

How to Troubleshooting Heroku DB specific Issue

heroku pg:backups:capture

heroku pg:backups:download

pg_restore --verbose --clean --no-acl --no-owner -h local.docker -U postgres -d gridtime latest.dump

gridtime's Issues

Organization setup - should post welcome notification to journal when owner joins

Created new organization, which the owner joins by default, and no welcome message was shown in the journal.

Setup pilot testdata in local database

Need to write the test harness for executing the Idea Flow data pipeline, and the easiest way to test and debug the functionality is with actual data. Setup the exported testdata locally for preparation with running with the data pipeline.

Create dashboard status report for jobs running on grid

Update CircuitActivityDashboard to do something that makes sense wrt reporting status on calendar and dashboard refresh activity first. The dashboard report should summarize the work in a way that makes sense to the user, after running the integration test in gridtime.

Setup fresh heroku servers with configurable (and changed) credentials

Heroku is currently deployed with default hardcoded credentials.

Make the configurations configurable, and then change the user/password of the deployed instances to something other than the defaults.

Upgrade to Java 11

The current version of gradle isn't compatible, and needs to be upgraded too (to Gradle 5)

Ability to explore detailed tile data from the CLI, starting from a gridtime coordinate position.

Once I find a WTF that's interesting, I should be able to zoom in to the details and go look at the tile where the WTF occurs, and see the files I was looking at, flow metric calculations, and all the details to help me understand what happened, and why it took so long.

I can pan left and right to see the activity that happened just before, or just after, wherever I'm looking.

Come up with a nice ascii representation of the data that makes it easy to read.

Create test harness for running Idea Flow data pipeline

Write a test harness for manually executing the Idea Flow data pipeline for a specific data set.

Need to be able to run the Calendar job to generate calendar tiles.

Then run the TileGenerator job to generate the Idea Flow tiles.

Top Reports: What are the top WTFs by block, by week, by day?

Need to be able to list top WTFs in latest block, latest week, for individual and team.

Then add command line hooks for displaying the results in a table.

IdeaFlow engine - Refactor TeamBoxConfigurations into ProjectBoxConfigurations

Now that organization members can be on multiple teams, Idea Flow individual feeds can't be team specific, so need to refactor this logic so that the box configurations (component mappings for files) applied for a tile comes from the project, not the team.

Since the Idea Flow aggregation is still done per team, and the members can be on multiple teams, whenever the member tile is generated, all the teams that the user is on will need to receive an aggregation event. Then we will need a way for the team to specify what projects they're interested in, even if team members are working on other projects for other teams...

So maybe a team projects list? Something like that. Applied at the team level as a filter on top of the individual feeds.

Create a CLI for the /terminal that works outside of torchie-shell

The gridtime /terminal client includes many command for starting/stopping grid, for setting up projects, inviting new team members, and many other things.

Ultimately we want the terminal commands to be integrated into the torchie-shell terminal, but for now, create a command line interface that works independently.

Organization setup - If an organization domain is setup under the wrong user, no way to fix it

Accidentally created an organization using the wrong API key, and now the domain is in use, and I can't create the organization under the right user. Need some way to be able to reverse these sorts of things, or at least not have the subscription be valid until the email for the domain is validated.

Need to think through these use cases, and setup some better handling around invalid organization setup.

Organization setup - validate email gives a 403 error

When validation email is sent out for organizational setup (email address is different than root account email used to setup subscription), clicking the email link gives a 403 error.

This resource path probably needs to be white listed in the security filters and isn't.

The validation email also should provide a bit more context about what's happening.

Setup Intellij Idea Flow plugin and configure to use local server

Setup the Intellij plugin, and configure to connect to local server...

Will need a local account, and local API key. Make this as easy as possible to set these up.

Create /terminal command for showing dashboard reports for jobs running on grid

Grid needs an interface to see what's going on!

Add a new terminal command that shows the status of our new dashboard reports.

IdeaFlow engine - handle failure and eviction of torchie processes on error

When a torchie process fails, the circuit monitor should count the failure, and the program should be evicted from the execution engine, and the failure should display in the dashboard.

IdeaFlow engine - duplicate key exceptions on feature cache when running concurrently

When there's multiple concurrent attempts to create the same feature key, duplicate key exceptions will occur. Need to debug the specific use cases, where this is happening, but since there is no guarding against this case in place right now, will likely need to implement a retry, so that if two threads try to key the same feature at the same time, the second thread can pick up the key generated from the first, and keep on going.