stem-c / casmm

Computation and Science Modeling through Making Full Stack Web App

CaSMM

Computation and Science Modeling through Making

Cloud-based programming interface

Application

client

client is the frontend of the application. It is powered by React and Blockly.

server

server is the web server and application server. It is powered by Node and Strapi.

compile

compile is an arduino compiler service. It is an unofficial fork of Chromeduino.

Environments

The project is divided into three conceptual environments.

Development

Structure

The development environment is composed of five servers. The first one is run with the Create React App dev server. The later four are containerized with docker and run with docker compose.

• casmm-client-dev - localhost:3000

• casmm-server-dev - localhost:1337

• casmm-compile-dev

• casmm-db-dev - localhost:5432

  The first time the db is started, the init_db.sh script will run and seed the database with an environment specific dump. Read about Postgres initialization scripts here

• casmm-compile_queue-dev

Running

casmm-client-dev

1. Follow the client setup
2. Run yarn start from /client

casmm-server-dev, casmm-compile-dev, casmm-db-dev, and casmm-compile_queue-dev

1. Install docker

2. Run docker-compose up from /

   Grant permission to the scripts and server directories if you are prompted

Staging

Structure

The staging environment is a Heroku app. It is composed of a web dyno, compile dyno, Heroku Postgres add-on, and Heroku Redis add-on.

• casmm-staging - casmm-staging.herokuapp.com
  • The web dyno runs server
  • The compile dyno runs compile

Running

casmm-staging is automatically built from the latest commits to branches matching release/v[0-9].[0-9]. Heroku runs the container orchestration from there.

Production

Structure

The production environment is a Heroku app. It is composed of a web dyno, compile dyno, Heroku Postgres add-on, and Heroku Redis add-on.

• casmm - www.casmm.org
  • The web dyno runs server
  • The compile dyno runs compile

Running

casmm is automatically built from the latest commits to master. Heroku runs the container orchestration from there.

Maintenance

All three components of the application have their own dependencies managed in their respective package.json files. Run npm outdated in each folder to see what packages have new releases. Before updating a package (especially new major versions), ensure that there are no breaking changes. Avoid updating all of the packages at once by running npm update because it could lead to breaking changes.

Strapi

This is by far the largest and most important dependency we have. Staying up to date with its releases is important for bug/security fixes and new features. When it comes to actually upgrading Strapi make sure to follow the migration guides!

CI/CD

All of the deployments and releases are handled automatically with GitHub Actions. The workflows implement custom Actions that live in the auto repo.

Contributing

Git Flow

We will follow this git flow for the most part — instead of individual release branches, we will have one to streamline staging deployment

Branches

Protected

Locked for direct commits — all commits must be made from a non-protected branch and submitted via a pull request with one approving review

• master - Production application

Non-protected

Commits can be made directly to the branch

• release - Staging application
• develop - Working version of the application
• feature/<scaffold>-<feature-name> - Based off of develop
  • ex. feature/cms-strapi
• hotfix/<scaffold>-<fix-name> - Based off of master
  • ex. hotfix/client-cors

Pull Requests

Before submitting a pull request, rebase the feature branch into the target branch to resolve any merge conflicts.

• PRs to master should squash and merge
• PRs to all other branches should create a merge commit

casmm's Issues

Capture/Restore Student Saves

Right now student code is only saved in browser while at the workspace view. This view needs to be updated to allow students to save and load "code submissions" when they want to. This also entails updating the backend services to handle student saves.

Frontend work:

• The workspace needs to have auto save features on load, close, on submission, and every x minutes
• The workspace needs a manual save button
• Students should be able to load their individual saves of a specific activity on the workspace

Backend work:

• Student saves should be stored per activity id, student id, and session id
• An endpoint needs to exist that can access a specific save given the day Id and student jwt
• An endpoint needs to exist that can save students work given the workspace, day id, and student jwt

Code templates

Frontend:

• Load block template from activities.template

Backend:

• Add template field to activities

Improve Continuous Delivery

• Create review github action to create/delete test apps
• create db init script
• init basic tests

Workflows
PRs open/reopen (develop):

• Create review app
• Build and push images tagged as review-pr-#
• Tag and push to heroku registry
• Release

PR close (develop):

• Delete review app

PR synchronize (develop)

• Build and push images tagged as review-pr-#
• Tag and push to heroku registry
• Release

Push (release/vX.X)

• Build and push image as staging-vX.X
• Tag and push to heroku registry
• Release

Release Published (master)

• Build and push image as release
• Tag and push to heroku registry
• Release

Upgrading Production

After we stress test the application, we may need to scale up our production environment. This would start by serving our static content via a CDN, then scaling up our backend web containers

Phase 1

• Serve static content via S3+Cloudfront
  
  • Remove frontend build from docker
  • Create a github action to build/deploy client to s3
  • Create/configure s3 bucket
  • Create/configure cloudfront distro
• Scale heroku web containers
  • Upgrade to horizontally scaled containers

Phase 2

• Migrate to AWS (Cloudfront, load balancer, fargate, rds, sqs, lambda)

Resources

• https://www.quora.com/When-using-node-js-do-you-still-need-Nginx-or-Apache
• https://medium.com/@derrickmehaffy/using-strapi-in-production-with-https-717af82d1445
• strapi/strapi#2617
• https://stackoverflow.com/questions/25439566/using-heroku-postgres-do-i-still-need-nginx

System Logging - Readable Issues for Production Support

We need to think of the ways that the application might fail and create readable issues that allow us to quickly identify issues and fixes to reduce downtime or system failure while in production.

• common ways users can crash the system
• bugs that are likely to cause system crash
• create reable issues to slackbot can report so that we can quickly address them

Student/Teacher Login Views

For Login:

• Student join code is now next to teacher login on same page

For student auth:

• Let mentor upload roster and upload animals as passwords
• Student page shows multiple "user dropdown"
• Students see two users by default, can add or remove
• Let students select their name from dropdown
• Let students select their character from dropdown
• Allow jwt to hold 1 or more student's auth

UI should resemble:

Content creator portal

Strapi should be customized to allow content creators to create and modify different blocks, activities, and topics. This is a V2 issue and should be expanded upon at a later date once V1 is nearing release.

Multiple activity page routing

Requirements for this feature:

• User can select an activity from a given list of activities
• List of activities must be shown from a list given at "/activities"
• Each activity must be able to show a unique list of available blocks and show given pictures for different activities.

Setup Server Development Environment

• Create a docker compose for the app
  • Add a mysql image
  • Add a strapi image
  • Create/add react image
  • Create/add compile image
    • Add compile service to project

Authentication: Login View

This feature is based upon the following layout of views, this feature only requires that the numbered views be completed:

1. This is the basic login view. This should allow a teacher to login on one screen and then view a basic dashboard of activities in the next screen.

• Teacher/Mentor can view page to login
• Teacher/Mentor route is protected
• Teacher/Mentor can view basic dashboard of topics/activities
• Teacher/Mentor can view students in a given session
• Teacher/Mentor can enable or disable sessions
• Components are restyled using new antd standard.

Stress testing

Stress test the application

Phase 1

• Using custom tool
• Testing static load
• Testing activities load

Phase 2

• Using proper service
• loading the frontend w/ sim. interactions
• load on the rest endpoints
• load on the compile server

Server scaffold

Find a headless CMS based on Node that will generate REST endpoints, database, authentication, and an admin panel. Implement the current data entity and generate sample activities.

• Init the ER model
• Init server documentation

Release automation

Right now there is a consistent workflow that is manually done. This process could be easily automated with github actions.

• On a merge into master, tag that point as a new version based on the PR title. "Release v1.0" would tag master as v1.0 for example.
• On a merge into master, also merge into develop

Update avrgirl dependency

Currently we are using a modified file from the avrgirl-arduino package and importing that as a library. This did cause an issue with some code that had to be commented out. The code was not needed, but at some point it would be useful to fork the package and create a custom package. The edits that need to be made are in the avrgirl-arduino.js file in this repository with comments which include "STEM+C".

• Create a repository and link as new dependency
• Edit package to allow hex code instead of a file path as an arg for ".flash"

We need auth

adfas

Stats/Analytics

We need to look into a stats engine for getting page views, user interaction, etc.

• For anonymized usage, we could just use google analytics and custom events
• For specific usage by classroom/student, we would need a custom pipeline (kinesis + elastic search)

User Logging

We need a way to track student progress on activities. Block picked up, deleted, state of blocks when he/she runs the code, modifications etc.

These logs will be used to analyze student usage of the tool from a learning an a usability perspective.

CMS sqlite3 dependency

During project installation, sqlite3 won't download properly and then lead to the server not starting. We no longer use it, so it should be removed from the dependency list.

• Remove from package.json
• Remove from database.json

New Block & Backend & frontend visualization - Capability - Ability to Read from Arduino

AnalogueRead (associated with Analogue Pins on arduino) - INPUT
Reads in a value based on resistance of circuit or voltage.

Staging and production monitoring

• Create a heroku slack channel
• Create a slack app w/ incoming web hooks
• Setup web hooks on heorku
• Create CMS endpoint to handle data transformation

Code block comments

This issue mostly deals with creating comments for code blocks and saving them.

Frontend:

• Display code block comments to user

Backend:

• Associate comments with different code blocks unique to each assignment
• Create a content type for comments

API Documentation

• Create full documentation
• Document on readme

Lock Blocks

The blocks in the user workspace should be able to be locked. This would prevent the user from making changes to the block once the xml form is loaded. This issue does require #38 to be completed since loading a xml workspace is a prerequisite.

• Allows user to load template with locked blocks
• User is unable to unlock blocks from workspace

Run/compile cooldown

We need to implement some sort of compile cool down. We don't want students spam clicking the run button and flooding the compile server w/ redundant requests. Here is an example of how leetcode limits their remote code execution:

This could be very simple and lead to a robust solution with the help of #6

Additionally, we need the client to limit the number of requests it sends.

Backend

• Check the last student submission for a time threshold and/or a workspace diff

Frontend

• Render an error message when trying to compile within the cool down
• Render an error message when trying to compile the same code as last time
• Have a request counter - if it hits the counter log an err message / try again

Styling of the COM port Pop-up window

If possible, it would be great to style the COM port Pop-up window that asks the user to select the USB port. This will ensure that it looks like a feature of the software and not an error.

Load Activities

Allows a user to load an activity based upon a join code. The views required are specified here:

The required features are as follows:

• Create view for join code like shown in 1 in the image
• Bring user to a workspace based upon code from view 1 into view 2
• Query user for name
• Restyle application using antd

Organize Readme

Migrate to strapi latest

Strapi has now officially been released with additional features and bug fixes

• Update cms strapi dependencies
• Update documentation links to the latest version

COPPA Privacy & Security Issues

Identify common privacy and security issues we need to address in this project since we are working with students and collecting data.

Data we would like to store or need to store

• username or student name
• lunchID or studentID
• artifacts created on the site/application
• logs of mouse interactions or typing on the site only

To Do

• Review COPPA policy & identify privacy and security issues that we need to address
• write a simple policy statement for our web application
• identify tests that we can run to ensure privacy and security

Resources
https://dataqualitycampaign.org/wp-content/uploads/2016/03/DQC-Student-Data-and-Consent-Policies.pdf

Federal Trade Commission - Children's Online Privacy Protection Rule ("COPPA")
The official website of the Federal Trade Commission, protecting America’s consumers for over 100 years.
Website - https://www.ftc.gov/enforcement/rules/rulemaking-regulatory-reform-proceedings/childrens-online-privacy-protection-rule

Link to the PDF - https://www.ftc.gov/sites/default/files/documents/rules/children%E2%80%99s-online-privacy-protection-rule-coppa/coppasurvey.pdf

Testing

• Create testing plan/protocol
• Write tests for the endpoints we currently use
  • /topics
  • /activities/toolbox/:id
  • /compile
• Write tests for core UI functions

We should use Jest as it works rly well w/ react and es6 modules

Blockly Sandbox View

• Users should be able to access a sandbox view without authenticating
• The sandbox should show a complete blockly canvas and toolbox
• The User should have access to all available blocks
• The user should have access to all additional functionality available in workspace view such as compile and arduinoToJS

Setup milestones and clarify all V1 tasking

Right now the issues are somewhat disorganized and some are missing. At a minimum we need to review all v1 requirements and establish all v1 tasking. This then should be broken down into separate sprints with their own milestones (i.e. v0.2 or v0.3).

• Finalize all v1 tasking
• Establish separate sprint tasking during v1 as minor versions

CaSMM branding

Frontend:

• Add CaSMM to the interface header
• Add colors to client readme
• Update favicon
• Give index.html a title

Backend:

• Add CaSMM logo to admin
• Update favicon
• Implement "brand book" styling

Implement continuous delivery

• Configure heroku.yml to build docker containers
• Create heroku account
• Create heroku pipeline
• Create staging app w/ db
• Create production app w/ db
• Create compile app
• Update deployment documentation
• Setup automatic deployments

Strapi limitation on nested group data

There seems to be an issue in strapi where nested group data (such as an array) does not render correctly and will always appear null. Testing locally showed this was always true regardless of other factors (link type, data type, data requirements).

This issue was discussed with a working solution here.

This either needs to be fixed or the entity model needs to be adjusted to account for this. Due to time restraints there will be a temporary fix setting blocks to be a 1st level array. This temporary change will be reflected in the schema as well as code in the workspace render.

• Implement a fix that allows nested group data to appear in strapi
• Adjust the frontend workspace render method to account for the corrected schema

about page

This page will give general information about the project until a webpage for the project is setup.

• Brief description of the project & functionality of the software & project team
• logo of UF
• logo of TAMU
• logo of NSF

Add sockets to classroom connection

Socket connections

• Frontend listen to socket events
• Implement socket.io in bootstrap function
• Implement emit on student join

Rename org

Per Dr. CHu's request, the org should be renamed to STEM-C

Update git flow

Production will still automatically deploy from master. Staging will now automatically deploy from the release branch. Develop will be the latest and will not deploy anywhere. PRs made onto master will be squashed and tagged with release.

• Update documentation
• Create release branch
• Update automatic deployments for staging app
• Make develop non-protected

Session Management

For the session management, we need to implement new content types and custom controllers

• Update IR digram
• Implement new content types
• Write custom session functions

Client error handling

Currently, all the api calls have no error handling leading to a bad UX if requests fail

• In requests.js, add a wrapper for the await calls to log and return the error the caller
• In the views, create a notification like "x could not be loaded, try again later"

User Friendly Compile Errors

When student code does not compile we need a way to send user friendly compile errors and suggestions to help them resolve the issues.

Completing this task will require

• research of common compile errors & common coding errors that student may make
• rewrite common compile and coding errors in child friendly language
• create a view/modal to display compile errors

Capture Student Submissions

• Update documentation

Server

• Create submission content type
• Add REDIS docker service
• connect to queue on start
• Override the create controller to add to queue
• Create sandbox endpoints

Compile

• rewrite compile to pull from queue

Automation

• add redis addon to review app create
• build and use compile image

Client

• On upload code
  • create a save
  • POST /submissions with the { day: dayId, workspace: "block code", board: "arduino:avr:uno", sketch: "arduino code" }
• After submitting make sequential GET /submissions/:submission-id till status is COMPLETED (hex will be in the final request response)

Wiki

As we get closer to a journal, we need to start to move our documentation to a wiki. We can try github's own wiki or pull a 3rd party tool

Rolling Documentation Updates

• Dyno restart policy - https://devcenter.heroku.com/articles/dynos#automatic-dyno-restarts

Cloud cost breakdown

• Create a unit of analysis
• Create a cost analysis for each service we need

Heroku Pricing:

Configure Roles/Permissions & Policies

• Create roles & permissions for all the endpoints
  • ie. the public role can only access register/login/etc
• Create policies to add specific auth logic to key endpoints
  • ie. a mentor should only be able to pull classrooms they belong to

React builds but is not usable

The react fronted builds correctly but when index.html is accessed will error out with the following message:

This is (likely) caused by the following code:

Client scaffold

• Init a react app in the client scaffold
• Add blockly
• Serial api
• Integrate with backend via REST calls to access activity list.
• Integrate with backend via REST calls to customize workspace.
• Finalize Client documentation BEFORE making a PR
• Setup frontend middleware to properly serve react + routing
• Configure build to handle new serving

Production Security

• Change the admin path from /admin to something less notable for security
• Configure cors to only accept same site
• Add some sort of encryption to the jwt tokens to hide payload contents (possible bcrypt)
• Switch over to using httpOnly in place of localstorage for jwt's