cloudhubs / graal-prophet-utils

This repository is a version of the prophet-utils repository, adapted to the GraalVM static analysis of microservice applications project

Graal Prophet Utils

This repository is a version of the prophet-utils repository, adapted to the GraalVM static analysis of microservice applications project.

For the high-level overview and instructions how to perform the analysis, please go to Graal MVP.

Running MicroGraal

• Build graal
  • git clone https://github.com/cloudhubs/graal.git
  • Install mx
  • Install jdk with jvmci and set JAVA_HOME
    • cd graal/substratevm && mx fetch-jdk
  • Compile graal
    • cd graal/substratevm && mx build
    • export GRAAL_PROPHET_HOME=$(mx -p graal/substratevm graalvm-home)
• Build graal-utils
  • Install Maven
  • Compile project
    • cd graal-prophet-utils && mvn clean package -DskipTests=true -X
• Analyze a project
  • Clone microservice project to be analyzed (e.g. train-ticket)
  • Create microservice metadata file
  • Compile microservice project
    • cd train-ticket && mvn install -DskipTests=true
  • Unzip jars
    • Example:
      • Set env. MS_ROOT
      • Run the help script ./microservice-setup.sh config/trainticket-microservice.json (requires jq json parser tool)
  • Run the analysis
    • $JAVA_HOME/bin/java -jar target/graal-prophet-utils-0.0.8.jar ./config/trainticket-microservices.json
  • Output will be in output_SYSTEMNAME

Note:

• This program only currently accepts Java microservices on Spark
• Program only runs on Linux machines
• This program depends on cloudhubs/graal repository

Setup:

1. Install or build your every microservice

2. Run microservice-setup.sh

   • You will need to configure the 'directory' variable to contain the path to your system for your machine. It currently contains all train-ticket microservices, but for the bash script to unzip all microservice jars for another project, you must replace the current list with one of all their names.
   • This script unzips all of the project jars to expose the BOOT-INF directory. It also deletes the SnakeYAML jar from the projects target if it exists.

3. ProphetUtils first argument is: the path to the microservices JSON. The format of the microservices JSON is as follows

   •   {
           "systemName": "systemnameHere",
           "microservices":[
               {
               "baseDir": "/home/person/work/microservice_system/microservice_root/",
               "basePackage": "edu.baylor.ecs.cms",
               "microserviceName": "cms"
           },
           {
               "baseDir": "/home/person/work/microservice_system/microservice_root/",
               "basePackage": "edu.baylor.ems",
               "microserviceName": "ems"
           },
           .
           .
           . etc
           ]
       }
     

   • Other options are dependant on the version below

TWO VERSIONS

main branch

1. This is a generalized approach and implementation for all microservice systems
2. Uses Levenshtein distance and approximates the distance for service dependancy linking
3. OPTIONS: <percent_match>
   • percent_match: integer value, similarity matching threshold

Austin-adding-new-params-for-requests branch

1. This branch is for analysis of train ticket
2. This one is not recommended for generalized approach as it parses patterns specific to train ticket
   • Parses destination microservice from URI and or host name depending on version of train ticket
3. Uses partial signature matching for the URI, body param, and HTTP method
4. OPTIONS: <is_train_ticket> <percent_match>
   • is_train_ticket: specify option if it is train ticket or not
   • percent_match: integer value, similarity matching threshold

NOTE: Both approaches parse by body param and HTTP method

FUTURE IMPROVEMENT RECOMMENDATIONS:

• Partial signature matching for the URI could be implemented for the generalized approach in the main branch instead of approximating with distance
• Partial signature matching is defined as only matching hard coded parts for the URI and ignoring path params

graal-prophet-utils's Issues

Where is `PROPHET_PLUGIN_HOME` used?

After some experiments, it seems that graal, graal-prophet-utils and the analyzed project repositories all need to be located in PROPHET_PLUGIN_HOME for analysis to work correctly.

The only reference in this repo to PROPHET_PLUGIN_HOME seems to occur when expanding the paths to all the microservices if they are defined relatively in the config file. So, I am assuming that this variable is also hardcoded in the graal repository?

At least for the microservice project, I think we need to decouple and introduce something like MS_ROOT variable to resolve the paths with. This will allow to have the project in an arbitrary location.

Split the NativeImage and Linking stages

We discussed before that it would be good to refactor the code so that the two stages are clearly split, and it is in theory possible to use them separately.

I already have some commits from some months ago working on it, so I can resume the work and make a PR for that.

Is it necessary to re-try analyzing a service?

Currently, the code works so that if a certain microservice fails to be processed, its try-count is increased by 1.
The loop iterates over all unprocessed services until either all are exhausted or all the leftovers have a try-count over 3.

Is this really necessary? Could the analysis first fail and then succeed during the same run?

This arrangement complicates the Native Image logic and data structures.

I can remove this while working on #9

Decouple from `train-ticket`

Currently, several places in the code have train-ticket specific processing:

• CLI option isTrainTicket
  • This option is only required in one place during the linking stage

    graal-prophet-utils/src/main/java/baylor/cloudhubs/prophetutils/visualizer/LinkAlg.java

    Line 138 in 3efc79e

    if (this.isTrainTicket) {

  • We can enforce that the system must have the name 'train-ticket', and set this parameter internally
• Data class contains hardcoded names of train-ticket "common" entities (ts-common), which are unused

  • graal-prophet-utils/src/main/java/baylor/cloudhubs/prophetutils/contextmap/Data.java

    Line 21 in 3efc79e

    private List<String> tsCommon = new ArrayList<>(Arrays.asList("Account", "AdminTrip", "Assurance", "AssuranceType", "Config", "Consign", "Contacts", "DocumentType", "Food", "FoodOrder",

  • Can be safely removed
• ReadCreate class contains hardcoded names of train-ticket "common" entities (ts-common)

  • graal-prophet-utils/src/main/java/baylor/cloudhubs/prophetutils/contextmap/ReadCreate.java

    Line 36 in 3efc79e

    private List<String> tsCommon = new ArrayList<>(Arrays.asList("Account", "AdminTrip", "Assurance", "AssuranceType", "Config", "Consign", "Contacts", "DocumentType", "Food", "FoodOrder",

  • They are then written to entities.json

Remove duplicate classes in `Data`

The Data class implements internally classes Annotation, Field, Name, Entity, which are also defined separately in systemcontext subpackage.

Their functionality should be merged and the classes should be left in systemcontext.

Remove unused classes

Classes Type and BoundedContext are not used, according to my IDE detectors.