lapplislazuli / microtope

Multi-Language Microservice Playground

Hi There! I'm Leonhard. 👋

Welcome to my Github Profile.

I am currently doing my PhD at the TU Delft in the Software Engineering Research Group (SERG) where I work in the CISELab. My research track is labelled "AI Testing and Testing AI" - So I do twice the AI and twice the Testing that other researchers do. I am very much the Thermomix of Software Engineering AI. But recently, I have spent most of my work-time with Matthi on improving tooling for Haskell!

I like:

• ☕ Java & 💜 Haskell
• 🔍 Testing
• 👥 Humans

I try to do everything open source, so most of the repositories you see here have .

If you want to reach me, you can either

• 📧 email me at [email protected]
• ❗ Open an issue anywhere around and @ me

microtope's Issues

Add PiTest to Maven

Two-Sentence Summary:
MutationTests show the effectiveness of tests, and are part of a solid

Proposed Solution
Add Pitest to the Maven files of Pulser and Worker, and run them on every build.

As this is also some kind of coverage, it should fail after a certain threshhold.

Possible Alternatives:
There are afaik no other modern, maintained libraries for MutationTests, but I haven´t looked.

Possible Problems:
Maybe tests could need refreshment after PiTest added

Related Issues:
Issue #106 touches the same code

Additional Context:
A good example is given at https://www.mkyong.com/maven/maven-pmd-example/ with some explanation.

More documentation and a wait to run with threshhold is under https://pitest.org/quickstart/maven/

Here´s an example how to add the config https://stackoverflow.com/questions/54888240/possible-to-fail-maven-build-when-any-pit-mutation-fails-rather-than-mutation-c

Split ActiveMQ-All Dependency to needed ones

Two-Sentence Summary:
At the moment the big "ActiveMQ-All" Dependency is used.
This does not seem necessary and should therefore be avoided.

Proposed Solution
Lookup the "smaller" activemq packages and insert them in worker and pulser appropriately

The output jar should be smaller, therefore nicer docker images, and maybe faster (download wise).

Additional, the modularisation could work afterwards (but not for sure)

Possible Alternatives:
Keep as is

Possible Problems:
There should be no problems occurring.

Related Issues:
There were actually problems with the classpath to something similiar when trying to add modules in worker. #30

Also there were build issues because of bad dependencies in 4461169

Additional Context:
Current Maven Dep:
https://mvnrepository.com/artifact/org.apache.activemq/activemq-all/5.15.10

Jasmin Test for Writing and Reading PlayerName

After #23

Write a test which takes one of the known player-id´s in the standard-db,
sets his name, and reads the id if the new name is set.

This is my first component-test!

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Move to JDK 13

Increase the Java Versions to 13.x

Proposed Solution
It should be enough to increase all the maven deps to 13.x regarding java.
The Docker Containers are still running with Zulu 13 and Java didn't add that many features.

Possible Alternatives:
Keep Java 12, if so: Revert the changes from #226 and #227 to have als jre12 containers.

Possible Problems:
There should be little to none.

Additional Context:
#226 and #227 show that everything is ok with JDK 13.

Tidy Up Pom.xmls

The Pom.xmls of Worker and Pulser look awefull.

ToDo:

• Clean them
• Automate formatting?
• Check in Linting?

Maybe look for alternatives via google first.

Nginx Forwarding to https only

After #32

This should be quite easy implementable, I think a lot of people need this.

Jasmin TestContainer

Put the Jasmin Tests inside a container which runs the tests against the api.

Be carefull with Api´s CORS

Let the Container fail if the api does not come alive and is otherwise not running

Also do the healthcheck of the api

Make Shared Maven Module

The Pulser and Worker share a lot of code actually, and parsing everything with Objects would be much better (and also easier to write a fully tested module).

Nicest thing would be to get my Github repository and build it into my pipeline, having the other components pull the latest version from this github.

This will maybe introduce massive overhead in terms of integration.

Dockerbuildfile for Maven starves on Ubuntu

When running docker build with the new dockerfiles it starves after the testphase downloading some java-dom "xenia" images.
The Download Rate drops to b/s and I have not seen it finish.

This error does not occur in the pipeline.

MessageReciever Integrationtests sometimes not terminate

Reproduce:
Run the Message-Reciever tests ~5 times.

Guess:
The "while !touched" is quite goofy and seems to hurt.
However, it would be interesting to see why the test is failing.
I expect that the vm amq shares connections or something related, which gives me sideeffects.

Workaround:
Make a timer to fail the test after the time.
Other workaround: Run them in Order (e.g. with Prio).

Refactor Top3Component

Currently the Top3 is quite overengineered.

Just let it take @input like a normal item,
have a "top3" item subscribed to it.

Oh and have #34 in Place.

Remove all Generics, only use the interface IRankable.

So whenever the @input changes it is also updated and "cut" again. It´s somewhat important to not sort the input array to keep the allList unsorted (or sorted in a different matter).

Selection will be a little harder then - but one fix at a time.

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*.

Dependabot tried to authenticate with the details you previously provided, but authentication failed. If they are no longer valid you will need to provide Dependabot with new credentials.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Flickering Build

On c688b83 there was a build failure, which has been resolved by rerunning.

To Reproduce
Uff?

This is the failing run.

Expected behavior
Not being solved by just re-running - I want persistant or no problems

Screenshots

Additional context
the failing build was the "build all" which failed on a commit that changed a different sub-project (nothing even remotely related to the failing component).

The POM looked right, so i rerun and it succeeded. Maybe there was a problem with the maven repository?

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*.

Dependabot tried to authenticate with the details you previously provided, but authentication failed. If they are no longer valid you will need to provide Dependabot with new credentials.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Dependabot PR´s don´t trigger related github ci/cd

The PR´s from Dependabot only trigger the non-specific actions.
Those are labeler and greeter.

But the job for building the component is not run.

Steps to reproduce the behavior:

1. See #112

Expected behavior
The Action "buildAPI" should be run.

Additional context
The Job for UI works as expected, so maybe this is an api thing.

User-Creation Error

This Error Appears sometimes on Docker Compose Startup.
It's visible as there is no new User created and you will only see the 3 standart-users (on a fresh system) or the last users (on a system with existing database).

The console of docker compose will show mostly the error that "steps" could not be inserted due to constraint violation as there is no regarding player.
The real reason is (probably) that the player was not created.

From first investigation this happens if the team "0" is used as players team, which is not in the database and therefore the player cannot be created.

Possible Solution:

1. Add Null Team
2. Increase Player-Team by 1

the latter should be prefered, as otherwise i have one team too much which will never get newly generated data.

Also:
Decrease the "catch" block of the insertPlayer method of worker to see such errors better in the future.

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Migrate big pipeline from Azure to Github Actions

One major problem with the azure pipeline, as found out with help of dependabot, is that it stops after a certain amount of build-minutes. (Thx microsoft)

Also, the github actions seem to be faster.

Proposed Solution
Migrate the azure-yml to a github-workflow yml with a daily or maybe bi-daily build.
This should include every job and should be mainly reproducible from the smaller task.

It should not be triggered on PR, for this will just multiply jobs.

Alternatives
Alternatively, Jenkins or Travis or CircleCI could be used. But after github actions seems to be good so far, and is already in place, maybe sticking to one tec is better (see the reason for this issue)

Related Issues
None

Additional context
Most of the problems arising from azure could also be used if not the full pipeline is run on every build.
However, i wanted to migrate to github actions in the first place only my beta-access was a little delayed.

Add Nginx Reverse Proxy to API-Container

I think it would be best to have a nice Nginx before the Api, so only the http and https port are used and exposed.

I think it´s best to have it actually integrated into the container. It´s still "one service" so to say, and it should not blow up the files. The Nginx is also not supposed to serve for anything else than the api.

However if this is big-design, I want to try it and see how it works.

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*.

Dependabot tried to authenticate with the details you previously provided, but authentication failed. If they are no longer valid you will need to provide Dependabot with new credentials.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Dependabot couldn't find a pom.xml for this project

Dependabot couldn't find a pom.xml for this project.

Dependabot requires a pom.xml to evaluate your project's current Java dependencies. It had expected to find one at the path: /core/pom.xml/pom.xml.

If this isn't a Java project, or if it is a library, you may wish to disable updates for it from within Dependabot.

View the update logs.

Limit JUnit-Deps to Test-Scope

Two-Sentence Summary:
Limit the Scopes of JUnit Dependencies to "test".

Proposed Solution
Edit every Pom.xml to limit scopes as required. This should give smaller fat-jars.

Possible Problems:
it could break, but that should be seeable on local builds.

Maybe this does not affect the fat-jar compiler, then this needs to be done in the compiler plugin (if possible)

Related Issues:
None

Additional Context:
Baeldung Tut and Explanation: https://www.baeldung.com/maven-dependency-scopes
Stack Overflow Topic related: https://stackoverflow.com/questions/26975818/what-is-scope-under-dependency-in-pom-xml-for

Get Angular Unit Tests to run

Remove NPM install docker from ui/Dockerfile

The Dockerfile for ui currently installs angular@latest before the normal npm install.

I dont think that is necessary and also gives a warning when there is a version missmatch.

Additonally, this leads to other docker-images without any changes from devs.

Proposed Solution
Remove line 11 from The UI Dockerfile.
RUN npm install @angular/cli@latest -g

The version specified in package.json should be installed.

Alternative Solutions
It could be kept as is, but the problems are described above.

Angular NGinx with Https

It would be nice to serve the container with a known certificate.

Tino used let´s encrypt to make a new certificate for every container, but i don´t like this very much.

I would save both certificates in this repo for dev purposes and supply them with volumes. So if i trust the certificate once i see that this is my known dev-environment.

More restrictive CORS for api

Right now CORS allows everything from everywhere.

With #36 I can only allow localhost.

Also I can limit the actions as there are currently only "get" and "put" requests and nothing else planned at the moment.

Share "Wait-for-it".sh with Symlinks across sub-projects

I wanted to mark this as "Idea" but actually i think it´s quite important to have a stable environment with exactly the same files on every project.

Add Code Coverage to Karma Conf

Active MQ Api

It would be nice to have an api which can insert messages into the activemq which are parsable.

when using the api you could have some curls laying around which have direct effect in the ui.

Additional this would be a nice possibility to have a new language in the project.

As two more ideas the pulser could use the api instead of amq directly or a gui can be build on top of the api. (maybe a non-web-gui?)

Dockerbuildfile has wrong locations

Seen on the pipeline:

In Stage 2 the copy paths for target are wrong.
They need to be absolute (/app/target/...)

Publish Docker Images to Packages

As a first step, maybe before #30 , it would be a nice start to publish the docker images to github packages with a github action.

Proposed Solution

• When the master is updated (not on PR) a github Action should be triggered
• this should only run for changed components, selected by branch
• the action should build and publish the container into the github package registry

The github action should use secrets in/from github.

Alternatives
Do the same Action but for DockerHub

I prefer keeping everything in git.

Additional context
Good StartingPoint: https://github.com/marketplace/actions/publish-docker

That way it should be run when the PR is accepted

Add Docker Healthchecks

I can define Healtchecks both in compose and dockerfiles for my applications to run.
Good first candidate would be the API or the Database for a try

https://docs.docker.com/engine/reference/builder/#healthcheck

Dependabot needs permission to see docker

Dependabot couldn't resolve your project's dependencies as it couldn't access docker.

You can provide additional git credentials in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'. If you use a custom token for the host github.com make sure it has read access to this repo, too.

Refactor TopN Rankable

Currently the "top3"-Component has the logic for getting the top 3 items.

However, this is a pure function of IRankable. Therefore extract both "compare(IRankable,IRankable)" and "sort(IRankable[])" to own files and only use them in the component.
Additonally these are easier testable with jasmine.

CI/CD Check on shared-resource equality

Problem
As shown in The meta readme there is a problem with hardlinks, symlinks and the way github and docker are used.

The problem is that docker does not support symlinks and github does not really support hardlinks. This is ok, but this project would need a way to check whether every linked file is still the same or one was altered, and should fail-on-build.

Describe the solution you'd like
A simple script which should check for file-equalities from shared-resources to the occurances and fails on difference.

This file should be run in a CI/CD job on every commit.

Describe alternatives you've considered
There are wrapper-tools to support checked hardlinking git-repositories, but I had not the time to check on them.

I think doing some shell scripting is a nice way to give another example to other interested persons. This is rather a common usecase and solving it with a simple github action and a shell script might help the world.

Related Issues

The problem popped up with #19

Add Code Coverage to Java Parts

Use Jacoco

Make UML Diagram

https://marketplace.visualstudio.com/items?itemName=jebbs.plantuml

That´s a lightweight tool to get it running and in version control

Dependabot couldn't find a pom.xml for this project

Dependabot couldn't find a pom.xml for this project.

Dependabot requires a pom.xml to evaluate your project's current Java dependencies. It had expected to find one at the path: /core/pom.xml/pom.xml.

If this isn't a Java project, or if it is a library, you may wish to disable updates for it from within Dependabot.

TruffleHog CI/CD Action

TruffleHog is a known python-program to check for secrets in Code. It would be nice to check the code with trufflehog in every PR to not introduce new secrets (if any).

Proposed Solution
A docker container which should:

• Install Pip & Trufflehog
• Install Git and pull Repository (the full)
• Run Trufflehog (With certain files excepted)
• Fail (the container) if Trufflehog finds any secrets.

Then:

• Make a github action to run the container
• impement the github action on EVERY pr

Alternatives:
There are actually many tools such as trufflehog on the opensource-market,

But TruffleHog has a lot of functionalty needed and seems quite easy to use.

Additional Context

Trufflehog repository

If done after #100 it can be added to the same github action.

This would be maybe a mighty by-product to contribute to github actions.

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*

Dependabot couldn't authenticate with https://maven.pkg.github.com/twonki/*.

Dependabot tried to authenticate with the details you previously provided, but authentication failed. If they are no longer valid you will need to provide Dependabot with new credentials.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Run Angular Unit Tests in Container & Pipeline

Requires #27

There is some ng test --chrome headless running around, the only thing is to get it into the build container properly and understandable.

Test bad-language remover

Fuck

Change PlayerName / TeamName on API Level

Use REST to make the Database changes.

Sign Docker Images in Package Registry

It would be great to have a signed package registry.

Proposed Solution

• When the Github Action for Package registry is performed
• A git-secret for the gpg key should be hurled
• And the Package should be Signed with it

Alternative
There is no real alternative IMO.

Related Issues
This should be done after #101

Additional context
Maybe the "Trust" is not related to the Docker image, but also to the Package registry. I am not sure about that to be honest

Linter-Container for CI/CD

Inside (maybe explicit at the start?) of the Github Pipeline on every PR should a job run to check linting.

Proposed Solution

• A docker container to checkout the repo
• Run every linter in place (checkstyle, ts lint)
• Fail on Linter fail
• (Make failure visible on console)
• Github Action to run the Container on Pull-Requests

Describe alternatives you've considered
The Linter-Container could be split on Technologies, and maybe only run on

Related Issues
This should be added to #97 aswell

Additional context
This could be combined with #86 .

Also a .md linter would be very nice.

Git action to post PR´s into Telegram Maintainer Group

It would be great to post a short telegram message into a maintainer group.

Proposed Solution
If a new PR is made, a github actions should trigger a telegram bot which posts into a "microtope" group.
The PR-title and a link should be posted there.

This group has to be created and every maintainer who wants to join must be (manually?) invited.

Describe alternatives you've considered
Similiar function could be achieved with discord. But i´m not sure how the privacy is handled in discord, while it´s easy possible for everyone to be very anonymus on Telegram.

Another alternative would be to send to the people directly, but i wanted to have an additional layer of privacy.

Additional context
I have not yet looked around on the github market place, maybe most of the things are already somewhere in place

Add SpotBugs to Java Parts

Two-Sentence Summary:
The SpotBugs-Library should be run on every change to java-parts for every java-part.

Proposed Solution

• Add Spotbug to Mvn
• Fix current Spotbug problems
• If any Configuration is required, add it to the directories
• Run normally with the CI/CD Pipeline

Possible Problems:
It could take very long to fix the current Spotfix problems, and maybe not all of them are usefull / necessary. This may needs a (smart) configuration.

Additional Context:
A good example how to add it to a java project is https://github.com/mkyong/maven-examples

Maybe you can add PMD on the run https://www.mkyong.com/maven/maven-pmd-example/

Extract SQLiteDB from finished MariaDB Container

This is just a nice idea:

There is a script called MySQL2Sqlite which could help to provide the db as a nice sqlite and therefore inmem database.

Reason for this is that the mariadbwriter in worker was untestable in unit and integrationtests without very strange mocking.

Following Goals would be great:

• Make a new Docker Container above the microtope/database
• Add the Script
• Run the Script
• Move the resulting dumb to a specified volume
• Make a docker compose to run the script and bind a volume next to the compose

MariaDB Timewalking

There is a possibility to enable "timewalking" for my Database.
That means that i can make queries "SELECT * FROM player_details AS IT WAS 2016".

That´s an awesome feature and I want it!