retest / gui-state-machine-api Goto Github PK

A REST API might be too slow when the state machine is updated constantly.
Hence, provide a simple Scala API which can be called directly in Java.

We use only the build on travis-ci.com. It is confusing to have both builds.

The NFA might be updated concurrently in the future. Add concurrency support to protect critical section from data races.

The issues #17, #21 and #24 change the API, so create a new release as soon as both are resolved to provide a stable API.

A release would require a release of surili-commons, so maybe just publish the snapshot and adapt it in Surili?

• https://doc.akka.io/docs/akka-http/current/?language=scala
• https://www.playframework.com/ (maybe to advanced - model view controller, we don't want a web application, only a RESTful service)

Akka HTTP is not a real framework but a suite of libraries. Since we do not provide a web application here but a simple REST interface, it should be sufficient.

TODO instead of using rest.model, map our types from api into JSON and back.

Clarify if we need the following parts from the legacy code and how they were used:

• The execution counter for the total number an action has been executed StateGraph.actionExecutionCounter? The execution counter for the number an action has been executed from a certain state StateGraph.executionCounter? Both execution counters weren't used except for verifications in unit tests and to get the never explored actions (which we can also just store in a field). So basically, we don't really need these counters. However, it could be interesting for the visualization or analysis of the NFA to see how often actions are repeated by the GA etc.
• Blacklisted and whitelisted components? They were used to filter all available actions when getting a state. See Environment. Since we provide no getState with a ExecutingTestContext, we do expect in our getState and executeActions the filtering to be already done. So it would have to be done from the client side not this API.
• Getting a state by an execution context only? Should never be called if the filtering is not done in this API (see above).
• StateGraph.getRandomAction and StateGraph.getRandomActions? getRandomActions seems to get all never explored actions from a state and to add all explored actions which means it simply gets all actions (confusing method name). StateGraph.getRandomAction chooses one from StateGraph.getRandomActions and randomizes it with the method Action.randomize which has to be implemented by the concrete action type. getRandomAction is required by Monkey Testing. getRandomActions is used to produce a random action and directly by the ML module only when there are no unexplored actions where also getTransitions().keySet() could have been used directly. Therefore, only make getRandomAction public for Monkey Testing.
• StateGraphListener with a method called whenever a transition was added. It was only used by MonkeyTestingModel to update the information about the progress in the GUI. We know that when we execute executeAction there will be a new transition. Therefore, the information is already given for the calling client.
• StateGraph.unknownUIState and UnknownState. The method unknownUIState is only used for ActionStateSequenceOld when cloning.
• StateGraph.groundState and GroundState. Its purpose is to identify a state graph without transmitting the whole state graph in form of a UUID. StateGraph.getGroundState is only used to get the execution date and to create the ExitState and UnknownState. Every state stores its corresponding GroundState. It is then used in ActionStateSequenceOld and when converting it to ActionSequence.
• PrimitivesPool
• StateGraph.exitState and ExitState was created in the constructor when there was a groundState although no transition led to it?
• AbstractState.canPossiblyExecuteActionSequence and AbstractState.canPossiblyExecuteAction.
• StateGraph used MD5 hash values to compare states and use keys. Better performance than comparing their descriptors?
• StateGraph.getAllNeverExploredActions has to be supoorted for chromosomes to generate new test cases. Store it globally. Otherwise, we would have to collect it from all states etc. like in the legacy code -> bad performance.
• StateGraph.getAllExploredActions was only used in the registered transitionAdded to show in the GUI how many actions have been executed by Monkey Testing. Is not hard to support as global variable in the state machine.

Some links:

• https://github.com/sbt/sbt-release
• https://github.com/verizon/sbt-rig
• https://github.com/scalacenter/sbt-release-early
• http://timperrett.com/2016/10/02/continuous-delivery-for-scala-with-travisci/
• https://central.sonatype.org/
• https://stackoverflow.com/questions/42514143/how-can-i-publish-a-snapshot-artifact-using-sbt-release

Commands sbt release and sbt publish.

Other retest artifacts are released and published this way: https://github.com/retest/recheck/blob/master/pom.xml

With #1 it is not that important to provide releases/publish them online since the REST service can just be deployed and run on a separate machine.

For instance, GuiStateMachineApi and its implementation look as follows:

trait GuiStateMachineApi { /* ... */ }

object GuiStateMachineApiImpl extends GuiStateMachineApi { /* ... */ }

Is this a thing in Scala to let objects implement traits? In Java, it looks a bit odd since I cannot code against the actual API anymore. Instead, I have to use the implementation directly:

Id stateMachineId = GuiStateMachineApiImpl.createStateMachine();
// ...

See #5 for the investigation of GraphDBs.

Neo4J allows transactions which provide concurrency support.
It can also visualize the state machines/graphs.

Labels group nodes together, so a SutState is not really a label.
An action type can be a label.

Some information about OGM:

• https://neo4j.com/docs/ogm-manual/current/tutorial/#tutorial:introduction
• https://github.com/neo4j-examples/neo4j-ogm-university

We store only the hashes and some additional information of the SutStates and actions in the graphdb.
Otherwise, it would be an antipattern.
Note that we need to use the Bolt protocol to utilize most of the tools for visualization and management.
Embedded databases are useful for storage only.

Open questions:

• How can we separate graphs with Bolt? Different port numbers?
• How does the user specify username and password for one graph database using the API?
• Does the user have to setup a Bolt Neo4J database before he/she can use the API or does our API automatically set it up for the first time?
• Which tools are the best for management and visualization? I have found the Desktop application https://neo4j.com/developer/neo4j-desktop/ which allows you to start Bolt graph databases and brings a browser which can visualize them. The browser's UI is available in a web browser as well. There are import/export apoc procedures which can be installed for GraphML etc.: http://neo4j-contrib.github.io/neo4j-apoc-procedures/3.5/export-import/graphml/

At least as part of the exposed API as this is our domain class to represent a set of RootElements.

Being able to access the HTML attributes of the target elements from the list of actions/action identfiers from getIncomingActionTransitions would be really helpful.

Possible attributes:
Identifying Attributes:
type
path
absolute outline

Attributes:
autofocus
contenteditable
disabled
download
draggable
height
hidden
href
multiple
name
onabort
onblur
oncanplay
onchange
onclick
oncontextmenu
ondbclick
ondrag
ondrop
onended
onerror
onfocus
oninvalid
onkeydown
onkeydowns
onkeypress
onkeyup
onmousedown
onmousemove
onmouseover
onmousewheel
onresize
onscroll
onsearch
onsubmit
readonly
size
sizes
tabindex
shown
target
type
value
width

Since we have all exploredActions and a different service/API to get ALL actions, we don't necessarily need this information here, too.

Actually, it is not an NFA because it is incomplete (missing complete transition table).
It's a set of action execution paths.
Maybe change the name or at least remove NFA from the description.
Can we keep the name state machine?
It's not a finite state machine because we do not know the number of states? The number of states should be finite?
It's a set of execution paths.

The trait should be implemented for GML, Java object streams, etc.

In the legacy code an MD5 checksum was used for the identifcation of states (not for the serialization).
We do not want to store the whole XML structure of states which would take too much space.
Instead we want to generate a unique checksum/hash as identifier.
We cannot reproduce the SutState or Action from this identifer but it does not matter for the monkey who only needs to know for its current SutState if it has already been there.
Additional properties for the states and actions could still be added in the future.

Instead of doing the following on the client side:

SutState beforeState = ...
Action action = ...
SutState afterState = ...
Set<Action> unexploredActions = ...
final State state = stateMachine.getState( beforeState, unexploredActions );
stateMachine.executeAction( state, action, afterState, unexploredActions );

Could we pass SutState directly?

SutState beforeState = ...
Action action = ...
SutState afterState = ...
Set<Action> unexploredActions = ...
stateMachine.executeAction( beforeState, action, afterState, unexploredActions );

Apparently the field Element.retestId is generated with random values and is not the same every time. We have to filter it when generating the hash value or make sure that Surili does always generate the same value.

recheck now contains all necessary dependencies from both retest-model and retest-sut-api, while I'm working on a new release of surili-commons (formerly surili-model). As soon as this is done, you should be able to go from this:

libraryDependencies += "de.retest" % "surili-model" % "0.1.0-SNAPSHOT" % "provided" withSources () withJavadoc ()
libraryDependencies += "de.retest" % "retest-sut-api" % "3.2.0" % "provided" withSources () withJavadoc ()

To this:

libraryDependencies += "de.retest" % "surili-commons" % "0.1.0" % "provided" withSources () withJavadoc ()

We need to use a specific location where the state machines are stored and loaded from.
This is also required for database backends. Otherwise, the user of the API does not know how to load persisted state machines from previous runs and extend them.
It will help Surili to get the existing state machines and to save them by specifying the appKey only.

Let the user of the API define a name and directly return the new state machine.
This allows us to remove the custom type IdMap and hence simplifies the code.

Before:
val stateMachine = GuiStateMachineApi().createStateMachine()

After:
val stateMachine = GuiStateMachineApi().createStateMachine("test")

The NFA can be represented by a directed graph but do we have any weights or do we need specific graph algorithms?

Visualization can also be done with graph file formats:
https://en.wikipedia.org/wiki/GraphML
Or with the graph database:
https://neo4j.com/developer/guide-data-visualization/

Some links:

• https://en.wikipedia.org/wiki/Graph_database
• https://en.wikipedia.org/wiki/Graph_database#List_of_graph_databases
• https://neo4j.com/docs/developer-manual/current/introduction/
• https://neo4j.com/developer/example-project/
• https://neo4j.com/developer/graph-algorithms/

Graph database use custom query languages such as https://en.wikipedia.org/wiki/Cypher_Query_Language. These are used for querying and updating the graphs.
Neo4j allows CQL transactions via REST: https://neo4j.com/docs/developer-manual/current/http-api/

For more complex strategies like ToStateWithUnexploredActions in the legacy code, we need to know which actions have not been explored in all states.
Note that we cannot really limit actions such as ChangeValueOfAction because we have to fill them with random values.
Hence, it would make more sense to store types of never explored actions together with the XPath of the target element.
Note that these never explored action types must be passed when getting a state like in the legacy API.

The package de.retest.guistatemachine.model contains model definitions based on the thesis by Furrer.
Currently, the REST service uses custom case classes.
Clarify, how the actual model looks like/should look like and adapt traits/classes in the model package.
Remove the case classes for the REST service afterwards.

The Furrer model is invalid. Specify the model by debugging the application.

RootElements from https://mvnrepository.com/artifact/de.retest/retest-model identify a state.
A Selenium API Action identifies an action/transition.

The current route unit test does not test a real running HTTP server?
This could be done with TravisCI using sbt run and concurrently curl queries.
Maybe Akka provides some testing framework.

It would be really useful to have a helper method in the GuiStateMachine that returns all actions that lead to one SutState, instead of only the getAllExploredActions method. Something like: getIncomingActions(SutState state).

If the following methods are not used anywhere, remove them:

trait GuiStateMachine {
  def getAllExploredActions: Set[Action]
  def getActionExecutionTime: Map[Action, Int]
}

They have only been added due to legacy code.

Then we can use a TrieMap or concurrent hash map for the states and remove synchronized.

We can also remove the execution counter for the action transitions and use a TrieMap here, too.

The legacy code provided the interface Action and the abstract class AbstractAction which reimplemented equals and hashCode by comparing ActionIdentifyingAttributes. This is important when the action is used as key type for maps or in sets etc.

As long as there is no abstract representation, the legacy API could be used:

<dependency>
			<groupId>de.retest</groupId>
			<artifactId>retest</artifactId>
			<version>3.0.1</version>
			<exclusions>
				<exclusion>
					<groupId>com.google.guava</groupId>
					<artifactId>guava</artifactId>
				</exclusion>
			</exclusions>
</dependency>

See de.retest.ui.actions.AbstractAction.

All state machines should be stored on the disk and restored from it in case the API/service crashes or the state machines become quite big and the creation/modification is interrupted.