soosterwaal / operias Goto Github PK

Next up is to exclude files which are completely unchanged

Next up, it the construction of the actual xml file containing the information about the coverage

After an error, we stil have to clean up the temp directories and empty destionation directories

It seems some of the coverage shown by SERG-Delft/jpacman-framework#21 (comment) is not correct.

In the MapParser.html, the makeGhostSquare method is covered 'half':

where as in the plain cobertura file (which I manually created with mvn cobertura:cobertura on top of the pull request) it is fully covered:

It looks like the Operias coverage is a bit messed up here.

Initially, operias will need 2 arguments. The source directory in which your new source code is in, and the repository directory, which is where the original source code is in.

The git arguments should be revised to:

The rest of the arguments will stay the same. If a branch name was given, that no specific commit is needed. If no branch name was given, it will use the master branch as default.

Operias should generate a XML report containing some specific metrics, like:

• How many lines were deleted, and what % was covered
• How many lines were added, and what % of those lines are covered
• How many lines were changed and what is the change of coverage for those lines
• How many test lines were changed

Several arguments should be able to be passed to operias through the commit line:

-d (--destination-folder), The destination folder where the generated html will go to
-r (--revised-folder), The folder with the revised source which needs to be compared to the original code
-o (--original-folder), The original source folder to which the code will be compared to
-c (--commit-id), The commit id to which the revised code will be compared to
-v (--verbose), Print out the status of operias

Either -o or -c should be provided

The next is an important step, file should be marked as changed if it is either changed in source file of changed in coverage, then, this information of both diff reports must be combined and stored

If a pull request was made and the other branch was some commits behind, it will show differences in de code/coverage from the changed files in the commits that were not on the branch.

So, in order to work with Operias correctly for your pull request, you should update the branch before creating the pull request

Some of my tests require a UI. For that reason, I've configured travis as follows:

language: java
before_install:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"
script: "mvn clean test"

I suspect that Operias doesn't do this, which is why some of my tests fail.

Would something like that be possible for operias as well?

Or: Should operias use the .travis.yml if it is there?

If a class is embedded within another class, cobertura will make 2 "classes" for this file the original one, and for every embedded class it adds $1, $2, $3 etc after the class name. This is not handled properly yet in operias and the combined view can be a bit tricky because you don't know what part of a file is the embedded class when you show the source differences

The percentages next to the bars are rounded, this should be rounded to 2 decimals instead of 0

If you have a subsubpackage, it will be shown in the subpackage, and in the main package

The file diff report contains the state of the source code for every file or directory, changed, new, deleted or same. The next step is to add another state to this, the state of the coverage, whether it is changed or stayed the same

In the overview, in the bars, even a slightest change in covered should be forced to have a least 1 pixel.

Often you see no changes in colors, but a small change did happen. If you force it to be at least 1 pixel, you can always see that the coverage or source was changed

The usage instructions should make it clear which "kinds of directories" are expected in the parameters:

--revised-directory -rd This directory contains the revised source code, this directory will be compared to the original directory
--original-directory -od This directory contains the original source code, unchanged.

Whether if should be the project dir or the source dir for instance.

When opening a file, you should see 4 different views:

• The original file, with the original coverage
• The revised file, with the revised coverage
• The source difference
• The merged view

In SERG-Delft/jpacman-framework#23 (comment)

The line coverage increased from 38.03% to 38.79%
The line coverage stayed the same at 30.82%

Seems confusing to me: Did it increase or stay the same?

As you can see in the image below. The coverage of the package was decreased to 0%. This is because the package was deleted. It should be clear that the package indeed was deleted. You can see it because 100% of the lines were removed. But if we look at the class, it said +15 lines (-100%) which is obviously wrong.

Also when opening a deleted class, the rows are not shaded.

And, same as the issue for the light/dark green bar, sometimes the light/dark red combination breaks

The overview page should be changed to a similar tree view as Eclemma provides. You start out with a list of changed packages, and you can open this package to view the sub packages and classes within the same table.

Also every package and class should get a bar containing the a visual representation of the changes in coverage

The next step is to execute the test suite of the project to start measuring the coverage.

This should be done for both ant project and maven projects.

The next step is to implement the instrumentation phase of cobertura. Instrumenting can either be done using the command line of using an ant build.xml script, if the project in an ant project

File names were changed in some commits ago, but the links in the html were not, so 404 pages will appear

The final step is to visualize this combined diff report into a html site which is easy to navigate and can clearly show you which lines has a different coverage or was changed

Same as for the classes

Set up the parameters for cobertura and execute mvn cobertura:cobertura on the correct project folders

The initial set up contains the initial project and the set up of the arguments for the tool and the set up of the test suite (if necessary).

• Create the initial (maven) project
• Determine which arguments must be used for the working of the tool
• Set up coveralls and travis-ci so see what how coveralls is doing it's work

If we look at the image below, we can see that the package org.junit has a decreased coverage of 1,04%. But if we look at the files in the package, we don't see any changes. That is because the file AssumptionViolatedException was added, but not fully covered. We can see that it is new, only by the increase of relevant lines by 100%. But this could also mean that the source was doubled. So there must be a clear indiciation that the was indeed new, and that this is the cause of the coverage drop.

Add a module to the project, which can be used to run operias as a github webhook on your own server

Next is to determine which files were added/deleting/editted, because we want to combine this information with the information retrieved from cobertura.

In some cases coverage may be missing because tests fail -- which may be due to the different environment in which operias runs the tests.

Therefore, in the Operias report have a separate page with info on pass/fail per test would be useful.

The output of information should be disabled by default. Now i will be turned of when the verbose argument is not given. It should be the other way around. Turn it ON when the verbose argument is given

The next step is to determine how cobertura will be executed. Initially, the tool will support maven projects only. Operias should execute the cobertura goal on maven and retrieve the report and parse this

As stated in the title, set up the project and work with coveralls

In the sample below, you can see a class which has 0% change in coverage and 0% change in source.

In this case, the amount of relevant lines stayed the same, but some lines were just replaced

A next step would be the implementation of the pulling of a repository and a specific commit or pull request and use this as the original folder

The next step is select the correct branch of the target directory. For this you need to execute git command line commands to checkout the correct branch.

• Three arguments needed at least, source directory, git directory, branch name
• Select the correct branch locally

The xml reports must be parsed and checked for any difference between them. The files that differ in coverage should be marker and remembered for a next phase

When passing wrong or insufficient arguments to operias.Main class, no error message is presented. It would be useful to present the list of all possible arguments and its description, or potentially what may be missing.

In the final site, deleted files are not shown because they are not yet parsed and marked as changed files

Line coverage diff for ImageSprite in attached figure has an extra line break.

Next up is to retrieve the difference in source between both directories. This can be done with the diff tool for example.

Operias now parses every types of file, it should only parse java files