systelab / allure-reporter Goto Github PK

Describe the bug
When it has loaded in the Allure Reporter a large number of .json files and the user select to upload the results to Jama, not all the results are uploaded, just a few ones.

To Reproduce
Steps to reproduce the behavior:

1. Have created in Jama a Test plan with test groups and a test cycle.
2. Load in Allure Report a large number of .json files (at least 150)
3. Upload the result to Jama selecting the correct Test plan, test cycle and test groups.
4. See error

Expected behavior
Only a few number of test cases are modified with the results of the executions.

Describe the bug
If we generate the report files with an action that has several steps-results, when one of them fails that causes that all these step-results are marked as fail being passed steps.

If the step that fails is in the middle (second level, in the given example), the following steps-results are not executed nor displayed in the report.

Expected behavior
Each one of the steps-results in the action should display their individual status (not all failed if that is not actually the case)

Screenshots

Description
With a set of Json files generated by the seed-jee tests and uploading them to generate the reports, the Allure Reporter doesn't do anything and remains in the same screen.

@systelab/angular-team

DevTools Console Error
Uncaught TypeError: Cannot read property 'length' of undefined
at AppComponent.push../src/app/app.component.ts.AppComponent.addTest (app.component.ts:103)
at FileReader.reader.onload (app.component.ts:67)
at ZoneDelegate.push../node_modules/zone.js/dist/zone.js.ZoneDelegate.invoke (zone.js:388)
at Zone.push../node_modules/zone.js/dist/zone.js.Zone.runGuarded (zone.js:151)
at FileReader. (zone.js:129)

The steps (@test, the first level) must be alphabetically ordered inside each Test Case (@tmslink)
It would be better to also order alphabetically the Test Cases inside a suite.

In order to facilitiate the troubleshooting and customization of non-java-generated json files it would be nice to have a mapping table where the map between allure-json model and allure-reporter internal model is cleary shown, including special assumptions like Action: prefix in step names (and what they do). That would help a lot when customizing the report to what is intended.

And, if a map from the allure-reporter to JAMA exists, that will be great!

This request is related to issue #46 and is referred to the management of the value of 'Executed in version'.

The value of the version where the automatic TC is executed is currently being reported using the Test Cycle name. Recently I faced projects not using it, so this mandatory information is not being set. So we met and decide a best way to report this value:

• JAMA will have a new field for Test Run item called 'Executed in Version'. This field will contain the list of Releases that each project have in JAMA.

• Allure Reporter UI will contain a new field [Radio-button selection] to support the setting of this value, deciding if Releases from Jama have to be used or using the already set 'New Cycle Name'(avoiding its repetition):
  "Executed in Version (Test Run):
  o Release: combo Releases (X.Y.Z b)
  o Use New Test Cycle name"

! Olga: this proposal will not work if I set in JAMA that values will come just from Releases list, if Cycle Name set is not exactly a value in the list**!**

Having this field is also a current need for Traceability matrix building.

In the .xml file generated from Protractor, if the spec has anidated steps like this example:

allure.createStep('Action: Click on the "Add Test" button', () => {
...
allure.createStep('The window dialog is present', () => {
...
})()
...
})()

then the generated report has duplicated rows for each of the inner Steps (not for the "Action:" one)

Describe the bug
The link in the main page to an example of how to write a Test Case is not working now, because it points to a non-existing file.

To Reproduce
Steps to reproduce the behavior:

1. Go to main screen, at the bottom.
2. Click on: See the example at "login.e2e-spec.ts" (here)
   the link leads to a not-found page.

Expected behavior
The page of the file that is supposed to be displayed:
https://github.com/systelab/seed-angular/blob/master/e2e/login/login.e2e-spec.ts

Describe the bug
In Angular, an it that no contains any allure.createStep is no documented in actions.

To Reproduce
An IT without createStep obtain this XML

<test-case start='1542892322024' status='passed' stop='1542892322612'> <name>Validate that clicking on a row opens Patient Maintenance</name> <title>Validate that clicking on a row opens Patient Maintenance</title> <labels> <label name='tms' value='TC0001_PatientManagement_e2e'/> <label name='feature' value='Purpose: This TC is intended to verify the CRUD of a Patient'/> <label name='appVersion' value='Version: 1.4. (build 2012)'/> <label name='tester' value='xcalvo'/> <label name='testExecutionDateTime' value='22/11/2018 14:11:26'/> <label name='browser' value='chrome'/> </labels> <parameters/> <steps/> <attachments> <attachment title='Screenshot' source='1dd79233-5e73-43f7-bc86-e741fbe486ca-attachment.png' type='image/png' size='42502'/> </attachments> </test-case>

In the seed-angular, the Allure libraries will be updated at systelab/seed-angular#51
Ensure Allure-reporter works as intended with the new versions.

Setup:

We have a Java class with JUnit tests, with the @feature tag set at class level, and one @TmsLink tag for each test method (since each method corresponds to an independent test case within the feature).

Steps:

1. Run the test class.
2. Collect the JSON files with allure results and show them in the allure-reporter available in GitHub.

Observed behavior:
The Allure Reporter displays each test method as an independent test case, each with the content of its @TmsLink as a header (which is correct), but for each of them, the content of the @feature tag is replicated.
See attached screenshot at the bottom, where each test method/test case is displayed separetely under a green checkbox with the @TmsLink content, but the @feature is replicated for all of them (marked with a red vertical line in the screenshot).

Desired behavior:
Since the @feature tag is a general description of the feature tested by the class, display it only once at the top, where the summary section for the classs is shown, and not repeated for each test method/test case.

Error
Using the Microsoft Edge browser, when you login to Jama in the allure report, it response that the user can not be found.

To Reproduce
Steps to reproduce the behavior:

1. Open Microsoft Edge
2. Go to https://systelab.github.io/allure-reporter/
3. Upload the json files
4. Click on login buttom
5. Enter your user and password
6. See error

Expected behavior
Login ok.

Screenshots

Desktop (please complete the following information):

• OS: Windows
• Browser: Microsoft Edge
• Version: 44.17763.1.0

When the user does the 'Update Test Case' option, in order to update the Description and the Steps to a Test Case in JAMA, other fields than the expected ones are affected being deleted.
In the case detected 'Test Design Status' and 'Last Tested Version' values are removed (see image attached). This is not expected at all.

This error blocks the use of this feature.

Steps to reproduce the behavior:

1. Go to Allure Reporter v.2.0
2. Drop a jason or xml with a TC result existing in JAMA, with values on all TC fields.
3. Log in JAMA and Click on 'Update Test Case'.
4. Go to JAMA to check that the TC has the uploaded Description and Steps and See that values in other fields have been deleted by error
   
   .
   Note: You probably will not see this error using the item 'Test Case (IL)', as it does not use the deleted fields mentioned here.

When trying to connect to jama and transfer the report information, the test group combo gets disabled and no selection can be done. If we click on Report an error message is displayed and no information is sent to jama. (tested in version 1.0)

It would be better to define an order in the test case (TmsLink) execution, and in the report.
Order: Epic (optional), alphabetic using TmsLink

When trying to print a generated report we obtain just a page printing with the view of the page in screen and scrollbars, this is not usable when the report takes a lot of pages. (tested in v1.0)

jsons created after execution can not content the new label 'actual results'.
This works for xml, and corresponding Angular code but not for java.

During verification I modified both json and xml manually, after being generated by the execution and I didn't realize that the automation code could not use this new label.

It is not a severe bug, because we can always use the UI text field, but that is applied to all TCs in the report and not just to a individual TC, as could be done if this info comes from the automation test code.

In behalf of @joaquimvila , it has been proposed to evaluate the possibility of having the test results uploaded to JAMA, without manual actions using Allure-Reporter UI. That is, using directly the JAMA REST-API.

I (Manel) am ok with the proposal (in fact the original update used in QL4 was in this way), but my conditions here would be:

1. The username registering the results can not be an impersonal user. It shall be a team member.
2. The CI tool shall upload results just labelled/formal versions, to avoid having in JAMA hundreds of useless results.
3. Current UI and features can always be used, being NOT replaced by the proposal.
4. The direct upload satisfies the required information to be filled in a formal Test Run

https://github.com/systelab/seed-angular/blob/allure-validation/e2e/allure/val.e2e-spec.ts
https://github.com/systelab/seed-jee/blob/allure-validation/src/test/java/com/systelab/seed/allure/AllureValidationTest.java

It is needed to re-define what information do we want to be uploaded with Test Runs to 'Actual Results' field in JAMA.

1. Review current info being uploaded.
2. Provide ways to include Location of Test Evidences (logs, allure report containing input&output data, db, screenshots, etc.)
3. Provide ways to include Test Environment used

This is for each Test Run

Currently, when we execute a test case and report to JAMA via allure, there is a field "Actual Result" which contains a table with the Java Function and the Test Description.

Different people has reported that this is not clear because is not the typicall test case format. The interestant thing should be store the test case executed. Options:

1. Store in the Actual Results the Test Description and the Expected Results (Easier)
2. Store in the Steps table of the Test Run AND the Test Case the Description and the Expected Results (Preferred).

NOTE: With the second option we should consider projects that are writing the steps manually because an automatic execution could modify the steps introduced manually

Is your feature request related to a problem? Please describe.
When there's a Failed step, the whole Test Case is reported as Failed without indicating the step that fails. It only happens with XML and works properly with JSON formats.

Scenario:
XML file --> TC_failed_xml_allure1.txt
Report in allure -->

Describe the solution you'd like
Display as "Failed" with a red cross the failed step. Given the failed information is only at step level, all the expected results rows will have the red cross.

Additional context
The version 2.0.0-beta.6 of allure-js-commons has been tested and it also happens.

The results detailed in the report, by step are not updated to JAMA at this same level. It is uploaded just the global Test Run status and then it appears an incoherence between Allure Report and JAMA item.
This is needed and evident just when the test steps are synchronized, when they are not I don't think that we could improve current behaviour.

The solution is to also update each test step result using the REST API.

When some long words are included, for example, in the column "Expected result" (it may happen ih the rest of the columns too), the app doesn't do the wrap and the remaining text are shown all over the next rightest columns. Please see attached files and the proposed solution that was tested in Edge v42.17134, Chrome v72.0, Firefox v65.0 and Opera v58.0.

When trying to connect to Jama for transfer the report, if the configured user is not found in Jama, or the Jama server is down, the 'Report to Jama' dialog shouldn't be displayed, as no transfer can be made wihtout correct Jama user nor Jama server down.

(This issue is 'cloned' from #53 , but in this case refered just to the action 'Update Test Case')

When users upload test steps to JAMA they needed to receive feedback about:

Upload is in progress
Upload has finished (all Test Cases have been uploaded)
How many Test Cases (optionally how many Test steps) have been uploaded and how many were not (if possible provide information about the failure cause: TC not found / matched; communication failure; Wrong content....)

• Each Test Case should contain a table with the columns Step, Action, Expected Result and P/F.
• The Step number column should start at 1 for each Test Case
• The Step number should be increased for each Action/Step (it does not apply to sub-steps)

Dropping files to upload could be slow when there are many. To know if all files have been processed and open the Jama report dialog, you have to see the tables and wait until the numbers don't change anymore.

It could be better to have a progress bar to know how many files have been processed and how many are missing.

Allure Reporter test deployment on:
https://systelab.github.io/allure-reporter-test/

is labelled as 2.0.0 instead of 3.0.0

I've tried to transfer a file from a test in Angular environment, the creation of a new cycle worked well, but the information about the Test Case wasn't updated at all.

Here is the file I used to generate the report:
SampleBank-testsuite.zip

When users upload results to JAMA (using 'Report' button) they needed to receive feedback about:

• Upload is in progress
• Upload has finished (all Test Runs have been uploaded)
• How many Test Runs have been uploaded and how many were not (if possible provide information about the failure cause: TC not found / matched; communication failure; Wrong content....)

Is your feature request related to a problem? Please describe.
Technical update on this tool
Update Angular from 7 to 11

Describe the solution you'd like
All the features must work after the update.

Only should be enabled when the required fields are informed: Project, Test Plan, Test Cycle, Test Group, New Test Cycle name (only if Test Cycle is new)
For the new "Update TC" button only is needed to inform the field Project (it should be enabled with only the field project informed)

The table that appear on the UI documentation is updated but there is a pending link to update:
https://github.com/systelab/seed-angular/blob/master/e2e/login/login.e2e-spec.ts
should be
https://github.com/systelab/seed-angular/blob/master/e2e/login/tests/login.e2e-spec.ts

(Attention: This item is very close related to #51 )

As explained in #51 , the upload of results shall be improved with the version where the test have been executed; but it refers just to Test Run item. This same value should be set to the field 'Last Tested Version' of the Test Case corresponding to each Test Run where a result is being set.
The values to be set are a closed enumerated list already defined in JAMA for each project. P.ex.: see this example:

So the implementation of this request must take into account the implementation of #51 because each time a result is uploaded to a Test Run, using a release value, this same value must be used to set it on the according Test Case.

BUT THERE IS AN EXCEPTION FOR THE PREVIOUS GENERAL RULE: when I execute test runs for a HF, patch, SP or Cybersecurity patch Validation OVER an OLDER version (different from the current in development) I DO NOT WANT TCs to be updated with the label of the old version.

So the implementation of this request shall allow to optionally set release to TCs when setting releases for Test Runs, using the most adequate UI controls.

As a wish, the feedback informing about the upload of results, could also mention the Test Cases modification, in some manner)

Allure 2 with JUnit 5 does not include in the generated JSON the attributes @tmslink and @feature
1f01c96c-fa31-4358-bf10-38a5eb71746b-result.json.txt

It works with JUnit 4.
,
{
"name": "feature",
"value": "Goal: This test case..."
}],
"links": [{
"name": "MLG_TC1086_LAB_QC_StatisticalCalculations",
"type": "tms"
}],

TmsLink and feature are mandatory tags, so the execution is not displayed in allure-reporter.
Find out a solution. Proposal:

• Remove the restriction to display an execution only when there's a tmslink
• Use the name or the description for the test case name

Description
After setting the configuration of JAMA instance different from the default one, the application ignores it an keeps using the default one.

To Reproduce
Steps to reproduce the behavior:

1. Open the configuration dialog and set the credentials of JAMA server different from the default one (i.e https://snowjamaserver.systelab.net/rest/latest)
2. Save the new configuration
3. Open the "Upload to JAMA" dialog.
4. All the dialog fields are shown as empty
5. Open the browser console and check that there is an error regarding the default URL address

Expected behavior
The JAMA configuration set on step 1 should be considered and used to fill the fields of the "Upload to JAMA" dialog.

Desktop:

• OS: Windows 10
• Browser: Chrome
• Version 79.0.3945.117

Screenshots

There are some labels present on the .xml file that may appear on the top (or bottom) of the report.

As they appear on the .xml file:

label name='appBeingTested' value='laboratory'
abel name='appName' value='Modulab'
label name='appDescription' value='Laboratorio'
label name='appVersion' value='Versión: 3.1.03 (build 1)'
label name='testVersion' value='0.8.0'
label name='testCreatedBy' value='[email protected]'
label name='testLastModifiedBy' value='[email protected]'
label name='testStartedAt' value='OCT/24/2018 13:28:53'
label name='company' value='company_name'
label name='browserName' value='chrome'
label name='browserVersion' value='70.0.3538.67'
label name='platform' value='Windows NT'

Developed solution:

• All the labels are displayed except from 'tms', 'feature', 'suite' and 'package'. The intended use of this feature is to include the tester, execution date, build and browsers.

For the rows at the beginning of the report that contains the total for "Test Summary by Test Suite", "... Severity" & "... Tag":
- Currently: The columns Pass, Fail & Other for each of this rows only shows the percentage.

 - Desired: Will be useful to show the quantity that generates that percentage too.

Is your feature request related to a problem? Please describe.
When an step is disabled using xit, the Test Case is reported with the status 'pending' in the xml. Allure-reporter displays this Test Case as Failed when it's not failed.
<test-case start='1543092080213' status='pending' stop='1543092080215'>
Describe the solution you'd like
Allure Reporter must omit the 'pending' status. The disabled Test Cases must not fail the whole Test Case, so the Test Case status must be "passed".