allure-framework / allure1

Allure 1 isn't supported any more, please consider using Allure 2 https://github.com/allure-framework/allure2 instead

[DEPRICATED]

Please follow https://github.com/allure-framework/allure2

Allure 1

A flexible, lightweight multi-language test report tool, with the possibility of adding to the report of additional information such as screenshots, logs and so on.

Who uses Allure

Initially Allure Framework was widely used in internal testing of Yandex software products. Three years later thousands of software testers all over the world are giving a high note to their experience with Allure.

Getting Started

Please consider using Allure 2 https://github.com/allure-framework/allure2 instead!

Contact us

• Mailing list: [email protected]
• Gitter chat room: https://gitter.im/allure-framework/allure-core
• StackOverflow tag: Allure

allure1's Issues

Hot keys to navigate between cases in Testsuite

For parameterized tests it is convenient to navigate in testsuite with Up and Down keys. Choice of Up and Down is an option, other hot keys are acceptable.

Features and stories support

add feature.json

Junit. Incorrect @step value processing

If we use Russian symbols, we get some problem. For example, for step

    @Step("Ищем: query = {0}")
    public void searchFor(String query) {
        onTestPage().searchFor(query);
    }

we get xml report:

and allure report:

Сделать archetype для testNG

Model Improvements Proposition

ref: allure.xsd

• I.
  skipped status is not exhaustive, there is two huge difference:
  test - canceled, due to some previous fail, or environment settings (there is a kind of failure message)
  test - pending, test is not yet implemented
  I met several frameworks, that use this concepts intensively.
• II.
  severity is very similar to labels, except that it is test-case only field. imho, it may be removed from model, and labels used instead. when distribution graph is been built, it make sense to use labels in general.
• III.
  I don't see much difference between parameters and attachment, maybe it may be unified to special attachment 'key-value-storage'. "other" is not the case, because key-value-storage attachment may be directly used in reports for dimensions.
  Note: First I thought they might be labels, but than understood that this is runtime, and should be closer to attachments.
• IV
  Test allerts/warnings. What cases it should cover:
• PerformanceTests/MicroBenchmarkTest have thresholds. Critical threshold should fail test, but if threshold is high it should be kind of warning.
• Some frameworks, are able to re-execute tests in case of failure. Just to have less blinking, but if I want to set thresholds "Ok, if pass in three times, fail after 10th try"... so between 3-7 this is test alert.
• ScalaTest, have "PendingUntilFixed", test that cannot pass due to some not yet implemented dependencies. The usage of such thing is an alert... This is not skipped due to static rule.
• Workarounds (HotFix on Test)... generally we assume that everything is fine, but if something went wrong, we implemented code, that will save tested functionality, but a such code cannot leave in test code forever.
• TestRequires human eye, for final evaluation. In my exp, when tests took the steps, and verifications is too complex. In other words steps are written, asserts not yet.
• V
  Some parametrised tests, or even tests with loops may be interested to dig in numbers.
  In results it maybe kind-a attachment rendered as chart. I assume, that html attachment may resolve this, but this means that each framework if it has numbers, should implement html-rendering. More convenient if adopters send data object, with marker "chart of type Pie", "chart of type stack", etc. Google-charts-api is nice candidate for 'json attachment" to "chart-html attachment" transformer.
  ... rereading, probably this maybe achieved if labels will be treated specially with json attachments.
• VI
  Tagging. Labels now requires value, it may be convenient just to make it optional. For reports this means no need to provide distribution of such dimension.

Tooltip above the chart doesn't fit to screen

When tooltip is close to the edge of the screen it can not fit. Need to take this into account when calculating its position.

Before & after

• Show error if test failed in before/after class

Invalid xml characters (0x0 for ex.) in *-testsute.xml break report

Documentation – Build instructions

As a C# Dude
I want a short build instructions
So that I’ll be able to build the latest allure-framework on my machine and do some hacking

Ну, пожалуйста!

Ability to log steps in console (when developing in IDE, or to see maven log)

It will be nice to build log some kind of:

-> START STEP: SettingsPageSteps.clicksOnFolder(folder)
<- DONE STEP: SettingsPageSteps.clicksOnFolder
-> START STEP: DefaultSteps.shouldBeEnabled([Кнопка «Новая папка», Кнопка для папки «Создать правило»])
<- DONE STEP: DefaultSteps.shouldBeEnabled

With all args and name of method (and class where is it). Ability to turn on/off logging in view as in allure-report (with customization of step-names) also would be great.

Add the ability to change suite name in runtime

Results directory creation problem

Unsynchronized method createResultsDirectory() can throw AllureException when running tests in parallel

JUnit: miss of description in test suites

Build report with only broken and failed test-cases

Sometimes we need to launch a huge amount of test-cases at once (for example 40k test-cases) and it is almost useless to generate report with passed tests because nobody will look at them. Plus report becomes very heavy: it takes about 3 min to finish all tests and 10 min to generate a report which becomes so heavy that lags occur while browsing it

It would be nice to create a property like "allure.only.defects=true/false" and if it is true then build report only with failed and broken test-cases.

TestCase uuid generation bug

Генерируются одинаковые уиды у тескейсов, из-за этого артифакты отображения

Print varargs in steps not as array, but as list of strings

When use

@Step("Step: {0}")
public void clickOn(HtmlElement... elements) {
...
}

Expects: Step: ["Element1", "Element2"]
But: Step: [Lru.yandex.qatools.htmlelements.element.HtmlElement;@42ed46a7 occurs.

It will be nice to print varags as list of String

Dont forget about primitive types (int... ints)

Use "Helvetica Neue",Helvetica,Arial,sans-serif to all text

Now all text use arial,sans-serif, and it ugly.
It will be nice to use "Helvetica Neue",Helvetica,Arial,sans-serif everywhere, because of it pretty! :)

"Helvetica Neue",Helvetica,Arial,sans-serif

arial,sans-serif

Еще мне сложно придумать как написать это по английски, но простой arial не такой сглаженный и очень угловатый. При большом количестве буков это утомляет

Grouping of typed tests

For JUnit:
Parametrized, Suites, Categories

JSON files refactoring

Now we use two big JSONs. Need to split it to more smaller files. For each tab should match at least one special JSON.

• xUnit.json - based on testsuite.json. Info about testsuite and short info about testcase
• testcases.json - array of short info about all testcases (one for graph and timeline tab)
• {uid}-testcase.json - full info about single testcase

Bug in getTitle() method

Когда в степ передается null:

@Test
public void test() throws Exception {
    step(null);
}

@Step("{0}")
public static void step(String array) {
}

возникает следующая ошибка

java.util.NoSuchElementException: null
    at java.util.LinkedList.getLast(LinkedList.java:255)
    at ru.yandex.qatools.allure.storages.StepStorage.getLast(StepStorage.java:21)
    at ru.yandex.qatools.allure.Allure.fire(Allure.java:43)
    at ru.yandex.qatools.allure.aspects.AllureStepsAspects.stepFailed(AllureStepsAspects.java:50)
    at my.company.MyTest.step(MyTest.java:70)
    at my.company.MyTest.test(MyTest.java:56)

Test mechanism  Time elapsed: 0.19 sec  <<< ERROR!
java.lang.NullPointerException: null
    at ru.yandex.qatools.allure.Allure.fire(Allure.java:65)
    at ru.yandex.qatools.allure.junit.AllureRunListener.testFinished(AllureRunListener.java:39)
    at org.junit.runner.notification.RunNotifier$7.notifyListener(RunNotifier.java:191)
    at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:61)
    at org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:188)
    at org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:39)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:277)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

Переход от thucydides на Allure

Fix Sonar Violations

test

Add @Documented to all annotations

For view titles and descriptions in javadocs for tests, annotations should have java.lang.annotation.Documented annotation

Не показываются failure из skip-тых тестов

Суть:
Есть тест со status='skipped' со вложенным failure:

<?xml version="1.0" encoding="utf-8"?>
<ns0:test-suite xmlns:ns0="urn:model.allure.qatools.yandex.ru" start="1392388427993" stop="1392388428016">
  <description>
This module contains basic examples of tests
  </description>
  <name>demo.test_basic_examples</name>
  <test-cases>
    <test-case severity="normal" start="1392388428000" status="skipped" stop="1392388428000">
      <name>test_skip</name>
      <description>this test is skipped</description>
      <failure>
        <message>skipped</message>
        <stack-trace>Skipped: for a reason!</stack-trace>
      </failure>
      <attachments/>
      <steps/>
    </test-case>
  </test-cases>
</ns0:test-suite>

Но причина skip'а не показывается: вместо нее стандарная плашка This test has been skipped:

Ожидается, что Failure так же будет показываться с причиной, которая берется из stack-trace -- или как-то еще.

Missed steps and attachments in junit tests with timeout

Проблема в том, что у нас сейчас степы хранятся в thread local storage. А JUnit запускает тесты с таймаутом в отдельном треде.

Fix issues after the code-review

Fix issues in webdriver tests found in #150

• Assign names for all elements
• Use hashShouldBe for assertion
• Move not inside matcher
• Do additional assert before click

Thanks @lanwen

Open retry rule as allure junit extension

use this repo https://github.com/allure-framework/allure-junit-extensions

Failed to execute goal allure when using along with the Spring and Camel

I've prepared the demo to reproduce: https://github.com/smecsia/allure-camel-spring-test

[ERROR] Failed to execute goal ru.yandex.qatools.allure:allure-junit-plugin:1.2.2:allure (default) on project allure-spring-test: Unable to execute mojo: Compiler errors:
[ERROR] error at (no source information available)
[ERROR] /tmp/allure-spring-test/src/test/java/test/SimpleCamelSpringTest.java:0::0 The type org.springframework.test.context.junit4.SpringJUnit4ClassRunner cannot be resolved. It is indirectly referenced from required .class files

Add allure-config module

This module must contain global allure config, for example name conventions

JUnit. Failure in before/after class not displayed

Move exception to step

If exception was thrown in step, it should be displayed inside the step.

Logging.

Write log to allure-report.log file to allure-results folder

• Add logging to allure-commons
• Add logging to allure-junit
• Add logging to allure-report-data
• Add logging to allure-report-plugin

Show loader while changing route

We need to inform user that some resources are loading and he will have to wait.
This module will be suitable http://chieffancypants.github.io/angular-loading-bar/. @allure-framework, do you like?

Replace skipped status by canceled & pending

• canceled, due to some previous fail, or environment settings (there is a kind of failure message)
• pending, test is not yet implemented
• Replace skipped status in allure-model by canceled and pending
• Refactor status logic in allure-junit
• Refactor status logic in allure-testng
• Refactor status logic in allure-commons
• Implement new statuses in report-face

Don't show num of tests if not enought space to do it

JUnit. Show ignored tests

Ability to ignore only current step, not whole test

Sometimes it needs to do smth like this:

@Step
public void step() {
  if(...) {
     //do 
  }
  // else - nothing
}

It will be useful to throw some kind of exception, which ignores only this step, but dont fail test.

For example:

@Step
public void step() {
  dontIf(object, matcher);
  // do smth
}

Show-stacktrace button bug

Add ScalaTest Listeners-Adaptor

It seems need to implement Reporter with specific to
Allure functionality, taking in account that ScalaTest may have nested suites

I hope 'ru.yandex.qatools.allure.annotations' will work as expected without any actions.

This should allow to start using it with sbt by passing reporter param to Runner:
something like

val project = project(".")
    .libraryDependencies(
"ru.yandex.qatools.allure" % "allure-scalatest-adaptor" % Versions.allure
)
#and for scopes...
testOptions in scope <+= Tests.Argument(TestFrameworks.ScalaTest, "-C", "ru.yandex.qatools.allure.scalatest.AllureTestReporter")

and then, incrementally,
new sbt-plugin might add this dependencies for all test, providing step that executed after test, testQuick, testOnly sbt command to generate report.

@bvenners, @cheeseng, you might be interested in this. If you have ideas how to do it better, please share your thought.

Inspired By: Selenium Camp 2014

Go to test by click it on timeline

What the error is it?

Now it needs to remember test name and search it manually

Fix report bugs when test using junit retry rule

JUnit. Add support of ErrorCollector rule

Add the ability to change parametrized test name correctly

add variable {parametrizedName} to Title annotation

Show when steps executed in before, after, beforeClass, afterClass in log

It will be nice to see, if step not executed in TEST method, but in before or smth else

Как запустить построение отчета локально для Python

Allure step listener

Add ability to create custom step listener. It should have before & after method for each of:

• step started
• step failed
• step finished
• attach started
• attach finished

More funny and greeny allure report if all tests are passed

Encoding problem with NONASCII

http://teamcity.qatools.ru/repository/download/allure_core_pull_request_build/534:id/index.html#/home/832e0603-1a9d-47d3-a5a1-aaf9f79e3fac/0a9e2b3b-2075-4d46-b94c-26220cdd954b

При отсутствии тайтла пишем пустую строку в степе

Вот так отображается степ без тайтла