Carina automation BDD framework (TestNG + Cucumber)
Home Page: https://zebrunner.github.io/carina/cucumber/
Carina-cucumber is independent Carina module that supports Behaviour-Driven Development (BDD)
We offer to use our configured Java code formatter for Eclipse. To use the same formatter in IntelliJ IDEA, you should install and configure Eclipse Code Formatter.
Code - Apache Software License v2.0
Documentation and Site - Creative Commons Attribution 4.0 International License
to reuse new log4j2x 2.17.1 we have to jump onto the 7.3.16 carina-core
Enhancement: On Test failure with Cucumber tests, we get failed scenario number in the text file
There should be a provision to save this failed text file at the last run results.
Please find the attached runner file
TestRunner.txt
In this file please look at line 15: "rerun:reports/qa/failed.txt"
So suppose, we have 10 scenarios for @administration tag and 6 scenarios out of 10 are passed then
failed text file show below text
src/test/resources/features/ZAdministration.feature:12,15,18,21
this gives us start line numbers of failed scenarios.
When we do the second run at that time this file will get overwritten.
Hence can we implement to generate this file (failed.txt) in our run results?
D:/Project/reports/1613739999290/
currently, it is getting generated at D:/Projects/reports/qa/failed.txt
If tests are executed in parallel with multiple runners (single feature->runner mapping) then only last run results will be shown.
the workaround did here to get this resolve
Runner 1:
plugin = {"pretty"2
"html:target/cucumber-core-test-report",
"pretty:target/cucumber-core-test-report.txt",
"json:target/cucumber-core-test-report.json",
"junit:target/cucumber-core-test-report.xml"}
Runner 2:
plugin = {"pretty",
"html:target/cucumber-core-test-report1",
"pretty:target/cucumber-core-test-report1.txt",
"json:target/cucumber-core-test-report1.json",
"junit:target/cucumber-core-test-report1.xml"}
Runner 3:
plugin = {"pretty",
"html:target/cucumber-core-test-report2",
"pretty:target/cucumber-core-test-report2.txt",
"json:target/cucumber-core-test-report2.json",
"junit:target/cucumber-core-test-report2.xml"}
Actual Result: All runners result should be displayed
Expected result: Only last run results are getting displayed.
Result with a workaround: All runners results are getting displayed, but some runner results are shown in duplicate.
as of now carina-demo and regular carina archetype already has example of the cucumber implementation. As result too common inside the classpath we have a conflicts because carina-cucumber is updated not so often. let's generate independent carina-cucumber archetype and sample project where we could track only cucumber based tests, dependencies etc
In 7.3.19 in the reports folder the name of the folder for each scenario equals the scenario name. From 7.4.21 version the name of the folder consists from udid
Add opportunity to rename in advance before starting test.
Also add log info about name of the renaming
Suppose you have set the below properties in the config file
thread_count=1
data_provider_thread_count=3
feature file as given
@featuretest
Feature: Test 1234
Scenario Outline: Scenario 1
Given Verify1
Examples:
| 1 |
| 2 |
| 3 | > SKIPPED
Scenario Outline: Scenario 2
Given Verify2
Examples:
| 1 |
| 2 |
| 3 | > SKIPPED
Scenario Outline: Scenario 3
Given Verify3
Examples:
| 1 |
| 2 |
| 3 | > SKIPPED
It skips the last row of example each please refer to the screenshot
subj
subj
java.util.ConcurrentModificationException on io.cucumber.testng.TestNGCucumberRunner.finish when running huge suite in parallel.
related tickets to review:
cucumber/cucumber-jvm#591
testng-team/testng#1168
https://stackoverflow.com/questions/40296350/testng-concurrentmodificationexception-while-running-with-test-invocation-count
https://groups.google.com/g/testng-users/c/don61IVYFIQ
Hi Team,
For the below versions, cucumber reports are not getting generated.
POM.xml
<carina-core_version>7.0.0-beta2-SNAPSHOT</carina-core_version>
<carina-cucumber_version>1.2-beta2-SNAPSHOT</carina-cucumber_version>
_config.properties:
project_report_directory=reports/qa
report_url=C:/Automation/QA_Automation/reports/qa
Issue #1:MAJOR
In a report (file:///C:/Automation/QA_Automation/reports/qa/emailable-report.html) when you click on Open Cucumber Report in new Window
It navigates to file:///C:/Automation/QA_Automation/reports/qa/1608016465879/cucumber-reports/cucumber-html-reports/overview-features.html
but it shows error logs:(attaching screenshot1.jpg for reference)
Issue #2:MINOR/COSMETIC
Run results show extra characters like 'Optional'. (attaching screenshot2.jpg for reference)
Current test name generation contains pseudo name like EX001. This number should contains more useful information like number of scenario line in the feature file
Earlier shown Feature and Scenario Name Value is not getting displayed.
An issue occurred in Release: 7.4.24
File name: src/main/java/com/qaprosoft/carina/core/foundation/cucumber/CucumberRunner.java
Line no 92 : ReportContext.setCustomTestDirName(testName);
The above line should be uncommented.
I have API with 73 data combinations which I placed under cucumber examples.
When I am running these tests in parallel with configuration as
thread-count=5
data_provider_thread_count=10
Tests are completed in 5 minutes whereas in status it shows 35 minutes.. which gives the wrong impression.. even though we are finishing it quickly it shows wrong numbers.
Subj
Today I had done one test with one feature file containing multiple scenarios and each scenarios have multiple examples
Just to show you
Abc.feature
Feature : f desc
Scenario outline: sc1 desc
GIVEN WHEN THEN
example
Row 1
Row 2
Scenario outline: sc2 desc
GIVEN WHEN THEN
example
Row 1
Row 2
Row 3
Scenario outline: sc3 desc
GIVEN WHEN THEN
example
Row 1
Row 2
Row 3
Row 4
Zebrunner output shows
f desc ( sc1 desc ) --> reset [L001]
f desc ( sc1 desc ) [L001]
f desc ( sc2 desc ) [L002]--> reset [L001]
f desc ( sc2 desc ) [L003]
f desc ( sc2 desc ) [L004]
f desc ( sc3 desc ) [L005]--> reset [L001]
f desc ( sc3 desc ) [L006]
f desc ( sc3 desc ) [L007]
f desc ( sc3 desc ) [L008]
Expected output :
subj
With the newest zebrunner agent that's possible to attach external links to entire test run.
Need to use this functionality for cucumber report
Suppose there is below feature file
ABC.feature
@tagFeature
feature:
@tagScenario
scenario outline:
Given statement
When statement
Then statement
Examples:
|Column|
|example1| --> Browser session started for this
|example2| --> Existing Browser session used here, instead of that new browser session should start as it nothing but new test case with the change in test data.
it should be carina-cucumber-archetype. fix should be done in pipeline code
[INFO] Uploading to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/7.4.21/carina-archetype-7.4.21.jar
[INFO] Uploaded to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/7.4.21/carina-archetype-7.4.21.jar (98 kB at 13 kB/s)
[INFO] Uploading to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/7.4.21/carina-archetype-7.4.21.pom
[INFO] Uploaded to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/7.4.21/carina-archetype-7.4.21.pom (2.6 kB at 5.2 kB/s)
[INFO] Uploading to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/7.4.21/carina-archetype-7.4.21.jar.asc
[INFO] Uploaded to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/7.4.21/carina-archetype-7.4.21.jar.asc (488 B at 2.9 kB/s)
[INFO] Uploading to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/7.4.21/carina-archetype-7.4.21.pom.asc
[INFO] Uploaded to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/7.4.21/carina-archetype-7.4.21.pom.asc (488 B at 3.0 kB/s)
[INFO] Downloading from QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/maven-metadata.xml
[INFO] Uploading to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/maven-metadata.xml
[INFO] Uploaded to QPS_Nexus: https://oss.sonatype.org/service/local/staging/deploy/maven2/com/qaprosoft/carina-archetype/maven-metadata.xml (311 B at 416 B/s)
How to run multiple features file in parallel without creating multiple runner files.
subj
let's try to switch again into the RELEASE version of the carina by default.
We don't plan any serious refactoring in carina for cucumber part so it is not so risky as earlier.
Using latest RELEASE version should allow to jump onto the latest carina and zebrunner reporting agents asap without release cycle
Hi,
When I try to run the script it gives me below error.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by cucumber.deps.com.thoughtworks.xstream.core.util.Fields (file:/C:/Users/prasad_shimpi/.m2/repository/info/cukes/cucumber-jvm-deps/1.0.5/cucumber-jvm-deps-1.0.5.jar) to field java.util.TreeMap.comparator
WARNING: Please consider reporting this to the maintainers of cucumber.deps.com.thoughtworks.xstream.core.util.Fields
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Showing
import com.qaprosoft.carina.core.foundation.utils.R;
It should be
import com.zebrunner.carina.utils.R;
Cucumber-java should be upgraded to the latest 7.5.0 which has annotations like @BeforeAll and @afterall
On Carina-Cucumber v8.0.0, method 'super.tearDownClass(context)' is throwing an error shown below. This is called in the Cucumber Test Runner class which extends Carina's 'CucumberRunner' class.
java.lang.RuntimeException: java.io.IOException: Stream closed
at io.cucumber.core.plugin.JsonFormatter.finishReport(JsonFormatter.java:150)
at io.cucumber.core.eventbus.AbstractEventPublisher.send(AbstractEventPublisher.java:51)
at io.cucumber.core.eventbus.AbstractEventPublisher.sendAll(AbstractEventPublisher.java:36)
at io.cucumber.core.plugin.CanonicalOrderEventPublisher.handle(CanonicalOrderEventPublisher.java:18)
at io.cucumber.core.eventbus.AbstractEventPublisher.send(AbstractEventPublisher.java:44)
at io.cucumber.core.eventbus.AbstractEventBus.send(AbstractEventBus.java:12)
at io.cucumber.core.runtime.SynchronizedEventBus.send(SynchronizedEventBus.java:47)
at io.cucumber.core.runtime.CucumberExecutionContext.emitTestRunFinished(CucumberExecutionContext.java:111)
at io.cucumber.core.runtime.CucumberExecutionContext.finishTestRun(CucumberExecutionContext.java:98)
at io.cucumber.testng.TestNGCucumberRunner.finish(TestNGCucumberRunner.java:145)
at com.qaprosoft.carina.core.foundation.cucumber.CucumberRunner.tearDownClass(CucumberRunner.java:124)
at com.spdji.t3.runner.TestRunner.afterSuiteTearDown(TestRunner.java:55)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:400)
at org.testng.SuiteRunner.run(SuiteRunner.java:330)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
at org.testng.TestNG.runSuites(TestNG.java:1099)
at org.testng.TestNG.run(TestNG.java:1067)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)
Caused by: java.io.IOException: Stream closed
at java.base/sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:45)
at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:118)
at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:208)
at io.cucumber.core.internal.com.fasterxml.jackson.core.json.WriterBasedJsonGenerator._flushBuffer(WriterBasedJsonGenerator.java:2023)
at io.cucumber.core.internal.com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.close(WriterBasedJsonGenerator.java:992)
at io.cucumber.core.internal.com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4573)
at io.cucumber.core.internal.com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3804)
at io.cucumber.core.plugin.JsonFormatter.finishReport(JsonFormatter.java:147)
... 29 more
Incorrect counts and logs are displayed on zafira report.
Even though 10 scenarios are executed it just 1 scenario is displayed and status/logs are not displayed for other sources.
cucumber html report is not generated due to the
java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonKey
at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.hasAsKey(JacksonAnnotationIntrospector.java:1094)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:496)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:421)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getPropertyMap(POJOPropertiesCollector.java:386)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getProperties(POJOPropertiesCollector.java:233)
at com.fasterxml.jackson.databind.introspect.BasicBeanDescription._properties(BasicBeanDescription.java:164)
at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findProperties(BasicBeanDescription.java:239)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._findCreatorsFromProperties(BasicDeserializerFactory.java:328)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:272)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:223)
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:261)
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:150)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:415)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:350)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:609)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.createContextual(ObjectArrayDeserializer.java:145)
at com.fasterxml.jackson.databind.DeserializationContext.handleSecondaryContextualization(DeserializationContext.java:856)
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:647)
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4805)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4675)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3642)
at net.masterthought.cucumber.ReportParser.parseForFeature(ReportParser.java:93)
at net.masterthought.cucumber.ReportParser.parseJsonFiles(ReportParser.java:72)
at net.masterthought.cucumber.ReportBuilder.generateReports(ReportBuilder.java:97)
at com.qaprosoft.carina.core.foundation.cucumber.CucumberRunner.generateCucumberReport(CucumberRunner.java:179)
at com.qaprosoft.carina.core.foundation.cucumber.CucumberRunner.tearDownClass(CucumberRunner.java:129)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:135)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:65)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:381)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:319)
at org.testng.internal.invokers.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:220)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:130)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.testng.TestRunner.privateRun(TestRunner.java:808)
at org.testng.TestRunner.run(TestRunner.java:603)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:429)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:423)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:383)
at org.testng.SuiteRunner.run(SuiteRunner.java:326)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1249)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
at org.testng.TestNG.runSuites(TestNG.java:1092)
at org.testng.TestNG.run(TestNG.java:1060)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:115)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.annotation.JsonKey
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 55 more
Right now carina-cucumber code doesn't use cucumber options coming from testng xml properties if they present.
To support it TestNGCucumberRunner object should be created with properties taken from testng xml.
check this pull request for example: cucumber/cucumber-jvm@03ea225
Once implemented properties might be specified in next format: https://github.com/cucumber/cucumber-jvm/blob/v7.5.0/testng/README.md#configure-cucumber-options-via-testng-xml
It will allow to e.g. override cucumber tags on suite level
Requirement : when we run a above feature file at that we get results in ./reports/qa/<unique_id (timestamp)>/ /snapshots
can we save this as
I have a feature file it has 5 scenarios as given
Test1: Pass
Test2: Pass
Test3: Fail
Test4: Fail
Test5: Fail
When I execute a test I did the setting of retry_count=3 in config.properties. this means if any of the test fails it should retry 3 times, But this is happening only for the first scenario where as for the 2nd or next scenarios it retries one more time.
below shown execution run results:
Test1 : Pass
Test2 : Pass
Test3 : Fail
Test3 : Fail > Retry > 1
Test3 : Fail > Retry > 2
Test3 : Fail > Retry > 3
Test4 : Fail
Test4 : Fail > Retry > 1
Test4 : Fail > Retry > 2
Test4 : Fail > Retry > 3
Test4 : Fail > Retry > 4
Test5 : Fail
Test5 : Fail > Retry > 1
Test5 : Fail > Retry > 2
Test5 : Fail > Retry > 3
Test5 : Fail > Retry > 4
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.