Comments (18)
Hello @fatahillahardhi
Case 1
This is as expected.
When using MobileDeviceType.SIMULATOR_AND_REAL_DEVICE
, Courgette calculates the max number of device types per type and orders the runs based on this.
The reason is Courgette cannot pre-allocate devices without actually knowing the scenario that's about to run next. So Courgette determines the strategy before any tests can run so it knows how to allocate devices based on the provided devices types. If you have more devices of a certain type then this will be used to determine the run order.
So for example, if you provide:
3 simulators and 1 real device
- All scenarios without a matching
realMobileDeviceTag
will first run on simulators in parallel. Then all scenarios that require a real device (i.e. the scenario that has a matching realMobileDeviceTag) will run next.
3 real devices and 1 simulator
- All scenarios that has a matching
realMobileDeviceTag
will first run on real devices in parallel. Then all scenarios that require a simulator will run next.
Case 2
If you use MobileDeviceType.SIMULATOR_AND_REAL_DEVICE
and provide both simulators and real devices without specifying any matching realMobileDeviceTag
then all scenarios will run on simulators.
You need to be explicit on which tests can + should run on real devices. Courgette does not know whether all tests can run on both simulators + real devices so it defaults to simulators.
If you don't specify a realMobileDeviceTag
then you better of either using MobileDeviceType.SIMULATOR
or MobileDeviceType.REAL_DEVICE
from courgette-jvm.
As this is not currently supported, I will investigate if we can differentiate between simulators and real devices in the same run.
from courgette-jvm.
Hi @prashant-ramcharan , hope you're doing well,
Thank you for the answer. I'm very interested to wait for the updates!
from courgette-jvm.
@fatahillahardhi This is now supported in Courgette version 6.11.0
Usage example
Courgette will only allocate a real device to tests matching any of the tags in realMobileDeviceTag
. For all other tests, one of the simulators will be allocated.
@RunWith(Courgette.class)
@CourgetteOptions(
...
plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
mobileDevice = {
"simulator1",
"simulator2"
"realDevice:00000000-000-0000-0000-000000000001",
},
realMobileDeviceTag = {"@UploadPhotos"},
)
from courgette-jvm.
Hi @prashant-ramcharan, it's really awesome!
However, I have encountered a problem. When I try to run it with SIMULATOR_AND_REAL_DEVICES
, it always runs on Real Devices.
Here are my Test Runner:
@RunWith(Courgette.class)
@CourgetteOptions(
runLevel = CourgetteRunLevel.SCENARIO,
reportTargetDir = "build",
testOutput = CourgetteTestOutput.CONSOLE,
environmentInfo = "project=iOS Test",
plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
mobileDevice = {
"iPhone 14",
"iPhone Real:xxxxxxx-xxxxxxx"
},
realMobileDeviceTag = {"@UploadPhotos"},
cucumberOptions = @CucumberOptions(
features = "src/test/resources/features",
publish = true,
plugin = {
"pretty",
"json:build/cucumber.json"}
))
When I run the command: ./gradlew clean test -Dcourgette.threads=2 -Dcucumber.tags="@focus" -Dcourgette.rerunFailedScenarios=false
, it consistently runs on Real Devices.
Moreover, when I try to print CourgetteMobileDeviceAllocator.DEVICE_NAME
to the terminal, it shows "iPhone 14," but in reality, the execution occurs on Real Devices, and it doesn't run in parallel.
from courgette-jvm.
Hi, if you remove the realMobileDeviceTag
option from your runner, does all tests run on the simulator?
Also, could you please share how your @UploadPhotos
scenario is tagged?
Is it tagged on scenario level, scenario outline (example table) level or at the feature level?
from courgette-jvm.
I tested with this
@RunWith(Courgette.class)
@CourgetteOptions(
threads = 3,
runLevel = CourgetteRunLevel.SCENARIO,
reportTargetDir = "build",
testOutput = CourgetteTestOutput.CONSOLE,
environmentInfo = "app=iOS test application; project_info=Courgette-JVM is awesome!",
plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
mobileDevice = {
"iPhone 15",
"iPhone 15:123",
},
realMobileDeviceTag = { "@upload" },
cucumberOptions = @CucumberOptions(
features = "src/test/resources/features",
glue = "steps",
tags = "@ios",
publish = true,
plugin = {
"pretty",
"json:build/cucumber-report/cucumber.json",
"html:build/cucumber-report/cucumber.html"}
))
public class IosTestRunner {
}
@ios
Feature: Test iOS application
Scenario: Verify application alert on device 1
Given I launch the app
When I show the alert
Then I verify the alert shows this alert is so cool.
And I accept the alert
@upload
Scenario: Verify application alert on device 2
Given I launch the app
When I show the alert
Then I verify the alert shows this alert is so cool.
And I accept the alert
Output was as expected: 1 passed on the simulator and 1 failed (test tagged with @upload
) because the real device does not exist.
───────────────────────────────────────────────────
Courgette Test Statistics
───────────────────────────────────────────────────
Summary: 50% passed, 50% failed
Duration: 0 min, 27 sec
Run Level: Scenario
Total: 2
Passed: 1
Failed: 1
───────────────────────────────────────────────────
from courgette-jvm.
I have tried several tests with several cases:
Devices:
- 1 Simulator
- 1 Real Device
Case 1: Run with 2 scenarios, one of which contains the @UploadPhotos
tag.
Results:
- Automation test does not run in parallel ❌
- Runs on both Simulator and Real Devices ✅
- Scenario with the @UploadPhotos tag does not run on the Simulator ✅
Case 2: Run with 3 scenarios, one of which contains the @UploadPhotos tag, and remove the realMobileDeviceTag
.
Results:
- Automation test does not run in parallel ❌
- Only runs on the Simulator ❌
- Scenario with the @UploadPhotos tag runs on the Simulator ❌
Case 3: Run with 3 scenarios without the @UploadPhotos tag and remove the realMobileDeviceTag
.
Results:
- Automation test does not run in parallel ❌
- Only runs on the Simulator ❌
- Scenario with the @UploadPhotos tag runs on the Simulator ❌
Case 4: Run with 3 scenarios without the @UploadPhotos tag and with realMobileDeviceTag
.
- Automation test does not run in parallel ❌
- Only runs on the Simulator ❌
- Scenario with the @UploadPhotos tag runs on the Simulator ❌
Overall I think realMobileDeviceTag
is doing well, but the issues is the test do not run in parallel but instead run sequentially, similar to conventional testing.
Scenario:
@iOS @Example
Feature: Scenario with Upload Tags
@Test1
Scenario: Login with already registered phone number
Given User is on screen
When User click button login
And User insert phone number "0823141284"
And User input correct OTP
Then User is on bibit main page
@Test2 @UploadPhotos
Scenario: Login with unregistered phone number
Given User is on screen
When User click button login
And User insert phone number unregistered "081231231231"
Then User direct otp page without get the otp
@Test3 @UploadPhotos
Scenario: Login with unregistered phone number 2
Given User is on screen
When User click button login
And User insert phone number unregistered "081231231231"
Then User direct otp page without get the otp
Test Runner
@RunWith(Courgette.class)
@CourgetteOptions(
threads = 2,
runLevel = CourgetteRunLevel.SCENARIO,
rerunFailedScenarios = true,
rerunAttempts = 1,
reportTargetDir = "build",
testOutput = CourgetteTestOutput.CONSOLE,
environmentInfo = "project=iOS Test",
plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
mobileDevice = {
"iPhone:xxxxxxx-xxxxxxx",
"iPhone 14"
},
realMobileDeviceTag = {"@UploadPhotos"},
cucumberOptions = @CucumberOptions(
features = "src/test/resources/features",
tags = "@Example",
publish = true,
plugin = {
"pretty",
"json:build/cucumber.json"}
))
from courgette-jvm.
Hi, with regards to this point:
Automation test does not run in parallel
You would need to specify more than 1 simulator or device to be able to run in parallel as Courgette can only allocate 1 device per run at a time.
We wont be able to run multiple tests in parallel if we only have 1 device / simulator per run.
Example:
threads = 3,
mobileDevice = {
"iPhone:xxxxxxx-xxxxxxx",
"iPhone 14",
"iPhone 15"
"iPhone 15 Pro"
},
The above will run 3 tests in parallel on the simulators
and only run tests on the real device if it matches the realMobileDeviceTag
Did you try adding more simulators / devices to see it runs in parallel?
Could you also try running the example project and let me know how you get on?
https://github.com/prashant-ramcharan/courgette-jvm-appium-ios
from courgette-jvm.
I have just tried adding more devices, but the result remains the same
Automation test does not run in parallel
mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
mobileDevice = {
"iPhone 14",
"iPhone 1:00008020-00146850340B402E",
"iPhone 2:00008020-001531EA2E22402E",
"iPhone 4:00008030-001419043428C02E",
"iPhone 6:3c240073ed0e0ecf6f786ebed31a9ee0b147e8bf",
},
realMobileDeviceTag = {"@UploadPhotos"}
from courgette-jvm.
Could you please try using just simulators to test?
Are you able to provide a demo project to reproduce this issue?
The tests run fine in parallel for me.. Link to video here
from courgette-jvm.
Oh sorry, I forgot to tell you.
If I use MobileDeviceType.SIMULATOR
or MobileDeviceType.REAL_DEVICE
, it can run properly (run with parallel), but if I use MobileDeviceType.SIMULATOR_AND_REAL_DEVICE
, the Automation test does not run in parallel.
from courgette-jvm.
Hi @fatahillahardhi , no worries 🙂
I'm also using MobileDeviceType.SIMULATOR_AND_REAL_DEVICE
and it runs fine in parallel.
Please see video demo here
As you can see in the video, the tests ran in parallel in the simulators and failed when running on the real device because the device did not exist (as expected)
If you want to share your Appium logs during the run, I can take a look. Might be worth re-building / signing your web driver agent again on the real devices.
Not sure how to reproduce this, so please provide a demo project if possible.
from courgette-jvm.
I attempted to run the example project here and encountered a unique result.
I used both MobileDeviceType.SIMULATOR
and MobileDeviceType.SIMULATOR_AND_REAL_DEVICE
. Surprisingly, the results were the same whether I ran the tests with 3 simulators or with 3 simulators + 1 real device. Unfortunately, the automation tests did not run in parallel.
Case 1, when using either MobileDeviceType.SIMULATOR
or MobileDeviceType.SIMULATOR_AND_REAL_DEVICE
and running with 3 simulators or 3 simulators + 1 real device:
- The automation test was expected to run in parallel but encountered errors.
- Out of 4 scenarios, only 1 scenario passed, while the other 3 failed consistently after multiple attempts.
- You can find the logs here (running with simulators only).
Case 2, when using MobileDeviceType.SIMULATOR
and running with 1 simulator:
- All scenarios passed successfully.
- You can find the logs here.
I hope this information helps in identifying and resolving the issues.
Test Runner:
@RunWith(Courgette.class)
@CourgetteOptions(
threads = 3,
runLevel = CourgetteRunLevel.SCENARIO,
reportTargetDir = "build",
testOutput = CourgetteTestOutput.CONSOLE,
environmentInfo = "app=iOS test application; project_info=Courgette-JVM is awesome!",
plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
mobileDeviceType = MobileDeviceType.SIMULATOR,
mobileDevice = {
"iPhone 15",
"iPhone 15 Plus",
"iPhone 15 Pro Max",
},
cucumberOptions = @CucumberOptions(
features = "src/test/resources/features",
glue = "steps",
tags = "@ios",
publish = true,
plugin = {
"pretty",
"json:build/cucumber-report/cucumber.json",
"html:build/cucumber-report/cucumber.html"}
))
Features:
@ios
Feature: Test iOS application
@Test1 @UploadPhotos
Scenario: Verify application alert on device 1
Given I launch the app
When I show the alert
Then I verify the alert shows this alert is so cool.
And I accept the alert
@Test2
Scenario: Verify application alert on device 2
Given I launch the app
When I show the alert
Then I verify the alert shows this alert is so cool.
And I accept the alert
@Test3
Scenario: Verify application alert on device 3
Given I launch the app
When I show the alert
Then I verify the alert shows this alert is so cool.
And I accept the alert
@Test4 @UploadPhotos
Scenario: Verify application alert on device 4
Given I launch the app
When I show the alert
Then I verify the alert shows this alert is so cool.
And I accept the alert
from courgette-jvm.
I'm using the same project as shown in the videos above so cannot replicate your issue.
Are you able to provide any example project for me to reproduce this? I cannot reproduce this with the Courgette example project as shown in the videos.
Case 1, when using either MobileDeviceType.SIMULATOR or MobileDeviceType.SIMULATOR_AND_REAL_DEVICE and running with 3 simulators or 3 simulators + 1 real device:
Could you please share your Courgette html report? This should show any exceptions thrown in the test.
from courgette-jvm.
@fatahillahardhi In the mean time, I will release a new Courgette option to log the run details for each parallel run.
from courgette-jvm.
@fatahillahardhi I just released Courgette 6.12.0
with a new courgette option generateCourgetteRunLog
Could you try updating your runner and include the following option?
generateCourgetteRunLog = true
At the end of the test run you should have a courgette run log saved to your report directory.
Please share the run log and Courgette html report if you still want me to look into this.
from courgette-jvm.
hi @prashant-ramcharan , sorry for the wait, I can only get back into this project now.
I have reattempted with some cases with version 6.12.0
Case 1: Simulator & Real Devices using realMobileDeviceTag @UploadPhotos
- 2 Simulators and 1 Real Device
- Initially, only the 2 simulators with tags not containing @UploadPhotos are executed in parallel.
- Once these 2 simulators are running, the real device does not react at all, more like waiting for the scenario without @UploadPhotos to finish.
- When the @UploadPhotos scenario finishes, with only scenarios containing @UploadPhotos remaining, the Real Device starts running, resulting in no parallel execution (with Real Devices). Is this expected?
Case 2: Simulator & Real Devices not using realMobileDeviceTag @UploadPhotos
- 2 Simulators and 1 Real Device
- Only simulators run in parallel; the real device does not run.
- Everything else remains as expected.
Here's the report: Link
Update: I also added the video on that link
from courgette-jvm.
Related Issues (20)
- Not able to run Courgette class HOT 1
- Run on Appium 2 HOT 1
- How to have the status as WARNING in the Courgette Report? HOT 5
- How to embed a URL in the courgette report as URL instead of just String? HOT 2
- How to integrate courgette with docker container HOT 4
- How fix java.lang.ClassNotFoundException: Files\Git\cmd;;/ HOT 3
- Exposing Scenario hooks HOT 4
- Parallel execution is not working running test on real devices (Android or iOS), it works fine on emulator and simulator HOT 1
- With Appium version 2.x is throwing error "org.openqa.selenium.UnsupportedCommandException: Method has not yet been implemented" HOT 1
- Courgette error processing reports HOT 2
- No Courgette Report is being produced if there are no tags as specified by the user, not in the project repo though. HOT 1
- Execute Courgette having test suite in a FAT JAR HOT 10
- Unable to override cucumber tags from the maven command line HOT 5
- Maven Build is failing with this error 'java: package com.fasterxml.jackson.datatype.jdk8 does not exist' HOT 3
- Query : How to retrieve System and environment info details HOT 5
- Report output get only show all failed HOT 1
- Reports not genenated HOT 5
- error=206, The filename or extension is too long trying to execute my project with courgette-jvm HOT 1
- Courgette report index.html is not generated properly HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from courgette-jvm.