Test case execution order about red

[michalanglart]

Hi,

all those different settings inside run configuration are used to build command line which will run your tests. The Test Suite(s) box just specifies which suites or tests inside them should be executed which translates to --suite and --test switches in command line call (see http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#command-line-options-for-test-execution ).

I'm not 100% sure about this, but it seems that there are only 2 possibilities to influence the order in which Robot will execute tests:

1. by randomizing the order (see http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#randomizing-execution-order ) with command line switch. RED does have dedicated setting for this in Run Configuration, but you can always add e.g. '--randomize all' switch in 'Additional Robot Framework arguments' box
2. suites order can be influenced with file name prefixes (see http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#execution-order ). It should work with RED, but this will not affect the order of tests inside the suites if I understood that correctly.

As long as Robot Framework does not provide some mechanism to specify tests order there is not much we can do about it unfortunately.

BR,
Michal

[TimoLinden]

Thanks for your explanation. Unfortunately not the answer I was hoping for.. I need more flexible way to select which test cases and in which order to execute. Is there something that prevents RED providing this kind of functionality? Please consider to put more effort on functionalities releated to test execution.
Br,
Timo

[michalanglart]

RED does not execute tests itself. It builds command line basing on run configuration which looks (in simple example) something like this:

python -m robot.run -s mySuite -t myTest filepath_to_project

and issues this command to operating system. The command will run python interpreter using robot.run module (which is an entry point for test execution in RF) and passing the arguments to it.

So the answer for "Is there something that prevents RED providing this kind of functionality?" is that currently RED is limited to functionalities which are offered by RF command line switches/arguments. So for example robot offers --includes or --excludes switches which selects tests to execute basing on its tags and RED also provides possibility to define such tags in Run Configuration.

However since it seems impossible to tell robot: "I want to first run test_x and then test_y" using command line (see http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#command-line-options-for-test-execution ), it also impossible for RED in current architecture.

[michalanglart]

We will have a deeper look into this issue as it seems that maybe ordering would be possible using SuiteVisitor API and -prerunmodifier switch, but it would be available with RF in version 2.9 or newer.

[adam-wypych]

@TimoLinden if am I understand You correctly, You want to have some functionality which will record sequence of adding test cases to Your run configuration and execute it in that sequence?

PS. I fully agree with @michalanglart that the test execution is mostly connected how RF run test suites. And from my experience this is mostly connected to how OS provide list of files in directory to RF to read. This is why this guide for ordering presave was prepared: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#execution-order .

Before RED project was started, I was in the team where we had similiar issue, it have been solved mostly base on the idea with numbers as prefix, with a little bit extend this workaround mostly because in our application we have many features. So we added some additionall prefix for each future like i.e. A00001. We increase this 0 numbers to do not have issues where we will have over 100 test cases.

Additionally we strongly prevent mess so every test suite for future should be in corresponding directory FEATURE__A__simpleShortName and test suites name has also name and prefix in place like A00001, so we guide RF how it should execute test cases. Base on that we achieve two things:

1. test execution could be split for multiple machines just by guide, which feature test should be executed and where (this decrease the time to get results in case of thousands long running test cases)
2. it will be always execute each test suite in the order which You want - if we add just option to guide RF via our SuiteVisitator API still You will have problem in Your CI.

Unfortunatelly base on the RF response to issue robotframework/robotframework#329 and also connected to it other issues robotframework/robotframework#1770 plus robotframework/robotframework#2385 .
You can't guarantee ordering of execution base on sequence of --test arguments presents in command line. This should be done by --prerunmodifier.

[TimoLinden]

Thanks again for your comprehensive answer. Currently I see RED usefull for test case development but not for test case execution. Here are few examples of my use cases:

• I want to run my tests in a loop for days
• I want to run some cases more often than the others
• I want to select the execution order of my test cases
  At the moment I need to use an external script to do this.

[michalanglart]

As to the execution order I'm currently evaluating possibility to define it inside RED using aforementioned prerunmodifier script. It looks very promising so far, but we'll talk about this within team in next week.

If this feature would get into RED it would be limited to Robot at least 2.9 since the visitor API was introduced in this version.

Also can you elaborate on this use case: 'I want to run some cases more often than the others"? What do you mean by this?

[adam-wypych]

@TimoLinden: "I want to run my tests in a loop for days" do You execute some stability tests? I am asking, because execute tests in loop without change the report gives to override it every time You execute new test, so finally You will see last result.
I think better for it is just have one of CI application like i.e. Jenkins where You can execute tests in loop every day and reports will be not override since its stored in BUILD_NUMBER directory.

[TimoLinden]

Ok, thanks for the tips. We'll have a look at Jenkins.

[adam-wypych]

Ok.
For Your "I want to run some cases more often than the others" I'm not fully ensure if I am get Your issue correctly. But if You want to execute some representative test cases after You commit\push change to Your application, You can use then tags.
In my previous project we used tag QUICK which execute only test cases, which covers most of the logic and gives feedback as soon as possible. All test cases was executed every 2 hours, which works fine for use.