Question: Is it possible to configure all test executions from a central point about avocado-misc-tests

[apahim]

You can use the YAML test loader:
http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html

Long story short, there's a specific YAML key (test_reference) that can be understood by the Avocado loader to be used as the test reference. The rest of the file is just a regular Avocado YAML parameters file, but the variants will be created per test_reference, meaning each test_reference will have a different set of variants and parameters. Example:

!mux
testcase_01:
    test_reference: passtest.py
    !include : passtest.py.data/passtest.yaml
testcase_02:
    test_reference: sleeptest.py
    variants: !mux
        slow:
            timeout: 1
        fast:
            timeout: 1
testcase_03:
    test_reference: failtest.py
    variants: !mux
        foo:
            p1: 1
        bar:
            p1: 10

And then run with:

$ avocado run testcases.yaml

[PraveenPenguin]

i think we can close this issue , feel free to open if discussion not helped

[narasimhan-v]

@fkromer Can you explain in detail please ?

[fkromer]

@narasimhan-v Right now a user of the repository needs to fork it to being able to configure the tests specific to the users system in a lot of different configuration files. If one would be able to configure the overall set of test suites/tests the user could hold this information in one central "place" potentially separate from the repository. However this is no proposal but a question. If this is not possible to fork the repository is totally fine.

[narasimhan-v]

@fkromer Since the config (yaml) files are dependent on the test scripts, I guess having a separate repository for config will add a lot of maintenance overhead, and deviations. But having one single yaml file for more test / test suites is possible, and can be maintained as your fork downstream, as per the requirements.

[fkromer]

@narasimhan-v

Since the config (yaml) files are dependent on the test scripts, I guess having a separate repository for config will add a lot of maintenance overhead, and deviations.

That is true. It is not about to separate the tests and the config into separate repos. It is about to override the default config in the separate yaml files with user specific config values from one single yaml file. (This single yaml file could be kept separate from the repo more easily.)

But having one single yaml file for more test / test suites is possible, and can be maintained as your fork downstream, as per the requirements.

Where can I find info about how to configure several tests/test suites in one single yaml file?

[narasimhan-v]

It is about to override the default config in the separate yaml files with user specific config values from one single yaml file.

You can do that.

Where can I find info about how to configure several tests/test suites in one single yaml file?

@fkromer That is just to collate all yaml parameters of tests into one single yaml, though it is not as easy. You would need to take care of conflicting parameters

[fkromer]

@narasimhan-v I just keep our fork of the repo with custom configuration. Makes maintenance a lot easier.

[apahim]

Of course, you have to make sure that both test_references and !includes are pointing to valid path locations.

[fkromer]

For to the point reference: other YAML to mux loader examples.

[fkromer]

@apahim I am using avocado v56 with the plugin loader_yaml from PyPi. The plugin seems to be installed:

$ avocado plugins | grep loader_yaml
loader_yaml YAML test loader options for the 'run' subcommand

I added a testcase.yaml file in the root directory of the test repo with your content from above. I tried to run the tests but I get an error:

$ avocado run testcases.yaml 

Unable to resolve reference(s) 'testcases.yaml' with plugins(s) 'file', 'yaml_testsuite', 'external', try running 'avocado list -V testcases.yaml' to see the details.

I don't know if this makes any difference: I have the plugin yaml_to_mux from PyPi installed as well.

Executing the basic example with avocado run basic.yaml works fine.

[fkromer]

Is it possible to define a yaml file in the root directory which loads other yaml files from other directories? (That would allow directory based configuration and test selection.) Could e.g. alltests.yaml (in the root directory) load cpu.yaml (in /cpu), fs.yaml (in /fs), and so on?

[narasimhan-v]

Works for me:

# cat yaml_test.yaml 
!mux
softwareraid:
    test_reference: tests/avocado-misc-tests/io/disk/softwareraid.py
    !include : tests/avocado-misc-tests/io/disk/softwareraid.py.data/softwareraid.yaml
blkdiscard:
    test_reference: tests/avocado-misc-tests/io/disk/ssd/blkdiscard.py 
    !include : tests/avocado-misc-tests/io/disk/ssd/blkdiscard.py.data/blkdiscard.yaml

# avocado run yaml_test.yaml 
JOB ID     : f3d4b65bc5494be91791e54c413436106d77cf31
JOB LOG    : /root/avocado-korg/io/results/job-2017-12-08T10.23-f3d4b65/job.log
 (1/7) tests/avocado-misc-tests/io/disk/softwareraid.py:SoftwareRaid.test_run;raidlinear-scenario-5e7c: FAIL (0.32 s)
 (2/7) tests/avocado-misc-tests/io/disk/softwareraid.py:SoftwareRaid.test_run;raid0-scenario-4c2c: FAIL (0.32 s)
 (3/7) tests/avocado-misc-tests/io/disk/softwareraid.py:SoftwareRaid.test_run;raid1-scenario-4dbc: FAIL (0.32 s)
 (4/7) tests/avocado-misc-tests/io/disk/softwareraid.py:SoftwareRaid.test_run;raid5-scenario-2827: FAIL (0.32 s)
 (5/7) tests/avocado-misc-tests/io/disk/softwareraid.py:SoftwareRaid.test_run;raid10-scenario-8715: FAIL (0.32 s)
 (6/7) tests/avocado-misc-tests/io/disk/softwareraid.py:SoftwareRaid.test_run;raid6-scenario-86dc: FAIL (0.32 s)
 (7/7) tests/avocado-misc-tests/io/disk/ssd/blkdiscard.py:Blkdiscard.test;scenario-f897: PASS (16.45 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 6 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 19.71 s
JOB HTML   : /root/avocado-korg/io/results/job-2017-12-08T10.23-f3d4b65/results.html
#

[apahim]

Yep, my example was not intended to work, considering the paths and the files are not real. If you provide accurate paths, it should work.

[fkromer]

@apahim Ah, was not clear to me.