edgexfoundry / edgex-taf

TAF

TAF stands for Test Automation Framework. It is used for automating end-to-end UI and API tests of the EdgeX services.

Those tests can separate to different suites:

• Functional test
• Integration test
• Performance test

Usage

• Users can refer to the the document to learn how to run functional test and integration test.
• Users can refer to the the document to run performance metrics collection on local.

Community

• Discussion: https://github.com/orgs/edgexfoundry/discussions
• Mailing lists: https://lists.edgexfoundry.org/mailman/listinfo

License

Apache-2.0

edgex-taf's Issues

TAF-Demo is not able to clone

I tried to clone the demo code which described in the README, it seems need to provide the username and password.

$ git clone --recurse-submodules https://github.impcloud.net/IVS/TAF-Demo.git
Cloning into 'TAF-Demo'...
Username for 'https://github.impcloud.net': 

@ramya-ranganathan Could you help check the TAF-Demo repository?

Missing tests for CBOR events

This issue is moved from edgexfoundry/blackbox-testing#338

Add performance metrics collection

Move performance collection into this repository.

Add edgex-taf docs

Add documents:

• CH-Edgex-taf.rst for QA tester to implement test cases.
• CH-Self-Assessment-DS.rst for DS developer to verify DS common behavior

Create Core Data V2 API Test Cases - Positive

Create core-data positive testcases based on core-data V2 API. Reference: blackbox-testing core-data.postman_collection.

Create test cases for exporting data to backend

• Export data to HTTP Server
• Export data to MQTT Server

Need changing modbus simulator image which is built by ourself

Currently, used modbus simulator image panterdsd/diagslave didn't support arm64, need changing it to following image.

nexus3.edgexfoundry.org:10003/edgex-devops/edgex-modbus-simulator-arm64:latest
nexus3.edgexfoundry.org:10003/edgex-devops/edgex-modbus-simulator:latest

Extract run.py to edgex-taf-common

Since run.py extract to taf-common, common.robot need to modify the config and logs path
This change should work after edgexfoundry/edgex-taf-common#4 finished.

Dockerize edgex-taf and robotframework

I saw edgex-taf invokes the robot via robot API,

            logging.info("Run test {0}".format(testfile))
            if DRY_RUN is False:
                logging.debug("Output to {0}, {1}".format(tmpOutputDir,noReport))
                if noReport:
                    kwargs["output"] = testname+".xml"
                run(testfile, **kwargs)

but we need to dockerize the edgex-taf and robotframework for running the testing on the Jenkins server.

For performance-summary-collection, we use robotframework and docker-compose docker container to execute full testing :

1. Mount all test script to robotframework container

docker run --rm --network host -v $PWD:$PWD -w $PWD -v /var/run/docker.sock:/var/run/docker.sock  \
    iotech-releases.jfrog.io/robotframework-x86_64:1.0.0 -d report .

2. Test script execute docker-compose command to deploy EdgeX

docker run --rm --env-file x86_64.env -v $PWD:$PWD -w $PWD -v /var/run/docker.sock:/var/run/docker.sock   \
    docker/compose:1.24.0 up -d

3. Execute test in the robotframework container.

@juan-pablo-lopez-gutierrez
Is it possible to modify the edgex-taf to execute testing in docker container?
Because It's good to dockerize all components(edgex-taf, robot, docker-compose) to make it easy to run testing on different testing machine like gateway, raspi ...etc.

Incorrect modbus simulator port

Modbus simulator runs on port 1502, not 502.

Add eNotation for value checker

Float value can be eNotation when verify testing result

Add start db to deploy edgex step

Compose files of developer-script had removed db from depend_on, we have to start it from the script.

Modify 6_scheduling test script to support C DS

Scheduling001 - Test Resource and Frequency for autoEvent

This case tests the device service should execute AutoEvent correctly.

Once the device-opc-ua adds a device with AutoEvent and then the device-opc-ua will execute the AutoEvent immediately, but the device-modbus-go will wait for the frequency time.

For example, DS add a device with the auto event as the following device entity:

{
  "description": "AutoEvent-Device",
  "id": "...",
  "name": "AutoEvent-Device",
   ...
  "autoEvents": [
     {
       "frequency": "8s",
       "resource": "DeviceValue_UINT64_R"
     }
   ]
}

After 16 seconds:

The device-opc-ua generates 3 readings

The device-modbus-go generates 2 readings

Fail to run demo.robot

I follow the README instruction and it seems not work to me.
@ramya-ranganathan Could you help check this error?

$ cd TAF/testScenarios
$ python3 run.py -u UC_Color_Log_Demo/
INFO:root:Running usecase ['UC_Color_Log_Demo/']
DEBUG:root:Testing demo.robot
INFO:root:Run test demo.robot
DEBUG:root:Output to ../../testArtifacts/robot/UC_Color_Log_Demo/, True
==============================================================================
Demo                                                                          
==============================================================================
TC0001 - Read Config File                                             | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
TC0002 - Test Debug Output                                            | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
TC0003 - Test Info Output                                             | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
TC0004 - Test Warn Output                                             | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
TC0005 - Test Error Output                                            | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
TC0006 - Test Log Exception Output                                    | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
TC0007 - Test PASS Output                                             | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
TC0008 - Test Fail Output                                             | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
TC0009 - Test Heading Output                                          | FAIL |
Parent suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'
------------------------------------------------------------------------------
Demo                                                                  | FAIL |
Suite setup failed:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/logs/UC_Color_Log_Demo.log'

Also suite teardown failed:
AttributeError: '__SettingsInfo' object has no attribute 'TestLog'

9 critical tests, 0 passed, 9 failed
9 tests total, 0 passed, 9 failed
==============================================================================
Output:  /Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/robot/UC_Color_Log_Demo/demo.xml
Log:     /Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/robot/UC_Color_Log_Demo/log.html
Report:  /Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/robot/UC_Color_Log_Demo/report.html
Log:     /Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/robot/UC_Color_Log_Demo/_log.html
Report:  /Users/bruce/Documents/workspace/git/edgexfoundry-holding/edgex-taf/TAF/testArtifacts/robot/UC_Color_Log_Demo/_report.html

Incorrect reading result order cause TAF can't fetch latest reading

• Use http://localhost:48080/api/v1/reading/device/{name}/limit instead of the query API to fetch the latest reading

Issue description:
The API http://localhost:48080/api/v1/reading/name/{name}/device/{name}/limit can't fetch the latest reading because RedisDB implementation is different to MongoDB.

Issue opened edgexfoundry/edgex-go#2394

Update TAF doc according to the new changes

Update documentation:

• edgex-taf/README.md
• edgex-taf/docs

Those changes are base on:

• Directory structure change #27
• TAF common usage change https://github.com/edgexfoundry/edgex-taf-common/blob/master/README.md

Enhancement: Add helper functions for testing

This issue is moved from edgexfoundry/blackbox-testing#252

Add checking if the app service start up

Found error, when executing get device command without app-service startup, backend didn't receive export data.

Get docker-compose file from developer-script

Make sure the testing always uses the latest docker-compose file.

Missing BACKWARD variable in deploy-edgex.sh

Should add ${BACKWARD} in docker-compose.yaml

docker run --rm -v ${WORK_DIR}:${WORK_DIR}:rw,z -w ${WORK_DIR} -v /var/run/docker.sock:/var/run/docker.sock \
        --env WORK_DIR=${WORK_DIR} --env PROFILE=${PROFILE}  \
        --env DS_PROFILE=${DS_PROFILE} --env CONF_DIR=${CONF_DIR} --env REGISTRY_URL=${REGISTRY_URL} \
        ${COMPOSE_IMAGE} -f "${WORK_DIR}/TAF/utils/scripts/docker/docker-compose.yaml" up -d consul data metadata command logging notifications scheduler

Create test cases for support-scheduler

Set scheduler to clean up data.

Geneva release version verification

Update to get geneva compose files.

Update App Service Configurable overrides in docker-compose-end-to-end.yaml

Need to update App Service Configurable, base on the change of blackbox-testing PR #440 and issue #447.

Refactor the edgex and DS deploy script

TAF should deploy edgex and DS with separate script

Add test suites fot DS certification

Implement the following test suites for common device service behaviors:

• 1_service_operation.robot
• 2_configuration_setting.robot
• 3_device_profile.robot
• 4_device.robot
• 5_data_transformation.robot
• 6_query_commands.robot
• 7_actuation_commands.robot

Move deploy edgex robot file out of functional-test directory

Currently, deploy edgex robot file put in functional-test directory will move to the parent directory to also use to another test.

Services fail to start because consul didn't start

Config-seed has removed on developer-script #235. It also cut the dependence between logging and consul.
Needs to start consul without any dependence.

Core-Data: AddEvent -- Negative Test for Mal-formed Event

This issue is moved from edgexfoundry/blackbox-testing#265

Add test case for querying readings by device name and value descriptor

This test case is used to test reading querying API with device name and value descriptor as the path parameters.

Change keyword "set environment variable" to "set test/suite variable"

If the variable only uses on suit or testcase level, please change the keyword to "set test/suite variable".

Create device
    [Arguments]  ${device_file}
    Create Session  Core Metadata  url=${coreMetadataUrl}
    ${data}=  Get File  ${WORK_DIR}/TAF/config/${PROFILE}/${device_file}  encoding=UTF-8
    ${newdata}=  replace string  ${data}   %DeviceServiceName%    ${DEVICE_SERVICE_EDGEX_NAME}
    ${headers}=  Create Dictionary  Content-Type=application/json
    ${resp}=  Post Request  Core Metadata  ${deviceUri}  data=${newdata}  headers=${headers}
    run keyword if  ${resp.status_code}!=200  log to console  ${resp.content}
    Should Be Equal As Strings  ${resp.status_code}  200
    set environment variable  deviceId   ${resp.content}

Refactor the device name of configuration.py

• Modify DEVICE_SERVICE_NAME and DEVICE_SERVICE_EDGEX_NAME to SERVICE_NAME
• Modify docker-compose file to match the SERVICE_NAME for deploying

Using jenkins server to run tests automatically

• Define jjb file on the ci-management to auto-create jenkins job
• Define jenkinsfile to describe the job steps for runing tests

Create test cases to test behavior when running multiple device services at once.

Create test cases to test behaviors when running multiple device services at once.

Got incorrect configuration.toml when running device-virtual fuji release

Can't use same --registry parameter on both release, fuji and geneva.

Need to remove device profile when shutdown device service

Device service used the same device profile name when starting device service. For that reason, when tests multiple device services by order, the device profile should be removed before tests the next device service.

Need to set compose image to variable

There are several files contain compose image. Once the compose version changed, we have to modify several files. Set it to a variable, just modify once.

Handle float value checking properly

For C DS example, put command is

curl http://localhost:48082/api/v1/device/name/Test-Device/command/GenerateDeviceValue_FLOAT32_RW  \
  -X PUT -H 'Content-Type: application/json' \
  -d '{"DeviceValue_FLOAT32_RW":"1.4"}'

When execute read command and the result is

{
   "readings" : [
      {
         "name" : "DeviceValue_FLOAT32_RW",
         "origin" : 1582265858354364854,
         "value" : "1.39999998e+00"
      }
   ],
   "origin" : 1582265858354364854,
   "device" : "Test-Device"
}

We should handle the float value checking properly by comparing the offset.

Run blackbox twice for both Mongo and Redis database

This issue is moved from edgexfoundry/blackbox-testing#344

Create test cases for support-notification

Add test cases for support-notification.

Integration tests for Store and Forward capability

Add integration tests to cover the Store and Forward capability.

Acceptance Criteria is as follows:

• Data is stored for later retry when export function (HTTP or MQTT) fails and persistOnFailure is true and config setting Writeable.StoreAndForward.Enable = true
• Stored data is exported when retry occurs and connection is good just as if it was exported without error originally.
• Stored data is cleared from the DB once successfully retried.
• Stored data retry count is incremented each time the export retry fails and is updated in DB.
• Stored data is cleared from the DB after the maximum configured retires. (Writeable.StoreAndForward.MaxRetries config setting)
• Changing the function pipeline results in any data stored for that app service to be cleared on the next retry.
• Retry loop is only run when config setting Writeable.StoreAndForward.Enabled is true
• Retry loop interval is set by the Writeable.StoreAndForward.RetryInterval config setting
• Export retries will resume after application service is restarted, i.e. store data is persisted between restarts of the application service.
• RetryCount resumes after application service restart, i.e. it isn't reset.
• Changes to Writeable.StoreAndForward values are dynamically processed without restarting the application service.

Need to update docker-compose file and configuration.toml based on the latest sdk

Update the docker-compose.yaml and configuration.toml based on the changing of device-service.

• device-virtual PR#94
• device-modbus PR#142

Append docker-compose failing for device-service

Since the developer-script PR #258 change device-virtual's dependencies, edgex-taf generate docker-compose file. Need to update the keyword which using to append the docker-compose-device-service file.

Need to deploy the security services when security enabled

Currently, edgex-taf only deploys non-security service. Need to deploy the security services when security enabled.

Add Tests For Environment Variable Override

This issue is moved from edgexfoundry/blackbox-testing#243

Update file structure

Separate testcases for Functional Test, Integration Test, and Performance Test.

Create test case to check service logs from support-logging

Add test cases to check the other services log from support-logging.

Can't append app-service container to docker-compose file

Since the portainer moved from developer-script/docker-compose-nexus-*.yml file. Update the generate compose file don't use portainer as keyword.

Change back the compose file to nightly-build

Currently, use the geneva compose file, need to change back to nightly-build.

Coredata Tests Need Device Data

This issue is moved from edgexfoundry/blackbox-testing#27

Remove edgex-taf-common submodule

After resolving the issue edgexfoundry/edgex-taf-common#10, edgex-taf-common setup as python module, user can install TAF common by pip3 command