authorjapps / zerocode Goto Github PK

The runner should look like below-

@LoadWith("load_config_sample.properties")
@TestMapping(testClass = TestGitGubEndPoint.class, testMethod = "testGitHubGET_load")
@RunWith(ZeroCodeLoadRunner.class)
public class LoadTest {

}

Basically a map accepting a method by its class name.

I would like to run same test case for API testing in parallel. e.g. call GET api around 10 times in parallel and see response so I can check if any issue with my code in case of multi-threaded env.

@authorjapps Excited about this awesome project, I've ran into a scenario where it require to pass number of query parameters in url. Can't we have the support for GET requests as in this POST example. Please ignore if there is existing support, I couldn't find and unable to get my trial and error working.

sample json:

{
  "scenarioName": "A scenario which has huge number of url query parameters",
  "steps": [
    {
      "name": "get_user_details",
      "url": "/example",
      "operation": "GET",
      "request": {
        "queryParams": {
          "queryParam1": "...",
          "queryParam2": "...",
          "queryParam3": "...",
          "queryParam4": "...",
          "queryParam5": "..",
          "more": "..."
        }
      },
      "assertions": {
        "status": 200,
        "body": {
          "...": "...",
        }
      }
    }
  ]
}

otherwise url will be like "url":"/examplequeryParam1=...&queryParam2=...&queryParam3=...&queryParam4=...&queryParam5=..."
which can grow more and become harder to grasp. Can't we have the support for this kind of scenario? Once get more familiar with the code I'd love to contribute. Thanks.

This is probably very useful feature to test the stress or capacity of an application under multiple type of load/stress e.g.

1. One user is just firing a request payload with one record to create one entity on server
2. Another user firing a request payload with 10 records to create multiple entities on server
3. Another user is firing similar POST, then PUT and then GET to check CRUD operation went well
4. Another user just firing GET to view the results from server due it has only read-only access

I want to generate load or stress with the above scenarios.

Basically if I configure 4 threads(mean 4 users) with 4 secs ramp-up time, I want all 4 threads to be picked up in parallel for all 4 scenarios respectively.

That means if I configure 8 threads, then I want to repeat the same requests to be fired again in circular manner till all my threads are used up i.e. thread-5 firing case-1, thread-6 firing case-2, thread-7 firing case-3, thread-8 firing case-4 and so on.

Which means if I configure 10 threads, then thread-9 firing case-1, thread-10 firing case-2 again and so on in similar pattern.

Can I use ZeroCodeLoadRunner.class for this?
If so how?

Work branch: issue6_number_equals
i.e. https://github.com/authorjapps/zerocode/tree/issue6_number_equals

e.g. this should work

      "assertions": {
        "status": 200,
        "body": {
          "id": "$NUMBER.EQ.1",
          "name": "Shower",
          "availability": true
        }
      }

also, the following should PASS

"assertions": {
        "status": 200,
        "body": {
          "id": "$NUMBER.EQ.${$.another_get_call.response.body.id}",
          "name": "Shower",
          "availability": true
        }

See implementation of $GT.
Trace code: 01_verification_test_cases/05_number_equals_test.json

@authorjapps, Can you redirect/persist the testcase and test-run data into a in-memory DB e.g. H2 or give provision to configure in a DB as well as offline?

Advantages
This will provide some bit of useful information as well as some kind of aggregated charts etc along with, it should allow to navigate to particular test log by a color code.

@JsonSerialize(using = LocalDateTimeSerializer.class)

• Amend this to generate or accept nano secs.
• Useful for load aka stress test reports in case loads are generated at same secs e.g. within 0(zero) second, you launch 5 to 10 users.
  There is no fine grain than nanoseconds in Java, so considere until the finest grain.

Something like the below flag will help.
interactive.html.report.disabled=false >> This is default and html report is generated.
interactive.html.report.disabled=true >> If you do not want the html reports be generated.

This is suggestion only.

please introduce a property e.g.
continuous.run.time.in.seconds
This will tell the framework for how many minutess or hours(1hr=60min=3600seconds) to run.
continuous.run.time.in.seconds=3600
or
continuous.run.time.in.minutes=60

Review Comment at PR by @nirmalchandra .

can you upgrade to <version>1.58</version> to see if the port is not repeated?
You can have as a tech-debt.

How to execute a SOAP service, given the WSDL file i.e. -
Given WSDL file having soapAction, port, endPoint and request xsd or the request xml etc, how to execute the SOAP service?

e.g.
You are developing a REST api GET:/api/v1/persons/p001

• and to build up the person details, this service needs to get address details of this person from another REST service
• i.e. Get the address details for p001 using API GET:/api/v1/addresses/pid/p001

Many times while designing an integration test we need to mock GET:/api/v1/addresses/pid/p001 to provide response as e.g.

GET:/api/v1/addresses/pid/p001
Response:
{
   "personId": "p001",
   "address": { 
          "line1" : "10 Downing Street"
    }
}

Then the GET person will respond with:

GET:/api/v1/persons/p001
Response:
{
   "personId": "p001",
   "name": "Peter",
   "age": 33,
   "address": { 
          "line1" : "10 Downing Street"
    }
}

Git branch: issue5_overwrite_http_client

e.g.
@UseHttpClient can be used to specify the CustomHttpClient class

@UseHttpClient(MyProjectHttpClient.class)
@TargetEnv("config_hosts.properties")
@RunWith(ZeroCodeUnitRunner.class)
public class ZeroCodeRunnerWithProjectHttpClient{
   ...
   ...
}

Does it have any LocalDate or LocalDateTime to generate current date which can work as below-

"date_today": "${LOCAL.DATE.TODAY:yyyy-MM-dd}",   //<---- should translate to e.g. 2018-02-11
"year_day": "${LOCAL.DATE.NOW:yyyy}" //<---- should translate to e.g. 2018

and it should support the official Java format given in the spec link below
https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

Display code-coverage report by integrating with travis.yaml file.
See here sample:
https://github.com/TechnionYP5777/SmartCity-Market/wiki/Integrating-Codecov-with-a-GitHub-project-(Coverage-tool)
It's free for OS.

Google search for-
Zerocode TDD BDD for MuleSoft API

You will find link to-
https://dzone.com/articles/zerocode-test-framework-for-restsoap-api-tddbdd-ap

Also check with other search params

• Create a Wiki page, provide the link from README file.
• Clearly define each properties, the theory/thought-process behind it.
• Provide examples of usage in the hello-world project or create a separate project/module
• Provide charts/graphs(not fancy ones) which will help to visualize the load issues(if any)
  -List our RESTful and SOAP end point load test result with linear-graph and bar-graphs
• This will help the community to understand and anyone can integrate/reuse the test code easily.

@authorjapps , can you please support LocalDateTime for generating current timestamp which can be useful during test run?
e.g.

• For generating current timestamp in a format supplied at the runtime of tests
• For generating a random number using datetime with milli of the day or datetime with nano of the day
• Or simply print the date or year etc

@authorjapps , the below feature is needed for the some project teams if they are using the same test suites for different environments e.g. DEV, SIT, UAT, PRE-PROD etc.
Based on the env parameter value, the specific property file should be picked up by zerocode runner.
Is this is already supported in the framework?

Currently it just fires tests the tests, if bad name found as Test-Method, then it goes to next thread.
Instead it should complain, that method not found

Something like below
e.g.
Currently framework supports the below.
Response:

{
   "currentStatus": "Searching"
}

In certain use-cases, the "currentStatus" can be one-of multiple values.
Then it would be nice to support like below.
Assertions:

{
       "currentStatus": "$ONE.OF:[Found, Searching, Not Looking]"
}

That means when the actual response with currentStatus having "Found" or "Searching" or "Not Looking" will pass.

Create a section in the README file-

Who uses Zerocode?

• Home Office(GOV.UK) - Micro-Services Contract Tests
• HSBC Bank - MuleSoft APIs Consumer Contract Tests and E2E Integration Tests

Currently interactive reports display the individual steps pass/failure in specific test.
It would be nice to have requests/responses at the step level displayed as well.

@authorjapps , Currently the time difference between the requestTimeStamp and responseTimeStamp seems like done on the millisecs part only.

e.g.

requestTimeStamp: "2018-02-05T19:46:31.631"
responseTimeStamp: "2018-02-05T19:46:32.731"

Then the response time delay should be:

responseTimeDelay: 1100 milli seconds 
ie (32-31=1sec=1000 millisec) + (731-631=100 millisec) = 1100 milli sec

But it prints(I think so):

responseTimeDelay: 100 milli seconds 

e.g.

• Description
  GIVEN I have a single class and single method
  WHEN I pick the class
  THEN I will be able to run that method for the load/stress

• Feature:
  This will also run all methods in the class. Those methods(in case of more than one) will run sequentially. This feature might have an impact on the performance-testers to think that the methods will run in parallel.

• Solution:
  Either allow this feature with a note or README updated.
  OR
  Warn the developer/programmer about the sequential behavior.

• Note-
  This will be an additional feature only.
  This is also already achieved in the parallel runner currently(JUnit4), for all the parallel users to do their tasks in sequence mimicking the production load/stress.

• So what's new?
  Ans: Only syntactical sugar.

For PASS tests show blue, for failed ones show RED

TOC(Table of content) navigation

• Work in Chrome
• but not in mobile devices eg Android

reproduce:

• Click on the "Step with more assertions", doesnt navigate to the section

To run fully independently WireMock should be supported for mocking the end points

@authorjapps
Observation-
It seems like two reports are generated per run, in a maven module.

Could you please explain why two?

Is it possible to add the static methods for JSON comparison. There are many asserters already available in the open source communities, but by those not all of the below are covered, hence it will be useful to add these asserters.

e.g.
DEFAULT mode is FIELDS_ANY_ORDER_IGNORE_NULL
JsonAsserter.assertEquals(actual, expected, mode); //<-- compare mode FIELDS_ANY_ORDER_IGNORE_NULL

actual={"name": "Harry", "age": 20}, expected={"name": "Harry", "age": 20} : will **PASS**
actual={"name": "Harry", "age": 20}, expected={"age": 20,"name": "Harry"} : will **PASS**
actual={"name": "Harry", "age": 20}, expected={"name": "Harry"} : will **PASS**
actual={"name": "Harry", "age": null}, expected={"name": "Harry"} : will **PASS**
actual={"name": "Harry", "age": 20}, expected={"age": 20} : will **PASS**

JsonAsserter.assertEquals(actual, expected, mode); //<-- compare mode FIELDS_STRICT_ORDER_EXACT_MATCH

actual={"name": "Harry", "age": 20}, expected={"name": "Harry", "age": 20} : will **PASS**
actual={"name": "Harry", "age": 20}, expected={"name": "Harry", "age": "$NOT.NULL"} : will **PASS**
actual={"name": "Harry", "age": 20}, expected={"name": "$NOT.NULL", "age": "$NOT.NULL"} : will **PASS**
actual={"name": "Harry", "age": 20}, expected={"age": 20,"name": "Harry"} : will **FAIL**
actual={"name": "Harry", "age": 20}, expected={"name": "Harry"} : will **FAIL**

Any other scenarios needed?

Can you have a DB executor to execute a SQL query on a DB e.g. Oracle and return result in a HashMap, but translated to JSON format.
e.g.
select ID, NAME from CUSTOMERS where SALARY < 40000;

If it returns result-

The response should be translated to-

"response": [
   {
       "ID": "1",
       "NAME": "Tom"
   },
   {
       "ID": "2",
       "NAME": "Harry"
   }
]

• First make request-timestamp and responseDelayMilliSec adjacent to each other for load-run
• Create a auto throughput section for load reports

Fixed by #130

This will help the community to understand and anyone can integrate/reuse the test code easily.

Is it call back from a message queue/topic?

Work branch: issue5_after_before_date
i.e. https://github.com/authorjapps/zerocode/tree/issue5_after_before_date

e.g.
"startDateTime": "$DATE.AFTER:2015-09-14T09:49:34.000Z"
"endDateTime": "$DATE.BEFORE:2015-09-14T09:49:34.000Z"

See implementation of $GT.
Trace code: public static final String ASSERT_VALUE_GREATER_THAN = "$GT.";

Update:

$DATE.BEFORE to $LOCAL.DATETIME.BEFORE to avoid confusion in the future.
Also same for
$DATE.AFTER to $LOCAL.DATETIME.AFTER

Currently it generates by timestamp.
Better if it generates by correlation-id which will be easy to track/calculate/ci-integration/log-tracking etc in case of a load generation or stress testing. The target will have single-step results by their correlation-id which makes sense.

Also it will be easy to project throughput, avg response delay etc.

e.g.
GET: https://somehost:443/api/customers
Sometime works for certain hosts.
Other times it fails with following exception, might be due to the Certificate issue.

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

or

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

What is the way out ?

Can you create a new client or change the http client to accept the certificate?

Any generator for it?
e.g. ${random.uuid} types ?

• Literally output of java.util.UUID#randomUUID
  e.g. 123e4997-e99b-92d3-a456-996642441212

How to assert both are same? @authorjapps
e.g.

Actual Response:
{
   "firstName": "Amit", //But in run time we dont know this,
   "callName": "Amit" ////But in run time we dont know this
}

In run time we dont know this is Amit, but we know firstName and callName are same.
How to assert ?

"assertions":{
   "firstName": "?", //what?
   "callName": "?" //what?
}

Defined the new place holder structure here.
e.g.

{
    "name": "add_500_employees_load",
    "url": "/google-emp-services/home/employees",
    "operation": "PUT",
    "request": {
        "body": "${JSON.FILE:common_files/request/500_employees_each_with_5_addresses.json}"
    },
    "assertions": {
        "status": 200
    }
}

More:

Hello, I am using this in a financial project as it it's quite easy and helps to write quality tests.
How does zerocode support spike chart showing PASS/FAIL result of the tests run? Can you plugin a dashboard to the CSV report?

How to trace a test into the log file?- e.g.
In case I need to know which particular step failed in a scenario just by looking at the chat/report? Color code will be useful! I don't want to see a aggregated fancy chart and drill down to get into it.

I am ok with optional time delay counter value if it provides against step info, not against the test scenario.
Anyway JUnit, maven sure file or maven-fail-safe plugin provides Test case wise and scenario wise report out of the box in a tabular format, but only assertion failures level, not enough to trace a TestCase.

Any pointer towards this will help, Thanks

• Explain the problem
• Explain proposed solution
• Explain usage with examples

todo--

When there were no reports generated, it should have been validated and appropriate message shown to the user. e.g target/zerocode-test-reports was not present or generated or permission issue was encountered.

Is it possible to fix this aspect of the library?

But below error breaks the reports generation, as -

Stacktrace was: java.lang.NullPointerException at java.util.Objects.requireNonNull(Objects.java:203) at java.util.Arrays$ArrayList.<init>(Arrays.java:3813) at java.util.Arrays.asList(Arrays.java:3800) at org.jsmart.zerocode.core.report.ZeroCodeReportGeneratorImpl.getAllEndPointFilesFrom(ZeroCodeReportGeneratorImpl.java:201) at org.jsmart.zerocode.core.report.ZeroCodeReportGeneratorImpl.readZeroCodeReportsByPath(ZeroCodeReportGeneratorImpl.java:174) at org.jsmart.zerocode.core.report.ZeroCodeReportGeneratorImplTest.testReportFolderNotPresentInTarget_validation

Review comments from- #62

 authorjapps - Owner
cover with test and create a sample please!

Work branch: issue6_number_equals
i.e. https://github.com/authorjapps/zerocode/tree/issue6_number_equals

e.g. this should work

      "assertions": {
        "status": 200,
        "body": {
          "id": "$NUMBER.EQ.1",
          "name": "Shower",
          "availability": true
        }
      }

also, the following should PASS

"assertions": {
        "status": 200,
        "body": {
          "id": "$NOT.EQ.${$.another_get_call.response.body.id}",
          "name": "Shower",
          "availability": true
        }

See implementation of $EQ.
Trace code: 01_verification_test_cases/06_number_not_equals_test.json

• Spike chart is disabled by default
• When Spike chart is enabled, provide a link on the top of the page of interactive chart

This will help the testers to navigate easily to spikes to figure out the most delayed test-steps and the failed test steps without going through the entire interactive html.

This is an enhancement.

Refer the ReportListener code for this.

Example of basic auth - Requested by Charann_twitter in gitter.

GIVEN I have basic username and password for a web server
WHEN I fire my http request with basic auth header
THEN I will be authorised to the resources

GIVEN I have basic username and password for a web server
WHEN I fire my http request with WRONG basic auth header
THEN I will not be authorised

Charaan needs an example of a custom HttpClient with auth header injected from a configurable properties file, so that he can use it across his test suite. This means he doesnt have to add it every time in the test steps header.

For SOAP messages "Content-Type":"text/xml" is generally set. How to override this in the request?

POST /endpoint.pl HTTP/1.1
Content-Type: text/xml
Content-Length: 167
SOAPAction: urn:example-org:demos#Method

<s:Envelope 
  xmlns:s='http://schemas.xmlsoap.org/soap/envelope/' >
 <s:Body>
  <m:Method xmlns:m='urn:example-org:demos' />
 </s:Body>
</s:Envelope>