nativescript / functional-tests-core

Appium based framework for testing Android and iOS native mobile apps.

Functional Tests - Core Framework

General purpose automation framework for native mobile apps based on Appium.

Demo Project

NativeScript/functional-tests-demo

Features

• Manage emulators, simulators and physical devices.
  • Create emulators and simulators
  • Install/Uninstall applications
  • Start/Stop/Restart applications
  • Get console logs
  • Get screenshots
  • Simulate environment (like geo location for example)
  • [Android] Get performance stats (memory usage, startup time, application size)
  • All of the above managed via configs files
• Interaction with native apps on iOS Simulators, Android Emulators as well as physical Android and iOS devices.
  • Locate elements via accessibility of the mobile platforms
  • Cross-platform locators that work for both Android and iOS (and also with different backend automation)
  • Locate elements via images and OCR
  • Perform gestures
• Flexible image verification.
  • Verify your page looks as expected
  • Verify particular element on screen looks as expected
  • Allow setting tolerance as % and number of pixels
  • Logic to exclude status bar from comparison
  • Logic that helps when you have shadows or pixels that looks almost the same, but RGB is different
• Locating elements by image and OCR (based on Sikuli).
• Logs, Reports, Artifacts
  • Console log from devices, emulators and simulators
  • Appium server logs
  • Test execution logs
  • Screenshots
  • Junit style xml reports with meta info that helps generating html reports with images.

Supported Platforms

Host

Tests based on this framework can be executed from Windows, Linux and macOS machines.

However, there are some limitations:

• Testing on iOS Simulators and devices can be done only from macOS hosts.
• Android testing is possible from any hosts, however some features do not work on Windows:
  • Executing tests on multiple Android emulators/simulators simultaneously.
  • Create emulators feature.

Mobile

Tests based on this framework can be executed against Android Emulators, iOS Simulators, Android physical devices and iOS physical devices.

Supported versions:

• iOS: 8+
• Android: 4.2+

...and yes, we work on iOS 10 and Android 7.1 :)

Most likely we also work on Android < 4.2, but we can not confirm since we test only Andoird 4.2+

Prerequisites and Setup

Prerequisites

Prerequisites to run tests based on this framework.

• The latest Node LTS release
• JDK 1.8+
• Appium
• (Android only) Android SDK
• (iOS only) Latest Xcode

Environment Setup

Please read Environment Setup document.

NativeScript Functional Tests

This framework is used for testing all sample apps @NativeScript.

Tests are available in NativeScript/functional-tests repository.

Issues

All work-items (issues, features, questions) are handled in GitHub.

If you find valid issue, please log it (as good is the report as good is the chance to fix it).

Clarification:

We do not guarantee all issues will be fixed, but we will be happy to see pull requests.

Build and Contribute

Build

Gradle is used as build system.

Project use Gradle Wrapper so global installation of Gradle is not required.

Run tasks

• Unix-like: ./gradlew <task>
• Windows: gradlew.bat <task>

List of tasks:

• tasks - Displays the tasks runnable from root project.
• build - Assembles and tests this project.
• check - Runs all style checks in config/checkstyle/checkstyle.xml; outputs reports in ./build/reports/.
• fatJar - Assembles a jar archive with dependencies; outputs a .jar file in ./build/libs/.

Contribute

If you see an area for improvement, want to fix some git issues or just have an idea for a new feature, we will appreciate your pull requests

Just make sure check task passes successfully.

functional-tests-core's Issues

Query about usage of latest appium java client

Can you please clarify when are you planning to migrate to latest version of Appium Java client

https://github.com/NativeScript/functional-tests-core/blob/master/build.gradle#L14, is quite old.

Use the library in gradle build system

https://github.com/NativeScript/functional-tests-demo#test-results

Currently, the functional-tests-core has report structure based on a maven project. It should be independent from the build system in use.

Consider a global config file of the tests to setup this.

App not re-installed properly on iOS 8/9 Simulators

If you already have old application with name TestApp installed and then you want to test different app, which is also named TestApp the framework will not re-install the app on iOS8/9 Simulators and tests will run against old app.

Notes: Not reproducible for iOS 10 Simulators, Android emulators and devices.

Folder structure for Sikuli image procession is not correct.

Let's imagine in settings we have testAppName=hybridapp.apk.

If you want to get locate by images working now you should place images in:
resources\images\hybridapp.apk\Emulator-Api19-Default\<image name>

It should be in in something like:
resources\images\hybridappmap\Emulator-Api19-Default\<image name>

Extensions should be ignored and map suffix should be added.

Package does not exists

Hi,

I just can't load the .jar dependencies through my tests suite cause the dependency is not resolved

On the first line of my test page

import functional.tests.core.mobile.element.UIElement;

And when I launch the test with mvn clean test -P sim.iphone5.ios10

package functional.tests.core.mobile.element does not exist

I registered the .jar with mvn install:install-file already

Need Info about parallel run capability

Hi,

Can you please clarify below

1. if your framework support parallel run with help of selenium grid ? If yes how do you achieve it ?

2. I observed it has quite old appium java client version , any reason for including latest one ?

3. Do I need to use junit or testng ?

Thanks & Regards,
Vikram

Question about license

Hi,

the project has no LICENSE file.
While all your other project have the Apache License 2.0.

Was it just forgotten?

What license has this project?

Build system improvements

A few things to improve developer ergonomy

Maven conventions:

• Use org.nativescript as groupId
• Clear up the version confusion. Mostly use a -SNAPSHOT suffix, and only remove it when releasing a version. Functional-tests-demo/libs contains something which claims to be 0.0.5, so I would suggest changing 0.0.6-SNAPSHOT to avoid more confusion.
• Instead of copying the dependency to functional-tests-demo/libs, simply use the m2 approach: use mvn install to publish to the local m2 repository, and depend on org.nativescript:functional-tests-core in the demo.
• Release a 0.0.6 version to the maven central repository so you can use functional-tests-demo without having to build functional-tests-core yourself and without having to do mvn install:install-file

Respect Android SDK Tools 25.3.0

Changes:

• android avd command-line functionality replaced with new avdmanager tool.
• Obsolete/deprecated tools have been removed: android and others.

https://developer.android.com/studio/releases/sdk-tools.html
SDK Tools, Revision 25.3.0 (March 2017)

Not able to execute the test cases for iOS.

Hi,

I am trying to execute the test cases for the iPhone 7 simulator by using following command,

functional-tests-demo: mvn clean test -P nativeapp.sim.iphone7.ios10

but I am getting following error,

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building functional.tests 0.0.1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ functional.tests ---
[INFO] Deleting /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ functional.tests ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ functional.tests ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ functional.tests ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ functional.tests ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/target/test-classes
[WARNING] bootstrap class path not set in conjunction with -source 1.6
[INFO]
[INFO] --- maven-surefire-plugin:2.19:test (default-test) @ functional.tests ---

---

T E S T S

Running TestSuite
Tests run: 4, Failures: 2, Errors: 0, Skipped: 2, Time elapsed: 45.856 sec <<< FAILURE! - in TestSuite
beforeSuiteUIBaseTest(nativeapp.tests.SmokeTests) Time elapsed: 32.257 sec <<< FAILURE!
java.lang.RuntimeException: Appium server not running!

afterSuiteUIBaseTest(nativeapp.tests.SmokeTests) Time elapsed: 40.767 sec <<< FAILURE!
java.lang.NullPointerException

Results :

Failed tests:
SmokeTests>UIBaseTest.afterSuiteUIBaseTest:187 » NullPointer
SmokeTests>UIBaseTest.beforeSuiteUIBaseTest:95 » Runtime Appium server not run...

Tests run: 4, Failures: 2, Errors: 0, Skipped: 2

[ERROR] There are test failures.

Please refer to /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/target/surefire-reports for the individual test results.
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (default) @ functional.tests ---
[WARNING] Parameter tasks is deprecated, use target instead
[INFO] Executing tasks

main:
[mkdir] Created dir: /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/target/surefire-reports/html
[xslt] Processing /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/target/surefire-reports/testng-results.xml to /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/target/surefire-reports/html/index.html
[xslt] Loading stylesheet /Users/rohanpeshwe/Desktop/appiumTest/functional-tests-demo/resources/report/testng-results.xsl
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51.624 s
[INFO] Finished at: 2017-04-19T18:38:54+05:30
[INFO] Final Memory: 22M/170M
[INFO] ------------------------------------------------------------------------

I have followed all the steps provided on the blog,

https://www.nativescript.org/blog/nativescript-open-source-functional-testing-framework

I have just tweaked some configs because my xcode's version is 8.2.1 and updated configs are as follows:

appiumVersion=1.6.3
platformName=iOS
deviceName=iPhone 7 102
platformVersion=10.2
testAppName=TestApp.app
testAppArchive=nativeapp.tgz
deviceType=Simulator
simulatorType=iPhone 7

Node version
v6.9.5

Appium Version
1.6.3

Java version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

Can somebody please point me the right direction and let me know if something is missing by my side.