plu / pxctest

Execute tests in parallel on multiple iOS Simulators

CV - Johannes Plunien

Summary

• Working in Software Engineering since 2004, with focus on iOS since 8 years
• Had different roles from backend/frontend to DevOps to iOS engineer
• Living north of Munich with my wife and two daughters, working remotely for eBay Kleinanzeigen (Berlin) as an iOS Developer

I see myself as a generalist, not afraid to touch other technologies than what I was hired for. Often launching Android Studio, comparing the Android app to the iOS app in my current job to debug something, or to check with Charles Proxy how and when they make a request to the backend to ensure the two different platforms behave the same.

Recent Open Source Contributions

• cli/cli - GitHub command line interface - Support gh release edit
• apple/swift-package-manager - Swift Package Manager - Limit concurrent HTTP requests when downloading binary dependencies
• fastlane/fastlane - fastlane - Fix duplicate use_system_scm option

Experience

+++ means I got a lot in touch with that technology while being in this position, does not mean I'm an expert in it.

05/2014 - today: Senior iOS Developer @ eBay Kleinanzeigen (remote since 2016)

+++ Swift
++  Jenkins, Objective-C, Ruby
+   GitHub Actions, Go, Java, Groovy, TypeScript

11/2013 - 04/2014: Senior Software Engineer @ metaquark GmbH

+++ Objective-C, Ruby
++  Bamboo, Capistrano
+   Puppet

01/2012 - 10/2013: Senior Software Engieer @ XING

+++ Objective-C, Perl, Ruby
++  Capistrano, Chef, Jenkins
+   VMware

04/2011 - 12/2011: Software Consultant @ ConSol MENA

+++ ColdFusion, Perl
++  Microsoft SQL Server

07/2009 - 03/2011: Senior Software Engineer @ XING

+++ MySQL, Perl, Ruby
++  German, Jenkins
+   RabbitMQ

09/2004 - 06/2009: Software Consultant @ ConSol Software GmbH

+++ Perl
++  Oracle

Side Projects

Current

simctl - Ruby interface to xcrun simctl (used by fastlane) - not very actively maintained, at least no new features are being added currently.

Past

Between 05/2017 and 07/2018 I've built and maintained the iOS Poloniex App (and Android App) as my own product in my personal App Store Account. Was finally acquired by Poloniex in 07/2018. Also started the Android App and implemented the first few screens, until I finally asked a friend for help, who knows a lot more about Android than I do.

pxctest's Issues

Query: usage with another framework which is in java

Hi,

Can you please clarify if I can use this framework with my existing framework which is based in Java ?

Thanks in advance.

Regards,
Vikrma

Is there a new version later？

Time out waiting for all required services

Anyone seeing this issue?, my simulators are boots however the test never starts.

Error Domain=com.facebook.FBSimulatorControl Code=0 "Timed out waiting for all required services [com.apple.SpringBoard, com.apple.medialibraryd, com.apple.backboardd, com.apple.mobile.installd, com.apple.SimulatorBridge] to start" UserInfo={NSLocalizedDescription=Timed out waiting for all required services [com.apple.SpringBoard, com.apple.medialibraryd, com.apple.backboardd, com.apple.mobile.installd, com.apple.SimulatorBridge] to start}

run-tests UnitTest doesn't work

when I run test with .xctestrun file. it has UnitTest and UITest. I run only UnitTests.
But, output is NilError. What did I do wrong?

I did

xcodebuild \
    -derivedDataPath "$PWD/derivedData" \
    -scheme 'MyApp' \
    -workspace 'MyApp.xcworkspace' \
    -destination 'platform=iOS Simulator,name=iPhone 7 Plus,OS=10.3' \
    build-for-testing

and

pxctest \
    run-tests \
    --testrun derivedData/Build/Intermediates/CodeCoverage/Products/MyApp_iphonesimulator10.3-x86_64.xctestrun \
    --destination 'name=iPhone 5,os=iOS 10.3' \
    --destination 'name=iPhone 6,os=iOS 10.3' \
    --only UnitTests

Run different tests on different simulators in parallel

So I can run the same tests in parallel in different simulators but is it possible to run different tests at the same time?

headless simulator boot timeout

These two work fine:

$ pxctest boot-simulators --destination "name=iPhone 7,os=iOS 10.3 --deviceset /tmp/test-simulators"
$ pxctest boot-simulators --destination "name=iPhone 6s,os=iOS 9.3 --deviceset /tmp/test-simulators"

These two don't:

$ pxctest boot-simulators --destination "name=iPhone 6,os=iOS 8.4 --deviceset /tmp/test-simulators"
$ pxctest boot-simulators --destination "name=iPhone 6,os=iOS 8.3 --deviceset /tmp/test-simulators"

They both end up with the following error:

Timed out waiting for all required services [com.apple.SpringBoard, com.apple.medialibraryd, com.apple.backboardd, com.apple.mobile.installd, com.apple.SimulatorBridge] to start

$ xcodebuild -version
Xcode 8.3.2
Build version 8E2002

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12
BuildVersion:	16A323

Unable to install pxctest via brew as doesn't build on OS X 11.6 using Xcode 8.2

Installation fails with the following log

==> Installing pxctest from plu/pxctest
==> Cloning https://github.com/plu/pxctest.git
Updating /Users/samhan/Library/Caches/Homebrew/pxctest--git
==> Checking out tag 0.3.1
==> ./scripts/build.sh /usr/local/Cellar/pxctest/0.3.1/libexec
Last 15 lines from /Users/samhan/Library/Logs/Homebrew/pxctest/01.build.sh:
2017-02-03 12:05:29 +0530

./scripts/build.sh
/usr/local/Cellar/pxctest/0.3.1/libexec

/private/tmp/pxctest-20170203-2407-4xnvuf/Dependencies/Commander /private/tmp/pxctest-20170203-2407-4xnvuf
Cloning https://github.com/kylef/Spectre.git
git: The build tool has reset ENV; --env=std required.
swift-package: error: No version tag found in (/private/tmp/pxctest-20170203-2407-4xnvuf/Dependencies/Commander/Packages/Spectre.git) package. Add a version tag with "git tag" command. Example: "git tag 0.1.0"

READ THIS: https://git.io/brew-troubleshooting
If reporting this issue please do so at (not Homebrew/brew):
  https://github.com/plu/homebrew-pxctest/issues

Has anyone used partitions?

Has anyone used partition? if so, is it working fine?I would also like to implement splitting my scheme into 2 simulators, I am using a different way but I think this would be easier.

My tests require photos in the Camera Roll but this is empty

Some of my tests have a requirement to attach photos. For this we have a number of photos stored in the Camera Roll. We then can attach a photo in our app. However, we delete the simulator each time (starting fresh each time we found to be more reliable) and there are no photos in the Media folder of the device type so when we go to add a photo as an attachment in our app the tests fail. Would it be possible for the media folder to be pre-populated with a sample photo? I have looked at adding something like:
xcrun simctl addmedia {UUID} ./Automation/Media/*.jpg or
xcrun simctl addmedia booted ./Automation/Media/*.jpg
Any suggestions on the best approach ? Thank you for any help.

Generate coverage data from each of the tests

Issue type: Feature Request

Why:

I need to gather coverage data from running tests.

How:

This is available in xcodebuild using the following:

xcodebuild \
    -derivedDataPath /tmp/some/directory \
    -enableCodeCoverage YES \
   -destination 'platform=iOS Simulator,name=iPhone 6,OS=10.0' \
   -xctestrun path/to/file.xctestrun \
   test-without-building

to extract the coverage data with LLVM

xcrun llvm-cov report -instr-profile /tmp/some/directory/Build/Intermediates/CodeCoverage/Coverage.profdata /tmp/some/directory/Build/Intermediates/CodeCoverage/Products/Debug-iphonesimulator/Example.app/Example

I don't know if its possible to pass xcodebuild those command line arguments using pxctest or if it is something that fbxctest would need to incorporate (or neither).

Support target names in --only option

• Change --only to VaradicOption
• Support TARGET[:Class/case[,Class2/case2]]

Cannot launch simulator

I have a config like below:

rm -rf /tmp/IS24_PR/test-simulators
mkdir /tmp/IS24_PR/test-simulators

pxctest boot-simulators \
--deviceset /tmp/IS24_PR/test-simulators \
--destination 'name=iPhone 7,os=iOS 10.3' \
--destination 'name=iPhone 6,os=iOS 9.3'

pxctest run-tests \
--testrun /tmp/IS24_PR/derivedData/Build/Intermediates/CodeCoverage/Products/ImmoScout24-Alpha_iphonesimulator10.3-x86_64.xctestrun \
--deviceset /tmp/IS24_PR/test-simulators \
--destination 'name=iPhone 7,os=iOS 10.3' \
--destination 'name=iPhone 6,os=iOS 9.3' \
--only 'ImmoScout24-AlphaTests'

It can run quite well for some time. But then I got this error.

NSUnderlyingError=0x7f91bc001b40 {Error Domain=com.apple.CoreSimulator.SimError Code=163 "Unable to boot device because it cannot be located on disk." UserInfo={NSLocalizedRecoverySuggestion=Use the device manager in Xcode or the simctl command line tool to either delete the device properly or erase contents and settings., NSLocalizedDescription=Unable to boot device because it cannot be located on disk., NSLocalizedFailureReason=The device's data is no longer present at /path/to/device

Do you have any idea how to fix it?

Application not running error on parallel tests

Seeing this error somewhat frequently, not sure how its cropping up...

UI Testing Failure - Application is not running, unable to get Accessibility data. Did you call -launch?

Unable to install via homebrew

This is the output of the command

brew tap plu/pxctest
brew install pxctest

Output:

==> Installing pxctest from plu/pxctest
==> Cloning https://github.com/plu/pxctest.git
Updating /Users/ignazioc/Library/Caches/Homebrew/pxctest--git
==> Checking out tag 0.2.6
error: no such remote ref f64bef8536fcfc7998df68846460e5adf93be347
Fetched in submodule path 'Dependencies/FBSimulatorControl', but it did not contain f64bef8536fcfc7998df68846460e5adf93be347. Direct fetching of that commit failed.
Error: Failed to download resource "pxctest"
Failure while executing: git submodule -q update --init --recursive

I was able to install only using the HEAD commit.

➜  ~ brew install pxctest --HEAD
==> Installing pxctest from plu/pxctest
==> Cloning https://github.com/plu/pxctest.git
Updating /Users/ignazioc/Library/Caches/Homebrew/pxctest--git
==> Checking out branch master
==> ./scripts/build.sh /usr/local/Cellar/pxctest/HEAD-c4d456a/libexec
🍺  /usr/local/Cellar/pxctest/HEAD-c4d456a: 227 files, 9.9M, built in 50 seconds
➜  ~

Additional info:

➜  ~ brew --version
Homebrew 1.1.2
Homebrew/homebrew-core (git revision ac83; last commit 2016-12-01)
➜  ~ git --version
git version 2.9.3 (Apple Git-75)

Is there any way to re-run failed tests?

Is it possible to re-run failed tests and if not, would it take much effort to add this option?

Unable to install - error

Getting the following error, please advise!

brew tap plu/pxctest && brew install pxctest

`
/private/tmp/pxctest20161207-1608-wajtb0/Dependencies/Commander /private/tmp/pxctest20161207-1608-wajtb0
Cloning https://github.com/kylef/Spectre
git: The build tool has reset ENV; --env=std required.
swift-package: error: No version tag found in (/private/tmp/pxctest20161207-1608-wajtb0/Dependencies/Commander/Packages/Spectre) package. Add a version tag with "git tag" command. Example: "git tag 0.1.0"

`

Error when running the tests

After running pxctest run-tests the simulators launch but before the app is loaded

Error Domain=com.facebook.FBSimulatorControl Code=0 "Failed start test manager" UserInfo={NSLocalizedDescription=Failed start test manager, NSUnderlyingError=0x7fafc1d1d3a0 {Error Domain=com.facebook.XCTestBootstrap Code=0 "Failed to determine test runner process PID" UserInfo={NSLocalizedDescription=Failed to determine test runner process PID}}}

i get the above issue.

Using iphone 6 and 7 iOS 10.1 Simulators

What is the .xctestrun file for?

how to run different test targets in same version of simulators parallel

Hi
I want to run my test targets parallel in same version of simulators.Is that possible?
to be more clear what I want to do is
targetA -----ios 6 version 10.1
targetB------ios 6 version 10.1

can't find .xctestrun file

Hi
After running build command I get
/var/folders/sc/b19_wblj5bv5wyph7hw5y5b95jc55b/T/SchemeScriptAction-Qjp2Sy: line 2: $: command not found
** TEST BUILD SUCCEEDED **
And i cant see that build/Products file generated.Is there something i missed.
And I should inform you also i get the message below while installing it.
_

You are using OS X 10.12.
We do not provide support for this pre-release version.
You may encounter build failures or other breakages

_.

`/tmp/test-simulators` is not a `path`

When I try to run this command in my terminal:
pxctest boot-simulators --deviceset /tmp/test-simulators --destination 'name=iPhone 6’ --destination 'name=iPhone 7’

Its says /tmp/test-simulators is not a path.

My folder structure: Desktop > Github > pxctest > tmp > test-simulators

Open too many simulators (I wrote only 2 destinations)

Using this
pxctest
run-tests
--destination 'name=iPad Air,os=iOS 10.2'
--destination 'name=iPad Air,os=iOS 10.2'
--testrun "/Users/username/Library/Developer/Xcode/DerivedData/MyApp-erqmuczdcogowjdmkomhdxdfgnfa/Build/Intermediates/CodeCoverage/Products/MyApp_iphonesimulator10.2-x86_64.xctestrun"

I see more than 5 simulator opening and it keeps opening more, I have to kill command line to stop it.

'pxctest run-test ' fails to launch tests

I have followed exactly what is given in the RM file. How ever when I execute pxctest run-test, I was always given the error below:

Error Domain=com.facebook.XCTestBootstrap Code=0

it opens the application and immediately minimising the application and after while raises the above issue.

I tried pxctest on Xcode 9 beta and I get "Abort trap: 6" on every command

I tried pxctest on Xcode 9 beta and I get "Abort trap: 6" on every command.

cbennett-70ap:iOSxm8 i58288$ sudo xcode-select -s ../../.././../../../../Applications/Xcode-beta.app/Contents/Developer/
Password:
cbennett-70ap:iOSxm8 i58288$ pxctest     run-tests     --destination 'name=iPad Air 2,os=iOS 10.3'     --testrun build/Products/iOSxm8_iphonesimulator11.0-x86_64.xctestrun
Abort trap: 6
cbennett-70ap:iOSxm8 i58288$ pxctest     run-tests     --destination 'name=iPad Air 2,os=iOS 11.0'     --testrun build/Products/iOSxm8_iphonesimulator11.0-x86_64.xctestrun
Abort trap: 6
cbennett-70ap:iOSxm8 i58288$ pxctest run-tests --help
Abort trap: 6
cbennett-70ap:iOSxm8 i58288$ 

Any support for Xcode beta?

Taking a long time to launch app

Since updating pxctest and xcode (to 8.3) I have noticed that testing takes a long time to actually begin. It looks like the app is installed on the simulator but then there is a few minutes delay until the app is actually launched. I have tried with both the latest pxctest version and pxctest --HEAD and see the same issue with both. Here is the command I'm running:

pxctest run-tests --destination 'name=iPhone 6s,os=iOS 10.3' --destination 'name=iPhone 6s Plus,os=iOS 10.3' --testrun "<path to .xctestrun>" --output ~/Desktop/test

Partial log output at point of delay:

2017-04-07 11:13:55.849858 pxctest[8314:71970] A4625354-FCE6-4834-BD82-0ED20668CD6C: TestManager Did Connect => TestManager Awaiting Result for (Test Host PID 8597 | Test Host Bundle com.apple.test.TravelerUITests-Runner | Session ID 9220E72D-3C1D-4EF2-B82A-BB3A931A17E4)
2017-04-07 11:13:55.878928 pxctest[8314:71970] A4625354-FCE6-4834-BD82-0ED20668CD6C: Test Plan Started
2017-04-07 11:13:55.879154 pxctest[8314:71970] A4625354-FCE6-4834-BD82-0ED20668CD6C: _XCT_didBeginExecutingTestPlan
2017-04-07 11:13:55.879551 pxctest[8314:71970] A4625354-FCE6-4834-BD82-0ED20668CD6C: Started initilizing for UI testing.
2017-04-07 11:16:11.458411 pxctest[8314:71970] 2E83107C-1BA9-46A9-A858-BA37C069A6DE: _XCT_testSuite:didStartAt:
2017-04-07 11:16:11.458937 pxctest[8314:71970] 2E83107C-1BA9-46A9-A858-BA37C069A6DE: _XCT_testSuite:didStartAt:
2017-04-07 11:16:11.459320 pxctest[8314:71970] 2E83107C-1BA9-46A9-A858-BA37C069A6DE: _XCT_testSuite:didStartAt:

I have also tried with a single simulator and I see the same issue.

How to shutdown gracefully the simulators for headless testing?

I am trying the headless testing with the boot-simulators command. It looks like new simulator devices are being created and booted in the provided folder (/tmp/test-simulators in your example).

How can I gracefully shutdown (and possibly delete) the newly created simulator devices after being done with the tests?

brew install pxctest fails

➜  github brew install pxctest                     
Updating Homebrew...
==> Installing pxctest from plu/pxctest
==> Cloning https://github.com/plu/pxctest.git
Updating /Users/dmytrogolub/Library/Caches/Homebrew/pxctest--git
==> Checking out tag 0.3.3
==> ./scripts/build.sh /usr/local/Cellar/pxctest/0.3.3/libexec
Last 15 lines from /Users/dmytrogolub/Library/Logs/Homebrew/pxctest/01.build.sh:
           ^
/tmp/pxctest-20170507-41569-1606bgn/PXCTestKit/Extension/String+PXCTestKit.swift:9:8: error: could not build Objective-C module 'CommonCrypto'
import CommonCrypto
       ^

** BUILD FAILED **


The following build commands failed:
	CompileSwift normal x86_64 /tmp/pxctest-20170507-41569-1606bgn/PXCTestKit/Reporter/TestReporter.swift
	CompileSwift normal x86_64 /tmp/pxctest-20170507-41569-1606bgn/PXCTestKit/Command/RunTests/RunTestsReporters.swift
	CompileSwift normal x86_64 /tmp/pxctest-20170507-41569-1606bgn/PXCTestKit/Command/RunTests/RunTestsPartitionManager.swift
	CompileSwift normal x86_64 /tmp/pxctest-20170507-41569-1606bgn/PXCTestKit/FileReader.swift
	CompileSwiftSources normal x86_64 com.apple.xcode.tools.swift.compiler
(5 failures)

If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
https://github.com/plu/homebrew-pxctest/issues

the failure seems to be happening due to the following line in module.modulemap of CommonCrypto.framework

module CommonCrypto [system] {
    header "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/CommonCrypto/CommonCrypto.h"
    export *
}

I have few versions of Xcode installed and none of them is in Applications folder. Changing the path to my Xcode location fixed the problem.
Not sure how to fix it for any Xcode location.

pxctest doesn't find tests in the test run

After updating to Xcode 8.3 I have switched to the master build of pxctest (because 0.3.1 doesn't work with Xcode 8.3) and tried to run our test suite. Pxctest reported Test run was empty, no tests were executed error.

If I run the test suite with xcodebuild using the command below, all the test files are found and executed .

xcodebuild test-without-building \
    -xctestrun './xcode-build-dir/Build/Intermediates/CodeCoverage/Products/MyApp_iphonesimulator10.3-x86_64.xctestrun' \
    -destination "OS=10.3,name=iPhone 7 Plus"

To run pxctest, I use the following command:

pxctest run-tests \
    --destination 'name=iPhone 7 Plus,os=iOS 10.3' \
    --testrun './xcode-build-dir/Build/Intermediates/CodeCoverage/Products/MyApp_iphonesimulator10.3-x86_64.xctestrun' \
    --output './xcode-build-dir/reports'

How to use it with appium ?

I am using appium to run my test on real iPhone device. Now I want to run it paralleled on multiple real iPhone devices. How can I use this tool is my project.

No log file created when using Xcode UI Tests

See discussion here:

dc69f3b

Duplicate symbols when installed via Homebrew

objc[31962]: Class _TtCs18_EmptyArrayStorage is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs24_ContiguousArrayStorage1 is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs17_CocoaSetIterator is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs24_CocoaDictionaryIterator is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs17NonObjectiveCBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs17_stdlib_AtomicInt is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs19_SwiftNativeNSArray is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs24_SwiftNativeNSDictionary is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs17_SwiftNativeNSSet is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs24_SwiftNativeNSEnumerator is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs18_SwiftNativeNSData is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs26_SwiftNativeNSCharacterSet is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs31_stdlib_ReturnAutoreleasedDummy is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs20_SwiftNativeNSString is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs19_NSContiguousString is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs40_SwiftNativeNSArrayWithContiguousStorage is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs21_SwiftDeferredNSArray is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs27_ContiguousArrayStorageBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs14_VaListBuilder is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtCs13VaListBuilder is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSError is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class SwiftObject is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftValue is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNull is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSArrayBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSDictionaryBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSSetBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSStringBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSEnumeratorBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSDataBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSCharacterSetBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftNativeNSIndexSetBase is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftCore.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtC8Dispatch16DispatchWorkItem is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftDispatch.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftDispatch.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtC10Foundation15NSSimpleCString is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtC10Foundation16NSConstantString is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtC10Foundation25NSFastEnumerationIterator is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtC10Foundation25_NSErrorRecoveryAttempter is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtC10Foundation20_SwiftNSCharacterSet is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _TtC10Foundation12_SwiftNSData is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib. One of the two will be used. Which one is undefined.
objc[31962]: Class _SwiftTypePreservingNSNumber is implemented in both /usr/local/Cellar/pxctest/0.2.1/libexec/PXCTestKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib and /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib. One of the two will be used. Which one is undefined.

Test Scheme Pre-Action not being executed

I've noticed that Pre-action scripts of test schemes do not get executed. It works when running with Xcode. Should this work with pxctest? Thanks in advance.

This is where I have the pre-action script:

Even with boot-simulators, devices show UI

Been trying this out with Xcode 8.2. Using boot-simulators seems to correctly put the simulators into the booted state (verified via xcrun simctl --path <simset> list). From that point, using run-tests causing the previously "headless" simulators to suddenly grow a head. I've checked & double checked the options I'm passing (even going as far verify them by debugging pxctest).

What else could this issue be?

FYI, we've fully automated our iOS builds with CI & CD using build-for-testing and 'test-without-building`. Would love to help out with this project to, hopefully drastically, reduce our integration times.

If the selected test class doesn't exists, raise an error.

I wold like to run only the tests in one class, therefore I run:

bundle exec rake test:unit[MyAwesomeTestClass]

A small issue happens if the class name is misspelled. In that case the project is still built, and the output looks like:

UnitTests - iPhone 5 iOS 10.2 - Finished executing 0 tests after 0.002s. 0 Failures, 0 Unexpected
UnitTests - iPhone 5 iOS 9.0 - Finished executing 0 tests after 0.001s. 0 Failures, 0 Unexpected
Total - Finished executing 0 tests. 0 Failures, 0 Unexpected

It clearly says Finished executing 0 tests but I would rather prefer a big red error message :P

Use pxctest to create simulators

Is it possible to create simulators in a specific location, e.g. --deviceset /tmp/test-simulators, without booting them in a headless state using boot-simulators?

I would like to be able to create simulators, then "run-tests" and delete the simulators once testing has completed.

Currently, I can use boot-simulators but am forced to run in a headless state using this.

pxctest - build and run-tests issue

Any idea what's going on here? The build succeeded but it didn't generate $PWD/build/Products/. xctestrun file. But an .xctest run file is generated at Build/Intermediates/CodeCoverage/Products/MyApp_iphonesimulator10.1-x86_64.xctestrun, when run this location getting an error as shown below.

`
Xcode 8.1
Build version 8B62

xcodebuild
-IDEBuildLocationStyle=Custom
-IDECustomBuildLocationType=Absolute
-IDECustomBuildProductsPath="$PWD/build/Products"
-scheme 'MyApp'
-workspace 'MyApp.xcworkspace'
-destination 'platform=iOS Simulator,name=iPhone 5,OS=10.1'
build-for-testing

pxctest \

run-tests
--destination 'name=iPhone 5,os=iOS 10.1'
--testrun Build/Intermediates/CodeCoverage/Products/MyAppUnitTests_iphonesimulator10.1-x86_64.xctestrun

Error Domain=com.facebook.XCTestBootstrap Code=0 "Failed to find test host application" UserInfo={NSLocalizedDescription=Failed to find test host application, NSUnderlyingError=0x7f845045a1d0 {Error Domain=com.facebook.FBControlCore Code=0 "Could not obtain Bundle ID for app at path /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Xcode/Agents/xctest" UserInfo={NSLocalizedDescription=Could not obtain Bundle ID for app at path /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Xcode/Agents/xctest}}}
`

build.sh not working

At latest master (0.3.2) build fails. (either via Xcode Cmd+B or ./script/build.sh fails)
I could work-around it by removing Spectre from embedded Commander.xcodeproj .
Homebrew fails with same error.

$ brew unlink pxctest && brew tap plu/pxctest && brew install pxctest
Unlinking /usr/local/Cellar/pxctest/0.3.1... 0 symlinks removed
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
amazon-ecs-cli                       jmxtrans                             rtv

==> Installing pxctest from plu/pxctest
==> Cloning https://github.com/plu/pxctest.git
Updating /Users/a14786/Library/Caches/Homebrew/pxctest--git
==> Checking out tag 0.3.2
==> ./scripts/build.sh /usr/local/Cellar/pxctest/0.3.2/libexec
Last 15 lines from /Users/a14786/Library/Logs/Homebrew/pxctest/01.build.sh:
<unknown>:0: error: no such file or directory: '/tmp/pxctest-20170331-10773-1wigcy1/Dependencies/Commander/.build/checkouts/Spectre.git--7655155069707042687/Sources/Context.swift'
<unknown>:0: error: no such file or directory: '/tmp/pxctest-20170331-10773-1wigcy1/Dependencies/Commander/.build/checkouts/Spectre.git--7655155069707042687/Sources/Expectation.swift'
<unknown>:0: error: no such file or directory: '/tmp/pxctest-20170331-10773-1wigcy1/Dependencies/Commander/.build/checkouts/Spectre.git--7655155069707042687/Sources/Failure.swift'
<unknown>:0: error: no such file or directory: '/tmp/pxctest-20170331-10773-1wigcy1/Dependencies/Commander/.build/checkouts/Spectre.git--7655155069707042687/Sources/Global.swift'
<unknown>:0: error: no such file or directory: '/tmp/pxctest-20170331-10773-1wigcy1/Dependencies/Commander/.build/checkouts/Spectre.git--7655155069707042687/Sources/GlobalContext.swift'
<unknown>:0: error: no such file or directory: '/tmp/pxctest-20170331-10773-1wigcy1/Dependencies/Commander/.build/checkouts/Spectre.git--7655155069707042687/Sources/Reporter.swift'
<unknown>:0: error: no such file or directory: '/tmp/pxctest-20170331-10773-1wigcy1/Dependencies/Commander/.build/checkouts/Spectre.git--7655155069707042687/Sources/Reporters.swift'
Command /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc failed with exit code 1

** BUILD FAILED **


The following build commands failed:
	CompileSwiftSources normal x86_64 com.apple.xcode.tools.swift.compiler
(1 failure)

If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
https://github.com/plu/homebrew-pxctest/issues

Installing from brew and boot-simulators command is unknown

After installing from brew according to the info in the README.md, the boot-simulators command is not available.

$ pxctest boot-simulators
Unknown command: boot-simulators

$ pxctest
Usage:

    $ pxctest

Commands:

    + run-tests
    + version

$ pxctest version
pxctest 0.2.9 (7bb7c7e)

Is it possible to generate a .plist file instead of junit/xml file

Is it possible to generate a .plist file with the test results instead of junit/xml file.?

Output to CircleCi is ......FF..........

I use --output command to output pxctest result to circle CI
but I want the JUnit file to be passed to Circle CI