mikeger / xcodeselectivetesting

Xcode selective testing: Run only tests relevant to the changeset.

Xcode Selective Testing

Run only tests relevant to the changeset.

Watch Video

What is it?

“Insanity is doing the same thing over and over and expecting different results.”

Albert Einstein, probably

Imagine we have the following dependencies structure:

If the 📦Login module is changed, it would only affect the 📦LoginUI and the 📱MainApp.

Does it make sense to test all the modules if we know only the 📦Login module is changed? No. We can only run 50% of the tests and get the same results.

This technique saves time when testing locally and on the CI.

Prerequisites

• Your project must have multiple targets or modules

Installation

Xcode

Add to Xcode as SPM dependency.

• Open your project or workspace in Xcode
• Select yout project in the file list in Xcode
• In the right menu select "Project", open tab "Package Dependencies"
• Select "+"
• In the new window, paste [email protected]:mikeger/XcodeSelectiveTesting in the search field
• Select project if necessary, put a checkbox on "XcodeSelectiveTesting" in the list
• Click "Add Package"

Using Swift Package Manager

Add .package(url: "[email protected]:mikeger/XcodeSelectiveTesting", .upToNextMajor(from: "0.9.3")) to your Package.swift's dependencies section.

Use SPM to run the command: swift run xcode-selective-test.

Using Mint

mint install mikeger/[email protected]

Manually

• Checkout this repository
• Compile the tool: swift build -c release

Integration

Use case: Swift Package Manager-based setup

In case you are using Swift Package Manager without Xcode project or workspace:

Run swift test --filter "$(swift run xcode-selective-test . --json | jq -r ". | map(.name) | join(\"|\")")"

NB: This command assumes you have jq tool installed. You can install it with Homebrew via brew install jq.

Use case: Xcode-based project, run tests locally

1. Install the tool (see Installation: Xcode)
2. Select your project in the Xcode's file list
3. Right-click on it and select SelectiveTestingPlugin
4. Wait for the tool to run
5. Run tests normally, SelectiveTesting would modify your test plan according to the local changes

Alternatively, you can use CLI to achieve the same result:

1. Run mint run mikeger/[email protected] YourWorkspace.xcworkspace --test-plan YourTestPlan.xctestplan
2. Run tests normally, XcodeSelectiveTesting would modify your test plan according to the local changes

Use case: Xcode-based project, execute tests on the CI, no test plan

• Requires jq installed (brew install jq)

1. Add code to install the tool
2. Use xcodebuild to run only selected tests: xcodebuild test -workspace Workspace.xcworkspace -scheme Scheme $(mint run --silent XcodeSelectiveTesting@provide-if-target-is-test-target --json | jq -r "[.[] | select(.testTarget == true)] | map(\"-only-testing:\" + .name) | join(\" \")")

Use case: Xcode-based project, execute tests on the CI, with test plan

1. Add code to install the tool
2. Add a CI step before you execute your tests: mint run mikeger/[email protected] YourWorkspace.xcworkspace --test-plan YourTestPlan.xctestplan --base-branch $PR_BASE_BRANCH
3. Execute your tests

How does this work?

1. Detecting what is changed

Git allows us to find what files were touched in the changeset.

Root
├── Dependencies
│   └── Login
│   ├── ❗️LoginAssembly.swift
│   └── ...
├── MyProject.xcodeproj
└── Sources

2. Build the dependency graph

Going from the project to its dependencies, to its dependencies, to dependencies of the dependencies, ...

Dependencies between packages can be parsed with swift package dump-package and dependencies between Xcode projects and targets can be parsed with XcodeProj.

BTW, This is the moment your Leetcode graph exercises would pay off

2.5. Save the list of files for each dependency

This is important, so we'll know which files affect which targets.

3. Traverse the graph

Go from every changed dependency all the way up, and save a set of dependencies you've touched.

4. Disable tests that can be skipped in the scheme / test plan

This is the hardest part: dealing with obscure Xcode formats. But if we get that far, we will not be scared by 10-year-old XMLs.

Command line options

• --help: Display all command line options
• --base-branch: Branch to compare against to find the relevant changes. If emitted, a local changeset is used (development mode).
• --test-plan: Path to the test plan. If not given, tool would try to infer the path.
• --json: Provide output in JSON format (STDOUT).
• --dependency-graph: Opens Safari with a dependency graph visualization. Attention: if you don't trust Javascript ecosystem prefer using --dot option. More info here.
• --dot: Output dependency graph in Dot (Graphviz) format. To be used with Graphviz: brew install graphviz, then xcode-selective-test --dot | dot -Tsvg > output.svg && open output.svg
• --verbose: Provide verbose output.

Configuration file .xcode-selective-testing.yml

It is possible to define the configuration in a separate file. The tool would look for this file in the current directory.

Options available are (see selective-testing-config-example.yml for an example):

• basePath: Relative or absolute path to the project. If set, the command line option can be emitted.
• testPlan: Relative or absolute path to the test plan to configure.
• exclude: List of relative paths to exclude when looking for Swift packages.
• extra/dependencies: Options allowing to hint tool about dependencies between targets or packages.
• extra/targetsFiles: Options allowing to hint tool about the files affecting targets or packages.

Support

Supported operating systems:

• macOS 12.0+ (Monterey) : Xcode 14.2 and above
• Linux: Swift 5.8 and above

Contributing

Contributions are welcome. Consider checking existing issues and creating a new one if you plan to contribute.

License

See LICENSE

Authors

• 🇺🇦 Michael Gerasymenko <mike (at) gera.cx>

Contributors

• Sam Woolf

If you like this product, consider donating to my hometown's charity project Monsters Corporation 🤝

xcodeselectivetesting's Issues

Do you plan to change you license to MIT ?

Hi,

Thanks for making this great library.

Do you plan to change your license to MIT? Because GPL is not acceptable for many companies.

Using a library under the GPL-3.0 (GNU General Public License, version 3.0) license in your app for distribution on the Apple App Store can be challenging due to the terms of the GPL-3.0 license, which is a strong copyleft license. The GPL-3.0 has certain requirements, including the obligation to make your source code available if you distribute a program that uses GPL-3.0-licensed code.

When you distribute an app through the Apple App Store, you don't typically provide access to the app's source code, which could be a problem when using GPL-3.0-licensed code. You would need to provide the source code for the entire app, not just the modified parts related to the GPL-3.0-licensed library.

Security Concerns with DependencyGraph

The DependencyGraph+DOT class has a couple of built in features which will cause significant security concerns for many companies.

Specifically, there is an option to renderToASCII, which would also send the project architecture information to another private site (dot-to-ascii.ggerganov.com/dot-to-ascii.php) with a URLRequest.

Also, it automatically uploads project architectural information to cdn.jsdeliver.net to use Mermaid for more visualizations. This seems to use open source versioning, but it does so by running a shell command that opens a browser and sends the architecture info to it through a hard-codes string url. It would be much preferable to use more common dependency definitions, like how you're doing with your other package dependencies (AEXML, etc.).

I'm sure the visuals are good and helpful, but as long as they are private or using unorthodox dependency management, this will be a significant obstacle for many companies to use. Even for basic architecture information, IP is closely guarded.

If these utilities could be added as standard open source project dependencies, instead of private websites, that would allow users to review the code for security concerns and build confidence. For many companies, that is a requirement before adopting a solution like this.

Alternatively, if the visualizations can't be ported into open source dependencies which don't require sending sensitive architectural info to private servers, perhaps those features could be removed to encourage adoption.

Get both staged and unstaged change set when analysing git repository

First of all, thank you for this great tool.

I was trying to create a git pre-commit hook using this script to run required unit tests before committing but the script always return 0 change set.

After taking a look at the Git command, it looks like that we need to use git diff --cached to get the staged changes.

For better coverage, I think we should get both staged and unstaged changes to analyse what tests should be executed.

Support multiple test plans

I received this request from fellow colleagues.

It is possible right now to execute the tool for every test plan individually, but it's not ideal since it would have to analyze the changeset several times.

One approach I was thinking about was to let pass an array of paths for the test plans so that the tool can process them all at once.

tool breaks when pr branch contains an & character

Its a bit of an edge case - but it did happen in our repo - A dev named his branch 'blah&blah', which broke the tool
I think because the tool uses git diff \(baseBranch)..\(currentBranch) if the branch name contains an & it is seen as a separate command.

Solution could be as simple as git diff '\(baseBranch)'..'\(currentBranch)'

Cannot run a diff with HEAD

For running on CI that creates a detached HEAD like GitHub actions I am unable to run the job. It returns the following error

fatal: ambiguous argument '<branch-name>..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Prior to running the command I did a shallow fetch of the base branch to make sure both branches are available.

Cannot detect a dependency from URL

If there is a change only in a dependency with no other source code changes then it is not detected. Updating on a new version of a dependency like Firebase for example should trigger tests to run for any frameworks that depend on it.

Use case: regression analysis

Develop a use case to support a version regression testing:

The tool can be potentially used to highlight changed packages/targets for the release, so it would be possible to know which modules have more changes and focus regression testing on them.

In this use case, it would not only be important to see IF the module was changed or its dependencies changed but also to see to which extent it was changed compared to other modules.