leancodepl / patrol

Flutter-first UI testing framework. Ready for action!

Patrol

Simple yet powerful Flutter-first UI testing framework overcoming limitations of flutter_test, integration_test, and flutter_driver. Created and supported by LeanCode.

Learn more about Patrol:

• Our extensive documentation
• The article about the test bundling feature in Patrol 2.0
• The first stable 1.0 release article
• The article about the first public release

Patrol custom finders

Flutter's finders are powerful, but not very intuitive to use.

We took them and made something awesome.

Thanks to Patrol's custom finders, you'll take your tests from this:

testWidgets('signs up', (WidgetTester tester) async {
  await tester.pumpWidget(AwesomeApp());
  await tester.pumpAndSettle();

  await tester.enterText(
    find.byKey(Key('emailTextField')),
    '[email protected]',
  );
  await tester.pumpAndSettle();

  await tester.enterText(
    find.byKey(Key('nameTextField')),
    'Charlie',
  );
  await tester.pumpAndSettle();

  await tester.enterText(
    find.byKey(Key('passwordTextField')),
    'ny4ncat',
  );
  await tester.pumpAndSettle();

  await tester.tap(find.byKey(Key('termsCheckbox')));
  await tester.pumpAndSettle();

  await tester.tap(find.byKey(Key('signUpButton')));
  await tester.pumpAndSettle();

  expect(find.text('Welcome, Charlie!'), findsOneWidget);
});

to this:

patrolTest('signs up', (PatrolIntegrationTester $) async {
  await $.pumpWidgetAndSettle(AwesomeApp());

  await $(#emailTextField).enterText('[email protected]');
  await $(#nameTextField).enterText('Charlie');
  await $(#passwordTextField).enterText('ny4ncat');
  await $(#termsCheckbox).tap();
  await $(#signUpButton).tap();

  await $('Welcome, Charlie!').waitUntilVisible();
});

Learn more about custom finders in the docs!

Patrol's custom finders are also available standalone in the patrol_finders package.

Patrol native automation

Flutter's default integration_test package can't interact with the OS your Flutter app is running on. This makes it impossible to test many critical business features, such as:

• granting runtime permissions
• signing into the app which through WebView or Google Services
• tapping on notifications
• much more!

Patrol's native automation feature solves these problems:

void main() {
  patrolTest('showtime', (PatrolIntegrationTester $) async {
    await $.pumpWidgetAndSettle(AwesomeApp());
    // prepare network conditions
    await $.native.enableCellular();
    await $.native.disableWifi();

    // toggle system theme
    await $.native.enableDarkMode();

    // handle native location permission request dialog
    await $.native.selectFineLocation();
    await $.native.grantPermissionWhenInUse();

    // tap on the first notification
    await $.native.openNotifications();
    await $.native.tapOnNotificationByIndex(0);
  });
}

CLI

See packages/patrol_cli.

The CLI is needed to enable Patrol's native automation feature in integration tests. It also makes development of integration tests much faster thanks to Hot Restart.

To run widget tests, you can continue to use flutter test.

Package

See packages/patrol.

Patrol contracts generator

1. (Optionally) add new request type:

class OpenAppRequest {
  late String appId;
}

2. Add new method to NativeAutomator:

abstract class NativeAutomator<IOSServer, AndroidServer, DartClient> {
  ...
  void openApp(OpenAppRequest request);
  ...
}

3. Run gen_from_schema script, few files will be updated

Develop patrol_cli

If you have previously activated patrol_cli run:

dart pub global deactivate patrol_cli

then

cd packages/patrol_cli
flutter pub global activate -s path .

patrol's Issues

WebView

• #26

Fix every APK artifact version having different certificates

After update, first maestro drive fails with INSTALL_PARSE_FAILED_INCONSISTENT_CERTIFICATES.

It's easy to fix by:

$ adb uninstall pl.leancode.automatorserver.test
$ adb uninstall pl.leancode.automatorserver

but it should be automated

Create default config file

Don't overwrite `maestro.toml` if it exists

Turn on/off internet connection on the device

Add option for defining desired device capabilities

What if the user wants to run tests on all Android devices?
What if the user wants to run tests on all devices that can be called "a tablet"?

Appium has this kind of device capabilities definition, maybe we want to do something like that, obviously in a simplified form (especially in the beginning)

AC: user can select any device with Android version 12

Find out what method of screenshot taking is the best

Can we use flutter screenshot or maybe we'll have to use the screenshot server like Appium does

Login to a customer project using WebView

TBD a video from the process

Bootstrap

• #19
• #20

Remove server .apk from git

And add it to GitHub Releases.

Improve `maestro_cli` versioning process

Currently, to bump maestro_cli version, we have to change it in:

• packages/maestro_cli/pubspec.yaml
• packages/maestro_cli/CHANGELOG.yaml
• packages/maestro_cli/lib/src/common/constants.dart
• AutomatorServer/app/build.gradle

This is suboptimal.

Device selection

• #14
• #16
• #15
• #31

Maestro Dart API

• #25
• #29
• #1

Read a single push notification

Introduce our own WidgetTester

AC: user doesn't have to create Automator's WidgetTester themselves, it can be injected automatically like WidgetTester already is.

Simple selector engine

Use `ktlint` to enforce Kotlin code style for `AutomatorServer`

it's cool

https://github.com/pinterest/ktlint

Support running on multiple devices simultaneously

☂ Config file

• #12
• #19
• #14
• #16

Close keyboard if opened

There's no easy way to do this using UiAutomator.

There's a hack though. An ugly one, but it works it works only through ADB.

Here's the link

View logs from `pl.leancode.automatorserver`

If we pass maestro drive -vv ("very verbose"), then maestro_cli does adb logcat and prints to stdout (or adb -s <device_id> logcat, when there is more than 1 active emulator).

We'll have to filter the logs, because there are tons of them

Fix tests failing because of ADB daemon not being active on first test run

Reported by @jBorkowska

Tap on the first push notification from target app

Load config from TOML file

It'd be convenient to have an extensible method for parsing config.

Split public API and CLI into separate packages

spec takes an interesting approach – they have packages/spec that developers import and use, and they also have packages/spec_cli.

CLI is activated by dart pub global activate spec_cli.

I think we could do the same to clearly separate what belongs to the CLI and what belongs to the public API.

cc @mateuszwojtczak

Readme/docs for QAs

rename `package:maestro` to `package:maestro_test`

Wrap Flutter Driver API methods

The goal is to have Flutter Driver API method wrappers in our Automator API, that do necessary wait/sync/pump jobs so the user doesn't have to write them and produce boilerplate code.

Example:

final counter = tester.firstElement(find.byKey(const ValueKey('counterText'))).widget as Text;

vs

final counter = tester.findByKey<Text>('counterText');

AC: user doesn't have to use any methods beyond Automator API to find a widget by key, tap it / type input and scroll

• Finders (by key, text, type)
• Tap
• Type input
• Scroll

Waiting for async UI actions (for existing features)

Windows support

Go to Gmail and handle reset password flow

Create branding

It'd be nice if we had some basic branding prepared when we release Maestro to the public.

Examples:

• https://github.com/felangel/bloc
• https://github.com/felangel/mason
• https://github.com/invertase/melos

Basically an icon + some colors :)

Support for iOS

• #23

Add native swipes

Create basic driver and test files

AC: Basically, we'd like to run:

maestro bootstrap
maestro drive

and get a successful example test drive.

Split `integration_test/app_test.dart` into 2 variants – generic and for counter app

Proposed usage:

Generic

$ maestro bootstrap

Takes $projectName from pubspec.yaml and uses it to paste the following line into integration_test/app_test.dart:

import 'package:$projectName/app.dart';

We'll have to assume that the project has app.dart file where class App extends Widget is located.

We'll also have to provide providers (e.g using GlobalProviders), if necessary. This requires some testing on real-world examples.

For counter app

$ maestro bootstrap --for counter

The current behavior remains unchanged.

Add support for using existing apk

This will free us from having to do a Gradle build on each maestro drive.

flutter drive has --use-application-binary option, which takes path to APK as an argument.

Testing Biometrics features

Wrap `adb` in Dart

This is just an idea that came to my mind today, might not necessarily make much sense.

We could create a small Dart package wrapping adb tool, so instead of writing adb commands as strings, we'd just use nice Dart API.

In the beginning, we'd implement just small subset of functionality of, let's say, Activity Manager.

final port = 5554;
final adb = Adb(port);

final apkPath = "./app-release.apk";
adb.install(apkPath);

final packageName = "pl.leancode.someApp";

adb.activityManager.start(packageName);

I've seen adb command only a few times in the codebase, so this is probably an overkill.

Push notifications

• #21
• #22
• #157

Taking screenshots and recording videos

• #24

First PoC for iOS simulator (home button and automation server)

We have to do the research about how we can design automator server & communication on iOS side and how much it has to differ from the Android.

Add config option for multiple devices array

AC: user can use --devices option in maestro drive --devices=MyDevice1,MyDevice2 to select devices.

Automatically release artifacts

I think we should follow this tagging convention:

• automator-v1.2.3 for AutomatorServer
• maestro-v1.2.3 for packages/maestro (or maestro_test, see #42)
• maestro_cli-v1.2.3, for packages/maestro_cli

This will make it easy to create some release.yaml GitHub Action which will build and upload server.apk and automator.apk to Azure (or GitHub Releases, in the future).

And also we should somehow make all 3 projects require the same version from each other, to avoid any problems that might arise from version mismatches.

Rewrite history to delete `.apk`s in git

.apks are deleted in #41 but they're still in git history.

We need to remove them from history.

We can also disable LFS afterwards.

Log in with Facebook flow

This is a bit of a spike, but it's necessary to do a research about how we can operate with WebViews so later we figure out the correct API tradeoffs.

https://medium.com/azimolabs/android-webview-automation-testing-6311df8fe42e

Add option for setting device (like in Flutter) which is compatible also with ADB

AC: user can use -d option in maestro drive -d MyDevice to select a device

Use default parameter values from config file

Supported entries:

• host
• port
• driver file path
• test file path

AC: if user sets a value for the key in the config and then runs maestro command without this key:value pair specified, then a value from config is going to be used.

Add `maestro drive --debug` flag

Scope:

• use $MAESTRO_ARTIFACT_PATH/server.apk and $MAESTRO_ARTIFACT_PATH/instrumentation.apk when --debug flag is passed
• create some "copy artifacts" Gradle task which would copy artifacts to $MAESTRO_ARTIFACT_PATH

Will make development easier.

Let users change `~/.maestro`

If env var MAESTRO_ARTIFACT_PATH is set, then we'll use it. If it's not set, then we'll fallback to $HOME/.maestro (current behavior).