automattic / dot-handy

Make your life testing WordPress.com things easier.

dot-handy

Just a handy utility to make your life testing dotcom things easier.

As dotcom developers, we often need to configure our browser instance in certain ways and perform repetitive actions upon a task, so this utility aims to ease the burden by:

1. Let you put the browser configuration you need to apply for what you are working on into a configuration file, so you don't need to tweak them again and again. Say goodbye to messing up with your browsers for daily uses.
2. Manage your own browser automation scripts for repetitive tasks in a way that's less involving than a formal e2e test.

dot-handy uses Playwright internally to implement the above two concepts as configuration and actions.

Configuration here simply means browser configuration you want to use to initiatialize your browser instance. e.g. browser type, locale, cookies, and local storage. Except for general browser settings, there are also dotcom-specific properties available like --currency. Configuration can be supplied through either a JSON configuration file or commandline options. A configuration file is first looked up from local-configs directory and then configs directory. The coresponding properties supplied by the commandline options will override. -C or --config-files options are used to supply config files; the corresponding commandline options for the properties are a bit ad hoc. For the most up-to-date list of properties and their commandline options, referring to the source code will be the best.

Actions here refers to browser automation scripts that will be run after initiating. Currently they can only be supplied by -A or --action-files commandline option. e.g. yarn start -A new-user pick-free-domain pick-free-plan means to open a new browser instance and then to create a new user, pick a free domain, and then choose the Free plan. It's kind of limiting right now because we can't use them in-between our own manual testing. An action file is looked up in a similar manner like configuration files: local-actions will be looked up first, and then actions directory. Actions can require certain configuration properties to work, like the action for setting ExPlat variations.

Getting started

If you have yarn installed already, running yarn install should be all that you need to start playing it.

Sample Use Cases

1. Start a browser instance of chromium, webkit, or firefox

> yarn start                 # default to chromium
> yarn start -B webkit
> yarn start -B firefox

2. Open the local calypso

> yarn start -E local

3. Set locale

> yarn start --locale "zh-TW"

4. Set currency

> yarn start --currency "PHP"

5. Login as a test account

> yarn start -A login --username TestUserName --password YourSuperSecretPassword

or, you can supply username and password through a local configuration file. It shouldn't be in the shared configs file because ... it's just not a good idea. A configuration file with the same name as the action file will be loaded by default if no other is supplied. e.g. putting the following in local-configs/login.json:

{
	"username": ".......",
	"password": "..................."
}

and then simply yarn start -A login will do.

6. Signup for a new Business account.

1. Enable the store sandbox and sandbox the public API. Otherwise the dummy credit card info won't work.
2. Create a local-configs/new-user.json with newUserGmailPrefix and password field. This new-user action relies on the gmail + trick to create an account, and suppliying a password is required. It's easy to put a random password, but that has given me a hard time closing test accounts. Thus it is implemented this way for now. e.g. Mine looks like this:

{
        "newUserGmailPrefix": "supertestingemail",  <- this is not real
        "password": "MySuperSecretPassword"   <- this is also not real, of course.
}

3. Run yarn start -A new-user pick-free-domain pick-business-plan checkout

This will sign up a new account with the Business plan by appending a random string by the gmail + trick. e.g. in this case, it could be [email protected].

Caution: By doing this rapidly, your test email could be blacklisted. The real blacklisting conditions are unknown to me though. I have a test email that never gets blacklisted while some others are not so lucky.

7. Create a test account and set ExPlat variation

At the moment of writing this, the configuration properties for setting ExPlat variations can't be supplied by commandline options. Thus, we'd need to create a local config for it. e.g. Add a local-configs/test-explat.json like:

{
        "explatExperiments": [
                [ "explat_experiment_slug", "treatment" ]
        ]
}

then run:

> yarn start -A set-explat new-user pick-free-domain pick-free-plan -C test-explat

8. Create my own handy config that isn't appropriate to put in a public repo

I often want to login to one of my test accounts and enable the analytics debug log, thus I have a local-configs/mine.json like this:

{
  "username": "..........",
  "password": ".............",
	"localStorage": [
		[ "debug", "calypso:analytics:*"]
	]
}

then I'll run the following command to begin with:

> yarn start -A login -C mine

9. Close a test account which owns only one simple site.

yarn start -A login close-account --username xxx --password yyy

It's also possible to combine this with some good old shell trick to close test accounts in batch. Let's say all the test accounts are having the same password yyy and they are all listed in a plain text file, test-accounts, line-by-line. Then this will do:

> cat test-accounts | xargs -L1 yarn start -A login close-account --password yyy --username

10. Use with GNU parallel to run several instances side by side

When you have to compare several variations side by side, combining dot-handy with GNU parallel can be very powerful. On a macOS, run brew install parallel will have it installed for you.

e.g.

Comparing LOHP side by side per 3 different currencies:

> parallel ::: "yarn start --currency TWD" "yarn start --currency PHP" "yarn start --currency EUR"

Comparing the control and the treatment variation of an ExPlat variation:

> parallel ::: "yarn start -A set-explat login -C test-account-1 exp-1-control" "yarn start -A set-explat login -C test-account-2 exp-1-treatment"

We all know the domain resolution of webkit isnt' affected by /etc/hosts. Thus it can happen to be a convenient way to compare sandboxed vs unsandboxed version:

> parallel ::: "yarn start -B webkit" "yarn start"

dot-handy's Issues

Idea: configurable host resolution rules.

One common thing we do is to point certain parts of WP.com to our developer sandbox. It would be very handy to compare if we could simultaneously run a browser instance resolving to the developer sandbox and the other one resolving to the production.

The challenge here is that there doesn't seem to be a reliable way across all the browsers for this: microsoft/playwright#2767.

One way guaranteed to work is to run through a docker container, but I'd try not to go that heavy-weighted route unless absolutely necessary.

Add support for URL query strings.

Since there are quite a few things across WP.com that can be configurable via URL query strings, e.g. the feature flags, it should be configurable through a configuration property.

How to smart navigate to `initialPath` when necessary?

Currently, an action uses initialPath property just for checking whether it's time to run the sequence of Playwright script in its function body. It's safe and reasonable when the action depends the previous action to navigate to the right path for it. e.g. the series actions for the signup flow, pick-free-domain, checkout, etc.

However, for cases that we just want to go right in the correct path, we will need to navigate there explicitly. e.g. close-account has this line:

await page.goto( getRootUrlFromEnv( extra.config.env ) + '/me/account/close' );

The downside is that these actions with explicit navigation won't be able to chain with other actions, making them inflexible.

How could we make it smarter when necessary?

Action: remove-purchase doesn't handle the case of Quick Start Session.

This issue is just for leaving a note here. Currently there is a bug causing the subscription removing button to be missing sometimes. We should fix that instead of making this action smarter.

Unify the schema of configurable properties.

Currently dot-handy provides two ways to configure: commandline parameters and configuration files. However, they are connected loosely in a hard-coding way. The configuration files are flexible JSON files that can be written arbitrarily, and the overrides from the command line are bound to those properties through hard-coding code in main.js:

// TODO: this shouldn't be too hard to generalized to enable complete overriding of config flags through commandline params. I should consider to implement a config schema.
const parseOverrides = ( argv ) => {
	const eligibleParams = [
		'browser',
		'cookies',
		'currency',
		'env',
		'path',
		'username',
		'password',
		'localstorage',
		'locale',
	];

	const overrides = {};

	for( const key of eligibleParams ) {
		if ( argv[ key ] ) {
			// FIXME: terrible!
			if ( key == 'localstorage' || key == 'cookies' ) {
				overrides[ key ] = JSON.parse( argv[ key] );
				continue;
			}
			overrides[ key ] = argv[ key];
		}
	}

	return overrides;
};

Ideally,

1. There should be a unified schema defining what's available in a configuration object.
2. How configuration properties bound to command line parameters won't need to be coded explicitly like this.

Inconsistent-looking `require` module paths.

Currently all the module paths are written in relative paths, which looks inconsistent across the codebase, and is one extra mental burden to keep track of. e.g. in main.js:

const {
	setLocalStorage,
	initialize,
} = require( './lib/init.js' );
const { readConfigFiles, mergeConfig } = require( './lib/config.js' );
const { readActionFiles } = require( './lib/action.js' );
const { getRootUrlFromEnv, parseNonSpaceSeparatedList } = require( './lib/misc.js' );

in any of the action files, e.g. close-action.js:

const { createAction, readActionFile } = require( '../lib/action.js' );

It'd be better if the module searching path can be set like Calypso does.

For calypso-specific actions, rely on calypso-e2e package instead of re-implementing them here.

Props to @lsl for this wonderful suggestion :)

Its a bit of a trade off but depending on the page and component definitions directly will get you free updates as the underlying code changes.

Refactor: the path resolving logic of actions and configs is similar. Can they be unified?

The path resolving of config files and of action files, although look different from code, the logic is exactly the same:

1. Look into the local file directory, i.e. local-actions and local-configs. These files are considered case specific that are not appropriate to commit to the public repo, so they are higher priority; sort of like various conf.d things on a Linux-like system.
2. If nothing, look into the shared file directory, i.e. actions and configs.

They look very different now because an action file is read by require(), while a config file is read as a plain json file. How could we unify them so we don't have maintain two path resolving code?

Unit tests for the config lib.

It's easy enough to write unit tests for yet its logic keeps growing without a safety net. Time to do that.

Unit tests for the action library.

Just like the title said, it's time to do it before something explodes.

Idea: find a way for running actions whenever needed.

Currently there is no way to interweave manual testing with the automated actions. It would make the actions be used in a more flexible way if we can initiate them whenever needed, e.g. filling the checkout form by running checkout action.

I think it's possible to implement an interactive commandline prompt or another floating GUI dialog communicating with the underlying node process, which in turn communicates to the browser instance.

Action parameters.

A very clear flaw of the current design is that actions can't be easily reused in sequence since they all read from the same config object. e.g. it's not possible to use navigate for navigating to 2 different URLs, it's also not possible to .

Actions should be able to define their own parameters, the main program should provide a way to elaborate the arguments, and finally runActions() should feed arguments accordingly.

The implementation won't be hard; how to design the command line option is more crucial. Specifically, for a sequence actions A, B, and C. How could users put down the arguments for A and C if B doesn't need one?