push-based / user-flow Goto Github PK
View Code? Open in Web Editor NEWπ¦ Combine Chrome tooling like Lighthouse userflows and DevTools reconder scripts in your CI
License: MIT License
π¦ Combine Chrome tooling like Lighthouse userflows and DevTools reconder scripts in your CI
License: MIT License
Error
> npx @push-based/user-flow -p packages/ngx-fast-icon-demo/perf collect
internal/fs/utils.js:312
throw err;
^
Error: EISDIR: illegal operation on a directory, read
at Object.readSync (fs.js:614:3)
at tryReadSync (fs.js:383:20)
at readFileSync (fs.js:420:19)
at readFile (C:\Users\micha\git\ngx-fast-icon\node_modules\@push-based\user-flow\src\lib\core\utils\file.js:13:38)
at readRcConfig (C:\Users\micha\git\ngx-fast-icon\node_modules\@push-based\user-flow\src\lib\core\rc-json\index.js:9:48)
Catch error and log that the rc path needs to point to file and not a directory.
Hey, I noticed that the versioning job in the CI is failing. Seems like your last tag is not attached to main
so semver fails (130839c).
I wonder how you end up with that state as it's not possible with the actual configuration. Are you doing manual steps that I ignore when you release?
Change the markdown table to use emojis instead of using the name of gather modes
Example
<span title="navigation">π§<span>
The setup command targets scaffolding of tests, budgets, .rc files etc.
It should focus on a good user experience from the CLI perspective. This means the user should be guided by questions and options to setup a testing repository for lighthouse user flows.
As we want to do a good job for our users I will map some challenges a potential user can have.
As there are different users let's define a persona first
In general this issue targets direct users of @push-based/user-flows, developer, performance engineers etc that install and run the CLI.
Personal data
Chang, 38, located in Brazil in a IT city, good developer in front-end development, tooling and some experience with node scripts with a high level of experience
Background
Heard about it on twitter and is curios what it really is.
Knowledge
He knows lighthouse already but is not a power user.
In general he sensed or know already that there is no good tool out there to measure runtime performance.
There is no puppeteer knowledge present in his bent, but he already wrote cypress tests and back then a small protractor tests suit.
The concept of Fixtures/Page-Objects etc. in testing architectures is nothing he uses on a daily basis but he knows the general idea and practical benefits.
Environment and Tooling
There is no CI setup for run-time performance in place in their infrastructure/company.
Not even for bootstrap performance with LHCI as the setup and maintenance was too time consuming.
Motivation
He is actively looking for new tools and tests them occasionally if the docs seem promising, in particular if he would already see how the user interaction and the resulting report is viewed.
He would even request some time to do a PoC on their product if the results are convincing and easy to explain to the product manager.
First contact and usage
In the afternoon he saw a tweet about the tool on twitter:
π£π£π£
I released push-based/user-flow 1.1.xa CLI that helps with @____lighthouse
Runtime #WebPerformance performance tests
- π CI integration
- π GH Actions
- πβπ¨PR comments
- π¦Ί scaffolding and nice prompts
- π updated docs
Uploading SpZcfNnbmKnbub3l8FWe.avifβ¦βor sponsorship β₯ on GH
https://github.com/push-based/user-flow/tree/main/packages/cli#readme
Gripped by the cool animation of the CLI he clicked the tweet to see details and swipes the images. "Pretty cool.... hmm"
He clicks the link and lands on the main readme page. He scrolls through the page and sees the npm badge, some logo and a toc.
As the ToC is too much to read he scrolls further down over the CLI option tables and discovers the image about the user interactions and the different scores... then the Architecture headline with the ufo and some links... Then he spots the letters "budgets" and the images which shows the highlighted area "Over budget".
Goosebumps... damn, I have to go...
As he had to drive home by car quickly to meet someone there was no time to read more about the budgets, so he did not know that they only work for the initial navigation. But the misconception intensified his interest enough to have the lib in mind for the next moment there was enough time to at least read more of the docs.
...
It's Thursday, so there most probably will be at least a little time in the coming the weekend to look it up.
The meeting with his friend was pretty cool so he stayed longer and had some nice conversations about headspace, a "observation technique" on thoughts that is described there and the movie inception. It was quite late when he came home so there was no energy left for anything else then getting ready to sleep.
After he arrived on Friday afternoon after quite a hard day (never do important stuff on a Friday) he sits on the sofa and scrolls the phone when the new tool popped up again in his mind triggered by the twitter notification badge on the app icon.
He quickly opens the notifications, scrolls some centimeters in the thread, closes it and opens the @push-based/user-flow docs on GitHub.
He skims the Quick start section for some information on how to connect the tool to his applications URL, it's little enough code to not scroll further and really read the lines.
package.json
to run the measure e.g. "user-flow":"npx user-flow collect"
To reduce the friction for new users to create user flow we should implement some aid to convert Chrome DevTools Recorder outputs into user-flow scripts.
Recorder user flows can be exported and shared in 3 ways/formats:
Potentially we could create a Chrome extension to export replay scripts in user-flow format directly.
As there are already multiple extensions for costume exports (examples: Cypress extension and WebPage Test extension) as well as a Recorder extention API.
As we could have a wrong directory set the CLI should tell us straight away there is something wrong. An error is good here as the collect command makes no sense if there is nothing to collect
implement JSON export
We have a couple of functions handling the passed CLI parameters.
This is normally handled by yargs's option normalization features , but as as we have to derive a couple of options before yargs starts we have to rely on a custom implementation.
As a side note I want to mention that yargs's new version will ship a couple of features making this possible more easily. After the upgrade we most probably can delete this code.
Implement an alias for interactive as i
Add if necessary more details descriptions to each command
When running the collect file and passing in an additional parameter like format it will overwrite the rc json file.
npx user-flow --format md
will overwrite the json file and remove the previous configuration.
Every change in the code base could potentially impact the application's performance.
We need a way to inform developers if the changes were performance slowdowns or improvements.
To accomplish this, we need a way to measures from a current audit with measures from a previous audit.
There are different building blocks necessary to implement this feature:
Building blocks
Report Comparison
The report would expand the reduced report by adding baseline measures.
This reduced report comparison would then be used to generate the Md table.
{
"name": "Sandbox Setup StaticDist",
"steps": [
{
"name": "Navigation report",
"gatherMode": "navigation",
"results": {
"performance": 0.95,
"accessibility": 1,
"best-practices": 0.92,
"seo": 1,
"pwa": 0.3
},
"baseline": {
"performance": 0.89,
"accessibility": 1,
"best-practices": 0.95,
"seo": 0.9,
"pwa": 0.2
}
},
]
}
Selecting Comparison
TODO
Related question:
Specifying Baseline
TODO
Related question:
Report Comparison Visualization
The report would add the comparison number next to the current measurement
Step Name | Gather Mode | Performance | Accessibility | Best Practices | Seo | Pwa |
---|---|---|---|---|---|---|
Navigation report | navigation | 95 (+10) | 100 | 82 (-11) | 100 | 30 (-5) |
Timespan report | timespan | 10/11 (-2) | - | 5/7 (-1) | - | - |
Snapshot report | snapshot | Γ 3/4 | 10/10 | 4/5 | 9/9 | - |
The report would be styled to make it easier to spot which measures when up or down.
Some options for this styling are:
<br />
Step Name | Gather Mode | Performance | Accessibility | Best Practices |
---|---|---|---|---|
Option 1 | navigation | 95 (+10 π) | 100 | 82 (-11 π») |
Option 2 | navigation | 95 (+10 π’) | 100 (βͺ) | 82 (-11 π΄) |
Option 3 | navigation | 95 (+10 β«) | 100 (βͺ) | 82 (-11 β¬ ) |
Option 4 | navigation | 95 π +10 |
100 |
82 π» -11 |
Option 5 | navigation | 95 π’ +10 |
100 βͺ |
82 π΄ -11 |
Option 6 | navigation | 95 +10 β« |
100 |
82 -11 β¬ |
Persistance Option
TODO
Related question:
Types:
Docs:
CLI options:
--budget-path
- path to budget config. e.g. --budget-path=budget.json
Resources:
related to #172
These key combinations may be used on prompts that support multiple choices, such as the multiselect prompt, or the select prompt when the multiple
options is true.
command | description |
---|---|
space | Toggle the currently selected choice when options.multiple is true. |
number | Move the pointer to the choice at the given index. Also toggles the selected choice when options.multiple is true. |
a | Toggle all choices to be enabled or disabled. |
i | Invert the current selection of choices. |
g | Toggle the current choice group. |
Currently the docs start with:
Install
Run npm i @push-based/user-flow --save-dev or yarn add @push-based/user-flow --dev to install the library.After that you can run:
user-flow --helpor user-flow --helpRun without install
You can also use npx to run it in e.g. the CI setup: npx @push-based/user-flow --help
but the output of the command is:
% npx user-flow --help
Options:
--help Show help [boolean]
--version Show version number [boolean]
code is loacted in packages/cli/src/lib/commands/init/processes/setup-rc-json.ts
consider this as default if no format is given
Reports persisted with collect should have:
github.com-20221216T184910.json
)Typing changes:
flow name should be:
<url>-<name(optional)>-<date>
init Command:
Documentation:
The "second contact" focuses on more advanced tests and CI integration.
It should focus on a good user experience from the CLI perspective, as well as a smooth setup of the GitHub action to integrate user-flow into the CI. This means the user should be guided by good documentation and maybe a command with questions and options to set it up.
In general this issue targets direct users of @push-based/user-flows, developer, performance engineers etc. that have user-flow already installed and running, but not implemented a real live integration.
This story is a follow up on #32
Furthermore it will help to separate the CLI and the GH action repositories.
Related repositories:
@push-based/user-flow
@push-based/user-flow-gh-action
Chris, 26, full-stack developer, self educated
Needs an easy way to stop regression on an ongoing JS application. He tried user flow already #32 and remembered it after another uncaught regression in his live system.
He knows lighthouse and puppeteer, also knows user-flow.
There are no performance tests in the application
There are very few tests in the CI
Chris already tested out user-flow and was impressed by this new method of testing the performance of user actions. However, he only tested the basics and left it at that point as there was no more time.
Some 1,5 month later, while testing the performance of the application he was working on, he noticed the performance of the application got worse and he had to figure out why.
The app is pretty busy maintained from many developers and there is no good release management in place to track changes, nor any continuous performance measurement, so it was quite a hassle to spot the issue.
After all the issue was found, and additionally, not only the performance that had gotten worse, but had also a couple of SEO critical bugs introduced.
What a nice ending of that Friday... and hello to a weekend of cleaning up others mess.
In the middle of the next week, not quite recovered from the work intense weekend he started to reason about possible solutions to this unfortunately recurring problem and user-flow jumped in his mind again.
He suggested to set up using user-flow to avoid regressions, but the conversation with his CTO was unpleasant.
Just by looking at some GitHub repository and no clear time estimation on how much it takes to get it running it just got zero attention next to all the high priority tickets in the queue.
But Chris's curiosity got the best of him and he keep playing around with it, knowing that he could convince his CTO to implement user-flow if a PoC could catch some regression in the CI.
To fix for second contact
To fix for third contact
init --action
It is nice to have a user friendly error message but we also need to have a way to understand the real cause. therefore I suggest to add the traces from the original error to the new one.
Example of missing traces:
Error while running user flows. TypeError: Cannot set properties of undefined (setting 'config')
It would be really helpful to see the actual file and the full trace here.
Solution:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/cause
try {
connectToDatabase();
} catch (err) {
throw new Error('Connecting to database failed.', { cause: err });
}
TODO:
Error while executing npm run start:ssr:prod
Current verbose output:
run "collect" as a yargs command with args:
potentialExistingCfg: {
collect: {
url: 'https://coffee-cart.netlify.app/',
ufPath: './packages/user-flow-ci-integration/user-flows',
serveCommand: undefined,
awaitServeStdout: undefined
},
persist: {
outPath: './packages/user-flow-ci-integration/measures',
format: 'md'
}
}
25l25l25lUpdate config under ./packages/user-flow-ci-integration/.user-flowrc.json
readRcConfig: ./packages/user-flow-ci-integration/.user-flowrc.json {
collect: {
url: 'https://coffee-cart.netlify.app/',
ufPath: './packages/user-flow-ci-integration/user-flows'
},
persist: {
outPath: './packages/user-flow-ci-integration/measures',
format: [ 'md' ]
}
}
No updates for ./packages/user-flow-ci-integration/.user-flowrc.json to save.
{
collect: {
url: 'https://coffee-cart.netlify.app/',
ufPath: './packages/user-flow-ci-integration/user-flows',
serveCommand: undefined,
awaitServeStdout: undefined
},
persist: {
outPath: './packages/user-flow-ci-integration/measures',
format: [ 'md' ]
}
}
run user flows without serve command
Set headless to true as we are running in __run_3 mode
Collect: Order Coffee in CI from URL https://coffee-cart.netlify.app/
File path: /home/runner/work/user-flow/user-flow/packages/user-flow-ci-integration/user-flows/order-coffee.uf.ts
Duration: Order Coffee in CI: 46.605
25h25h25hrun "collect" as a yargs command with args:
potentialExistingCfg: {
collect: {
url: 'https://coffee-cart.netlify.app/',
ufPath: './packages/user-flow-ci-integration/user-flows',
serveCommand: undefined,
awaitServeStdout: undefined
},
persist: {
outPath: './packages/user-flow-ci-integration/measures',
format: 'md'
}
}
25l25l25lUpdate config under ./packages/user-flow-ci-integration/.user-flowrc.json
readRcConfig: ./packages/user-flow-ci-integration/.user-flowrc.json {
collect: {
url: 'https://coffee-cart.netlify.app/',
ufPath: './packages/user-flow-ci-integration/user-flows'
},
persist: {
outPath: './packages/user-flow-ci-integration/measures',
format: [ 'md' ]
}
}
No updates for ./packages/user-flow-ci-integration/.user-flowrc.json to save.
{
collect: {
url: 'https://coffee-cart.netlify.app/',
ufPath: './packages/user-flow-ci-integration/user-flows',
serveCommand: undefined,
awaitServeStdout: undefined
},
persist: {
outPath: './packages/user-flow-ci-integration/measures',
format: [ 'md' ]
}
}
run user flows without serve command
Set headless to true as we are running in __run_3 mode
Collect: Order Coffee in CI from URL https://coffee-cart.netlify.app/
File path: /home/runner/work/user-flow/user-flow/packages/user-flow-ci-integration/user-flows/order-coffee.uf.ts
Duration: Order Coffee in CI: 46.605
25h25h25h
Document the different attribute names that are knows by user-flow:
We can already give a teaser on the future geh action discussed here: push-based/user-flow-gh-action#9
We need a way to reference the produced output with a user flow.
I suggest to add the following data to those formats:
When running the init command the file produce imports the use flow from '../../../' instead of '@push-based/user-flow'
Current output:
// Your custom interactions with the page
import {
UserFlowContext,
UserFlowInteractionsFn,
UserFlowProvider,
} from "../../..";
Expected output:
// Your custom interactions with the page
import {
UserFlowContext,
UserFlowInteractionsFn,
UserFlowProvider,
} from '@push-based/user-flow';
Command to reproduce:
npx user-flow init
At the moment the handling of cli options and the rc.json file and the defaults.
As it is used all over the place it would be a good thing to have that clean from the beginning.
See details here:
https://github.com/yargs/yargs/blob/main/docs/advanced.md#rc-files
--config-path
options to CLIAdd a new parameter the the collect
command named config-path
. This path should specify the location of a lighthouse config file.
TODOS:
--config-path
- path to configuration . e.g. --config-path=confiuration.json
Files in the collect user-flow path/directory should only be executed if:
ts
or js
extensionSeveral things in the way how the CLI works are specifically designed for humans that use and interact with it.
A couple of examples are prompts, file generation, visualizing data, reporting progress etc
There are a couple of scenarios where we have no humans involved and or just no need for a specific feature to be active.
One way of doing this is to configure the CLI over comments. E.g. don't open a report automatically --open false
.
To avoid a large set of CLI parameters or many different config files I suggest to do some of those configurations automatically.
CI Configurations Draft:
--open false
- No one will look at it in the CI--interactive false
- no one can answer--format md --format <all others>
- Because we need it for the GitHub action as md textheadless
- we need to configure the lighthouse for a headless runBefore:
npx user-flow -p ./packages/user-flow-ci-integration/.user-flowrc.json --interactive false -e false -f html -f md
After:
npx user-flow -p ./packages/user-flow-ci-integration/.user-flowrc.json
Test Configurations Draft:
--open false
- We only need it to test that the --open
parameter is working--interactive false
- Only the prompts need it, the rest is defined by config files--dryRun false
- We want to have as many tests in dry run to save timeheadless
- we need to test a headful run only onceBefore:
npx user-flow -p ./packages/user-flow-ci-integration/.user-flowrc.json --interactive false -e false --dryRun true
After:
npx user-flow -p ./packages/user-flow-ci-integration/.user-flowrc.json
To group the different pre-configurations I suggest to introduce the concept of environments to the CLI.
Environments:
LOCAL
- on the developers machineCI
- in the a CI setup like GitHub pagesTEST
- in the test environmentIn general the name CI
is used and set to true
or 1
.
The following platforms have a CI
key:
CI=true
- github
CI=true
- circleci
CI=true
- gitlab
CI=true
- travis
CI=true
- netlify
CI=1
- vercel
Unknown:
???
- Jenkins???
- Azure Pipelinesuser-flow CLI is set up now! π
in collect commandWe should create a custom RunnerExtension
This extension will detect user-flow modes over the type
property.
We can later either execute the user-flow modes programmatically or add a custom replay action in the json
export.
We can start with some experiments and document them in a readme.
Later we can work on a custom runner-extension and document everything.
We also have to save raw exports of original and enriched recordings.
run user flows without serve command
Set headless to true as we are running in __run_3 mode
Collect: Order Coffee in CI from URL https://coffee-cart.netlify.app/
File path: /home/runner/work/user-flow/user-flow/packages/user-flow-ci-integration/user-flows/order-coffee.uf.ts
// -- Add out path in verbose long
Output path: /example/path ...
// --
Duration: Order Coffee in CI: 46.425
25h25h25h
When setting multiple persist formats it should save the results in those formats
"persist": { "outPath": "./measures", "format": ["html", "json"] }
The first option is saved but an error occurs when attempting to generate the second report.
type PersistFn = (cfg: Pick<PersistOptions, 'outPath'> & { flow: UserFlow, name: string }) => Promise<string>;
const _persistMethod = new Map<string, PersistFn>();
_persistMethod.set('html', async ({ outPath, flow, name }) => {
const report = await flow.generateReport();
const fileName = join(outPath, `${toFileName(name)}.uf.html`);
writeFile(fileName, report);
return fileName;
});
_persistMethod.set('json', async ({ outPath, flow, name }) => {
const report = await flow.createFlowResult(); // <== Error occurs here
const fileName = join(outPath, `${toFileName(name)}.uf.json`);
writeFile(fileName, JSON.stringify(report));
return fileName;
});
export async function persistFlow(flow: UserFlow, name: string, { outPath, format }: PersistOptions): Promise<string[]> {
// @Notice: there might be a bug in user flow and Promise's
return Promise.all(format.map((f: string) => (_persistMethod.get(f) as any)({ flow, name, outPath })));
}
Error: no known mark: lh:runner:audit
at Object.exports.stop (/Users/username/Applications/user-flow/node_modules/marky/lib/marky.cjs.js:104:13)
at Function.timeEnd (/Users//usernamer/Applications/user-flow/node_modules/lighthouse-logger/index.js:129:11)
at Function.audit (/Users//username/Applications/user-flow/node_modules/lighthouse/lighthouse-core/runner.js:82:11)
at async auditGatherSteps (/Users//username/Applications/user-flow/node_modules/lighthouse/lighthouse-core/fraggle-rock/user-flow.js:215:20)
Instead of using a new Map
for the _persistMethod we could use an object method that returns the list of fileNames.
The format options would then be handled with a condition switch statement.
The potential solution was already tested by persisting both formats on the same Map.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.