zachleat / performance-leaderboard Goto Github PK

View Code? Open in Web Editor NEW

96.0 7.0 11.0 88 KB

A plugin to run Lighthouse against a set of urls to see which site is the fastest.

JavaScript 100.00%

speedlify

performance-leaderboard's Introduction

performance-leaderboard

A plugin to run Lighthouse against a set of urls to see which site is the fastest.

See also: performance-leaderboard-pagespeed-insights (runs Lighthouse through PageSpeed Insights)

Installation

npm install performance-leaderboard

Features

Median Run Selection: performance-leaderboard will run Lighthouse on the same site multiple times and select the Median run. It factors in First Contentful Paint, Largest Contentful Paint, and Time to Interactive when selecting the median run.

Usage

Create a test file, say sample.js:

const PerfLeaderboard = require("performance-leaderboard");

(async function() {

	let urls = [
		"https://www.gatsbyjs.org/",
		"https://nextjs.org/",
		"https://www.11ty.dev/",
		"https://vuejs.org/",
		"https://reactjs.org/",
		"https://jekyllrb.com/",
		"https://nuxtjs.org/",
		"https://gohugo.io/",
	];

	// Create the options object (not required)
	const options = {
		axePuppeteerTimeout: 30000, // 30 seconds
		writeLogs: true, // Store audit data
		logDirectory: '.log', // Default audit data files stored at `.log`
		readFromLogDirectory: false, // Skip tests with existing logs
		// onlyCategories: ["performance", "accessibility"],
		chromeFlags: ['--headless'],
		freshChrome: "site", // or "run"
		launchOptions: {}, // Puppeteer launch options
	}

	// Run each site 3 times with default options
	console.log( await PerfLeaderboard(urls) );

	// Or run each site 5 times with default options
	console.log( await PerfLeaderboard(urls, 5) );

	// Or run each site 5 times with custom options
	console.log( await PerfLeaderboard(urls, 5, options) );
})();

Run node sample.js.

Sample Output

[
	{
		url: 'https://www.11ty.dev/',
		requestedUrl: 'https://www.11ty.dev/',
		timestamp: 1623525988492,
		ranks: { hundos: 1, performance: 1, accessibility: 1, cumulative: 1 },
		lighthouse: {
			version: '8.0.0',
			performance: 1,
			accessibility: 1,
			bestPractices: 1,
			seo: 1,
			total: 400
		},
		firstContentfulPaint: 1152.3029999999999,
		firstMeaningfulPaint: 1152.3029999999999,
		speedIndex: 1152.3029999999999,
		largestContentfulPaint: 1152.3029999999999,
		totalBlockingTime: 36,
		cumulativeLayoutShift: 0.02153049045138889,
		timeToInteractive: 1238.3029999999999,
		maxPotentialFirstInputDelay: 97,
		timeToFirstByte: 54.63900000000001,
		weight: {
			summary: '14 requests • 178 KiB',
			total: 182145,
			image: 124327,
			imageCount: 10,
			script: 7824,
			scriptCount: 1,
			document: 30431,
			font: 15649,
			fontCount: 1,
			stylesheet: 3914,
			stylesheetCount: 1,
			thirdParty: 15649,
			thirdPartyCount: 1
		},
		run: { number: 2, total: 3 },
		axe: { passes: 850, violations: 0 },
	}
]

Rankings

In the return object you’ll see a ranks object listing how this site compares to the other sites in the set. There are a bunch of different scoring algorithms you can choose from:

ranks.performance
- The highest Lighthouse performance score.
- Tiebreaker given to the lower SpeedIndex score.
ranks.accessibility
- The highest Lighthouse accessibility score.
- Tiebreaker given to lower Axe violations.
- Second tiebreaker given to highest Axe passes (warning: each instance of an Axe rule passing is treated separately so this will weigh heavily in favor of larger pages)
ranks.hundos
- The sum of all four Lighthouse scores.
- Tiebreaker given to the lower Speed Index / Total Page Weight ratio.
ranks.cumulative (the same as hundos but with an Axe tiebreaker)
- The sum of all four Lighthouse scores.
- Tiebreaker given to the lower Axe violations.
- Second tiebreaker given to the lower Speed Index / Total Page Weight ratio.

Changelog

v10.0.0 Update lighthouse to v10, requires Node 16.
v9.0.0 Update lighthouse to v9.0. Removes carbonAudit, upstream API was removed.
v5.3.0 Update lighthouse from v8.2 to v8.5
v5.2.0 Update lighthouse from v8.0 to v8.2
v5.1.0 Adds axePuppeteerTimeout option. Adds carbonAudit option.
v5.0.0 Update lighthouse to v8.0
v4.1.0 Update lighthouse to v7.3
v4.0.0 Major version upgrade of lighthouse dependency from v6.5 to v7.2

performance-leaderboard's People

Contributors

Stargazers

Watchers

Forkers

nishtrack bridgestew-llc stedman sebastianbenz nfjodor studiometa hackthings fershad allrude murtuzaalisurti kumarsuresh1525

performance-leaderboard's Issues

Reintroduce carbon estimate

Would there be an appetite to add web page carbon estimates back to the results (removed in #16) without depending on an external API?

The calculations outlined in this post https://sustainablewebdesign.org/calculating-digital-emissions/ could be used. They are the same as is used by the Website Carbon API which was previously used.

I'd be happy to contribute a PR (with some guidance).

Allow setup scripts per URL

It would be very useful to be able to run a setup script for each URL in Chrome before lighthouse runs.

For example:

To log into a secured area
To set cookies
Even (and I don't know if this is possible - I've been spoiled by WPT) to change where a domain resolves

How?

I don't know. This issue is really a discussion starter to see if it's something you are interested in adding, and to think about how/if it's feasible.

Using Puppeteer to script interactions seems sensible. I've done some Googling for ideas.

I've seen this project uses chrome-launcher to set up Chrome rather than Puppeteer directly so that could be a complication.

I don’t know if this is an official API or not but I think we may need to remove the carbon testing part, unfortunately.

Report lighthouse version in results

Top level lighthouseVersion in lighthouse results.

Upgrade to Lighthouse 9

Also switch our versioning to make the major version of performance-leaderboard always match the major version of lighthouse.

I have the following error when using this package inside a Docker container.
(Site 1 of 1, run 1 of 1): https://google.nl Logged an error with https://google.nl: TypeError: Cannot read property 'score' of undefined at ResultLogger.mapResult (/app/node_modules/performance-leaderboard/src/ResultLogger.js:237:56) at runLighthouse (/app/node_modules/performance-leaderboard/performance-leaderboard.js:88:37) at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:95:5) at async /app/src/index.js:113:30 [ { url: 'https://google.nl', error: '{}', sidequests: { '-weight.total': 1, '+weight.total': 1, '-weight.document': 1, '+weight.document': 1, '-weight.script': 1, '+weight.script': 1, '-weight.image': 1, '+weight.image': 1, '-weight.font': 1, '+weight.font': 1, '+weight.fontCount': 1, '-timeToFirstByte': 1, '-totalBlockingTime': 1, '-largestContentfulPaint': 1 } } ]

My code is:
`(async function() {

        let urls = [
            "https://google.nl",
        ];

        const options = {
            //axePuppeteerTimeout: 120000, // 120 seconds
            writeLogs: false, // Store audit data
            readFromLogDirectory: false, // Skip tests with existing logs
            onlyCategories: ["performance"],
            chromeFlags: [
                '--headless',
                '--disable-gpu',
                '--disable-dev-shm-usage',
                '--no-first-run'
            ],
        }

        try {
            const data = await PerfLeaderboard(urls, 1, options);
            console.log(data);
        } catch (error) {
            console.log(error);
        }

    })();`

What can this be?

Testing one site getting two different speedindices

When I run performance-leaderboard to test only one of my 11ty sites, first without any number of pass console.log( await PerfLeaderboard(urls) );
and second with 5 passes console.log( await PerfLeaderboard(urls, 5) );, I get a big difference (43%) between the first one (3 passes) and the second one (5 passes).

Here is the log:

Testing 1 sites:
(Site 1 of 1, run 1 of 3): https://corsa.netlify.app/posts/unfolding.html
(Site 1 of 1, run 2 of 3): https://corsa.netlify.app/posts/unfolding.html
(Site 1 of 1, run 3 of 3): https://corsa.netlify.app/posts/unfolding.html
[
  {
    url: 'https://corsa.netlify.app/posts/unfolding.html',
    finalUrl: 'https://corsa.netlify.app/posts/unfolding.html',
    lighthouseScore: 1,
    firstContentfulPaint: 1156.5130000000001,
    firstMeaningfulPaint: 1156.5130000000001,
    speedIndex: 1395.6202760075153,
    rank: 1
  }
]
Testing 1 sites:
(Site 1 of 1, run 1 of 5): https://corsa.netlify.app/posts/unfolding.html
(Site 1 of 1, run 2 of 5): https://corsa.netlify.app/posts/unfolding.html
(Site 1 of 1, run 3 of 5): https://corsa.netlify.app/posts/unfolding.html
(Site 1 of 1, run 4 of 5): https://corsa.netlify.app/posts/unfolding.html
(Site 1 of 1, run 5 of 5): https://corsa.netlify.app/posts/unfolding.html
[
  {
    url: 'https://corsa.netlify.app/posts/unfolding.html',
    finalUrl: 'https://corsa.netlify.app/posts/unfolding.html',
    lighthouseScore: 1,
    firstContentfulPaint: 969.73,
    firstMeaningfulPaint: 991.73,
    speedIndex: 969.73,
    rank: 1
  }
]

At the same time, DevTools Lighthouse gives me a result similar to the second one (speedIndex: 0.9s).

Is there any explanation?

Create a log all runs for later analysis

Per feedback here it would be nice to see a full log of all of the runs, rather than throwing non-median runs away (should later analysis be needed)

https://twitter.com/reubenlillie/status/1241866530591301632

cc @reubenlillie

Multiple reports on the same day

Problem

If you run the script multiple times the report is overwritten, because the report folder name is the current date.

Axe scan navigation timeout errors

Started getting these when running tests this week. Only two sites, both of which do have some Axe errors (that I'm working on) - but unsure why those errors would cause timeouts. Any ideas here? Results in this file:

leaderboard.txt