GithubHelp home page GithubHelp logo

johansatge / psi-report Goto Github PK

View Code? Open in Web Editor NEW
11.0 4.0 4.0 617 KB

Crawls a website, gets PageSpeed Insights data for each page, and exports an HTML report.

License: MIT License

JavaScript 68.57% HTML 31.43%
crawler pagespeed-insights html-report cli

psi-report's Introduction

Version Downloads Dependencies devDependencies

psi-report

Crawls a website or get URLs from a sitemap.xml or a file, gets PageSpeed Insights data for each page, and exports an HTML report.


Installation

Install with npm:

$ npm install psi-report --global
# --global isn't required if you plan to use the node module

CLI usage

$ psi-report [options] <url> <dest_path>

Options:

    -V, --version               output the version number
    --urls-from-sitemap [name]  Get the list of URLs from sitemap.xml (don't crawl)
    --urls-from-file [name]     Get the list of URLs from a file, one url per line (don't crawl)
    -h, --help                  output usage information

Example:

$ psi-report daringfireball.net/projects/markdown /Users/johan/Desktop/report.html

Programmatic usage

// Basic usage

var PSIReport = require('psi-report');
var psi_report = new PSIReport({baseurl: 'http://domain.org'}, onComplete);
psi_report.start();

function onComplete(baseurl, data, html)
{
    console.log('Report for: ' + baseurl);
    console.log(data); // An array of pages with their PSI results
    console.log(html); // The HTML report (as a string)
}

// The "fetch_url" and "fetch_psi" events allow to monitor the crawling process

psi_report.on('fetch_url', onFetchURL);
function onFetchURL(error, url)
{
    console.log((error ? 'Error with URL: ' : 'Fetched URL: ') + url);
}

psi_report.on('fetch_psi', onFetchPSI);
function onFetchPSI(error, url, strategy)
{
    console.log((error ? 'Error with PSI for ' : 'PSI data (' + strategy + ') fetched for ') + url);
}

Crawler behavior

The base URL is used as a root when crawling the pages.

For instance, using the URL https://daringfireball.net/ will crawl the entire website.

However, https://daringfireball.net/projects/markdown/ will crawl only:

  • https://daringfireball.net/projects/markdown/
  • https://daringfireball.net/projects/markdown/basics
  • https://daringfireball.net/projects/markdown/syntax
  • https://daringfireball.net/projects/markdown/license
  • And so on

This may be useful to crawl only one part of a website: everything starting with /en, for instance.

URLs from a sitemap.xml or a file

Instead of crawling the website, you can set the URL list with a sitemap.xml or a file.

  • --urls-from-sitemap https://example.com/sitemap.xml
  • --urls-from-file /path/to/urls.txt

Only the URLs inside this file will be processed.

Changelog

This project uses semver.

Version Date Notes
2.2.1 2018-01-19 Fix missing source files on NPM (@blaryjp)
2.2.0 2017-11-27 Prepend baseurl if not present, for each urls in file (@blaryjp)
2.1.0 2017-11-19 Add --urls-from-sitemap and --urls-from-file (@blaryjp)
2.0.0 2016-04-02 Deep module rewrite (New module API, updated CLI usage)
1.0.1 2016-01-15 Fix call on obsolete package
1.0.0 2015-12-01 Initial version

License

This project is released under the MIT License.

Credits

psi-report's People

Contributors

blaryjp avatar dasrick avatar johansatge avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

psi-report's Issues

Ability to run pagespeed report on static list of pages/urls

I have a static list of urls ( which represent the major pages of my site ) that i would like to run page speed tests on. It would be awesome if there was a way to feed in this list so i don't have to wait for the long process of the crawler crawling my whole website.

Crashes randomly

tool crashes randomly, tried it on two different machines
node --version
v11.10.1
npm --version
6.7.0
psi-report --version
2.2.1

Any idea?


/usr/local/lib/node_modules/psi-report/src/psi.js:106
score: rule_groups['SPEED'].score,
^

TypeError: Cannot read property 'SPEED' of undefined
at _parseData (/usr/local/lib/node_modules/psi-report/src/psi.js:106:39)
at Request._callback (/usr/local/lib/node_modules/psi-report/src/psi.js:54:52)
at Request.self.callback (/usr/local/lib/node_modules/psi-report/node_modules/request/request.js:185:22)
at Request.emit (events.js:197:13)
at Request. (/usr/local/lib/node_modules/psi-report/node_modules/request/request.js:1161:10)
at Request.emit (events.js:197:13)
at IncomingMessage. (/usr/local/lib/node_modules/psi-report/node_modules/request/request.js:1083:12)
at Object.onceWrapper (events.js:285:13)
at IncomingMessage.emit (events.js:202:15)
at endReadableNT (_stream_readable.js:1129:12)
at processTicksAndRejections (internal/process/next_tick.js:76:17)

Question about the lib

does this lib only get the speed and usability ?
The original psi lib gets additional things like:

CSS size                                   | 35.7 kB
HTML size                                  | 3.14 kB
Image size                                 | 178 kB
JavaScript size                            | 6.27 MB
CSS resources                              | 1
Hosts                                      | 1
JS resources                               | 5
Resources                                  | 12
Static resources                           | 11
Total size of request bytes sent           | 963 B

Minimize render blocking resources         | 48
Prioritize visible content                 | 8
Size content to viewport                   | 50

so I was wondering if it was possible to show this in the HTML report also

TypeError: Cannot read property 'replace' of undefined

System:

  • Ubuntu 16.04 (can reproduce the problem on OS X)
  • Node 8.9.3
  • psi-report installed locally with "yarn add psi-report --dev" or "npm install psi-report --save-dev" (same problem when installed globally)

When running psi-report I get the following traceback:

$ node_modules/psi-report/src/cli.js https://kitconcept.com report.html
Found https://kitconcept.com/
Found https://kitconcept.com/blog/
Found https://kitconcept.com/blog/pastanaga-sprint-bonn-2017-report/
Found https://kitconcept.com/blog/headless-and-mobile-the-future-of-plone/
Found https://kitconcept.com/blog/plone-beethoven-sprint-2017-sprint-report-day-3/
Found https://kitconcept.com/blog/plone-beethoven-sprint-report-day-2/
Found https://kitconcept.com/blog/plone-beethoven-sprint-2017-sprint-report-day-1/
Found https://kitconcept.com/blog/continuous-integration-for-plone/
2017-12-21T08:34:31.157Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.159Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.160Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.164Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.167Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.186Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
Found https://kitconcept.com/plone-beethoven-sprint-bonn-2017/
2017-12-21T08:34:31.230Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.238Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.265Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.276Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.311Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.350Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.369Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.373Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.384Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.393Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.401Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
2017-12-21T08:34:31.402Z - warn: CRAWLER response body is not HTML, skip injecting. Set jQuery to false to suppress this message
Got PSI data (mobile) for https://kitconcept.com/blog/
Got PSI data (desktop) for https://kitconcept.com/blog/
Got PSI data (desktop) for https://kitconcept.com/
Got PSI data (mobile) for https://kitconcept.com/
Got PSI data (mobile) for https://kitconcept.com/blog/pastanaga-sprint-bonn-2017-report/
Got PSI data (desktop) for https://kitconcept.com/blog/pastanaga-sprint-bonn-2017-report/
Got PSI data (mobile) for https://kitconcept.com/blog/headless-and-mobile-the-future-of-plone/
Got PSI data (mobile) for https://kitconcept.com/blog/plone-beethoven-sprint-2017-sprint-report-day-3/
Got PSI data (desktop) for https://kitconcept.com/blog/plone-beethoven-sprint-2017-sprint-report-day-3/
Got PSI data (mobile) for https://kitconcept.com/blog/plone-beethoven-sprint-report-day-2/
Got PSI data (desktop) for https://kitconcept.com/blog/headless-and-mobile-the-future-of-plone/
Got PSI data (mobile) for https://kitconcept.com/blog/plone-beethoven-sprint-2017-sprint-report-day-1/
Got PSI data (desktop) for https://kitconcept.com/blog/plone-beethoven-sprint-2017-sprint-report-day-1/
Got PSI data (desktop) for https://kitconcept.com/blog/plone-beethoven-sprint-report-day-2/
Got PSI data (mobile) for https://kitconcept.com/blog/continuous-integration-for-plone/
Got PSI data (desktop) for https://kitconcept.com/blog/continuous-integration-for-plone/
Got PSI data (mobile) for https://kitconcept.com/plone-beethoven-sprint-bonn-2017/
Got PSI data (desktop) for https://kitconcept.com/plone-beethoven-sprint-bonn-2017/
/home/timo/workspace/elif/frontend/node_modules/psi-report/src/html_report.js:47
                html = html.replace('<!--results-->', _buildItem(templates.item, page) + '<!--results-->');
                            ^

TypeError: Cannot read property 'replace' of undefined
    at /home/timo/workspace/elif/frontend/node_modules/psi-report/src/html_report.js:47:29
    at Array.map (<anonymous>)
    at _onTemplatesLoaded (/home/timo/workspace/elif/frontend/node_modules/psi-report/src/html_report.js:45:18)
    at /home/timo/workspace/elif/frontend/node_modules/psi-report/node_modules/async/dist/async.js:3874:9
    at /home/timo/workspace/elif/frontend/node_modules/psi-report/node_modules/async/dist/async.js:473:16
    at iterateeCallback (/home/timo/workspace/elif/frontend/node_modules/psi-report/node_modules/async/dist/async.js:980:24)
    at /home/timo/workspace/elif/frontend/node_modules/psi-report/node_modules/async/dist/async.js:958:16
    at /home/timo/workspace/elif/frontend/node_modules/psi-report/node_modules/async/dist/async.js:3871:13
    at ReadFileContext.callback (/home/timo/workspace/elif/frontend/node_modules/psi-report/src/html_report.js:31:21)
    at FSReqWrap.readFileAfterOpen [as oncomplete] (fs.js:420:13)

@johansatge thank you for sharing your work! If you have a pointer how to fix this or work around the issue that would be highly appreciated!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.