styfle / links-awakening

🔗 Recursively check a website for broken links

Links Awakening

Recursively check a website for broken links.

Usage

npx links-awakening 'https://styfle.dev'

Output

✅ https://styfle.dev/
✅ https://styfle.dev/projects
✅ https://styfle.dev/blog
✅ https://styfle.dev/contact
✅ https://twitter.com/styfle
✅ https://keybase.io/styfle
✅ https://github.com/styfle
✅ https://www.npmjs.com/~styfle
✅ https://styfle.dev/blog/copee-released
✅ http://googlechromereleases.blogspot.com/
✅ https://developers.google.com/web/updates/
✅ https://developers.google.com/web/updates/2015/04/cut-and-copy-commands
❌ http://zeroclipboard.org/ (status: 500, referer: https://styfle.dev/blog/copee-released)
✅ http://styfle.github.io/copee/
✅ https://styfle.dev/blog/d3js-graph-prod
❌ https://styfle.dev/slides/d3js.html (status: 404, referer: https://styfle.dev/blog/d3js-graph-prod)

Programmatic API

import { awaken, type AwakenResult } from 'links-awakening';

const onAwaken = ({ url, status }: AwakenResult) => {
  const icon = status >= 200 && status <= 299 ? '✅' : '❌';
  console.log(`${icon} ${url}`);
};

const url = new URL('https://example.com/blog');

const results = new Map<string, AwakenResult>();

await awaken({ url, onAwaken, results });

console.log(`Done! Crawled ${results.size} links.`);

links-awakening's Issues

Automatically publish on merge

I think semantic-release would be a good choice.

For example:

• vercel/ncc#1015
• vercel/ncc#1016

Add support for `http-equiv`

Check for <meta http-equiv="status" content="404"> and treat the same as a status code

[Bug] It doesn't recognize `mailto:` scheme

Hello! 👋
Thanks for this awesome utility, that might be run in CI to avoid broken links. 👍

I tried it on my website, and it seems like all my links are working. 😄
However, I've got this: mailto:[email protected] (status: unknown scheme, referer: https://divlo.fr/).
It doesn't yet recognize mailto: scheme, I don't know if it's easy to know if a email is valid or not, but if it is complicated, and not in the scope of this utility, I'd rather having a warning to tell me that it couldn't be validated instead of an error.

Crash on invalid url

Hello,
the line

links-awakening/src/index.ts

Line 77 in 400fd6f

absoluteUrl = new URL(href, url);

(and

links-awakening/src/index.ts

Line 73 in 400fd6f

absoluteUrl = new URL(href);

) assumes that href is a well formatted URL, which is not necessarily the case.

I have encountered links like http://tel:[phone]xx which crashes the program.
An AwakenResult should be returned instead, with a message like 'Invalid URL'.

node:internal/errors:490
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_INVALID_URL]: Invalid URL
    at new NodeError (node:internal/errors:399:5)
    at URL.onParseError (node:internal/url:565:9)
    at new URL (node:internal/url:645:5)
    at Object.onopentag (file:///Users/jack/.nvm/versions/node/v18.14.0/lib/node_modules/links-awakening/dist/index.js:50:35)
    at Parser.endOpenTag (file:///Users/jack/.nvm/versions/node/v18.14.0/lib/node_modules/links-awakening/node_modules/htmlparser2/lib/esm/Parser.js:193:86)
    at Parser.onopentagend (file:///Users/jack/.nvm/versions/node/v18.14.0/lib/node_modules/links-awakening/node_modules/htmlparser2/lib/esm/Parser.js:204:14)
    at Tokenizer.stateBeforeAttributeName (file:///Users/jack/.nvm/versions/node/v18.14.0/lib/node_modules/links-awakening/node_modules/htmlparser2/lib/esm/Tokenizer.js:404:22)
    at Tokenizer.parse (file:///Users/jack/.nvm/versions/node/v18.14.0/lib/node_modules/links-awakening/node_modules/htmlparser2/lib/esm/Tokenizer.js:763:22)
    at Tokenizer.write (file:///Users/jack/.nvm/versions/node/v18.14.0/lib/node_modules/links-awakening/node_modules/htmlparser2/lib/esm/Tokenizer.js:154:14)
    at Parser.write (file:///Users/jack/.nvm/versions/node/v18.14.0/lib/node_modules/links-awakening/node_modules/htmlparser2/lib/esm/Parser.js:432:28) {
  input: 'http://tel:[phone]xx',
  code: 'ERR_INVALID_URL'
}

Node.js v18.14.0

[Bug] It always exit with status-code 0 even with broken links

Hello! 👋

Currently this utility, prints all the links of the website, and prints if they are valid or not.
If some links are unvalid, it's still exiting with a exit status code of 0.
However, I find that this utility could be really useful in CI, and it might make more sense to exit with 1 for exemple, so it failsthe CI if there are broken links.