urlint / urlint

Checking the availability of links from your CLI

Features

• Collect all the links behind a URL.
• Get HTTP status code and group (2xx, 3xx, etc) per every link.
• HTML & XML (RSS/Atom/Feed/Sitemap) markup supported.
• Universal UNIX interface, easy to integrate with any existence system or tooling.

Usage

as CLI

The nature of the program is to be used as CLI:

$ npx urlint

We recommend execute it using npx over install it globally to have the benefits of the latest version.

Also, the CLI supports load the configuration via:

• A .urlintrc file, written in YAML or JSON, with optional extensions: .yaml/.yml/.json/.js.
• A urlint.config.js file that exports an object.
• A urlint key in your package.json file.

See options to know what kind of settings can be place there.

as Node.js module

Alternatively you can use it from Node.js.

See @urlint/core to know more.

License

urlint © Kiko Beats, released under the MIT License.
Authored and maintained by Kiko Beats with help from contributors.

kikobeats.com · GitHub @Kiko Beats · Twitter @Kikobeats

urlint's Issues

An in-range update of @urlint/core is breaking the build 🚨

Version 1.4.3 of @urlint/core was just published.

Branch	Build failing 🚨
Dependency	@urlint/core
Current Version	1.4.2
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

@urlint/core is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build failed Details

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Install on MacOS High Sierra 10.13.3

Hello

After installation # npm insall urlint-cli, this message display :

$ npm install urlint-cli
npm WARN temp No description
npm WARN temp No repository field.
npm WARN temp No license field.

• [email protected]
  updated 1 package in 8.131s

---

But when you want tu use :

$ urlint
zsh: command not found: urlint

Seb

Add notification

• email
• slack

Update deps

not sure why greenkepeer is not working

Print total time

Support cookie file

That way we can validate links after a user has logged-in

Error: Chromium revision is not downloaded

I just came across this package and decided to give it a go. Seems like there is an issue following the bump of Puppeteer in adblocker@1.4.0. Reproduction steps on macOS (Node 12):

npm install urlint -g

/usr/local/bin/urlint -> /usr/local/lib/node_modules/urlint/bin/cli/index.js
npm WARN @cliqz/[email protected] requires a peer of puppeteer@^2.0.0 but none is installed. You must install peer dependencies yourself.

+ [email protected]

urlint example.com
42.4s ⠹  ← timer keeps ticking forever

ctrl+c ↓

Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"
    at Launcher.launch (/usr/local/lib/node_modules/urlint/node_modules/puppeteer-core/lib/Launcher.js:119:15)
    at async pRetry.onFailedAttempt.debug.attemptNumber (/usr/local/lib/node_modules/urlint/node_modules/browserless/src/index.js:35:26)
    at async RetryOperation._fn (/usr/local/lib/node_modules/urlint/node_modules/p-retry/index.js:41:12)
Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"
    at Launcher.launch (/usr/local/lib/node_modules/urlint/node_modules/puppeteer-core/lib/Launcher.js:119:15)
    at async pRetry.onFailedAttempt.debug.attemptNumber (/usr/local/lib/node_modules/urlint/node_modules/browserless/src/index.js:35:26)
    at async RetryOperation._fn (/usr/local/lib/node_modules/urlint/node_modules/p-retry/index.js:41:12)
Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"
    at Launcher.launch (/usr/local/lib/node_modules/urlint/node_modules/puppeteer-core/lib/Launcher.js:119:15)
    at async pRetry.onFailedAttempt.debug.attemptNumber (/usr/local/lib/node_modules/urlint/node_modules/browserless/src/index.js:35:26)
    at async RetryOperation._fn (/usr/local/lib/node_modules/urlint/node_modules/p-retry/index.js:41:12)
Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"
    at Launcher.launch (/usr/local/lib/node_modules/urlint/node_modules/puppeteer-core/lib/Launcher.js:119:15)
    at async pRetry.onFailedAttempt.debug.attemptNumber (/usr/local/lib/node_modules/urlint/node_modules/browserless/src/index.js:35:26)
    at async RetryOperation._fn (/usr/local/lib/node_modules/urlint/node_modules/p-retry/index.js:41:12)
Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"
    at Launcher.launch (/usr/local/lib/node_modules/urlint/node_modules/puppeteer-core/lib/Launcher.js:119:15)
    at async pRetry.onFailedAttempt.debug.attemptNumber (/usr/local/lib/node_modules/urlint/node_modules/browserless/src/index.js:35:26)
    at async RetryOperation._fn (/usr/local/lib/node_modules/urlint/node_modules/p-retry/index.js:41:12)

An in-range update of @urlint/core is breaking the build 🚨

Version 1.3.4 of @urlint/core was just published.

Branch	Build failing 🚨
Dependency	@urlint/core
Current Version	1.3.3
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

@urlint/core is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build failed Details

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of beauty-error is breaking the build 🚨

The dependency beauty-error was updated from 1.0.0 to 1.0.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

beauty-error is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Release Notes for v1.0.1

Bug Fixes

• add files metadata (07ab4e9)

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Add notification

Suggested notifiers:

• email
• slack
• sms
• twitter

Cannot read property 'toString' of undefined

When I try to run on http://www.rdias.com.br I get the following error, after it verified some of the urls:

TypeError: Cannot read property 'toString' of undefined
at getStatusCodeId (/home/paulo/.npm-packages/lib/node_modules/urlint/bin/view/colorize.js:13:50)
at getStatusCodeColor (/home/paulo/.npm-packages/lib/node_modules/urlint/bin/view/colorize.js:16:14)
at byStatusCode (/home/paulo/.npm-packages/lib/node_modules/urlint/bin/view/colorize.js:21:17)
at renderLink (/home/paulo/.npm-packages/lib/node_modules/urlint/bin/view/render.js:29:44)
at /home/paulo/.npm-packages/lib/node_modules/urlint/bin/view/render.js:48:19
at arrayMap (/home/paulo/.npm-packages/lib/node_modules/urlint/node_modules/lodash/lodash.js:639:23)
at map (/home/paulo/.npm-packages/lib/node_modules/urlint/node_modules/lodash/lodash.js:9554:14)
at /home/paulo/.npm-packages/lib/node_modules/urlint/bin/view/render.js:36:18
at /home/paulo/.npm-packages/lib/node_modules/urlint/node_modules/lodash/lodash.js:3543:27
at /home/paulo/.npm-packages/lib/node_modules/urlint/node_modules/lodash/lodash.js:4905:15

An in-range update of html-get is breaking the build 🚨

Version 1.0.3 of html-get was just published.

Branch	Build failing 🚨
Dependency	html-get
Current Version	1.0.2
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

html-get is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

---

Commits

The new version differs by 2 commits.

• f5fb07c Release 1.0.3
• 7b487e5 Expose create browserless

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of @urlint/core is breaking the build 🚨

Version 1.4.1 of @urlint/core was just published.

Branch	Build failing 🚨
Dependency	@urlint/core
Current Version	1.4.0
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

@urlint/core is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build failed Details

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of html-get is breaking the build 🚨

Version 1.0.2 of html-get was just published.

Branch	Build failing 🚨
Dependency	html-get
Current Version	1.0.1
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

html-get is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

---

Commits

The new version differs by 3 commits.

• 74e886a Release 1.0.2
• 48b0ec8 Add puppeteer as peer dependency
• 4425c3c Update README.md

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Add serialization

Be possible serialize output in different format

• JSON
• CSV
• HTML (https://github.com/bahmutov/term-to-html)

Configuration file support

Using https://github.com/davidtheclark/cosmiconfig for avoid passing the same parameters all the time

Add follow urls

Currently, the only way to follow more than one URL is providing a sitemap URL.

The problem is, if your site doesn't have a sitemap, then you need to provide one by one URL ¯_(ツ)_/¯

Two ideas to avoid that situation could be:

Discover URLs dynamically

Be possible lint new URLs at the moment you expand the initial set of URLs.

For leveraging the feature, it should be limit the deep to avoid an infinite process that never ends.

Example

# Default value. The initial url of sets detected is not expanded.
$ urlint https://kikobeats.com --deep-level=0

# After the initial set of urls detected
# the urls will be expanded an added in the process 
$ urlint https://kikobeats.com --deep-level=1

More than one URL as input

Currently just is possible provide one url, so let make possible provide more than one.

Then unify all the urls detected for avoid analyze the same url more than once.

The URLs could be provided from different ways, for example, using a JSON, CSV, Text plain.

Sort status results by count

Setup CI url properly

The current Travis URL is malformed, need create the job instead of build url

Related:

• travis-ci/travis-ci#8935
• siddharthkp/ci-env#19

An in-range update of reachable-url is breaking the build 🚨

Version 1.1.3 of reachable-url was just published.

Branch	Build failing 🚨
Dependency	reachable-url
Current Version	1.1.2
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

reachable-url is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Commits

The new version differs by 2 commits.

• 59152c4 Release 1.1.3
• ca9ce00 Update deps

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Add prerendering support

Using https://github.com/Kikobeats/get-html

Install on MacOS High Sierra 10.13.3 - With Install Global

Hello

I have also test with install on global, but i have this error.

$ urlint
/usr/local/lib/node_modules/urlint-cli/bin/cli/index.js:67
;(async () => {
^

SyntaxError: Unexpected token (
at createScript (vm.js:56:10)
at Object.runInThisContext (vm.js:97:10)
at Module._compile (module.js:542:28)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:389:7)
at startup (bootstrap_node.js:149:9)

Sébastien

Consider migrate from neat-log to diffy

Avoid animation on CI environment

Show redirects origin → dist

Sort results by status code

Markdown support

An in-range update of @urlint/core is breaking the build 🚨

Version 1.4.2 of @urlint/core was just published.

Branch	Build failing 🚨
Dependency	@urlint/core
Current Version	1.4.1
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

@urlint/core is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build failed Details

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

[Question] Does urlint account for broken anchor links (website.com/link#anchor)?

Quickly glancing at the source code it wasn't clear whether or not this library accounts for checking whether anchor links are broken or not. In other words, does urlint check to see if the link's resolved content has an associated id element with a value of anchor in its markup, for the hypothetical link of link.com/page#achor?

Base64 images reported as 404

When running npx urlint our-test-domain.com, it gives a 404 status on a bunch of base64 images. Perhaps it should not try to follow base64 uri's, or data-uri's in general?

2xx 270
3xx 6

4xx 14

404 data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAlgCWAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCAAGABQDAREAAhEBAxEB/8QAFgAAAwAAAAAAAAAAAAAAAAAAAgQI/8QAIxAAAQIFAwUAAAAAAAAAAAAAAQIDAAQFERIGITEjMkFRcf/EABUBAQEAAAAAAAAAAAAAAAAAAAQA/8QAHBEBAAEEAwAAAAAAAAAAAAAAAQACEUHRAxPB/9oADAMBAAIRAxEAPwC2NQvTUpq2nzLT/SsJcsG+JCnWgT9sr14EDqau8Bw+biSxxNzO4NXGNRdvbuHEMhokpJyVsOYpT//Z +5ms
404 data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAlgCWAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCAAUABQDAREAAhEBAxEB/8QAGQAAAgMBAAAAAAAAAAAAAAAABAUBAgYI/8QAKBAAAQMDAwMDBQAAAAAAAAAAAQIDBAUREgAGISJBUQcUMRMzUrHR/8QAFwEBAQEBAAAAAAAAAAAAAAAAAwACBP/EACMRAAICAgEEAgMAAAAAAAAAAAECAAMRIRIEUWFxQbExodH/2gAMAwEAAhEDEQA/AOyBl9YJAPx/dUpLsR1zE2Pb9apQtunvpTbA6pRlJl0yiJffnlEVhnguuLAB6lJvza3xrne9KwWsOAPkkD7jrSzkBBknsCfqKa3vdmmGplmE3Iahxm5DbiXvvZ4gC2PFsvJ7eeBbqgOXEZwM/n1/Yw6YnGdZ8e/3rfsSuxJUTddGcl1SE29ObeLTjpcKgvpSq6fxHV8DxrNDrcvJxuau5VNxQ4E1VR21TqqHI02OJbDqs1IdJIvkpQt45Ons6eq1Sli8gfgwEvsrYNWcEdplN1emFMfcrM8zakn3cVEcxUyj7dtKbEYN2sDwNK6K6cCMDGNagVFqnLgkknO9+NePHeGemO04dB20WWVvuhx9bilPLBN7AdgOyRo6qEqXisay1rDlp//Z +1ms
404 data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAMAAAC6V+0/AAACr1BMVEUAAADj5eTm5+jf4+SssLXh4uODh46HjJPg4uFqb3eprrXe4N/d397c3t3c3d/d39/c3t/d3t/c3d3b3dzj5eXk5uXi5OTi5OTi4+PZ2NnV1dfh4uTi5OTb29za2tvW1tjU1NXMzc/g4uPh4uPb29zLy83PztHV1dbT09Xg4eLg4uLU1NbT09XX1tfW1Na1ur/g4eHS0tTR0dO9u7XFv8HPz9DOztC7u7dvdnHZ29vGxsnOzc+sq6qen5SMjYDf4eDKzc+5ur7MzM5xcG4cHSXf4eDS1dWxtLjHx8oeHik5Oj/e4N/e4N/Y29qzt7vBwcTQ0NIrKjAgICWHi5HHy87d397e397c3t2+w8W4ub21t682Nj9BQkq1u8DP0dTc3d/e397d397Hy82ztLjBwsIqJit0dny9wsfY2dvd3+Dd397d397P0tStsLW1trUoKDI5NjuyuL3IzdDb3N7d397d397X2dmxtblkZ2kbHCNfYWq8wcXV19nc3t/c3t3d397b3d2/w8ewtbrHy87b3d7c3t3c3t3b3d3Y2trc3d3Z2drZ2NrX1tiamZ+rq7DT0tTa2dnb2tna2drS0tTU1NXW1dfGxMfNy8zV0tLHxsXS0NDU0tLa2NrQ0NLR0dPT09XY19jMycnLx8XPysnb19TY1c3m5OPFxcjg4ODT09TW1tfKycrJxsbX0dDk29rf2NXm4+G6ubzCwsXOztDY2NnS0tPLyszZ19fg3Nvi29nJwrrc2NTk4uLW1tjDw8bZ2NmchomTiYbV1dbY1tja2NjU0s/LxcHAuLS8u7XU0s7b2NjOycqajI5HREjHx8TU09THmqm4aY2gaIKnboSbmY7Oy8ekoaLJysm0d42Ya4GQa33OeZ+yrKN7eXu6ura8lKDKo7G2sK5DOTzDxcOHiIv1E3O7AAAAinRSTlMAAAAAAAAAAAAAAAAAAAAAAAAAAAEBAQQIY0MCBjrI/YkGAQQWl/zKJgEFWubyYAECIbOmDyHt3DcHVu/5dgIDDUDauRgEDi683ToBBg0hlv3+piYLAQYMGm706mMXCQMFChVN4bgxEAUBAwkSM8P0dhoKAwIHDiGbzDwRBgICBgsVFwoDAQMFBAFixcEVAAAACXBIWXMAABcRAAAXEQHKJvM/AAAAB3RJTUUH5AMDBzYLD5V4cgAAANtJREFUGBmNwbFKQmEcxuH316fH4HgmCxok0itoaWhyLZy6gUAMnB1aihxqESoaGhpqafBC6gKChKaGdjG0xUqC/H+f52Bjz6N/cwpYib9+lUHGlTCDH6WWJOXXoihq58uVSHNIVcyB4Ja3b3ksV4AG3vQenmXYhAYZ3fDyKbFFCyYxqSv6E7ENzZiFLjxSA/aq/HGKG24497q6Xlh4yKE6XofUMUbFXbwTgg6elOzEmCPMGaaHTHEfc8g55u5DyEtaZK7HEgqSNsHlSMYpmD7VcuZiJA9lStK75mZu0ijisbffhQAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAAASUVORK5CYII= +1ms

... more

Being possible follow a set of steps

Related:

• https://github.com/checkly/puppeteer-recorder
• https://netflix.github.io/pollyjs
• https://www.cypress.io

Exclude mailto urls

HTTP Code error not correct

Hello

I have make a test :

When i see the scan result, you delete "www."

And you can see lot of 500, but it is not 500 but 301 :)

Have a good day :)

Detect dynamic urls from scripts

Dynamic urls create for scripts are not detected, for example:

	<script>
		(function(i,s,o,g,r,a,m){i["GoogleAnalyticsObject"]=r;i[r]=i[r]||function(){
		(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
		m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
	})(window,document,"script","//www.google-analytics.com/analytics.js","ga");
		ga("create", "UA-34372837-2", "auto");
		ga("send", "pageview");
	</script>

Related:

• https://github.com/gajus/url-extractor

Support specify HTML selector