jsdelivr / dash-directus

The jsDelivr dashboard backend

jsDelivr - Open Source CDN

A global super-fast and production-focused CDN, tightly integrated with NPM and GitHub
with support for on-the-fly optimizations, ES modules, detailed download stats and more

www.jsdelivr.com

Website Repo · Public API · Blog · @jsDelivr · Discord

---

We are looking for contributors. Please check open issues in the above repos if you think you could help, or open a new one if you have an idea you'd like to discuss.

jsDelivr is a free CDN for open-source files. We are tightly integrated with Github and npm, allowing us to automatically provide a reliable CDN service to almost every open-source project out there.

We offer a stable CDN that can be used in production on popular websites with huge amounts of traffic. There are no bandwidth limits or premium features, and it's completely free to use.

How does it work - A simple infographic

Why jsDelivr?

Ready for production

Our public CDN is built to be used in production by even the largest websites. Everything is optimized and constantly improved to offer all users maximum speed and uptime. Performance is monitored at all times, and we are always looking into new technologies and providers that may further improve our CDN. Downtime, timeouts, or slow responses are simply unacceptable.

We do everything possible to ensure our CDN will NEVER break any websites, regardless of the use case. If a file is available via our CDN, we assume it's used in production and ensure it will continue to work no matter what.

This includes dynamic endpoints such as /npm/, /gh/ and /combine/. When a file is first accessed, it gets permanently stored in a reliable file system. This means that even if a npm package gets deleted or an existing file gets removed by a developer, jsDelivr will continue to serve the stored copy forever without breaking any websites or causing any issues.

On top of that, we also do version fallback. This means that if a file used in version 1.0.1 is no longer available in 1.0.2 and a user requests the non-existent file, we will fall back to the previous 1.0.1 version and serve it instead of failing with a 404 error.

Multi-CDN

Unlike the competition, jsDelivr uses multiple CDN providers, resulting in the best possible uptime and performance. We currently use CloudFlare, and Fastly.

If a CDN goes down, websites that use jsDelivr won't have any issues because all traffic will be instantly redirected to remaining operational providers.

Smart Load Balancing

jsDelivr uses real user performance data (also known as RUM) to make its routing decisions. These metrics are gathered from hundreds of websites and are used in our load-balancing algorithm to make accurate decisions for serving content.

All providers (CDNs and custom servers) are tested millions of times per day by real users from all over the world. Based on this information, jsDelivr knows what provider is the fastest for each user. Each user gets a unique response based on his or her location, ISP, and the providers' uptime in real-time.

This system also responds immediately to performance degradation and downtime of providers. If a CDN is under a DDoS attack, and its performance drops in some locations, in a matter of seconds, the algorithm will pick up the change and start serving a different provider to all affected users.

Failover

We have multiple layers of failover to protect our users from any downtime.

We use 2 DNS providers at the same time. For jsDelivr to go down, both of these companies would have to go down at the same time.

Both of our DNS providers monitor our load-balanced endpoint, and if they detect problems, they will automatically switch all traffic to a single CDN provider.

Our load balancer monitors the uptime of all CDN providers using both RUM and synthetic data. If any of those detect downtime or performance degradation, that CDN provider will be removed immediately without any impact on our users.

Our origin consists of multiple servers in different data centers. If a server goes down, the CDNs will automatically switch to using the remaining healthy servers.

In total, we have one of the most resilient systems out there, ready to be used in production by even the biggest companies.

China

jsDelivr has multiple locations close to Chinese urban centers to ensure low latency and high performance for all Chinese users.

Usage Documentation

jsDelivr provides mirrors for npm, GitHub, WordPress plugins, and custom endpoints for several other projects with special requirements. If our regular endpoints don't work for your use case, let us know and we'll figure something out!

If you are a package author, check our tips for package authors to make using your package as easy as possible.

Root endpoint is always https://cdn.jsdelivr.net

npm

jsDelivr can instantly serve any file from any npm package in the public registry. New versions pushed to npm are instantly available via our CDN as well. No maintenance is required.

If a package, version, or file gets removed from npm, then jsDelivr will continue to serve that file from our permanent storage without breaking any websites using it.

We use a permanent S3 storage to ensure all files remain available even if npm goes down or a package is deleted by its author. Files are fetched directly from npm only the first time or when S3 goes down.

Load any project hosted on npm:

/npm/package@version/file

Load exact version:

/npm/[email protected]/dist/jquery.min.js

Use a version range instead of an exact version:

/npm/jquery@3/dist/jquery.min.js
/npm/[email protected]/dist/jquery.min.js

---

NOTE

If you use this feature and a file you requested is not available in the newest version of the package, the link will keep working thanks to our version-fallback feature. We'll continue to serve the file from the older version of the package instead of failing with a 404 error.

---

Load by tag (Not recommended for production usage):

/npm/jquery@beta/dist/jquery.min.js

Omit the version completely or use "latest" to load the latest one (not recommended for production usage):

/npm/jquery@latest/dist/jquery.min.js
/npm/jquery/dist/jquery.min.js

---

NOTE Requesting the latest version (as opposed to "latest major" or "latest minor") is dangerous because major versions usually come with breaking changes. Only do this if you really know what you are doing.

---

Add ".min" to any JS/CSS/SVG file to get a minified version - if one doesn't exist, we'll generate it for you. All generated files come with source maps and can be easily used during development:

/npm/[email protected]/github-markdown.min.css

---

NOTE Minifying a large file can take several seconds. However, we store all generated files in our permanent storage, so this delay only applies to the first few requests.

---

Omit the file path to get the default file. This file is always minified:

/npm/[email protected]
/npm/jquery@3
/npm/jquery

Get a directory listing:

/npm/[email protected]/
/npm/[email protected]/dist/

GitHub

We recommend using npm for projects that support it for better UX - npm packages are searchable on our website, and package pages show additional useful information, such as descriptions and links to homepages.

We use a permanent S3 storage to ensure all files remain available even if GitHub goes down or a repository or a release is deleted by its author. Files are fetched directly from GitHub only the first time or when S3 goes down.

Load any GitHub release, commit, or branch:

/gh/user/repo@version/file

Load exact version:

/gh/jquery/[email protected]/dist/jquery.min.js
/gh/jquery/jquery@32b00373b3f42e5cdcb709df53f3b08b7184a944/dist/jquery.min.js

Use a version range instead of an exact version (only works with valid semver versions):

/gh/jquery/jquery@3/dist/jquery.min.js
/gh/jquery/[email protected]/dist/jquery.min.js

---

NOTE If you use this feature and a file you requested is not available in the newest release, the link will keep working thanks to our version-fallback feature. We'll continue to serve the file from older release instead of failing with a 404 error.

---

Omit the version completely or use "latest" to load the latest one (only works with valid semver versions): (not recommended for production usage)

Falls back to the master branch if there are no tagged releases.

/gh/jquery/jquery@latest/dist/jquery.min.js
/gh/jquery/jquery/dist/jquery.min.js

---

NOTE Requesting the latest version (as opposed to "latest major" or "latest minor") is dangerous because major versions usually come with breaking changes. Only do this if you really know what you are doing.

---

Add ".min" to any JS/CSS/SVG file to get a minified version - if one doesn't exist, we'll generate it for you. All generated files come with source maps and can be easily used during development:

/gh/jquery/[email protected]/src/core.min.js

---

NOTE Minifying a large file can take several seconds. However, we store all generated files in our permanent storage, so this delay only applies to the first few requests.

---

Get a directory listing:

/gh/jquery/[email protected]/
/gh/jquery/[email protected]/dist/

Combine multiple files

Our combine endpoint allows you to load several files from npm and GitHub endpoints in one request:

/combine/url1,url2,url3

All features that work for individual files (version ranges, minification, main modules) work here as well. All combined files come with source maps and can be easily used during development.

Examples:

/combine/gh/jquery/[email protected]/dist/jquery.min.js,gh/twbs/[email protected]/dist/js/bootstrap.min.js
/combine/npm/[email protected]/dist/css/bootstrap.min.css,npm/[email protected]/dist/css/bootstrap-theme.min.css

---

NOTE Combining large/many files can take several seconds. However, we store all generated files in our permanent storage, so this delay only applies to the first few requests.

---

Publishing packages

All packages hosted on npm and tagged releases on GitHub are automatically available on jsDelivr. If you are a package author, here are a few tips to make using your package as easy as possible:

• Use semver for versioning (this is enforced by npm but not by GitHub)
• If a file listed as main in package.json isn't meant to be used in a browser, set a browser or jsdelivr field
• If you distribute minified JS/CSS files, also include source maps for those files
• If you don't want to provide minified files, it's fine - we'll handle that for you

Configuring a default file in package.json

For packages hosted on npm, we support serving "default" files with shorter URLs. The default file can be configured by setting one of the following fields in package.json, with jsdelivr having the highest priority:

1. jsdelivr
2. browser
3. main

We will first attempt to locate a minified version of the file provided here (by removing the extension and looking for the same file .min.js). If we can't find one, we will minify ourselves.

Be advised that you must include the file extension in the values, for example:

"main": "./index" // this will NOT work
"main": "./index.js" // this is the correct way

For projects having both a JS and a CSS file, use one the above fields for JS and a style field for the CSS file (example).

Restrictions

• Packages larger than 150 MB or single files larger than 20 MB (in the case of GitHub) are not supported by default. We recommend removing files that are not needed from your package when possible. If you need to set a higher limit for your package, open an issue in this repo.
• HTML files are served with Content-Type: text/plain for security reasons.

WordPress

Our WordPress endpoint works for plugins and themes hosted in the WordPress.org plugin directory and Wordpress.org theme directory, and mirrors the WordPress.org plugins SVN repo.

Load any plugin from the WordPress.org plugins SVN repo:

/wp/project/tags/version/file

Load exact version:

/wp/wp-slimstat/tags/4.6.5/wp-slimstat.js

Load the latest version (not recommended for production usage):

/wp/wp-slimstat/trunk/wp-slimstat.js

Load any theme from the WordPress.org themes SVN repo:

https://cdn.jsdelivr.net/wp/themes/project/version/file

Load an exact version of a file:

https://cdn.jsdelivr.net/wp/themes/twenty-eightteen/1.7/assets/js/html5.js

Add ".min" to any JS/CSS file to get a minified version - if one doesn't exist, we'll generate it for you. All generated files come with source maps and can be easily used during development:

https://cdn.jsdelivr.net/wp/themes/twenty-eightteen/1.7/assets/js/html5.min.js

Caching

Our caching logic and headers are optimized for production use and apply to all non-custom endpoints.

• Static Versions and commit hashes - Effectively forever. The caching headers are set for 1 year but we also permanently cache the files in our S3 storage. So all future requests that bypass the CDN will hit our S3 storage with no option or way to update the contents of that file.
• Version aliasing - 7 days. This also includes latest versions. They are cached on our CDN for 7 days with the option to purge the cache using our API to speed up the release of your project to your users.
• Branches - 12 hours.

In certain cases, purgeable files can get updated faster due to low-cache hit ratio or forced CDN purge from our side for maintenance reasons.

We use permanent S3 caching even with dynamic URLs, such as version aliasing, meaning once we download your tagged files, there is no way for you to update them. If there is a critical issue in your latest release the best course of action is to tag a new semver valid release with the fix and purge the CDN URLs using our purge API.

Purge cache

jsDelivr has an easy-to-use API to purge files from the cache and force the files to update. This is useful when you release a new version and want to force the update of all version-aliased users.

Please note:

• It will not work for static files as explained above.
• Valid semver releases must be used for purge to work
• Rate-limiting applies to all users

To avoid abuse, access to purge is given after an email request (for now - [email protected]).

Custom CDN Hosting

We can work together and set up a custom configuration for your project. This way, you can have full control over your files and the ability to utilize the full power of jsDelivr.

This kind of custom hosting can be suitable for:

• Binary hosting. Windows executable files and zips.
• Frequently updated files.
• Projects that can't follow jsDelivr file structure.
• Some other use that will blow all of our minds.

Simply send an email to [email protected] with a request for more information.

Current OSS projects using custom configs:

• webjars
• webpack webpackbin and codesandbox
• emojione.com
• hex.pm

Privacy Policy

cdn.jsdelivr.net

jsDelivr might use information about downloaded files to build download stats per project and per file.

jsDelivr does not store any user data and does not track any users in any way.

Here are the relevant policies of our CDN providers:

• Cloudflare
• Fastly

The bus factor

dash-directus's Issues

Staging server

Before we make the dashboard public, we should set up a separate staging environment. I'm thinking a separate render.com service, and the DB server can stay the same, just use a separate database.

Add probe data

• hardware version, if any
• status must be in sync

Probe adoption issue

I've run several times into a situation where the API took more than 5 seconds to respond, so we run into a timeout here and didn't store the code as valid. However, the probe actually did get the code a second later, and it was confusing the code didn't work then.

Partly, this is a problem of not showing the timeout error in the current UI, but I think we could also store the code regardless of the API response. If the user happens to get it, they can use it without issues, even if the request "timed out". If the user doesn't get it, they'll request a new one that overrides the old one, so no harm is done.

Add a user type field

We already have the system that assigns users GP credits for sponsoring, but we also need an attribute on the user like isActiveSponsor: boolean that can be used for UI customizations and enabling some extra non-GP features. For one-time sponsors, this should probably say false all the time.

Create prod deployment dockerfile and scripts

1. single docker image that contains all built files
2. npm run migrate - connects to the configured (remote) instance and runs any new migrations
3. npm run init - initial setup script

Cache GitHub requests

As we're making more and more requests to the GitHub API, we'll need to consider its API limits and make some improvements. One thing that helps significantly without affecting our use in any way is caching and making conditional (ETag-based) requests, which don't count toward the API limits if the response is a 304.

Minor refactoring

• tokens:
  • change the generator endpoint from GET to POST and require the user to be logged in
  • for the Origins input, use the same UI element as for Probe tags, and validate each separate value on the backend via new URL(); automatically add "https://" protocol if the user doesn't include it - the point is that there needs to be a clear visual indication of how the values were understood/parsed after saving
• add prefix gp_ to tables: adopted_probes, credits and move the two collections under new "Globalping" directory in the UI
• rename tokens to jsd_purge_tokens and in the UI to "Purge API Tokens" under jsDelivr
• add gp_tokens collection with the same functionality as jsd tokens; call these "Globalping Tokens"

Ability to select username/organization as a tag prefix

User should be able to select an org from a dropdown and free type the tag part.

UI:

• a list with 2 items per row, prefix select on the left and tag input on the right;
• ability to add new tag;
• ability to remove existing tag;
• “Org Sync” button to fetch new org values from GitHub.

New tags prefixes are validated to exist in current user orgs list. Organisations list is synced on every login + on button click.

Sync of users github usernames

Original conversation: jsdelivr/globalping#427 (comment)

Updating usernames on login seems not very obvious from my perspective.
While updating with 5 min intervals cron means "Usernames in your tags are updated automatically within 5 minutes." which is clear. So I'd personally go with that option.

User settings

Users should be able to update their:

• name
• email
• light/dark theme preference

The username is always synced from GitHub.

Credits generated per probe

We plan to show the generated credits in the past month for each probe, but right now, we don't have the credits <-> probe connection. I'm thinking an optional reference to a probe in the credits additions will be best?

DB config

We need to update the collation/character set values in the server config:

[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_unicode_ci
init-connect = 'SET names utf8mb4 COLLATE utf8mb4_unicode_ci;'

[client]
default_character_set = utf8mb4

[mysql]
default_character_set = utf8mb4

Auto-ban accounts if banned by Github

Spammers like creating fresh Github accounts to abuse services like our CDN. They will try to abuse our dashboard too.
The good thing is that Github seems to be quick to ban those accounts after a while.
So we need a way to sync those bans and disable accounts on our side too.

Seed data

Create a script to seed mock data, for ease of development and testing.

Geodata of an IP range

Need to implement a way for an admin to modify the geodata of an IP range using dashboard.

Users should be able to overwrite the admin on adopted probes.

Adopted probes migration broken

I ran into a couple of issues while deploying #21, most notably the columns in the old and new probe tables ended up in an entirely different order, which broke the INSERT INTO gp_adopted_probes SELECT * FROM adopted_probes Let's check what that happened and fix it in case we ever need to run this again. Also, check the other collections being migrated.

Since we're not live yet, I've just dropped all existing probes right now.

Split GP/JSD

After https://github.com/jsdelivr/meta/issues/50, we'll need to split this into two repos.

Deleting accounts

Users must be able to delete their accounts. This means:

• adopted probes are deleted,
• existing credits and their history are deleted (though if they keep sponsoring, new additions will appear over time),
• tokens are deleted,
• sponsor records are kept as before.

Auth integration flows

We have the basic tokens functionality, but it hasn't been integrated into any of the services and clients. I'll describe the flows I expect to use in each integration, and if we agree it's all good, I'll then create the issues in the affected repos.

jsDelivr Purge / Globalping homepage GUI app

We'll configure Directus to set its cookies for jsdelivr.com so that any subdomain can read them. Then, we'll be able to detect if the user is logged in to the dashboard from the website and perform any API calls. This way, our front-end code on the website can request a token for the specific API and use it without the user having to do anything.

Tokens created this way will have a special flag in the DB and will not be visible in the user's dashboard. For security reasons, they also have a short TTL (1 day). This doesn't impact the UX in any way since the UI can always request a new token in the background as long as the user is signed in the dashboard (and prompt to sign if not).

Globalping CLI

A token must be created and added manually, probably either set via an env var or a config file. A new command can also be added that:

1. Prints a link like https://dashboard.jsdelivr.com/tokens/new/?name=Globalping%20CLI.... which brings the user to the correct page with values pre-filled.
2. Accepts the token as input and saves it to the correct file.

Of course, if the user already has a token, they skip step 1 and simply paste it.

Globalping Discord / Slack

A token must be created and added manually via a new command. It is then stored for that app installation in our DB. The flow can be similar to the CLI, or we can optionally not provide the link here (step 1) if we don't want it to be "too easy" as we previously discussed.

jsDelivr Purge / Globalping API

Requirements:

• new tokens must work instantly, without any delay
• deleted / revoked tokens should stop working reasonably fast
• don't query the DB on every single request

Suggestion:

• query the DB once a minute for all tokens; for each token found, store it in memory as valid for the next two minutes
• if a request comes with a token that isn't in memory - query the DB for the specific token and store the result - valid/invalid - for the next two minutes

Note that this applies only to requests not using credits. If the users exceeds their time-based requests quota and has credits, that'll be handled separately in a next step.