Bundling for browser usage about shiki

[LukasBombach]

So if anyone arrives here having troubles

✅ Bundling and self-hosting with Webpack 5 / Next js

this is a way to make it work.

When you simply import and use shiki like this

import { getHighlighter } from "shiki";

getHighlighter().then(highlighter => /* do stuff */);

You might get the error

TypeError: WebAssembly: Response has unsupported MIME type 'text/html; charset=utf-8' expected 'application/wasm'

This is not really a MIME-Type-problem. What is happening here is that the browser bundle loads a wasm file (and some other files) from the default location

• /dist/onig.wasm
• /dist/themes/[yourtheme].json

and those files are not there. You can change the path as described here, but you might also want to serve those files yourself. For this, you can copy the npm package to your static files in next

next.config.js

const path = require("path");
const CopyPlugin = require("copy-webpack-plugin");

const nextConfig = {
  webpack: (config, { webpack }) => {
    /**
     * Copying the whole npm package of shiki to static/shiki because it
     * loads some files from a "cdn" in the browser (semi-hacky)
     * @see https://github.com/shikijs/shiki#specify-a-custom-root-directory
     */
    config.plugins.push(
      new CopyPlugin({
        patterns: [
          {
            from: path.resolve(path.dirname(require.resolve("shiki")), ".."),
            to: "static/shiki/",
          },
        ],
      })
    );
    return config;
  },
};

module.exports = nextConfig;

And then set the CDN to your own served folder

- import { getHighlighter } from "shiki";
+ import { getHighlighter, setCDN } from "shiki";

+ setCDN("/_next/static/shiki/");

getHighlighter().then(highlighter => /* do stuff */);

[octref]

@canibanoglu Even if that's doable, is it desirable at all? Loading several MB of WASM and then the grammars to do syntax highlighting on client side?
I initially thought of shiki as an library, on either a static site generator system or locally, to faithfully reproduce VS Code's highlighting. I still think that's useful enough. If you need to dynamically do syntax highlighting in browser, maybe Monaco or Highlight.js would be better choices?

[zikaari]

Author of onigasm here. There is absolutely no problem running it in browsers, in fact, it was designed to be run in browsers for one of my own projects (now defunct).

It works perfectly fine with webpack with minimal config changes, just define a rule in webpack config to use file-loader for any import asking for *.wasm file.

Input

// src/index.ts
await loadWASM(await import('onigasm/lib/onigasm.wasm'))`

Output

// out/bundle.eu21df1.js
loadWASM('build/onigasm.8ug81e3.wasm').then(...)`

Note: Webpack might try to use it's on WASM loader, despite the explicit rule. If that happens, fix this like this - webpack/webpack#7264 (comment) The entire thread is a great read

[canibanoglu]

Oh, I quite agree with you there. I was just thinking that changing the way onigasm is loaded slightly would enable people who were inclined to bundle it for browser use could do so without too much trouble.

Instead of reading the WASM file from the file system with fs, we could use a loader to import it into code and webpack to take care of bundling for browser use if asked. Haven't tried anything yet though.

On a kinda related note: Is there a reason for not using node-oniguruma instead of onigasm for non-browser use?

[octref]

It should be possible, but I haven't had time for it. You should start by making Onigasm runnable in browsers.

[sammndhr]

If you're using Vue and gridsome, there's a plugin for it. https://github.com/EldoranDev/gridsome-plugin-remark-shiki @ipelekhan

[canibanoglu]

If you're using Vue and gridsome, there's a plugin for it. https://github.com/EldoranDev/gridsome-plugin-remark-shiki @ipelekhan

That library actually still runs on node.

[canibanoglu]

Hmm, I looked into this a bit but I'm left a bit confused. As it is shiki depends on onigasm instead of node-oniguruma. The onigLibs.ts file loads the WASM and takes care of everything but due to the way it reads the binary file it won't bundle with webpack due to the errors reported by @ipelekhan. If the goal is to depend solely on onigasm then we need to make sure that this works on both web and node without any adjustments by the end user by replacing the fs-dependent code segments.

One idea could be to introduce webpack to the project and depend on loaders to pull the raw data from files instead of using fs calls. Thoughts?

[octref]

vscode-textmate depends on either onigasm or oniguruma so I was using it.

[orta]

Related: microsoft/monaco-editor#1915

[lukeed]

Blocked by zikaari/onigasm#20

Hopefully onigasm can use an isomorphic lru-cache alternative. I'll even make one with a dispose hook if they need me to

[orta]

This is in with 0.9.0 - see the readme and #109

[Nedilko]

So if anyone arrives here having troubles

✅ Bundling and self-hosting with Webpack 5 / Next js

this is a way to make it work.

When you simply import and use shiki like this

import { getHighlighter } from "shiki";

getHighlighter().then(highlighter => /* do stuff */);

You might get the error

TypeError: WebAssembly: Response has unsupported MIME type 'text/html; charset=utf-8' expected 'application/wasm'

This is not really a MIME-Type-problem. What is happening here is that the browser bundle loads a wasm file (and some other files) from the default location

• /dist/onig.wasm
• /dist/themes/[yourtheme].json

and those files are not there. You can change the path as described here, but you might also want to serve those files yourself. For this, you can copy the npm package to your static files in next

next.config.js

const path = require("path");
const CopyPlugin = require("copy-webpack-plugin");

const nextConfig = {
  webpack: (config, { webpack }) => {
    /**
     * Copying the whole npm package of shiki to static/shiki because it
     * loads some files from a "cdn" in the browser (semi-hacky)
     * @see https://github.com/shikijs/shiki#specify-a-custom-root-directory
     */
    config.plugins.push(
      new CopyPlugin({
        patterns: [
          {
            from: path.resolve(path.dirname(require.resolve("shiki")), ".."),
            to: "static/shiki/",
          },
        ],
      })
    );
    return config;
  },
};

module.exports = nextConfig;

And then set the CDN to your own served folder

- import { getHighlighter } from "shiki";
+ import { getHighlighter, setCDN } from "shiki";

+ setCDN("/_next/static/shiki/");

getHighlighter().then(highlighter => /* do stuff */);

I'm trying to do like you suggested, but I'm getting a lot of errors in nextjs.

[Nedilko]

It outputs no file nord.json at .... .next/server/app/(blog)/article/themes/nord.json

[Nedilko]

next 13 server components

[iamnbutler]

I found this issue looking for how to use shiki with Next.js, if you did as well the discussion you are looking for is likely here: #398

[zm-cttae-archive]

If the problem is WASM bundling in Webpack this works - zikaari/onigasm#2 (comment)

[max-programming]

So if anyone arrives here having troubles

✅ Bundling and self-hosting with Webpack 5 / Next js

this is a way to make it work.

When you simply import and use shiki like this

import { getHighlighter } from "shiki";

getHighlighter().then(highlighter => /* do stuff */);

You might get the error

TypeError: WebAssembly: Response has unsupported MIME type 'text/html; charset=utf-8' expected 'application/wasm'

This is not really a MIME-Type-problem. What is happening here is that the browser bundle loads a wasm file (and some other files) from the default location

• /dist/onig.wasm
• /dist/themes/[yourtheme].json

and those files are not there. You can change the path as described here, but you might also want to serve those files yourself. For this, you can copy the npm package to your static files in next

next.config.js

const path = require("path");
const CopyPlugin = require("copy-webpack-plugin");

const nextConfig = {
  webpack: (config, { webpack }) => {
    /**
     * Copying the whole npm package of shiki to static/shiki because it
     * loads some files from a "cdn" in the browser (semi-hacky)
     * @see https://github.com/shikijs/shiki#specify-a-custom-root-directory
     */
    config.plugins.push(
      new CopyPlugin({
        patterns: [
          {
            from: path.resolve(path.dirname(require.resolve("shiki")), ".."),
            to: "static/shiki/",
          },
        ],
      })
    );
    return config;
  },
};

module.exports = nextConfig;

And then set the CDN to your own served folder

- import { getHighlighter } from "shiki";
+ import { getHighlighter, setCDN } from "shiki";

+ setCDN("/_next/static/shiki/");

getHighlighter().then(highlighter => /* do stuff */);

This worked for me! Thanks @LukasBombach