Automatic tree-shaking of messages about next-intl

[nbouvrette]

Great to see I'm not the only one wanting to solve this problem. I don't know if you guys saw but I split the next-multilingual logic that dynamically loads message modules into this package: https://github.com/Avansai/messages-modules

It uses Babel to dynamically inject the message in their local scope which is pretty neat but one optimization I would like to fix is that it loads all locales and not just the one being viewed... which of course would end up causing performance issues in the long run. But I think it's still better than a huge message file containing messages that are not even used

The big problem ahead is that from what I understood the app directory cannot use Babel... so basically I will need to rewrite this package in Rust so that it can work with SWC.

If you guys are interested to team up, I would be more than happy to get some help. Otherwise, I will most likely work on this later this year (Q3?) which should also give extra time to make the app directory more stable

[stylehuan]

Is there any progress?

[amannn]

An alternative (or possibly addition) is a concept based on the recently announced server components.

Code like this:

function LatestFollower({user}) {
  const t = useTranslations('LatestFollower');

  function onFollowClick() {
    UserAPI.follow(user.id);
  }

  return (
    <>
      <Text>{t('latestFollower', {username: user.name})}</Text>
      <IconButton
        aria-label={t('followBack')}
        icon={<FollowIcon />}
        onClick={onFollowClick}
      />
    </>
  );
}

… could be turned into:

function LatestFollowerServer({user}) {
  const t = useTranslations('LatestFollower');

  return (
    <LatestFollowerClient
      $1={<Text>{t('latestFollower', {username: user.name})}</Text>}
      $2={t('followBack')}
      $3={<FollowIcon />}
    />
  );
}

function LatestFollowerClient({$1, $2, $3, user}) {
  function onFollowClick() {
    UserAPI.follow(user.id);
  }

  return (
    <>
      {$1}
      <IconButton aria-label={$2} icon={$3} onClick={onFollowClick} />
    </>
  );
}

These are the interactive features that React currently has:

• useState
• useReducer
• use(Layout)Effect
• refs (and thus useImperativeHandle)
• event listeners

Most are easy to detect, but event listeners seem a bit tough, since it could potentially be a render prop. A convention like on[A-Z].* would definitely help here.

[amannn]

For very performance critical apps, it might make sense to use components that make use of next-intl only on the server side and to provide labels as strings to interactive client components. That way you wouldn't have to tree shake anything.

[amannn]

A manual approach seems easy and reliable: #62

[nbouvrette]

Thanks @amannn for sharing this with me. For those who might be interested, we built a similar Babel plugin solution that works on Next.js: https://github.com/Avansai/next-multilingual

Of course, as called out, one of the drawbacks is that all locales are loaded in the same bundle which increases the size. I am planning to try to find a solution as we are adding an SWC version of the plugin.

[amannn]

@yashvesikar just shared a POC for a webpack plugin that extracts messages based on page entry points with me: https://github.com/yashvesikar/next-intl-translation-bundler-example

I'm still mind blown 🤯

AFAICT what we'd have to do (assuming we'd implement this for the app directory) is:

1. Detect all entry points. In the POC, every file is treated as an "entry point", meaning that it will emit translations. This would need to be adapted to only handle modules that have the 'use client' banner as these are the boundaries where we're switching to the client side.
2. For every entry point, traverse the module graph and find all references to useTranslations/t and collect the namespaces.
3. Emit the translations per entry point.

The other part is that this is a webpack plugin, no idea if these will be supported by Turbopack or if we'd have to implement it in Rust instead (if Turbopack even supports plugins at this point).

Edit: An improved version of the webpack plugin that supports the App Router was implemented in https://github.com/jantimon/next-poc-i18n-split.

---

Another related project: https://lingui.dev/guides/message-extraction#dependency-tree-crawling-experimental

[yashvesikar]

@amannn Unfortunately, I don't have much knowledge of the app directory and all that RSC entails. I can speak to some of the advancements I implemented to this logic in the past and some of the ones I didn't get to.

• Support for colocated translations by component — Similar to CSS modules. This has a few advantages:
  • IMO it's easier to spot and fix & address translation bugs and errors.
  • Gives the translation plugin full control over the key names and allows the user to use TS type checking! (rough idea) i.e.

// src/components/my-component/i18n/en-US.json
{"title": "Title in English", "other": "Some other Key that won't be bundled"}
// src/components/my-component/i18n/fr.json
{"title": "Title in French"}

// src/components/my-component/i18n/index.ts
// loader can check that the keys match the ones in the main locale json 
let translations = {title: "title"} as const

// src/components/my-component/my-component.tsx
import { translations } from "/.i18n

export const MyComponent() => {
	// by modifying the useTranslations hook we can make t(key) have key be typeof translations
	const t = useTranslations(translations); 
	
	return(
		<div>{t(translations.title)}</div>
	)
}

• Figuring out how to get Webpack to load the translation files as dependencies.
  Right now in the POC, I made the server is loading the files from the disk using fs.readFileSync. This is probably the worst way to read that data for high-traffic scenarios because it is a blocking call. Even making that call async will be much better, even better might be to figure out how to have Webpack resolve the dependency itself such that the translations for a given page/path are available to the getServerSideProps function the same way any other import would be. I spent a little time looking into that but Webpack beat me 😅
  An open question is whether caching would be better done this way or via a separate network request with a long max-age.

• Improving the HMR, right now the HMR in dev kinda works. The POC plugin will build the right translation output files the first time the page is loaded in dev, but if the translation JSON file is changed the server needs to be restarted.

Some known limitations are that the Webpack parser is very limited and fickle. It's not a great way to traverse the AST to get all the keys, but since it's a single toolchain it is nice. I don't have any experience in SWC or TurboPack parsers but I have a hard time imagining they are worse than Webpack.

There are a few more things but I will leave it here for now. Sorry for the wall of text, happy that this may be useful to the project :)

[amannn]

Hey @yashvesikar, thanks a lot for chiming in here and sharing your thoughts in detail!

Support for colocated translations by component

That's an interesting one, I think @nbouvrette is exploring something similar to this.

At least for next-intl, I'm currently not pursuing this direction though and rather try to make it easy to work with message bundles per locale as from my experience this integrates well with using translation management systems. We do have type checking though btw.!

Improving the HMR

Good point! That's something that's working fairly well currently, I'd have to investigate this for sure.

I don't have any experience in SWC or TurboPack parsers but I have a hard time imagining they are worse than Webpack.

😆 Well, I've read that Turbopack is adding support for Webpack loaders, I'm not sure what the story for plugins is though. This topic is generally a bit futuristic at this point and Turbopack is still quite experimental from what I know. At least for the time being, a webpack plugin would definitely be the way to go.

(continuing from our Twitter thread) in nextjs every page is an entrypoint + a main entrypoint for shared assets. so extracting all keys for a page is the same as by entrypoint. the webpack docs are simultaneously really useful and not useful enough 😅

Hmm, I just console.log'd the entry points—and I can confirm the part with the pages being entry points! But e.g. if pages/A.tsx imports components/Profile.tsx, how can I find this reference to process the dependency graph recursively?

Generally I'm currently still quite busy with the RSC integration for next-intl, but your thoughts are really helpful here, I hope to find some time to revisit this topic in the future! Made me laugh the other day when I saw that this was issue #1! 😄

[gurkerl83]

Maybe something like like this can help to omit loading all translation files at ones, esp. the caching strategy is interesting here one variant of the server part. https://github.com/QuiiBz/next-international/blob/v2/packages/next-international/src/app/server/create-get-scoped-i18n.ts
The client part especially how translations are instructed to load is also interesting, also this requires no plugin.

Maybe this helps
Thx!

[amannn]

In next-international, getScopedI18n is about being able to provide a namespace, which is not supported by getI18n as far as I understand. Loading of messages upfront via createI18nServer/createI18nClient is the same for either API. That's at least my understanding.

next-intl handles both cases with one API via getTranslations(namespace?) and useTranslations(namespace?).

The current plan for this feature is to collect namespaces and keys with a compiler!

[gurkerl83]

I apologize for the misunderstanding in my earlier message about scoped translations - I should have read the documentation more thoroughly. Scoped translations help streamline code by reducing redundancy.

getI18n vs getScopedI18n

const t = getI18n();
console.log(t('pages.settings.published'));
console.log(t('pages.settings.title'));

const t = useScopedI18n('pages.settings');
console.log(t('published'))
console.log(t('title'));

About the last sentence...

The current plan for this feature is to collect namespaces and keys with a compiler!

In which context, the issue about tree shaking or how next-intl currently handles internals?

Thx!

[amannn]

No worries! Yep, I agree that scoping translations is helpful. next-intl is heavily based on that too! 🙂