mozilla / potools

Debug tools for po files

Potools

A simple CLI utility to provide assistance with po-file based translations

Installation

To install globally use:

yarn global add potools or npm install -g potools

If you prefer to install this as a project dep then use:

yarn add potools --dev or npm install --save-dev potools

The command will be found in node_modules/.bin/potools.

Commands

debug

This command helps write out debug versions of pofile contents. It is both HTML and placeholder aware.

potools debug <potfile> [output] [format]

Options:
  --help    Show help                                                  [boolean]
  --format  The output format.
                             [choices: "unicode", "mirror"] [default: "unicode"]
  --output  Output path. If not provided outputs to stdout   [default: "stdout"]

Output Formats

There are two formats you can use:

• unicode: converts input to use unicode characters.
• mirror: converts input to use mirrored characters (recommended to be used via an RTL locale).

Placeholder Support

The debug command handles the following placeholder styles:

• %(placeholder)s
• %(placeholder)d
• %s
• %d
• {placeholder}

See caveats for special handling of named placeholders for HTML substitution in the mirror format output.

Debug Examples:

potools debug ../bar/locale/templates/LC_MESSAGES/messages.pot

Will output to stdout in unicode format.

potools debug ../bar/locale/templates/LC_MESSAGES/messages.pot --format mirror

Will output to stdout in mirror format.

potools debug ../bar/locale/templates/LC_MESSAGES/messages.pot --format mirror --output ../test.txt

Will write the ouptut to ../test.txt.

Debug command caveats

The RTL debug locale generation (--format mirror) generally inverts input to emulate what an RTL translation will look like. It will also handle HTML by reversing parts of HTML AST during processing in order to keep a sane order of elements but reverse the order of text and HTML.

E.g: foo <a href="#whatevs">bar <span>test</span></a> becomes <a href="#whatevs"><span>ʇsǝʇ</span> ɹɐq</a> ooɟ.

When placeholders are used to substitute HTML, this will cause problems if the placeholders are inverted. To fix this placeholders that are prefixed with start or end and have a matching suffix will remain in their original placement order.

E.g: foo %(startSpan)sbar%(endSpan)s will become %(startSpan)sɹɐq%(endSpan)s ooɟ to preserve order.

Development

Requirements

• git
• Node6+
• yarn (npm can be used in place of this if you prefer).

To get started first fork and clone this repo. Then run yarn install (or npm install) to install the dependencies.

Testing

To run the tests there are various commands:

• yarn test or npm test: Runs the lint + tests
• yarn test-coverage or npm run test-coverage: runs the tests with coverage
• yarn test-watch or npm run test-watch: runs the tests with a watcher.

potools's Issues

debug command mangles date formating strings

Mentioned by @EnTeQuAk on irc:

452:msgid "%%Y-%%m-%%d" -> 453:msgstr "%d%-ɯ%%-⅄%%"

Please add a LICENSE to this repository.

Typically Mozilla folks use MPL2, but I defer to whatever considerations are appropriate for this public repository.

Make placeholder support more comprehensive

The regexes used for placeholders are very basic and should be expanded to support a more comprehensive set of conversion modifiers.

For our needs this should cover the combined set of JS sprintf-js modifiers and the python ones.

We might also want to include some of the additional special placeholder syntax.

mirror format: Only reverse child nodes with a mix of text and HTML.

The reversal of the list of child nodes in the mirror transform should only really happen when there's a mix of text and HTML.

This way the main outline of any HTML will be preserved e.g. Headings and list order etc.

Wiki changes

FYI: The following changes were made to this repository's wiki:

• defacing spam has been removed

• Restricting write access to contributors is strongly encouraged. Please make that change (documentation).

These were made as the result of a recent automated defacement of publically writeable wikis.

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Add proper plural configuration

Unfortunately potools doesn't add any proper plural configuration header to the pofiles so errors like https://sentry.prod.mozaws.net/operations/olympia-dev/issues/608333/ keep occouring whenever we re-extract.

Let's maybe add them to the headers. See mozilla/addons-server@aba4b7c for an example configuration.

Don't pseudolocalise template literal variable names

I'm using ttag which allows me to express simple localised strings as:

t`User ${some.obj.name}`

(Complex cases, such as plurals. are supported too, but are not as concise)

This results in a .po file that looks like:

msgid "User ${ match.params.id }"
msgstr ""

After running potools debug I get:

msgid "User ${ match.params.id }"
msgstr "Ŭşḗř ${ ḿȧŧƈħ.ƥȧřȧḿş.īḓ }"

While I'd much rather get:

msgid "User ${ match.params.id }"
msgstr "Ŭşḗř ${ match.params.id }"

This is somewhat similar to #11

Version 10 of node.js has been released

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

• Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

More information on this issue

Greenkeeper has checked the engines key in any package.json file, the .nvmrc file, and the .travis.yml file, if present.

• engines was only updated if it defined a single version, not a range.
• .nvmrc was updated to Node.js 10
• .travis.yml was only changed if there was a root-level node_js that didn’t already include Node.js 10, such as node or lts/*. In this case, the new version was appended to the list. We didn’t touch job or matrix configurations because these tend to be quite specific and complex, and it’s difficult to infer what the intentions were.

For many simpler .travis.yml configurations, this PR should suffice as-is, but depending on what you’re doing it may require additional work or may not be applicable at all. We’re also aware that you may have good reasons to not update to Node.js 10, which is why this was sent as an issue and not a pull request. Feel free to delete it without comment, I’m a humble robot and won’t feel rejected 🤖

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

mirror format: A textNode that's pure whitespace shouldn't invoke reversal

If a text node is just whitespace it:

• Should be formatted
• It shouldn't invoke the childNode reversal (added in #5)

Placeholders that provide HTML get reversed when substituted

We occasionally have strings like:
foo %(startSpan)sbar%(endSpan)s

Which becomes:
foo <span>bar</span>

The problem is the mirror transform will make this into:
%(endSpan)sɹɐq%(startSpan)s ooɟ

This will be a problem when it gets used because this will become:
</span>sɹɐq</span> ooɟ

Replace NPM token

All NPM tokens were revoked so a new one needs to be issued.

Add a potools lint command

Dennis is a nice po linter but it does have some issues where it only warns for things that we'd consider errors see also mozilla/dennis#101

I'd like to consider making a lint command for potools that can error for the specific things we care about.

There's some additional useful prior art here too: https://github.com/shane-tomlinson/grunt-l10n-lint

HTML entities should be ignored in the mirror transform.

Currently entities like   will get inverted it would be nice to avoid that.