locize / locize-cli Goto Github PK

Bug Report

I needed to remove 652 keys from all (5) languages, and there is no bulk removal command (that would accept e.g. a file as arg, that lists all the keys to remove) so I ran a script that calls locize remove translation <key> for every key ("translation" is the namespace).
It was very slow for each key, in total the script took over 7 hours to run. Why is it so slow?
Ideally, removing one key should take only a few seconds maximum.

To Reproduce

Have ~3k locize keys in 5 languages. Remove 652 with the CLI like described above.

Expected behavior

Removing one key takes only a few seconds maximum.

Your Environment

• locize version: locize CLI v7.6.14
• os: Linux

When calling SYNC

othing to update for en/functionLists
adding 29 keys in en/functionUnits...
adding 3888 keys in en/functions...
adding 1017 keys in en/menus...
adding 23 keys in en/modules...
adding 12 keys in en/tabs...
adding 12 keys in en/viewTypes...
Error: Please consider to page your requests! Currently sending 1017 keys, but should send max 1000 keys!
at Request.request [as _callback] (/usr/local/lib/node_modules/locize-cli/sync.js:371:19)

When calling MIGRATE

transfering latest/de/functionLists...
transfering latest/de/functionUnits...
transfering latest/de/functions...
transfering latest/de/menus...
transfering latest/de/modules...
transfering latest/de/tabs...
transfering latest/de/viewTypes...
transfering latest/en/functionLists...
transfer failed for latest/de/tabs...
Error: Project 04c3b685-8a27-442d-9f86-d6c827364c7a has no language de!
at Request.request [as _callback] (/usr/local/lib/node_modules/locize-cli/migrate.js:85:63)

A high severity security advisory (as well as a medium one) has been issued for js-yaml, which is a dependency for locize-cli

Update version to 3.13.1 in order to satisfy the advisory

% locize download --api-key xxx --project-id 3f516219-6546-4f2a-9030-fc31659dc115 --ver latest --language en-US --format yaml --namespace translation
Invalid content for "yaml-rails" format!
unacceptable kind of an object to dump [object Undefined]

Other formats - json or yaml - work fine.

🚀 Feature Proposal

Add command to rename a key

Motivation

When a bunch of keys should be renamed, it's not equivalent to removing the keys and adding them under a new name because then the values of the non-reference languages are deleted.
It would be useful to have a subcommand for renaming keys.
It allows bulk changes much easier.

From documentation:

if you have less keys (you have deleted some keys) in your local namespace it will remove them in locize too

This can be a problem when different people work in their branches and run sync. They may delete each other keys, because person A doesn't have keys from person B's branch and vice versa.

Is it possible to add an option to run sync without deletion of missing keys?

Reference lang won't get updated values if skip-delete is enabled and there are extra keys in locize. This seems to be caused by 'somethingToUpdate' flag which removes reference lang from downloadAll if any changes are detected (even if skipped).

Would assume that values are always updated from locize and skipDelete would just skip the extra keys.

I think you should turn sync command a bit safer.
If by accident we execute sync command out of languages directory, all directories on the current path will be deleted with rm -rf, so not recoverable.
I made that mistake, fortunately, inside of a project folder without any uncommitted code. But it could be much dangerous, for example inside home dir.
There are many options available to avoid this.

• Require a path argument with the path of the languages dir
• Require a confirmation when rm is about to delete anything.
• Do rm with -i

I had a json translation file with nested structure like the following

"errors": {
  "401": "Not authorized",
  "404": "Page not found"
}

When running the sync command the first time it added the keys as expected in locize.io. But trying to download the namespace as nested json I got an json with lots of null keys in an array. Discovered it when the sync command reported "removing 498 keys from xyz". I only had like 40 keys in total. Somewhere along the line numeric keys isn't handled nicely, might not be this project. Hard for me to tell though.

"errors": [
    null,
    null,
    null,
    null,
    null,
    null,
    null,
    ...

When syncing reference language from project to locize.com via sync command. The sync process reorders all the translation keys in the file alphabetically.
Since we try to organize the translations inside the file in logical order, it would be really nice to suppress this functionality via configuration (command line argument).

I started implementing the locize-cli today, when i tried adding the api-key and project-id i get this error

dding myNewKey to latest/en/ns1...
add failed for myNewKey to latest/en/ns1...
Unauthorized

In case a new key is added locally and another key's value is updated on locize. In that case only new key is added to locize. But the key value change of some other does not get applied locally. So only upstream works and down stream doesn't.
Are we only doing either upstream or downstream at a time? Both doesn't happen in one sync?

example:

Local file:

{
"oldKey":"oldValue",
"newKey":"Some Value" // new key
}

on Locize:

{
"oldKey":"newValue"
}

After Sync

Local file:

{
"oldKey":"oldValue", // Still old value here.
"newKey":"Some Value"
}

on Locize:

{
"oldKey":"newValue",
"newKey":"Some Value" // new key added 
}

The only workaround for this is to call sync twice.

I'm running into an issue that is preventing me from syncing locize locally with my repository. I've got my reference language in my repo and within Locize set to en and I'm running locize sync in the locales directory and I end up with the error saying Error: Reference language not found!

My directory structure looks like the following:

I feel like I'm missing something obvious here. Both of those file names are namespaces which exist in the Locize interface.

Additionally is there any React documentation on disabling locize in production, and instead forcing i18n to use the synchronized json files in the locales directory instead of the API?

sync has option for cleaning local files. I would assume this would affect only localization dir but it seems to delete everything in the working dir. In my case everything in project folder as this was tried as a npm script.

Hello!
I use locize-cli to synchronize localization files with locize.
In the project we use the PO file format.
Here is an example of a valid key.

msgid "tied with% {count} other% {test_one} test"
msgid_plural "tied with% {count} others% {test_one} test"
msgstr [0] "à égalité avec% {count} other% {test_one} test"
msgstr [1] "à égalité avec% {count} others% {test_one} test"

After locize sync --api-key api_key --project-id project_id --format po
we see the following fields in locize:

"tied with% {count} other% {test_one} test": "à égalité avec% {count} other% {test_one} test",
"tied with% {count} other% {test_one} test_plural": "à égalité avec% {count} others% {test_one} test"

When you need to synchronize translation files or download them using locize download, we get the following:

msgctxt "one} test"
msgid "tied with% {count} other% {test"
msgid_plural "tied with% {count} other% {test"
msgstr [0] "à égalité avec% {count} other% {test_one} test"
msgstr [1] "à égalité avec% {count} others% {test_one} test"

From this result questions follow.
In msgid and msgid_plural identical keys are stored. When requesting the key tied with% {count} others% {test_one} test we will not receive a translation.
And the second question.
Why is everything going after "_" placed in msgctxt?
As far as I know, msgctxt is used for Disambiguating Contexts.

Currently some of our strings in Android project are formatted in HTML format. When we add translations in other languages in HTML format, then downloaded strings from locize escape characters < and > of HTML. Thus our strings.xml is not valid and builds fail.

For e.g., we have a string

<![CDATA[
            For <b>Sports</b>, try sport types, teams or leagues.<br />
            For <b>Movies</b>, try titles, actors or genres.<br />
            For <b>Shows</b>, try titles or genres.
]]>

Translated version is

<![CDATA[
            Para <b>Deportes</b>, intente clase de deporte, equipos o ligas.<br />
            Para <b>Películas</b>, intenta títulos, actores, o géneros.<br />
            Para <b>Series</b>, intenta títulos o géneros.
]]>

when strings.xml for Android is downloaded, then locize downloads the following translation

<?xml version="1.0" encoding="utf-8"?>
<resources>
  <string name="error_no_search_results_subtitle">&lt;![CDATA[            
            Para &lt;b&gt;Deportes&lt;/b&gt;, intente clase de deporte, equipos o ligas.&lt;br /&gt;
            Para &lt;b&gt;Películas&lt;/b&gt;, intenta títulos, actores, o géneros.&lt;br /&gt;
            Para &lt;b&gt;Series&lt;/b&gt;, intenta títulos o géneros.
]]&gt;</string>
</resources>

And this is an invalid string

{ "foo": { "100": "bar" } }

When imported and exported from locize, this becomes:

{ "foo": [ null, null, null, null, .........., "bar" ] }

With 99 null entries. Expected is that the exported file is the same as the original.

It's possible to create translation file that can't be downloaded using locize download. Program will fail download when translation namespace contains "message" or "errorMessage" key with any value.

I think it's caused by this line, but this logic is repeated in other places.

How to reproduce:

• Create project in locize.
• Create namespace with "message" or "errorMessage" key with value "WAT?"
• Use locize-cli to download all translations: locize download --project-id ${LOCIZE_PROJECT_ID}

What happens:

• locize-cli stops downloading translations and returns non-zero exit code
• locize-cli gives user error message (it's the value of the "message" or "errorMessage" key). In this case it's "WAT?"

What should happen:

• locize-cli should download all translations
• locize-cli should return 0 exit code

After updating from version '7.0.3' to '7.6.3'.
I have started getting this error when I try to sync.
npx locize sync -k xxx -i xxx

Package exports for '/project/node_modules/gettext-converter' do not define a valid './cjs/i18next2po' target

More Info:
react: "^16.13.1",
react-native: "0.61.5"
npm : 6.13.1

Hi

Inside locize.io, when I download a nested/flat json, my empty strings are preserved, e.g. { "welcome.title: "" }.

When I download via locize download (or use the locize API directly), the result does not contain those keys anymore.

What gives?

I'm trying to sync using a naming format like the below:

locales
    ├── global
    │   ├── en-GB.translation.json
    │   ├── fr-FR.translation.json

I've tried using the arguments:
--path-mask: {{namespace}}/{{language}}.translation
--namespace: global
--language: en-GB

However I'm not seeing the keys copied into the locize portal, and the CLI is outputting the message FINISHED in green text colour. Have I made an error with the path-mask?

We are trying to set up a script in our package to do a i18next parse, followed by a locize sync. In doing so we want to be able to detect errors as well as outputting the success, but can't seem to get any.

At first we thought it might be a sync problem where the exec wouldn't have time to finish. But making it async didn't solve the problem. stdout and stderror are still null.

This is the locize part of the script:

const util = require('util');
const exec = util.promisify(require('child_process').exec);

async function main() {
  const { stdout, stderr } = await exec( 
    `locize sync` +
    `--api-key [API_KEY]` + 
    `--project-id [PROJECT_ID]` + 
    `--format json`
  );

    if (stderr) {
    console.error(`error: ${stderr}`);
  }
  console.log(`success: ${stdout}`);
}

main();

In our test case we just passed --api-key 1 --project-id 2, expecting stderr to return Reference language not found since that's what it does when running the command straight in the terminal.

Any suggestions?

🐛 Bug Report

When I run locize sync to sync strings against the Locize service, it truncates the trailing newline from my json files. This seems like a minor issue but it does cause an issue with some other tooling we have.

🐛 Bug Report

We've selected the 'flattened json' option as in our project setup - however, the cli downloads json files with nested keys. We have several keys with periods within them, and this is causing issues with our automated build scripts.

To Reproduce

• Goto project settings.
• Under the 'Project Information' panel, select 'json flat' from the publish format menu.
• Download translation files through the locize cli.

Expected behavior

Nested keys in the translated files should be flattened.

"FirstPartOfTheString. SecondPartOfTheString." : "FirstPartOfTheTranslation. SecondPartOfTheTranslation."

should not become

"FirstPartOfTheString" : { 
    "SecondPartOfTheString": "FirstPartOfTheTranslation. SecondPartOfTheTranslation."
}

Your Environment

• runtime version: i.e. node v10.15.0
• Locize CLI: >=7.5.0
• os: Mac, Linux

Since local reference language is master, if we modify some keys value in local repo, we should keep locize data up to date with local repo by using cli. there is no need to modify on locize by hand directly.

When I use the "Export JSON Flat" option directly in the namespace of locize.io, the file is exported in alphabetical order and this very useful.

Is it possible to have the same result with locize-cli?

If it's not supported now, is it something doable in the near future?

Thank you

Let say we have these strings we need to translate to a language without plural form. For Example Korean.

Wat|Got %{count} bonus -> %{count} 보너스를 받았습니다
Wat|Got %{count} bonuses -> %{count} 보너스를 받았습니다

When locize download --format po is called, we've got this for KO language

msgid ""
msgstr ""
"Project-Id-Version: locize\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=1; plural=0\n"
"Language: ko\n"

msgid "Wat|Got %{count} bonus|#|Wat|Got %{count} bonuses"
msgstr "%{count} 보너스를 받았습니다"

Which obviously do not allow translation to be found.

λ locize --version
7.0.3

Is it possible to use _plural for strings like these anyway?

msgid "Wat|Got %{count} bonus"
msgid_plural "Wat|Got %{count} bonuses"
msgstr[0] "%{count} 보너스를 받았습니다"
msgstr[1] "%{count} 보너스를 받았습니다"

Not sure this is the correct but cannot find a better solution.

Hi,
Not sure where to ask this question so apologies if it's not the right place.
I'm using a webpack setup with locize and i18next to load my translation files instead of downloading them directly from locize in production. Currently if we export form locize the format of the exported file is something like this:

key: value
key: value

This is a little bit odd because if we then try to add them as a resource in i18next's init method, the format ought to be:

namespace:
  key: value
  key: value

I checked the API and that request also returns the former format, is there any way to have the namespace including inside the translation file?

me again,

On the locize web interface, it is possible to download a json nested file wich contains all the language and all the name space.

It'd be very cool to be able to do the same with the cli.

First I thought I'd use sync, but it seems it's for a one way sync (local -> locize) and not the other way around.

Then I thought I could do that with the download command, but it seems that if I don't specify a language, it takes the reference language only. So would it be possible to have a --all option to the download command to get the same result as the web interface ? Or have I miss something ?

Thanks

We would like to be able to update the values for the reference language using the save-missing command similar to how the sync command behaves with --update-values true.

We are using the save-missing command to upload new keys to Locize in the workflow explained in #41. For the reference language we are simply setting the value as the key before attempting to upload to Locize.

I'm running locize sync --api-key myApiKeyHere --project-id myProjectIdHere and getting the following error.
My locize project is brand new and has a single version called latest. It has two languages: es(reference) and en.

I'm running it from a directory that has 2 directories inside: es and end. Each directory has a file for each namespace called, for example Login.json.

...
adding 1 keys in es/CreateModal...
adding 14 keys in es/CreateOfferModal...
adding 3 keys in es/CreatedProductsModal...
adding 2 keys in es/CsvDropzone...
Error: Endpoint request timed out
    at Request.request [as _callback] (/usr/local/lib/node_modules/locize-cli/sync.js:374:22)
    at Request.self.callback (/usr/local/lib/node_modules/locize-cli/node_modules/request/request.js:185:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (/usr/local/lib/node_modules/locize-cli/node_modules/request/request.js:1161:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/locize-cli/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)

Using web interface (locize.io) for downloading translations we get proper result:

msgid "tied with %{count} other"
msgid_plural "tied with %{count} other"
msgstr[0] "à égalité avec %{count} other"
msgstr[1] "à égalité avec %{count} others"

Using Locize CLI "download" method, we get strings only in plural form:

msgid "tied with %{count} other"
msgstr "à égalité avec %{count} others"

Switching on any other format from GetText, we get proper result too:

{
  "tied with %{count} other": "à égalité avec %{count} other",
  "tied with %{count} other_plural": "à égalité avec %{count} others"
}

We use Locize CLI, v4.8.0.

A project only requires a single namespace. It would be good to be able to download all language files for a namespace.

Hi

locize download fetched json files from locize in a normalized format (indentation, alphabetized, etc.).

When adding keys for translations into the files manually, it would be nice if I could run something like locize normalize or locize format to reformat the json. The reason is that downloading the locize files and committing them to the repo produce needlessly big diffs (reordering).

Bonus if there would be a --dry-run flag which exits with 0 or 1 depending on whether the files need formatting at all (to include it into our CI).

When I run

node_modules/.bin/locize --project-id MY_PROJECT_ID --api-key MY_API_KEY s

(with the my project Id and API key) locize-cli deletes my .git and a lot of other directories!

The offending line seems to be

locize-cli should not be calling rimraf.sync on random directories in the first place.

Likely, there should also be a check for correct arguments.

As an additional failsafe, locize-cli should check if one of the directories it is about to delete starts with a dot. If so, crash out immediately, because that can't be right, and can potentially destroy very important data.

Our team is using i18next-scanner to extract keys from our codebase which we then upload to Locize using the sync command. However, this seems to overwrite any existing translations for existing keys on Locize. Is this the intended behaviour?

We are not using the --update-values flag, and looking at the README for the sync I don't see any mention that the values will be overwritten, only that keys will be added or removed.

🚀 Feature Proposal

It would be nice to have a way to use the download command with an argument to bypass the TTL caching on the CDN.

Motivation

• Because we use sync we would like to be able to download the latest translations right after sync because by default it updates the keys but not the values, so values might be different between master and locize + developers might have new keys added that we also want to re-download.
• We have offline mode support in our app and for that reason we fetch translations during build time and use them as a fallback. The problem is that because we have caching configured on the project in Locize sometimes our translations are more stale than the time of build.

Example

Something like:
locize download --ver production --bypass-cache=true

🚀 Feature Proposal

A subcommand to sync only keys of reference language. (Not changing values.)

Motivation / Example

Consider the following workflow:
(Reference language is en. Path mask is {{namespace}}.{{language}}. Namespace is translation.)
Individual developers add keys to translation.en.json in their branches and do PRs against master.
Periodically, the master branch shall be synced with locize:

1. Keys added to translation.en.json should be uploaded with their values
2. Keys removed from translation.en.json should be removed from locize's server
3. Existing keys' values should not be uploaded: Keys in translation.en.json whose values differ from their values in locize should NOT have their values changed on either side. (This is because translators can edit the English values in locize, and those changes should not be overwritten.)
4. (Afterwards, locize dl is executed to download the latest values for all languages.)

Currently, step 2. cannot be done either with sync nor save-missing:

• Not with sync because:

all non reference languages will always be just locally replaced by what is published on locize

(But non-reference languages' files should not be changed!)

• Not with save-missing because it doesn't remove deleted keys.

Sync doest not work when using a filename that does not contain namespace.

Sync works only locize => local files but local changes are not shown in locize.

• File structure:
  Resources
  ├SharedResources.de.resx
  ├SharedResources.en.resx
  ├SharedResources.it.resx

• Command
  locize s -i KEY-k ID --format resx --namespace ApiTranslations -m SharedResources.{{language}}

Since there is a provided --path-mask parameter I would expect it to override the filename and work in a two way manner.

The locize-cli tool doesn't honor the xliff tags in Android.

I copied the translation example on the Android developer site (minus the ...) and ran it through the format tool. Unfortunately, it stripped out all of the xliff tags.

$ locize format android-docs.xml --format android --dry true
<resources xmlns:xliff="urn:oasis:names:tc:xliff:document:1.2">
  <!-- Example placeholder for a special unicode symbol -->
  <string name="star_rating">Check out our 5
    <xliff:g id="star">\u2605</xliff:g>
  </string>
  <!-- Example placeholder for a for a URL -->
  <string name="app_homeurl">
    Visit us at <xliff:g
    id="application_homepage">http://my/app/home.html</xliff:g>
  </string>
  <!-- Example placeholder for a name -->
  <string name="prod_name">
    Learn more at <xliff:g id="prod_gamegroup">Game Group</xliff:g>
  </string>
  <!-- Example placeholder for a literal -->
  <string name="promo_message">
    Please use the "<xliff:g id="promotion_code">ABCDEFG</xliff:g>" to get a discount.
  </string>
</resources>

<?xml version="1.0" encoding="utf-8"?>
<resources>
  <string name="app_homeurl.value"> Visit us at http://my/app/home.html </string>
  <string name="prod_name.value"> Learn more at Game Group </string>
  <string name="promo_message.value"> Please use the ABCDEFG to get a discount.\n  </string>
  <string name="star_rating.value">Check out our 5 u2605 </string>
</resources>
reformatting /private/tmp/android-docs.xml...
would have reformatted /private/tmp/android-docs.xml...
FINISHED

Here's the example I used. android-docs.xml

The --skip-delete option that you've added for #24 doesn't seem to work. It says in console "skipping the removal of 1 keys in en/common..." but the key still gets deleted.

I'm running locize sync --api-key myApiKeyHere --project-id myProjectIdHere and getting the following error.
My locize project is brand new and has a single version called latest. It has two languages: es(reference) and en.

I'm running it from a directory that has 2 directories inside: es and end. Each directory has a file for each namespace called, for example Login.json.

/usr/local/lib/node_modules/locize-cli/sync.js:436
    opt.isPrivate = downloads[0].isPrivate;
                                 ^

TypeError: Cannot read property 'isPrivate' of undefined
    at getDownloads (/usr/local/lib/node_modules/locize-cli/sync.js:436:34)
    at Request.request [as _callback] (/usr/local/lib/node_modules/locize-cli/sync.js:213:5)
    at Request.self.callback (/usr/local/lib/node_modules/locize-cli/node_modules/request/request.js:185:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (/usr/local/lib/node_modules/locize-cli/node_modules/request/request.js:1161:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/locize-cli/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)

Is there something I'm missing? Do I need to add an extra parameter to the command?

Hi

I am worried that occasionally, the normal workflow would be broken and the reference language files would be changed in our repo, and not inside of locize. When that happens, I'd like to basically force push the repo files, as it sounds very hard to find the exact difference and manually fix the entries in locize.

Right now for example I got an old en version on locize while integrating the workflow, and in the meantime the repo file has changed. How to reset?

I am basically looking for locize sync, which also overwrites existing key values.

How to do that without deleting and restarting the whole project?

Thanks

I'm trying to use this tool with gettext po files, but this is not working.

The sync tool has a weird behavior. I already lost some translations, and my billing skyrocketed.

To reproduce the problem get a po file and a bunch of untranslated keys. The sync tool will add them all. Without changing anything, the second time the sync tool runs, it will delete all keys and add them all again. Somewhere in between all untranslated keys on locize disappear.

As you can see on the screenshot, the sync tool removes and adds keys without any changes on locize nor the po file.

I am trying to add a command in package JSON to just download my translations files (English and Arabic)

I used this

"download": "locize download"

and I created a new file in the root called .locize and I added my project id, API key, version and language but when running the above command it gives me an error

error: missing required argument `projectId`

🚀 Feature Proposal

Add command to change a key's value for a given language.

Motivation

It's much faster to run a CLI command than to login to locize's UI to change a key's value.
And it allows bulk changes much easier.

When running the sync command with the default setting for the reference-language-only option (true), the code does not read the namespace and namespaces option, so it's not possible to filter to only certain namespaces while also using the reference language only option.

Working on a PR for this now.

We are using the locize cli to upload keys for translation. We are using the PO format.
We have found that locize sync is not deleting keys in locize that are not in our local PO files.

This is the command we are using:

locize sync --api-key "${API_KEY}" --project-id "${PROJECT_ID}" --format po --namespace backend --path locales

We also try using -S false, and it does not work neither.

Hello,

It'd be great to have the cli looking for the config file in the current directory in addition to the home dir.