mozilla / webmaker-download-locales

cli to download all locales from S3 bucket

webmaker-download-locales

cli to download all locales from S3 bucket

npm install webmaker-download-locales

Usage:

Run this in the root directory of the app only.

node webmaker-download-locales "<app_name>"

If you installed the module globally you can do it like this:

webmaker-download-locales "<app_name>"

This will download all the translation files from S3 bucket and write the files to locale/<locales>/*<files>.json

Options

-l (--languages)

If you want to specify a subset of languages, add the -l (or --languages) flag:

webmaker-download-locales -l en_US,fr,bn_BD

webmaker-download-locales's Issues

"make-valet" and "webliteracymap" gives error

If i run one of these it works:
.bin\webmaker-download-locales "make-valet"
.bin\webmaker-download-locales "webliteracymap"

If i run like this i get error:
.bin\webmaker-download-locales "make-valet" -l da_DK
.bin\webmaker-download-locales "webliteracymap" -l da_DK

events.js:72
        throw er; // Unhandled 'error' event
              ^
TypeError: Object #<Object> has no method 'map'
    at C:\[...]\npm\node_modules\webmaker-download-locales\lib\utils.js:61:73
    at C:\[...]\npm\node_modules\webmaker-download-locales\lib\utils.js:13:5
    at Parser.<anonymous> (C:\[...]\npm\node_modules\webmaker-download-locales\node_modules\xml2js\lib\xml2js.js:384:20)

    at Parser.emit (events.js:95:17)
    at Object.onclosetag (C:\[...]\npm\node_modules\webmaker-download-locales\node_modules\xml2js\lib\xml2js.js:348:26)
    at emit (C:\[...]\npm\node_modules\webmaker-download-locales\node_modules\xml2js\node_modules\sax\lib\sax.js:615:33)

    at emitNode (C:\[...]\npm\node_modules\webmaker-download-locales\node_modules\xml2js\node_modules\sax\lib\sax.js:620:3)
    at closeTag (C:\[...]\npm\node_modules\webmaker-download-locales\node_modules\xml2js\node_modules\sax\lib\sax.js:861:5)
    at Object.write (C:\[...]\npm\node_modules\webmaker-download-locales\node_modules\xml2js\node_modules\sax\lib\sax.js:1294:29)
    at Parser.exports.Parser.Parser.parseString (C:\[...]\npm\node_modules\webmaker-download-locales\node_modules\xml2js\lib\xml2js.js:403:31)

Untranslated strings get filled in with the original (english) strings

When i use webmaker-download-locales to download the untranslated strings get filled in with the original strings.

So lets say i download a file that is 95% translated and then upload it, then it appears to be 100% translated because the untranslated strings gets filles in with the english strings.

CLI crashes if invalid app is specified

Jons-MacBook-Pro:webmaker.org jon$ npm run download-locales

> [email protected] download-locales /Users/jon/Sites/webmaker.org
> webmaker-download-locales webamaker


/Users/jon/Sites/webmaker.org/node_modules/webmaker-download-locales/bin/index.js:63
  body.ListBucketResult.Contents.forEach(function(data) {
                                 ^
TypeError: Cannot call method 'forEach' of undefined
    at /Users/jon/Sites/webmaker.org/node_modules/webmaker-download-locales/bin/index.js:63:34
    at IncomingMessage.<anonymous> (/Users/jon/Sites/webmaker.org/node_modules/webmaker-download-locales/bin/index.js:41:7)
    at IncomingMessage.EventEmitter.emit (events.js:117:20)
    at _stream_readable.js:920:16
    at process._tickCallback (node.js:415:13)

npm ERR! [email protected] download-locales: `webmaker-download-locales webamaker`
npm ERR! Exit status 8
npm ERR! 
npm ERR! Failed at the [email protected] download-locales script.
npm ERR! This is most likely a problem with the webmaker.org package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     webmaker-download-locales webamaker
npm ERR! You can get their info via:
npm ERR!     npm owner ls webmaker.org
npm ERR! There is likely additional logging output above.
npm ERR! System Darwin 13.0.0
npm ERR! command "node" "/usr/local/bin/npm" "run" "download-locales"
npm ERR! cwd /Users/jon/Sites/webmaker.org
npm ERR! node -v v0.10.23
npm ERR! npm -v 1.3.17
npm ERR! code ELIFECYCLE
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/jon/Sites/webmaker.org/npm-debug.log
npm ERR! not ok code 0

Download all categories

Hi if i use "webmaker" i only get some files.

Can you make option like "*" or something to download all files.

Allow to have an option for download locale specific

Right now running webmaker-download-locales it will simply download all supported locales for that apps, but we should allow to download a list of specified locales as well.

webmaker-download-locales webmaker th-TH,bn-BD,fr ??

languages are downloaded many, many times

The utils.getAllObjects function, paired with https://github.com/mozilla/webmaker-download-locales/blob/master/lib/utils.js#L29, means that for any X languages, we end up with a list that is size (X+1)*X/2 - for 1 language, size 1. For 10 languages, size 55.

The getAllObjects function clearly has some bugs in it, and should probably not live inside the list_files function, but outside of it, since it doesn't make a lot of sense to have it rebuilt every time. I'll be hunting down how to fix it so it doesn't generate that permutative list but gives a clean listing instead.

On a fast computer (fast internet + SDD, for instance), the repeated URL fetching and writing to the same file over and over can cause race conditions between stream writes, causing amazingly nondeterministic failures =)

Timeouts occur when downloading appmaker locale files

There are 3500+ files that are downloaded as a part of the appmaker locale system. webmaker-download-locales appmaker is run as a part of appmaker's npm install process, but no progress is reported, and errors aren't well managed (and reported).

Added some debugging to my local copy. Once in a while a ENOTFOUND pops up, but mostly ECONNTIMEOUT when downloading files from s3.

Changing the simultaneous download magic number (https://github.com/mozilla/webmaker-download-locales/blob/master/webmaker-download-locales.js#L19) to something less than 16 helps.

Get alot of meta files when downloading appmaker

When i use this program to download the appmaker translation i get a file called "meta-component-[something].json" for each of the compenent files.

Like for "component-addform.json" there is "meta-component-addform.json" and so on.

There is 35 component files which i want to get.
But then there is also 35 files starting with "meta-" that i dont want to get.

Could you change the code so i dont get these meta files ?