nicksnyder / go-i18n Goto Github PK

How do you handle the format for dates, numbers, etc.. ?

How about stealing this: https://github.com/jbnicolai/closure-library/blob/20702eff5afe0fd9d8c5a42254879fc18c34253b/closure/goog/i18n/datetimepatternsext.js ?

For example, the angular folks leverage it through their https://docs.angularjs.org/api/ng/filter/date filter.. same with number and currency filters..

Hi!

We started using go-i18n in one of our projects some time ago. Thank you for making our life easier!

Currently we are in the process of translating everything to several languages and started using an online service to coordinate the translations. We can export the translations from there in json format, and using them is as easy as replacing the name of the keys:

term => id
definition => translation

Is there any chance to handle term and definition as synonyms of id and translation? In that case we would be able to use the files directly without converting them at all.

Here is the format definition of the service we use: https://poeditor.com/localization/files/json

Best Regards
Tobias

I would love to include my translated files with go-bindata thus not providing access to a file but a string/byte pointer of the file content.

Is there a way i can load translations from string or an go-bindata Asset?
see: https://github.com/jteeuwen/go-bindata

It looks like the recent change to support a flatter format might have some backwards-compatibility issues with yaml

https://github.com/nicksnyder/go-i18n/blob/master/i18n/bundle/bundle.go#L125

This line assumes that a standard format yaml file will start with a -, which is true unless there's a comment at the start of the file. In that case, it'll assume it's a flat format file and panic/error.

from @bubenkoff #87 (comment)

@nicksnyder extract doesn't seem to work.
ran it with json format and got empty toml file.

goi18n extract -sourceLanguage en-us -format json -outdir locales objects/ service/

got

active.en-US.toml

which is empty

see https://github.com/getlantern/i18n

We coudl incorporate the ability to pick detect the users locale. I need this for Desktop apps :)

Please forgive my question as I'm somewhat new to Golang.

A recent change in go-i18n/i18n/bundle/bundle.go to import gopkg.in/yaml.v2 started causing my build to fail with this error....

src/github.com/nicksnyder/go-i18n/i18n/bundle/bundle.go:7:2: cannot find package "gopkg.in/yaml.v2"

I use "git clone" to pull down all depend projects so I'm not sure if I can only get the package above via "go get" or if there should be a github repo with the changes that I can clone from.

Let's say I have two files as follows:

// en-us.json
[
    {
        "id": "sample_test",
        "translation": {
            "other": "{{.Type}} by {{.Owner}}",
            "zero": "Nobody here but us chickens!"
        }
    }
]

// ru-ru.json
[
    {
        "id": "sample_test",
        "translation": {
            "other": "{{.Type}} от {{.Owner}}",
            "zero": "Здесь никого, кроме нас, цыплят!"
        }
    }
]

After I get a merge files the following lines:

// en-us.all.json
[
    {
        "id": "sample_test",
        "translation": {
            "one": "",
            "other": "{{.Type}} by {{.Owner}}"
        }
    }
]

// ru-ru.all.json
[
    {
        "id": "sample_test",
        "translation": {
            "few": "",
            "many": "",
            "one": "",
            "other": "{{.Type}} от {{.Owner}}"
        }
    }
]

So... WTF?
I want to use the translation only in two cases:

• If the variable is equal to 0 - show "empty" translation (text only)
• If the variable is greater than 0 - show the value of a universal translation

I do not want to duplicate the line, for example, for a value of 1, this is an extra information trash and additional fixes.

Hello

I am looking for a way to build translation into a web-based project. I came across this and it really looks impressive. However, I have no idea where to start; I am puzzled how the whole framework is to be integrated in a project. It would be great if there was a minimum sample project somewhere, showing how this is done ?

Is the support for locale based number and currency handling in your timeline?

Create tag for 1.0 and start using versioned import urls everywhere.
http://gopkg.in/v1/docs

This replaces the rejected #81

While @nicksnyder is right about thread safety being the client's responsibility, the global pluralSpecs makes it harder than it should.

And then having RegisterPluralSpec mutate that state.

In Hugo we run most tests with t.Parallel(). And while this may report some data races we may not realistically see in real use scenarios, it is well worth it.

In this case, it reports a data race in pluralSpecs when using the RegisterPluralSpec. We can probably fix that by adding some global locks.

But these globals also has another unfortunate side-effect:

We cannot run multiple go-i18n configurations side-by-side. And this has nothing to do with concurrency.

So, it would be much better if you could provide a constructor func that takes the options needed (plural specs etc.)

Hi, I'm want to override the language used in templates in the handler but I can't find out how to do that... I load the default en-US and in this case I want to switch to nl-NL. Can you let me know how I should do that?

Could you make a more thorough example for V2?

I'm not sure why NewBundle accepts language.English when later Accept-Language is used as an input. What is a bundle, for that matter?

RegisterUnmarshalFunc is set to toml. Where is there any TOML used?

I want to use Sourcegraph code search and code review with go-i18n. A project maintainer needs to enable it to set up a webhook so the code is up-to-date there.

Could you please enable go-i18n on @sourcegraph by going to https://sourcegraph.com/github.com/nicksnyder/go-i18n and clicking on Settings? (It should only take 15 seconds.)

Thank you!

I've spent a few hours trying to figure this out, but it looks like it is not possible for an argument (/insert) on a call to i18n.T to contain a percent character.

anonymized code:

func main() {
	input := "%"
	fmt.Printf(i18n.T("<my translation ID>",
		map[string]interface{}{
			"Insert": input,
		}))
}

Output (with preceding message text removed):

%! (MISSING)

I've tried substituting % with %25 , url escaping , passing in as byte slice rather than string.

But it looks like a percent anywhere within a value (of the args template data) gets garbled.

Comments are useful for translators so they can understand additional context about the translation.

{
  "id": "helloName",
  "translation": "Hello {{name}}",
  "comment": "This is a welcome message for a user. {{name}} is the user's first name."
}

This is more of a question than a bug report. Given that I am starting a new project which I will want to serve in English and German let's say I start with this file

~/i18n ᐳ cat en-US.all.json 
[
  {
    "id": "point_count",
    "translation": {
      "one": "Only {{ .Count }} Point...",
      "other": "{{ .Count }} Points!"
    }
  }
]

When I generate the untranslated file I get this

~/i18n ᐳ goi18n *all.json
~/i18n ᐳ cat de-de.untranslated.json 
[
  {
    "id": "point_count",
    "translation": {
      "one": "{{ .Count }} Points!",
      "other": "{{ .Count }} Points!"
    }
  }
]

If I send that to be translated they won't know how English handles the one case to generate an appropriate German translation. This seems to be expected by the test files under goi18n/testdata/expected/ but it was unexpected to me. Is this intended?

Thanks!

As discussed in this thread:

gohugoio/hugo#2577

We use go-i18n in Hugo and it works great, but the number one related question we get on the support forum comes from people having a hard time getting their language files in YAML right (indentation issues, mostly).

So it would be nice if go-i18n could support the simpler TOML format and also optionally support a slightly flatter data format, so we could write:

[[d_days]]
one = "{{.Count}} day"
other = "{{.Count}} days"

[[string_id_2]]
other = "Some other translation"

/cc @moorereason @BoGeM

I was wondering if we can support ARB format, so i can use google translate toolkit.

Here is a good summary:
https://github.com/dart-lang/intl#extracting-and-using-translated-messages

Background info:
At the moment i use Dart for frontend and golang for backend.

Blocked on response from these tickets

http://unicode.org/cldr/trac/ticket/9006
http://unicode.org/cldr/trac/ticket/9022

Hi Nick, Found out today my vendored version is not compatible with the new version. Specifically this function of mine does not work anymore because the locale package is gone.

My Call to AddTranslation:
i18n.AddTranslation(locale.MustNew(userLocale), tran)

What is the new mechanism for this. I spent a handful of minutes looking but found nothing obvious.

Thanks

// LoadJSON takes a json document of translations and manually
// loads them into the system
func LoadJSON(userLocale string, translationDocument string) error {
tracelog.Startedf("localize", "LoadJSON", "UserLocale[%s] Length[%d]", userLocale, len(translationDocument))

tranDocuments := []map[string]interface{}{}
err := json.Unmarshal([]byte(translationDocument), &tranDocuments)
if err != nil {
    tracelog.CompletedErrorf(err, "localize", "LoadJSON", "**************>")
    return err
}

for _, tranDocument := range tranDocuments {
    tran, err := translation.NewTranslation(tranDocument)
    if err != nil {
        tracelog.CompletedError(err, "localize", "LoadJSON")
        return err
    }

    i18n.AddTranslation(locale.MustNew(userLocale), tran)
}

tracelog.Completed("localize", "LoadJSON")
return nil

}

A flatter data file structure would be beneficial. Implementation should maintain support of existing format.

JSON

{
  "d_days": {
    "one": "{{.Count}} day",
    "other": "{{.Count}} days"
  },
  "string_id_2": {
    "other": "Some other translation"
  }
}

YAML

d_days:
  one: "{{.Count}} day"
  other: "{{.Count}} days"

string_id_2:
  other: Some other translation

TOML (assuming #61)

[d_days]
one = "{{.Count}} day"
other = "{{.Count}} days"

[string_id_2]
other = "Some other translation"

Hey, @nicksnyder! Ran into an issue today you might find interesting. Here's my situation.

I have three files, all for English: en.all.json, en-US.all.json, and en-GB.all.json. The first holds all English translations that are common between all variations of English we support. The second and third files hold translations specific to their respective locales.

A user requests something from us, and sends us the locale en-US. We make a TranslationFunc with preferences en-US, en. Now here's where we run into a problem: if I ask for a translation that is not in en-US.all.json, it returns the translationID instead of looking inside en.all.json for a relevant translation.

This is due to the way bundle.TfuncAndLanguage handles language preferences. When I ask for en-US and the language has any translations, it limits the search to this map, ignoring the remaining language preferences.

I'd like to support the preference fallback at the translation level, instead of the language level. In pseudocode, this is:

translation_id = "my_string"
preferences = %[en-US en]

# Upon request for a translation, iterate through each & return
# if any of the preferences contain the translation.
preferences.each do |pref|
  if translations[pref] && translations[pref][translation_id]
    # A translation was found.
    return translations[pref][translation_id]
  end
end
# If no langs have matching translations, return the translation ID.
return translation_id

What do you think?

Hey @nicksnyder!

Thanks for writing this great series of packages. We're looking into using them and we're running into a fairly large issue: when we call a Tfunc, it is automatically cast to a map[string]interface{} instead of being left alone. As a result, my struct doesn't resolve properties.

Thoughts?

Don't need to distinguish these since they have the same pluralization. Also seems inconsistent with pt-BR.

CLDR provides XML data for all locales so I would like to write a tool that parses the file and generates code and tests for all locales.

I think it could be very useful to be able to add custom functions to the internally used template engine.

The Template method that allows it is https://golang.org/pkg/text/template/#Template.Funcs .

I'm not sure where it would the best place would be to implement this. Maybe adding another Field Funcs of the type template.funcMap to the Message struct would be a good idea?

Given I've added a string to the translations for the default language and run goi18n *.all.json to update the other languages with the new id without a translation, my program is now in a state where it will return ids for all non-default languages rather than something intelligible, like the value present in the default language.

The README explains that you're meant to send out the *.untranslated.json files for translations after running goi18n *.all.json, but am I supposed to wait until I get those translations back before I can ship my code?

It would be far more preferable if I could use a default language for when a translation does not exist.

It seems to me that this is not presently possible with go-i18n. I have to merge the untranslated translation files (which contain the default language), which means then that the untranslated files get emptied. Now I cannot easily add another string as my untranslated files will then only contain the new untranslated string.

I would like to be able to maintain untranslated files with all of the untranslated strings such that whenever I get a chance to send them out I have the complete list, and have a default language used when a specific translation ID does not exist.

I have started to prototype what v2 of this library might look like.

The overall goals of v2 are to

1. Address all issues that are currently open on the project
2. Apply some Go best practices to the codebase (e.g. remove global state, explicit error returns)
3. Revisit API design and make breaking changes which are necessary to accomplish (1) and (2).

My development plan is to work on this in a branch until I am happy with the results. Then, I will merge this into master under a subfolder called v2-beta and tag a 1.x.0 release. Projects can then begin to try out the new API and provide feedback. APIs under v2-beta subfolder will be subject to change pending community feedback. Once I am happy with the results, I will make a final 1.x.0 release that contains the v2 api under the v2 subfolder. Immediately following that, I will delete the v1 package files, move v2-beta to the root of the repository, and tag 2.0.0.

I do not have an estimate for when this work will be complete as I am working on it in spare time here and there.

If you have general feedback about this process it can be shared here.
If there are other things you would want to see in a world where breaking API changes are possible, please create a new issue and I will consider it.

Problem

Adding to the bundle in a concurrent program causes panics

Resolution

Make bundle safe for concurrent writes

Stack trace

fatal error: concurrent map read and map write

goroutine 49987 [running]:
runtime.throw(0x36a3440, 0x21)
    /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc821f58bc0 sp=0xc821f58ba8
runtime.mapaccess1_faststr(0x28527e0, 0xc8203d3ef0, 0x3458300, 0xd, 0xc8203c5338)
    /usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc821f58c20 sp=0xc821f58bc0
github.com/-/-/vendor/github.com/nicksnyder/go-i18n/i18n/bundle.(*Bundle).translate(0xc82040cc90, 0xc8203f4d60, 0x3458300, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/-/-/vendor/github.com/nicksnyder/go-i18n/i18n/bundle/bundle.go:237 +0x15c fp=0xc821f58ce8 sp=0xc821f58c20

Hi,

In one particular situation i want to translate some materials using a cascade of IDs.

Specifically for form error handling.

Reason for that is to be able to have a general translation, and if needed a set of more and more specific translations ID alternative for that specific (input + error) of that specific (form+input+error).

Right now i m doing it by trying to translate each possibility until it returns non empty value which is considered as the translation to consume.

I d prefer to use a HasFunc(ID) to remove that test on empty response.

what do you think ?

I don't know what to do anymore.

Hi,

First of all, thank you for this nice tool !

My use case :

• use cobra and its templates ;
• use go-i18n to have translation of the templates.

I am facing what follows with templates :

1. in the toml files : the template contains client application functions : goi18n application fails to convert it the flat file into json because those client applications functions are not known at that moment ;
   • I have to write by my self the json files ;
2. in my application, I just want to get the template as pass it to the client application for processing : again, it fails cause the Execute() function is called when getting the template and there is no arguments so far to make the execution correctly.
   • I don't known how to get this working in the end ...

Digging into the code, I must admit that the Execute() function on template is always called even if there is no arguments (https://github.com/nicksnyder/go-i18n/blob/master/i18n/translation/template.go#L37 called by https://github.com/nicksnyder/go-i18n/blob/master/i18n/bundle/bundle.go#L387).

Do you have any clue on how to manage this use case : let client application handle template execution ?

Hi Nick,

Adding strings to json by hand is cumbersome, goi18n could parse the source looking for Tfunc invocations and treat the arguments of such calls as candidates for translation.

I have carved a proof of concept here: https://github.com/rtfb/go-i18n/commits/sift-WIP

You can try it out with $ goi18n -sift src/ l10n/*, which will extract untranslated stuff to *.untranslated.json.

Would you be interested in such functionality? I would then shape it up and submit a PR.

CLDR zero key support

[
  {
    "id": "{{.Count}} users online",
    "translation": {
      "zero": "Никого нет",
      "other": "{{.Count}} пользователей онлайн"
    }
  }
]

Random translations

The idea is to be able to return a random phrase, if the term translation is an array

[
  {
    "id": "Hello",
    "translation": ["Привет", "Здарова", "Привет, братюня", "Как сам?"]
  }
]

...or so...

[
  {
    "id": "{{.Count}} apples",
    "translation": {
      "one": "{{.Person}} яблоко",
      "other": ["{{.Count}} яблок", "{{.Count}} яблочек"]
    }
  }
]

Hi!

This might be similar to #30 but that issue is quite old so I'm rather starting a new one here. I'm not sure it's completely related.

My setup

• de.all.json
• en.all.json

I am using simply de and en because there is just one translation for each language, no flavours and I want them to be used for any kind of locales of those languages that come along, be it en-GB, en-US, de-CH, de-DE, etc. Default language is en in any case, and I really want to get the English text if the German version should not be found for some reason (which best should never happen).

The issue

With Accept-Language header de-DE, the de translations are not found.

This is becauselang.MatchingTags() is only applied when adding translations (so in my case de, en, and of course the matching tags are only de and en).
When the translation function is requested for de-DE, only lang.Tag is used to lookup the fallback language and this is then only de-de (in bundle.translatedLanguage()).

So, my question is, would it not make more sense to use lang.MatchingTags() also in bundle.translatedLanguage() where the fallback is looked up?

Hi,

Thank you for creating this great tool!

I'm using the go-i18n with the PhraseApp, which is a platform that provides the API to download the translated file. I want to reload the translated file on the fly when the translators do some changes without re-deploying our microservices.

I noticed that the AddTranslation method can import the translations and it will apply the lock to the bundle, but I also noticed that you mentioned that "Your Go program should load translations during its initialization". So I wonder that if it's appropriate to reload the translations after the service initialization using AddTranslation.

The second question is about the goroutine safety of Tfunc. Is it safe that I get a Tfunc during the initialization and use it as a global sharing function? From my understanding, it should be ok. I just wanna confirm it here.

Thank you again.

In my templates I'm trying to do something like:
{{T "simple_user_greeting" .UserName}}

With translation:

[
  {
    "id": "simple_user_greeting",
    "translation": "Hello {{.}}"
  }
]

It outputs:
Hello map[Count:user1]

I'm expecting:
Hello user1

where user1 is .UserName.

Aparently, the first variable is parsed as 'Count' (unless not int nor string) and then mapped as such in a map. Is there a way to override this? Is it planned? Because I don't want to specify the variable name in the translation file (I'd like to do it in the template).

On this line is it necessary to have the 'string' type too? If it isn't there, I think this would work normally (although I'd understand if you don't want to change that).

Thank you for the great project!
One thing really missing is message extraction utility.
Any plans for implementing that?
Thanks again!

Is it possible to write translation strings for html/template templates which contain HTML fragments?

For example:

  {
    "id": "userUnauthorized",
    "translation": "You are not authorized to view user <span class='entityName'>{{ .userName }}</span>. "
  },

Calling {{T "userUnauthorized" .}} results in escaped HTML. I would prefer to use HTML in my translations because it reduces the total number of keys required, and I want to give my translators more flexibility with sentence construction.

If I were using html/template directly, I could use the template.HTML function to declare that the input is HTML and not a string. Is there a clean way to use something like this technique for translations?

I'd love to be able to just write

program_greeting:  "Hello world"

instead of

program_greeting:
  other: "Hello world"

While I understand the use case of the latter, couldn't it just support a direct string value and default to the "other" mechanic?

I have a translation like this

[customer]
  one = "Kunde"
  other = "Kunden"

And call

{{T "customer"}}

The restult is "customer" where I would expect one of the defined translations. I would suggest to use "other" for undefined amounts.

When using {{T "customer" 2}} everything works fine.

{
  "id": "UPDATE_CART",
  "translation": "Updated cart{{if .Items}} with {{end if .Items}}{{.Items}}{{if .Items}} items{{end if .Items}}"
}

So if {{.Item}} is present then the output would be

Updated cart with 12 items

else

Updated cart 

If user wants to have custom language code that aren't in Unicode CLDR (like at.yaml, dk.yaml, ch.yaml, gb.yaml, etc.), bundle.ParseTranslationFileBytes returns no language found in "dk.yaml".
I don't think that users should be limited only with CLDR languages. They can freely use language codes they want and it can be technically implemented.
Discussion: https://discuss.gohugo.io/t/i18n-country-codes-as-part-of-the-url/6881

I'm a newb to this project, and I think I'm following the instructions, but perhaps I'm missing something?

This code generates the error in the subject line:

bundle := i18n.NewBundle(language.English) bundle.RegisterUnmarshalFunc("toml", toml.Unmarshal) bundle.MustLoadMessageFile(path.Dir(os.Args[0]) + "/i18n/active.fr.toml")

The toml file originates from a goi18n extract.

In addition, the MustLocalize example on this page generates the same error:

https://godoc.org/github.com/nicksnyder/go-i18n/v2/i18n#pkg-examples

Thank you in advance.

I have an English file with strings foo and bar. I run the tool which generates a de-DE.all.json with empty translations for those strings. I removed the foo string from the English file and re-ran the tool intending to regenerate the German files and it panics.

~/project ᐳ cat en-US.all.json
[
  {
    "id": "bar",
    "translation": "Bar"
  }
]
~/project ᐳ cat de-DE.all.json
[
  {
    "id": "bar",
    "translation": ""
  },
  {
    "id": "foo",
    "translation": ""
  }
]
~/project ᐳ goi18n *all.json
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x50 pc=0x100afa]

goroutine 1 [running]:
github.com/nicksnyder/go-i18n/i18n/translation.(*singleTranslation).Backfill(0x82043a3a0, 0x0, 0x0, 0x0, 0x0)
        /Users/jwalker/go/src/github.com/nicksnyder/go-i18n/i18n/translation/single_translation.go:37 +0xba
main.(*mergeCommand).execute.func2(0x88205bc318, 0x82043a3a0, 0x0, 0x0)
        /Users/jwalker/go/src/github.com/nicksnyder/go-i18n/goi18n/merge.go:71 +0x11a
main.filter(0x820444de0, 0x820481b00, 0x0, 0x0, 0x0)
        /Users/jwalker/go/src/github.com/nicksnyder/go-i18n/goi18n/merge.go:100 +0x12e
main.(*mergeCommand).execute(0x820481ef0, 0x0, 0x0)
        /Users/jwalker/go/src/github.com/nicksnyder/go-i18n/goi18n/merge.go:69 +0xe11
main.main()
        /Users/jwalker/go/src/github.com/nicksnyder/go-i18n/goi18n/goi18n.go:78 +0x1ee

It is not used within the package and will be ambiguous when zh-Hant and zh-Hans are merged into just zh for #1

Does not support single subtag IETF language codes or anything else but 2-subtag codes, while the IETF language code does define other-than-2-subtag syntax.

Since browsers often send short IETF language tags, this doesn't really work with go-i18n. For example, my browser sends: Accept-Language:en-US,en;q=0.8,de;q=0.6,nl;q=0.4. Only the en-US is useful for go-i18n, but en/de/nl are not.

It would be great if you request en-US with i18n.Tfunc and only en exists, it uses en. This can be extended when any request for a more specific language tag (more extensions) is boiled down to the closest supported language tag

Working with Android/Java the IDE generates a dynamic class R.java with all strings ids, this method avoid a lot of bugs. In a big project is easy to reference an invalid id or to have a lot of unused ids.

What do you think the command goi18n generates R.go ?
Example:

en-US.all.json

[
  {
    "id": "settings_title",
    "translation": "Settings"
  }
]

$ goi18n -genconst en-US.all.json -outdir ./

this command generates R.go

package R
const SettingsTitle string = "settings_title"

In code the new way will be:

T(R.SettingsTitle)

This is only a suggestion, let me know what do you think.
I can help with this change
thanks