Develop a translation workflow about almanac.httparchive.org

[MSakamaki]

I propose https://gitlocalize.com as a tool for translators to use.
this is also use at https://developers.google.com/web/fundamentals/.
this solution should also be able to handle the separate directory translations suggest above.

[ilyaspiridonov]

Feel free to contact me for any help with Gitlocalize, [email protected]

[c-torres]

@MSakamaki I think approach 1 is the best option.

[MSakamaki]

@rviscomi

integrate Git Localize

Yes, I think I can help this.
In order not to add unnecessary files to the translation service, I think it would be nice to have a directory like this:

• src
  • contents
    • en
      • word.yaml or json - small things like words
      • Chapter markdown file or directory
    • ja
      • ...Japanese content and words
  • static
  • templates
    • base.html (base template)

1. yaml/json to bundle small words and metadata (etc. lang=ja) for template.
2. Prepare the main content as a markdown file.
3. Use contents according to the first path (/en or /ja).
    3.1. and redirect to /en when accessed as root (/)

I'm new to Flask and Python. I'm in the midst of exploring how to do this in a cloned repository.
Postscript: Now that I cannot login to git localize, it may take some time.

[MSakamaki]

If so does that mean we'd need to have a build process to reconstruct the content from the translations?

At first I was thinking about a mechanism to translate dynamically from keywords.
I think that html is as common as possible, and a mechanism to convert words is good.
english is the main language, and english words is used for those that lack translation.
I tentatively implemented that example on the branch translation-spike.
You can see the translation works at http://localhost:8080/en and http://localhost:8080/ja .

The point of concern is that the process of extracting words from json can interfere with site construction. And there are performance concerns.
may need to provide a build process as a performance plobrem solution.

I was hoping that we could write HTML templates and markdown files, which Gitlocalize picks up and generates translated versions of the HTML templates and markdown files.

I think this idea is good, deploy in the contents directory both the html need to translate and the markdown.

I examined the characteristics of flask, and realized that all the html templates need to be placed in the tempalte_folder.

So, if you want to include html in the target of translation, src/en/templates/ and src/ja/templates/ in the first proposal seems more straightforward.

[MSakamaki]

Thank you for Flask's easy-to-understand example!
And I am sorry to mislead you.
Use cases that require words are minimal parts such as buttons, menus, and metadata. I think that long sentences (main contents) should have markdown separately.
However, this leaves the possibility that words that are out of context may be placed (words that look like buttons, even though they are labels).

I am concerned that if HTML is included in the target of translation, the layout may be broken.
And, as the English layout is updated, the translation html will also need to be updated.

Yes, words can be inconsistent with context, and including html can break the layout, which we consider to be a trade-off. (In either case, it is necessary to check the final layout)
my can accept either way.

[MSakamaki]

I prefer an approach like that where there are no build or run time costs.

Thanks for the answer! I was able to understand your opinion.

Certainly, there may be layout conflicts and translation conflicts after site construction is complete, but the impact may be minor.
It is important to stabilize the layout early.

I think that the create issues is good when there is a problem with the copy of the translation html.

I agree to src/templates/en/index.html, i can help it.

[MSakamaki]

There is one question, is this translation like a langage(en/ja)? Or do think it is better to be aware of the location(en-US/ja-JP)?
I think that it is necessary to create a directory structure that also corresponds.
I like to recommend a structure that is conscious of location(en-US/ja-JP).

[MSakamaki]

motivated more by being precise/explicit with the intentioned reader of the translation?

Yes, I thought it would be nice to add localization code to the directory so that not only readers but also translators at start-up can be made explicit localization aware.

As I am not familiar with the idea of translation solutions, there may be some wastes or problems with this idea.
I am glad if you can point out without hesitation.

[rviscomi]

Sounds good, let's add the location to the directory.

[rviscomi]

Agreed, let's remove the locale from the directories for now. Filed this PR to do the renaming: #43.

Curious to hear more about Jinja compatibility with gitlocalize.

[rviscomi]

How about we do manual PR-based translations for all of the Jinja-based content, and when we have the chapters written we use markdown for them and gitlocalize to do the translation. Does that work?

cc @mikegeyser this is related to #59.

[MSakamaki]

@rviscomi

How about we do manual PR-based translations for all of the Jinja-based content

Okay, copying html by hand is a bit cumbersome, but I think that there is no serious problem.

we use markdown for them and gitlocalize to do the translation. Does that work?

Yes, as soon as we have a markdown content in the en-US directory, i are ready to integrate it.

I want to confirm one.
Is there a problem with translation branch strategies at gitlocalize-XXXX => master?

[rviscomi]

Naming branches gitlocalize-XXXX SGTM. I'll update the top comment to summarize our discussion.

[rviscomi]

Thanks @MSakamaki! That looks great to me.

Is this something you'd be able to set up for us?

• submit a pull request to add the en directory to src
• integrate Git Localize

You should have repo write access to be able to do the integration.

After we get that set up, we should also reorganize the base template to be language-agnostic:

• create a new base template without any localization or user-facing text, just the basic scaffolding
• rename what's currently called base.html to english_base.html or similar
• english_base.html and each language should extend this new base template and fill in localized things like the page title and body contents

Markup like the analytics script, meta tags, and stylesheets are probably not going to change for each i18n page, so they are good candidates for the base template.

Another open question is web fonts. We currently use Open Sans but I understand it might not be good for Japanese text. So maybe that should be included in the localized templates. Any ideas? Thoughts?

[rviscomi]

Thanks @MSakamaki. The src/contents/en directory structure makes sense to me, let's do it.

word.yaml or json - small things like words

Hmm does this mean we'd need to manually extract every word from the English contents and put it in a yaml or json file? Is the translated version a similar map of words to translations? If so does that mean that we'd need to have a build process to reconstruct the content from the translations? cc @ilyaspiridonov

I was hoping that we could write HTML templates and markdown files, which Gitlocalize picks up and generates translated versions of the HTML templates and markdown files.

[rviscomi]

How reliable is word-for-word translation? For example, idioms and phrases might be best translated in groups of words or entire sentences. Also, things like pluralization rules are different across languages. So it would seem like replacing the entire document with a translated version (as opposed to individual translations for each word) would be more reliable.

I examined the characteristics of flask, and realized that all the html templates need to be placed in the tempalte_folder.

Good point about Flask requiring the templates directory. By default it actually needs to be a top level directory under src, but we can have subdirectories in it. For example, see how we use them for error and report subdirs in the httparchive.org repo. There's a param to change the default template folder, but it works at the entire app-level and I don't see a way to customize it at the request-level.

So maybe something like src/templates/en/index.html is the best approach?

Then in the handlers for each page, we check for a locale and insert it into the template path:

@app.route('/')
def home():
	return render_template('en/index.html')

@app.route('/ja/')
def home_ja():
	return render_template('ja/index.html')

Could look into a way to do this dynamically while validating language codes.

[rviscomi]

Ah I see in https://github.com/HTTPArchive/almanac.httparchive.org/blob/translation-spike/src/templates/index.html how you've implemented what you're describing.

I'm reluctant to go with that approach because even for the English version it requires mapping text to variables, which seems like a cumbersome way to write HTML.

After all of the English content is written how easy is it to copy/paste all of the content to a src/templates/ja/.. directory and translate the content manually in the HTML files? I prefer an approach like that where there are no build or run time costs. Apologies if that's a naive solution with obvious flaws, I'm regretfully too used to building English-only websites.

And, as the English layout is updated, the translation html will also need to be updated.

This is a good point. We don't want to repeat every HTML change N times for N languages. I'm hoping that things like the layout will be stable by the time we're ready to start translation late in the project. If we need to choose between a build time or run time word/phrase substitution approach, I'd much prefer a built time approach so it doesn't impact the UX.

[rviscomi]

Good question! Could you explain more of the benefits of including the region in the language code? Do you anticipate having multiple regions per language or is it motivated more by being precise/explicit with the intended reader of the translation?

We can also name the directories short codes like en and ja but use more precise lang attribute values in the HTML.

[MSakamaki]

Postscript: Now that I can not login to git localize, it may take some time.

This problem has been resolved now, I'm will try to integrate gitlocalize.com.

[MSakamaki]

I'm sorry, I have checked at gitlocalize.com integration, there was one problem.

found that setting of locale code (en-US) seems to be difficult in directory unit.
It seems to supported only areas that may be necessary such as Portugal, China. (confirmed to gitlocalize.com)
There are three approaches, and I would like to hear other translators' opinions. cc: @c-torres @Pavel-Evdokimov

1. need to change the folder structure back to something like /en-US/ to /en/ for gitlocalize.com integration.
2. crowdin.com supports full locales, and you can add custom locales (but there are many functions, so the configuration is complicated and it is necessary to apply) https://crowdin.com/page/open-source-project-setup-request
3. Perform pull request based translation.

For now, I'm thinking about the approach of 1.
If there seems to be a better way in the other translators approach, I would like to adopt an opinion.

[ilyaspiridonov]

Hi there, this is Ilya from GitLocalize.
Actually, we used to have locale codes before. We got rid of those because Google Translate only accepted language codes ("en", "de" etc.), so sometimes our users would have troubles using Google Translate on GitLocalize if one of the target languages was "non-standard" (e.g. en-UK instead of en).
What we could do is this: we could restore locale codes, and map all locales to standard language codes inside GitLocalize (that way Google Translate would receive "en" instead of "en-US" or "en-UK").
Please let me know what you think. I believe it should take us a week or two to get this done.

[MSakamaki]

Thank you for answer!

In my think, glad that there is a locale code, but it was not required.
(locale code was just nice to let the start-up translators be aware of the locale)

If it is possible to integrateion with gitlocalize.com simply by removing the locale code, I think it is better to start with language code only.

@rviscomi
I'm sorry overturned, is there no problem?

@ilyaspiridonov
additional questions:
What kind of approach is ideal when managing jinja templates with gitlocalize.com? (Manualy translation and PR?)
Editing the jinja template file with gitlocalize.com, in UI breaked.

jinja template
http://jinja.pocoo.org/docs/2.10/templates/

Markdown, html, yaml seems to support.
https://docs.gitlocalize.com/file_formats.html

[rviscomi]

@MSakamaki @ilyaspiridonov is it possible to use gitlocalize with Jinja templates?

[MSakamaki]

@rviscomi
In gitlocalize, I think that it is difficult to edit jinja.
(Simple html is also difficult to edit.)
The translation of gitlocalize would be better for pure markdown and yaml editing.

I forked this repository and added jinja templates, simple html, and markdowns.
https://github.com/MSakamaki/almanac.httparchive.org/tree/master/src/templates/en-US

Here you can try out the gitlocalize behavior of it.
https://gitlocalize.com/repo/2364

If the solution is not given by @ilyaspiridonov and you use gitlocalize.com, the translation of jinja templates will be PR based.

Other

If want to translate all content in the same experience, you may need to unify it into a PR-based update or submit a community application to crowdin.

This is a sample created with the crowdin trial account.
It is not as smooth as html, but it seems to work without breaking jinja templates.
https://crowdin.com/project/MSakamaki-almanac-httparchive-org