tldr-pages / tldr-translation-pairs-gen Goto Github PK

In commands, tokens can be translated.

We can index and pull out the tokens from each command string, and include those mappings in the output as well.

Example

- Add a new task to the work list:

`todo new {{thing_to_do}} --list {{work}}`

- Aggiungere un nuovo task alla lista delle cose da fare per lavoro:

`todo new {{cose_da_fare}} --list {{lavoro}}`

• thing_to_do → cose_da_fare
• work → lavoro

Some commands have square brackets in the command description, which is a hint to what argument it's referring too. However, this information isn't helpful when pairing translations. This may be worth removing while building the corpus.

Example

pages/common/am.md

`am start -n {{com.android.settings/.Settings}}`

- Start an activity and pass [d]ata to it:

pages.de/common/am.md

- Starte eine Aktivität und übergib ihr Daten:

`am start -a {{android.intent.action.VIEW}} -d {{tel:123}}`

The resulting dataset then has:

<tu>
  <tuv xml:lang="de">
    <seg>Starte eine Aktivität und übergib ihr Daten </seg>
  </tuv>
  <tuv xml:lang="en">
    <seg>Start an activity and pass [d]ata to it </seg>
  </tuv>
</tu>

In the project we have certain lines that are repeated frequently, namely the template text like for alias pages.

We should do something to avoid writing out duplicates. Cases where lines are similar but not the same like "More Info…" is perfectly fine since the link is changing, but cases like the alias pages are just 100% the exact same.

The primary motivation for this project is to produce a dataset for OPUS, yet this is not one of the supported outputs.

Rather than throwing a CSV, XML, or JSON file for them to process further, it'd be nice if we could just understand the format and support it out of the box in the first place.

Here is the format we should export to:
https://opus.nlpl.eu/trac/wiki/DataFormats.html

The current interface is clearly rushed, let's clean this up a bit.

The --format argument currently only accepts 1 format at a time. In the scenario that we want to produce multiple datasets, this means redundantly processing the files multiple times.

To reduce wasted computation/energy, we should support either multiple formats or an all that processes the files once, but exports in all formats specified.

We need to publish builds of this somewhere, likely npm so that it's more convenient to use.