After training Nada's model for 20 epochs (like in her paper) this is what we get

When we take a random German sentence from the dev set, the model does quite well:

$ vipe < random_german.txt |
    while read line; \
    do \
        echo $line \
        echo $line | bash translate_from_stdin.sh 2>/dev/null \
    done

> aber was konnte ich weiter thun die bequemlichkeit des dienstes in
< aber was kommte ich weiter thum die bequenlichkeit des diemstes i

Clearly there are some mistakes like s/n/m/g but overall the performance is remarkably good.

The model is also able to handle just a little chunk of the sentence:

> aber was konnte ich weiter thun
< aber was kommte ich weiter thu

However the model is quite brittle.
If we think up a random sentence that is not taken from the development data but is nevertheless similar to the sentence above, we get pseudo-Swedish output:

> sag mir was kann ich tun
< nar som han fatt ock du

When we use a sentence whose syntax is a bit closer to the dev sentence, the model fares better. At least the output is now pseudo-German.

> aber sag mir was konnte ich weiter tun
< aber mag fur dam wollte uns deuter th

When we modify the sentence to be a bit more "Yiddish-like" (konnte is not valid Yiddish), the output degrades substantially:

> ober zog mir vos konnte ikh vayter tun
< aber was fur daz hattie uhl domier in

What's cool to notice is that the model does pick up on (aber, ober) and (was,vos).

Using PanLex alone is not feasible as the strings are too short.
This will cause the frequency-encoded "ciphertexts" to be awfully short and, as a result, many words get encoded to something like 0 1 2 3 4 5.
While this might've been a good idea with plain decipherment with tons of data, it's a bit hard to do with just a bilingual dictionary.

Understandably the results were abysmal with SER over 100%.
To fix this, I moved from PanLex to Tatoeba.
Those experiments are still running but seem to exhibit better loss dynamics (no divergence).

Notes on corpora

• Corpora used: Tatoeba for yid-deu as well as PanLex bilingual dictionary

Corpus counts

Number of lines/sentences

PanLex

11450 train
2454 dev
2454 test

Tatoeba

train 2387
dev 512
test 512

Number of space-separated tokens

PanLex

11450 train
2454 dev
2454 test

Tatoeba

train 12562
dev 2618
test 2792

Number of unique tokens

PanLex

6673 train
2050 dev
2048 test

Tatoeba

1575 train
893 dev
933 test

Quick example with Ladino:

Cipher length ~300:

Line: poko despues un muevo grupo de imigrantes de rusya konosidos komo bilu beit yaakov lekhu venelkha ke es kaza de yaakov suveremos se enstalaron en petah tikvah i kon el ayudo de edmund rothschild un filantropo fransez siyonisto i muy riko drenaron los bataklikes de la rejion lo ke permityo a los muevos rezidentes a sembrar guertas de agrios i esto trusho un desvelopamyento ekonomiko

Deciphered: katakyliselikenkveljakumesakylkovormonylikylkmeimoktanaioyaiktavakpoiekyloyk

Clearly it doesn't work for these longer ones. What if I take the first 151 characters?

Line: poko despues un muevo grupo de imigrantes de rusya konosidos komo bilu beit yaakov lekhu venelkha ke es kaza de yaakov suveremos se enstalaron en petah

Deciphered: poco despues un mueto grupo de amagranves de rusla conosados como habu feavilacostreno deritano po demano demano demarito chematitro destro destrichono demano de po chamantritechestecontalachono de de

It seems to get the beginning sort of right but not much more.