oggy22 / translator Goto Github PK

ASSERT(x) in Debug throws exception, but it is invisible to the user.

Find a way to have a dialog pop-up when the ASSERT is breached in Debug.

Add more parsing tests including those that would exercise more than one rule, for example:
"ja idem u skolu" would exercise:

1. ->
2. u -> PrilOdredba

Add parsing tests for English.

Currently class pattern works only with jokers.

For pronouns in Serbian e.g:
{ L"ја", L"мој", зам, прид },
{ L"ти", L"твој", зам, прид },
There are no jokers.

Implement Assert as:

• in Debug, it will throw
• in Release, it will __assume

Create a rule:
verb * -> adjective *able

Currently, many tests only assert without a good failure message. This is due to the fact that many classes don't have string/wstring converters. Write these converters and use them in the tests.

In Debug, make sure that every single word rule was used. It can be done as:

• as soon as the rule is used set a flag to true
• check all the rules that they have the flag set (either in a test or in init)

Make cpp file for each language
Put everything in static fields, so no language instance is required.

Make sure you always use Assert instead of throwing an exception

The following warning is issued sporadically. Try to resolve it.

Severity Code Description Project File Line Suppression State
Warning MSB8028 The intermediate directory (Debug) contains files shared from another project (TranslatorCppTest.vcxproj, UnitTest1.vcxproj). This can lead to incorrect clean and rebuild behavior. TestTranslatorCpp C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppBuild.targets 392

The class attribute_manager should have the following methods:

// True if it has all the attributes that the argument has
bool has_all(const attribute_manager&);

// True if there is no conflict, i.e. no category has mismatching arguments
bool accepts(const attribute_manager&);

This should always hold:
am.has_all(am2) =>am.accepts(am2)
am.accepts(am2) <=> am2.accepts(am)

Implement joker logic.
Currently only patterns with "%" are allowed.
There should be:
"?" any letter - for any language
"@" any vowel - language defined
"#" any consonant - language defined

Introduce a base class for languages:

template <Lang>
class Language

so that

class Serbian : public Language<Serbian>
class English : public Language<English>

Make command line application work with various arguments e.g:

• translator.exe -random:SR
  This will generate a random proper sentence in Serbian language.
• translator.exe -listwords:EN
  Lists all the words in English language
• translator.exe -translate:EN-SR text.txt
  Translate the given file from English to Serbian.
• translator.exe -help
  Prints all the commands

Make Serbian adjectives have all 3x2x7x3=126 word forms

There are:

• 3 genders
• 2 plurals (singular and plural)
• 7 cases
• 3 comparatives (positive, comparative and superlative)

Test failing: each_adjective_has_3genders_2plurals_7cases

Write a prototype of WPF C# which writes to std input and reads from std output of C++ Console App and displays it in a textbox.

The C++ Console App can simply echo the message or do some simple modifications to it.

The C++ Console App needs to be run only once and therefore there must be a message-end-character. E.g. Null-termination, once received on WPF, the text is displayed.

Might experience threading issues in WPF as only UI Thread can change the textbox.

Introduce determiniers for English:
http://www.ef.com/english-resources/english-grammar/determiners/

Write the following tests:

• check_some_noun_forms:
  konj - > konja (akuz)
  covek -> covece
  orah -> orasi, orahe
  mis -> miseva
• check_some_verb_forms:
  ici->idem, idu, isao

etc.

Basically, whenever there is a special rule or rule exception, write a test that checks that the rule is applied in an appropriate case.

Make smart use of precompiled header so the compilation is faster.

Make TranslatorWPF actually work with TranslatorCPP.
TranslatorCPP should have command line arguments e.g. "parse:Serbian" or "parse:English" or "translate:English-Serbian"

On parse only, the app should output something like "Parse ok"/"Parse failed"

Currently used joker which matches everything is '%'. This was probably SQL influence, because when I started writing Translator in C#, I used SQL DB to store languages and thought using a similar syntax would be beneficial.
Actually, it's never been so, and now I think it should be '*' as in most scripts syntax (e.g. DOS command prompt)

Currently attributes and categories are defined as enum classes each:
enum class attributes { singular, plural, masculine, feminine, ... }
enum class categories { number, gender, .... }
and there is mapping defined as:
std::map<attributes, categories> belongs_to = ...

I am thinking of:
enum class gender { masculine, feminine, neutrum }
enum class number { singular, plural }

This should be investigated and possibly the code be switched to such.

In language.h, parse method has switch statement for 1, 2 and 3 length rules. This should be re-written using variadic templates.

*ti -> *
*eti -> *i //sedi, prdi
*ati -> aj // cekaj
*iti -> ij // pij

Both unordered_set and set, and unordered_map and map are used.
Make them more consistent.

Consider having different flavours (i.e. non-unordered/unordered) for Debug/Release, as allegedly unordered versions are faster.

Put some effort in generation of English words:

• nouns: singular and plural
• verbs: present, past participle, past

Write a prototype app that prints wide strings to standard output.

In addition to parsing and translation the system has potential of generating random proper sentences/texts by generating random abstract syntax trees as follows:

• Start with a syntax node, probably sentence
• Randomly choose rules and apply them where possible
• Once no rule can be applied anymore, choose random words and place them on leafs in proper word forms
• Walk the tree (DFS) to yield a sentence/text

Note: This is completely a separate task, independent of others, just for fun.

Release/Debug x64 configuration fails to build.

There needs to be a nice visualization of parsing_triangle e.g:

Ja idem u skolu
++ - Zamenica
+----+ - glagol
+----+ - imenica
+------+ - recenica
+-----+ - priloska odredba
+----------------+ - recenica

There should be a command line option with an input file. Input file will contain a text in Serbian (or potentially any other supported language). The system will try to parse the text and provide the following stats:

• number of recognized/unrecognized words (potentally list them in output files)
• the longest (successfully) parsed phrase counting characters
• the longest (successfully) parsed phrase counting words

By adding new words these stats should gradually increase.

Extra: Create a service which would regularly download a text from http://www.politika.rs, parse it and store results.

Sometimes it happens that all tests are passing but the program crashes from the command line.
Therefore, create tests which simulate common command line scenarios and which will fail iff command line fails. This may include:

• Listing all the words and word forms for a given language
• Parsing a sentence in the given language
• Translating from/to languages

I had problems moving belongs_to_category into the base Language class. Namely, attributes and attribute_categories are inaccessible from the base class.

Try solving this somehow, and having belongs_to_category in the base class.

Make system support these:
Човек - човече - људи (plural basis "ljud")
коњ - коњи
миш - мишеви (plural basis "mishev")
муж - мужеви (plural basis "mishev")
слон - слонови (plural basis "slonov"
сто - стола - столови (singular basis "sto")
мушкарац - мушкарца - мушкарци
nouns in akuzative animate vs inanimate

Rules (for words) should be checked in reverse order so the most general rule comes first in the list and therefore it's checked last. For example plural noun rules in English should be written as:
% -> %s
%y -> %ies
%o -> %oes

Check:
http://www.myenglishpages.com/site_php_files/grammar-lesson-plurals.php

1. • Traverse all the dictionary words for duplicates.
2. • When creating each derived dictionary word check that it doesn't exist already.

This should be only in Debug.

All x64 both Release and Debug are failing with the following message:
A 64-bit test cannot run in a 32-bit process. Specify platform as x64 to force test run in x64 mode on x64 machine.

The file language.h became too big containing many classes. Break it down into multiple files, possibly each containing one class/struct

Figure out how to link TranslatorCpp from the cpp test project. The problem is linking error of static field defined in cpp.

Create natvis file for classes and types in language.h
Natvis file instructs debugger how to present types in debugger.
For reference check cpprest.natvis

Write a rule in Serbian to create diminutive.

Create the following tests for Serbian language:

• each noun has gender
• each noun has 7x2 word forms 7 cases x 2 numbers
• each verb has 3x2 forms in present

Sometimes, VS Git doesn't mark files which have been modified. Even when they are explicitly added as:
git add file.cpp
they seem to be unmarked eventually. Marked files are seen as with a "red check" in VS Solution Explorer.

Find ouy why and fix it.

Write tests which will count the number of words and compare it against explicitly given numbers:

• Dictionary words
• Derived dictionary words
• Word forms

Also, write a test to check that each dictionary word produces at least one word form

Consider using smart pointers:

• uniqure_ptr
• shared_ptr
• weak_ptr

For example, the rule in Serbian:
IS(broj, padez) -> imen(broj, padez)
doesn't seem to work.

The problem might be the way new nodes are added into the parsing matrix for rule of length 1. Left and right of the rule are hitting the same spot in the matrix.

For example pronouns "ja", "ti", "on" need to have derived dictionary words as "moj", "tvoj", "njegov". Every derived dictionary word may/will have word forms e.g:
nom: moj/moja/moje
gen: mog(a)/moje/mog(a)
dat: mom(e)/mojoj/mom(e)
akuz: moj/moje/moje

Write some basics English-Serbian translator class:
template <class Lang1, class Lang2>
class word_translation
{
dictionary_word* p_word1;
dictionary_word* p_word2;
};

template <class Lang1, class Lang2>
class translator
{
static vector<pair<Lang1::wordtypes, Lang2::wordtypes>> wordtypes;
static vector<pair<Lang1::attributes, Lang2::attribtues>> attrs;
static vector<pair<Lang1::categories, Lang2::categories>> cats;
static vector<word_translation> words;
static vector<rule_translation> rules;
};

There are TestTranslator and Translator which are C# projects. Remove them.