Fields often contain "?" about jmdict-simplified

[scriptin]

I can confirm it is a Zorba bug: 28msec/zorba/issues/228

[scriptin]

Fixed by adding a function which checks if a string is duplicated, and if it is, returns only a half.

[aehlke]

Ah after running a build myself, it's fine! For some reason the archive I've downloaded from the release has the question marks.

[scriptin]

Yes, all tags are broken in the release. Thank you for reporting that.

Question mark is a default value when converting tags. If none of cases match, it inserts a question mark. This whole tagging thing in original files is so yucky because it uses custom XML entities (e.g. <!ENTITY oK "word containing out-dated kanji">), which are substituted with text, and because of that I had to implement the backwards conversion.

Since all tags are broken, and original XML file hasn't changed, I think it's something related to how Zorba handles XML entities. I'll have to figure out what it is.

Can you tell which version of Zorba you are using?

[aehlke]

Zorba NoSQL Query Processor, Version: 3.1.0

[scriptin]

All entities appear to be parsed "doubled" inside sense → pos elements, e.g. &n; is interpreted as noun (common) (futsuumeishi)noun (common) (futsuumeishi) instead of just noun (common) (futsuumeishi). The only exception is the very first entry of the dictionary. (I discovered that by adding trace messages into the code.) At first it made me think that I somehow concatenate variables, but I can't find any part of the code which could do that, unless I terribly misunderstand how variables in XQuery work.

I also added a script for generating tag-related code to make sure I haven't messed that up. It doesn't resolve this problem either.

My version of Zorba is the same.

Can you possibly pull the master as it is right now and re-run the build as ./build.sh dev dev and tell me if you get the same error as below? I'm starting to think if it's a bug in Zorba.

$ ./build.sh dev dev
Processing a full EN version
</home/d/jmdict-simplified/src/tags.xq>:180,18: error [unknown-tag]: Unknown tag 'noun (common) (futsuumeishi)noun (common) (futsuumeishi)' on entity 1000010
Processing EN version with common words only
</home/d/jmdict-simplified/src/tags.xq>:180,18: error [unknown-tag]: Unknown tag 'noun (common) (futsuumeishi)noun (common) (futsuumeishi)' on entity 1000110
Done

[aehlke]

May I recommend documenting a docker-based build process to avoid system environment differences?

[scriptin]

Well, Docker image uses the same version we both use, so there is no difference. As for documenting, I'd throw away Zorba altogether rather than documenting it. After using it a bit I absolutely hate it: its error messages are really bad, and it's not in active development anymore.

If you know good alternatives, please tell me.