GithubHelp home page GithubHelp logo

Comments (13)

notani avatar notani commented on August 18, 2024 1

How about this?
https://github.com/knowitall/morpha

from bpe_analysis.

justhalf avatar justhalf commented on August 18, 2024

Should we keep the plus sign?
In my Indonesian morphology normalizer script I didn't. Also, are you working on this?

from bpe_analysis.

notani avatar notani commented on August 18, 2024

Should we keep the plus sign?

No, + is just for an illustration purpose.

I will do English and Japanese normalization. Were you also working on this?

from bpe_analysis.

justhalf avatar justhalf commented on August 18, 2024

Yes, the scripts for Indonesian is at https://github.com/justhalf/bpe_analysis/blob/master/morphind/process_txt.py
It uses MorphInd for the morphology analyzer.

No, + is just for an illustration purpose.

I asked because in the Indonesian one I explicitly remove +. I think we should remove the plus sign, yeah.

from bpe_analysis.

notani avatar notani commented on August 18, 2024

Did you start English and Japanese normalization, too?

from bpe_analysis.

justhalf avatar justhalf commented on August 18, 2024

Did you start English and Japanese normalization, too?

No, I haven't started. I didn't know which morphology analyzer to use. But if we have them, we can simply replace the subprocess call with the corresponding call.

from bpe_analysis.

justhalf avatar justhalf commented on August 18, 2024

In this paper it says the lexicon (19MB) is large:
image

Does anyone know a good English morphology analyzer? I was surprised to find none, only Morfessor, which was automatic.

from bpe_analysis.

justhalf avatar justhalf commented on August 18, 2024

Maybe this one? http://wiki.apertium.org/wiki/Lttoolbox

from bpe_analysis.

notani avatar notani commented on August 18, 2024

Maybe this one? http://wiki.apertium.org/wiki/Lttoolbox

Does this output surface forms of morphemes?

This statistical morphological segmenter can generate normalized surface forms like un+test+able+ly:
https://github.com/ryancotterell/treeseg

We can search for similar studies by "morphological segmentation" rather than "morphological analysis"

from bpe_analysis.

justhalf avatar justhalf commented on August 18, 2024

Based on my cursory look, it seems so.

This statistical morphological segmenter can generate normalized surface forms like un+test+able+ly:
https://github.com/ryancotterell/treeseg

That's a good one, since it is modern. I was looking at more that has more manual analysis, since it will be less automatic, e.g., FST. But couldn't find FST for English.

from bpe_analysis.

justhalf avatar justhalf commented on August 18, 2024

That looks good. You have one for Japanese as well? (I guess we don't need this for Chinese?)

from bpe_analysis.

notani avatar notani commented on August 18, 2024

Fortunately, Japanese segmentation by UDPipe is already morpheme segmentation and has normalized forms. I think we don't need normalization for Chinese.

Can you do English normalization?

from bpe_analysis.

justhalf avatar justhalf commented on August 18, 2024

I am trying. Morpha apparently only handles plural nouns and verb inflections, but not derivations. So happiness stays as is.

from bpe_analysis.

Related Issues (7)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.