Comments (5)
Yes, this is a somewhat annoying thing about hacking on duckling.
from duckling.
I've been thinking about how we could approach this issue. My idea is to add a rule to each language that splits single words so we can reuse the rules that already exist.
Assume that we have Language X
and it has rules for hello
, world
and intersection
, so it correctly parses hello
(with hello rule
), world
(with world
rule), and hello world
, world hello
, hello hello
, world world
(with intersection
rule).
What I would like to do is add a single-word decomposition
rule that will handle helloworld
, hellohello
, worldhello
and worldworld
. The idea is that this rule will only split the word into subwords which then will be picked up by the intersection
rule.
For example, rule single-word decomposition
converts helloworld
into hello world
. Then, rule intersection
picks hello world
and works as expected.
The single-word decompositon
rule would need to know in advance all possible subwords so then it can separate them. A rough implementation would be something like:
ruleSingleWordDecomposition :: Rule
ruleSingleWordDecomposition = Rule
{ name = "single-word decomposition"
, pattern =
[ regex "(\\S+)" -- we match 'single' words
]
, prod = \tokens -> case tokens of
(Token RegexMatch (GroupMatch (w:_)):_) -> do
let rgx = "(hello|world)" :: String -- we need to compute in advance a big regex of possible "subwords"
let ws = Text.pack <$> getAllTextMatches (show w =~ rgx)
Just $ Token RegexMatch $ GroupMatch ws -- How do we return the "subwords" so other rules can pick them up?
_ -> Nothing
}
Note: we need to split the "single word" in Haskell (let ws = Text.pack <$> getAllTextMatches (show w =~ rgx)
) since we can't do it with pure regexes AFAIK
In this case, our single-word decomposition
rule knows that "single words" can be composed only from the "subwords" hello
and world
.
The main issue that I found with this is how to use the output of this rule as input for the remaining rules (they don't pick it up). So, even if we're able to build the required regex and split the "single word" into "subwords" I can't seem to redirect them to other rules.
I'm trying to avoid adding special cases or hacking into the internals of Duckling (Engine.hs
and the like), so any hint would be appreciated.
from duckling.
One quick note, the mentioned examples in the first comment aren't permalinks so I can't be sure to what they refer to. I understand that this issue is related to:
from duckling.
@chessai Do you think that this could work?
from duckling.
Regarding @emlautarom1's interesting suggestion, I see two problems that should be considered before going this route.
First, decomposing German number words like "dreiundzwanzig" 'three.and.twenty' (23) can lead to unnwanted ambiguities because "drei und zwanzig" 'three and twenty' is also a grammatical phrase in German. Consider, for instance, the sentence "sie wählten zwischen dreiundzwanzig Kandidaten" 'they made a selection between twenty three candidates'. The problem is that the sentence "sie wählten zwischen drei und zwanzig Kandidaten" 'they made a selection between three candidates and twenty candidates'. Hence, the suggested decomposition makes the two sentences indistinguishable.
Similarly, if single-word decomposition
is applied the compound "zweihundertjährige" 'two.hundred.year.old.PL' and the phrase "zwei hundertjährige" 'two hundred.year.old.PL' are converted to the same string. Thus, the difference between sentences like "sie fällten zweihundertjährige Eichen" 'they chopped oak trees that were two hundred years old' and "sie fällten zwei hundertjährige Eichen" 'they chopped two oak trees that were one hundred years old' is erased.
So the upshot is that we lose information by applying single-word decomposition
.
from duckling.
Related Issues (20)
- Support Persian(fa) Languages HOT 4
- [Persian][Numeral] Extract incorrect number from text
- ghc: panic! when stack build.
- Error building on MacOS with M1 Pro HOT 4
- Can't build Docker image HOT 2
- [it_IT] add time corpus/rules
- Default log files are created even if different paths are provided via options
- [ERROR] Network.Socket.sendBuf: resource vanished HOT 1
- Time intervals are too large by one grain HOT 1
- [Question] Which elements are relevant for Test-Failures? HOT 3
- Will not run on new Macbook HOT 1
- Reject common phrases like "guten morgen"
- Up Text and Hashable constraints
- Ne dimension is not working. HOT 1
- Custom Dimension doesn't return the actual value!
- Intervals with leap year dates return incorrect years
- how can I detect the date and time separately? anyone knows?
- Inconsistent date range parsing HOT 1
- Incorrect abbreviation of 'millions' in locale DE HOT 1
- overalaping body in the extracted entities
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from duckling.