fgribreau / node-language-detect

🇫🇷 NodeJS language detection library using n-gram

Node Language Detect

LanguageDetect is a port of the PEAR::Text_LanguageDetect for node.js.

LanguageDetect can identify 52 human languages from text samples and return confidence scores for each.

Installation

This package can be installed via npm as follows

npm install languagedetect --save

Example

const LanguageDetect = require('languagedetect');
const lngDetector = new LanguageDetect();

// OR
// const lngDetector = new (require('languagedetect'));

console.log(lngDetector.detect('This is a test.'));

/*
  [ [ 'english', 0.5969230769230769 ],
  [ 'hungarian', 0.407948717948718 ],
  [ 'latin', 0.39205128205128204 ],
  [ 'french', 0.367948717948718 ],
  [ 'portuguese', 0.3669230769230769 ],
  [ 'estonian', 0.3507692307692307 ],
  [ 'latvian', 0.2615384615384615 ],
  [ 'spanish', 0.2597435897435898 ],
  [ 'slovak', 0.25051282051282053 ],
  [ 'dutch', 0.2482051282051282 ],
  [ 'lithuanian', 0.2466666666666667 ],
  ... ]
*/

// Only get the first 2 results
console.log(lngDetector.detect('This is a test.', 2));

/*
  [ [ 'english', 0.5969230769230769 ], [ 'hungarian', 0.407948717948718 ] ]
*/

API

• detect(sample, limit) Detects the closeness of a sample of text to the known languages
• getLanguages() Returns the list of detectable languages
• getLanguageCount() Returns the number of languages that the lib can detect
• setLanguageType(format) Sets the language format to be used. Suported values:
  • iso2, resulting in two letter language format
  • iso3, resulting in three letter language format
  • Any other value results in the full language name

Benchmark

• node.js 1000 items processed in 1.277 secs (482 with a score > 0.2)
• PHP 1000 items processed in 4.835 secs (535 with a score > 0.2)

Credits

Nicholas Pisarro for his work on PEAR::Text_LanguageDetect

License

Copyright (c) 2013, Francois-Guillaume Ribreau [email protected], Ruslan Zavackiy [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

node-language-detect's Issues

Work right without php.js

Adding more n-gram, extending the list of available languages

Hi there!

This has been really useful to me, for quality control. Is accurate and very important: fast.

I would like to ask for an update of the detected languages. Maybe adding some instructions? or scripts that parse text and extend the database?

Many thanks

Improve benchmark tests

What do you think about changing english languages names on ISO 639‑1 or ISO 639‑3 language codes.

Improve language detection accuracy

Detects "Тест" as bulgarian

Тест is a Russian word.

Detects "Hello" as "'hawaiian"

const LanguageDetect = require('languagedetect');
const lngDetector = new LanguageDetect();
console.log(lngDetector.detect('Hello'));

Output:
[ [ 'hawaiian', 0.3786666666666666 ], [ 'italian', 0.2813333333333333 ], [ 'icelandic', 0.2553333333333333 ], [ 'danish', 0.2326666666666667 ], [ 'welsh', 0.22866666666666668 ], [ 'dutch', 0.19666666666666666 ], [ 'english', 0.18600000000000005 ], [ 'finnish', 0.17933333333333334 ], [ 'norwegian', 0.16666666666666663 ], [ 'slovak', 0.15399999999999991 ], [ 'slovene', 0.12466666666666659 ], [ 'pidgin', 0.12466666666666659 ], [ 'hungarian', 0.10733333333333328 ], [ 'spanish', 0.09600000000000009 ], [ 'czech', 0.09533333333333338 ], [ 'azeri', 0.08666666666666667 ], [ 'swedish', 0.08466666666666656 ], [ 'tagalog', 0.08333333333333337 ], [ 'german', 0.07733333333333325 ], [ 'french', 0.07600000000000007 ], [ 'croatian', 0.05733333333333335 ], [ 'portuguese', 0.012666666666666715 ] ]

Unexpected translation results

I'm attempting to use your library in an IRC bot that has a twitter feed, to help reduce the non-English tweets that make it through the filtering. The filtering was mostly successful, except for a couple like this. (The first result from your language detector is printed first, then the tweet itself. I have left out the user names attached to the tweets.)

 [ 'pidgin', 0.22029761904761902 ]
 node-db-oracle - Oracle database bindings for Node.js http://bit.ly/l7T8mS

 [ 'danish', 0.14746268656716421 ]
 Hiring Systems Engineer blog.nodejitsu.com scaling node.js ... http://t.co/IBUYTHk

I can completely understand where an algorithm would have difficulty with the strange text that most Tweets tend to consist of - but do you have any thoughts here? Even if I had to make a list of three or four languages that can count as 'probably English', I'd be okay with it, but I haven't really used this kind of library before - so I'd like your input.

Thanks again for putting this together - it's a big improvement for me already!

NPM and releases.

Hi!
In this repository exists tag named 1.1.1, and the lastest relase on NPM is 1.1.0.
Can this be normalized/synchronized ?

Detect Cyrillic / Japanese / Thai / Gurmukhi

Cyrilic Hiragana Thai Gurmukhi:

var text = tweet.getStatus().text;

// 0400 - 04FF: Cyrillic
// 3040 - 309F: Hiragana (Japanese)
// 0E00 - 0E7F: Thai
// 0A00 - 0A7F: Gurmukhi (Indian)
return !(/[\u0400-\u04FF\u3040-\u309F\u0E00-\u0E7F\u0A00-\u0A7F]+/.test(text));

Version 1.1 code review & release

GH-7, GH-8, GH-9, GH-10, GH-11, GH-12, GH-13

any chance for a deno version?

Test async implementation

Need to test, if using async in loops will improve performance.

Types error: Parameter 'languageType' implicitly has an 'any' type

The most recent release broke TypeScript support:

node_modules/languagedetect/index.d.ts(55,25): error TS7006: Parameter 'languageType' implicitly has an 'any' type.

Seems like the types file has a small error here. I'll send a PR to fix it.

Add more unicode languages for tests

TypeError: path must be a string

I use the library with Node 4.1. I get the following error:
fs.js:878
binding.stat(pathModule._makeLong(path), req);
^

TypeError: path must be a string
at TypeError (native)
at Object.fs.stat (fs.js:878:11)
at new module.exports (/home/nmp/WorkSpace/NaSp/NodeJS/AppHunt/node_modules/language-detect/language-detect.js:30:6)
at Object. (/home/nmp/WorkSpace/NaSp/NodeJS/AppHunt/build/handlers/apps_handler.js:86:19)
at Module._compile (module.js:413:34)
at Object.Module._extensions..js (module.js:422:10)
at Module.load (module.js:357:32)
at Function.Module._load (module.js:314:12)
at Module.require (module.js:367:17)
at require (internal/module.js:16:19)
at Object. (/home/nmp/WorkSpace/NaSp/NodeJS/AppHunt/test/helper/dbhelper.js:3:19)
at Module._compile (module.js:413:34)
at Object.Module._extensions..js (module.js:422:10)
at Module.load (module.js:357:32)
at Function.Module._load (module.js:314:12)
at Module.require (module.js:367:17)
at require (internal/module.js:16:19)
at Object. (/home/nmp/WorkSpace/NaSp/NodeJS/AppHunt/test/app_test.js:5:16)
at Module._compile (module.js:413:34)
at Object.Module._extensions..js (module.js:422:10)
at Module.load (module.js:357:32)
at Function.Module._load (module.js:314:12)
at Module.require (module.js:367:17)
at require (internal/module.js:16:19)
at /home/nmp/WorkSpace/NaSp/NodeJS/AppHunt/node_modules/mocha/lib/mocha.js:216:27
at Array.forEach (native)
at Mocha.loadFiles (/home/nmp/WorkSpace/NaSp/NodeJS/AppHunt/node_modules/mocha/lib/mocha.js:213:14)
at Mocha.run (/home/nmp/WorkSpace/NaSp/NodeJS/AppHunt/node_modules/mocha/lib/mocha.js:453:10)
at Object. (/home/nmp/WorkSpace/NaSp/NodeJS/AppHunt/node_modules/mocha/bin/_mocha:393:18)
at Module._compile (module.js:413:34)
at Object.Module._extensions..js (module.js:422:10)
at Module.load (module.js:357:32)
at Function.Module._load (module.js:314:12)
at Function.Module.runMain (module.js:447:10)
at startup (node.js:141:18)
at node.js:933:3

Any ideas why?

Please add support for asian languages.

Is it possible to add asian languages such as chinese,japanese and korean?

Unable to checkout

When I clone the repo, git says:

warning: remote HEAD refers to nonexistent ref, unable to checkout.

And I can not see anything in the cloned repository.

Unify JS code style

Language name ISO639 support in return array

Problem in detecting Farsi Language

Hi,

I tried to use this project in order to detect Farsi ( Persian ) Language. It seems it does not work.

I think the problem is in ord function ( phpjs.js:224 ) or in _next_char ( Parser.js:363 ). It does not recognize farsi characters. here is my code:

var ld = require('node-language-detect');

var obj = new ld();

console.log(obj.detect('سلام. این یک متن فارسی است که باید توسط این ماژول شناسایی شود.'));

// it returns []

Remove .coffee files & clean tests files

Performance improvement

http://code.google.com/p/language-detection/source/browse/trunk/src/com/cybozu/labs/langdetect/Detector.java

http://code.google.com/p/language-detection/source/browse/trunk/#trunk%2Fprofiles

English text is detecting as Danish

I am using the npm module languagedetect (version: ^2.0.0).
For simple English string, it is returning the language as danish with the highest score and English with less score.
const text = 'Test from backend'; const languages = lngDetector.detect(text, 2);

The response which I am getting in languages is:
[["danish",0.2868627450980392],["english",0.2733333333333333]]

Node Language Detect

console.log(lngDetector.detect('Language Detect'));

/**[ [ 'danish', 0.3608888888888889 ],
  [ 'norwegian', 0.29933333333333334 ],
  [ 'german', 0.2757777777777778 ],
  [ 'dutch', 0.21777777777777785 ],
  [ 'swedish', 0.20977777777777784 ],
  [ 'spanish', 0.20377777777777772 ],
  [ 'italian', 0.19844444444444442 ],
  [ 'french', 0.19600000000000006 ],
  [ 'indonesian', 0.1948888888888889 ],
  [ 'romanian', 0.19377777777777772 ],
  [ 'tagalog', 0.18800000000000006 ],
  [ 'cebuano', 0.18000000000000005 ],
  [ 'slovene', 0.17977777777777781 ],
  [ 'portuguese', 0.1691111111111111 ],
  [ 'turkish', 0.15955555555555556 ],
  [ 'azeri', 0.15022222222222226 ],
  [ 'pidgin', 0.13088888888888883 ],
  [ 'english', 0.12911111111111118 ],
  [ 'latin', 0.12644444444444447 ],
  [ 'estonian', 0.10177777777777786 ],
  [ 'swahili', 0.08711111111111114 ],
  [ 'hawaiian', 0.08066666666666666 ],
  [ 'lithuanian', 0.0802222222222222 ],
  [ 'albanian', 0.07733333333333325 ],
  [ 'latvian', 0.07422222222222219 ],
  [ 'hausa', 0.0662222222222223 ],
  [ 'hungarian', 0.06399999999999995 ],
  [ 'somali', 0.06022222222222218 ],
  [ 'czech', 0.04133333333333322 ],
  [ 'finnish', 0.023777777777777787 ],
  [ 'icelandic', 0.013555555555555543 ],
  [ 'welsh', 0.00955555555555554 ] ]*/

Option to return language as iso2 or iso3

Would it be possible to add an optional third parameter to LanguageDetect.detect() that would allow add a property to return the languages as iso2 or iso3. If it is already possible, could this be added to the docs?

Example:

LanguageDetect.detect(text, 1, { langType: 'iso2' } )

Buble sorting?

Hey guys, I think it's sweet module - I quickly glanced at code, I can see https://github.com/FGRibreau/node-language-detect/blob/master/lib/Parser.js#L221 you are implementing your own buble sorting algorithm.

Buble sorting are famous as not the best algorithm, my question is why it's been used and why Array.sort is not?

language detection failing on basic English input

hey folks -

testing out this library with some sample posts from our application, and I'm getting some strange results:

console.log
  post text: take me back home

console.log
  language detection results: [ [ 'pidgin', 0.3275 ], [ 'hawaiian', 0.2816666666666666 ] ]

console.log
post text: i like your hair

console.log
  language detection results: [ [ 'hawaiian', 0.26625 ], [ 'norwegian', 0.25145833333333334 ] ]

I installed via npm install languagedetect --save

Add support for Japanese

There is no support for Japanese, however, it's a popular enough language that I think it should be supported.

Current behavior is to guess the language is English due to Japanese characters not being recognized since it's a unique character set.

Sample: "シャーロック・ホームズ (Sherlock Holmes) は、19世紀後半に活躍したイギリスの小説家・アーサー・コナン・ドイルの創作した[1]、シャーロック・ホームズシリーズの主人公である、架空の探偵"

Result:

[
  [ 'english', 0.030795454545454626 ],
  [ 'somali', 0.026553030303030245 ],
  [ 'estonian', 0.021590909090909105 ],
  [ 'hungarian', 0.021098484848484755 ],
  [ 'danish', 0.019962121212121264 ],
  [ 'albanian', 0.019053030303030183 ],
  [ 'hawaiian', 0.015946969696969737 ],
  [ 'french', 0.015643939393939377 ],
  [ 'latin', 0.015606060606060623 ],
  [ 'german', 0.015454545454545388 ],
  [ 'hausa', 0.01435606060606065 ],
  [ 'swedish', 0.012575757575757462 ],
  [ 'welsh', 0.011325757575757489 ],
  [ 'portuguese', 0.010909090909090868 ],
  [ 'czech', 0.010833333333333361 ],
  [ 'spanish', 0.010492424242424137 ],
  [ 'latvian', 0.01041666666666663 ],
  [ 'swahili', 0.010227272727272751 ],
  [ 'norwegian', 0.009356060606060645 ],
  [ 'pidgin', 0.00920454545454541 ],
  [ 'vietnamese', 0.007348484848484826 ],
  [ 'dutch', 0.006212121212121224 ],
  [ 'icelandic', 0.005113636363636487 ],
  [ 'indonesian', 0.003901515151515156 ],
  [ 'lithuanian', 0.0012499999999999734 ]
]

Not accepting short input - e.g. "Hi" returns empty result.

Not accepting short input - e.g. "Hi" returns empty result.

Also lacks of support for ja, ko

Source of language datasets

Where is the source text dataset for the Ngrams of those 52 languages? Would like to see if it is different from wooorm/franc#78 usage of UDHR, and if it is more accurate than them.

Chinese language detection?

Is it possible to add support for simplified Chinese?

Logic error?

I found a small logic error on LanguageDetect.js#L176

if I'm reading this equality conversion chart correctly you're checking if the argument 'sample' is equal to 0 (zero), the empty array [], the empty string '', null, and false, but the first expression being true makes the second expression false.