shataniya / cncut

Chinese word segmentation tool based on javascript

cncut

Chinese word segmentation tool based on javascript【基于javascript的中文分词工具】

// npm
npm i cncut

// yarn 
yarn add cncut

const cncut = require("cncut")
const cn = cncut()
var text = "我是这个世界上最帅的男人"
console.log(cn.cut(text).join("/"))

// result：
我/是/这个/世界/上/最/帅/的/男人

cncut's Issues

hanging up while there are something like punctuation 有标点符号的时候会卡住

好像这个项目不需要python也能单独使用，点赞……

但是，有标点符号的时候会卡住，例如

const cncut = require("cncut")
const cn = cncut()
var text = "寂寞，寂寞"
console.log(cn.cut(text).join("/"))

cncut crashes

When I run the following

var title = '性命古訓辨證'
const dict = require('cncut/dict.json')
const cncut = require('cncut')(dict)

console.log(cncut.cut(title))

I get

<--- Last few GCs --->

[27868:0x104654000]    34084 ms: Scavenge 3522.5 (3584.7) -> 3517.4 (3589.7) MB, 25.0 / 0.0 ms  (average mu = 0.524, current mu = 0.531) allocation failure 
[27868:0x104654000]    34127 ms: Scavenge 3527.5 (3589.7) -> 3522.5 (3594.9) MB, 26.0 / 0.0 ms  (average mu = 0.524, current mu = 0.531) allocation failure 
[27868:0x104654000]    34170 ms: Scavenge 3532.6 (3594.9) -> 3527.5 (3599.9) MB, 26.0 / 0.0 ms  (average mu = 0.524, current mu = 0.531) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
 1: 0x1000a86ec node::Abort() [/usr/local/bin/node]
 2: 0x1000a8862 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1001deb0d v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x1001deab7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10032361f v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 6: 0x10030711a v8::internal::Factory::NewClosureFeedbackCellArray(int) [/usr/local/bin/node]
 7: 0x100422411 v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::ConvertElementsWithCapacity(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::FixedArrayBase>, v8::internal::ElementsKind, unsigned int, unsigned int, unsigned int) [/usr/local/bin/node]
 8: 0x100421057 v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::GrowCapacity(v8::internal::Handle<v8::internal::JSObject>, unsigned int) [/usr/local/bin/node]
 9: 0x1005684f3 v8::internal::Runtime_GrowArrayElements(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
10: 0x10082ef39 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
11: 0x1a9b9a40591f 
Abort trap: 6

real	0m34.211s
user	0m41.107s
sys	0m2.570s