Comments (10)
不好意思,过了这么久才看到
from node-segment.
建议在分词前使用 split(/\s+/)
来分割
from node-segment.
这个不准备修复了么?
from node-segment.
1、导致自动把一 一
合并的原因是:分词过程中是没有记录空格的,因此一 一
的分词结果是['一', '一']
,由于默认启用了一个优化功能,把相邻两个数字合并了;
2、如果要修复的话,目前最简单的方法就是分词前使用 split(/\s+/)
来分割一次;
3、如果直接在 segment 模块中修复的话,我需要再考虑一下这个改动是否会对一些现有的程序产生影响。
from node-segment.
已修复。请使用 v0.0.5 版本
from node-segment.
from node-segment.
OK,已打上标签“v0.0.5”
在 Node.js 上使用,直接用 npm install [email protected]
即可安装指定版本
from node-segment.
还是有问题,建议 Reopen。
0.0.5 版把空白字符当前分词要素,但是最终结果中还是忽略了空白字符本身:
segment.doSegment("a a")
// 输出结果:
[ { w: 'a', p: 16 }, { w: 'a', p: 16 } ]
// 正确结果应该是:
[ { w: 'a', p: 16 }, { w: ' ', p: 16 }, { w: 'a', p: 16 } ]
from node-segment.
这不是Bug,而是设计的时候分词结果自动去掉了“无用”的空白字符。
不知道是否有必要保留这些空格
from node-segment.
程序处理的时候,空白文本是内容的一部分,不应该被忽略掉。
pinyin 处理时,分词模块忽略空白字符,会导致输出不一致:
han = "a a";
py = pinyin(han);
// 如果分词模块忽略掉空白字符:
py === "aa";
// 正确的应该是。
py === "a a";
from node-segment.
Related Issues (20)
- 对于英文分词,单复数等是否需要处理 HOT 3
- 如何获取TF? HOT 1
- 如何忽略一些无意义的词 HOT 2
- 能提供javascript版本的分词吗? HOT 3
- 这句话解析不了 HOT 7
- 关于同义词,我写的好像没问题啊 HOT 1
- es6中出现数组遍历的错误 HOT 2
- 某些文本,node进程莫名退出。 HOT 1
- 是否有错别字纠正功能呢?感觉还挺有必要的 HOT 1
- 字典中能否用通配符,比如日期,不想写无数个某年某月某日 HOT 4
- 如何找到联想词汇
- Cannot find module "URLTokenizer" HOT 3
- Action required: Greenkeeper could not be activated 🚨
- An in-range update of should is breaking the build 🚨 HOT 4
- An in-range update of mocha is breaking the build 🚨 HOT 24
- 特定的省市分不出来 HOT 1
- Version 10 of node.js has been released
- 同义词替换无法保留原始内容结构,空格、换行丢失 HOT 1
- Out of memory HOT 4
- 请勿用此模块来对较长且无任何标点符号的文本进行分词,否则会导致分词时间成倍增加 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-segment.