Comments (6)
I have plans to further optimize the Matcher
class and add optional parameters which will use heuristics to quickly filter down potential candidates (e.g. sequence of two or more characters matching). There is some overhead when loading in libraries and models, which I hope wasn't included in the time for your testing. Another idea I am considering implementing is a batch/multiprocessing option.
from hmni.
Agreed, I think one or more matching consonant letters would also be a safe candidate filter. I will have to run some tests on my international names dataset to confirm. If there are a few edge cases where it doesn't work, they can even be hard-coded into the algorithm.
from hmni.
Batch processing was the first thing that came to my mind. We just pulled your library down yesterday, so I think it's current. Maybe we (@odinolav) could help further things with you?
from hmni.
Thanks, I am very open to collaborating with other developers. The latest release is v0.1.6
which I uploaded last night.
from hmni.
For what it's worth, I'm first trying to speed up the batch test I have going, and I've found (unsurprisingly) that preemptively making sure at least two characters in any order match up can help quite a bit. At first I was thinking matching by two consecutive characters could be safe... but then there are names like Jim => James. 2 non-consecutive seems safe though.
from hmni.
I ended up using a modified version of the Soundex algorithm to filter for disjointed encodings. This candidate filter is a new feature in v0.1.8
as default behavior Matcher(prefilter=True)
.
Note: This alone will not make the cartesian product of two arrays in the millions computable, as the implementation is still O(m x n). I will be labelling this issue as help wanted for anyone who can improve the performance of the library.
from hmni.
Related Issues (19)
- Tensorflow Dependency Error HOT 6
- matcher.similarity gives an error HOT 1
- pandas fuzzy merge with particular distance algorithm HOT 1
- ImportError: cannot import name 'Iterable' from 'collections' HOT 4
- Code from your tutorial HOT 1
- ModuleNotFoundError: No module named 'syllable_tokenizer' HOT 1
- AttributeError: module 'hmni.syllable_tokenizer' has no attribute 'tokenize' HOT 1
- UserWarning: Trying to unpickle estimator MinMaxScaler from version 0.23.1 when using version 1.0.2. HOT 3
- ImportError: cannot import name 'MyVocabularyProcessor' from 'preprocess' (D:\Anaconda\lib\site-packages\preprocess.py) HOT 1
- TypeError: 'Matcher' object is not callable
- licence compatibility HOT 2
- Can we alter this model such that the recall is prioritised instead?
- Cannot import name 'float' from 'numpy' HOT 2
- Could you share the training part? HOT 1
- AttributeError: MinMaxScaler Error HOT 7
- fuzzymerge throws float indexing error HOT 1
- fuzzymerge throws error if match cannot be found
- featurize(df) throwing error: AttributeError: module hmni.syllable_tokenizer has no attribute 'tokenize' HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hmni.