tdebatty / java-string-similarity

Implementation of various string similarity and distance algorithms: Levenshtein, Jaro-winkler, n-Gram, Q-Gram, Jaccard index, Longest Common Subsequence edit distance, cosine similarity ...

java-string-similarity

A library implementing different string similarity and distance measures. A dozen of algorithms (including Levenshtein edit distance and sibblings, Jaro-Winkler, Longest Common Subsequence, cosine similarity etc.) are currently implemented. Check the summary table below for the complete list...

• Download
• Overview
• Normalized, metric, similarity and distance
• Shingles (n-gram) based similarity and distance
• Levenshtein
• Normalized Levenshtein
• Weighted Levenshtein
• Damerau-Levenshtein
• Optimal String Alignment
• Jaro-Winkler
• Longest Common Subsequence
• Metric Longest Common Subsequence
• N-Gram
• Shingle (n-gram) based algorithms
  • Q-Gram
  • Cosine similarity
  • Jaccard index
  • Sorensen-Dice coefficient
• Ratcliff-Obershelp
• Experimental
  • SIFT4
• Users

Download

Using maven:

<dependency>
    <groupId>info.debatty</groupId>
    <artifactId>java-string-similarity</artifactId>
    <version>RELEASE</version>
</dependency>

Or check the releases.

This library requires Java 8 or more recent.

Overview

The main characteristics of each implemented algorithm are presented below. The "cost" column gives an estimation of the computational cost to compute the similarity between two strings of length m and n respectively.

		Normalized?	Metric?	Type	Cost	Typical usage
Levenshtein	distance	No	Yes		O(m*n) 1
Normalized Levenshtein	distance similarity	Yes	No		O(m*n) 1
Weighted Levenshtein	distance	No	No		O(m*n) 1	OCR
Damerau-Levenshtein 3	distance	No	Yes		O(m*n) 1
Optimal String Alignment 3	distance	No	No		O(m*n) 1
Jaro-Winkler	similarity distance	Yes	No		O(m*n)	typo correction
Longest Common Subsequence	distance	No	No		O(m*n) 1,2	diff utility, GIT reconciliation
Metric Longest Common Subsequence	distance	Yes	Yes		O(m*n) 1,2
N-Gram	distance	Yes	No		O(m*n)
Q-Gram	distance	No	No	Profile	O(m+n)
Cosine similarity	similarity distance	Yes	No	Profile	O(m+n)
Jaccard index	similarity distance	Yes	Yes	Set	O(m+n)
Sorensen-Dice coefficient	similarity distance	Yes	No	Set	O(m+n)
Ratcliff-Obershelp	similarity distance	Yes	No		?

[1] In this library, Levenshtein edit distance, LCS distance and their sibblings are computed using the dynamic programming method, which has a cost O(m.n). For Levenshtein distance, the algorithm is sometimes called Wagner-Fischer algorithm ("The string-to-string correction problem", 1974). The original algorithm uses a matrix of size m x n to store the Levenshtein distance between string prefixes.

If the alphabet is finite, it is possible to use the method of four russians (Arlazarov et al. "On economic construction of the transitive closure of a directed graph", 1970) to speedup computation. This was published by Masek in 1980 ("A Faster Algorithm Computing String Edit Distances"). This method splits the matrix in blocks of size t x t. Each possible block is precomputed to produce a lookup table. This lookup table can then be used to compute the string similarity (or distance) in O(nm/t). Usually, t is choosen as log(m) if m > n. The resulting computation cost is thus O(mn/log(m)). This method has not been implemented (yet).

[2] In "Length of Maximal Common Subsequences", K.S. Larsen proposed an algorithm that computes the length of LCS in time O(log(m).log(n)). But the algorithm has a memory requirement O(m.n²) and was thus not implemented here.

[3] There are two variants of Damerau-Levenshtein string distance: Damerau-Levenshtein with adjacent transpositions (also sometimes called unrestricted Damerau–Levenshtein distance) and Optimal String Alignment (also sometimes called restricted edit distance). For Optimal String Alignment, no substring can be edited more than once.

Normalized, metric, similarity and distance

Although the topic might seem simple, a lot of different algorithms exist to measure text similarity or distance. Therefore the library defines some interfaces to categorize them.

(Normalized) similarity and distance

• StringSimilarity : Implementing algorithms define a similarity between strings (0 means strings are completely different).
• NormalizedStringSimilarity : Implementing algorithms define a similarity between 0.0 and 1.0, like Jaro-Winkler for example.
• StringDistance : Implementing algorithms define a distance between strings (0 means strings are identical), like Levenshtein for example. The maximum distance value depends on the algorithm.
• NormalizedStringDistance : This interface extends StringDistance. For implementing classes, the computed distance value is between 0.0 and 1.0. NormalizedLevenshtein is an example of NormalizedStringDistance.

Generally, algorithms that implement NormalizedStringSimilarity also implement NormalizedStringDistance, and similarity = 1 - distance. But there are a few exceptions, like N-Gram similarity and distance (Kondrak)...

Metric distances

The MetricStringDistance interface : A few of the distances are actually metric distances, which means that verify the triangle inequality d(x, y) <= d(x,z) + d(z,y). For example, Levenshtein is a metric distance, but NormalizedLevenshtein is not.

A lot of nearest-neighbor search algorithms and indexing structures rely on the triangle inequality. You can check "Similarity Search, The Metric Space Approach" by Zezula et al. for a survey. These cannot be used with non metric similarity measures.

Read Javadoc for a detailed description

Shingles (n-gram) based similarity and distance

A few algorithms work by converting strings into sets of n-grams (sequences of n characters, also sometimes called k-shingles). The similarity or distance between the strings is then the similarity or distance between the sets.

Some of them, like jaccard, consider strings as sets of shingles, and don't consider the number of occurences of each shingle. Others, like cosine similarity, work using what is sometimes called the profile of the strings, which takes into account the number of occurences of each shingle.

For these algorithms, another use case is possible when dealing with large datasets:

1. compute the set or profile representation of all the strings
2. compute the similarity between sets or profiles

Levenshtein

The Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.

It is a metric string distance. This implementation uses dynamic programming (Wagner–Fischer algorithm), with only 2 rows of data. The space requirement is thus O(m) and the algorithm runs in O(m.n).

import info.debatty.java.stringsimilarity.*;

public class MyApp {
    
    public static void main (String[] args) {
        Levenshtein l = new Levenshtein();

        System.out.println(l.distance("My string", "My $tring"));
        System.out.println(l.distance("My string", "My $tring"));
        System.out.println(l.distance("My string", "My $tring"));
    }
}

Normalized Levenshtein

This distance is computed as levenshtein distance divided by the length of the longest string. The resulting value is always in the interval [0.0 1.0] but it is not a metric anymore!

The similarity is computed as 1 - normalized distance.

import info.debatty.java.stringsimilarity.*;

public class MyApp {
    
    public static void main (String[] args) {
        NormalizedLevenshtein l = new NormalizedLevenshtein();

        System.out.println(l.distance("My string", "My $tring"));
        System.out.println(l.distance("My string", "My $tring"));
        System.out.println(l.distance("My string", "My $tring"));
    }
}

Weighted Levenshtein

An implementation of Levenshtein that allows to define different weights for different character substitutions.

This algorithm is usually used for optical character recognition (OCR) applications. For OCR, the cost of substituting P and R is lower then the cost of substituting P and M for example because because from and OCR point of view P is similar to R.

It can also be used for keyboard typing auto-correction. Here the cost of substituting E and R is lower for example because these are located next to each other on an AZERTY or QWERTY keyboard. Hence the probability that the user mistyped the characters is higher.

import info.debatty.java.stringsimilarity.*;

public class MyApp {

    public static void main(String[] args) {
        WeightedLevenshtein wl = new WeightedLevenshtein(
                new CharacterSubstitutionInterface() {
                    public double cost(char c1, char c2) {
                        
                        // The cost for substituting 't' and 'r' is considered
                        // smaller as these 2 are located next to each other
                        // on a keyboard
                        if (c1 == 't' && c2 == 'r') {
                            return 0.5;
                        }
                        
                        // For most cases, the cost of substituting 2 characters
                        // is 1.0
                        return 1.0;
                    }
        });
        
        System.out.println(wl.distance("String1", "Srring2"));
    }
}

Damerau-Levenshtein

Similar to Levenshtein, Damerau-Levenshtein distance with transposition (also sometimes calls unrestricted Damerau-Levenshtein distance) is the minimum number of operations needed to transform one string into the other, where an operation is defined as an insertion, deletion, or substitution of a single character, or a transposition of two adjacent characters.

It does respect triangle inequality, and is thus a metric distance.

This is not to be confused with the optimal string alignment distance, which is an extension where no substring can be edited more than once.

import info.debatty.java.stringsimilarity.*;

public class MyApp {


    public static void main(String[] args) {
        Damerau d = new Damerau();
        
        // 1 substitution
        System.out.println(d.distance("ABCDEF", "ABDCEF"));
        
        // 2 substitutions
        System.out.println(d.distance("ABCDEF", "BACDFE"));
        
        // 1 deletion
        System.out.println(d.distance("ABCDEF", "ABCDE"));
        System.out.println(d.distance("ABCDEF", "BCDEF"));
        System.out.println(d.distance("ABCDEF", "ABCGDEF"));
        
        // All different
        System.out.println(d.distance("ABCDEF", "POIU"));
    }
}

Will produce:

1.0
2.0
1.0
1.0
1.0
6.0

Optimal String Alignment

The Optimal String Alignment variant of Damerau–Levenshtein (sometimes called the restricted edit distance) computes the number of edit operations needed to make the strings equal under the condition that no substring is edited more than once, whereas the true Damerau–Levenshtein presents no such restriction. The difference from the algorithm for Levenshtein distance is the addition of one recurrence for the transposition operations.

Note that for the optimal string alignment distance, the triangle inequality does not hold and so it is not a true metric.

import info.debatty.java.stringsimilarity.*;

public class MyApp {


    public static void main(String[] args) {
        OptimalStringAlignment osa = new OptimalStringAlignment();
        
        System.out.println(osa.distance("CA", "ABC"));;
    }
}

Will produce:

3.0

Jaro-Winkler

Jaro-Winkler is a string edit distance that was developed in the area of record linkage (duplicate detection) (Winkler, 1990). The Jaro–Winkler distance metric is designed and best suited for short strings such as person names, and to detect transposition typos.

Jaro-Winkler computes the similarity between 2 strings, and the returned value lies in the interval [0.0, 1.0]. It is (roughly) a variation of Damerau-Levenshtein, where the transposition of 2 close characters is considered less important than the transposition of 2 characters that are far from each other. Jaro-Winkler penalizes additions or substitutions that cannot be expressed as transpositions.

The distance is computed as 1 - Jaro-Winkler similarity.

import info.debatty.java.stringsimilarity.*;

public class MyApp {


    public static void main(String[] args) {
        JaroWinkler jw = new JaroWinkler();
        
        // substitution of s and t
        System.out.println(jw.similarity("My string", "My tsring"));
        
        // substitution of s and n
        System.out.println(jw.similarity("My string", "My ntrisg"));
    }
}

will produce:

0.9740740656852722
0.8962963223457336

Longest Common Subsequence

The longest common subsequence (LCS) problem consists in finding the longest subsequence common to two (or more) sequences. It differs from problems of finding common substrings: unlike substrings, subsequences are not required to occupy consecutive positions within the original sequences.

It is used by the diff utility, by Git for reconciling multiple changes, etc.

The LCS distance between strings X (of length n) and Y (of length m) is n + m - 2 |LCS(X, Y)| min = 0 max = n + m

LCS distance is equivalent to Levenshtein distance when only insertion and deletion is allowed (no substitution), or when the cost of the substitution is the double of the cost of an insertion or deletion.

This class implements the dynamic programming approach, which has a space requirement O(m.n), and computation cost O(m.n).

In "Length of Maximal Common Subsequences", K.S. Larsen proposed an algorithm that computes the length of LCS in time O(log(m).log(n)). But the algorithm has a memory requirement O(m.n²) and was thus not implemented here.

import info.debatty.java.stringsimilarity.*;

public class MyApp {
    public static void main(String[] args) {
        LongestCommonSubsequence lcs = new LongestCommonSubsequence();

        // Will produce 4.0
        System.out.println(lcs.distance("AGCAT", "GAC"));
        
        // Will produce 1.0
        System.out.println(lcs.distance("AGCAT", "AGCT"));
    }
}

Metric Longest Common Subsequence

Distance metric based on Longest Common Subsequence, from the notes "An LCS-based string metric" by Daniel Bakkelund. http://heim.ifi.uio.no/~danielry/StringMetric.pdf

The distance is computed as 1 - |LCS(s1, s2)| / max(|s1|, |s2|)

public class MyApp {

        public static void main(String[] args) {

        info.debatty.java.stringsimilarity.MetricLCS lcs = 
                new info.debatty.java.stringsimilarity.MetricLCS();

        String s1 = "ABCDEFG";   
        String s2 = "ABCDEFHJKL";
        // LCS: ABCDEF => length = 6
        // longest = s2 => length = 10
        // => 1 - 6/10 = 0.4
        System.out.println(lcs.distance(s1, s2));

        // LCS: ABDF => length = 4
        // longest = ABDEF => length = 5
        // => 1 - 4 / 5 = 0.2
        System.out.println(lcs.distance("ABDEF", "ABDIF"));
    }
}

N-Gram

Normalized N-Gram distance as defined by Kondrak, "N-Gram Similarity and Distance", String Processing and Information Retrieval, Lecture Notes in Computer Science Volume 3772, 2005, pp 115-126.

http://webdocs.cs.ualberta.ca/~kondrak/papers/spire05.pdf

The algorithm uses affixing with special character '\n' to increase the weight of first characters. The normalization is achieved by dividing the total similarity score the original length of the longest word.

In the paper, Kondrak also defines a similarity measure, which is not implemented (yet).

import info.debatty.java.stringsimilarity.*;

public class MyApp {

    public static void main(String[] args) {
        
        // produces 0.583333
        NGram twogram = new NGram(2);
        System.out.println(twogram.distance("ABCD", "ABTUIO"));
        
        // produces 0.97222
        String s1 = "Adobe CreativeSuite 5 Master Collection from cheap 4zp";
        String s2 = "Adobe CreativeSuite 5 Master Collection from cheap d1x";
        NGram ngram = new NGram(4);
        System.out.println(ngram.distance(s1, s2));
    }
}

Shingle (n-gram) based algorithms

A few algorithms work by converting strings into sets of n-grams (sequences of n characters, also sometimes called k-shingles). The similarity or distance between the strings is then the similarity or distance between the sets.

The cost for computing these similarities and distances is mainly domnitated by k-shingling (converting the strings into sequences of k characters). Therefore there are typically two use cases for these algorithms:

Directly compute the distance between strings:

import info.debatty.java.stringsimilarity.*;

public class MyApp {
    
    public static void main(String[] args) {
        QGram dig = new QGram(2);
        
        // AB BC CD CE
        // 1  1  1  0
        // 1  1  0  1
        // Total: 2

        System.out.println(dig.distance("ABCD", "ABCE"));
    }
}

Or, for large datasets, pre-compute the profile of all strings. The similarity can then be computed between profiles:

import info.debatty.java.stringsimilarity.KShingling;
import info.debatty.java.stringsimilarity.StringProfile;


/**
 * Example of computing cosine similarity with pre-computed profiles.
 */
public class PrecomputedCosine {

    public static void main(String[] args) throws Exception {
        String s1 = "My first string";
        String s2 = "My other string...";

        // Let's work with sequences of 2 characters...
        Cosine cosine = new Cosine(2);

        // Pre-compute the profile of strings
        Map<String, Integer> profile1 = cosine.getProfile(s1);
        Map<String, Integer> profile2 = cosine.getProfile(s2);

        // Prints 0.516185
        System.out.println(cosine.similarity(profile1, profile2));
    }
}

Pay attention, this only works if the same KShingling object is used to parse all input strings !

Q-Gram

Q-gram distance, as defined by Ukkonen in "Approximate string-matching with q-grams and maximal matches" http://www.sciencedirect.com/science/article/pii/0304397592901434

The distance between two strings is defined as the L1 norm of the difference of their profiles (the number of occurences of each n-gram): SUM( |V1_i - V2_i| ). Q-gram distance is a lower bound on Levenshtein distance, but can be computed in O(m + n), where Levenshtein requires O(m.n)

Cosine similarity

The similarity between the two strings is the cosine of the angle between these two vectors representation, and is computed as V1 . V2 / (|V1| * |V2|)

Distance is computed as 1 - cosine similarity.

Jaccard index

Like Q-Gram distance, the input strings are first converted into sets of n-grams (sequences of n characters, also called k-shingles), but this time the cardinality of each n-gram is not taken into account. Each input string is simply a set of n-grams. The Jaccard index is then computed as |V1 inter V2| / |V1 union V2|.

Distance is computed as 1 - similarity. Jaccard index is a metric distance.

Sorensen-Dice coefficient

Similar to Jaccard index, but this time the similarity is computed as 2 * |V1 inter V2| / (|V1| + |V2|).

Distance is computed as 1 - similarity.

Ratcliff-Obershelp

Ratcliff/Obershelp Pattern Recognition, also known as Gestalt Pattern Matching, is a string-matching algorithm for determining the similarity of two strings. It was developed in 1983 by John W. Ratcliff and John A. Obershelp and published in the Dr. Dobb's Journal in July 1988

Ratcliff/Obershelp computes the similarity between 2 strings, and the returned value lies in the interval [0.0, 1.0].

The distance is computed as 1 - Ratcliff/Obershelp similarity.

import info.debatty.java.stringsimilarity.*;

public class MyApp {


    public static void main(String[] args) {
        RatcliffObershelp ro = new RatcliffObershelp();
        
        // substitution of s and t
        System.out.println(ro.similarity("My string", "My tsring"));
        
        // substitution of s and n
        System.out.println(ro.similarity("My string", "My ntrisg"));
    }
}

will produce:

0.8888888888888888
0.7777777777777778

Experimental

SIFT4

SIFT4 is a general purpose string distance algorithm inspired by JaroWinkler and Longest Common Subsequence. It was developed to produce a distance measure that matches as close as possible to the human perception of string distance. Hence it takes into account elements like character substitution, character distance, longest common subsequence etc. It was developed using experimental testing, and without theoretical background.

import info.debatty.java.stringsimilarity.experimental.Sift4;

public class MyApp {

    public static void main(String[] args) {
        String s1 = "This is the first string";
        String s2 = "And this is another string";
        Sift4 sift4 = new Sift4();
        sift4.setMaxOffset(5);
        double expResult =  11.0;
        double result = sift4.distance(s1, s2);
        assertEquals(expResult, result, 0.0);
    }
}

Users

• StringSimilarity.NET a .NET port of java-string-similarity
• OrientDB string-metrics wraps java-string-similarity to provide different string similarity and distance measures as SQL functions in OrientDB

Use java-string-similarity in your project and want it to be mentioned here? Don't hesitate to drop me a line!

Security & stability

java-string-similarity's Issues

Is there any String Subsequence Kernel JAVA implementation?

SSK is a kernel method for measuring the similarity between strings.
Any implementations?
Hope to see.

sift4

I've made a java port of the sift4 common algorithm. https://siderite.blogspot.com/2014/11/super-fast-and-accurate-string-distance.html

Is there any interest in adding it here?

NGram exact match varying results

I am using this library in an Apache Spark application (using scala).

I have been seeing variable results using the NGram algorithm where exact matches result is either "0.0" or "1.0". Below are some examples.

`QGram dig = new QGram(2);

dig.distance("S","S")
//result = 1.0

dig.distance("Kirk","Kirk")
//result = 0.0

dig.distance("07426796542","07426796542")
//result = 0.0`

Should all these examples not result in a score of 1.0 as they are exactly the same?

A Ratcliff-Obershelp implementation would be helpful

There is a library which has an accurate implementation but its based on scala.

Implement Kondrak n-gram similarity

multilanguage similarity

Hello!
Can the java-string-similarity be applied to many languages such as Germany,French,Chinese and so on.Can it be is relative to language type?

Make KShingling threadsafe

https://github.com/tdebatty/java-string-similarity/blob/master/src/main/java/info/debatty/java/stringsimilarity/KShingling.java

In this way precomputing profiles could happen in parallel:
https://github.com/tdebatty/java-string-similarity/blob/master/src/main/java/info/debatty/java/stringsimilarity/examples/PrecomputedCosine.java

Re-authorize Coveralls

I was trying to look at the Coveralls data here and it can't show the source code because it says the repo owner must re-authorize with GitHub:

SOURCE NOT AVAILABLE
The owner of this repo needs to re-authorize with github; their OAuth credentials are no longer valid so the file cannot be pulled from the github API.

Was version 0.15 removed from Maven?

I could swear I successfully used version 0.15 before, whereas right now the latest published version seems to be 0.13. Am I only hallucinating? :)

using this code in netbeans 12.5

Hi here,
I want to use this algorithm for a problem i'm trying to solve between and inventory excel and images related to the products inside the excel spreedsheet so i'm working into my own code in Netbeans 12.5 with Java but i'm not able to use them i tried to clone the code and the librarie into the project creat and object
Cosine cosine = new Cosine(2);
for example but there is no reference for importing the class. i tried also to make an import of the package
import src.test.java.info.debatty.java.*;
but still not working. I am not a full Java developer , i just trying to solve a problem but with Software, if you can give a better guide of how to implement it, much appreciate it.

This is great! Thank you! 🙏

🙏🙏🙏

Jaro Winkler similarity on short strings

I am trying to use jaro wrinkler similarity to check colors strings coming from user inputted form against a palette of fixed colors.

Using jaro wrinkler similarity, I get these kind of results for very short strings:

• s1 = "ed" - s2 = "red" -> similarity = 0
• s1 = "nude" - s2 = "red" -> similarity = 0.5833333134651184

Is it correct to get similarity = 0 in the first case?

Implement Damerau optimal string alignment

Implement Smith-Waterman-Gotoh algorithm

"An Improved Algorithm for Matching Biological Sequences", Gotoh (1982):
The algorithm of Waterman et al. (1976) for malchirlg biological seyuences was modified under some limitations to be accomplished in essentially MN steps, instead of the M²N steps necessary in the original algorithm.
http://www.genome.ist.i.kyoto-u.ac.jp/~aln_user/archive/JMB82.pdf

Implement BLAST heuristic

accelerate calculate Jaccard

java-string-similarity/src/main/java/info/debatty/java/stringsimilarity/Jaccard.java:

        Map<String, Integer> profile1 = getProfile(s1);
        Map<String, Integer> profile2 = getProfile(s2);

        Set<String> union = new HashSet<String>();
        union.addAll(profile1.keySet());
        union.addAll(profile2.keySet());

        int inter = 0;

        for (String key : union) {
            if (profile1.containsKey(key) && profile2.containsKey(key)) {
                inter++;
            }
        }

        return 1.0 * inter / union.size();

can be modified to:

        Map<String, Integer> profile1 = getProfile(s1);
        Map<String, Integer> profile2 = getProfile(s2);

        Set<String> union = new HashSet<String>();
        union.addAll(profile1.keySet());
        union.addAll(profile2.keySet());

        double x = profile1.size();
        double y = profile2.size();
        double z = union.size();
        return (x + y - z) / z;

Feature Request: Ability to store and specify profiles

Use-case: Searching for a best match

Given a string Q, and a list of strings L, I want to find the distance between Q and every element of L and then pick the element with the shortest distance.

Current solution

The current solution is to call a distance function with two String parameters. This function assumes nothing about the Strings and hence some information is recomputed every time (for example, profiles of a string).

Proposal

Two additional APIs can be provided to improve performance:

   Profile getProfile(String)

   double distance(Profile p1, Profile p2)

For convenience, a third API would also be useful:

  double distance(Profile p, String s) {
    return distance(p, getProfile(s));
  }

This can be used like this:

  String query = "alex";
  QGram qg = new QGram(2);
  Profile queryProfile = qg.getProfile(query);
  list forEach { element ->
    println(qg.distance(queryProfile, element));
  }

Further, if the list is going to be persistent, it could also be possible to serialize the profile of each element of the list into the persistent store. Then, both the query string and the list element's profile need not be recomputed every time!

---

If the overall idea sounds good to you, I will write more about how the Profile type could be made type safe across the different implementations of StringSimilarityInterface.

传说中的字符串编辑距离

传说中的字符串编辑距离

two questions about code architecture

1. Would you explain why you decided not to make StringDistance.distance() static? Why is it necessary to create an instance of StringDistance (or of a derived class) to compute the distance? The method does not use this anyway...

2. Also, why StringDistance derives from java.io.Serializable?

KShingling and StringProfile classes are not available

These classes are not the latest release:

info.debatty.java.stringsimilarity.KShingling;
info.debatty.java.stringsimilarity.StringProfile;

Jaro-Winkler handling empty strings

Hi,

When using jaro-winkler method I see that when comparing empty strings '' and '' the score return is 1, indicating a complete mismatch.

here is an example below implemented in scala

val jaro = new JaroWinkler()
println(jaro.distance("",""))
println(jaro.distance("match","match"))

Can this be altered so that when comparing empty strings it returns an exact match of 0?

Thanks

Fix README

Can you check the sections about the Jaccard index and the Sorensen-Dice coefficient in the README file? They both report the distance as computed as 1 - cosine similarity.

Suggestion: "Typical Uses" column in Overview table

Would it be possible for you to add a column called "Typical Uses" in the table in Overview? Perhaps to briefly mention where the algorithm is typically used (for example "diff utility, Git reconciliation" for LCS).

Not sure if you will have or can find this info. It will be helpful to pick out the algorithm best suited for the user's needs.

multilanguage similarity

Hello!
Can the java-string-similarity be applied to many languages such as Germany,French,Chinese and so on.Can it be is relative to language type?

Licensing?

Can you attach a license to this project? I'm interested in using and possibly contributing, but without a license, I can't be sure of your intentions.

Damerau–Levenshtein and triangle equality

Hi. I noticed something whilst browsing this git...

The readme states that the triangle inequality does not hold for the Demerau-Levenshtein algorithm. This is only true if you use a restricted edit distance implementation of the algorithm. If you allow for the 'unrestricted' case (of adjacent transpositions) the inequality will hold. This is, however, at the expense of added computational complexity. See here

Implementing the adjacent transposition calc will allow you to use Demerau-Levenshtein in a metric tree, for example.

Cheers

Chris

Thread Safety

Hi,

Could you please document the if NormalizedLevenshtein or other implementation class is thread safe or not.

Thanks

Jaro winkler similarity on Empty strings

I am using jaro wrinkler similarity to check similarities between names. In one of the use case, i found this issue.
s1 = "SOME NAME" - s2 = "" -> similarity = 1
Why is the output "1". shouldn't "1" be for exact matches ?
please help

version details : java-string-similarity -> 2.0.0

String similarity techniques issue

We are working on Record linkage project. We are observing a strange behavior from all of the standard technique like Jaro Winkler, Levenshtein, N-Gram, Damerau-Levenshtein, Jaccard index, Sorensen-Dice
which we used from your repository.

Say,
String 1= MINI GRINDER KIT
String 2= Weiler 13001 Mini Grinder Accessory Kit, For Use With Small Right Angle Grinders
String 3= Milwaukee Video Borescope, Rotating Inspection Scope, Series: M-SPECTOR 360, 2.7 in 640 x 480 pixels High-Resolution LCD, Plastic, Black/Red

In the above case string 1 and string 2 are related the score of all the methods as shown below.
Jaro Winkler -> 0.391666651
Levenshtein -> 75
N-Gram, -> 0.9375
Damerau -> 75
Jaccard index -> 0
Sorensen-Dice -> 0
Cosine -> 0

But string 1 and string 3 are not at all related, but distance method are giving very high score.
Jaro Winkler -> 0.435714275
Levenshtein -> 133
N-Gram, -> 0.953571439
Damerau -> 133
Jaccard index -> 1
Sorensen-Dice -> 0
Cosine -> 0

Any thoughts .?

Couldn't build the library

Have I tried to build the library after downloading on my machine using maven, it failed and there an error in NGram.java distance() about overriding. I am not sure how to resolve this. Let me know how to execute the example you mentioned in read.me. Thank you.

API changes review for Java String Similarity

The review of API changes for the Java String Similarity library since 0.1 version: https://abi-laboratory.pro/java/tracker/timeline/java-string-similarity/

The report is updated 3 times a week. Hope it will be helpful for users and maintainers of the library.

The report is generated by the https://github.com/lvc/japi-tracker tool for jars at http://central.maven.org/maven2/info/debatty/java-string-similarity/ according to the article https://wiki.eclipse.org/Evolving_Java-based_APIs_2.

Thank you.

Cosine Similarity

Hello,

I just tried cosine similarity and the result was above 1!
My code is attached.

ConsineSimilarity.zip

@override annotation in distance

@override annotation in NGram.java shows error in Java 6 and above. It should be removed as the method is not being overridden. It is just being implemented.

Where is the SparseIntegerVector (and related) code?

Where is the SparseIntegerVector (and related) code?

Need to update README

In README paragraph:
https://github.com/tdebatty/java-string-similarity#shingle-n-gram-based-algorithms
There are import of classes that does not exist already:
import info.debatty.java.stringsimilarity.KShingling;
import info.debatty.java.stringsimilarity.StringProfile;

There is no such package on npm, and I am trying to do this!

I have not found such perfect package on npm, so, could I translate this package from Java to TypeScript?

This is the semi-finished repo. Please look it over: https://github.com/hellojayjay/string-metric

Will you allow me to do so?

Coveralls stats is not updated

Starting from this build, the build command of mvn clean cobertura:cobertura coveralls:report in .travis.yml failed. Hence the coveralls stats was not updated since this commit.

Error message:
[ERROR] Failed to execute goal org.codehaus.mojo:cobertura-maven-plugin:2.7:instrument (default-cli) on project java-string-similarity: Execution default-cli of goal org.codehaus.mojo:cobertura-maven-plugin:2.7:instrument failed: Plugin org.codehaus.mojo:cobertura-maven-plugin:2.7 or one of its dependencies could not be resolved: Could not find artifact com.sun:tools:jar:0 at specified path /usr/local/lib/jvm/openjdk11/../lib/tools.jar -> [Help 1]

Turned out it was because the cobertura-maven-plugin is not supporting java 8 and beyond. False alarm? Consider JaCoCo instead?

N-Gram Example Comment Incorrect?

This issue was reported to my team's .NET port of your library but I confirmed that it is an issue here as well.

The example on the README shows that the N-Gram code expects values of 0.416666 and 0.97222. However, different results are given when the code is actually ran. I am not sure if this is a bug in the code, or that the README comment is outdated/incorrect.

I created a unit test for the README example, and sure enough it fails:

    @Test
    public void exampleFromReadme() {
        // produces 0.416666
        NGram twogram = new NGram(2);
        assertEquals(0.416666, twogram.distance("ABCD", "ABTUIO"), 0.001);

        // produces 0.97222
        String s1 = "Adobe CreativeSuite 5 Master Collection from cheap 4zp";
        String s2 = "Adobe CreativeSuite 5 Master Collection from cheap d1x";
        NGram ngram = new NGram(4);
        assertEquals(0.97222, ngram.distance(s1, s2), 0.001);
    }

Results:

java.lang.AssertionError: 
Expected :0.416666
Actual   :0.5833333134651184

This result (0.583) is the same that we get on the .NET side of things. As I am not an expert in these algorithms, I am unsure if this is a code bug or a need to update the README.

C# .NET Port

I've created a port of your library to the .NET Framework using C#. The repository for the port is here: https://github.com/feature23/StringSimilarity.NET

Aside from changes to C# syntax, I made a few changes to make the code a bit more idiomatic for the language, but it's as true to the original as possible. Also I have, of course, linked back here to your original library. In any case, I just wanted to make you aware of the port, and please let me know if you have any interested in collaborating!

Kotlin Multiplatform port

Similar to #59 and #19, I decided to make a pure Kotlin port of this library.

Although on the server side, Kotlin has compat with Java libraries, I wanted to make it a Multiplatform lib because why not.

Currently there is no README, but when I finish it, I'm going to mention this repo in the README.

The port is here, under kt-string-similarity: https://github.com/solo-studios/kt-fuzzy

Any problem using singleton? [Question]

Hi Team,

Can anyone imagine or already get any problem to use a that lib (RatcliffObershelp specificly) in a Singleton class?

My use :

import info.debatty.java.stringsimilarity.RatcliffObershelp;

public class StringSimilarityCalculator {

    private static StringSimilarityCalculator stringSimilarityCalculator;
    private static RatcliffObershelp calculo = new RatcliffObershelp();
    private static final double minSimilaridadeEndereco = 0.90;

    private StringSimilarityCalculator(){}

    public static StringSimilarityCalculator getInstance() {

        if(stringSimilarityCalculator == null){

            stringSimilarityCalculator = new StringSimilarityCalculator();
        }

        return stringSimilarityCalculator;
    }

    public boolean isStringEnderecoSimilar(String enderecoPagador, String enderecoPagadorBanco){

        return calculo.similarity(enderecoPagador.toLowerCase(),
                enderecoPagadorBanco.toLowerCase()) >= minSimilaridadeEndereco;
    }
}

Thanks !!

Ngram.distance returns a similarity score

All of the metrics set a distance of 0 to imply exact string match and 1 to imply distance strings. Ngram.distance is backwards and returns 1 for an exact match and 0 for distant strings.

Publish version 1.2.0 to maven central repo

Currently the latest version on maven central is 1.1.0
The latest released version in 1.2.0 according to releases page.

Thanks