pharo-ai / edit-distances

Several edit distances algorithms implemented in Pharo

Table of contents

• Table of contents
• Description
• How to install it
• How to depend on it
• Implemented distances

Description

Edit distance is a way of quantifying how dissimilar two strings are to one another by counting the minimum number of operations required to transform one string into the other. The distance between the two documents is defined as "The minimum number of insertions, deletions, and substitutions required to transform one string into the other". Edit-based distances for which such weights can be defined are usually refered to as generalized distances. These methods are independent of a searching algorithm, i.e. Levenshtein or Hamming edit distances can be applied separately to a searching algorithm.

Edit distances find applications in natural language processing, where automatic spelling correction can determine candidate corrections for a misspelled word by selecting words from a dictionary that have a low distance to the word in question.

This package provides methods to determine the distance between two objects, for example, between Strings. Those objects, often represent documents. Normally they are strings but also they can be arrays of numbers that represent a document.

• An edit distance does NOT count matches.
• Some commonly refered "edit distances" compare corresponding elements and require objects of equal length (examples: Euclidean, Manhattan, Hamming,...)
• To speed up computation, some distances are based in "tokens", and also referred as token-based distances (example: Cosine similarity).

How to install it

EpMonitor disableDuring: [
	Metacello new
		repository: 'github://pharo-ai/edit-distances/src';
		baseline: 'AIEditDistances';
		load ]

How to depend on it

If you want to add the AIEditDistances to your Metacello Baselines or Configurations, copy and paste the following expression:

spec
	baseline: 'AIEditDistances' 
	with: [ spec repository: 'github://pharo-ai/edit-distances/src' ]

Implemented distances

These are the currently implemented distance.

Note that we are currently working on this project so we will be implementing more distance in the future.

• Euclidean norm, also known as Euclidean length, L2 norm, L2 distance, l^2 norm
• Manhattan distance, also known as City Block Distance.
• Cosine similarity, note this is not the same as TF-IDF.
• Levenshtein distance
• Restricted Damerau-Levenshtein
• Damerau-Levenshtein
• Kendall Tau distance (with and without normalization)
• Szymkiewicz-Simpson coefficient
• Shingles similarity
• Hamming distance
• Damerau Levenshtein distance
• Restricted Damerau Levenshtein distance
• Helling distance
• Brew distance

All the distances are polymorphic. They have the same API.

To use them, send the message distanceBetween: and:.

aDistance distanceBetween: xCollection and: yCollection

Euclidean norm

The Euclidean distance between two points in Euclidean space is the length of a line segment between the two points. It can be calculated from the Cartesian coordinates of the points using the Pythagorean theorem, therefore occasionally being called the Pythagorean distance.

euclideanDistance := AIEuclideanDistance new.

euclideanDistance distanceBetween: #( 0 3 4 5 ) and: #( 7 6 3 -1 ). "9.746794344808963"

Manhattan distance

In the Manhattan distance, the Euclidean geometry is replaced by a new metric in which the distance between two points is the sum of the (absolute) differences of their coordinates.

More formally, we can define the Manhattan distance, also known as the L1-distance, between two points in an Euclidean space with fixed Cartesian coordinate system is defined as the sum of the lengths of the projections of the line segment between the points onto the coordinate axes.

manhattanDistance := AIManhattanDistance new.

manhattanDistance distanceBetween: #( 10 20 10 ) and: #( 10 20 20 ). "10"

Cosine similarity

Cosine similarity is a metric used to determine how similar the documents are irrespective of their size. Mathematically, it measures the cosine of the angle between two vectors projected in a multi-dimensional space.

cosineDistance := AICosineSimilarityDistance new.

cosineDistance distanceBetween: #(3 45 7 2) and: #(2 54 13 15) "0.9722842517123499"

Levenshtein distance

The Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.

levenshteinDistance := AILevenshteinDistance new.

levenshteinDistance distanceBetween: 'zork' and: 'fork' "1"

Restricted Damerau-Levenshtein distance

The restricted Damerau-Lavenshtein distance, also known as the optimal string alignment distance or restricted edit distance is a string metric for measuring the edit distance between two sequences.

This distance differs from the classical Levenshtein distance by including transpositions (swap) among its allowable operations in addition to the three classical single-character edit operations (insertions, deletions and substitutions).

How to use it on Playground:

restrictedDamerauLevenshtein := AIRestrictedDamerauLevenshteinDistance new.

restrictedDamerauLevenshtein distanceBetween: 'an act' and: 'a cat' "2".

Brief explanation :

To go from 'an act' to 'a cat' we need to add an 'n' after the 'a' and swap the 'ac' into 'ca' (instead of substituting 'a' for 'c' and 'c for 'a'). So in total we did 2 changes, that's why when we press command G (CMD+G) on the code above we have the value of 2.

Damerau-Levenshtein distance

The Damerau-Lavenshtein distance is a string metric for measuring the edit distance between two sequences. The difference between this distance and the restricted Damerau-Levenshtein distance consists in that the restricted one computes the number of edit operations needed to make the strings equal under the condition that no substring is edited more than once, whereas this algorithm presents no such restriction.

How to use it on Playground:

damerauLevenshtein := AIDamerauLevenshteinDistance new.

damerauLevenshtein distanceBetween: 'a cat' and: 'a abct' "2".

Brief explanation :

To go from 'a cat' to 'a abct' we need to swap the 'ac' into 'ca' (instead of substituting 'a' for 'c' and 'c for 'a') and add 'b' between the edited substring. This last edit wouldn't be possible because that would require the substring to be edited more than once, which is not allowed in OSA(optimal string alignment). So in total we did 2 changes, that's why when we press command G (CMD+G) on the code above we have the value of 2.

Kendall Tau distance

The Kendall tau rank distance is a metric that counts the number of pairwise disagreements between two ranking lists. The larger the distance, the more dissimilar the two lists are.

Kendall tau distance is also called bubble-sort distance since it is equivalent to the number of swaps that the bubble sort algorithm would take to place one list in the same order as the other list. The Kendall tau distance was created by Maurice Kendall.

This distance a a little special. The distance itself is the number of discordant pairs that there is between the two ranked lists. But, often the result is normalized. In this implementation we normalize the distance by default. To not normalize the result you must send the message normalizeResult: false.

kendallTauDistance := AIKendallTauDistance new.
kendallTauDistance normalizeResult: false.

There are two normalizers implemented. If no normalizer is specified, then the default normalized will be used. AIKendallTauDistance class>>#defaultNormalizer. Which is the AIBKendallTauNormalizer.

• The first normalizer is AIBKendallTauNormalizer. It uses the following formula to normalize the discordant pairs. This is the default normalizer.

tau_b = (P - Q) / sqrt((P + Q + T) * (P + Q + U))

Where P is the number of concordant pairs, Q the number of discordant pairs, T the number of ties only in x, and U the number of ties only in y. If a tie occurs for the same pair in both x and y, it is not added to either T or U.

• The second normalizer that we have is the AICKendallTauNormalizer. It has the following formula:

tau_c = 2 (P - Q) / (n**2 * (m - 1) / m)

Where P is the number of concordant pairs, Q the number of discordant pairs. n is the total number of samples, and m is the number of unique values in either x or y, whichever is smaller.

Example with normalization:

kendallTauDistance := AIKendallTauDistance new.

kendallTauDistance distanceBetween: #( 1 2 3 4 5 ) and: #(3 4 1 2 5 ). "(1/5)"

Example using another normalizer:

kendallTauDistance := AIKendallTauDistance new.
kendallTauDistance useNormalizer: AICKendallTauNormalizer.

kendallTauDistance distanceBetween: #( 1 2 3 4 5 ) and: #(3 4 1 2 5 ). "(1/5)"

Example without normalization:

kendallTauDistance := AIKendallTauDistance new.
kendallTauDistance normalizeResult: false.

kendallTauDistance distanceBetween: #( 1 2 3 4 5 ) and: #(3 4 1 2 5 ). "4"

Szymkiewicz-Simpson coefficient

Also called overlap coefficient, is a similarity measure that measures the overlap between two finite sets. As this distance is conceptually only for sets, it is defined only in the Set class. So, we method is expecting an instance of Set as an argument.

szymkiewiczSimpsonDistance := AISzymkiewiczSimpsonDistance new.

szymkiewiczSimpsonDistance
	distanceBetween: #( 1000 2 0.5 3 6 88 99 ) asSet
	and: #( 1000 0.5 99 ) asSet. "1.0"

Shingles similarity

A string similarity metric based on Shingles encoding [1]. It is well suited for detecting strings that underwent small modifications. Shingles encoding changes a little when string changes a little.

[1] Broder, Andrei Z. "On the resemblance and containment of documents." Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171). IEEE, 1997.

shinglesSimilarity := AIShinglesSimilarity
  slidingWindowSize: 2
  maxEncodingSize: 5.

shinglesSimilarity distanceBetween: #(lorem ipsum dolor sit amet) and: #(hello world lorem ipsum) "(7/24)"

slidingWindowSize and maxEncodingSize have default values. Note that the result will change acording to the values that are set.

shinglesSimilarity := AIShinglesSimilarity new.

shinglesSimilarity distanceBetween: #(lorem ipsum dolor sit amet) and: #(hello world lorem ipsum) "0"

edit-distances's Issues

Implement Episode Distance

Also known as "Episode matching".
Allows only insertions, which cost 1.

Example:

MARADONA
M ARADONAS

Resulting distance = 1

[Cleanup] Refactor the tests

Now the tests are kind of unreadable. Maybe it is better to move the all the tests of one distance to a class.

Implement Szymkiewicz–Simpson coefficient

Reference: https://en.wikipedia.org/wiki/Overlap_coefficient

PolyMath is not loaded when installing the project using the Metacello script

Steps to reproduce:

1. Create a new Pharo image (8, 9 or 10).
2. Run the How to install it? scrip that is available in the README.
3. You will se that PolyMath is not loaded:
   

Implement Jensen-Shannon distance

Reference: https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence

Don't load unnecessary external PolyMath tests from Baseline

Implement Brew Distance

Original algorithm documentation: https://web.archive.org/web/20090510012823/http://ling.ohio-state.edu/~cbrew/795M/string-distance.html

Implement Damerau-Levenshtein distances

Note there are two versions:

• The full Damerau-Levenshtein distance
• The restricted Damerau-Levenshtein distance

There's two implementation of distance the Levenshtein

There's an iterative and a recursive implementation of the Levenshtein distance. They both produce the same result. We need to see which one is the most efficient in terms of time execution and remove the other one.

Implement Hamming distance

It should work for both numbers and strings

Do we need to delete the Levenshtein distance implementation in Array because it's never used and produces an exception?

Create CI and GitHub action for this repo

And also put the CI badge in the README

Add test for Array>>#levenshteinDistanceTo:

The method Array>>#levenshteinDistanceTo: is never called by any test. Only the ones that are called are the levenshtein methods defined in String class.

Implement Minkowski and make Euclidean and Manhattan using it

Minkowski Distance is the generalization of both Manhattan and Euclidean distance. Euclidean is the Minkowski distance to the power of 2 and Manhattan to the power of 1

Implement Tau Distance

Note there are 3 versions: tau-a, tau-b and tau-c

Scipy reference: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html#scipy.stats.kendalltau

Implement Hellinger Distance or Sørensen–Dice coefficient

Reference: https://en.wikipedia.org/wiki/Hellinger_distance

Implement keyboard or typing distance

Reference implementations: https://stackoverflow.com/questions/18949684/keyboard-distance-in-python

Add missing defaultDistanceMetric

When evaluating without specifying the metric algorithm to use:

#(0 3 4 5) distanceTo: #(7 6 3 -1) 

A "Instance of Array class did not understand #defaultDistanceMetric" is raised.

Add the missing method.

Implement or port Mahalanobis distance

Reference: https://en.wikipedia.org/wiki/Mahalanobis_distance

[Think] about the architecture

Currently, we have a lot of the logic of the distances as extensions methods.

Like:

AICosineSimilarityDistance>>#distanceBetween:and:

anObject cosineSimilarityDistanceTo: anotherObject

Array>>#cosineSimilarityDistanceTo:

	| num size1 size2 |

	num := (self * anArray) sum.
	size1 := (self * self) sum sqrt.
	size2 := (anArray * anArray ) sum sqrt.
	
	^ num / (size1 * size2)

We have to be consistents... Need to discuss further

Implement Jaro-Winkler

Not to be confused with the Jaro distance which is a special case of the Jaro-Winkler distance with p = 0.

This is the original author implementation:

/* strcmp95.c   Version 2						      */

/* The strcmp95 function returns a double precision value from 0.0 (total
   disagreement) to 1.0 (character-by-character agreement).  The returned 
   value is a measure of the similarity of the two strings.                   */

/* Date of Release:  Jan. 26, 1994					      */
/* Modified: April 24, 1994  Corrected the processing of the single length
             character strings.
   Authors:  This function was written using the logic from code written by
             Bill Winkler, George McLaughlin and Matt Jaro with modifications
             by Maureen Lynch. 
   Comment:  This is the official string comparator to be used for matching 
             during the 1995 Test Census.                                     */

#include <ctype.h>
#include <string.h>

#define NOTNUM(c)	((c>57) || (c<48))
#define INRANGE(c)      ((c>0)  && (c<91))
#define MAX_VAR_SIZE 61
#define NULL60 "                                                            "

double  strcmp95(char *ying, char *yang, long y_length, int *ind_c[])

{
/* Arguments:

   ying and yang are pointers to the 2 strings to be compared.  The strings
   need not be NUL-terminated strings because the length is passed.

   y_length is the length of the strings. 

   ind_c is an array that is used to define whether certain options should be 
   activated.  A nonzero value indicates the option is deactivated.  
   The options are:
     ind_c[0] Increase the probability of a match when the number of matched
              characters is large.  This option allows for a little more
              tolerance when the strings are large.  It is not an appropriate
              test when comparing fixed length fields such as phone and
              social security numbers.
     ind_c[1] All lower case characters are converted to upper case prior
              to the comparison.  Disabling this feature means that the lower 
              case string "code" will not be recognized as the same as the 
              upper case string "CODE".  Also, the adjustment for similar 
              characters section only applies to uppercase characters.

   The suggested values are all zeros for character strings such as names.    */

static	int	pass=0,	adjwt[91][91];
static	char	sp[39][2] =
 {'A','E',  'A','I',  'A','O',  'A','U',  'B','V',  'E','I',  'E','O',  'E','U',
  'I','O',  'I','U',  'O','U',  'I','Y',  'E','Y',  'C','G',  'E','F',
  'W','U',  'W','V',  'X','K',  'S','Z',  'X','S',  'Q','C',  'U','V',
  'M','N',  'L','I',  'Q','O',  'P','R',  'I','J',  '2','Z',  '5','S',
  '8','B',  '1','I',  '1','L',  '0','O',  '0','Q',  'C','K',  'G','J',
  'E',' ',  'Y',' ',  'S',' '};

char    ying_hold[MAX_VAR_SIZE],
        yang_hold[MAX_VAR_SIZE],
        ying_flag[MAX_VAR_SIZE],
        yang_flag[MAX_VAR_SIZE];

double  weight,	Num_sim;

long    minv,   search_range,   lowlim,    ying_length,
        hilim,  N_trans,        Num_com,   yang_length;

int	yl1,	yi_st,	N_simi;

register        int     i,      j,      k;

/* Initialize the adjwt array on the first call to the function only.
   The adjwt array is used to give partial credit for characters that 
   may be errors due to known phonetic or character recognition errors.
   A typical example is to match the letter "O" with the number "0"           */
if (!pass) {
  pass++;
  for (i=0; i<91; i++) for (j=0; j<91; j++) adjwt[i][j] = 0;
  for (i=0; i<36; i++) {
    adjwt[sp[i][0]][sp[i][1]] = 3;
    adjwt[sp[i][1]][sp[i][0]] = 3;
} }

/* If either string is blank - return - added in Version 2                    */
if (!strncmp(ying,NULL60,y_length)) return(0.0);                 
if (!strncmp(yang,NULL60,y_length)) return(0.0);                 

/* Identify the strings to be compared by stripping off all leading and 
   trailing spaces.							      */
k = y_length - 1;
for(j = 0;((ying[j]==' ') && (j < k));j++);
for(i = k;((ying[i]==' ') && (i > 0));i--);
ying_length = i + 1 - j;
yi_st = j;

for(j = 0;((yang[j]==' ') && (j < k));j++);
for(i = k;((yang[i]==' ') && (i > 0));i--);
yang_length = i + 1 - j;

ying_hold[0]=yang_hold[0]=0;
strncat(ying_hold,&ying[yi_st],ying_length);
strncat(yang_hold,&yang[j],yang_length);

if (ying_length > yang_length) {
  search_range = ying_length;
  minv = yang_length;
  }
 else {
  search_range = yang_length;
  minv = ying_length;
  }

/* If either string is blank - return                                         */
/* if (!minv) return(0.0);                   removed in version 2             */

/* Blank out the flags							      */
ying_flag[0] = yang_flag[0] = 0;
strncat(ying_flag,NULL60,search_range);
strncat(yang_flag,NULL60,search_range);
search_range = (search_range/2) - 1;
if (search_range < 0) search_range = 0;   /* added in version 2               */

/* Convert all lower case characters to upper case.                           */
if (!ind_c[1]) {
  for (i = 0;i < ying_length;i++) if (islower(ying_hold[i])) ying_hold[i] -= 32;
  for (j = 0;j < yang_length;j++) if (islower(yang_hold[j])) yang_hold[j] -= 32;
}

/* Looking only within the search range, count and flag the matched pairs.    */
Num_com = 0; 
yl1 = yang_length - 1;
for (i = 0;i < ying_length;i++) {
  lowlim = (i >= search_range) ? i - search_range : 0;
  hilim = ((i + search_range) <= yl1) ? (i + search_range) : yl1;
  for (j = lowlim;j <= hilim;j++)  {
    if ((yang_flag[j] != '1') && (yang_hold[j] == ying_hold[i])) {
        yang_flag[j] = '1';
        ying_flag[i] = '1';
        Num_com++;
        break;
} } }

/* If no characters in common - return                                        */
if (!Num_com) return(0.0);                          

/* Count the number of transpositions                                         */
k = N_trans = 0;
for (i = 0;i < ying_length;i++) {
  if (ying_flag[i] == '1') {
    for (j = k;j < yang_length;j++) {
        if (yang_flag[j] == '1') { 
         k = j + 1;
         break;
    } }
    if (ying_hold[i] != yang_hold[j]) N_trans++;
} }
N_trans = N_trans / 2;

/* adjust for similarities in nonmatched characters                           */
N_simi = 0;
if (minv > Num_com) {   
  for (i = 0;i < ying_length;i++) {
    if (ying_flag[i] == ' ' && INRANGE(ying_hold[i])) { 
      for (j = 0;j < yang_length;j++) {
        if (yang_flag[j] == ' ' && INRANGE(yang_hold[j])) {
          if (adjwt[ying_hold[i]][yang_hold[j]] > 0) {
            N_simi += adjwt[ying_hold[i]][yang_hold[j]];
            yang_flag[j] = '2';
            break;
} } } } } }
Num_sim = ((double) N_simi)/10.0 + Num_com;

/* Main weight computation.						      */
weight= Num_sim / ((double) ying_length) + Num_sim / ((double) yang_length)
   + ((double) (Num_com - N_trans)) / ((double) Num_com);
weight = weight / 3.0;

/* Continue to boost the weight if the strings are similar                    */
if (weight > 0.7) {

  /* Adjust for having up to the first 4 characters in common                 */
  j = (minv >= 4) ? 4 : minv;
  for (i=0;((i<j)&&(ying_hold[i]==yang_hold[i])&&(NOTNUM(ying_hold[i])));i++); 
  if (i) weight += i * 0.1 * (1.0 - weight);

  /* Optionally adjust for long strings.                                      */
  /* After agreeing beginning chars, at least two more must agree and 
       the agreeing characters must be > .5 of remaining characters.          */
  if ((!ind_c[0]) && (minv>4) && (Num_com>i+1) && (2*Num_com>=minv+i)) 
  if (NOTNUM(ying_hold[0])) 
    weight += (double) (1.0-weight) *
   ((double) (Num_com-i-1) / ((double) (ying_length+yang_length-i*2+2)));
 }

return(weight);

} /* strcmp95 */