msuchane / near-facsimile Goto Github PK

The complete run might take a long time. It would be useful to display the current file number out of the total number of comparisons, each time when printing out similar files.

Given a module title, find modules in the repository that already have a similar title.

This might be useful when deciding whether to write a new module or reuse an existing one.

Present the results as a graph for better readability and interpretation. The natural candidate for this graph would be a histogram.

Either draw the graph internally, or provide instructions for how to plot it with external tools.

One issue here is that accurate numbers are only available above the threshold that the user configures. Possible solutions:

• Only plot the data above the threshold. Probably the cleanest solution, because it scales with the threshold and the user can opt in for 0% threshold to get the full graph.
• Fill the rest of the graph with some sort of mock data for the remaining files below the threshold.
• Use trigram data below the threshold. even though it's inaccurate and usually overlaps with the accurate data.

Add an option to find all files in the repository that are similar to a given, single file.

Add an option where the user can specify regular expressions. The tool then skips all lines in files that match the regular expressions.

This is useful for ignoring code comments or other such boilerplate that might otherwise skew the statistics.

Before comparing the file content using the Levenshtein or Jaro distance, first compare the two files using word-level trigrams to get the general sense of their similarity. Then, use the distance metric only on files that are relatively similar by trigrams.

Resources:

• https://en.wikipedia.org/wiki/Trigram_search
• https://en.wikipedia.org/wiki/Trigram