microsoft / near-duplicate-code-detector

A simple tool for detecting near-duplicate source code

Near-Duplicate Code Detector

This cross-platform sample tool detects exact and near duplicates of code maintained by the Deep Program Understanding group in Microsoft Research, Cambridge, UK. It has been created for the purpose of deduplicating code corpora for research purposes.

Requirements: .NET Core 2.1 or higher. For parsing code, an appropriate runtime for each of the languages that needs to be tokenized is also required.

To run the near-duplicate detection run:

$ dotnet run /path/to/DuplicateCodeDetector.csproj [options] --dir=<folder> <output-file-prefix>

This will use all the .gz files in the <folder> and output an <output-file-prefix>.json with the groups of detected duplicates. Invoke --help for more options.

Input Data

The input data should be one or more .jsonl.gz files. These are compressed JSONL files where each line has a single JSON entry of the form

{
    "filename": "unique identifier of file, such as a path or a unique id",
    "tokens" : ["list", "of", "tokens", "in", "file"]
}

Alternative formats can be accepted by providing the --tokens-field and --id-fields options.

The tokenizers folder in this repository contains tokenizers for C#,F#, Java, JavaScript and Python. Please, feel free to contribute tokenizers for other languages too.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

near-duplicate-code-detector's Issues

[Sloved] Run script issue

Environment: Windows 10

I tried to run this srcipt $ dotnet run /path/to/DuplicateCodeDetector.csproj [options] --dir=<folder> <output-file-prefix> but failed and it showed a hint like Usage: ...CloneDetectorCli.... It may be caused by the difference of dotnet versions.

If you have the same problem, try to add --project before name of project, like $ dotnet run --project /path/to/DuplicateCodeDetector.csproj [options] --dir=<folder> <output-file-prefix> and you can sucessfully run this detector program. : -)

ArgumentOutOfRangeException

For anyone trying this tool out and encountering similar issue.

Problem:
While testing on a very small scale, I encounterent ArgumentOutOfRangeException in CloneGorups.cs:line 47.

Cause:
If there are 2 entries in duplicationFactors variable, meaning there are only 2 unique clone clusters, then the midpoint becomes 1. Then the duplicationFactors[midpoint + 1] is causing an index out of range exception.

Solution:
Use on bigger projects and assume that the duplicationFactors includes more entries or add a special case to the code where duplicationFactors.Count == 2.

Error log:

Unhandled Exception: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at NearCloneDetector.CloneGroups..ctor(IEnumerable`1 clonePairs) in /near-duplicate-code-detector/DuplicateCodeDetector/CloneGroups.cs:line 47
   at NearCloneDetector.CloneDetectorCli.DetectClones(IDictionary`2 arguments) in /near-duplicate-code-detector/DuplicateCodeDetector/CloneDetectorCli.cs:line 68
   at NearCloneDetector.CloneDetectorCli.Main(String[] args) in /near-duplicate-code-detector/DuplicateCodeDetector/CloneDetectorCli.cs:line 30

Non deterministic output

Running the solution with the same tokens jsonl file, the output json of clone groups varies upon each execution.
Is there a more deterministic approach to near duplicate detection?

Cheers
Sky

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request