Topic: deduplication Goto Github

👇 Here are 407 public repositories matching this topic...

alephdata / fingerprints
deduplication,Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
Organization: alephdata
company-data names fingerprint entity clustering nlp deduplication

arsenetar / dupeguru
deduplication,Find duplicate files
User: arsenetar
Home Page: https://dupeguru.voltaicideas.net
python deduplication

borgbackup / borg
deduplication,Deduplicating archiver with compression and authenticated encryption.
Organization: borgbackup
Home Page: https://www.borgbackup.org/
python compression ssh deduplication backup borgbackup encryption

borgmatic-collective / borgmatic
deduplication,Simple, configuration-driven backup software for servers and workstations
Organization: borgmatic-collective
Home Page: https://torsion.org/borgmatic/
python backup borg borgbackup servers deduplication compression borgbase cronhub cronitor

cargo-limit / cargo-limit
deduplication,Productivity improvements for Rust ecosystem: warnings are skipped until errors are fixed, LSP-independent Neovim integration, etc.
Organization: cargo-limit
rust wrapper filter limit cargo cargo-wrapper deduplication crates neovim nvim

cupcakearmy / autorestic
deduplication,Config driven, easy backup cli for restic.
User: cupcakearmy
Home Page: https://autorestic.vercel.app/
restic backup cli config pruning incremental-backup incremental deduplication config-driven

deajan / backup-bench
deduplication,Quick and dirty backup tool benchmark with reproducible results
User: deajan
backup benchmark benchmarking borgbackup compression deduplication duplicacy kopia restic bupstash

dm-vdo / kvdo
deduplication,A kernel module which provide a pool of deduplicated and/or compressed block storage.
Organization: dm-vdo
kernel-modules storage vdo linux-kernel deduplication compression

dm-vdo / vdo
deduplication,Userspace tools for managing VDO volumes.
Organization: dm-vdo
compression deduplication storage vdo

dpc / rdedup
deduplication,Data deduplication engine, supporting optional compression and public key encryption.
User: dpc
data-deduplication backup encryption deduplication

elemental-lf / benji
deduplication,Benji Backup: A block based deduplicating backup software for Ceph RBD images, iSCSI targets, image files and block devices
User: elemental-lf
Home Page: https://benji-backup.me
backup deduplication ceph lvm block-based s3 b2 kubernetes iscsi

f483 / dejavu
deduplication,Quickly detect already witnessed data.
User: f483
Home Page: https://f483.github.io
command-line command-line-tool deduplication duplicate-values duplicates go golang history memory probabilistic

fake-name / intraarchivededuplicator
deduplication,Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
User: fake-name
postgresql image-search deduplication bk-tree python cython

fcorbelli / zpaqfranz
deduplication,Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
User: fcorbelli
zpaq compression deduplication backup solaris

google / unisim
deduplication,UniSim is a package for efficient similarity computation, fuzzy matching, and clustering of data.
Organization: google
deduplication fuzzy-matching nlp onnx similarity tensorflow2 vector-search

ibm / data-prep-kit
deduplication,Open source project for data preparation of LLM application builders
Organization: ibm
Home Page: https://ibm.github.io/data-prep-kit/
data-preparation finetuning llm llmapps data data-prep data-preprocessing data-preprocessing-pipelines datacuration large-language-models

j535d165 / data-matching-software
deduplication,A list of free data matching and record linkage software.
User: j535d165
data-matching record-linkage entity-resolution deduplication fuzzy-matching software awesome awesome-list machine-learning open-source

j535d165 / recordlinkage
deduplication,A powerful and modular toolkit for record linkage and duplicate detection in Python
User: j535d165
Home Page: http://recordlinkage.readthedocs.io/
record-linkage entity-resolution dedupe string-distance machine-learning privacy python python-library data-matching deduplication

jaskey / rocketmqdeduplistener
deduplication,RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用
User: jaskey
rocketmq rocketmq-client deduplication

jvirkki / dupd
deduplication,CLI utility to find duplicate files
User: jvirkki
Home Page: http://www.virkki.com/dupd
c deduplication duplicate-files duplicatefilefinder duplicates fdupes

kdeldycke / mail-deduplicate
deduplication,📧 CLI to deduplicate mails from mail boxes
User: kdeldycke
Home Page: https://kdeldycke.github.io/mail-deduplicate
python mail dedupe cli maildir mailbox mbox deduplication cleanup babyl mh mmdf email

kopia / kopia
deduplication,Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Organization: kopia
Home Page: https://kopia.io
deduplication backup google-cloud-storage encryption cloud hacktoberfest

lakshmipathi / dduper
deduplication,Fast block-level out-of-band BTRFS deduplication tool.
User: lakshmipathi
btrfs btrfs-progs btrfs-tools deduplication dedupe

lostatc / acid-store
deduplication,[UNMAINTAINED] A transactional and deduplicating virtual file system
User: lostatc
deduplication encryption acid redis s3 sqlite filesystem sftp rclone fuse

markusressel / py-image-dedup
deduplication,CLI utility to find near duplicate images and remove all but the best copy.
User: markusressel
python python3 python-3 image-analysis duplicate-detection duplicate-images deduplication dedup find-duplicates image-comparison

mattilyra / lsh
deduplication,Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
User: mattilyra
cython deduplication duplicate-documents lsh minhash

mhx / dwarfs
deduplication,A fast high compression read-only file system for Linux, Windows and macOS
User: mhx
gpl-license filesystem squashfs compression fuse-filesystem cpp lzma zstd lrzip zpaq

moj-analytical-services / splink
deduplication,Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
Organization: moj-analytical-services
Home Page: https://moj-analytical-services.github.io/splink/
record-linkage spark em-algorithm deduplication deduplicate-data entity-resolution data-matching fuzzy-matching data-science duckdb

netinvent / npbackup
deduplication,A secure and efficient file backup solution that fits both system administrators (CLI) and end users (GUI)
Organization: netinvent
backup cli deduplication gui restic vss compression healthcheck monitoring prometheus-metrics

nlfiedler / fastcdc-rs
deduplication,FastCDC implementation in Rust
User: nlfiedler
Home Page: https://crates.io/crates/fastcdc
rust deduplication chunking-algorithm

nvidia / nemo-curator
deduplication,Scalable data pre processing and curation toolkit for LLMs
Organization: nvidia
data-curation llm data data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes

opengene / gencore
deduplication,Generate duplex/single consensus reads to reduce sequencing noises and remove duplications
Organization: opengene
sequencing ngs bioinformatics sequencing-error consensus deep-sequencing sequencing-noise duplication duplex deduplication

opensanctions / nomenklatura
deduplication,Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources
Organization: opensanctions
data-integration deduplication record-link

openvenues / libpostal
deduplication,A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
Organization: openvenues
address-parser machine-learning nlp address international c deduplication record-linkage deduping natural-language-processing

pjdude / dude
deduplication,Duplicates Detector is a cross-platform GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for efficient and convenient operation.
User: pjdude
Home Page: https://pjdude.github.io/dude/
duplicate-detection duplicate-files duplicates duplicates-removal python tkinter deduplication sha1 utility utility-application python3 gui gui-application threads easy easy-to-use easyui duplicate cli

prometheus / alertmanager
deduplication,Prometheus Alertmanager
Organization: prometheus
Home Page: https://prometheus.io
monitoring alertmanager pagerduty email notifications opsgenie slack deduplication hacktoberfest

restic / restic
deduplication,Fast, secure, efficient backup program
Organization: restic
Home Page: https://restic.net
go restic backup deduplication dedupe secure-by-default

rin-nas / postgresql-patterns-library
deduplication,Коллекция готовых SQL запросов для PostgreSQL по часто возникающим задачам (получение и модификация данных, ускорение запросов, обслуживание БД)
User: rin-nas
sql postgresql recipes phone email validation parse syntax check function

ropeladder / record-linkage-resources
deduplication,Resources for tackling record linkage / deduplication / data matching problems
User: ropeladder
data-matching deduplication entity-resolution java javascript python record-linkage

rustic-rs / rustic
deduplication,rustic - fast, encrypted, and deduplicated backups powered by Rust
Organization: rustic-rs
Home Page: https://rustic.cli.rs
backup deduplication encryption restic rust

sahib / rmlint
deduplication,Extremely fast tool to remove duplicates and other lint from your filesystem
User: sahib
Home Page: http://rmlint.rtfd.org
c python filesystem lint duplicates fdupes deduplication

sreedevk / deduplicator
deduplication,Filter, Sort & Delete Duplicate Files Recursively
User: sreedevk
deduplication duplicate-detection duplicate-files duplicatefilefinder filesystem rust

tsileo / blobstash
deduplication,You personal database. Mirror of https://git.sr.ht/~tsileo/blobstash
User: tsileo
blob-store backup content-addressed document-store go blobstash deduplication storage

unreadablewxy / fs-curator
deduplication,Automation for the serious data hoarder that wants to have their data and use it
User: unreadablewxy
file-sorting file-renamer hard-links organizer directory-tree deduplication

usc-isi-i2 / rltk
deduplication,Record Linkage ToolKit (Find and link entities)
Organization: usc-isi-i2
linkage similarity similarity-metric string-similarity record-linkage entity-resolution deduplication

vintasoftware / entity-embed
deduplication,PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
Organization: vintasoftware
Home Page: https://entity-embed.readthedocs.io/en/latest/
entity-resolution record-linkage representation-learning embeddings approximate-nearest-neighbors deduplication entity-matching data-matching deep-learning python

yomguithereal / talisman
deduplication,Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
User: yomguithereal
Home Page: https://yomguithereal.github.io/talisman/
natural-language-processing machine-learning fuzzy-matching clustering record-linkage information-retrieval deduplication

yornaath / batshit
deduplication,A batch manager that will deduplicate and batch requests for a certain data type made within a window. Useful to batch requests made from multiple react components that uses react-query
User: yornaath
Home Page: https://batshit-example.vercel.app/
async batch-processing concurrency deduplication fetch react-query typescript react tanstack

zinggai / zingg
deduplication,Scalable identity resolution, entity resolution, data mastering and deduplication using ML
Organization: zinggai
fuzzymatch fuzzy-matching deduplication dedupe masterdata dataengineering data-transformation analytics-engineering entity-resolution identity-resolution

zouzias / spark-lucenerdd
deduplication,Spark RDD with Lucene's query and entity linkage capabilities
User: zouzias
spark lucene rdd spatial-search record-linkage deduplication entity-linking linkage hacktoberfest