List of duplicate file finders

Python

ActiveState Recipe - a minimal python command line tool that only detects duplicates

C

duff - a Unix command-line utility for quickly finding duplicates in a given set of files

C

{{?}}

{{?}}

dupedit - Compares many files at once without checksumming. Avoids comparing files against themselves when multiple paths point to the same file.

C++

Duplicate Files Finder - GUI Application for Windows and Linux. Project site.

C

{{?}}

{{?}}

{{?}}

dupmerge - POSIX C compliant and runs on various platforms (Win32/64 with Cygwin, *nix, Linux etc.)

Python

fastdupes fast and small python command line tool to find duplicates

Perl

{{?}}

{{?}}

fdupe - a small script written in Perl. Doing its job fast and efficiently.

Java

findrepe - free Java-based command-line tool designed for an efficient search of duplicate files, it can search within zips and jars.(GNU/Linux, Mac OS X, *nix, Windows)

Perl

{{?}}

{{?}}

{{?}}

{{?}}

freedups - Perl script that hardlinks duplicates to save space, caches file checksums.

Python

{{?}}

{{?}}

hardlinkpy - A tool to hardlink together identical files in order to save space. It is a complete rewrite and improvement over the original hardlink.c code (which was written by: Jakub Jelinek <jakub@redhat.com>). Performance is orders of magnitude faster than hardlink.c due to a more efficient algorithm.

Python

liten2 - A rewrite of the original Liten, still a command line tool but with a faster interactive mode using SHA-1 checksums (Linux, Mac OS X, *nix)

C++

{{?}}

rdfind - One of the few which rank duplicates based on the order of input parameters (directories to scan) in order not to delete in "original/well known" sources (if multiple directories are given). Uses MD5 or SHA1.

C, Perl, SH

{{?}}

{{?}}

repeats - C and SH, from littleutils. File sizes, then partial-read hashes, then full-read hashes, then (optionally) byte-for-byte comparisons. Highly efficient. (Linux, *nix, Cygwin)

C

rmlint - Fast finder with command line interface and many options to find other lint too (uses MD5), claims to be better than rdfind and fdupes.

Java

{{?}}

DFS - search by content / size / name