Remove duplicate images from the two given directories recursively
Too many images that are the same but might have different dates and other metadata. This tool compares images based on the following criteria:
- SHA-256 hash of the file, since
- File size in bytes, will be in
- Width and height, will be in
- Bit depth, will be in
- Number of unique colors, will be in
- Image contents comparison, will be in
The idea is to start from stronger method, while progressing towards a weaker method.
Underneath SQLite is used for storing any meta information about the files, as sometimes the amount of files to compare is huge.
It will be possible to reuse the resulting database file and skip reading the files, in the
Reading the information from image file is the most time consuming part within the execution of this tool.
Starting with version
0.3.0, additional tools are needed.
brew install graphicsmagick
In Ubuntu it is usually available via:
sudo apt-get install graphicsmagick
Windows users could download an installer package.
Now install the command line tool globally, which might need increased privileges:
[sudo] npm install --global image-duplicate-remover
Please note that while in Linux and with
sudo, some of the dependencies might fail to install,
which can be fixed in some case by
sudo npm install --global --unsafe-perm image-duplicate-remover.
See more details at docs.npmjs.com.
Command line options
The available command line options can be seen via command:
The output will be something similar to:
image-duplicate-remover [options] <primary directory> <secondary directory>-h, --help Help and usage instructions-V, --version Version number-v, --verbose Verbose output, will print which file is currently being processed-D, --database String SQLite database to use-S, --skip-reading Skip reading the directories, just use the existing database. Requires database-n, --dry-run Try it out without actually removing anythingVersion 0.2.0
The following command shows how two folders are compared, but nothing will be removed since the
--dry-run option is used.
image-duplicate-remover -vn a b
First thing to do is to file an issue.
Linting is done with ESLint and can be executed with
npm run lint.
npm installnpm run lint
Unit tests are written with
tape and can be executed with
Code coverage is inspected with
can be executed with
npm run coverage after running
Please make sure it is over 90% at all times.
- Update dependencies to avoid any security issues
- Update tested GraphicsMagick to version
npm-shrinkwrap.jsonfor locking the working set of 3rd party dependencies
- Previous migration from
better-sqlite3was not properly tested #25
- Renamed the column
hashin the database table
- Switched from using
- Switched from using
- Enable the use of existing SQLite database
filesizeto find matches too, when
sha256was not a match
- Minimum Node.js version lifted from
- Gets the job simply done with SHA-256 :neckbeard: checking, hence first release. Released 6 months late.
Licensed under the MIT license.