Add remove-duplicates script.

Author: bamos

README.md

@@ -65,22 +65,6 @@ and inserted directly into the README as markdown.
 
 
 
-## [python2.7/music-organizer.py](https://github.com/bamos/python-scripts/blob/master/python2.7/music-organizer.py)
-+ Authors: [Brandon Amos](http://bamos.github.io)
-+ Created: 2014.04.19
-
-
-This script (music-organizer.py) organizes my music collection for
-iTunes and [mpv](http://mpv.io) using tag information.
-The directory structure is `<artist>/<track>`, where `<artist>` and `<track>`
-are lower case strings separated by dashes.
-
-See my blog post
-[Using Python to organize a music directory](http://bamos.github.io/2014/07/05/music-organizer/)
-for a more detailed overview of this script.
-
-
-
 ## [python2.7/mt.py](https://github.com/bamos/python-scripts/blob/master/python2.7/mt.py)
 + Authors: [Brandon Amos](http://bamos.github.io)
 + Created: 2014.11.30
@@ -93,78 +77,39 @@ of the output.
 
 
 
-## [python3/github-repo-summary.py](https://github.com/bamos/python-scripts/blob/master/python3/github-repo-summary.py)
-+ Authors: [Brandon Amos](http://bamos.github.io)
-+ Created: 2014.11.02
-
-
-Produces a Markdown table concisely summarizing a list of GitHub repositories.
-
-
-
-## [python3/link-checker.py](https://github.com/bamos/python-scripts/blob/master/python3/link-checker.py)
+## [python2.7/music-organizer.py](https://github.com/bamos/python-scripts/blob/master/python2.7/music-organizer.py)
 + Authors: [Brandon Amos](http://bamos.github.io)
-+ Created: 2014.02.06
-
-
-Script to be run by crontab to report broken links.
-
-Builds upon linkchecker (Ubuntu: sudo apt-get install linkchecker)
-to hide warnings and to send a concise email if bad links are found.
-
-![Link checker screenshot](https://raw.githubusercontent.com/bamos/python-scripts/master/link-checker-screenshot.png?raw=true)
-
++ Created: 2014.04.19
 
 
-## [python3/phonetic.py](https://github.com/bamos/python-scripts/blob/master/python3/phonetic.py)
-+ Authors: [Brandon Amos](http://bamos.github.io)
-+ Created: 2014.02.14
-
+This script (music-organizer.py) organizes my music collection for
+iTunes and [mpv](http://mpv.io) using tag information.
+The directory structure is `<artist>/<track>`, where `<artist>` and `<track>`
+are lower case strings separated by dashes.
 
-Obtain the NATO phonetic alphabet representation from short phrases.
+See my blog post
+[Using Python to organize a music directory](http://bamos.github.io/2014/07/05/music-organizer/)
+for a more detailed overview of this script.
 
-```
-$ phonetic.py github
-g - golf
-i - india
-t - tango
-h - hotel
-u - uniform
-b - bravo
-```
 
 
+## [python3/eval-expr.py](https://github.com/bamos/python-scripts/blob/master/python3/eval-expr.py)
++ Authors: J. Sebastian, [Brandon Amos](http://bamos.github.io)
++ Created: 2013.08.01
 
-## [python3/rank-writing.py](https://github.com/bamos/python-scripts/blob/master/python3/rank-writing.py)
-+ Authors: [Brandon Amos](http://bamos.github.io)
-+ Created: 2014.02.14
 
+A module to evaluate a mathematical expression using Python's AST.
 
-`rank-writing.py` ranks the writing quality of my
-blog's Markdown posts and my project's Markdown README files.
++ Original by: J. Sebastian at http://stackoverflow.com/questions/2371436.
++ Modifications by: [Brandon Amos](http://bamos.github.io).
 
-The following programs should be on your `PATH`:
-+ [aspell](http://aspell.net/)
-+ [write-good](https://github.com/btford/write-good)
-+ [diction](https://www.gnu.org/software/diction/)
+If you want a command-line expression evaluator, use
+[Russell91/pythonpy](https://github.com/Russell91/pythonpy).
 
 
 ```
-$ rank-writing.py *.md
-
-=== 2013-05-03-scraping-tables-python.md ===
-Total: 53
-├── aspell: 34
-├── diction: 0
-└── write-good: 19
-
-...
-
-=== 2013-04-16-pdf-from-plaintext.md ===
-Total: 0
-├── aspell: 0
-├── diction: 0
-└── write-good: 0
+$ eval-expr.py '(((4+6)*10)<<2)'
+(((4+6)*10)<<2) = 400
 ```
 
 
@@ -182,10 +127,10 @@ delete the current wallpaper.
 
 
 ### Warning
-+ This approach doesn't work with multiple monitors.
++ This approach doesn't work with multiple monitors or virtual desktops.
 
 ### Tested On
-+ OSX Yosemite 10.10.2 with a single monitor on a MBP.
++ OSX Yosemite 10.10.2 with a single desktop on a MBP.
 
 ### Usage
 Ensure `db_path` and `wallpaper_dir` are correctly set below.
@@ -220,6 +165,38 @@ alias rm-wallpaper='rm $(get-osx-wallpaper.py) && killall Dock'
 
 
 
+## [python3/github-repo-summary.py](https://github.com/bamos/python-scripts/blob/master/python3/github-repo-summary.py)
++ Authors: [Brandon Amos](http://bamos.github.io)
++ Created: 2014.11.02
+
+
+Produces a Markdown table concisely summarizing a list of GitHub repositories.
+
+
+
+## [python3/link-checker.py](https://github.com/bamos/python-scripts/blob/master/python3/link-checker.py)
++ Authors: [Brandon Amos](http://bamos.github.io)
++ Created: 2014.02.06
+
+
+Script to be run by crontab to report broken links.
+
+Builds upon linkchecker (Ubuntu: sudo apt-get install linkchecker)
+to hide warnings and to send a concise email if bad links are found.
+
+![Link checker screenshot](https://raw.githubusercontent.com/bamos/python-scripts/master/link-checker-screenshot.png?raw=true)
+
+
+
+## [python3/merge-mutt-contacts.py](https://github.com/bamos/python-scripts/blob/master/python3/merge-mutt-contacts.py)
++ Authors: [Brandon Amos](http://bamos.github.io)
++ Created: 2014.01.08
+
+
+Merges two mutt contact files.
+
+
+
 ## [python3/merge-pdfs-printable.py](https://github.com/bamos/python-scripts/blob/master/python3/merge-pdfs-printable.py)
 + Authors: [Brandon Amos](http://bamos.github.io)
 + Created: 2014.10.17
@@ -257,6 +234,70 @@ PS file.
 
 
 
+## [python3/phonetic.py](https://github.com/bamos/python-scripts/blob/master/python3/phonetic.py)
++ Authors: [Brandon Amos](http://bamos.github.io)
++ Created: 2014.02.14
+
+
+Obtain the NATO phonetic alphabet representation from short phrases.
+
+```
+$ phonetic.py github
+g - golf
+i - india
+t - tango
+h - hotel
+u - uniform
+b - bravo
+```
+
+
+
+## [python3/rank-writing.py](https://github.com/bamos/python-scripts/blob/master/python3/rank-writing.py)
++ Authors: [Brandon Amos](http://bamos.github.io)
++ Created: 2014.02.14
+
+
+`rank-writing.py` ranks the writing quality of my
+blog's Markdown posts and my project's Markdown README files.
+
+The following programs should be on your `PATH`:
++ [aspell](http://aspell.net/)
++ [write-good](https://github.com/btford/write-good)
++ [diction](https://www.gnu.org/software/diction/)
+
+
+```
+$ rank-writing.py *.md
+
+=== 2013-05-03-scraping-tables-python.md ===
+Total: 53
+├── aspell: 34
+├── diction: 0
+└── write-good: 19
+
+...
+
+=== 2013-04-16-pdf-from-plaintext.md ===
+Total: 0
+├── aspell: 0
+├── diction: 0
+└── write-good: 0
+```
+
+
+
+## [python3/remove-duplicates.py](https://github.com/bamos/python-scripts/blob/master/python3/remove-duplicates.py)
++ Authors: [Brandon Amos](http://bamos.github.io)
++ Created: 2015.06.06
+
+
+Detect and remove duplicate images using average hashing.
+
+http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html
+
+
+
 ## [python3/word-counter.py](https://github.com/bamos/python-scripts/blob/master/python3/word-counter.py)
 + Authors: [Brandon Amos](http://bamos.github.io)
 + Created: 2014.11.7
@@ -288,45 +329,15 @@ $ word-counter.py shakespeare.md --numWords 4 --maxTuples 3
 
 
 
-## [python3/eval-expr.py](https://github.com/bamos/python-scripts/blob/master/python3/eval-expr.py)
-+ Authors: J. Sebastian, [Brandon Amos](http://bamos.github.io)
-+ Created: 2013.08.01
-
-
-A module to evaluate a mathematical expression using Python's AST.
-
-+ Original by: J. Sebastian at http://stackoverflow.com/questions/2371436.
-+ Modifications by: [Brandon Amos](http://bamos.github.io).
-
-If you want a command-line expression evaluator, use
-[Russell91/pythonpy](https://github.com/Russell91/pythonpy).
-
-
-```
-$ eval-expr.py '(((4+6)*10)<<2)'
-(((4+6)*10)<<2) = 400
-```
-
-
-
-## [python3/merge-mutt-contacts.py](https://github.com/bamos/python-scripts/blob/master/python3/merge-mutt-contacts.py)
-+ Authors: [Brandon Amos](http://bamos.github.io)
-+ Created: 2014.01.08
-
-
-Merges two mutt contact files.
-
-
-
 # Similar Projects
 There are many potpourri Python script repositories on GitHub.
 The following list shows a short sampling of projects,
 and I'm happy to merge pull requests of other projects.
 
 Name | Stargazers | Description
 ----|----|----
-[averagesecurityguy/Python-Examples](https://github.com/averagesecurityguy/Python-Examples) | 18 | Example scripts for common python tasks
+[averagesecurityguy/Python-Examples](https://github.com/averagesecurityguy/Python-Examples) | 20 | Example scripts for common python tasks
 [ClarkGoble/Scripts](https://github.com/ClarkGoble/Scripts) | 26 | My scripts - primarily using python and appscript
-[computermacgyver/twitter-python](https://github.com/computermacgyver/twitter-python) | 40 | Simple example scripts for Twitter data collection with Tweepy in Python
-[gpambrozio/PythonScripts](https://github.com/gpambrozio/PythonScripts) | 40 | A bunch of Python scripts I made and that might interest somebody else
-[realpython/python-scripts](https://github.com/realpython/python-scripts) | 56 | because i'm tired of gists
+[computermacgyver/twitter-python](https://github.com/computermacgyver/twitter-python) | 45 | Simple example scripts for Twitter data collection with Tweepy in Python
+[gpambrozio/PythonScripts](https://github.com/gpambrozio/PythonScripts) | 39 | A bunch of Python scripts I made and that might interest somebody else
+[realpython/python-scripts](https://github.com/realpython/python-scripts) | 59 | because i'm tired of gists

python3/remove-duplicates.py

@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+
+__author__ = ['[Brandon Amos](http://bamos.github.io)']
+__date__ = '2015.06.06'
+
+"""
+Detect and remove duplicate images using average hashing.
+
+http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html
+"""
+
+import argparse
+import imagehash
+import os
+import sys
+
+from collections import defaultdict
+from PIL import Image
+
+
+def getImgs(d):
+    """Get the images from the test directory, partitioned by class."""
+    exts = ["jpg", "png"]
+
+    imgClasses = []  # Images, separated by class.
+    for subdir, dirs, files in os.walk(d):
+        imgs = []
+        for fName in files:
+            (imageClass, imageName) = (os.path.basename(subdir), fName)
+            if any(imageName.lower().endswith("." + ext) for ext in exts):
+                imgs.append(os.path.join(subdir, fName))
+        imgClasses.append(imgs)
+    return imgClasses
+
+
+def getHash(imgPath):
+    """Get the hash of an image, and catch exceptions if the image
+    file is corrupted."""
+    try:
+        return imagehash.average_hash(Image.open(imgPath))
+    except:
+        return None
+
+
+def runOnClass(args, imgs):
+    """Find and remove duplicates within an image class."""
+    d = defaultdict(list)
+    for imgPath in imgs:
+        imgHash = getHash(imgPath)
+        if imgHash:
+            d[imgHash].append(imgPath)
+
+    numFound = 0
+    for imgHash, imgs in d.items():
+        if len(imgs) > 1:
+            print("{}: {}".format(imgHash, " ".join(imgs)))
+            numFound += len(imgs) - 1  # Keep a single image.
+
+            if args.delete:
+                largestImg = max(imgs, key=os.path.getsize)
+                print("Keeping {}.".format(largestImg))
+                imgs.remove(largestImg)
+                for img in imgs:
+                    os.remove(img)
+    return numFound
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('inplaceDir', type=str,
+                        help="Directory of images, divided into "
+                        "subdirectories by class.")
+    parser.add_argument('--delete', action='store_true',
+                        help="Delete the smallest duplicate images instead "
+                        "of just listing them.")
+    args = parser.parse_args()
+
+    numFound = 0
+    for imgClass in getImgs(args.inplaceDir):
+        numFound += runOnClass(args, imgClass)
+    print("\n\nFound {} total duplicate images.".format(numFound))

requirements-3.txt

@@ -2,3 +2,4 @@ Jinja2==2.7.2
 PyGithub==1.25.2
 PyPDF2==1.23
 toolz==0.7.1
+imagehash==0.3