Unpicking and sorting out TBs of Photos/Videos

Hi, given the state of the world I've decided to try do a virtual spring clean of a myriad of my external hard drives from years of family photos.

I've about 10Tb of files, of which I think at least 3Tb are backups.

I've moved everything to a new huge RAID and tried to move everything into a central folder structure and am trying to use Hazel to do the heavy lifting.

Desired outcome is the following structure:

/<externaldrive>/Photos/YEAR/YYYY-MM-DD/
/<externaldrive>/Home Movies/YEAR/YYYY-MM-DD/<DeviceModel>

With a view to importing the photos into CaptureOne/Lightroom per year and the videos into FCPX libraries per year.

I've run a few rules as described below and then a summary of the issues I've faced so far. If anyone has any tips to help this process, I would be immensely grateful!

Photos Strategy

1)Remove empty folders

IF:
Kind is folder
Sub-file/folder Count is 0

DO:
Move to trash

2) Sort photos by date taken

IF:
Kind is Image or Extension is NEF

DO:
Move to folder: SortedPhotos
Sort into subfolder with pattern: Date taken (hazel) YYYY > Date taken (hazel) YYYY-MM-DD

3)Go into subfolders:

IF: Kind is Folder
DO: Run rules on folder contents

Throw away Duplicate files.

Video Strategy

1)Remove empty folders

IF:
Kind is folder
Sub-file/folder Count is 0

DO:
Move to trash

2) Sort videos by date taken

IF:
Kind is Movie

DO:
Move to folder: SortedPhotos
Sort into subfolder with pattern: Date taken (hazel) YYYY > Date taken (hazel) YYYY-MM-DD > device Model

3)Go into subfolders:

IF: Kind is Folder
DO: Run rules on folder contents

Throw away Duplicate files.

1) Many files do not have exif info, as I did a dump down from FLICKR and somehow many of them lost their date taken.
2) Files with the same file name have been sent to the trash during this process, even though they are completely different files
3) Many files with different names, but that are duplicates -1, -2, (1), remain.

To help with 1, I wrote another rule per 'unsorted' files like ones that ended up in 0001-12-31 > 0001, I rewrote the move rule to use "Date Created" which at least helps group similar ones together to manually try and sort.

I tried using tools like Duplicate Detective 2 to remove a bunch of these post-hazel to get rid of the -1 -2 files, but their auto select always wants to pick to keep the -1 or -2 files, and delete the 'original' which irks my OCD somewhat, since they have the same date-time stamp (I've contacted their support on that but guessing reply will be "It doesn't do that, sorry"). I thought the "Throw away Duplicate Files" rule in Hazel would solve this but I can't get it working.

Anyone got hazel rules ninja skills enough to help me combine my above rules together so that in one go, it will:

1) Remove exact duplicates with the same name, or -1, -2, (1)
2) Whilst also sorting into a folder based on date taken, as long as the date taken exists, and if it doesn't, do it based on "Date Created"

please?

Many thanks in advance!

Is there a reason why you have it going into subfolders? That seems a bit dangerous as it will refile the images into subfolders again and again. Note that the Sort into subfolders action has an option to remove duplicates at the destination.

Mr_Noodle wrote:Is there a reason why you have it going into subfolders? That seems a bit dangerous as it will refile the images into subfolders again and again. Note that the Sort into subfolders action has an option to remove duplicates at the destination.

Most of the source files are in subfolder after subfolder, I did out a limit on the initial sort to stop it going too deep (I think 3 layers was as deep as it needed to go).

Most are now in Photos/yyyy/yyyy-mm-dd/ structure but I’ve still got loads that are in /0001/0001-12-31/ and need resorting from there. Others have the shot taken in the name in there so need to some how add date pattern matching if can’t find date taken info, just struggling to work out how to write IF, ELSEIF, ELSEIF, ELSE type logic into the rules. Is it just a case of the list sequence of the rule? First rule = if, second is ELSEIF, last is ELSE?

I’ve got all still on my original drives, so could start from scratch.

How would you approach this task differently? (Other than filing neatly originally!)

Thanks for the reply.

You want to make sure the rule doesn't match the folders as it can end up filing those again into subfolders.

If a rule doesn't match it goes to the next so that is basically the if/else type construct.