Noodlesoft Forums

After reading the "My Perfect Photo Management Process using Dropbox & Flicker" I thought I would post my own experience with Dropbox photo management and Hazel.

In 2012, I read Federico Viticci's Moving from iPhoto to Dropbox and immediately liked his ideas. I subscribed to a 100GB Dropbox account, purchased Hazel and began to implement his methods. I encountered problems along the way, had to do a lot of research and learn a LOT about Phil Harvey's Exiftool application. I wanted to share my research and experience both to have a record that I can reference and to maybe try to assist others.

I will admit that many will not like my file naming scheme as it includes "undesirable" characters (spaces, dashes, quotes and especially the slash). I have a decent understanding of quoting and escaping file names in a unix environment so it is my cross to bear.

I like my file names to convey as much info about the file as possible and I virtually copied Federico's entire process with a few tweaks to make it mine.

• I name my photos with: yyyy-mmdd-hhmm.ss[lat/lon]"camera model_camera filename".
  
• I go down to the second to capture photos taken in rapid succession or burst mode.
  
• I prefer the camera's original filename (usually IMG_####) to again preserve sequence in rapid-fire photos.
  
• In the future, with scripts I can easily isolate individual pieces of info about photos or groups of photos by using the delimiters in my naming scheme (lat/lon between brackets, filename between quotes, date/time with dashes, dots and a bracket).

- Filename example: 2014-0522-0011.09[+32.230/-112.211]"iPad_IMG_0845".jpg

I know the slash is not recommended but it makes the coordinates much more readable for me personally. The backslash could have been used as well, but I was worried about its "escaping" behavior.

Federico later recommended the Unbound app for IOS for viewing remotely stored photos which I also purchased. I adopted his folder hierarchy as well to make Unbound display content in chronological order. So I have individual folders for each year and within each year there is a folder for each month named as: YYYY - MM - MMMMMMMM.

- Folder example: 2014 - 05 - May

I initially used the Dropbox IOS app "Camera Upload" feature to upload all my photos. After several months, I decided that I did not like the naming scheme that Dropbox uses for newly imported photos. All uploaded photos are named with a date/time naming scheme of: yyyy-mm-dd hh.mm.ss
For example, Dropbox would have named the above example photo as: 2014-05-22 00.11.09.jpg

I find two downsides to this...

1. it works well as long as all of your photos are one second or more apart. With rapid photos or burst mode photos I began to discover sequence problems. It could also be related to when the file finished copying to the Dropbox server. The first file to finish uploading gets the base name then all the subsequent uploads get the -# at the end of the name. I had many burst photos of 3-5 photos out of order after Dropbox upload. I also suspect that Hazel was affecting this because of the order in which it received the files.
2. by renaming the files with their scheme Dropbox totally eliminates the camera's original name for the file. It no longer exists even in the exif data. So that opportunity for sequencing photos is gone.

As an upside, however, after deep examination of exif, stat, mdls and GetFileInfo outputs, I found that Dropbox accurately preserves the file creation information (date and time).

In a later post Federico recommended the CameraSync app for uploading to Dropbox. I really liked this
app for its much greater control for when, how and how much is uploaded to Dropbox and began to use it instead of the Dropbox app. One of my most-liked features of CameraSync is the "Use Original Filenames" which uploads all my files as IMG_####.

As mentioned in the article "the Dropbox api is not letting third-party developers set timestamps". As a result of this, all the filesystem creation/modification dates and times of the file are set to the moment that CameraSync uploads the file to dropbox. The actual creation date still exists in exif, but Hazel uses the filesystem timestamps for rules and sorting.

This wreaked havoc with my organization system! Federico's article did show how to paste a php script into Hazel to correct dates and times but since I don't know php and don't want to use code I don't understand, I looked for my own options.

After learning a great deal about exiftool I wrote two scripts to include in my Hazel rules. One is a simple bash script to "touch" each file with the creation date pulled from the exif data. The second is another bash script to rename the file using exiftool instead of Hazel.

As a final organizational note, unlike many posts I found on the internet I actually desire to have both my movies and screenshots incorporated right in to my photo gallery. The movies are for fairly obvious reasons. As far as the screenshots, when attending events I will often take screenshots of the compass or weather apps documenting where I was at the time. I will also take screenshots of internet information pertaining to the event. So I want screenshots in my gallery and can always go back and delete non-relevant shots.

My photo workflow is as follows...

1. Use CameraSync to upload to Dropbox "Camera Uploads" folder
2. Hazel watches the "Camera Uploads" folder and applies one of three rule sets based on whether the file is a photo, movie or screenshot.
3. Hazel's rules "touch" each file with a "CreateDate" pulled directly from the exif data of the file and then rename the file utilizing the exiftool command rather than Hazel. (this prevents any naming discrepancies that may be caused by upload or Hazel timing).
4. Finally, Hazel moves and sorts the file into the file structure of my Photo Gallery.

Here are the details...


For screenshots, movies and photos, the sequence of the rules is the same

1. shell script to set creation date in filesystem
2. set color label for screenshots and movies
3. shell script to rename file
4. move file to main photos folder
5. sort file into proper subfolder

For screenshots, movies and photos, the renaming shell script is identical

Code: Select all

exiftool -P -d "%Y-%m%d-%H%M.%S" -c "%+.3f" '-FileName<${FileModifyDate}[]"%f%-c".%e' '-FileName<${CreateDate}[:]"%f%-c".%e' '-FileName<${CreateDate}[:]"${Model}_%f%-c".%e' '-FileName<${CreateDate}[${GPSLatitude}:${GPSLongitude}]"${Model}_%f%-c".%e' "$1"

The renaming script is a cascade. Phil Harvey explained in his forums that the LAST VALID filename argument is the one used to name the file:

• Code: Select all

  exiftool -P -d "%Y-%m%d-%H%M.%S" -c "%+.3f"

  sets the date and GPS naming scheme.

• Code: Select all

  '-FileName<${FileModifyDate}[]"%f%-c".%e'

  if "FileModifyDate" exists in exif with no other information

• Code: Select all

  '-FileName<${CreateDate}[:]"%f%-c".%e'

  if the file ALSO has a "CreateDate" tag but no GPS information

• Code: Select all

  '-FileName<${CreateDate}[:]"${Model}_%f%-c".%e'

  if the file ALSO has "CreationDate" PLUS a camera model but still no GPS (airplane mode)

• Code: Select all

  '-FileName<${CreateDate}[${GPSLatitude}:${GPSLongitude}]"${Model}_%f%-c".%e'

  if the file ALSO has "CreationDate" PLUS a camera model and GPS info

I imagine it is possible to have CreateDate and GPS but NO camera model and should probably write that in as well in the future.

Note the colon between the brackets ([:]). The colon in an exiftool filename translates into a slash (/) in the Mac OS. I found that little tidbit direct from Phil Harvey somewhere deep in the exiftool forums. In a final file name it looks like [/] which indicates to me that exiftool found no GPS info.

The "touch" scripts are slightly different. Screenshots have almost no exif information while photos and movies have a convenient "CreatDate" in exif. For "touch"ing and naming screenshots I use the "FileModifyDate" exif tag.

I created a more simple touch script for screenshots.

Versus a slightly more complex version for movies and photos. I made this one more complex just in case I ever import from a camera that doesn't use a "CreateDate" tag.

The move to the PhotoGallery folder and subsequent sort into subfolder is again the same for screenshots, movies and photos. I discovered the trick of moving to a folder first and then having the sort rule from the Noodlesoft forums. This avoids the manual step of having to create a new year folder and month subfolders for each new year. If there is no "2015" folder in my PhotoGallery when Hazel tries to put a photo there next January, it will automatically be created as well as the "January" subfolder.

The final product looks like this...

It can be easily browsed via Finder on my Macs or with the Unbound app (or any other viewing apps) on my IOS devices in full chronological sequence.

I have not activated or used Dropbox's Carousel app as I am afraid it might try to make changes to something in my system. Comments?

I look forward to comments and criticisms that will help further refine my workflow.

This post was very helpful--thanks!

I'm aiming for a somewhat simpler file naming convention:
mm Month YYYY at hh:mm:ss PM (GMT -offset) - (latitude x longitude) - name.extension

It's working out well for me, but I'm encountering the problem you and Federico referenced where CameraSync is uploading the file to Dropbox but the date stamp is set to the upload date and not the original EXIF created date. This is highly problematic for my intended organizational structure, for much the same reason as you.

I've tried two things. First, I tried the PHP script that Federico included. I used a "Run shell script" action, pasted the PHP into the embedded script field, and it failed with an error running the script. (Generic error; nothing useful). So I thought maybe I'd have to run it through a .php file, so I put the code into a .php file, and changed the embedded script to:

php script.php

And absolutely nothing happened when it ran. So the second thing I tried was "slightly more complex" version of your touch script above. Again I used a Run shell script action, and pasted the following code into the embedded script field:

Code: Select all

#!/bin/bash
for f in "$@"
do
   if
      exiftool -s "$1" | grep -Fq "CreateDate"
   then
      year='exiftool -s3 -"CreateDate" "$1" -d %Y'
      month='exiftool -s3 -"CreateDate" "$1" -d %m'
      day='exiftool -s3 -"CreateDate" "$1" -d %d'
      hour='exiftool -s3 -"CreateDate" "$1" -d %H'
      minute='exiftool -s3 -"CreateDate" "$1" -d %M'
      second='exiftool -s3 -"CreateDate" "$1" -d %S'
   else
      year='exiftool -s3 -"FileModifyDate" "$1" -d %Y'
      month='exiftool -s3 -"FileModifyDate" "$1" -d %m'
      day='exiftool -s3 -"FileModifyDate" "$1" -d %d'
      hour='exiftool -s3 -"FileModifyDate" "$1" -d %H'
      minute='exiftool -s3 -"FileModifyDate" "$1" -d %M'
      second='exiftool -s3 -"FileModifyDate" "$1" -d %S'
   fi
   touch -t $year$month$day$hour$minute.$second "$1"
done


That, also, is generating the same error as before:

Error processing shell script on file file.jpg

Any ideas what I'm doing wrong?


dt

I'm not an expert on ANY of this but after looking at your script I have two thoughts.

First, in the Hazel shell script editor you must have the proper path to the shell at the top. This is the field labeled "Shell:" at the top of the editor drop down window.

For a bash script it needs to be /bin/bash (or wherever your bash shell is located)
For a php script it needs to be /usr/bin/php (or again, wherever your php is installed)

The second observation is that your variable definition lines appear to be using single quotes ( ' ) to encapsulate the definition statement. In my code I used the "tick" mark ( ` ) which on my keyboard is to the left of the number 1 and below the esc key.

You have this:

Code: Select all

year='exiftool -s3 -"CreateDate" "$1" -d %Y'

I think it needs to be this:

Code: Select all

year=`exiftool -s3 -"CreateDate" "$1" -d %Y`

Note the subtle difference between the "tick" and the "quote".

Hope this helps.

-k-

First of all, pooter your solution is brilliant, all my buddies seem to be impressed that I can access photos from the 90s in no time. Anyway, there was one problem for me that it took me a couple days to resolve so I thought I would post it here, if others face this problem.

Pooter's solution worked flawlessly for images, but some of my videos were labeled with an incorrect timestamp. After narrowing down the problem, I found it was just .MOV files I shot on my iPhones and it was just the hour off 6 hours ahead. It turns out for some bizarre reason Phil Harvey's ExifTool defaults to using the local timezone for most everything but QuickTime movies default to using UTC timezone.

All you have to do is put the code below in a text file say tempfix.txt in your home directory:

Code: Select all

# Specify default ExifTool option values
%Image::ExifTool::UserDefined::Options = (
    QuickTimeUTC => 1,  # assume QuickTime date/time values are in UTC
);

Then it needs to be renamed to .ExifTool_config
The only way to do this on a mac is to open a terminal and type in:

Code: Select all

mv ~/tempfix.txt ~/.ExifTool

Pooter's solution should then just magically work even for MOV files

http://owl.phy.queensu.ca/~phil/exiftool/config.html
http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,4178.msg19857.html#msg19857

@pooter - thanks for the detailed steps!

I have been researching for the last few days. Been very motivated to learn some bash scripting first, but decided that I need to get my photos mess sorted out before I can learn some Bash scripting.

Have a few question for you, if you can help. The questions relate to the naming convention you have chosen.

Can you enumerate any benefits of including special characters in the file name, and any down side to doing so?
I think, you hinted how it can help you in searches. Can you share some more on that front?
For e.g., I wonder what will be the use of including GPS coordinates in the file name - it might help instead if you included a location name if that can be done.

As for downside, I am wondering what the downside is of using these special characters in the file name.

Thanks for sharing your expertise!

at first thanx for the described steps for sorting photos.

after 4 days of hard work it seems to work in nearly all files.

I use photosync and owncloud.

is there a chance to move the files without the exiftool scripting, when i use the filename from photosync ?

this program generates 2015-09-12_12-45-03_img2345.jpg

is there a chance to use the 2015 for the first folder and the 2015-09 for the subfolder?

in Excel i would do @Left(Filename;4) and then @Left(Filename;9)

is this also possible in hazel script?

Hi,

Have an issue the follows the frame of this discussion.

My goal is to periodically back up my files from within the Apple Photos library (in the even I dump or lose iCloud). I export a photo with the photo taken date of 8/30/2015.

When I run the hazel recipe with the photo script above, the file renames to the correct date '8302015' but the creation date of the file is 12/1/2015 (the date the photo library was created).

Question... I do I amend the creation date of the file to get it to read 8/30/2015 so I can make certain all exif data is exactly preserved?

The creation date of the file in the filesystem shouldn't have any bearing on the EXIF data, which is inside the file itself. Also, you can try using the Sync action instead as I think (need to double-check) that it preserves the creation date.

Hello -

First, I hope someone sees this a few months later

I am using the tips and code above to do as outlined at the top post. I seem to have it working for photos. However, for movies, when the script runs, they are getting named with the file creation date (which is the export date from Photos - i.e., today as I am working on this today).

I am very novice at EXIF Tool and have a hard time understanding its complexities (I don't write scripts at all). I can't tell what is in the EXIF Tool shell scripts that would be causing it to run differently than it does on the photos....but I think this is what is happening.

Help! Please and thank you!!

Kelli

For those of you that are having trouble with getting the creation dates of movies/videos to be correct, I have found a solution! If you get you files renamed in the format "YYYY-MM-DD at HH-MM-SS" this might work for you too! I used other peoples suggestions on photos and got them renamed and sorted, there's also a awesome Mac App called "Renamer" that works for photos and doesn't require an code writing to make it work, just drag and drop what you want.

Anyways, based on https://discussions.apple.com/thread/2739681?tstart=0, I used this to edit to code to accept my format "YYYY-MM-DD at HH-MM-SS.filetype" , open Script Editor, paste the script below, run, compile, save as application, drop your movie files on top. This changed the creation dates and modification dates on the videos that I exported from Photos app to match their names (the corrected and true dates they were shot).

Code: Select all

on open of theFiles
   -- This is executed when files are dropped on the script
   repeat with i in theFiles
      if folder of (info for i) then
         processAFolder(i)
      else
         processAFile(i)
      end if
   end repeat
end open

on processAFile(theFile)
   set theName to name of (info for theFile)
   -- Assumes file syntax begins with YYYYMMDD_hhmmss
   try
      set yy to characters 3 through 4 of theName as text as integer
      if yy < 10 then set yy to "0" & yy
      set mm to characters 6 through 7 of theName as text as integer
      if mm < 10 then set mm to "0" & mm
      set dd to characters 9 through 10 of theName as text as integer
      if dd < 10 then set dd to "0" & dd
      set hr to characters 15 through 16 of theName as text as integer
      if hr < 10 then set hr to "0" & hr
      set min to characters 18 through 19 of theName as text as integer
      if min < 10 then set min to "0" & min
      set sec to characters 21 through 22 of theName as text as integer
      if sec < 10 then set sec to "0" & sec
      do shell script "touch -t " & yy & mm & dd & hr & min & "." & sec & " " & quoted form of POSIX path of theFile
   end try
end processAFile

on processAFolder(theFolder)
   tell application "Finder"
      set theContents to items of folder theFolder as alias list
   end tell
   repeat with i in theContents
      if folder of (info for i) then
         processAFolder(i)
      else
         processAFile(i)
      end if
   end repeat
end processAFolder