Weird issue - maybe a race condition?

I am seeing some strange behaviour from Hazel and I suspect it may be some kind of timing issue or race condition...

I have a few questions to help me troubleshoot this:

1. When multiple files get added to a folder that is monitored by Hazel, does it process each file, through all of the rules defined on that folder, first before starting the next one? Or does it process the files in parallel?

2. When a file gets renamed as part of rule processing that file may now match other rules that it didn't previously match? Is this re-evaluation performed as part of this processing pass i.e. will any newly matching rules be processed 'now' for the file or does Hazel wait for the next 'scan' of the folder and essentially treat the renamed file as a new file?

3. If a rule calls a shell script as an action and that shell script renames the file, how is that handled? Since Hazel doesn't know that the file has been renamed will the renamed file not get processed now until the next 'scan' of the folder? And if so, presumably other files may get processed prior to that happening?

4. What exactly triggers Hazel to 'rescan' a folder? I assume it uses file system event notification and/or folder actions but those are not foolproof in my experience; does it also use a time based polling mechanism as a fallback?

Thanks,

Chris

Reading the section 'Understand the Logic of Rules' I think I have partial answers for some of these...

For (1), it seems like it processes files one at a time not in parallel?

For (2), I'm guessing that other actions in the same rule will be applied to the renamed file but unless the rule contains the 'Continue' action as it's last action the renamed file will next be examined on the next 'scan' of the folder?

For (3), since Hazel doesn't know that the file has been renamed by the shell script it will re-consider the renamed file on the next 'scan' of the folder? This leads me to another question; if the school script action that renames the file is no the last action in the list for the rule, will subsequent actions be able to act on the file even though it has a different name? i.e. does Hazel identify the file by inode number or some other unique id or just by file name when it is processing the rules for an action?

My question (4) still stands; what triggers the initial and subsequent scans of a folder?

Thanks, Chris

It's probably more productive to post what the original issue is that you are seeing but to answer your questions.

1. One at a time, not parallel

2. It happens on the next scan, unless you use the "Continue" action. And even then, only rules after the rename will be evaluated that pass.

3. While Hazel may try and track the file across renames, it may not work. Note that if you use AppleScript, there is a way to communicate back to Hazel the new path/name of the file.

4. File events are sufficient. For time based occurrences (like doing "Date last modified is not in the last 5 minutes" will set up specific timers to handle those events but no polling is done unless file events are not supported (like on network drives).

The original problem was that when we put a large number of files into our workflow's 'inbox' folder at one time, some of them just 'disappeared' while being processed. If we did them one at a time, or just a few at a time, then they were all processed correctly.

I have several shell scripts that run as part of different rules that (a) extract data from the files (that cannot sensibly be extracted using Hazel pattern matching) and then rename the files based on the extracted data and also appending a '.proc' extension to the end of the new name (which already has an extension). So a file named 'Email.pdf' going into a script might end up being renamed as '2018-02-26 Email from=xxx@yyy.com to=abc@def.com.pdf.proc'. A rule later in the folder ruleset moves all '.proc' files elsewhere and removes the '.proc' suffix in the process, but of course that doesn't fire until the next time the folder is scanned so the .proc files may hang around for a little while and there is a non-zero chance of a file name conflict. I have added code to the shell scripts to ensure they generate unique names to avoid potentially overwriting existing and not yet moved .proc files (seems somewhat unlikely but certainly isn't impossible). I will monitor the situation to see if this resolves the issue.

I'm interested in your assertion that file events are sufficient. I'm not an expert on exactly what file operations generate these events on macOS (for example do file renames using the 'mv' command generate file events?) but I have seen it stated elsewhere that file events are not always 100% reliable (I don't know if that is really true) and I certainly do sometimes observe files hanging around for a significant time even though they have not been previously matched and Hazel says that they match a rule which should cause them to change / move immediately. Hence my curiosity as to whether there is some kind of timed poll as a fallback to ensure timely processing of files.

Chris

You'll have to provide evidence of file events not firing as I've not seen a case of that unless the filesystem in question does not support them. It's an all or nothing thing.

Cases where things may not be happening are due to other factors. They can range from a faulty installation (sometimes people manage to install Hazel twice so two instances are interfering with each other) to rule misconfiguration to a possible bug. In all cases, the file events come in.

There are maybe one or two exceptions but they are quite rare. The one that comes to mind is monitoring an FTP server folder. Files uploaded via FTP using the FTP server built in to macOS somehow is able to bypass sending file events.