Excessive RAM Usage

I decided to make a clean install of Hazel and all my rules. After putting 84 folders with 155 rules in total, Hazel is using already more than 3 GB of RAM. And I have still 94 folders with 262 rules more to go…

Mac OS X 10.7.5

Any idea about this? Thanks!

[edit]: After a couple of hours working in Hazel, the RAM usage is permanently in between 4 and 5.5 GB. When starting up Hazel, it's even 7.5 GB!

Which process is using up memory specifically? Also, 84 folders and 155 rules? You may want to look into consolidating those. Some tips:

- If the folders are under a common folder, have Hazel monitor that folder and descend into the subfolders as needed. See the sticky article near the top of the forums on how to deal with subfolders.

- Search the help for "match patterns". This will allow you to use patterns to consolidate a lot of repetitive rules.

You may want to describe in more detail why you have so many rules. I think with that many, it might be unavoidable that Hazel will have to use resources to run those.

The process which is using up memory is "System Preferences", after having Hazel started to define rules. Besides of this, there is a bunch of "hazelworkers" using between 10 and 70 MB each.

OK, let's give it a try.

I've got a picture archive, with pictures organised by folders and subfolders. Whenever a picture is added to the archive, or edited i.e. in Photoshop, it has to be uploaded onto a server with an online archive. So far, no problem.

In the same time, I need to rename a copy of the picture file, and to add certain tags to the filename. For this, the related picture files are copied into a certain folder whose name is containing all the needed tags. There, the picture files will be resized to a minimum picture size (Automator Workflow), and renamed using a certain pattern which includes the name of the source folder.

Example:

The folder "01 Hotels HR" contains 80 subfolders like these:

Banjole
Baska Voda
Biograd
Bol
Cres
Dubrovnik
…

These folders contain picture files. Depending on the origin folder, the containing pictures are copied to a destination folder. One destination folder for each origin folder:

y-y-HR-IS-PUY-52-y-y-y-y-u
y-y-HR-IS-PUY-55-y-y-y-y-u
y-y-HR-KV-RJK-61-y-y-y-y-u
y-y-HR-KV-RJK-64-y-y-y-y-u
y-y-HR-MD-SPU-86-y-y-y-y-u
y-y-HR-MD-SPU-89-y-y-y-y-u
…

Here, the pictures will be resized (Hazel & Automator Workflow) and renamed (Hazel), and then moved to an upload folder where a FTP client with hotfolder watching is taking them over. This part is done with one single rule, using the option "run rules on folder contents".

I don't have any idea how to realise this without having a rule defined for each single folder.

Regarding my "mass od rules", there are other folders besides this archive where I need 27 rules for one folder. Depending on a certain pattern (27 possibilities), these files need to be moved to a related folder (27 folders).

I hope that I could point it out.

I've got some new evidence:

1st
Before editing rules in Hazel, I stop Hazel. Otherwise, I am totally running out of memory, and can hardly do anything. After closing the System Preferences, I start Hazel again. This is somehow working, but RAM usage after opening the Hazel Pref pane is still somewhere about 5 or even 7 GB.

2nd
Hazel was running fine this morning. After a certain time, even when there hasn't been any action and Hazel didn't have anything to do, I see in the Activity monitor that there are 68 (!) "hazelworkers" active. RAM usage in average between 6 and 30 MB, one of them about 300 MB, but the peak was just over 5 GB! See my screenshot from the Activity Monitor:

Even after stopping Hazel, the "hazelworkers" are still active and don't free any RAM. The only way is to quit the processes.

Could this be an issue under Mac OS Lion? I had almost the same number and kind of rules running before under Mac OS Snow Leopard, and never faced such problems…

I would be very appreciated if there is a way to get this sorted out.

EDIT: Hazel is still stopped, but the remaining "hazelworkers" are using around 100 MB of RAM each now.

One thing I have noted when doing live testing is that Hazel seems to queue up the job and create the hazel helper. This means that once you tell Hazel to stop, it seems to only stop adding to the queue of jobs, but the jobs already in the queue get executed. Essentially what I am saying is you have to be patient and let the jobs run until they are completed before you will see the usage drop. Doing a "Run Rules Now" will allow you to see what is happening on a per folder basis, without running the remainder of the folders and their rules.

This should help you narrow down what folder(s) is causing your issues. It does sound like you have a lot of folders that may be condensed. This will cause you to lower the number of simultaneous hazel helpers you have going. Remember, a "Run Rules on Subfolders" rule can me a major ally in this situation.

Mr. Noodle, please correct me if I am wrong.

Bryan

Hi Bryan,

I'm not really sure about "to be patient and let the rules run until they are completed". The first time when I ran out of memory, all jobs were done. I didn't copy any more picture files to be treated. In the end, the screensaver freezes, and there's nothing to do except to push the power button.

Exactly this happened two more times today. Just an hour or so after booting the Mac, there are numerous hazelhelpers active, one of them using 4 gigs or more. And I didn't touch the Mac at all, except wakening it from the screensaver to watch the Activity Monitor. No actions for Hazel to do. Maybe I should add that this Mac is working exclusively as a local server for the picture archive (at this moment not used except for testing purposes), as an FTP server for incoming data (really not too much), to run the Hazel rules, and to upload the files passed by Hazel to a certain web server.

I'm really getting desperate. More or less the same folders and rules have been working perfectly on another Mac. The only significant change is that I added one folder with 78 rules.

I would like to use "Run Rules on Subfolders" even more. I already use them for a bunch of jobs (resizing and renaming of files in subfolders). I tried with others, but didn't succeed. Here's the typical example:

I've got a folder with X subfolders. If files within the subfolders are added or last modified today, they should be copied to a certain destination, depending on the name of the folder they are contained in. In details:

– Folder AAA (watched by Hazel)
– Subfolders aaa, bbb, ccc, and ddd, each of them containing picture files.
– If a picture file in one of the subfolders is added or modified today, it should be copied:
– from aaa > 111, from bbb > 222, from ccc > 333, from ddd > 444 and so on.

I don't have an idea how to solve this by "Run Rules on Subfolders"… At least, all my attempts failed…

To do subfolders, read the article carefully. People, for some reason, ignore the picture with the big red X through it. That means don't do that. If you only have one rule and you say that it does run rules on folder contents, then chances are you are doing that thing that the article tells you not to do, which is to not combine different logic into one rule. Keep that subfolder rule separate. Do not modify or add to the subfolder rule unless you actually understand what you are doing. Your logic should not be in that rule but in separate rules.

Now, I don't have enough details on these patterns but again, look at the pattern matching. You have way too many rules and folders for it not to take up a ton of resources. X folders multiplied by Y rules multiplied by Z files (including those in subfolders) is a ton of processing that has to happen. Your #1 focus right now should be consolidating and reducing your ruleset. I suggest creating a test folder with a smaller number of files and build your ruleset up gradually.

OK, I'll try to consolidate the whole thing, and try to make more use of the option to let have Hazel working on subfolders. I'll go slowly step by step. But I already stuck at the very beginning.

I have a folder "01 Hotels HR" which contains subfolders which have to be processed. For this, I defined this rule first:

Then, in one of the subfolders (named "Banjole"), I want to find all files added or last modified today. These files should be copied to a certain destination. For this, I defined this rule:

This doesn't work. The 1st rule matches, but the 2nd doesn't.

The second rule will only match files that are directly in the "Banjole" folder. It won't match (a) folders in there since the first rule matches folders (you'd need to reverse the order of the rules) and (b) files deeper down. I'm not sure if this is what you want or not. Also, do you expect the files in "Banjole" to have different criteria than another folder underneath "01 Hotels HR" ? If you could provide more specific details I can better advise on how to do the rules for it.

The second rule will only match files that are directly in the "Banjole" folder.

This is correct and should be like this. But it doesn't work as it should. Again, I ran two tests now. The first time, all files in the "Banjole" folder have been matched and copied, not only the 4 newly added files. The second time (same test, adding 4 files to the folder), nothing has been matched and copied.

It won't match (a) folders in there since the first rule matches folders (you'd need to reverse the order of the rules) and (b) files deeper down.

There are no subfolders within the "Banjole" folder.

Also, do you expect the files in "Banjole" to have different criteria than another folder underneath "01 Hotels HR" ?

Yes, and the different criteria is exclusively the name of the enclosing folder (here "Banjole").

Further explanation:

Besides "Banjole", there are 66 more folders in "01 Hotels HR". Each folder represents a place. Within these folders are pictures of hotels. It doesn't matter how a picture (hotel) is called. Only the name of the enclosing folder (place) is important.

Does this help a bit to understand?

It should match. You should check the preview, click on one of the files and click on the "i" button. Look at the attributes of the file and check carefully to see which field is making the file match or not.

How are the rules different for each hotel?

There are no different rules for each hotel. All hotels in ONE subfolder have the same rules and conditions, only depending of their enclosing folder.

But anyway, thanks for now. I'll be travelling for the next 3 weeks, therefore I cannot go on with this item for now. I'll get back to you, most probably with an extensive description of the whole thing.