keywords - where and how does Hazel store them?

Hi

When someone writes a keyword using a rule with Hazel, where does the keyword go?

Two things I am wondering:

1) Do you use MDItemSetAttribute to set the keywords in spotlight?

2) Do you set keywords in the file attributes (xattr)?

--Tom

Hi Tom,

We already discussed this in email but thought others may be curious about this. In short, 1) yes 2) no.

Keywords is a bit of a problematic feature because they are only stored in the Spotlight index. Apple never really provided a UI to change them and it appears that the de facto convention for tagging files is to add them to the comments field. Therefore, I'm considering retiring the keywords feature in favor of having people use the comments field for tags.

I will probably post more on this but if you are using keywords now, you should start migrating towards using the comments field instead as it's a better supported and more robust solution. As to how to phase it out, I'm trying to figure that one out.

Mr_Noodle wrote:you should start migrating towards using the comments field instead as it's a better supported and more robust solution.

Maybe relative to using keywords but …

… it appears that the de facto convention for tagging files is to add them to the comments field.

I don't trust this rather vague "de facto convention" of using Spotlight Comments as a reliable, robust solution for file tagging and won't use or recommend it.

I've already expressed and questioned that in discussions about tagging (mostly relative to Leap) on the Ironic Support Forum, as Tom might recall. If Hazel were able to directly and reliably read/write Leap tags I'd make use of that functionality, more willing to accept clearly defined Leap-specific limitations than allow Spotlight Comments to be a muddled method for storing/sharing tags.

I'm all in favor of a generalized file tagging solution, just adamantly resistant to Spotlight Comments being satisfactory even if it works fine for other people.

Could you specify how Hazel does not work reliably with Leap tags? Hazel is able to filter on comments and also write them out so I'm not sure what you are alluding to.

As far as using Spotlight comments for tags, it's the best solution there is right now. It is common to everyone and is accessible by users in Finder. The only other solution I can think of using using an xattr (extended attribute). The problem with that is that you have to get all the devs to start using the same one. On top of that, users will not be able to see or change the tags without a special program.

If you have something else in mind, please share it but right now I'm not really sure where the problem is.

Simply, as mentioned at the end of my previous post, I'm strongly resistant to using Spotlight Comments for file tagging.

Mr_Noodle wrote:Could you specify how Hazel does not work reliably with Leap tags?

I didn't mean to imply that, though Hazel only supports Leap's user-defined tags indirectly through Spotlight Comments. I'm not (and won't be) using those for file tagging even if Hazel's interaction is reliable and:

As far as using Spotlight comments for tags, it's the best solution there is right now.

Probably, though it's still just an informal solution. Part of the trouble is Spotlight Comments allow tags to be loosely defined, e.g. using different prefixes and/or delimiters.

It is common to everyone and is accessible by users in Finder.

Sure, which are sufficient reasons for some people.

The only other solution I can think of using using an xattr (extended attribute).

That's the only possibility I'm aware of now, too.

SpotMeta's approach always seemed more appealing to me for tagging than "stealing" Spotlight Comments, with spotmeta-leopard | Google Groups recently attempting to revive that effort.

The problem with that is that you have to get all the devs to start using the same one.

I'd love to see more developer ideas and consensus about alternatives to Spotlight Comments for tagging. It's a topic I'd surely bring up with folks I'd meet at WWDC, if I were fortunately enough to be attending.

On top of that, users will not be able to see or change the tags without a special program.

Free format Spotlight Comments already have to be specially structured/treated to conform to any tagging pseudo-conventions, which has proven to be problematic (more often than I'm comfortable with).

If you have something else in mind, please share it but right now I'm not really sure where the problem is.

So far nothing specific in mind beyond SpotMeta and some Ironic Support Forum discussions (e.g. here and here).

It's not hard to see problems with building a file tagging system with Spotlight Comments as the foundation. At best it's a well intended interim, but ultimately inadequate, solution.

Maybe you already know, under normal conditions Spotlight Comments are saved in the com.apple.metadata:kMDItemFinderComment xattr on Leopard. At least they're not completely reliant on being stored in .DS_Store files anymore.

From your original reply to Tom:

I'm considering retiring the keywords feature in favor of having people use the comments field for tags.

I agree with retiring that feature, with Spotlight Comments for tags as the currently preferred alternative (caveat emptor).

I'm concerned too many developers will become satisfied explicitly or implicitly offering/promoting Spotlight Comments as the method for tagging files, having less incentive to pursue and encourage superior solutions.

I'm still not sure why tags in comments is such a terrible thing. It is the result of users coming up with a system using the interfaces available to them. While having some specialized program to do it how you want it in particular may be "cleaner" to you, keep in mind that unless it ships with the system, it's not something you could rely on as being widespread or somewhat standardized.

I'm considering talking to other devs about it but it will require a lot of coordination amongst several devs for something that for the majority of people seems to work fine.

Mr_Noodle wrote:I'm still not sure why tags in comments is such a terrible thing.

Being aware of certain problems that can occur using Spotlight Comments for storing file tags is why I choose to avoid using (and recommending) them for that purpose. That doesn't make it terrible, just for some people (including me) those risks outweigh the benefits.

It is the result of users coming up with a system using the interfaces available to them. While having some specialized program to do it how you want it in particular may be "cleaner" to you, keep in mind that unless it ships with the system, it's not something you could rely on as being widespread or somewhat standardized.

I realize things like that have motivated and can still favor using Spotlight Comments for tags.

Maybe I'm misguided or too optimistic but I'd like an implementation of file tags that makes them essentially peers with file names (that have a well known, predictable, trustable behavior), which definitely disqualifies Spotlight Comments (that require too much diligence to preserve their integrity). That's an attempt to densely summarize my point.

I'm considering talking to other devs about it but it will require a lot of coordination amongst several devs for something that for the majority of people seems to work fine.

If file tags are important to me they certainly can't be trusted to something that seems to work fine.

I'd welcome and be interested in coordinating a thorough open discussion about using Spotlight Comments for file tagging (and viable alternatives), with contributions from people who see issues differently and/or understand pros/cons better than I do. Unless I'll discover something sooner that discourages the effort, though I'd be surprised if I'm the only devil's advocate about this.

Even if we apparently disagree I've appreciated your feedback. Thanks!

I think this points to a more general issue of user defined metadata. Currently, Spotlight only handles intrinsic metadata, that is, metadata that is already in the file. Except for comments and xattrs, there is no way for users to define their own extrinsic metadata (and for xattrs, there is no common UI for it).

At some level, Apple will have to get involved as it requires more than coordination. For instance, these attributes would need to be part of the standard set of things indexed by Spotlight (i.e. not specific to a importer/file type). I had a brief discussion with one of the Spotlight engineers at WWDC and the best we can do now is to file bugs against it.

Also, Tom and I had a brief talk at WWDC. I think it might be useful to have some 3rd party devs band together to come up with a standard schema for xattrs so that the apps can interoperate and users do not have to be conscious of the reverse-domain naming convention. Once I can come up with a more cogent solution, I'll probably post something on the blog for other devs to consider.

Thanks for taking some initiative with this at WWDC, Paul. I agree with your observations, especially:

I think it might be useful to have some 3rd party devs band together to come up with a standard schema for xattrs so that the apps can interoperate and users do not have to be conscious of the reverse-domain naming convention.

Unless OS bugs or other irresolvable technical issues prevent it, xattrs seem a better standard, interoperable foundation for user defined metadata than Spotlight Comments. They're both vulnerable to accidental deletion/modification but there's less contention with xattrs being used specifically and only for this purpose.

There's no reason for me to blindly speculate what, if anything, might help get Apple more involved beyond filing bugs against Spotlight. Maybe someone who's more enlightened can, though.

Hi guys,

Can I please interrupt the geekfest a moment ... I'm a Hazel newbie, and have been snared by the confusion between "comment" and "keywords". As a result I have a couple of unwanted keywords, which are messing up my display in Leap. Can someone please tell me how to get rid of them? Thanks.

You need to contact Tom about Leap stuff. If you have some files with keywords, drop me an email and I'll see if I can help you to get rid of them (though keeping them there shouldn't cause any harm).

Can someone explain what exactly is wrong with the current keyword situation? The comment about it being only in the Spotlight index seems wrong. (they are actually put into xattrs if you use MDItemSetAttribute). And the comment about there being no standard UI is valid, but hardly any worse of a problem than the other kludges being suggested.

BTW, what is this supposed defacto standard about keywords in the comment field? Is it just space separated, or comma separated words? And these keywords somehow supposed to be separated from other text somehow? Tell me more about this.

I think using xattr's is a Leopard-only thing so the volatility of keywords was a valid point on Tiger. Also, the problem with no UI is that it does confuse users as they may set keywords and then have no idea where else they come into play. At least with the comments, users can see the effects.

As for the defacto standard, I always forget what the exact standard is. As they say, the great thing about standards is that there are so many to choose from. IIRC, I think it's keywords preceded by an ampersand and separated by spaces but maybe someone who uses other tagging software can elaborate.

countach wrote:they are actually put into xattrs if you use MDItemSetAttribute

Based on what I've read MDItemSetAttribute is still private in 10.5 -- it's not in Apple Developer documentation.

Mr_Noodle wrote:As they say, the great thing about standards is that there are so many to choose from.

Yep, and easily applicable to the defacto ones for OS X file tagging.

IIRC, I think it's keywords preceded by an ampersand and separated by spaces but maybe someone who uses other tagging software can elaborate.

Some software uses '@'-prefixed and comma-separated tags, sometimes as alternatives to &/Space. And I think some allows double-quoted multi-word tags, e.g. @"This is one tag".

Being vehemently opposed to using Spotlight Comment for tag storage is one reason I don't know more specific details offhand, though I'll always recommend checking directly with developers which variations are supported by their software and confirmed (when possible) to interoperate with others'.

I'm quite curious about this recent Ironic Software news:

We are now working on a new system that is proving to be more robust and stable than the current methods and eliminates the need for Spotlight Comment Tags (which we no longer advocate as a reliable mechanism).

We at ironic are working to release a new xattr based tagging system to the public domain within a week or three. We will open source the technology (which is just a few cocoa files).

Basically, it will:
1) Allow programs to set/retrieve tags, ratings, etc on files using a simple API. This api will enforce a few 'rules' about tags - (eg no duplicates, case insensitive, case preserving, etc). Rules are generally 'good' - they allow for consistent user experiences.

2) These tags are automatically indexed with Spotlight.

3) Uses no Apple 'secret' api.

4) Allows the setting of (non spotlight indexed) 'larger but still small (<4k)' blobs of meta data, such as workflows , etc.

There are a few 'gotchas' that we found while working this all out, which is one reason why we thought it was important to not only release the idea, but also some source to implement it.

Deep (see ironicsoftware.com) uses this tech.

--Tom