Noodlings

As mentioned in my last post, I had various issues with codesigning and notarization during Hazel 5’s launch. Now before I get into it, remember during all this that Hazel is codesigned and notarized with no warnings or errors. In my release build process, I do a codesign —verify as well as spctl to make sure everything is hunky-dory before submitting to Apple for notarization.

The biggest problem at launch was some users getting an “Unidentified developer” alert when opening the dmg. I had various users send in logs, but it was only when someone found a log message pertaining to the rpath for one of the binaries in the bundle that I was able to identify the problem. Strangely enough, that person didn’t receive the “Unidentified developer” error alert.

Turns out, I had mistakenly added a binary to the top level bundle (it was originally in a framework). The binary had an rpath which escaped the framework bundle as it depended on a sibling framework. This worked fine when the binary was in the framework bundle but when it was at the top level of the main app bundle, it escaped the app bundle itself. Not sure how the binary got added to the Copy Files phase there but it was an easy fix to remove.

Another issue was that the installer wasn’t triggering for some people. For the installer to trigger, Hazel needs to detect that it is running from a disk image. Further investigation turned up that Hazel was being translocated. If you don’t know what translocation is, I suggest reading this article: https://lapcatsoftware.com/articles/app-translocation.html

When translocated, the binary is no longer on the disk image, instead it is copied to a temp location on disk. Hazel is unaware of this and as a result, doesn’t run the installer. Why was Hazel being translocated? I’m still not sure. It’s my understanding that if an app and its containing dmg is signed and notarized, it shouldn’t be translocated. As a workaround, I ended up using the private SecTranslocate.h APIs to detect the translocation and compensate appropriately.

Possibly related to the above, there was also an issue with the embedded helper app not running. Logs from users showed that the quarantine flag was still set on the helper and that was preventing it from being run. When the user copies an app, like say from a disk image to /Applications, the quarantine flag should be cleared for the app and everything inside but for some reason it was not clearing it for the embedded binaries. Note that unlike when a user launches an app from Finder where they will be asked to run the app, a login item helper will fail to launch without any prompt. As a short term workaround, on first run, I had Hazel recursively remove its quarantine flag before it attempted to run the helper.

Even with the workarounds, something didn’t sit right with me. I suspected there was something wrong with my bundle that was causing these issues. I ended up filing a DTS incident to get some feedback from Apple. The DTS engineer was able to reproduce the issue on a VM and dig up a log message indicating that it could not clear the quarantine flag on the Autoupdate helper app in the Sparkle framework. For those that don’t know, Sparkle is a very commonly used framework for apps to update themselves. The Autoupdate helper runs when your app is terminated so that it can install and run the new version.

This all lead to an investigation into where should Autoupdate, or executable binaries in general, be put in a framework. Now, I had looked into this once before . Sparkle originally put Autoupdate in Resources which is incorrect and can cause subtle issues. The solution I came up with back then was to put any executables alongside the framework binary, which would be in Versions/A with a symbolic link at the top level of the framework bundle. This seemed to work at the time. It got through codesigning and notarization checks without issue and ran fine after that.

But it seems that the problem is more subtle than that as whatever code that clears quarantine flags on embedded bundles didn’t like that. I was pointed to this article which suggests I put the program in Contents/Helpers. Despite the insistence that this was the way to do it, it flat out did not work. An important thing to note is that framework bundles do not have a Contents directory. Putting a Contents directory, either in Versions/A or at the top level will make codesign barf, complaining about an invalid bundle format. My guess is that when codesign sees a Contents directory, it considers the bundle to be another (non-framework) type of bundle. After wasting a good bit of time on this, I went with what I thought made more sense, putting it in a Helpers directory (sans Contents). If you have an Xcode “Copy Files” phase copy to Resources for a framework, for instance, it doesn’t put it in Contents/Resources, but in Resources so it made some sense to do it this way.

Sure enough, that worked. Of course, after fixing it, someone pointed out how some of the system frameworks use the Helper directory. I guess I should’ve checked there first. At least I have some confirmation that I was on the right track. Note that you should put the Helper directory under Versions/A. You can symlink to the top level but that’s optional. I did it mainly to give myself a more consistent path to access it in the very unlikely event that I end up shipping different versions of the framework.

Overall, this experience has been very frustrating. There is little to no documentation on all the various things that can go wrong in your bundle that will cause things to fail, and when it does fail, the logging is very inconsistent. Lastly, none of the tools or processes in place (codesign, spctl, notarization) catch these cases. These are all issues related to the static structure of the app bundle so it seems like they should be detectable. Having a tool that developers can run on their bundles before submitting them would be very helpful as the current ones are far from complete in that regard.

After all this, I’m still not sure if the translocation problem is still there or not. I’m not about to remove a workaround just to have it fail for users just so I can get some feedback so that question will remain unanswered for the time being. The lack of definitive and correct information during this whole ordeal is a bit disturbing and makes me think that there’s no one that actually knows for sure how things really work.

For those at Fruit Company, I’ve filed FBAs FB8981011 and FB8981016.

Recently, I took some time to do some clean up on my Xcode project as it has collected its share of odd scripts and settings over the years. Little did I know that I’d make a couple of discoveries surprising enough that I’d actually bother to blog about it.

Speeding up builds

The first thing I wanted to address was build speed. Over the years, it’s felt like build times have gotten longer. Of course, before we start mucking with things, we needed a way of measuring which leads me to the first tip: you can have Xcode display build times in the main activity display in the titlebar. Just use this default:

After doing that and performing a build, it should look something like this:

I tried tightening things up: getting rid of unnecessary steps, removing redundancies, making sure all script phases had inputs and outputs specified. They had minimal impact. Looking at my build logs in Xcode (which helpfully show the time for each step), I noticed that code signing was taking up a significant amount of time. Around 3-8 seconds each occurrence, even for standalone binaries. I have quite a few of these as well as frameworks so that time adds up. Note that for debug builds, I had the “—timestamp=none” option set so that was not the culprit in this case.

Poking around, I came across this thread. While I didn’t have the main problem described there of duplicate certificates, buried in that thread was the following advice: trim ~/Library/Preferences/com.apple.security.plist 

Opening that file up revealed that I had several entries, all except one pointing to non-existent files with the one valid entry pointing to my login keychain. After removing the invalid entries, code signing only took up to 1 second, max. This shaved 40-60 seconds off of my full release builds and 10 seconds off of incremental ones. Huge savings.

Cleaning out frameworks

Another thing I noticed in the cleanup was that some of my frameworks were being copied without their headers. I had a script of my own to remove the headers after copying for deployment builds, but the frameworks remained header-free even when I disabled/removed this script. To make things even stranger, this was was only happening for one target as another target would always have the headers in included frameworks. I checked all sorts of settings but it was only after asking in a Slack channel did I find the answer. Thanks to Nicholas Riley, who didn’t miss a beat and pointed me to this thread.

Apparently, there’s a hidden setting in your project.pbxproj file for copying frameworks where you can specify whether headers get copied over. This is not exposed anywhere in Xcode’s UI, as far as I can tell. It’s also a mystery why it gets set on some targets and not others. The only way to enable/disable this is to edit the project.pbxproj by hand. 

The setting looks something like this:

The flag in question is RemoveHeadersOnCopy. It appears the setting sticks even if Xcode writes to the file after your edit. Of course, if you add new frameworks to copy, you may need to edit the file again.

Hopefully the above helps you out with your builds. If you have any eye-opening tips to make your build process quicker/better, comment below as I’d love to hear them.

In my frameworks, I have extra programs that my framework code needs to use. Before, I would have these auxiliary programs outside the framework bundle and would set a variable in the framework at runtime to point to the location. Now, having everything self-contained in the framework bundle makes things a bit nicer.

This approach is not without it’s little gotchas, which I’ll go through here.

The basics are straightforward enough. In Xcode, for your framework target, add the binary to the target dependencies and add a Copy File build phase which copies it to the Executables location.

With a normal app bundle, these binaries are put in the MacOS subdirectory. In your code, you can use -[NSBundle pathForAuxiliaryExecutable:] which knows to look in that directory.

With frameworks, it’s not as simple. First off, there is a Versions directory with (potentially) various versions underneath. There is a symlink to the current version as well as symlinks at the top level of the framework directory. These symlinks point to the individual directories and files in the framework, including a symlink to the main framework image. Binary images are not put in their own subdirectory as they are with app bundles.

Any other executables aside from the framework binary are not automatically symlinked. On top of that, NSBundle can’t find them unless they are at the top level. So, in addition to the above steps, you need to add a Run Script build phase which symlinks your extra binaries to the top level. Make sure to use the -h flag (in addition to -s) to ln so as to not resolve the Current link.

Now, suppose your executable links against another framework in your app? For example, let’s take the following situation:

In this case, let’s say someprogram links against the Bar framework. We’ll need to specify an @rpath but what isn’t clear here is what @loader_path is (for an overview of @rpath and @loader_path, check out this post).

It would make sense that @loader_path is the top level of the framework. After all, NSBundle will return the top level symlink to your binary which is what you actually run in your code. In such a case, the rpath would be @loader_path/../ (we want to go one level up from someprogram so that we can see Bar.framework).

Unfortunately, that isn’t the case. Even though the resulting crash report you would get from doing the above will show the path to the symlinked binary, the loader seems to resolve the link, and then set @loader_path to that. So, in this case, it is referring to the original binary in Versions/Current/. Taking this into account, the correct rpath would be @loader_path/../../../

Not a huge deal but hopefully this will save someone from some confusion. As usual, it’s possible I’m missing out on something here, in which case, please comment below on how wrong I am.

I’ve finally gotten around to consolidating most of the bits of code I’ve posted on this blog over the years and put it in a repository. While the real NoodleKit is a much more extensive and cohesive toolkit that I use internally, this one will serve as a place where I put the odd scraps that I decide to open source. I guess I could’ve called it NoodleScraps but it doesn’t quite sound as nice.

If you’ve been following my blog for a while then there’s not much new here, codewise. There are a tweaks here and there plus the stuff should be 64-bit ready so it might be worth re-downloading just for that. It will be the place where I put future code so you should keep an eye on it.

The project is structured to build a framework but also has an examples directory (with corresponding targets) to demonstrate the use of the different classes. The current notion is to keep things compatible with 10.5. And, as usual, all code is released under the MIT license.

NoodleKit (hosted at github)

Enjoy.

As you may or may not know, Hazel is packaged as a preference pane. While you do all the configuration via the preference pane interface, the actual work of running your rules is done by background processes. The commandline tool that actually runs the rules is stored inside the pref pane bundle. I’ve kept it in the Resources part of the bundle for no good reason except that’s the catch all for all your bundle stuff. But then I thought, “why not stick it in the MacOS dir. That’s where executables are supposed to go, right?”

So, I went into XCode and created a new “Copy Files” phase to copy the built executables into the “executables” dir (which ends up being the MacOS dir in your bundle). I modified the code that launches the program to look in the right place (NSBundle’s -pathForAuxiliaryExecutable: for those of you keeping score at home). It seemed to work. My bundle was now a tad tidier with things in their proper place.

Later on, I was testing something else and noticed that when Hazel was running certain rules, the pref pane icon would show up in the dock and then disappear when the background program was done. Strange enough that a pref pane icon would appear in the dock; stranger that it was being triggered by a non-GUI program. More testing revealed that it happened when Hazel was executing AppleScripts. I tried isolating the AppleScript parts. I checked all sorts of paths. I couldn’t figure out what was causing it. The Pope was in town so maybe he was messing things up somehow.

I checked previous versions to isolate when the bug was introduced and discovered it was only in the last set of changes. Moving the executables back to Resources fixed it.

Problem solved but I had no idea why. After muttering about it in IRC, Mike Ash casually throws out the explanation: “the binary is hitting the window server, the window server notices that it’s in an executable directory of a bundle, sticks it in the Dock.”

Of course, it all sounds so simple when you hear the answer. Calling the AppleScript created the window server connection which then prompted it to throw the icon in the dock as if it were a regular app. Not sure how long it would have taken me to figure that out on my own.

The lesson here is that when faced with a crazy bug, ask Mike Ash. Really, though, don’t do that. He’ll crash his glider into me next chance he gets. The real lesson is that you should be careful about putting executables for background programs in the MacOS directory of your bundle.

Ok, so I can’t pin this one on the Pope. All I know is that I don’t get these types of problems when the Dalai Lama is in town.

It’s been a while since I posted. Since then, Leopard has been released and it’s been keeping me busy. Now that I’ve finally upgraded my dev machine to Leopard, I’ve started implementing Leopard-specific functionality. I still want to maintain Tiger-compatibility but there are cases where I need to reference new symbols in Leopard. These articles ^{1 2} tell you what you need to know, though I thought I’d provide a boiled down guide to what you need to do.

For the purposes of this article, we’ll assume we want a binary that will run on Tiger or later but have access to Leopard-only functionality. In this case, the deployment target is Tiger and the Target SDK (which API version you are using) is Leopard.

• In Xcode, Project->Edit Project Settings->General. For Cross-Develop Using Target SDK, select the latest OS version whose features you want to use. In this case, Mac OS X 10.5



• Click on the Build tab. Make sure the Configuration pop-up is set to All Configurations. Set Mac OS X Deployment Target to the minimum OS version you want to support (Mac OS X 10.4).

In most cases, this is all you need to do. Why’s that? It’s because Apple defines Availability macros that they use to set what symbols are available for different OS releases, based on the settings you just made above. If your target SDK is a later version than your deployment target, the symbols unique to the later target are weak-linked. This means that if the symbol does not exist, it will be set to NULL. As indicated in the articles linked above, you can compare these symbols to NULL before using them. Note that this mostly pertains to things like C functions. For Objective-C, you can check for the existence of classes using NSClassFromString(). For methods, just use -respondsToSelector:. What this all allows is for your program to dynamically use Leopard-specific functionality if it’s available. You can test for specific functionality without relying on doing a broader OS version check.

Of course, there are cases where things don’t work out of the box. For instance, it seems the headers for some lower level APIs aren’t set up with the Availability macros. In such a case, you can do your own prototype, this time asserting the weak linking:

extern void somefunction() __attribute__((weak_import));

If you don’t do this, somefunction() will end up being non-null even on older OS versions which will screw up your check for the function’s existence.

And then there are issues that aren’t based on API so much as behavior. For instance, if there was a bug in Tiger that was fixed in Leopard. It’s not the type of thing your program can detect so in such cases you’ll have to resort to version checking.

Nothing particularly new here but thought it might be helpful for people doing the OS version straddle for the first time.

It is a bit serendipitous that Gus Mueller’s discussion of Automated Builds on the Late Night Cocoa podcast should happen soon after I finally got my own act together and automated most of my build process. So, taking advantage of the momentum I thought I’d ramble on for a bit on the topic.

My process does not do as much as Gus’ (I don’t automatically upload) and it’s all triggered from within XCode when I do a Release build (Gus uses a script) but the important part is in the end I have a procedure that does most of the tedious parts for me. The problem with automating the process with XCode is that there are certain types of phases or resources that it doesn’t not handle natively so you have to script them yourself or it’s not obvious how to do it the right way. I thought I’d go over a couple of the oddball cases. Not all will be applicable but hopefully you can find something useful.

Code Stripping

This is something I neglected to deal with for the longest time. The notion is to strip your code but keep an unstripped version around so that you can use it with atos when you get a stack trace. I never got around to figuring out the right combination of settings on my own but this post gives you everything you need. Just stick this in a shell script phase after the product is built and use the XCode settings outlined in the comments. A big thanks to Andy O’Meara and Rob Barris for figuring it all out and sharing.

Building DMGs

This is one a bit trickier. Everyone seems to have their own methods and requirements for this so I’ll just point out some specific things here. First off, it’s all done with hdituil. If you don’t feel like learning hdiutil there are higher-level commandline programs and scripts you can use such as buildDMG or DropDMG (the latter is a GUI app which also comes with a commandline program).

In my case, I don’t construct the DMG from scratch. I created a template disk image with placeholders for my files. There are ways to deal with icon positioning programmatically but really, it’s just easier to do it by hand, save the image and use it over and over again. During a build, I mount the disk image, replace the files and then unmount. The disk image is then converted to a compressed format. For the mounting part, use the “-private” flag. It prevents Finder from seeing it so you don’t have to deal with the DMG window popping up while it’s being packaged. Unfortunately, my current script has some quirks specific to me so I’m not posting it but feel free to drop me a line if you have questions.

Also, credit goes to Jonathan Wight for providing the ideas and script that formed the basis of the DMG build procedure outlined above.

Useful Commandline Tools

If you are building an app then this is not an issue but for preference panes, it’s common to have a customized Finder icon for the .prefPane file itself (note that the icon specified in your bundle is for within the System Preferences app). For this, I downloaded the osxutils package. It contains a commandline program called “seticon” which, as the name indicates, sets an icon on a file. There are other little commandline programs to do things like set the Finder comment or the color label.

[Update: Tom Harrington was kind enough to point out my boneheadedness. Setting bundle icons in Info.plist for pref panes works fine so using the seticon program above is totally unnecessary. There was a misconfiguration in my project that lead to my faulty assumption that you had to set it separately.]

There’s also SetFile (comes with the dev tools) which is useful for setting HFS+ attributes. For instance, if you want to script setting the background image file in your DMG to be invisible, a SetFile -a V yourfile will do the trick.

So, basically most of the things you can and want to do to files in the Finder can be done on the commandline (therefore making them available for scripts).

XCode and External Targets/Dependencies

This is something that I always forget how to do. If your project has a dependency on some other project, such as a framework, you can drag the xcodeproj file into your project. But to create an actual dependency between your target and the external target, you can’t drag the thing underneath your target nor can you do a “Get Info” on the external target and check off the targets it’s a part of. Instead, select your target and do a “Get Info” on it. Under the “General” tab, hit the plus (+) button at the bottom to add the external product as a dependency. Also, you need to have both projects build to the same directory (“Get Info”->”General” tab). Now, building your target will build the dependent target as necessary. The important thing is that you do not depend on specific files. If you find yourself typing in paths that refer to another project’s build directory, you are probably doing it incorrectly.

XCode and Applescripts

I have some Applescripts in my project. Before, I saved them as .scpt files which are already compiled. Because they were already compiled, XCode’s “Compile AppleScripts” phase didn’t apply. Since I wanted to convert them to be “Run only”, I added a shell script build phase to use osacompile but this was a little clunky. I had to run it once to get XCode to generate the new .scpt files and then I added those to the XCode project so that they could be copied into the bundle during a build.

Instead of using .scpt, I now use plain, uncompiled .applescript files. XCode knows how to deal with these via the “Compile AppleScripts” phase. To make the compiled script “Run only”, add “-x” to the OTHER_OSAFLAGS setting in your build settings. Note that you may have to add “OTHER_OSAFLAGS” if it doesn’t exist.

• • •

So, that’s it for my random XCode tips. My build procedure is not perfect. I need to better automate versioning but for now it’s relatively pain free. If you have any tidbits, post them here.

This issue came up twice in the past couple weeks thus triggering my rule that if something comes up more than once then others may be interested in it. So, here’s a little tip that may save you some frustration.

When using regular expressions with Xcode’s search, the docs mention that XCode uses the ICU library. Naturally, you’d think it would use ICU’s syntax for specifying backreferences in the replace string, which would be to use variables consisting of a dollar sign ($) followed by the number of the capture group being referenced.

Of course, if that were the case, I probably wouldn’t be writing this tip. Using $ for backreferences turns “Search & Replace” into “Search & Destroy”. The syntax is to use backslash (\) instead of dollar sign. In short, \1 instead of $1.

Type the following in an Xcode editor window: “$1 works in replace strings.”

Now perform the following search & replace:

Ok, maybe this example makes things more confusing. If in doubt, just remember: $ = bad, \ = good in Xcode Find panel.

As to why Xcode does not use the ICU syntax in the replace strings, beats me. If anyone has a simple explanation, send it my way though I also welcome apocryphal anecdotes and crackpot conspiracy theories (extra points if you can convincingly implicate the Trilateral Commission).