Noodlings

As mentioned in my last post, I had various issues with codesigning and notarization during Hazel 5’s launch. Now before I get into it, remember during all this that Hazel is codesigned and notarized with no warnings or errors. In my release build process, I do a codesign —verify as well as spctl to make sure everything is hunky-dory before submitting to Apple for notarization.

The biggest problem at launch was some users getting an “Unidentified developer” alert when opening the dmg. I had various users send in logs, but it was only when someone found a log message pertaining to the rpath for one of the binaries in the bundle that I was able to identify the problem. Strangely enough, that person didn’t receive the “Unidentified developer” error alert.

Turns out, I had mistakenly added a binary to the top level bundle (it was originally in a framework). The binary had an rpath which escaped the framework bundle as it depended on a sibling framework. This worked fine when the binary was in the framework bundle but when it was at the top level of the main app bundle, it escaped the app bundle itself. Not sure how the binary got added to the Copy Files phase there but it was an easy fix to remove.

Another issue was that the installer wasn’t triggering for some people. For the installer to trigger, Hazel needs to detect that it is running from a disk image. Further investigation turned up that Hazel was being translocated. If you don’t know what translocation is, I suggest reading this article: https://lapcatsoftware.com/articles/app-translocation.html

When translocated, the binary is no longer on the disk image, instead it is copied to a temp location on disk. Hazel is unaware of this and as a result, doesn’t run the installer. Why was Hazel being translocated? I’m still not sure. It’s my understanding that if an app and its containing dmg is signed and notarized, it shouldn’t be translocated. As a workaround, I ended up using the private SecTranslocate.h APIs to detect the translocation and compensate appropriately.

Possibly related to the above, there was also an issue with the embedded helper app not running. Logs from users showed that the quarantine flag was still set on the helper and that was preventing it from being run. When the user copies an app, like say from a disk image to /Applications, the quarantine flag should be cleared for the app and everything inside but for some reason it was not clearing it for the embedded binaries. Note that unlike when a user launches an app from Finder where they will be asked to run the app, a login item helper will fail to launch without any prompt. As a short term workaround, on first run, I had Hazel recursively remove its quarantine flag before it attempted to run the helper.

Even with the workarounds, something didn’t sit right with me. I suspected there was something wrong with my bundle that was causing these issues. I ended up filing a DTS incident to get some feedback from Apple. The DTS engineer was able to reproduce the issue on a VM and dig up a log message indicating that it could not clear the quarantine flag on the Autoupdate helper app in the Sparkle framework. For those that don’t know, Sparkle is a very commonly used framework for apps to update themselves. The Autoupdate helper runs when your app is terminated so that it can install and run the new version.

This all lead to an investigation into where should Autoupdate, or executable binaries in general, be put in a framework. Now, I had looked into this once before . Sparkle originally put Autoupdate in Resources which is incorrect and can cause subtle issues. The solution I came up with back then was to put any executables alongside the framework binary, which would be in Versions/A with a symbolic link at the top level of the framework bundle. This seemed to work at the time. It got through codesigning and notarization checks without issue and ran fine after that.

But it seems that the problem is more subtle than that as whatever code that clears quarantine flags on embedded bundles didn’t like that. I was pointed to this article which suggests I put the program in Contents/Helpers. Despite the insistence that this was the way to do it, it flat out did not work. An important thing to note is that framework bundles do not have a Contents directory. Putting a Contents directory, either in Versions/A or at the top level will make codesign barf, complaining about an invalid bundle format. My guess is that when codesign sees a Contents directory, it considers the bundle to be another (non-framework) type of bundle. After wasting a good bit of time on this, I went with what I thought made more sense, putting it in a Helpers directory (sans Contents). If you have an Xcode “Copy Files” phase copy to Resources for a framework, for instance, it doesn’t put it in Contents/Resources, but in Resources so it made some sense to do it this way.

Sure enough, that worked. Of course, after fixing it, someone pointed out how some of the system frameworks use the Helper directory. I guess I should’ve checked there first. At least I have some confirmation that I was on the right track. Note that you should put the Helper directory under Versions/A. You can symlink to the top level but that’s optional. I did it mainly to give myself a more consistent path to access it in the very unlikely event that I end up shipping different versions of the framework.

Overall, this experience has been very frustrating. There is little to no documentation on all the various things that can go wrong in your bundle that will cause things to fail, and when it does fail, the logging is very inconsistent. Lastly, none of the tools or processes in place (codesign, spctl, notarization) catch these cases. These are all issues related to the static structure of the app bundle so it seems like they should be detectable. Having a tool that developers can run on their bundles before submitting them would be very helpful as the current ones are far from complete in that regard.

After all this, I’m still not sure if the translocation problem is still there or not. I’m not about to remove a workaround just to have it fail for users just so I can get some feedback so that question will remain unanswered for the time being. The lack of definitive and correct information during this whole ordeal is a bit disturbing and makes me think that there’s no one that actually knows for sure how things really work.

For those at Fruit Company, I’ve filed FBAs FB8981011 and FB8981016.

Recently, I took some time to do some clean up on my Xcode project as it has collected its share of odd scripts and settings over the years. Little did I know that I’d make a couple of discoveries surprising enough that I’d actually bother to blog about it.

Speeding up builds

The first thing I wanted to address was build speed. Over the years, it’s felt like build times have gotten longer. Of course, before we start mucking with things, we needed a way of measuring which leads me to the first tip: you can have Xcode display build times in the main activity display in the titlebar. Just use this default:

After doing that and performing a build, it should look something like this:

I tried tightening things up: getting rid of unnecessary steps, removing redundancies, making sure all script phases had inputs and outputs specified. They had minimal impact. Looking at my build logs in Xcode (which helpfully show the time for each step), I noticed that code signing was taking up a significant amount of time. Around 3-8 seconds each occurrence, even for standalone binaries. I have quite a few of these as well as frameworks so that time adds up. Note that for debug builds, I had the “—timestamp=none” option set so that was not the culprit in this case.

Poking around, I came across this thread. While I didn’t have the main problem described there of duplicate certificates, buried in that thread was the following advice: trim ~/Library/Preferences/com.apple.security.plist 

Opening that file up revealed that I had several entries, all except one pointing to non-existent files with the one valid entry pointing to my login keychain. After removing the invalid entries, code signing only took up to 1 second, max. This shaved 40-60 seconds off of my full release builds and 10 seconds off of incremental ones. Huge savings.

Cleaning out frameworks

Another thing I noticed in the cleanup was that some of my frameworks were being copied without their headers. I had a script of my own to remove the headers after copying for deployment builds, but the frameworks remained header-free even when I disabled/removed this script. To make things even stranger, this was was only happening for one target as another target would always have the headers in included frameworks. I checked all sorts of settings but it was only after asking in a Slack channel did I find the answer. Thanks to Nicholas Riley, who didn’t miss a beat and pointed me to this thread.

Apparently, there’s a hidden setting in your project.pbxproj file for copying frameworks where you can specify whether headers get copied over. This is not exposed anywhere in Xcode’s UI, as far as I can tell. It’s also a mystery why it gets set on some targets and not others. The only way to enable/disable this is to edit the project.pbxproj by hand. 

The setting looks something like this:

The flag in question is RemoveHeadersOnCopy. It appears the setting sticks even if Xcode writes to the file after your edit. Of course, if you add new frameworks to copy, you may need to edit the file again.

Hopefully the above helps you out with your builds. If you have any eye-opening tips to make your build process quicker/better, comment below as I’d love to hear them.

Yes, another journey into the world of email hosting. This time it’s a bit different. Instead of receiving email, today we talk about sending it.

It seems that during my recent launch I became collateral damage in the war against spam. Between a couple of email campaigns and a bunch of license emails, I had sent out a good amount of email; enough to cause me a ton of headaches as certain mail services, actually, mainly one: Gmail, decided to mark a lot of it as spam (if I was lucky) or make them disappear entirely.

The way things were set up, my email campaigns were sent out by Campaign Monitor, my license emails by my server and my regular “interactive” emails via Rackspace’s email service (as described in a previous installment). The ones I had the most problems with were the first two, as they were repetitive messages sent out in volume.

For a while, I’ve had SPF records set up. What are SPF records? In a nutshell, it’s a way for you to specify which mail servers are the “official” servers your email comes from. This is to help identify mail coming from you as opposed to a spammer posing as you. The way you set it up is to create special DNS TXT records listing out the specific servers for your domain.

Apparently, this isn’t enough. Seems like there’s another layer you can implement: DKIM. With this, you have your mail server sign outgoing emails so mail servers at the receiving end can know that emails are definitely from you and definitely not, once again, from a spammer imposter. So, I went ahead and set up OpenDKIM on my server. You can find various guides out there on how to install it on your OS and integrate it with your MTA (I use postfix and hooking the two up was pretty easy). You also have to add a DNS record listing your public key so other mail servers can verify your signatures.

Even after doing that, it still didn’t seem to appease the Gmail gods. I found this page which recommended yet another thing: DMARC. Here, you specify a policy as a guideline to mail servers on how to handle your email. One of the things you can specify is an email address where mail services can send you reports on the emails you send them. And you guessed it, you implement it by creating a DNS record.

Being desperate, I thought I’d do it, hoping that Gmail would send me a report telling me what I’m doing wrong. Soon after, I started receiving DMARC reports from all sorts of mail services (Microsoft, Yahoo, AOL, etc.). Over ten days later and guess who still hasn’t sent me one.

I’ve been getting fewer reports about not receiving emails lately but that’s mainly because of decreased volume since launch. It’s still unclear whether Gmail is binning my emails at a high rate or not. Nonetheless, if sending out emails are an important of your business, I recommend doing the above. Even if Gmail seems to be hard to please, other mail services are more appreciative of the gesture.

Note that there is also the option of relaying all my mail through Rackspace. It’s still a possibility but (a) I’m afraid of poisoning the well since my email is already being marked and (b) using a shared relay opens you up to being blacklisted because of someone else’s misdoings. All in all, I feel that some level of redundancy is ok here.

When implementing the above, you can check the headers of an email received at the other end to make sure everything is set up properly. Here’s one from an email sent from my server to my Gmail account:

Authentication-Results: mx.google.com; dkim=pass header.i=@noodlesoft.com; spf=pass (google.com: domain of www-data@noodlesoft.com designates 2001:4801:7824:103:be76:4eff:fe11:5179 as permitted sender) smtp.mailfrom=www-data@noodlesoft.com; dmarc=pass (p=NONE dis=NONE) header.from=noodlesoft.com

As you can see, it shows that my SPF and DKIM passes. That doesn’t guarantee anything but it helps.

You can also check out Google’s Postmaster tools site. It will give you feedback on various metrics concerning email from your site. To set it up, you have to create a DNS record (see a running theme here?) with text it supplies you so that it can see that you control the domain. After that, it will track your site.

Also, yes, DNS once again: make sure all your regular DNS records are set up properly. Not only do you want an A and PTR record for IPv4, but also a AAAA and PTR record for IPv6 as more mail servers nowadays are checking for that.

Until next time, here’s hoping I don’t have to resort to human sacrifice to get Gmail to accept my messages.

Over the weekend, Gus Mueller turned me on to the LLVM/Clang static analyzer. And just in time, too, as I was polishing up my 2.2 release (which went up earlier today).

It’s an offshoot of the LLVM and Clang projects (read the respective pages on what they are if you don’t know already). The static analyzer analyzes your code and looks for problems, focusing mainly on memory allocation patterns, in this case, including Objective-C/Cocoa (retain/release/autorelease) and CF (CFRetain/CFRelease) semantics.

Take this contrived example for instance:

  id foo()
  {
      NSArray       *array = [[NSArray alloc] init];

      if ([array count] > 0)
      {
        return nil;
      }
      return [array autorelease];
  }

The example above will get you a report like this (it generates html):

Drilling down you get this (still in html):

(click to enlarge)

Here you can see it pointing out [1] where the object was allocated [2] the branch it took and [3] the point where you leaked it. Pretty neat. It tries to follow every possible branch finding paths where you may have leaked an object. It also finds what it calls “dead stores” (when you assign a value to a variable but never use it again) and missing dealloc methods

As the project page says, it is very early in development. You’ll find that it does turn up a lot of false positives, especially with the missing deallocs. False positives for memory leaks seem to occur when you release something in a different scope than where you created it. For instance, I have this chunk of Apple code that wraps CFRelease() with it’s own function that checks for NULL first. The checker complained about this every time. Nonetheless, it did turn up some real leaks for me.

Aside from reducing the number of false positives, I’d also like to see the entries grouped by source file (it’s annoying jumping around between files) as well as some way to bring up the original source file by clicking on its name in the source code view. You will also see multiple entries for the same leak when the code traverses multiple paths that end up with the same leak which can be annoying.

In any case, I recommend downloading it and giving it a try. I’m not sure how thorough it is (i.e. whether it can supplant running your program through MallocDebug/Instruments/leaks) but it makes a great additional tool to add to your arsenal. Chances are it will look at some code path that you don’t test. Oh, and a couple tips:

• Make sure you do a clean build on your project first. The checker only runs on files that would normally be compiled (it sits in as your compiler). If your project is already built, then no files will be compiled/analyzed.
• Use the -V option, which will pop open a browser with the analysis page when done. Normally, it sticks the files somewhere under /tmp but only shows the actual path when you start the run. Needless to say, that bit of text scrolls off pretty quickly.
• While the tool does come up with false positives, you’ll find that sometimes it finds something subtle that you may blow off as a false positive on first glance. Make sure you understand what it is flagging, even if it ends up being wrong.

I haven’t used it with a garbage collected program so I don’t know if it uses different techniques in such a case or is just plain unnecessary. Maybe the dead store detection becomes more important. Reports from anyone using this with GC are welcome.