Noodlings

Let’s talk about APIs. Like UIs, APIs provide an interface (hence the “I”) to a class of users. In this case, the users are programmers. Because of this, many times you have APIs that are cluttered and overly difficult to use since programmers can probably figure that stuff out. It’s unfortunate as many just see APIs as a way to expose functionality without any regard for design.

I feel a hallmark of a good UI is one that avoids presenting the user with unnecessary decisions. The more you can either filter or figure out on the user’s behalf, the better. I’ve touched upon this before: 1 2

I feel the same holds true for APIs. APIs should do more than just provide functionality. They should strive to do more for the user without burdening them with unnecessary details. Provide power with little effort on the user’s part. Let the programmer worry about the “what” and not about the “how”.

This leads me to my favorite API: NSArray

Yes, NSArray.

What is an NSArray? It’s an ordered collection of objects. Doesn’t sound terribly exciting but ask yourself this: what data structure does it implement? If you ask around, you’ll get a bunch of different answers but the real answer here is “It doesn’t matter.”

Now, you may be thinking that NSArray is insufficient to cover all the different cases where you’d need an ordered list. Surely, you have to use a skip list in this one case or a circular queue in this other one. Before going further, I highly recommend reading this oldie but goodie (and one of my favorite blog posts): Array.

As it turns out, NSArray is adaptive. It’s not implementing a single data structure but instead changes depending on usage. Every time I, or a colleague, has implemented their own special data structure, it has ended up performing no better, or sometimes even worse, than NSArray. Now, I’m sure you can come up with some special set of circumstances where you’d need to roll your own, but from my experiences with Cocoa over the years, I’d say that for 99% of the cases, NSArray is fine.

Another example of adaptive behavior is Introsort, which will use either quicksort or heapsort depending on the circumstances. Quicksort has great average performance but has a horrible worst case (O(n²)). Introsort is able to detect when things are headed towards that worst case and switches to heap sort instead (worst case: O(n log n)). No need for the user to evaluate the data to figure out which sort algorithm to use.

Now, sometimes things are inherently complex (threading comes to mind). It does take a bit of care and experience to make sure that instead of making something simple, you end up making it simplistic. Nonetheless, I feel that NSArray hits that sweet spot of providing what you need while at the same time cutting out what you really don’t.

Hopefully this shows that APIs can be so much more than a simple wrapper around some code. The use of proper abstraction as well as going the extra mile on the user’s behalf can go a long way towards making a merely functional API something wonderful.

I’ve added support for JSON Feed to this blog. You can find a link to it under the “Meta” section in the lower right corner.

It’s very early and hardly any clients support it yet but it was easy enough to add support for it on my end. I felt like I should try being the chicken in the chicken-and-egg problem that comes with a new standard being introduced.

Let me know if you run into any problems with it.

Things have settled down with my 4.1 release so I thought I’d tend to my neglected blog.

Today, I’m going to talk about getting info about your iTunes library, playlists and albums.

Now, the obvious way to do this would be through AppleScript/AppleEvents. The problem with this approach is that it requires iTunes to be running for it to work. To actually change stuff in your iTunes library, you will have to use AppleScript but for the purposes of this article, we are just looking at ways to get information about it.

The old way to do this was to parse the iTunes Library.xml file (located in ~/Music/iTunes). This file contains a dump of everything you need. Unfortunately, recent versions of iTunes do not generate this automatically; you have to enable it in its preferences (it’s in the “Advanced” tab).

Now, the notion of having to parse an XML file does sound painful but luckily it’s actually a property list so you can load it that way and access everything as NSArrays, NSDictionaries, NSStrings, etc.

With OS X 10.9 (yes, I know it’s macOS now but back then it was still OS X), Apple introduced the MediaLibrary API (ML). This provided a way of not just accessing the iTunes library, but the libraries of all of Apple’s other media apps (Photos, GarageBand, even Final Cut). No parsing involved but it is a fully asynchronous API so expect to deal with a lot of callbacks.

The last one (I don’t know exactly when it was released but it was the last one I discovered) is iTunesLibrary (ITLib) and it’s a bit easy to miss because, from what I can tell, it is not a part of the OS, but iTunes itself. You can find the framework under /Library/Frameworks (not /System/Library/Frameworks). It provides similar functionality to the MediaLibrary framework, but specific to iTunes.

With so many options, which do you use? For the most part, they return the same information though it depends on the version of iTunes. With some versions, one API may return some special playlist that the others don’t. Also, ML is asynchronous while the others are synchronous. I don’t see that as a big deal either way as you can convert one into the other with a tiny bit of work.

The big differentiating factor is performance. Here are the results for getting a list of playlists for my library:

XML: 1.664939s
ML: 18.792855s
ITLib: 0.844415

I don’t think my library is the biggest in the world but it is significant (about 2000 albums). As you can see, ML is pretty damn slow with ITLib being much, much, much faster. What is surprising to me is how much faster parsing the XML file is than ML. It is possible, though, that I’m holding it wrong.

So, ITLib is my recommended API for this. It’s what Hazel uses starting with 4.1 (previously, it was parsing the XML when it was available and falling back to ML). That said, there is one wrinkle.

The persistent IDs of entities in iTunes are hex strings. In the XML file, they are zero padded. In ML, though it returns strings, they are not zero padded (rdar://26624642 for you Apple folks watching from home). To add even more confusion to the mix, they are NSNumbers in ITLib.

If you are using any of these APIs in isolation then it’s probably not a big deal, but if you want to later manipulate that entity using AppleScript (like importing files into a playlist), you will need to provide the zero-padded hex string. Something to keep in mind.

Today marks the 10th anniversary of Hazel. On September 6, 2006, Hazel 1.0 was released and after all this time, it’s still going strong. I’ve already shared my thoughts on why I took this journey and how I managed to stay on this road so I won’t rehash that here. What started as a product developed in my bedroom is, well, still a product developed in my bedroom, but now I can say it’s a successful one.

Hazel back then was a much more basic product. You can check out the release notes page, which documents everything back to the beginning. It’s interesting to see what features were added later, many of which are probably considered essential now.

As for the future, who knows? While the list of big features I want to implement is much shorter now, there are still a bunch of smaller things I want to put in. I can’t say whether I’ll still be doing this in another ten years but I have no intention of quitting now.

And with today being the anniversary, I need to celebrate in some way so, from today until 3am (EDT) tomorrow (Sept. 7), I am offering Hazel at its original price from 10 years ago. That’s $16 (50% off). The family pack and upgrade didn’t exist back then but they’re 50% off as well. No special code or link. Just go to the store and buy. If you don’t have a copy yet, here’s your chance.

I’ve kept my hat out of the discussions of Swift vs. Objective-C for several reasons, but the main one is that frankly, I’ve been busy. I ship a product, one that I would consider successful, and recently shipped a major release. What does that mean? It means that debating these types of issues and playing with the latest thing is a luxury. Because I have a shipping product with a lot of users, I can’t just change things unless I find that it adds value to my users. My job is to solve my users’ problems. Now that I have a little bit of time to do so, this is one lens I’m hoping to add to this discussion

In addition, years ago, when Java was introduced, I jumped on that bandwagon, leaving behind Objective-C. It started out writing replacements for Foundation and AppKit in Java and ended with writing large scale servers. I was on that wagon for a good 10 years before returning, so I wasn’t a tourist there. This is yet another lens I also want to add to the discussion as history seems to be repeating itself and many younger people seem to be missing out on this part.

But let’s start with the current issue about dynamism since that’s the hot topic.

One thing many people seem to overlook about the dynamism of Objective-C is that it enabled NeXT (and Apple) to provide better GUI tools. Using dynamism, they were able to make GUI building declarative in nature. Connect this to that. Call this method. All stored in a file that was (and still is) data, not code. Competitors at the time (and today) resorted to code generation which is fragile and, ironically, unsafe. Yes, you could have a more declarative file format, but implementing that in using a static language required a lot of hard-coding and switch statements. Not the elegance that many people claim to be moving towards.

I’m not saying that a language has to be purely dynamic but it shouldn’t be purely static either. It think it’s spurious not to credit a level of dynamism for the quality of apps on Apple platforms over the years, and to be pedantic, the NeXT ones as well – many of which were considered the best on any platform at the time. To deny that, I feel, shows a lack of understanding of what has made the platform great all these years.

Moving from a dynamic language to a more static one seems refreshing at first. I went through the exact same thing when switching to Java. Yes, I can hear some of you grumbling “Java” out there but the irony is that there are more similarities in the evolution between Java and Swift than you may want to admit.

Everything seemed so much “tighter” in Java. Interfaces (Java’s version of protocols) everywhere! No more pointers! Checked exceptions! It’s all so new and exciting. You might just have to take my word on this, but any new language that you really like will always seem to make you “more productive”. A colleague and I (plus maybe one, or one and a half, devs who were on the periphery) implemented our own version of Foundation and AppKit in what I would consider a pretty short amount of time. Other people at Lighthouse can correct me on this but my recollection was that it was on the scale of months, not years. You can read up more on it here.

Was it the language? At the time we might say yes but in hindsight, I’d say it was just that we were just super excited and into the project. I was working with an exceptional team and being a new platform, we were excited about making our mark on it. Yeah, there were bits about Java that were better than Objective-C. But there were bits that were worse. The point here, though, is that until the honeymoon is over and you are maintaining a project over several years, it’s hard to make real proclamations about productivity. Truth be told, I’d say the biggest impact on software quality and productivity is the quality of engineers regardless of any other factors.

Looking back on my years with Java, I find the biggest problems with the platform was this idea that everything can be done by the compiler. It sounds great on paper but when implemented, it resulted in all sorts of contortions to work around not letting any dynamism in. It resulted in a lot of glue being written that I wouldn’t have to write in a more dynamic platform. It resulted in cascading changes across code to make trivial modifications just to appease the compiler. Maybe in a future post I’ll go into specific language features that I found problematic and show parallel trends/features in Swift but let’s just say that less is more. It’s an aesthetic that I always felt that put Apple above the rest and something which I feel like in some ways is being abandoned.

Now, I’m sure many of you will just paint me as an old, stubborn Obj-C luddite. And it’s true on some level but that doesn’t mean you should discount my experience. Many of us old fogeys are working on products that have shipped for years. That means that (1) we are solving user problems (2) have immense experience with doing app development (3) really understanding the frameworks (4) have been maintaining a code base.

Let’s go through these points:

1. We are solving user problems.

We have real problems to solve for people outside of ourselves. We need to be practical to survive. Users don’t care about the tech underneath. We have to ship them solutions. This oftentimes requires rapid response. This also oftentimes involves working around Apple bugs, bugs that never get fixed and will only grow in number as Apple maintains their yearly release schedule (something to think about for proponents of the “final” keyword). Agility and flexibility are paramount here. Reality is not pretty. The user could care less how beautiful your code is if you aren’t solving their problems.

2. We have immense experience with doing app development

I want to emphasize the “app” part of that. A common argument for very strict type safety are for cases which aren’t apps. The predominant software in the Apple ecosystem is end user applications. Let’s try and not lose focus here and argue the case for categories of software that aren’t Apple’s forte. If you are writing some life-critical software on an Apple platform, god help you. Even with Swift, you are stuck with Xcode and Apple’s insistence on yearly new OS releases.

3. We really understand the frameworks

I would take many of the arguments for Swift more seriously if there weren’t strawman examples set up on the Obj-C side. You’ll have to dig up Brent Simmons’ posts on this, but when I see the kooky examples of horrible old Obj-C way, what it tells me is more that you don’t understand the frameworks. It resembles informercials where people are fumbling opening a jar and then are presented with some useless gadget to solve a non-existent problem. I’m sure things can be done better than they are now but providing bad examples indicating ignorance of what it is you are trying to replace doesn’t help your argument. And I have seen plenty of examples of new people replacing what they don’t understand. Please don’t be one of those people.

4. We have been maintaining a code base for a while

I have been writing Hazel for ten years. Many of us old-time app developers are in similar positions. Speaking with many other devs, most of whom write iOS apps, I’d be lucky to find those that have maintained a code base for a third of that time. Sure, there are parts of the code that are ugly, but that happens on any platform. Same thing happened on Java. No matter what you do, time is always against you. When I was working with Java, I found the technical debt grew in a different fashion. It was hard to change anything. This was because changes in stricter languages tended to cascade. Because everything has to declare everything about itself, changing code turns into a Facebook feed, except where you and everyone else are obligated to respond to every stupid pic and forward of Snopes-bait. With a more dynamic code base, I can say I have less code to maintain and that I can be more agile. Not that it’s without it’s own problems. But let’s not act like type-safety is a panacea.

And to give more of my personal perspective on the language debate, I’d love to have something better than Objective-C. But I’d also like to have something better than Swift. My personal pet peeve is that we are in the 21st century and we still have to give a shit about the storage type of numbers. I’d be happy to get rid of scalar primitives entirely.

Ultimately, though, I need something that solves my problems. My problems aren’t performance. They aren’t type-safety (maybe it’s just me, but I rarely have issues with it and when I do, the time it takes to fix it is far less than the time specifying everything to a tee everywhere else). They aren’t being able to write clever VDLs. For me, it’s writing apps that solve my users’ problems and getting them out in a timely fashion. As it stands now, Swift (at least pure-Swift, or even current Swift as a non-ABI-stable moving target) does not do that for me.

Now maybe these same problems can be solved in a static way but what I’m not seeing from the static-camp are (decent) solutions. What I’m seeing are either hand-waving or the same crufty code-generation, write tons of repetitive boilerplate type of solutions that I had hoped we had left behind in the 90s.

What I do see makes me worry that it’s not the experienced app-writers that are being heard. Unfortunately, many of us are too busy to sit in the various mailing lists and forums. Yes, ok, we lose. But we all lose if those creating the platform don’t draw from the experience of those who have built upon it successfully. As someone who has done both (write frameworks and apps) and seen what happens when the former ignore the latter (the rise and fall of Java), I’m hoping history doesn’t repeat itself.

That’s my somewhat meandering brain dump for the moment as I do have to get back to working on my next patch. Sure, these are just my opinions but my hope is that we can at least look for more balance in our approaches to solving actual problems.

Yes, another journey into the world of email hosting. This time it’s a bit different. Instead of receiving email, today we talk about sending it.

It seems that during my recent launch I became collateral damage in the war against spam. Between a couple of email campaigns and a bunch of license emails, I had sent out a good amount of email; enough to cause me a ton of headaches as certain mail services, actually, mainly one: Gmail, decided to mark a lot of it as spam (if I was lucky) or make them disappear entirely.

The way things were set up, my email campaigns were sent out by Campaign Monitor, my license emails by my server and my regular “interactive” emails via Rackspace’s email service (as described in a previous installment). The ones I had the most problems with were the first two, as they were repetitive messages sent out in volume.

For a while, I’ve had SPF records set up. What are SPF records? In a nutshell, it’s a way for you to specify which mail servers are the “official” servers your email comes from. This is to help identify mail coming from you as opposed to a spammer posing as you. The way you set it up is to create special DNS TXT records listing out the specific servers for your domain.

Apparently, this isn’t enough. Seems like there’s another layer you can implement: DKIM. With this, you have your mail server sign outgoing emails so mail servers at the receiving end can know that emails are definitely from you and definitely not, once again, from a spammer imposter. So, I went ahead and set up OpenDKIM on my server. You can find various guides out there on how to install it on your OS and integrate it with your MTA (I use postfix and hooking the two up was pretty easy). You also have to add a DNS record listing your public key so other mail servers can verify your signatures.

Even after doing that, it still didn’t seem to appease the Gmail gods. I found this page which recommended yet another thing: DMARC. Here, you specify a policy as a guideline to mail servers on how to handle your email. One of the things you can specify is an email address where mail services can send you reports on the emails you send them. And you guessed it, you implement it by creating a DNS record.

Being desperate, I thought I’d do it, hoping that Gmail would send me a report telling me what I’m doing wrong. Soon after, I started receiving DMARC reports from all sorts of mail services (Microsoft, Yahoo, AOL, etc.). Over ten days later and guess who still hasn’t sent me one.

I’ve been getting fewer reports about not receiving emails lately but that’s mainly because of decreased volume since launch. It’s still unclear whether Gmail is binning my emails at a high rate or not. Nonetheless, if sending out emails are an important of your business, I recommend doing the above. Even if Gmail seems to be hard to please, other mail services are more appreciative of the gesture.

Note that there is also the option of relaying all my mail through Rackspace. It’s still a possibility but (a) I’m afraid of poisoning the well since my email is already being marked and (b) using a shared relay opens you up to being blacklisted because of someone else’s misdoings. All in all, I feel that some level of redundancy is ok here.

When implementing the above, you can check the headers of an email received at the other end to make sure everything is set up properly. Here’s one from an email sent from my server to my Gmail account:

Authentication-Results: mx.google.com; dkim=pass header.i=@noodlesoft.com; spf=pass (google.com: domain of www-data@noodlesoft.com designates 2001:4801:7824:103:be76:4eff:fe11:5179 as permitted sender) smtp.mailfrom=www-data@noodlesoft.com; dmarc=pass (p=NONE dis=NONE) header.from=noodlesoft.com

As you can see, it shows that my SPF and DKIM passes. That doesn’t guarantee anything but it helps.

You can also check out Google’s Postmaster tools site. It will give you feedback on various metrics concerning email from your site. To set it up, you have to create a DNS record (see a running theme here?) with text it supplies you so that it can see that you control the domain. After that, it will track your site.

Also, yes, DNS once again: make sure all your regular DNS records are set up properly. Not only do you want an A and PTR record for IPv4, but also a AAAA and PTR record for IPv6 as more mail servers nowadays are checking for that.

Until next time, here’s hoping I don’t have to resort to human sacrifice to get Gmail to accept my messages.

A little pet peeve of mine are the turd files that Emacs leaves behind all over the place.

For those that don’t use Emacs, when you edit files with it, it will keep a backup of the original file in a file with the same name, except with a tilde (~) at the end. The problem is that you have to manually clear them out. When you are jumping around in Terminal editing config files left and right, trying to get something working, you tend to forget where those files are. And yes, you can turn that off, but I like having the backups and yes, you can specify that it store its backup files in a common place but it makes it harder to recover files should I need to check the backup version, especially when I’m editing a bunch of files with the same name (index.php anyone?).

Since Hazel before version 4 only operated on actual folders/directories in the filesystem, it couldn’t really handle cases like this where the files are strewn all over the place. Now with Smart Folder support, you can create one to match all the turd files and have Hazel clean them up for you. Here’s a Smart Folder and Hazel rule you can use yourself:

Turd Cleanup

One important bit about the Smart Folder conditions is that I use “Filename” and not the more readily available “Name” or “File extension” attributes as the ~ is tacked onto the end regardless of whether there is an extension or not.

And of course, no reason to limit it to Emacs turds. Go ahead and edit the Smart Folder to include whatever other poops you have on your system.

I’m finally going to start transitioning off of Feedburner. I haven’t done this before so I stand a good chance of screwing it up. From what I understand, when I switch things over, Feedburner should do a redirect to the new feed. If you find that it doesn’t work, you can always re-subscribe by going to the site.

As for where I’m taking the feed, I’m going back to hosting it myself via WordPress. There seem to be plugins that do the minimal statistics collection that I care about and I’m tired of using services that just end up shutting down.

Hopefully, you won’t notice a thing but I thought I’d post just in case something goes wrong.

It’s been a bit hectic because of the launch yesterday but I finally have a moment to post. Yes, Hazel 4 is finally out. You can find the release notes here.

Development was a bit rocky. I played with a bunch of different features but some of them didn’t quite pan out in a way that I liked. It felt like wasted effort in that that work didn’t result in a usable feature but many of them were only shelved temporarily. Oftentimes I end up having that eureka moment which would allow a shelved feature to be finally realized so something to look out for in future point releases.

That said, I’m happy with the features that I did get to work. They seem simple on paper but involved a bit more thought than would be expected. Sync is always tricky and getting the preview feature down to something as simple as it is now took a little doing.

Along with that was the site re-design (courtesy of the folks at Brotherhood). The previous site was mostly static. Adding content involved editing raw html pages and adding them. It was enough to discourage me from doing it often and discourage me it did, as I ended up leaving the site very outdated. The new site is backed by WordPress which will hopefully remedy that. The point here being that content can be added more easily using tools like MarsEdit or WPs web editor. I’ve already added a few posts (a review and a couple of knowledge base articles) since the launch.

Also, the new site design is a bit more stripped down and streamlined. I’ve tried to reduce navigation in favor of search. Most of the site is searchable via the form on the support page so I recommend going there first and doing a search if you ever have a question about Hazel.

The launch itself went relatively smoothly (except one incident – more details below). I can credit most of this to using a VPS (virtual private server). VPSes are great as you can clone, rebuild and resize them as needed. It gives you an amazing amount of agility when deploying servers.

Before the launch, I set up a clone server so I could set up and test the new site. Since it’s a clone, no need to reinstall and reconfigure everything (though you do have to make some changes in places where the IP address or hostname is stored). You end up with pretty much an identical server to play with which went a long way towards making sure things were working properly.

When I launched, I transferred the new stuff over to the live server. That part went with little drama but then disaster happened: I underestimated the load from tons of eager customers. The problem was that I had sent two email campaigns. One to those on my mailing list and another to those who purchased recently. The latter group received a message with instructions on how to get their free upgrade license. And guess what all those people decided to do immediately upon seeing that.

The result was that the site got slammed. More specifically, apache was overloaded. Enter VPS awesomeness #2: I was able to resize the server on the fly. It took a little while (maybe 15 minutes though it felt much longer) but the old server was still able to run, albeit very sluggishly, until the last minute when the conversion finished and it rebooted. After that, the site ran like butter and it was smooth sailing (at least as far as the server went).

Aside from some minor glitches (version 4.0.1 released this morning should address some of them), the launch has been pretty great. I just had the best day in sales in Hazel’s history so I’m pretty happy about that. My thanks to everyone who contributed, including Brotherhood, Jono Hunt for his icon and UI work, my beta testers, my friends in the Mac dev community and of course, all my customers who’ve been very supportive of Hazel all these years.

In case you haven’t been keeping up with the Twitter feed or the forums, Apple introduced a glaring bug in Gatekeeper in 10.11.4. Any non-app bundles (that includes preference panes, screensavers, plugins, etc.) are rejected by Gatekeeper regardless of whether they are signed or not. For those of us that ship such bundles, this is what I’d consider a big deal.

The workaround itself isn’t so bad: either right-click and select “Open” or drag the pref pane into System Preferences. For users curious enough to email support, it’s not such a bad thing as I can suggest one of the workarounds. The problem it gives new users installing it for the first time a bad impression of the software, not realizing that it’s actually Apple’s fault. I can’t really measure this type of thing but I think it’s safe to say a good number of them just nope out of there without contacting me. The result is that my software has an association with being untrusted.

Daniel Jalkut gives a more detailed account of the mechanics of it here so I won’t go retread that ground.

After filing a DTS incident, Apple confirmed it as a bug (I’ve filed rdar://25466753). There was no information about when a fix would be made available and given that Apple is not known for being nimble about these things and that I was losing customers, I followed the advice of some colleagues and took matters into my own hands.

The result is that starting with Hazel 3.3.8, it will ship with an installer app. The installer app still goes through System Preferences as I still think that its installation process works well. Note that an installer package was also an option but I couldn’t figure out if/how to make it use a previous install location if the software was already installed (If anyone knows, I’d be happy to hear about it just in case I have to resort to it in the future). Hopefully now I can direct my efforts back to getting Hazel 4 shipped.

One can debate how much Apple cares about non-app-bundled software but when the workaround is to suggest people bypass Gatekeeper, they should be very concerned. False positives only erode the confidence people have in your security systems and you don’t want them to get in the habit of casually bypassing them.