Preference Panes and Catalina

August 28th, 2019 — 1:09pm

Apple announced numerous changes in Catalina in WWDC but there’s one that they failed to mention. System Preferences has had a major change: preference panes now load in a separate process. Apple ones get their own while all the third party ones get stuck in a process called “legacyLoader (System Preferences)”.

It’s becoming more common nowadays to split up programs into separate processes to isolate them from each other, usually done in the name of stability and security. The upshot of this to me is that all display and events happen via IPC between my pane and System Preferences. If done right, this should all be seamless and I shouldn’t notice a thing:

Window2

Of course, it’s never that simple.

First off, dark mode is broken. You can test this out by creating a blank preference pane in Xcode and loading that. For you Apple folks, that’s FB7076716. Interesting side note: problem doesn’t manifest itself if SIP is disabled.

The other problem is that magenta. What is that? Apparently, it’s a security feature. Any windows besides the main preference window will have their transparent areas colored magenta. I’ve been trying to pin down the rules as to when this occurs but it’s been slow going and also the screenshot above contradicts what I’ve been told. Note that my software is notarized so it’s not like this is only applying to unknown software.

To top this all off: none of this has been announced or documented. I’ve only found out about this through backchannels and then later, via a “conversation” via Feedback Assistant (FB6758586). It’s a big change and requires some major overhauls on my part. Aside from leaving things as they are, my options are to either rework my whole UI to avoid using any extra windows or convert my preference pane to an app. The latter is a long term goal but neither of those are realistic for doing over the summer between WWDC and Catalina’s launch. And mind you, that’s assuming they actually announced and documented this stuff up front (which they didn’t). How Hazel is going to look on Catalina is very much up in the air right now.

Maybe this will help other people writing pref panes as info on this is nowhere to be found (if I’m wrong on this, please send a link my way as I dying for any clarification on this). I don’t know how much of this Apple will fix as a good part of it is meant as a feature but one can hope that it will somehow sort itself out in the end.

Comment » | Hazel, OS X, Software, User Interface

Frameworks and Auxiliary Binaries

April 4th, 2019 — 7:09pm

In my frameworks, I have extra programs that my framework code needs to use. Before, I would have these auxiliary programs outside the framework bundle and would set a variable in the framework at runtime to point to the location. Now, having everything self-contained in the framework bundle makes things a bit nicer.

This approach is not without it’s little gotchas, which I’ll go through here.

The basics are straightforward enough. In Xcode, for your framework target, add the binary to the target dependencies and add a Copy File build phase which copies it to the Executables location.

With a normal app bundle, these binaries are put in the MacOS subdirectory. In your code, you can use -[NSBundle pathForAuxiliaryExecutable:] which knows to look in that directory.

With frameworks, it’s not as simple. First off, there is a Versions directory with (potentially) various versions underneath. There is a symlink to the current version as well as symlinks at the top level of the framework directory. These symlinks point to the individual directories and files in the framework, including a symlink to the main framework image. Binary images are not put in their own subdirectory as they are with app bundles.

Any other executables aside from the framework binary are not automatically symlinked. On top of that, NSBundle can’t find them unless they are at the top level. So, in addition to the above steps, you need to add a Run Script build phase which symlinks your extra binaries to the top level. Make sure to use the -h flag (in addition to -s) to ln so as to not resolve the Current link.

Now, suppose your executable links against another framework in your app? For example, let’s take the following situation:

In this case, let’s say someprogram links against the Bar framework. We’ll need to specify an @rpath but what isn’t clear here is what @loader_path is (for an overview of @rpath and @loader_path, check out this post).

It would make sense that @loader_path is the top level of the framework. After all, NSBundle will return the top level symlink to your binary which is what you actually run in your code. In such a case, the rpath would be @loader_path/../ (we want to go one level up from someprogram so that we can see Bar.framework).

Unfortunately, that isn’t the case. Even though the resulting crash report you would get from doing the above will show the path to the symlinked binary, the loader seems to resolve the link, and then set @loader_path to that. So, in this case, it is referring to the original binary in Versions/Current/. Taking this into account, the correct rpath would be @loader_path/../../../

Not a huge deal but hopefully this will save someone from some confusion. As usual, it’s possible I’m missing out on something here, in which case, please comment below on how wrong I am.

Comment » | OS X, Programming, Xcode

AEftermath

January 29th, 2019 — 1:30pm

Now that the dust has mostly settled, I thought I’d follow up on my previous post concerning Mojave and its new privacy protections.

When Mojave was first released, there were complaints about Hazel not working. In these cases, Hazel was in the privacy list, but not checked. Users claimed to have not seen any sort of prompt. It’s hard to say if the prompt was just missed somehow or if there was a bug where the prompt was not showing up.

Months later, the issues stopped (or at least people stopped reporting them). I can’t say whether some sort of bug was fixed or if people are just now more familiar with how all the new privacy protections work. I’m leaning towards a bug of some sort as the reports have stopped altogether; I’d expect the occasional report of confusion if it was just a knowledge thing.

Regarding unit tests, it seems that changing my unit tests to run within an app, and adding the appropriate usage strings to the test app, was enough to get them to run. There is an issue of including Appkit in unit tests that are testing Appkit-independent functionality, possibly in resulting in subtle false positives in some tests. I suspect those cases will be very rare and, frankly, I’m just glad I can actually run my tests on Mojave now.

One issue that remains is filesystem level protections. With Mojave, certain directories (which I believe still aren’t documented) are protected, not by permissions, but a separate system tied in with SIP. The problem with this is lack of transparency both to us devs as well as users. I’ve had users waste a bunch of time trying to fix the issue by tweaking permissions and ACLs, not realizing that there is another system in place that is invisible to them. Very few users were able to suss out that they needed to add Hazel to the “Full Disk Access” list (instructions on how to do that can be found here).

Overall, it’s confusing since there’s little to no indication in the OS. Finder can view these folders fine whereas other Apple apps (like Terminal) can’t. The red badge on the folder icon looks the same as one for a folder protected by UNIX permissions.

At least, there seems to be one way as a dev to be able to make sense of things. Accessing protected directories (for instance, by using access() ) will fail with an EPERM error. This differs from accessing a directory protected by UNIX permissions. In those cases, you will get an EACCESS error. While that’s great for differentiating between the two cases what’s unclear to me is if there are other situations, outside of Mojave privacy protections, that would give me an EPERM error. Nonetheless, in the latest release, I’ve tried to detect this in Hazel and put up an alert. It’s a fairly uncommon case so I suspect most users will not run into this.

By now, most of the kinks have been worked out but it would be nice if Apple were to document this stuff. The path getting to this point was far more rocky than it should have been.

Comment » | Hazel, OS X, Programming

Hazel and the AEpocalypse

September 4th, 2018 — 1:13pm

This morning I released a beta of Hazel with a good deal of trepidation. If you’ve been following the dev blogosphere, you should know that the AEpocalpyse is coming. For those not in the know, Mojave introduces privacy protections for access to AppleEvents, prompting the user whenever an app tries to send AppleEvents to another app.

Here’s some reading to get you up to speed:

Felix Schwarz: macOS Mojave gets new APIs around AppleEvent sandboxing – but AEpocalypse still looms
Daniel Jalkut: Apple Events Usage Description
Dave Nanian: macOS Mojave: Opening New Vistas in Security for Mac Users
Paulo Andrade: Apple Events And The State Of macOS Automation

The result of all this is that we have a not-very-well-thought-out and buggy feature which was added late in the beta with no documentation. The last part makes it particularly hard as we have no way of knowing if the current behavior is a bug or by design, nor whether there will be any more changes or fixes before Mojave ships.

As it stands right now, I am faced with the following problems:

  • I cannot run a number of my automated tests.
    My tests test functionality that uses AppleEvents or the tests use AppleScripts themselves to verify test results. Adding the usage description keys to the Info.plists of the xctest bundles does not work. Also, it’s unclear which app the keys should be added to as it’s not documented whether a helper app, or its enclosing app needs to include them. On top of that, I don’t own the running app in this case so changing those keys will break the code signature. If anyone has a workaround for this, I’m all ears.
  • If the user is away when the prompt comes up, the request can time out and it will silently default to denying authorization thereafter.

    As pointed out by Dave Nanian in his post above, users may never see the prompt to provide access and later find that key functions are not working. To fix it requires using the commandline (provided the user knows what’s going on). Update: This is fixed in beta 10.
  • In some cases, users may be prompted twice for the same feature within Hazel.
    For example, when using the “Import into Photos” action, users will be prompted in the UI as Hazel needs to fetch the list of Photos albums via the MediaLibrary API. Then to do the import itself, Hazel has to use AppleEvents, which brings up another prompt. Guarding AppleEvents is a bit orthogonal to the other access restrictions as it is focusing on an access mechanism and not the information being accessed, resulting in redundancies like this.
  • Running user scripts is problematic
    Since Hazel allows users to run their own scripts, I may end up having to specify usage strings for everything, in the off-chance a user’s script accesses that area of the OS.
  • AEDeterminePermissionToAutomateTarget doesn’t work very well
    Felix’s post outlines this function. It has a critical bug making it near useless for me. On top of that, it requires the app you want to access to be running to determine permissions which leads to a less than ideal user experience.

In short, unless Apple fixes this soon, or even better, rolls it back to be released later, things are going to be a bit bumpy in Mojave. As a user, you may be surprised how often AppleEvents come up in the apps that you use.

The Hazel beta is out (forum login required) so if you are on Mojave (or even if you’re not; I need to make sure I didn’t break things on previous OS versions), give it a try. Reports from the field will help greatly. My options may be limited given the scant documentation and tools provided by Apple but hopefully something can be done.

Update (Sept 13): Beta 11 is out. It fixes a couple of issues I’ve found:

  • It doesn’t seem to prompt anymore when using the MediaLibrary API so the “double-prompting” issue is no longer a concern. I was wrong on this. I thought I had reset permissions when doing the test but trying it again, it seems I didn’t. So, the double-prompt is still a thing.
  • If the user denies access, the app now appears in the SysPrefs Privacy settings, unchecked. This is huge as the user can now enable access without having to resort to tccutil

tccutil itself is still broken in that it does not work when specifying a bundle identifier. Makes it such that if you want to reset permissions, you have to do it for all apps, not just for the one you care about. This results in you getting re-prompted all over again. The fix to the Privacy settings above does alleviate much of the need for this but this still indicates a lack of thoroughness when implementing this new system.

Comment » | Hazel, OS X, Programming

The Invisible Interface: APIs

May 25th, 2017 — 11:21am

Let’s talk about APIs. Like UIs, APIs provide an interface (hence the “I”) to a class of users. In this case, the users are programmers. Because of this, many times you have APIs that are cluttered and overly difficult to use since programmers can probably figure that stuff out. It’s unfortunate as many just see APIs as a way to expose functionality without any regard for design.

I feel a hallmark of a good UI is one that avoids presenting the user with unnecessary decisions. The more you can either filter or figure out on the user’s behalf, the better. I’ve touched upon this before: 1 2

I feel the same holds true for APIs. APIs should do more than just provide functionality. They should strive to do more for the user without burdening them with unnecessary details. Provide power with little effort on the user’s part. Let the programmer worry about the “what” and not about the “how”.

This leads me to my favorite API: NSArray

Yes, NSArray.

What is an NSArray? It’s an ordered collection of objects. Doesn’t sound terribly exciting but ask yourself this: what data structure does it implement? If you ask around, you’ll get a bunch of different answers but the real answer here is “It doesn’t matter.”

Now, you may be thinking that NSArray is insufficient to cover all the different cases where you’d need an ordered list. Surely, you have to use a skip list in this one case or a circular queue in this other one. Before going further, I highly recommend reading this oldie but goodie (and one of my favorite blog posts): Array.

As it turns out, NSArray is adaptive. It’s not implementing a single data structure but instead changes depending on usage. Every time I, or a colleague, has implemented their own special data structure, it has ended up performing no better, or sometimes even worse, than NSArray. Now, I’m sure you can come up with some special set of circumstances where you’d need to roll your own, but from my experiences with Cocoa over the years, I’d say that for 99% of the cases, NSArray is fine.

Another example of adaptive behavior is Introsort, which will use either quicksort or heapsort depending on the circumstances. Quicksort has great average performance but has a horrible worst case (O(n2)). Introsort is able to detect when things are headed towards that worst case and switches to heap sort instead (worst case: O(n log n)). No need for the user to evaluate the data to figure out which sort algorithm to use.

Now, sometimes things are inherently complex (threading comes to mind). It does take a bit of care and experience to make sure that instead of making something simple, you end up making it simplistic. Nonetheless, I feel that NSArray hits that sweet spot of providing what you need while at the same time cutting out what you really don’t.

Hopefully this shows that APIs can be so much more than a simple wrapper around some code. The use of proper abstraction as well as going the extra mile on the user’s behalf can go a long way towards making a merely functional API something wonderful.

 

Comment » | Cocoa, OS X, Programming

JSON Feed

May 24th, 2017 — 3:51pm

I’ve added support for JSON Feed to this blog. You can find a link to it under the “Meta” section in the lower right corner.

It’s very early and hardly any clients support it yet but it was easy enough to add support for it on my end. I felt like I should try being the chicken in the chicken-and-egg problem that comes with a new standard being introduced.

Let me know if you run into any problems with it.

Comment » | Web

Getting Info From iTunes

May 19th, 2017 — 12:35pm

Things have settled down with my 4.1 release so I thought I’d tend to my neglected blog.

Today, I’m going to talk about getting info about your iTunes library, playlists and albums.

Now, the obvious way to do this would be through AppleScript/AppleEvents. The problem with this approach is that it requires iTunes to be running for it to work. To actually change stuff in your iTunes library, you will have to use AppleScript but for the purposes of this article, we are just looking at ways to get information about it.

The old way to do this was to parse the iTunes Library.xml file (located in ~/Music/iTunes). This file contains a dump of everything you need. Unfortunately, recent versions of iTunes do not generate this automatically; you have to enable it in its preferences (it’s in the “Advanced” tab).

Now, the notion of having to parse an XML file does sound painful but luckily it’s actually a property list so you can load it that way and access everything as NSArrays, NSDictionaries, NSStrings, etc.

With OS X 10.9 (yes, I know it’s macOS now but back then it was still OS X), Apple introduced the MediaLibrary API (ML). This provided a way of not just accessing the iTunes library, but the libraries of all of Apple’s other media apps (Photos, GarageBand, even Final Cut). No parsing involved but it is a fully asynchronous API so expect to deal with a lot of callbacks.

The last one (I don’t know exactly when it was released but it was the last one I discovered) is iTunesLibrary (ITLib) and it’s a bit easy to miss because, from what I can tell, it is not a part of the OS, but iTunes itself. You can find the framework under /Library/Frameworks (not /System/Library/Frameworks). It provides similar functionality to the MediaLibrary framework, but specific to iTunes.

With so many options, which do you use? For the most part, they return the same information though it depends on the version of iTunes. With some versions, one API may return some special playlist that the others don’t. Also, ML is asynchronous while the others are synchronous. I don’t see that as a big deal either way as you can convert one into the other with a tiny bit of work.

The big differentiating factor is performance. Here are the results for getting a list of playlists for my library:

XML: 1.664939s
ML: 18.792855s
ITLib: 0.844415

I don’t think my library is the biggest in the world but it is significant (about 2000 albums). As you can see, ML is pretty damn slow with ITLib being much, much, much faster. What is surprising to me is how much faster parsing the XML file is than ML. It is possible, though, that I’m holding it wrong.

So, ITLib is my recommended API for this. It’s what Hazel uses starting with 4.1 (previously, it was parsing the XML when it was available and falling back to ML). That said, there is one wrinkle.

The persistent IDs of entities in iTunes are hex strings. In the XML file, they are zero padded. In ML, though it returns strings, they are not zero padded (rdar://26624642 for you Apple folks watching from home). To add even more confusion to the mix, they are NSNumbers in ITLib.

If you are using any of these APIs in isolation then it’s probably not a big deal, but if you want to later manipulate that entity using AppleScript (like importing files into a playlist), you will need to provide the zero-padded hex string. Something to keep in mind.

Comment » | Cocoa, Hazel, OS X, Programming, Software

Ten Years

September 6th, 2016 — 7:00am

Today marks the 10th anniversary of Hazel. On September 6, 2006, Hazel 1.0 was released and after all this time, it’s still going strong. I’ve already shared my thoughts on why I took this journey and how I managed to stay on this road so I won’t rehash that here. What started as a product developed in my bedroom is, well, still a product developed in my bedroom, but now I can say it’s a successful one.

Hazel back then was a much more basic product. You can check out the release notes page, which documents everything back to the beginning. It’s interesting to see what features were added later, many of which are probably considered essential now.

As for the future, who knows? While the list of big features I want to implement is much shorter now, there are still a bunch of smaller things I want to put in. I can’t say whether I’ll still be doing this in another ten years but I have no intention of quitting now.

And with today being the anniversary, I need to celebrate in some way so, from today until 3am (EDT) tomorrow (Sept. 7), I am offering Hazel at its original price from 10 years ago. That’s $16 (50% off). The family pack and upgrade didn’t exist back then but they’re 50% off as well. No special code or link. Just go to the store and buy. If you don’t have a copy yet, here’s your chance.

2 comments » | Business, Hazel, Noodlesoft, Software

On Dynamism

May 23rd, 2016 — 7:15pm

I’ve kept my hat out of the discussions of Swift vs. Objective-C for several reasons, but the main one is that frankly, I’ve been busy. I ship a product, one that I would consider successful, and recently shipped a major release. What does that mean? It means that debating these types of issues and playing with the latest thing is a luxury. Because I have a shipping product with a lot of users, I can’t just change things unless I find that it adds value to my users. My job is to solve my users’ problems. Now that I have a little bit of time to do so, this is one lens I’m hoping to add to this discussion

In addition, years ago, when Java was introduced, I jumped on that bandwagon, leaving behind Objective-C. It started out writing replacements for Foundation and AppKit in Java and ended with writing large scale servers. I was on that wagon for a good 10 years before returning, so I wasn’t a tourist there. This is yet another lens I also want to add to the discussion as history seems to be repeating itself and many younger people seem to be missing out on this part.

But let’s start with the current issue about dynamism since that’s the hot topic.

One thing many people seem to overlook about the dynamism of Objective-C is that it enabled NeXT (and Apple) to provide better GUI tools. Using dynamism, they were able to make GUI building declarative in nature. Connect this to that. Call this method. All stored in a file that was (and still is) data, not code. Competitors at the time (and today) resorted to code generation which is fragile and, ironically, unsafe. Yes, you could have a more declarative file format, but implementing that in using a static language required a lot of hard-coding and switch statements. Not the elegance that many people claim to be moving towards.

I’m not saying that a language has to be purely dynamic but it shouldn’t be purely static either. It think it’s spurious not to credit a level of dynamism for the quality of apps on Apple platforms over the years, and to be pedantic, the NeXT ones as well – many of which were considered the best on any platform at the time. To deny that, I feel, shows a lack of understanding of what has made the platform great all these years.

Moving from a dynamic language to a more static one seems refreshing at first. I went through the exact same thing when switching to Java. Yes, I can hear some of you grumbling “Java” out there but the irony is that there are more similarities in the evolution between Java and Swift than you may want to admit.

Everything seemed so much “tighter” in Java. Interfaces (Java’s version of protocols) everywhere! No more pointers! Checked exceptions! It’s all so new and exciting. You might just have to take my word on this, but any new language that you really like will always seem to make you “more productive”. A colleague and I (plus maybe one, or one and a half, devs who were on the periphery) implemented our own version of Foundation and AppKit in what I would consider a pretty short amount of time. Other people at Lighthouse can correct me on this but my recollection was that it was on the scale of months, not years. You can read up more on it here.

Was it the language? At the time we might say yes but in hindsight, I’d say it was just that we were just super excited and into the project. I was working with an exceptional team and being a new platform, we were excited about making our mark on it. Yeah, there were bits about Java that were better than Objective-C. But there were bits that were worse. The point here, though, is that until the honeymoon is over and you are maintaining a project over several years, it’s hard to make real proclamations about productivity. Truth be told, I’d say the biggest impact on software quality and productivity is the quality of engineers regardless of any other factors.

Looking back on my years with Java, I find the biggest problems with the platform was this idea that everything can be done by the compiler. It sounds great on paper but when implemented, it resulted in all sorts of contortions to work around not letting any dynamism in. It resulted in a lot of glue being written that I wouldn’t have to write in a more dynamic platform. It resulted in cascading changes across code to make trivial modifications just to appease the compiler. Maybe in a future post I’ll go into specific language features that I found problematic and show parallel trends/features in Swift but let’s just say that less is more. It’s an aesthetic that I always felt that put Apple above the rest and something which I feel like in some ways is being abandoned.

Now, I’m sure many of you will just paint me as an old, stubborn Obj-C luddite. And it’s true on some level but that doesn’t mean you should discount my experience. Many of us old fogeys are working on products that have shipped for years. That means that (1) we are solving user problems (2) have immense experience with doing app development (3) really understanding the frameworks (4) have been maintaining a code base.

Let’s go through these points:

1. We are solving user problems.

We have real problems to solve for people outside of ourselves. We need to be practical to survive. Users don’t care about the tech underneath. We have to ship them solutions. This oftentimes requires rapid response. This also oftentimes involves working around Apple bugs, bugs that never get fixed and will only grow in number as Apple maintains their yearly release schedule (something to think about for proponents of the “final” keyword). Agility and flexibility are paramount here. Reality is not pretty. The user could care less how beautiful your code is if you aren’t solving their problems.

2. We have immense experience with doing app development

I want to emphasize the “app” part of that. A common argument for very strict type safety are for cases which aren’t apps. The predominant software in the Apple ecosystem is end user applications. Let’s try and not lose focus here and argue the case for categories of software that aren’t Apple’s forte. If you are writing some life-critical software on an Apple platform, god help you. Even with Swift, you are stuck with Xcode and Apple’s insistence on yearly new OS releases.

3. We really understand the frameworks

I would take many of the arguments for Swift more seriously if there weren’t strawman examples set up on the Obj-C side. You’ll have to dig up Brent Simmons’ posts on this, but when I see the kooky examples of horrible old Obj-C way, what it tells me is more that you don’t understand the frameworks. It resembles informercials where people are fumbling opening a jar and then are presented with some useless gadget to solve a non-existent problem. I’m sure things can be done better than they are now but providing bad examples indicating ignorance of what it is you are trying to replace doesn’t help your argument. And I have seen plenty of examples of new people replacing what they don’t understand. Please don’t be one of those people.

4. We have been maintaining a code base for a while

I have been writing Hazel for ten years. Many of us old-time app developers are in similar positions. Speaking with many other devs, most of whom write iOS apps, I’d be lucky to find those that have maintained a code base for a third of that time. Sure, there are parts of the code that are ugly, but that happens on any platform. Same thing happened on Java. No matter what you do, time is always against you. When I was working with Java, I found the technical debt grew in a different fashion. It was hard to change anything. This was because changes in stricter languages tended to cascade. Because everything has to declare everything about itself, changing code turns into a Facebook feed, except where you and everyone else are obligated to respond to every stupid pic and forward of Snopes-bait. With a more dynamic code base, I can say I have less code to maintain and that I can be more agile. Not that it’s without it’s own problems. But let’s not act like type-safety is a panacea.

And to give more of my personal perspective on the language debate, I’d love to have something better than Objective-C. But I’d also like to have something better than Swift. My personal pet peeve is that we are in the 21st century and we still have to give a shit about the storage type of numbers. I’d be happy to get rid of scalar primitives entirely.

Ultimately, though, I need something that solves my problems. My problems aren’t performance. They aren’t type-safety (maybe it’s just me, but I rarely have issues with it and when I do, the time it takes to fix it is far less than the time specifying everything to a tee everywhere else). They aren’t being able to write clever VDLs. For me, it’s writing apps that solve my users’ problems and getting them out in a timely fashion. As it stands now, Swift (at least pure-Swift, or even current Swift as a non-ABI-stable moving target) does not do that for me.

Now maybe these same problems can be solved in a static way but what I’m not seeing from the static-camp are (decent) solutions. What I’m seeing are either hand-waving or the same crufty code-generation, write tons of repetitive boilerplate type of solutions that I had hoped we had left behind in the 90s.

What I do see makes me worry that it’s not the experienced app-writers that are being heard. Unfortunately, many of us are too busy to sit in the various mailing lists and forums. Yes, ok, we lose. But we all lose if those creating the platform don’t draw from the experience of those who have built upon it successfully. As someone who has done both (write frameworks and apps) and seen what happens when the former ignore the latter (the rise and fall of Java), I’m hoping history doesn’t repeat itself.

That’s my somewhat meandering brain dump for the moment as I do have to get back to working on my next patch. Sure, these are just my opinions but my hope is that we can at least look for more balance in our approaches to solving actual problems.

36 comments » | Cocoa, OS X, Programming

Adventures in Email Hosting 2

May 19th, 2016 — 3:10pm

Yes, another journey into the world of email hosting. This time it’s a bit different. Instead of receiving email, today we talk about sending it.

It seems that during my recent launch I became collateral damage in the war against spam. Between a couple of email campaigns and a bunch of license emails, I had sent out a good amount of email; enough to cause me a ton of headaches as certain mail services, actually, mainly one: Gmail, decided to mark a lot of it as spam (if I was lucky) or make them disappear entirely.

The way things were set up, my email campaigns were sent out by Campaign Monitor, my license emails by my server and my regular “interactive” emails via Rackspace’s email service (as described in a previous installment). The ones I had the most problems with were the first two, as they were repetitive messages sent out in volume.

For a while, I’ve had SPF records set up. What are SPF records? In a nutshell, it’s a way for you to specify which mail servers are the “official” servers your email comes from. This is to help identify mail coming from you as opposed to a spammer posing as you. The way you set it up is to create special DNS TXT records listing out the specific servers for your domain.

Apparently, this isn’t enough. Seems like there’s another layer you can implement: DKIM. With this, you have your mail server sign outgoing emails so mail servers at the receiving end can know that emails are definitely from you and definitely not, once again, from a spammer imposter. So, I went ahead and set up OpenDKIM on my server. You can find various guides out there on how to install it on your OS and integrate it with your MTA (I use postfix and hooking the two up was pretty easy). You also have to add a DNS record listing your public key so other mail servers can verify your signatures.

Even after doing that, it still didn’t seem to appease the Gmail gods. I found this page which recommended yet another thing: DMARC. Here, you specify a policy as a guideline to mail servers on how to handle your email. One of the things you can specify is an email address where mail services can send you reports on the emails you send them. And you guessed it, you implement it by creating a DNS record.

Being desperate, I thought I’d do it, hoping that Gmail would send me a report telling me what I’m doing wrong. Soon after, I started receiving DMARC reports from all sorts of mail services (Microsoft, Yahoo, AOL, etc.). Over ten days later and guess who still hasn’t sent me one.

I’ve been getting fewer reports about not receiving emails lately but that’s mainly because of decreased volume since launch. It’s still unclear whether Gmail is binning my emails at a high rate or not. Nonetheless, if sending out emails are an important of your business, I recommend doing the above. Even if Gmail seems to be hard to please, other mail services are more appreciative of the gesture.

Note that there is also the option of relaying all my mail through Rackspace. It’s still a possibility but (a) I’m afraid of poisoning the well since my email is already being marked and (b) using a shared relay opens you up to being blacklisted because of someone else’s misdoings. All in all, I feel that some level of redundancy is ok here.

When implementing the above, you can check the headers of an email received at the other end to make sure everything is set up properly. Here’s one from an email sent from my server to my Gmail account:

Authentication-Results: mx.google.com; dkim=pass header.i=@noodlesoft.com; spf=pass (google.com: domain of www-data@noodlesoft.com designates 2001:4801:7824:103:be76:4eff:fe11:5179 as permitted sender) smtp.mailfrom=www-data@noodlesoft.com; dmarc=pass (p=NONE dis=NONE) header.from=noodlesoft.com

As you can see, it shows that my SPF and DKIM passes. That doesn’t guarantee anything but it helps.

You can also check out Google’s Postmaster tools site. It will give you feedback on various metrics concerning email from your site. To set it up, you have to create a DNS record (see a running theme here?) with text it supplies you so that it can see that you control the domain. After that, it will track your site.

Also, yes, DNS once again: make sure all your regular DNS records are set up properly. Not only do you want an A and PTR record for IPv4, but also a AAAA and PTR record for IPv6 as more mail servers nowadays are checking for that.

Until next time, here’s hoping I don’t have to resort to human sacrifice to get Gmail to accept my messages.

Comment » | Business, System Administration, Tools, Web

Back to top