On Mac OS X programming
I’ve added support for JSON Feed to this blog. You can find a link to it under the “Meta” section in the lower right corner.
It’s very early and hardly any clients support it yet but it was easy enough to add support for it on my end. I felt like I should try being the chicken in the chicken-and-egg problem that comes with a new standard being introduced.
Let me know if you run into any problems with it.
Yes, another journey into the world of email hosting. This time it’s a bit different. Instead of receiving email, today we talk about sending it.
It seems that during my recent launch I became collateral damage in the war against spam. Between a couple of email campaigns and a bunch of license emails, I had sent out a good amount of email; enough to cause me a ton of headaches as certain mail services, actually, mainly one: Gmail, decided to mark a lot of it as spam (if I was lucky) or make them disappear entirely.
The way things were set up, my email campaigns were sent out by Campaign Monitor, my license emails by my server and my regular “interactive” emails via Rackspace’s email service (as described in a previous installment). The ones I had the most problems with were the first two, as they were repetitive messages sent out in volume.
For a while, I’ve had SPF records set up. What are SPF records? In a nutshell, it’s a way for you to specify which mail servers are the “official” servers your email comes from. This is to help identify mail coming from you as opposed to a spammer posing as you. The way you set it up is to create special DNS TXT records listing out the specific servers for your domain.
Apparently, this isn’t enough. Seems like there’s another layer you can implement: DKIM. With this, you have your mail server sign outgoing emails so mail servers at the receiving end can know that emails are definitely from you and definitely not, once again, from a spammer imposter. So, I went ahead and set up OpenDKIM on my server. You can find various guides out there on how to install it on your OS and integrate it with your MTA (I use postfix and hooking the two up was pretty easy). You also have to add a DNS record listing your public key so other mail servers can verify your signatures.
Even after doing that, it still didn’t seem to appease the Gmail gods. I found this page which recommended yet another thing: DMARC. Here, you specify a policy as a guideline to mail servers on how to handle your email. One of the things you can specify is an email address where mail services can send you reports on the emails you send them. And you guessed it, you implement it by creating a DNS record.
Being desperate, I thought I’d do it, hoping that Gmail would send me a report telling me what I’m doing wrong. Soon after, I started receiving DMARC reports from all sorts of mail services (Microsoft, Yahoo, AOL, etc.). Over ten days later and guess who still hasn’t sent me one.
I’ve been getting fewer reports about not receiving emails lately but that’s mainly because of decreased volume since launch. It’s still unclear whether Gmail is binning my emails at a high rate or not. Nonetheless, if sending out emails are an important of your business, I recommend doing the above. Even if Gmail seems to be hard to please, other mail services are more appreciative of the gesture.
Note that there is also the option of relaying all my mail through Rackspace. It’s still a possibility but (a) I’m afraid of poisoning the well since my email is already being marked and (b) using a shared relay opens you up to being blacklisted because of someone else’s misdoings. All in all, I feel that some level of redundancy is ok here.
When implementing the above, you can check the headers of an email received at the other end to make sure everything is set up properly. Here’s one from an email sent from my server to my Gmail account:
Authentication-Results: mx.google.com; dkim=pass header.i=@noodlesoft.com; spf=pass (google.com: domain of www-data@noodlesoft.com designates 2001:4801:7824:103:be76:4eff:fe11:5179 as permitted sender) smtp.mailfrom=www-data@noodlesoft.com; dmarc=pass (p=NONE dis=NONE) header.from=noodlesoft.com
As you can see, it shows that my SPF and DKIM passes. That doesn’t guarantee anything but it helps.
You can also check out Google’s Postmaster tools site. It will give you feedback on various metrics concerning email from your site. To set it up, you have to create a DNS record (see a running theme here?) with text it supplies you so that it can see that you control the domain. After that, it will track your site.
Also, yes, DNS once again: make sure all your regular DNS records are set up properly. Not only do you want an A and PTR record for IPv4, but also a AAAA and PTR record for IPv6 as more mail servers nowadays are checking for that.
Until next time, here’s hoping I don’t have to resort to human sacrifice to get Gmail to accept my messages.
It’s been a bit hectic because of the launch yesterday but I finally have a moment to post. Yes, Hazel 4 is finally out. You can find the release notes here.
Development was a bit rocky. I played with a bunch of different features but some of them didn’t quite pan out in a way that I liked. It felt like wasted effort in that that work didn’t result in a usable feature but many of them were only shelved temporarily. Oftentimes I end up having that eureka moment which would allow a shelved feature to be finally realized so something to look out for in future point releases.
That said, I’m happy with the features that I did get to work. They seem simple on paper but involved a bit more thought than would be expected. Sync is always tricky and getting the preview feature down to something as simple as it is now took a little doing.
Along with that was the site re-design (courtesy of the folks at Brotherhood). The previous site was mostly static. Adding content involved editing raw html pages and adding them. It was enough to discourage me from doing it often and discourage me it did, as I ended up leaving the site very outdated. The new site is backed by WordPress which will hopefully remedy that. The point here being that content can be added more easily using tools like MarsEdit or WPs web editor. I’ve already added a few posts (a review and a couple of knowledge base articles) since the launch.
Also, the new site design is a bit more stripped down and streamlined. I’ve tried to reduce navigation in favor of search. Most of the site is searchable via the form on the support page so I recommend going there first and doing a search if you ever have a question about Hazel.
The launch itself went relatively smoothly (except one incident – more details below). I can credit most of this to using a VPS (virtual private server). VPSes are great as you can clone, rebuild and resize them as needed. It gives you an amazing amount of agility when deploying servers.
Before the launch, I set up a clone server so I could set up and test the new site. Since it’s a clone, no need to reinstall and reconfigure everything (though you do have to make some changes in places where the IP address or hostname is stored). You end up with pretty much an identical server to play with which went a long way towards making sure things were working properly.
When I launched, I transferred the new stuff over to the live server. That part went with little drama but then disaster happened: I underestimated the load from tons of eager customers. The problem was that I had sent two email campaigns. One to those on my mailing list and another to those who purchased recently. The latter group received a message with instructions on how to get their free upgrade license. And guess what all those people decided to do immediately upon seeing that.
The result was that the site got slammed. More specifically, apache was overloaded. Enter VPS awesomeness #2: I was able to resize the server on the fly. It took a little while (maybe 15 minutes though it felt much longer) but the old server was still able to run, albeit very sluggishly, until the last minute when the conversion finished and it rebooted. After that, the site ran like butter and it was smooth sailing (at least as far as the server went).
Aside from some minor glitches (version 4.0.1 released this morning should address some of them), the launch has been pretty great. I just had the best day in sales in Hazel’s history so I’m pretty happy about that. My thanks to everyone who contributed, including Brotherhood, Jono Hunt for his icon and UI work, my beta testers, my friends in the Mac dev community and of course, all my customers who’ve been very supportive of Hazel all these years.
Comment » | Business, Hazel, Software, System Administration, Web
As you may or may not have noticed (more likely the latter), this blog was down for a chunk of the afternoon. I had to fix something, and, well, it took a bit longer than usual. You may have noticed that you’d see garbage characters like “ö” pop up in posts and comments. That’s because some time ago a WordPress upgrade changed the character encodings. I didn’t consider it a high priority issue and let it sit until now.
Following this article, I converted everything over only to realize that none of the actual characters were converted properly. Instead of trying to debug SQL scripts that could potentially destroy all my data, I went through and edited every character encoding screw-up by hand. It wasn’t so bad with my posts since I pretty much remember what I put in there. Fixing user comments was a different matter. Being on a perfectionist tear, I used the Wayback Machine to find the comments before I performed the fateful WP upgrade just to figure out if somebody used a smart quote or an em-dash. Fun.
Hopefully everything is back up and fixed. If you notice any other garbage characters floating around, please post here so I can fix it.
And yes, I’m overdue for a real post. All you have to do is ÃâπÀìâ,öå¢Ã,Å,ìãâπÃ∫, and I just might be compelled to write something.
I just switched PotionStore to use Phusion Passenger. Also known as mod_rails, Passenger is an Apache module that allows you to run Rails with Apache. Unlike other Apache plugins like mod_php, your application is still run in separate processes. Previously, I had been using Apache as a proxy to a mongrel cluster. On the surface, this doesn’t sound much different but Passenger does give you a couple things:
Luckily, Andy Kim already played guinea pig and tried it out to make sure it worked. Many thanks to him for that (and for the whole PotionStore thing to begin with, of course).
While the setup was fairly simple, I ran into a couple odd issues. For one, the Enterprise Ruby installer seemed to screw up the permissions of some of its files. All of its .so files and a directory or Ruby file here and there were set to be only readable by the owner. Make sure to check this before deploying. Note also that it installs as a totally separate Ruby installation so run its version of gem to make sure your Ruby packages match what you had on “regular” Ruby. For those of you are running PotionStore, make sure to do a rake rails:update otherwise it’ll bomb and log a message telling you to do so.
Unfortunately, I didn’t record the memory usage beforehand so I don’t know the exact gain. Based on my recollection, it does seem like I have maybe 20M or so more than I did before (for two Ruby processes). One odd thing I’ve noticed in my graphs is that my interrupts and context switches plummeted immediately. Not sure why that is but it seems like a good thing to me.
While this doesn’t remove Rails’ lack of thread-safety problem (resulting in a separate process per request), it does at least make the deployment much, much easier and with the memory savings, a bit more scalable as you take less of a memory hit with each extra Ruby process. Especially for those of you that have not deployed yet, this will save you a bit of a headache in configuration (no proxy and mongrel setup). It’s only been up for a couple days so it may be too early to tell but so far it’s been running fine.
Comment » | Ruby on Rails, Software, System Administration, Web
Some months ago I had arranged to sponsor Daring Fireball for sometime in April. Then a certain promotion came up that overlapped with it. Not being big on overlapping promotions I bumped it until later. Then I bumped it again. Then another bump. Then a jump to the left and a step to the…whoops, wrong card.
But this week, finally, I am sponsoring Daring Fireball’s RSS feed. There’s even a promo coupon. I’m not going to tell you what it is as that would defeat the purpose of the promotion. Subscribe to the feed and look for yourself. If it’s not showing up for you, wait towards the end of the week when it will get posted again.
And on a side note, I suggest hanging around the forums (signup required for the beta forum in particular) as there may be some beta news by the end of the week.
Comment » | Hazel, Noodlesoft, Web
It’s been over a month since I’ve been with Slicehost so I figure it’s enough time to make an assessment. Especially now that the MacUpdate promo is over, I actually have a sense of how well things hold up under load.
For those who don’t know, Slicehost is a hosting provider. What sets them apart from shared hosting providers is that they provide you with what they call a “slice” (other similar providers may call it a virtual private server or VPS). What this is is a virtualized server of your own. From your perspective, it’s like getting a dedicated server. You choose what OS you want (which right now consists of different Linux distributions) and you get root access so you can do whatever you want.
It differs from shared hosting in that your slice is like it’s own machine. It gets a guaranteed amount of memory and CPU so even if your neighbor is a hog, it won’t affect your slice. Because of the way that the Xen virtualization works, it is impossible to oversell on capacity.
Compared to getting a dedicated machine, it’s much cheaper and you aren’t tied to specific hardware. Blown power supply? Not your problem. I don’t know exactly what they do in this case, but I imagine they can move your slice to different hardware as needed.
You can read my original report on getting set up. You are expected to set up and administer the slice yourself. If you have the inclination and need a high level of control, then this is probably for you.
It only took me a weekend from getting my slice to having everything migrated over. Of course, being the tweaky type that I am, I spent some days playing with it and optimizing it. One of the benefits and dangers of having full control.
When I launched the new store, there was a problem. Ruby on Rails is a memory hog and as a result, I needed to upgrade to a bigger slice. Fortunately, Slicehost automates all that. Just log into the management console and request the larger slice. It takes a little while for the slice to get prepped but during that time your slice is still up. The downtime for the reboot was short (less than a minute) and that was it. The fact that it’s automated is a big deal to me as it means I can do it on the fly without doing a drawn out back and forth with a support person.
As you may or may not know, Hazel was included in the recent MacUpdate bundle. Before the launch, I upgraded my slice again to 1024M in anticipation of the load. Turns out, this was unnecessary. The 1024 slice never broke a sweat. The load went up briefly to 0.4 once. Apache connections stayed below the upper limits of what a smaller slice would have been able to handle. Traffic was about 150K requests a day at its peak. I don’t really have a frame of reference for that except that it’s a good bit more than I usually get. In short, the 512 slice would have been able to handle it fine. The slice has performed better than with my previous providers and I haven’t noticed any slowdowns or downtime (except for when I restart things for maintenance). With the promotion over, I’ve downgraded my slice and things are still running smoothly.
After all this, I’ve only contacted support twice. Once in the beginning just to say hi and once today for an issue that ended up being an Ubuntu thing. In both cases, I received responses within the hour. Granted, I’m doing a lot of the things that support at other hosting services would do for you. It’s a trade-off between effort and control and I’m at that point where I need more of the latter. For the things I really care about, keeping the machine and network reliable, there has been nothing to report, and that is how it should be.
It’s too bad most providers price on bandwidth and storage space. I would have happily paid more per month if I could get higher availability and reliability (with the ability to run RoR – sorry Pair). Of course, with everyone claiming 99.999999% availability, it’s hard to differentiate oneself on this front so providers seem to just pile on the bandwidth/space like extra gravy hiding the bad meat.
I feel like VPSes are the future of hosting. The amount of computing power that you can cram into a 1U rack space is far more than most of us need or want to pay for. But virtualize it and divvy it up and you have a great scalable model for doing dedicated hosting. It’s probably greener too but I’ll let the hippies make a determination on that.
In getting Potion Store integrated with my site, I had to learn Ruby on Rails. RoR has always been one of those things I’ve been curious about so I saw it as an opportunity to dive in. I present here a first impression. Note that these are impressions. They represent how I perceived things coming to it for the first time. While chances are that these perceptions are inaccurate, there’s something to be said for seeing how things appear to fresh eyes.
Ruby is an interesting language. It appears to be very flexible and dynamic, allowing you to do a bunch of really neat things that you can’t in most other languages. Note that these features aren’t really unique to Ruby. My sense is that Ruby is not a distinctive language in itself so much as a mostly good mix of other languages. I say “mostly” because I’m not sold on the syntax. It reminds me too much of Perl. It’s like they created a great new flavor of ice cream and then mixed in glass and razor blades.
My advice to those picking up Rails for the first time, get the Rails book and read the first few chapters before diving in. I found the online tutorials kinda annoying and not as good at explaining the structure and conventions that Rails relies on. Rails’ strongpoint is streamlining common tasks and functionality but to do so, it expects things to be laid out in a certain way. The other thing that takes time to figure out is the “magic” that Rails performs at certain points. One of the great things about Rails is how certain things are automatic. Very little glue needs to be coded and I feel that’s a wonderful thing. On the downside, sometimes this magic makes it hard to figure out what’s going on. For instance, some classes had naming patterns for their methods that you had to find out about in the documentation. I found the quality of the documentation to be a bit uneven which didn’t help in some cases.
I feel most of the problems with Rails are less about the design and more about the implementation. For instance, Rails is a memory hog. And because Rails is single-threaded, you need to have at least two of these things sitting there to get any sort of concurrency. When I deployed my store, I had to upgrade my SliceHost slice as the 256M slice wasn’t cutting it (remember I’m also running mysql and PHP stuff). Also, I felt deployment was a bit clumsy, as I had to set up a cluster of mongrel instances to run my app with apache acting as a proxy. While I see that there is a mod_ruby for apache out there, no one I know recommended it. Nonetheless, I felt like I had to deal with more moving parts than was necessary.
In a nutshell, Rails: great, Ruby: enh. If you are going to buy a book on it, get the Rails book first. It has an appendix on Ruby that should be sufficient for most things. If you want to go deeper with the Ruby side of things, you can get the Ruby book as well but personally, I could have done without it.
Overall, I like the platform but at the same time, I can see that it’s relatively immature and has some ways to go. I’m pretty much tied to RoR for now since it’s what the store is written in but that’s not a bad thing. That said, I also am not so amazed that I am going to join the Ruby cult.
Nonetheless, RoR feels like how a web framework should and less like a bloated platform where you end up installing a hundred frameworks requiring just as many config files to be edited to solve non-existent problems that are justified because someone gave it a fancy technical name along with a contrived design pattern to fix it (if you need a hint as to what I’m referring to: “starts with a J”). RoR seems to be able to identify and address the actual problems people run into when creating web applications, all the while keeping it relatively simple, and that is refreshing to see.
In my recent quest to deal with some long needed “home improvement,” I’ve just rolled out my own store based on PotionStore. For those that don’t know, PotionStore is an open-source store implementation done by Andy Kim. It’s used by PotionFactory, of course, as well as a few other shops.
Until now, I’ve been using Kagi for all my payment processing. Kagi is great for someone starting out as they provide the store, deal with taxes and can even provide a license number generator. I think when you are doing your first release, outsourcing the commerce side to someone else can save a good deal of time.
So why am I switching off Kagi? Mainly much lower commissions on each transaction and more control over the store itself. I was finding that the commissions were adding up to significant money as my volume rose and there were certain features I wanted to add that I couldn’t. On top of that, there was a bit of confusion regarding having customers dealing with a 3rd party.
Unfortunately, now with my own store, I am going to have to charge sales tax to New York residents. One of the downsides to living in the same state as me. Trust me, I’m not looking forward to doing the tedious sales tax filings.
Using PotionStore has been great. The integration took a bit longer as I had to learn Ruby on Rails at the same time (more on this possibly in a future post). I also had to change the code a bit for certain requirements I had (sales tax, license files, family packs, etc.). My thanks to Andy for sharing his code. I hope to see other devs pick it up and maybe even contribute to it. Oh, and a thanks to Gus Mueller for letting me crib some Javascript from his store.
The result of all this is a store which I hope looks nicer and has a more streamlined flow. I’ve added the ability to download your licenses immediately as well. You’ll still get them emailed to you but this way, you’ll get instant satisfaction coupled with some level of redundancy.
So, give it a spin. Maybe even buy something :). If anything goes wrong, or even if everything goes right, I’d love to hear about it.
In the past couple weeks, I’ve been observing some strange traffic. I’ve been getting a good number of hits to an old blog article of mine. Close to 1000 hits a day which is a lot for an old article that hasn’t been linked to in months.
The traffic has the following characteristics:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)Seeing as how the article is Mac-centric and the user agent is for a Windows client, I suspect that these are automated hits and that whoever is hitting the article doesn’t care about its content. But if it’s some sort of bot attack, I don’t see the point of it.
Unfortunately, I can’t figure out much more than that. So, yes, consider this a “Dear Lazyweb” post. If you have any ideas on what’s going on, please post here. Oh, and don’t ask me how, but I have eliminated the Illuminati from the list of suspects. UFOs/aliens are still fair game though.
Update (Oct. 16, 2007): Not sure what happened but it stopped, pretty much completely. It’s like someone flicked a switch. I still have no idea what that was all about but the fact that it stopped pretty suddenly implies there’s a central source to all this.
Also, something else I realized was that it seems as if those hits were hitting the article page but not any of the associated files (css, graphics, etc.). Very strange.
Again, any theories are welcome.