Noodlings

Video games are on the forefront of what kinds of rich interactions people can have with computers. In the past decade, there’s been a push for more and more immersive virtual environments resulting in more advanced APIs and hardware to provide things such as super-fast 3D rendering. In recent years, OS X has leveraged these advances in the predominantly 2D world of user interfaces, often in brilliant ways as seen with QuartzGL, CoreAnimation and CoreImage.

In video games, it’s quite common to exploit stereo output or even better, surround sound, to provide positional audio cues. Just as graphics can simulate a 3D space, so can sounds be placed positionally in the same space. If you, super-genetically-modified-mutant-soldier, are running around on the virtual battlefield and there is some big-bad-alien-Nazi-demon-zombie dude shooting at you from the side, you will hear it coming from that direction and react accordingly. Directional audio cues can supplement visual cues or even supplant them if visual ones cannot be shown (i.e. something requiring attention outside your field of view).

On OS X, sound is used rather sparingly in the interface, which is probably a good thing. But for those cases where it’s use is warranted, why not take advantage of technology available? Just as animation can be used to guide the user’s focus, why not sound? OS X does ship with OpenAL, which is to sound what OpenGL is to graphics, providing a way to render sounds in a 3D space.

I’ve put together a quick proof of concept app (download link near the end of the article). Move the window around the screen and click the button to make a sound. Based on the window’s position, the sound will appear to come from the different sides, which, for the most part is left/right, most sound output systems not being designed to articulate things in the up/down direction. The program itself basically maps the window position to a point in the 3D sound space. Right now, it doesn’t really use the z-axis (the axis that goes into your screen) but conceivably you can do things like make the sound appear further away based on window ordering. Try using headphones if the effect is not as apparent using speakers.

There is a significant technical issue, though. You can’t really know the actual physical dimensions and layout of a user’s screens. In addition, the position of the speakers relative to the screens is also not known. While you can get screen resolutions and relative positions of the screens, these are mostly hints at the actual layout. In my demo program, it is assumed that the screens are relatively close to each other forming one gigantic screen. It is also assumed that the speakers produce a soundstage roughly centered on the primary display (the one with the menubar). It assumes a model like this (the circle is the user and the thin slabs are the monitors, from a top-down view):

In reality, it’s probably more likely the user would have a setup like the following:

But who knows, it could possibly be something like this:

The point here is that the effectiveness of this is dependent on the user’s setup. A particular idealized model would have to be chosen that hopefully works well enough for most people. While pinpoint accuracy is not really feasible, it probably isn’t required either. Human hearing is imprecise, otherwise ventriloquists would never be able to pick up a paycheck. Just an indication of left, right or center is probably enough for these purposes.

Where would this be useful? Well, this all came up yesterday when I received an IM (via Adium). I had my IM windows split up across two screens so I had to scan around a bit to find out which window had the new message. Though the window was on the screen to the left, the audio alert made me look at the main screen since the sound was centered straight ahead. It would be great to see an idea like this implemented in Adium and I’ve filed a feature request with them for their consideration. It’s ticket #11292 so you don’t go and submit a duplicate request.

It would be interesting to see more use of this in user interfaces out there. I don’t want to encourage people to add sounds to their apps if they weren’t already using them but for those that are, it’s something to consider. Overall, the effect is quite subtle but with some tweaking, it can be quite effective.

The link to download the demo program is below. Sorry, no source is provided this time. The code is a hacked together mess of stuff copied and pasted from an Apple example as I have never used OpenAL before. This can probably also be implemented in CoreAudio by adjusting the balance between the channels. If you are considering implementing something like this, email me and I’d be happy to discuss details as long as they don’t involve audio APIs since, well, I don’t know them particularly well.

Download PositionalAudioAlertTest.zip (Leopard only)

Thanks to Mike Ashe and Chris Liscio for advice on CoreAudio, which I ended up not needing as Daniel Jalkut suggested I use OpenAL instead which made things easier.

Yes, it’s free code time again. I’ve been neglecting the blog for some time so hopefully this will make up for it. Think of it as that conciliatory heart-shaped box of chocolates used as a sorry way to make up for forgetting about your birthday, after which, I go back to my old ways of sitting on the couch all day watching sports, ignoring you.

In version 2.2 of Hazel, I added mini AppleScript and shell script editors so that people could enter scripts inline without having to go to another program and saving it to an external file. I’ll admit, I didn’t set out to make an uber-editor since it was intended for small scripts. Nonetheless, a user recently pointed out that when a line wraps, it’s hard to tell if it’s a continuation of the previous line or a new one. One of his suggestions was putting line numbers in the left gutter. If you don’t know what I’m talking about, look at TextMate (the example he cited) or XCode (you need to turn it on in preferences). I thought it might be overkill for a script editor that will mostly be used for scripts less than ten lines long. I’m instead considering doing an indented margin for continuation lines. Less visual clutter and addresses the problem at hand.

Nonetheless, I was curious about implementing line numbers. Poking around, I found some tips on how to do it but it seemed like there were odd problems implying it wasn’t as straightforward as one would think. So, snatching some free time in between other things, I decided to tackle the problem.

I looked into subclassing NSRulerView. The problem is that NSRulerView assumes a linear and regular scale. Now, to make it clear, I am talking about numbering logical lines, not visual ones. If a line wraps, it still counts as one line even if it takes two or more visually. The scale is solely dependent on the layout of the text and can’t be computed from an equation. Despite these limitations, I went ahead and subclassed NSRulerView. If anything, NSScrollView knows how to tile it.

I had this notion that NSRulerView was a view that synced its dimensions with the document view of the scrollview. With a vertical ruler, I assumed it would be as tall as the document and the scroll view just scrolls it in tandem with the document. Not so. It’s only as tall as the scrollview. That means you have to translate the scale depending on the clipview’s bounds.

I added some marker support via an NSRulerMarker subclass that knows about line numbers. The line number view will draw the markers underneath the labels a la XCode (with the text inversed to white). The sample project uses another subclass which will toggle markers on mouse click. While NSRulerView usually delegates this to its client view it made more sense to just do it in a subclass of NSRulerView. You have to subclass something to get it to work and it made more sense to subclass the ruler view since the code to handle markers never interacts with anything in the client view anyways. Personally, I find it an odd design on Apple’s part and would have preferred a regular delegate.

The project is linked below. The main classes are NoodleLineNumberView and NoodleLineNumberMarker. Some notes:

• To integrate: just create the line number view and set it as the vertical ruler. Make sure the document view of the scrollview is an NSTextView or subclass. Depending on the order of operations, you may have to set the client view of the ruler to the text view.
• The view will expand it’s width to accommodate the widths of the labels as needed.
• The included subclass (MarkerLineNumberView) shows how to deal with markers. It also shows how to use an NSCustomImageRep to do the drawing. This allows you to reset the size of the image and have the drawing adjust as needed (this happens if the line number view changes width because the line numbers gained an extra digit).
• Note that markers are tied to numerical lines, not semantic ones. So, if you have a marker at line 50 and insert a new line at line 49, the marker will not shift to line 51 to point at the same line of text but will stay at line 50 pointing at whatever text is there now. Contrast with XCode where the markers move with insertions and deletions of lines (at least as best as it can). This is logic that you’ll have to supply yourself.

More details, including performance notes, can be found in the Read Me file included in the project.

I’m putting this out there because I’m probably not going to use it and it seems like a waste of some useful code. Also, my apologies to the user who asked for this feature. I feel like somewhat of a jerk going through the trouble of implementing the feature and not including it. It was more of a fun exercise on my part but I still feel it’s not suitable for Hazel. That said, I may consider adding it and having it available via a hidden default setting. Votes for or against are welcome.

In the meantime, you can use the code however you want. MIT license applies. Please send me any bug reports, suggestions and feedback.

Enjoy.

Download Line View Test.zip (version 0.4.1)

Update (Oct. 6, 2008): Uploaded version 0.3. Fixes bugs found by Jonathan Mitchell (see comments on this post). Also made line calculations lazy for better performance.

Update (Oct. 10, 2008): Uploaded version 0.4. Fixes bugs mentioned in the comments as well as adds methods to set different colors. There is a display bug that happens when linking against/running on 10.4. See the Read Me for details.

Update (Oct. 13, 2008): Uploaded version 0.4.1. Figured out the 10.4 display bug. Apparently, NSRulerView’s setRuleThickness: method doesn’t like non-integral values. Rounding up solves the problem. Thanks to this page for identifying the problem.

Update (Sep. 29, 2009): I have included this class in my NoodleKit repository so you should check there for future updates.