CocoaDust

I attended WWDC last week, it was good, and while I can’t talk about most things, I would like to talk about what Apple has made public.

So excuse me if this isn’t an original thought, but after what people are calling Steve Jobs’s worst keynote ever I did some thinking and had a lot of discussions with people about what that really means for Leopard in general.

The thing about Leopard is that is shows how mature the OS is now when the biggest thing they can trumpet is backup software (Time Machine), which looks to be a fantastic implementation in demo form, and there’s Spaces, the stacks in the Dock are pretty handy and the consolidated UI is really, truly welcome. And there’s more to it than just that, of course.

So Leopard actually sorts out a lot of things that have been annoying people, but what many people won’t be aware is just how much stuff it adds for developers – more than any release of Mac OS X that I can think of and I’ve seen them all. Also, they really seem to be taking the HIG bull by the horns and reining in the craziness. The Apple Design Awards showed that pretty well too.

I heard someone say (and can’t remember who – sorry) “Apple seems to be counting on developers to sell Leopard”. No bad thing!

That keynote sucked though. Oy.

I appear as a guest discussing Search Kit, the framework for indexing and searching files and text on episode 14 of the Late Night Cocoa podcast, out now.For anyone who doesn’t know, Late Night Cocoa looks at a different topic every week with a guest developer and is suitable for both novice and experienced Cocoa developers. It stands out because its host, Steve Scott, aka Scotty, does an excellent job producing a podcast that is not only useful and informative but also sounds good too. A lot of work goes in behind the scenes to make that happen.And, as always with a podcast, you can listen to what you want when it suits you, so if one topic doesn’t interest you, there are bound to be plenty more that do. So, check it out.

By now (hours after the announcement) Leopard’s postponement until October is not news, but I’m just documenting my thoughts.

I’m a little disappointed, it means I have to rearrange my plans a little, do more work, etc but on the other hand it gives me breathing room and, crucially, we finally know where we stand, because that is the most difficult thing.

You see, it’s one thing if the OS is planned to come out before you expect to release – fine. It takes a while for everyone to get up to speed. But, when you’re (ahem) developing a product that could depend on or benefit from Leopard technology, you don’t want it sitting around the shelf for months. For anything more than a few weeks, that’s not an option for me.

Now we know where we stand I have a choice: to work on something else or target Tiger.

Apart from the WordPress blog, the dynamic parts of my Reinvented Software site is all Ruby on Rails. The pages there could be better (i.e. sexier), but I ran out of self-imposed time. I spent a few weeks last year rewriting those pages and what used to be an entire J2EE backend in Rails and felt cleansed as a result.

(Yes, I know J2EE was complete overkill – it would be for running the Universe – but when I set up the business, I wanted to keep my hand in just in case things didn’t work out and I’d end up back in corporate hell. Thankfully, that didn’t happen.)

My biggest concern was how to serve the files. My server is running Apache 1.3.x, I could have upgraded to Apache 2.0 but that would been a lot of work. I also heard Apache 2 working with FastCGI could be quite heavy on resources and that was off-putting. I could have moved entirely to lighttpd, but that was too scary, so I decided it would be better to serve the Rails stuff with lighttpd and have Apache do the rest.

I wrote about this on my personal blog at the time (boring my non-geek friends in the process). From the original post:

Apache proxies requests to Lighttpd to serve the Rails stuff (so, for example, you see http://reinventedsoftware.com/store/ instead of http://reinventedsoftware.com:3000/store/), it all works seamlessly.

While I could find instructions out there on how to get Apache proxy to Lighttpd to serve an entire Rails application from a specific URL (e.g. http://reinventedsoftware.com/myrailsapp/) this mixing and matching wasn’t documented – so I will probably put something about this on the Rails Wiki. It was quite straightforward once I had worked it all out.

Zach Hale came across the post and asked how to set up Apache to work alongside lighttpd, so here is how I did it. Note these instructions are for Apache 1.3.x, I don’t know how it would work with Apache 2:

Apache Setup

In Apache’s httpd.conf, ensure mod_proxy is loaded:

LoadModule proxy_module       libexec/libproxy.so

Then use the ProxyPass and ProxyPassReverse directives to lighttpd for certain paths (note the standard Rails stylesheets and javascripts directories also):

ProxyPass /news http://localhost:3000/news
ProxyPassReverse /news http://localhost:3000/news

ProxyPass /support http://localhost:3000/support
ProxyPassReverse /support http://localhost:3000/support

ProxyPass /stylesheets http://localhost:3000/stylesheets
ProxyPassReverse /stylesheets http://localhost:3000/stylesheets

ProxyPass /javascripts http://localhost:3000/javascripts
ProxyPassReverse /javascripts http://localhost:3000/javascripts

Lighttpd

In lighttpd.conf ensure server.modules includes mod_alias and mod_proxy:

server.modules           = ( "mod_rewrite", "mod_accesslog", "mod_fastcgi",
                             "mod_compress", "mod_expire", "mod_alias", "mod_proxy" )

And further down, add this to handle the aliasing:

$HTTP["url"] =~ "^/" {
  server.document-root = CWD + "/public/"
  alias.url = ( "/" => CWD + "/public/" )
  accesslog.filename = CWD + "/log/access.log"
  server.error-handler-404 = "dispatch.fcgi"
  server.errorlog          = CWD + "/log/lighttpd.error.log"
  server.indexfiles = ( "index.html" , "dispatch.fcgi" )
  # rails stuff
  fastcgi.server = ( ".fcgi" =>
    (
        (
       "socket" => CWD + "/log/code.socket",
       "min-procs" => 2,
       "max-procs" => 2,
       "bin-path" =>  CWD + "/public/dispatch.fcgi",
       "bin-environment" => ( "RAILS_ENV" => "production" )
  )))
}

I barely understand this stuff, so don’t ask me to explain it.

Finally, I created symlinks to all the root level folders in the Apache served public_html directory, so that lighttpd could find the files when the user navigates. Here’s a cut down ls -l:

dispatch.rb
dispatch.fcgi
dispatch.cgi
stylesheets
javascripts
support -> ../../../public_html/support/
news -> ../../../public_html/news
feeder -> ../../../public_html/feeder
kit -> ../../../public_html/kit
about -> ../../../public_html/about
blog -> ../../../public_html/blog
404.html
500.html

So, it’s a bit of a hybrid system. Apache is in control and serves most files, but lighttpd takes care of all the Rails stuff. It’s been running for 10 months now through some very heavy moments and works really well.

In this Ars post on Leopard’s ship date (I’m pretty sure it will be around WWDC, BTW), the paragraph that gets the most attention in the comments is this one:

One more tip we got regarding Leopard, is that InputManager plugins are no longer allowed. That’s right… no more little hacks from anybody besides Apple. No more Apple menu hacks. No more Safari plugins. (InputManager is not exactly the same as APE, by the way.) “Apple isn’t really broken up about it since InputManagers were often used for nefarious purposes anyway,” our sources said, but the loss of InputManager control will break a lot of shareware and commercial software that currently makes use of that control.

I have to say, this will be a huge help to me. The trouble with Input Managers is that they load into every application, and the problem with that is, they are almost always involved in the crash reports I receive for KIT. The telltale sign is when the crash is happening entirely in Apple code, sometimes drawing standard widgets, sometimes just as part of the run loop, with the Input Managers listed below.

The trouble is that KIT gets the blame. I’m not sure why KIT is so susceptible to these, as Feeder hardly seems to get affected, but I suspect it’s for a combination of reasons.

Firstly, the sort of people who use apps use KIT like the enhancements these Input Managers provide. Secondly, KIT uses a whole heap of Apple frameworks and is always swapping views in and out. Finally, KIT is very multithreaded, scanning for changes to files, indexing, searching, etc. Maybe some of the problems are due to the way KIT is designed, but I have no way of finding out.

I have no issue with people running Input Managers, as long as they accept that this means their set up is unsupported when crashes occur.

The simple fact is the application works perfectly on a regular system and I can’t fix problems in code that aren’t mine, so less hour spent sifting through crash reports that have nothing to do with me, the better.

One thing that particularly interests me with Spotlight is how an application like my own Keep It Together can set additional metadata attributes on files owned by generic or other applications’ Spotlight importers.

KIT keeps all the original files on disk, not in a database or file package, so they can be found by Spotlight and indexed by the relevant importers. What’s missing right now is the extra metadata from KIT itself – rating, tags, etc, that isn’t stored in the actual files.

iPhoto works in a similar way, so it’s a good case study. Metadata in the image files in iPhoto’s library will be indexed using the generic Image importer. iPhoto adds some attributes of its own, such as a rating and keywords. Looking at iPhoto’s importer, it only works on .ipspot files. iPhoto writes out a single iPhoto.ipspot file, which has one line pointing to the XML library that changed.

At first, I wasn’t sure how the importer could handle that, since a number of different items can be changed in one operation, yet the importer can only return details for a single item, namely the file that triggered the change.

I did some digging and it seems the iPhoto importer doesn’t return details for the .ipspot file at all, but rather writes out the file to trigger the importer, which manually sets metadata attributes on the files in its library using a private function, MDItemSetAttributes. Looking at the Metadata framework, there’s also a corresponding MDItemSetAttribute function, also private.

Boolean MDItemSetAttributes (MDItemRef item, CFDictionaryRef dict);
Boolean MDItemSetAttribute (MDItemRef item, CFStringRef name, CFTypeRef attribute);

Caution would be wise with that approach because, private function aside, performance is a crucial factor. The importer needs to do its work quickly as it will be called every time the trigger file is changed.

Another thought is that you could actually use these functions in your application, rather than rely on an importer to do the work for you, but as the function is undocumented, there is no way of knowing whether that would also cause issues.

The missing piece of the jigsaw here is how iPhoto opens and selects the image whenever the file is chosen from Spotlight’s results, since opening the file in the Finder would normally open its default application, such as Preview, Photoshop, or whatever. That’s not something I’d want to do for KIT, but it would be interesting to find out how it’s done. If anyone knows, please comment.

Update: My understanding now is that selecting the photos in iPhoto as mentioned above is something entirely hardcoded into Spotlight and not something possible for just any app. See the AppleScript statements in /System/Library/CoreServices/Search.bundle/Contents/Resources/iPhotoScripts.plist

One of the interesting things about Spotlight is that it only works on individual files. When it sees that a file has changed on disk, it uses the relevant importer to update the metadata store.

For each time it is called, a Spotlight importer can only provide data for a single item that will be shown in Spotlight’s results. A different approach is required for multiple items, such as bookmarks in your web browser or iCal events, that are contained in a database or single file (including file packages, because Spotlight won’t index the files inside).

It’s a very simple workaround, but seems a little clumsy, considering. Every time such an item is changed, you need to change a file on disk that can trigger Spotlight to invoke the appropriate importer.

To use iCal as an example, in ~/Library/Caches/Metadata, you’ll see a number of folders for different applications, including iCal itself. Looking in that folder, you’ll see a number of other folders with UUID-style file names that refer to the calendars in the application, and in those folders various files with .icalevent or .icaltodo extensions. These files contain data that the iCal importer will use to provide metadata to Spotlight. It’s iCal’s responsibility to ensure those files are created, updated and deleted as necessary.

In some cases, looking at files in the Metadata folder can reveal an surprising amount of duplicate data, with files filled with text. This is probably because it would be too much of an overhead, or too difficult for the importer to extract the actual data from the database.

If you’re designing a new application that uses a database or single-file format to contain multiple items that you want indexed by Spotlight, you should bear this in mind.

That last post was a tangent from my original thought bubble of when we were likely to get a ship date for Leopard and the futility of planning for future versions of Mac OS X.

It matters to me because I need to know what’s coming and plan for it. I have a fair idea of what I am doing for the rest of the year; what makes things difficult is the way software sales nosedive during the summer months.

Timing is everything. I need to get major releases out during peak periods, no good releasing software if everyone is up a mountain or at the beach. If Leopard is going to be out in June, just before sales start to dip (I think Leopard can get away with such seasonal trends) then great, I can do my best to get updates out before then.

However, if it’s going to be around April or early May and I need to do Leopard-specific work (because we just don’t know) then my current plans would need some adjustment, because there are plenty of other things I could be doing too.

Of course, if Apple thought a lot of work would be required, they’d let us know about it sooner, but I work up to the wire. It is so easy for my time to be devoured by events in ways that simply allow me to maintain the status quo. This year I am really determined to go beyond that.

So, Apple has succinctly redefined itself with the Macworld keynote in name, while the transformation has been taking place over the last few years it now seems somehow more substantial. One could easily say this started with the iPod, but arguably the iMac was Jobs’s first real statement following his return to the company that pitched Apple in a different direction to its business-oriented competitors. That path led us here.

I think developers in particular could feel a little slighted by the Macworld keynote and iPhone. As it stands the phone is not extensible, available or even suitable for everyone’s phone contract arrangements, but it is utterly desirable, runs Mac OS X and more than worthy of Apple brand, so that’s good. The trouble is what wasn’t announced at the keynote leaves developers and consumers hanging. I come at this from the developer perspective and will concentrate on the software here.

What’s happening with Leopard? It seems pretty certain we will see it by the time of the launch of the iPhone, if not before, and that Leopard is partly the cause of the delay. iLife, iWork? Well, maybe they are waiting for the big cat too, which would suggest that they will truly embrace Leopard technology in a way no iLife suite has done with the current OS before, and then the possibilities boggle the mind.

The annual update cycle now seems broken, maybe permanently. Perhaps not a bad thing, it is quite a demanding and artificial timeframe for any developer. It hardly seems like Apple would launch these products without fanfare as the very appeal of the Mac on a consumer level begins with iLife; the iPod, .Mac, soon AppleTV and later the iPhone are all affected by it in some way, and the suite influences and impacts countless third party applications.

So what do we know now and where could that lead us?

We know, because Mr Jobs told us, that there is more to Leopard than we have been shown. Many people think this will bring an updated UI, and sure enough there are aspects to the iPhone that have an iTunes 7ish look about them, although we do not know that is final or that it will have anything to do with the desktop version of Leopard. Seems likely though, given the fact that the phone runs a version of OS X and Apple is keen on making a point of this.

We know that Apple really *should* show such UI updates to developers some time before the release goes out, because everyone (including Apple) is using a billion custom controls to compensate for a lack of such standard controls in shipping versions of the OS and that anything more than minor changes could prove rather ugly.

Trouble is, we don’t know that is coming, but assuming there is something to significantly revamp the Mac OS X experience in Leopard and that this something would appear in developer builds, then such an update would get leaked in a heartbeat, so it really needs to be shown to the public first or there goes any ooh-ahh announcement, without which Steve Jobs simply wouldn’t exist. Vanish in a puff of smoke, he would.

It’s also safe to assume we know updates to the iLife and iWork applications are coming in whatever shape and form. To go beyond the middle of the year with a pair of suites suffixed with ‘06 will just be too… last year. I think AppleTV is planned to work as things are, so its February availability doesn’t matter here.

All this seems to narrow the timescale for a preview of Leopard to around April at the latest and I would predict (hahaha!) sometime by the end of March, for no other reason than a gut instinct. On iLife, who can be so sure? Maybe we won’t get to see that until the summer. iWork seems neither here nor there unless spreadsheets get you moist. And, of course, there is everything we don’t know because it doesn’t exist yet. There could be a lot of that.

I think the iPhone is the first shot of what should turn out to be an exceptional year for the Mac and it seems all the more fitting that Apple is keeping us in suspense on all of this.

Note to long-term followers of my various blogs: Yes, I’ve already had to resort to song titles in order to name my posts. Things can only get better.

I’m not really one for Macworld predictions, unless I have my eye on something (a new Mac, etc) and then it’s more a case of whether it will appear, not what it will be. It’s the same with gifts – it spoils the surprise for me to guess what’s inside. So rather than have expectations and end up disappointed, I don’t pay much attention to the rumours.

Of course, there could also be announcements that affect my business, but those would just cause me to worry and worrying doesn’t do much good either. I can’t plan for speculation, but I can anticipate it. Last year I was pretty sure there would be something to do with podcasting in iLife (I was tipped off, and it was pretty obvious in hindsight) and there was, but it wasn’t really a threat to my business.

One thing I look forward to every year are the Mac OS X demonstrations. This year’s should be good as it was clear a few things were held back from last year’s WWDC. With Vista gone GM, I expect we’ll see some new stuff next week and (very importantly for me) maybe even get a release date for Leopard. Even if we don’t, it’s always a great time to be a Mac user as there should be plenty of surprises in store. I like surprises.

Update, Jan 9th: Or maybe I could forget about all that and just sit here for two hours gawping at a presentation about a phone. But truly, it deserved it. See – surprises!