King Software Designs

An app I’m working on uses the AVFoundation API’s for direct access to the camera.   Soon after opening the camera it’s clear how easy it is to shoot yourself in the foot juggling 5mp images around.   My first approach used a Core Data “cache” that I could use to get images at specific sizes, and resize them and store them in the cache if required.   It worked, was quick to develop, but slow as a dog!   Startup and runtime performance was pretty good, but writing new images from the Camera were in the 5-6 second range.  Obviously not acceptable!

Core Data should not be used to store large blobs.

I knew it when I wrote the code, so this wasn’t a shock, but boy does this cause a lot of thrashing.   I learned a lot from the WWDC10 Advanced Performance Optimization on iPhone part 2 – A very good presentation, one I wish I revisited sooner.  Apparently, blob’s under 2k are fine, but blobs over that interefere with SQLite’s page caching.   Marcus Zarra advises keeping Blob’s under 100k.    Regardless who’s advice you take, do not store 5mp images in Core Data.

Timestamping Image Operations can be deceiving

In the sake of efficiency, where the image is decompressed varies, and printing out timstamps is not accurate for all actions.   For example, I can setup a CGImageRef to my 5mp image in .02 seconds on an iPhone4!   However, when Quartz goes to render the image, CAPrepareTransaction balloons up decompressing the image on the fly.   This is a great feature, but it will force your technique for profiling data.   This is a general issue for profiling.   What occurs under the hood on most of the functions I’m investigating is completely context dependent.   All the times in this article I attempt to isolate the work being done over a chunk of time.   I did some investigation with Instruments, but all the times were taken with timestamps.

Getting Data from the Camera

When you setting up a AVCaptureStillImageOutput, make sure the AVVideoCodecKey is ‘jpeg’.    BGRA will give you faster access to the bytes if you need to do on the fly processing of the bits, but that is only a concern if you’re doing image processing.   Getting the jpeg bytes to disk took around .7 seconds, as opposed to *6* seconds for BGRA.    The average time to get access to the CMSampleBufferRef was .5 for jpeg and 2.0 for BGRA.  I’m guessing the increased access time is due to the increased memory bandwidth and the increased write time due to compressing the software on the CPU instead of dedicated hardware.   I was always using jpeg output, but did some tests with BGRA to see if they would resize faster.   They didn’t.

Do not fight the Compression

Using JPEG  is great for getting the data to disk, but getting a UIImage I can use in my app was still slow.   To resize using Core Graphics, you still need a CGImageRef of the 5mp image to downsample, meaning the image needed to be decompressed.   Resizing to 1024 was taking 1.4-1.8 seconds.   What this was doing was probably a combination of resizing, memory IO and jpeg compression.   I tried a few different methods of resizing but nothing dented the times.   This approach gave me a 2 second delay before I could show an image to the user.

ImageIO

New in iOS4 is CGImageSource, which can efficiently create thumbnails with CGImageSourceCreateThumbnailAtIndex.    It appears to be smart with resizing jpegs, using the format to it’s advantage and creating jpegs without de-compressing the whole image.   It also claims to store the thumbnail inside the image for faster loading in the future.   This is something I’m still tinkering with, but just changing to this access method cut my resize time down from 1.4-1.8 seconds to 0.7-0.9 seconds.   A nice speed up!

Writing to disk

Finally to get that image to disk, write the original NSData blob to disk atomically.   When I wrote the image with CGImageDestination and copied over the image source, it was in the 2-3 seconds vs 0.1 seconds for NSData’s writeToFile:atomically:.   Again, do not fight your compression!    I was hoping that CGImageDestination would write out the thumbnail I created, but it didn’t.   I’ll have to look at that again, my hunch is that the format of the image is not jpeg2000 and that regular jpeg can’t store thumbnails.   Oh well, I can cache in a separate file!

Grand Central Dispatch

Once the timing is tuned, and the memory usage is efficient as it gets, take everything that takes time off of the main thread.    I originally put a spinner on the camera button because I had would get an OOM crash if a few snaps backed up onto eachother.   Now I can take about 30 images before the low memory killer strikes – I’m guessing a backup of 10-15 images.

Hope this was informative, and if you are doing anything with the camera, I urge you to review the WWDC Performance Optimization presentations.   They are great!

I finally wrapped up some of my efforts with Pocket Sphinx into a library, VocalKit.   My main goal is to make it easy for developers to over come the setup and learning curve with Pocket Sphinx in their iPhone / iPad apps.   VocalKit is a set of Dual-Architecture static libraries for PoketSphinx and Flite, and an Objective-C Wrapper that integrates with AudioQueue.   Also there is a Test Project you can build and run on your iPhone with out any code modification.   I hope to add Julius support in the future.

Still plenty of work to do, but it’s a good start!

I was looking for a way to tune an animation to get the right feel.  It was getting frustrating to modify, rebuild, recompile and retest for every little change.    When the tuning is more of a feel than a right / wrong check, this was driving me mad.   By the time you changed the code, the feel was gone!

I remembered two XCode features that I thought might help, Runtime variable modification and  Breakpoint Actions.

• Variable Modification

If  constants are moved into the function that starts the animation and they are static, they show up in the debug window.

If variables are static, the state sticks around to subsequent invocations.  This allows you to tune the constants, get a feel for it, and tune it again.

• Breakpoint Actions

Another thing that helps when getting a feel for how your code is interacting with the phone is to add breakpoint sounds.   This will can cause XCode to play a sound or print a variable.   Select the breakpoint in the breakpoint group in the main tree, expand and press the + button.

This wasn’t as helpful as I thought as the sound plays inline and causes a UI hickup.   Still an interesting trick so I thought I’d throw it out there.  Also while I was mucking around I found that you could print out variables right in GDB, which could be nice if you want to log something but not rebuild.

Anyone have any other good XCode tricks?

I often find the need for a rails backed iPhone app. I do some modeling of my domain objects in the iPhone and get the basic lay of the land.  Then I go off and create the rails world, determine the resource structure and add a JSON interface to the rails app.  Then I go back to the iPhone and add the HTTP Requests and JSON loading. As the app develops, this definitely diverges, but for the initial spike, I keep thinking “Can’t I just generate my rails model off Core Data?”.

Enter MOGenerator, which I think everyone should be using on their CoreData models. It is a project that links a template language (MiscMerge, an apparently abandoned package) with the NSEntityDescription objects to build great wrappers for CoreData. This separates your code from the machine generated code and keeps you from dreading changes to your data model. Turns out this template language can generate any text. My first thought was to use this to generate rails code, and soon there after I realized how much work that would be. I don’t want Core Data to be the authority of my rails data model though, I just want to cut out some tedious work in bootstrapping the rails app.

So I modified MOGenerator to… generate ./script/generate commands.  So yes, generating a shell command to generate rails code.   It doesn’t attempt to be a perfect (or complete) mapping of CoreData to Rails, just to save some time.

The modifications to MOGenerator are very simple. MOGenerator is dynamic in what it can generate in it’s template files, BUT it hardcodes filenames in the program.  My fork pushes the filename selection out into the template. I found another github project that forked just to change the hardcoding, so I’m hoping that rentzsch incorporates the patch.

Moving forward I see two main things

• Modifying templates to use [EntityName].[hm] and [EntityName]+custom.[hm] rather than inheritance. It’s just cleaner in my book to seperate with Categories rather than inheritance.
• Pushing some busy-work JSON loading into [EntityName]+loadJSON.[hm]. This would define the JSON keys for each attribute into userInfo if it’s source was JSON.

And if I were to dream, I see pushing the authority for my CoreData model into rails and auto-generating that from my rails model. That way if I wanted an object or attribute to show up in my app, I could add a key and it would get added to the whole stack, and add versioning support…..   But reality quickly squashes that dream !

BrowserCMS is a new CMS system for rails that was “first” introduced in their talk at acts_as_conference 2009.   Watching the last 10 or so minutes will give you a feel for the Usage Model

A solid CMS solution for my clients is essential for my peace of mind! There’s my value add which is the web application, and then there’s the rest of the site. Mixing and matching PHP CMS solutions or static HTML files has been the bane of my life for a while now.

One thing that I really enjoy about the project is that there’s a company committed to it’s success. That’s one of the biggest deciders for me in any open-source project. I loaded up the demo, and the UI looked very nice by default which is always encouraging.

So the final arbitor is to look under the hood and see what exactly is there.

• Basic Design

BrowserCMS is composed of a library of behaviors and  MVC objects that utilize these behaviors.  These behaviors are usually pulled into the Model, and the CMS Framework allows the management of these various Models.   Then a site is constructed entirely with these model objects and managed by the CMS interface.   I loved how they loaded the demo site.

You can also add your own objects by subclassing ContentBlock.   I need to play around more to get a feel for how much work it is, but it looks like you will get a lot of knobs in the CMS system for versioning, publishing and connecting into the site map for ‘free’.

• Code

All the code is very well laid out and readable.   I was largely interested in how they implemented Versioning, and it didn’t pervert the model it was versioning which I was curious about.   Basically the columns that you want versioned are specified and then another table keeps track of all the modified versioned columns.   All the features I investigated seemed to encapsulate specific features and function independently.

• Community

I just jumped on the googlegroup, and decided to take on a little cleanup that I would like to see, just to get my hands wet.   Basically none of their Model objects are in the Cms Module, so I can’t have their code live along side my code temporarily while I do the migration.   I could easily just move my code around, but it seemed like a good excuse to get playing around with an opensource project!

• Authentication

Out of the box permssions and groups works well.   It has an implementation generated from Restful Authentication.   It says ‘Use standard Authentication tools’ on the wishlist, so we’ll see where that goes.

Also, for future, more public sites, I am hooked on open-ID logins, like the system provided by RPX so hopefully the plugin system can support an open-id system.

Looking forward to playing around some more!