Managing Camera Data
An app I’m working on uses the AVFoundation API’s for direct access to the camera. Soon after opening the camera it’s clear how easy it is to shoot yourself in the foot juggling 5mp images around. My first approach used a Core Data “cache” that I could use to get images at specific sizes, and resize them and store them in the cache if required. It worked, was quick to develop, but slow as a dog! Startup and runtime performance was pretty good, but writing new images from the Camera were in the 5-6 second range. Obviously not acceptable!
Core Data should not be used to store large blobs.
I knew it when I wrote the code, so this wasn’t a shock, but boy does this cause a lot of thrashing. I learned a lot from the WWDC10 Advanced Performance Optimization on iPhone part 2 – A very good presentation, one I wish I revisited sooner. Apparently, blob’s under 2k are fine, but blobs over that interefere with SQLite’s page caching. Marcus Zarra advises keeping Blob’s under 100k. Regardless who’s advice you take, do not store 5mp images in Core Data.
Timestamping Image Operations can be deceiving
In the sake of efficiency, where the image is decompressed varies, and printing out timstamps is not accurate for all actions. For example, I can setup a CGImageRef to my 5mp image in .02 seconds on an iPhone4! However, when Quartz goes to render the image, CAPrepareTransaction balloons up decompressing the image on the fly. This is a great feature, but it will force your technique for profiling data. This is a general issue for profiling. What occurs under the hood on most of the functions I’m investigating is completely context dependent. All the times in this article I attempt to isolate the work being done over a chunk of time. I did some investigation with Instruments, but all the times were taken with timestamps.
Getting Data from the Camera
When you setting up a AVCaptureStillImageOutput, make sure the AVVideoCodecKey is ‘jpeg’. BGRA will give you faster access to the bytes if you need to do on the fly processing of the bits, but that is only a concern if you’re doing image processing. Getting the jpeg bytes to disk took around .7 seconds, as opposed to *6* seconds for BGRA. The average time to get access to the CMSampleBufferRef was .5 for jpeg and 2.0 for BGRA. I’m guessing the increased access time is due to the increased memory bandwidth and the increased write time due to compressing the software on the CPU instead of dedicated hardware. I was always using jpeg output, but did some tests with BGRA to see if they would resize faster. They didn’t.
Do not fight the Compression
Using JPEG is great for getting the data to disk, but getting a UIImage I can use in my app was still slow. To resize using Core Graphics, you still need a CGImageRef of the 5mp image to downsample, meaning the image needed to be decompressed. Resizing to 1024 was taking 1.4-1.8 seconds. What this was doing was probably a combination of resizing, memory IO and jpeg compression. I tried a few different methods of resizing but nothing dented the times. This approach gave me a 2 second delay before I could show an image to the user.
ImageIO
New in iOS4 is CGImageSource, which can efficiently create thumbnails with CGImageSourceCreateThumbnailAtIndex. It appears to be smart with resizing jpegs, using the format to it’s advantage and creating jpegs without de-compressing the whole image. It also claims to store the thumbnail inside the image for faster loading in the future. This is something I’m still tinkering with, but just changing to this access method cut my resize time down from 1.4-1.8 seconds to 0.7-0.9 seconds. A nice speed up!
Writing to disk
Finally to get that image to disk, write the original NSData blob to disk atomically. When I wrote the image with CGImageDestination and copied over the image source, it was in the 2-3 seconds vs 0.1 seconds for NSData’s writeToFile:atomically:. Again, do not fight your compression! I was hoping that CGImageDestination would write out the thumbnail I created, but it didn’t. I’ll have to look at that again, my hunch is that the format of the image is not jpeg2000 and that regular jpeg can’t store thumbnails. Oh well, I can cache in a separate file!
Grand Central Dispatch
Once the timing is tuned, and the memory usage is efficient as it gets, take everything that takes time off of the main thread. I originally put a spinner on the camera button because I had would get an OOM crash if a few snaps backed up onto eachother. Now I can take about 30 images before the low memory killer strikes – I’m guessing a backup of 10-15 images.
Hope this was informative, and if you are doing anything with the camera, I urge you to review the WWDC Performance Optimization presentations. They are great!
