Welcome everyone. My name is David Hayward and I am excited to talk about enhancements to Core Image, and its support for RAW image files.

I will talk about four main topics. First, I will review how RAW image files from cameras, are supported by Core Image on Apple platforms. And show you significant RAW quality improvements that are coming in iOS, iPadOS, macOS, and visionOS 27.

After that, I will discuss how to get optimal performance when rendering RAWs.

And lastly, I will describe features, that Apple added to the CIImageProcessor class for RAW.

To start, here is a quick summary of how Core Image supports RAW. RAW files come from a diverse set of camera makes and models, but unlike HEIFs and JPEGs, they need special handling before they can be displayed.

The first step, is to parse the file's metadata, and unpack the RAW sensor values. At this stage, each pixel location only has a red, green, or blue value arranged in a mosaic pattern. This image is cropped way in, so that this pattern is visible.

The next step, is to demosaic the sensor data, so that each pixel contains red, green, and blue values.

The following stage is to denoise the pixels, so that the image is free of photon noise, read noise and thermal noise. After this, convolutions are applied to sharpen edges and add local contrast.

The last major step is to adjust white balance, exposure, color and tone to make a pleasing final image.

The algorithms for all these steps are built in to iOS, iPadOS, macOS, and visionOS.

This means that people can view RAW files in system apps like Finder, Preview and Freeform.

In fact, any app or framework that uses the Image IO API gets RAW support automatically.

And beyond basic viewing, apps that use the CIRAWFilter API, can present advanced editing controls.

This API is used by apps like Photos, Pixelmator Pro, and others like Nitro, Acorn, and many more.

Starting back in 2006, the system included hand-tuned calibrations for just 21 camera models. In the years since, that number has grown to 784 models, across all major camera vendors. This even includes support for Apple Pro RAW files for iPhone cameras.

But the key feature of RAW, is that a beloved photo you took years ago, can be reprocessed using the latest state-of-the-art algorithms. Apple's RAW processing pipeline has been updated eight times, each improving demosaic, denoise and color. Some older versions have been maintained, so that people can still use them if desired.

Now, Apple has made its biggest update yet, RAW 9! This version dramatically improves the rendering of RAW files. It is built atop a tiled CoreML model, that combines demosaic with denoise for best quality. And the model is run on device using the Apple Neural Engine cores, for optimal performance.

Let me demonstrate the improvements up close.

This is a zoomed-in crop of a low noise image using RAW 8. This Sony Alpha 7 II image of a vintage dial indicator actually looks quite good. However, when you explore that same image under RAW 9, the image is sharper, clearer, and the fine text is easier to read.

The differences are even more dramatic, when you view high noise images. First, observe the actual RAW data that is contained in this very noisy ISO 51,200 image.

In this example from a Canon 5D Mark III, the image is a 10x Crop of a box of crayons. There is so much luma and chroma noise in the RAW data, that it's impossible to discern the unique color of each crayon. Using our previous algorithms, this is the result! RAW 8 did an acceptable job of recovering the actual colors in the scene. But if you examine the results under RAW 9, the output is significantly better. The colors are accurate and well defined. Even the shiny specular highlights on the crayons are visible.

This last example is a crop of a photo of embroidery yarn, shot with a Fujifilm X-T5 at ISO 12,800. This camera has a non-traditional sensor pattern, which is challenging to demosaic. In the RAW 8 results, there are some color artifacts and loss of detail in the yarn. But if you observe the same image under RAW 9, the results are discernibly better. The small text is more legible, and the texture in the yarn much clearer.

So here is how you can enable RAW 9 in your application. First, use the CIRAWFilter API to load a RAW file. This API is well described in the "Capture and process ProRAW images" from WWDC 21.

However, it is important to know that RAW 9 is not enabled by default. You should check that the supportedDecoderVersions property contains the version9 enumeration. If so, you can opt in by setting the decoderVersion property to version9.

To know what camera models support RAW 9, there's now a class method called supportedCameraModels. This API will provide your app, an array of all the models that are supported with a given version. In the release of iOS, iPadOS, macOS, and visionOS 27, there will be hundreds models that can use RAW 9, including all major professional camera vendors. This camera list will grow, via over-the-air updates to the operating system. RAW 9 is also automatically supported by cameras that shoot DNG natively, such as Apple iPhones.

The true power of the CIRAWFilter API is unleashed, when your app modifies its properties. This allows people to customize how RAW files are displayed.

There are currently 20 calibrated properties that can be adjusted. Adopt these adjustments in your app, to unlock the full power of editing RAW images. These are the most important controls for editing RAW: exposure, which controls how much to brighten or darken the image, luminanceNoiseReductionAmount, which adjusts how much fine luma grain is visible, sharpnessAmount, which determines how much the edges are sharpened, and contrastAmount, which alters how much local contrast is applied near edges.

All these controls work even better in RAW 9 than they did in prior versions.

There are some properties that are no longer needed in RAW 9. The colorNoiseReductionAmount property now has no effect, because the CoreML model handles color noise reduction automatically.

Also, the detailAmount and moireReductionAmount properties, are no longer needed nor supported in RAW 9.

You can call the is supported properties, to check if properties work for the filter instance.

Now that you have learned how RAW 9 appears and behaves, I would like to discuss its performance. Fundamentally, in order to get such quality improvements, RAW 9 is more performance and resource intensive than prior versions. Even though RAW 9 runs the CoreML model hundreds of times per image, when an app edits CIRAWFilter properties, subsequent renders are fast and responsive.

This is because Core Image caches intermediate results.

The best practices to get optimal performance, depends on how your app uses RAW files. I will give some recommendations for the two most common use cases.

First, here is advice on the interactive editing use case.

Interactive editing, is when one RAW file is rendered multiple times at screen resolution. Typically this is in response to the app adjusting a CIRAWFilter property, such as exposure or sharpness.

When interactive editing RAWs, use the scaleFactor property of the CIRAWFilter, when displaying the image at a reduced size. This is important because it reduces the work to render images that have more megapixels than the display.

Use one CIContext per view and set its cacheIntermediates option to true. Caching allows the intensive CoreML work to be skipped, while the app is adjusting CIRAWFilter properties.

Core Image will use more memory for caching between renders, if you add the "Extended Virtual Addressing entitlement" to your application. There is detailed documentation for this on Apple's website. Also, render directly to Metal-backed views. This improves performance of repeated renders, because Metal can start work on the next frame before, the previous frame is completed. For more information on rendering to a MTKView, consult the "Display EDR content with Core Image, Metal, and SwiftUI" from WWDC 22.

After that advice on the interactive editing use case, here are the most important tips when exporting files: The exporting use case, is when multiple RAW files are each rendered once at full resolution to other formats like HEIF or JPEG.

The best practice for this case is to export with a CIContext, with the option cacheIntermediates set to false.

Also, you can tell Core Image to use more memory during each export, by setting the context memoryLimit option. On iOS, the default limit is a conservative 256 megabytes. Setting the limit to 512 or 1024 megabytes can significantly improve performance.

And you can get extra memory savings by using the context methods heifRepresentation or jpegRepresentation, instead of calling Image IO directly.

In the last section, I want to talk about improvements to the CIImageProcessor API.

RAW 9 uses the CIImageProcessor API, because it enables algorithms that use CoreML in conjunction with other CIKernels.

As a result of this effort, the Core Image team added two features to the CIImageProcessor API that you may want to use in your app. The first I'll talk about, is support for explicit output tile sizes.

This is an example of a typical CIImageProcessor class. It implements the region of interest callback, that defines how much of its input is needed for a given output rectangle.

Most importantly, within the process callback, you must be aware of the input.region and output.region, that the processor should operate on. If there is plenty of memory, then CoreImage will call the process function, with an output.region for the entire image. But when memory is limited, the output.region might be considerably smaller.

Now, a processor can explicitly control output tile sizes. The code in your processor class remains as before, but you tell CoreImage to use the output regions of your choice.

First create an array, and then fill the array with all the desired tiles to cover your image. In this example, the tiling strategy takes the input image extent, and breaks it up into 512x512 pixel tiles. When you create an image which uses the processor, call its apply method and pass in the tile array that covers the image.

So now that I've described explicit output tile sizes, my last topic is the temporary buffers feature.

It is common for a CIImageProcessor that calls CoreML to use temporary buffers. This is because CoreImage uses interleaved image buffers, which must be converted to planar data for CoreML. When a processor callback is called for many tiles, the temporary buffers will be created and destroyed repeatedly. This can impact performance.

The CIImageProcessorOutput class, now has methods to help. Here's how this works! Here's an example of a simple CIImageProcessor class that uses the temporary buffer feature.

Instead of processing directly, from input to output, this callback requests a scratch buffer by asking the output object for a temporary CVPixelBuffer.

When doing so, provide an identifier. This is vital for process callbacks that use more than one temporary buffer.

Then, this example processor copies from the input pixel buffer to the temporary buffer, alters the temporary buffer pixels in-place, and finally copies that to the output. Core Image will manage the lifecycle of temporary buffers for you. They will be released automatically and released at the correct time and recycled when the processor is called for the next tile.

Here are the key points to take away.

Try RAW 9 in your app. It's a major quality improvement and it's just a few lines of code to enable.

Follow the performance best practices, I outlined for fast exporting and responsive editing.

Give your app access to the powerful editing properties of CIRAWFilter, that let people finetune how their RAW images appear.

And finally, for optimal performance in your CIImageProcessor, here use the explicit tiling and temporary buffer APIs. Personally, I have really enjoyed revisiting my library, of over 7000 family photos that I have shot in RAW over the past 20 years. It has been really great to see the improved quality of noise reduction. I think that people using your app, will also enjoy these improvements. Thank you for watching!