Developing iPhone raw images using MATLAB

This guide will describe how to capture and process raw (undeveloped) images from an Apple iPhone. Typically, the jpeg files from a modern smartphone are excellent for most use-cases, and if not there are nice commercial and open-source raw development tools out there. The main reason for going the route described here is that you want to gain some insight into what goes on, i.e. that you are a tinkerer.

I make no claims of novelty or finesse, most of this could be learned by spending some time googling (see references at the bottom), but I have presented it in the way that made sense to me.

Requirements:

• An Apple device running iOS 10 or later
• A third party photo app (e.g. ProCam)
• A computer
• Adobe DNG converter
• EXIFtool (recommended)
• MATLAB (no toolboxes needed)

Capturing images on the phone

The built-in camera app of iOS devices does not support the system APIs that Apple provide for doing raw image capture. Some 3rd party apps do, I have used ProCam, where the RAW option is prominently featured in the UX. Snap some image and transfer it to your computer.

Convert compressed DNG to uncompressed DNG

It is great that Apple chose to use a standard format for their raw image files. Unfortunately, they are using a lossless compressed variant that is not supported directly in MATLAB. Fear not, as we will use the (free) Adobe DNG converter to decompress the DNG content. You need to select settings->compatibility->custom->uncompressed.

Reading raw DNG files in MATLAB

Use the built-in tiff library to load the uncompressed DNG file:

fname = 'my_iphone_image.dng';
info = imfinfo(fname);
info.SubIFDs{1}
warning off MATLAB:tifflib:TIFFReadDirectory:libraryWarning
t = Tiff(fname,'r');
offsets = getTag(t,'SubIFD');
setSubDirectory(t,offsets(1));
cfa = read(t);
close(t);

The variable cfa now contains a 2-d array of uint16 image data:

>> whos cfa
  Name         Size                 Bytes  Class     Attributes

  cfa       3024x4032            24385536  uint16 

Convert it to (single-precision) floating-point:

cfa = single(cfa);
[h,w] = size(cfa);

Simple debayer

A pretty minimal debayer, doing bilinear interpolation on each color plane in isolation. The cfa_pat variable is hardcoded for my iPhone 6s in landscape mode. You might have to shift things around in other cases (or parse the meta-data properly).

cfa_pat = cat(3, [1 0; 0 0], [0 1; 1 0], [0 0; 0 1;]);
im_cfa = repmat(cfa_pat, h/2, w/2) .* cfa;
 
rb_kernel = [1 2 1;...
             2 4 2;...
             1 2 1]./4;
g_kernel =  [0 1 0;...
             1 4 1;...
             0 1 0]./4;
kernels = cat(3, rb_kernel, g_kernel, rb_kernel);
for ch = 1:3
    rgb(:,:,ch) = conv2(im_cfa(:,:,ch), kernels(:,:,ch), 'same');
end

Remove bias/normalize:

The digital system of the camera (e.g. A/D converter) might not represent pixel values as numbers representing linear light. By compensating for the indicated black level and white level, we get a normalized number that does.

bl = info.SubIFDs{1}.BlackLevel;
wl = info.SubIFDs{1}.WhiteLevel;
rgb = (rgb - bl)/(wl - bl);

White balance

Doing "as shot" WB is our goal here (whatever the camera chose):

rgb_stripe = reshape(rgb, [], 3);
wb = info.AsShotNeutral(2)./info.AsShotNeutral;
rgb_stripe = rgb_stripe .* wb;
rgb = reshape(rgb_stripe, h, w, []);

Color correction

Let us get some header info from EXIF tool. This can be skipped, but then you need to provide your own color correction data below.

[status, result] = system(['/usr/local/bin/exiftool ', fname])

Manually enter the Color Matrix 1 (or 2) from the EXIF tool result. The DNG standard wants you to interpolate between the two depending on illumination (1 is "Standard Light A", 2 is D65), I just picked the one that looked best to me for this scene. Use sRGB->XYZ from Bruce Lindbloom.

XYZ_to_RAW = [0.7762 -0.249 -0.0233;...
             -0.6623  1.3668 0.3295;...
             -0.1233  0.1441 0.6442];
sRGB_to_XYZ = [0.4124 0.3576 0.1805;...
               0.2126 0.7152 0.0722;...
               0.0193 0.1192 0.9505];
sRGB_to_RAW = XYZ_to_RAW * sRGB_to_XYZ;
sRGB_to_RAW = sRGB_to_RAW ./ repmat(sum(sRGB_to_RAW, 2), 1, 3);

Now we can generate camera raw colors from sRGB. Invert to do the other way around, and apply:

RAW_to_sRGB = sRGB_to_RAW^-1;
rgb_stripe = reshape(rgb, [], 3);
rgb_stripe = rgb_stripe*RAW_to_sRGB';
rgb = reshape(rgb_stripe, h, w, []);

Gain and clip:

Play with the gain until a subjectively "pleasing" brightness is reached.

gain = (1/4)/mean(rgb(:));
rgb4 = gain.*rgb;
rgb4(rgb4>1) = 1;
rgb4(rgb4<0) = 0;

sRGB Gamma:

Our displays usually expects a non-linear distributed 8-bit value:

threshold = 0.0031308;
out = rgb4;
out(rgb4<=threshold) = 12.92*rgb4(rgb4<=threshold);
out(rgb4>threshold) = 1.055*rgb4(rgb4>threshold).^(1/2.4)-0.055;
out = 255*out;

Finale

Have a look at the results (top) versus jpeg straight out of phone (bottom) (note that these are two similar but separate captures).

imagesc(uint8(out), [0 255])

The jpeg straight out of the device appears to have some deep blacks tone mapping and far more saturated colors than this raw conversion. As expected, it also appears to feature sharpening and noise reduction.

Resources