At Halide HQ, we’ve been busily working away on a very deep, technical analysis of what’s new in iPhone 12 cameras this year. All four of ’em. On Friday, we got our hands on the final iPhones in this year’s line-up, the iPhone 12 mini and the iPhone 12 Pro Max.

The Pro Max was the wildcard this year. Apple devoted a whole section of their keynote to it, and we got a ton of questions about it on our Twitter. Some reviewers on the internet panned it as an indistinguishable improvement from the iPhone 12 camera, while others called it quite good.

We’ll get into the iPhone 12 camera — we have a lot of thoughts on that. But the iPhone 12 Pro Max tests were quite surprising. So surprising we’ve decided to create this whole separate post about it.

What Makes the iPhone 12 Pro Max Special

We had an advanced look at the Pro Max specs, but to recap, it has:

• A 47% larger sensor
• A faster ƒ/1.6 lens
• A brand-new ‘sensor shift’ stabilization system for low-light
• ISO sensitivity is 87% higher
• A new telephoto lens, reaching a new length of 65mm (full-frame equivalent)

Specs are cool, but now that we have the hardware we can take a deeper look at how these changes affect real-world photography.

A bigger sensor

The biggest news was that the iPhone 12 Pro Max got a bigger sensor. 47% bigger. Sounds big. Is it? What does that even mean? Let’s visualize it:

That’s significant. When iFixit took this chonker apart, they also noted how large it was:

Ok, practically speaking, what does this mean?

Imagine a camera sensor as a collection of lots of smaller sensors. Each collect red, green, or blue light. These sensors are packed together to get an image that measures 3024 by 4032 pixels. (Technically each pixel on that sensor is called a ‘photosite,’ as they collect, yes, photons)

You’d think a bigger sensor means more pixels — and indeed, a bigger sensor could allow you to pack in more pixels. But we’re at a point of diminishing returns in megapixel wars.

Instead, Apple decided to make the the photo sites bigger, because one most important aspects of image quality images (and really, life in general) is signal to noise.

Giving these sensor sites more room and making them larger makes them more sensitive to light. More light means more signal, less noise, and sharper results. Let’s look at a comparison:

Can you tell the difference? For reference, the iPhone 12 Pro Max is at the bottom. I can’t.

Here’s why we’re seeing stories that the camera is a minor difference at best: Most people who aren’t seeing the dramatic difference are shooting in daylight, with a fast ƒ/1.6 lens. On top of that, Apple’s intelligent image processing combines multiple shots together, which makes it harder to look into the hardware.

We get a better look with RAWs (shot with Halide, of course). RAWs omits steps like multi-exposure combination and noise reduction, there will be some noise; but that lets us see how much noise the camera really has to deal with.

Since the iPhone 12 has a bit more noise, there’s less detail. Still, the detail isn’t easy to spot in bright daylight. Everything changes between the iPhone 12 and 12 Pro Max the moment sunset begins. As shadows get darker and light gets dimmer, you can start seeing shifts in detail.

The difference gets dramatic as the sunset progresses:

Again, these are RAW files shot in Halide. Meanwhile, photos in the first-party Camera app get a bit… smudgy:

We can zoom in on this a bit to see that better:

It seems Apple’s Camera app applies just as much noise reduction to the 12 Pro Max image as it does for regular iPhone 12. This unfortunately ‘blurs’ superior details out of the iPhone 12 Pro Max.

This is even more evident when you look at images closely and there’s even less light. Take this shot I took passing a restaurant in Chinatown (the owner has consented with me putting up this image):

Smart HDR rescues detail in the highlights and shadows here, but it is at the expense of a lot of sharpness.

Mind you, this is a really close-up zoom of a cellphone photo. Our expectation is that it is kind of mushy and lacks detail. Instead, the iPhone 12 Pro Max packs in incredible levels of detail down to the pixel level.

It might not pack more resolution in terms of the image size, but the resolution in that 12-megapixel image is far higher. There’s just much more detail in every shot, especially in RAW.

The iPhone 12 Pro shot of the same scene shows the kind of noise that it normally being smudged out:

Seeing how there’s just that much more detail in the raw image data from the iPhone 12 Pro Max, it could stand to have less processing to show off its higher signal to noise ratio. It’s possible Apple will tweak the image pipeline in the future to bring some of that detail out better.

In the meantime, we got RAW photography. Hooray!

Now, with the sun past the horizon, there’s no contest between the iPhone 12 and the Pro Max anymore. In these, close-to-nighttime to nighttime shots the Pro Max is a clear winner.

We’d say this sensor is a big leap ahead in image quality.

In our first look at the hardware changes, we said larger sensors offer a different ‘look’. That means more depth of field, but it may also render scenes differently. If you’re a photographer that has been in the field for long enough to have made the move from APS-C to full-frame, you might have hoped for a similar sudden, dramatic change of ‘look’. Sorry: that’s not happening here.

Sensor shift

We’d need to point out that all examples were shot handheld. While you could do a comparison of the iPhone 12 vs. the iPhone 12 Pro Max by putting it on a tripod, it’s kind of a pointless benchmark unless you mount your phone on a tripod whenever you shoot. We believe good comparisons should be done in real-world conditions, which means handheld shots.

That bring us to the next big change. Since the iPhone 7 Plus, there’s been magical stabilization technology that counteract shaky hand movements, so you can land sharper shots. However, the stabilization was previously applied to the optics system.

The iPhone 12 Pro Max applies stabilization to the sensor. Since this is a a light, flat plane, it’s technically feasible to move further and faster than an entire lens assembly.

At first you may think this will help video. In practice, video stabilization is absurdly good on as-is, thanks to intelligent software that smoothly crops and transforms your video frames, even as you run down a street. Stabilizing sensor can help, but only so much. This isn’t a Steadicam.

Low light photography is a different story. Software can’t un-shake your hand. Night Mode on iPhone gets around this by taking a ton of photos and just throwing out the ones that ended up too shaky.

Now we see the iPhone 12 Pro Max pull ahead — way ahead, really — of its smaller iPhone 12 mini, 12 and 12 Pro siblings:

That’s a RAW shot. Taking a single frame, there’s a clear winner. But even in that multi-image-taking, auto-stabilizing Night Mode, the difference in detail is obvious:

(images via Austin Mann’s fantastic iPhone 12 Pro Max initial review)

Even though Night Mode involves heavy computational photography, it ultimately benefits from having sharper images to work with before it merges them into the final shot.

Achieving the impossible

The iPhone 12 Pro Max’ new sensor-shift technology really shines when you take away that computational magic and focus on traditional, single-shot photography.

Notice that sharp RAW shot we just looked at? That was taken at a fraction of a second. The longer you open the shutter, the more light you can let in, and the less noise you get.

With a traditional DSLR, nobody would dare take a one-second photos handheld.

Yet the iPhone 12 Pro Max can take sharp images at ridiculous exposure times.

How about a third to half a second?

This looks like an image I get out of a modern mirrorless camera. Here’s a comparison with the Night Mode in the Camera app, and the (poor thing) iPhone 12 Pro:

In this last comparison, you can see that Night Mode can get this shot, but it sacrifices lots of fine detail and makes the scene look a bit unnatural. The iPhone 12 Pro, with its more basic stabilization, can’t get a steady frame.

It would be impossible to get a shot like this on the iPhone 12, 11 Pro, or heck, the high-end digital cameras I have lying around the studio. This is very cool stuff.

65mm lens

As a professed lover of the 52mm telephoto lens on the iPhone 11 Pro, I was particularly excited about having an even longer telephoto lens. Yes, it’s just a bit longer, now reaching 65mm — 2.5× instead of the previous 2× zoom — but reach is reach.

I quite enjoying getting the extra reach:

A longer lens in this case means a slower aperture, so a bit less light collected. That, too, isn’t that noticeable until sunset:

After all: less light collected means a bit more noise. That’s OK before it gets really dark: you can see what kind of noise the image pipeline has to deal with well in the comparison above.

The iPhone 11 Pro and 12 Pro both feature a regular 52mm (2×) lens with an ƒ/2.0 aperture, while the 12 Pro Max’s makes due with ƒ/2.2. In practice, that makes it worse at night. And on top of receiving less light, longer lenses amplify any camera shake.

With Night Mode coming to all lenses on the iPhone 12 except the telephoto lens, that makes it less than ideal when it’s dark out. I found that once the sun sets, you are sometimes better off using the new and incredibly sharp, stabilized wide angle sensor and cropping the result.

It seems the first party camera agrees with me. Once it gets dark out, open up the app, tap the “2.5x” button, and cover the telephoto camera (on the iPhone 12 Pro Max, the top lens) with your finger. You won’t see it covered.

In low light, the first-party camera quietly changes over to the wide angle lens, which receives more light. It then crops the result. It’s deception, sure, but it’s classic Apple: they know what’s best for you.*

*a note: We don’t do this in Halide, since knowing exactly what lens you’re shooting with is important for creative control. But we tackle a different audience than Apple. We really do believe that their way of auto-magically switching and swapping lenses is the right decision when you’re trying to serve the broadest possible audience.

An Aside about the future: ProRAW

A few days ago, Apple released a beta of iOS 14.3, which includes ProRAW. Initially teased during the announcement of the iPhone 12 Pro, ProRAW is a new format by Apple that ostensibly brings the best of the huge computational advancements in the iPhone camera to a format that offers complete creative control.

We’re writing a deep dive on what ProRAW means, but there’s one practical implication worth talking about today. One reason people choose RAW is for fine grained control over noise-reduction, and unfortunately ProRAW files have already had some noise reduction applied. Not as much the JPEGs of the past, but it’s there.

For this and a few other reasons, ProRAW is less of a ‘true’ RAW, and more of a very high quality format for your shots. Aside from the noise reduction, it does expose fine control over magical processes like Smart HDR.

So far ProRAW has shown to be fantastic quite often, but we’ve also observed tradeoffs. However, ProRAW is currently in beta. We’ve been submitting feedback to Apple. Once ProRAW ships, we’ll publish our analysis. That being said, we can say today — with great confidence — that ProRAW is coming to Halide.

Anyway, back to regular ol’ RAWs…

Halide and the iPhone 12 Pro Max

Halide, as a pro camera app, has never captured RAW files this great out of an iPhone by a wide margin. We’re incredibly excited to begin work on getting even better results out of this camera.

So let’s get started: we’ve just released Halide 2.0.4, with updates for the new iPhone 12 Pro Max.

We have improved framelines and lens controls for the new 65mm lens, tweaked exposure logic and more to take full advantage of this new phone and its tiny mini sibling. We’re not done there, though: there seems to be a lot left for us to squeeze out of this camera.

Conclusion

There are varied opinions about the iPhone 12 Pro Max. Is it a huge leap forward in iPhone camera quality? We say yes— the results speak for themselves.

But why did so many first impressions fail to pick up on the improved quality? It’s in the name of the device: iPhone 12 Pro Max.

As developers of a camera app, the results mind-blowing. It achieves images previously only seen in dedicated cameras, with sensors four times its size. It allows photographers to get steady and well exposed shots in conditions that weren’t imaginable a year ago. It captures low-light shots beyond anything we’ve seen on an iPhone. By a lot.

If you are an advanced user, the type that uses more than just the stock apps, this phone offers a lot.

Compare it to when Apple releases a new chip. They’re already ahead of the curve, and then they release an even newer chip that blows the old one away. Some will ask: “What can this even be used for?” Before you know it, you have console quality graphics and amazing AR experiences.

We’d compare this camera to a next-generation chip. In daily use, an average person won’t notice the signal to noise ratio or longer lens or superb stabilization and sensitivity. But as developers like us build apps for it, and this camera system works its way into future iPhones over the next few years, we can expect even the ‘traditional’ camera market to be forced to step up.

The iPhone 12 Pro Max is a Pro photographer’s iPhone. And we couldn’t be more excited to start building for it.

The iPhone 12 Pro Max: Real Pro Photography was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s that time again. Every year, we take a deep dive into what is new in the iPhone camera. This year is special: Apple is introducing several new cameras in the iPhone 12 and a whole new sensor in the iPhone 12 Pro Max.

We’ll be doing a thorough analysis of the iPhone 12 and 12 Pro Max cameras, comparing them to previous iPhones and tearing down the new processing in Smart HDR 3 and the benefits of their new lenses. First, however, we’ll take a quick look at what is new in the iPhone 12 Pro Max.

Why just the iPhone 12 Pro Max?

The iPhone 12 Pro Max is the first iPhone in a while to get a bigger sensor. Not just a bit bigger: the sensor is a whopping 47% larger, which should not just help it in low light but also allow for more depth of field, more detail in photos and generally a different ‘look’.

A note: These are just the hardware specs

Thanks to Halide’s Technical Readout feature, we can take a look at what an iPhone reports as its camera specifications.

iPhones tend to get better every year even when not changing their sensors and lenses; the real leaps in camera performance are made in computational photography, with merging many images and using AI and machine learning to augment the tiny sensors that fit into these magical devices we hold in our hands.

As usual, we’re following up with a look at the work that is being done to turn simple raw sensor data into such great shots.

What’s New?

47% Larger Sensor, 85% More Sensitive

A new sensor means new specs. Paired with the new, faster aperture the iPhone 12 gets across the line of ƒ/1.6, the iPhone Pro Max does not just gain low-light ability with its sheer sensor size or new in-body sensor-shift stabilization: the maximum ISO sensitivity of the sensor is up 85% to a maximum ISO of 7616.

That’s the highest ISO sensitivity we’ve seen in an iPhone, ever.

This should make for a pretty fantastic iPhone for night- or low-light time shooting.

While computational photography can help a lot with collecting more light over more shots, higher ISO can also help freeze motion better; this makes this iPhone uniquely more suited for capturing action in lower light. You can do a lot in software, but not everything.

A new telephoto lens — 25% zoomier

The technical readout confirms we’re going to about a 65mm (Full-frame equivalent) telephoto lens on the rear, with a slight sacrifice to light-collecting ability: we lose our nice ƒ/2.0 aperture and go back to the iPhone XS and X telephoto-standard ƒ/2.2. Not too terrible.

When composing, this is the change in your field of view:

That’s a nice 25% of extra zoom.

Here’s an actual comparison of how it would shoot and what kind of image you would get:

It’s not a huge change, but it might be more dramatic in real-life use.

What isn’t new

There’s changes, but there’s also things that stayed the same.

Ultra-wide ultra-same

Every lens seems to have gotten a revision with this new, larger phone, and very eagle-eyed iPhone users might notice that it is also… larger:

However, it seems by the technical readout and from reviews that the ultra-wide lens and sensor is the same one as in all the other iPhone 12s. It might get a larger bump on the back of your phone, but it’s not collecting more light as a result. Sorry, Pro Max-ers.

You do get a slightly improved ISO sensitivity and night mode on all iPhone 12 Ultra-Wide cameras. Yay!

Longer lens, same sensor

No changes in the telephoto sensor. This is unfortunate, as the telephoto lens is my favorite:

Catching the sunrise on Twin Peaks. 52mm lens, iPhone 11 Pro, @halidecamera Mark II. pic.twitter.com/HBrzSycdf7

Its sensor has been a bit… well, it is just not as great as that awesome regular wide-angle sensor. I don’t want to call it ‘underwhelming’, but it’s been the same size since the iPhone XS and it still struggles a bit in low light. I would certainly be excited by a revision next year.

You might notice this even more on the iPhone 12 Pro Max, as the iPhone’s default camera app will just crop the image from your (now even faster, better-in-the-dark) Wide lens when the light gets low or conditions are otherwise non-optimal for the tiny telephoto sensor. It’ll be interesting to see if the Pro Max more aggressively forces the telephoto to step back in low light.

Same old TrueDepth

Since the iPhone X, the infamous ‘notch’ of the iPhone has stayed the same size. That little lip at the top houses a camera and a few other interesting tidbits like an IR flood illuminator and sensor; they have not been miniaturized or changed in this revision. Last year, the sensor got a small bump to 12 megapixels and a wider lens; perhaps next year we’ll see a smaller notch and corresponding camera changes.

iPhone 12 Pro Max Technical Readout Comparison

That’s it for my take — here’s the raw specs you’ve been waiting for. Changes from last year’s iPhone 12 Pro Max are bold.

A note on these changes

Every year we are tempted to look at just the bare specs of iPhone cameras. As there’s now competing smartphones throwing around terms like 108 megapixels and 100-times zoom, you would be tempted to make a judgment based purely on numbers. We’d caution against that.

Every year we re-emphasize that quality in mobile photography is now largely coming from the integration of hardware and software and increasingly significant leaps in processing in the form of computational photography. This year is no exception, new sensor size or not. When looking at the actual images coming out of smartphones, the iPhone continues to come out on top — and we wouldn’t be surprised if the iPhone 12 Pro Max sets the new standard for mobile photography quality, regardless of its megapixel count.

We’re curious to take these new iPhones’ new software pipeline apart in our upcoming analysis, to see just how much they can squeeze out of a single RAW frame. With ProRAW coming this year, it’s an exciting year for iPhone photography as Apple increasingly begins to treat it as a real camera, with proper RAW support, various settings for managing the imaging pipeline, and technology like sensor-shift image stabilization.

One thing is for sure: the new iPhone 12 is a fantastic camera:

yup yup yup I like the iPhone 12 camera RAW files pic.twitter.com/5jLbgaC8xA

Check back in with us for our upcoming software- and RAW analysis of iPhone 12 and iPhone 12 Pro Max.

This technical readout was brought to you by Halide. You can shoot RAW with every iPhone with Halide, and get pro-level controls. We just introduced a new version of Halide called Mark II, which packs some of our best tools and a total redesign. For more information on the app, check out our website.

iPhone 12 Pro Max Preview: The Camera Hardware Changes was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s iPhone 12 day! A lot was announced today, from tiny HomePods to new iPhones and magnetic accessories, but as we make one of the more well known camera apps on iOS, we really cared about one thing:

What’s new in iPhone photography?

Here at Halide HQ (Halide HQ in COVID 2020 being, all three of us at home) we were reeling with the amount of information dropped on us. Here’s a very quick summary.

You can also read our initial impressions on Twitter, of course:

OK, there's a lot of stuff to be exited about with cameras and photography on iPhone 12 and 12 Pro. Let's unpack (thread). pic.twitter.com/ylNrmLxohE

Hardware

Even though most iPhone users will not likely be upgrading from an iPhone 11, we’ll use that phone from last year as our baseline to see what is new in iPhone camera hardware-land.

Note that most major improvements in photography on phones come in software nowadays; we usually expect to see minor changes in hardware. Evolution, not revolution.

Let’s start with what’s unchanged: On the iPhone 12, 12 mini and 12 Pro (non-Max), the sensor on the main (or “wide”) and ultra-wide cameras is unchanged. The 11 Pro and 12 Pro will both still come with the same telephoto lens and sensor as well.

That’s where similarities end. That main wide camera gets a new lens, which according to Apple lets in 27% more light thanks to its faster (that is, bigger) ƒ1.6 aperture (previously ƒ1.8) and sharper 7-element construction.

There is some mention of sharpness improvements on the ultra-wide lens, but we’re unsure if this is a software correction improvement or a new lens design — or both. The images certainly look sharper.

But if you like large phones, this is your year. The iPhone 12 Pro Max has the real goods.

In addition to a better lens, the 12 Pro Max has the room to pack a new, 47% larger sensor. That means bigger pixels, and bigger pixels that capture more light simply means better photos. More detail in the day, more light at night. That combines with the lens to result in almost twice as much light captured: Apple claims an 87% improvement in light capture from the 11 Pro. That’s huge.

But that’s not its only trick: the 12 Pro Max’s Wide system also gets a new sensor-shift OIS system. OIS, or Optical Image Stabilization, lets your iPhone move the camera around a bit to compensate for your decidedly unsteady human trembly hands. That results in smoother video captures and sharp shots at night, when the iPhone has to take in light over a longer amount of time.

Again, only the 12 Pro Max gets this; it will allow it to do thousands of corrections per second; it will be interesting to see how much of a difference this makes against its smaller siblings.

Photos out of this camera system will be pretty fantastic. Conspicuously absent is a big megapixel boost. Android flagship phones have been touting absurd megapixel counts (like Samsung’s 108 megapixel camera); Apple is simply not participating in this race.

The megapixels across the board are unchanged. This huge new sensor? 12 megapixels. The ultrawide? 12 megapixels. The telephoto? You guessed it, 12. The selfie camera? 12 megapixels.

On the Pro Max, the telephoto camera also gets an upgrade. It can look a bit further now, to 65mm vs. its previous 52mm. Why only the Pro Max gets this is a bit of a mystery, but it will enable a nice, high zoom range (again, nothing crazy here like Huawei’s 100x zoom). The new 65mm telephoto lens does lose a little bit of light gathering capability as its lens is a bit slower at ƒ2.2 vs. the iPhone 11 Pro / 12 Pro’s ƒ2.0 52mm lens.

Finally, the 12 Pro and 12 Pro Max both gain something we’ve seen in the new iPad Pro: a LiDAR sensor. What’s a LiDAR sensor? We did a very deep dive, complete with a proof of concept app for mapping 3D space in this blog post:

LIDAR: Peek Into The Future with iPad Pro

LiDAR on the iPad Pro did more for AR and very little for photography. On the iPhone 12 Pro, it promises to vastly speed up autofocus and help with Portrait mode.

As it stands, Portait mode combines data from multiple cameras as well as using Machine Learning to add depth to a scene. More data is always better, so we’re excited to dive into it with Halide’s Depth Mode to see how high resolution the new combined depth sensing is.

Hardware Summary:

iPhone 11 to iPhone 12 : A better wide camera lens that collects more light, an unspecified but better ultrawide camera.

iPhone 11 Pro to iPhone 12 Pro: The same improvements as above, and the same telephoto camera we get in the iPhone 11 Pro. But: LIDAR!

iPhone 11 Pro to iPhone 12 Pro Max: Okay, now there’s no comparison. This is a much better Wide camera, with almost double the light collection, a new sensor, improved OIS and more. It has a new and greater zoom range on its telephoto lens. This thing is a beast. And it also gets LiDAR!

Software

As we said before, we really expect to see the greatest leaps in photographic improvement on phone cameras to come in software. Google has demonstrated this, with the first Night Mode that does seemingly impossible night shots on tiny phone cameras to various other computational tricks.

The key here is combining the very powerful processors in our phones with the ability to take many photos at once; something that small sensors can do better than the large and high-megapixel sensors of large professional mirrorless and SLR digital cameras. This is how our app Spectre (which Apple chose as 2019’s App of the Year) can do handheld 9-second long exposures. It’s all in the smarts.

We noticed that with the iPhone XS, Apple started to publicly shed light on, and market, its smart computational photography systems. Smart HDR was introduced then, which promised to combine many exposures at once in an instant when you press the shutter to produce images with better detail in shadows and highlights. It wasn’t perfect; some felt like it was aggressively smoothing over faces, for instance.

With Smart HDR 3 on iPhone 12 and 12 Pro, Apple segments images to prevent that kind of thing happening. This was already happening on iPhone 11, but this is a software challenge that only gets better with more processing power. Its other computational photography technique, Deep Fusion — a technology that kicks in in low and medium light — is promised to be improved as well.

Once you get more processing power, you can also start combining all these smart photography processes. We previously got Night Mode on iPhone 11 and 11 Pro.

Now it’s everywhere: Night mode is coming to the ultra-wide camera and selfie (TrueDepth) camera. What’s more, iPhone 12 and 12 Pro do just that smart combining of technologies: you can now get Night Mode to kick in on your Portrait photos.

This is big: a lot of crazy computational photography happening real-time. For this to work, the phone has to compare data from LIDAR and multiple cameras to infer depth, render real-time blur while also doing some crazy Night Mode enhancements. This is where that incredibly fast, Neural Engine and bigger ISP (image signal processor) toting A14 chip comes in handy.

Apple benefits from this tight integration of hardware and software: the new chip and better software enable better noise reduction, which means better video in low light (and photos, probably). Apple says the new ISP (Image Signal Processor) on the chip is to thank, but I bet some machine learning is involved. Apple claims 87% better low-light video on the iPhone 12 Pro Max, and states that on all phones, the new ISP allows better noise reduction for more detail. I bet some Machine Learning is used in this noise reduction, too.

RAW comes to iPhone

This was scary to us for a moment: a lot of people buy Halide to shoot RAW on iPhone. And why not? The cameras on iPhones are incredibly impressive and can shoot shots like this in RAW:

NYC moods. 52mm lens, iPhone 11 Pro, @halidecamera RAW. 🏙 pic.twitter.com/AbtZowNstg

Now, RAW gets stage time. Apple announced ProRAW. We had previously written about how RAW is both handicapped and benefits from not getting the ‘smart’ photography benefits of Smart HDR and Deep Fusion, which prompted us to release a Halide update with a feature called Smart RAW:

Halide 1.10: The Smartest RAW

ProRAW, according to Apple, gives you the standard RAW along with this pipeline information, which should offer some fantastic flexibility when editing. Note that this might be a custom format; little is known, and it seems it was only announced and might be limited to the iPhone 12 Pro.

It’s also listed as ‘coming soon’:

More on that as we find out about it. Apple announced an API, so we’re eager to get our hands dirty on it.

Other improvements

There’s a small slew of other improvements: Machine learning improvements are said to have further improved Portrait mode. Night mode now comes to Timelapses, and video can now be captured in 10-bit HDR with Dolby Vision, a remarkable first. As we learn more about the soft- and hardware of these devices, including its Technical Readouts, we’ll post them here.

Feel free to send us any questions on Twitter!

iPhone 12 Event: All The Camera News was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

Our team is excited and inspired by the technologies and announcements of this year. However, we can’t let products overshadow the time we live in. Normally we use this space for product announcements, but today we’d talk to talk about a few things we believe in deeply.

First off, Black Lives Matter. As a team and as people, we are dedicated to doing the necessary listening and work needed to fight systemic racism. We stand in solidarity with the Black Lives Matter movement, and we pledge to stay committed to fight racial injustice. It should not have taken us until 2020 to get here. We ask that you join us in supporting this cause.

We are actively working on long-term initiatives to promote Black photographers and look forward to using our resources and channels to create a lasting impact beyond this news cycle. We look forward to sharing some of our efforts with our new product releases this year. We also urge you to reach out to your local government representatives and demand they rethink their law enforcement and defund the police.

What can we do as a company? Since starting Lux Optics, we’ve donated over $25,000 to charities. It’s a good start, but we want to do more.

2% for Causes

In the last month we’ve thought about how to make charity an ongoing effort in our company. We did some research, and 1% for the Planet caught our attention. It is a formal organization where the member companies pledge to donate 1% of profits toward environmental charities.

To start, today we are pledging 1% for Society. 1% of all profits will go toward helping create a society that is equitable and good for everyone. To that effect, so far we’ve selected the following charities:

• Campaign Zero
• ACLU
• Equal Justice Initiative
• Black Girls Code

However, society and the planet are not two separate issues. Often times, societal oppression directly overlaps with environmental injustice. That’s why we’re also applying to join 1% for the Planet. With these commitments, we will have an ongoing effort to support the causes we consider the most important issues facing us right now.

For transparency, we will have a post every year to detail the organizations that we donate to and solicit input on other great charities we can support.

We Are Not Done

We consider 2% the minimum of what we will donate every year, and we will continue to donate in other ways.

For example, we’ve decided that instead of giving new hires a signing-bonus, we will have a “charity bonus,” where they will have $5,000 donated to a charity of their choosing. Earlier this year saw our first hire, Rebecca, and she selected The Vanessa T Marcotte Foundation, which strives to stop violence against women in society.

In Closing

Our little 3 person company wouldn’t exist without your support — we want to thank you for helping us support these causes when it matters: now, and tomorrow. Feel free to let us know what you think of our efforts, and how we can do better.

Doing The Right Thing was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

Two weeks ago, Apple dropped a new iPad Pro into our lives via their website. One of its headline features? A brand new camera array. In this quick article, we look at what’s changed, including a whole new “3d” sensor — and what this new technology enables.

If you’re just joining us: we’re the team that makes Halide, an iPhone camera app.

A New Camera Array

Does the new iPad Pro pack a great camera? We have to look at the whole package. Today’s high-end smartphones pack a cluster of cameras, with multiple lenses (and sensors!). Google entered the fray last year with the Pixel 3 double-shooter. Now the iPad Pro joins this camera superstructure elite.

Let’s start at the ‘primary camera’:

The Wide Angle

Not to be confused with the ultra-wide angle lens we’ll get to shortly, this 28mm wide-angle camera is similar to all cameras ever put on the iPad, going back to the iPad 2. It’s table stakes. How does it compare?

iPad Pro 2020 vs iPad Pro 2018

Shot head-to-head with the iPad Pro that preceded it, the changes are… minimal. It appears the sensor and lens is either identical or almost identical to the last generation iPad Pro. They certainly match physically, with new accessories like the upcoming magic keyboard fitting both generation iPads.

The camera module hasn’t moved much; it just grew a little bit, adding two new sensors to its mesa-like structure.

Processing appears a little different. The sensors show wider sensitivity to light with their ISO ranges, with low-light image processing slightly improved.

These minor changes are likely only software. Big changes like ‘Deep Fusion’ typically require new chips, but the iPad Pro’s chipset is basically identical to the 2018 model, with an extra GPU core enabled.

Also unchanged? The lens focal length, which determines the field of view. iPhone XS and XR moved to an ever-so-slightly wider 26mm lens; this carried over to the iPhone 11 and 11 Pro, with their ultra-wide lens being exactly half that at 13mm.

Not the iPad Pro. iPhone 8 and iPhone X had 28mm lenses — and so does this iPad Pro, like the iPad before it.

Here’s our full Technical Readout and comparison:

If you need something to compare it to, it’s the iPhone 8 camera. Don’t expect parity with Apple’s latest iPhone 11 Pro shooters, but it’s still a great set of cameras.

(Yes, our test setting definitely looks like a picturesque Apple marketing mockup.)

The ultra wide angle camera

Here’s a big and obvious change: this is the first iPad to have a second camera. The ultra-wide, famous from last year’s iPhone 11 and iPhone 11 Pro arrives on the iPad. There’s some changes compared to the iPhone, however:

Remember when I said the iPad Pro’s wide angle lens was a bit less wide than the iPhone XS/XR/11 series of phones? That applies here, too: the ultra-wide has a grand difference of 1mm with the iPhone 11 and 11 Pro’s ultra-wide lens, bringing it to a tidy 14mm. In daily usage, it’s not noticeable.

What’s most interesting is the mere 10 megapixel resolution. It’s up from previous iPads, which packed an 8 megapixel rear camera, but this is the lowest resolution on a “new” rear camera since the iPhone 6.

Of course course megapixels are only one factor in an image, and don’t always tell the full story. So let’s compare:

In short, the iPhone 11 and 11 Pro pack a significantly larger (and better) sensor with its wide-angle camera, compared to iPad. The ultra-wide sensor on iPhone is comparable to the ultra-wide on iPad in quality, but the iPad is lower resolution.

It appears the hardware just isn’t there to support night mode, Deep Fusion, and even portrait mode. The iPhone XR pulled off single-camera Portrait Mode thanks to a larger sensor, and tapping into focus pixels — we even got single camera Portrait mode working fornon-human subjects in Halide!).

Some have asked if it’s possible to add portrait mode with the third little disc on the back of this new iPad, which is the…

The Depth Sensor

Here’s the biggest and totally new thing, then: The LIDAR sensor.

This is the first time a new imaging capture technology has appeared on iPad before iPhone. So we were a bit surprised by a lack of marquee applications to show off its potential. The AR Measure app that Apple bundles makes handy use of it:

AR experiences are way more seemless and accurate. You no longer having to ‘calibrate’ the device by waving it around in space. It can detect and classify features in your room like windows. It can even measure how tall people are. So what is this LIDAR wizardry?

How LIDAR Works

The LIDAR sensor, also known as a 3D ‘Time of Flight’ sensor (ToF for short) is a sensor that is exceptionally good at detecting range.

Regular camera sensors are good at focused images, in color. The LIDAR sensor doesn’t do anything like this. It emits small points of light, and as they bounce off your surroundings, it times how long it took the light to come back.

This sounds crazy, but it’s timing something moving at the speed of light. This window of time that amount to hundreds of picoseconds. Pico? Yes, pico — that’s an order of magnitude smaller than nanoseconds! A picosecond is 0.000000000001 seconds. Count the zeros.

If this sounds familiar to all you rocket scientists out there, the name LIDAR is a play on RADAR, but instead of using RAdio waves, it uses infrared light.

Could this Power Portrait Mode?

Today, the iPad doesn’t support portrait mode on the rear cameras. Could they fix this in a software update in the future, by using this depth data? While I wouldn’t say it’s impossible, it’s very unlikely.

The problem is that the depth data just isn’t high resolution. This fantastic experiment from iFixit shows how big the dot pattern is that this sensor projects:

Compare that to the infrared dot pattern that your iPhone (or iPad Pro!) projects for Face ID:

The Face ID sensor is trying to only resolve an area the size of a human face, with enough accuracy you can use it as a security device. The LIDAR sensor is made for room-scale sensing. It’s basically optimized for scanning rooms in your house.

The only reason we won’t say it could never support portrait mode is that machine learning is amazing. The depth data on the iPhone XR is very rough, but combined with a neural network, it’s good enough to power portrait mode. But if portrait mode were a priority, we’d put our money on Apple using the dual-cameras.

How We Can Use LIDAR

With Halide, we’d love to use the depth data in interesting ways, even if it’s low resolution. There was only one problem: there are no APIs for us as developers to use to get access to the underlying depth data. They only expose the processed 3d surface.

What if we re-thought photographic capture, though? We built a proof-of-concept we’re calling Esper.

Esper experiments with realtime 3D capture using the cameras and LIDAR sensor at room scale. It’s a fun and useful way to capture a space.

In the GIF above, we scan a kitchen. Panning around helps the resolution of the LIDAR sensor immensely, as you can see it forming — and refining a 3D mesh of its surroundings. When you tap the capture button, it textures the 3D model, using the camera data.

Once captured, you can view the space in 3D or in AR. You can change the way it looks:

And thanks to it being a three dimensional capture, you can even walk around in it!

The smallest ‘resolution’ we found this to be feasible in is about furniture-size. This chair captures pretty well:

If you’re interested in seeing how this works, download one of Ben’s chairs that he ‘captured’ in Esper. (insert ‘you wouldn’t download a car’ joke here)

You can view it on any device and even place it into the ‘real world’ using AR. Thanks to the LIDAR sensor, its scale is accurate.

That’s Esper! It’s our quick proof of concept to show that while this new LIDAR sensor isn’t something that is (currently) able to augment our ‘traditional’ photography, it opens the door to new applications that are powerful and creativity-enabling in their own right.

Let us know what you think of Esper on Twitter!

We think this is a super exciting thing about the way camera hardware is evolving. It’s not simply new sensors used to augment photography or video capture as we know it: it will enable entirely new ways to capture reality around us.

It’s up to us as developers to re-imagine the art of capturing the reality around us and give users the tools to explore what’s possible.

Your questions

We asked for some questions, and we got them. Let’s answer some:

Tinko Vialard on Twitter

@sdw @halidecamera How much better is lidar compared to a dual camera setup. And is it worth adding it to a dual camera setup?

It’s not better or worse, because it’s going for something different — room scale scanning. Just like having a gyroscope or GPS opens up new possibilities, it would be great for this sensor to find its way to an iPhone.

Call of Facchin - Modern Warfare on Twitter

@sdw @halidecamera Does it offer anything better than the iPhone 11? :)

With LIDAR, this iPad is easily Apple’s best device for sensing three-dimensional space. Whether it comes to measurements or AR, there’s no contest. Outside of that, your several-year old iPhone will still take better photos.

Johannes Freund on Twitter

@sdw @halidecamera Could it be used to scan objects for 3D printing?

Ah, objects. Unfortunately, the mesh output by the system right now isn’t accurate enough to send to a 3d printer. If you look at Ben’s chair scan, you’ll see that the surface is rough, and it has trouble with details like chair legs. But it’s a great starting point for a 3d model, since all the proportions will be very accurate.

David Klein ☕️ on Twitter

@sdw @halidecamera Are you excited as a photographer? Are you excited as the developer of Spectre/Halide? Where do you see the industry in 2 years, 5 years?

We’re super excited as photographers and developers to see what the next few years bring . Photography isn’t traditionally taking photos anymore; it’s combining all the data and smarts in our devices into allowing totally new interpretations of what ‘photography’ can mean. And if you’re not excited about that, we’re at a loss!

Thread on Twitter

@sdw @halidecamera The 2018 iPad Pro models lost optical image stabilization. Does these new models have it and if not isn't it a big deal?

No optical image stabilization (OIS) was added, and while it’s not exactly missed at the ultra-wide end (the wider the lens, the less OIS is necessary), it’s a bummer that the camera did not improve at all on the wide end. Next year?

Ryan McLeod on Twitter

@sdw @halidecamera If I put my mouth around it could it scan the inside of my entire mouth?

Ah, the hard hitting questions.

Unless the inside of your mouth is about the scale of a room (like a blue whale), you could conceivably do that a lot better with the infrared, front-facing TrueDepth camera on your iPhone X (or newer). Halide will let you capture the raw depth data. Let us know how it goes!

Wrapping up

Thanks for your questions, we hope you enjoyed our regular virtual ‘tear down’ of this camera module! We’re excited to see cameras get ever more interesting with additional sensors and modules.

Happy snapping!

LIDAR: Peek Into The Future with iPad Pro was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

2020 is shaping up to be an eventful year, and we’ve been hard at work in 2020 on new features for Halide, Spectre, and beyond .

Today we’re breaking our silence with our latest update: here’s Halide 1.16, a ‘groundwork’ update with several features that improve everything about RAW — it improves how Halide displays your RAW captures, how it saves RAW (and regular compressed) files, improves Halide’s file output, and makes your life a bit better. On top of that, we’ve improved Halide’s performance even more.

Let’s start with a feature that merits its own description: Rescue.

Rescue

There’s no worse of feeling than going into your camera roll after a day of shooting, only to discover that perfect shot is disappeared. This is extremely rare, but not impossible:

• You taking a photo at 5% battery, and suddenly the phone shuts down.
• You get a phone call in the middle of a shoot
• Or, a bug in iOS or (believe it or not) Halide causes an issue

Even it’s a one in a million chance, we cannot accept even the slightest chance of losing a photo. We treat data loss as a critical problem, and that’s why we built Rescue.

In the past, when you took a photo with Halide, we would simplify hand it off to iOS to upload to iCloud. We assumed everything worked as expected, so long as iOS doesn’t hand us back an error. In Halide 1.16 we trust but verify.

Now when you capture a photo, we keep a backup copy around, similar to how your email client might have an “outbox.” We don’t remove that backup copy until we’ve verified for ourselves that the photo exists in your library.

If you ever feel like, “I swore I took more photos,” and you can’t find it in your iCloud library, take a peak inside Image Rescue.

1. Open the photo reviewer.
2. Tap the “Grid View” icon.
3. Tap the “Lifesaver” icon in the upper right.

Rescue is a feature that we’ve put together thanks to feedback from our community, which includes photographers that have pushed iPhones to their limits in extremely remote locations or extreme conditions. We’re excited to release this update to show that Halide is the ultimate photography tool that everyone can truly rely on, even when the going gets tough.

Smart RAW is Smarter

We first introduced our Machine Learning based Smart RAW adjustment last year. We gather data continuously to improve camera settings for better RAW output with more editing headroom and lower noise.

Halide 1.16 packs a nice update to Smart RAW, based on iOS 13 behavior and our own research. You’ll notice better highlight recovery during day scenes.

Here’s a few sample photos which were captured with the parameters that Smart RAW now uses:

Flat RAWs Previews

This is a big one.

RAW files deserve several posts of their own to explain just how they work, what the power of RAW is, and why you should — or shouldn’t — shoot in RAW. We did just that (so click here if you’d like to read up!) but it’s still not exactly intuitive.

The Power of RAW on iPhone, Part 1: Shooting RAW

Many people just getting started with RAW complain that at a glance, the image looks much worse than a picture taken with the default camera app.

Yikes, what’s happening with the sky? It looks like the highlights are blown out:

So what’s going on? The iOS RAW engine is trying to develop your RAW file for you. You see, a completely unadjusted RAW is pretty “flat” looking. When studying this RAW file, you’ll find that it actually has more detail in the sky and shadows than the picture straight out of the default camera app.

So why do RAWs look like that in most apps?

The standard iOS RAW engine adds a contrast curve and other adjustments to punch it up. Unfortunately, it doesn’t employ any sort of sophisticated computational photography algorithms— it twiddles some knobs and calls it a day. Very often this makes it appear as though your RAW has clipped data, making you think it’s worse than a JPEG.

Halide 1.16 renders flat RAWs in the Photo Reviewer. While the image appears less vibrant, it makes it much more obvious what areas are actually clipped, without having to jump into an editor. Images do appear a bit ‘bland’ and flat, but the beauty of RAW is what you can do with them in an editor. You can now even export the ‘Flat JPG’ straight to apps that support it.

The Power of RAW on iPhone, Part 2: Editing RAW

From looking at the underlying RAW, it’s obvious we only need to stop down the image a little bit to recover more sky.

Faster Saves

While updating our plumbing to support Rescue, we improved the way we saved photos so narrow the window where bad things can happen. We make saving photos the highest priority task in the app, and throttle other behavior (e.g. fetching photos from the library) while there’s a queue of photos to be saved.

Reduced Memory

When we say this is a groundwork update, we mean it. One of the most essential things of iOS apps is performance and memory usage. We’ve reduced the chance of running out of memory in depth mode, causing a crash. We say “chance,” because crash-worthy memory usage is less predictable than you think. Want details? We’ve got them:

Warning: Technical Details Ahead.

Apple doesn’t share hard numbers on memory limits. You can force a crash on an iPhone 7 by deliberately eating up to 1.4 gigs of memory, but iOS can consume memory on its own in surprising way. For example, if your email client wakes up in the background to fetch new emails, it’s suddenly going to need more memory than when it’s chilling in the background.

Last year we measured that Halide typically used between 100mb and 150mb of memory. That seems reasonable when you consider a single full-resolution photo consumes nearly 50MB of memory.

However, in depth mode, Halide would peak a 576MB when generating our portrait effect — it has a lot of moving parts. We planned to get that number down, but traced most memory usage to internals of Apple’s Core Image framework. Apple didn’t expose knobs for us to adjust to fix it, so this could take a while to find a solution. “Well, 576MB is still way below that 1.4 gig limit, so things are safe for now, right?” Well…

When any app uses the iPhone camera, it fires up an iOS daemon called mediaserverd. Normally this daemon consumes 230MB of memory. We found out that in depth mode, it would hover around 850MB, which sends iOS into a “Danger! Free up memory!” zone. Sometimes you’d be unlucky, and iOS would decide to kill Halide. We solved this experimenting with different camera configurations, and ultimately got Apple’s daemon to only use 515MB. We were back in a safe zone.

In this update, we’ve further optimized our Core Image usage, and reduced Halide’s peak memory use in Depth Mode to 300MB. As before, when shooting RAW, it peaks at 150MB. In addition to getting us into an even safer zone, it makes us a better iOS citizen.

Spectre

Today we’re also issuing a small Spectre update — it has been in solid shape since being awarded with Apple’s pick as iPhone App of the Year. This update brings it up to speed with Siri Shortcuts and fixes a few bugs.

The Future

Spectre will receive Rescue (and our optimizations) in an upcoming release. In fact, we’re working toward launching these sort of common features simultaneously on Halide and Spectre by sharing code.

This has slowed down some Halide feature development, but once we’re through those woods, it will be worth it.

In addition to this update, we’ve had our heads down on something really big. Stay tuned: this year will be a great one for Halide and Spectre users alike.

Stay healthy, stay safe, and happy snapping from the Halide team!

Halide 1.16: Safer, Better, Faster was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

Well, this was a surprise.

We were thrilled and honored to be contacted by Apple a few weeks ago, asking us to attend a special event in New York City with several other great developers to celebrate their picks for the Apps of the Year. The big bombshell was dropped when we were told that Spectre was chosen as the iPhone App of the Year.

At the event, we were honored to shake the hands of some of the people at Apple that helped us achieve a personal dream: to make photography apps with an incredible level of polish and cutting edge technology.

Thanks to the App Store, me and Ben have become a full-time two man company creating Halide and Spectre, something we could only dream about when we first launched Halide in 2017.

Spectre is young — we launched the app in February, and were overwhelmed with the positive response. It was the top paid app in the United States for almost a full week. We’re honored and humbled with this incredible award from Apple.

We’re not done with Spectre. We think there’s lots of exciting opportunities for computational photography left for us to explore. We have huge plans in 2020, and we can’t wait to show you what we have been working on.

A huge thanks go out to everyone at Apple for making this possible and for choosing us as the App of the Year. And a huge thanks to all of you, our users, for sharing so many incredible shots tagged with #SpectreShot — We’ll be featuring our favorites of the year over on our Instagram and Twitter — and for spreading the word about our little magical photography tool.

We truly couldn’t have done it without you.

Spectre is Apple’s iPhone App of the Year was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last month, we took a look at what is new in the iPhone 11 and 11 Pro’s camera hardware. You might’ve noticed two things from Apple’s iPhone announcement event and our blog post: the hardware changes seem fairly modest, with more attention directed at this generation’s software based processing.

It’s true: The great advances in camera quality for these new iPhones are mostly to blame on advanced (and improved) software processing.

I’ve taken some time to analyze the iPhone 11’s new image capture pipeline, and it looks like one of the greatest changes in iPhone cameras yet.

What is a photo?

That sounds like we’re off to a rather philosophical start, or delivering the punchline of an iPad photography commercial, but to highlight what makes the iPhone 11 camera unique we have to understand our expectations of photography.

For a while now, you haven’t been the one taking your photos. That’s not a slight at you, dear reader: When your finger touches the shutter button, to reduce perceived latency, the iPhone grabs a photo it has already taken before you even touched the screen.

This is done by starting a sort of rolling buffer of shots as soon as you open the Camera app. Once you tap the shutter, the iPhone picks the sharpest shot from that buffer. It saves a shot you, the user unaware of this skullduggery, assumes you have taken. Nope. You merely provided a hint, to help the camera pick from the many shots it had taken on its own.

We can argue this is still deliberate photography. Without your action, there would be no photo.

The resulting image is not a single exposure, either. The iPhone takes a multitude of exposures — going by Apple’s presentation, a whole lot— and intelligently merges them.

Last year’s iPhone XS introduced Smart HDR, which combined over- and underexposed shots to ensure your image has more detail in the shadows and highlights. HDR is shorthand for ‘High Dynamic Range’, and for decades it’s been something of a white whale in photographic technology.

Smart HDR Gets Smarter

Ever since humans put silver halide (hey, that’s our app’s name) on a plate to capture a still image, we have been frustrated by a photograph’s limited ability to capture details in both light and dark areas.

Our eyes are actually really good at this sort of thing. It’s estimated they have 20 ‘stops’ of dynamic range. As a comparison, one of the latest and greatest cameras, the Sony A7R4, can capture 15 stops. iPhone 11 is around 10 stops.

Long before digital, film photographers ran into the same problems when they went from film negative (13 stops) to paper (8 stops). They would dodge and burn the image when developing to get everything within a printable range.

Today, a skilled photographer can recover dynamic range from a single RAW photo through editing. It’s just a manual process, and in extreme situations you’ll run up against limits of the sensor.

Smart HDR takes a very different approach by ‘cheating’ its way to better dynamic range. You don’t need a 15-stop sensor that can resolve more detail in light and dark areas at the same time — you just need a sensor that can take images fast enough, and smart software to merge them. The ‘editing’ happens in real time, without any action from the photographer.

As we wrote last year, it comes with trade-offs. Keeping the iPhone sensor biased towards rapid, short exposures causes much noisier images, something RAW files from iPhone XS and XR cameras confirmed. This requires noise reduction, which has a side effect of losing details.

Interestingly, this noise reduction led to the accusation Apple was applying a skin-smoothing ‘beauty filter’ to every shot. In fact every texture was getting smoothed. It made for better shots much of the time, but the loss of detail was noticeable. Images looked soft:

That made shooting in RAW a compelling option for advanced iPhone photographers. The detail was retained, and with something like Halide’s Smart RAW or manual exposure settings noise was kept (mostly) manageable. With a RAW editor, very often you could recover dynamic range just as well as Smart HDR. I preferred it to the iPhone XS/XR’s ‘processed’ image look.

But when the iPhone 11 arrived, I immediately noticed a staggering improvement from its predecessor. Smart HDR got… well, smarter.

On iPhone XS and XR, Smart HDR was heavy-handed at times, smoothing out texture in places or flattening highlights when unnecessary. Sometimes it failed entirely, and refusing to let an overexposed shot just be overexposed.

This is a classic problem in HDR. Sometimes you’ll recover dynamic range at the cost of a flat image without contrast; a sunset shining brightly on someone’s face could look a bit dull. At worst, it totally ruined photos. Smart HDR was so biased against highlights that it became very easy to spot images shot with the iPhone XS and XR by their ‘look’ alone.

On the iPhone 11, the camera knows that sometimes you want highlights blown out. That’s smart — smarter Smart HDR, if you will.

The Devils of Detail

iPhone 11 photos showcase a remarkable amount of detail.*

*For the purposes of this post of the series, I will judge the typical iPhone camera output. In iOS 13.2, Apple sometimes uses ‘Deep Fusion’ to further improve detail and sharpness. We will delve into that — how it works, what triggers it, and how it stacks up vs. RAW photography — in future post. Stay tuned.

Detail is something that we always want more of in photography. There’s a nearly irrational chase of ‘sharpness’ in lenses, a seemingly never-ending push for more megapixels in cameras and a constant discussion about how big of a sensor is required for a halfway decent photo.

We can see on iFixit (and in our hardware breakdown) that as far as the sensor goes, the size and megapixel count are the same as the XS. What is Apple doing to achieve so much more detail? There are two changes:

Hardware

While the changes in hardware are minor, there are clearly some improvements under the hood (under the lens?). Notable is the improved ISO range, which has gains of 33% and 42% on the main (wide) and telephoto camera, respectively.

What’s ISO? It’s a measurement of sensitivity to light. In the previous century, it specified the sensitivity of photographic film. For example, an ISO number above 800 is very sensitive, and even vulnerable to Airport X-Rays.

Today, ISO is a setting of your sensor. When you have less light, you can increase the ISO setting. This lets you can capture with a faster shutter speed, which creates a sharper shot. The trade-off is that a higher ISO produces more noise. That requires more noise-reduction, which results in less detail, like we showed earlier.

Many thought this higher maximum ISO on iPhone 11 sensors was used solely for night mode, but an improved ISO range also makes a noticeable improvement to everyday noise. You don’t have to use the new maximum ISO to see a difference. You’ll see less noise in most — if not all — of your shots.

There’s one notable exception: the minimum ISO of the iPhone 11 is a bit higher, which means at the baseline it is ever so slightly noisier than the iPhone XS and XR. However:

It appears the sensor resolves more detail despite the slightly added noise. It’s an improvement at every level.

At the same time, there’s only so much you can do with hardware. Folks have compared the iPhone 11’s results to a DSLR or other large cameras, yet this is how tiny the sensor is compared to the full-frame Sony A7R4 camera:

Yep, we’ll need magic for it to get good images. This year the magic has gotten even more sophisticated. Or rather, discerning.

The Big One: Selective Post-Processing

When editing your photo, you might opt to make changes to the entire image, like brightening the scene or enhancing sharpness across the shot. But you might also apply selective editing to some parts of the image, similar to how magazines ‘airbrush’ a model’s skin to look smoother.

Apple added similar retouching to its capture process. The iPhone 11 now breaks down your image in various areas automatically and then applies different amounts of noise reduction to those areas. You probably can’t spot the noise reduction in this shot:

If we distort the contrast in an image, we can bring out the ‘watercolor artifacts’ in the clouds:

As you can see, the shot receives aggressive noise reduction in the sky. Since skies tend to be uniform surfaces of color, iOS figures it can apply its harshest, smoothest noise reduction to make the skies silky smooth, with nary a nodule of grain to be seen.

Other areas get different levels of noise reduction. Cacti and other features that the camera deemed worthy of detail get very little.

However, this technique isn’t perfect. Notice that in the original image, the shrubs that weren’t well defined lose a little detail. iOS doesn’t know,“This is a cactus, this is sky.” It seems to apply noise reduction based on the amount of initial detail.

When iOS detects a face, it’s a very different story. It can specifically target detail in skin texture, hair, and clothing.

Apple refers to it as “Photo Segmentation.” They use machine learning to split the photo into tiny pieces and processes each individually. What’s so impressive is that all of that is done in an instant.

This targeted post-processing is what makes the iPhone 11 an entirely new camera. Techniques like (last year’s) Smart HDR much less discriminate.

This year’s change to the imaging pipeline selectively edits every shot you ever take to help retain particular areas or smooth over texture.

In the iPhone 11 Night Mode, you can also see detail vanish in some areas. Except that it really seems to only affect parts of the image that you don’t really care that much about. Night Mode has a remarkable if not uncanny ability to extract an image that is sometimes even sharper than the regular mode, with strong sharpening and detail retention occurring in areas that are selected by the camera during processing.

It’s ironic that there wasn’t much truth in last year’s ‘Beautygate’ controversy, because this year’s iPhones actually do airbrush particular parts of your image more aggressively than others. But since the camera is now smart enough to apply this selectively, in areas where you don’t even notice it, there are no controversies roiled the tech world this year. Images just looks sharper and smoother

Better Noise Reduction

While it is somewhat difficult to make an objective comparison of the quality of noise reduction used on the iPhone 11 vs. the iPhone XS (it’s impossible to turn off semantic mapping to get ‘vanilla’ noise reduction), it does appear noise reduction on iPhone 11 is better at retaining details.

We could speculate the new noise reduction might uses machine learning to ‘fill in’ detail that could be lost to noise— many modern noise reduction technologies do. Regardless, this is a welcome improvement.

Post-Processing and the Ultra Wide Lens

We said it already: we’re very sad we cannot get RAW output from the ultra wide angle lens. Perhaps it will come in a software update. In this photo, you can see that detail is largely smoothed over:

The ultra wide lens is the slowest of the three rear lenses, forcing it to use a higher ISO. If you want to see this in real time, just use the nifty new zoom wheel on your iPhone camera app as soon as the sun starts going down. As you tick down to 0.9x and below, you’ll see texture in say, your carpet, just kind of… disappear.

The ultra wide also faces a unique challenge: ‘chromatic aberration,’ or in plain English, purple fringes. The telephoto faces this to a lesser degree, and we can see it in its RAWs. Notice purple/pink-ish fringing around the leaves in this photo taken in the Hong Kong botanical gardens against direct sunlight:

This is something photographers that shoot with ‘fast lenses’ (large aperture; the range between f/2.0 and f/0.95 — a smaller number being a larger aperture) are used to: the more light a lens can let in, the more it is likely to suffer from the dreaded fringing. Fortunately apps like Lightroom can remove fringing just fine; now Apple’s automatic-post processing saves shots from this somewhat unflattering chromatic aberration.

If we do ever get RAW output from the ultra-wide angle lens, I’d love to see it without any correction: from fish-eye distortion to fringing, I want to see the (ugly) truth.

A Compelling DSLR Competitor?

This year’s changes are huge. Thanks to hardware improvements, semantic adjustments on every frame, Smart HDR improvements and tweaks to noise reduction, the iPhone 11 and 11 Pro are real contenders for the smartphone camera throne.

In the past, iPhones made great photos for sharing on social media, but blown up on a big screen, the shots didn’t hold up. It’s why I frequently still pack a ‘big’ camera with me on trips.

With these huge improvements in processing, the iPhone 11 is the first iPhone that legitimately challenges a dedicated camera.

Thanks to some hefty software processing, the sharpness and detail is now on the level of being usable for prints and higher-resolution screen viewing.

One of the shots below was taken with a $4000 Sony A7R4 — a brand new flagship full-frame camera that came out in September — paired with a $1500 G-Master lens with a focal length close to the iPhone’s wide (24mm vs. iPhone’s 26mm). The other shot was taken on an iPhone 11 Pro with its 52mm telephoto lens. ‡

You can probably figure out which is which, but the question is not when we achieve parity, but when we achieve ‘good enough’. Camera makers should be concerned.

‡ obviously, the Sony’s actual image was much larger: but it is also 61 megapixels compared to the iPhone’s 12 megapixels. The iPhone image only had to be scaled down slightly to match the 100% crop of the Sony shot.

Does This Make RAW Obsolete?

No.

With all this detail, it would seem that there’s no real purpose to shooting RAW anymore. After all, if we can have all the detail and no noise, what’s the point?

You’ll find there are still reasons to shoot RAW on iPhone if you want to get more out of that tiny, but powerful array of cameras.

There’s still more detail in RAW

This might astound you, but yes: there’s still more detail in RAW photos. This is particularly noticeable in fine detail that semantic mapping and noise reduction doesn’t quite respect as being meaningful and when Smart HDR has to do a lot of merging:

Yep, detail is still better in RAW. Just look at this 100% crop. The sweater, the hair, grass, and even the trees in the background:

Yes, there’s some grain. The little bit of grain you get in most shooting conditions really only adds to the character of the shot, in my opinion.

You can, of course, always use your favorite editing app to apply noise reduction later. That’s something you can do, rather that being stuck with an image that has all the noise — and with it, underlying detail and texture— removed permanently and irreversibly.

Flexibility in Editing

When a photo is saved as an JPEG or HEIC, it throws away a lot of data useful in editing. Pushing around the highlight and shadows sometimes reveal strange fall-off areas and edges around objects because of exposure merging:

At times, skies appear strange when recovering highlights, whereas the extra data in the RAW file will prevent banding and create a graceful roll-off (a fancy word for gradients that look natural):

Current iPhone 11 RAW files bias towards a rather bright, flat look. Despite this seemingly overexposed look, aggressively pushing down the exposure when editing reveals no detail in the highlights was lost:

For the pros, here’s what the levels looked like initially:

No HDR necessary. Beautiful, smoothly rolled off highlights.

A similarly exposed JPG or HEIC might get a similar look with Smart HDR, but when editing allow far less flexibility.

Editing white balance is a superpower

Cameras face a variety of lighting conditions, and have to figure out what’s white in a shot. You might look at a beautiful snowy scene outside on Christmas Day — hooray a white Christmas! — and snap a photo only to find it blue, or yellow. The sky might cast a blue light on the white snow, or at sunset cast an orange glow, confusing your camera.

If you take a photo in the Camera app, that white balance is committed to your shot and can’t be changed. You can change all the colors in your image by adjusting the tint or hue, but you can’t simply set the white balance after the fact. RAW files let you do this, which gives you the ability to make dramatic shots of wildly varying fall moods:

Or get completely creative with color and color balance and create shots that fit right in on a subreddit like /r/outrun:

As cameras increasingly ‘edit’ our shots for us as we capture them, there’s also something to be said for denying the aesthetic dictated upon us by camera processing and taking matters into your own hands. Photography is art, after all!

There’s no need to choose

If that all sounds a bit overwhelming, we found the best way to enjoy both worlds — the processed magic and the raw power of RAW — was to simply give you both. Apps like Halide are able to take a RAW and processed JPG simultaneously (Halide does this by default) to let you choose later on whatever works best.

That is, if you’re not shooting with the ultra-wide lens.

Halide 1.15: Smart, Smarter, Smartest

Choice is good. In iOS 12, our HEIC/JPG shots in Halide got the majority of software magic on the iPhone XS, XR and 11/11 Pro, but not quite all of it. For instance, we noticed slightly reduced dynamic range in our Smart HDR shots compared to the stock camera app output.

With the introduction of new APIs from Apple in iOS 13, we have released a new update that gives you one more choice.

Halide 1.15 adds a new option: “Enable Smartest Processing”. This allows the camera to take as much time as it needs to get a shot with the exact same process as the built-in camera app, letting you get all the dynamic range or even Deep Fusion in your shots.

The only catch is that you have to disable RAW capture, as it is not possible to take a shot with all the smart processing and a RAW at the same time. (Unfortunately, it’s a limitation outside of our control.)

This new setting is available on devices running iOS 13 and makes the biggest difference on iPhone XS, XR and the 11 and 11 Pro. Deep Fusion is only available on iPhone 11 and 11 Pro, starting with iOS 13.2.

We’re really excited to let you play with both RAW and fully-processed captures in one app, and look forward to hearing which one you prefer.

Speaking of choice: For quick switching between the two, we suggest simply moving around the RAW toggle in the Quick Bar. We’ve made a quick tutorial on how you can do that here.

Happy shooting!

Check back with us soon for Part 2 of our look at the iPhone 11 camera process, featuring a breakdown of Night Mode, Deep Fusion and several other magic tricks. In the mean time, we’re always happy to get questions to answer on Twitter!

Inside the iPhone 11 Camera, Part 1: A Completely New Camera was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last year, before the iPhone XS hit the store shelves, we took a look at the changes in camera hardware on the iPhone XS compared to the iPhone X. We can do this thanks to Halide’s Technical Readout feature.

Much like last year, some kind individuals have shared a few technical readouts with us from iPhone 11 Pro (and iPhone 11) units in the wild.

A few words before we dig in:

Why just the iPhone 11 Pro?

The iPhone 11 and iPhone 11 Pro use the same kind of camera system apart from a key difference: the Pro gains a telephoto camera. All camera, sensor and performance specifications appear identical across the board, so there’s no need for us to compare the iPhone 11 to the iPhone XS. Performance-wise, though, the iPhone 11 looks to be a much better camera than the iPhone XS.

These are just the hardware specs

Every year, cameras including those in your iPhone become far more powerful thanks to both hardware changes and accompanying software processing.

Incremental hardware improvements enable cool new features, but the real magic is increasingly in the software. We’ll follow up with a look at the work that is being done to turn simple raw sensor data into such good images. This year, iPhones can do multi-second long exposures by fusing possibly dozens of frames at night. It will analyze separate parts of the image semantically and render them differently to maximize detail. This is a machine learning camera.

What’s New?

The ultra wide-angle lens and sensor.

Photography defines ‘ultra-wide’ as wider than a 15mm equivalent focal length. What does that mean? That means it’s very wide; it can capture twice as much field of view as your iPhone XS’ regular wide angle lens, which already got a bit wider last year compared to the iPhone X.

13mm is very comparable to an existing type of camera: the action camera, and in particular GoPro. GoPro cameras, depending on their mode, are pretty close to 14mm.

This is a shot taken with a 16mm lens on a full-frame sensor. iPhone 11 Pro’s field of view will be even wider; you could totally use this chest-mounted when mountain biking.

iPhone XS used two different size sensors: the sensor in the main (wide) camera was a bit bigger and better than the one in the telephoto sensor, which meant photos in Portrait mode or some telephoto shots wouldn’t be quite as good.

This is sadly still the case, and both the ultra wide-angle and telephoto lenses use smaller, slightly worse sensors in the iPhone 11 and 11 Pro compared to the sensor they use in the regular ‘wide’ camera.

Of note is that the sensor seems to be lacking some ‘skills’ the other sensors have: it clearly has less dynamic range in some reviewer sample shots, and Matthew Panzarino mentions a lack of focus pixel coverage and as such, does not do Night Mode. That also means you won’t be able to do Portrait Mode (yet…?) with it. It also lacks OIS (Optical Image Stabilization), though ultra wides fortunately suffer far less from errant camera movement because of their large field of view.

New, sensitive sensors all around

We got new sensors! The biggest notable change across the board this year is what we already teased: the max ISO of the sensors has gone up significantly. The Wide camera’s maximum ISO sensitivity is up 33%; the telephoto 42%!

ISO is a number used to describe how sensitive a camera can be to light. A higher maximum ISO simply means a sensor can make itself even more sensitive to light in the dark. This is at the expense of noise: the higher your ISO setting, the noisier your images.

Maximum ISO doesn’t tell a whole story, as it’s completely possible for a sensor to just produce a noisier result. From our initial impressions, however, this higher maximum ISO also means that the new sensors simply produce a lot less noise at regular ISO ranges, and take better images in the dark. A welcome change from the iPhone XS, which tended to produce especially noisy RAW files.

Good news for telephoto lens lovers: Pair that with the telephoto’s new f/2.0 aperture and you have a lot more light-gathering which should prevent the Camera app from constantly picking the wide-angle image over the dark, muddy telephoto one in mixed lighting.

Bokeh-tastic f/2 telephoto aperture

We’re expecting to see some really nice natural depth of field from the new, faster f/2.0 aperture on the telephoto lens. This faster aperture, paired with its higher sensitivity to light, should allow much nicer Portrait mode and regular telephoto shots.

Long exposure times

It was big news to see the iPhone XS wide-angle sensor to go up to a full second of exposure last year; this year brings that 1 second capture ability to the telephoto and new ultra-wide angle sensor. While it appears Night Mode doesn’t work with the ultra-wide, it does seem to allow a second long exposure if you have a tripod and manual camera app handy.

Minimum exposure times

Wow, this is a quick sensor. The wide angle camera can now capture images at up to 1/125000th of a second. That’s over six and a half times faster than the iPhone XS. We don’t yet know what this means, practically, but it’s a very small slice of a second.

TrueDepth changes

Not something most people are foaming at the mouth for, but a respectable upgrade in the front-facing (a.k.a. ‘TrueDepth’) camera department. We’re seeing a slightly wider lens of 24mm vs. 30mm on the iPhone XS and a resolution upgrade to 12 megapixels. It’s an all new sensor behind the lens, enabling ‘slofies’ (sorry, we said it) which require the sensor to be able to capture high-FPS video. It, too, has a new and higher maximum ISO: 28% higher than last year.

Regarding its new 24mm range: By default, the Camera app will crop this back to the expected 30mm range until you rotate it to landscape or toggle the new, wider view. The extra few mm in focal length is likely there to facilitate a wider angle for Face ID unlocking. We’d love to see an even wider lens here, or perhaps two lenses like on the Pixel 3. Ultra-wide selfies?

iPhone 11 Pro’s Technical Readout Comparison

That’s it for the bird’s eye view — here’s the raw specs you’ve been waiting for. Changes from last year’s iPhone XS are bold.

And here’s the comparison of the TrueDepth system, with changes in bold.

Not too shabby.

A note on these changes

We said it last year, and we’ll say it even louder this year: this is only a look at what changed in the camera hardware.

It’s kind of unbelievable that even with the glowing reviews out today, Apple has said that there’s more software processing yet to come. We’re told Deep Fusion is a very big leap in post-processing quality, but with the changes to Smart HDR, Semantic Mapping in the imaging pipeline and discrete situational processing like Night Mode, these specs are the furthest from the whole story on the new iPhone cameras yet.

It’s 2019, and your camera isn’t just a module that takes a photo anymore. Cameras compose images from dozens of exposures, mixing and matching pixels from various frames, changing the output creatively and intelligently to ensure you get an image that looks, ironically, more faithful to what we see. All of this means that with fairly modest changes in hardware (except for the new ISO ranges), the iPhone 11 and 11 Pro appear to deliver one of the biggest leaps in camera quality in iPhones yet.

This technical readout was brought to you by Halide. We can’t wait to take a deep dive into the software processing and compare them to the raw sensor output — stay tuned for that next week.

iPhone 11 Pro Preview: The Camera Hardware Changes was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ve had an incredible amount of submissions to our contest! While we already got a steady stream of excellent photos tagged with #ShotWithHalide on Instagram and Twitter, the absolute deluge of stellar shots we got during the contest period was overwhelming!

It took us a bit of extra time to look through and select the winners, so pardon the delay. With that said, let’s get to our winners!

First place

The grand winner of the contest is Travis Ross. This photo captures everything that makes a great photo: emotion, movement, composition. In this absolute chaos, Travis used Halide to snap a photo worth framing.

That won him a fantastic Leica Q2 camera. It should be able to cover any cases in which an iPhone and Halide just doesn’t quite cut it!

Second place

Our second place winner, Tomás Francisco Lemos, has this spectacular Spectre shot taking second place:

Winning a Polaroid Classics OneStep+, to take even more great photos — but now developed as Polaroids!

Third place

Coming in in third place, Jordan Duke’s Halide capture struck us with the power of, well, a lightning strike:

Jordan will get Studio Neat goodies including a Glif tripod mount for iPhone, a notebook and a wireless charging base. Score!

The rest

We have selected 47 other winners who we’re contacted so we can send them a set of Halide and Spectre merch, including t-shirts, stickers and pins.

Didn’t win, or miss out on the contest? Check back soon when we have news about our new app updates we’ve been working on — and new Halide and Spectre merch store. 👀

Halide: Our Contest Results was originally published in Halide on Medium, where people are continuing the conversation by highlighting and responding to this story.