NeuralHash update: of attempts at explaining, hackers and the backdoor that supposedly isn’t

Dominik Bärlocher

20-8-2021

Translation: Veronica Bielawski

Apple’s vice president Craig Federighi opens up about the NeuralHash child porn detector. A hacker has already replicated the system, and child safety advocates are taking action against critics.

Apple has been making waves with its plan to crack down on child pornography and abuse. And it didn’t take long for the world to react. Independent researchers were quick to raise a loud outcry, which was publicly ignored. Internally at Apple, it isn’t so.

Here’s a look at a discussion gone off course.

Apple NeuralHash and communication safety in Messages: the lowdown

This article is about two systems, both of which aim to protect children. But NeuralHash – called CSAM detection in its implementation – and the somewhat awkwardly named «communication safety in Messages» work in completely different ways.

NeuralHash/CSAM detection

Apple has developed the CSAM detection system in cooperation with the child safety organisation National Center for Missing and Exploited Children (NCMEC). CSAM stands for «child sexual abuse material».

The system scans photos on your iPhone that you want to sync to iCloud. It compares the images to images of child pornography known to the NCMEC. This is done on your device, i.e. before your images are uploaded to iCloud.

Apple NeuralHash scans the NCMEC’s database and gives each image a new hash that doesn’t just index it, but understands its content. This is supposed to allow NeuralHash to correctly identify images even if they’ve been modified, such as recoloured or resized.

In short: NeuralHash is supposed to understand the content of the image instead of just the data in the image file. It doesn’t just see «1024 pixels wide, 768 pixels tall, this pixel here is red and that one is blue». Rather, it sees something like «picture on a beach with a sunset and a dog in the sand».

Should Apple detect «a certain number of» matches on an iPhone or iPad, an alert will be triggered. These images are then transmitted to Apple unencrypted, where they’re reviewed by a human. If the suspicion of child pornography is confirmed, the NCMEC is alerted. The NCMEC in turn alerts the appropriate authorities.

CSAM detection does not detect newly generated material.

For now, CSAM detection will only be rolled out in the United States in an upcoming version of iOS 15 and iPadOS 15. Users have neither insight nor influence on the workings of CSAM detection.

Communication safety in Messages

The second child safety feature Apple has implemented is called communication safety in Messages. This is a tool for parents whose children have an iPhone and whose Apple ID is linked to that of the parents.

If the feature is turned on, then incoming and sent images are monitored. Communication safety in Messages recognises nude pictures – both existing and newly snapped ones – and places a filter over them. The images are displayed blurred.

In addition, the child receives a reassuring message. They’re informed that the image is intentionally blurred. The message also tells the child that they don’t have to view the image and that resources on how to protect themselves are available.

If the child is under 12 years of age, Apple also gives parents the option to be notified when communication safety in Messages intervenes.

Craig Federighi admits to mistakes in communication

On 13 August 2021, Apple’s senior vice president of Software Engineering, Craig Federighi, made a media appearance. He concedes that the communication of the two systems has led to confusion.

«We wish that this would’ve come out a little more clearly for everyone because we feel very positive and strongly about what we’re doing and we can see that it has been widely misunderstood,» says Federighi at the beginning of the video.

According to him, the mistake lies in the fact that Apple introduced the two features – CSAM detection and child safety in Messages – simultaneously. Since both features have the same goal in mind, he claims they are easy to confuse with each other.

Apple doesn’t scan the pictures on your iPhone? Apples scans the pictures on your iPhone!

Craig Federighi says that the sound bite that got out – «oh my God, Apple is scanning my phone for images» – is wrong. He clarifies that what's actually happening is that images that are being stored in the cloud are being checked against a database. An alarm is only supposed to be raised if there’s a match according to the NeuralHash system.

Supposedly malicious gossip has it that Apple is scanning pictures on iPhones, since CSAM detection is a kind of scanning tool. One that looks at pictures. On iPhones. Images are compared to the database locally – on your device – before being uploaded to iCloud. iCloud itself is not scanned. «Find» is a rather odd choice of word, since it implies a preceding search. And that would mean Apple is looking for child pornography on your iPhone.

NeuralHash does do something in iCloud, after all

Starting at about the 2:45 mark in the video, Craig Federighi explains that not all of NeuralHash’s functionality is built into the iPhone; «the other half» is in iCloud. The vice president stresses that no user is obligated to use iCloud.

NeuralHash was reportedly invented so that questionable images like these could be found without Apple having to «look at pictures». Federighi does, however, provide some insight into how NeuralHash works: The first scan happens locally on the phone, the second in the cloud.

«Seeing» is a matter of definition

For the first time, Craig Federighi mentions an actual figure. Before NeuralHash shares any data with a human, CSAM detection must detect «something on the order of 30» images in your iCloud library. In other words, only once the threshold of about 30 images is reached are these 30 images transmitted to Apple unencrypted. Then, an actual person at Apple looks at these images and, if needed, alerts the NCMEC. Apple employees don’t have access to any other pictures on your iPhone.

It’s clear that Apple defines «seeing» and «viewing» in its own way and, in doing so, distorts the notion of an invasion of privacy. Apple seemingly defines the two terms to mean that an Apple employee can look at your pictures and conclude, «That’s one cute cat there!» Now, with NeuralHash, a human only has access to photos after the automated, non-human system has detected «something on the order of 30» questionable images – and then only those particular images are transmitted unencrypted. Apparently, Apple doesn’t consider this to be «seeing» or «viewing» photos.

But privacy activists define «seeing» and «viewing» as a manufacturer being able to access images on your iPhone. So, a manufacturer automatically evaluating images and reserving the right to transmit them, if necessary, to a human who then looks at them and judges them to be criminal or not is, in fact, an invasion of privacy.

Bathtub pictures are okay

Craig Federighi gives the all-clear on photos you take of your child as they grow up. Especially in their first few months of life, babies are often photographed with little to no clothing.

While CSAM detection does examine these images, it doesn’t raise an alarm.

It only raises an alarm on images that already exist in the NCMEC’s database and have been indexed by NeuralHash. NeuralHash doesn’t classify newly taken pictures as child pornography and won’t raise the alarm when you’re at the pool with your kid.

«NeuralHash is in no way a backdoor»

Craig Federighi says that NeuralHash is «in no way» a backdoor into the system. He says that NeuralHash is only being applied as part of the process of storing something in the cloud,

and that he really doesn’t understand its characterisation as a backdoor.

To clarify: according to dictionary.com, a backdoor is a «secret access point or undocumented vulnerability in a software program, hardware component, or digital network, sometimes intentionally maintained as for remote developer access, but also sometimes created or exploited for unauthorised access by hackers».

So, if Apple can send a picture from your iPhone to an employee without your cooperation, that would by definition be a backdoor. If NeuralHash detects «something on the order of 30» images on your iPhone, then Apple gets access to your data – despite encryption, your passcode and face unlock.

This may contradict Apple CEO Tim Cook’s statements, depending on how you define the term «backdoor».

Over 60 organisations ask Apple not to do this

The Center for Democracy and Technology (CDT) reported on an open letter addressed to Apple CEO Tim Cook on 19 August 2021.

In the letter, over 60 signatory organisations oppose NeuralHash and – depending on the degree of misunderstanding – communication safety in Messages. In the first paragraph of the letter, they make clear how this discussion should be conducted.

The danger of this technology must be included in the discussion. And a differentiation between anecdotal and systemic must be made; anecdotally, CSAM detection cracks down on child pornography. Systemically, however, the system could also be used to detect homosexuals in Poland, to track down Uyghurs in China, or even to detect skin cancer in its early stages.

If NeuralHash’s AI can build up a model of something, then it can be used to find that «something» and raise an alarm – be it red cars, for whatever reason, or women in Iran not wearing the hijab.

NeuralHash is freely available and already exists in iOS 14.3

The hacker Asuhariet Ygvar has «rebuilt» NeuralHash. In a Reddit post, the hacker describes what he or she did.

The NeuralHash model on the iPhone is written in MobileNetV3 architecture, a computer vision model by Google. Asuhariet Ygvar exported the model to ONNX and rewrote the whole NeuralHash algorithm in the programming language Python.

This has allowed NeuralHash to leave the Apple platform and to even run on Linux.

AppleNeuralHash2ONNX is freely available on GitHub and can be freely configured. From a cynical point of view, China now has its Uyghur detector.

With this, Asuhariet Ygvar exposes another danger of the system: its very notion. Sure, Apple may try its hardest to be the only one able to configure the system. It has implemented a human review so that false alarms can be avoided as much as possible. The NCMEC is also generally considered to be trustworthy.

But the idea alone was enough to get someone to replicate the system. It’s safe to assume that Asuhariet Ygvar is not the only person who has already done this or will still do this. By introducing NeuralHashes, Apple has proven the concept. That alone can inspire people to think, «If Apple can do this, so can I.»

Asuhariet Ygvar is a pseudonym, by the way. The name comes from the manga/anime «Rotte no Omocha!».

First artificially generated false positives

Asuhariet Ygvar’s AppleNeuralHash2ONNX is a new toy for hackers around the world to play with. It was published on GutHub on 16 August 2021. Two days later, Cory Cornelius posted two images that he claims generate the same hash. A hash is a number that should clearly indicate what’s in the image. Craig Federighi describes hashes as non-human-readable «gobbledygook». A hash is an identification number assigned to an image based on an abstraction of what a Neural Network sees in said image. Basically: NeuralHash looks at an image, interprets it according to the given parameters and assigns a corresponding number.

This number tells the network what’s on the image.

NCMEC gives NeuralHash access to its database of child pornography. NeuralHash creates hashes of those images. These hashes are uploaded to your smartphone, where they’re compared to the hashes of your pictures.

Cory Cornelius asks if these two images really do have identical hashes, or if he’s the only one getting that result.

$ python3 nnhash.py NeuralHash/model.onnx neuralhash_128x96_seed1.dat beagle360.png
59a34eabe31910abfb06f308
$ python3 nnhash.py NeuralHash/model.onnx neuralhash_128x96_seed1.dat collision.png
59a34eabe31910abfb06f308

This is, of course, an unfavourable result. But Apple says a match like the one pointed out by Cory Cornelius and confirmed by Asuhariet Ygvar is a case of one in a trillion.

NCMEC responds to criticism... and puts itself in the doghouse

The harsh criticism of Apple has left its mark in Cupertino. In a memo to his employees, Sebastien Marineau-Mes, one of Apple’s software vice presidents, hoped to encourage his employees and explain next steps. In addition to keeping children safe being «such an important mission», he wrote the following:

Included in this internal memo, obtained by the Apple news magazine 9to5Mac, is a memo from the NCMEC. It’s written by Marita Rodriguez, executive director of strategic partnerships. In the memo, she hopes to give Apple encouragement. She also responds to initial criticism, adding fuel to the flames:

As was to be expected, the NCMEC did itself no great favours by publishing this memo. It’s the line about the «screeching voices» that breeds resentment and a lack of understanding.

Reddit user u/classycatman summed it up pretty well:

Dominik Bärlocher

Senior Editor

dominik.baerlocher@digitecgalaxus.ch

Journalist. Author. Hacker. A storyteller searching for boundaries, secrets and taboos – putting the world to paper. Not because I can but because I can’t not.

These articles might also interest you