Insta's AI finds nudes in encrypted DMs

Instagram nudes sent or received by under-18s will be detected and blurred with device-based software able to scan end-to-end encrypted (E2EE) direct messages (DMs).

The launch of the client-side feature coincides with Facebook Messenger’s current, and Instagram’s planned, global rollout of default E2EE, which police and regulators have opposed.

E2EE is not interoperable with the server-based software Meta currently uses to detect, remove and report child sexual exploitation material (CSEM).

Following pushback against mandatory E2EE backdoors, online safety watchdogs in the UK and Australia have more recently pitched scanning communications from users’ devices, or from government-owned, intermediary servers, as an alternative method of detecting and removing illegal content. 

However, unlike the device-based content scanning technology that UK and Australian regulators are pushing E2EE providers to deploy, Instagram’s Nudity Protection feature does not block illicit material.

The settings, default for teens and available to adults, warn users of the dangers of sexual exploitation scams and revenge porn, but still allows them to unblur detected nudes. 

“Meta won’t have access to these images – unless someone chooses to report them to us,” the company, which, in the last financial year, handed 5054 users’ data to various Australian law enforcement and regulatory bodies, said in a statement.

Acting eSafety Commissioner Toby Dagg told iTnews that he “welcomes” the feature, but “would also welcome further information from Meta about the long-term efficacy and uptake of these tools over time.” 

Scanning platforms for crime

Dagg said that eSafety’s “transparency notices to 29 services, including those owned by tech giants Apple, Meta, Microsoft and Google” had mapped out the use of language analysis processing to detect child grooming, which Xbox Live, Facebook, Instagram, TikTok and Twitch currently use. 

“Reports to our investigators show that the criminals behind these [sexual extortion] scams initially make contact posing as an attractive young woman on social media services, with Instagram and Snapchat the most frequently targeted.” 

When unencrypted, Facebook, Instagram, TikTok, Twitch, Google Chat, Twitter, TikTok, Snapchat, Xbox Live, and Discord messages are scanned for verified CSEM, reports eSafety released in 2022 [pdf] and 2023 [pdf] revealed. 

Instagram, Facebook, YouTube, Discord and Twitch also use AI trained on verified CSEM to detect new CSEM. 

Automate detect and block regime

Dagg said that eSafety’s “transparency powers” to evaluate platforms’ detection software “work hand in hand with new mandatory codes which require providers of online products and services in Australia to do more to address the risk of harmful material, including child sexual exploitation material and grooming.” 

In the UK and Australia, Meta, Apple, Signal and other E2EE providers are pushing back against their inclusion in industry codes that could mandate solutions that scan, detect and block content before encryption when the regulator deems it "technically feasible" for the provider in question. 

“Technical feasibility” depends on “whether it is reasonable for service providers to incur the costs of taking action, having regard to the level of risk to the online safety of end-users.”

eSafety has said that scanning communications from a device or government-owned server would not amount to mandating “companies to design systematic vulnerabilities or weaknesses into any of their end-to-end encrypted services.”

Meta’s submission [pdf] to eSafety said that, unless the industry codes explicitly defined “technically feasibility” to exclude solutions that could “render methods of encryption less effective”, they could force providers to design systematic vulnerabilities or weaknesses.

Like Apple’s similar, device-side child safety features for iMessage, Meta’s nudity protection feature is likely aimed at demonstrating to authorities that harmful E2EE material can be reduced without third-parties directly blocking or reporting it.

eSafety’s Updated Position Statement on End-to-end encryption [pdf], released in October, said Apple’s child safety feature “demonstrates – at scale – that device side tools can be used alongside E2EE, without weakening encryption and while protecting privacy.”

It adds, however, that “Apple’s intervention is limited in that it does not prevent the sharing of illegal material or activity, or enable accounts to be banned by the service.”

eSafety was more supportive of Apple’s iCloud solution [pdf], which Apple discontinued after a backlash from privacy advocates. 

The iCloud solution would have scanned content from users' devices before it was uploaded to their backup; police would have been alerted when illegal material was detected.

Apple's director of user privacy and child safety Erik Neuenschwander said in an email [pdf] obtained by Wired that the project was ditched over concerns it could create new “threat vectors for data thieves to find and exploit" and lead to authoritarian surveillance through function creep.

“How can users be assured that a tool for one type of surveillance has not been reconfigured to surveil for other content such as political activity or religious persecution?” the email - sent to a child rights group that supported Apple readopting the solution - said in August.