Google has an automated tool to detect child pornography, but sometimes the system gets it wrong.
Mark noticed something was wrong with his toddler. His son's penis looked swollen and sore. Mark, a dad who lives in San Francisco, grabbed his Android smartphone and took photos to document the problem so he could track its progress.
It was a Friday night in February 2021. His wife called a nurse at their healthcare provider to schedule an emergency visit for the next morning, via video conference, because it was a Saturday and there was a pandemic. The nurse told Mark to send photos so the doctor could review them in advance.
Mark’s wife grabbed her husband’s phone and sent some high-quality close-up shots of their son’s groin to her iPhoneso he could upload them to the healthcare provider’s system. In one of the photos, Mark’s hand was visible so that the swelling could be seen better for comparison. Mark and his wife didn’t think at all about the tech giants that made this rapid digital data capture and sharing possible, or what those giants might think of the images.
With the help of the photos, the doctor diagnosed the problem and prescribed antibiotics, which quickly cleared the infection. The incident, however, left Mark with a much bigger problem that would cost him more than a decade of stored contacts, emails and photos and make him the target of a police investigation. Mark, who asked to be identified only by his first name out of fear for his reputation, had been caught in an algorithmic web designed to trap people sharing child sexual abuse material.
Because tech companies often record so much data, they have been pressured to act as watchdogs, examining what passes through their servers to detect and prevent criminal behavior. Child advocates say the companies’ cooperation is essential to combat the rampant online spread of child pornography. But that may include inspecting private files, such as digital photo albums. The unexpected scrutiny has wrongly labeled innocent behavior as malicious in at least two cases the Times has uncovered.
Jon Callas, a technologist at the Electronic Frontier Foundation, a digital civil liberties organization, said these cases are early warning signs.
"There could be dozens, hundreds, thousands more of these cases," he said.
See also: Google: Repel record-breaking HTTPS DDoS attack
Given the toxic nature of the allegations, Mr. Callas assumed that most people wrongly accused of distributing child pornography would not go public with what had happened.
"I knew these companies were tracking everything and that privacy is not what we would want it to be," Mark said. "But I haven't done anything wrong.".
The police agreed but Google did not back down.
After setting up a Gmail in his mid-40s, Mark, now in his 40s, relied heavily on Google. He synced appointments with his wife on Google Calendar. His Android smartphone camera backed up his photos and videos to the Google cloud. He even had a phone plan with Google Fi.
Two days after taking the photos of his son, Mark’s phone beeped: his account had been disabled due to “harmful content” that was a “serious violation of Google policies and may be illegal.” A “learn more” link led to a list of possible reasons, including “child sexual abuse and exploitation.”
Mark was confused at first but then he remembered his son's infection. "Oh my God, Google probably thinks it was child pornography," he thought.
In an unusual twist, Mark had worked as a software engineer on a large tech company's automated tool for removing video content reported by users as malicious. He knew that such systems often have a human in the loop to ensure that the computers don't make mistakes, and he assumed his case would be cleared up once he reached that person.
He filled out a form requesting a review of Google’s decision, explaining his son’s infection. At the same time, he discovered the domino effect of Google’s rejection. Not only did he lose emails, contact information for friends and former colleagues, and data from his son’s early years, but his Google Fi was also closed, meaning he had to get a new phone number with another company. Without access to his old phone number and email address, he couldn’t get the security codes he needed to log into other online accounts, cutting him off from much of his digital life.
"The more eggs you have in one basket, the more likely the basket is to break," he said.
In a statement, Google said: "Child sexual abuse material is abhorrent and we are committed to preventing its spread on our platforms.".
A few days after Mark filed the appeal, Google responded that it would not reinstate the account, without further explanation.
Mark didn't know it, but Google's review team had also flagged a video that prompted the San Francisco Police Department to investigate him.
The day after Mark’s problems began, the same scenario was playing out in Texas. A child in Houston had an infection in his “inner parts,” his father wrote in an online post. At the pediatrician’s request, Cassio used an Android to take photos of his child, which were automatically backed up to Google Photos. He then sent them to his wife via Google’s chat service.
See also: Google Down: Tech giant apologizes for outage
Cassio was in the middle of buying a house and signing countless digital documents when his Gmail was deactivated. He asked his credit broker to change his email address, which made him suspicious until Cassio's real estate agent vouched for him.
"It was a headache," said Cassio.
Images of children being exploited or sexually abused are reported by tech giants millions of times each year. In 2021, Google alone filed over 600,000 reports of child abuse material and disabled the accounts of over 270,000 users as a result. Mark and Cassio’s experiences were 2 drops in a big bucket.
The tech industry’s first tool to seriously disrupt the massive online distribution of child pornography was PhotoDNA, a database of known abuse images that were converted into unique digital codes , or hashes. It could be used to quickly scan a large number of images to find matches, even if the image had been altered. After Microsoft released PhotoDNA in 2009, Facebook and other tech companies used it to root out users who were circulating illegal and harmful images.
"It's an amazing tool," the president of the National Center for Missing and Exploited Children.
A bigger breakthrough came nearly a decade later, in 2018, when Google developed an artificial intelligence tool that could identify child abuse images it had never seen before. That means that in addition to images of known children who have been sexually abused, the tool could generally identify images of unknown victims who could be saved by authorities. Google has made its technology available to other companies, including Facebook.
When Mark and Cassio ’s photos were automatically uploaded from their phones to Google’s servers, the tool flagged the images as child pornography . Jon Callas of the EFF called the tool’s scanning intrusive, saying that a family photo album on someone’s personal device should be within the “private sphere.”
"This is exactly the nightmare that concerns us all," Mr. Callas said. "They'll scan my family album and then I'll be in trouble.".
A human content moderator for Google would review the photos after they were flagged by the AI tool to confirm that they met the federal definition of child pornography material. When Google makes such a discovery, it locks the user's account, searches for other similar material, and as required by federal law makes a report to the CyberTipline at the National Center for Missing and Exploited Children.
The nonprofit received 29.3 million reports last year, or about 80,000 reports a day of child abuse material. Fallon McNulty, who runs the CyberTipline, said most of those are images that have been reported in the past and are circulating online consistently. So 40 analysts are focusing on potential new victims so they can prioritize those cases for law enforcement.
Ms. McNulty said NCMEC staff prioritizes reviewing CyberTiplinethat include previously unreported material. “This may be a child who has not yet been identified or protected and is not safe.”
Ms McNulty said Google's astonishing ability to identify these images so her organisation could report them to the police for further investigation was "an example of the system working as it should".
CyberTipline staff members add any new images to the hashed database that they share with tech companies for review purposes. When Mark's wife learned of this, she deleted the photos of their son that Mark had taken, fearing that Apple might report her account. Apple announced plans last year to screen iCloud photos for child pornography, but the move was delayed indefinitely after opposition from privacy groups.
In 2021, CyberTipline reported that it had notified authorities of “over 4,260 potential new child victims.” Among them were Mark and Cassio’s sons.
In December 2021, Mark received a package in the mail from the San Francisco Police Department. It contained a letter informing him that he was being investigated, as well as copies of search warrants served on Google and internet service provider . An investigator had requested everything in Mark’s Google account: his web searches, his location history, his messages, and any documents, photos, and videos he had stored.
The investigation, related to "child pornography videos," took place in February, a week after the photos of his son were taken.
Mark called the investigator, Nicholas Hillard, who said the case was closed. Mr. Hillard had tried to contact Mark, but his phone number and email address had not worked.
“I found that the incident did not meet the criteria of a crime and that nothing malicious occurred,” Mr. Hillard wrote in his report. The police had access to all the information Google about Mark and determined that it was not child abuse or exploitation.
Mark asked if Mr. Hillard could tell Google he was innocent so they could get his account back.
“You need to talk to Google,” Mr. Hillard said, according to Mark. “There’s nothing I can do.”.
Mark appealed case to Google, providing the police report, but to no avail. After receiving notice two months ago that his account was being permanently deleted, Mark spoke with a lawyer about suing Google and asked how much it might cost.
"I decided it probably wasn't worth $7,000," he said.
Kate Klonick, a law professor at St. John’s Universitywho has written about online content policing, said it can be difficult to “account for things that are invisible in a photo, like the behavior of the people sharing the image or the intentions of the person taking it.” False positives, where people are wrongly accused, are inevitable because billions of images are scanned. While most people would likely ignore false reports to make sure they aren’t distributing abusive images, Ms. Klonick said companies need a “robust process” to redress innocent people who are wrongly thought to be distributing child pornography.
“False reports would be considered problematic if the issue was simply about content restriction and censorship,” Ms. Klonick said. “But this is doubly dangerous because it also leads to someone being reported to law enforcement.” A parent could lose custody of their children because of these false reports.
Cassio was also being investigated by police. A detective from the Houston Police Department called in the fall of 2021, asking him to come to the department .
After Cassio showed the detective his communications with the pediatrician, the misunderstanding was quickly cleared up. But he also couldn't get his Google account back, despite being a paying Google user. He now uses a Hotmail for email, which people make fun of him for, and creates multiple backups of his data.
Not all photos of naked children are pornographic or abusive. Carissa Byrne Hessick, a law professor at the University of North Carolina who writes about child pornography crimes, said the legal definition of what constitutes sexually abusive images can be complicated.
But Ms. Hessick said she agreed with police that the medical images did not qualify. "There is no child abuse," she said. "The image was taken for non-sexual reasons."
Through machine learning, a computer program is trained by feeding it “right” and “wrong” information until it can distinguish between the two. To avoid reporting photos of babies in the bathroom or children running naked through sprinklers, Google’s AI was trained with both images of child pornography and non-abuse images so it could more accurately understand what should be reported.
“We recognize that in an era of telemedicine and especially Covid, it has been necessary for parents to take photos of their children in order to get a diagnosis,” said Claire Lilley, Google’s head of child safety. The company has consulted with pediatricians, she said, so that its human reviewers understand potential conditions that may appear in photos taken for medical reasons.
Suzanne Haney, chair of the American Academy of Pediatrics' Council on Child Abuse and Neglect, advised parents not to take photos of their children's genitals, even when asked to do so by a doctor.
“The last thing you want is for a child to feel comfortable with someone taking pictures of their genitals,” Haney said. “If it absolutely has to be done, avoid uploading to the cloud and delete the images immediately.”
He said most doctors were probably unaware of the risks when they asked parents to take such photos.
"I applaud Google for what it's doing," Haney said of the company's efforts to combat abuse. "We have a horrible child pornography problem. Unfortunately, this was tied to parents trying to do the right thing for their children.".
Cassio was told by a customer support representative that sending the images to his wife using Google Hangouts violated the chat service's terms of service. "Do not use Hangouts in any way that exploits children," the terms state. "Google has a zero-tolerance policy against this content."
As for Mark, Ms. Lilley, at Google, said that reviewers had not detected a rash or redness in the photos he took and that a subsequent review of his account turned up a video from six months earlier that Google also deemed child pornography, of a young child lying in bed with a woman without clothes.
Mark didn't remember this video and no longer had access to it, but he said it sounded like a private moment he'd been inspired to capture, not realizing anyone else would ever see or judge it.
"I can imagine it. We woke up one morning. It was a beautiful day with my wife and son and I wanted to capture the moment," Mark said. "If we had just slept in our pajamas, this could have been avoided.".
A Google spokeswoman said the company stands by its decisions, even though law enforcement cleared the two men of child pornography charges.
Ms. Hessick said the cooperation that technology companies provide to law enforcement to address and eliminate child sexual abuse is “incredibly important,” but she thought that corrections should be allowed.
“From Google’s perspective, it’s easier to deny these people access to the services,” he speculated. Otherwise, the company would have to resolve more difficult questions about “what is appropriate behavior toward children and what is appropriate to photograph or not.”.
Mark still has hope that he can get his data back. The San Francisco police department is keeping the contents of his Google account in a drive. Mark is now trying to get a copy. A police spokesman said the department is willing to help him.
Source: nytimes.com
