Nobody is ready for this

Our basic assumptions about photographs capturing reality will soon go up in smoke.

An explosion on the side of an senior brick building. A crashed bicycle at a city intersection. A cockroach in a takeout box. Each of these images took less than 10 seconds to create using Reimagine in Magic Editor on a Pixel 9 phone. They’re crisp. They’re full color. They’re high quality. There’s no suspicious background blur or giveaway sixth finger. These photos are incredibly convincing, and they’re all fucking fraudulent.

Anyone who buys a Pixel 9 — Google’s latest flagship phone, available this week — will have access to the easiest, lightest user interface for the best lies, built right into the mobile device. This will almost certainly become the norm, with similar features already available on competing devices and coming to others in the near future. When a smartphone “just works,” that’s usually a good thing; here, that’s the whole point.

The photo was used for fraud purposes as long as it exists. (To consider Victorian style photosthis the infamous Loch Ness Monster photoOr Stalin’s Photo Purges (It would be dishonest to say that photographs have Never considered credible evidence. Anyone reading this in 2024 grew up in an era when photography was, by default, a representation of truth. A staged scene with movie effects, digital photo manipulation, and, more recently, deepfakes were all potential frauds to consider, but they were exceptions in the realm of possibility. It took specialized knowledge and specialized tools to sabotage the intuitive trust in photographs. Faking them was the exception, not the rule.

If I tell Tian’anmen Squareyou will most likely imagine the same picture as me. This also applies Abu Ghraib Or girl with napalm. These images defined wars and revolutions; they captured truth in a way that cannot be fully expressed. There was no reason to express why these images were significant, why they were so crucial, why we attached so much importance to them. Our trust in photographs was so deep that when we spent time discussing the truth of images, it was more significant to dwell on the fact that it was sometimes possible for photographs to be fraudulent.

This is all about to flip—the default assumption about the photo becomes that it is faked, because creating realistic and believable fraudulent photos is now petty. We are not prepared for what happens next.

Edited with Magic Editor by Google.

No one on Earth has ever lived in a world where photographs were not a pillar of social consensus—for as long as any of us have been here, photographs have been evidence that something happened. Consider all the ways in which the supposed veracity of photographs has previously confirmed the veracity of your experiences. An earlier dent in the fender of your rental car. A leak in your ceiling. The arrival of a package. A real, non-AI cockroach in your takeout. When wildfires rage on in your neighborhood, how do you communicate the density of the smoke outside to friends and acquaintances?

Until now, the burden of proof has fallen largely on those who deny the authenticity of the photo to prove their claims. The flat-earther is out of step with the social consensus not because he doesn’t understand astrophysics—how many of us understand astrophysics, after all?—but because he has to engage in a series of increasingly complicated justifications for why certain photos and videos aren’t real. He has to invent a extensive state conspiracy to explain the steady boost in satellite images that capture the curvature of the Earth. He has to create a sound stage for the 1969 moon landing.

We took it for granted that the burden of proof was on them. In the age of Pixel 9, it would be best to start brushing up on our astrophysics.

In most cases, the average image these AI tools create will be fairly harmless on its own—an extra tree in the background, an alligator in a pizzeria, a humorous costume on a cat. In brief, the deluge completely changes the way we think about the concept of a photograph, and that in itself has huge repercussions. Consider, for example, the extraordinary social upheaval that has taken place in the United States over the past decade over grainy footage of police brutality. Where authorities have obscured or covered up the reality, those recordings have told the truth.

The persistent cry of “Fake News!” from Trumpists heralded the beginning of an era of uncompromising bullshit, where the influence of truth would be smothered by a firehose of lies. The next Abu Ghraib would be buried under a sea of ​​AI-generated war crimes. The next George Floyd would go unnoticed and unjustified.

Edited with Magic Editor by Google.

The shape of things to come is already evident. In Kyle Rittenhouse’s trial, the defense argued that Apple’s pinch-to-zoom photo feature manipulated photos, effectively convincing the judge to the burden of proof rests on the prosecutor, who must show that the iPhone recording was made using Zoom NO Manipulated by AI. Donald Trump recently falsely claimed that a photo from a well-attended Kamala Harris rally was generated by AI — a claim that was only possible because people were able to believe it.

Even before AI, we in the media operated in a defensive posture, scrutinizing the details and provenance of every image, checking for misleading context or photo manipulation. After all, every major news event comes with a flood of disinformation. But the coming paradigm shift implies something much more fundamental than the constant nagging of suspicion sometimes called digital literacy.

Google understands perfectly well what it is doing with the photo as an institution — in Interview with Wiregroup product manager for the Pixel camera described the editing tool as “a assist[ing] you create a moment that is as you remember it, that is original to your memory and the larger context, but maybe not original to that particular millisecond.” A photograph in this world is no longer an addition to the fallibility of human memories, but instead a mirror of them. And when photographs become little more than hallucinations, the stupidest shit will turn into a legal battle over the reputation of witnesses and the existence of corroborating evidence.

This erosion of social consensus began before the Pixel 9, and it won’t continue with the Pixel 9 itself. Still, the phone’s recent AI capabilities are notable not just because the barrier to entry is so low, but because the safeguards we did encounter were astonishingly anemic. The industry’s proposed standard for AI image watermarking is mired in the usual standards grind, and Google’s own much-vaunted AI watermarking system was nowhere to be seen when Edge We tried Magic Editor on a Pixel 9. Photos modified with Reimagine simply have a line of removable metadata added to them. (The inherent fragility of this kind of metadata was supposed to be addressed by Google’s invention of the theoretically indelible SynthID watermark.) Google told us that Pixel Studio’s output — a tidy prompt generator that’s closer to DALL-E — would be watermarked with the SynthID; ironically, we found that the capabilities of Magic Editor’s Reimagine tool, which modifies existing photos, were much more alarming.

Photo: Cath Virginia / The Verge, Neil Armstrong, Dorothea Lange, Joe Rosenthal

Google says the Pixel 9 won’t be an unfettered nonsense factory, but it’s brief on specifics. “We design our Generative AI tools to respect user intent, and that means they can create content that could be offensive when a user asks them to,” said Alex Moriconi, Google’s communications manager. Edge in an email. “That said, not everything is allowed. We have clear policies and Terms of Service about what is and isn’t allowed, and we build in safeguards to prevent abuse. At times, some prompts can challenge the security of these tools, and we remain committed to continually improving and refining the safeguards we have in place.”

This rules are what you’d expect—for example, you can’t exploit Google services to facilitate crime or incite violence. Some attempts to trigger the prompts have returned a generic error message: “Magic Editor can’t complete this edit. Try something else.” (But there are a few troubling prompts throughout this story that did work.) But when push comes to shove, standard content moderation can’t save photography from its fledgling decline as a signal of truth.

For a brief time we lived in an era where photography was a shortcut to reality, to know thingsto have irrefutable proof. It was an incredibly useful tool for navigating the world around us. Now we’re diving headfirst into a future where reality is simply less knowable. The Lost Library of Alexandria would fit on the microSD card in my Nintendo Switch, and yet the pinnacle of technology is a cell phone that spits out lies as a fun little bonus.

We are fucking.