/cdn.vox-cdn.com/uploads/chorus_asset/file/25809036/IMG_9569E5A1FC48_1.jpeg?w=696&resize=696,0&ssl=1 "Does your iPhone share Apple Photos by default?")
/cdn.vox-cdn.com/uploads/chorus_asset/file/25809036/IMG_9569E5A1FC48_1.jpeg?w=1200&resize=1200,0&ssl=1&description=Does+your+iPhone+share+Apple+Photos+by+default%3F){kind=link}
Sure enough, when I checked my iPhone 15 Pro this morning, the switch was on. You can find this by going to Settings > Photos (or System Settings > Photos on a Mac). Enhanced Visual Search lets you search for landmarks you’ve taken photos of, or search for those images using the names of those landmarks.
To see what the Photos app can do, swipe up on the photo you took of a building and select “Search for a landmark,” and a card will appear that will perfectly identify it. Here are some examples from my phone:
On the surface, it’s a handy extension of the Photos app’s Visual Look Up feature, introduced by Apple in iOS 15, which lets you identify plants or, say, figure out what they are symbols on the washing label. But Visual Look Up doesn’t need special permission to share data with Apple, and it does.
The description below the toggle says you’re giving Apple permission to “privately match locations in your photos to a global index maintained by Apple.” As for how, details are in the file Apple Machine Learning Research Blog about the improved visual search Johnson linked to:
The process begins with an on-device ML model that analyzes a given photo to determine whether there is a “region of interest” (ROI) that may contain a landmark. If the model detects an ROI in the “landmark” domain, a vector embedding is computed for that area of the image.
According to the blog, this vector embedding is then encrypted and sent to Apple for comparison to its database. The company offers a very technical explanation of vector embedding in research articleBut IBM put it more simplywriting that an embedding transforms “a data point, such as a word, sentence, or image, into a file N-dimensional array of numbers representing the characteristics of this data point”
Like Johnson, I don’t fully understand Apple’s research blogs, and Apple didn’t immediately respond to our request for comment on Johnson’s concerns. The company appears to have gone to great lengths to maintain data privacy, in part by condensing image data into a format readable by the ML model.
Still, including a toggle option to, say, share analytics, recordings, or interact with Siri, rather than something users have to discover, seems like a better option.