New Live AI and Meta’s Ray-Ban Translation, Hands-on: Signs of AR Glasses to Come

I activated Meta Ray-Ban new live AI feature and took a morning walk through Manhattan. It was a strange experience. A white LED in the corner of my eye remained on as my glasses continued to report on my life. I asked questions awkwardly: about the pigeons, about the construction workers, about whether he knew what car was nearby or who owned those trucks across the street. I received mixed responses, sometimes no response at all. And then my connection ended due to bad Bluetooth in the city.

My first steps with an ever-sentient AI companion have been strange and even more sci-fi than what I’ve experienced over the past year. Very similar to a recent demo with Google always-on glasses with Gemini technology, Ray-Ban by Meta (which are now available) are taking the next steps toward becoming something of an ever-attentive assistant. EITHER agentas the AI ​​landscape now calls it. Live AI and live translation; once turned on, stay there. AI is supposed to see what you see. And maybe it will help you do something you don’t know how to do.

But these features also seem like previews of what could be a whole new set of Meta glasses coming next year, ones that could have their own screen and maybe even a gesture control bracelet, based on hints given by Mark Zuckerberg. in threads last week after a written story by Joanna Stern of The Wall Street Journal.

At the moment, Live AI feels like a strange glimpse into a more always-on, more intrusive AI future, which is more of a companion than a help in my early attempts. And yet the translation, when it works, is surprisingly useful… even if it works with a bit of a delay.

Live AI: a state of mind of constant listening and observation

Activating Live AI means starting live video recording. Although the video is not saved for you to watch later, Meta’s AI processes it through your phone and transmits it to the glasses. The LED light stays on to notify people that it is on, but in my experience, people don’t notice the LED light much or don’t seem to care. Everything you say can be interpreted by Meta AI, so forget about conversations with others. At the office, I seemed like a weird guy talking to myself or perhaps seemingly talking to others (only for people to try to talk to me and realize I wasn’t talking to them). But Live AI can be paused by tapping the side of the glasses.

You can end Live AI by saying “Stop Live AI”, but sometimes Meta AI thought you were asking if was a live AI: a “Who starts first?” moment. I had to scream several times before it stopped.

The challenge with live AI is figuring out what to do with it. I walked around the office asking about furniture placement and was told that everything seemed fine: “the room appears to be well designed and functional, with no obvious changes needed.” I asked him about a story I was writing on my laptop and he said, “The text seems to be a cohesive, well-structured piece, with no parts that seem unnecessary.” I kept trying to get constructive feedback and it was difficult to get anything that wasn’t generic, even if it pointed out some notable lines and summarized my points.

As he walked out, he told me what street he was on, but it was wrong – I corrected him and then he just acknowledged it and moved on. He knew the Chase bank I was looking at and told me the bank’s hours, and he knew Joe’s Pub when I stood at the entrance to the Public Theater, but he couldn’t tell me what they were playing that night. It could recognize common pigeons, mistakenly recognized a car on the sidewalk as a Mercedes (it was a Lincoln), and recommended, for some reason, a bar on the street that was now, according to Meta AI, “defunct.”

Live AI is largely an early access beta at the moment, but I also need to understand what I’ll do with it. The early beta feel and unclear purpose can combine to make it seem ridiculous. Or unexpectedly deep. Either way, keeping it running takes a toll on battery life: 30 minutes of use, rather than the hours Ray-Bans typically run.

Translation: Useful, for some languages.

Live translation works the same way, starting on demand. But you need to download language packs for the specific languages ​​you want to translate: Spanish to English, for example. Currently only Spanish, French, Italian and English are supported, which is a disappointment.

I chatted with a CNET colleague danni santana in noisy Astor Place, near our New York office. He spoke in Dominican Spanish and I spoke in English. The translated answers appeared in my ears a few seconds later, and during our conversation, I felt like I was picking up enough to understand. It wasn’t perfect: the translation AI didn’t seem to understand some phrases or idioms. The delay made it difficult to know when the translation would be finished or if there was more to come. I had trouble judging the timing of my responses to Danni as he waited patiently for me to speak across the table.

Meta also displays a live transcript of the conversation in the Meta View phone app, which you can refer to while wearing the glasses to show the person you’re talking to or clarify what was said.

Ray-Ban’s translation feature seems much more instantly useful than Live AI, but that’s also because Live AI still doesn’t make it clear what you should use it for. Maybe I could turn it on while I’m cooking, building IKEA furniture, or playing a board game? I don’t know. Help me figure this out, Meta. Also, not having any head-up display makes Live AI look like I’m guessing what the glasses are looking at.

Of course, you can use Google Translate on your phone. Meta is using his glasses to translate similar to how you would use a pair of headphones. But Meta’s glasses can also see and translate written elements, but that is not part of the live conversational translation mode.

What’s next: screen or gestures? Or both?

The one-year-old Ray-Bans from Meta have now obtained multiple important AI functionseach changing the equation in surprising ways. However, the latest live AI additions appear to be pushing the limits of the hardware, reducing battery life. I wish I had better ways to know what the AI ​​can see or what it can point with my hand to indicate what I want to ask.

The glasses of the future could move in this direction: both with front screens and gesture recognition. Meta CTO Andrew Bosworth, in a conversation I had with him at the end of the year, acknowledges that these are the next steps, but the timeline is unclear. Orion Goggles (an ambitious pair of future-proof glasses with 3D displays and a wrist-based gesture tracker that I tested earlier this year that can recognize taps and pinches with your fingers) are still years away from being real. But Meta’s wrist-worn neural band could emerge sooner, or perhaps a way for camera-equipped glasses to recognize hand gestures. And as for smart glasses screens, Meta could explore a smaller head-up display to display information before moving to larger, more immersive AR screens. Bosworth points out next-generation AR glasses in a recent blog postBut will any of this be possible in the next generation of Ray-Ban glasses next year?

“Gesture-based controls require downward-facing cameras and probably some lighting,” Bosworth says of the future Meta glasses. “You could do it on the current Ray-Ban Metas (in Live AI, we played with it), but you only have to do it (from) the camera’s field of view.” However, he recognizes the possibilities of adding an EMG band to the glasses sooner rather than later. “Now you’re adding a device that needs to be charged, it’s an extra cost, it’s an extra weight, but it’s very convenient.” But Bosworth believes the EMG band is useful only when there is a screen on the glasses, something Ray-Bans don’t have…yet. Chances are, when Ray-Bans get some sort of display screen, an entrance band will debut alongside them. I have seen some attempts ideas like this in other products.

And then there’s the question of battery life: how will these always-on glasses work for more than a few hours at a time? Or how would all this increase the cost of a next-generation pair of glasses?

Meanwhile, Meta’s AI could also be extended to areas like fitness, as something that also serves as a bridge to virtual reality, where Meta has another version of Meta AI. “It would be very unusual if, a year from now, the AI ​​you are using to track your steps in the world and give you advice was not aware that you are also doing these training (in virtual reality)” says Bosworth.

As Live AI continues to evolve, it may be absolutely necessary to have better ways to add gestures. Bosworth sees pointing things out as a key way to train AI to improve in the future. “As AIs improve, the need for these much simpler and more intuitive gestures increases significantly.”

Meta’s Ray-Bans don’t let me point things out right now, and it makes Live AI look a little confusing to use at times. But maybe that requires newer hardware and additional gestures and displays to make the next leap.