Google Lens Adds New Feature to Answer Questions About Videos

Google is upgrading its visual search app, Lens, with a new feature that allows users to ask questions about objects in videos in near-real-time. English-speaking Android and iOS users who have the Google app installed can now capture videos via Lens and receive answers about the objects of interest within those videos.

AI-Powered by Google's Gemini Model

The new feature uses Google’s Gemini AI model, which powers various products across Google’s portfolio. Lou Wang, director of product management for Lens, explained that the Gemini model helps analyze the video and respond to user questions. For instance, if you record a video of fish swimming, Lens can explain why they’re swimming in a circle and provide additional resources for learning more.

How to Access Lens' Video Analysis

To access the new video analysis feature, users must be enrolled in Google’s Search Labs program and opt in to the experimental “AI Overviews and more” feature. Once activated, holding down the shutter button in the Google app triggers Lens’ video capture mode, allowing users to ask questions as they record.

Lens then uses AI Overviews, a feature in Google Search, to summarize web-sourced information and provide relevant answers based on the frames in the video. The system identifies the most important and relevant frames to "ground" its AI-generated responses.

Inspired by User Curiosity

According to Wang, the development of the video feature was motivated by observing how people are already trying to use Lens. "If you lower the barrier of asking these questions and helping people satisfy their curiosity, people are going to pick this up pretty naturally," Wang said during a press briefing.

Racing to Lead AI Video Analysis

Google’s addition of video search to Lens arrives just ahead of similar features being developed by other tech giants. Meta recently previewed real-time AI video capabilities for its Ray-Ban Meta AR glasses, and OpenAI has teased a tool in Advanced Voice Mode that will eventually be able to analyze videos in real time. However, Google has launched this feature first, though Lens’ functionality remains asynchronous rather than real-time.

Additional Features for Image and Product Search

Beyond video analysis, Lens has introduced additional features for both image and e-commerce searches. Users can now search with images and text simultaneously, allowing them to take a photo and ask questions about it in one go. This feature is available to all English-speaking users, including those not enrolled in Labs.

For e-commerce, Lens will now recognize products and display key information, such as price, brand, reviews, stock, and deals. This functionality works on both uploaded and newly snapped photos, though it is currently limited to certain categories like electronics, toys, and beauty in select countries.

Monetizing the Lens Shopping Experience

Given that around 4 billion Lens searches per month are related to shopping, Google has introduced relevant shopping ads on the results page for identified products. These ads will display options and prices, capitalizing on the immense volume of shopping-related queries made through Lens.

What This Means for Users and the Future of Search

Google Lens’ new video analysis feature marks a significant advancement in how users can interact with their surroundings and get immediate insights. By leveraging AI to answer questions in near real-time, Google is positioning itself at the forefront of visual search technology. This tool has the potential to revolutionize the way people gather information, making the search process more interactive and intuitive. With similar features on the horizon from competitors like Meta and OpenAI, the race to integrate AI-powered video capabilities into everyday tools is heating up, promising more user-friendly and intelligent ways to explore and understand the world.