Apple Upgrades Siri With Screen-Aware AI

(MENAFN- The Arabian Post) Apple is preparing to equip the iPhone with a more visually intelligent assistant as it integrates a new artificial intelligence model, known as Ferret-UI, into the software foundation of iOS 18. The development marks a shift in how Siri may interpret and respond to what appears on a user's screen, enabling more precise app navigation, icon recognition and contextual understanding.

The Ferret-UI model, introduced through Apple's machine learning research division, is designed to comprehend graphical user interfaces rather than relying solely on text prompts. By training the system on large volumes of annotated screenshots, engineers have taught it to recognise buttons, icons, menus and layout patterns within mobile applications. That capability could allow Siri to move beyond voice queries and execute actions based on visual cues, such as tapping specific on-screen elements or extracting information from within apps.

Apple has faced mounting competition in generative AI from companies including Google and OpenAI, prompting a broader overhaul of its software strategy. At its Worldwide Developers Conference in 2024, the company unveiled a suite of AI features under the banner of“Apple Intelligence,” positioning them as privacy-centric tools embedded directly into iOS, iPadOS and macOS. Ferret-UI aligns with that direction by focusing on on-device understanding of interfaces rather than cloud-dependent responses.

Researchers behind the model reported that Ferret-UI demonstrated strong performance across benchmark tests involving user-interface tasks. These include identifying small icons within complex layouts, distinguishing between visually similar elements and following multi-step navigation instructions. In comparative evaluations against other multimodal AI systems, Apple's approach showed gains in accurately grounding language instructions in visual elements on mobile screens.

See also Rubrik expands sovereign cloud push amid data control demands

For users, the practical implications could be significant. A visually aware Siri may interpret commands such as“open the settings for this app” or“find the send button on this screen” with greater accuracy. The assistant could also assist users with limited vision by describing interface components in more detail or guiding them step by step through unfamiliar apps. Accessibility advocates have long called for smarter screen interpretation tools that bridge the gap between voice control and graphical interfaces.

Apple has historically emphasised accessibility features across its devices, including VoiceOver and Live Text. Integrating Ferret-UI into Siri could deepen those efforts, particularly for users who rely on spoken navigation. By understanding where elements appear and how they relate to each other, the assistant might help automate repetitive tasks such as booking appointments, filling out forms or adjusting in-app settings without manual taps.

The technological foundation of Ferret-UI reflects a broader industry shift towards multimodal AI, which combines language processing with visual comprehension. Companies such as Google have advanced models capable of analysing images and video, while OpenAI's systems can interpret screenshots and graphical content. Apple's research suggests that specialised training on mobile interfaces yields better performance in user-interface scenarios than general-purpose vision models.

Industry analysts view the move as a strategic response to perceptions that Siri has lagged behind rival assistants in contextual understanding. While Apple has steadily improved natural language processing over the years, critics have argued that Siri struggled with complex or layered commands. Embedding a UI-focused model may narrow that gap by allowing the assistant to see and reason about the same screen elements visible to the user.

See also Reddit emerges as a destination for answer-led searches

Privacy considerations remain central to Apple's positioning. The company has stated that much of its AI processing will occur on device, reducing the need to transmit sensitive data to remote servers. Ferret-UI's design supports that approach, as screenshot interpretation can be handled locally using Apple's custom silicon, including the neural engines integrated into A-series and M-series chips.

Developers may also benefit from the technology. If integrated into application programming interfaces, Ferret-UI could enable third-party apps to expose structured interface metadata that Siri can interpret more effectively. That, in turn, might encourage more seamless app automation and cross-app workflows within iOS 18. For example, a user could instruct Siri to compare prices between two shopping apps displayed on screen or extract booking details from a travel application without manual copying.

The timing of the enhancement comes amid growing expectations that smartphone platforms will become more proactive and contextually aware. Generative AI has reshaped user expectations around chatbots and assistants, creating pressure on device makers to deliver more sophisticated capabilities. Apple's approach differs in that it integrates AI directly into system-level interactions rather than presenting it solely as a conversational interface.

Notice an issue? Arabian Post strives to deliver the most accurate and reliable information to its readers. If you believe you have identified an error or inconsistency in this article, please don't hesitate to contact our editorial team at editor[at]thearabianpost[dot]com. We are committed to promptly addressing any concerns and ensuring the highest level of journalistic integrity.

MENAFN22022026000152002308ID1110773811