<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Temirlan J. on Medium]]></title>
        <description><![CDATA[Stories by Temirlan J. on Medium]]></description>
        <link>https://medium.com/@zhekenov?source=rss-2821e90d6937------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lnqER9trUmavQv4_</url>
            <title>Stories by Temirlan J. on Medium</title>
            <link>https://medium.com/@zhekenov?source=rss-2821e90d6937------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:22:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zhekenov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unlocking the Future of Mobile UI Understanding: A Deep Dive into Ferret-UI and UI-JEPA]]></title>
            <link>https://zhekenov.medium.com/unlocking-the-future-of-mobile-ui-understanding-a-deep-dive-into-ferret-ui-and-ui-jepa-f0fb8df92d84?source=rss-2821e90d6937------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0fb8df92d84</guid>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ui-jepa]]></category>
            <category><![CDATA[ferretui]]></category>
            <dc:creator><![CDATA[Temirlan J.]]></dc:creator>
            <pubDate>Sat, 14 Sep 2024 21:21:48 GMT</pubDate>
            <atom:updated>2024-09-14T21:21:48.324Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Temirlan J.</em></p><p><strong>Imagine a world where your smartphone doesn’t just respond to your taps and swipes but truly understands your intentions, anticipates your needs, and interacts with you as seamlessly as a human assistant would.</strong> This isn’t a scene from a futuristic movie; it’s the exciting reality being shaped by cutting-edge research in mobile user interface (UI) understanding. Two groundbreaking papers — <strong>Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs</strong> and <strong>UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity</strong> — are pushing the boundaries of how machines comprehend and interact with mobile UIs.</p><p>In this deep dive, we’ll explore the innovations these papers bring to the table, analyze their approaches, and discuss the implications for the future of AI-driven UI understanding.</p><h3>The Challenge: Making Sense of Complex Mobile UIs</h3><p>Mobile applications have become extensions of ourselves, handling everything from banking to social connections. However, the complexity and variability of mobile UIs present significant challenges for machines:</p><ul><li><strong>Diverse Visual Elements</strong>: Mobile screens are a mosaic of icons, widgets, text, and interactive elements that vary widely across apps.</li><li><strong>Dynamic Changes</strong>: UIs can change based on user actions, making it hard to predict and understand context.</li><li><strong>Small Objects of Interest</strong>: Important elements like icons or buttons are often small and densely packed, making them difficult to detect and interpret.</li></ul><p>Traditional models struggle to process and understand these intricate interfaces effectively. This is where multimodal large language models (MLLMs) come into play, integrating visual and textual data to enhance comprehension.</p><h3>Ferret-UI: A Specialized MLLM for Mobile UIs</h3><p><strong>Ferret-UI</strong> is designed to bridge the gap between general MLLMs and the specific needs of mobile UI understanding. Built upon the existing <strong>Ferret</strong> model, which excels in referring and grounding tasks within natural images, Ferret-UI adapts these capabilities for mobile screens.</p><h3>Key Innovations</h3><ol><li><strong>Any-Resolution (AnyRes) Capability</strong>: Recognizing that mobile screens have elongated aspect ratios and contain small objects, Ferret-UI introduces an “any-resolution” approach. Screens are divided into sub-images based on their orientation — horizontal division for portrait screens and vertical for landscape. This magnifies details and allows the model to focus on smaller elements without losing the broader context.</li><li><strong>Enhanced Referring and Grounding</strong>: The model can interpret and interact with specific regions or objects within the UI, whether specified as points, boxes, or free-form shapes. This is crucial for tasks like icon recognition, text identification, and widget classification.</li><li><strong>Comprehensive Training Data</strong>: The researchers curated training samples from a wide range of elementary UI tasks, such as icon recognition, find text, and widget listing. They also included advanced tasks like detailed descriptions, perception conversations, interaction conversations, and function inference.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_vqo3MsoyXH5MDVtikZCkg.png" /><figcaption>Ferret-UI is able to perform referring tasks (e.g., widget classification, icon<br>recognition, OCR) with flexible input formats (point, box, scribble) and grounding<br>tasks (e.g., find widget, find icon, find text, widget listing) on mobile UI screens.</figcaption></figure><h3>How It Works</h3><p>Ferret-UI uses a pre-trained image encoder (like CLIP-ViT-L/14) to process both the full screen and its sub-images. These visual features are then combined with text embeddings and fed into a language model that generates responses based on both visual and textual cues. The architecture allows for precise region annotations and enables the model to execute open-ended instructions effectively.</p><p>According to You et al, 2024, a pre-trained image encoder and projection layer produce image features for the entire screen. For each sub-image obtained based on the original image aspect ratio, additional image features are generated. For text with regional references, a visual sampler generates a corresponding regional continuous feature. The LLM uses<br>the full-image representation, sub-image representations, regional features, and text embeddings to generate a response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*wbI19tQr1tvrvXiAmm6Clw.png" /><figcaption>Overview of Ferret-UI-anyres architecture.</figcaption></figure><h4>Architecture Overview</h4><p><strong>1. Building Upon Ferret’s Foundation</strong></p><p>Ferret-UI is an extension of the original Ferret model, which is known for its strong referring and grounding capabilities in natural images. Ferret integrates a pre-trained visual encoder (like CLIP-ViT-L/14) with a decoder-only language model (such as Vicuna). The key innovation in Ferret-UI is adapting this architecture to handle the unique challenges of mobile UIs.</p><p>Mobile UI screens come with elongated aspect ratios and contain numerous small interactive elements. To address this, Ferret-UI incorporates an “any-resolution” (AnyRes) strategy:</p><ul><li><strong>Sub-Image Division</strong>: Screens are divided into sub-images based on their aspect ratios. Portrait screens are split horizontally, while landscape screens are split vertically.</li><li><strong>Separate Encoding</strong>: Each sub-image is independently encoded using the same pre-trained image encoder. This ensures that fine-grained details of UI elements are captured effectively.</li><li><strong>Enhanced Visual Features</strong>: By processing both the global image and its sub-images, the model retains the context of the entire screen while focusing on detailed regions.</li></ul><p>Ferret-UI leverages a hybrid representation that combines continuous visual features with spatial coordinates. This allows the model to handle inputs like points, bounding boxes, and free-form scribbles, enabling precise referring and grounding.</p><h4>Training Pipeline</h4><p>The training data for Ferret-UI is curated to cover a wide range of tasks:</p><ul><li><strong>Elementary Tasks</strong>: Include referring tasks (e.g., icon recognition, OCR) and grounding tasks (e.g., find widget, widget listing).</li><li><strong>Advanced Tasks</strong>: Involve detailed descriptions, perception conversations, interaction conversations, and function inference.</li></ul><p>All training samples are formatted in an instruction-following manner. This aligns with the model’s ability to process open-ended instructions and generate coherent responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z2C01a_f05bUDk1dxK3DlQ.png" /><figcaption>Elementary task data generation overview. A UI detector outputs all<br>detected elements, with each element’s type, text, and bounding boxes. These detections<br>are used to create training samples for elementary tasks</figcaption></figure><p><strong>Training Procedure</strong></p><ul><li><strong>Vision Encoder Freezing</strong>: The pre-trained visual encoder’s weights are kept frozen to retain its general visual understanding capabilities.</li><li><strong>Updating the Decoder and Projection Layer</strong>: Only the decoder (language model) and the projection layer that maps visual features to the language model’s input space are updated during training.</li><li><strong>Loss Function</strong>: The model is trained using a language modeling objective, focusing on generating accurate and contextually appropriate textual responses based on the visual inputs.</li></ul><p><strong>4. AnyRes Training Considerations</strong></p><ul><li><strong>Enhanced Detail Capture</strong>: By integrating AnyRes, the model is trained to handle inputs of varying resolutions and aspect ratios, improving its ability to recognize small UI elements.</li><li><strong>Performance Optimization</strong>: Training with AnyRes requires careful handling to ensure that the additional sub-image encodings do not introduce computational bottlenecks.</li></ul><p>In evaluations, Ferret-UI demonstrated:</p><ul><li><strong>Superior Performance</strong>: It outperformed existing models, including GPT-4V, on elementary UI tasks.</li><li><strong>Advanced Understanding</strong>: Showed proficiency in handling advanced tasks that require reasoning and open-ended interactions.</li><li><strong>Transferability</strong>: Exhibited strong capabilities across different operating systems, working well with both iPhone and Android screens.</li></ul><p>The implications are significant for areas like accessibility, automated testing, and personalized user experiences. By enabling machines to understand and interact with mobile UIs more effectively, Ferret-UI paves the way for more intelligent and responsive applications.</p><h3>UI-JEPA: Lightweight, On-Device User Intent Understanding</h3><p>While models like Ferret-UI push the boundaries of UI understanding, they often require significant computational resources, making them impractical for on-device applications. <strong>UI-JEPA</strong> addresses this by offering a lightweight framework capable of understanding user intent from sequences of UI actions, all while running efficiently on mobile devices.</p><h3>Key Innovations</h3><ol><li><strong>Self-Supervised Learning with Masking Strategies</strong>: UI-JEPA employs Joint Embedding Predictive Architectures (JEPA) to learn abstract UI embeddings from unlabeled data. By masking parts of the input — both spatially and temporally — the model learns to predict missing information, effectively capturing temporal relationships and the abstract meaning within UI sequences.</li><li><strong>Integration with a Fine-Tuned LLM Decoder</strong>: The framework combines the strengths of self-supervised learning with the language generation capabilities of a fine-tuned language model. This allows UI-JEPA to generate coherent and accurate descriptions of user intent based on UI interactions.</li><li><strong>New UI-Grounded Multimodal Datasets</strong>: Recognizing the lack of high-quality datasets, the researchers introduced two new datasets — <strong>Intent in the Wild (IIW)</strong> and <strong>Intent in the Tame (IIT)</strong> — designed for few-shot and zero-shot UI understanding tasks.</li></ol><h3>How It Works</h3><p>UI-JEPA consists of two main components:</p><ul><li><strong>Video Encoder (ViT)</strong>: Processes sequences of UI actions, capturing both spatial and temporal dynamics.</li><li><strong>Language Model Decoder</strong>: Generates textual descriptions of user intent from the video embeddings.</li></ul><h4>Architecture Overview</h4><p>UI-JEPA leverages the JEPA framework to learn abstract representations from unlabeled data. The core idea is to predict representations of masked portions of the input based on the unmasked context, enabling the model to capture high-level semantic information.</p><p><strong>Video Encoder: Vision Transformer (ViT)</strong></p><ul><li><strong>Model Configuration</strong>: UI-JEPA uses a ViT-based video encoder tailored for processing UI action sequences. Specifically, it adopts a ViT architecture pre-trained on large-scale video datasets, such as Kinetics-400 or Kinetics-600, to leverage learned temporal and spatial features.</li></ul><p>The training process involves two stages:</p><ol><li><strong>JEPA Tuning</strong>: The video encoder is fine-tuned using self-supervised learning, with various masking strategies to learn from unlabeled data.</li><li><strong>Fine-Tuning the LLM Decoder</strong>: A small amount of labeled data is used to fine-tune the language model, enabling it to generate accurate intent descriptions.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Fdc3hiAFsdfeqmIA5Vl6Q.png" /><figcaption>Training and Inference Processes of UI-JEPA:</figcaption></figure><h3>Results and Implications</h3><p>UI-JEPA’s performance is impressive:</p><ul><li><strong>Efficient and Lightweight</strong>: Achieves performance comparable to state-of-the-art MLLMs but with significantly reduced computational costs — up to 50.5x less — and a 6.6x improvement in latency.</li><li><strong>On-Device Deployment</strong>: Suitable for scenarios requiring low latency and heightened privacy, as it can run efficiently on mobile devices.</li><li><strong>Enhanced UI Understanding</strong>: Demonstrates strong capabilities in predicting user intent from UI action sequences.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IdHdbZD0B0mD6S17kTY9aA.png" /><figcaption>Examples of inputs and corresponding labels from the IIWand IIT datasets</figcaption></figure><p>This has profound implications for applications like digital assistants, which can become more responsive and context-aware, and for privacy-conscious users who prefer on-device processing.</p><h3>Approaches to UI Understanding</h3><ul><li><strong>Ferret-UI</strong> focuses on enhancing the model architecture to handle the unique challenges of mobile UIs, such as elongated aspect ratios and small objects. It emphasizes precise referring and grounding capabilities, enabling it to interact with specific UI elements effectively.</li><li><strong>UI-JEPA</strong> takes a different approach by leveraging self-supervised learning to reduce reliance on large datasets and heavy computational resources. It focuses on understanding user intent through sequences of UI actions and aims for efficient, on-device deployment.</li></ul><h3>Strengths and Limitations</h3><p><strong>Ferret-UI Strengths</strong>:</p><ul><li>Exceptional performance on both elementary and advanced UI tasks.</li><li>Strong grounding and referring capabilities.</li><li>Broad applicability across different operating systems.</li></ul><p><strong>Ferret-UI Limitations</strong>:</p><ul><li>May require significant computational resources, making on-device deployment challenging.</li><li>Relies heavily on UI detection models, which can be a bottleneck.</li></ul><p><strong>UI-JEPA Strengths</strong>:</p><ul><li>Lightweight and efficient, suitable for on-device use.</li><li>Effective in scenarios where privacy and low latency are crucial.</li><li>Reduces the need for large amounts of labeled data.</li></ul><p><strong>UI-JEPA Limitations</strong>:</p><ul><li>May struggle with zero-shot generalization to unfamiliar apps.</li><li>Granularity of user intent prediction can be limited, particularly in tasks requiring detailed text recognition.</li></ul><h3>Potential Synergies</h3><p>Combining the strengths of both models could lead to even more powerful UI understanding systems. For instance:</p><ul><li><strong>Hybrid Models</strong>: Integrate Ferret-UI’s precise grounding capabilities with UI-JEPA’s efficient, self-supervised learning approach to create models that are both accurate and lightweight.</li><li><strong>Enhanced Datasets</strong>: Utilize the datasets introduced by UI-JEPA to further train and refine Ferret-UI, potentially improving its performance and reducing its reliance on computational resources.</li></ul><h3>The Road Ahead: Implications for AI and Mobile UX</h3><p>The advancements presented by Ferret-UI and UI-JEPA signal a transformative shift in how machines interact with mobile UIs.</p><ul><li><strong>Improved Accessibility</strong>: Enhanced UI understanding can make apps more accessible to users with disabilities by providing better support for assistive technologies.</li><li><strong>Personalized Experiences</strong>: Apps can become more intuitive and responsive, adapting to individual user behaviors and preferences.</li><li><strong>Privacy-Preserving AI</strong>: Models like UI-JEPA demonstrate that it’s possible to achieve high-performance AI on-device, reducing the need to send user data to the cloud and enhancing privacy.</li><li><strong>Efficient Development</strong>: Developers can leverage these models to automate testing and UI analysis, saving time and resources.</li></ul><h3>Era of Intelligent Interaction</h3><p>The future of mobile interaction lies in machines that can understand and anticipate our needs, providing seamless and intuitive experiences. <strong>Ferret-UI</strong> and <strong>UI-JEPA</strong> represent significant strides toward this future, each tackling the challenge from unique angles.</p><p><strong>Ferret-UI</strong> pushes the boundaries of what MLLMs can achieve in terms of precise UI understanding, while <strong>UI-JEPA</strong> offers a practical, efficient solution for on-device intent prediction. Both models have their strengths and limitations, but together they paint a promising picture of what’s possible.</p><p>As researchers continue to refine these models and explore new approaches, we can look forward to a new era where our devices don’t just respond to us — they understand us.</p><h3>References</h3><ul><li>Fu, Yicheng, et al. “UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity.” <em>arXiv preprint arXiv:2409.04081</em> (2024).</li><li>You, Keen, et al. “Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs.” <em>arXiv preprint arXiv:2404.05719</em> (2024).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0fb8df92d84" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the Future of AI, Technology, and Innovation]]></title>
            <link>https://zhekenov.medium.com/test-1-b6639f89f4b5?source=rss-2821e90d6937------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6639f89f4b5</guid>
            <dc:creator><![CDATA[Temirlan J.]]></dc:creator>
            <pubDate>Tue, 10 Sep 2024 22:52:41 GMT</pubDate>
            <atom:updated>2024-09-14T23:23:54.364Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello! I’m Temirlan, an AI engineer with a passion for exploring the cutting-edge developments in artificial intelligence, machine learning, and how these technologies are shaping the future.</p><p>This blog will be my platform to dive into the innovations and breakthroughs that excite me the most, and I hope to make complex topics accessible and engaging.</p><p>What you can expect:</p><ul><li><strong>AI in Action:</strong> I’ll be writing about real-world applications of AI, from computer vision to natural language processing, and how these technologies are transforming industries.</li><li><strong>Research Deep Dives:</strong> Expect insights into the latest AI research papers and technologies, such as multimodal models and advancements in mobile UI understanding.</li><li><strong>Tech and Career Tips:</strong> Whether you’re looking to break into AI or sharpen your skills, I’ll share tips, resources, and personal experiences from my journey.</li><li><strong>Project Insights:</strong> I’ll take you behind the scenes of my AI projects, sharing lessons learned, tools used, and challenges overcome.</li></ul><p>Join me as we explore the exciting possibilities that AI brings to our world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6639f89f4b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>