Welcome to the second instalment in our series exploring multimodal AI work at GovTech Singapore’s AI Practice. In this article, we delve into a common challenge faced by organisations across various sectors in their digital transformation journeys: the efficient cataloguing and management of vast collections of digitised physical assets.

The Digital Dilemma

The digitisation of physical assets — such as unique objects and inventories — promises enhanced searchability and analysis at scale. However, this laudable intention often collides with a daunting reality. As the volume of digitised assets grows, the task of cataloguing becomes increasingly onerous, consuming significant time and labour resources.

Consider this: In a traditional setting, cataloguing a single complex item might take an expert 15 minutes. For a collection of 100,000 items, that’s 25,000 hours of work — equivalent to over 12 years of full-time effort for a single cataloguer. The scale of this challenge is staggering.

Many organisations find themselves trapped in a cycle of “digitise now, catalogue later,” where assets are photographed and tagged with only basic metadata such as titles, postponing more detailed cataloguing for an indefinite future. This approach, while expedient, ultimately hampers the very accessibility and analysability that digitisation aimed to achieve.

A Common Challenge, A Shared Solution

This predicament is not unique to any single organisation or sector. It’s a shared challenge faced by institutions ranging from museums and libraries to government agencies and corporate archives. Recognising this common ground, we present our recent proof-of-concept project exploring an AI-assisted approach to cataloguing.

The potential impact is transformative: An AI-assisted approach could reduce the cataloguing time per item to ideally just 3 minutes, including human verification. For that same collection of 100,000 items, this could slash the total cataloguing time to just 5000 hours — a reduction of up to 80%. This means a project that might have taken years could now be completed in months.

Our goal is twofold:

1. To address the immediate cataloguing challenges presented to us
2. To provide a blueprint that other organisations can adapt and learn from in their own digital asset management efforts

While we’ve tailored our solution to a specific use case (details of which are omitted due to sensitivities), the underlying principles and methodologies are broadly applicable. We believe that by sharing our experiences and insights, we can contribute to the broader conversation on leveraging AI in digital transformation initiatives.

In the following sections, we’ll dive into the details of our approach, discussing both the successes and the hurdles we encountered along the way. Whether you’re grappling with similar cataloguing challenges or simply curious about the potential of AI in asset management, we hope you’ll find valuable takeaways from our journey.

The Challenge: Efficient Cataloguing of Diverse Assets

Our project focused on a vast repository of digitised physical assets, each representing a unique piece of information crucial for educational, organisational, and research purposes. The collection comprises diverse items, each an image-metadata pair. While efficient cataloguing is key for effective search and retrieval, only a small percentage had been manually catalogued due to the sheer volume.

In this context, “uncatalogued” refers to asset records lacking metadata tags for specific fields in the cataloguing framework. This creates challenges for staff needing to search and study the collection efficiently, and affects external users like researchers who face limitations in searchability.

Our Approach: Multimodal AI-Assisted Cataloguing

To address these challenges, we developed an AI-assisted cataloguing process aimed at:

1. Accelerating the cataloguing timeline
2. Reducing manual workload on cataloguers
3. Improving accuracy and consistency of asset descriptions

Our solution leverages two cutting-edge AI techniques:

1. Multimodal Retrieval-Augmented Tagging: Generates accurate tags by leveraging existing knowledge about similar assets.
2. Multimodal Retrieval-Augmented Physical Description Generation: Creates detailed, context-aware descriptions drawing on information from similar assets.

Breaking Down the Challenge

We focused on two main objectives:

1. Tagging: Addressing three key attributes:

• Object Types: A hierarchical taxonomy with four levels of specificity and controlled vocabulary
  - Level 1 (Most General): e.g., “Documents & Records”
  - Level 2: e.g., “reports”
  - Level 3: e.g., “financial reports”
  - Level 4 (Most Narrow): e.g., “quarterly financial statements”
• Colours
• Shapes: Both general and object-specific descriptors

2. Description Generation: Creating detailed physical descriptions focusing on visual appearance and content. The requirements for this are nuanced, with a need for a specific style, format and tone that goes beyond what a simple “describe this image” prompt can achieve.

Our approach embraces a human-in-the-loop system, where AI generates initial tags and descriptions, and human cataloguers verify and refine the AI-generated content.

To API or Not to API, That is The Question

In developing our AI-assisted cataloguing system, we faced a critical decision: should we leverage commercial API models or build a custom pipeline using self-hosted models? Embracing innovation, we decided to explore both paths.

Why the dual approach?

1. Performance Benchmarking: We aimed to compare the capabilities of large-scale commercial API models against more compact, open-source alternatives. This comparison would provide valuable insights into the trade-offs between model size, performance, and resource requirements in the context of asset cataloguing.
2. Addressing Data Sensitivity: While our proof-of-concept involved public data, we recognised the need to prepare for future scenarios involving sensitive or classified information. Developing a self-hosted solution would allow us to maintain stricter control over data handling and processing.
3. Enhancing In-house Capabilities: This project presented an ideal opportunity to bolster our AI expertise. By developing a custom pipeline, we could deepen our understanding of multimodal AI systems and their application to real-world challenges.

Search & Retrieval

Given our access to a pre-existing catalogue of a selected subset of assets, we determined that a search and retrieval approach would be most effective for our AI-assisted cataloguing system. This method allows us to leverage existing knowledge to inform the cataloguing of new items.

The cornerstone of our approach is the use of multimodal embedding models. These sophisticated AI models can process both images and text, creating numerical representations (embeddings) that capture the essence of each asset. We experimented with three state-of-the-art models:

1. CLIP ViT-SO400M-14-SigLIP-384
2. Jina CLIP v1
3. Nomic Embed v1.5

Each of these models excels at generating embeddings that allow for efficient and meaningful comparisons between different assets, regardless of whether they’re images or text representations of the asset, such as names or descriptions. This capability is crucial for our task of finding similar items in our catalogue to assist with tagging and description generation. For a more in-depth explanation of how multimodal embedding models are used and what embeddings are, take a look at this article.

Data Hydration: Building Our Knowledge Base

To create a robust knowledge base, we needed an efficient way to store and retrieve embeddings. Our journey led us from starting with FAISS to ending with LanceDB, a serverless vector database well-suited for proof-of-concept work.

For each asset, we indexed four distinct types of data:

1. Image: The visual representation of the asset.
2. Name: Although not always descriptive, it provides valuable context.
3. Zero-shot Captions: Generated using a state-of-the-art open-source vision-language model.
4. Metadata: Partially available for some assets (e.g. missing fields)

Each data type was embedded and stored in separate tables within our vector database as seen in the figure below, creating a rich, multi-faceted representation of our asset collection.

Search & Retrieval: Connecting the Dots

For uncatalogued assets, typically only an image and name are available. Our search process involves:

1. Generating embeddings for the query asset’s image and name.
2. Performing cross-modal and same-modal retrieval across all four data tables.
3. Retrieving the k-most similar assets for each query.

This approach allows us to leverage the full spectrum of our knowledge base, drawing connections between visual and textual information to enhance our tagging and description generation tasks.

As we move forward, we’ll be exploring the practicality and value-addedness of further improving our retrieval process, experimenting with different k values and exploring similarity thresholds to optimise performance.

Retrieval-Augmented Tagging

With our retrieval system in place, we move on to the core task of tagging.

Self-Hosted Model-Based Retrieval-Augmented Tagging

Our self-hosted model-based retrieval-augmented tagging pipeline follows these steps:

1. Tag Embedding: We pre-embed all possible tags.

2. Similarity Calculation: We compute cosine similarities between:

• Image embeddings and all tag embeddings
• Name embeddings and all tag embeddings

3. Similarity Fusion: For this proof-of-concept, we simply average the two similarity scores. Future iterations will explore more sophisticated fusion techniques.

4. Probability Distribution: We apply softmax to the fused similarities to obtain initial tag probabilities.

5. Context Integration: We leverage the tags from retrieved similar assets:

• Tally the frequency of each tag and its hierarchical branches
• Boost probabilities of frequently occurring tags by a scale factor

As the task is a hierarchical tagging one, during the actual user interaction on the user interface, we present the user with tags level by level. The user would select the relevant tag(s) from the first level first, and second and so on, to a depth that the user deems appropriate.

API-Based Retrieval-Augmented Tagging

For comparison, we implemented a retrieval-augmented tagging pipeline using commercial API models:

1. Prompt Generation: We use Claude’s meta prompt generator to create a prompt including all possible tags.
2. Context Integration: We incorporate the most common tags from retrieved assets into the prompt.
3. Tag Prediction: We request the top-5 most likely tags from the API model.
4. Hierarchy Reconstruction: As API models struggle with generating full tag branches, we manually reconstruct the hierarchical structure. For the self-hosted model-based approach, there is no need to do this as we are tagging level by level rather than just getting five tags from a black box model.

Scope of Retrieval-Augmented Tagging

It’s important to note that our retrieval-augmented tagging approach is primarily beneficial for the object type tagging. Our findings show that shape and colour tagging do not significantly benefit from search and retrieval:

• Colour Tagging: We found that this task doesn’t significantly benefit from retrieval augmentation. The colour of an asset is typically unique and straightforward to identify, making additional context from similar assets unnecessary.
• Shape Tagging: Currently, our knowledge base lacks sufficient tagged shape data to make retrieval augmentation effective for this attribute. However, we see potential for improvement as our database grows.

For colour and shape tagging, our current pipeline operates without the probability boosting from retrieved assets. As we accumulate more diverse and comprehensively tagged data, we plan to reassess the potential of retrieval augmentation for shape tagging.

Tagging Performance Metrics

To rigorously evaluate our tagging system, we developed a set of metrics that account for the complexities of hierarchical and multi-tag classifications:

Hierarchical Tagging (Object Type):

• True Positive: A predicted tag branch with at least one intersection with the ground truth.
• Top-k Accuracy: The presence of at least one intersection with the ground truth among the top-k predicted branches.

Multi-tag Classification (Colour and Shape):

• Top-k True Positive: A prediction containing at least half of the ground truth tags within the top-k predictions.

These metrics allow us to capture the nuanced performance of our system across different tagging tasks and hierarchical levels.

Performance Comparison

Our evaluation on a carefully curated test subset of fully-tagged assets yielded compelling results as seen in the table below, with the highlighted green cells being the top performers for each metric context.

Key Findings:

1. Superior Performance of Self-Hosted Models: Our self-hosted, model-based retrieval-augmented tagging pipeline consistently outperformed API-based solutions across various metrics.
2. Fine-Grained Control: The ability to customise each stage of the process in our self-hosted solution proved invaluable, especially for handling the nuances of our specific cataloguing needs.
3. Specialisation Matters: These results challenge the notion that large, general-purpose API models are always the best solution. In specialised domains like ours, tailored approaches can yield significant improvements.

Implications and Future Directions

1. Choosing Self-Hosted Solutions: Based on our findings, we’ve decided to proceed with our self-hosted, model-based retrieval-augmented tagging approach for future development.
2. Beyond “API-First”: Our results demonstrate that while API solutions are powerful, they’re not always the optimal choice for specialised tasks. There’s significant value in developing custom AI solutions for domains requiring nuanced understanding and precise categorisation.
3. Continuous Improvement: As we expand our knowledge base and refine our models, we anticipate further improvements, particularly in areas like shape tagging where our current approach has limitations.

Retrieval-Augmented Physical Description Generation

Building on our success with retrieval-augmented tagging, we applied a similar approach to generating physical descriptions of assets. This process leverages retrieved similar assets as additional context in the prompt, enhancing the quality and accuracy of the generated descriptions.

Methodology

Our ideal pipeline for physical description generation is consistent across both local vision-language models and API-based models, as both accept prompts as input. We construct these prompts using Claude’s meta prompt generator, incorporating:

• The name and image of the query assets
• Images and physical descriptions of similar assets

However, for the current scope of this proof-of-concept project, we did not use the ideal pipeline above due to a lack of a large and diverse enough knowledge base for physical descriptions. The results below are from using a simple prompting pipeline without the search and retrieval.

Model Diversity: From Lightweight to Large-Scale

We generated physical descriptions using nine different models for the same test subset of catalogued assets:

Self-hosted Vision-Language Models:

1. Florence-2-Large-FT (0.77B parameters)
2. Phi-3-Vision-128K-Instruct (4B parameters)
3. MiniCPM-Llama3-V 2.5 (8B parameters)
4. InternVL2–8B (8B parameters)
5. LLaVA 1.6 (34B parameters)

API Models:

1. GPT-4o
2. GPT-4o-mini
3. Claude 3 Haiku
4. Claude 3.5 Sonnet

This diverse selection allowed us to compare the performance of models across different scales, from sub-billion parameter models suitable for edge deployment to large-scale API solutions.

Performance Evaluation

Evaluating the quality of generated physical descriptions presents unique challenges:

1. Subjective Variations: Descriptions can vary based on individual interpretations, level of detail, terminology preferences, and consistency standards.
2. Limitations of Traditional Metrics: Standard metrics like CLIP-Score or RefCLIP-Score may not adequately capture the nuances of domain-specific terminology and descriptions.

To address these challenges, we adopted a “model-as-a-judge” approach:

1. We used API models (GPT-4o, GPT-4o-mini, Gemini 1.5 Flash, Claude 3.5 Sonnet) to rate descriptions on a scale of 1 to 10, including both the generated and reference descriptions.
2. Descriptions were anonymised and randomly ordered to minimise bias.
3. The asset image was included in the evaluation prompt to provide necessary context.

Performance Results

Our evaluation yielded several significant insights:

1. API Model Superiority: Claude 3.5 Sonnet and GPT-4o consistently produced the highest-rated descriptions, showcasing the current capabilities of large-scale language models.
2. Impressive Performance of Smaller Models: Notably, MiniCPM-LLama3-V 2.5 and InternVL2–8B, both with only about 8 billion parameters, outperformed the human-generated reference descriptions. They achieved scores just ~1.25 points (out of 10) below the top-performing API models.
3. Potential for Self-Hosted Solutions: These results demonstrate that relatively small, self-hosted models can compete with API solutions for specialised tasks, offering a promising direction for organisations with specific needs or data privacy concerns.

Validating Our Approach: Expert Cataloguer Insights

To ensure the reliability of our “model-as-a-judge” approach, we engaged expert cataloguers to rate descriptions from one self-hosted model and two API models. This validation process revealed:

1. Strong Alignment: Expert ratings closely mirrored our model-based evaluations in terms of relative quality rankings, with only a slight discrepancy between GPT-4o and Claude 3.5 Sonnet.
2. Consistent Quality Assessment: There was high agreement between expert and model judgments on which specific descriptions warranted high scores (9–10) or low scores (0–5).
3. Practical Implications: Descriptions scoring 9–10 were deemed to require minimal to no changes before acceptance, highlighting the potential for significant time savings in the cataloguing process

Future Directions and Ongoing Improvements

While our current results are promising, we see several avenues for further enhancement:

1. Refining the Model-as-a-Judge Approach: We plan to engage more deeply with cataloguers to understand their scoring criteria and refine our automated evaluation process.
2. Exploring Model Fine-Tuning: There’s significant potential for improving self-hosted small vision-language models through fine-tuning on domain-specific data.
3. Balancing Performance and Practicality: For now, we’re utilising API-generated descriptions while working to enhance the self-hosted models. This hybrid approach allows us to benefit from high-quality descriptions immediately while investing in more flexible, potentially more cost-effective solutions for the future.

Conclusion: Paving the Way for AI-Assisted Asset Cataloguing

Our proof-of-concept project demonstrates the potential of AI-assisted cataloguing for large-scale digital asset collections. Key takeaways include:

1. The Power of Multimodal Search & Retrieval: By leveraging existing knowledge through search & retrieval techniques like retrieval-augmented tagging, we’ve significantly enhanced tagging accuracy. This approach proves particularly valuable in domains with specialised vocabularies and nuanced categorisations.
2. Custom Solutions vs. API Models: Our self-hosted model-based retrieval-augmented tagging pipeline outperformed API-based solutions, highlighting the value of tailored approaches in specialised domains. This underscores that “just call an API” isn’t always the optimal solution for complex, domain-specific tasks.
3. Promising Performance of Small Models: The impressive performance of smaller, self-hosted models like MiniCPM-LLama3-V 2.5 and InternVL2–8B suggests that resource-efficient AI solutions can compete with larger API models in specialised tasks. This opens up possibilities for more accessible and cost-effective AI implementations.
4. Innovative Evaluation Techniques: The “model-as-a-judge” approach, validated by expert human evaluations, offers a promising pathway to explore automated methods for assessing natural language outputs in domains where traditional metrics fall short.
5. The Importance of Human Oversight: While AI shows great promise in accelerating the cataloguing process, the role of human experts remains crucial. A human-in-the-loop system ensures that the final catalogued data maintains high standards.

Looking ahead, we see opportunities for:

• Fine-Tuning: The performance of our self-hosted models suggests that fine-tuning could yield even better results, potentially surpassing API solutions.
• Expanding Applications: The techniques developed here could find applications in other domains requiring specialised knowledge and nuanced categorisation.

While challenges remain, this project lays the groundwork for more efficient, AI-assisted management of digital assets across various use cases. At the same time, we do acknowledge that its effectiveness on a full-scale deployment will depend on eager adoption, ongoing evaluation, and adaptation. As we move forward, the lessons learned and techniques developed in this project will undoubtedly inform and inspire future AI initiatives at GovTech Singapore, furthering our mission to leverage technology for the public good.

Multimodal Search & Retrieval: A Case Study in Asset Cataloguing was originally published in AI Practice and Data Engineering Practice, GovTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction: Understanding Multimodal AI

This post kicks off a series exploring multimodal AI and its potential applications across various use cases in the public sector. First things first, what exactly is multimodal AI?

At its core, multimodal AI combines and processes different types of data — called modalities — in both input and output. This approach allows AI systems to understand and generate information across various formats, much like humans do when interpreting their environment. A prime example is OpenAI’s GPT-4o. Users can interact with it through speech, text, images, or videos, and receive responses in speech, text, or generated images.

Vision-Language Models (VLMs) are one of the most common and well-known types of multimodal AI. These models can:

• Process images, and more recently videos
• Understand accompanying text
• Generate text responses (and maybe images for some models)

Think GPT4o but without voice.

Another kind of multimodal model that is crucial in our work are multimodal embedding models, with one commonly used example of it being CLIP (Contrastive Language-Image Pre-training). The unique feature of multimodal embedding models is the ability to generate embeddings within a shared embedding space, in this case for images and text. This means:

• It can take either images or text as input
• It outputs embeddings (compressed numerical representations)
• The embeddings for related concepts are similar across modalities. For instance, the word “apple” and an image of an apple would have very similar embedding values, enabling powerful cross-modal search and retrieval capabilities.

While these models represent significant advancements, true multimodality — where a single model can process and generate any type of data seamlessly — remains an aspirational goal.

In this article and a few upcoming ones, we’ll explore how we’re leveraging these technologies to create innovative solutions for government agencies, always with an eye toward future developments in this rapidly evolving field.

Clarifying Our Approach to Multimodal AI

To set the stage, let’s address some common misconceptions and clarify our approach to multimodal AI:

1. We’re not training models from scratch — “So, are y’all building a ‘GovTech Griffin’ or something?”

• We leverage existing research and models developed by well-resourced organisations. Our focus is on efficient use of our limited GPU resources, not training a model from scratch for the sake of it.

2. We’re not just calling APIs — “Why not just use ChatGPT for everything?”

• While large API models are impressive, they can be sometimes too generic for specialised tasks. Our approach is to construct custom multimodal pipelines, developing self-hosted solutions for sensitive data, and fine-tuning permissively licensed open-source models when appropriate and practical. This strategy can outperform generic large API models in specific use cases.
• However, we don’t over-engineer solutions. If a problem is simple enough to be solved with an API call and the data classification and sensitivity allows for it, we take that approach.

3. We’re focused on multimodal being vision and language (for now) — “But can your AI speak like ChatGPT?”

• Our current scope of exploration centres on vision and language modalities as these two areas offer numerous opportunities across government agencies. We plan to expand to other modalities in the future, but the core of our multimodal work is on mastering VLMs.

4. We’re building flexible infrastructure, not chasing specific models — “Have you tried the latest model that just dropped?”

• Our strategy is to develop best practices and robust infrastructure and create adaptable pipelines for various use cases so that we can easily swap in newer, better models as they emerge. This approach ensures our work remains relevant despite rapid model evolution.

Our approach to multimodal AI is one of pragmatism with a focus on efficiency. Our goal is to create lasting value through thoughtful application of technology. We are building not just for the present but also the future and seek to not just be use case-driven but also inspire use cases.

Ongoing Explorations in Multimodal AI

Our vision is to develop versatile multimodal AI solutions adaptable to specific agency needs while exploring broader techniques. We aim to strike a balance between versatility and specificity in our work.

VLMs typically excel at task-specific applications like image captioning and visual question answering, with some models capable of OCR, open-vocabulary object detection, and region-specific captioning.

Here are some specific areas that we have been (and are currently) exploring.

Small VLM Fine-tuning

We are investigating the potential of fine-tuning small, task-specialised VLMs. This approach aims to create small expert models that can potentially outperform their larger and more general counterparts in specific tasks. Recent trends in AI have seen a resurgence of smaller LLMs and VLMs, typically within a single-digit billion parameters range, or even in the millions range like Florence-2. These compact models offer the advantage of being able to run on-premise or on-device, making them more accessible and practical for our resource constraints.

To clarify, we are not suggesting that fine-tuning these VLMs will outperform other models in classic vision tasks like object detection or segmentation. Classic vision models like YOLO and RT-DETR will still excel at these tasks. Rather, we are focusing on more general and open-ended tasks, such as tagging and captioning, comparing against larger VLMs, not classic vision models.

Some explorations we have done encompass fine-tuning VLMs from model families such as Bunny, LLaVA, MiniCPM, and InternVL. We’re applying these models to tasks like detailed image captioning and tagging, utilising publicly available data that aligns with various agency use cases.

Through our explorations, we’ve gained several key insights:

• Data quality is paramount for small models — We’ve observed that smaller models are significantly more sensitive to data quality compared to their larger counterparts. The adage “garbage in, garbage out” is particularly relevant here. To address this, we are prioritising thorough data cleaning processes and strive to ensure that our ground truth data closely aligns with our desired outcomes. This attention to data quality is crucial for the successful fine-tuning of small VLMs.
• Synthetic data proves valuable for fine-tuning — Obtaining high-quality, real-world data can be challenging, especially during exploratory phases where formal project agreements may not be in place. Even when we do have access to real-life data, it’s often insufficient in quantity or quality for effective training. To overcome this hurdle, we’ve adopted the strategy of generating synthetic data. This involves using larger, more capable models to create ideal training examples through extensive prompt engineering. This approach allows us to supplement our training data with high-quality, task-specific examples.
• Task-tuned small VLMs can outperform larger, general VLMs in specific benchmarks — Our internal benchmarks have yielded encouraging results. We’ve found that small VLMs, when finetuned on high-quality, task-specific data, can perform on par with — and sometimes even outperform — generic VLMs that are several times larger. This finding underscores the potential of specialised, efficient models in tackling specific tasks, challenging the notion that bigger is always better in the world of AI.

These learnings are shaping our approach to developing practical, efficient, and effective multimodal AI solutions for WOG. By focusing on small, specialised models, we’re aiming to create AI tools that are not only powerful but also resource-efficient and adaptable to specific use cases. We hope to share more specifically about our finetuning efforts in future articles.

Multimodal Prompting

Most VLMs typically require text prompting, necessitating prompt engineering exploration. Our investigations have yielded several interesting observations:

• Effectiveness of simple prompts for VLMs: Open-source small to mid-sized VLMs respond best to straightforward, task-specific instructions. Unlike text-only LLMs, which often benefit from elaborate prompts, VLMs perform optimally with simple directives. For tasks such as image captioning or visual question answering, a concise prompt like “Describe this image in detail” typically outperforms more complex prompting schemes.
• VLM training effects on prompt responsiveness: The training process of VLMs appears to influence their ability to handle varied text inputs. As these models are fine-tuned on vision-language tasks, they seem to lose some flexibility in processing diverse text prompts — a characteristic that differs from pure language models. For example, anecdotally, VLMs would tend to ignore instructions in prompts more often than LLMs. This reduced versatility may be due to the specific instruct-tuning datasets used in VLM training.
• VLM-LLM chaining is useful for formatting prompts: Since VLMs are not so responsive to complex instructions currently, we find that chaining a VLM’s output to a LLM to further reformat or rephrase can help meet complex desired outcomes. The LLM need not be large, just good enough to be a bit more adept at language tasks. This approach allows us to leverage the strengths of both model types: the VLM’s ability to interpret visual information and the LLM’s proficiency in language manipulation and formatting.

As open-source VLMs continue to advance, there may be potential for more sophisticated prompting strategies in the future.

Multimodal Search and Retrieval

Search and retrieval is a well-established concept in the vision domain, predating the era of generative AI. In the generative AI era, it got rebranded to retrieval-augmented generation (RAG).

Extending search and retrieval to multimodal AI is a natural progression. It encompasses not only image-image or text-text retrieval but also cross-modal retrieval between image and text. The key to enabling this cross-modal functionality lies in unified embedding spaces provided by models like CLIP.

We also explored modality conversion which converts the image into text modality by using a VLM to generate a zero-shot caption. This grants an additional knowledge base to perform retrieval on, which is arguably a more “grounded” one compared to an image knowledge base for retrieval using text modality.

We’ve applied multimodal search and retrieval to two specific tasks:

1. Hierarchical image tagging: Incorporating search and retrieval into our existing tagging pipeline has yielded significant performance improvements up to 40+%.
2. Detailed image captioning: RAG has proven effective in aligning output captions with expected styles, as the model learns from retrieved context.

We hope to release an article in the coming weeks to go a lot more in depth into this area with a specific case study and use case.

Key learnings from our multimodal search and retrieval explorations:

• Cross-modal retrieval efficacy: Multimodal embedding models have emerged as a cornerstone of multimodal operations, enabling cross-modal retrieval between images and text. This capability is crucial for effective multimodal search and retrieval.
• Modality conversion enhances search and retrieval performance: Leveraging a VLM to generate zero-shot captions for images works. These captions serve as an additional knowledge base for RAG, effectively bridging the gap between visual and textual information.
• RAG as a cost-effective performance booster: Compared to the resource-intensive process of model fine-tuning, RAG offers a relatively low-effort approach to significantly improve model performance. This makes it an attractive option for enhancing multimodal AI systems efficiently.

Multimodal Benchmarking

The challenge of benchmarking and performance evaluation in the era of LLMs extends to VLMs, presenting unique complexities. Public benchmarks and leaderboards, and traditional metrics such as BLEU, ROUGE, and CLIPScore often fall short in accurately reflecting real-world task performance, particularly for natural language outputs like detailed image captions. A significant limitation of reference-based performance metrics is that the reference itself is not always a “golden standard.”

While the model-as-a-judge approach, which utilises large API-based models to score responses for hard-to-quantify tasks, shows promise with alignment to human preferences in our preliminary tests, its scalability is limited by potentially high costs and its subjective nature.

Where possible, we create in-house benchmarks for specific tasks that are customised for success in those applications. These tailored benchmarks allow us to evaluate models based on criteria that are directly relevant to our use cases, providing a more accurate assessment of a model’s potential real-world performance.

As for how we evaluate new models for potential adoption, we follow a three-step process:

1. Vibes Check: We begin with an intuitive assessment to gauge the model’s initial feel and performance from a user perspective. This also sometimes involves looking at user sentiments from online sources like r/localllama or X. There’s nothing empirical about this, but sometimes intuition can be very useful.
2. Hard Sample Testing: If the model passes the vibes check, we proceed to test it against a set of challenging examples that have proven difficult for previous models.
3. Detailed Evaluation: Only models that show promise in the first two steps undergo our comprehensive evaluation process. This tiered approach helps us manage resources efficiently while ensuring thorough assessment of promising candidates.

Key learnings from our benchmarking efforts:

1. Limitations of Standard Metrics: Online leaderboards, popular benchmarks, and traditional metrics are good high-level indicators, but often fail to capture the nuances of real-world, task-specific performance. This observation underscores the need for developing custom evaluation methods or adapting existing metrics to better align with our specific use cases and contexts.
2. Efficacy of Incremental Evaluation: Our tiered approach (Vibes check -> Hard sample check -> Proper check) has proven effective in balancing thoroughness with efficiency. By starting with small-scale assessments and progressively moving to more comprehensive evaluations, we can identify promising models without expending unnecessary resources on less suitable candidates.
3. Value of Custom Benchmarks: Our in-house, task-specific benchmarks have proven invaluable in assessing models for particular applications. These custom evaluations provide insights that generic benchmarks often miss, allowing us to select models that are truly better for our specific needs.
4. Limitations of Reference-Based Metrics: We’ve recognised that reference-based performance metrics can be misleading when the reference itself isn’t a perfect standard. This is particularly true in tasks with multiple valid outputs, necessitating more flexible and nuanced evaluation approaches.

This benchmarking strategy allows us to navigate the complex landscape of VLM evaluation, ensuring that we select models that not only perform well on paper but also meet the practical needs of our specific applications.

Possible Multimodal Use Cases

After discussing our explorations and learnings in multimodal AI, it’s crucial to address how these efforts can be applied across various government sectors. Here are several examples of possible multimodal AI use cases that could be implemented in real-world applications, grouped by their primary function:

1. Image-Text Relevance

Image-text relevance refers to checking whether image-text pairs are relevant to each other. This use-case is primarily enabled by multimodal embedding models like CLIP to compare natural language text with images. This capability can be realised by converting both textual and visual data into a common format for comparison. The system would then assess the similarity between the text and image representations, establishing criteria to classify their level of relevance.

In practical use, this use case can enable and augment workflows that involve document processing, compliance verification, and public communication in government settings. For instance, it can be used to verify that images submitted with official reports or applications match the accompanying textual descriptions. This technology can also streamline administrative processes by automatically flagging discrepancies between visual and textual content, reducing the manual workload on officers and improving the accuracy of reviews.

Additionally, it can be employed in public-facing government websites and publications to ensure that images are appropriately matched with their captions or surrounding text, enhancing the clarity and effectiveness of government communications. In scenarios involving large-scale data management, this capability can assist in organising and validating extensive collections of government records, ensuring that visual data is correctly associated with its textual metadata.

2. Open Vocabulary Object Detection

Open vocabulary object detection refers to the ability to identify and locate objects in images or video without being limited to a predefined set of object categories. This use-case leverages AI models that can recognise a wide range of objects based on their general understanding of the visual world. Open vocabulary object detection is currently available on the Video Analytics System.

In practical applications, this technology can significantly enhance various monitoring, inspection, and analysis tasks. For instance, in environmental monitoring, it can identify diverse wildlife species in nature camera footage, even those not explicitly included in training data. This flexibility is particularly valuable in government contexts where the range of potential objects of interest is vast or constantly evolving. It reduces the need for frequent model updates and allows for more comprehensive and adaptable visual analysis across different departments and scenarios.

3. Media Tagging, Captioning, and Search

Media tagging, captioning, and search involve automatically assigning descriptive labels, generating detailed textual descriptions for images and videos, and enabling efficient retrieval of visual content. This use-case leverages advanced AI models to understand, describe, and index visual content with high specificity.

This technology can significantly enhance content management, accessibility, and information retrieval across various government applications:

• Digital Archives: It can automatically tag and describe historical photographs, making vast collections searchable. Researchers and the public can then easily find specific images using natural language queries, such as “find photos of urban development in the 1960s.”
• Public Communications: For government publications and websites, it can generate accurate captions for images, ensuring clarity and context for all citizens, including those with visual impairments. The search capability allows staff to quickly locate appropriate images for specific communication needs.
• Cross-Modal Search: By leveraging models like CLIP, the system enables powerful cross-modal search capabilities. Users can find images using text descriptions or even use an image to find similar visual content across large databases.

This capability is particularly valuable in contexts where large volumes of visual data need to be processed, organised, and made accessible. It can dramatically reduce the manual effort required for media management while improving the discoverability and usability of visual information. The addition of advanced search functionality transforms static visual archives into dynamic, easily navigable resources, enhancing decision-making processes and public service delivery.

This capability could be implemented using a combination of vision-language embedding and generative models for content analysis and tagging. The system might incorporate techniques such as RAG to enhance tagging and captioning accuracy by leveraging relevant contextual information. It could be designed to generate hierarchical tags, from broad categories to specific descriptors, facilitating multi-level organisation and retrieval. Additionally, the system could include captioning models capable of generating detailed, contextually relevant descriptions of visual content.

Conclusion: The Future of Multimodal AI in Government

As we’ve explored throughout this article, GovTech’s journey into multimodal AI is driven by a commitment to practical, efficient solutions that address real government needs. Our approach balances innovation with pragmatism, focusing on creating lasting value rather than chasing the latest trends.

Key takeaways from our multimodal AI initiatives thus far include:

• The power of small, specialised models: Our initial explorations suggest that carefully tuned small VLMs can often match or exceed the performance of larger, more resource-intensive models for specific tasks.
• The importance of adaptable infrastructure: Our focus on building flexible pipelines allows us to quickly incorporate new models and technologies as they emerge.
• The value of multimodal search and retrieval: By extending search and retrieval and retrieval-augmented generation to the visual domain, we’ve significantly improved performance in tasks like image tagging and captioning.
• The need for custom benchmarking: We’ve developed task-specific evaluation methods that more accurately reflect real-world performance than generic benchmarks.

These insights are shaping our approach to developing practical, efficient, and effective multimodal AI solutions for whole-of-government use. The potential use cases we’ve outlined — including image-text relevance checking, open vocabulary object detection, and advanced media tagging and captioning — demonstrate the wide-ranging impact that multimodal AI could have on government operations.

As multimodal AI technologies evolve, GovTech remains committed to harnessing their potential to create more responsive, efficient, and citizen-centric government services. By integrating cutting-edge AI research with practical government applications, we’re not just improving existing processes — we’re reimagining what’s possible in public sector technology.

Multimodal AI: Our Approach, Explorations, and Use Cases was originally published in AI Practice and Data Engineering Practice, GovTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the realm of AI, the focus has been fixated on Large Language Models (LLMs) and leveraging powerful APIs like OpenAI’s. However, my recently completed project was in an entirely different domain — computer vision at the edge.

Exploring Balefire

Recently, we’ve wrapped up the third iteration of Balefire, an edge AI solution aimed at identifying smoking activities in public spaces. This project is an exploration with the National Environment Agency (NEA), a collaborative partnership that has been flourishing since Balefire’s inception in 2019. The principal aim of Balefire in this third iteration is to assist NEA in detecting smokers in smoking-prohibited places. The system provides vital information, such as the frequency and patterns of smoking activities, as well as overall foot traffic data. These insights facilitate NEA in optimising the allocation of enforcement officers to these identified hotspots.
While the first two iterations served primarily as proof-of-concept demonstrations, the latest iteration of Balefire represents our largest scale pilot to date, with 20 installations around Singapore. This expanded pilot is a strategic move towards applying the valuable insights gained from these field deployments towards full operationalization in the future.

The Intricacies of Smoking Detection

At first glance, detecting smoking activities might seem straightforward. Many AI professionals might initially view Balefire as a mere cigarette detection tool and wonder about its complexity. I shared this sentiment until I delved into the project, where I quickly realized the substantial challenges involved. Cigarettes are notably difficult to detect — they’re small, often partly occluded. Moreover, numerous everyday objects resemble them, especially when viewed through cameras in an outdoors setting. Common misidentifications include straws, shiny phone edges, fingers positioned in certain ways, and even certain types of food, all of which can be easily mistaken for a cigarette. Relying on smoke detection or the cigarette’s glowing tip as detection cues also proved error prone. Going beyond the cigarette and looking at the entire person, such as through pose estimation, also resulted in an unacceptable level of false positives. All these insights, gained from lots of experimentation, led to the understanding that an end-to-end detection model isn’t feasible, particularly in an edge AI context with its inherent compute limitations and relatively small model sizes, coupled with the need for near-instantaneous detection.

Furthermore, the challenge extends beyond mere detection. It’s crucial to avoid duplicate detections for the same smoker as that will skew trends and prevent meaningful analysis. This required a reidentification process within the edge pipeline, adding another layer of complexity to the task.

Pre-Existing Solutions

When confronted with the task of smoking activity detection, it seemed logical to assume that existing solutions would be readily available. This was my initial belief as well. However, upon exploring the market, it became evident that while there are service providers offering smoking detection capabilities, their offerings are often constrained by specific limitations. For instance, many of these systems require the smoker to be fully visible within the camera’s field of view, leading to a high likelihood of false negatives. Others demand that the subjects be positioned quite close to the camera, a requirement stemming from their need for a minimum size of detectable objects.

NEA’s requirements, however, called for a more versatile solution. They needed a system capable of identifying as many smokers as possible across the entire span of a camera’s field of view and doing so almost instantaneously. This set of needs was not met by the existing solutions and thus necessitated the development of a custom solution tailored specifically to these unique operational demands.

Balefire Pipeline

Determining the most effective approach for smoking activity detection required extensive experimentation and iterative improvements. Eventually, we established a robust pipeline specifically tailored for this task:

1. Head Detection and Processing: The pipeline initiates with the camera frames being fed into a head detector, which identifies the coordinates of all heads within the frame.
2. Heuristic-Based Filtering: Post-detection, these heads undergo a series of heuristic filters designed to eliminate potential erroneous heads. These filters are the product of accumulated learnings and detailed analysis of deployment data.
3. Head Tracking: An object tracker then follows the detected heads across successive frames, linking them with previously detected heads wherever possible. This ensures that, for identified smokers, repeated alerts are not triggered each time they are recognized in a new frame.
4. Smoke/No-Smoke Classification: Heads not previously classified as belonging to smokers are then processed through a binary head classifier. This classifier determines whether the individual is smoking or not.
5. Reidentification Module: If the classifier indicates smoking activity, a reidentification module attempts to match the detected smoker against a watchlist of recent smokers. If there is no reidentification, an alert is triggered. The watchlist is updated with the latest appearance of the smoker and other relevant information.

Typically, reidentification is integrated with object tracking in computer vision applications. However, to maintain real-time analysis capabilities in a compute-limited edge environment, we chose to apply reidentification solely for detected smokers, rather than for all tracked heads. The addition of reidentification after classification helped to further reduce double/multiple counting of the same person.

In Version 1, we used a face detector instead of a head detector. However, the inconsistency in facial visibility, such as when individuals turn away from the camera, led to tracking issues and downstream double/multiple counting where multiple alerts are sent out for the same smoker on different frames.

In Version 2, an attempt was made to combine head and face detectors for enhanced filtering, but with finetuning of the head classifier and the addition of the reidentification module, this became redundant.

While some of these design choices might seem unconventional, they are justified within the context of edge AI deployment. Empirical data from internal benchmarks and on-the-ground operational improvements support these decisions.

From the first to the third version of Balefire, these strategic decisions, coupled with continuous model fine tuning using data collected over the course of the project, have led to a reduction of false positives and double counts by approximately 90%, while maintaining a high true positive rate and minimizing false negatives.

For data collection and annotation for training of the constituent models, we used actual footage from the current and past iterations of Balefire and used a semi-supervised approach to annotate them. Simply put, we used our existing models to annotate the new data for us and corrected any errors from that process. We iteratively added in specific profiles of images that the existing models were error prone in, such as persons wearing helmets, or persons who are eating or drinking. This helped to improve the performance of the models significantly over the course of the project.

Operationalization Considerations

Operational considerations played a critical role in the development of Balefire. Our focus extended beyond merely proving the viability of smoking activity detection, a milestone achieved by the previous iteration in 2020. The challenge was to create a system that not only enhanced NEA’s enforcement operations but also incorporated a set of quantitative performance metrics addressing key issues like false positives and double counting.

In the realm of performance evaluation, accuracy is a fundamental and very commonly used metric. However, for Balefire’s specific operational context, precision and recall are more relevant. A high precision rate is crucial to ensure that NEA is not overwhelmed with false positive alerts, while a high recall rate is essential to identify as many smokers as possible within the operational scope.

However, neither precision nor recall sufficiently address the issue of double counting. To tackle this, we introduced a novel metric: strict precision. This metric is an adaptation of the standard precision formula, integrating double counts into the calculation. By doing so, strict precision evaluates the detections not only for their accuracy but also for their necessity and relevance, ensuring that each alert sent to NEA is both valid and essential.

To rigorously assess Balefire’s performance, we established two benchmarks using video footage from all deployment locations. These benchmarks utilized three main performance metrics — accuracy, precision, and our newly defined strict precision — to evaluate and compare the efficacy of Balefire across its three iterations and various experimental modifications. This comprehensive approach has been instrumental in refining Balefire’s capabilities and ensuring its operational effectiveness for NEA.

Lessons Learned

Some valuable non-technical lessons that I have learnt from working on Balefire include:

• Revisiting Old Solutions: Previously discarded ideas should be reassessed from time to time, as they might offer unexpected benefits in a different context. For instance, the reidentification model was initially set aside after not working well together with the object tracker for real-time tracking. In subsequent iterations, I tried reintegrating this model from another perspective, and this resulted in a significant improvement in Balefire’s performance.
• The Swiss Cheese Approach: For Balefire, we embraced the Swiss Cheese Model of Risk, a concept traditionally used in risk management and safety engineering. This model visualizes a system’s defenses as layers of Swiss cheese, where each layer represents a safeguard, and the holes symbolize potential failures or weaknesses. The key is that the weaknesses in one layer are covered by another, reducing errors overall.
• Reality Check on Academic Benchmarks: New model releases, while impressive on research benchmarks, often fall short in real-world applications as these benchmarks often do not adequately represent the challenges and nuances that can arise in practical settings. There needs to be real-world, context-specific internal benchmarks developed to properly evaluate experiments.
• Resourcefulness and Pragmatism: Innovation doesn’t always mean inventing something entirely new. Often, the most efficient solutions come from creatively using existing resources and technologies. It’s crucial, however, to be mindful of legal aspects like software licenses when utilizing available tools.

These insights are often reiterated by many AI practitioners but only become profoundly meaningful when experienced firsthand.

Conclusion

Deploying complex AI systems at the edge, as we do at DSAID, presents a unique set of challenges and opportunities distinct from traditional cloud-based solutions. Edge computing, characterized by its limited computational capacity compared to cloud environments, necessitates a fundamentally different approach to AI algorithm optimization. The evolution of the Balefire project underlines the importance of a tailored approach to edge AI deployment. This project has shown how technical expertise must be integrated with a profound understanding of the specific operational needs. In the case of NEA’s requirements, it was evident that off-the-shelf models were insufficient, leading us to develop bespoke solutions that precisely addressed the unique challenges at hand.

Looking ahead, the future of edge AI remains vibrant. With recent developments like the impressive benchmark results achieved by vision models on the Raspberry Pi 5, and the substantially enhanced capabilities of the Jetson Orin platform, the field of edge AI is at the forefront of technological innovation. These advancements open a plethora of exciting possibilities for us at DSAID. From enhancing public service delivery to exploring new realms of AI applications, the future of edge AI is replete with opportunities for groundbreaking tech for public good.

Balefire: Navigating the Challenges of Edge AI in Smoking Detection was originally published in AI Practice and Data Engineering Practice, GovTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my last article, I shared how to run large language models and vision language models on the Raspberry Pi. This time around, instead of LLMs and VLMs, we shall run an image generation model — Stable Diffusion XL (SDXL) Turbo — on the Raspberry Pi 5. It’s another impossible sounding feat, but open-source wonders do exist, and running a SDXL Turbo model on a very resource constrained environment is one of them.

OnnxStream

OnnxStream is an open-source project created by Vito Plantamura with the original intention of running Stable Diffusion 1.5 (SD1.5) on a Raspberry Pi Zero 2 by minimising memory consumption as much as possible, albeit at the cost of increased inference latency/throughput.

At the time of writing, it has expanded to support not only Stable Diffusion 1.5 but also Stable Diffusion XL 1.0 Base (SDXL) and Stable Diffusion XL Turbo 1.0. I won’t go into detail about how exactly this amazing feat is being achieved since the GitHub repository already explains it very well.

Instead, let’s just jump right into getting it working.

Technical Requirements

All you need is the following:

• Raspberry Pi 5 — Or Raspberry Pi 4 or any other Raspberry Pi, just expect it to be slower
• SD Card — Minimally 16GB, with Raspbian or some Linux distro already setup. The SDXL Turbo weights are around 8GB.
• An internet connection

Setting Up OnnxStream

The instructions here are from the GitHub repository, but I’m breaking it down and explaining it a bit more.

1. Building XNNPack

First, we have to install XNNPack, which is a library from Google that provides “high-efficiency floating-point neural network inference operators”. But we can’t just get the latest version in case of any breaking changes. Instead, we shall get the version that the OnnxStream creator has verified to be working at the time of writing. In a terminal, run:

git clone https://github.com/google/XNNPACK.git
cd XNNPACK
git checkout 579de32260742a24166ecd13213d2e60af862675
mkdir build
cd build
cmake -DXNNPACK_BUILD_TESTS=OFF -DXNNPACK_BUILD_BENCHMARKS=OFF ..
cmake --build . --config Release

XNNPack will take a couple of minutes to build. Go get coffee or something.

2. Building OnnxStream

Next, we have to build OnnxStream. In a terminal, run:

git clone https://github.com/vitoplantamura/OnnxStream.git
cd OnnxStream
cd src
mkdir build
cd build
cmake -DMAX_SPEED=ON -DXNNPACK_DIR=<DIRECTORY_WHERE_XNNPACK_WAS_CLONED> ..
cmake --build . --config Release

Make sure to replace <DIRECTORY_WHERE_XNNPACK_WAS_CLONED> with the path at which XNNPack was cloned to (not the build folder). In my case, it was at /home/admin/XNNPACK/.

3. Downloading Model Weights

Now, we need to download the model weights for SDXL Turbo. In a terminal, run:

git lfs install
git clone --depth=1 https://huggingface.co/AeroX2/stable-diffusion-xl-turbo-1.0-onnxstream

If you have not installed git-lfs yet, do so first. This will take even longer than the step before since the model weights are quite big. Go get lunch!

You can also run the other two models supported — Stable Diffusion 1.5 and Stable Diffusion XL 1.0 Base by downloading their weights from the links provided in OnnxStream’s GitHub repository. Make sure your SD card has enough space if you are downloading all these models!

Once done, that’s it! We are ready to generate images on a Raspberry Pi!

Generating Images

To generate images, run the code block below:

cd ~/OnnxStream/src/build/
./sd --turbo --models-path /home/admin/stable-diffusion-xl-turbo-1.0-onnxstream --prompt "An astronaut riding a horse on Mars" --steps 1 --output astronaut.png

Replace the prompt with what you want to generate. I’m just using the go-to classic astronaut prompt here. I set steps to just 1 as SDXL Turbo doesn’t need many steps to generate a good-looking image.

There are other arguments you can pass too, such as — neg-prompt for negative prompts (SDXL Turbo does not support negative prompts but you can use it for the other two models), — steps to set the number of generative steps and — seed to set the random seed.

The arguments required will change according to the type of model used, so please take a look at OnnxStream’s GitHub repository for the full list of arguments to pass if you’re using something other than SDXL Turbo.

As you can see in the image above, on the Raspberry Pi 5, each diffusion step takes around 1 minute, and in total with pre-processing and decoding, it around 3 minutes to generate a single image.

Here’s a comparison and progression of the same prompt with the same seed from step 1 to 10. You can see that even with a single step with refinement, the generated image is already really well-done. This is in contrast to SDXL or SD1.5 which requires quite a few steps to reach that quality.

Conclusion

With it taking around at least a couple of minutes to generate an image, the question of use cases for it comes begging.

The most obvious and fun use case to me is an ever-changing photo frame that will generate a new image every few minutes. There is actually a project along this tangent that uses OnnxStream, by rvdveen on GitHub, where he uses OnnxStream on a Raspberry Pi Zero 2 W to generate images for news articles and shows it on a photo frame using an e-ink display. It takes around 5 hours to generate an image on the Pi Zero 2 W with what should be SD1.5, but hey it’s not like you need a photo frame to be changing what it’s showing in real-time.

Or maybe you just want your own locally hosted image generator which can produce decent quality images while not hogging any major compute devices on your network.

Have fun playing around with SDXL Turbo on the Raspberry Pi!

Disclaimer: I have no affiliation with OnnxStream or StabilityAI. All views and opinions are my own and do not represent any organisation.

Generating Images with Stable Diffusion and OnnxStream on the Raspberry Pi was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ever thought of running your own large language models (LLMs) or vision language models (VLMs) on your own device? You probably did, but the thoughts of setting things up from scratch, having to manage the environment, downloading the right model weights, and the lingering doubt of whether your device can even handle the model has probably given you some pause.

Let’s go one step further than that. Imagine operating your own LLM or VLM on a device no larger than a credit card — a Raspberry Pi. Impossible? Not at all. I mean, I’m writing this post after all, so it definitely is possible.

Possible, yes. But why would you even do it?

LLMs at the edge seem quite far-fetched at this point in time. But this particular niche use case should mature over time, and we will definitely see some cool edge solutions being deployed with an all-local generative AI solution running on-device at the edge.

It’s also about pushing the limits to see what’s possible. If it can be done at this extreme end of the compute scale, then it can be done at any level in between a Raspberry Pi and a big and powerful server GPU.

Traditionally, edge AI has been closely linked with computer vision. Exploring the deployment of LLMs and VLMs at the edge adds an exciting dimension to this field that is just emerging.

Most importantly, I just wanted to do something fun with my recently acquired Raspberry Pi 5.

So, how do we achieve all this on a Raspberry Pi? Using Ollama!

What is Ollama?

Ollama has emerged as one of the best solutions for running local LLMs on your own personal computer without having to deal with the hassle of setting things up from scratch. With just a few commands, everything can be set up without any issues. Everything is self-contained and works wonderfully in my experience across several devices and models. It even exposes a REST API for model inference, so you can leave it running on the Raspberry Pi and call it from your other applications and devices if you want to.

There’s also Ollama Web UI which is a beautiful piece of AI UI/UX that runs seamlessly with Ollama for those apprehensive about command-line interfaces. It’s basically a local ChatGPT interface, if you will.

Together, these two pieces of open-source software provide what I feel is the best locally hosted LLM experience right now.

Both Ollama and Ollama Web UI support VLMs like LLaVA too, which opens up even more doors for this edge Generative AI use case.

Technical Requirements

All you need is the following:

• Raspberry Pi 5 (or 4 for a less speedy setup) — Opt for the 8GB RAM variant to fit the 7B models.
• SD Card — Minimally 16GB, the larger the size the more models you can fit. Have it already loaded with an appropriate OS such as Raspbian Bookworm or Ubuntu
• An internet connection

Like I mentioned earlier, running Ollama on a Raspberry Pi is already near the extreme end of the hardware spectrum. Essentially, any device more powerful than a Raspberry Pi, provided it runs a Linux distribution and has a similar memory capacity, should theoretically be capable of running Ollama and the models discussed in this post.

1. Installing Ollama

To install Ollama on a Raspberry Pi, we’ll avoid using Docker to conserve resources.

In the terminal, run

curl https://ollama.ai/install.sh | sh

You should see something similar to the image below after running the command above.

Like the output says, go to 0.0.0.0:11434 to verify that Ollama is running. It is normal to see the ‘WARNING: No NVIDIA GPU detected. Ollama will run in CPU-only mode.’ since we are using a Raspberry Pi. But if you’re following these instructions on something that is supposed to have a NVIDIA GPU, something did not go right.

For any issues or updates, refer to the Ollama GitHub repository.

2. Running LLMs through the command line

Take a look at the official Ollama model library for a list of models that can be run using Ollama. On an 8GB Raspberry Pi, models larger than 7B won’t fit. Let’s use Phi-2, a 2.7B LLM from Microsoft, now under MIT license.

We’ll use the default Phi-2 model, but feel free to use any of the other tags found here. Take a look at the model page for Phi-2 to see how you can interact with it.

In the terminal, run

ollama run phi

Once you see something similar to the output below, you already have a LLM running on the Raspberry Pi! It’s that simple.

You can try other models like Mistral, Llama-2, etc, just make sure there is enough space on the SD card for the model weights.

Naturally, the bigger the model, the slower the output would be. On Phi-2 2.7B, I can get around 4 tokens per second. But with a Mistral 7B, the generation speed goes down to around 2 tokens per second. A token is roughly equivalent to a single word.

Now we have LLMs running on the Raspberry Pi, but we are not done yet. The terminal isn’t for everyone. Let’s get Ollama Web UI running as well!

3. Installing and Running Ollama Web UI

We shall follow the instructions on the official Ollama Web UI GitHub Repository to install it without Docker. It recommends minimally Node.js to be >= 20.10 so we shall follow that. It also recommends Python to be at least 3.11, but Raspbian OS already has that installed for us.

We have to install Node.js first. In the terminal, run

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - &&\
sudo apt-get install -y nodejs

Change the 20.x to a more appropriate version if need be for future readers.

Then run the code block below.

git clone https://github.com/ollama-webui/ollama-webui.git
cd ollama-webui/

# Copying required .env file
cp -RPp example.env .env

# Building Frontend Using Node
npm i
npm run build

# Serving Frontend with the Backend
cd ./backend
pip install -r requirements.txt --break-system-packages 
sh start.sh

It’s a slight modification of what is provided on GitHub. Do take note that for simplicity and brevity we are not following best practices like using virtual environments and we are using the — break-system-packages flag. If you encounter an error like uvicorn not being found, restart the terminal session.

If all goes correctly, you should be able to access Ollama Web UI on port 8080 through http://0.0.0.0:8080 on the Raspberry Pi, or through http://<Raspberry Pi’s local address>:8080/ if you are accessing through another device on the same network.

Once you’ve created an account and logged in, you should see something similar to the image below.

If you had downloaded some model weights earlier, you should see them in the dropdown menu like below. If not, you can go to the settings to download a model.

The entire interface is very clean and intuitive, so I won’t explain much about it. It’s truly a very well-done open-source project.

4. Running VLMs through Ollama Web UI

Like I mentioned at the start of this article, we can also run VLMs. Let’s run LLaVA, a popular open source VLM which also happens to be supported by Ollama. To do so, download the weights by pulling ‘llava’ through the interface.

Unfortunately, unlike LLMs, it takes quite some time for the setup to interpret the image on the Raspberry Pi. The example below took around 6 minutes to be processed. The bulk of the time is probably because the image side of things is not properly optimised yet, but this will definitely change in the future. The token generation speed is around 2 tokens/second.

To wrap it all up

At this point we are pretty much done with the goals of this article. To recap, we’ve managed to use Ollama and Ollama Web UI to run LLMs and VLMs like Phi-2, Mistral, and LLaVA on the Raspberry Pi.

I can definitely imagine quite a few use cases for locally hosted LLMs running on the Raspberry Pi (or another other small edge device), especially since 4 tokens/second does seem like an acceptable speed with streaming for some use cases if we are going for models around the size of Phi-2.

The field of ‘small’ LLMs and VLMs, somewhat paradoxically named given their ‘large’ designation, is an active area of research with quite a few model releases recently. Hopefully this emerging trend continues, and more efficient and compact models continue to get released! Definitely something to keep an eye on in the coming months.

Disclaimer: I have no affiliation with Ollama or Ollama Web UI. All views and opinions are my own and do not represent any organisation.

Running Local LLMs and VLMs on the Raspberry Pi was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

‘Today I Learned’ (TIL) was one of the two competitions that the Defence Science and Technology Agency (DSTA) of Singapore organised this year as part of ‘Brainhack’, its first iteration of an event that combined technology showcases and competitions on cybersecurity and artificial intelligence (AI). Just like the name suggests, TIL is the AI related competition. There were two categories, university/polytechnics and junior college/integrated programme for which I joined the former.

The Dataset & The Challenge

We are provided with 11 classes of human actions/poses, with around 85–90 images per class. The images are all taken by the organisers themselves and is a private dataset so unfortunately I won’t be able to show them in this post, but what’s below are lookalikes from Google. Thankfully, the classes were all rather balanced in terms of numbers.

The task is basically image classification with the sole metric being accuracy. A validation set of about 20 images per class was also provided. The whole dataset was structured in terms of folder classes.

We are given seven days from the release of the dataset to develop our pipeline and train our models. Then there will be a competition day where two test sets will be released. The second test set will come after a ‘curveball challenge’ for which we were given no information on but it was hinted that it was a second dataset with related classes to the first. The curveball challenge was only to last 3 hours, so that meant that we would have less than 3 hours to train our new models.

Only 30 minutes would be given for us to evaluate the second test set with only one submission allowed, which means that our pipeline cannot be some convoluted process that takes forever to process the images.

Due to all these constraints, I expected my first test set to perform the best and the second one to do a lot worse, so I had to push my accuracy for the initial dataset to as high as possible.

Framework

For training the model, I used FastAI’s wonderful library. It’s basically the perfect wrapper on top of PyTorch with optimised presets and amazing features that makes it a breeze to use. The FastAI community is also very helpful and there’s always a few people in it trying to incorporate whatever the latest development is into FastAI as soon as possible.

Baseline

I started off looking at a simple solution, pluck the original dataset into a Convolutional Neural Network (CNN) and hope for the best. Using a pretrained resnet50 model, I managed to get around 70% accuracy after going through the usual pretrained model training pipeline (training with the last layer unfrozen, then unfreezing the whole model and retraining again), with a total of 20 epochs.

The image size was set to 224x224 and the transformations were the default transformations from get_transforms(). Training was done using the one-cycle policy (with cyclic learning rate and momentum) which is an awesome built-in feature of FastAI so I didn’t need to implement it from scratch.

Changing Image Size

I tried progressive image resizing during training which has been shown to improve accuracy and went from 128x128 to 224x224 to 256x256 to 300x300 which did show some improvements to around 72% in accuracy. But this did mean that training would take in theory at least 4 times longer for around 2% increase in performance which was a tradeoff that I’m not sure I wanted, given the time constraints.

A good example of this image resizing strategy is this cifar-10 notebook by FastAI:

https://github.com/fastai/fastai/blob/master/courses/dl1/cifar10.ipynb

Results Interpretation

The confusion matrix below is for the initial baseline model with 70% accuracy. Some classes like KungfuCrane performed very badly while other classes like KoreanHeart and ChildPose performed very well. KungfuCrane gets mistaken for other classes like ChairPose easily, which slightly explains why it performs badly, since the poses are somewhat similar. WarriorPose also gets confused with KungfuCrane often, probably because of how both poses has the person with his/her arms outstretched.

I further ventured into the dataset to look at what else might be causing these problems. It turns out that the images were intentionally ‘unclean’ in the sense that certain poses had several different variations which can be mistaken by the model for another pose. For example, HulkSmash had the person in the image in various stages of ‘smashing’ from arms high above the head to arms already on the ground. There were also images that were taken from the side view instead of the person facing the camera inside the validation set, but such images were present in very small numbers in the training set.

Plotting the images with the highest losses also revealed that sometimes furniture or the background can be misinterpreted as the person. I can’t show the plot for the highest losses for reasons of data privacy. For instance for false positives of KungfuCrane, a right angled object in the background next to the person might be seen as a hand in the crane pose.

Gathering Data

Given the small number and low quality of images in the initial dataset, it was crucial that we got more data from wherever we can, which means online and ourselves. So we went around mass downloading images of the poses using Bing Image API and some other tools but it required quite a bit of data cleaning since most images were not what we wanted.

There was also the problem of new data being introduced not being similar to the original data and making the performance worse. So we had to set a strict criteria of the new images being real life, not being in a group, not having too many words and not having just white backgrounds. We also took our own photos using our own group members at different angles and pose variations.

Once all that was done, the size of the resultant dataset was huge because of how high resolution the new images can be, so I resized them into 300x300 jpg images using Pillow and the size problem was solved with each image becoming around 25 KB.

Even with all this, our dataset size remained relatively small, at around 1500 images. But our accuracy improved from 72% to 78% on the validation set. This was probably due to the fact that we covered a lot more variations in poses at different angles that were similar to the validation set.

Pose Estimation

At this point, I’ve tried the usual methods of improving results and so I wanted some new novel method to try out, in case some other teams did it as well (which many did). Human pose estimation is quite a developed area in Computer Vision and it played quite well into our dataset since most images had only one person in it doing a unique pose.

I tried out OpenPose on the images but couldn’t get it to work well, so I ended up going with AlphaPose instead. It worked rather well visually, most of the skeleton diagrams were drawn accurately and sometimes too well until it detected reflections as well. Again, for reasons of data privacy, I am unable to show the images. AlphaPose also provided a list of coordinates for each keypoint in the skeleton as seen below:

My original idea was to use the superimposed images (like the one on the right) to train the CNN (since it has already highlighted the figure for the model and so it would be easier to identify the pose) and then using the coordinates output to compute a vector of L2 distances from an anchor point (like the neck), feeding it into a tabular learner and finally to use a very simple average ensemble to get the predictions.

But the CNN showed marginal improvements for accuracy compared to just a normal image dataset which could be attributed to random chance and the output coordinates from AlphaPose were quite inaccurate for certain points which can skew the vector greatly. Given more time and more mathematical knowledge, I probably could have made it work since the confidence level for the coordinates was given as well. There was also the concern about time since it took around 4 seconds per image and the test set could have up to 2000 images. So this idea was scrapped.

Other teams that made it work used DensePose instead and made it part of their data pipeline for image classification, without the coordinates vector part.

Generating Data — Background Replacement

The one thing that makes this challenge unique was that the images for all test sets and validation sets were taken in the same premises with a limited group of people so the variance in images wasn’t as high as public datasets. That also meant that certain locations popped up occasionally with the same limited group of people in it. There was also the problem observed earlier about the background interfering with the inference.

Since pose estimation did not work out well, I decided to go to another avenue instead, of semantic image segmentation. The initial idea was to crop out the humans from the images entirely and train the model based on that, but once again the concern about time constraints in the final test set surfaced since image segmentation would have to be done on those images as well.

The workaround I came up with was to come up with several commonly seen backgrounds in the original dataset and then paste the image-segmented human figures into each background. This would effectively increase the amount of images by a multiple of however many backgrounds there were. In theory it would also reduce the inference errors due to the background.

Semantic image segmentation was done using ImageAI’s library and pretrained models. The backgrounds were sourced from the images themselves. The humans inside were removed and the inpainting was done using Pluralistic Inpainting although that did not give a good inpainting since the pretrained model for it was training on building exteriors and so I had to Photoshop them as well.

With all of this, the expanded dataset size was around 9000 images.

More Data Augmentation and Training

Some more augmentations were added in, specifically cut-out and wrapping of the images. At this stage, changing the image size was no longer a viable option since it took around 3–4 minutes an epoch and changing the size would mean a total run time of around 3 hours, so images were resized to 256x256 at training time instead. Mixup data augmentation was also introduced at this stage.

Efficientnets

Just a few days before the competition began, the Efficientnet paper dropped and the results on Imagenet and transfer learning were astounding to say the least. Naturally, many AI practitioners sought to replicate the model for the different frameworks and luckily, PyTorch and FastAI had quite a few very talented individuals working on it. I was able to use one such implementation to train, in addition to the already available resnets and densenets.

Efficientnet-B3 was the largest efficientnet model with pretrained weights available at that time so that was what I used to train on my dataset. It initially showed equivalent results with resnet50 for the original dataset although it trained in around half the time (living up to the paper’s claims somewhat and its name). I probably couldn’t match the paper’s performance because of how arbitiary a lot of the hyperparameters were for my version of the implementation but still, it worked rather nicely.

When I used resnet50 on the expanded dataset, the accuracy went up to 80% which I thought was probably the upper cap. But when Efficientnet-B3 was used, the accuracy went all the way to 86% which was honestly a shocker. I believe that the rise in accuracy was not only because of Efficientnet or using larger models, but also because of the increase in the number of images (albeit very similar images) allowing for a lot of random data augmentations to take place.

Now I was certain that increasing the data using my strategy worked and that bigger/deeper architectures means better performance on this dataset.

Further Tweaks and Final Training

We added some more images and expanded our final dataset to 11000 images. Accuracy was eventually capped at 88% for Efficientnets. So instead of just Efficientnets, I went back to resnets and densenets and used resnet152 and densenet161 for training as well, both of which gave an accuracy of 90–91%. At this point, I was scared of going any higher in case of overfitting, so the models were done.

Ensemble Voting

Taking the earlier scrapped idea of ensembling a tabular learner and a CNN, I decided to just ensemble several CNNs instead. I trained 20 models of resnets, densenets and efficientnets on different datasets (original, expanded and final) and did up a very simple average ensemble and weight average ensemble.

The average ensemble summed up the probabilities from each model for each image for each class, took the mean and then argmax-ed to get the final prediction. The weight average ensemble weighted each model using its accuracy on the validation set instead of just the mean.

The results improved further for both ensembles from 91% to 92%.

Competition Day

Test Set 1

The first test set gave around 90% accuracy for each individual model, with the ones trained on the original and expanded datasets doing worse. So for the ensemble, we cut off those that scored below 90% individually. That left us with around 5 models for the ensemble. Using the 5 models ensemble, the average ensemble gave a wonderful 95% accuracy, which was way above the other teams.

The Curveball

The curveball wasn’t that much of a curveball. It was just the addition of 4 new classes so we got to work immediately to start taking new photos, finding new images off the web and doing background replacement. That resulted in a curveball dataset of 15 classes and 15000 images. There was a ‘trial test set’ also released that we can try out our model on which wouldn’t be counted in the final grading.

At that point, we had 2 hours left to train the models and each epoch was taking 7–8 minutes and we had 4 laptops which means 8 instances of training (using normal and incognito tabs). Using pretrained imagenet models were giving 65–75% accuracy which wasn’t what we wanted at all. Some teams played a fool and went to hand label the trial test set and screwed up the leaderboard with scores of 99.78% and such which just added to the pressure.

What I did in the end was to use the previously trained model on the 11 classes and retrain it on the 15 classes by changing the classification layer manually. Turns out, that was all we needed to do. The best result of 85% was obtained in just 1 epoch, which I guess means all the feature maps for the humans were already optimised and the new classes did not need to change the feature maps too significantly.

The final ensemble of two models (both resnet152s) gave 87.75% accuracy on the trial test set, which was the highest accuracy without hand labelling.

As for the final test set with the 30 minutes to submit, the two model ensemble gave an accuracy of 85.5%.

After that it was just pitching of our strategies to the judges on the next day and soon after that we were notified that we were the champions.

Things I Wanted To Try

1. GANs to generate new data, but time limitations and I’ve never worked with GANs before. Would data being generated from a GAN even count as valid data for training since training data is used to generate the new data?
2. An actual weight ensemble learner layer for the different models (inspired by one of the groups), but this means an additional learner to train after training each model, and it can’t fit into the pipeline of a single submission
3. Taking snapshots/slices of a video for more images (inspired by one of the groups)
4. AutoAugment Policies, but of course time and memory limits.
5. Test Time Augmentation (TTA). TTA on the validation set actually worsened performance, and it will take X times longer for evaluation on the test set but time limits.
6. KNN as another avenue of predictions to put into the ensemble.

Some Thoughts and Disclaimers

There may be some misconceptions in whatever I have claimed above, I’ve just started on AI and more specifically deep learning less than 6 months ago. Not everything I claim might be accurate but at least I was on the right track. At the same time, winning the competition felt great when I realised who I was up against, considering I was the only one with any sort of experience in AI before this. Also, other teams had excellent programmers, those who were about to or had already graduated from university or even PhD students while my team had 3 of us who were only about to enter university and 1 freshman student. It was a very eye opening journey where I learnt a whole lot more about deep learning.