In the years since starting Mantis, we’ve worked on a lot of projects with many organisations. We’re confident that we’ve been able to deliver value on all the projects that we have worked on, not least because our clients tell us so.

But, despite our best efforts and the best efforts of our clients, sometimes the course of a project can be bumpy. AI projects are complex, so too are the organisations that commission them, and it’s sometimes in the overlap of this complexity that issues can arise.

This blog post summarises the lessons we’ve learned from many projects. We wanted to articulate these lessons so that both we at Mantis and the organisations that we work with are aware of the most common issues that can arise in projects, so that we tackle them together in advance.

Broadly speaking there are three common issues that we’ve seen cause projects to go awry:

• Poorly defined project outcomes
• Issues with data
• Lack of ongoing support

Poorly defined project outcomes

Defining the desired outcomes for a project seems like an obvious thing to get right, but at the outset of a project it’s not always clear that the outcomes have been well defined. The most common cause, we find, is not having the right people in the room from the start of the project.

Getting the right people in the room on the client side is critical so that the project actually delivers business value.

We’re used to working closely with technical teams, but sometimes, the business leader — who is often also the budget holder — is not close enough to the work to spot that the technical deliverables do not align with the business value that they want to get from the work.

Ensuring that this doesn’t happen is critical, but it is also difficult, because each organisation has its own structure and politics, and it can be difficult for us to navigate that based on just a few meetings. For this reason, this critical stakeholder management is as much the responsibility of the team commissioning the work, as it is ours to get right, and we need to work together to ensure shared success.

We’ve learned from experience that whilst we may deliver a very successful project from the technical point of view, it will be viewed as a failure more widely if it later fails to deliver business value in the client organisation.

For this reason, we involve a client delivery manager in all our projects. Part of their job is to help to identify key stakeholders, and to advocate for their involvement in the project at an early stage. The client delivery manager also plays an important role in translating important business objectives into technical requirements.

We also aim to involve the engineers who will work on the project as early as possible in scoping it, so that there are no unexpected surprises for them.

Data issues

Complaints about data are an old cliche among data professionals, and Google has the memes to prove it. But the reality is that many projects run into difficulties due to issues with data. These issues typically fall into two broad categories: availability and quality.

Many projects run into difficulties due to issues with data

Data availability

AI models typically need to be “trained” on data relevant to the problem that we’re trying to solve. If that data doesn’t exist, or isn’t available, then it can make it impossible for a project to get off the ground.

The good news is that, as I’ve written about before, in recent years the arrival of Generative AI models like the Large Language Models (LLMs) GPT-4 and Gemini mean that it’s not always necessary to have data in advance. Nonetheless, even if we can leverage these kinds of models (and we may not want to in some cases) we almost always need some data to evaluate the performance of these models.

In some cases, we may need to ask the client to use their subject matter expertise to help us to annotate some data so that we can evaluate our work against their expectations. This annotation can be time consuming and a little onerous, but it is an essential step to ensure that we can meet client expectations.

Sometimes the data we need exists, but getting access to it takes time due to lengthy internal processes. This can be the case with large organisations where many stakeholders are involved in authorising the sharing of data, especially to an external organisation. This all takes time, and if we don’t anticipate this, the project budget can be eaten away by delays.

Data Quality

Not all data is good data, and there is an old adage “rubbish in, rubbish out”, meaning that an AI model is only as good as the data that it is trained on. But assessing the quality of data is hard, especially when the quantities are large. Moreover, data means a lot of things to different people, and data experts who use data for one thing may be unaware of the issues that beset the data when used for something else, like training an AI model.

Mitigating data issues

The best way to mitigate data issues is to investigate the data at the earliest opportunity in advance of starting a project. For this reason we always advocate for a short period of discovery during which we will assess the quality of the data, and it allows both sides to see how easy it will be to get access to the appropriate data. During discovery we’ll also be able to see if we will need to ask the client to annotate some data, so that they can plan for this.

Even with a discovery phase, there may still be unexpected surprises arising as a result of the data. It’s not possible to mitigate all the ‘unknown unknowns’, but it does at least allow us to quantify the ‘known unknowns’ and ensure that these are well communicated from the offset.

Lack of ongoing support

Singing the praises of an ongoing support contract is going to sound like a sales pitch. But honestly, our biggest failures have been in cases where we didn’t advocate strongly enough for providing additional support to follow on at the end of a project.

There are many reasons why it might make sense to budget some extra time to account for unexpected things that come up after the bulk of the work is done. Two that we have seen repeated are: not accounting for the pace of change of the AI sector, and not underestimating how long it will take for our clients to fully take ownership of a piece of work that we deliver.

The pace of change in AI is so rapid now, that many things we considered impossible twelve months ago, are now eminently possible, perhaps even trivial

The pace of change in AI is so rapid now, that many things we considered impossible twelve months ago, are now eminently possible, perhaps even trivial. So budgeting for some “office hours” during which we are available to help keep the project up to date with the latest developments can be a good investment.

Taking full ownership of a piece of work that we deliver to a client can take time. There are lots of complexities to AI projects, and integrating a model into an organisation’s systems can throw up unexpected issues, especially in enterprise organisations or where the team is not used to working with AI.

The worst outcome for everyone is that the value of an otherwise successful project is not fully realised because there are minor unresolved issues after the main contract was concluded, but that could easily have been resolved with a small number of flexible “office hours” during which we are available to help.

Planning for success

At the end of every project we conduct a retrospective both with the client, and internally, and take forward those learnings into future projects. This blog post provides the rationale for why we advocate strongly for:

• Getting the business users in the room early on, and involving them in defining successful project outcomes.
• A discovery period to look into the project in detail, and amongst other things, investigate the data readiness.
• A follow-on “office hours” contract to allow any issues to be addressed after the bulk of the work has been completed.

Why AI projects go wrong, and what we can do about it was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the second part of our series about Generative AI (you find the first post here), we’re looking at the difference between proprietary and open source Large Language Models (LLMs), and under which circumstances it makes sense to choose one over the other. This is something that we’re frequently being asked by clients, and this post represents some of our current thinking.

Some caveats

More so than with part I, we need to caveat everything in this post with the warning that things are moving really really fast, so this post represents our thinking today, but we fully expect it may be outdated tomorrow (in fact we’ve needed to rewrite parts of it multiple times since we wrote the first draft).

The other caveat worth stating is that the only way to access proprietary LLMs is via API, and we’re really comparing apples with oranges by comparing proprietary APIs with open source models. That’s because we don’t really know what is going on behind the proprietary model APIs. We don’t know if we are getting a response from a single model or many models, nor do we know about the post-processing steps or guard rails that are in place.

With open source models that you manage yourself it’s a lot more transparent, we’re getting the ‘warts and all’ output from the model, with all that that implies. Nonetheless, in this blog post, I will refer to the proprietary APIs as models, just to make the text easier to follow.

It’s also worth thinking about what open source actually means. Many of the models we lazily refer to as open source in this post are not truly open source in the sense that they, and the data on which they were trained, are published under a permissive Apache, MIT, or GPL licence. If you’re going to work with these models, you need to understand the licences under which they are published, and the implications that this has for your specific use case.

Finally, the eagle-eyed amongst you will notice the conspicuous absence of a key class of models, namely: phi, Vicuna, Alpaca, WizardLM, and Orca. We’ve deliberately not included these models because they exist in somewhat of a grey area, having been trained on data produced by a large proprietary model (usually GPT4).

With all that said, let’s dive in.

tldr;

As we mentioned in our last post, our position is broadly the following: if you’re prototyping or you want best in class performance, then it probably makes sense to use a proprietary model through an API. If you want control or customisability then it probably makes sense to use an open source model deployed in an environment of your choice.

That being said, let’s step through a series of concerns that frequently come up when we talk to clients about LLMs, and then we’ll dive deeper into more specific models across the proprietary and open source landscapes.

Performance

If your number one concern is that the model provides the absolute cutting edge performance — in terms of how well the model solves your problem — and nothing else will do, then you should use a proprietary model API.

We’ll discuss the relative performance of different models later, but the short answer is that — so far — open source models have not matched the performance of the largest proprietary models, and nor are they likely to in the near future.

Stability and Reliability

Using a proprietary model API you have very few levers to ensure that the model you are using is stable and reliable. As with any third party service, you basically need to accept the capabilities of the API from the provider, and whilst these capabilities are constantly improving, using popular proprietary APIs can be subject to large variations in latency and periods of degraded performance or outages. You may also not have complete flexibility over which version of a model you are using and when that version changes.

If you are building the model for a production use case for which stability and reliability are critical, then it may make sense to use an open source model. You’ll then be in control of the amount of computing resources available to the system, and you’ll be able to optimise this to ensure a balance of cost and stability.

You’ll also be in control of the level of redundancy that you build into the system to ensure that the service is as reliable as it needs to be. None of this is without cost, but for now at least, if you want complete control over these aspects of a deployment, the only way is for you to deploy an open source model in a computing environment of your choice.

Inference Speed

Similar to stability and reliability, when using a proprietary API you need to accept the inference speed as a given; there are no levers for you to pull.

With an open source deployment you can decide how much computing power to allocate — you can scale up or down your GPU size, or bring new devices online to handle the load across the system and optimise the model’s inference speed (and cost) for your use case.

Privacy, Security, and Intellectual Property

If your organisation is particularly concerned about the privacy of your users, the security of your data, and the sensitivity of your intellectual property, an open source solution deployed in an environment which you totally control probably makes sense.

In time, as LLMs become more commoditised it’s likely that we will all come to trust third party providers with our data just as many organisations trust cloud service providers with their data now. But, there will always be a subset of organisations for whom guarantees of third party providers are not enough, and will want to host their own models in their own environment — and in this case an open source deployment makes sense.

Models

The landscape of models is confusing — let’s be honest. There are hundreds of models out there, and more being released each week. It’s a wilderness of Llamas, Vicunas, Alpacas, Falcons, Beagles… and it helps to try to break down this complexity down into something more manageable.

The way we tend to describe the landscape is like a pyramid with a few really high performing models at the top like GPT-4, and the more traditional language models (small by today’s standards) like BERT at the base of the pyramid. For conceptual simplicity, we split the pyramid into four layers which we’ll describe in turn.

“Pinnacle” (GPT4 level) models

The top of the pyramid is dominated by a few proprietary models: OpenAI’s GPT4, Anthropic’s Claude, Google’s Gemini2 and the new Mistral Medium which has recently shown impressive performance on popular benchmarks such as the LMSys chatbot arena.

We don’t try to define what these models are capable of here, but it’s worth looking at the respective homepages for lots of examples of what these impressive models can do (https://openai.com/gpt-4, https://deepmind.google/technologies/gemini/#hands-on, https://www.anthropic.com/news/introducing-claude). Bubeck et al (2023) also give a very thorough deep dive into an early version of GPT4 in Sparks of Artificial General Intelligence: Early experiments with GPT-4, and for a hands-on approach to getting a feel for these models you should experiment with the LMSys chatbot arena.

If you need best in class performance on complicated tasks, then these are the models you should be using, and as of now: all are proprietary.

Mid (GPT3.5) level models

The next group of models we loosely call the GPT3.5 level group. This is the current default model in use on ChatGPT, so its capabilities are likely to be familiar to many. This group includes a mix of proprietary and open source models, such as:

• GPT3.5 (proprietary, 175B?)
• Mixtral 8x7B (open source, 8x7B)
• Llama2 (open source, 70B)
• Palm 2 (proprietary, 340B)
• Claude Instant (proprietary, >130B?)
• Falcon (open source, 180B)

These models all produce impressive results, and are excellent for a range of generative applications such as Question-Answering and summarisation, as well as producing high quality zero and few shot performance on extractive tasks such as text classification and NER.

If you have any of the concerns that we discussed previously such as privacy security, or IP protection, or want to have fine grained control over stability, latency, cost, and performance, but don’t need best in class performance, then one of the open source models from this layer in the pyramid may be a good option.

Lower performance models (~1–20B)

After the mid group, there exists a large pool of mostly open models that are suitable for specific use cases, but lack the general applicability of the mid and high performance models.

These models are good for simple generative use cases, for example easy summarisation where less nuance is required, or Question-Answering over generic topics. These models may also be good candidates for prototyping: to prove out a use case before moving on to a larger more costly model. These models have also shown excellent performance when fine tuned for the specific use case. We’re thinking primary here of:

• LLAMA2 7–13B
• Mistral 7B
• Smaller proprietary models from Google (Gemini Nano), etc.

The motivations for using these models are several, but may include: infrastructure size constraints, inference speed, or compute cost.

Another motivation may be specificity; for example the IGEL family which is a family of models fine tuned specifically for the German language.

Small(!) models

Finally, we have the language models which are no longer considered to be ‘large’, and are much more uniformly open source. Here we’re thinking of smaller transformer models such as BERT and its many variations like RoBERTA.

These models perform very well on well structured (extractive) problems for which there are a lot of annotated data. These open source models excel where size, speed, and cost are a concern. They also overcome the stability and privacy concerns, as we can easily train or fine tune our own models and deploy them wherever they need to be deployed.

It’s these models that we were referring to in the first part of this series when we talked about using LLMs to generate data to train smaller transformer models.

Summing up

In this post we’ve outlined the way we think about the landscape of open and proprietary LLM models. It’s not perfect, but it helps us to frame the conversation with clients around the various capabilities of different models, and whether those models should be proprietary or open source.

We’d love to hear if you’ve found this post helpful or if you disagree! Please reach out to us in the comments below. Equally, if you’re trying to make sense of how your organisation should make use of LLMs, we may be able to help. This kind of strategic LLM consultancy is our fastest growing type of client engagement.

We’re also working on a series of workshops for decision makers and technical experts. We’ll make an announcement about this in the coming weeks, but if this sounds interesting in the meantime, check out the landing page or reach out to us at hi@mantisnlp.com.

In the last blog in this series, we’ll talk about the trends that we see in the development of LLMs and how these trends are once again changing the way we think about space. Stay tuned!

How we’re thinking about Generative AI: Proprietary vs Open Source was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

In our last blog aimed at the not for profit sector, we talked about how we can use Artificial Intelligence (AI) to help measure the impact of interventions. The techniques we talked about in that blog fall under the category of ‘extractive’ AI techniques. In this blog, we’ll focus instead on the latest generation of ‘generative’ AI tools, and how not for profits can make best use of these tools.

What is Generative AI?

Until relatively recently almost all AI was ‘extractive’. This means that: given a collection of data the AI would extract relevant information, for example mentions of people, places, topics, and themes or applying labels to the document, for example MeSH tags.

Generative AI is different. Generative AIs are capable of synthesising high quality content or data, and the release of models such as OpenAI’s ChatGPT has unlocked a whole range of new use cases from convincingly human-like chatbots to synthetic data generation.

We’ll look at three use cases in this blog post that are of particular relevance to not for profits. All involve the use of Large Language Models (LLMs) like ChatGPT: a type of Generative AI model that can be used to understand and synthesise text.

Support for Grants

For grant giving organisations there are a number of use cases for LLMs around the grant application and evaluation process.

The first of these is to help applicants to write grant applications. We’re a long way from LLMs being able to write applications from start to finish — and this may not ever be feasible or desirable. But it is certainly the case that these tools can assist applicants in a number of ways, for example by synthesising drafts of generic text such as the problem background, or suggesting structure for longer free text fields that must be completed as part of the application.

The widespread use of ChatGPT by academics to help write parts of sometimes onerous research grant applications was the subject of a recent nature article. Whilst many applicants are using these tools on their own accord, pro-active grant funding organisations can build generative AI directly into the application process.

Pro-active grant funding organisations can build generative AI directly into the application process.

For example, the application process can be streamlined by creating tools such as chatbots to walk applicants through it, making suggestions about how to answer questions in a way that is helpful to the funder, and providing some early instant feedback to the applicant to help them refine the application before it is seen by a human reviewer. LLMs are also very good at producing summaries and synthesising text from a broad range of supplied sources, and could produce lay summaries and other free text answers based on reference material automatically, or when prompted.

This leads to the other obvious application: evaluating grant applications. There has been some debate recently surrounding the use of LLMs for reviewing academic articles — some positive, some negative.

The short answer is that it looks like LLMs have a place in checking for consistency of argument, clarity and style (all of which could instantly be fed back to the applicant), but probably not for assessing novelty, and providing a deep and nuanced critique of scientific methods. This will likely change over time as more specific LLMs for these tasks are produced.

We can extrapolate this into fields other than science: there will need to be human evaluator, but the LLM can assist by assessing grants for softer criteria, and help the evaluator by summarising or improving clarity of the application.

Finally, generative tools can be used to identify the most relevant reviewer, and provide them with a summary of other similar grants that have previously been funded, saving much time in reviewing databases of funded grants, ensuring that funding aligns with the organisation’s strategy.

generative tools can be used to identify the most relevant reviewer, and provide them with a summary of other similar grants that have previously been funded

In summary, Generative AI can be used to chip away at the grants application process from both sides, helping to make a smoother and quicker process for all concerned.

Dynamic Information Retrieval

One of the benefits of LLMs is that for many use cases they do not need to be trained — the process by which we ‘teach’ a model about a certain topic. Compare this to the WellcomeBertMesh model which we developed with the Wellcome Trust which applies an ontology of tags (Medical Subject Headings) to free text documents such as grant summaries: this model was trained for multiple days and required very specific data to be prepared in advance.

Imagine a different scenario: a not for profit that has sent out a survey, or has a dataset describing the multiple ways its money is being used at the community organisation level. If we want to identify cases where vulnerable adults, or minority groups are mentioned using AI, we would need to collect and annotate data and train a model for the task which would take days. With LLMs we can just ask the model to find the information we’re looking for, and it will probably do a pretty good job out of the box.

With LLMs we can just ask the model to find the information we’re looking for, and it will probably do a pretty good job out of the box

That’s not to say that we can be complacent and assume that the model is right, or that it doesn’t need to be assessed for bias — it does — but that is the same as any model. What LLMs offer us is a way to get started very quickly, and for generic use cases like this, it is likely to perform very favourably against trained models, or simpler naive approaches such as keyword searches.

Marketing and Education

The final use case for generative AI is probably the most popular across all industries at the moment: content creation. This has two obvious use cases for Not for Profits: marketing, and education.

For marketing, GenAI tools have proven to be very adept at producing marketing content to help get the message of the organisation out to a wider audience via social media. This is nothing new — many businesses are experimenting with LLMs for this use case.

A more specific use case for not for profits is education. Many organisations have a stated aim of increasing awareness of an issue, for example a medical condition, or a humanitarian situation, and others wish to educate, e.g. providing educational materials for sufferers of rare diseases.

In this educational use case LLMs can be used to provide content targeted to the user through mediums that the user is familiar with, such as text message, Whatsapp, or even synthesised phone calls. For example, chatbots can be built on top of the organisation’s existing knowledge base of blog posts and other authoritative material to create a conversational interface which allows users to get answers to specific questions asked in their own words.

chatbots can be built on top of the organisation’s existing knowledge base of blog posts and other authoritative material to create a conversational interface which allows users to ask specific questions in their own words

There are dangers, of course, in ensuring accuracy and appropriateness of the messaging, but these are being overcome in a number of ways as the technology matures, such as the use of Retrieval Augmented Generation (RAG) systems.

This has been a brief tour of possible use cases for Large Language Models, which are a type of Generative AI, in the not for profit sector. This is just one facet of Generative AI, there are likely to be many more use cases presenting themselves, for example as large vision models (LVMs) — models which can process and interpret images and videos — become more widely available and useful.

We’ll continue this series by looking at more concrete examples in the coming weeks. In the meantime, if anything in this post has chimed with you or what your organisation is trying to do, feel free to get in touch for a chat at hi@mantisnlp.com.

If you’re a decision maker and you’re wondering how you can plan for Generative AI at a strategic level in your organisation, you might also be interested in our workshop package Generative AI for Decision Makers, which will equip you with the tools you need to speak authoritatively about AI, and make informed decisions about how to integrate it into your organisation.

What role can Generative Artificial Intelligence play for not for profits? was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

Organisations that fund programs, or perform or fund research are naturally interested in the impact their funding is having, but tracking and measuring impact can be extremely challenging.

Some of the reasons for this are that Impact can mean different things for different grants and organisations, and the full impact of funding may not be apparent for a long time after the funding is distributed.

At Mantis, we specialise in Natural Language Processing (NLP) — the field of AI concerned with processing human language and documents. So, when we work with NGOs to help measure their impact it tends to be using written documents such as policy documents, mainstream media, academic publications, or some other type of document entirely.

In this blog we’ll look at three ways that Artificial Intelligence (AI) can be used to measure the impact of funding, based on lessons we’ve learned working with some of the world’s biggest grant funders, such as The Wellcome Trust.

1. Finding Impactful needles in haystacks

The first step of looking for evidence of impact in written sources is to collect candidate documents that might point to the impact a grant has had. There are a number of ways we might go about this, but we usually end up with a collection of many thousands to millions of candidate documents, of which usually only a proportion are relevant to us.

The naive approach to finding the relevant documents is to use keyword searches, but this usually results in a lot of false positives: documents which contain relevant keywords, but are not actually relevant.

Instead, we can use AI as a sophisticated filter to find documents that really are relevant, including those that don’t contain any relevant keywords. We can do this by using a technique called semantic similarity. This is where we compare the meaning of documents to see if the overall meaning is similar to what we are looking for.

Another approach is to use an AI model to categorise the documents by applying “tags”. This can be a simple yes/no indicating whether the document is relevant, or it could be a comprehensive system of tags such as Medical Subject Heading tags. In either case the aim is simply to identify relevant documents based on the tags the AI applies, so that we can progress to the next stage with a smaller subset of all the documents that we have collected.

2. This isn’t the Jane Smith you’re looking for

Once we have collected and filtered the documents that are relevant to our subject of interest, we can look at the documents in more detail. The most straightforward evidence of impact is when the work that is funded is mentioned or cited.

A mention could be something in a mainstream media article such as:

“Researchers at the University of Cambridge discovered a groundbreaking new vaccine for Malaria…Dr. Jane Smith explained the vaccine was developed with mRNA technology…the research was funded by the Fictional institute”.

As before, we can get some of the way just by searching for keywords, but we will run into the same issues we highlighted already, and it assumes that we know everything we want to search for in advance, which may not be the case.

Instead, we can use an AI to both identify the sorts of things we want to find: techniques, collaborators, institutions, locations, etc. and to ensure that the mention is referring to our “Jane Smith”, not another person who just happens to have the same name. The Wellcome Trust have written a blog post about this.

3. Not all citations are equal

Like mentions, academic citations are an important way of measuring the reach of research that was funded by a grant. In academic journals, citations and their accompanying references are generally well structured, making analysis simple. In fact, academics already generate all sorts of metrics to measure impact of academic publications — for example journal Impact Factor, a measure of how impactful an academic journal is based on how often it is cited.

Not everyone is as diligent when it comes to references though, and in some documents, particularly policy documents created by national governments, the citations can be poorly structured making it hard to process automatically.

AI can help us here to detect citations and references, even if they are badly formatted, and extract the individual components such as author names, title, publication, etc., making the process of linking the reference to the original research article trivial. We previously worked with The Wellcome Trust on a model to do exactly this, which is available openly.

The other way that AI can help here is to confirm that a mention or citation is indicative of real impact. Consider for example:

Dr. Jane Smith was forced to retract publications relating to her research on mRNA vaccines…

Or

The project funded by the Fictional Institute was considered to have no impact on the prevalence of Malaria in The Fictional Republic of Fictionia.

Neither of these examples are indicative of the positive impact that we are usually asked to measure, but AI can help us to detect and discard them — or indeed report them.

To conclude

To round this up, we’ve gone through some of the ways that AI can be helpful when trying to measure the impact of grant funding. We’ve just scratched the surface here, but these are some of the common tasks that we find ourselves performing for NGOs again and again.

These are all examples of “extractive” AIs which extract information from data. In a future blog post we’ll look at the ways the latest generative AI can be used by NGOs to measure impact.

In the meantime, if you have any questions or comments about this post, or want to discuss other ways that AI can help to track impact, please feel free to get in with us at hi@mantisnlp.com.

AI for grant funding NGOs: Three ways Artificial Intelligence can help measure impact was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s no exaggeration to say that the world has changed in the last year thanks to the developments in Generative Artificial Intelligence (AI). Models such as GPT3 and Dall･E 2 changed our expectations about what was possible from AI, much faster than anticipated by experts in the field (such as Geoffrey Hinton).

At Mantis, we’ve seen a lot of requests from clients, old and new, about how they should be using Generative AI such as Large Language Models (LLMs) like ChatGPT. We’re in the privileged position of having spoken to lots of companies about how they are making use of LLMs, and we’re helping a number of organizations to map out their strategy for how to integrate LLMs into their businesses.

This is the first in a series of blogs that describes our current thinking about LLMs based on our experience working with clients over the last year. This first post is about when we should use them, and what a typical LLM journey looks like for us — so far!

When should you not use an LLM?

Recent developments have put cutting edge models in the hands of anyone with an internet connection, and this has sparked many people’s imaginations about the ways that AI can be applied to their business.

Quite often however, we find ourselves counseling against using LLMs for a number of use cases because there are cheaper, faster, and better performing alternatives. When all you have is a model like ChatGPT, all problems start to look like LLM problems, even if it is the computational equivalent of hammering a nail with a very costly microscope.

LLMs are huge models that are relatively expensive, slow, and can have a large environmental footprint. There are also some disadvantages to relying on LLMs provided by third parties related to the reliability and stability of these models, for example it is well documented that the performance of models such as GPT3.5 and 4 changes over time, and this is something over which you will have little control.

Consequently, there are a subset of tasks for which you probably shouldn’t use an LLM in a business critical application (if you’re prototyping, it’s a different story: see below). These include tasks such as sentiment analysis, text classification, and Named Entity Recognition (NER) for which high performing, smaller models existed before the arrival of LLMs. These problems are well structured, extractive AI tasks that smaller models can do really well, and given the right data will outperform an LLM in speed, cost, reliability, and performance.

When should you use an LLM?

We think that there are two concrete use cases for which LLMs are ideal.

Prototyping

LLMs are ideal for prototyping. That is, quickly testing an idea that would traditionally have required:

• AI development expertise
• Relevant data to train a model
• Time

LLMs are able to do so-called: zero and few-shot prediction: meaning that they are able to produce an output with either no, or few, examples of what you would like the output to be. This includes extractive AI tasks for which we would expect smaller, more specialist models to perform better.

LLMs are a great way to get started, prove a use case with minimum development time, and then help you to transition to a smaller, better, more specialist model for a production use case (more on this later).

True Generative use cases

True to the name Generative AI, LLMs excel at producing high quality text outputs. LLMs are unparalleled for use cases such as content generation (e.g. copywriting, or writing research grants), and summarisation, whilst anyone who has used ChatGPT will have seen that LLMs can also power impressively human-sounding chatbots.

In recent months, particular interest has built around chatbots and assistants — and more generically Question-Answering systems that are powered by Retrieval Augmented Generation (RAG) systems, utilizing LLMs. We’ve posted about RAG systems before and even produced a demonstration using the content from the UK Government’s website GOV.UK. These RAG systems allow for the latest information to be made available to the model, and can also help to reduce one of the most important failure modes of LLMs: hallucinations.

Another impressive use case that we have seen in recent weeks, at which LLMs do very well, is for reviewing, for example as part of the scientific peer review process. Whilst LLMs are unlikely to fully replace human reviewers in this highly technical field, they can provide helpful suggestions to human reviewers around topics such as clarity of presentation.

The LLM Journey

OK, so you’ve decided you are going to use an LLM, now what’s the journey to getting the performance you want? Here we outline the steps we take to tuning LLM performance.

Zero-shot and Few-shot prediction

We’ve touched already on the use of LLMs for zero and few shot prediction. Our first steps are simply to prompt the LLM to produce the output we want: a so-called zero-shot prompt, meaning that the model doesn’t see in advance an example of what you want it to output.

If we don’t get the performance we want, we can likely improve performance by providing the model with a few examples of what we want: so called few-shot prediction. This is a simple step that can be implemented very easily.

Prompt optimisation

The quality of the prompt can greatly affect the quality of the output of an LLM, and so if you want to improve the performance of the LLM, you should spend some time optimizing the prompt. Ideally you would set up an automated way of evaluating the effect of manual changes to your prompt (see some examples in our blog), or better still: automate the entire process of improving the prompt. This process, called Automated Prompt Engineering (APE), is the subject of another of our blog series, in which we outline how you can implement this kind of system.

Data generation

If the use case is not generative, and up until now we’ve been using an LLM for prototyping, this is when we’d look to transition to a smaller, more performant model. We’ll need data to train these models on data, but LLMs can help us in two ways.

Firstly, we can ask an LLM to annotate real data that we already have, but that has not been annotated yet. Even if we don’t fully trust the model’s annotations, it may be much faster to annotate with an LLM, and then have a human correct the annotations (or a subset of them) rather than relying totally on human annotators.

In some cases we can rely on the model to generate entirely synthetic data, with or without labels. Again, this may need some human intervention to ensure quality, or at the very least evaluation.

Once we have sufficient data, we can train a smaller model for example using SetFit or a custom transformer model if we have more data available (see our recipe for this approach). We can then involve the LLM in an iterative process of improving the data, and retraining the model, perhaps incorporating the LLM into an active learning process, so that we can iteratively improve the quality of the training data.

Be aware though that some LLMs have licenses which prohibit their use for data generation in this way.

Customizing the LLM

The last step in the journey is to customize the LLM itself. There’s a couple of ways that we might do this.

The first is through a process called supervised fine tuning (SFT), where we continue to train (fine tune) the model with data from our domain of interest. Generic LLMs are trained on a wide variety of text sources (such as Wikipedia, Reddit, etc.), but for fine tuning we would choose text specific to our domain of interest, such as finance: for example BloombergGPT: a model fine tuned specifically for the finance domain.

Another option is to tune the model to align better to our preferences. There are a number of ways we might do this: for instance we can ask the model to generate a response to a query multiple times, and then pick our favorite response (Direct Preference Optimisation). Another approach is to train a second model to recognise the kind of responses that we prefer, and then use this model to adjust the outputs of the LLM, so-called: Reinforcement Learning through Human Feedback (RLHF).

This step makes sense when we want to adjust the tone of voice in a chat application (and prompt engineering was not enough), or if we want the model to follow instructions or perform better on our task. We leave this step until last because it is also the most expensive and time consuming, and requires us to have data with which to fine tune the model.

We’ve published a series of four blog posts on the topic of supervised fine tuning with our friends at Argilla, the first of which you can find here.

Next time…

In this article we’ve outlined our thinking about when you should use an LLM, and our approach to tuning the performance of the LLM. In the next post in this series, we’ll introduce the LLM pyramid which describes our thinking about the capabilities of large and small models, both proprietary and open source, to help answer a very common question we receive: should you use a proprietary LLM or an open source one.

As ever, if anything I’ve talked about here chimes with what you’re doing in your organisation and you’d like to get some help with your Generative AI journey, please feel free to reach out to us at hi@mantisnlp.com.

How we think about Generative AI was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Economist recently published a piece entitled Your employer is (probably) unprepared for artificial intelligence (AI). It’s a thought provoking article that considers how technologies diffuse through an economy, and the negative consequences for firms of failing to keep up with technological advances.

Most striking for me were two quoted statistics: according to official statistics, only 1.6% of US firms made use of Machine Learning (an important sub-type of AI) in 2020; and: 40% of small businesses in the US are not interested in AI tools.

These are remarkable statistics given the benefits that AI is expected to bring. In June, Mckinsey published a report stating that modern Generative AI, like ChatGPT, could add as much as $4.4 trillion to the global economy. If that sounds like a lot, it is: for reference, the entire UK economy — the sixth largest in the world — was $3.1 trillion in 2021.

Hopefully this isn’t news to you, and your organisation has already got a comprehensive AI strategy in place, but if not, this blog post is for you.

It’s never been easier to get started

Thankfully, it’s never been easier for an organisation to get started with AI. One important way to do this is to execute a successful Pilot Project which can demonstrate the benefits to the organisation, and help to build momentum behind the idea of using AI.

Stanford Professor, and AI luminary Andrew Ng in his AI Transformation Playbook, advocates Pilot Projects as the first step for companies wishing to make an AI transformation.

In this blog post I’ll talk about what makes a good Pilot Project, and three simple rules for identifying one in your organisation.

What is a Pilot Project?

A Pilot Project is a project undertaken within an organisation to prove the value of AI. The idea is to choose a problem that is big enough that solving it would be important to the business, but not so big that demonstrable progress cannot be made within a 3–6 months. It should also be both technically feasible and have a measurable impact.

Now is not the time to shoot for the singularity, the objective is to get something important done, within the timescale, demonstrate value, and unlock budget for the next step in your organisation’s AI transformation.

There should be data

It may be necessary to train a new AI model during the pilot project, and data is essential to this process. Exactly what data means differs for every project, but if you want to show progress on a problem in 3–6 months, then the pilot project should tackle a problem for which some relevant data already exists, or can easily be produced.

It’s possible to generate synthetic data, or work with internal or external partners to annotate data, but it takes time to get these projects off the ground, and the fact that the data do not already exist may mean that the problem is not yet amenable to an AI approach in a short timescale.

The exception to this rule is that the new wave of Generative AI models such as OpenAI’s ChatGPT, Google’s Bard, and Anthropic’s Claude, are capable of doing few and even zero shot prediction. This means that you may need very little or no data at all to get started with these models. However, they may not perform well on problems that are very specific to your business, or are not related to natural language processing (NLP) which is the field of AI concerned with understanding and generating language

Moreover, even if you can use these so-called Large Language Models (LLMs), data may still be required to make a meaningful evaluation of the outputs, and there are trade-offs to working with these new models that need to be evaluated carefully, such as: high cost, lack of stability, and privacy concerns. And in the long run, a bespoke model trained on your organisation’s data is likely to perform better.

It should be easy to measure impact

At the end of the pilot project, you want to be in a position to demonstrate to your stakeholders that the project was successfully completed with a measurable impact. Measuring the impact of AI projects can be hard if we don’t have existing KPIs or metrics to act as a baseline against which we can evaluate performance. Let’s look at an example.

When I worked at the Government Digital Service (the part of the UK Government that manages the GOV.UK website) we did a pilot project to help a team there to understand feedback received from visitors to GOV.UK. There was already a team of people working on the problem, and we knew that it took a person in that team about forty-five minutes to process one hundred pieces of feedback. This meant that we could measure performance of our project in terms of hours saved per week for the team.

This is a good metric that was easy to understand, but it wasn’t a great metric because it didn’t speak to important business objectives of the organisation. What GDS really cared about was how to make content easier to find, and how to optimise the user experience, and our metric didn’t speak to these objectives at all.

Choose mosaics over monoliths

Another experience I had working in the UK Government is that our stakeholders from other Government departments often wanted to shoot for the moon, and try to build a monolithic solution that could solve all of their thorniest problems at once.

Aside from being a poor choice for a pilot project in terms of technical feasibility and timescales, the route to being an AI enabled organisation is not (yet) about building your own HAL9000. Instead, mature AI enabled organisations tend to incorporate AI as a mosaic of specialised tools and services that solve specific business problems.

This can be an important message to put across to stakeholders, and can help to manage expectations, especially in the early stages of an organisation’s AI transformation. It’s also important to remember that a pilot project is as much about the organisation learning how to execute these projects as it is about the project itself. AI projects often involve more uncertainty than traditional software engineering projects, and learning how to deal with this uncertainty is a key outcome in itself.

Who should execute the pilot project?

If you’re lucky enough to have internal capacity to take on your first few pilot projects, then that is ideal. It’s often the case however that organisations want to do a pilot project to prove out a potential use case for AI, before allocating the budget for building an internal team. In this case it makes sense to work with an agency on contract to get started.

You’re probably thinking that this is exactly what the founder of an AI consultancy would say, and you’re right, so let me restrict my advice to the following. It can be hard to select one agency over another when you don’t have specific expertise in the field, so take your time to select a partner who has the right experience for your problem.

I recommend spending some time in ‘discovery’ during which the agency will look to understand your problem better through a series of meetings with the key stakeholders, and looking at samples of your data. Ideally this discovery work would be discrete, and result in a report detailing the findings and options for a route forward. Should you decide that the agency isn’t the right fit for your organisation, you can use the report to scope work with another agency without having to start from scratch.

What’s next?

It’s a good idea to take on a few pilot projects to get a good understanding of the lifecycle of an AI project, but once enough momentum has been built within the organisation, you can progress on to the next phase of AI transformation. Andrew Ng’s recommendation is to build internal AI capability, and this makes sense as you’ll want to start to lay out a long term strategy for the organisation with full awareness of the organisation’s goals and objectives for the coming year, and it may be easier to achieve this with an internal team.

To conclude

Pilot projects are a fantastic way to take your organisation’s first steps into the world of AI. Keep them small and measurable, and use it as an opportunity to learn how AI projects differ from the projects your organisation is used to delivering, and the skills and profiles needed to deliver them.

And of course, it goes without saying; if you are looking to complete a pilot project Mantis is here to help. Drop us a line at hi@mantisnlp.com.

How not to get left behind by AI was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

Spoiler: it lets you deploy an Inference Endpoint in one line of code:

What is hugie?

Hugie is a small, Python based command line interface (CLI) for managing Inference Endpoints via the API. We started using Inference Endpoints to support a production use case, and we needed an easy way to deploy Inference Endpoints as part of our CI/CD pipeline powered by GitHub Actions.

How do I pronounce it?

Huggy, like a hug but a bit more? From Hugging Face Inference Endpoints. With a little reference to Huggy Bear Brown.

How do I get it?

You can install hugie directly with pip:

pip install hugie

How do I create an endpoint?

1. Export your Hugging Face API token into an environment variable called HUGGINGFACE_READ_TOKEN, e.g.:

export HUGGINGFACE_READ_TOKEN=api_XXXXXXXX

2. Define a config file for the endpoint you want to create. For this we can just download one for a small translation model with:

curl -O https://raw.githubusercontent.com/MantisAI/hugie/main/examples/development.json

This config contains information about the model to be deployed, the size of the instance to deploy it on, the region, replicas, etc. We’ll look at the config in more detail later.

3. Create the endpoint:

$ hugie endpoint create development.json
>>> 
Endpoint development created successfully on aws using t5-small

How do I check the endpoint status?

We can check the status of all of our endpoints with the endpoint list command:

$ hugie endpoint list 
>>>
Name          State          Model      Revision   Url 
-----------   ------------   --------   --------   ----
development   initializing   t5-small   main       None

Once the endpoint is deployed, the URL field will be populated, it will look something like this:

https://abcdefghijklmnop.us-east-1.aws.endpoints.huggingface.cloud

We can also use the watch command to keep a constant watch on what endpoints we have running with:

$ watch hugie endpoint list

How do I query the endpoint?

If you want to test that the endpoint works, you can use the endpoint test command:

$ hugie endpoint test development “This is a test”
>>>
[{‘translation_text’: ‘Dies ist ein Test’}]

Notice that we pass the instance name (defined in the config) and a string that we want to do inference on. You can also pass a path to a JSON file containing multiple queries with the — input-file <PATH> option.

For production use cases, we query the API using an HTTP POST request. We need to pass a Bearer token for authentication. Here’s how to do it using curl:

curl https://abcdefghijklmnop.us-east-1.aws.endpoints.huggingface.cloud \
    -X POST -d ‘{“inputs”: “This is a test”}’ \
    -H “Authorization: Bearer ${HUGGINGFACE_READ_TOKEN}” \
    -H “Content-Type: application/json”
>>>
[{“translation_text”:”Dies ist ein Test”}]

Cool, so what’s in the config file?

The config files lets you set parameters such as the cloud vendor you want to use, the region where the endpoint will be deployed, the size of the endpoint, etc. In our example we create a small endpoint in the AWS us-east-1 region running a translation model t5-small. It’s default behavior is to translate texts from English into German:

{
  "accountId": null,
  "compute": {
    "accelerator": "cpu",
    "instanceSize": "small",
    "instanceType": "c6i",
    "scaling": {
      "maxReplica": 1,
      "minReplica": 1
    }
  },
  "model": {
    "framework": "custom",
    "image": {
      "huggingface": {}
    },
    "repository": "t5-small",
    "revision": "main",
    "task": "translation"
  },
  "name": "development",
  "provider": {
    "region": "us-east-1",
    "vendor": "aws"
  },
  "type": "protected"
}

You can read more about the config settings in the Inference Endpoint docs.

What’s coming next?

• Tighter integration with the Hugging Face Inference Endpoints API so that you have access to all the functionality of the API.
• A more streamlined flow to allow you to create endpoints on the fly, and some functionality around setting things like provider, and type defaults, to reduce the amount of work you need to do to deploy an endpoint.

👋Get in touch!

Hugie is an open source project. We welcome your contributions in comments, issues, and PRs! And if you want help with deploying transformers into production where you work, get in touch with us at hi@mantisnlp.com, or drop us a line.

Deploy a cutting edge machine learning model with one line of code was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ve been experimenting a lot with ChatGPT and other Large Language Models (LLMs). In this post we talk about a demonstration application that we have built which layers ChatGPT on top of the UK Government website GOV.UK, allowing you to query the website in natural language.

This post is about what we have built, why we did it, and how it performs better than vanilla ChatGPT.

tl;dr

Sign up at https://chat.mantisnlp.com and have a play. To improve the application, we will record the conversations, so please don’t share any sensitive information with the chat bot.

What have we built?

We’ve built a layer on top of ChatGPT that improves its performance when asking questions related to GOV.UK. You can ask the chat bot for answers to questions that you would normally need to search on GOV.UK. For example if we want to know how to get a UK driving license:

The chat bot will return the relevant information along with links to the pages where it has found the information. We can follow up with more questions that build on the first one.

Why have we built this?

Over the last year or so, we’ve had several requests from companies who wanted a chat bot that can be used to query internal company documents in natural language.

Large companies especially, that have huge volumes of knowledge, want systems that can make that data more easily searchable and accessible to their employees.

Imagine for example that you could ask a chat bot, in natural language, specific questions about your company’s past projects, business processes, or corporate intranet and get answers back immediately as if you were messaging a friend.

We’ve used GOV.UK as a demonstration because the content is well known (at least in the UK), is well structured (and therefore easy to verify), and is published under the permissible Open Government License.

How is it better than ChatGPT?

If you’ve experimented with ChatGPT (or other large language models LLMs) you may have come up against some limitations. OpenAI are transparent about these, they state that outputs from ChatGPT can be both out of date, and untrustworthy (hallucinated). Obviously this makes it difficult to justify using a chat bot like ChatGPT in a production setting where accuracy is important.

We’ve addressed these problems by anchoring the responses of our chat bot to a current version of GOV.UK, so that it provides more up-to-date information, and reduces the likelihood that the chat bot responds with something that is not accurate. We also provide links to the source of the information that the chat bot provides.

We can see this in action if we ask ChatGPT and our demo the same question: “Am I eligible for the UK Winter Fuel Payment”.

If we ask this question of ChatGPT, we get the answer (truncated for brevity):

This sounds plausible, but is not correct. According to GOV.UK, (https://www.gov.uk/winter-fuel-payment):

If you were born before 26 September 1956 you could get between £250 and £600 to help you pay your heating bills. This is known as a ‘Winter Fuel Payment’.

If we ask the same question of our chat bot, satisfyingly, we get the correct answer:

What next?

You can try our chat bot at https://chat.mantisnlp.com. You’ll need to request access, and then we will create you an account. Make sure that you allow us to send you updates when you sign up (and marketing if you are interested) so that you can stay up to date as we make changes to the demo.

We’ve also got a more technical blog post planned looking at what we have done in more detail, and some insights from early user tests. Keep an eye out for it over the next couple of weeks.

And of course, if you are interested in deploying something similar for your own documents, please reach out to us at hi@mantisnlp.com for a chat.

ChatGPT for GOV.UK was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

• Deploy (almost) any model on Hugging Face Hub
• To any cloud (AWS, and Azure, GCP on the way)
• On a range of instance types (including GPU)

We’re switching some of our Machine Learning (ML) models that do inference on a CPU to this new service. This blog is about why, and why you might also want to consider it.

What were we doing?

The models that we have switched over to Inference Endpoints were previously managed internally and were running on AWS Elastic Container Service (ECS) backed by AWS Fargate. This gives you a serverless cluster which can run container based tasks. Our process was as follows:

• Train model on a GPU instance (provisioned by CML, trained with transformers)
• Upload to Hugging Face Hub
• Build API to serve model (FastAPI)
• Wrap API in container (Docker)
• Upload container to AWS Elastic Container Repository (ECR)
• Deploy model to ECS Cluster

Now, you can reasonably argue that ECS was not the best approach to serving ML models, but it served us up until now, and also allowed ML models to sit alongside other container based services, so it reduced cognitive load.

What do we do now?

With Inference Endpoints, our flow looks like this:

• Train model on a GPU instance (provisioned by CML, trained with transformers)
• Upload to Hugging Face Hub
• Deploy using Hugging Face Inference Endpoints.

So this is significantly easier. We could also use another managed service such as SageMaker, Seldon, or Bento ML, etc., but since we are already uploading our model to Hugging Face hub to act as a model registry, and we’re pretty invested in Hugging Face’s other tools (like transformers, and AutoTrain) using Inference Endpoints makes a lot of sense for us.

What about Latency and Stability?

Before switching to Inference Endpoints we tested different CPU endpoints types using ab.

For ECS we didn’t test so extensively, but we know that a large container had a latency of about ~200ms from an instance in the same region. The tests we did for Inference Endpoints we based on text classification model fine tuned on RoBERTa with the following test parameters:

• Requester region: eu-east-1
• Requester instance size: t3-medium
• Inference endpoint region: eu-east-1
• Endpoint Replicas: 1
• Concurrent connections: 1
• Requests: 1000 (1000 requests in 1–2 minutes even from a single connection would represent very heavy use for this particular application)

The following table shows latency (ms ± standard deviation and time to complete test in seconds) for four Intel Ice Lake equipped CPU endpoints.

+--------+--------------+-------------+----------+-----------------+
|  size  | vCPU (cores) | Memory (GB) | ECS (ms) |     🤗 (ms)     |
+--------+--------------+-------------+----------+-----------------+
| small  |            1 |           2 | _        | ~ 296           |
| medium |            2 |           4 | _        | 156 ± 51 (158s) |
| large  |            4 |           8 | ~200     | 80 ± 30 (80s)   |
| xlarge |            8 |          16 | _        | 43 ± 31 (43s)   |
+--------+--------------+-------------+----------+-----------------+

What we see from these results is pretty encouraging. The application that will consume these endpoints serves requests in real time, so we need as low latency as possible. We can see that the vanilla Hugging Face container was more than twice as fast as our bespoke container run on ECS — the slowest response we received from the large Inference Endpoint was just 108ms.

What about the cost?

So how much does this all cost? The table below shows a price comparison for what we were doing previously (ECS + Fargate) and using Inference Endpoints.

+--------+------+--------+----------+----------+--------+
|  size  | vCPU | memory |   ECS    |    🤗    | % diff |
+--------+------+--------+----------+----------+--------+
| small  |    1 |      2 | $ 33.18  | $ 43.80  |   0.24 |
| medium |    2 |      4 | $ 60.38  | $ 87.61  |   0.31 |
| large  |    4 |      8 | $ 114.78 | $ 175.22 |   0.34 |
| xlarge |    8 |     16 | $ 223.59 | $ 350.44 |    0.5 |
+--------+------+--------+----------+----------+--------+

We can say a couple of things about this. Firstly, we want a managed solution to deployment, we don’t have a dedicated MLOPs team (yet), so we’re looking for a solution that helps us minimize the time we spend on deploying models, even if it costs a little more than handling the deployments ourselves.

Inference Endpoints are more expensive that what we were doing before, there’s an increased cost of between 24% and 50%. At the scale we’re currently operating, this additional cost, a difference of ~$60 a month for a large CPU instance is nothing compared to the time and cognitive load we are saving by not having to worry about APIs, and containers. If we were deploying 100s of ML microservices we would probably want to think again, but that is probably true of many approaches to hosting.

Some notes and caveats:

• You can find pricing for Inference Endpoints here, but a different number is displayed when you deploy a new endpoint from the GUI. I’ve used the latter, which is higher.
• The values that I present in the table for ECS + Fargate are an underestimate, but probably not by much. I extracted them from the fargate pricing page and it includes just the cost of hosting the instance. I’m not including the data ingress/egress (probably the biggest thing is downloading the model from Hugging Face hub), nor have I included the costs related to ECR.

Other considerations

Deployment Options

Currently you can deploy an Inference Endpoint from the GUI or using a RESTful API. You can also make use of our command line tool hugie (which will be the subject of a future blog) to launch Inference Endpoints in one line of code by passing a configuration, it’s really this simple:

hugie endpoint create example/development.json

For me, what’s lacking is a custom terraform provider. It’s all well and good deploying an inference endpoint from a GitHub action using hugie, as we do, but it would be better if we could use the awesome state machine that is terraform to keep track of these. I’m pretty sure that someone (if not Hugging Face) will write one soon enough — if not, we will.

Hosting multiple models on a single endpoint

Philipp Schmid posted a really nice blog about how to write a custom Endpoint Handler class to allow you to host multiple models on a single endpoint, potentially saving you quite a bit of money. His blog was about GPU inference, and the only real limitation is how many models you can fit into the GPU memory. I assume this will also work for CPU instances, though I’ve not tried yet.

To conclude…

We find Hugging Face Inference Endpoints to be a very simple and convenient way to deploy transformer (and sklearn) models into an endpoint so they can be consumed by an application. Whilst they cost a little more than the ECS approach we were using before, it’s well worth it because it saves us time on thinking about deployment, we can concentrate on the thing we want to: building NLP solutions for our clients to help solve their problems.

If you’ve enjoyed this blog, you can get more content relating to transformers, Hugging Face, and applied NLP at the MantisNLP blog. If you want to talk about deploying your own NLP models into production, we’d love to help you. Get in touch with us at hi@mantisnlp.com.

Why we’re switching to Hugging Face Inference Endpoints, and maybe you should too was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.

Named Entity Recognition

At Mantis, we’ve worked on NER problems for numerous clients in a range of sectors, for example extracting key dates, values, and jurisdictions from contracts; or extracting items, prices, and quantities from tender documents.

Most recently we worked with The Wellcome Trust (hereafter Wellcome), a large Charitable Foundation that funds science to improve people’s lives across the globe, to extract entities from funding applications. This blog post is an abridged version of a blog post we jointly wrote with Wellcome about the project, which you can read here.

Wellcome grant applications

Wellcome holds a database of over 130,000 applications predominantly from scientists applying for research funding. A lot of information about the planned research is captured in free text fields, such as the title of the application, and a synopsis of the research to be undertaken.

These free text fields are not easy to analyze, but we can use NER methods to extract some of the critical information. Wellcome stakeholders were particularly interested in geographic information, which will help them to answer questions like: who are the beneficiaries of the research funding, i.e. the people who derive ultimate benefit from the research, aside from the people and institutions that receive the funding. So, we worked with them to develop a solution to extract entities such as cities, countries, and geographic landmarks.

We experimented with a number of approaches, starting with the simplest possible solution which exactly matched obvious location entities such as Paris, France, and Denmark. This ‘naive’ approach is very quick to implement, and gives us a good baseline. We also explored two machine learning approaches using pre-trained models that provide a more state of the art solution.

We’ve built an application to demonstrate a machine learning approach to this problem. It uses a pre-trained transformer model (a modern type of neural network) to identify two types of location entities: geopolitical entities or GPEs (countries, administrative regions, cities, etc.) and locations (non GPE locations such as geographical regions, mountains, lakes, etc.). The application uses Wellcome grant data published openly through the website https://www.threesixtygiving.org/ under a CC BY 4.0 license.

If you experiment with the model, you’ll notice that it makes mistakes: sometimes it misses locations, and other times it will make incorrect suggestions. This is expected; Wellcome grant applications come from a very specific domain for which this generic model is not specialized.

Balancing Precision and Recall

In this project, the machine learning approaches we tried had a good balance of precision and recall. That is to say that they were reasonably good at finding location entities (high recall), but they produced quite a few false positives (low precision). The naive approach on the other hand, produces close to zero false positives (high precision), but it missed a lot of locations, because it will only find locations on the list we have given it (low recall).

The need to balance precision and recall is common to all machine learning problems, and the relative weight given to each will depend on the use case. In this project it was more important to ensure that we added data of the highest quality to Wellcome’s database, so we implemented the naive solution to begin with. In future we can improve the performance of the machine learning models for this domain by ‘fine-tuning’ them using human annotated data, something that was not available for this project.

35% more locations

Despite using a naive approach our work with Wellcome increased the useful location information available to grants by around 35%. This new information can help Wellcome’s analysts to answer important questions like: who are the beneficiaries of the research funding, i.e. the people who derive ultimate benefit from the research, aside from the people and institutions that receive the funding. For details on why this is a hard question to answer, see our other blog post written in collaboration with Wellcome.

Finally, if you need to extract some information from documents, whether they are entities or something more complex, we may be able to help. Feel free to reach out to us for a chat at hi@mantisnlp.com. You can also find me on twitter at @m_a_upson.

Extracting useful information from documents with Named Entity Recognition was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.