Do note that all the codes used in this blog are available at the repo — https://github.com/aryabhattaslair/RAG_Evaluations.git.

Chain of Thought explores asking the LLM to generate a series of intermediate reasoning steps when answering a question. This significantly improves LLMs performance in logic, math and reasoning tasks. If you are using one of the available LLM APIs (OpenAI, Claude, Gemini etc) the recent models should come with this ability. Most likely they have been fine-tuned to think step by step for reasoning tasks. One of the example with gpt-3.5-turbo

Query: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
Each can has 3 tennis balls. How many tennis balls does he have now?

LLM Response: To find out how many tennis balls Roger has now, we first calculate the number of tennis balls in the cans he bought:
Number of tennis balls per can = 3
Number of cans bought = 2
Total number of tennis balls from the new cans = 3 * 2 = 6

Now, we add the original number of tennis balls Roger had to the new ones he bought:
Original number of tennis balls = 5
Total number of tennis balls now = 5 + 6 = 11

Therefore, Roger now has 11 tennis balls.

By default, it appears that the model is trained (and instructed) to think Step by Step. This implies that most likely, the model is fine-tuned with CoT responses and CoT instructions are part of the system instructions.

However, step by step thinking does not solve all our reasoning problems. Lets take a look at this example

Query: The bakers at the Beverly Hills Bakery baked 200 loaves of bread on Monday morning. They sold 93 loaves in the morning and 39 loaves in the afternoon. A grocery store returned 6 unsold loaves. How many loaves of bread did they have left?

LLM Response: To find the number of loaves left, we need to subtract the total number of loaves sold and the number of loaves returned from the total number of loaves baked.

Total loaves baked = 200
Loaves sold in the morning = 93
Loaves sold in the afternoon = 39
Loaves returned = 6

Total loaves left = 200 — (93 + 39) + 6
Total loaves left = 200–132 + 6
Total loaves left = 68

Therefore, the bakers at the Beverly Hills Bakery had 68 loaves of bread left.

We can see even though the model is thinking step by step it made an error in addition. Lets look at a more advanced method to further improve models reasoning capability. The ideas are discussed in more detail here and here. At its core, its the idea that LLMs can critique their own responses. Lets take the example of previous query and try to generate response using Critique and Revise approach.

Here I will use SmartLLMChain module available within the LangChain framework. SmartLLMChain generates response in 3 steps — Ideation, Critique and Resolve. Lets look at the API

We can see we have defined separate LLM calls for each of the step. Here we are using same models but different models can also be used. Ideation and Critique should use more powerful LLMs with higher temperatures while Resolver can be a less powerful LLM with lower temperatures.

Lets run the chain. The Ideation step, since we have set n_ideas=3, it generates 3 potential solutions
Idea 1:
Sure, let’s break it down step-by-step:

1. **Total loaves baked:** The bakers baked 200 loaves of bread on Monday morning.
2. 2. **Loaves sold in the morning:** They sold 93 loaves.

3. **Loaves sold in the afternoon:** They sold 39 loaves.

4. **Total loaves sold:** Add the loaves sold in the morning and afternoon.
\[
93 + 39 = 132 \text{ loaves}
\]

5. **Loaves returned:** A grocery store returned 6 loaves. These are loaves that were previously counted as sold but need to be added back to the inventory because they were returned.

6. **Total loaves remaining:** Subtract the total loaves sold from the total baked and add back the returned loaves.
\[
200–132 + 6 = 74 \text{ loaves}
\]

So, after all the transactions, the Beverly Hills Bakery had 74 loaves of bread left.
Idea 2:
1. **Total loaves baked**: The bakers made 200 loaves of bread.

2. **Loaves sold in the morning**: They sold 93 loaves.

3. **Loaves sold in the afternoon**: They sold an additional 39 loaves.

Adding the loaves sold in the morning and afternoon gives us:
\[
93 (\text{morning}) + 39 (\text{afternoon}) = 132 \text{ loaves sold total}
\]

4. **Loaves returned by the grocery store**: A grocery store returned 6 unsold loaves.

5. **Total loaves sold or returned**: To find out the total number of loaves either sold or returned, add the number sold to the number returned:
\[
132 (\text{total sold}) + 6 (\text{returned}) = 138 \text{ loaves}
\]

6. **Loaves left at the end of the day**: To find out how many loaves are left, subtract the total number of loaves sold or returned from the total baked:
\[
200 (\text{total baked}) — 138 (\text{sold or returned}) = 62 \text{ loaves left}
\]

Therefore, the number of loaves left at the end of the day is **62 loaves**.
Idea 3:
Certainly! Let’s calculate how many loaves of bread the Beverly Hills Bakery had left at the end of the day step by step:

1. **Total loaves baked:** 200 loaves.

2. **Loaves sold in the morning:** 93 loaves.

3. **Loaves sold in the afternoon:** 39 loaves.

4. **Total loaves sold:** 93 (morning) + 39 (afternoon) = 132 loaves.

5. **Loaves returned by the grocery store:** 6 loaves.

6. **Net loaves sold (after accounting for returns):** 132 sold — 6 returned = 126 loaves.

7. **Loaves left:** We started with 200 loaves and sold (net) 126, so to find the loaves left, we subtract the net loaves sold from the total loaves baked:
\[
\text{Loaves left} = \text{Total loaves baked} — \text{Net loaves sold} = 200–126 = 74.
\]

Thus, the Beverly Hills Bakery had **74 loaves of bread left** at the end of the day.

We can see that in the 3 ideas, ideas 1 and ideas 3 have the correct solution. Now we run critique on these ideas to identify the right ideas.

Critique:
Certainly! Let’s analyze each response to identify potential flaws or instances of faulty logic.

### **Analysis of Idea 1:**

**Flaws:**
1. **Misinterpretation of Returned Loaves:** The calculation seems to assume that returned loaves should be subtracted from the total sold. In most scenarios, returned items should be added back to the inventory rather than be subtracted from sales, as they were not finalized sales.
2. **Miscount in Final Inventory:** Based on typical inventory logic, the store would have loaves left over by subtracting total sold (including returns) from what was baked, without subtracting returns from sales.

### **Analysis of Idea 2:**

**Flaws:**
1. **Incorrect Final Calculation:** The response states that 138 loaves are “accounted for” and subtracts this from the total baked to get 62 loaves left, ignoring the fact that returned loaves are added back to the available inventory, not just subtracted from total sales.
2. **Logic Misstep in Return Addition:** The response treats returned loaves as if still part of the “sold” classification, which could be misleading since these are effectively back in inventory for potential resale.

### **Correct Approach:**
To find the correct number of loaves left:
- Total loaves baked: 200
- Total loaves sold: \(93 \, \text{(morning)} + 39 \, \text{(afternoon)} = 132\)
- Total effective loaves after considering returns: \(132 \, \text{(sold)} + 6 \, \text{(returned)}\)
- Loaves left: \(200–132 = 68 \, \text{loaves left}\)

### **Analysis of Idea 3:**

**Flaws:**
1. **Faulty Return Subtraction:** The logic applied in subtracting the loaves that were returned from the total loaves sold is faulty, as these loaves should be added back to inventory.
2. **Misinterpretation of Returns Impact:** Returns have been incorrectly dealt with as a reduction in sales rather than an addition back into available stock.

### **Summary:**
Each response contains an error in how returns are treated. Returned goods should be added back to the stock rather than subtracted from sales totals. The correct number of loaves left should account for loaves baked minus loaves actually sold, not taking returns from sales but adding returned loaves back to the stock for a correct inventory count. The correct calculation should have been:
- Loaves left = Total baked — Total sold
- Loaves left = 200–132 = 68 loaves left

This error in handling returns and final tally calculation resulted in the incorrect number of loaves being reported as leftover in each scenario.

And eventually resolution which uses this criticism to arrive at the final response
Resolution:
To determine how many loaves of bread were left at the Beverly Hills Bakery after Monday’s activities, we need to consider the total number of loaves baked, the number sold, and the number returned. Here’s the correct calculation:

1. **Total Loaves Baked**: The bakery baked 200 loaves of bread on Monday morning.

2. **Loaves Sold**: Throughout the day, the bakery sold a total of 132 loaves (93 loaves in the morning plus 39 loaves in the afternoon).

3. **Loaves Returned**: A grocery store returned 6 unsold loaves. These returned loaves are added back to the bakery’s inventory since they were not sold.

4. **Loaves Remaining**: To find the remaining loaves, subtract the total loaves sold from the total loaves baked, and then add the returned loaves since they are still part of the inventory.

\[
\text{Loaves Remaining} = (\text{Total Baked} — \text{Total Sold}) + \text{Returned Loaves}
\]
\[
\text{Loaves Remaining} = (200–132) + 6 = 74
\]

Therefore, after accounting for the sales and returns, the Beverly Hills Bakery had 74 loaves of bread left at the end of Monday.

An excercise for readers, try Critique and Revise to solve Prof. Lecun’s gear question (https://x.com/ylecun/status/1639690596364308482?lang=en). Is it able to solve this problem?

This framework can also be applied to real world problems such as to reduce hallucinations in RAG applications. Added a notebook in the Github repository (notebooks/Critique_Revise_RAG.ipynb) for deep dive.

Critique and revise is a powerful technique to help LLMs reduce hallucinations and solve complex problems. This comes at a significant cost in terms of compute and latency. In the above example, we make 7 LLM calls to get the final answer. But if the comparison is between the additional LLM calls and 1 hour of human time, the choice is clear.

We will look at how Critique and Revise can be used for more advanced tasks as we look into Agents.

Though evaluating LangChain was a valuable experience, I felt that a lot more needs to be explored in the library as well as in the RAG ecosystem which has developed (libraries like LLamaIndex) around this field. I decided to make the repository public and will continue to add more interesting examples, use-cases and modules. Link to the repository — https://github.com/aryabhattaslair/RAG_Evaluations

I demonstrated 2 applications in the workshop

• Building a News QA module where I used a Pinecone Vector database to support the QA application. The database here was a set of news articles scraped from a news website.
• Building a PDF QA module where I used a Chroma local vector db. The database here was a set of PDFs. A local vector DB is useful for cost-effective development as it allows testing applications with small local datasets.’

There are 2 major flows in the QA module. We first generate the Search Query using the create_history_aware_retriever

The Search Query Generator created here is used in the second flow to generate the actual conversation

Overall I was impressed by the width of connectors and modules available within LangChain. It allows me to prototype applications fairly quickly. I found connectors for almost all Vector DBs I have worked with, LLM module for all major models and ingestion module for various input sources. This also helps avoid a lot of boilerplate code.

However, I also felt that LangChain library is difficult to work with especially when the use case is more complex and requires custom modifications.

• Class hierarchy seems inconsistent. For example, what is the difference between chains and runnables. Runnables seem to run from chains then why are they in different hierarchy?
• LangChain in its efforts to provide high level API for a lot of use cases have made the API very brittle. This makes integration very difficult. If I want to make any changes to the input to a chain, I may need to change the internal code of the Chain.
• Library structure keeps changing. Every time I revisit an app I wrote, I have to make significant changes to LangChain code. This may not be LangChain’s fault but they could maintain the interface same and keep changing the implementation as the vendor side changes.
• Documentation appears incomplete and insufficient

Right now, I am recommending using LangChain to prototype and experiment but develop own library for production use case. Please note that I may not have completely understood why those design choices are made. Also the library is in aggressive development, and a lot of these issues should be resolved.

I will continue to explore this area. I am exploring new techniques such as DeepRAG, Chain of Thought prompting, using Knowledge Graphs to improve query and responses, Critique and Revise etc. I will also explore how to optimize RAG pipelines for performance and accuracy and different design choices made for different objectives.

Please stay tuned.

This is for those who might be skeptical about AI. Some might find what I’m saying pretty straightforward. When it comes to products, we often talk about the engineering needed for scalability, while demos usually emphasize the core ideas. In this blog, I won’t dwell too much on the platform and scaling aspects, though scaling does pose challenges for Retrieval Augmented Generators (RAGs). Dealing with issues like processing data, choosing Vector DBs, managing latency and throughput, inference on Large Language Models (LLMs), authentication, and logging is crucial for RAGs at scale. However, my main focus is on the fundamental tech necessary for the product to function effectively.

Let me elaborate on my reasons. Do note that these are only my own thoughts and experiences and it may be different from your experience. I will focus on the generation aspect and then talk about challenges with retrieval.

1. Lets start with discussing the Generation aspect of the LLMs.

LLMs can be highly unpredictable when generating answering from context. Formal discussion on my claims below can be read from this excellent survey paper on factuality (link). LLMs have difficulty answering from context specially if the request is more complex or the data itself is complex (fast changing, incomplete etc) . Some of the challenging scenarios which I have seen

• If LLMs have to combine information from its knowledge and context to answer a question. For example, if you are asking questions about changes to a recent product but you are assuming that lot of the product information LLMs should have intrinsically. LLMs can have difficulty robustly decoding responses in such scenarios.
• Large Language Models (LLMs) face challenges when trying to understand quickly changing and messy information. They can easily get confused or misinterpret information that is similar but not quite the same. For instance, when dealing with rapidly changing business data (like sales information), LLMs might struggle to pick out the important details or sort through the relevant information.
• Large Language Models (LLMs) might struggle with understanding complicated questions. These systems are most useful when it comes to finding and piecing together intricate and hard-to-find information. However, their performance in this regard can be shaky because they might lack proper instruction following and reasoning capabilities. For example, if users only wants to see information with specific filters LLMs may not follow it reliably.
• LLMs are very susceptible to hallucinating numbers. You have to be very careful when using it to get accurate numbers.
• LLMs have difficulty saying they do not know. If the context is not relevant to the query LLMs can generate very bizarre output.

Some of these can be improved by fine-tuning models (look at this excellent paper)(or pre-training) with close to real application examples and queries. But that is not an off-the-shelf solution and may be prohibitively costly. Some of the factuality issues can also be reduced by using Critique and Revise (https://arxiv.org/abs/2305.11738) approaches like Chain of Thoughts (https://arxiv.org/abs/2201.11903). However, these are high latency, high cost inference options and may require ML expertise to integrate with the RAG pipeline.

2. Lets look at the retrieval aspect of the RAG pipelines

• Creating a strong context is trickier than it seems, especially considering the diverse queries your application will encounter. Ranking enterprise data can be challenging due to a significant overlap in documents, a wide range of domain data, and various types of queries. To build an effective ranking system for your retrieval pipeline, you might need a lot of customized engineering. Additionally, for a unique and new application, establishing a good ranking system could be tough initially, especially if you face a “cold start” problem, meaning you lack sufficient data. Simple search methods like ANN or keyword search based methods like BM25 may not be enough to establish a robust context. (other interesting thoughts on this here).
• Users can write queries with lot of fluff and irrelevant information. You may have to rewrite user query into one or more search queries for better retrieval. This may require you to generate such queries during inference using ML or search based system. You may also hit a cold start problem here.
• You may require a summarization or a filtering system if you don’t have enough control on the document DB side. Search may return lot more information/data you want to consume and you may need to further trim it down. Having a good metadata on your data might help here but lot of enterprise data specifically unstructured data may not have any metadata.
• A lot of information users are interested in is in tables, images, excel sheets (and variety of other proprietary format) which may be harder to combine with text based context. You may also need to write custom pipelines to ingest this data so it becomes available for vector search and for building your context. This may become tedious, expensive and error prone (specifically if this has to be done frequently).

While these challenges aren’t impossible, addressing them may demand a significant amount of engineering on both the search backend and machine learning fronts to ensure the system works reliably. Additionally, there’s the challenge of establishing effective metrics and benchmarks for Retrieval Augmented Generation (RAG) pipelines, given the inherent difficulty in rating factuality and the further challenge of generalizing benchmarks across a variety of use cases.

Creating top-notch products for businesses would involve doing substantial work on both the retrieval and generation aspects of these systems. In my opinion, a more effective approach would be to build a specialized platform for specific industries or domains and customize it for each customer. Considering the valuable impact these products can have on making business processes more efficient, I see great opportunities for startups in this space.

1. Can you solve it using Dynamic Prompting? (and not do SFT)

Is it possible to get to the relevant performance and behavior by using zero shot or few shot prompting? You could also try to use Dynamic Prompting (DP) by using other signals available from your user. For example, for an enterprise application perhaps you have different prompts for different organizations or different verticals. I have seen amazing level of adaptability in Instruction Tuned LLMs by just using DP. A good reference paper for this — https://arxiv.org/pdf/2303.02909.pdf.

2. Less data is more.

Try to be conservative with data being used for SFT. Bad (or even ok) data hurts performance more than it helps. This goes back to the adaptability point that having small amount of good data is way better than having lot of data you are not clear on the quality. Invest in the quality of data being added to SFT both in employee-hours and in tooling. Make sure to run relevant quality checks on data before it gets added to your SFT mix. Usually you need examples in 100s not in 1000s for a given feature unless there is additional variability. A good reference on this — https://arxiv.org/pdf/2305.11206.pdf.
This is more difficult than it looks. A lot of times you will see interference in SFT data between features. Data you add to improve feature A starts hurting feature B. So be careful about it. I think there is art there. If you can do it well, it can become a definite competitive advantage.

3. Tune the model to use context well

Your SFT context should be as close to the inference as possible and you should provide clear examples where model should use context to answer questions or adjust behavior. A good example of it is using external knowledge base for answering questions (similar to a RAG applications). Add examples in your fine-tuning data on how the results from knowledge base added to the context are used to answer questions. Having a good mix of data representing different behavior is key here and could be the difference between an average and excellent model behavior. Definitely an art and can be a competitive advantage if done well.
Another interesting example of using context well is to help LLM understand the current time (or the time at which a query is made) and how to use it.

4. Distill from larger models

LLM scaling laws have shown that larger models have much better generalization, better instruction following and better reasoning but these models are costly and have high latency. A good idea is to distill from these models through fine-tuning. One potential approach is to generate the necessary fine tuning data by using a larger instruction tuned model. For example using an instruction tuned LLAMA 2 70B model to finetune LLAMA 2 7B model. Some reference on this (https://blog.research.google/2023/09/distilling-step-by-step-outperforming.html)

5. Use LoRA to experiment

Fine-tuning sometime requires experimentation as you are not sure how the data will result in model behavior. It can be more optimum to use LoRA for experimentation (and evaluate the model) and full fine-tuning for the final model. LoRA can save you significant money (reduced GPU memory requirement in the LoRA paper https://arxiv.org/pdf/2106.09685.pdf by a factor of 3) and fasten your experimentation velocity. Also look at other PET methods like QLoRA.
However, its a good idea to tune the final production ready model using the standard approach to avoid losses.

6. Invest in Evaluation and Regression Sets

If you are regularly updating your model through SFT, invest in building good evaluation and regression sets for testing your models. Getting domain experts or users to create this sets can be a bit expensive but will pay off well in the future.
This is before any Teamfood or Dogfood step for testing your application. SFT is a probabilistic exercise and can have unpredictable behavior. To avoid pushing a problematic model change, run auto and rated evaluations for any new model. This will also help improve experimentation velocity and help you clearly gauge progress.
Also, do regularly update these sets based on logs from your application.

7. Use Critique and Revise to improve data quality

I am still exploring this but methods like Chain of Thoughts prompting (https://arxiv.org/abs/2201.11903) and self critiquing (https://arxiv.org/pdf/2206.05802.pdf) are showing lot of promise in identifying and correcting poor SFT data. One can get very high quality training data by combining Large Model Distillation with Critique and Revise.
These methods can also be used during inference to answer more difficult reasoning questions or queries where users request rewrite or further clarification.

Above is a very high level summary of some of the key areas I believe are relevant for Supervise Fine Tuning. In future, I will try to go into more details of some of these ideas and expand them. Thank you for reading through and do reach out to me on LinkedIn if you have questions, comments or need advice on SFT.