<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vineet Singh on Medium]]></title>
        <description><![CDATA[Stories by Vineet Singh on Medium]]></description>
        <link>https://medium.com/@vineet.vinsing?source=rss-f162140d81db------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*a9vx_ECsr-UmLBVLyDUJhQ.jpeg</url>
            <title>Stories by Vineet Singh on Medium</title>
            <link>https://medium.com/@vineet.vinsing?source=rss-f162140d81db------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:50:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vineet.vinsing/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Under the Hood of Farmer.chat: Journey to an optimised, production ready RAG powered Chatbot]]></title>
            <link>https://medium.com/digitalgreen-techblog/under-the-hood-of-farmer-chat-journey-to-an-optimised-production-ready-rag-powered-chatbot-589bf5716e27?source=rss-f162140d81db------2</link>
            <guid isPermaLink="false">https://medium.com/p/589bf5716e27</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[agriculture-technology]]></category>
            <dc:creator><![CDATA[Vineet Singh]]></dc:creator>
            <pubDate>Mon, 08 Jul 2024 05:59:57 GMT</pubDate>
            <atom:updated>2024-07-08T06:30:20.231Z</atom:updated>
            <content:encoded><![CDATA[<p>In our<a href="https://medium.com/digitalgreen-techblog/unlocking-generative-ai-for-agriculture-farmer-chat-2ae2d62f1946"> previous article</a>, we gave detailed description about functional blocks of <a href="https://farmerchat.digitalgreen.org/">Farmer.chat</a>, an AI-powered chatbot designed to bridge the information gap for extension workers and farmers. We highlighted the value Farmer.chat delivers by providing timely and accurate agricultural knowledge in a user-friendly conversational format. In this article, let’s delve deeper into the journey of creating a production-ready RAG-powered chatbot for Farmer.chat. While there are excellent resources on Medium and all across the web on RAG and advanced RAG techniques, there are few resources which provide practical case study on how to evolve into production ready systems solving real business problems. We hope this provides a case study for the AI community and helps them adopt some of the important points.</p><blockquote><strong>The perspective for the readers is the way to build technical blocks based on the user needs rather than a prescriptive RAG pipeline.</strong></blockquote><h4>Why did we chose RAG to start with?</h4><p>We get this question a lot of times: why we chose RAG and why not a fine tuned model. Although it has been debated at various forums and papers it is worthwhile to spend some time and relate it to the business problem itself. As mentioned in the previous article, the main challenges of agriculture extension in nutshell are:</p><ul><li><strong>Un-organised information:</strong> generic and staggered information makes finding the right answer difficult.</li><li><strong>Context matters:</strong> Farming success depends on local factors and agriculture domain knowledge.</li><li><strong>Accuracy is key:</strong> Bad advice can hurt a farmer’s livelihood and trustworthy information sources are a must.</li></ul><p><a href="https://platform.openai.com/docs/guides/optimizing-llm-accuracy/llm-optimization-context">This article</a> last year on openai gives an interesting quadrant approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U7TlR50UaTy2fK_A" /><figcaption>RAG vs Fine tuning</figcaption></figure><p>The article puts it out clearly:</p><blockquote><strong>Context optimisation:</strong> You need to optimise for context when the model lacks contextual knowledge because it wasn’t in its training set or it requires knowledge of proprietary information. This axis is <strong>where you need RAG.</strong></blockquote><blockquote><strong>LLM optimisation:</strong> You need to optimise the LLM when the model is producing inconsistent results with incorrect formatting, or the tone or style of speech is not correct. This axis <strong>is where you need LLM fine tuning.</strong></blockquote><p>There is an<a href="https://arxiv.org/abs/2401.08406"> in-depth study</a> done in the agriculture domain by the Mircrosoft team. The key conclusions are covered from tables 18–20 in this paper.</p><blockquote>A key observation is that a base model like GPT4 + RAG is almost as good as a fine tuned base model.</blockquote><blockquote>Delta improvement is observed with fine tuned base model + RAG but not consistent.</blockquote><p><strong>RAG is cost effective as compared to fine tuning</strong>. Fine tuning + RAG may be slightly beneficial but if we want relatively good contextual answers, developing a RAG powered chatbot is a faster and cost effective.</p><blockquote>Taking this decision early helped immensely as we will see in the article further. These early decisions are key to success in your technology and product roadmap.</blockquote><p><strong>Note</strong>: we took this decision way back in June 2023 while these in depth study and reports are mostly from the end of 2023 to beginning of 2024.</p><blockquote>In the initial days it is not important to back your decisions with a very thorough study but quick indicators. Validation from studies by others is a further reinforcement and <strong>if it is not in line, one should reconsider the decision</strong>.</blockquote><h4>Agile approach: Build — Test — Deploy — REPEAT</h4><p>We adopted a rigorous “build-test-deploy” cycle within an agile development framework to optimise Farmer.chat. This iterative approach allowed us rapidly prototype, test, and refine the chatbot, ensuring continuous improvement.</p><p>It is important to understand that RAG, though it sounds simple, has many moving parts and a lot of techniques to improve each of those moving parts. <strong>In real world development, we don’t have the luxury of implementing a super refined RAG pipeline in one go. One needs to start simple and focus on one thing at a time.</strong></p><blockquote>One needs to know which part to focus on first to maximise the improvement to effort.</blockquote><p>These decisions are not straightforward and depend on your team structure, strength, type of content, business problem at hand and many factors. Here we present our journey starting with a small team how we upgraded each version and gradually increasing complexity.</p><h4>Version 1: 2 — week sprint</h4><p>Our version 1, almost a year ago, was just two week sprints. Agriculture is a vast subject, we focused on one or two crops in one region and a curated set of questions (50 odd) for those crops for evaluation.</p><p><strong>Key decisions</strong></p><ol><li><strong><em>Messenger platform</em></strong> : we developed bot for two messenger channels, Whatsapp and Telegram. Reason for choosing messenger applications was <strong>ease of onboarding users and familiarity of the users with the interface</strong>. While WA is more widely used, telegram offers simpler APIs to integrate, is free and provides better user experience for multi-modal content. Getting users to install telegram was a one time effort.</li><li><strong><em>Langchain’s Power </em>:</strong> We opted for Langchain, a popular open-source library. Langchain provides powerful tools for building LLM enabled apps using ready to chain of thoughts, agents and configurability to call different LLM or embedding models. Langchain provided the ConversationalRetrievalChain class well suited for RAG.</li><li><strong><em>ChromaDB for Knowledge Storage </em>:</strong> We selected ChromaDB as the vector store. We considered pinecone, but chromadb satisfied the need to have option of self hosting.</li></ol><p><em>Prompt engineering played a crucial role in ensuring that the bot doesn’t answer and start conversing on topics outside agriculture domain.</em> It performed reasonably on the test questions for the crops.</p><blockquote>The test in this version was about the <strong>ability to answer agriculture related questions and not on the completeness, relevance and accuracy</strong>.</blockquote><p>Here is the information flow of the ConversationalRetrievalChain class. <em>Please note that </em><strong><em>this class is now deprecated in langchain and we moved away from langchain after our first version</em></strong><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/0*ZQ5rVSLxXhR_Jb-s" /><figcaption>Version 1: farmer.chat</figcaption></figure><p><strong>Issues with version 1</strong></p><ol><li><strong>Hallucination</strong>: While this version didn’t answer out of context questions (not related to agriculture), <strong>it did answer the questions which were related to agriculture and the crop but were not covered in the content</strong>.</li><li><strong>Limitation with langchain</strong>: while langchain is an excellent starting point, we realised some limitations like:</li></ol><ul><li>we couldn’t log the rephrased query and we really couldn’t see if it is adding some keywords that cause issues in retrieval</li><li>retrieval method (at least at that point) had two options and in our production environment, the retrieval method that worked had opposite sense to cosine similarity!!</li><li>we found that retrieval method itself when done using directly chroma client provided different and better results as opposed to the one through langchain</li></ul><p>3. <strong>Chromadb in memory problem</strong>: while chromadb is easy to set up, AI native and fast, it is in memory and as the number of bot instances in different regions increased, the cloud compute requirement grew exponentially.</p><p>While hallucination is a business problem the other issues were more technical in nature. However, both were intertwined in the sense that langchain’s limitations didn’t allow us to properly address the issue of hallucination.</p><blockquote>As a result, we decided to <strong>move away</strong> <strong>from langchain</strong> to get better control.</blockquote><p><strong>Please note:</strong> this is not to suggest that langchain is bad. On the contrary, we will advise anyone to <strong>use langchain or llamaindex</strong> if that satisfies your business needs <strong>but suggest testing it thoroughly</strong>.</p><h4>Version 2: focus on fixing hallucination</h4><p>We set up the same flow but with our own calls to LLM and control over the prompts in rephrasing the query based on history as well as generation. Once we got the rephrased query and retrieved chunks, we noticed the cause of hallucination.</p><blockquote>Hallucination in RAG occurs when irrelevant text chunks are passed in generating the answer.</blockquote><p>It is a well known little quirk of LLM. It is good in following <strong>positive instructions</strong> but does not properly <strong>respect negative instructions</strong>, for instance in this case — <em>do not generate answers if the chunks in context are not relevant to the question is </em><strong><em>not respected all the time</em></strong>.</p><p>We looked at each of the moving part and first ruled out what not to focus on:</p><ul><li><strong>Embeddings: </strong>the available embeddings were better but delta improvement was really small as reported on MTEB leaderboard</li><li><strong>Chunking strategy</strong>: although it has significant impact as we see later, there were too many variations and not clear where to start</li><li><strong>Query decomposition</strong>: it does maximise the retrieved text chunks but logically, it will lead to completeness of answer rather than improving the hallucination as the irrelevant chunks would remain as is</li><li><strong>Vectordb: </strong>it was easy to do quick trials and check that there would be some improvement but not much and it is more related to performance</li><li><strong>Augment knowledge base</strong>: quick trials revealed that the GPT models were themselves not providing hallucination free question answer pair and generating the same was not that easy</li></ul><blockquote><strong>We decided rather than more upstream steps, focus on the downstream</strong>. In essence, we want to maximise text chunks with relevant information to be passed on for generation and minimise text chunks with irrelevant information. <strong>Why not just add a filter and rank the retrieved text chunks rather.</strong></blockquote><p>Though there are many algorithms, <strong>we leveraged LLM itself to reflect and reason through and filter/ rank chunks</strong>. The key to evaluate the efficacy was to look at the retrieved chunks for each question and identify which is completely relevant and which is completely irrelevant manually and verify if the results match with LLM.</p><blockquote>With LLM based filtering and ranking, we were able to <strong>correctly filter out 98.3 % of irrelevant chunks on test data</strong>.</blockquote><blockquote><strong>This led to overall improvement in completely factually correct responses from </strong>~ 76 % to ~ 100%<strong> on the test data of 30 odd questions.</strong></blockquote><p>We also changed chromadb in-memory to chromadb client. Here is what the new flow looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*dbrR-M1zhqWLtT72" /><figcaption>Version 2: farmer.chat</figcaption></figure><p><strong>Issues with version 2</strong></p><ol><li>While chroma db client worked, it is a patch and chroma still doesn’t provide elegant way and we had to look for other options.</li><li><strong>high faithfulness -&gt; high unanswered questions</strong>: what we were able to answer was correct and grounded in the content but it meant a lot of questions now got unanswered !!</li></ol><h4>Version 3: fixing vectorstore db</h4><p>As we had got a positive response initially from the bot, we were required to scale it in different geographies and larger, more complex content. This meant vector store db in chroma which was selected with minimal testing needed thorough analysis.</p><blockquote><strong>This section can be a bit technical and therefore we have provided the complete details </strong><a href="https://docs.google.com/document/d/1vdmd_gRzndoSYL-KnxJApTCwhYZuNVSUPWRA6lAaK7E/edit?usp=sharing"><strong>here</strong></a><strong>. We present the conclusions and results here.</strong></blockquote><p><strong>Benchmark Result Analysis</strong></p><p>The detailed result sheet can be found here<a href="https://docs.google.com/spreadsheets/d/1Fm46hIpGrt94-GJanCiJR5cbfbEdQsbMeacRURMpUng/edit#gid=0"> Vector DB Benchmarking</a> The important points observed from the result sheet are here,</p><blockquote>Qdrant DB is the best performing among all, it has the highest QPS, Highest Recall@100 (at efConstruct 256)</blockquote><blockquote>Vespa and ElasticSearch db are more refined on the memory utilization side, however their latency is on the higher side.</blockquote><p>Here is the revised information flow for version 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*qsqCeptbc2fQVnnG" /><figcaption>Version 3: farmer.chat</figcaption></figure><h4>Version 4: fixing unintentional unanswered questions</h4><p>As we enhanced our RAG pipeline, we focused also on creating an auto evaluation pipeline which was benchmarked against human evaluation. Our user research team and product — program teams pointed out that inability to answer questions causes drop off. This was corroborated with the data but what came out further was that around 11% of unanswered questions were the users asking about a different crop than what they have selected and 2–3 % were further remarks (welcome message, exit) which got converted to some question because of chat history. This also resulted in more of a question answering machine than a conversational flow.</p><p>We decided to have a user intent based flow so as to create conversational flow as well as route to the RAG only when the intent is farming related.</p><blockquote>Tests showed that chatgpt-4 achieved 100 % accuracy in user intent classification sample with few shot learning.</blockquote><blockquote><strong>Achieving near 100 % is important</strong> because if this is wrong, it can lead to more absurd answers.</blockquote><p>This was quickly rolled out and led to change in the information flow as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/0*TyRpe92yqiMeQB3n" /><figcaption>Version 4: Farmer.chat</figcaption></figure><h4>Version 5: retrieval error due to common words</h4><p>With more deployment and usage, there were some <strong>issues reported where users’ questions remained unanswered even though it seems correct</strong>.</p><p>A closer inspection of retrieved chunks revealed that certain words which are very frequently used in some text chunks and used in query cause totally irrelevant chunks to be retrieved. Example, <em>what are common varieties of coffee? </em><strong><em>VS </em></strong><em>what are common varieties of coffee in </em><strong><em>Kenya</em></strong><em>?</em></p><p>In this particular case, there are document sections where Kenya comes too often or a phrase which is realted to Kenya like KES. This resulted in irrelevant retrieved text chunks which were rejected by filtering and ranking.</p><blockquote>The fix to this we added another LLM call to shrink and simplify the query to basic theme for retrieval.</blockquote><p>This added another block in version 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/0*Q7rlBzi78nU5YAou" /><figcaption>Version 5: Farmer.chat</figcaption></figure><h4>Version 6: Semantic chunking</h4><p>One of the key limitations with RAG is you can know <strong>if the answer is correct by looking at the text chunks</strong> passed at generation step but <strong>it is hard to ascertain if the answer is complete</strong> because there could be text chunks that have been missed.</p><p>Plainly speaking:</p><blockquote>We can be sure that <strong>whatever answered is correct</strong> but can we be sure that <strong>whatever can be answered is answered</strong>.</blockquote><p>The latter depends on the context recall which can vary depending on the content and query. Context recall measures what percentage of relevant text chunks are retrieved on an average. To calculate the same, <strong>one needs to know how many relevant chunks are there in the first place</strong>.</p><p>One thing that we considered initially and literature suggests is having a good chunking strategy. We had quickly implemented linear chunking and though it works, it is not the best way. What if we can group together chunks based on topics so that if that topic is asked about, the entire chunk gets extracted. There is a<a href="https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb"> great jupyter notebook</a> describing five levels of chunking by<a href="https://x.com/GregKamradt"> Greg Kamradt</a>. We request reader to read through the notebook and watch the associated video. We decided to implement semantic chunking (level four). This approach doesn’t rely on LLM but just on calculating the cosine similarity between the adjacent sentences.</p><blockquote>Intuitively, semantic chunking is simply saying that <strong>if the topic changes the similarity distance between new sentence will see a sudden jump with the previous sentence(s)</strong>.</blockquote><p>This makes sense. However, we had to split the tables and images. We had tried out grobid earlier which is widely used for scientific documents but it didn’t work on our content. We found<a href="https://github.com/Unstructured-IO/unstructured"> untructerd-io</a> as way better and though it misses out on captions of the images or tables, it is able to reasonably extract the text.</p><blockquote>Having a good pdf parser suitable for your content is necessary for semantic chunking.</blockquote><p>Once we were able to do semantic chunking, we used it to create chunks and their respective topics.</p><p>This resulted in yet another change in the information flow of RAG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*5I_P7djXnzo5eRqS" /><figcaption>Verison 6: farmer.chat</figcaption></figure><blockquote>It is important to understand shift in the evaluation beyond <strong>RAGAS</strong></blockquote><p>Our focus was earlier on <strong>factual correctness and relevance</strong>. Now, let us assume that there are certain questions from the users which can be answered and certain questions that can’t be answered. And let us assume there are some factually correct answers and factually incorrect answers. This creates a two by two matrix as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/0*qBIcGJQl_tXqfKmY" /><figcaption>Evaluation: focusing on RAGAS</figcaption></figure><p>If we are looking at RAGAS framework on faithfulness and relevance, we are trying to maximise the upper half while minimise the lower half.</p><p>What if we change the y axis of the quadrant as is answered and is not answered, creating a revised confusion matrix for RAG as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/0*PjWFvpYl1qqp93kG" /><figcaption>Evaluation: focus on what is answered</figcaption></figure><p><strong>In this revised approach, we maximise the first and third quadrant such that it is factually correct.</strong> This provides a clear definition of hallucination and retrieval error if what can be answered is answered provided it is factually correct.</p><blockquote>It is important to relook at your evaluation criteria as per the business needs.</blockquote><blockquote>Our overall metric became multiplication of percentage faithful sentence and percentage answered.</blockquote><blockquote>With semantic chunking we were able to reduce the number of unanswered questions to nearly half.</blockquote><h4>Our journey: In a nutshell</h4><p>Our journey involved multiple version iterations, each focusing on specific optimisation goals:</p><ul><li><strong>Version 1:</strong> A rapid prototype built using LangChain to demonstrate core functionalities. While functional, it highlighted limitations in complexity.</li><li><strong>Version 2:</strong> Focused on addressing “hallucination” (generating irrelevant responses). We moved away from LangChain.</li><li><strong>Version 3:</strong> Optimised the Vector Database (Vectordb) for efficient information retrieval.</li><li><strong>Version 4:</strong> Streamlined user intent flow routes to minimise unanswered questions. This improved user experience as well.</li><li><strong>Version 5:</strong> Implemented “shrink query for retrieval” to handle complex or edge-case queries.</li><li><strong>Version 6:</strong> Employed semantic chunking to group sentences realted to topics together, reducing unanswered questions.</li></ul><h4>The road ahead</h4><p>Our work on Farmer.chat continues. In the next article, we’ll explore <strong>how we employed automated evaluation techniques to accelerate</strong> the iteration process and further optimise Farmer.chat for real-world impact.</p><p>We currently have many LLM calls and there are ways we have optimised it for latency in the acceptable range. There are further enhancements under way to decrease latency and cost. We would be exploring the same in detail in future articles. Stay tuned.</p><h4>Gratitude</h4><p>We are grateful to our donors and projects which made this work possible, in particular AIEP and GAIA project by Bill and Melinda Gates Foundation and GIZ.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=589bf5716e27" width="1" height="1" alt=""><hr><p><a href="https://medium.com/digitalgreen-techblog/under-the-hood-of-farmer-chat-journey-to-an-optimised-production-ready-rag-powered-chatbot-589bf5716e27">Under the Hood of Farmer.chat: Journey to an optimised, production ready RAG powered Chatbot</a> was originally published in <a href="https://medium.com/digitalgreen-techblog">digitalgreen-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Generative AI for Agriculture: Farmer.chat]]></title>
            <link>https://medium.com/digitalgreen-techblog/unlocking-generative-ai-for-agriculture-farmer-chat-2ae2d62f1946?source=rss-f162140d81db------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ae2d62f1946</guid>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[agriculture-technology]]></category>
            <dc:creator><![CDATA[Vineet Singh]]></dc:creator>
            <pubDate>Fri, 31 May 2024 10:56:14 GMT</pubDate>
            <atom:updated>2024-05-31T10:56:14.292Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/u/1ac150b0b18b">Rajsekar</a> <a href="https://medium.com/u/f02a5c1a841e">Waseempasha</a> <a href="https://medium.com/u/d8fe58a55d77">Ramaskanda</a> <a href="https://medium.com/u/d6fa51cfd62d">Sanjeev Mishra</a> <a href="https://medium.com/u/227a37d4a4c2">Jayasankar G K</a></p><p>At <a href="https://digitalgreen.org/">Digital Green</a>, we are passionate about empowering smallholder farmers, who play a critical role in global food security. Timely access to accurate information is essential for these farmers to make informed decisions and improve their yields. However, farmers and agriculture extension workers often face challenges in accessing the right information which is timely and contextual due to the scattered agricultural data. <a href="http://famrer.chat/">Farmer.chat</a>, our AI-powered chatbot, aims to bridge this gap by offering extension workers and farmers a smarter tool to access and share crucial agricultural knowledge which is trustworthy, contextual and in the local language in multiple channels.</p><p>While farmer.chat is relatively recent innovation, Digital Green has been at the forefront of innovations in agriculture extension for more than a decade and a half. Our journey started with farmer first approach: capturing and curating locally made videos, facilitating screenings in farmer groups and enabling them with software tools for feedback and data collection.</p><blockquote>Through our interventions, we were able reduce the cost of adoption of agriculture practice from $35 to $3.5.</blockquote><p>Over the last fifteen years, we have been able to reach 6.3 Mn farmers, through a network of &gt; 100K agriculture extension agents which has resulted in estimated 24% increase in income.</p><p>While these numbers are impressive, we still have to push the envelop. There are estimated 500 Mn smallholder farmers globally and agriculture extension agents who interact with farmers at typically in the ratio of 1:1000. Reaching the agriculture extension workers and farmers through partnership and technology remains the key.</p><h4>Challenges of agricultural extension</h4><p>Imagine an extension worker in a rural village, helping a farmer struggling with a crop disease. To provide the best advice, they need to juggle a vast amount of information — from local soil conditions to government regulations on pesticides. But sifting through research papers, policy documents and websites can be time-consuming, especially when a farmer needs a quick answer.</p><p>Here’s the problem in a nutshell:</p><ul><li><strong>Unorganised information:</strong> There’s a lot of agricultural data out there, but it is generic and staggered and farmer being a practitioner, finding the right answer can be like searching for a needle in a haystack.</li><li><strong>Context matters:</strong> Farming success depends on local factors like soil type and weather. Conversational interface helps narrow down the question to the specific context.</li><li><strong>Accuracy is key:</strong> Bad advice can hurt a farmer’s livelihood. Extension workers need trustworthy information sources not only for the farmers but for themselves to gain confidence and reaffirm their knowledge.</li></ul><h4>Unlocking Answers in the Field: Smarter Tools</h4><p>We have experimented with chatbots to augment our work for long but with the advances of generative AI, these AI tools can be game-changers. We immediately saw the potential of developing intelligent chatbots powered by <a href="https://www.promptingguide.ai/research/rag">Retrieval-Augmented Generation (RAG)</a> on approved, curated information. These intelligent chatbots can utilise advancement in generative AI to enable:</p><ul><li><strong>Fast answers:</strong> GenAI outperforms conventional ways of ingesting unstructured content formats in docs, pdfs, crop tables, images, videos etc. Chatbots can quickly find the most relevant information based on a farmer’s question.</li><li><strong>Conversational flow:</strong> GenAI can be trained to drive a conversational flow to narrow down the problem. This enables gathering necessary contextual information on one hand and generating byte sized relevant responses on the other.</li><li><strong>Reliable sources:</strong> RAG enabled chatbots can be linked and trained to generate answers grounded in databases of officially approved agricultural content, guaranteeing accurate and trustworthy advice for farmers.</li></ul><p>By empowering extension workers with smarter tools, the agentic RAG-enabled chatbots can help bridge the knowledge gap between farmers and the information they need to succeed.</p><blockquote>Our findings point to potential of decreasing cost of adoption further from $3.5 to $0.35</blockquote><p>In this article, we present a detailed description of information flow in <a href="https://farmerchat.digitalgreen.org/">farmer.chat</a>.</p><h4>Farmer.chat overview</h4><p><a href="https://farmerchat.digitalgreen.org/">Farmer.chat</a> is an end to end solution that can be deployed by any organisation for enhancing outreach to the farmers . It is highly customisable and can be plugged in IVR or messenger or an app and can ingest any kind of structured or unstructured knowledge source.</p><p>In the span of over a year, we have continuously evolved our product. In this small timeframe we have been able to:</p><blockquote>Reached more than 16k extension workers</blockquote><blockquote>Answering &gt; 200k questions</blockquote><blockquote>&gt; 6 languages , 40 value chain crops</blockquote><blockquote>~ Zero biases or toxic responses</blockquote><p>Here is an example of advanced agent driven chatbot:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FXhqzekbhZ4cDjh63nY%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FXhqzekbhZ4cDjh63nY&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExcHNnOTAyNG10a3F6b3Bra2VlMWF2MTV4Znh2cjBnM29hbDJpMTZzcCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FXhqzekbhZ4cDjh63nY%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/67574bd063e21c93b5c0e113fd6cd2f2/href">https://medium.com/media/67574bd063e21c93b5c0e113fd6cd2f2/href</a></iframe><p>This is what it happens here if you pay closer look. The user starts with a query seeking a resource: video on wheat rust. The bot understands and presents the relevant video link. The user further asks about the specific chemical for seed treatment . The bot gets the chemical from <strong><em>what is being spoken in the video. </em></strong>The user further asks about the weather without specifying the necessary information like the location. The bot understands that the <strong><em>information is incomplete and prompts back </em></strong>to the user to share location. The user enters the location in free form (if you notice, makes a grammatical error). The bot is able to extract the district name and responds in a structured and readable weather summary. The user further asks about the possibility of pest and disease attack and it looks in the PDF documents and respond with the previously extracted weather information.</p><blockquote>This is how a GenAI powered intelligent bot seamlessly uses different tools: video search, video generated answers, weather API and answer generation from PDF.</blockquote><p>Next, let’s look how farmer.chat works under the hood, step by step.</p><h4>Step 1: Knowledge base builder</h4><p>This is the heart of the system. It is what powers the entire bot by ingesting content and datasets from different sources.</p><p>Here are the functional blocks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zq26EB5PJwckQgoxvm1foQ.png" /><figcaption>Knowledge base builder: powered by <a href="http://farmstack.co/">FarmStack</a></figcaption></figure><p>This is a web interface powered by FarmStack, which is a digital public good developed by Digital Green and currently enabling DPI like kadp.kalro.org.</p><p>The key components of knowledeg base builder are:</p><ul><li><strong>Connectors: </strong>act as pathways that allow the knowledge base builder to ingest various types of data from outside (eg, Googel drive, youtube)</li><li><strong>Content processing engine: </strong>the ingested data is processed and transformed into a format suitable for the knowledge base and involves auto summarisation, content categorisation, auto tagging</li><li><strong>Semantic chunking: </strong>the organised files with metadata are processed with sentences similar in meaning grouped together in text chunks</li><li><strong>Vector db creation: </strong>each text chunk is converted into vector representation using an embedding model</li><li><strong>Video processing: </strong>auto transcription and translation to create description of the videos</li><li><strong>Video indexing:</strong> LLM to generate description of videos and automated sections</li></ul><p>Using the above components, the knowledge base builder serves to ingest, organise and convert the data into vectordb format which can be used by AI application.</p><h4><strong>Step 2: AI modules</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lxrK4OXn1QUGzjANz9pZKQ.png" /><figcaption>AI modules: augmenting the knowledge base builder</figcaption></figure><p>This is the module which is the brain of the system. It is where the query of the user is understood, processed, the necessary information to respond to the user query is obtained and the apt response is generated that is grounded in the knowledge base.</p><p><strong><em>2.1 Response Generator (RAG):</em></strong></p><p>This sub-module utilises a Retrieval-Augmented Generation (RAG) approach to create a response for the user. It is what ensures that the information delivered is grounded in the content and not outside.</p><p>RAG as many are aware of combines two broad streams:</p><ul><li><strong>Information retrieval:</strong> Searching the knowledge base for relevant information that matches the user’s query. This involves calling the vector database API created in knowledge base builder to get necessary text chunks.</li><li><strong>Generation:</strong> Using the retrieved information in the text chunks and the user query, the generator calls LLM and generates a human-like response that addresses the user’s needs.</li></ul><p><strong><em>2.2 Query Orchestration (Generative AI agent)</em>:</strong></p><p>This sub-module leverages large language model (LLM) to process the user query and plan what steps need to be taken. There are three main components:</p><ul><li><strong>Planning and reasoning agent:</strong> understand what user is trying to ask and which tools can be used to answer the user question.</li><li><strong>Execution agent:</strong> call the specific tools with necessary parameters so as to synthesise the responses.</li><li><strong>Tooling service:</strong> Use the API end points to create tools and their description that the LLM can use.</li></ul><blockquote>This sub module is the core to create conversational flow and helps narrow down the user input.</blockquote><p><strong><em>2.3 Local Language &amp; Voice ( Traditional AI):</em></strong></p><p>This sub-module employs language and ASR modules to:</p><ul><li><strong>Detect language :</strong> Identify the language used in the user query.</li><li><strong>Translation:</strong> If necessary, translate the query into a language understood by the system.</li><li><strong>Speech recognition (for voice queries):</strong> Convert spoken user queries into text.</li><li><strong>Text to speech: </strong>to convert the final response to an audio note.</li></ul><h4>Step 3: Integrating with frontend system</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L8epmSjWRweoVhQoLlAbuQ.png" /><figcaption>Final system: enabling responses</figcaption></figure><p>Finally, we have the frontend API handlers consisting of two main modules:</p><p><strong><em>3.1 User Interaction</em></strong></p><p>The process starts with a user interacting with the chatbot through various channels like a messenger app, IVR (Interactive Voice Response) system, or even a mobile application.</p><p><strong><em>3.2 Frontend API handlers</em></strong></p><p>This module handles the user interface (UI) aspects of the chatbot. It ensures a smooth user experience by tailoring the interaction based on the chosen platform (messenger, IVR, etc.)</p><h4>Step 4: Plugging it all back in knowledge base</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qelwszS_MEanDv14PkbUEA.png" /><figcaption>farmer.chat flow of information</figcaption></figure><p>Although the bot can function with the flow in step 3, the major addition here is feedback of all the conversations back to knowledge base builder.</p><blockquote>It is important that the conversations with associated logs are fedback in the knowledge base that can be reviewed manually to curate golden question — answers.</blockquote><p><strong><em>4.1 Conversation Logs:</em></strong></p><p>The system keeps track of user interactions and chatbot responses. These conversation logs are valuable for:</p><ul><li><strong>Reviewing and Refining Responses:</strong> Human evaluators can analyze logs to identify areas where chatbot responses can be improved.</li><li><strong>Generating Golden Q/A Pairs:</strong> Subject matter experts in the field of language, climate, gender, agronomy, extension can co-create high-quality interactions from the logs as question-and-answer pairs. These “golden” Q/A pairs are then used to further train and improve the LLMs used for query orchestration and response generation.</li></ul><p><strong><em>4.2 Analytics:</em></strong></p><p>This module provides real-time insights into the performance of the chatbot system. These insights can be used to:</p><ul><li>Identify areas for improvement in content.</li><li>Monitor user engagement.</li><li>Track the effectiveness of the chatbot in answering user queries.</li></ul><h4><strong>Conclusion</strong></h4><p>This article we presented the flow of information to make <a href="http://farmer.chat/">farmer.chat</a> come to life. The generative AI space is evolving fast and so are we. This is just the overview and in the next articles we will take you to deeper dive into each of the modules. Stay tuned!!!</p><h4><strong>Big Thanks</strong></h4><p>To our gracious donors and government partners for making this possible.</p><p>To the readers!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ae2d62f1946" width="1" height="1" alt=""><hr><p><a href="https://medium.com/digitalgreen-techblog/unlocking-generative-ai-for-agriculture-farmer-chat-2ae2d62f1946">Unlocking Generative AI for Agriculture: Farmer.chat</a> was originally published in <a href="https://medium.com/digitalgreen-techblog">digitalgreen-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lord Of the Identities -1]]></title>
            <link>https://medium.com/@vineet.vinsing/lord-of-the-identities-1-6bc27713fc8d?source=rss-f162140d81db------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bc27713fc8d</guid>
            <category><![CDATA[identity]]></category>
            <category><![CDATA[background-checks]]></category>
            <category><![CDATA[ideas]]></category>
            <category><![CDATA[government]]></category>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[Vineet Singh]]></dc:creator>
            <pubDate>Fri, 10 Nov 2017 05:34:40 GMT</pubDate>
            <atom:updated>2017-11-10T05:34:40.908Z</atom:updated>
            <content:encoded><![CDATA[<h4>The fellowship for identity</h4><p>Continuing from the <a href="https://medium.com/@vineet.vinsing/lord-of-the-identities-af22a52bbb66">last post</a>, which was to serve as the seed, I will attempt to analyze the Gordian knot of different elements that should be brought over together for the task of integrated, singular, privacy shielded and self-sovereign identity. I am happy to have received questions by many people in person following my last post which I will try to attempt to answer here and in subsequent posts. This post is expository as well as descriptive in the sense of forming different models, the information by no means is complete and wherever possible, I have tried to give the link of the source of information. The focus is much lesser on solution than understanding the problem in detail.</p><h4>Terrain of Identity</h4><p>Let us start with understanding the terrain, which as per Sun Tzu is the most critical element.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*E0lDciboCQQai6lW264b4Q.jpeg" /><figcaption>The Palm Model of Identity (Image modified from<a href="https://www.youidraw.com/apps/drawing/?templateid=1309088704497ee4ac76f36618cae9c7"> youidraw</a>)</figcaption></figure><p><strong>The Palm Model of Identity: </strong>This model can be understood as sort of importance of various types of identities we hold.</p><p>First, we have a thumb which represents the Government Identity (GI) in the form of some ID proof given to the individuals by the respective governments and its agencies. This is the most important finger because this proves our citizenship, entitles us to participate in government formation in the form of voting, gives us the right to drive on road or open a bank account or travel etc. It also helps governments to track individuals and provide the targeted set benefits of the policies and provide negative reinforcements to other set who don’t comply. Therefore, this provides a necessary risk-benefit scenario for individuals to opt for and necessary spending by the Governments.</p><ol><li>The index finger is something we use most often daily and is the Virtual Identity (VI) of ourselves on net. The sites like facebook, Google provide the platform for us to create the identity that we use to login many other sites, interact with other people continuously in our daily lives.</li><li>The middle finger is the identity which we hold with the regulated government authorized systems like Banks and Telecom which we generally get only after due diligence from the service providers. We use this identity at the back and sparingly but is very important for us.</li><li>The Ring finger is again the privately regulated identity we have with the organizations like our employer some of the which are regulated and verified but not used very frequently.</li><li>The tiny/cute finger is the privately unregulated identity that we have and some times are ourselves unaware of, like our consumer ids with various merchants or service providers that they typically maintain to try to ensure loyalty.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*Xxi3j-PqnpMMMBn4yqxOTQ.jpeg" /><figcaption>Puzzle Model of Identity</figcaption></figure><p><strong>Puzzle Model of Identities: </strong>It is also important to understand the various ways the interaction of different identities we hold occurs. On one hand GI (shown in the puzzle on left in grey) are formed either in solo or some of them also take the information of one of the regulated VI like bank or Telecom, if not as data then as the necessary proof of some data like address etc.</p><p>The Regulated VIs in yellow also require the information from the GI and give back the same to other. For example, bank requires government Id as well as shares the overlapping information with the Mobile Network Operator which provides its own ID as contact for some Government ID.</p><p>Finally, we also have set of unregulated VIs, of which some are formed in isolation with the regulated VI as the red right hand corner or may require some interaction with government like our employee number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*zEvawsoBxQbKPfnL_u_VvA.jpeg" /><figcaption>Triumvirate functional relationship among different actors</figcaption></figure><p><strong>Triumvirate relationship: </strong>Apart from the types of identities we hold, there are also various players and actors in this complex landscape which can roughly be fit into an overlapping triumvirate model with a shadow on one side. Please note that this is the most simplistic model I could think of and may not necessarily be the best one to represent. There is one set of entities which acts as Identity Providers, these are primarily government whose pure mandate is providing a high level of assurance identity cards to individuals. The banks and mobile network operators also fall into this category giving an identity to an individual which is considered for most of the cases as an acceptable verifiable source. There are enterprise identity solution provider who generally provide the services at the back end of these identity providers. These are the entities not facing directly the end customers, a prominent example of such organization is <a href="https://www.morpho.com/en">safran/morpho</a> who provides the backend technology for <a href="https://uidai.gov.in/">Aadhar</a> program in India and other programs in various other countries. There are well established players like Safran, <a href="https://www.gemalto.com/">Gemalto</a> and new players like <a href="https://backstage.forgerock.com/">Foregrock</a> competing in non trivial manner in this space. The products vary from the hardware for acquiring biometric of individuals to the magnetic stripe card to the cloud based web solutions.</p><p>The second edge of the triumvirate represents the entities which provide sort of provisional Identity, if at all that is the right word to use. The level of assurance may be very low and is not the aim, the identity is provided to be used for accessing their own products or services on web. The prominent example being Facebook and Google, who in spite of not verifying the identity provide the single sign on for various multiple websites. There are again start up in this space in <a href="https://www.yoti.com/">Yoti </a>and <a href="https://shocard.com/">Shocard </a>which aim to provide a singular digital identity on the top of Government/Bank/Mobile identities to access all.</p><p>The third overlapping edge is that of verification service, prominent global example being <a href="https://www.trulioo.com/">Trulioo</a>. The function of these entities is to provide a verification service for the claim made by an individual. It can be seen that the way facebook/Google etc provide the single sign on, they act as verification service too. The mobile number, which is generally used as verification in multi-factor authentication, is on the other end of the spectrum where the user acts as the verification using the identity and not the service provider.</p><p>This model very roughly represents the functional relationship which is often overlapping between different players. The model can be made more complex by incorporating a fourth or fifth dimension of attestation service providers and also the back end service providers for the private network, IdaaS.</p><p>The different actors associated with each group in the triumvirate may not necessarily function the same way and may overlap with other edges in some functionalities. There is an increasing level of overlap and integration in different functions, for instance axis bank in India has started offering <a href="https://www.axisbank.com/webforms/ping-pay/faq.aspx">ping pay</a> service to transfer funds through fb/whatsapp accounts, thereby relying on the identity provided by the social media platforms on web which is non KYC compliant. Another interesting start up in the domain of identity verification is <a href="http://www.veridu.com/">Veridu</a> which aims to extract information from various social media sites with user’s permission and create trust/confidence score using ML tools. It is yet to be seen whether a probabilistic model of such type can be legally accepted for credit scoring that the banks can rely on for providing the credit scoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*CbXUQUSzy7vaKz_cTPDvng.jpeg" /><figcaption>Network of actors and consumers of identity</figcaption></figure><p><strong>Network of identity: </strong>Finally, the section of understanding landscape is incomplete without spending time to understand the network. Networks are pervasive and can be found everywhere, identity being no exception. In the simplified matrix on left, the rows are the various sets of entities. The columns represent the four different ways the identities can be accessed, the simplest being password and the level of security is generally believed to increase towards right as we go to Multi-Factor Authentication, Biometric and Password + Biometric. There is debate on the security of biometric and there is a level of merit on both sides of argument, for those interested can read a small crisp article <a href="https://heimdalsecurity.com/blog/biometric-authentication/">here</a>. Each of the players offer services to other players within this or outside this network and that is the source of revenue for all. The end customer at the bottom uses these identities interchangeably day in day out through different levels of security. The end customers generally receive the services for free or with minimal cost and possess the Identity from various actors.</p><p>Except government whose aim in providing the identity is generally for delivering targeted policy benefits as well as reducing cost and leakage in delivery, every other actor deals with identity as a tool for achieving their respective business goal. For backend service provider the third party could be a privately run company or the government as the consumer. For facebook, Google and likewise, it is the advertisement or marketing agencies or influencers who can shape the behavior of their end customer. The identity is a way to help these organizations understand user behavior and the focus is on more personalized experience on these platforms. The verification service providers at the other end of the spectrum also use the identity issued by different actors simply for their own business processes. This network model is critical for understanding the value proposition and also realize the complexity of the landscape. Not all players within each horizontal line are integrated and there is a huge gap in the vertical direction. <em>One thing that can be asserted with full confidence is that the value token in the network is the element of identifying an individual which is through different identities that an individual holds and the way individuals and systems interact</em>.</p><h4>Identity’s Mount Doom: Forged there — Finish there</h4><p>The value chain created is linear in current network. The government agencies provide identity to the individuals which can be used to get the identity on the KYC compliant service providers like Banks and Mobile Network Operators which are used as verification for the non KYC identity and also to get the identity on credit card or similar system in association. The starting point here is the GI (Government Identity) and the value is built on top of the same.</p><p>Although, web is free and anyone can come and surf or create identity here, there will be practical limitations to what activities one can do without GI. And as have been various cases recently, we should continue to see more debates between tech companies like Apple and Governments on ability to track individual and data associated with him/her. The web has been left dissociated from the GI in an attempt to expand the network primarily because of the high barrier of establishing the true verified identity of an individual. Technologically, the problem is two fold, how to authenticate that the GI documents are correct and second is how to ensure that an individual furnishing these documents is the one mentioned in GI. This problem become extremely high barrier to solve remotely. What further compounds the problem at global scale is the different level of trust established between the governments and the citizens and different level of assurance required by different regulatory bodies.</p><p><strong><em>The association between the GI and the other VIs is the Mount Doom. </em></strong>Most of the new age companies are trying to enter this domain through the use of new technology advancements like ML coupled with the continuous rise of mobile based internet access. The problems trying to be solved range from voice based security credentials to taking photograph and matching with the ones on the ID proofs and digitizing the id proofs scanned copy on cloud. The algorithmic way of solving is inherently probabilistic in nature and it needs to be seen how less can false positives or negatives be before this could be trusted by the regulatory bodies as well as society at large. One can argue that this integration is already done by Banks and mobile operators, but neither is this association complete nor is this association the primary task of their businesses. There is also a very strong unrealized factor of trust between the end customers and banks, for countries like Finland and Norway the government has nodded the use of bank id because of citizens’demands while in US, the survey shows 27 % Millenials trust banks compared to &gt; 70 % showing trust in Apple or Google. If at all, the flexibility in delivering products and services combined with massive reach of tech giants such as Amazon, Google, Facebook will enable them to compete with banks before banking system can figure out the value chain itself (Sorry Banking sector, harsh but true).</p><h4>The selection of the Ring bearer</h4><p><strong>Candidate 1- Government Initiatives: </strong>The government can play a key role in taking the initiatives in this direction as they are the provider of this identity. There are already success stories in the form of <a href="https://e-estonia.com/solutions/interoperability-services/x-road/">xroad</a>. There are also significant initiatives taken by other Governments, such as <a href="https://www.gov.uk/government/publications/introducing-govuk-verify/introducing-govuk-verify">UK’s verify</a> or US Government’s <a href="https://www.nist.gov/itl/tig">National Strategy for Trusted Identities in Cyberspace</a>. Then there is very successful enrollment by Government of India in <a href="https://uidai.gov.in/">Aadhar</a> program. While the UK and US governments initiatives are aimed with focus on private industry’s participation as the identity providers, the Aadhar system is totally wholly owned by Indian Government, providing the interface APIs for private sector organizations to build the platform on top of it. Although being developed by very limited number of Governments, these models provide a proof-of-concept models to enter this market. A case in point is Estonia’s program which is an integrated framework of Government and the private sector using the decentralized network. Aadhar’s success story is based on the fact that the model is based on three biometric details of fingerprints, iris scan and photograph of individual while the data stored is the Gender, Date of Birth, Name and address, making it extremely easy for enrollment. There have been questions raised over past few years on the privacy infringement and complete centralization of data as well as the analytics thereof, but primarily, it has received a good response from industry and general public. The GOI claims to have saved millions of dollars in last mile delivery of the subsidy schemes for vast populace, but the long term benefits are still to be realized. The Government has pushed for extensive linkage of Aadhar with the Banking system and mobile service providers and there is push for linking with insurance sector as well. Another salient point of Aaddhar that is often missed is that it is open for enrollment by every resident of the country and not necessarily the citizens. Estonia’s identity program has developed into matured e-residency program and several private sector players have started to shown interest and developing applications for e-residents of Estonia, making its scope even wider offering Country as a Service (CaaS) model. Estonia is also taking lead to leverage on eIDAS regulations of EU and bring in more nations to use its model, thereby deftly making for its small population. Finland and Sweden have already shown interest in adopting the system. On one hand it helps government identify the individuals with criminal backgrounds or those involved in tax evasion and on the other hand provides significant convenience to citizens in accessing the government services. The people have shown particular interest and the identity is widely used for banking and other connected services through one identity. Like aadhar, there is a physical id but unlike Aadhar, there is no biometric involved. Whereas challenges for Aadhar is privacy related issues and safety of centralized database, especially biometric, the challenges for Xroad is the scale that Aadhar has reached because of the sizable population and belief in democracy. <em>One particular and heartening change these initiatives have brought is the focus of policy formulation by the respective governments has shifted to creating standards for verification and building trust framework addressing the privacy from directly providing the identity.</em></p><p>The major drawbacks with the system is the non global nature and varying level of governance and regulations from country to country. As per recent reports form UN, a total of 1 to 1.5 billion people in the world are without identity. A global consensus on the framework is difficult to arrive at, for example, the general population in India is much less concerned about the privacy issues compared to that of EU or US and it may just be cultural difference.</p><p><strong>Candidate 2 — Godzilla of Identities(Facebook): </strong>Facebook with its subsidiaries as whatsapp, instagram and messenger has the largest user base in the digital world and truly is a Godzilla. Furthermore, the user base keeps growing with alarming rate and there is strong retention of the users on the network. Facebook is uniquely positioned as compared to other giants such as Google, Apple, Amazon or Microsoft and can build massive systems that can be used for daily transaction. This makes facebook a powerful organization in provisioning identity which cuts across countries. However, the biggest disadvantage is the trust factor and the fact that facebook’s primary business has to do with leveraging identity for advertisement purpose by third parties. Coupled with the fact that it forms a centralized private sector organization makes it a risky choice.</p><p><strong>Candidate 3-Mobile Connect by GSMA: </strong><a href="https://www.gsma.com/identity/">Mobile Connect</a> is one of the three main focus areas of GSM Association. GSMA is the administrator of TAC/IMEI code database that uniquely identifies the wireless device. The verification service of IMEI is already being widely used to identify the model and make of the device. GSMA comprises of more than 800 mobile network operators and another 300 affiliate members with the aim to provide common standard to achieve scale and interoperabillity of mobile technologies. Identity is one of the three focal programs along with future networks and IoT by GSMA. The biggest favorable factor for GSMA is the rising number of mobile users across world with significant larger number of people accessing internet through mobile. Furthermore, identity in future will not be limited to humans alone and will have to incorporate devices with rising penetration of IoT, making it uniquely positioned and the only substantial challenger to facebook. While fb operates today at extremely low level of assurance, mobile operators today operate in most of the countries complying with strong regulations also ensuring the privacy of the individuals, making them represent two extremes. The mobile network operators have significant inroads and physical presence in the rural sector in emerging economies as well.</p><p>However, there are challenges in terms of extending the mobile operators in the digital marketing value as evident by the failed effort of GSMA to capture the market. The deeper association of the mobile network operators with the customer may be required and is going to be definitely a point of concern as the operators bear the responsibility of transmitting the messages aimed at destabilizing the law and order situation in different countries. another major challenge is the “softwarization” (my own word) of SIM card. The physical chip that brings the operators and its customers in contact through various affiliates at various small or big shops can be broken if it goes into the software stack of device manufacturers, especially with more players in IoT space.</p><p><strong>Candidate 4-Truly decentralized network: </strong>The internet and subsequently world wide web have proven the effectiveness of an open decentralized system. All the above candidates represent either the extension of Government or creating a framework where identity is provided by the centralized system like fb or GSMA. Estonia has already proven the power of decentralized system in action. The salient points for each candidate is trust through regulations by government, social network scale by fb and privacy plus deeper penetration in the territory by mobile network operators. Each have its own limitations and challenges, the Government is limited by the political boundaries, a web based company is limited by the level of assurance required while the mobile operator will have difficulty in carrying the entire responsibility. The mobile operator also face dilemna of opening the network for marketing, thereby reducing the privacy guarantee edge that they have over the like of fb but missing the revenue opportunity.</p><p>If we carefully examine, all the above candidates would have to create a single container that they have to maintain replacing the network of identities. What if there is a truly decentralized network on which everyone can connect. In fact, the private sector organization are themselves legal entity who can use this identity network for registration just like individuals. This entire network will be based on being attested by other individuals and the information shared through Zero Knowledge Proofs for verification agencies who can focus on running analytics and fraud detection techniques.</p><p>There are significant challenges in Decentralized technology, first and foremost being the scalability. <em>The more the number of nodes in a network the more is the security but the more time consuming and difficult is the consensus. This inherently opposing forces lead to an equilibrium in the number of transactions per second</em>. Second, the current state of blockchain has limit of the size of the blocks and the higher the size, more difficult is the distributed storage. To this effect, most of the blockchain based decentralized system have a limit on the block size. Third, the end user comfort and the means to incorporate biometric as part of the signature.</p><p>The positive side is that there are lots of developments taking place by a very active and increasing community globally in this sector specifically targeting the technical issues. There are hosts of stuff being tried out and quickly adopted in not so liked cryptocurrency market. This is another area where the blockchain has to very clearly distinguish itself from the volatility of cryptocurrency and try to market the core tech to various stakeholders. The likes of <a href="https://www.evernym.com/">evernym </a>and <a href="https://blockstack.org/">blockstack </a>have made significant progress in this domain and seem to move in right direction.</p><p>Only if I may draw some parallels, the Government seems like elves, the web based social network can be likened to the men, and the Mobile Network Operator to the dwarfs who have dug deep. The decentralized network puts the entire onus on the common individual, can it be the Hobbit?</p><h4>The fellowship</h4><p>It seems clear that this task can’t be achieved without the collaboration of different actors. There is tremendous value and also power and control for one institution to grow too big but the identity of an individual on any network should reside with the individual to make it self -sovereign, which at present seems to be achievable by a true decentralized system. The journey could be long and arduous but there is hope with the collective effort of a very effective community.</p><p>One of the critical questions is how to ensure adoption of this system. In my opinion, a product gets to the scale by three main factors: convenience, incentive or gratification. The mail or chat provides a great degree of convenience in exchanging information, so does the mobile, or the ticket booking websites. When convenience is an order of magnitude different from the current systems or processes, its an instant recipe of success. If a product doesn’t offer the same difference in the level of convenience, it generally requires incentive. The prominent example being the e-commerce companies offering big incentives in addition to the convenience. Th network effect based firms like Uber too offer convenience to the riders but a good deal of incentives for the drivers to expand the network. The third part of gratification is much more experiential part than the other two and the gaming industry is the best example. The level of gratification in scoring higher in temple run or candy crush is much higher compared to ordering the stuff on demand on e-commerce or sending message on whatsapp. Twitter seems to be one platform to offer an instant gratification on speaking your mind on any trending issue and answering people you agree/disagree with.</p><p>An identity network doesn’t provide any of these instantly. In the long run, sharing documents or attesting documents digitally is a level of convenience that will be appreciable but it can’t be compared with other products. This is where the institutions can play a very important role by acting as anchors. The mobile payment wallets are case in point in getting adopted, many of the users didn’t install the app on their own but created wallet inside an existing app like Uber or a food ordering or e-commerce app. The government, the banks, mobile service providers or even education institutions can play the most important role as an anchor for all such systems, making users to create a self sovereign identity.</p><p>In terms of convenience to the businesses, there is a very strong use case of automating KYC by the FinTech organizations. However, targeting this as the use case in general creates a narrow version of complying with the regulations of a particular country. The approach should be the other way, creating a network and then opening it for Banks or any other service provider to integrate with the Banks as per the KYC guidelines of country. Same way, employee background check and verification or digitally attested copies of transcripts are an important use case which provides convenience to the businesses and institutions, but it narrows down scope of the network to one function in the triumvirate. The network should be made open for anyone to join and perform background check and verification or create digital locker.</p><p>I hope that this post is helpful in giving a complete picture. The approach for the solution and comparison of different approaches will be the focus in next post. I will be very happy to take up the questions in comments section and get feedback.</p><p>To be continued…..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bc27713fc8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lord of the Identities]]></title>
            <link>https://medium.com/@vineet.vinsing/lord-of-the-identities-af22a52bbb66?source=rss-f162140d81db------2</link>
            <guid isPermaLink="false">https://medium.com/p/af22a52bbb66</guid>
            <category><![CDATA[background-checks]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[identity]]></category>
            <category><![CDATA[trust]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Vineet Singh]]></dc:creator>
            <pubDate>Wed, 25 Oct 2017 17:57:04 GMT</pubDate>
            <atom:updated>2017-10-25T17:57:04.833Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6dJAV2IgfsLf35JE29NYBg.jpeg" /><figcaption>source: <a href="https://www.theodysseyonline.com/digital-identity">odysseyonline</a></figcaption></figure><h3>One identity to rule them all</h3><p>Let us sit back, relax and just think for a while who we are and how we claim to be who we are in our day-to-day lives:</p><ol><li><em>Scene 1: </em>We go for a formal meeting and exchange our business cards. Our business card tells our name, designation, phone number, mail id, company name etc. We are saved in someone’s account as our number, mail id and that is us for each other. We share this information with others and know that the person can only contact me on this phone or mail but can’t hack it, <em>we trust the service providers</em>.</li><li><em>Scene 2: </em>We want to open a bank account and we give required documents issued by different government departments as per KYC. We buy a home and we want to avail a loan and we again furnish the same documents. We make an investment and we again furnish almost the same set of documents. The documents that we have is a proof of who we claim to be, and often also have other details of our Parents name, spouse, children, date of birth etc. <em>These documents are very well regulated by the government that we trust.</em></li><li><em>Scene 3: </em>We change the job and are asked to give our experience letter on company letter head and give the same set of documents that we gave to our telecom service provider or the bank service provider. The company accepts it because we are giving it, <em>we possess those documents and we give all the information in our presence, trust is implicit.</em></li><li><em>Scene 4: </em>We love to be connected in our digital world through our various social media apps. Almost all of these apps/websites know and agree you are what you say you are, there was no check wherever and whenever you created yet another user id. At a single point of time, we could be logged in Google, facebook, twitter, amazon, whatsapp, linkedIn, instagram, Uber, Ola, gaming apps and so on……. God knows what. Many people keep lot of passwords saved in drafts of these mails, <em>they trust us and we trust them, no questions asked.</em></li><li><em>Scene 5: </em>Every time we make some payment or login from a different device, we might be asked to enter OTP as the two factor authentication. We get the OTP on our phone number through SMS, the number that we obtained submitting documents issued by certified government institutions. <em>We don’t mind this extra security step and feel more trust in the system.</em></li><li><em>Scene 6: </em>Almost all these apps have some form of credit card details being used and saved except perhaps expiry date and 3-digit cvv number and we pride in doing things as fast as we can, booking flights on the go, paying utility bills, booking movie tickets and so on….. <em>We trust them that it is secured.</em></li></ol><p><strong><em>Is this how it should be?</em></strong></p><ol><li>Why can’t I just say to my bank “Hey, look I submitted these docs with my telecom operator and he verified it” and the telecom operator says, “ Oh yes, I did…” and bank says <strong>“yes, you are a reputed service provider” and verifies the data and accepts it</strong>.</li><li>My phone number or my bank account number/CRN number are as much a virtual identity as gmail/facebook/amazon, why should I get authentication to my mail id or mobile. If my device is lost and is hacked, this authentication is of no use. Why can’t there be some other way to say <strong>“yeah, its me”</strong> in digital world as we do in physical so that trust is implicit.</li><li>My documents are all issued and maintained by some authorities who I trust, what if they fail or are hacked some how !! <strong>Is the data in all my documents coherent? Is the address on my passport same as driving license is same as Aadhar? Highly unlikely, how does anyone know which is the correct one. Is there a single source of truth?</strong></li><li>Every document was formed at different point of time and some information was taken from some document. Not a single one has complete and correct information. <strong>Why should I be not concerned that someone may have used my address proof by submitting a forged electricity/gas/water bill? I mean, just look at how secured these proof of address are, literally ;)</strong></li><li>If I change my address or my phone number or my name, can it not be updated with every virtual identity? <strong>Why should I go to each and every place and tell that I have updated some info?</strong></li><li><strong>Why should I as a company rely on as tamper proof a document as company letter head of another company !!</strong> How can I be sure that the educational qualifications of the person are indeed correct and the copy of transcripts is original?</li><li><strong>Why can’t my relationships be added simply with other people and stored at one place</strong> from where it can be retrieved and updated and modified as we move along in the life?</li></ol><h3>What’s amiss?!!</h3><blockquote><strong><em>Trust</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6NWVY2fMNwTuCRys0nx5Q.jpeg" /><figcaption>source: <a href="https://identitywoman.net/the-identity-spectrum/">identitywoman</a></figcaption></figure><p>We require identities on various platforms and that can’t be changed but we must realize that all these identities, be it the bank/ phone number/ facebook/ google, <em>all of them are virtual identities and extension of our true self</em>. The documents issued to us are certificates by the respective Government agencies and these are our “documental” identities stating some information about us. When a bank verifies my documents, there is an information transaction taking place: <em>I tell them that this is who I am, they ask for supporting documents from a trusted third party which I give them, I sign the copy of those documents, they verify that the details mentioned in the document copies are indeed matching what we are stating, they then provide me a bank account number and a customer id through which I can access my bank accounts</em>. When this transaction takes place, they place trust on the central authority who has issued the documents and <em>since I possess those documents, they trust me</em>. The fact that I possess those documents is indeed implicit in building that trust. <em>However, this trust system is sitting in a “silo” with the service provider and I have to rebuild the same trust every time I opt for another service.</em></p><p>In the digital world, I can go on and create multiple user ids, which can be used to access and create user ids for other platforms, no trust is placed till the time we are doing financial transactions. Isn’t this fundamentally ripe for all sorts of attack and fraud. More often than not, we share these virtual identities as a way to contact us by our service providers like Banks. <em>When we are doing any financial transaction the bank is “authorized” to do the necessary transaction and to authenticate, it uses one of our virtual identities that we have provided, </em><strong><em>assuming that our other virtual identity is US</em></strong>.</p><p>Internet is not owned by anyone, yes there are big firms that exercise centralized power, but there is nobody that owns internet. What we require is one single digital identity which we own, which a) acts as a locker of information of our “documental” identities b) is owned by us as we own the documents, that is, we can create/ update/ share/ purge c) is native of internet, nobody owns the system on which the identity is created d) is secured enough to be trusted. Loss of credentials in any virtual identity does not lead to loss of any shared information on this identity. e) is the super set of all the information in all the virtual identities. f) can act as a watermark on critical digital documents.</p><blockquote>Such an identity can be used to authenticate any transaction made by any virtual identity on net and the trust should inherently be there. The chances of this identity being compromised should be astronomically low, so low that it is futile to attempt something like that.</blockquote><h3>BLOCKCHAIN Solution</h3><p>I would leave the introduction of blockchain for readers to study from various sources and articles on net and give a very basic understanding. Blockchain enables Distributed Ledger Technology (DLT) where everyone has a copy of the global ledger and to hack this requires hacking every single copy. The information in the DLT is stored in blocks of valid transactions which get closed in some average time (seconds to minutes) when everyone reaches a consensus and point to the previous blocks in a chain like a hashed linked list. To hack the system means to change entire series of blocks till genesis in a small time frame on all the nodes, which makes it extremely improbable. The consensus algorithm is the key which is achieved by the most famous Proof-of-Work like in Bitcoin, but there are other ways like PBFT/DBFT/POS etc which have been tried out or are under development. The reason consensus is the most important part of DLT because this is what keeps the system secured and is achieved through incentives for people trying to keep the system secured, the value of the incentive is because the system is healthy and secured. Well, let me explain the last sentence in another way, a cyclic process: <em>Secured and healthy system requires people/node to work to make it secure which in turn requires some incentives to be given to the nodes on the network which in turn requires that people value the incentives which is only possible if the system is healthy and secured.</em> The incentives and the tasks that nodes have perform to make network secure varies as per the consensus algorithm. So, Bitcoin essentially is a token of incentive for making the Bitcoin network secured and this token is valued by the people who are trading goods in Bitcoin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*D6lm3nIl0cMy98WcRlODCA.jpeg" /><figcaption>An example of different blockchain interaction between DI network and VI networks</figcaption></figure><p>We have just described “documental” identity (DocI), provided by the government agencies and virtual identity (VI), that we create with lots of independent service providers. We have defined loosely what we want from the new Digital Identity (DI): one identity to rule them all. Imagine, that you can create your DI on net, filling details of your DocI and other information and create a key — value pair through which you can share part of information securely. Imagine that this data is sharded and distributed in multiple copies with different people so that nobody has any significant piece of this data and every time you retrieve this data, you do it from multiple sources (~ like BitTorrent). The DIs are in a block chain and distributed this way throughout the network and it is improbable to break the security of this network. The list of DI created in a block is governed by consensus algorithm and size of the blocks. Just like cryptocurrency, a user can do transaction of the information field in the DI securely. The transaction here is CUSP (Create, Update, Share. Purge). Let us say that the user wants to create a VI with some service provider, the service provider can have its own “permissioned” blockchain in which the VIs are created in blocks. If the user wants, he can share the information with VI service provider and if the VI verifies this information, it can digitally sign it. Hence, the ledger of a user would look like this: Info 1,2..n created -&gt; info 1,2 shared with VI service provider and verified -&gt;….</p><p>In this new system, the DI can also act as an authentication system, were the information is hashed and passed on. Hashing has two properties that are very useful: a) if x is not equal to y, then H(x) is not equal to H(y) and b)if you know H(x), you can’t find x (If you are a mathematician, you probably find this blasphemous statement and hold my neck, but wait that is what it is for a layman :)). Now, when we share the hash of information, the information can’t be revealed and if we give the same hash of information on the service provider’s network, it just needs to be checked that two hashes are same and H(x) from DI is digitally signed by the DI holder.</p><p>We can authorize the VI to share the info with it which needs to be authenticated by DI that we hold. So, if I have indeed shared my info with a telecom operator and go to some bank and share the info, bank will accept if and only if it is authenticated by a DI. So, no information can be transmitted or accepted by any VI without verification from DI.</p><p>The DI can be locked through hashed information of biometric as per the user’s wish.</p><h4>Actors and players in this network</h4><ol><li>Validators: in the DI network, there can be people who are validating that they know this particular user and verified this.</li><li>Attesters: VI service providers act as attesters of the information of a DI and create a VI. The VI service providers can pass on this information as if attested by them on request of VI.</li><li>Verifiers: The VI service providers can have nodes or users/agents who can act as verifiers on the network.</li><li>Government Agency: The government agency can act as issuers of DI after due verification.</li></ol><h3>What does it hold for the future</h3><p>Identity is a key concept in today’s society and one’s identity is associated with basic benefits and also risks. If you have no identity, no record whatsoever, you can do anything without getting traced but it also implies that you will be devoid of any social guarantee schemes and citizenship rights of the nation. Citizenship, usually associated with birth gives the first and basic record and identifies an individual. As more identity documents are generated, there is either an increase in risk/benefit to the identifying party or the identified party. If one gets a PAN card or a passport, there is risk of being subject to scrutiny by the government on the income/travel etc, but also comes certain benefits. Needless to say the risk associated with identity are far higher than benefits and act as a deterrent to not create a bad history. It also favors the rich and powerful who have good history compared to poor or less powerful who have bad or no history at all. The risks for rich can also be lower due to better access to legal resources and influence.</p><p>The concept of a single unifying DI makes this a level paying field and gives access to different service providers to act as attesters and allows self-assertion by every individual. The individual bears the responsibility for any wrong information provided and the service providers can be enabled to offer the service to attest an individual, be it employment or bank details or phone number. Another aspect is that this would retain the anonymity of an individual as well as provide trace for regulatory authorities. Furthermore, in the long run the relationships and social validation makes this a network which acts as self registering system, does away with many registrar offices for marriage and forms the backbone of legal process in transfer of immovable or movable properties. To list few of the processes in various industry sectors that would get disrupted:</p><ol><li>KYC related process get seriously simplified and assured.</li><li>Background verification of employees.</li><li>Extension in web browser to use DI to access net. All the login ids can be traced to the DI.</li><li>Cyber security/patrolling.</li><li>Education records of individual.</li><li>Unified medical records.</li><li>Trust in P2P lending network.</li><li>International trade/travel and granting of Visa by the government.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af22a52bbb66" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>