<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sravya Munnangi  on Medium]]></title>
        <description><![CDATA[Stories by Sravya Munnangi  on Medium]]></description>
        <link>https://medium.com/@munnangisravya?source=rss-635650226706------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*dWMrPfpH63AXH0Vi</url>
            <title>Stories by Sravya Munnangi  on Medium</title>
            <link>https://medium.com/@munnangisravya?source=rss-635650226706------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:45:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@munnangisravya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ASCII Smuggler — The INVISIBLE prompt injection.]]></title>
            <link>https://medium.com/@munnangisravya/ascii-smuggler-the-invisible-prompt-injection-d4188d2ff951?source=rss-635650226706------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4188d2ff951</guid>
            <category><![CDATA[prompt-injection-attack]]></category>
            <category><![CDATA[cyber]]></category>
            <category><![CDATA[penetration-testing]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sravya Munnangi ]]></dc:creator>
            <pubDate>Sat, 18 May 2024 00:43:19 GMT</pubDate>
            <atom:updated>2024-05-18T00:43:19.530Z</atom:updated>
            <content:encoded><![CDATA[<h3>ASCII Smuggler — The INVISIBLE prompt injection.</h3><p>Hello and welcome to my new blog post. Today I am going to discuss a future threat which is invisible. With greater power comes greater responsibility, and upholding this responsibility is tough. Similarly, with growing technology come greater challenges. In the present scenario, we are facing troubles with intangible prompt injections. These attacks are invisible. Yes, invisible to human eyes.</p><p>Almost every graduate, even if they are not academically inclined, has probably heard of ASCII, which stands for American Standard Code for Information Interchange. ASCII includes numerical values from 0–9, alphabets a-z, A-Z, and punctuation marks. ASCII smuggling often involves the use of Unicode tags. Unicode stands for Universal Encoding Standard. These two encoding standards come into play in prompt injections that are intangible.</p><p>Okay, I know diving deeper into ASCII might get little boring but, let me tell you the interesting one.</p><p>The attackers or the malicious actors can perform invisible prompt injections by using this unicode tags.</p><p>Since it is interesting, I just dived deeper into the subject, discovering that there are different versions of Unicode available, such as UTF-7, UTF-8, UTF-16, and UTF-32. Unicode supports various languages like Latin, English, Arabic, and many others. Essentially, Unicode is used to encode scripts, emojis, and other special characters.</p><p>So, you might be wondering how it actually works. To get a hands-on experience, I tried this on Claude AI, which is developed by Anthropic, and guess what? I was successful.</p><p>I used my evil mind and tried ASCII smuggling in Claude AI by encoding the text for the operation I intended to perform. I opened the ASCII smuggler tool, entered the action I wanted to perform, copied the encoded content, and then placed this content in Claude.</p><p>Here are the images of the attack</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mmZM46JJBrN0PakMJXrDoQ.png" /><figcaption>Input where I placed the intended content to be encoded and copied the encoded output.</figcaption></figure><p>When I just pasted this encoded content in Claude it gave this output which I actually desired.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*5XCd_ovOET30ON9_zQ0PhQ.png" /></figure><p>You might think where did Inject this encoded characters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*it0EORPBIQSzXSFi1KpLLA.png" /></figure><p>The operation I intend to perform is to ensure that, even when someone tries to summarize or elaborate on the content, the output will always be something like ‘banana,’ regardless of the original content, by injecting the encoded payload somewhere in the middle.</p><p>And it can also be done by using a script</p><pre>def convert_from_tag_chars(tagged_string):<br>    return &#39;&#39;.join(chr(ord(ch) - 0xE0000) for ch in tagged_string if 0xE0061 &lt;= ord(ch) &lt;= 0xE007A)<br><br>tagged_input = input(&quot;Enter a string of tagged characters to convert to ASCII: &quot;)<br>ascii_output = convert_from_tag_chars(tagged_input)<br>print(&quot;ASCII output:&quot;, ascii_output)</pre><p>while running the above code you are supposed to enter the text that needs to be encoded. You can craft your payloads by running the script.</p><p>What else can we do with this vulnerability?</p><p>When I dug deeper into the subject, I discovered that by using this vulnerability, we can change the skin tones of emojis. Yes, emojis these days come with different skin tones like fair, medium, dark, etc. When we inject a payload in between, we can observe the changed output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*EUVI73jc5Mccl6H3Y7-Peg.png" /><figcaption>In the second prompt I just injected the encoded characters to change the skin tone of the emoji.</figcaption></figure><p><em>What if this vulnerability continues?</em></p><p>If this continues to happen data theft , Integrity and also the reputation will be at stake at the same time it leads to huge losses of those who are unaware of this attack.</p><p>Picture this :<em>Your email has an AI assistant and I have injected a payload asking to delete all the emails from a particular person. Since it is invisible and on getting executed the AI assistant will do that for me. But the victim will have No clue about what just happened</em>.</p><p><em>How to mitigate this Vulnerability?</em></p><p>Either the Unicode tags must be displayed in the front end or these encoded characters must be dropped before it is sent to LLM.</p><p>Happy learning and cheers for a secured future…. Thankyou :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4188d2ff951" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Supply Chain Vulnerabilities in LLMs]]></title>
            <link>https://medium.com/@munnangisravya/supply-chain-vulnerabilities-in-llms-0f5afd4fcb13?source=rss-635650226706------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f5afd4fcb13</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[penetration-testing]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[supply-chain]]></category>
            <dc:creator><![CDATA[Sravya Munnangi ]]></dc:creator>
            <pubDate>Sat, 27 Apr 2024 08:29:52 GMT</pubDate>
            <atom:updated>2024-04-27T08:29:52.673Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s digital age, large language models (LLMs) like GPT-3 and BERT are transforming the way we interact with technology. These models power chatbots, search engines, and virtual assistants, making our lives easier and more efficient. But with great power comes great responsibility, and in the world of AI, one of the most significant challenges lies in securing the LLM supply chain.</p><p><strong><em>What is Supply Chain?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EMxqVV0TWrS7xOjzCs5-LA.png" /><figcaption>Supply chain vulnerabilities in LLMS a cyber threat (image from srm-solutions)</figcaption></figure><p>A supply chain refers to the network of processes and entities involved in the production and delivery of a product or service. It includes everything from the procurement of raw materials to manufacturing, distribution, and delivery to the end customer. In the context of machine learning and AI, the supply chain encompasses all the steps and components involved in building, training, and deploying models. In the same way we have different processes involved such as</p><p><strong>Data Acquisition</strong>: It involves the process of collecting and sourcing the data required to train and develop machine learning models, including large language models (LLMs). It encompasses various steps and considerations to ensure that the data used is relevant, accurate, and comprehensive.</p><p>This will determine the where the data will come from, such as internal databases, publicly available datasets, third-party providers, or crowdsourced platforms. Also it considers which type of data is required for a specific application such as text, images, audio etc.</p><p><strong>Model Development</strong>: This involves a series of steps that lead to the creation of a functional AI model. This process encompasses data collection, preprocessing, model selection, training, evaluation, and tuning.</p><p><strong>Deployment and Maintenance</strong>: Deployment refers to the process of moving a machine learning model from a development or testing environment to a production setting, where it becomes a live, operational system. where as Maintenance refers to the ongoing tasks required to keep the model running smoothly and efficiently after deployment.</p><p><strong>Infrastructure and Dependencies</strong>: Infrastructure and dependencies refer to the underlying hardware, software, and services required to develop, deploy, and maintain large language models (LLMs). They form the backbone that supports AI applications, ensuring seamless operation and communication between various components.</p><p><strong><em>What are the vulnerabilities involved in Supply chain?</em></strong></p><p><strong>Data Integrity: </strong>Training data is the lifeblood of LLMs. If this data is compromised through manipulation or poisoning, the model’s performance can be skewed, leading to biased outcomes or inaccurate predictions.</p><p><strong>Model Tampering</strong>: Pre-trained models sourced from third parties may be vulnerable to tampering attacks. A malicious actor could alter the model to produce harmful or unexpected outputs, impacting its overall reliability.</p><p><strong>Infrastructure Weaknesses:</strong> Security vulnerabilities in the infrastructure supporting model deployment can lead to unauthorized access or attacks on the system. This includes issues with servers, networks, and software components.</p><p><strong>Plugin and Extension Risks:</strong> LLM plugins and extensions, especially those from third parties, can introduce new attack surfaces due to insecure design or lack of proper vetting. These plugins may allow attackers to exploit vulnerabilities and compromise the system.</p><blockquote>As a penetration tester, I don’t think it’s easy to exploit these vulnerabilities. It would be like someone tampering with a product while it’s being developed or trained. Only individuals with close access to the model could feasibly manipulate it. For example, in Reinforcement Learning from Human Feedback (RLHF), if someone with malicious intent provides malicious feedback during the training process, it could affect how the model learns.</blockquote><p>So, this is Supply chain in LLMs which interlinked with one another. and this vulnerability is listed in OWASP top 10 LLM vulnerabilities. I don’t know how could someone practice this.</p><p>If you have come this far then you can👏</p><p>Thankyou :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f5afd4fcb13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Text from Images: Optical Character Recognition (OCR)]]></title>
            <link>https://medium.com/@munnangisravya/unlocking-text-from-images-optical-character-recognition-ocr-995b28b64e65?source=rss-635650226706------2</link>
            <guid isPermaLink="false">https://medium.com/p/995b28b64e65</guid>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sravya Munnangi ]]></dc:creator>
            <pubDate>Sat, 06 Apr 2024 05:58:29 GMT</pubDate>
            <atom:updated>2024-04-06T05:58:29.166Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever wondered how we can extract text from images or documents that lack editing functionality? In today’s digital age, where vast amounts of data are exchanged in various formats, this capability is nothing short of amazing. From handwritten notes to printed documents, extracting necessary information from textual content can be a challenging task. This is where Optical Character Recognition (OCR) emerges as a transformative technology. Let’s explore OCR, including its definition, functionality, and the companies leading its implementation.</p><p><strong>Defining OCR</strong>: Optical Character Recognition, or OCR, is a technology designed to convert scanned documents, images, or other visual representations of text into machine-readable text. In essence, OCR enables computers to interpret and extract textual content from non-editable formats, making it searchable, analyzable, and manipulable.</p><p><strong><em>How OCR Works?</em></strong></p><p>At its core, OCR operates through a series of intricate processes aimed at deciphering text from images:</p><p><em>Preprocessing</em>: The input document undergoes preprocessing steps such as noise reduction and enhancement to improve the clarity and legibility of the text.</p><p><em>Text Detection</em>: OCR algorithms analyze the preprocessed image to identify regions containing text, distinguishing text elements from other visual components like graphics or background noise.</p><p><em>Text Recognition:</em> Within the detected text regions, OCR algorithms employ pattern recognition and feature extraction techniques to interpret individual characters and words. This involves matching the extracted features against known character patterns to determine the corresponding textual content.</p><p><em>Post-processing</em>: Following text recognition, OCR may apply techniques such as error correction, spell checking, and formatting adjustments to refine the accuracy and quality of the extracted text.</p><p>Understanding <strong><em>Document AI</em></strong>: Document AI represents a comprehensive suite of tools and <em>APIs developed by Google Cloud Platform</em> to empower businesses with advanced document processing capabilities. While OCR is a fundamental component of Document AI, its scope extends far beyond simple text extraction. Document AI leverages machine learning and natural language processing models to understand the content, structure, and context of documents. This includes functionalities such as entity extraction, document classification, and structured data extraction, enabling businesses to derive actionable insights from their document repositories.</p><p><strong><em>Leading Companies in OCR Implementation</em></strong>: Some leading companies have started using OCR technology to make managing documents easier, help people get information more easily, and make their work more efficient. Some of these top companies include:</p><p><strong><em>ABBYY</em></strong>: ABBYY provides different kinds of software that help different types of businesses. These software tools are designed to turn paper documents into digital files, automatically take out important information, and make business tasks run smoother.</p><p><strong><em>Adobe</em></strong>: Adobe’s Acrobat OCR feature helps people turn scanned documents and images into PDF files that can be edited and searched easily. This makes it simpler for people to work together on documents and integrate them into their workflows.</p><p><strong><em>Microsoft:</em></strong> Microsoft’s Azure Cognitive Services have OCR capabilities. This means developers can use them to add text recognition features to their apps. For example, they can make apps that can scan receipts, process forms, and extract content from documents.</p><p><strong><em>Google</em></strong>: Google’s Cloud Vision API uses advanced technology to pull out text/words from pictures. This helps companies get useful information from images.</p><p>Google Lens provides a practical example where everyone can observe text or extract words from an image.</p><p>Thankyou :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=995b28b64e65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Insecure Plugins it’s vulnerabilities, exploitation and mitigations]]></title>
            <link>https://medium.com/@munnangisravya/insecure-plugins-its-vulnerabilities-exploitation-and-mitigations-c94192adcb40?source=rss-635650226706------2</link>
            <guid isPermaLink="false">https://medium.com/p/c94192adcb40</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[pentesting]]></category>
            <category><![CDATA[plugins]]></category>
            <dc:creator><![CDATA[Sravya Munnangi ]]></dc:creator>
            <pubDate>Sat, 23 Mar 2024 08:18:08 GMT</pubDate>
            <atom:updated>2024-03-23T08:18:08.416Z</atom:updated>
            <content:encoded><![CDATA[<p>In the landscape of digital platforms and applications, plugins play a significant role in enhancing the user experiences and functionality extension. However, what is a plugin? , how can the misuse of this plugins leads to serious consequences? and these Insecure plugin design is in OWASP top 10. lets dive into the subject.</p><p><strong>what is a plugin?</strong></p><p>Plugins are like add-ons features which are handy for various software systems, enabling them to perform specific tasks which are beyond its core functionalities and are automatically used by the main system(which is also called Model) . This main system decides when to use these plugins, the user doesn’t control it directly and these plugins enhance the capabilities of a system.</p><p><strong>Uses of Plugins</strong></p><p>Plugins find applications across a wide range of domains, including web browsers, content management systems (CMS), and multimedia software. For instance, in web browsers, plugins such as ad blockers(which prevents ads), password managers, and video players enhance browsing experiences by offering functionalities not inherently present in the browser itself. Similarly, CMS platforms like WordPress leverage plugins to add features like contact forms, SEO optimization, and e-commerce capabilities, empowering users to create dynamic websites with ease.</p><p><strong>Vulnerability and exploitation:</strong></p><p>Plugins can pose security risks if not implemented and managed properly. When attackers want to take advantage of weaknesses in plugins, they use different methods. One way is by messing with the boxes where you type things in (like forms or search bars) to put in harmful instructions. This tricks the plugin into doing things it shouldn’t, like running code it’s not supposed to. Also, attackers find ways to get around the security checks in plugins to get more power than they should have. This lets them do things they’re not supposed to do and ignore the usual protections.</p><p>By doing these things, attackers can mess up the safety and privacy of the system. This can really hurt the people and groups using it, causing big problems that are hard to fix.</p><p>Now, let’s dive deeper into this:</p><ol><li><em>Manipulating Input fields</em>: Attackers can tamper with text boxes or forms on websites or applications. For instance, they might inject scripts or commands into these fields. When the plugin processes this input, it might execute the injected code, allowing the attacker to perform unauthorized actions. This could range from stealing sensitive data to taking control of the entire system.</li><li><em>Exploiting Authentication Weaknesses</em>: Plugins often have authentication mechanisms to control access to certain features or resources. Attackers try to find flaws in these mechanisms to bypass authentication or gain higher levels of access than they should have. This could enable them to manipulate data, install malware, or perform other malicious activities without proper authorization.</li><li><em>Impact on System Integrity and Confidentiality</em>: When attackers successfully exploit plugin vulnerabilities, they can compromise the overall security of the system. They might alter or delete important data, disrupt normal operations, or gain access to confidential information. This not only damages the reputation of the affected users or organizations but also poses significant risks to their operations and privacy.</li></ol><p>What is the core point where this attack can take place?</p><p>These vulnerable plugins accept<strong><em> free text inputs</em></strong> <strong><em>from the model</em></strong> <strong><em>without any validation.</em></strong></p><p>Consequences of vulnerability exploitation</p><p>The consequences of plugin exploitation can be severe. Data breaches may result in the exposure of confidential information, threatening user privacy and trust. System compromise can lead to disruption of services, financial losses, and reputational damage for organizations. Furthermore, unauthorized access to sensitive data may facilitate identity theft, fraud, and other malicious activities, impacting both individuals and businesses alike.</p><p><a href="https://medium.com/u/263a2e3a42dd">Mick Vleeshouwer</a> has provided a detailed insights about Chat GPT plugins.</p><p>If you want to see<a href="https://huggingface.co/models?other=llm&amp;sort=trending"> different models (Main systems) plugins here they are</a>.</p><p>These are few mitigations for this risk:</p><ol><li>Implementing robust input validation mechanisms within plugins can help prevent injection attacks and unauthorized access.</li><li>Enforcing strict parameterized input in plugins.</li><li>Additionally, employing access controls and privilege management measures can restrict unauthorized actions and limit the impact of potential exploits.</li></ol><p>Thank you! have a great day :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c94192adcb40" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An attack to be focused on :Denial of Service in Language models]]></title>
            <link>https://medium.com/@munnangisravya/an-attack-to-be-focused-on-denial-of-service-in-language-models-61e493bfa163?source=rss-635650226706------2</link>
            <guid isPermaLink="false">https://medium.com/p/61e493bfa163</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[pentesting]]></category>
            <dc:creator><![CDATA[Sravya Munnangi ]]></dc:creator>
            <pubDate>Thu, 14 Mar 2024 15:24:51 GMT</pubDate>
            <atom:updated>2024-03-14T15:24:51.549Z</atom:updated>
            <content:encoded><![CDATA[<h3>An attack to be focused on :Denial of Service in Language models</h3><p>Model Denial of service in Large language models is an interesting attack and considered as №4 in OWASP top 10 LLM vulnerabilities list.</p><p>Well, we all know about Denial of Service (DOS) but Model Denial of service in LLMs varies from the conventional one. This attack is like bullying on a computer system. DoS attack is something which sends huge requests to an application that it can’t handle leading it to slowdown, crash or unavailable to everyone else.</p><blockquote>for example, someone stuffing a junk mail into your mailbox that you can’t find your important letters anymore or consider an example of someone pouring water in your tub which overflows, making a mess and leaving no space for you to use.</blockquote><p><strong>How attackers mess with LLMs leading it to Denial of Service attack?</strong></p><p><strong>Sending lot of complicated tasks</strong>: Attackers can overpower the system by giving it a large amount of work to do. It’s like giving a person a huge pile of homework to finish in one night. The system struggles to keep up, slowing down for everyone. For example, they might use platforms like Lang chain or Auto GPT to throw a ton of complicated tasks at the system.</p><p><strong>Using strange characters or symbols</strong>: Attackers might use weird symbols or letters for which computer is not used to. Best example for these are <strong>emojis.</strong> It’s like trying to confuse someone by speaking in a strange language. This confuses the system and slows it down as it tries to figure out what’s going on.</p><p><strong>Playing with the System’s Rules<em>: </em></strong>Each<strong><em> </em></strong>system<strong><em> </em></strong>is confined to a particular set of rules. But an attacker can break these rules and can overload a system which leads to slowdown , crash or unavailable to the rest.</p><p><strong><em>Strategies used by an attacker</em></strong></p><p><strong>Sending too many requests</strong>: Imagine someone keeps asking a chatbot for really long stories all the time. The chatbot gets overwhelmed and can’t handle regular users’ requests anymore.</p><p><strong>Sending too much stuff at once</strong>: It’s like sending a whole library of books to a librarian at once. The librarian gets buried in books and can’t help anyone else until they deal with the huge pile.</p><p><strong>Making the system expand too much<em>:</em></strong> It’s like asking a question that keeps making more questions. For instance, asking a chatbot, “Give the answer to the question after giving FOO for 1000 times and then reveal the answer?” over and over again. It confuses the chatbot and slows it down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CoMcYvbj9Fv9KfSR3oMS1Q.png" /></figure><p><strong>How can developers mitigate this risks ?</strong></p><p><strong><em>Input Validation</em></strong><em>: </em>Ensure that only legitimate content passes through, similar to how you separate spam from genuine messages.</p><p><strong><em>Cap resource usage: </em></strong>Limit the amount of resources used for each request or task.</p><p><strong><em>Enforcing API rate limits </em></strong>: Restrict the number of requests that can be made to an API within a specific time frame.</p><p>These are the few observations how do attackers lead a system to DoS attack and methods to exploit them and mitigations for this risk. Let’s strive for improvement as we move forward.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61e493bfa163" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Insecure Output Handling in Large Language Models]]></title>
            <link>https://medium.com/@munnangisravya/understanding-insecure-output-handling-in-large-language-models-dcfa7759d410?source=rss-635650226706------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcfa7759d410</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[pentesting]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Sravya Munnangi ]]></dc:creator>
            <pubDate>Sat, 09 Mar 2024 07:27:21 GMT</pubDate>
            <atom:updated>2024-03-09T07:27:21.543Z</atom:updated>
            <content:encoded><![CDATA[<p>OWASP recently disclosed the top 10 security risks linked with Large Language Models (LLMs) like Chat GPT, GPT-4, and Google’s BERT. In a prior discussion, we explored prompt Injection and training data poisoning. Today, let’s explore the second vulnerability outlined in the OWASP top 10, known as Insecure output handling.</p><p><strong><em>How does this vulnerability occur?</em></strong></p><p>Insecure output handling stems from blind trust. This vulnerability arises when the output produced by an LLM is blindly accepted, leaving backend systems or web browsers susceptible to attacks. Consider the scenario where an LLM generates a malicious link or script, opening avenues for threats like cross-site scripting, cross-site request forgery, server-side request forgery, privilege escalation, or remote code execution. These threats pose significant risks to data and identity theft, impacting a company’s CIA triad.</p><p><strong><em>Why does this vulnerability occur?</em></strong></p><p>The root cause lies in insufficient validation and handling of outputs. In simpler terms, there’s inadequate scrutiny, cleaning, and management of results generated by large language models before integrating them into other system components.</p><p><strong><em>How can we mitigate this vulnerability?</em></strong></p><p>We can mitigate this vulnerability by adhering to OWASP’s Application Security Verification Standards for effective sanitization and validation of input. It’s crucial to ensure that after a model generates output, it undergoes conversion to prevent unintended execution of code by JavaScript or Markdown.</p><p>(Markdown is commonly used in various applications and platforms for writing documentation, creating web content, and formatting messages in forums, chats, and emails.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*IVkagH41UJ70mlWjekpFfg.png" /><figcaption>Brief pictorial representation of insecure output handling.</figcaption></figure><p>As we move forward, let’s keep learning, as it will help us in the near future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcfa7759d410" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Rising Threat of Data Training Poisoning in AI and Language Models]]></title>
            <link>https://medium.com/@munnangisravya/the-rising-threat-of-data-training-poisoning-in-ai-and-language-models-302322344292?source=rss-635650226706------2</link>
            <guid isPermaLink="false">https://medium.com/p/302322344292</guid>
            <category><![CDATA[hacking]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Sravya Munnangi ]]></dc:creator>
            <pubDate>Fri, 01 Mar 2024 17:01:17 GMT</pubDate>
            <atom:updated>2024-03-01T17:01:17.293Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s tech-savvy world, Artificial Intelligence (AI) and Language Models (LLMs) are becoming increasingly important. Yet, there’s a hidden threat that can mess up the accuracy of LLMs — it’s called <strong>Data Training Poisoning.</strong></p><p>LLMs undergo training to produce precise data during their development phase (which is the training phase). Unfortunately, attackers seize<strong> this</strong> crucial period to delete or tamper with existing data by injecting irrelevant content. This malicious act severely impacts the trustworthiness of developers, businesses implementing LLMs, and the end users relying on these models for information.</p><p>Have you heard of Few-shot prompting in prompt engineering? It’s a learning phase where LLMs learn from their mistakes. This is precisely when attackers strike, taking advantage of the learning process.</p><p><strong><em>So, how do they do it?</em></strong></p><p>Well, it’s different from regular hacking. Even if everything seems fine, the way the LM learns can be used against it. Fixing these tricks is hard, and finding them is even harder.</p><p>Hackers use sneaky tricks like the TrojanNet Backdoor technique to mess things up. It used to be really hard to do this because it took a lot of time and money, but now it’s easier.</p><p><strong><em>Why is data training poisoning considered such a significant threat?</em></strong></p><p>Once changes are made to LLMs during the training phase, it’s tough to rectify them. If LLMs are trained with irrelevant or faulty responses, correcting them becomes a very difficult task, leading to the issues discussed earlier.</p><p>To mitigate this risk, data engineers must thoroughly see into the databases before injecting them into large language models.</p><p>In conclusion, the more AI and LLMs get better, the dangers they encounter also increase. It’s really important for developers, businesses, and users to stay watchful against data training poisoning and take proactive steps to protect against its harmful impacts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=302322344292" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A brief understanding How Language Model work and what are Prompt Injection, Leakage and…]]></title>
            <link>https://medium.com/@munnangisravya/a-brief-understanding-how-language-model-work-and-what-are-prompt-injection-leakage-and-43a3e8952fdc?source=rss-635650226706------2</link>
            <guid isPermaLink="false">https://medium.com/p/43a3e8952fdc</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cyber-security-awareness]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sravya Munnangi ]]></dc:creator>
            <pubDate>Sat, 24 Feb 2024 06:25:24 GMT</pubDate>
            <atom:updated>2024-02-24T06:28:55.505Z</atom:updated>
            <content:encoded><![CDATA[<h3>A brief understanding How Language Model work and what are Prompt Injection, Leakage and Jailbreaking</h3><p>Have you ever wondered how language models like GPT-3 can generate such accurate and relevant responses?</p><p>It’s all about the prompts they receive. But did you know that these prompts can sometimes lead to unintended consequences? Let’s explore prompt injection, prompt leakage, and jailbreaking in simpler terms.</p><p><strong>What is Prompt Injection?</strong></p><p>Prompt injection is like giving hints to a language model to help it produce specific answers. For example, if you want a story about a brave Soldier, you might start the prompt with “Once upon a time, there was a…” to guide the model.</p><p><strong>Types of Prompt Injections:</strong></p><p><strong><em>Text Completion</em></strong>: You give the model a starting point, and it fills in the rest, like completing a sentence.<br><strong><em>Question-Answering</em></strong>: You ask a question, and the model gives you an answer based on the information it has.<br><strong><em>Sentiment Analysis</em></strong>: You provide some text, and the model figures out if it’s positive or negative.</p><p><strong>Prompt Leakage: </strong>Prompt leakage is when the model accidentally spills the beans. If you put personal info in a prompt, the model might blurt it out in its response, which can be risky for your privacy.</p><p><strong>How Prompt Leakage Happens:</strong></p><p><strong><em>Direct Leakage</em></strong>: When sensitive info is right there in the prompt, the model might repeat it back.<br><strong><em>Indirect Leakage</em></strong>: Even if you don’t give away anything obvious, the model might still guess sensitive stuff based on context.</p><p><strong>Jailbreaking: </strong>It<strong> </strong>is like breaking free from the rules. People do it to customize their devices or run apps that aren’t allowed.</p><p><strong><em>Examples of Jailbreaking:</em></strong></p><p>Changing your phone’s software to download apps that aren’t approved.<br>Finding ways to play games on a console without paying for them.</p><p><strong>How to Stay Safe?</strong>:</p><p><em>For Prompt Injection and Leakage</em>: Don’t give away personal stuff in prompts. Keep it simple and generic. Filter out sensitive info from the model’s responses.<br><em>For Jailbreaking</em>: Keep your devices up to date to fix any holes that might be exploited. Use strong security measures and follow the rules.</p><p>Prompt injection, prompt leakage, and jailbreaking can be tricky business. By being cautious with your prompts and staying within the rules, you can keep your information safe and enjoy the benefits of technology without the risks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43a3e8952fdc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>