<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nisarg Negi on Medium]]></title>
        <description><![CDATA[Stories by Nisarg Negi on Medium]]></description>
        <link>https://medium.com/@nisargnegi?source=rss-5dbe41d3e692------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7ro_S9tt51jKDkveYmAoSA@2x.jpeg</url>
            <title>Stories by Nisarg Negi on Medium</title>
            <link>https://medium.com/@nisargnegi?source=rss-5dbe41d3e692------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:59:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nisargnegi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Getting Started with Python Programming: A Beginner’s Guide]]></title>
            <link>https://medium.com/@nisargnegi/getting-started-with-python-programming-a-beginners-guide-36fba75ce4ce?source=rss-5dbe41d3e692------2</link>
            <guid isPermaLink="false">https://medium.com/p/36fba75ce4ce</guid>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Nisarg Negi]]></dc:creator>
            <pubDate>Mon, 07 Oct 2024 07:08:46 GMT</pubDate>
            <atom:updated>2024-10-08T05:12:40.295Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xydnsVbow-_CPe9p" /></figure><h3>Chapter 1.1 Getting Started</h3><p><strong>Python</strong> is one of the most popular programming languages in the world today. Whether you’re a beginner or an experienced developer, Python’s versatility, ease of use, and vast community support make it an ideal language for various applications like web development, automation, artificial intelligence, and beyond.</p><p>In this introductory guide, we’ll cover the basics of Python and give you the foundation to understand how computers work, what a program is, how programming languages differ from human languages, and what makes Python stand out.</p><h3>1.1.1 What is a Computer?</h3><p>At its core, a modern computer can be described as a <strong>machine that stores and manipulates information under the control of a changeable program</strong>. In simpler terms, a computer processes data and instructions to produce meaningful results.</p><p>For instance:</p><ul><li>Find students with the highest and lowest GPA.</li><li>Sort the monthly sales of a bakery to identify the best-performing months.</li><li>Predict house prices based on past data.</li></ul><p>All these tasks involve taking raw data, processing it with a program, and transforming it into valuable insights.</p><h3>1.1.2 What is a Program?</h3><p>A <strong>computer program</strong> is essentially a detailed, step-by-step set of instructions that tells the computer exactly what to do. If we change the program, the computer performs different actions and produces different results.</p><p>For example, computers understand basic mathematical operations like addition or division. But you can write a program that uses these basic functions to calculate something more complex, like the average of several numbers. By combining simple operations into larger ones, you can create complex programs capable of handling any task.</p><p>The process of writing and combining these operations is what we call <strong>programming</strong>. It’s the heart of what makes a computer so powerful. Without software (programs), computers are just expensive paperweights!</p><h3>1.1.3 What is a Programming Language?</h3><p>Programming languages are the tools we use to communicate with computers. Think of them as the equivalent of human languages, but instead of speaking to each other, we speak to machines.</p><p>Just as human languages like English or Spanish can be ambiguous (e.g., “I saw a man on a hill with a telescope”), programming languages are designed to be <strong>precise and unambiguous</strong>. This clarity is what allows computers to follow instructions without confusion.</p><p>Python, for example, uses straightforward syntax. To add two numbers, you simply write 3 + 4. Compare that to another programming language, Scheme, where the same operation looks like this: (+ 3 4). While the syntax might differ, both languages achieve the same result.</p><p>In the background, computers don’t understand these high-level languages directly. They require <strong>machine language</strong>, which is composed of detailed instructions for the computer’s hardware. This is where <strong>compilers</strong> and <strong>interpreters</strong> come into play.</p><h3>1.1.4 Compilers vs. Interpreters</h3><ul><li>A <strong>compiler</strong> translates the entire program into machine language at once. This compiled program can be run repeatedly without needing the source code or the compiler.</li><li>An <strong>interpreter</strong>, on the other hand, translates the program line by line as it runs, which is more flexible but usually slower than compiled programs.</li></ul><p>Python uses a hybrid approach, where it first compiles code into an intermediate form known as bytecode before interpreting it. This allows for faster execution while retaining flexibility.</p><h3>1.1.5 Features of Python</h3><p>Why is Python so popular? Let’s look at some key features:</p><ol><li><strong>High-Level Language</strong>: Python abstracts away many of the complex details of the computer’s hardware, so you can focus on solving problems rather than worrying about how memory is allocated or how data is managed.</li><li><strong>Interpreted Language</strong>: You don’t have to compile your code to see the results. Python reads your code line-by-line and executes it, making debugging and testing much easier.</li><li><strong>Object-Oriented</strong>: Python allows you to bundle data and the functions that modify that data into objects, making your code reusable and easier to maintain.</li><li><strong>Easy to Learn</strong>: Python has a clean and straightforward syntax that’s easy for beginners to pick up, especially compared to other programming languages like C++ or Java.</li><li><strong>Versatile</strong>: Python can be used in a variety of applications — from web development and automation to machine learning and cloud administration.</li><li><strong>Popular and Well-Supported</strong>: Python boasts a massive community of developers and third-party libraries that extend its capabilities, particularly in web development, automation, and beyond.</li></ol><h3>1.1.6 How to Start Using Python</h3><p>There are several ways you can use Python, depending on your needs and environment. Here are the four most common methods:</p><ol><li><strong>Command Line</strong>: Run Python scripts directly in the terminal. This is the simplest method and great for quick tests.</li><li><strong>IDLE</strong>: Python’s integrated development environment, which comes with the Python installation. It’s a basic editor, suitable for small projects.</li><li><strong>Jupyter Notebook</strong>: A popular tool, especially in the data science community, that allows you to write and run Python code interactively in a web-based environment. You can mix code with visualizations and explanations, making it ideal for learning and prototyping.</li><li><strong>Microsoft Visual Studio Code</strong>: One of the most powerful code editors available, with extensive support for Python through plugins. It’s highly recommended for larger projects.</li></ol><h3>1.1.7 A Basic Python Program</h3><p>Let’s take a look at a simple Python program to get a feel for the language. If you haven’t installed Python yet, download the <a href="https://www.anaconda.com/products/distribution">Anaconda Python Distribution</a>, which comes with all the tools you need.</p><p>Here’s an example of a Python function that adds two numbers:</p><pre>def add(x: int, y: int) -&gt; int:<br>    return x + y</pre><pre>def main():<br>    x = 1<br>    y = 2<br>    result = add(x, y)<br>    output_string = f&#39;The output of add({x}, {y}) is {result}.&#39;<br>    print(output_string)</pre><pre>if __name__ == &#39;__main__&#39;:<br>    main()</pre><p>This program defines a function add() that adds two numbers and then uses it in the main() function to add 1 and 2, printing the result.</p><h3>1.1.8 Recommended Tools and Extensions</h3><p>If you’re using <strong>Microsoft Visual Studio Code</strong>, here are some recommended extensions to enhance your Python development experience:</p><ul><li><strong>File Utils</strong>: Helps manage files in your projects.</li><li><strong>Markdown All in One</strong>: Useful if you write documentation or comments in Markdown.</li><li><strong>SQLite Viewer</strong>: Great if you work with databases and need to view your data within the editor.</li></ul><h3>Chapter 1.2 Definitions</h3><h3>Key Programming Definitions: Building Blocks of Python</h3><p>In this section, we will break down some of the essential terms that are fundamental to understanding programming. Whether you’re new to coding or need a refresher, these definitions will help you build a solid foundation as you continue your Python journey.</p><h3>Objectives</h3><ul><li>To learn the key terms used to comprehend programming.</li></ul><h3>1.2.1. Recap of Previous Terms</h3><p>Before we dive into new terms, let’s quickly review some basic concepts we’ve already encountered:</p><ul><li><strong>Hardware</strong>: The physical components of a computer system, such as the CPU, memory, and hard drive.</li><li><strong>Computer</strong>: A machine that stores and manipulates information according to a set of instructions, known as a program.</li><li><strong>Program</strong>: A sequence of instructions that tell a computer how to perform a task.</li><li><strong>Programming</strong>: The process of writing a program to solve a specific problem or automate a task.</li><li><strong>Programming Language</strong>: A formal language, such as Python, Java, or C++, used to write programs.</li><li><strong>High-level Language</strong>: A programming language that is easier for humans to read and write, like Python.</li><li><strong>Low-level Language</strong>: A language that is closer to machine code and less abstract than high-level languages, often requiring the programmer to manage details like memory usage.</li><li><strong>Machine Language</strong>: The raw binary code that a CPU understands directly, consisting of specific instructions for performing tasks like moving data or performing calculations.</li></ul><h3>1.2.2. New Terms</h3><p>Now, let’s expand our vocabulary with some key programming concepts. These terms will help you better understand how Python (or any programming language) works under the hood.</p><ul><li><strong>Syntax</strong>: This refers to the set of rules that defines the structure of a valid program. Just as grammar guides sentence structure in a human language, syntax dictates how operators, operands, and other components can be combined in code. Refer Indentation, Variable, Comments, Data Types, Operators, Functions.</li><li><strong>Operator</strong>: Operators are symbols or keywords that perform operations on data. For example, + is an arithmetic operator used for addition, and == is a comparison operator that checks if two values are equal.</li><li><strong>Operand</strong>: The operands are the values or variables that the operator acts upon. In the expression 3 + 4, both 3 and 4 are operands.</li><li><strong>Expression</strong>: An expression is a combination of operators and operands that, when evaluated, produces a value. For example, 3 + 4 is an expression that evaluates to 7.</li></ul><h3>1.2.3. Example: Expressions and Operators in Action</h3><p>Let’s walk through a couple of examples to see how these terms come together in real Python code.</p><h4>Example 1: Basic Arithmetic Expression</h4><pre>3 + 4</pre><ul><li><strong>3</strong> is an operand.</li><li><strong>+</strong> is an operator.</li><li><strong>4</strong> is an operand.</li></ul><p>The whole line is an <strong>expression</strong> because it involves an operator (+) and two operands (3 and 4). When evaluated, this expression produces the value 7.</p><h4>Example 2: More Complex Expressions</h4><pre>2 - (3 / 5)</pre><ul><li><strong>2</strong> is an operand.</li><li><strong>-</strong> is an operator (subtraction).</li><li><strong>(3 / 5)</strong> is another expression, where:</li><li><strong>3</strong> and <strong>5</strong> are operands.</li><li><strong>/</strong> is an operator (division).</li></ul><p>In this case, the result of dividing 3 by 5 is calculated first, and then the subtraction is performed. The entire expression evaluates to 1.4.</p><h4>Example 3: An Expression That’s Just a Value</h4><p>Not every expression needs an operator. A value on its own can also be considered an expression.</p><pre>212</pre><p>In this case, the number 212 is both an <strong>expression</strong> and a <strong>value</strong>. When evaluated, the expression simply returns the value 212.</p><h3>Conclusion</h3><p>Understanding these core terms — syntax, operator, operand, expression, and statement — will make it easier to grasp the more advanced concepts as we move forward in our Python programming journey. Think of them as the building blocks of any program you write.</p><h3>Chapter 1.2 Arithmetic</h3><h3>Python Arithmetic Operators: A Comprehensive Guide</h3><p>In this section, we will dive into the arithmetic operators available in Python. These operators allow us to perform a variety of mathematical operations, from basic addition and subtraction to more complex tasks like integer division and exponentiation.</p><p>Understanding how these operators work is essential for solving problems and writing efficient code. Let’s explore each operator in detail, along with examples.</p><h3>1.3. Arithmetic Operators</h3><p>Table 1.1 provides a list of the most commonly used arithmetic operators in Python.</p><h3>Table 1.1: Arithmetic Operators</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*vhybpEREIN07HXDTNNpOwA.png" /></figure><ul><li>Operators that involve two operands (e.g., 3 + 4) are called <strong>binary operators</strong>.</li><li>The <strong>negation</strong> operator (-5) is a <strong>unary operator</strong> because it applies to only one operand.</li></ul><h3>1.3.1. Negation</h3><p>The <strong>negation</strong> operator negates a value. Here’s a simple example:</p><pre>-5<br># Output: -5</pre><p>You can negate a negative value, which results in a positive value:</p><pre>--5<br># Output: 5</pre><h3>1.3.2. Addition, Subtraction, Multiplication, and Division</h3><p>The <strong>addition</strong>, <strong>subtraction</strong>, <strong>multiplication</strong>, and <strong>division</strong> operators are straightforward and follow the same rules you’d expect from basic math.</p><h4>Examples:</h4><pre>11 + 3.1<br># Output: 14.1</pre><pre>5 - 19<br># Output: -14</pre><pre>8.5 * 4<br># Output: 34.0</pre><pre>11 / 2<br># Output: 5.5</pre><blockquote><strong><em>Note:</em></strong><em> When operands are of mixed data types (e.g., one is an integer and the other a float), Python automatically converts the result to a float.</em></blockquote><h3>1.3.3. Integer Division and Modulus</h3><p>The <strong>integer division</strong> operator (//) returns the quotient of a division, discarding the fractional part. To find the remainder of a division, use the <strong>modulo</strong> operator (%).</p><h4>Example: Integer Division</h4><p>To determine how many 24-hour days fit into 53 hours:</p><pre>53 // 24<br># Output: 2</pre><h4>Example: Modulo</h4><p>To find out how many hours are left over:</p><pre>53 % 24<br># Output: 5</pre><h3>1.3.3.1. Working with Negative Numbers</h3><p>When dealing with negative numbers in integer division and modulo, Python follows a couple of key rules:</p><ul><li><strong>Rule 1</strong>: Python’s integer division rounds towards negative infinity, meaning it “rounds down” regardless of whether the result is positive or negative.</li><li><strong>Rule 2</strong>: The equation n = q * base + r governs how modulo operations work, where:</li><li>n is the dividend.</li><li>q is the quotient from integer division.</li><li>base is the divisor.</li><li>r is the remainder, determined by the modulo operator.</li></ul><p>Let’s examine a few cases involving negative numbers.</p><h4>1.3.3.2. Case 1: Positive Dividend and Negative Divisor</h4><pre>17 // -10<br># Output: -2</pre><pre>17 % -10<br># Output: -3</pre><ul><li><strong>Explanation</strong>: Dividing 17 by -10 results in -1.7, which rounds down to -2. Using the equation 17 = -2 * -10 + r, we find that the remainder r = -3.</li></ul><h4>1.3.3.3. Case 2: Negative Dividend and Positive Divisor</h4><pre>-17 // 10<br># Output: -2</pre><pre>-17 % 10<br># Output: 3</pre><ul><li><strong>Explanation</strong>: Dividing -17 by 10 results in -1.7, which rounds down to -2. Solving -17 = -2 * 10 + r gives r = 3.</li></ul><h4>1.3.3.4. Case 3: Negative Dividend and Negative Divisor</h4><pre>-17 // -10<br># Output: 1</pre><pre>-17 % -10<br># Output: -7</pre><ul><li><strong>Explanation</strong>: Dividing -17 by -10 results in 1.7, which rounds down to 1. Solving -17 = 1 * -10 + r gives r = -7.</li></ul><h3>1.3.4. Exponentiation</h3><p>The <strong>exponentiation</strong> operator (**) raises a number to the power of another number.</p><h4>Example:</h4><pre>3 ** 6<br># Output: 729</pre><p>This expression calculates 3 raised to the power of 6, which equals 729.</p><h3>Conclusion</h3><p>Python provides a variety of arithmetic operators that allow you to perform a range of mathematical operations, from basic addition and subtraction to more complex tasks like exponentiation and modulus. Understanding how these operators work — especially when dealing with negative numbers — will help you write more accurate and efficient programs.</p><h3>1.4 Numeric Data Types</h3><p>In this chapter, we’ll explore Python’s <strong>numeric data types</strong>, particularly <strong>int</strong> and <strong>float</strong>. Understanding how these data types work is essential because every value in Python is tied to a specific type, which influences how operations on those values behave.</p><p>Python supports several numeric data types, but for now, we will focus on the most common ones: <strong>int</strong> and <strong>float</strong>. You can always check the data type of a value using Python’s type() function.</p><pre>type(2)<br># Output: &lt;class &#39;int&#39;&gt;</pre><pre>type(3.14)<br># Output: &lt;class &#39;float&#39;&gt;</pre><h3>1.4.1. Int and Float</h3><ul><li><strong>int</strong>: Represents whole numbers, both positive and negative, such as 4, -7, and 1000. There is no limit to the size of an integer in Python.</li><li><strong>float</strong>: Represents numbers with a decimal point, such as 2.5, -17.0, or 3.14. Floats have a limited precision due to the way they are stored in memory.</li></ul><h4>Examples:</h4><ul><li><strong>int</strong> values:</li></ul><pre>type(4)   # Output: &lt;class &#39;int&#39;&gt; <br>type(-7)  # Output: &lt;class &#39;int&#39;&gt;</pre><ul><li><strong>float</strong> values:</li></ul><pre>type(2.5)    # Output: &lt;class &#39;float&#39;&gt; <br>type(-17.0)  # Output: &lt;class &#39;float&#39;&gt;</pre><h4>Numeric Operations</h4><p>The result of an operation depends on the data types involved:</p><ul><li><strong>Two </strong><strong>int operands</strong> produce an int result:</li></ul><pre>1 + 2 <br># Output: 3 <br>type(1 + 2)   # Output: &lt;class &#39;int&#39;&gt;</pre><ul><li><strong>Two </strong><strong>float operands</strong> produce a float result:</li></ul><pre>1.0 + 2.0 # Output: 3.0 <br>type(1.0 + 2.0)   # Output: &lt;class &#39;float&#39;&gt;</pre><ul><li><strong>Mixed </strong><strong>int and </strong><strong>float operands</strong>: Python automatically converts the int to a float, and the result is a float:</li></ul><pre>17.0 - 10 <br># Output: 7.0 <br>type(17.0 - 10) # Output: &lt;class &#39;float&#39;&gt; <br><br> 17 - 10.0 <br># Output: 7.0 <br>type(17 - 10.0)   # Output: &lt;class &#39;float&#39;&gt;</pre><p>While Python allows you to write numbers like 10., it’s generally considered poor style to omit the zero after the decimal point. It’s clearer to write 10.0 for readability and consistency.</p><pre>10.   # Poor style<br>10.0  # Preferred style</pre><h3>1.4.1.1. Differences Between int and float</h3><p>Here are some important distinctions between the two data types:</p><ul><li><strong>Space in Memory</strong>:</li></ul><ol><li><strong>float</strong> values occupy a fixed amount of memory space.</li><li><strong>int</strong> values use variable space, depending on the size of the number. Python’s int is implemented as a <strong>bignum</strong>, meaning it can handle arbitrarily large integers.</li></ol><h4>Example of Memory Usage:</h4><p>You can check how much memory an object occupies using the sys.getsizeof() function:</p><pre>import sys<br>sys.getsizeof(2.0)<br># Output: 24<br><br>sys.getsizeof(2**30)<br># Output: 32<br><br>sys.getsizeof(2**130)<br># Output: 44</pre><p>As seen in the examples, the memory usage of an int increases as the number grows, while a float uses a fixed 24 bytes of memory.</p><ul><li><strong>Precision</strong>:</li><li><strong>int</strong> values are precise because they represent exact whole numbers.</li><li><strong>float</strong> values, however, can only approximate real numbers due to the limited memory available to store them. This can lead to rounding errors in calculations.</li></ul><h4>Example of Precision with float:</h4><pre>2 / 3<br># Output: 0.6666666666666666</pre><pre>5 / 3<br># Output: 1.6666666666666667</pre><p>Because float values are represented with a limited precision, you may notice small inaccuracies in the result.</p><h3>Key Rule: Use int When You Don’t Need Fractions</h3><p>If you’re working with whole numbers and don’t need to represent fractional values, it’s better to use the int type. Not only does this avoid potential rounding errors, but it can also save memory when working with large datasets.</p><h3>Conclusion</h3><p>Understanding Python’s numeric data types is essential for performing mathematical operations effectively. The int type handles whole numbers with unlimited range, while the float type is used for decimal values with limited precision. Knowing when and how to use these types will help you avoid common pitfalls in your code.</p><h3>1.5 Adding a Large Number and a Very Small Number</h3><p>In Python, handling very large and very small numbers together can sometimes result in unexpected outcomes due to the limitations of floating-point arithmetic. Let’s dive into this issue and explore how Python deals with large numbers and precision, and how you can mitigate such problems.</p><h3>Example: Adding a Large Number and a Very Small Number</h3><p>Consider the following example:</p><pre>10000000000 + 0.00000000001</pre><p>What’s the expected result?</p><p>You might think the result should be:</p><pre>10000000000.00000000001</pre><p>However, the actual output is:</p><pre>10000000000.0</pre><h3>Why Does This Happen?</h3><p>The result should have included <strong>twenty zeros</strong> between the first and last significant digit. However, the computer cannot accurately represent such a large number combined with such a small one due to the limitations of how floating-point numbers are stored. The small number (0.00000000001) gets lost in the magnitude of the large number (10000000000), and as a result, the sum remains essentially unchanged.</p><p>This issue is known as <strong>floating-point precision error</strong>, and it’s particularly problematic when working with very large and very small numbers together.</p><p>For example, imagine a bank adding very small interest values to large account balances. A floating-point error could lead to significant discrepancies over time if the small values are “lost” in the computation.</p><p>When dealing with the addition of floating-point numbers, it’s advisable to add them in ascending order of magnitude to minimize the error.</p><p>Alternatively, utilizing a specialized library is a preferable option:</p><ul><li><a href="https://mpmath.org/">mpmath: a Python library for arbitrary-precision floating-point arithmetic</a></li></ul><h3>1.6. Operator Precedence</h3><h3>What is 8/2(2+2)?</h3><p>Using the PEMDAS rule, we can break down the expression 8/2(2+2).</p><h3>PEMDAS: “Please Excuse My Dear Aunt Sally”</h3><ul><li><strong>P</strong> — Parentheses</li><li><strong>E</strong> — Exponents</li><li><strong>MD</strong> — Multiplication and Division (evaluated left to right)</li><li><strong>AS</strong> — Addition and Subtraction (evaluated left to right)</li></ul><h3>Order of Precedence:</h3><ol><li><strong>Parentheses</strong>: Perform operations inside parentheses first.</li><li><strong>Exponents</strong>: Apply exponents next.</li><li><strong>Multiplication and Division</strong>: Perform multiplication and division from left to right.</li><li><strong>Addition and Subtraction</strong>: Perform addition and subtraction from left to right.</li></ol><h3>Precedence in Python:</h3><ol><li><strong>Parentheses</strong></li><li><strong>Exponents</strong></li><li><strong>Unary plus, Unary minus</strong></li><li><strong>Multiplication, Division, Integer division, Modulus (Remainder)</strong></li><li><strong>Addition, Subtraction</strong></li></ol><p>Now, let’s evaluate 8/2(2+2):</p><ol><li><strong>Parentheses</strong>: Calculate the value inside the parentheses first, 2 + 2 = 4.</li><li><strong>Multiplication and Division</strong> (left to right): Now the expression becomes 8 / 2 * 4.</li></ol><ul><li>First, divide: 8 / 2 = 4.</li><li>Then multiply: 4 * 4 = 16.</li></ul><p>Thus, the result of 8/2(2+2) is 16.</p><h3>1.7. Variables in Python</h3><h3>Variables</h3><p>Variables in Python act as labels for values (such as numbers, strings, or None) and provide a way to reference those values later in your code. They are identifiers that can store and retrieve data, and you can modify their values by assigning new data to them.</p><p>Before you use a variable in an expression, it must first be assigned a value. In Python, variables do not require an explicit declaration of the data type; the type is automatically inferred from the value assigned. This characteristic makes Python a <strong>dynamically typed language</strong>. In contrast, <strong>statically typed languages</strong> require that the data type of a variable be declared at the time of assignment.</p><p>In Python, you can even reassign a variable to a value of a different data type:</p><pre>x = 3  # data type is int<br>x = 3.14  # data type is now float; this is allowed!</pre><p>Variable names are <strong>case-sensitive</strong>, so ph and pH are treated as two distinct variables.</p><h3>1.7.1. Restrictions for Variable Names</h3><p>There are specific rules you must follow when naming variables:</p><ul><li><strong>Must start</strong> with a letter (a-z, A-Z) or an underscore (_).</li><li>Can consist of <strong>letters</strong>, <strong>numbers</strong>, and <strong>underscores</strong>, but <strong>no spaces</strong>.</li><li><strong>Symbols</strong> like (), @, +, -, *, / cannot be used in variable names.</li><li><strong>Cannot use Python keywords</strong> as variable names.</li></ul><h3>1.7.2. Python Keywords</h3><p>Keywords in Python are <strong>reserved</strong> and cannot be used as variable names. These words have special meaning and are part of Python’s syntax.</p><p>Here are some examples of Python keywords:</p><pre>False      await      else       import     pass<br>None       break      except     in         raise<br>True       class      finally    is         return<br>and        continue   for        lambda     try<br>as         def        from       nonlocal   while<br>assert     del        global     not        with<br>async      elif       if         or         yield</pre><h3>1.7.3. Conventions for Variable Names</h3><p>While Python allows flexibility in naming variables, there are <strong>best practices</strong> you should follow to keep your code readable and consistent:</p><ul><li>Use <strong>snake_case</strong> for variable and function names, not camelCase.</li><li>Variable names should be <strong>lowercase</strong>.</li><li><strong>Uppercase</strong> is reserved for constants, like PI = 3.14.</li><li><strong>UpperCamelCase</strong> is used for class names.</li><li>Use __private__ (double underscore) to signify a private variable, meaning it shouldn’t be accessed directly. This is similar to private variables in other languages.</li><li>Avoid using Python built-in functions or keywords as variable names (e.g., int, type, float).</li></ul><p>If you accidentally use a Python keyword as a variable, you can restore the keyword using del:</p><pre>int = 3  # this overwrites the built-in int<br>del int  # restores the built-in int</pre><pre>print = &#39;Hello&#39;  # overwrites the built-in print function<br>del print  # restores the print function</pre><h3>1.7.4. Variable Assignment</h3><p>Variables are created by executing <strong>assignment statements</strong>, where a value is assigned to a variable using the equals sign (=).</p><p>For example:</p><pre>degrees_celsius = 26.0</pre><p>In this case, the value 26.0 is assigned to the variable degrees_celsius. From that point on, whenever Python encounters degrees_celsius in an expression, it substitutes the value it holds.</p><h4>Example of Using a Variable in an Expression</h4><pre>degrees_celsius = 26.0<br>result = 9 / 5 * degrees_celsius + 32<br>print(result)<br># Output: 78.80000000000001</pre><p>Here, degrees_celsius is used to compute the Fahrenheit equivalent, and Python substitutes the value 26.0 into the calculation.</p><p>Variables can be reassigned with new values as the program executes, which is why they are called <strong>variables</strong>. Their value can “vary” over time.</p><h4>Example of Reassigning a Variable</h4><pre>degrees_celsius = 15.5<br>result = 9 / 5 * degrees_celsius + 32<br>print(result)<br># Output: 59.900000000000006</pre><p>In this case, degrees_celsius is reassigned to 15.5, and the new value is used in the calculation.</p><p>Understanding how to create, name, and assign values to variables is fundamental in Python programming. Follow the best practices and conventions to write clear, maintainable code!</p><h3>1.8. Memory Management in Python</h3><p>The way Python handles memory management is quite different from other programming languages. In many languages, variables are often thought of as <strong>storage locations</strong> in memory, or “boxes” where values are stored. When a value in a variable is changed, the old value is simply replaced with the new one.</p><h3>Python’s Sticky-Note Model</h3><p>In Python, the memory model is more flexible. Instead of thinking of variables as boxes where values are stored, think of them as <strong>sticky notes</strong> placed on values in memory. These sticky notes (variables) point to values, which can be stored anywhere in memory. This is why Python variables are better described as <strong>references</strong> or <strong>pointers</strong> to values, which are technically <strong>objects</strong>.</p><p>When you assign a value to a variable in Python, you are essentially placing a sticky note (the variable name) on an object (the value). This model is often referred to as the <strong>sticky-note model</strong>.</p><h3>Example Memory Model</h3><p>Consider the following code:</p><pre>difference = 20<br>double = 2 * difference<br>print(double)  # Output: 40<br>difference = 5﻿<br>print(double)  # Output: 40</pre><p>Here’s how Python handles memory management step by step:</p><ol><li><strong>difference = 20</strong>: Python evaluates 20, creates an object for this value in memory, and then sticks the difference label onto this object.</li><li><strong>double = 2 * difference</strong>: Python calculates 2 * difference (which is 2 * 20), creates an object for the result 40, and sticks the double label onto it.</li><li><strong>difference = 5</strong>: Python creates a new object for 5 and moves the difference label to this new object. However, the double variable still references the original value 40.</li><li><strong>Output</strong>: When print(double) is called, it prints 40, as double still points to the object holding the value 40.</li></ol><h3>Memory Management Steps in Python</h3><p>When executing a statement like this:</p><pre>variable = expression</pre><p>Python performs the following steps:</p><ol><li><strong>Evaluate the expression</strong> on the right-hand side of the = sign to produce a value. This value is stored in memory at a particular address. This value becomes an <strong>object</strong> (which has a type and a memory address).</li><li><strong>Store the memory address</strong> of this value in the variable on the left-hand side of the = sign. If the variable doesn’t already exist, Python creates it. If it does exist, Python updates the variable to reference the new memory address.</li></ol><h3>Example in Action</h3><pre>number = 3<br>print(number)  # Output: 3</pre><pre>number = 2 * number<br>print(number)  # Output: 6</pre><pre>number = number * number<br>print(number)  # Output: 36</pre><h4>Step-by-Step Breakdown:</h4><ol><li><strong>number = 3</strong>: Python creates an object for the value 3 and places the number label on it.</li><li><strong>number = 2 * number</strong>: Python evaluates 2 * number (which is 2 * 3), creates a new object for the value 6, and moves the number label to this new object.</li><li><strong>number = number * number</strong>: Python evaluates number * number (which is 6 * 6), creates an object for the value 36, and moves the number label to this object.</li></ol><h3>Conclusion</h3><p>In Python, variables act as <strong>labels</strong> that point to values (or objects) stored in memory. This is different from languages where variables are considered fixed memory storage locations. Python’s flexible memory management approach allows for easy reassignment of variables, enabling powerful dynamic programming.</p><h3>1.9. Augmented Assignments</h3><p>An <strong>augmented assignment</strong> combines an assignment with an operator, providing a more concise way to write expressions. Instead of writing a separate expression and assignment, an augmented assignment performs both in one step.</p><h3>How Augmented Assignment Works</h3><p>When an augmented assignment is executed, Python follows these steps:</p><ol><li><strong>Evaluate the expression</strong> on the right-hand side of the = to produce a value.</li><li><strong>Apply the operator</strong> to the variable on the left-hand side and the evaluated value on the right-hand side. This produces a new value.</li><li><strong>Store the new value</strong> in the variable on the left-hand side.</li></ol><h3>Example:</h3><p>Consider the augmented assignment x += 2. Here’s how it works:</p><ul><li>Python evaluates 2 and adds it to the current value of x.</li><li>The result is then assigned back to x.</li></ul><h3>Table 1.2: Augmented Assignments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*XdfkcH0_J9eqBuWemmMDdw.png" /></figure><h3>1.10. Errors in Python</h3><p>Errors in Python occur when the interpreter encounters something it cannot process. Python provides clear error messages that help identify the problem and offer clues on how to fix it. This section introduces three common types of errors that you are likely to encounter at the early stages of programming in Python.</p><h3>1.10.1. NameError</h3><p>A <strong>NameError</strong> occurs when you reference a variable or name that has not been declared or assigned a value.</p><p>Example:</p><pre>3 + moogah</pre><p>Output:</p><pre>---------------------------------------------------------------------------<br>NameError                                 Traceback (most recent call last)<br>Cell In[1], line 1<br>----&gt; 1 3 + moogah</pre><pre>NameError: name &#39;moogah&#39; is not defined</pre><p>In this case, the variable moogah has not been defined, leading to a <strong>NameError</strong>.</p><h3>1.10.2. SyntaxError</h3><p>A <strong>SyntaxError</strong> occurs when you write code that does not conform to Python’s grammar rules. These errors typically occur when operators, parentheses, or other structural elements are arranged incorrectly.</p><p>Example 1:</p><pre>2 +</pre><p>Output:</p><pre>Cell In[2], line 1<br>    2 +<br>        ^<br>SyntaxError: invalid syntax</pre><p>This error occurs because the plus operator (+) is incomplete and does not have an operand on the right side.</p><p>Example 2:</p><pre>12 = x</pre><p>Output:</p><pre>Cell In[3], line 1<br>    12 = x<br>    ^<br>SyntaxError: cannot assign to literal here. Maybe you meant &#39;==&#39; instead of &#39;=&#39;?</pre><p>In this case, the error occurs because Python does not allow assignment to a literal (12). The error message also suggests that you might have intended to use == instead of =.</p><h3>1.10.3. ZeroDivisionError</h3><p>A <strong>ZeroDivisionError</strong> occurs when you attempt to divide a number by zero. Division by zero is undefined in mathematics, and Python raises an error to alert you to this issue.</p><p>Example:</p><pre>3 / 0</pre><p>Output:</p><pre>---------------------------------------------------------------------------<br>ZeroDivisionError                         Traceback (most recent call last)<br>Cell In[4], line 1<br>----&gt; 1 3 / 0</pre><pre>ZeroDivisionError: division by zero</pre><p>In this case, dividing 3 by 0 is not allowed, so Python raises a <strong>ZeroDivisionError</strong>.</p><h3>Conclusion</h3><p>Errors are a natural part of programming, and Python helps identify and explain them through detailed messages. As you write more code, you will likely encounter these errors. Understanding how to read and interpret error messages is key to debugging your code and becoming a more effective programmer.</p><h3>1.11. Formatting Code in Python</h3><p>In Python, writing clean and readable code is important, especially when working with long statements. If a single statement becomes too long, Python provides ways to format it over multiple lines for better readability. You can either use <strong>parentheses</strong> or the <strong>line-continuation character (</strong><strong>\)</strong> to achieve this.</p><h3>1.11.1. Using Parentheses</h3><p>You can enclose a long statement within <strong>parentheses</strong> to split it across multiple lines. This makes the code more readable and is commonly used for complex expressions.</p><p>Example:</p><pre>(2 +<br> 3)</pre><p>Output:</p><pre>5</pre><h3>1.11.2. Using the Line-Continuation Character (\)</h3><p>Another way to split a long statement across multiple lines is by using the <strong>line-continuation character</strong>, which is a backslash (\). This tells Python that the statement continues onto the next line.</p><p>Example:</p><pre>2 + \<br>3</pre><p>Output:</p><pre>5</pre><p>Both methods provide flexibility when dealing with long statements, helping you maintain clean and readable code.</p><h3>Conclusion</h3><p>Whether you use parentheses or the line-continuation character (\), Python offers convenient ways to format long statements over multiple lines. This not only helps with readability but also makes your code easier to maintain and understand.</p><h3>1.12. Describing Code in Python</h3><p>Including <strong>comments</strong> in your code is a great way to describe what the code is doing, making it easier to understand for yourself and others who may read it. Comments do not affect how the code runs; they are purely for the benefit of the programmer.</p><h3>1.12.1. Single-Line Comments</h3><p>In Python, you can write single-line comments by using the # character. Everything after the # on that line is ignored by Python.</p><p>Example:</p><pre>x * (x + 1)  # secret formula</pre><p>Here, # secret formula is a comment that explains the purpose of the expression, but it does not affect the execution of the code.</p><h3>1.12.2. Multiline Comments</h3><p>If you need to write longer comments that span multiple lines, you can use triple quotes — either &#39;&#39;&#39; (single quotes) or &quot;&quot;&quot; (double quotes). While these are technically string literals, they can be used for commenting as Python ignores them if they are not assigned to a variable.</p><p>Example with <strong>double quotes</strong>:</p><pre>&quot;&quot;&quot;<br>This is a multiline comment.<br>Test 1234<br>&quot;&quot;&quot;<br>x * (x + 1)</pre><p>Example with <strong>single quotes</strong>:</p><pre>&#39;&#39;&#39;<br>This is a multiline comment.<br>Test 1234<br>&#39;&#39;&#39;<br>x * (x + 1)</pre><p>Both methods can be used for multiline comments, and Python will ignore the text within the triple quotes if it’s not assigned to a variable.</p><h3>Conclusion</h3><p>Using comments in your Python code is a valuable practice for documenting your code and improving its readability. Whether using single-line comments (#) or multiline comments with triple quotes (&#39;&#39;&#39; or &quot;&quot;&quot;), they help explain the logic behind your code and make it easier to maintain in the long run.</p><h3>1.13. Simultaneous Assignment in Python</h3><p><strong>Simultaneous assignment</strong> is a powerful feature in Python that allows you to assign multiple variables at the same time. This is particularly useful when you need to <strong>swap values</strong> between variables. Instead of using a temporary variable to swap values, Python provides a cleaner, more concise approach.</p><h3>Traditional Way of Swapping Values</h3><p>In many programming languages, swapping values typically involves using a temporary variable:</p><pre>temp = x<br>x = y<br>y = temp</pre><p>Here, temp is used to hold the value of x temporarily while swapping the values of x and y.</p><h3>Simultaneous Assignment in Python</h3><p>In Python, you can achieve the same swap in a single line without needing a temporary variable:</p><pre>x, y = y, x</pre><p>This method makes the code cleaner and easier to read.</p><h3>Example</h3><p>Before swapping:</p><pre>x = 5<br>y = 10</pre><p>Swapping using simultaneous assignment:</p><pre>x, y = y, x</pre><p>After swapping, the values of x and y are:</p><pre>x = 10<br>y = 5</pre><h3>Conclusion</h3><p>Simultaneous assignment is a convenient and concise way to assign multiple variables or swap values in Python. It helps simplify code and eliminates the need for temporary variables when performing tasks like swapping values.</p><h3>1.14. Exercises</h3><p>For each of the following expressions, what value will the expression give? Verify your answers by typing the expressions into Python.</p><ol><li><strong>9 - 3</strong></li><li><strong>8 * 2.5</strong></li><li><strong>9 / 2</strong></li><li><strong>9 / -2</strong></li><li><strong>9 // -2</strong></li><li><strong>9 % 2</strong></li><li><strong>9.0 % 2</strong></li><li><strong>9 % 2.0</strong></li><li><strong>9 % -2</strong></li><li><strong>-9 % 2</strong></li><li><strong>9 / -2.0</strong></li><li><strong>4 + 3 * 5</strong></li><li><strong>(4 + 3) * 5</strong></li></ol><h3>Write Two Assignment Statements</h3><ol><li>Create a new variable, temp, and assign it the value 24.</li><li>Convert the value in temp from Celsius to Fahrenheit by multiplying by 1.8 and adding 32; make temp refer to the resulting value. What is temp&#39;s new value?</li></ol><h3>Subexpression Evaluation</h3><p>For each of the following expressions, in which order are the subexpressions evaluated?</p><ol><li><strong>6 3 + 7 4</strong></li><li><strong>5 + 3 / 4</strong></li><li><strong>5 - 2 * 3 ** 4</strong></li></ol><h3>Exercise: Assignments and Operations</h3><ol><li>Create a new variable x, and assign it the value 10.5.</li><li>Create a new variable y, and assign it the value 4.</li><li>Sum x and y, and make x refer to the resulting value. After this statement has been executed, what are the values of x and y?</li></ol><h3>SyntaxErrors</h3><p>Which of the following expressions result in <strong>SyntaxErrors</strong>?</p><ol><li><strong>6 * ———--8</strong></li><li><strong>8 = people</strong></li><li><strong>((((4 ** 3))))</strong></li><li><strong>(-(-(-(-5))))</strong></li><li><strong>4 += 7 / 2</strong></li></ol><p>Use these exercises to practice and understand how Python evaluates expressions, manages variables, and handles errors.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36fba75ce4ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sherlocked: A RAG-Powered Guide to the World of Sherlock Holmes using Llama3 70B & Llama Parser]]></title>
            <link>https://medium.com/@nisargnegi/sherlocked-a-rag-powered-guide-to-the-world-of-sherlock-holmes-using-llama3-70b-llama-parser-67cbdde6ff37?source=rss-5dbe41d3e692------2</link>
            <guid isPermaLink="false">https://medium.com/p/67cbdde6ff37</guid>
            <category><![CDATA[llama-3]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llamaparse]]></category>
            <category><![CDATA[sherlock-holmes]]></category>
            <dc:creator><![CDATA[Nisarg Negi]]></dc:creator>
            <pubDate>Mon, 20 May 2024 08:13:46 GMT</pubDate>
            <atom:updated>2024-05-20T08:13:46.585Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K8JYte7zFzTwURR2" /><figcaption>Ask Sherlock</figcaption></figure><p>Have you ever wondered how Sherlock Holmes would solve a modern-day mystery? With “Sherlocked,” you can delve into the world of the famous detective and explore his adventures using cutting-edge technologies. In this tutorial, we’ll walk you through the steps of setting up Sherlocked, from scraping the entire Sherlock Holmes collection to querying the knowledge base for insightful answers. Get ready to harness the power of advanced parsing, efficient processing, and natural language understanding!</p><h3>The Tools of the Trade</h3><p>Before we dive into the code, let’s familiarize ourselves with the essential tools and libraries we’ll be using:</p><ul><li>LangChain: For seamless interaction in plain English.</li><li>LLaMA Parse: For powerful and precise text processing.</li><li>GROQ: For throughput Llama3 70B model access.</li><li>Flashrank Reranker: To prioritize the most pertinent information.</li><li>Qdrant: For vector storage and similarity search.</li></ul><h3>Setting Up the Environment</h3><p>First, let’s install the necessary dependencies:</p><pre>pip -qqq install pip --progress-bar off<br>pip -qqq install langchain-groq==0.1.3 --progress-bar off<br>pip -qqq install langchain==0.1.17 --progress-bar off<br>pip -qqq install llama-parse==0.1.3 --progress-bar off<br>pip -qqq install qdrant-client==1.9.1 --progress-bar off<br>pip -qqq install &quot;unstructured[md]&quot;==0.13.6 --progress-bar off<br>pip -qqq install fastembed==0.2.7 --progress-bar off<br>pip -qqq install flashrank==0.2.4 --progress-bar off</pre><h3>The Case of the API Keys</h3><p>To use LLaMA Parse and GROQ, you’ll need to create API keys.</p><ol><li>LLaMA Parse:</li></ol><ul><li>Sign up at <a href="https://cloud.llamaindex.ai/">LLaMA Cloud</a>.</li><li>Create an API key and store it securely.</li></ul><p>2. GROQ:</p><ul><li>Sign up at <a href="https://console.groq.com/playground">GROQ Cloud</a>.</li><li>Create an API key and store it securely.</li></ul><p>Ensure you have your API keys ready before proceeding.</p><h3>The Case of the Parsed Document</h3><p>Next, we’ll download and parse the Sherlock Holmes books. We’ll download PDFs from a dedicated website and merge those under 500KB into a single document.</p><pre>import requests<br>from bs4 import BeautifulSoup<br>import os<br>from urllib.parse import urljoin<br>from PyPDF2 import PdfMerger</pre><pre>url = &#39;https://sherlock-holm.es/pdf/a4/1-sided/&#39;</pre><pre>os.makedirs(&#39;sherlock_pdfs&#39;, exist_ok=True)<br>os.makedirs(&#39;data&#39;, exist_ok=True)</pre><pre>response = requests.get(url)<br>response.raise_for_status()</pre><pre>soup = BeautifulSoup(response.content, &#39;html.parser&#39;)<br>pdf_links = soup.find_all(&#39;a&#39;, href=True)<br>pdf_urls = [link[&#39;href&#39;] for link in pdf_links if link[&#39;href&#39;].endswith(&#39;.pdf&#39;)]</pre><pre>merger = PdfMerger()<br>size_limit_kb = 500<br>size_limit_bytes = size_limit_kb * 1024</pre><pre>for pdf_url in pdf_urls:<br>    full_pdf_url = urljoin(url, pdf_url)<br>    filename = pdf_url.split(&#39;/&#39;)[-1]<br>    print(f&#39;Downloading {filename}...&#39;)<br>    pdf_response = requests.get(full_pdf_url)<br>    pdf_response.raise_for_status()<br>    <br>    pdf_path = os.path.join(&#39;sherlock_pdfs&#39;, filename)<br>    with open(pdf_path, &#39;wb&#39;) as f:<br>        f.write(pdf_response.content)<br>    <br>    file_size = os.path.getsize(pdf_path)<br>    if file_size &gt; size_limit_bytes:<br>        print(f&#39;{filename} is larger than 500KB and will be skipped.&#39;)<br>        os.remove(pdf_path)<br>    else:<br>        print(f&#39;{filename} downloaded and added to the merger.&#39;)<br>        merger.append(pdf_path)</pre><pre>output_path = os.path.join(&#39;data&#39;, &#39;merged_sherlock.pdf&#39;)<br>merger.write(output_path)<br>merger.close()</pre><pre>print(f&#39;All PDFs under 500KB have been merged into {output_path}.&#39;)</pre><h3>Document Parsing</h3><p>We use LLaMA Parse to process the merged PDF and extract the text content.</p><pre>instruction = &quot;&quot;&quot;The provided document is a Sherlock Holmes book.<br>This text contains details about the characters, plot, and setting of a Sherlock Holmes story.<br>It may include descriptions of crimes, investigations, and deductions.<br>Try to focus on factual information and avoid subjective analysis.&quot;&quot;&quot;</pre><pre>parser = LlamaParse(<br>    api_key=os.environ.get(&quot;LLAMA_PARSE&quot;),<br>    result_type=&quot;markdown&quot;,<br>    parsing_instruction=instruction,<br>    max_timeout=5000,<br>)</pre><pre>llama_parse_documents = await parser.aload_data(&quot;./data/merged_sherlock.pdf&quot;,)</pre><h3>Creating the Memory Palace</h3><p>We need to split the document into smaller chunks and embed them into a vector store using Qdrant.</p><pre>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain_community.document_loaders import UnstructuredMarkdownLoader</pre><pre>document_path = Path(&quot;data/parsed_document.md&quot;)<br>with document_path.open(&quot;a&quot;) as f:<br>    f.write(parsed_doc.text)</pre><pre>loader = UnstructuredMarkdownLoader(document_path)<br>loaded_documents = loader.load()</pre><pre>text_splitter = RecursiveCharacterTextSplitter(chunk_size=2048, chunk_overlap=128)<br>docs = text_splitter.split_documents(loaded_documents)</pre><pre>from langchain_community.embeddings.fastembed import FastEmbedEmbeddings</pre><pre>embeddings = FastEmbedEmbeddinregs(model_name=&quot;BAAI/bge-base-en-v1.5&quot;)</pre><pre>qdrant = Qdrant.from_documents(<br>    docs,<br>    embeddings,<br>    path=&quot;./db&quot;,<br>    collection_name=&quot;document_embeddings&quot;,<br>)</pre><h3>Querying the Knowledge Base</h3><p>Now, we can query the knowledge base to find relevant information about Sherlock Holmes cases.</p><pre>query = &quot;What is the most difficult case solved by Sherlock?&quot;<br>similar_docs = qdrant.similarity_search_with_score(query)</pre><pre>for doc, score in similar_docs:<br>    print(f&quot;text: {doc.page_content[:256]}\n&quot;)<br>    print(f&quot;score: {score}&quot;)<br>    print(&quot;-&quot; * 80)<br>    print()</pre><p>Finally, we use a retriever to refine the search results:</p><pre>retriever = qdrant.as_retriever(search_kwargs={&quot;k&quot;: 5})<br>retrieved_docs = retriever.invoke(query)<br><br>for doc in retrieved_docs:<br>    print(f&quot;id: {doc.metadata[&#39;_id&#39;]}\n&quot;)<br>    print(f&quot;text: {doc.page_content[:256]}\n&quot;)<br>    print(&quot;-&quot; * 80)<br>    print()</pre><h3>The Power of Flashrank Reranker</h3><p>To ensure we provide the most relevant information, Sherlocked employs the Flashrank Reranker. This tool helps prioritize the most pertinent details from the retrieved documents, improving the overall quality of responses.</p><p>The Flashrank Reranker works by reordering the retrieved documents based on their relevance to the query. Here’s how to integrate the Reranker into Sherlocked:</p><pre>from langchain.retrievers import ContextualCompressionRetriever<br>from langchain.retrievers.document_compressors import FlashrankRerank<br><br># Initialize the Flashrank Reranker<br>reranker = FlashrankRerank()<br><br># Create a ContextualCompressionRetriever using the Reranker<br>contextual_retriever = ContextualCompressionRetriever(<br>    base_retriever=retriever,<br>    document_compressor=reranker<br>)<br><br># Perform the query with the reranker to get the most relevant documents<br>reranked_docs = contextual_retriever.invoke(query)<br><br>for doc in reranked_docs:<br>    print(f&quot;text: {doc.page_content[:256]}\n&quot;)<br>    print(&quot;-&quot; * 80)</pre><h3>The Final Touch: Integrating the LLM</h3><p>After retrieving and reranking the most relevant documents, the final step is to generate a coherent and contextually accurate response using a language model (LLM). In Sherlocked, we use a LLaMA model to achieve this.</p><p>Here’s how to integrate the LLaMA model into Sherlocked:</p><pre>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_groq import ChatGroq<br>from langchain.chains import RetrievalQA</pre><pre>llm = ChatGroq(temperature=0, model_name=&quot;llama3-70b-8192&quot;)<br><br>prompt_template = &quot;&quot;&quot;<br>Use the following pieces of information to answer the user&#39;s question.<br>If you don&#39;t know the answer, just say that you don&#39;t know, don&#39;t try to make up an answer.<br><br>Context: {context}<br>Question: {question}<br><br>Answer the question and provide additional helpful information,<br>based on the pieces of information, if applicable. Be succinct.<br><br>Responses should be properly formatted to be easily read.<br>&quot;&quot;&quot;<br><br>prompt = PromptTemplate(<br>    template=prompt_template, input_variables=[&quot;context&quot;, &quot;question&quot;]<br>)<br><br>qa = RetrievalQA.from_chain_type(<br>    llm=llm,<br>    chain_type=&quot;stuff&quot;,<br>    retriever=contextual_retriever,<br>    return_source_documents=True,<br>    chain_type_kwargs={&quot;prompt&quot;: prompt, &quot;verbose&quot;: True},<br>)<br><br>response = qa.invoke(&quot;What is the most difficult case solved by Sherlock?&quot;)<br>print_response(response)</pre><h4>Explanation</h4><p>In this code:</p><ol><li>Prompt Template:</li></ol><ul><li>We define a prompt_template that instructs the LLaMA model to generate an answer based on the given question and context.</li></ul><p>2. Initialize the LLaMA Model:</p><ul><li>We use the ChatGroq class to initialize the LLaMA model with the specified API key.</li></ul><p>3. Create the RetrievalQA Chain:</p><ul><li>We create a RetrievalQA chain using the contextual retriever and the LLaMA model.</li><li>The ChatPromptTemplate is used to format the input question and context before passing it to the LLaMA model.</li></ul><p>4. Ask a Question:</p><ul><li>We define a question and invoke the qa_chain with this question to get the response.</li></ul><h3>The Power of the LLM</h3><p>By integrating the LLaMA model, Sherlocked can generate detailed and contextually accurate answers to user queries. This enhances the overall user experience, providing insights and information with the depth and clarity reminiscent of Sherlock Holmes himself.</p><h3>Example Questions &amp; Responses</h3><p>Here are some example questions you can ask Sherlocked and their responses:</p><p>Question: Who was the great hound really?<br>Response: The great hound was a curly-haired spaniel, not a supernatural creature.<br>Additional information:</p><ul><li>The hound was a real dog, not a spectral or hellish creature, as suggested by the family legend.</li><li>The dog was used by Stapleton to create the illusion of a supernatural hound, in an attempt to frighten Sir Charles to death.</li><li>The dog was kept in the out-house at Merripit House, and was occasionally taken out to create the illusion of a ghostly hound on the moor.</li><li>The dog was daubed with a luminous mixture to make it appear eerie and supernatural.</li></ul><p>Question: How did Irene trick Sherlock?<br>Response: Irene Adler outwitted Sherlock by escaping his detection in the case of the King of Bohemia.<br>Additional information:</p><ul><li>Irene Adler was a clever and cunning opponent who managed to outwit Sherlock Holmes, which was a rare occurrence.</li><li>Her cleverness and intelligence earned her a special place in Holmes’ memory, and he referred to her as “the woman” in later conversations.</li></ul><p>Question: How does Sherlock feel about his family?<br>Response: Sherlock is reserved about his family, but he mentions his brother Mycroft.<br>Additional information:</p><ul><li>Sherlock’s ancestors were country squires, and his grandmother was the sister of Vernet, a French artist.</li><li>He believes that his faculty of observation and deduction may be hereditary, citing his brother Mycroft as an example of someone who possesses these traits to a greater degree than he does.</li></ul><h3>Conclusion</h3><p>With Sherlocked, you now have the tools to explore the world of Sherlock Holmes like never before. By leveraging advanced parsing, vector embeddings, and high-performance data manipulation, you can dive deep into the mysteries solved by the legendary detective. Happy sleuthing!</p><p>For the full code and detailed instructions, visit the <a href="https://www.kaggle.com/code/nnisarg/ask-sherlock-rag-llama3-70b-reranking-llama-parser">Sherlocked Kaggle Notebook</a>.</p><h3>Future Enhancements</h3><p>To enhance Sherlocked with multi-user support, consider the following suggestions:</p><ol><li>UI Addition:</li></ol><ul><li>Use <a href="https://www.gradio.app/guides/creating-a-chatbot-fast">Gradio </a>to create an interactive web interface for Sherlocked.</li><li>Deploy the Gradio app on a platform like Hugging Face Spaces or Heroku for easy access.</li></ul><p>2. User Authentication:</p><ul><li>Implement user authentication using OAuth or JWT tokens.</li><li>Integrate with platforms like Google or GitHub for ease of use.</li></ul><p>3. Session Management:</p><ul><li>Create user-specific sessions to track interactions and maintain context.</li><li>Store session data in a database like MongoDB or PostgreSQL.</li></ul><p>4. Concurrency Handling:</p><ul><li>Ensure the backend can handle multiple concurrent users efficiently.</li><li>Use tools like Redis for caching and load balancing.</li></ul><p>Feel free to fork the <a href="https://github.com/nisargnegi/Sherlocked_RAG_Llama3_LlamaParse_Groq">Sherlocked GitHub repository</a> and contribute to this innovative project. If you have any questions or run into issues, don’t hesitate to open an issue or submit a pull request. Let’s keep the power of deduction alive !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67cbdde6ff37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Image Enhancement using Autoencoder]]></title>
            <link>https://medium.com/@nisargnegi/image-enhancement-using-autoencoder-3c48310ce801?source=rss-5dbe41d3e692------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c48310ce801</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[image-enhancement]]></category>
            <category><![CDATA[autoencoder]]></category>
            <dc:creator><![CDATA[Nisarg Negi]]></dc:creator>
            <pubDate>Wed, 25 Jan 2023 11:00:57 GMT</pubDate>
            <atom:updated>2023-01-25T11:44:53.644Z</atom:updated>
            <content:encoded><![CDATA[<p>Link to repo: <a href="https://github.com/nisargnegi/Intro-to-ML-Assignments/blob/main/Assignment3/Autoencoder_image.ipynb">https://github.com/nisargnegi/Intro-to-ML-Assignments/blob/main/Assignment3/Autoencoder_image.ipynb</a></p><ol><li><strong>Introduction</strong></li></ol><p>Autoencoder is a type of Artificial neural network which has the ability to learn the input representation. It can encode and compress data efficiently and then learn from the process how to reverse it and construct the input from the compressed data.</p><p>There are three main parts in an autoencoder: encoder, decoder and latent block(or bottleneck. The encoder extracts the best features, the latent space is used for storage of these features, and the decoder does the reverse of the encoder and reconstructs an image from using the features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*n53zBG7J7W0U4iy1qjxvww.png" /></figure><p>One application of Autoencoder is denoising data. Data can lose quality over transmission on the internet. Autoencoders can be used to get back the lost quality.</p><p><strong>2. Types of Autoencoder</strong></p><ol><li>Denoising Autoencoder: in this encoder, we create a copy of the input data with some noise added. This avoids the generalization of the input and makes the models learn features from the input data and train to recover the original data. It minimizes the loss function between the output and the noisy input. It is easy to setup. The only drawback is that his model does not memorize the training as the input and output keeps changing.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*YL7gw2ZwAJmH1gt3yQcJNA.png" /></figure><p>2. Sparse Autoencoder: in this encoder, we have a sparsity penalty in the training criterion. There are two ways to add sparsity: L1 regularization and KL-divergence. Adding sparsity helps to activate only fewer but insightful nodes, which helps the model learn from latent representations instead of redundant nodes. This also prevents overfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*wfc44m_bgAla56qWMLjXoA.png" /></figure><p>3. Deep Autoencoder: It works by creating two identical deep belief networks: one for encoding and one for decoding. Unsupervised layer-by-layer pretraining is used for this model. Each is a Restricted Boltzmann Machine which is the base of a deep belief network, a binary transform is performed after each layer. Encoding is fast using Deep autoencoder and they are used in topic modelling across a set of documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*QMlujECa3jEQUDkr7d6QXA.png" /></figure><p>4. Contractive Autoencoder: It is just like a sparse autoencoder, but here the regularizer used is calculated by the Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input. This model is robust and less sensitive to variations.</p><p>5. Undercomplete Autoencoder: They have a small number of hidden layers which help to capture the most important features in the data. They do not need any regularization as they maximize probability.</p><p>6. Convolutional Autoencoder: Convolutional Autoencoder uses a mixed sum of signals to reconstruct the input data rather than taking them one at a time like other models. It uses a convolution neural network, which is better at retaining connected information between the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*dgqd8yuXLA1OuynE62YbbA.png" /></figure><p><strong>3. Experiment</strong></p><p>We will use the ‘Labeled Faces in the Wild’ dataset. It contains photographs of faces used for face recognition. We will use this dataset to build a model for image resolution enhancement.</p><p>First, we will download the data set using wget and extract it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*5Fv56XZWOMZ7uWj6w0Illw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*vrhWDmZerxwPebg9wP2AJA.png" /></figure><p>We will reduce the size of our images as per the limitations of our device and reduce the training time. I have set it at 80x80.</p><p>Then we will split the data into training and testing. We will keep aside the test dataset and use it for model evaluation.</p><p>We then reduce the resolution of the images to 40% so as to add noise to the training data, which we will later enhance using our model. We will use this low-resolution image dataset as training data for our model.</p><p>We will then define our encoding model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*sn70Mocq7zJJufBXphpjow.png" /></figure><p>Our encoder consists of 2 Convolution neural networks followed by a max pool layer and then again a Convolution neural network. We have used the activation function as relu for all layers.</p><p>We will then define our decoding model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*VBG_xtTWFeMrX4a-OkF1Vg.png" /></figure><p>Our decoder consists of 1 Convolution neural network layer followed by an up-sampling layer and then 2 Convolution neural networks. We have used the activation function as relu for all layers.</p><p>Our model summary is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*UDrwA1P-TSJHLlz0WZV-mw.png" /></figure><p>I have set the model parameters as epochs = 5 and batch size = 256 depending on the limitations of my machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*CHO6cHtGc8BMqdsUxRPS9Q.png" /></figure><p>Now we can use the trained model on our test dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*2THgFAeyF-JZcVNvaZNi-w.png" /></figure><p><strong>4. References</strong></p><p>1. <a href="https://www.analyticsvidhya.com/blog/2020/02/what-is-autoencoder-enhance-image-resolution/">https://www.analyticsvidhya.com/blog/2020/02/what-is-autoencoder-enhance-image-resolution/</a></p><p>2. <a href="https://medium.com/ai%C2%B3-theory-practice-business/understanding-autoencoders-part-i-116ed2272d35">https://medium.com/ai%C2%B3-theory-practice-business/understanding-autoencoders-part-i-116ed2272d35</a></p><p>3. <a href="https://blog.francium.tech/enhance-images-with-autoencoders-58afa4fd638f">https://blog.francium.tech/enhance-images-with-autoencoders-58afa4fd638f</a></p><p>4. <a href="https://medium.com/@syoya/what-happens-in-sparse-autencoder-b9a5a69da5c6">https://medium.com/@syoya/what-happens-in-sparse-autencoder-b9a5a69da5c6</a></p><p>5. <a href="http://vis-www.cs.umass.edu/lfw/lfw.tgz">http://vis-www.cs.umass.edu/lfw/lfw.tgz</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c48310ce801" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to Support Vector Machine]]></title>
            <link>https://medium.com/@nisargnegi/intro-to-support-vector-machine-85de1d8e11d9?source=rss-5dbe41d3e692------2</link>
            <guid isPermaLink="false">https://medium.com/p/85de1d8e11d9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[svm]]></category>
            <dc:creator><![CDATA[Nisarg Negi]]></dc:creator>
            <pubDate>Wed, 25 Jan 2023 10:38:30 GMT</pubDate>
            <atom:updated>2023-01-25T10:39:18.347Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>1. Introduction</strong></p><p>The support Vector Machine (SVM) algorithm is a supervised machine learning algorithm used for regression classification analysis. In SVM, if we have a labelled dataset, data is divided into classes by drawing a hyperplane between them. SVM finds the maximum distance hyperplane from the data points, each side of the hyperplane forms 2 separate classes. In two-dimensional data, this hyperplane forms a line dividing the data into 2 different classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*fUJ_woGFGonyAhmhDjQuzQ.png" /><figcaption>Fig 1.1 hyperplane line dividing the data into 2 different classes</figcaption></figure><p>When we have more complex data in which we cannot draw a separating line as in fig 1.2 (a), we transform the data and add one more dimension say z-axis (Fig 1.2 (b)). We can observe that the points around the origin in xy plane will be clearly separated by the z-axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*4q9Hxc9u_EWszkNO5ya8CA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*tjyLsNOmMavlHwj-bnbfWA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*Ni3dZG2if4GUA8QKM2TKfQ.png" /><figcaption>Fig 1.2 a,b,c</figcaption></figure><p>For the datapoints in z plane we can consider the points in a circle with the equation w = x2 + y2 and manipulate these points as a function of distance from the z axis. If we transform them back to XY plane, we will get a circular boundary as in figure 1.2(c). These transforms are known as kernels.</p><p><strong>2. Parameters</strong></p><p>a. C Parameters:</p><p>It controls the distance between the data points and the hyperplane.</p><ul><li>large C: distance is small, it can overfit</li><li>small C: distance is large, it can underfit</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*AjgZwn_J2KXaHHnVfv5f5g.png" /></figure><p>b. Kernel:</p><p>as stated before, SVM uses kernel trick to transform nonlinear data. There are 4 types of kernels which we can use:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab3e30856ac18ad0125ec21d01783c4b/href">https://medium.com/media/ab3e30856ac18ad0125ec21d01783c4b/href</a></iframe><p>c. Gamma Parameter:</p><p>It defines how distant values from hyperplane must be considered in rbf, polynomial or sigmoid kernel.</p><ul><li>Low Gamma: far away points are considered</li><li>High Gamma: only close points are considered</li></ul><p><strong>3. Dataset</strong></p><p>We are using asteroseismology data which has the distinct features of oscillations of 6008 Kepler stars and using it to predict whether it is a Red Giant or a Helium burning star.</p><p>Features:</p><ul><li><strong>POP</strong>: Target Variable, if 0: Red giant star &amp; 1: Helium burning star</li><li><strong>Dnu</strong>: Mean large frequency separation of modes with the same degree and consecutive order</li><li><strong>Numax</strong>: Frequency of maximum oscillation power</li><li><strong>Epsilon</strong>: Location of the l=0 mode</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*UScGKZCT0Ous3ZSi0y_Org.png" /></figure><p><strong>4. Experiment</strong></p><p>From the asteroseismology dataset, we will use the features to predict whether the star is a Red Burning Giant or Helium Burning Star.</p><p>We will read the dataset from CSV and store it in a data frame. Then we will perform our initial analysis of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/1*Lqf96t1kwTTo8gIwuB-WUw.png" /></figure><p>We can see that all the features are integer or float so will not require any labelling or one hot encoding.</p><p>Then we will look for null values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/108/1*sE3ehbYg9-coUOYzgxE7Nw.png" /><figcaption>Count of Null values</figcaption></figure><p>We can tell that the data set does not have any null values and will not require imputation.</p><p>We will perform bivariate analysis using a pairplot. Pairplot helps us plot pairwise bivariate distributions in a dataset which helps us summarise the data visually. The x &amp; y axis has all the features plotting against the target variable POP and the diagonal has the distribution of each element against the target variable POP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/1*ZAi7ZcdPi_Gnxs6b--gUtA.png" /><figcaption>Pairplot</figcaption></figure><p>Then we will check for the count of each classification in the target variable POP to check if the data is not highly imbalanced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/1*tRLDrjSJydeRvtn9nh3mOQ.png" /></figure><p>Our data is not highly imbalanced.</p><p>We will then plot the correlation for each variable in the dataset. This helps us find highly correlated features which we can remove from our model training as they will not affect our training by much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPRm9FZKMKTfOmVHDqndHw.png" /></figure><p>We will then split our dataset into test(30%) and training(70%).</p><p>We will then fit our Training data into an SVM classifier and predict the result. We will compare this result with the actual values from the dataset by using a confusion matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*DVUZtgDowDPUsG0bp1Nx2A.png" /></figure><p>We are getting an accuracy of 92.03%. This accuracy is without any normalization and parameter tuning.</p><p>We can get a better result. We will then normalize our data and check if our accuracy increases:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*UNnmRuM9JGQKxIEXs1dD6w.png" /></figure><p>Normalizing the data increases our accuracy to 94.02%.</p><p>We can increase this further by parameter optimization. We will now perform parameter optimization by using grid search. For this will run SVM model prediction on different values of C, Gamma and Kernels.</p><ul><li>C: 0.1, 1, 10, 100</li><li>Gamma: 1, 0.1, 0.01, 0.001</li><li>Kernel: rbf, polynomial, sigmoid, linear.</li></ul><p>Using GridSearchCV we will run SVM for all the above parameters and select the best ones that fit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*3hrgn_UQ1NEh601TspzsiQ.png" /><figcaption>Grid search result</figcaption></figure><p>We find out that the best parameters are C = 10, gamma = 1 and kernel being rbf.</p><p>Now, finally, we will train our model using the best parameter that we found and build our confusion matrix and test for the accuracy score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*swGSmANDCQ-m_ek1Mpppww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*9i9KUrg1AHMtNMjQITAC4A.png" /></figure><p>We are now getting the accuracy of 96.01% which is greater than the previous conditions.</p><p>This tells us how using normalization and parameters optimization we can increase our accuracy.</p><p><strong>4. References</strong></p><p>a.<a href="https://medium.com/machine-learning-101/chapter-2-svm-support-vector-machine-theory-f0812effc72">https://medium.com/machine-learning-101/chapter-2-svm-support-vector-machine-theory-f0812effc72</a></p><p>b.<a href="https://medium.com/@cdabakoglu/what-is-support-vector-machine-svm-fd0e9e39514f">https://medium.com/@cdabakoglu/what-is-support-vector-machine-svm-fd0e9e39514f</a></p><p>c. <a href="https://cdsarc.cds.unistra.fr/viz-bin/cat/J/MNRAS/469/4578#/article">https://cdsarc.cds.unistra.fr/viz-bin/cat/J/MNRAS/469/4578#/article</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85de1d8e11d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>