<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rudraj on Medium]]></title>
        <description><![CDATA[Stories by Rudraj on Medium]]></description>
        <link>https://medium.com/@rudraj18110?source=rss-80d8ccba8703------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*-aap6Jr5H-lUJQLE</url>
            <title>Stories by Rudraj on Medium</title>
            <link>https://medium.com/@rudraj18110?source=rss-80d8ccba8703------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:24:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rudraj18110/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Handling Character Encodings in Python]]></title>
            <link>https://medium.com/@rudraj18110/handling-character-encodings-in-python-7b7590bb7dd5?source=rss-80d8ccba8703------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b7590bb7dd5</guid>
            <category><![CDATA[unicode]]></category>
            <category><![CDATA[character-encoding]]></category>
            <category><![CDATA[utf-8]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Rudraj]]></dc:creator>
            <pubDate>Sun, 04 Oct 2026 11:20:13 GMT</pubDate>
            <atom:updated>2026-10-04T11:20:13.919Z</atom:updated>
            <content:encoded><![CDATA[<p>When working with Python, we usually deal with text without thinking much about how computers actually store it. We type words, save files, read documents, and print messages on the screen. However, computers do not directly understand characters such as A, é, 中, or ₹. They store information as numbers and bytes. Character encoding is the system that allows these characters to be represented and exchanged correctly.</p><p>Understanding character encoding becomes especially important when a Python program reads files, works with databases, processes web data, or communicates with another system. If the wrong encoding is used, characters may appear incorrectly or Python may generate errors. This blog explains how Python handles character encodings and how developers can avoid common problems.</p><h3>What Is Character Encoding?</h3><p>Character encoding is a method of representing characters as numerical values that can be stored or transmitted by a computer.</p><p>One of the earliest and most well-known character encoding systems is <strong>ASCII (American Standard Code for Information Interchange)</strong>. ASCII represents basic English characters using numbers from 0 to 127. For example, the character A is represented by the number 65.</p><p>However, ASCII is limited because it mainly covers English letters, digits, punctuation, and a small number of control characters. It cannot represent the huge number of characters used in languages around the world.</p><p>This led to the development and widespread use of <strong>Unicode</strong>, which provides a standard way of representing characters from many writing systems. Unicode includes characters from languages such as Hindi, Japanese, Arabic, Chinese, and many others, along with symbols and emojis.</p><h3>Unicode and UTF-8</h3><p>Unicode and UTF-8 are related, but they are not the same thing.</p><p><strong>Unicode</strong> is a character standard that assigns code points to characters. <strong>UTF-8</strong> is an encoding method used to represent those Unicode characters as bytes.</p><p>UTF-8 is widely used because it is compatible with ASCII for basic English characters while also supporting a very large range of Unicode characters.</p><p>For example, Python can work with a Unicode string containing characters from different languages:</p><pre>text = &quot;Hello, नमस्ते, こんにちは&quot;<br>print(text)</pre><p>Python 3 handles strings as Unicode text by default. This means that programmers can normally work with characters from different languages without manually converting every character.</p><h3>Strings and Bytes in Python</h3><p>One of the most important concepts when dealing with encoding in Python is understanding the difference between a <strong>string</strong> and <strong>bytes</strong>.</p><p>A Python str represents text, while bytes represents a sequence of bytes.</p><p>For example:</p><pre>text = &quot;Hello ₹&quot;<br>print(type(text))</pre><p>The output will be:</p><pre>&lt;class &#39;str&#39;&gt;</pre><p>If we encode this string into UTF-8:</p><pre>text = &quot;Hello ₹&quot;<br>data = text.encode(&quot;utf-8&quot;)</pre><pre>print(data)<br>print(type(data))</pre><p>The result is a bytes object. The exact byte representation is handled by UTF-8.</p><p>The .encode() method converts a Python string into bytes using the specified encoding.</p><p>To convert those bytes back into a Python string, we use .decode():</p><pre>data = &quot;Hello ₹&quot;.encode(&quot;utf-8&quot;)<br>text = data.decode(&quot;utf-8&quot;)</pre><pre>print(text)</pre><p>This gives:</p><pre>Hello ₹</pre><p>Therefore, a simple way to remember the process is:</p><p><strong>String → encode → bytes</strong></p><p>and</p><p><strong>Bytes → decode → string</strong></p><h3>Reading Text Files with Encoding</h3><p>Character encoding becomes particularly important when Python reads a text file.</p><p>For example, suppose we have a file called message.txt containing:</p><pre>Hello, नमस्ते!</pre><p>We can explicitly specify UTF-8 while opening the file:</p><pre>with open(&quot;message.txt&quot;, &quot;r&quot;, encoding=&quot;utf-8&quot;) as file:<br>    content = file.read()</pre><pre>print(content)</pre><p>Here, encoding=&quot;utf-8&quot; tells Python how the bytes stored in the file should be interpreted as characters.</p><p>Specifying the encoding is a good practice because it makes the program’s expectation clear instead of relying on the computer’s default encoding.</p><p>We can also specify an encoding when writing a file:</p><pre>text = &quot;Python supports Unicode text.&quot;</pre><pre>with open(&quot;output.txt&quot;, &quot;w&quot;, encoding=&quot;utf-8&quot;) as file:<br>    file.write(text)</pre><p>The file is written using UTF-8, making it easier to exchange the file with other systems that support UTF-8.</p><h3>What Happens When the Wrong Encoding Is Used?</h3><p>Problems occur when the encoding used to decode data does not match the encoding that was originally used.</p><p>For example, imagine a file contains UTF-8 encoded text. If a program tries to read those bytes using an incompatible encoding, Python may raise a UnicodeDecodeError.</p><p>A simplified example is:</p><pre>data = &quot;₹&quot;.encode(&quot;utf-8&quot;)</pre><pre>print(data.decode(&quot;ascii&quot;))</pre><p>This can produce a UnicodeDecodeError because the UTF-8 byte sequence for the rupee symbol cannot be correctly interpreted as ASCII.</p><p>The important lesson is that encoding and decoding must match the actual representation of the data.</p><h3>Handling Encoding Errors</h3><p>Python provides error-handling options when encoding or decoding text.</p><p>For example:</p><pre>data = b&quot;Hello \xff&quot;</pre><pre>text = data.decode(&quot;utf-8&quot;, errors=&quot;replace&quot;)</pre><pre>print(text)</pre><p>The replace option substitutes characters that cannot be decoded correctly with a replacement character.</p><p>Another option is ignore:</p><pre>text = data.decode(&quot;utf-8&quot;, errors=&quot;ignore&quot;)<br>print(text)</pre><p>This skips bytes that cannot be decoded.</p><p>However, these options should be used carefully. Ignoring or replacing invalid data can result in information being lost. It is generally better to identify why the encoding is incorrect and use the correct encoding whenever possible.</p><h3>Why UTF-8 Is Commonly Preferred</h3><p>UTF-8 has become one of the most commonly used encodings for modern applications. One reason is its compatibility with ASCII. Basic English characters use the same byte values in both ASCII and UTF-8.</p><p>At the same time, UTF-8 can represent a wide range of Unicode characters. This makes it useful for applications that need to support multiple languages.</p><p>For example:</p><pre>languages = [&quot;English&quot;, &quot;हिन्दी&quot;, &quot;日本語&quot;, &quot;العربية&quot;]</pre><pre>for language in languages:<br>    print(language)</pre><p>Python can store these strings normally because Python 3’s str type supports Unicode text.</p><p>This is useful in applications such as websites, student management systems, databases, multilingual applications, and data-processing programs.</p><h3>Character Encoding in CSV Files</h3><p>Encoding problems can also appear when working with CSV files. For example, a CSV file containing names or addresses in different languages may not display correctly if it is opened using the wrong encoding.</p><p>Python can specify the encoding while reading a CSV file:</p><pre>import csv</pre><pre>with open(&quot;students.csv&quot;, &quot;r&quot;, encoding=&quot;utf-8&quot;, newline=&quot;&quot;) as file:<br>    reader = csv.reader(file)</pre><pre>    for row in reader:<br>        print(row)</pre><p>This is particularly useful when processing datasets containing international names or other non-English text.</p><h3>Common Encoding Mistakes</h3><p>There are a few common mistakes programmers should avoid.</p><p>The first is assuming that every text file uses the same encoding. Different files may have been created using different encodings.</p><p>The second is confusing strings with bytes. A Python string is text, while bytes are raw binary data. When moving between these two forms, encoding and decoding are required.</p><p>Another common mistake is ignoring encoding errors without understanding their cause. While errors=&quot;ignore&quot; can prevent an exception, it can also remove important characters.</p><p>Finally, relying completely on the operating system’s default encoding can make a program behave differently on another computer. Explicitly specifying the expected encoding is often a safer approach.</p><h3>Best Practices for Handling Character Encodings</h3><p>A few simple practices can prevent many encoding-related problems:</p><ol><li><strong>Use Unicode strings in Python 3</strong> for normal text processing.</li><li><strong>Prefer UTF-8</strong> when you control the format or storage of text.</li><li><strong>Specify the encoding explicitly</strong> when opening text files.</li><li><strong>Encode text only when bytes are required</strong>, such as when writing binary data or communicating with certain systems.</li><li><strong>Decode bytes using the correct encoding</strong> before treating them as text.</li><li><strong>Do not ignore encoding errors blindly</strong>, because important information may be lost.</li><li><strong>Test applications with non-English characters</strong> instead of testing only with English text.</li></ol><h3>Conclusion</h3><p>Character encoding is an important part of working with text in Python. While Python 3 makes Unicode text handling relatively simple, developers still need to understand what happens when text is converted into bytes and when bytes are converted back into text.</p><p>The key concepts are easy to remember: strings represent text, bytes represent binary data, encoding converts strings into bytes, and decoding converts bytes back into strings. UTF-8 is commonly used because it can represent a wide range of Unicode characters and works well with standard English text.</p><p>By explicitly specifying encodings when reading and writing files and by understanding errors such as UnicodeDecodeError, Python programmers can build applications that handle text more reliably. This becomes increasingly important as applications work with international users, multilingual data, APIs, databases, and files created on different systems.</p><p>Understanding character encoding may seem like a small part of programming, but it becomes very important whenever text moves between different systems. Knowing how Python handles Unicode, UTF-8, strings, and bytes helps prevent unexpected characters, corrupted data, and difficult-to-debug errors.</p><p>This blog is created as part of learning at Pillai College of Engineering (PCE). For more details, visit <a href="https://www.pce.ac.in/">https://www.pce.ac.in</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b7590bb7dd5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>