<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abdul Wasiue on Medium]]></title>
        <description><![CDATA[Stories by Abdul Wasiue on Medium]]></description>
        <link>https://medium.com/@abdulwasiueunk?source=rss-13b81e473468------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ERazST3X6raL8AlqzfofJg.jpeg</url>
            <title>Stories by Abdul Wasiue on Medium</title>
            <link>https://medium.com/@abdulwasiueunk?source=rss-13b81e473468------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:10:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abdulwasiueunk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Spark vs Hadoop — Understanding Big Data]]></title>
            <link>https://medium.com/@abdulwasiueunk/apache-spark-vs-hadoop-understanding-big-data-7778f7e8e31a?source=rss-13b81e473468------2</link>
            <guid isPermaLink="false">https://medium.com/p/7778f7e8e31a</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Abdul Wasiue]]></dc:creator>
            <pubDate>Wed, 04 Feb 2026 12:06:20 GMT</pubDate>
            <atom:updated>2026-02-04T12:06:20.882Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Spark vs Hadoop — Understanding Big Data</h3><p>The title “Apache Spark vs Hadoop” might seem straightforward, but it actually reflects how deeply you understand <strong>big data and its inner workings</strong>. We can’t simply say “Hadoop vs Spark” because Hadoop and Spark serve different purposes, and Spark often runs <strong>on top of Hadoop</strong>. Confusing, right?</p><p>Before Spark, there was Hadoop. After Spark, Hadoop is still around. Let’s start with Hadoop.</p><p>In simple terms, Hadoop can work standalone, integrating with Hive, MapReduce, Oozie, HBase, and other components. Spark, however, cannot run without Hadoop, because Hadoop is not just a framework - it’s an architecture and the foundation of distributed parallel processing.</p><h3><strong>What is Hadoop?</strong></h3><p>Apache Hadoop is an open-source framework designed for storing and processing <strong>massive amounts of data</strong> across clusters of computers in a distributed manner. Developed to handle <strong>big data</strong> efficiently, Hadoop allows organizations to process datasets that are too large for a single machine. Its strength lies in its <strong>scalability, fault tolerance, and cost-effectiveness</strong>, making it a cornerstone of modern data processing systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/0*z4KlNVh0q2POehlG.png" /><figcaption>Hadoop Architecture — GeeksForGeeks</figcaption></figure><h4><strong>Core Components of Hadoop</strong></h4><p>Hadoop Distributed File System (HDFS):</p><ul><li>HDFS is Hadoop’s storage layer.</li><li>It splits large files into smaller blocks and stores them across multiple nodes in a cluster.</li><li>It ensures <strong>fault tolerance</strong> by replicating each block on multiple nodes, so if one node fails, data is still safe.</li></ul><p>Hadoop works on the principle of <strong>distributed computing</strong>, breaking down large datasets into smaller chunks and processing them across a cluster of machines:</p><ol><li>Data is first <strong>split into blocks</strong> and stored in HDFS.</li><li>When a job is submitted, YARN <strong>allocates resources</strong> across the cluster.</li><li>MapReduce processes the data in parallel:</li></ol><ul><li>The <strong>Map function</strong> transforms input data into key-value pairs.</li><li>The <strong>Reduce function</strong> aggregates and summarizes the results.</li></ul><p>4. The final output is written back to <strong>HDFS</strong>, ready for analysis or reporting.</p><p>This architecture allows Hadoop to <strong>scale horizontally</strong>, meaning you can handle more data simply by adding more nodes to the cluster.</p><h3><strong>What is Apache Spark?</strong></h3><p>Apache Spark is an open-source <strong>distributed data processing framework</strong> designed for <strong>speed, ease of use, and advanced analytics</strong>. Unlike Hadoop’s MapReduce, Spark performs <strong>in-memory computation</strong>, which makes it <strong>much faster</strong> for many data processing tasks, especially iterative operations like machine learning and graph processing. Spark supports multiple programming languages, including <strong>Java, Scala, Python, and R</strong>, making it highly flexible for developers and data scientists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*EzZs4uEuO30lV51KV07_RA.png" /><figcaption>Apache Spark — Master-Slave</figcaption></figure><p><strong>Core Components of Apache Spark</strong></p><p>Spark has a rich ecosystem that allows it to handle a wide variety of big data tasks:</p><ol><li><strong>Spark Core</strong></li></ol><ul><li>The foundation of Spark, responsible for <strong>task scheduling, memory management, fault recovery</strong>, and basic I/O operations.</li><li>Supports in-memory computing, which dramatically increases performance compared to disk-based processing.</li></ul><p><strong>2. Spark SQL</strong></p><ul><li>Enables <strong>structured data processing</strong> using SQL queries or DataFrame API.</li><li>Integrates with various data sources like Hive, JSON, Parquet, and JDBC.</li></ul><p><strong>3. Spark Streaming</strong></p><ul><li>Processes <strong>real-time streaming data</strong> from sources like Kafka, Flume, or socket connections.</li><li>Supports near real-time analytics and monitoring.</li></ul><p><strong>4. MLlib (Machine Learning Library)</strong></p><ul><li>Provides scalable machine learning algorithms for classification, regression, clustering, and recommendation systems.</li></ul><p><strong>5. GraphX</strong></p><p>A library for <strong>graph processing</strong> and graph-parallel computation, useful for social networks, recommendation engines, and network analysis. Like mutual friends on facebook.</p><p><strong>How Spark Works</strong></p><ul><li>Spark reads data from <strong>HDFS or other storage systems</strong> into memory as <strong>Resilient Distributed Datasets (RDDs)</strong> or <strong>DataFrames</strong>.</li><li>Data is <strong>processed in parallel across the cluster</strong>, with transformations applied lazily and actions triggering execution.</li><li>Spark optimizes tasks using the <strong>Directed Acyclic Graph (DAG) scheduler</strong>, reducing unnecessary data shuffling and disk writes.</li><li>Final results can be <strong>written back to storage</strong> or fed directly into analytics pipelines.</li></ul><p>Because Spark keeps data <strong>in memory</strong> wherever possible, it can be <strong>10–100x faster than Hadoop MapReduce</strong> for certain workloads, especially those involving multiple iterations over the same dataset. In memory (like RAM and cache) do not confuse like its a rocket science.</p><p><strong>So, Why that means Hadoop is dead?</strong></p><p>No sir, never. See, spark is running on top of Hadoop. Only Map Reduce is replaced by spark. And spark can process data in batches as well streaming but hadoop is only processing the data in batches.</p><p>Ok. What is batch processing and streaming? Batch processing is tored data in somewhere (database) and we process it, where as streaming is live data which we process.</p><p><strong>Conclusion</strong>: Hadoop is not dead rather we still use its components on top of Spark, only map reduced is replaced by Spark. Hadoop supports only batch processing but Spark supports both batch and streaming porcessing. That being said, this blog is about the difference so don’t confuse yourself as whole architecture of Apache spark or Hadoop.</p><p>#hadoop #bigdata #spark #processing</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7778f7e8e31a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Production-Ready dbt Pipeline with Airflow (Astronomer) and Snowflake — Without Losing…]]></title>
            <link>https://medium.com/@abdulwasiueunk/building-a-production-ready-dbt-pipeline-with-airflow-astronomer-and-snowflake-without-losing-6aacf6470a8b?source=rss-13b81e473468------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aacf6470a8b</guid>
            <dc:creator><![CDATA[Abdul Wasiue]]></dc:creator>
            <pubDate>Mon, 05 Jan 2026 07:25:47 GMT</pubDate>
            <atom:updated>2026-01-05T07:50:43.891Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Production-Ready dbt Pipeline with Airflow (Astronomer) and Snowflake</h3><p>If you have ever tried to glue <strong>dbt</strong>, <strong>Airflow</strong>, and <strong>Snowflake</strong> together, you already know this truth:</p><blockquote><em>“The tools are powerful, but the integration can feel painfully fragile.”</em></blockquote><p>This blog is a walkthrough of how I built a <strong>production-ready dbt pipeline orchestrated by Airflow using Astronomer and Cosmos</strong>, what broke along the way, and more importantly how I fixed it.</p><p>No theory heavy fluff. Just practical lessons from real errors, real containers, and real frustration.</p><h3>Snowflake — The Cloud Data Warehouse</h3><p>Snowflake is a <strong>cloud-native data platform</strong> designed to store, process, and analyze large volumes of structured and semi-structured data. Unlike traditional databases, Snowflake separates <strong>storage from compute</strong>, which allows teams to scale resources independently, paying only for what they use. It supports multi-cluster processing, concurrency without performance hits, and has native support for JSON, Avro, Parquet, and other modern data formats.</p><p><strong>Core Concepts:</strong></p><ul><li><strong>Database &amp; Schemas:</strong> Organize data logically.</li><li><strong>Warehouses:</strong> Compute clusters that execute queries.</li><li><strong>Separation of Storage &amp; Compute:</strong> Scale independently for cost efficiency.</li><li><strong>Zero-maintenance:</strong> Cloud-managed infrastructure.</li></ul><p><em>In simple terms:</em> Snowflake is like a highly elastic, cloud-based library for your data fast, scalable, and always available.</p><h3>dbt (Data Build Tool) — Transformations Made Reliable</h3><p>dbt is a <strong>transformations-focused tool</strong> for analytics engineering. Instead of writing SQL scripts manually, dbt allows you to define <strong>modular models</strong> (SQL SELECT statements), test data quality, document data lineage, and version-control transformations. Its philosophy is “<strong>analytics engineering as software engineering</strong>”: everything is versioned, testable, and reproducible.</p><p><strong>Core Concepts:</strong></p><ul><li><strong>Models:</strong> SQL SELECT statements that create transformed tables/views.</li><li><strong>Tests:</strong> Validate data integrity (e.g., uniqueness, null values, relationships).</li><li><strong>Macros:</strong> Reusable SQL snippets/functions.</li><li><strong>Documentation &amp; Lineage:</strong> Auto-generate DAGs of transformations.</li></ul><p><em>In simple terms:</em> dbt takes raw data and helps you build clean, tested, and maintainable tables that your business can rely on.</p><h3>Airflow — Orchestration Engine</h3><p>Airflow is a <strong>workflow orchestration tool</strong>. It allows you to <strong>schedule, monitor, and manage complex data pipelines</strong> using Python code. Workflows are defined as DAGs (Directed Acyclic Graphs), where each task is a node in the graph, and dependencies define execution order. Airflow excels at <strong>reliability, retries, monitoring, and alerting</strong> for ETL pipelines and machine learning workflows.</p><p><strong>Core Concepts:</strong></p><ul><li><strong>DAGs:</strong> Represent the workflow and task dependencies.</li><li><strong>Operators:</strong> Individual tasks (BashOperator, PythonOperator, DbtRunOperator).</li><li><strong>Scheduler:</strong> Executes tasks based on schedule or triggers.</li><li><strong>Task Instance &amp; Logs:</strong> Track the state, success, or failure of each task.</li></ul><p><em>In simple terms:</em> Airflow is like the conductor of an orchestra — coordinating multiple instruments (tasks) to play in perfect harmony.</p><h3>Why dbt + Airflow + Snowflake?</h3><p>Let’s start with the <em>why</em>.</p><ul><li><strong>dbt</strong> is excellent for transformations, testing, and analytics engineering.</li><li><strong>Snowflake</strong> is fast, scalable, and widely adopted.</li><li><strong>Airflow</strong> is still the most flexible orchestration engine out there.</li></ul><p>But orchestrating dbt inside Airflow traditionally meant:</p><ul><li>BashOperators</li><li>Long shell commands</li><li>Poor visibility</li><li>Manual dependency management</li></ul><p>That’s where <strong>Astronomer Cosmos</strong> changes the game.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MrAE4yRp8mOzXOmO1ko8Tw.png" /><figcaption>Design Flow</figcaption></figure><p>Cosmos allows Airflow to <strong>understand dbt projects natively, </strong>each model becomes a task, dependencies are inferred automatically, and observability improves drastically.</p><h3><strong>The High-Level Architecture</strong></h3><pre>   Snowflake (Raw Tables)<br>        ↓<br>   dbt models (Transformation) - Documentation and Code Pipeline<br>        ↓<br>   Cosmos DbtDag (Integretion) <br>        ↓<br>   Airflow DAG  (Monitor)</pre><h3>Project Structure (What Actually Worked)</h3><p>Cosmos <em>will not guess your dbt project location</em>.<br> If dbt_project.yml is not exactly where you point to it, the DAG will break.</p><pre>.<br>├── dags/<br>│   └── dbt_dag.py<br>├── dbt_pipeline/<br>│   ├── dbt_project.yml<br>│   ├── models/<br>│   │   ├── staging/<br>│   │   └── marts/<br>│   ├── macros/<br>│   └── tests/<br>├── Dockerfile<br>├── requirements.txt<br>└── airflow_settings.yaml</pre><h3>Writing the DAG with Cosmos</h3><p>Instead of writing multiple operators, the DAG is defined using DbtDag</p><pre>from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig<br>from cosmos.profiles import SnowflakeUserPasswordProfileMapping<br>from datetime import datetime<br>import os<br><br>profile_config = ProfileConfig(<br>    profile_name=&quot;default&quot;,<br>    target_name=&quot;dev&quot;,<br>    profile_mapping=SnowflakeUserPasswordProfileMapping(<br>        conn_id=&quot;snowflake_conn&quot;,<br>        profile_args={<br>            &quot;database&quot;: &quot;dbt_db&quot;,<br>            &quot;schema&quot;: &quot;dbt_schema&quot;,<br>        },<br>    ),<br>)<br><br>dbt_dag = DbtDag(<br>    dag_id=&quot;dbt_dag&quot;,<br>    project_config=ProjectConfig(&quot;/usr/local/airflow/dags/dbt_pipeline&quot;),<br>    profile_config=profile_config,<br>    execution_config=ExecutionConfig(<br>        dbt_executable_path=f&quot;{os.environ[&#39;AIRFLOW_HOME&#39;]}/dbt_venv/bin/dbt&quot;<br>    ),<br>    schedule_interval=&quot;@daily&quot;,<br>    start_date=datetime(2023, 9, 10),<br>    catchup=False,<br>)</pre><p>Airflow gets automatically: One task per dbt model, Correct dependencies Retry handling and Logs per model. No BashOperators, No hard-coded commands.</p><h3>Docker: The Necessary Evil</h3><p>I will be honest: Docker caused most of the pain, Port 5432 Errors</p><p>At one point, Airflow refused to start with:</p><pre>ports are not available: exposing port 5432</pre><p>The issue wasn’t Docker itself, it was <strong>Postgres already running on the host machine</strong>. Fix options:</p><ul><li>Stop the local Postgres service</li><li>OR change the exposed port in Astronomer</li></ul><p>Lesson learned:</p><blockquote><em>Docker errors are often</em><strong><em> </em>host problems</strong><em>, not container problems.</em></blockquote><h3>Key Takeaways</h3><p>If you’re building something similar, here’s what I’d emphasize:</p><ol><li><strong>Always verify paths inside the container</strong></li><li><strong>Don’t include Snowflake domains in the account</strong></li><li><strong>Cosmos is strict, but predictable</strong></li><li><strong>Docker errors are often host-level</strong></li><li><strong>Warnings ≠ failures</strong></li></ol><p>Most importantly: <em>Once it works, it works </em>cleanly<em>.</em></p><p>Here is github link: <a href="https://github.com/Wasiue03/dbt-pipeline">https://github.com/Wasiue03/dbt-pipeline</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aacf6470a8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake and the Evolution of Modern Analytics]]></title>
            <link>https://medium.com/@abdulwasiueunk/snowflake-and-the-evolution-of-modern-analytics-75da672a4799?source=rss-13b81e473468------2</link>
            <guid isPermaLink="false">https://medium.com/p/75da672a4799</guid>
            <category><![CDATA[data-handling]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[dimensional-model]]></category>
            <dc:creator><![CDATA[Abdul Wasiue]]></dc:creator>
            <pubDate>Wed, 17 Dec 2025 04:18:57 GMT</pubDate>
            <atom:updated>2025-12-17T04:18:57.243Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Introduction: Why Snowflake Represents a Shift in Data Thinking</strong></h4><p>Over the past decade, the way organizations work with data has changed dramatically. What once revolved around on-premise databases, nightly batch jobs, and rigid reporting pipelines has evolved into cloud-native platforms that prioritize scalability, flexibility, and speed. Among these platforms, Snowflake has emerged not merely as a popular technology, but as a reflection of how modern analytics is actually practiced.</p><p>Understanding Snowflake, therefore, is not just about learning a tool; it is about understanding a shift in data thinking.</p><p>At first glance, Snowflake is often described as a cloud data warehouse. While this description is technically correct, it undersells the broader value of the platform. Snowflake is better understood as an analytics foundation that enables teams to ingest, transform, model, govern, and share data at scale. Its architecture and features encourage better design choices, cleaner data pipelines, and stronger alignment between technical teams and business users.</p><p><strong>Snowflake Architecture: </strong>One of the most important architectural decisions behind Snowflake is the separation of storage and compute. In traditional systems, these two concerns are tightly coupled, meaning that increasing query performance often requires duplicating data or overprovisioning hardware. Snowflake breaks this limitation by storing data centrally while allowing multiple independent compute clusters, known as virtual warehouses, to access it. This design enables different workloads to run simultaneously without interfering with one another. Data engineers can execute heavy transformations, analysts can run ad-hoc queries, and dashboards can refresh in parallel, all without contention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*HIlSrKmtdfHjAMNLY5rC8w.png" /><figcaption>Image by Snowflake</figcaption></figure><p>This architectural flexibility directly influences how teams work. Instead of negotiating for shared resources or scheduling queries during off-peak hours, teams can scale compute up or down based on actual needs. More importantly, this separation changes the conversation from infrastructure constraints to analytical intent. The focus shifts away from “Can the system handle this query?” to “What questions do we want to answer with our data?”</p><h3><strong>Dimensional Modeling in Modern Snowflake Projects</strong></h3><p>Dimensional modeling is a methodology designed to organize data around business processes rather than technical transactions. In practice, this means identifying facts, such as sales or transactions, and describing them through dimensions like customers, products, dates, or locations. Star schemas, where a central fact table connects to denormalized dimension tables, are particularly well suited for Snowflake because they align naturally with analytical queries and BI tools. Queries become easier to write, performance is predictable, and the data structure mirrors how business users think about their domain.</p><p><strong>Star Schema: </strong>Within dimensional modeling, the star schema is the most commonly used design pattern. In a star schema, a central fact table connects to multiple denormalized dimension tables. This structure simplifies query logic and improves performance by reducing the number of joins required.</p><p><strong>Snowflake Schema: </strong>While star schemas are the default, there are scenarios where a snowflake schema becomes appropriate. In a snowflake schema, dimension tables are normalized into multiple related tables, often to represent hierarchies such as geography or product classification.</p><p>There are cases where a snowflake schema, in which dimensions are normalized into hierarchical tables, becomes useful. This typically happens when dimensions grow large, complex, or highly hierarchical, such as geographic or product classification data. However, in many modern Snowflake environments, storage costs are relatively low compared to the value of simplicity. As a result, star schemas remain the default choice, reinforcing the idea that clarity often outweighs theoretical normalization in analytics systems.</p><p>In Snowflake-based ecosystems, semantic layers can be implemented in several ways. Business intelligence tools often provide their own semantic modeling capabilites, allowing teams to define measures, relationships, and calculations centrally. Modern data transformation tools such as dbt also play an important role by embedding business logic directly into version controlled models. In some cases, secure views within Snowflake are used to expose curated, governed datasets to downstream consumers.</p><h4>Dynamic Tables: Simplifying ELT and Data Refresh</h4><p>Once data is modeled correctly, keeping it up to date becomes the next challenge. Traditionally, this required complex orchestration using streams, tasks, and external schedulers. Snowflake’s Dynamic Tables introduce a simpler and more declarative approach.</p><p>With Dynamic Tables, engineers define transformation logic and specify how fresh the data should be. Snowflake then automatically manages refresh timing, incremental processing, and dependencies. This reduces operational overhead and allows teams to focus on data logic rather than pipeline mechanics.</p><p>This shift from imperative orchestration to declarative intent reflects a broader trend in data engineering. Teams increasingly want to describe what the data should look like rather than how to move it step by step. By reducing operational overhead, Dynamic Tables free engineers to focus on data quality, modeling, and business logic, which ultimately deliver more value than pipeline maintenance.</p><h4><strong>Semantic Layer: Translating Data into Business Language</strong></h4><p>Even well-modeled tables are not inherently meaningful to business users. Raw column names and join logic can still act as barriers to insight. The semantic layer addresses this problem by translating technical data into business-friendly metrics and dimensions.</p><p>Regardless of the implementation, the purpose of the semantic layer remains the same: to ensure that everyone is speaking the same data language. When revenue, active users, or conversion rates are defined once and reused consistently, organizations spend less time debating numbers and more time making decisions. Snowflake supports this approach by providing strong governance features, fine-grained access control, and the ability to expose curated datasets without duplicating data.</p><h4><strong>Data Profiling and Quality in Snowflake</strong></h4><p>Trust in data, however, extends beyond consistent definitions. It also depends on data quality and understanding. Data profiling is a critical practice that helps teams assess the completeness, accuracy, and distribution of data before it is used for reporting or machine learning. In Snowflake, data profiling is typically performed using SQL queries and metadata views rather than a single built-in tool. Analysts and engineers examine null rates, uniqueness, value ranges, and anomalies to validate assumptions and identify potential issues early.</p><h4><strong>Snowflake Data Marketplace and Secure Data Sharing</strong></h4><p>Snowflake extends beyond internal analytics through its Data Marketplace and secure data sharing capabilities. Instead of copying or ingesting external datasets, organizations can query live third-party data directly within Snowflake.</p><p>This zero-copy sharing model simplifies data enrichment and ensures that consumers always access up-to-date information. It also enables organizations to share their own data securely, opening new opportunities for collaboration and monetization.</p><h4><strong>Conclusion:</strong></h4><p>Snowflake’s real value lies not in any single feature, but in how its capabilities work together to support modern analytics. From scalable architecture and thoughtful modeling to automated transformations and consistent business definitions, Snowflake enables teams to build data systems that are both powerful and trustworthy.</p><p>Ultimately, Snowflake succeeds when it is used intentionally. The platform provides the tools, but it is the responsibility of data professionals to design systems that align with business goals and analytical needs. Those who understand this balance are not simply users of Snowflake — they are practitioners of modern analytics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75da672a4799" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Analysis: Raw Data To Insights]]></title>
            <link>https://medium.com/@abdulwasiueunk/data-analysis-raw-data-to-insights-22f0c6b7eb95?source=rss-13b81e473468------2</link>
            <guid isPermaLink="false">https://medium.com/p/22f0c6b7eb95</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Abdul Wasiue]]></dc:creator>
            <pubDate>Sat, 07 Sep 2024 15:57:06 GMT</pubDate>
            <atom:updated>2024-09-07T15:59:07.336Z</atom:updated>
            <content:encoded><![CDATA[<p>Business today has evolved significantly compared to 20 years ago, largely due to advances in technology and the rise of entrepreneurship. In the past, obtaining, understanding, and acting on information was a slow and challenging process. Entrepreneurs often relied on individuals with access to data, which took considerable time to gather and analyze.</p><p>One of the biggest challenges back then was the limited volume of available data. Decisions were often made based on incomplete or insufficient information. However, as technology has advanced, the amount of data generated has grown exponentially. Today, data is constantly being produced, with volumes increasing every second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*uPV3zqnFYHd6QpmT" /><figcaption>Image by istock</figcaption></figure><p>In fact, studies show that in the past two years alone, we’ve created as much data as was generated throughout the entirety of human history. This surge in data availability has transformed how businesses operate, enabling data-driven decision-making, faster responses, and deeper insights that were unimaginable just a couple of decades ago.</p><h3><strong>Data Analysis And Decision Making</strong></h3><p><strong>Data analysis</strong> is the process of cleaning, transforming, and interpreting raw data to extract valuable insights. These insights can help businesses understand trends, identify opportunities, and make decisions based on evidence rather than intuition. By converting data into actionable knowledge, organizations can validate hypotheses, optimize strategies, and predict future outcomes with greater confidence.</p><h4>Example: Data Analysis in Retail</h4><p>Imagine a large retail chain looking to optimize its inventory. Without proper analysis, the company might either overstock items, resulting in waste, or understock, leading to missed sales opportunities. Through <strong>data analysis</strong>, the company can examine historical sales data, customer behavior, seasonal trends, and supply chain efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*iiJPaJiW1b_VXjQQ.jpg" /><figcaption>Image By SelectHub</figcaption></figure><p>For instance, by analyzing purchasing patterns from previous years, the company might discover that a certain product sells out quickly during the holiday season in specific regions. They could also identify that another product tends to move slower and ties up capital. With this information, the retailer can <strong>forecast demand</strong>, optimize stock levels, and ensure products are available where and when they’re needed. This not only improves customer satisfaction but also reduces costs associated with overstocking or lost sales.</p><h3>Beyond Technical Skills: The Importance of Soft Skills for Data Analysts</h3><p>Many beginners in data analysis assume that mastering technical skills is the key to becoming a successful data analyst. While technical proficiency is crucial, it represents only about <strong>50% of the equation</strong>. The real challenge begins once the technical work is done—when you need to communicate your findings and insights effectively.</p><p>Let&#39;s say you&#39;ve built a dashboard, created a plot, or completed a complex analysis. <strong>Then what?</strong> Are you able to explain what the data reveals? Can you clearly communicate how your findings impact business decisions? This is where soft skills, particularly communication, come into play.</p><h3>Why Communication Skills Matter</h3><p><strong>Technical work</strong>—such as writing SQL queries, coding in Python, or building dashboards—is important, but <strong>being able to interpret and convey the story behind the data</strong> is what truly makes a great data analyst. The ability to communicate results in a clear, concise, and actionable way is often the difference between a good analyst and a great one.</p><h4>Imagine This Scenario:</h4><p>You&#39;ve spent hours analyzing customer data and created a dashboard showing various customer behaviors. However, if you can&#39;t <strong>effectively present</strong> your findings to the marketing team, your analysis loses value. The marketing team needs to understand:</p><ol><li><strong>What the data shows</strong> – Are there actionable trends?</li><li><strong>Why it matters</strong> – How does this affect their strategy?</li><li><strong>What actions to take</strong> – What decisions can be made based on your insights?</li></ol><h3>How to Improve Your Communication Skills as a Data Analyst</h3><ol><li><strong>Understand Your Audience: </strong>Tailor your communication based on who you’re presenting to. For example, executives may want high-level insights, while your technical team might want to dig into the details.</li><li><strong>Storytelling with Data: </strong>It’s not just about numbers or charts. Learn to weave a <strong>narrative</strong> around your analysis—what’s the problem you’re solving, what insights have you uncovered, and how should the business respond?</li><li><strong>Visualization Matters: </strong>Clear and intuitive data visualizations help communicate complex insights. A well-designed chart can often explain a trend far better than words alone.</li><li><strong>Presenting Findings: </strong>Whether you&#39;re presenting to a small team or an entire organization, practice explaining your analysis in <strong>simple, non-technical terms</strong>. Focus on <strong>why</strong> the insights matter and what should be done next.</li><li><strong>Collaboration and Active Listening: </strong>Engage with stakeholders throughout the process. Don’t just throw data at them—<strong>ask questions</strong>, understand their needs, and listen to their feedback. This helps in aligning your analysis with business objectives.</li></ol><h3>Tools For Data Analysis</h3><p><strong>Excel</strong></p><ul><li><strong>Usage</strong>: Widely used for data manipulation, analysis, and visualization. While it may seem basic, Excel is a powerful tool for handling smaller datasets, building pivot tables, and creating quick visualizations.</li><li><strong>Importance</strong>: It’s often the starting point for many analysts and remains crucial for rapid, ad-hoc analysis.</li></ul><p><strong>SQL</strong></p><ul><li><strong>Usage</strong>: SQL (Structured Query Language) is the backbone of data retrieval from relational databases. It allows analysts to interact directly with large databases, filter data, join tables, and aggregate results.</li><li><strong>Importance</strong>: Knowing SQL is a must, as it provides direct access to structured data in enterprise databases.</li></ul><p><strong>Python / R</strong></p><ul><li><strong>Usage</strong>: Both Python and R are programming languages widely used for data analysis, manipulation, statistical computing, and machine learning. Libraries like Pandas, NumPy (Python) and dplyr, ggplot2 (R) are essential for data handling and visualization.</li><li><strong>Importance</strong>: Mastering either Python or R allows analysts to perform complex analysis, build models, and automate tasks that would be difficult to do in Excel.</li></ul><p><strong>Tableau / Power BI</strong></p><ul><li><strong>Usage</strong>: These are popular data visualization tools that help analysts create interactive, dynamic dashboards and reports. They enable the exploration of large datasets visually and make insights accessible to non-technical stakeholders.</li><li><strong>Importance</strong>: Communicating insights effectively is just as important as finding them, and these tools excel in presenting complex data in a digestible format.</li></ul><p><strong>Jupyter Notebooks / Google Colab</strong></p><ul><li><strong>Usage</strong>: Jupyter Notebooks provide a user-friendly interface for writing Python code and visualizing results. It’s widely used for exploratory data analysis (EDA) and documenting the analysis process.</li><li><strong>Importance</strong>: Analysts use Jupyter for interactive coding, visualization, and sharing results in a collaborative environment.</li></ul><p><strong>Big Data Tools (Hadoop, Spark)</strong></p><ul><li><strong>Usage</strong>: As data volumes grow, analysts increasingly need to handle large datasets. Hadoop and Spark are popular frameworks for processing large-scale data in distributed environments.</li><li><strong>Importance</strong>: Knowing how to work with big data platforms allows analysts to manage and analyze datasets that are too large for traditional tools.</li></ul><h3>Conclusion</h3><p>Now a days, Large Language Models and Gen AI are the Race, If you stop at data analysis then eventually you will lose the race. Explore more, adapt new skills, deep dives into big data, how to build data pipelines, how to build models, understand the trend of Gen AI and Prompt Engineering.</p><p>I will be doing some practical blogs on data analysis and big data tools, till then keep learning and keep exploring!</p><p>#Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22f0c6b7eb95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieval Augmented Generation (RAG)]]></title>
            <link>https://medium.com/@abdulwasiueunk/retrieval-augmented-generation-rag-6bb8f7ab2f3c?source=rss-13b81e473468------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bb8f7ab2f3c</guid>
            <category><![CDATA[explore]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[prompt]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Abdul Wasiue]]></dc:creator>
            <pubDate>Sat, 24 Aug 2024 04:08:26 GMT</pubDate>
            <atom:updated>2024-08-24T04:10:25.906Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s rapidly growing digital age, the sheer volume of information available to users is overwhelming. Whether it’s academic research, technical documentation, or even entertainment content, the challenge lies not just in accessing this information but in retrieving the most relevant and accurate data quickly and efficiently. Traditional search engines and retrieval systems often fall short in providing precise answers, especially for complex queries that require a deep understanding of context or specialized knowledge.</p><p>This limitation is particularly pronounced in fields where up-to-date information is crucial, such as healthcare, legal studies, or scientific research. Users often face difficulties in finding the exact piece of information they need, leading to inefficient workflows, increased time spent on research, and even the risk of missing critical data. The problem is compounded by the fact that most retrieval systems are static, lacking the ability to adapt to the evolving nature of queries and the specific needs of individual users.</p><p>To address these challenges, a new paradigm known as Retrieval-Augmented Generation (RAG) is emerging. RAG models combine the strengths of retrieval-based systems with the capabilities of generative models to create a more dynamic and context-aware retrieval process. Unlike conventional systems that rely solely on pre-indexed databases, RAG models can retrieve information from vast datasets and generate responses that are tailored to the specific needs of the user, providing a more nuanced and insightful answer to complex queries.</p><h3>Introduction</h3><p>RAG models represents a significant advancement in the field of information retrieval. By integrating retrieval mechanisms with generative models, these systems can access a broader range of data sources, understand the context of the query more deeply, and produce responses that are both informative and actionable. This not only enhances the accuracy of the information retrieved but also improves the user experience by making the interaction more intuitive and effective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*mwKL3QSJMjVDEldG.png" /><figcaption>Taken from Hyperight</figcaption></figure><p>This diagram illustrates the process of how a Retrieval-Augmented Generation (RAG) model works to generate responses using both a retrieval system and a large language model (LLM).</p><p><strong>User Prompt</strong>: The process starts when a user submits a query or a prompt through their device, like a computer. This could be any question or request for information.</p><p><strong>Retrieval System Initiation</strong>: The prompt is sent to a retrieval system. This system’s job is to find the most relevant information related to the user’s query.</p><p><strong>Semantic Search</strong>: The retrieval system performs a “semantic search” in a large repository of documents. Semantic search means that it looks for documents not just based on exact keyword matches but also on the meaning and context of the query.</p><p><strong>Relevant Documents</strong>: Based on the search, the retrieval system pulls out documents or pieces of information that it considers most relevant to the user’s query.</p><p><strong>Contextual Integration</strong>: The retrieval system then combines the user’s original prompt with the relevant information it found. This combination is enriched with context, making it more comprehensive and ready for deeper analysis.</p><p><strong>Large Language Model (LLM)</strong>: The enriched prompt is sent to a large language model (like GPT). The LLM processes this information, understanding the context and the query deeply, and generates a response.</p><p><strong>Generated Response</strong>: Finally, the generated response is sent back to the user, providing them with a precise and contextually appropriate answer to their query.</p><p>As the demand for more sophisticated information retrieval systems continues to grow, the adoption of RAG models is poised to transform how users interact with data. This introduction to Retrieval-Augmented Generation sets the stage for exploring how these models work, their potential applications, and the impact they can have on various industries.</p><h3><strong>How RAG Models Differ from Traditional Retrieval Systems:</strong></h3><p>Traditional search engines and retrieval systems, which have been the backbone of information access for decades, operate primarily by indexing large amounts of data and returning results based on keyword matching and ranking algorithms. While these systems have served well for basic information retrieval, they exhibit several limitations when faced with more complex or nuanced queries:</p><p><strong>Keyword Dependency:</strong> Traditional systems rely heavily on keyword matching, which can lead to irrelevant or incomplete results if the query terms do not exactly match the indexed content. This often results in users having to sift through a large amount of data to find what they need.</p><p><strong>Contextual Insensitivity:</strong> These systems typically lack a deep understanding of context. For instance, they may struggle to distinguish between different meanings of the same word or to understand the nuanced requirements of a query. This leads to results that are often too broad or unrelated to the specific intent of the user.</p><p><strong>Static Responses:</strong> Traditional retrieval systems are static, meaning they return the same results for the same query regardless of the context or user’s intent. They do not adapt or refine their responses based on the evolving needs or the specific context in which the query is made.</p><p><strong>Limited Understanding of Complex Queries:</strong> When users pose complex, multi-faceted questions, traditional systems often fall short. They are not designed to synthesize information from multiple sources or to generate comprehensive answers that address the full scope of the query.</p><p><strong>Difficulty in Handling Large Volumes of Data: </strong>As the amount of available data continues to grow exponentially, traditional systems struggle to efficiently manage and retrieve the most relevant information. This can result in slower response times and less relevant results.</p><h3><strong>Advantages of Integrating Retrieval with Generation</strong></h3><p>Retrieval-Augmented Generation (RAG) models address many of these limitations by combining the strengths of both retrieval-based and generative approaches. This integration offers several key advantages:</p><p><strong>Improved Relevance:</strong> RAG models begin by retrieving relevant documents or data from a vast corpus, ensuring that the generated responses are grounded in accurate and pertinent information. This retrieval step filters out irrelevant data, significantly enhancing the relevance of the final output.</p><p><strong>Contextual Understanding:</strong> Unlike traditional systems, RAG models incorporate contextual information to better understand the intent behind a query. The generative component uses this context to create responses that are more aligned with the user’s needs, addressing not just what was asked but also the underlying purpose of the query.</p><p><strong>Dynamic and Adaptive Responses:</strong> RAG models are not static; they dynamically generate responses based on both the retrieved information and the specific context of the query. This adaptability ensures that the responses are more precise and tailored to the individual user’s needs.</p><p><strong>Handling of Complex Queries:</strong> RAG models excel at synthesizing information from multiple sources to answer complex questions. By combining retrieval with generation, these models can generate comprehensive, well-rounded responses that address all aspects of a query, even when it spans multiple topics or requires an in-depth understanding.</p><p><strong>Scalability:</strong> The retrieval component of RAG models is designed to efficiently handle large volumes of data, quickly identifying the most relevant pieces of information. This scalability is crucial in today’s data-rich environments, where the ability to quickly retrieve and process information is key to providing timely and accurate responses.</p><h3>Conclusion</h3><p>As data continues to grow in both volume and complexity, the adoption of RAG models is poised to transform how we interact with information. By offering a more nuanced understanding of queries and delivering more relevant results, RAG models not only improve the efficiency of information retrieval but also enhance the overall user experience. In a world where information is power, RAG models represent the future of how we access, interpret, and utilize data, paving the way for more informed decision-making across various domains.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bb8f7ab2f3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark: A Beginner Guide Overview]]></title>
            <link>https://medium.com/@abdulwasiueunk/apache-spark-a-rescue-hero-c6404ae185b6?source=rss-13b81e473468------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6404ae185b6</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[spark-mllib]]></category>
            <dc:creator><![CDATA[Abdul Wasiue]]></dc:creator>
            <pubDate>Thu, 15 Aug 2024 19:26:06 GMT</pubDate>
            <atom:updated>2024-08-24T03:51:36.577Z</atom:updated>
            <content:encoded><![CDATA[<p>Companies are drowning in data from social media, smart devices, and sales. Old systems can’t handle it fast enough. They’re slow, complicated, and use too much computer power. This causes delays and mistakes. Companies need to make quick decisions based on data, but if they can’t, they’ll miss out and fall behind.</p><p>In today’s fast-paced business world, speed and accuracy are crucial. Companies need to react quickly to changes in the market, customer behavior, and other trends. But with old systems, they’re stuck in the slow lane. They need a better way to process and analyze data, so they can make informed decisions and stay ahead of the competition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*GBwojVUns9mfDcz1.jpg" /></figure><p>Apache Spark addresses the limitations of traditional data processing frameworks by offering a unified platform that supports both batch and real-time data processing. Spark’s in-memory computing capabilities allow for faster data processing by reducing the need for disk I/O, which is a major bottleneck in traditional systems. Additionally, Spark’s distributed computing framework enables the parallel processing of large datasets across multiple nodes, thus significantly improving processing speed and scalability.</p><p>These are some real-world scenarios of companies that use Apache Spark to solve various problems:</p><p><strong>Uber: </strong>Uses Spark for real-time data processing and analytics to optimize ride-sharing services, predict demand, and improve customer experience.</p><p><strong>Netflix:</strong> Employs Spark for data processing, machine learning, and analytics to personalize recommendations, predict user behavior, and optimize content delivery.</p><p><strong>LinkedIn:</strong> Utilizes Spark for data analytics, machine learning, and natural language processing to improve user experience, recommend connections, and optimize advertising.</p><p><strong>Amazon:</strong> Uses Spark for data processing, analytics, and machine learning to optimize supply chain management, predict customer behavior, and improve recommendation systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*al_zyw9N4WGU2zak.png" /></figure><p>Spark’s versatility extends beyond simple data processing; it provides built-in libraries for machine learning, graph processing, and streaming, making it a comprehensive solution for a wide range of data analytics tasks. Organizations can leverage Spark’s robust ecosystem to build scalable data pipelines, perform advanced analytics, and derive insights in real-time, all while minimizing infrastructure costs and complexities.</p><h3>Introduction to Apache Spark Libraries</h3><p>Apache Spark’s libraries are designed to extend its core functionalities and simplify the development of complex data processing and analytics applications. There is alot but I will discuss some of them</p><p><strong>Spark SQL</strong>:</p><p><strong>Overview</strong>: Spark SQL is a powerful module that enables structured data processing using SQL queries. It supports reading data from various sources such as JSON, Parquet, ORC, Hive, and JDBC.</p><p><strong>DataFrame API</strong>: Provides a high-level abstraction for working with structured data, offering performance optimizations and support for various data formats.</p><p><strong>Integration with Hive</strong>: Spark SQL can integrate with Apache Hive, enabling users to query Hive tables directly from Spark.</p><p><strong>Catalyst Optimizer</strong>: An advanced query optimization engine that improves the execution of SQL queries by generating optimized execution plans.</p><p><strong>MLlib</strong>:</p><p><strong>Overview</strong>: MLlib is Spark’s scalable machine learning library, providing a range of algorithms and utilities for building machine learning models.</p><p><strong>Algorithms</strong>: Includes algorithms for classification, regression, clustering, collaborative filtering, and more.</p><p><strong>Pipelines</strong>: Supports the creation of machine learning pipelines, allowing for the seamless integration of data preprocessing, model training, and evaluation.</p><p><strong>Distributed Computing</strong>: MLlib’s algorithms are designed to scale out across a cluster, enabling the processing of large datasets in parallel.</p><p><strong>Spark Streaming</strong>:</p><p><strong>Overview</strong>: Spark Streaming enables real-time data processing, allowing users to process and analyze data as it arrives, rather than waiting for batch processing.</p><p><strong>DStream (Discretized Stream)</strong>: The basic abstraction in Spark Streaming, representing a continuous stream of data as a series of RDDs.</p><p><strong>Windowed Operations</strong>: Supports operations over sliding windows of data, enabling complex event processing and time-based aggregations.</p><p><strong>Integration with Various Sources</strong>: Spark Streaming can consume data from sources like Kafka, Flume, HDFS, and more, providing flexibility in building streaming applications.</p><p><strong>SparkR</strong>:</p><p><strong>Overview</strong>: SparkR is an R package that provides a frontend for using Apache Spark with R, allowing R users to perform large-scale data analysis using Spark’s distributed computing capabilities.</p><p><strong>DataFrame API</strong>: Similar to the DataFrame API in Spark SQL, SparkR provides an easy-to-use API for working with structured data.</p><p><strong>Machine Learning</strong>: Supports the integration of MLlib, allowing R users to build and evaluate machine learning models on large datasets.</p><p><strong>Interoperability</strong>: SparkR allows the seamless combination of Spark and R code, enabling the use of Spark’s computational power within R environments.</p><h3><strong>Spark Core:</strong></h3><p><strong>Spark Core</strong> is the foundation of Apache Spark, providing the basic functionalities that make Spark an efficient and powerful tool for large-scale data processing.</p><p>At the heart of Spark Core is the <strong>Resilient Distributed Dataset (RDD)</strong>. An RDD is a collection of data that is distributed across multiple machines in a cluster. RDDs are immutable, meaning once they are created, they cannot be changed. This immutability helps Spark recover from failures because it can always recreate the RDD from its original source. RDDs can be transformed into new RDDs using operations like map or filter, but these transformations are not executed until an action like count or collect is performed. This allows Spark to optimize the entire computation process.</p><p>A key feature of Spark Core is <strong>in-memory computing</strong>. Spark can store data in the memory of the cluster’s machines, which speeds up processing, especially for tasks that involve repeated operations on the same data. By avoiding the need to read from and write to disk repeatedly, Spark significantly boosts performance for many types of data processing tasks.</p><p>Spark also provides <strong>fault tolerance</strong>. If a part of the data is lost due to a failure in one of the machines, Spark can automatically rebuild the lost data using the transformations recorded in the RDD’s lineage. This ensures that data processing continues smoothly even in the face of hardware failures.</p><p>The <strong>task scheduling and distribution</strong> feature of Spark Core is responsible for managing how work is divided and executed across the cluster. Spark creates a Directed Acyclic Graph (DAG) that represents the flow of data and tasks. This DAG helps Spark schedule tasks in the most efficient way, making sure that the resources of the cluster are used effectively.</p><p>Finally, <strong>Spark Core</strong> provides APIs in several programming languages, including Java, Scala, Python, and R. This makes Spark accessible to a wide range of developers, allowing them to process large datasets using the language they are most comfortable with.</p><p>In essence, Spark Core is what enables Apache Spark to process large amounts of data quickly and reliably across distributed systems. Its features, like RDDs, in-memory computing, fault tolerance, and efficient task scheduling, make it a robust choice for big data processing.</p><p>I will be focusing on MLlib in this blog. Lets do some practical work in python. I am using Simple Linear Regression on Employee Dataset which I have generated.</p><pre>from pyspark.sql import SparkSession<br>from pyspark.ml.feature import VectorAssembler<br>from pyspark.ml.regression import LinearRegression<br>from pyspark.ml.evaluation import RegressionEvaluator</pre><p>We have imported the necessary libraries from pyspark. Vector Assembler is used to create vector of feature which will be used to get the target variable. If we notice it is different from Traditional approach where we do things manually, in spark we can do in single line.</p><pre>Initialize Spark Session<br>spark = SparkSession.builder.appName(&quot;LinearRegressionExample&quot;).getOrCreate()<br><br>data = spark.read.csv(&quot;emp.csv&quot;, header=True, inferSchema=True)<br><br>assembler = VectorAssembler(inputCols=[&quot;Age&quot;, &quot;Experience&quot;], outputCol=&quot;features&quot;)<br>data = assembler.transform(data)</pre><p>First We initialized the session and named it(whatever you do). Second we load the data. In last two lines we used vectorization of feature variable on which our target variable depend. If we notice it is different from Traditional approach where we do things manually, in spark we can do in single line.</p><pre>final_data = data.select(&quot;features&quot;, &quot;Salary&quot;)<br><br>train_data, test_data = final_data.randomSplit([0.8, 0.2])<br><br>lr = LinearRegression(labelCol=&quot;Salary&quot;)<br><br>lr_model = lr.fit(train_data)</pre><p>The code first selects the relevant columns, `features` and `Salary`, from the dataset to create a new DataFrame called `final_data`. The data is then split into two sets: `train_data`, which contains 80% of the data, and `test_data`, which contains the remaining 20%. This split allows the model to be trained on one portion of the data and tested on another. The linear regression model is then initialized with the label set to `Salary`, meaning the model will predict salary values based on the input features. Finally, the model is trained using the `fit` method on the `train_data`, allowing it to learn the relationships between the features and the target variable, `Salary`.</p><pre>predictions = lr_model.transform(test_data)<br>evaluator = RegressionEvaluator(labelCol=&quot;Salary&quot;, predictionCol=&quot;prediction&quot;, metricName=&quot;rmse&quot;)<br>rmse = evaluator.evaluate(predictions)<br>print(f&quot;Root Mean Squared Error (RMSE): {rmse}&quot;)<br><br>print(f&quot;Coefficients: {lr_model.coefficients}&quot;)<br>print(f&quot;Intercept: {lr_model.intercept}&quot;)</pre><p>Root Mean Squared Error (RMSE): 5100.0380743272<br>Coefficients: [334.97433478168324,1780.7997836512889]<br>Intercept: 15167.403641264884</p><p>Oops!!, That’s terrible results because our model is not trained well, data is not balance and enough. Anyways, Our Goal was to Utilize the Power of Apache Spark.</p><h3><strong>Conclusion:</strong></h3><p>It was the simple overview of Apache Spark and MLLib. You can further paly around and learn by applying different techniques apart from Lineat Regression. MLlib supports various techniques of Machine Learning.</p><p>Thank You :)</p><p>Github: <a href="https://github.com/Wasiue03/Apache-Spark">https://github.com/Wasiue03/Apache-Spark</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6404ae185b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Kafka: Real Time Streaming using Docker]]></title>
            <link>https://medium.com/@abdulwasiueunk/apache-kafka-real-time-streaming-using-docker-ff3c6d7eef9d?source=rss-13b81e473468------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff3c6d7eef9d</guid>
            <category><![CDATA[real-time-communication]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[apache-kafka-streams]]></category>
            <dc:creator><![CDATA[Abdul Wasiue]]></dc:creator>
            <pubDate>Fri, 26 Jul 2024 14:03:38 GMT</pubDate>
            <atom:updated>2024-07-26T14:06:43.456Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Kafka is an open-source streaming data platform developed by LinkedIn. As it expanded Kafka’s capabilities, LinkedIn donated it to Apache for further development.</p><p>Kafka operates like a traditional pub-sub message queue, such as RabbitMQ, in that it enables you to publish and subscribe to streams of messages. But it differs from traditional message queues in 3 key ways:</p><ol><li>Kafka operates as a modern distributed system that runs as a cluster and can scale to handle any number of applications.</li><li>Kafka is designed to serve as a storage system and can store data as long as necessary; most message queues remove messages immediately after the consumer confirms receipt.</li><li>Kafka handles stream processing, computing derived streams and datasets dynamically, rather than just passing batches of messages.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OWBfmnNkxcWSl3slxg7_ug.png" /><figcaption>Kafka Diagram</figcaption></figure><h3>What is Kafka used for?</h3><p>Kafka is a stream processing system used for messaging, website activity tracking, metrics collection and monitoring, logging, event sourcing, commit logs, and real-time analytics. It’s a good fit for large scale message processing applications since it is more robust, reliable, and fault-tolerant compared to traditional message queues.</p><p>Open-source Kafka or managed distributions are ubiquitous in modern software development environments. Kafka is used by developers and data engineers at companies such as Uber, Square, Strave, Shopify, and Spotify.</p><h4>1. Real-Time Processing</h4><p>Kafka is designed for low-latency data ingestion and processing. It supports real-time streaming, allowing organizations to make data-driven decisions quickly.</p><h4>2. Scalability</h4><p>Kafka’s distributed architecture allows it to scale horizontally. You can increase the capacity of your Kafka cluster by adding more brokers or partitions, handling increased data volumes effortlessly.</p><h4>3. Durability and Reliability</h4><p>Kafka ensures data durability through replication. Data is replicated across multiple brokers, which provides fault tolerance and high availability. In case of a broker failure, data can be recovered from other replicas.</p><h4>4. Fault Tolerance</h4><p>Kafka’s design supports fault tolerance. If a broker fails, Kafka can continue to operate with minimal disruption due to its replication and partitioning mechanisms.</p><h4>5. Integration</h4><p>Kafka integrates seamlessly with various data processing frameworks such as Apache Spark, Apache Flink, and Apache Storm. It also supports connectors for numerous data sources and sinks, making it a versatile tool in the data ecosystem.</p><h3>Core Components of Apache Kafka</h3><h4>1. Producer</h4><p>The Producer is responsible for sending records to Kafka topics. Producers push data into Kafka, typically as messages or events. Each record sent by a producer is tagged with a key, value, and timestamp. Producers are designed to handle high throughput and can send data to multiple partitions in parallel.</p><p><strong>Why use Producers?</strong></p><ul><li><strong>Scalability:</strong> Producers can be scaled horizontally to handle increased data volumes.</li><li><strong>Asynchronous Processing:</strong> Producers can send messages asynchronously to improve performance and throughput.</li></ul><h4>2. Consumer</h4><p>Consumers read records from Kafka topics. They subscribe to one or more topics and process the records in real time. Consumers work in groups, and each consumer within a group processes a portion of the data, allowing for parallel processing and load balancing.</p><p><strong>Why use Consumers?</strong></p><ul><li><strong>Load Balancing:</strong> Consumers within a group share the workload, enabling efficient data processing.</li><li><strong>Fault Tolerance:</strong> If one consumer fails, others in the group can take over, ensuring uninterrupted data processing.</li></ul><h4>3. Broker</h4><p>A Kafka cluster is composed of one or more brokers. Each broker is a server that stores data and serves client requests. Brokers manage the data replication and ensure data durability. They handle data partitioning and coordinate the distribution of data among producers and consumers.</p><p><strong>Why use Brokers?</strong></p><ul><li><strong>Scalability:</strong> Brokers can be added to a cluster to scale horizontally.</li><li><strong>Data Durability:</strong> Data is replicated across brokers to ensure durability and high availability.</li></ul><h4>4. Topic</h4><p>Topics are logical channels to which records are sent by producers and from which records are read by consumers. Each topic is divided into partitions, allowing Kafka to parallelize data processing and storage. Topics provide a way to organize data and manage data flow.</p><p><strong>Why use Topics?</strong></p><ul><li><strong>Organization:</strong> Topics help in categorizing data based on business needs.</li><li><strong>Parallelism:</strong> Data is split into partitions, enabling parallel processing and increasing throughput.</li></ul><h4>5. Partition</h4><p>Partitions are the fundamental unit of parallelism in Kafka. Each topic is divided into multiple partitions, which can be distributed across different brokers. Partitions allow Kafka to handle large volumes of data and provide high throughput by parallelizing data read and write operations.</p><p><strong>Why use Partitions?</strong></p><ul><li><strong>Parallel Processing:</strong> Partitions enable concurrent data processing, improving performance.</li><li><strong>Scalability:</strong> Adding partitions can help in scaling the system to handle more data.</li></ul><h4>6. Zookeeper</h4><p>Apache ZooKeeper is a distributed coordination service used by Kafka for managing and coordinating brokers. It handles tasks such as leader election for partitions, configuration management, and synchronization between brokers. ZooKeeper ensures that Kafka brokers are in sync and that the data is replicated correctly.</p><p><strong>Why use Zookeeper?</strong></p><ul><li><strong>Coordination:</strong> ZooKeeper helps in coordinating distributed systems and managing metadata.</li><li><strong>Fault Tolerance:</strong> It provides a mechanism for failover and recovery in the Kafka cluster.</li></ul><p>Simple Kafka Setup on local machine using docker container. For that you should install docker and create docker compose yaml file which will run the container and pull image from docker hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*7XZ6brwU0AerzmcWbHJAtg.jpeg" /><figcaption>Simple Design for Learning Purpose</figcaption></figure><pre>version: &#39;3.8&#39;<br>services:<br>  zookeeper:<br>    image: wurstmeister/zookeeper:latest<br>    ports:<br>      - &quot;2181:2181&quot;<br><br><br>  kafka:<br>    image: wurstmeister/kafka:latest<br>    ports:<br>      - &quot;9092:9092&quot;<br>    environment:<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092<br>      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br>    depends_on:<br>     - zookeeper</pre><p>This yaml will pull the images for zoojeeper and kafka, running on local host.</p><pre> redis:<br>    image: redis:7.2.5<br>    ports:<br>      - &quot;6379:6379&quot;</pre><p>To add redis, we specify the image along version and ports. My image is redis:7.2.5 and ports specified 6379:6379. We dont have to create another yaml file, rather add redis part in existing yaml.</p><pre>version: &#39;3.8&#39;<br>services:<br>  zookeeper:<br>    image: wurstmeister/zookeeper:latest<br>    ports:<br>      - &quot;2181:2181&quot;<br><br>  kafka:<br>    image: wurstmeister/kafka:latest<br>    ports:<br>      - &quot;9092:9092&quot;<br>    environment:<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092<br>      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br>    depends_on:<br>      - zookeeper<br><br>  redis:<br>    image: redis:7.2.5<br>    ports:<br>      - &quot;6379:6379&quot;</pre><p>Now lets create python file which will retrive data from exteranl api which then will produced by Producer</p><p><strong>Producer.py</strong></p><pre><br>class APIProducer:<br>    def __init__(self, api_url, kafka_topics, bootstrap_servers=&#39;localhost:9092&#39;):<br>        self.api_url = api_url<br>        self.kafka_topics = kafka_topics<br>        self.producer = KafkaProducer(<br>            bootstrap_servers=bootstrap_servers,<br>            value_serializer=lambda v: json.dumps(v).encode(&#39;utf-8&#39;)<br>        )<br><br>    def fetch_and_send(self):<br>        response = requests.get(self.api_url)<br>        response.raise_for_status()<br>        data = response.json()<br>        for item in data:<br>            for topic in self.kafka_topics:<br>                self.producer.send(topic, value=item)<br>                print(f&quot;Sent to {topic}: {item}&quot;)<br>                time.sleep(1)  # Simulate delay<br><br>if __name__ == &quot;__main__&quot;:<br>    api_url = &quot;https://jsonplaceholder.typicode.com/todos&quot;<br>    kafka_topics = [&quot;topic1&quot;, &quot;topic2&quot;, &quot;topic3&quot;]<br>    producer = APIProducer(api_url, kafka_topics)<br>    producer.fetch_and_send()</pre><p>APIProcuer class is resposible for parsing the url of api, number of topics and server connection.</p><p><strong>Kafka Producer Setup: </strong>Inside the __init__ method, a Kafka producer is created using the KafkaProducer class. The producer is configured to connect to the Kafka server and serialize the data to JSON format before sending.</p><p><strong>Fetching and Sending Data: </strong>The fetch_and_send method is responsible for:</p><ul><li>Fetching data from the API using the requests.get method.</li><li>Raising an exception if the API request fails.</li><li>Parsing the JSON response from the API.</li><li>Iterating over the fetched data and sending each item to all specified Kafka topics.</li><li>Printing a confirmation message for each sent item.</li><li>Introducing a 1-second delay between sending items to simulate a real-time data flow.</li></ul><p><strong>Consumser.py</strong></p><pre>from kafka import KafkaConsumer<br>import redis<br>import json<br><br>class ETLConsumer:<br>    def __init__(self, kafka_topics, redis_host=&#39;localhost&#39;, redis_port=6379, bootstrap_servers=&#39;localhost:9092&#39;):<br>        try:<br>            self.kafka_topics = kafka_topics<br>            self.consumer = KafkaConsumer(<br>                *kafka_topics,  <br>                bootstrap_servers=bootstrap_servers,<br>                value_deserializer=lambda x: json.loads(x.decode(&#39;utf-8&#39;))<br>            )<br>            self.redis_client = redis.StrictRedis(host=redis_host, port=redis_port, decode_responses=True)<br>        except Exception as e:<br>            print(f&quot;Error initializing Kafka consumer or Redis client: {e}&quot;)<br>            raise<br><br>    def transform_and_store(self):<br>        try:<br>            for message in self.consumer:<br>                data = message.value<br>                data[&#39;status&#39;] = &#39;active&#39; if data.get(&#39;completed&#39;) else &#39;inactive&#39;<br>                key = f&quot;task:{data[&#39;id&#39;]}&quot;<br>                self.redis_client.set(key, json.dumps(data))<br>                print(f&quot;Stored: {data}&quot;)<br>        except Exception as e:<br>            print(f&quot;Error processing Kafka message or storing data in Redis: {e}&quot;)<br>         <br><br>if __name__ == &quot;__main__&quot;:<br>    kafka_topics = [&quot;topic1&quot;]<br>    print(&quot;Starting Kafka consumer...&quot;)<br>    consumer = ETLConsumer(kafka_topics)<br>    print(&quot;Kafka consumer initialized successfully.&quot;)<br>    try:<br>        print(&quot;Starting message processing...&quot;)<br>        consumer.transform_and_store()<br>    except KeyboardInterrupt:<br>        print(&quot;KeyboardInterrupt: Stopping Kafka consumer.&quot;)<br>    except Exception as e:<br>        print(f&quot;An error occurred: {e}&quot;)</pre><p>The ETLConsumer class is responsible for initializing the Kafka consumer and the Redis client. The __init__ method sets up the connection parameters for both Kafka and Redis.</p><p>The kafka_topics parameter is a list of Kafka topics to subscribe to. The redis_host and redis_port parameters are the host and port for the Redis server, defaulting to localhost and 6379 respectively. The bootstrap_servers parameter is the address of the Kafka server, defaulting to localhost:9092. The KafkaConsumer is configured to deserialize the JSON messages from Kafka, and the redis.StrictRedis client is set up to interact with the Redis server.</p><h4>Transforming and Storing Data</h4><p>The transform_and_store method is responsible for reading messages from the Kafka topics, transforming the data, and storing it in Redis. The method iterates over the messages received from the Kafka topics. Each message is deserialized into a Python dictionary (data). A new field, status, is added to the data based on the value of the completed field. If completed is True, status is set to &#39;active&#39;. Otherwise, status is set to &#39;inactive&#39;. The data is then stored in Redis with a key formatted as task:{id}, where {id} is the unique identifier of the task. The data is serialized back to JSON before being stored in Redis.</p><p>We have run both Python files, first producer.py`which will fetch data from api and then consumer.py using the kafka server, it will send data to consumer in real time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4AfFpk0-CXO2mqKyy_91fA.png" /><figcaption>Real Time Streaming</figcaption></figure><p>We have used redis to store our streaming data using docker conainer. In powershell, execute docker exec -it kafka-redis-1 redis-cli it will execute the cli then check for the keys that will show the tasks or messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*v-UsFDuS0Ib5QuDYYai2bA.png" /><figcaption>Redis Data Store</figcaption></figure><p><strong>Conclusion</strong></p><p>In this article, we’ve explored the architecture and implementation of a data streaming pipeline using Apache Kafka. We began by understanding the fundamental components of Kafka, including producers, consumers, brokers, topics, partitions, and ZooKeeper. Each component plays a crucial role in ensuring the efficient handling of data streams, from ingestion to processing and storage.</p><p>The provided example illustrated a practical setup where an external API serves as the data source. A producer fetches data from this API and sends it to Kafka topics. Kafka acts as the central message broker, efficiently distributing data to consumers. ZooKeeper manages the Kafka brokers, ensuring high availability and fault tolerance. The consumer reads the streamed data, processes it, and stores it in Redis, an in-memory data store, enabling fast and reliable data access.</p><p>By leveraging Docker for containerization, the setup ensures a consistent and reproducible environment, simplifying the deployment and scaling of the Kafka infrastructure. This architecture showcases Kafka’s strengths in real-time data processing, scalability, and reliability, making it an excellent choice for building robust data streaming applications.</p><p>Whether you are tracking website activity, collecting metrics, monitoring logs, or performing real-time analytics, Apache Kafka provides a versatile and powerful platform to handle large-scale data processing needs. The example pipeline serves as a foundational blueprint that can be extended and customized to meet specific use cases and business requirements.</p><p>Thanks :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff3c6d7eef9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Language Translation Using Neural Networks (Understand the math)]]></title>
            <link>https://medium.com/@abdulwasiueunk/language-translation-using-neural-networks-understand-the-math-b01493163187?source=rss-13b81e473468------2</link>
            <guid isPermaLink="false">https://medium.com/p/b01493163187</guid>
            <category><![CDATA[neural-network-training]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[seq2seq]]></category>
            <category><![CDATA[encoder-decoder]]></category>
            <category><![CDATA[machine-translation]]></category>
            <dc:creator><![CDATA[Abdul Wasiue]]></dc:creator>
            <pubDate>Tue, 09 Jul 2024 08:26:53 GMT</pubDate>
            <atom:updated>2024-07-26T14:11:15.476Z</atom:updated>
            <content:encoded><![CDATA[<h3>Language Translation Using Neural Networks: English and Urdu</h3><p>In a world where borders are increasingly becoming blurred and communication knows no bounds, the role of language translation has become more critical than ever. As we navigate the vast landscape of global connectivity, the demand for seamless and accurate translation has given rise to advanced technologies that bridge linguistic gaps.</p><p>There have been many different architectures and models to resolve the problem to let machine understand the context especially the idiomatic sentences which vary from one place to another in terms of culture and language.</p><h3><strong>Google Neural Machine Translation</strong></h3><p>Google Neural Machine Translation (GNMT) and other Models that are being used for language translation are performing better but still there is space between the dots. There are many techniques and methods to do better in machine translation and we will discuss those as well.</p><p>One of the techniques in the machine learning for translation is Neural Networks which performed better results in many tasks. It is biological inspired programming paradigm which enables machines to learn from observational data. First, let’s understand what a function. A function is something you give a value x and it calculates a value y. In simple words, you give input and it gives us output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*kqmdi1DrRF_CcllfwAaYVA.png" /><figcaption>function with input x and output y</figcaption></figure><p>Let’s say we dont know the function but we know some values of x and y. Can we still find the pattern without knowing the function?. Answer is yes, and that is called function approximation where we find the pattern using values of x and y without knowing the function. That is what a neural network does. Neural Network is basically a function which takes input and gives us output, consider it as layers like input layer, hidden layer(s) and output layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*5tcTMuWpJrURY67jXgy4zQ.png" /><figcaption>Neural Network Structure</figcaption></figure><p>Hidden layers are the main the computation engine which do the computation using functions, it could be sigmoid function, ReLU and GRU. Hidden layer is responsible for transforming the input to the output that could be used. All the nodes are interconnected with each other that means the output of previous node is input of next node. Each node contains weight which multiplied by input then send it to the next node. And that is the reason why neural networks can learn almost anything. Now lets understand the architecture and types of neural networks. There are different types of Neural Networks Autoencoders, RNN, LSTM, Seq2Seq and many more. In all these models neural networks are being used and its interconnected structure but with small change and depends upon specific tasks.</p><p>Now lets dive into the code and we will be using seq2seq with encoder decoder RNN architecture for Urdu to English Translation. The most commonly used for language translation is seq2seq model which uses encoder-decoder architecture and we will follow that model for Eng to Ur translation. From the term it is obvious that it uses sequence of input and gives sequence of output.</p><pre>import unicodedata<br>import re<br>import random<br>import nltk<br>from nltk.stem import WordNetLemmatizer<br>from urduhack.preprocessing import remove_accents<br>from urduhack.tokenization import word_tokenizer<br>import pandas as pd<br></pre><p>Importing the libraries that will be used in preprocessing the text parallel corpus English and Urdu.</p><pre>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>print(device)</pre><p>Lets ensure that we are using cuda. Check the status whether you are using CPU or CUDA.</p><pre><br>SOS_token = 0<br>EOS_token = 1<br>UNK_token = 2<br><br>urdu_diacritics  = [&#39;ِ&#39;, &#39;ٰ&#39;, &#39;ُ&#39;, &#39;ٍ&#39;, &#39;ً&#39;, &#39;َ&#39;]<br><br>class Lang:<br>    def __init__(self, name):<br>        self.name = name<br>        self.word2index = {}<br>        self.word2count = {}<br>        self.index2word = {0: &quot;SOS&quot;, 1: &quot;EOS&quot;, 2: &quot;UNK&quot;}  # Include UNK_token<br>        self.n_words = 3  # Count SOS, EOS, and UNK<br><br>    def remove_diacritics(self, sentence):<br>      for letter in text:<br>        if letter in urdu_diacritics:<br>            text = text.replace(letter, &#39;&#39;)<br>        return text<br><br><br>    def addSentence(self, sentence):<br>        for word in sentence.split(&#39; &#39;):<br>            self.addWord(word)<br><br>    def addWord(self, word):<br>        if word not in self.word2index:<br>            self.word2index[word] = self.n_words<br>            self.word2count[word] = 1<br>            self.index2word[self.n_words] = word<br>            self.n_words += 1<br>        else:<br>            self.word2count[word] += 1<br><br></pre><p>First we are initializing the tokens for SOS(Start of Snetence), EOS(End of Sentence) and UNK(Unknown). It is best practice to identify the vocab and unkonwn words which are not present in the dataset or model doesn’t recongnise the vocab. In Urdu language, the diacritics are most challenging task because they pay pivot role and sometimes they change the context like: Urdu Word: “میں”</p><ul><li>Without Diacritics: “main” (means ‘I’ or ‘me’)</li><li>With Diacritic: “mēṁ” (means ‘in’ or ‘within’)</li></ul><p>Example Sentences:</p><ul><li>Without Diacritics: “میں کتاب پڑھ رہا ہوں۔” (I am reading a book.)</li><li>With Diacritic: “میں کتاب میں پڑھ رہا ہوں۔” (I am reading within the book.)</li><li>For that we have to remove the diacritics for better understanding to the model and training phase.</li></ul><p>The addWord method is responsible for adding a word to the vocabulary. If the word is not already in the vocabulary, it is assigned a unique index, and its count is set to 1. If the word is already in the vocabulary, its count is incremented.</p><p>The addSentence method takes a sentence, splits it into words, and then calls the addWord method for each word. This method is used to add sentences to the vocabulary.</p><pre>def unicodeToAscii(s):<br>    return &#39;&#39;.join(<br>        c for c in unicodedata.normalize(&#39;NFD&#39;, s)<br>        if unicodedata.category(c) != &#39;Mn&#39;<br>    )<br><br># Lowercase, trim, and remove non-letter characters<br><br><br>def normalizeString(s):<br>    s = unicodeToAscii(s.lower().strip())<br>    s = re.sub(r&quot;([.!?])&quot;, r&quot; \1&quot;, s)<br>    return s</pre><p>The Normalize function takes a Unicode string (s) and converts it to its ASCII equivalent by removing diacritics. It uses the unicodedata module to normalize the string to its decomposed form (NFD) and then filters out characters with the category &#39;Mn&#39; (non-spacing marks), effectively removing diacritics. This is other way similar to previous daicritics removal technique. Further normalizes the string by converting it to lowercase, stripping leading and trailing whitespaces, and adding spaces before punctuation marks (‘.’, ‘!’, ‘?’). This is commonly done in text processing to improve tokenization and model understanding.</p><p>For instance:</p><p>input: “Thérè áre sôme Ünicôde téxts wíth diacritics.”<br>Output: there are some unicode texts with diacritics .</p><p>One of the challenge in machine translation is to provide better preprocessed text to let model understand during training easily. Some tasks like language translation or compilers depends upon preprocessing and it is critical point for better accuracy and efficiency.</p><pre>def readLangs(file_path_lang1, file_path_lang2, reverse=False):<br>    print(&quot;Reading lines...&quot;)<br>    # Read the first file and split into lines<br>    lines_lang1 = open(&#39;/content/english-corpus.txt&#39;, encoding=&#39;utf-8-sig&#39;).read().strip().split(&#39;\n&#39;)<br><br>    # Read the second file and split into lines<br>    lines_lang2 = open(&#39;/content/urdu-corpus.txt&#39;, encoding=&#39;utf-8-sig&#39;).read().strip().split(&#39;\n&#39;)<br><br>    # Ensure that both files have the same number of lines<br>    if len(lines_lang1) != len(lines_lang2):<br>        raise ValueError(&quot;Both files should have the same number of lines.&quot;)<br><br>    # Combine lines from both files into pairs<br>    pairs = [[normalizeString(lang1), normalizeString(lang2)] for lang1, lang2 in zip(lines_lang1, lines_lang2)]<br>    print(pairs)<br><br>    # Reverse pairs, make Lang instances<br>    if reverse:<br>        pairs = [list(reversed(p)) for p in pairs]<br>        input_lang = Lang(&quot;Urdu&quot;)  # Replace &quot;lang2&quot; with the appropriate language name<br>        output_lang = Lang(&quot;English&quot;)  # Replace &quot;lang1&quot; with the appropriate language name<br>    else:<br>        input_lang = Lang(&quot;English&quot;)  # Replace &quot;lang1&quot; with the appropriate language name<br>        output_lang = Lang(&quot;urdu&quot;)  # Replace &quot;lang2&quot; with the appropriate language name<br><br>    return input_lang, output_lang, pairs</pre><p>I am using two txt files and you can find them below mentioned link. We will then combine them in one parallel corpus.</p><pre>MAX_LENGTH = 100<br><br>def filterPair(p):<br>    return len(p[0].split(&#39; &#39;)) &lt; MAX_LENGTH and len(p[1].split(&#39; &#39;)) &lt; MAX_LENGTH<br><br>def filterPairs(pairs):<br>    return [pair for pair in pairs]<br><br></pre><p>FilterPair function takes a pair of sentences p (presumably source and target sentences) and returns True if both the source and target sentences have a length (number of tokens) less than MAX_LENGTH, and False otherwise. The MAX_LENGTH constant is used to set a maximum allowed length for sentences. FilterPairs function takes a list of pairs of sentences (pairs) and filters them using the filterPair function. It returns a new list containing only those pairs for which filterPair returns True.</p><pre><br>def prepareData(lang1, lang2, reverse=False):<br>    input_lang, output_lang, pairs = readLangs(lang1, lang2, reverse)<br>    print(&quot;Read %s sentence pairs&quot; % len(pairs))<br>    pairs = filterPairs(pairs)<br>    print(&quot;Trimmed to %s sentence pairs&quot; % len(pairs))<br>    print(&quot;Counting words...&quot;)<br>    for pair in pairs:<br>        input_lang.addSentence(pair[0])<br>        output_lang.addSentence(pair[1])<br>    print(&quot;Counted words:&quot;)<br>    print(input_lang.name, input_lang.n_words)<br>    print(output_lang.name, output_lang.n_words)<br>    return input_lang, output_lang, pairs<br><br><br>input_lang, output_lang, pairs = prepareData(&#39;eng&#39;, &#39;urdu&#39;, True)<br>print(random.choice(pairs))</pre><p>This function takes two language codes (lang1 and lang2) and an optional reverse parameter. It performs the following steps:</p><p>Calls the readLangs function to read sentence pairs for the specified languages and possibly reverse the pairs if reverse is True.<br>Prints the number of sentence pairs read.<br>Calls the filterPairs function to filter out pairs that exceed the maximum length. Prints the number of sentence pairs after trimming.<br>Counts the words in the sentences and adds them to the language objects (input_lang and output_lang) using the addSentence method and prints the final count of words for each language. Returns the language objects and the filtered pairs.</p><p>Now lets implement the encoder decoder architecture and hidden layer. We will be using torch library and GRU mechanism. Please make sure to import correct libraries and versions.</p><pre>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br><br>class EncoderRNN(nn.Module):<br>    def __init__(self, input_size, hidden_size):<br>        super(EncoderRNN, self).__init__()<br>        self.hidden_size = hidden_size<br>        self.embedding = nn.Embedding(input_size, hidden_size)<br>        self.gru = nn.GRU(hidden_size, hidden_size)<br><br>    def forward(self, input, hidden):<br>        embedded = self.embedding(input).view(1, 1, -1)<br>        output, hidden = self.gru(embedded, hidden)<br>        return output, hidden<br><br>    def initHidden(self):<br>        return torch.zeros(1, 1, self.hidden_size, device=device)</pre><p>Initializes the EncoderRNN class. It takes input_size (size of the vocabulary) and hidden_size as parameters.<br>Creates an embedding layer (self.embedding) to convert input indices into dense vectors of a fixed size (hidden_size).<br>Initializes a GRU (Gated Recurrent Unit) layer (self.gru) with input and hidden size both set to hidden_size.<br>forward Method: Defines the forward pass of the encoder.<br>Takes an input tensor (sequence of indices representing words) and a hidden tensor representing the hidden state of the GRU.<br>Applies the embedding layer to convert input indices to dense vectors and reshapes it to have a batch size of 1 (view(1, 1, -1)).<br>Passes the embedded tensor through the GRU layer, updating the hidden state and producing an output tensor.Returns the output tensor and the updated hidden state. This EncoderRNN is designed to process one time step at a time (sequence length of 1). It’s a basic building block for sequence-to-sequence models, where the output of the encoder is used as the initial hidden state for the decoder.</p><p>initHidden Method:<br>Initializes the hidden state for the GRU. It returns a tensor of zeros with shape (1, 1, hidden_size).</p><pre><br>class DecoderRNN(nn.Module):<br>    def __init__(self, hidden_size, output_size):<br>        super(DecoderRNN, self).__init__()<br>        self.hidden_size = hidden_size<br><br>        self.embedding = nn.Embedding(output_size, hidden_size)<br>        self.gru = nn.GRU(hidden_size, hidden_size)<br>        self.out = nn.Linear(hidden_size, output_size)<br>        self.softmax = nn.LogSoftmax(dim=1)<br><br>    def forward(self, input, hidden):<br>        output = self.embedding(input).view(1, 1, -1)<br>        output = F.relu(output)<br>        output, hidden = self.gru(output, hidden)<br>        output = self.softmax(self.out(output[0]))<br>        return output, hidden<br><br>    def initHidden(self):<br>        return torch.zeros(1, 1, self.hidden_size, device=device)</pre><p>Initializes the DecoderRNN class. It takes hidden_size and output_size as parameters.<br>Creates an embedding layer (self.embedding) to convert output indices into dense vectors of a fixed size (hidden_size).<br>Initializes a GRU (self.gru) with input and hidden size both set to hidden_size.<br>Creates a linear layer (self.out) to map the hidden state to the output size.<br>Applies the LogSoftmax activation (self.softmax) along dimension 1.<br>forward Method:<br>Defines the forward pass of the decoder.<br>Takes an input tensor (the previous output index) and a hidden tensor representing the hidden state of the GRU.<br>Applies the embedding layer to convert input indices to dense vectors and reshapes it to have a batch size of 1 (view(1, 1, -1)).<br>Applies the ReLU activation to the embedded tensor.<br>Passes the tensor through the GRU layer, updating the hidden state and producing an output tensor.<br>Applies the softmax activation to the linear layer to obtain the output probabilities.<br>Returns the output tensor and the updated hidden state.</p><pre>import torch.nn.functional as F<br>class AttnDecoderRNN(nn.Module):<br>    def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):<br>        super(AttnDecoderRNN, self).__init__()<br>        self.hidden_size = hidden_size<br>        self.output_size = output_size<br>        self.dropout_p = dropout_p<br>        self.max_length = max_length<br><br>        self.embedding = nn.Embedding(self.output_size, self.hidden_size)<br>        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)<br>        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)<br>        self.dropout = nn.Dropout(self.dropout_p)<br>        self.gru = nn.GRU(self.hidden_size, self.hidden_size)<br>        self.out = nn.Linear(self.hidden_size, self.output_size)<br><br>    def forward(self, input, hidden, encoder_outputs):<br>        embedded = self.embedding(input).view(1, 1, -1)<br>        embedded = self.dropout(embedded)<br><br>        attn_weights = F.softmax(<br>            self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)<br>        attn_applied = torch.bmm(attn_weights.unsqueeze(0),<br>                                 encoder_outputs.unsqueeze(0))<br><br>        output = torch.cat((embedded[0], attn_applied[0]), 1)<br>        output = self.attn_combine(output).unsqueeze(0)<br><br>        output = F.relu(output)<br>        output, hidden = self.gru(output, hidden)<br><br>        output = F.log_softmax(self.out(output[0]), dim=1)<br>        return output, hidden, attn_weights<br><br><br>    def initHidden(self):<br>        return torch.zeros(1, 1, self.hidden_size, device=device)</pre><p>Initializes the AttnDecoderRNN class. It takes hidden_size, output_size, dropout_p, and max_length as parameters.<br>Creates an embedding layer (self.embedding) to convert output indices into dense vectors of a fixed size (hidden_size).<br>Initializes linear layers (self.attn, self.attn_combine) for attention mechanism.<br>Applies dropout to the embedded input (self.dropout).<br>Initializes a GRU (self.gru) with input and hidden size both set to hidden_size.<br>Creates a linear layer (self.out) for the final output.<br>forward Method:<br>Defines the forward pass of the decoder with attention.<br>Takes an input tensor (the previous output index), a hidden tensor representing the hidden state of the GRU, and encoder_outputs tensor.<br>Applies embedding and dropout to the input.<br>Calculates attention weights using a linear layer and applies them to the encoder outputs.<br>Concatenates the embedded input and attention-applied encoder output.<br>Passes the concatenated tensor through a linear layer (self.attn_combine).<br>Applies ReLU activation and passes through the GRU, updating the hidden state.<br>Applies log-softmax activation to obtain the final output probabilities.</p><pre>def indexesFromSentence(lang, sentence):<br>    return [lang.word2index[word] for word in sentence.split(&#39; &#39;)]<br><br><br>def tensorFromSentence(lang, sentence):<br>    indexes = indexesFromSentence(lang, sentence)<br>    indexes.append(EOS_token)<br>    return torch.tensor(indexes, dtype=torch.long, device=device).view(-1, 1)<br><br><br>def tensorsFromPair(pair):<br>    input_tensor = tensorFromSentence(input_lang, pair[0]).to(device)<br>    target_tensor = tensorFromSentence(output_lang, pair[1]).to(device)<br>    return (input_tensor, target_tensor)</pre><p>indexFromSentence function takes a language object (lang) and a sentence as input. It tokenizes the sentence into a list of words and retrieves the corresponding indices of each word from the language’s word2index dictionary. Returns a list of indices representing the words in the sentence. <br>tensorFromSentence function utilizes the indexesFromSentence function to obtain the indices for each word in the sentence.<br>Appends the End-of-Sentence (EOS) token index (EOS_token) to the list of indices. If our dictionary is like {‘I’: 1, ‘love’: 2, ‘coding’: 3}, and the input sentence is “I love coding”, this function would return [1, 2, 3].<br>Converts the list of indices into a PyTorch tensor of type torch.long and reshapes it into a column vector using .view(-1, 1).<br>Returns the resulting tensor.<br>tensorsFromPair function takes a pair of sentences (pair) as input.<br>Calls tensorFromSentence for both the input and target sentences using the corresponding language objects (input_lang and output_lang).<br>Converts the resulting tensors to the specified device (assumed to be defined elsewhere in the code).<br>Returns a tuple containing the input and target tensors.</p><pre>teacher_forcing_ratio = 0.5<br>def train(input_pair, target_pair, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion, max_length=MAX_LENGTH):<br>    input_tensor, target_tensor = input_pair.to(device), target_pair.to(device)<br>    encoder_hidden = encoder.initHidden()<br><br>    encoder_optimizer.zero_grad()<br>    decoder_optimizer.zero_grad()<br><br>    input_length = input_tensor.size(0)<br>    target_length = target_tensor.size(0)<br><br>    encoder_outputs = torch.zeros(max_length, encoder.hidden_size, device=device)<br><br>    loss = 0<br>    predictions = []<br><br>    for ei in range(input_length):<br>        encoder_output, encoder_hidden = encoder(<br>            input_tensor[ei], encoder_hidden)<br>        encoder_outputs[ei] = encoder_output[0, 0]<br><br>    decoder_input = torch.tensor([[SOS_token]], device=device)<br><br>    decoder_hidden = encoder_hidden<br><br>    use_teacher_forcing = True if random.random() &lt; teacher_forcing_ratio else False<br><br>    if use_teacher_forcing:<br>        # Teacher forcing: Feed the target as the next input<br>        for di in range(target_length):<br>            decoder_output, decoder_hidden, decoder_attention = decoder(<br>                decoder_input, decoder_hidden, encoder_outputs)<br>            loss += criterion(decoder_output, target_tensor[di])<br>            predictions.append(torch.tensor([torch.argmax(decoder_output, dim=1).item()], device=device))  # Store predictions<br> # Store predictions<br><br>            decoder_input = target_tensor[di].unsqueeze(0)  # Teacher forcing<br><br>    else:<br>        # Without teacher forcing: use its own predictions as the next input<br>        for di in range(target_length):<br>            decoder_output, decoder_hidden, decoder_attention = decoder(<br>                decoder_input, decoder_hidden, encoder_outputs)<br>            topv, topi = decoder_output.topk(1)<br>            predictions.append(topi.item())  # Store predictions<br>            decoder_input = topi.squeeze().detach().to(device)  # detach from history as input<br><br>            loss += criterion(decoder_output, target_tensor[di])<br>            if decoder_input.item() == EOS_token:<br>                break<br><br>    loss.backward()<br><br>    encoder_optimizer.step()<br>    decoder_optimizer.step()<br><br>    return loss.item() / target_length, predictions</pre><p>Training the Neural Network<br>In machine learning, training a neural network involves teaching it to make accurate predictions. This process is like guiding a student through exercises. In our case, the student is the neural network, and the exercises are pairs of sentences for translation.</p><p>Teacher Forcing<br>Think of “teacher forcing” as a method where the teacher (the correct translation) guides the student (the neural network) during learning. It’s like showing the correct answers to help the student learn more efficiently.</p><p>The train Function<br>This function is like a coaching session. It takes a pair of sentences, an encoder, a decoder, and optimizers (tools for adjusting the learning process). The goal is to minimize the difference between the predicted translation and the actual translation.</p><p>Encoder’s Role:</p><p>The encoder processes the input sentence, capturing its meaning in a hidden state.<br>The hidden state is then used to guide the decoder.<br>Decoding with Attention:</p><p>The decoder receives the hidden state from the encoder and generates the translation step by step.<br>Attention mechanisms allow the decoder to focus on different parts of the input sentence while generating each word.<br>Teacher Forcing vs. Self-Teaching:</p><p>use_teacher_forcing decides whether the network should learn with the help of correct translations or rely on its own predictions.<br>If using teacher forcing, the correct translation is fed as input at each step.<br>Without teacher forcing, the network uses its own predictions as input for the next step.<br>Loss Calculation:</p><p>The loss represents the difference between the predicted translation and the actual translation.<br>The goal is to minimize this difference to improve accuracy.<br>Updating Weights:</p><p>After calculating the loss, the optimizer adjusts the neural network’s “weights” to improve future predictions.</p><pre>def trainIters(encoder, decoder, n_iters, print_every=1000, plot_every=100, learning_rate=0.01):<br>    start = time.time()<br>    plot_losses = []<br>    print_loss_total = 0  # Reset every print_every<br>    plot_loss_total = 0  # Reset every plot_every<br>    total_correct = 0<br>    total_samples = 0<br><br>    encoder_optimizer = optim.SGD(encoder.parameters(), lr=learning_rate)<br>    decoder_optimizer = optim.SGD(decoder.parameters(), lr=learning_rate)<br>    training_pairs = [tensorsFromPair(random.choice(pairs))<br>                      for i in range(n_iters)]<br>    criterion = nn.NLLLoss()<br><br>    for iter in range(1, n_iters + 1):<br>        training_pair = training_pairs[iter - 1]<br>        input_tensor = training_pair[0]<br>        target_tensor = training_pair[1]<br><br>        loss, predictions = train(input_tensor, target_tensor, encoder,<br>                                  decoder, encoder_optimizer, decoder_optimizer, criterion)<br><br>        print_loss_total += loss<br>        plot_loss_total += loss<br><br>        # Calculate accuracy during training<br>        correct = (torch.tensor(predictions, device=device) == target_tensor).sum().item()<br><br><br>        total_correct += correct<br>        total_samples += target_tensor.size(0)<br><br>        if iter % print_every == 0:<br>            print_loss_avg = print_loss_total / print_every<br>            accuracy = total_correct / total_samples<br>            print_loss_total = 0<br>            total_correct = 0<br>            total_samples = 0<br>            print(&#39;%s (%d %d%%) Loss: %.4f, Accuracy: %.2f%%&#39; % (timeSince(start, iter / n_iters),<br>                                                                 iter, iter / n_iters * 100, print_loss_avg, accuracy * 100))<br><br>        if iter % plot_every == 0:<br>            plot_loss_avg = plot_loss_total / plot_every<br>            plot_losses.append(plot_loss_avg)<br>            plot_loss_total = 0<br><br>    showPlot(plot_losses)</pre><p><strong>The trainIters Function</strong><br>This function orchestrates multiple training sessions, iterating through the dataset several times. It keeps track of losses and accuracy, printing updates at regular intervals.</p><p><strong>Initialization:</strong></p><p>The function initializes optimizers, defines the criterion (loss calculation method), and prepares training pairs.</p><p><strong>Training Loop:</strong></p><p>It iterates through the training pairs, calling the train function for each.<br>Losses and accuracy are tracked and printed periodically.</p><p><strong>Plotting Loss:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*5SHZeEZa2JMTDHdMh_YBew.png" /></figure><p>The loss values are collected over iterations and plotted to visualize the network’s learning progress.<br>In a nutshell, this training process is akin to coaching a student through exercises, gradually refining their skills. The neural network learns to translate by adjusting its internal parameters based on the differences between predicted and correct translations.</p><pre>def test(input_tensor, target_tensor, encoder, decoder, criterion):<br>    # Set the model to evaluation mode<br>    encoder.eval()<br>    decoder.eval()<br><br>    input_length = input_tensor.size(0)<br>    target_length = target_tensor.size(0)<br><br>    encoder_hidden = encoder.initHidden()<br>    encoder_outputs = torch.zeros(MAX_LENGTH, encoder.hidden_size, device=device)<br><br>    loss = 0<br>    correct_predictions = 0<br><br>    with torch.no_grad():  # Disable gradient tracking during testing<br>        for ei in range(input_length):<br>            encoder_output, encoder_hidden = encoder(<br>                input_tensor[ei], encoder_hidden)<br>            encoder_outputs[ei] = encoder_output[0, 0]<br><br>        decoder_input = torch.tensor([[SOS_token]], device=device)<br>        decoder_hidden = encoder_hidden<br><br>        for di in range(target_length):<br>            decoder_output, decoder_hidden, _ = decoder(<br>                decoder_input, decoder_hidden, encoder_outputs)<br>            loss += criterion(decoder_output, target_tensor[di])<br><br>            topv, topi = decoder_output.topk(1)<br>            decoder_input = topi.squeeze().detach()<br><br>            if decoder_input.item() == target_tensor[di].item():  # Check if prediction is correct<br>                correct_predictions += 1<br><br>            if decoder_input.item() == EOS_token:<br>                break<br><br>    accuracy = correct_predictions / target_length<br>    average_loss = loss.item() / target_length<br><br>    return average_loss, accuracy</pre><p>Testing the Neural Network<br>In machine learning, testing a neural network involves evaluating its performance on unseen data. It’s akin to assessing a student’s understanding after they’ve learned a subject. The test function here evaluates how well the trained network translates sentences it hasn’t seen before.</p><p>The test Function:<br>This function takes an input sentence, the corresponding correct translation, the encoder, decoder, and a criterion (a measure of performance, typically the loss function).</p><p>Evaluation Mode:</p><p>The eval() method is called on both the encoder and decoder. This sets them to evaluation mode, indicating that they should not learn during this phase. We only want to evaluate their performance.<br>Encoding the Input:</p><p>The input sentence is processed by the encoder, capturing its meaning in a hidden state. The encoder’s output is stored in encoder_outputs.<br>Decoding and Evaluation:</p><p>The decoder then generates the translation one step at a time. At each step, the generated word is compared to the actual word in the correct translation.<br>Loss is accumulated based on the difference between predicted and actual translations.<br>Accuracy Calculation:</p><p>The function keeps track of correct predictions, checking if the predicted word matches the actual word at each step.<br>The final accuracy is the ratio of correct predictions to the total number of words in the correct translation.<br>Results:</p><p>The function returns the average loss and accuracy. The average loss is a measure of how well the network’s predictions align with the correct translation.<br>Use of torch.no_grad():<br>The with torch.no_grad(): block ensures that gradient tracking is disabled during testing. This speeds up the process since we don’t need to calculate gradients for parameter updates. Testing is purely about assessing performance.</p><p>In essence, the test function evaluates the neural network’s ability to translate sentences accurately without further learning. It’s like giving a student a quiz to see how well they’ve retained what they’ve learned.</p><pre>import time<br>import math<br><br><br>def asMinutes(s):<br>    m = math.floor(s / 60)<br>    s -= m * 60<br>    return &#39;%dm %ds&#39; % (m, s)<br><br><br>def timeSince(since, percent):<br>    now = time.time()<br>    s = now - since<br>    es = s / (percent)<br>    rs = es - s<br>    return &#39;%s (- %s)&#39; % (asMinutes(s), asMinutes(rs))</pre><p>“asMintues” function takes a duration in seconds (s) and converts it into a human-readable format, expressing the duration in minutes and seconds. For example, if s is 125 seconds, it would return ‘2m 5s’.<br>“timeSince” function helps track time during a process, like training a neural network. It calculates elapsed time, estimates total time based on completion progress, and shows the remaining time in a readable format. These functions are handy for keeping track of how much time has passed during a task and estimating how much time is left until completion.</p><pre>def testIters(encoder, decoder, test_input_tensors, test_target_tensors, criterion):<br>    total_loss = 0<br>    total_correct = 0<br>    total_samples = 0<br>    num_batches = len(test_input_tensors)<br><br>    with torch.no_grad():  # Ensure no gradients are calculated during testing<br>        for batch_idx in range(num_batches):<br>            input_tensor = test_input_tensors[batch_idx]<br>            target_tensor = test_target_tensors[batch_idx]<br><br>            loss, predictions = test(input_tensor, target_tensor, encoder, decoder, criterion)<br>            total_loss += loss<br><br>            # Calculate accuracy<br>            correct = (predictions == target_tensor).sum().item()<br>            total_correct += correct<br>            total_samples += target_tensor.size(0)<br><br>    average_loss = total_loss / num_batches<br>    accuracy = total_correct / total_samples<br><br>    return average_loss, accuracy</pre><p>The testIters function systematically evaluates the trained neural network’s performance on multiple batches of test data. Throughout the testing process, it keeps track of the total loss, correct predictions, and the overall number of samples. By iteratively calling the test function for each batch, it accumulates insights into the neural network’s accuracy and proficiency in translating sentences. The function then calculates the average loss and accuracy across all batches, providing a comprehensive evaluation of the model’s performance on the test data. This process, facilitated by the use of torch.no_grad() to disable gradient tracking during testing, mimics the act of grading a student on their comprehension and application of knowledge across various quizzes.</p><pre>hidden_size = 512<br><br><br>encoder1 = EncoderRNN(input_lang.n_words, hidden_size).to(device)<br>attn_decoder1 = AttnDecoderRNN(hidden_size, output_lang.n_words, dropout_p=0.1).to(device)<br><br># Train the model<br>trainIters(encoder1, attn_decoder1, 50000, print_every=5000)</pre><p>Here, you’re creating an EncoderRNN and an AttnDecoderRNN instance. These components are the building blocks of a sequence-to-sequence model for translation. The hidden_size parameter determines the size of the hidden state, essentially the internal representation of the input sentence.<br>Now, you’re initiating the training process with the trainIters function. It will train the model for 50,000 iterations, printing updates every 5000 iterations. During training, the model learns to map input sentences from input_lang to output sentences in output_lang. The process involves adjusting internal parameters (weights) based on the difference between predicted and actual translations.</p><pre># Save encoder and decoder models<br>torch.save(encoder1.state_dict(), &#39;encoder_model.pth&#39;)<br>torch.save(attn_decoder1.state_dict(), &#39;decoder_model.pth&#39;)</pre><p>We are saving the model so that we dont have to train everytime we translate the input text.</p><pre><br><br># User input function<br>def main():<br>    while True:<br>        input_sentence = input(&quot;Enter an Urdu sentence (or type &#39;exit&#39; to quit): &quot;)<br>        if input_sentence.lower() == &#39;exit&#39;:<br>            break<br>        translated_sentence = translateSentence(encoder1, attn_decoder1, input_sentence)<br>        print(&quot;Translation:&quot;, translated_sentence)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>In this function, a loop is created to continuously prompt the user for input. The user can enter an Urdu sentence, and if they type ‘exit’, the loop breaks, ending the interaction. For each entered sentence, the translateSentence function is called to translate it using the trained encoder1 and attn_decoder1 models. The translation is then printed.</p><h3><strong>Conclusion</strong></h3><p>This is the basic implementation of Machine Translation, you can enhance it further using different architecture, hyper parameter tunning and training it on even large dataset. Till then keep learning and exploring new AI/ML ideas.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b01493163187" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>