<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Linesh Malkam on Medium]]></title>
        <description><![CDATA[Stories by Linesh Malkam on Medium]]></description>
        <link>https://medium.com/@lineshmalkam?source=rss-c84538b492f3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*f6WJJd3btSujPgh1lLWptQ.jpeg</url>
            <title>Stories by Linesh Malkam on Medium</title>
            <link>https://medium.com/@lineshmalkam?source=rss-c84538b492f3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:40:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lineshmalkam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a LLM based parser for PDF files to extract tables, figures and more.]]></title>
            <link>https://medium.com/@lineshmalkam/building-a-llm-based-parser-for-pdf-files-to-extract-tables-figures-and-more-3a1792fda2bb?source=rss-c84538b492f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a1792fda2bb</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Linesh Malkam]]></dc:creator>
            <pubDate>Wed, 12 Feb 2025 13:19:10 GMT</pubDate>
            <atom:updated>2025-02-25T07:00:17.032Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I will walk you through a <strong>problem statement</strong> I received as part of the interview process for a <strong>Founding Engineer (LLM/AI Engineer) role</strong>. The task involves <strong>extracting, structuring, and analyzing financial data</strong> from large-scale scanned PDFs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*SvKJs-nClgu-h__nglxeqg.png" /></figure><p>We’ll break down the problem statement into key components, explore possible solutions, and discuss an <strong>LLM-powered approach</strong> to automate the process efficiently. Additionally, we’ll consider <strong>scalability challenges</strong> and strategies to handle thousands of similar documents in real-world applications.</p><p>Let’s dive in!</p><p><strong>Problem Statement</strong></p><p>Companies publish annual reports as scanned PDFs, making financial data extraction challenging due to unstructured tables, images, and visual elements. This assignment focuses on developing an automated pipeline to:</p><ol><li><strong>Extract tabular data</strong> from Tesco’s 2023 and 2024 annual filings and convert it into structured JSON.</li><li><strong>Merge financial data</strong> across both years for year-over-year comparison.</li><li><strong>Propose methods</strong> to extract insights from charts, images, and infographics.</li><li><strong>Optimize scalability</strong> to handle extensive documents (500+ pages) efficiently.</li><li><strong>Integrate LLMs</strong> (e.g., Gemini Flash 2.0) for enhanced extraction and processing.</li></ol><p><strong>Simplifying the problem</strong></p><p>After analyzing the problem statement, we understand that we need to process the scanned PDF to <strong>extract tables and key financial figures</strong> efficiently. The input file is a <strong>scanned PDF</strong>, approximately <strong>15MB in size</strong>, containing <strong>over 250 pages</strong> — making it computationally heavy to process.</p><p>Additionally, we need to consider how to <strong>scale the solution</strong> to handle <strong>multiple large files</strong> efficiently. Beyond extraction, we must also <strong>summarize each table</strong>, enabling us to <strong>compare tables across different years</strong> and identify similarities between them.</p><h3>Observations:</h3><ul><li>We must efficiently process <strong>large scanned PDFs</strong> without performance bottlenecks.</li><li>The input consists of <strong>scanned PDFs</strong>, requiring OCR-based extraction.</li><li>We need to <strong>extract tables and key figures</strong> while also summarizing the content.</li><li>The solution should <strong>maintain high accuracy</strong> to ensure reliable financial data extraction.</li></ul><p><strong>Approach</strong></p><p>The PDF consists of <strong>scanned pages</strong>, meaning we need to apply <strong>OCR</strong> (Optical Character Recognition) to extract text and structured data. Since the document is large, <strong>feeding it directly into an LLM</strong> is not feasible due to <strong>context window limitations</strong>.</p><p>To efficiently process the document, we must first <strong>detect and extract tables and relevant diagrams</strong> that are crucial for our use case. This requires a <strong>table detection mechanism</strong> combined with <strong>OCR-based text extraction</strong>, ensuring that only meaningful data is processed while minimizing computational overhead.</p><p>Doing OCR process manually by writing on our would consume a lot of time as the assignment was given limited time. For the table detection we need some kind of Computer vision model to detect the tables or diagrams. Now we can build OCR and detection mechanism from scratch but would take a lot of time to builts and test it. So we should find some opensource tool which can satisfy our requirements.</p><p>Manually implementing the <strong>OCR process</strong> from scratch would be <strong>time-consuming</strong>, especially given the <strong>time constraints</strong> of the assignment. For <strong>table and diagram detection</strong>, we need a <strong>computer vision model</strong> capable of identifying structured elements within the scanned PDF.</p><p>While building a <strong>custom OCR and detection mechanism</strong> is possible, it would require significant effort in <strong>development and testing</strong>. Instead, the best approach is to explore <strong>open-source tools</strong> that align with our requirements, enabling us to focus on <strong>data extraction, processing, and analysis</strong> rather than low-level implementation.</p><p><strong>Using Unstructured.io for OCR and table recognition</strong></p><pre>from unstructured.partition.pdf import partition_pdf<br><br>file_path =&#39;file.pdf&#39;<br>chunks = partition_pdf(<br>filename=file_path,<br>infer_table_structure=True, # extract tables<br>strategy=&quot;auto&quot;, # mandatory to infer tables<br>extract_image_block_types=[&quot;Table&quot;,&quot;Image&quot;], # Add &#39;Table&#39; to list to extract image of tables<br>extract_image_block_to_payload=True, # if true, will extract base64 for API usage<br>chunking_strategy=&quot;by_title&quot;, # or &#39;basic&#39;<br>max_characters=10000, # defaults to 500<br>combine_text_under_n_chars=2000, # defaults to 0<br>new_after_n_chars=6000,<br>)<br><br># separate tables from texts<br>texts = []<br><br>for chunk in chunks:<br><br>    if &quot;CompositeElement&quot; in str(type((chunk))):<br>        texts.append(chunk)</pre><p>The given code efficiently processes the <strong>report (a scanned PDF)</strong> by extracting structured data, including <strong>tables and images</strong>, using the unstructured.partition.pdf library. Since scanned PDFs don’t contain selectable text, <strong>OCR-based extraction is required</strong>. To handle this, the code <strong>automatically detects tables</strong> while preserving their structure and <strong>extracts relevant images</strong>, converting them to <strong>Base64 format</strong> for easier API integration. Additionally, it <strong>chunks the document intelligently</strong>, splitting content based on <strong>titles and character limits</strong> to optimize processing. This ensures that each extracted segment is <strong>meaningful and structured</strong>, making it easier to analyze large PDFs without overwhelming memory or hitting context window limits in LLMs. By leveraging <strong>open-source tools</strong> instead of building a custom OCR system, this approach saves time while ensuring high accuracy in extracting financial data.</p><pre># Get the Tables from the CompositeElement objects<br>def get_images_base64(chunks):<br>    images_b64 = []<br>    for chunk in chunks:<br>        if &quot;CompositeElement&quot; in str(type(chunk)):<br>            chunk_els = chunk.metadata.orig_elements<br>            for el in chunk_els:<br>                if &quot;Table&quot; in str(type(el)) :<br>                    images_b64.append(el.metadata.image_base64)<br>    return images_b64<br><br>tables = get_images_base64(chunks)<br>len(tables)</pre><p>The function get_images_base64(chunks) extracts <strong>Base64-encoded images of tables</strong> from the structured chunks generated by partition_pdf. It works by iterating through each chunk and identifying whether it is a <strong>CompositeElement</strong>, which contains multiple sub-elements like text, images, and tables. If a chunk contains a table, the function retrieves its <strong>Base64 representation</strong> from the metadata and stores it in a list. This allows for easy extraction of table images, which can later be processed using <strong>OCR for text recognition</strong> or stored for further analysis. Finally, calling len(tables) gives the <strong>total number of extracted tables</strong>, helping us assess how many tables were detected in the scanned PDF. This approach efficiently captures structured data while handling the complexities of scanned documents. we can do same for the diagrams where we just have to change “table” to “image”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2h03FSKHay3Ys_uJdkplMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhtJGv8bfjwSEnp7agDCvg.png" /></figure><p><strong>Converting to JSON format</strong></p><p>Now that we have successfully extracted the <strong>tables as images</strong>, the next step is to apply <strong>LLM-based parsing</strong> to convert the table contents into a structured <strong>JSON format</strong>. Using <strong>OCR</strong>, we can extract text from the table images and then feed the extracted content into an <strong>LLM</strong> to ensure proper structuring and formatting. This approach helps in handling cases where tables have <strong>merged cells, misaligned data, or complex structures</strong> that traditional parsers might struggle with.</p><p>Once the extracted data is processed, it can be <strong>converted into a JSON format</strong> optimized for our website, ensuring <strong>smooth rendering and easy integration</strong> with the frontend. This structured format will allow for better <strong>visualization, searchability, and comparisons</strong> across different tables, making the financial data more accessible and interactive.</p><p><strong>Using LangChain</strong></p><p>Since we now have the extracted <strong>tables as text</strong> (via OCR), we can use <strong>LangChain</strong> to structure this data efficiently. LangChain allows us to <strong>prompt an LLM</strong> to parse and format the extracted table data into a clean <strong>JSON format</strong>, ensuring consistency and accuracy.</p><pre>from langchain_google_genai import ChatGoogleGenerativeAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br><br># Set your Google Gemini API key<br>import os<br>os.environ[&quot;GOOGLE_API_KEY&quot;] = &quot;&quot;  # Replace with your actual API key<br><br># Define the prompt template<br>prompt_template = &quot;&quot;&quot;Extract the tables data and values in a json format stricly where Each table&#39;s JSON should include: Table Name (if available). Units (e.g., GBP, USD) if mentioned.The data contained in the table, accurately structured for ease of analysis. return only json nothing else&quot;&quot;&quot;<br>messages = [<br>    (<br>        &quot;user&quot;,<br>        [<br>            {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: prompt_template},<br>            {<br>                &quot;type&quot;: &quot;image_url&quot;,<br>                &quot;image_url&quot;: {&quot;url&quot;: &quot;data:image/jpeg;base64,{image}&quot;},<br>            },<br>        ],<br>    )<br>]<br><br># Create the prompt template<br>prompt = ChatPromptTemplate.from_messages(messages)<br><br># Initialize the Google Gemini model<br>llm = ChatGoogleGenerativeAI(<br>    model=&quot;gemini-2.0-flash-exp&quot;,  # Use the Gemini Flash model<br>    temperature=0,                # Set temperature for deterministic output<br>    max_tokens=None,              # No limit on output tokens<br>    timeout=None,                 # No timeout<br>    max_retries=2,                # Retry up to 2 times on failure<br>)<br><br># Create the chain<br>chaintables = prompt | llm | StrOutputParser()<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*LGMqOQxn3lRX320b1aMmUw.png" /></figure><p>This code sets up a <strong>LangChain pipeline</strong> to extract <strong>tables from images</strong> using Google’s <strong>Gemini LLM (via LangChain Google Generative AI)</strong> and convert them into <strong>structured JSON format</strong>. It first defines a <strong>prompt template</strong> instructing the model to extract tables while maintaining structure, including the <strong>table name, units (if available), and data in a structured JSON format</strong>. The extracted table images (in <strong>Base64 format</strong>) are passed to the <strong>Gemini Flash 2.0 model</strong>, which processes the images and outputs only <strong>JSON-formatted table data</strong>. The pipeline is built using <strong>LangChain’s prompt handling</strong>, model invocation, and output parsing utilities, ensuring a <strong>deterministic, structured, and reliable extraction</strong>. This allows for seamless integration with the frontend, making financial data easily <strong>searchable, comparable, and user-friendly</strong>.</p><pre>from langchain_google_genai import ChatGoogleGenerativeAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br><br># Set your Google Gemini API key<br>import os<br>os.environ[&quot;GOOGLE_API_KEY&quot;] = &quot;&quot;  # Replace with your actual API key<br><br># Define the prompt template<br>prompt_template = &quot;&quot;&quot;Extract Insights from Visual Elements such as diagram only . Be specific about graphs, such as bar plots and if it not relevant just skip it.&quot;&quot;&quot;<br><br>messages = [<br>    (<br>        &quot;user&quot;,<br>        [<br>            {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: prompt_template},<br>            {<br>                &quot;type&quot;: &quot;image_url&quot;,<br>                &quot;image_url&quot;: {&quot;url&quot;: &quot;data:image/jpeg;base64,{image}&quot;},<br>            },<br>        ],<br>    )<br>]<br><br># Create the prompt template<br>prompt = ChatPromptTemplate.from_messages(messages)<br><br># Initialize the Google Gemini model<br>llm = ChatGoogleGenerativeAI(<br>    model=&quot;gemini-2.0-flash-exp&quot;,  # Use the Gemini Flash model<br>    temperature=0,                # Set temperature for deterministic output<br>    max_tokens=None,              # No limit on output tokens<br>    timeout=None,                 # No timeout<br>    max_retries=2,                # Retry up to 2 times on failure<br>)<br><br># Create the chain<br>chainimages = prompt | llm | StrOutputParser()</pre><p>This code sets up a <strong>LangChain pipeline</strong> to extract <strong>insights from visual elements</strong>, specifically focusing on <strong>diagrams and graphs</strong> such as <strong>bar plots, line charts, and relevant visuals</strong>. It uses <strong>Google’s Gemini Flash 2.0 model</strong> to analyze extracted images and <strong>identify meaningful data representations</strong> while skipping irrelevant visuals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAgguznP3Qjuy3pFIppq6A.png" /></figure><h3>Main WorkFlow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dLo9PEx0x_NG19IcJw9frQ.png" /></figure><p>This workflow diagram represents a <strong>pipeline for extracting, summarizing, and retrieving data from scanned PDFs using an LLM and a vector store</strong>. Here’s a step-by-step breakdown:</p><h3>1. PDF/S3 Input:</h3><ul><li>The process starts with a <strong>PDF file</strong> stored locally or in an <strong>S3 bucket</strong> (Amazon Simple Storage Service).</li></ul><h3>2. Processing with unstructured.io:</h3><ul><li>The <strong>PDF is processed using </strong><strong>unstructured.io</strong>, a tool that extracts <strong>tables and images</strong> from documents.</li><li>These extracted elements are <strong>encoded in Base64 format</strong> to be passed to the LLM.</li></ul><h3>3. Table Processing Pipeline:</h3><ul><li><strong>Extracted tables</strong> are sent to the <strong>LLM (Large Language Model)</strong> for <strong>summarization and structuring</strong>.</li><li>Each <strong>table is assigned a document ID (</strong><strong>doc.id)</strong>, making it easier to reference later.</li><li>The <strong>summaries of the tables</strong> are stored for retrieval.</li></ul><h3>4. Image Processing Pipeline:</h3><ul><li><strong>Extracted images (e.g., charts, graphs)</strong> are also processed by the <strong>LLM</strong> to generate <strong>image summaries</strong>.</li><li>Like tables, each image is assigned a <strong>document ID (</strong><strong>doc.id)</strong>, and its summary is stored.</li></ul><h3>5. Storing Summarized Data in a Vector Store:</h3><ul><li>The summarized <strong>tables and images</strong> are stored in a <strong>vector database</strong>.</li><li>This enables <strong>efficient retrieval</strong> of relevant information based on user queries.</li></ul><h3>6. User Query &amp; LLM Response:</h3><ul><li>When a <strong>user submits a query</strong>, it is <strong>sent to the LLM</strong>.</li><li>The LLM <strong>retrieves relevant documents from the vector store</strong> using their <strong>document IDs</strong>.</li><li>The retrieved information is then <strong>formatted into a response</strong> and sent back to the user.</li></ul><h3>Key Takeaways:</h3><ul><li>This workflow automates <strong>table and image extraction</strong> from PDFs.</li><li>It ensures <strong>structured data storage</strong> using a <strong>vector database</strong>.</li><li>The <strong>LLM enables natural language retrieval</strong>, making financial reports more interactive and searchable.</li></ul><p><strong>Thanks for reading along till the end.</strong></p><p>LinkedIn : <a href="https://www.linkedin.com/in/linesh-malkam-ba613b133/">https://www.linkedin.com/in/linesh-malkam-ba613b133/</a></p><p>Twitter : <a href="https://x.com/the_linuxx">https://x.com/the_linuxx</a></p><p>Github : <a href="https://github.com/lmalkam">https://github.com/lmalkam</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a1792fda2bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka : Basics and Simple implementation]]></title>
            <link>https://medium.com/@lineshmalkam/kafka-basics-and-simple-implementation-499cf7590561?source=rss-c84538b492f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/499cf7590561</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[hld]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Linesh Malkam]]></dc:creator>
            <pubDate>Fri, 10 Jan 2025 05:19:45 GMT</pubDate>
            <atom:updated>2025-02-25T07:00:32.918Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka : Basics and Simple implementation</h3><h3>Challenges for Databases in Real-Time Data Input</h3><ul><li><strong>Write Performance</strong>: Struggles with high-throughput writes due to locking and contention.</li><li><strong>Scalability</strong>: Scaling horizontally is complex and resource-intensive.</li><li><strong>Concurrency</strong>: Cannot handle simultaneous writes from millions of producers efficiently.</li><li><strong>Latency</strong>: Slower due to indexing, locking, and transaction overhead.</li><li><strong>Event-Driven Systems</strong>: Lack built-in support for pub-sub or event-streaming models.</li></ul><h3>Why Kafka Excels at Real-Time Data Input</h3><ul><li><strong>High Throughput</strong>: Handles millions of messages per second with low latency.</li><li><strong>Horizontal Scalability</strong>: Partitions topics across brokers for distributed load handling.</li><li><strong>Decoupling</strong>: Asynchronous pub-sub model for producers and consumers.</li><li><strong>Data Retention</strong>: Configurable data storage for replay and recovery.</li><li><strong>Stream Processing</strong>: Real-time transformation and aggregation via Kafka Streams or ksqlDB</li></ul><p>Apache Kafka is an <strong>open-source event streaming platform</strong>. Think of it as a messaging system that helps different applications talk to each other by sending and receiving messages. It’s designed to handle large amounts of data in real-time.</p><h3>How Does Kafka Work?</h3><p>At its core, Kafka operates on three main components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*m0B69C_iabwAZUypfdWwog.png" /></figure><h4>1. Producers</h4><p>Producers are the sources of data. They send messages to Kafka. For example, a weather app could be a producer sending temperature data.</p><h4>2. Brokers</h4><p>A Kafka broker is like a post office. It receives messages from producers and stores them temporarily. Kafka is scalable, so you can have multiple brokers working together.</p><h4>3. Consumers</h4><p>Consumers are applications that read messages from Kafka. Continuing with our example, a weather display board could be a consumer that reads the temperature data and shows it.</p><h4><strong>4. Topic</strong></h4><p>A topic is like a category or channel where messages are stored.</p><h3>Role of ZooKeeper?</h3><p><strong>ZooKeeper</strong> is a centralized service used by Apache Kafka to manage and coordinate its cluster components. Think of ZooKeeper as Kafka’s “control room” where important tasks like tracking system state and managing configuration are handled.</p><h3>How does everything work together ?</h3><h4>1. Kafka Producers: The Data Senders</h4><p>Producers are like messengers. They send data (messages) to topics in Kafka. Here’s what they do:</p><ul><li><strong>Choose Where to Send Data:</strong> Each topic is split into sections called partitions. Producers decide which partition gets the data, often based on a key (like a customer ID).</li><li><strong>Make Sure Data is Delivered:</strong> Producers can ask Kafka to confirm when the data is safely received. They can:</li></ul><blockquote>Not wait for confirmation.</blockquote><blockquote>Wait for the main server handling the partition to confirm.</blockquote><blockquote>Wait for all copies (backups) to confirm.</blockquote><h4>2. Kafka Consumer Groups: The Data Readers</h4><p>Consumers are like workers that read data from Kafka. When they work in groups (consumer groups), they share the workload:</p><ul><li><strong>Divide the Work:</strong> Kafka assigns different partitions to different consumers in the group so they don’t overlap.</li><li><strong>Handle More Work:</strong> Adding more consumers to the group means they can process more data, as long as there are enough partitions.</li><li><strong>Remember Where They Left Off:</strong> Kafka keeps track of what data each consumer has already read, so they can pick up where they stopped if something goes wrong.</li></ul><h4>3. Data Streaming: How Data Moves</h4><p>Kafka is like a big logbook where data is written and read:</p><ul><li><strong>Producers Write Data:</strong> Producers add data to partitions in a topic.</li><li><strong>Consumers Read Data:</strong> Consumers can read new data as it arrives or go back and read old data, making Kafka useful for real-time updates and historical data.</li></ul><h4>4. Auto-Balancing: Keeping Things Running Smoothly</h4><p>Kafka has features to make sure everything runs evenly and reliably:</p><ul><li><strong>Reassigning Partitions:</strong> If a consumer joins or leaves a group, Kafka redistributes the work so that all partitions are covered.</li><li><strong>Handling Server Changes:</strong> If a Kafka server goes down or a new one is added, Kafka moves data and responsibilities around to keep things balanced.</li><li><strong>Backups:</strong> Each partition has copies on other servers. If the main one fails, a backup takes over.</li></ul><h4>5. Data Partitions: Kafka’s Secret to Speed</h4><p>Partitions are like lanes on a highway. They let Kafka handle lots of data at once:</p><ul><li><strong>Splitting Data:</strong> Topics are divided into partitions, so multiple producers and consumers can work at the same time.</li><li><strong>Storing Data in Order:</strong> Data in a partition is stored in the order it arrives, but Kafka doesn’t guarantee order between different partitions.</li><li><strong>Making Copies:</strong> Kafka keeps copies of each partition on different servers to make sure data isn’t lost.</li></ul><h3>Implementation of Kafka in Node.js</h3><ol><li>Setup the ZooKeeper on the port 2181 using Docker</li></ol><pre>docker run -p 2181:2181 zookeeper</pre><p>2. Start the Kafka container on different terminal and expose it on port 9092.</p><pre>docker run -p 9092:9092 \<br>-e KAFKA_ZOOKEEPER_CONNECT=&lt;PRIVATE_IP&gt;:2181 \<br>-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://&lt;PRIVATE_IP&gt;:9092 \<br>-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \<br>confluentinc/cp-kafka</pre><p>Note : Add your Computer’s IP in PRIVATE_IP</p><p>3. Code part</p><p>— client.js</p><pre>const { Kafka } = require(&quot;kafkajs&quot;);<br><br>const kafka = new Kafka({<br>  clientId: &quot;my-app&quot;,<br>  brokers: [&quot;192.168.137.76:9092&quot;],<br>});<br><br>module.exports = kafka;</pre><p>It will create a new Kafka instance for user app on the port 9092 of you PC.</p><p>— admin.js</p><pre>const { kafka } = require(&quot;./client&quot;);<br><br>async function init() {<br>  const admin = kafka.admin();<br>  console.log(&quot;Admin connecting...&quot;);<br>  admin.connect();<br>  console.log(&quot;Adming Connection Success...&quot;);<br><br>  console.log(&quot;Creating Topic [rider-updates]&quot;);<br>  await admin.createTopics({<br>    topics: [<br>      {<br>        topic: &quot;rider-updates&quot;,<br>        numPartitions: 2,<br>      },<br>    ],<br>  });<br>  console.log(&quot;Topic Created Success [rider-updates]&quot;);<br><br>  console.log(&quot;Disconnecting Admin..&quot;);<br>  await admin.disconnect();<br>}<br><br>init();</pre><p>It is a admin program where we can ultimately create topics which or on the broker where producers can produce the messages and consumers can consume it. Here we have a create a topic called “rider-updates”. Also we are mentioning here that we will be creating 2 data partitions for the specific topic.</p><p>— producer.js</p><pre>const kafka  = require(&quot;./client&quot;);<br>const readline = require(&quot;readline&quot;);<br><br>const rl = readline.createInterface({<br>  input: process.stdin,<br>  output: process.stdout,<br>});<br><br>async function init() {<br>  const producer = kafka.producer();<br><br>  console.log(&quot;Connecting Producer&quot;);<br>  await producer.connect();<br>  console.log(&quot;Producer Connected Successfully&quot;);<br><br>  rl.setPrompt(&quot;&gt; &quot;);<br>  rl.prompt();<br><br>  rl.on(&quot;line&quot;, async function (line) {<br>    const [riderName, location] = line.split(&quot; &quot;);<br>    await producer.send({<br>      topic: &quot;rider-status&quot;,<br>      messages: [<br>        {<br>          partition: location.toLowerCase() === &quot;north&quot; ? 0 : 1,<br>          key: &quot;location-update&quot;,<br>          value: JSON.stringify({ name: riderName, location }),<br>        },<br>      ],<br>    });<br>  }).on(&quot;close&quot;, async () =&gt; {<br>    await producer.disconnect();<br>  });<br>}<br><br>init();</pre><p>Here we are creating a producer which produces the data and puts into the kafka broker. Here we are giving input in the command line as “&lt;name&gt; &lt;location = {south,north}&gt;” which will be used for data partitions.</p><p>— consumer.js</p><pre>const kafka = require(&quot;./client&quot;);<br>const group = process.argv[2];<br><br>async function init() {<br>  const consumer = kafka.consumer({ groupId: group });<br>  await consumer.connect();<br><br>  await consumer.subscribe({ topics: [&quot;rider-status&quot;], fromBeginning: true });<br><br>  await consumer.run({<br>    eachMessage: async ({ topic, partition, message, heartbeat, pause }) =&gt; {<br>      console.log(<br>        `${group}: [${topic}]: PART:${partition}:`,<br>        message.value.toString()<br>      );<br>    },<br>  });<br>}<br><br>init();</pre><p>we can run this program as “node consumer &lt;group_id&gt;” where the specific consumer will be associated with the specific group_id. It will be necessary if we want to check the auto-balancing of the kafka for the consumer group. so will be having two consumers of same group_id to watch it in action. In the program the data will be shown in real time where each consumer will receiver according to their locations = {north or south} in the command shell.</p><p>OutPut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BFVj6C_vt1dxD1AFAvFXw.png" /></figure><p>GitHub: <a href="https://github.com/lmalkam/Kafka-Implmentation">https://github.com/lmalkam/Kafka-Implmentation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=499cf7590561" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Custom TCP Protocol and sniffing packets in C: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@lineshmalkam/building-a-custom-tcp-protocol-and-sniffing-packets-in-c-a-step-by-step-guide-45da0f7d71df?source=rss-c84538b492f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/45da0f7d71df</guid>
            <category><![CDATA[sniffing]]></category>
            <category><![CDATA[computer-networking]]></category>
            <category><![CDATA[c-programming]]></category>
            <category><![CDATA[3-way-handshake]]></category>
            <dc:creator><![CDATA[Linesh Malkam]]></dc:creator>
            <pubDate>Sun, 05 Jan 2025 06:38:27 GMT</pubDate>
            <atom:updated>2025-02-25T07:00:46.137Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong> In this article, we will explore a custom implementation of a TCP-like protocol in C. The project demonstrates socket programming concepts using raw sockets and a custom protocol, dubbed “Linesh_TCP.” This article will guide you through the code, explaining its components and functionality in a simple, digestible manner. By the end, you’ll have a solid understanding of how to create and manage your own TCP-like protocol.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*e_g0IS7jELtXMpoa9ZOhSQ.png" /></figure><h3>Key Concepts</h3><ol><li><strong>Raw Sockets</strong>: Unlike standard sockets, raw sockets provide low-level access to network layers, allowing you to craft custom headers.</li><li><strong>Custom Protocol Design</strong>: “Linesh_TCP” is a simplified version of the TCP protocol, with fields for connection requests, acknowledgments, and data transmission.</li><li><strong>Packet Sniffing</strong>: Using libpcap, the program monitors and decodes custom packets on the network.</li></ol><h3>Project Overview</h3><p>The project comprises three main components:</p><ol><li><strong>Client Program</strong></li><li><strong>Server Program</strong></li><li><strong>Packet Sniffer</strong></li></ol><h3>Custom Protocol Structure</h3><p>The Linesh_TCP structure encapsulates the protocol:</p><pre>struct Linesh_TCP {<br>    uint16_t source;<br>    uint16_t dest;<br>    uint8_t connect_req;<br>    uint8_t connect_acc;<br>    uint8_t final_acknow;<br>    uint8_t data_flg;<br>    uint16_t seq;<br>    uint16_t rec_seq;<br>};</pre><p><strong>Fields Explained:</strong></p><ul><li>source &amp; dest: Source and destination ports.</li><li>connect_req: Indicates a connection request.</li><li>connect_acc: Acknowledges a connection.</li><li>final_acknow: Final acknowledgment flag.</li><li>data_flg: Denotes the presence of data in the packet.</li><li>seq &amp; rec_seq: Sequence and acknowledgment numbers.</li></ul><h3>Client Program</h3><p>The client initiates communication using linesh_connect and sends messages via custom_msg.</p><h4>Highlights:</h4><ol><li><strong>Connection Establishment</strong>:</li></ol><pre>request-&gt;connect_req = 1; sendto(sfd, request, sizeof(request), 0, addr, sz);</pre><ul><li>Sends a connection request to the server.</li></ul><p><strong>2. Final Acknowledgment</strong>:</p><pre>request-&gt;final_acknow = 1; sendto(sfd, request, sizeof(request), 0, addr, sz);</pre><p>Completes the handshake process.</p><p><strong>3 . Data Transmission</strong>:</p><pre>memcpy(msg + sizeof(*buff), buffer, bufsiz); <br>sendto(sfd, msg, 1000, 0, addr, sz);</pre><ul><li>Encodes the message into a custom packet format and sends it.</li></ul><h3>Server Program</h3><p>The server listens for connection requests and processes incoming messages using linesh_accept and custom_receive.</p><h4>Highlights:</h4><ul><li><strong>Connection Handling</strong>:</li></ul><pre>buff-&gt;connect_acc = 1; sendto(sfd, buff, sizeof(buff), 0, addr, *sz);</pre><p>Accepts connection requests and sends an acknowledgment.</p><ul><li><strong>Message Reception</strong>:</li></ul><pre>memcpy(buffer, buffer1 + (ip-&gt;ihl * 4) + sizeof(*buff), bufsiz);</pre><p>Extracts the payload from the received packet.</p><h3>Packet Sniffer</h3><p>The sniffer captures and decodes custom protocol packets using libpcap.</p><h4>Highlights:</h4><ul><li><strong>Packet Capture</strong>:</li></ul><pre>handle = pcap_open_live(device, BUFSIZ, 0, 1000, error); pcap_loop(handle, 0, packet_handler, NULL);</pre><p>Captures packets on the specified device.</p><ul><li><strong>Packet Decoding</strong>:</li></ul><pre>ip = (struct iphdr *)(packet + 14); tcp = (struct Linesh_TCP *)(packet + 14 + (ip-&gt;ihl * 4));</pre><ul><li>Extracts IP and custom TCP headers for analysis.</li></ul><h3>How It Works</h3><ol><li><strong>Handshake Process</strong></li></ol><ul><li>The client sends a connection request (connect_req flag).</li><li>The server responds with a connection acknowledgment (connect_acc flag).</li><li>The client confirms the connection with a final acknowledgment (final_acknow flag).</li></ul><p><strong>2. Data Transmission</strong></p><ul><li>The client encodes and sends messages in custom packets (data_flg flag).</li><li>The server extracts and processes the message payload.</li></ul><p><strong>3. Packet Monitoring</strong></p><ul><li>The sniffer decodes packets to display IP and custom protocol details.</li></ul><h3>Key Learnings</h3><ul><li><strong>Protocol Design</strong>: Understanding and implementing communication protocols.</li><li><strong>Raw Sockets</strong>: Building low-level network applications.</li><li><strong>Packet Sniffing</strong>: Using tools like libpcap for debugging and analysis.</li></ul><h3>Challenges and Tips</h3><ol><li><strong>Permissions</strong>: Raw sockets require root privileges.</li><li><strong>Error Handling</strong>: Always check return values of socket and library functions.</li><li><strong>Protocol Testing</strong>: Use packet sniffers to debug and validate custom protocol behavior.</li></ol><h3>Conclusion</h3><p>This project is an excellent introduction to network programming and protocol design. By implementing a simplified TCP protocol, you gain hands-on experience with low-level networking and system programming concepts. Feel free to expand this project by adding features like reliability, congestion control, or encryption!</p><p>Code for this TCP program : <a href="https://github.com/lmalkam/Advanced-Networks/tree/master/ENDLAB">https://github.com/lmalkam/Advanced-Networks/tree/master/ENDLAB</a></p><p>Github Repository for more codes : <a href="https://github.com/lmalkam/Advanced-Networks">https://github.com/lmalkam/Advanced-Networks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45da0f7d71df" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>