<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sambit Mahapatra on Medium]]></title>
        <description><![CDATA[Stories by Sambit Mahapatra on Medium]]></description>
        <link>https://medium.com/@sambit9238?source=rss-241751fe4cf9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*poYd7UPh07SiMlM5cAitEA.png</url>
            <title>Stories by Sambit Mahapatra on Medium</title>
            <link>https://medium.com/@sambit9238?source=rss-241751fe4cf9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:06:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sambit9238/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MCP in Action: Tools, Resources, and Prompts for a Productive Assistant]]></title>
            <link>https://sambit9238.medium.com/mcp-in-action-tools-resources-and-prompts-for-a-productive-assistant-2c88ec7ae190?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c88ec7ae190</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 15:08:55 GMT</pubDate>
            <atom:updated>2025-08-22T15:24:43.339Z</atom:updated>
            <content:encoded><![CDATA[<h4>MCP servers have 3 building blocks</h4><ul><li><strong>Tools</strong>: actions the AI can take autonomously during a conversation — like “search emails,” “find free time,” or “create a doc.” The model decides when to call them and with what arguments.</li><li><strong>Resources</strong>: curated data the application fetches and supplies — like recent email threads, working hours for scheduling, or an agenda template. The app controls when and how these are used (often for UI or prompt context).</li><li><strong>Prompts</strong>: predefined, user-triggered workflows — one-click starters such as “Plan my project sync” that chain the right tools in the right order with sensible defaults.</li></ul><p>Together, these three let users kick off reliable workflows, the app provide safe and relevant context, and the AI execute concrete actions end to end.</p><p>Suppose we are building an AI app, which can:</p><ul><li>Read emails</li><li>Check and block calendar time</li><li>Create Google Docs with meeting agendas</li></ul><blockquote>User intent example:</blockquote><blockquote>“Plan my next project sync — summarize relevant emails, find a 60-minute slot next week, and create a shared agenda doc.”</blockquote><h4>1. Tools: LLM controlled actions</h4><p>Think of Tools as the assistant’s hands. They’re typed functions the LLM can call with arguments. When the LLM sees a task that needs external data or an action, it selects the right tool and proposes parameters.</p><p>Examples for this assistant:</p><p><strong>Email tools</strong>:</p><ul><li>email.search({ query: “project X”, since: “2025–08–01” })</li><li>email.get_message({ id: “abc123” })</li></ul><p><strong>Calendar tools</strong>:</p><ul><li>calendar.find_slots({ duration_min: 60, window: “next week”, attendees: […] })</li><li>calendar.create_event({ start, end, attendees, title, description })</li></ul><p><strong>Docs tools</strong>:</p><ul><li>gdocs.create_doc({ title: “Project Sync Agenda — YYYY-MM-DD” })</li><li>gdocs.write_content({ doc_id, content: “Objectives, Topics, Decisions, Action Items” })</li><li>gdocs.share_doc({ doc_id, emails: […], role: “commenter” })</li></ul><h4>Why Tools matter:</h4><ul><li>Autonomy: the AI can “do” things, not just talk about them.</li><li>Consistency: standardized inputs/outputs make calls predictable.</li><li>Composability: multiple tools can be chained to complete a workflow.</li></ul><h4>Practical guardrails:</h4><ul><li>The server decides which tools to expose.</li><li>Inputs are validated and can be augmented (e.g., default time zones, safe limits).</li><li>Sensitive operations can require explicit user confirmation.</li></ul><h4>2. Resources: app-controlled context and UI data</h4><p>Resources are how the application guides the assistant and keeps context safe, focused, and useful. They’re not invoked directly by the LLM; the app fetches them at the right time to power the UI or augment prompts.</p><p><strong>Examples for this assistant</strong>:</p><ul><li><strong>Email resources</strong>: recent threads, labels/folders, small curated snippets to seed the model.</li><li><strong>Calendar resources</strong>: working hours, holidays, time-zone rules, required attendees.</li><li><strong>Docs resources</strong>: recent files/folders for a picker, an agenda template, meeting notes skeletons.</li></ul><p><strong>Where Resources help most</strong>:</p><ul><li><strong>Better UX</strong>: autocomplete options, searchable pickers, sensible defaults.</li><li><strong>Reduced noise</strong>: only supply what’s relevant, minimize token usage and exposure.</li><li><strong>Safety</strong>: the app controls which data goes into the model’s context and how much.</li></ul><p><strong>Practical patterns</strong>:</p><ul><li>Preload small, targeted metadata for the UI.</li><li>Inject only the necessary excerpts into the prompt.</li><li>Keep a strict cap on size and sensitivity; redact as needed.</li></ul><h4>3. Prompts: user-controlled, predefined workflows</h4><p>Prompts are the shortcuts users click. They’re named workflows that chain Tools with best-practice defaults, so common tasks are fast, consistent, and auditable.</p><p>Examples for this assistant:</p><ul><li><strong>“Summarize emails for next sync”</strong>: email.search → email.get_message → model summary</li><li><strong>“Find a 60-minute slot next week”</strong>: calendar.find_slots → propose top options → (optional) confirm → calendar.create_event</li></ul><p>Why Prompts matter:</p><ul><li><strong>Reliability</strong>: the same steps run the same way every time.</li><li><strong>User control</strong>: people initiate the workflow and can review before committing.</li><li><strong>Speed</strong>: one click sets up multi-step actions with smart defaults.</li></ul><p>Practical touches:</p><ul><li>Ask for confirmation before sending invites or sharing docs.</li><li>Offer editable parameters (duration, attendees, template) before execution.</li><li>Show progress and results inline (created event, doc link, summary).</li></ul><h3>How they work together: an end-to-end flow</h3><ol><li><strong>The user clicks “Plan my project sync”</strong> (Prompt).</li><li><strong>The app fetches Resources</strong>: Recent email threads, working hours, preferred attendees, an agenda template.</li><li><strong>The LLM executes Tools</strong>:</li></ol><ul><li>Reads and summarizes relevant emails.</li><li>Finds a 60-minute slot next week; proposes top options.</li><li>On confirmation, creates the calendar event.</li><li>Creates a Google Doc agenda from the template, inserts the email-derived topics, and shares it with attendees.</li></ul><p>4. <strong>The app returns the final result</strong>: Summary of key email points, confirmed meeting time, and the doc reference.</p><p>Each component stays in its lane:</p><ul><li>Prompts standardize the workflow.</li><li>Resources provide safe, curated context.</li><li>Tools let the model perform the actual operations.</li></ul><p><strong>In short,</strong></p><ul><li><strong>Tools are the AI’s actions.</strong></li><li><strong>Resources are the app’s curated context and UI data.</strong></li><li><strong>Prompts are the user’s one-click workflows.</strong></li></ul><p>Reference: <a href="https://anthropic.skilljar.com/introduction-to-model-context-protocol">https://anthropic.skilljar.com/introduction-to-model-context-protocol</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c88ec7ae190" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MCP in Action: End-to-End Assistant Workflow Explained]]></title>
            <link>https://sambit9238.medium.com/mcp-in-action-end-to-end-assistant-workflow-explained-f3ead1aab82f?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3ead1aab82f</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 14:41:59 GMT</pubDate>
            <atom:updated>2025-08-22T14:41:59.118Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>What is MCP?</strong></p><p>MCP stands for Model Context Protocol. It is a communication layer that gives LLMs capability to securely access the context and tools they need — without building a bunch of tedious, custom integrations.</p><p><strong>Why it matters?</strong></p><p>Suppose we are building an AI app, which can:</p><ul><li>Read emails</li><li>Check and block calendar time</li><li>Create Google Docs with meeting agendas</li></ul><p>Traditionally, that means writing and maintaining separate integrations:</p><ul><li>One to read emails</li><li>Another to create calendar events</li><li>Another to write Google Docs</li><li>Plus Logic to pass data in between them</li></ul><p><em>That’s slow, breakable, and repetitive.</em></p><h4>How MCP changes this</h4><p>With MCP, the LLM talks to dedicated “MCP servers” for each system:</p><ul><li>Email MCP server for reading emails</li><li>Calendar MCP server for scheduling</li><li>Google Docs MCP server for writing agendas</li></ul><p>The LLM requests what it needs via the protocol, and each MCP server handles the heavy lifting with its underlying service. No custom, one-off integrations required — just a consistent, standardized way to connect.</p><h4>The result</h4><ul><li>Faster development: plug into standardized MCP servers instead of coding bespoke connectors.</li><li>Cleaner architecture: one protocol, many tools.</li><li>More capable AI: the model can pull real context and take actions across systems reliably.</li></ul><blockquote>Let’s say a user asks “Plan my Saturday”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rKvv7wpreqj1DWeB9BKLWA.png" /></figure><ol><li><strong>User → Server: “Plan my Saturday”</strong>: The user submits a natural-language request to the AI app server.</li><li><strong>Server → MCP Client: “List tools for LLM”</strong>: The server asks the MCP client to enumerate which tools are available right now (capabilities from connected MCP servers).</li><li><strong>MCP Client → Server: “Here’s the tool list”</strong>: The client returns a catalog (e.g., email.search, calendar.find_slots, gdocs.create_doc, etc.).</li><li><strong>Server → LLM: “Query + tools”</strong>: The server sends the user’s query plus the available tool definitions/schemas to the model so it knows what it’s allowed to use.</li><li><strong>LLM → Server: “Tools and arguments to use”</strong>: The LLM decides which tool(s) it needs and proposes arguments matching each tool’s schema (e.g., calendar.find_slots: { duration: 60, window: “next Saturday” }).</li><li><strong>Server → MCP Client: “Call these tools with these args”</strong>: The server instructs the MCP client to execute the selected tool calls with the proposed (and possibly validated/augmented) arguments.</li><li><strong>MCP Client → MCP Servers: “Request to server”</strong>: The MCP client sends a CallToolRequest to the appropriate MCP server (e.g., Calendar MCP Server). This is where the tool name and JSON arguments are handed off.</li><li><strong>MCP Servers → MCP Client: “Response from server”</strong>: The MCP server runs the real external API call (e.g., Google Calendar API), validates responses, and returns a CallToolResult to the MCP client.</li><li><strong>MCP Client → Server: “Results after running the tool”</strong>: The client passes the tool results back to the AI app server (e.g., available time slots, created event details, document ID/URL).</li><li><strong>Server → LLM: “Tool results”</strong>: The server provides the returned data to the model so it can synthesize a helpful, human-readable answer.</li><li><strong>Server → User: “Response to user”</strong>: The server delivers the final response (e.g., “Here’s your Saturday plan: event scheduled 3–4 pm; agenda doc created; summary of emails included”).</li></ol><h4>Why this flow is useful</h4><ul><li><strong>Clear separation of concerns</strong>: reasoning (LLM), orchestration (server), execution (MCP client/servers).</li><li><strong>Minimal custom integration</strong>: services are accessed through standardized MCP tools.</li><li><strong>Safety and control</strong>: the server controls what tools are exposed; arguments can be validated before execution.</li><li><strong>Composability</strong>: adding a new capability is adding a new MCP server/tool, not re-architecting the app.</li></ul><p>Reference:</p><p><a href="https://anthropic.skilljar.com/introduction-to-model-context-protocol">https://anthropic.skilljar.com/introduction-to-model-context-protocol</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3ead1aab82f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transforming Enterprise PDFs for LLM and RAG Applications]]></title>
            <link>https://sambit9238.medium.com/transforming-enterprise-pdfs-for-llm-and-rag-applications-a-guide-to-three-python-libraries-8a32f4249dfb?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a32f4249dfb</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[pdf]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Sat, 30 Nov 2024 17:27:09 GMT</pubDate>
            <atom:updated>2024-11-30T17:31:22.845Z</atom:updated>
            <content:encoded><![CDATA[<p>In various industries — such as construction, legal, and finance — critical information is often embedded within PDFs containing intricate tables and forms. To harness this data for Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) applications, converting PDFs into structured formats like Markdown is essential. This article explores three Python libraries — <strong>Docling</strong>, <strong>PyMuPDF4LLM</strong>, and <strong>pdfplumber</strong> — that facilitate this conversion with minimal setup.</p><p>We are going to experiment the performance of these libraries in parsing a pdf on this document: <a href="https://ada.nv.gov/uploadedFiles/adanewnvgov/content/home/features/ADAClasses/Merged%20Cell%20Table.pdf">https://ada.nv.gov/uploadedFiles/adanewnvgov/content/home/features/ADAClasses/Merged%20Cell%20Table.pdf</a></p><ol><li><a href="https://github.com/DS4SD/docling"><strong>Docling</strong></a></li></ol><p><strong>Docling</strong> is a versatile tool designed to read popular document formats and export them to Markdown and JSON. It offers advanced PDF understanding, including page layout analysis and table structure recovery, making it suitable for complex documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aiDvTJNM7bbj63ODc-gtvA.png" /></figure><p>The output markdown does look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7comJ-isdnABtUJ_pyuW6w.png" /></figure><p><strong>2. </strong><a href="https://pymupdf.readthedocs.io/en/latest/pymupdf4llm/"><strong>PyMuPDF4LLM</strong></a></p><p><strong>PyMuPDF4LLM</strong> is a high-level wrapper of PyMuPDF functions that converts PDF pages to text in Markdown format. It detects standard text and tables, arranging them in the correct reading sequence for GitHub-compatible Markdown text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DQ52HCqXO_RF9AplvoxL7A.png" /></figure><p>The output markdown does look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XKQL49JTANBIlBQcX2E8Dg.png" /></figure><p><strong>3. </strong><a href="https://github.com/jsvine/pdfplumber"><strong>pdfplumber</strong></a></p><p><strong>pdfplumber</strong> is a Python library that allows for the extraction of text, tables, and metadata from PDF files. While it doesn’t directly convert PDFs to Markdown, it enables users to extract content, which can then be formatted into Markdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQIDzZ94MoMo-D81UGIZrQ.png" /></figure><p>The output markdown does look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwGYjiW11pSnCd-GxppZ8A.png" /></figure><h4>Performance Comparison</h4><p>In a test with a one-page PDF:</p><ul><li><strong>Docling</strong> took approximately 6 seconds to process.</li><li><strong>PyMuPDF4LLM</strong> and <strong>pdfplumber</strong> each took about 0.1 seconds.</li></ul><p>Regarding output quality, <strong>PyMuPDF4LLM</strong> provided the most straightforward implementation and yielded better results, though it did not include column names properly in the table.</p><h3>Conclusion</h3><p>Each of these libraries offers unique advantages for converting PDFs to Markdown, catering to different needs and document complexities. It needs more testing on different layouts of pdfs and different sizes of pdfs to better judge the pros and cons of each option in different scenarios. As new solutions emerge, we will continue to update this guide to assist in the evolving challenge of efficient PDF to Markdown conversion. Please put in comment any other pure python library suggestion you have for this task.</p><p>Reference for pdfplumber processing: <a href="https://dev.to/rishabdugar/pdf-extraction-retrieving-text-and-tables-together-using-python-14c2">https://dev.to/rishabdugar/pdf-extraction-retrieving-text-and-tables-together-using-python-14c2</a></p><p><a href="https://github.com/sambit9238/Deep-Learning/blob/master/pdf_to_markdown.ipynb">Deep-Learning/pdf_to_markdown.ipynb at master · sambit9238/Deep-Learning</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a32f4249dfb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recurrent Circle of Machine Learning]]></title>
            <link>https://medium.com/geekculture/recurrent-circle-of-machine-learning-5aab2ab1a4ac?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/5aab2ab1a4ac</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Mon, 29 Mar 2021 19:30:43 GMT</pubDate>
            <atom:updated>2021-03-30T21:20:34.497Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*cUQ9eSrby9QQeRCXw3jOSA.png" /></figure><h4>How to manage a Machine Learning project?</h4><p>Machine Learning is the new tool to achieve software 2.0 of a product by shifting it from rule-driven to data-driven. Machine Learning enables the apps and services to go from only interactive to personalized.</p><p>Despite how hot and trendy of a topic Machine Learning is, in reality, the failure rates of these projects are shocking. Some surprising figures on the failure rates of Machine Learning projects:</p><ul><li><a href="https://blogs.gartner.com/andrew_white/2019/01/03/our-top-data-and-analytics-predicts-for-2019/">Gartner has estimated that 85% of machine learning projects fail.</a></li><li><a href="https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/">VentureBeat AI says ‘87% of ML projects never make it to production’</a>.</li></ul><p>The major reason for these failures as well as the high cost involved with these is the disconnect between Machine Learning and Software development. Software development tends to be predictable and measurable. Its’ life cycle flows in one direction mostly, <strong>Requirements -&gt; Design -&gt; Implementation -&gt; Testing -&gt; Deployment.</strong></p><p>However, <strong>Machine Learning is still research </strong>and the expectation of a <strong>100% success rate is unrealistic</strong>. Its’ life cycle goes through a recurrent circle with multiple iterations and experimentation across. The so famous Agile-based project deliveries may not be expected. Asking Data Scientists or ML Engineers to present progress in every standup will only result in ‘if-else’ clauses instead of neural networks. Some important prerequisites before even planning for a Machine Learning project are :</p><ol><li>Finding the technical feasibility. (project to generate video from the text as a side feature for your product might not be technically feasible in 2021).</li><li>Clarity on success criteria. (Should have clarity how much accuracy, inference time, etc. sentiment analysis model should have to go to production, before starting on it.)</li><li>Clear expectations from leadership. (Business leaders need to understand how machine learning works and what business benefit the project will give.)</li></ol><p>Let’s discuss the recurrent circle of a Machine Learning project life cycle.</p><h4>Plan &amp; Setup</h4><p>The first step of starting the project is whether to start it or not? You neither need a JCB to root up tomato plants or Machine Learning to perform addition. Once convinced of the values it will be providing, a proper project objective should be there. What trade-offs of accuracy and speed are acceptable, what frameworks can be used, how to set up the code base, need all these answers at this stage.</p><h4>Data Preparation</h4><p>At this step we need to collect and gather data from various sources.</p><ol><li>Prepare and process the data as per the problem statement.</li><li>If it is a supervised problem, need ground truths or labels of each data point.</li><li>Build data ingestion pipeline.</li></ol><p>While doing all this, we might need to go back to <strong>Plan &amp; Setup</strong> phase for cases like:</p><ol><li><strong>It is too hard to get labeled data.</strong></li><li><strong>Easier to label a different task that can give the same business value.</strong></li></ol><h4>Model Preparation</h4><p>At this step, we have sufficient data in the desired format to build a Machine Learning model for the task we are aiming for. It comprises a couple of sub-steps like Model Building and Debugging, Model Training and Tuning, Testing and Evaluation.</p><p><em>In Model Building and Debugging,</em></p><ol><li>Establish a baseline model to measure performance. (e.g. for sequential task a simple LSTM, for image data LeNet kind architecture, etc.)</li><li>Find out the SOTA models for the task (if available) and try to reproduce them.</li><li>Find out what other people trying or any pre-trained model available for the task (if available)</li><li>Give quick tries to different isolated ideas at the early stage.</li><li>Implement the models and train with a few data points.</li><li>Debug implementation if any. (75% time it will be an issue with the versions of libraries or shapes of tensors.)</li></ol><p><em>In Model Training and Tuning,</em></p><ol><li>Get the required infrastructure to feed the whole data to train (decide on the resources and GPUs if needed).</li><li>Define the metric e.g. accuracy, precision, etc from the success criteria defined at the project planning stage.</li><li>Train the model and measure the metric on the validation set.</li><li>If under-fitted i.e. metric on training data is worse than baseline increase the complexity of the model by tuning necessary hyperparameters. It might be the number of layers or number of neurons in a layer etc.</li><li>If over-fitted i.e. metric on training data is awesome but not so good on validation data, which will be mostly the case, try to deal with it. You may try different regularisation methods or dropout methods. Maybe normalization (newly published works preferring it over regularization) will do better.</li><li>Iteratively keep debugging the models and uncover common failure modes.</li></ol><p><em>In Testing and Evaluation,</em></p><ol><li>Evaluate the model performance on test data distribution. The assumption is that your test data distribution mimics the real-world data.</li><li>Revisit the decided model evaluation metrics to confirm the performance of the model is up to the desired level.</li><li>Also test the<strong> input data pipeline, model inference functionality, inference on explicit scenarios expected in production.</strong></li></ol><p>While going through this step if realized, <strong>collected data not sufficient, the feature requirement is different or data labels are wrong</strong>, then iterate back to the <strong>Data Preparation</strong> stage.</p><p>If<strong> the task is too hard to solve or trade-off requirements are not achievable,</strong> then go back to the <strong>Plan &amp; Setup </strong>stage.</p><h4>Deployment</h4><p>At this stage, you have the data pipeline ready, the desired model is ready. Run the model in pilot mode for a couple of weeks, evaluate biases, deal with them, and roll-out the model to production.</p><p>At this stage, if you need <strong>more accuracy or improvement on any metric</strong> then get back to the <strong>Model Preparation</strong> stage again.</p><p>If there is a <strong>data mismatch between production and your test data, you gathered more data, or certain scenarios model always failing to predict correctly </strong>go back to Data Preparation stage and cover the steps again.</p><p>After considering all the steps,<strong> if performance is still not great either in terms of accuracy and speed, or the success criteria defined at planning time not correct for production</strong>, then it needs a redo of the whole cycle again from the <strong>Plan &amp; Setup</strong> stage.</p><p><strong>PS: </strong>Learning from our previous experiments at <a href="https://www.csat.ai/">CSAT.AI</a>, we always make sure before starting the project we properly define the accepted trade-off between accuracy, inference time, and inference cost. Being an engineer we might always crave for State of the Art things, but putting a BERT on a GPU for a 1% extra accuracy on sentiment analysis might not provide the desired business value.</p><h3>References:</h3><ul><li><a href="https://www.datarobot.com/wiki/machine-learning-life-cycle/">Machine Learning Life Cycle | DataRobot Artificial Intelligence Wiki</a></li><li><a href="https://fullstackdeeplearning.com/spring2021/lecture-5/">Full Stack Deep Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5aab2ab1a4ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/recurrent-circle-of-machine-learning-5aab2ab1a4ac">Recurrent Circle of Machine Learning</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[INDIA The Future Superpower]]></title>
            <link>https://sambit9238.medium.com/india-the-future-superpower-b880a2157ea8?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b880a2157ea8</guid>
            <category><![CDATA[india]]></category>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[independence-day]]></category>
            <category><![CDATA[world]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Fri, 14 Aug 2020 18:51:45 GMT</pubDate>
            <atom:updated>2020-08-14T18:56:05.247Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*swcPvTviybqKGaP75KYf4w.jpeg" /></figure><p>India is a culture rich ancient civilization that has sustained and evolved through a journey of thousand years. A civilization which adopted the farming 5000 years back, had smarter architect city like Harappa, Mohenjodaro as early as 3200BC around. Around the 5th century, when the world community was learning to have a civilized society, it had made the oldest knowledge center named as Nalanda University to innovate and thrive. Modern India which got independent in 1947 is a descendant of this esteem ancient civilization. This country, which has managed to thrive its culture and way of life through hundreds of years of foreign invasion be it in the form of Mughals or British. Present India is even better than ever be it in disrupting the Tech, advancing in the economy, guiding the world through Yoga, Ayurveda, or protecting humanity through upliftment of downtrodden.</p><p>Here are my top 5 reasons why I think India as going to be the future superpower:</p><h3>Economy</h3><p>Chinese economy might look solid presently, but it is mostly based on foreign investment which can burst anytime. Whereas Indian economic growth is organic and dependent more on local sectors. It is the world’s fifth-largest economy by nominal GDP and the third-largest by purchasing power parity currently. And heavy foreign investments have started from multinational companies like Apple, Google, Cisco, GM, etc to boost it further.</p><p>Importantly, if we ignore the corona scenario India has managed to grow its GDP at 7+% constantly while even China facing troubles. If the trend continues, it will definitely become the second-largest economy by 2050.</p><h3>Military</h3><p>India is a major military power in the world across the land (Army), sea (Navy), and air (Air Force). With a strength of over 1.4 million active personnel, it is the world’s second-largest military force and has the world’s largest volunteer army. As per global firepower ranking, it is the 4th mightiest force in the world. The Indian military has a yearly budget of around 45–50 billion dollars each year, which might not sound that impressive. What makes India unique is it is battle-hardened and has fought many wars in the mountain terrains where most army and airforce fails thanks to the geographical challenges.</p><p>The further modernization of the military is it the arrival of rafales or the addition of apache further boosts its capability. I believe spectacular indigenous developments of missiles and satellite technology will be making a big difference in the coming decade, thanks to DRDO, ISRO, etc..</p><h3>Technology</h3><p>India is rocking as far as science and technology are concerned. It is already the world’s leading producer of computer software. It is also supporting and investing in research and development centers and is slowly undergoing a revolution in the field of science and technology. In this new era, India is rapidly integrating the AI to mainstream to add further 1 trillion dollars to its economy.</p><p>India is adopting digitization through break-neck speed, be it digitizing the administrative works through apps like umang or encouraging technological innovations through smart hackathon programs and sites like <a href="https://indiaai.in/">https://indiaai.in/</a>. I believe the New Education Policy of 2020 will give a big boost to this field in the coming years.</p><h3>Culture</h3><p>Hinduism defines the culture of India. It believes in <strong>‘sarb dharm sama bhaba’ </strong>that allows a prosperous society if diverse people across caste, creed, and religion. Indians not only milk nature, but they also pray and protect it. The diversity in language, food, religion, beliefs, architectural monuments makes it an amazing destination for travelers. This millennium-old culture empowers Indians to maintain strong ethics and honesty in life. The world also loves to adopt this rich culture, be it through International Yoga Day or gradual adoption of Ayurveda.</p><h3>Foreign Affairs</h3><p>India’s diplomatic standing is magnificent. It has no international conflicts and maintains a good friend, as well as trade relationships across the world, be it with west or middle east or Russia. It has even helped other countries in need which is more noticeable in COVID time. It believes ‘Basudhaiva Kutumbakam’ i.e. the whole world is a family.</p><p>India’s biggest advantage is its stable democracy, freedom of speech, freedom of expression. The country with a population of more than 135 crores where around 35% are youth definitely showing a huge potential to make the world better by leading it.</p><ul><li><a href="https://en.wikipedia.org/wiki/India_as_an_emerging_superpower#:~:text=The%20Republic%20of%20India%20is,rate%20(mid%20year%20terms).">India as an emerging superpower</a></li><li><a href="https://www.ibef.org/economy/indian-economy-overview">Indian Economy: Overview, Market Size, Growth, Development, Statistics...IBEF</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b880a2157ea8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Activation Functions in Neural Network]]></title>
            <link>https://medium.com/swlh/activation-functions-in-neural-network-eb0ab4bb493?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb0ab4bb493</guid>
            <category><![CDATA[activation-functions]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Fri, 07 Aug 2020 16:58:15 GMT</pubDate>
            <atom:updated>2020-08-08T02:28:51.788Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_YcNaLjxCR87rKplKOT4uQ.jpeg" /></figure><h4>A concise analysis of roles and types of activation functions in neural networks. Detailed pros and cons of 5 popularly used ones in an Artificial Neural Network.</h4><p>Activation function, as the name suggests, decides whether a neuron should be activated or not based on the addition of a bias with the weighted sum of inputs. Hence, it is a very significant component of Deep Learning, as they in a way determine the output of models. The activation function has to be efficient so the model can scale along the increase in the number of neurons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtC6qvZ300Elg0XOcaC6Bw.png" /></figure><p>To be precise, the activation function decides how much information of the input relevant for the next stage.</p><p>For example, suppose x1 and x2 are two inputs with w1 and w2 their respective weights to the neuron. The output Y = activation_function(y).</p><p>Here, y = x1.w1 + x2.w2 + b i.e. weighted sum of inputs and bias.</p><p>Activation functions are mainly of 3 types. We will analyze the curves, pros and cons of each here. The input we work on will be an arithmetic progression in [-10, 10] with a constant difference of 0.1</p><pre>x = tf.Variable(tf.range(-10, 10, 0.1), dtype=tf.float32)</pre><h4>Binary step</h4><p>A binary step function is a threshold-based activation function. If the input value is above or below a certain threshold, the neuron is activated and sends exactly the same signal to the next layer.</p><pre>#Binary Step Activation<br>def binary_step(x):<br>    return np.array([1 if each &gt; 0 else 0 for each in list(x.numpy())])<br>do_plot(x.numpy(), binary_step(x), &#39;Binary Step&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*hBxj14DgOO2Yh4eVGvX2gA.png" /></figure><p>The binary step is not used mostly because of two reasons. Firstly it allows only 2 outputs that don’t work for multi-class problems. Also, it doesn’t have a derivative.</p><h4>Linear</h4><p>As the name suggests, the output is a linear function of the input i.e. y = cx</p><pre>#Linear Activation<br>def linear_activation(x):<br>    c = 0.1<br>    return c*x.numpy()<br>do_plot(x.numpy(), linear_activation(x), &#39;Linear Activation&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*QxkbKI8kQyRx2e3ytxB5JA.png" /></figure><p>The linear activation function is also not used is neural networks because of two main reasons.</p><p>Firstly, with this activation at multiple layers, the last output just becomes a linear function of the input. That defeats the purpose of multiple neurons and layers.</p><p>Secondly, as y = cx, the derivative becomes dy/dx = c, which is a constant. So it can’t be used in backpropagation to train neural networks.</p><h4>Non-linear</h4><p>Non-linear activation functions are used everywhere in neural networks as they capture the complex pattern in data due to the non-linearity nature and support back-propagation as they have a derivative function.</p><p>Here we discuss 5 popularly used activation functions.</p><ol><li><strong><em>Sigmoid</em></strong></li></ol><p>Sigmoid function squashes the output between 0 to 1 and the function looks like sigmoid(x) = 1 / (1 + e^(−x))</p><pre>y = tf.nn.sigmoid(x)<br>do_plot(x.numpy(), y.numpy(), &#39;Sigmoid Activation&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*Mcfp_lTsreNfdhaJh_lUmw.png" /></figure><p>The major advantage of this function is that its gradient is smooth and output always lies between 0 to 1.</p><p>It has a few cons like, the output always lies between 0 to 1 which is not suitable for multi-class problems. With multiple layers and neurons, the training gets slower as it is computationally expensive.</p><pre>with tf.GradientTape() as t:<br>    y = tf.nn.sigmoid(x)<br>do_plot(x.numpy(), t.gradient(y, x).numpy(), &#39;Grad of Sigmoid&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*gyV7jYySszXh9274caUrIQ.png" /></figure><p>Also as seen in the gradient graph, it suffers from vanishing gradient problem. The input changes from -10 to -5 and 5 to 10, but the gradient output doesn’t change.</p><p><em>2. </em><strong><em>Tanh</em></strong></p><p>Tanh activation function is more like sigmoid but it squashes output between -1 to 1. The function looks like tanh(x) =(1-e^(-2x))/(1+e^(2x))</p><pre>y = tf.nn.tanh(x)<br>do_plot(x.numpy(), y.numpy(), &#39;Tanh Activation&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*6z6k7Jpibggpy6qnfIachw.png" /></figure><p>In addition to the advantages of the sigmoid function, its output is 0 centered.</p><p>With multiple layers and neurons, the training gets slower as it is also computationally expensive.</p><pre>with tf.GradientTape() as t:<br>    y = tf.nn.tanh(x)<br>do_plot(x.numpy(), t.gradient(y, x).numpy(), &#39;Grad of Tanh&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*lV8nODOri31sd4GcquKtBQ.png" /></figure><p>As seen in the gradient graph, it also suffers from a vanishing gradient problem. Here the input gets out of [-2.5, 2.5], but the gradient output doesn’t change no matter how much the input changes.</p><p><em>3.</em><strong><em> ReLU</em></strong></p><p>ReLU or Rectified Linear Unit either passes the information further or completely blocks it. The function looks like relu(x) = max(0, x)</p><pre>y = tf.nn.relu(x)<br>do_plot(x.numpy(), y.numpy(), &#39;ReLU Activation&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*m-zGWavVjLVRT1BMFh4F2g.png" /></figure><p>It is most popular due to its simplicity and non-linearity. Its derivatives are particularly well behaved: either they vanish or they just let the argument through.</p><pre>with tf.GradientTape() as t:<br>    y = tf.nn.relu(x)<br>do_plot(x.numpy(), t.gradient(y, x).numpy(), &#39;Grad of ReLU&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*ajahrLFDSrdVi_PyV7SqrQ.png" /></figure><p>One disadvantage it has is that it doesn’t retain negative inputs thus if inputs are negative, it doesn’t learn anything. That is called the <strong>dying ReLU</strong> problem.</p><p><em>4. </em><strong><em>Softmax</em></strong></p><p>Softmax activation function gives output in terms of probability and the number of outputs is equal to the number of inputs. The function looks like,</p><p>softmax(xi) = xi / sum(xj)</p><pre>x1 = tf.Variable(tf.range(-1, 1, .5), dtype=tf.float32)<br>y = tf.nn.softmax(x1)</pre><p>The major advantage of this activation function is it gives multiple outputs that make it popularly used in the output layer of a neural network. It makes it easier to classify multiple categories.</p><p>The main limitation of this algorithm is that it won’t work if data is not linearly separable. Another limitation is that it does not support null rejection, so you need to train the algorithm with a specific null class if you need one.</p><p>5. <strong>Swish</strong></p><p>Google Brain Team has proposed this activation function, named Swish. The function looks like swish(x) = x.sigmoid(x)</p><p>According to their <a href="https://arxiv.org/abs/1710.05941v1">paper</a>, it performs better than ReLU with a similar level of computational efficiency.</p><pre>y = tf.nn.swish(x)<br>do_plot(x.numpy(), y.numpy(), &#39;Swish Activation&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*aJBGuPJA1xbtyRTDSWOYjQ.png" /></figure><p>One reason swish might be performing better than ReLU is it addresses the dying ReLU issue as shown in the below graph.</p><pre>with tf.GradientTape() as t:<br>    y = tf.nn.swish(x)<br>do_plot(x.numpy(), t.gradient(y, x).numpy(), &#39;Grad of Swish&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*kHZe8PPNQUF82jF4GK3gLg.png" /></figure><p>On an additional note, ReLU has got other variances of its type also which are quite popular like leaky ReLU, parametric ReLU, etc.</p><ul><li><a href="https://github.com/sambit9238/Deep-Learning/blob/master/activation_function.ipynb">sambit9238/Deep-Learning</a></li><li><a href="https://towardsdatascience.com/activation-functions-in-neural-networks-83ff7f46a6bd">Activation Functions in Neural Networks</a></li><li><a href="https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/">7 Types of Activation Functions in Neural Networks: How to Choose?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb0ab4bb493" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/activation-functions-in-neural-network-eb0ab4bb493">Activation Functions in Neural Network</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Albert Vectorization (With Tensorflow Hub)]]></title>
            <link>https://medium.com/data-science/albert-vectorization-with-tensorflow-hub-c6b892ed7ed4?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6b892ed7ed4</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Sat, 11 Jul 2020 19:02:05 GMT</pubDate>
            <atom:updated>2020-07-12T18:39:54.229Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCPNtiH8SxU0RD9crIjbzA.jpeg" /></figure><h4>We vectorize input texts using pre-trained Albert embeddings from tensorflow-hub. It is used as a Keras layer, so can be extended to build deep learning models easily.</h4><p>The advent of BERT has disrupted the traditional paradigm of NLP. The downstream model building is now from a model primed with knowledge of a language rather than from scratch. ALBERT can be called a lite BERT with a greatly reduced number of parameters that use a transformer-encoder architecture. In the paper, vocabulary size is also of 30K as used in the original BERT. Three major things distinct ALBERT from BERT are Factorized Embedding Parameterization, Cross-Layer Parameter Sharing, Sentence Order Prediction.</p><h3>Advantage over BERT</h3><p>BERT like models can provide a poor-quality performance when one tries to simply enlarge the hidden size of the model. Parameter-reduction technique such as factorized embedding parameterization is used to separate the size of the hidden layers from the size of vocabulary embedding which makes it easy to grow the hidden size without significantly increasing the parameter size. While the cross-layer parameter sharing prevents the parameter from growing with the depth of the network. Thus, both the techniques significantly reduce the number of parameters for traditional BERT without worsening the performance and improving parameter-efficiency. The performance of ALBERT is further improved by introducing a self-supervised loss for sentence-order prediction (SOP).</p><p>In this article, we will get the ALBERT vectors of corresponding text using keras layer format of tensorflow-hub module.</p><h3>1. Setup Environment</h3><p>We will be making a model using tensorflow keras API which will give the ALBERT embedding for text input. The environment setup includes installing the required libraries and getting required tensorflow-hub modules to get ALBERT vectors. The libraries used are:</p><pre>#tensorflow==2.2.0<br>#tensorflow_hub=0.8.0<br>#sentencepiece==0.1.91<br>#tqdm==4.47.0<br>#seaborn==0.10.1<br>#numpy==1.19.0</pre><p>We will be using TF2 SavedModel format of ALBERT. You can directly use the module hosted on tfhub for the inference. But for the production scenario, having the module at local will be preferable. For that, we need to first get the zip file of the module and unzip it.</p><pre># mkdir albert_en_base<br># mkdir 1<br># wget <a href="https://tfhub.dev/tensorflow/albert_en_base/1https://tfhub.dev/tensorflow/albert_en_base/1">https://tfhub.dev/tensorflow/albert_en_base/1https://tfhub.dev/tensorflow/albert_en_base/1</a><br># tar-xzf albert_en_base_1.tar.gztar-xzf albert_en_base_1.tar.gz<br># rm -rf albert_en_base_1.tar.gzrm -rf albert_en_base_1.tar.gz</pre><h3>2. Tokenizing</h3><p>We will import the ALBERT module as a keras layer.</p><pre>albert_layer = hub.KerasLayer(&quot;albert_en_base/1&quot;, trainable=False)</pre><p>The tricky part is, ALBERT module can’t be directly fed with the text. It needs to go through a preprocessing layer.</p><p>First, we will tokenize the input text using ALBERT tokenizer which is based on a sentencepiece tokenizer i.e. a subword level tokenizer. It is a data-driven tokenizer to take care of the out of vocabulary words. Since we will have only one text at each input the token list of sentences would look like [“[CLS]”] +TOKENS+ [“[SEP]”].</p><pre>sp_model_file = albert_layer.resolved_object.sp_model_file.asset_path.numpy()<br>tokenizer = FullSentencePieceTokenizer(sp_model_file)<br>stokens = tokenizer.tokenize(sentence)<br>stokens = stokens[:MAX_LEN]<br>stokens = [&quot;[CLS]&quot;] + stokens + [&quot;[SEP]&quot;]</pre><p>For example, the text “the body is made of metallic and delivers high tension” generate a token list like:</p><pre>[&#39;▁the&#39;,<br> &#39;▁body&#39;,<br> &#39;▁is&#39;,<br> &#39;▁made&#39;,<br> &#39;▁of&#39;,<br> &#39;▁metallic&#39;,<br> &#39;a&#39;,<br> &#39;▁and&#39;,<br> &#39;▁deliver&#39;,<br> &#39;i&#39;,<br> &#39;▁high&#39;,<br> &#39;▁tension&#39;]</pre><p>Now we need 3 input sequences that can be fed to ALBERT module.</p><ul><li>Token ids: for every token in the sentence which is fetched from the ALBERT vocab dictionary.</li><li>Mask ids: for every token to mask out tokens used only for the sequence padding (so every sequence has the fixed length).</li><li>Segment ids: 0 for one-sentence sequence (our case here), 1 if there are two sentences in the sequence.</li></ul><pre>def get_ids(tokens, tokenizer, max_seq_length):<br>  &quot;&quot;&quot;Token ids from Tokenizer vocab&quot;&quot;&quot;<br>  token_ids = tokenizer.convert_tokens_to_ids(tokens,)<br>  input_ids = token_ids + [0] * (max_seq_length-len(token_ids))<br>  return input_ids</pre><pre>def get_masks(tokens, max_seq_length):<br>  return [1]*len(tokens) + [0] * (max_seq_length - len(tokens))</pre><pre>def get_segments(tokens, max_seq_length):<br>  &quot;&quot;&quot;Segments: 0 for the first sequence, 1 for the second&quot;&quot;<br>  segments = []<br>  current_segment_id = 0<br>  for token in tokens:<br>    segments.append(current_segment_id)<br>    if token == &quot;[SEP]&quot;:<br>      current_segment_id = 1<br>  return segments + [0] * (max_seq_length - len(tokens))</pre><pre>ids = get_ids(stokens, tokenizer, MAX_SEQ_LEN)<br><strong># [2, 13, 1, 589, 17378, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]</strong></pre><pre>masks = get_masks(stokens, MAX_SEQ_LEN)<br><strong># [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]</strong></pre><pre>segments = get_segments(stokens, MAX_SEQ_LEN)<br><strong># [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]</strong></pre><h3>3. Albert Vectorization</h3><p>Now we are ready with the required input processing now. We will build a model using tf keras API which will take the processed input and get the ALBERT vectors of the text. The output ALBERT vectors contain 2 vectors, one is pooled_output and sequence_output. The pooled_output is the sentence embedding of the dimension 1x768 and the sequence output is the token level embedding of the dimension 1x(token_length)x768.</p><pre>def get_model():<br>  input_word_ids = tf.keras.layers.Input(shape=(MAX_SEQ_LEN,), dtype=tf.int32,name=&quot;input_word_ids&quot;)<br>  input_mask = tf.keras.layers.Input(shape=(MAX_SEQ_LEN,), dtype=tf.int32,name=&quot;input_mask&quot;)<br>  segment_ids = tf.keras.layers.Input(shape=(MAX_SEQ_LEN,), dtype=tf.int32,name=&quot;segment_ids&quot;)</pre><pre>  pooled_output, sequence_output = albert_layer([input_word_ids, input_mask, segment_ids])</pre><pre>  model = tf.keras.models.Model(inputs=[input_word_ids, input_mask,  segment_ids], outputs=[pooled_output, sequence_output])<br>  return model</pre><p>The idea behind making a keras model is to extend it easily for any classification models with the required addition of layers and parameters.</p><p>Now all the hard work is done. Will just need an inference function to get corresponding ALBERT embeddings of the texts.</p><pre>s = &quot;This is a nice sentence.&quot;</pre><pre>def get_albert_vec(s):<br>  stokens = tokenizer.tokenize(s)<br>  stokens = [&quot;[CLS]&quot;] + stokens + [&quot;[SEP]&quot;]<br>  ids = get_ids(stokens, tokenizer, MAX_SEQ_LEN)<br>  masks = get_masks(stokens, MAX_SEQ_LEN)<br>  segments = get_segments(stokens, MAX_SEQ_LEN)<br>  input_ids = np.asarray(ids, dtype=np.int32).reshape(1,22)<br>  input_masks = np.asarray(masks, dtype=np.int32).reshape(1,22)<br>  input_segments = np.asarray(segments, dtype=np.int32).reshape(1,22)<br>  return input_ids, input_masks, input_segments</pre><pre>input_ids, input_masks, input_segments = get_albert_vec(s)<br>pool_embs, word_embs = albert_model.predict(<br>      [[input_ids, input_masks, input_segments]]<br>)</pre><p>To quickly check the quality of sentence embeddings let’s run it on a small set of examples and check the sentence similarity score of each pair using dot product of their corresponding normalized sentence embedding vectors.</p><pre>sentences = [<br># Smartphones<br>&quot;I like my phone&quot;,<br>&quot;My phone is not good.&quot;,</pre><pre># Weather<br>&quot;Recently a lot of hurricanes have hit the US&quot;,<br>&quot;Global warming is real&quot;,</pre><pre># Asking about age<br>&quot;How old are you?&quot;,<br>&quot;what is your age?&quot;]</pre><pre>run_and_plot(sentences)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*MgCEGzIvbJ1xiYXNDKA1kQ.png" /></figure><h3><strong>References</strong></h3><ul><li><a href="https://github.com/sambit9238/Deep-Learning/tree/master/sarcasm_classifier/albert_tfhub">sambit9238/Deep-Learning</a></li><li><a href="https://medium.com/doxastar/from-bert-to-albert-pre-trained-langaug-models-5865aa5c3762">From Bert to AlBert: Pre-trained Langaug Models</a></li><li><a href="https://ai.googleblog.com/2019/12/albert-lite-bert-for-self-supervised.html">ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations</a></li><li><a href="https://analyticsindiamag.com/googles-nlp-powered-pretraining-method-albert-is-leaner-meaner/">Google&#39;s NLP-Powered Pretraining Method ALBERT Is Leaner &amp; Meaner</a></li><li><a href="https://amitness.com/2020/02/albert-visual-summary/">Visual Paper Summary: ALBERT (A Lite BERT)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6b892ed7ed4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/albert-vectorization-with-tensorflow-hub-c6b892ed7ed4">Albert Vectorization (With Tensorflow Hub)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sarcasm Classification (Using FastText)]]></title>
            <link>https://medium.com/data-science/sarcasm-classification-using-fasttext-788ffbacb77b?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/788ffbacb77b</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[fasttext]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Sun, 28 Jun 2020 17:20:09 GMT</pubDate>
            <atom:updated>2020-06-29T03:37:16.686Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDonoHq7uVnPsWa1Bjk5Fg.png" /></figure><h4>We will build a sarcasm classifier for news headlines using the FastText python module.</h4><p>FastText is a library created by the Facebook Research Team for <strong>efficient learning of word representations</strong> and <strong>sentence classification</strong>. It has gained a lot of attraction in the NLP community especially as a strong baseline for word representation replacing word2vec as it takes the char n-grams into account while getting the word vectors. Here we will be using FastText for text classification.</p><p>The data is collected from <a href="https://www.kaggle.com/rmisra/news-headlines-dataset-for-sarcasm-detection">https://www.kaggle.com/rmisra/news-headlines-dataset-for-sarcasm-detection</a>. Here the sarcastic ones are from <a href="https://www.theonion.com/"><em>TheOnion</em></a><em> </em>and nos-sarcastic ones are from<em> </em><a href="https://www.huffingtonpost.com/"><em>HuffPost</em></a>. Let’s now jump to coding.</p><p>First, let’s inspect the data to decide the approach. You can download the data from <a href="https://www.kaggle.com/rmisra/news-headlines-dataset-for-sarcasm-detection/data?select=Sarcasm_Headlines_Dataset_v2.json">https://www.kaggle.com/rmisra/news-headlines-dataset-for-sarcasm-detection/data?select=Sarcasm_Headlines_Dataset_v2.json</a>.</p><pre><em>#load data</em><br><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong><br>df = pd.read_json(&quot;Sarcasm_Headlines_Dataset_v2.json&quot;, lines=<strong>True</strong>)<br>#shuffle the data inplace<br>df = df.sample(frac=1).reset_index(drop=<strong>True</strong>)<br><em># show first few rows</em><br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*zCujhnhDVoHe3djUG1kDeA.png" /></figure><p>Basically, on reading the json in tabular form with pandas, the dataset contains 3 columns where ‘headline’ contains the headline texts of the news and ‘is_sarcastic’ contains 1 and 0 signifying sarcastic and non-sarcastic respectively. If we see the representation of sarcastic and non-sarcastic examples are</p><p>0 14985<br>1 13634<br>Name: is_sarcastic, dtype: int64</p><p>Now to just look at how the texts are looking for sarcastic and non-sarcastic examples:</p><pre>#word cloud on sarcastic headlines</pre><pre>sarcastic = ‘ ‘.join(df[df[‘is_sarcastic’]==1][‘headline’].to_list())</pre><pre>plot_wordcloud(sarcastic, ‘Reds’)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*p5M6Ig66_Q524iHjbjSUng.png" /></figure><pre><em>#word cloud on sarcastic headlines</em> sarcastic = &#39; &#39;.join(df[df[&#39;is_sarcastic&#39;]==0][&#39;headline&#39;].to_list()) plot_wordcloud(sarcastic, &#39;Reds&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*fLwBc7Z0XIIBUDvmjLgAWg.png" /></figure><p>Now before going to building the classifier model, we need to do some cleaning of the text to remove noise. Since these are the news headlines, they don’t contain much crap. The cleanings what I thought of having are getting all string lowercase, getting rid of anything which is not alphanumeric and replacing the numeric with a specific label.</p><pre>df[&#39;headline&#39;] = df[&#39;headline&#39;].str.lower()<br>df[&#39;headline&#39;] = df[&#39;headline&#39;].apply(alpha_num)<br>df[&#39;headline&#39;] = df[&#39;headline&#39;].apply(replace_num)</pre><p>Now, we are ready with the cleaned text and their corresponding labels to build a binary sarcasm classifier. As discussed, we will build the model using FastText python module.</p><p>First, need to install the FastText python module in the environment.</p><pre><em>#Building fasttext for python\</em> <br>!git clone https://github.com/facebookresearch/fastText.git <br>!cd fastText<br>!pip3 install .</pre><p>We need to make the training and testing file ready in the format which FastText api can understand. The default format of text file on which we want to train our model should be __ label __ &lt;Label&gt; &lt;Text&gt;. We can use other prefix instead of __label__ by changing the parameter accordingly in the training time, which we will see ahead.</p><pre><em>#data preparation for fasttext</em><br><strong>with</strong> open(&#39;fasttext_input_sarcastic_comments.txt&#39;, &#39;w&#39;) <strong>as</strong> f:<br>    <strong>for</strong> each_text, each_label <strong>in</strong> zip(df[&#39;headline&#39;], df[&#39;is_sarcastic&#39;]):<br>        f.writelines(f&#39;__label__<strong>{each_label}</strong> <strong>{each_text}\n</strong>&#39;)</pre><p>The data in the file looks like this:</p><pre>!head -n 10 fasttext_input_sarcastic_comments.txt</pre><pre>__label__1 school friends dont find camp songs funny<br>__label__0 what cutting americorps would mean for public lands<br>__label__0 when our tears become medicine<br>__label__1 craig kilborn weds self in private ceremony<br>__label__1 white couple admires fall colors<br>__label__1 mom much more insistent about getting grandkids from one child than other<br>__label__0 diary of a queer kids mom<br>__label__1 sephora makeup artist helping woman create the perfect pink eye<br>__label__1 kelloggs pulls controversial chocobastard from store shelves<br>__label__0 winston churchills grandson introduces a new nickname for donald trump</pre><p>Here __label__0 implies non-sarcastic and __label__1 implies sarcastic. Now we are in good shape to start training the classifier model. For that will divide the dataset into training(90%) and testing(10%) dataset. The FastText function to be used for this supervised binary classification is <em>train_supervised</em>.</p><pre><em>&#39;&#39;</em><br><em>For classification train_supervised call will be used:</em><br><br><em>The default parameters to it:</em><br><em>    input             # training file path (required)</em><br><em>    lr                # learning rate [0.1]</em><br><em>    dim               # size of word vectors [100]</em><br><em>    ws                # size of the context window [5]</em><br><em>    epoch             # number of epochs [5]</em><br><em>    minCount          # minimal number of word occurences [1]</em><br><em>    minCountLabel     # minimal number of label occurences [1]</em><br><em>    minn              # min length of char ngram [0]</em><br><em>    maxn              # max length of char ngram [0]</em><br><em>    neg               # number of negatives sampled [5]</em><br><em>    wordNgrams        # max length of word ngram [1]</em><br><em>    loss              # loss function {ns, hs, softmax, ova} [softmax]</em><br><em>    bucket            # number of buckets [2000000]</em><br><em>    thread            # number of threads [number of cpus]</em><br><em>    lrUpdateRate      # change the rate of updates for the learning rate [100]</em><br><em>    t                 # sampling threshold [0.0001]</em><br><em>    label             # label prefix [&#39;__label__&#39;]</em><br><em>    verbose           # verbose [2]</em><br><em>    pretrainedVectors # pretrained word vectors (.vec file) for supervised learning []</em><br><em>&#39;&#39;&#39;</em><br>model = fasttext.train_supervised(&#39;sarcasm_train.bin&#39;, wordNgrams=2)</pre><p>To measure the performance in the testing dataset:</p><pre><em>#measuring performance on test data</em><br><strong>def</strong> print_results(sample_size, precision, recall):<br>    precision   = round(precision, 2)<br>    recall      = round(recall, 2)<br>    print(f&#39;{sample_size=}&#39;)<br>    print(f&#39;{precision=}&#39;)<br>    print(f&#39;{recall=}&#39;)<br><br>print_results(*model.test(&#39;sarcasm_test.bin&#39;))</pre><pre><strong><em>sample_size=2862 <br>precision=0.87 <br>recall=0.87</em></strong></pre><p>The results, though not perfect, look promising. We save the model object now for future inferences.</p><pre><em>#save the model</em><br>model.save_model(&#39;fasttext_sarcasm.model&#39;)</pre><p>FastText is also capable of compressing the model in order to have a much smaller size model file by sacrificing only a little bit of performance through quantisation.</p><pre># with the previously trained `model` object, call<br>model.quantize(input=&#39;sarcasm_train.bin&#39;, retrain=True)\<br># results on test set<br>print_results(*model.test(&#39;sarcasm_test.bin&#39;))</pre><pre><strong><em><br>sample_size=2862 <br>precision=0.86 <br>recall=0.86</em></strong></pre><p>If you see the precision and recall seem to suffer .01 score but, looking at the model file size:</p><pre>!du -kh ./fasttext_sarcasm*</pre><pre><strong><em>98M ./fasttext_sarcasm.ftz<br>774M ./fasttext_sarcasm.model</em></strong></pre><p>The compressed model is only 1/12th of the base model. So this is a trade-off between model size and performance which users have to decide depending on the use case. Since the classifier model is trained and ready, now is the time to have the inference script ready so you can plan the deployment.</p><pre>def predict_is_sarcastic(text):<br>    return SarcasmService.get_model().predict(text, k=2)</pre><pre>if __name__ == &#39;__main__&#39;:<br>    ip = &#39;Make Indian manufacturing competitive to curb Chinese imports: RC Bhargava&#39;<br>    print(f&#39;Result : {predict_is_sarcastic(ip)}&#39;)</pre><pre><strong><em>Result : ((&#39;__label__0&#39;, &#39;__label__1&#39;), array([0.5498156 , 0.45020437]))</em></strong></pre><p>As can be seen above the result label with maximum probability is __label__0 which means the used headline is non-sarcastic as per the trained model. In model.predict() call the value of k signifies the number of classes you want in output along with their respective probability score. Since we have used softmax activation (default one in FastText), the sum of the probability of the two labels is 1.</p><p>To conclude, FastText can be a strong baseline while doing any NLP classification and its implementation is very easy. All the source code of this article can be found at <a href="https://github.com/sambit9238/Deep-Learning/tree/master/sarcasm_classifier">my git repo</a>. The FastText python module is not officially supported but that shouldn’t be an issue for tech people to experiment :). In future post will try to discuss how can the trained model be moved to production.</p><ul><li><a href="https://github.com/facebookresearch/fastText/tree/master/python">facebookresearch/fastText</a></li><li><a href="https://github.com/sambit9238/Deep-Learning/tree/master/sarcasm_classifier">sambit9238/Deep-Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=788ffbacb77b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/sarcasm-classification-using-fasttext-788ffbacb77b">Sarcasm Classification (Using FastText)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demographics of Women in India (A data-centric view)]]></title>
            <link>https://sambit9238.medium.com/demographics-of-women-in-india-a-data-centric-view-515147690972?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/515147690972</guid>
            <category><![CDATA[women]]></category>
            <category><![CDATA[india]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Sat, 07 Mar 2020 18:42:23 GMT</pubDate>
            <atom:updated>2020-03-07T18:46:53.172Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/938/0*de_N13APk6dT4Cc8" /></figure><p>India is the fastest growing economy and the largest democracy of the world with the potential to become the next superpower in a coming couple of decades. A country like India where women are the binding force of the very culture and society, the empowerment of women is the empowerment of the country. During the foreign invasion and British colonization, the country had gradually become a men-centric one. But in the last couple of decades, we are back on the track.</p><p>Here we are going to have a data-centric view on the demographics of women in India as of 2011 census and their socio-economic growth in comparison to the last census in 2001.</p><p>The data-analysis done to get these insights are divided into 3 major steps.</p><h3>Data Collection</h3><p>Before starting the analysis, some of the required libraries for these are pandas for data processing and seaborn for plotting.</p><pre>import pandas as pd<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br>sns.set(style=&quot;ticks&quot;, color_codes=True)</pre><p>We got the raw data of Govt. of India Census, 2001 District-Wise and 2011 District-Wise from Kaggle at <a href="https://www.kaggle.com/bazuka/census2001">https://www.kaggle.com/bazuka/census2001</a> and <a href="https://www.kaggle.com/danofer/india-census/version/1#india-districts-census-2011.csv">https://www.kaggle.com/danofer/india-census/version/1#india-districts-census-2011.csv</a></p><p>Then we filtered out the required data for our analysis and do some cleansing to help in analysis.</p><pre>#Read the data<br>df_2011 = pd.read_csv(“india-districts-census-2011.csv”)<br>df_2001 = pd.read_csv(“india_census_2001.csv”)</pre><pre>#selecting required columns<br>df_2011 = df_2011[[‘District code’, ‘State name’, ‘District name’, ‘Population’, ‘Male’, ‘Female’, ‘Literate’, ‘Male_Literate’, ‘Female_Literate’, ‘Workers’, ‘Male_Workers’, ‘Female_Workers’,’Age_Group_0_29&#39;,’Age_Group_30_49&#39;,’Age_Group_50&#39;]]<br>df_2001 = df_2001[[‘State’,’District’,’Persons’,’Males’,’Females’,’Persons..literate’,’Males..Literate’,’Females..Literate’]]</pre><p>In the two datasets, the names of states were different which we had to map with each other to ease our state-wise analysis. The dataset now looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*tjqWzPDJj8Jr7jNdmacPoQ.png" /><figcaption>2011 Census India</figcaption></figure><p>We need to group by existing stats state-wise as we are doing a state-wise analysis but the dataset is a district-wise dataset. some code snippet here:</p><pre>#2011 data processing<br>df_2011_pr = pd.DataFrame()<br>df_2011_pr[&#39;state&#39;] = df_2011[&#39;State name&#39;].unique().tolist()<br>df_2011_pr[&#39;population&#39;] = df_2011.groupby([&#39;State name&#39;], sort=False)[&#39;Population&#39;].sum().tolist()<br>df_2011_pr[&#39;population_male&#39;] = df_2011.groupby([&#39;State name&#39;], sort=False)[&#39;Male&#39;].sum().tolist()<br>        ......</pre><pre>df_2001_pr = pd.DataFrame()<br>df_2001_pr[&#39;state&#39;] = df_2001[&#39;State&#39;].unique().tolist()<br>df_2001_pr[&#39;population&#39;] = df_2001.groupby([&#39;State&#39;], sort=False)[&#39;Persons&#39;].sum().tolist()<br>df_2001_pr[&#39;population_male&#39;] = df_2001.groupby([&#39;State&#39;], sort=False)[&#39;Males&#39;].sum().tolist()<br>df_2001_pr[&#39;population_female&#39;] = df_2001.groupby([&#39;State&#39;], sort=False)[&#39;Females&#39;].sum().tolist()<br>        ......</pre><p>Now we have 35 rows for each data-frame corresponding to census data of <strong>29 States + 6 Union Territories</strong> of India.</p><h3>Data Processing</h3><p>Now we will do the analysis of women demographics in terms of the population, literacy and working-class as of 2011 and the corresponding change from 2001. For this, we calculated 7 factors from existing data i.e. sex_ratio, sex_ratio_literate (sex ratio in literates), sex_ratio_workers ((sex ratio in workers), literacy_rate_female (female literacy rate), working_women_pr (female workers in total working class). All these stats show the numbers of women per 1000 of the respective value. some code snippet here:</p><pre>#new columns for stat 2011<br>df_2011_pr[&#39;sex_ratio&#39;] = df_2011_pr[&#39;population_female&#39;]*1000//df_2011_pr[&#39;population_male&#39;]<br>df_2011_pr[&#39;sex_ratio_literate&#39;] = df_2011_pr[&#39;literate_female&#39;]*1000//df_2011_pr[&#39;literate_male&#39;]<br>   .......</pre><pre>#new columns for stat 2001<br>df_2001_pr[&#39;sex_ratio&#39;] = df_2001_pr[&#39;population_female&#39;]*1000//df_2001_pr[&#39;population_male&#39;]<br>df_2001_pr[&#39;sex_ratio_literate&#39;] = df_2001_pr[&#39;literate_female&#39;]*1000//df_2001_pr[&#39;literate_male&#39;]<br>   .......</pre><h3><strong>Stats and Visualization</strong></h3><p>We have got quite a few stats from the census 2011 and the comparison to 2001, to check all analysis go through the github link attached in the post. In the post, we will go through some major stats among them.</p><p>When it comes to sex ratio, we always go for the women population for each 1000 men population. But we need to fill the gender gap in the literate group and working-class group as well. Here some interesting stats on the top 10 states having the highest sex ratio in India:</p><pre>#Important stats<br>df_2011_pr.sort_values(by=[&#39;sex_ratio&#39;], ascending=False).iloc[:10][[&#39;state&#39;,&#39;sex_ratio&#39;,&#39;sex_ratio_literate&#39;,&#39;sex_ratio_workers&#39;]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*9vOcPalnrq8ecIuid--PqA.png" /><figcaption>Top 10 states in terms of sex ratio (2011)</figcaption></figure><p>Like we see here, the states like Kerala and Pondicherry have high sex_ratio but need to do a lot to attract more women to the workforce. Another thing to notice is also the sex_ratio among literates is not that impressive e.g. in Chhatisgarh, there are 990 females for every 1000 males, but only 746 female literates for every 1000 male literates. An important observation is, the north-eastern states like Manipur which are having very high GDP growths lately(30–40%) are also having a very vital representation from women also in their socio-economy.</p><pre>df_temp = df_2011_pr.sort_values(by=[&#39;sex_ratio&#39;], ascending=False).iloc[:5][[&#39;state&#39;,&#39;sex_ratio&#39;,&#39;working_women_pr&#39;,&#39;literacy_rate_female&#39;, &#39;literacy_rate&#39;]]<br>df_temp = pd.melt(df_temp, id_vars=[&#39;state&#39;], value_vars=[&#39;literacy_rate_female&#39;, &#39;literacy_rate&#39;, &#39;working_women_pr&#39;])<br>plt.figure(figsize=(12, 5))<br>sns.barplot(y=&quot;state&quot;, x=&quot;value&quot;, hue=&#39;variable&#39;, data=df_temp)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*-WeyKXNwrwevLvIQC79qLQ.png" /><figcaption>Top 5 states in sex ratio (2011)</figcaption></figure><p>An important point to consider here is the gender gap is diminishing in terms of literacy but India has a lot to achieve in getting women workforce to thrive the economy.</p><p>We wanted to have a look at the youth demographics. We considered people having age 0–29 as a youth.</p><pre>df_temp = pd.melt(df_2011_pr.sort_values(by=[&#39;sex_ratio&#39;], ascending=False).iloc[:10], id_vars=[&#39;state&#39;], value_vars=[&#39;literacy_rate_female&#39;,&#39;working_women_pr&#39;,&#39;youth_population&#39;])<br>plt.figure(figsize=(18, 7))<br>sns.lineplot(x=&quot;state&quot;, y=&quot;value&quot;, hue=&#39;variable&#39;, data=df_temp)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*keAJrU926utT9G7UH5kOhg.png" /></figure><p>An important trend here is the more the population of youth, the more the representation of women in the working class. This indicates, gradually with time women are playing a major part in the workforce of India.</p><p>Now we wanted to compare the change in demographics with respect to population, literacy, and workforce. But we couldn’t get the workforce-related data from the 2001 census. The study w.r.t. population and literacy are interesting.</p><pre>df_2001_pr = df_2001_pr.sort_values(by=[&#39;state&#39;])<br>df_2011_pr = df_2011_pr.sort_values(by=[&#39;state&#39;])<br>df_comp = pd.DataFrame()<br>df_comp[&#39;state&#39;] = list(df_2011_pr[&#39;state&#39;])<br>df_comp[&#39;sex_ratio_change&#39;] = df_2011_pr[&#39;sex_ratio&#39;] - df_2001_pr[&#39;sex_ratio&#39;]<br>df_comp[&#39;sex_ratio_literate_change&#39;] = df_2011_pr[&#39;sex_ratio_literate&#39;] - df_2001_pr[&#39;sex_ratio_literate&#39;]<br>df_comp[&#39;literacy_rate_female_change&#39;] = df_2011_pr[&#39;literacy_rate_female&#39;] - df_2001_pr[&#39;literacy_rate_female&#39;]<br>df_comp[&#39;literacy_rate_change&#39;] = df_2011_pr[&#39;literacy_rate&#39;] - df_2001_pr[&#39;literacy_rate&#39;]</pre><pre>df_comp.sort_values(by=[&#39;sex_ratio_change&#39;], ascending=False).iloc[:5,:3]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*FKe3Y4_fwCx9465gAYx4uw.png" /><figcaption>Top 5 states in terms of growth in sex ratio (2011)</figcaption></figure><p>The important factor here is the sex_ratio growth in 3 out of 5 states are double in terms of literacy than in terms of population. This should be the scenario as there is a lot more gender gap in terms of literacy than in terms of population.</p><p>Now to conclude with some major stats:</p><pre>female_2001 = round(df_2001[&#39;Females&#39;].sum()*100/df_2001[&#39;Persons&#39;].sum(), 2)<br>female_2011 = round(df_2011[&#39;Female&#39;].sum()*100/df_2011[&#39;Population&#39;].sum(), 2)<br>female_growth = round(female_2011 - female_2001, 2)<br>female_literates_2001 = round(df_2001[&#39;Females..Literate&#39;].sum()*100/df_2001[&#39;Persons..literate&#39;].sum(), 2)<br>female_literates_2011 = round(df_2011[&#39;Female_Literate&#39;].sum()*100/df_2011[&#39;Literate&#39;].sum(), 2)<br>female_literates_growth = round(female_literates_2011 - female_literates_2001, 2)<br>female_workers_2001 = &#39;NA&#39;<br>female_workers_2011 = round(df_2011[&#39;Female_Workers&#39;].sum()*100/df_2011[&#39;Workers&#39;].sum(), 2)<br>female_workers_growth = &#39;NA&#39;</pre><pre>df_stat = pd.DataFrame()<br>df_stat[&#39;name&#39;] = [&#39;population&#39;, &#39;Literacy&#39;, &#39;Workers&#39;]<br>df_stat[&#39;2001&#39;] = [str(female_2001)+&#39;%&#39;, str(female_literates_2001)+&#39;%&#39;, female_workers_2001]<br>df_stat[&#39;2011&#39;] = [str(female_2011)+&#39;%&#39;, str(female_literates_2011)+&#39;%&#39;, str(female_workers_2011)+&#39;%&#39;]<br>df_stat[&#39;growth&#39;] = [&#39;+&#39;+str(female_growth)+&#39;%&#39;, &#39;+&#39;+str(female_literates_growth)+&#39;%&#39;, female_workers_growth]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*ueHt8ZziLwQBQG1xxWWX_A.png" /></figure><p>If we see, India has done impressively good in beefing up the female literacy by 3.12%, the sex ratio has increased reasonably by 0.27%. It has 31.12% female work-force as of 2011. After 2011, there has been a dramatic change in technology, economy, and culture focusing more on gender equality, so we are expected to see a hooping growth in 2021.</p><p>As seen from the above stats, as of 2011 census women have made great strides in many areas with notable progress in reducing some gender gaps and thriving the demographics and socio-economy plot of the oldest civilization. Waiting for the 2021 census, to track the trend in the ongoing decade.</p><p><a href="https://github.com/sambit9238/DataScience/blob/master/india_census/analysis.ipynb">sambit9238/DataScience</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=515147690972" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classification Model Building and Tracking in Tensorflow 2.0]]></title>
            <link>https://medium.com/data-science/classification-model-building-and-tracking-in-tensorflow-2-0-398f5fca8fcf?source=rss-241751fe4cf9------2</link>
            <guid isPermaLink="false">https://medium.com/p/398f5fca8fcf</guid>
            <category><![CDATA[tensorboard]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sambit Mahapatra]]></dc:creator>
            <pubDate>Sun, 24 Nov 2019 14:51:52 GMT</pubDate>
            <atom:updated>2019-11-26T06:46:10.872Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*78KMHVVecnTzkLbN7xW0tQ.png" /></figure><p>Last month Tensorflow 2 got released making the model development and deployment easier with tight integration of high level keras, cleaning up of redundant APIs, keeping eager execution as default, removal of global and usage of function instead of sessions.</p><p>In this article, we will build a CNN classifier for MNIST dataset with Tensorflow 2.0 and track the model performance with Tensorboard, keeping all the best practices in mind.</p><p>First step is to install the Tensorflow 2.0. The suggested way is to install it either in virtual environment or docker. To install through virtual env, you can follow this<a href="https://medium.com/@sambit9238/python-development-environment-set-up-with-virtual-environment-be26b1e96188."> link</a>.</p><p>Now we will divide our article into 2 section:</p><ol><li>CNN model building.</li><li>Tracking the model with Tensorboard.</li></ol><h3>CNN model building</h3><p>First we need to import the required libraries.</p><pre>from tensorflow.keras import Model</pre><pre>from tensorflow.keras.layers import Dense, Flatten, Conv2D</pre><pre>import tensorflow as tf</pre><blockquote>Make sure you have installed numpy version 1.16.4, otherwise user warning like this shows up:</blockquote><blockquote>FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’. np_resource = np.dtype((“resource”, np.ubyte, 1)) s</blockquote><p>Now, we will get the available<a href="https://www.tensorflow.org/api_docs/python/tf/keras/datasets"> mnist handwritten digit dataset from tf.keras API.</a> It has both training and testing set having 60,000 and 10,000 images respectively. Each image is black and white of dimension 28*28. Then we convert the samples from integers to floating point numbers to feed them to the model.</p><pre>mnist = tf.keras.datasets.mnist</pre><pre>(x_train, y_train), (x_test, y_test) = mnist.load_data()</pre><pre>x_train, x_test = x_train / 255.0, x_test / 255.0</pre><pre>print(x_train.shape)<br>#  (60000, 28, 28)</pre><p>Since we are going to feed the input to CNN, the input shape has to be in a specific format. For Tensorflow, the format should be (batch, height, width, channels). So we will add a new axis to format our inputs for CNN.</p><pre># Add a channels dimension<br>x_train = x_train[..., tf.newaxis]<br>x_test = x_test[..., tf.newaxis]</pre><pre>print(x_train.shape)<br># (60000, 28, 28, 1)</pre><p>Then we will shuffle and batch the dataset using tf.data API. It is a very handy API to design your input data pipelines to the models in production. For shuffling, we will use .shuffle() from tf.data.Dataset API. It randomly shuffles the elements of the provided dataset by filling a buffer with buffer_size elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. We have used a buffer size of 10000 considering the memory constraint of the machine.</p><pre>#tf.data to batch and shuffle the dataset</pre><pre>train_ds = tf.data.Dataset.from_tensor_slices((x_train,y_train))\<br>                                        .shuffle(10000).batch(32)</pre><pre>test_ds =  tf.data.Dataset.from_tensor_slices((x_test,y_test))\<br>                                        .batch(32)</pre><pre>print(train_ds.element_spec)</pre><pre>#(TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float64, name=None), TensorSpec(shape=(None,), dtype=tf.uint8, name=None))</pre><p>Since we are doing batch of 32, the last batch will have less than 32 elements, which if you want can drop by passing drop_remainder=True to .batch()</p><p>Now we have the data, it is time to define model architecture. We will use Keras <a href="https://www.tensorflow.org/guide/keras#model_subclassing">model subclassing API</a> for this. Another way is to use keras functional API though subclassing API is preferred for production.</p><pre>class MyModel(Model):<br>        def __init__(self):<br>                super(MyModel, self).__init__()<br>                self.conv1 = tf.keras.layers.Conv2D(32, 3, padding=&#39;same&#39;, activation=&#39;relu&#39;)<br>                self.pool = tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=2, padding=&#39;same&#39;)<br>                self.flatten = tf.keras.layers.Flatten()<br>                self.d1 = tf.keras.layers.Dense(256, activation=&#39;relu&#39;)<br>                self.d2 = tf.keras.layers.Dense(128, activation=&#39;relu&#39;)<br>                self.d3 = tf.keras.layers.Dense(10, activation=&#39;softmax&#39;)</pre><pre>        def call(self, x):<br>                x = self.conv1(x)<br>                x = self.pool(x)<br>                x = self.flatten(x)<br>                x = self.d1(x)<br>                x = self.d2(x)<br>                x = self.d3(x)<br>                return x</pre><p>Now, we need to have a optimizer and loss function for the training of the models.</p><pre># optimizer and loss function to train<br>loss_object = tf.keras.losses.SparseCategoricalCrossentropy()<br>optimizer = tf.keras.optimizers.Adam()</pre><blockquote>Since, we are providing labels as integers we will be using SparseCategoricalCrossentropy as loss function.</blockquote><p>We will also track the loss and accuracy on the training and testing data after each epoch.</p><pre># metrics to measure the loss and the accuracy of the model</pre><pre>train_loss = tf.keras.metrics.Mean(name=&#39;train_loss&#39;)<br>train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(<br>                            name=&#39;train_accuracy&#39;)</pre><pre>test_loss = tf.keras.metrics.Mean(name=&#39;test_loss&#39;)<br>test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(<br>                            name=&#39;test_accuracy&#39;)</pre><p>Now, we need to define the training and testing methods to start model building.</p><pre>@tf.function<br>def train_step(model, optimizer, images, labels):<br>         with tf.GradientTape() as tape:<br>                  predictions = model(images)<br>                  loss = loss_object(labels, predictions)<br>         gradients = tape.gradient(loss, model.trainable_variables)<br>         optimizer.apply_gradients(zip(gradients,                   model.trainable_variables))<br>         train_loss(loss)<br>         train_accuracy(labels, predictions)</pre><pre>@tf.function<br>def test_step(model, images, labels):<br>         predictions = model(images)<br>         t_loss = loss_object(labels, predictions)<br>         test_loss(t_loss)<br>         test_accuracy(labels, predictions)</pre><p>Lot of different things from Tensorflow 1 time here in the above code.</p><p>First about tf.function,</p><blockquote>In TensorFlow 2.0, eager execution is turned on by default. The user interface is intuitive and flexible (running one-off operations is much easier and faster), but this can come at the expense of performance and deployability.</blockquote><blockquote>tf.function constructs a callable that executes a TensorFlow graph (<a href="https://www.tensorflow.org/api_docs/python/tf/Graph">tf.Graph</a>) created by tracing the TensorFlow operations in func. This allows the TensorFlow runtime to apply optimizations and exploit parallelism in the computation defined by func. So, it is recommended for production.</blockquote><p>Second thing is the tf.GradientTape(),</p><blockquote>With eager execution enabled, to back propagate errors, you have to keep track of the gradients of your computation and then apply these gradients to an optimiser.</blockquote><blockquote>Obviously, Tensorflow could just keep track of every gradient for every computation on every tf.Variable. However, that could be a huge performance bottleneck. They expose a gradient tape so that you can control what areas of your code need the gradient information.</blockquote><p>Now, we have to instantiate a model object and feed the data to train the model. We have to track the loss and accuracy on test data to evaluate the model. We have used here only 5 epochs for demonstration purpose.</p><pre>EPOCHS = 5</pre><pre># Create an instance of the model<br>model = MyModel()<br>for epoch in range(EPOCHS):<br>        for images, labels in train_ds:<br>                train_step(model, optimizer, images, labels)</pre><pre>        for test_images, test_labels in test_ds:<br>                test_step(model, test_images, test_labels)</pre><pre>        template = &#39;Epoch {}, Loss: {}, Accuracy: {}, Test Loss: {}, Test Accuracy: {}&#39;<br>        print(template.format(epoch+1, train_loss.result(),<br>                              train_accuracy.result()*100,<br>                              test_loss.result(),<br>                              test_accuracy.result()*100))</pre><pre>#output<br>Epoch 1, Loss: 0.14411139488220215, Accuracy: 95.71333312988281, Test Loss: 0.04969984292984009, Test Accuracy: 98.44999694824219<br>2019-11-24 18:54:43.808050: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 376320000 exceeds 10% of system memory.<br>Epoch 2, Loss: 0.04587719216942787, Accuracy: 98.59500122070312, Test Loss: 0.06183547526597977, Test Accuracy: 98.04000091552734<br>Epoch 3, Loss: 0.029635081067681313, Accuracy: 99.05333709716797, Test Loss: 0.04673100262880325, Test Accuracy: 98.54000091552734<br>Epoch 4, Loss: 0.020683910697698593, Accuracy: 99.32167053222656, Test Loss: 0.04771297425031662, Test Accuracy: 98.6199951171875<br>Epoch 5, Loss: 0.01358255185186863, Accuracy: 99.57833099365234, Test Loss: 0.05094970762729645, Test Accuracy: 98.58000183105469</pre><p>If we see with 5 epochs it got around 98%+ accuracy.</p><h3>Tracking the model with Tensorboard</h3><p>Tensorboard is a very handy graph visualisation tool to track the model training progress in real time. It can be used for several usage from inspecting model architecture to tracking the behaviour of hyper parameters on model training. Here, we will just go through a short example of examining the accuracy and loss of the model while on training.</p><p>We need to first launch a Tensorboard instance first to display the data.</p><pre>import datetime<br>from tensorboard import program</pre><pre>tb = program.TensorBoard()<br>tb.configure(argv=[None, ‘ — logdir’, “.”])<br>url = tb.launch()<br>print(“tensorboard url: “, url)</pre><pre>#tensorboard url:  <a href="http://localhost:6006/">http://localhost:6007/</a></pre><p>The Tensorboard can be viewed at <a href="http://localhost:6006/">http://localhost:6007/</a> now. If you go there, you will see a screen like :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_brT5L0iN9TFtv9dNFtgLw.png" /></figure><p>It is because the Tensorboard displays the data by reading from the log directory in the supplied location. In tb.configure, we have provided the current working directory. So it will create a log directory and write the data to be shown on Tensorboard. So, we will first create a log writer for both training and testing</p><pre>current_time = datetime.datetime.now().strftime(&quot;%Y%m%d-%H%M%S&quot;)<br>train_log_dir = &#39;logs/gradient_tape/&#39; + current_time + &#39;/train&#39;<br>test_log_dir = &#39;logs/gradient_tape/&#39; + current_time + &#39;/test&#39;<br>train_summary_writer = tf.summary.create_file_writer(train_log_dir)<br>test_summary_writer = tf.summary.create_file_writer(test_log_dir)</pre><p>Then we have to write each epoch results w.r.t. to both training and testing to visualise them.</p><pre>for images, labels in train_ds:<br>        for images, labels in train_ds:<br>                train_step(model, optimizer, images, labels)<br>        with train_summary_writer.as_default():<br>                tf.summary.scalar(&#39;loss&#39;, train_loss.result(), step=epoch)<br>                tf.summary.scalar(&#39;accuracy&#39;, train_accuracy.result(), step=epoch)</pre><pre>        for test_images, test_labels in test_ds:<br>                test_step(model, test_images, test_labels)<br>        with test_summary_writer.as_default():<br>                tf.summary.scalar(&#39;loss&#39;, test_loss.result(), step=epoch)<br>                tf.summary.scalar(&#39;accuracy&#39;, test_accuracy.result(), step=epoch)</pre><p>Now while training is going on, if you want to see the result just go to Tensorboard url and the screen will look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WllH1EvXYCzJa9pli2X7bw.png" /></figure><p>Many things can be visualised in Tensorboard about the models which are not part of this article.</p><p>The code for this article can be found at :</p><p><a href="https://github.com/sambit9238/Deep-Learning/blob/master/mnist.py">sambit9238/Deep-Learning</a></p><h4>References:</h4><ul><li><a href="https://www.tensorflow.org/tutorials/quickstart/advanced">TensorFlow 2 quickstart for experts | TensorFlow Core</a></li><li><a href="https://pgaleone.eu/tensorflow/tf.function/2019/03/21/dissecting-tf-function-part-1/">Analyzing tf.function to discover AutoGraph strengths and subtleties - part 1</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=398f5fca8fcf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/classification-model-building-and-tracking-in-tensorflow-2-0-398f5fca8fcf">Classification Model Building and Tracking in Tensorflow 2.0</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>