<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Raffa Nimir on Medium]]></title>
        <description><![CDATA[Stories by Raffa Nimir on Medium]]></description>
        <link>https://medium.com/@raffanimir?source=rss-de23824a4dab------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dZ4pzSj2raFrPf05v_DgXA.png</url>
            <title>Stories by Raffa Nimir on Medium</title>
            <link>https://medium.com/@raffanimir?source=rss-de23824a4dab------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:01:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raffanimir/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Do We Return to Ourselves in the Noise of the World?]]></title>
            <link>https://raffanimir.medium.com/how-do-we-return-to-ourselves-in-the-noise-of-the-world-793f032adc0e?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/793f032adc0e</guid>
            <category><![CDATA[random-thoughts]]></category>
            <category><![CDATA[reflections]]></category>
            <category><![CDATA[bias]]></category>
            <category><![CDATA[decision-bias]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Mon, 27 Apr 2026 16:06:01 GMT</pubDate>
            <atom:updated>2026-04-27T16:06:01.613Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mkAwXRYwcPR3vgjiEMR-rw.jpeg" /><figcaption>photo credit : Jr Korpa Unsplash</figcaption></figure><p>In my very first <em>systems engineering</em> lecture, the professor didn’t introduce himself, didn’t explain the syllabus — he simply wrote one sentence on the board and paused:</p><blockquote>“All models are wrong, but some are useful.”<em> — George Box</em></blockquote><p>It didn’t feel like an academic definition.<br> It felt like a stance — almost a philosophy — on truth itself.</p><p>Because simply put:<br> Every model, no matter how precise, is just an attempt to approximate a reality far more complex than it.<br> A reality filled with details that don’t reveal themselves upfront,and effects that only make sense after they happen — randomness, the imprint of life itself.</p><p>Truth, by its nature, is often understood <em>retrospectively</em>.</p><p>And as simple as that idea sounds, it made me quieter — <br> in my expectations,<br> and in my confidence in any “complete picture” that hasn’t fully unfolded yet.</p><p>From there, I found myself extending the same lens to something we live with every day:</p><p><strong>Advice.</strong></p><p>If you look at it closely, advice can also be seen as a kind of modeling.</p><p>Someone else is trying to understand your reality<br> with the limited information they have,<br> filling in the gaps with their own experiences,<br> and offering you a direction that <em>might</em> be useful.</p><p>And in that sense, I found myself simplifying it like this:</p><blockquote><strong><em>All advice is wrong, but some is useful.</em></strong></blockquote><p>This isn’t a dismissal of advice — far from it.<br> If anything, it’s a softer, more grounded way to understand it.</p><p>Advice can expand your perspective.<br> It can open a new angle.<br> It can bring you closer to a decision.</p><p>But at the same time, it will always be incomplete.<br> Simply because no one else lives the full depth of your experience but you.</p><p>This idea might have stayed simple — <br> if we lived in a quieter world.</p><p>But in the age of social media, advice is no longer something we seek.<br> It’s something that reaches us — sometimes even intrudes on us.</p><p>A constant stream of opinions, experiences, and ready-made models:<br> how to think, how to love, how to live.</p><p>And within that stream, I started noticing something else:</p><p>It’s not just that we’re hearing more advice — <br> it’s that we’re most influenced by what we heard <em>last</em>.</p><p>That’s where <strong>recency bias</strong> comes in.</p><p>In simple terms:<br> our minds tend to give more weight to the most recent information,<br> because it’s easier to recall and closer to our awareness.</p><p>So something you just saw or heard<br> might feel more convincing — more “true” — <br> even if it isn’t.</p><p>When you connect both ideas:</p><ul><li>A constant influx of advice</li><li>A natural bias toward the most recent input</li></ul><p>You begin to see how easily your decisions can shift.</p><p>Not necessarily because your beliefs changed,<br> but because the latest input occupied more space than it deserved.</p><p>And maybe this is where a small pause becomes important:</p><p>When you find yourself suddenly leaning toward an idea,<br> it might be worth asking:</p><p><strong>Is this truly an extension of my experience?<br> Or just the echo of the last thing I consumed?</strong></p><p>If we go back to the beginning:</p><p>Models are incomplete.<br> Advice is incomplete.</p><p>And maybe the point isn’t to reject them — <br> but to stay aware of their limits.</p><p>I’m not saying we should shut the world out to return to ourselves.<br> But maybe we can learn to hold a balance:</p><p>To listen… without dissolving.<br> To learn… without losing our voice.</p><blockquote><strong><em>You are your greatest source of truth.</em></strong></blockquote><p>Not because you’re always right — <br> but because you are closest to your own context, your own details, your own reality.</p><p>In a world full of noise,<br> maybe real awareness isn’t about finding the <em>right</em> answer —</p><p>but about noticing<br> <strong>where your thoughts came from in the first place.</strong></p><p>Because sometimes,<br> what feels like a “new understanding”…</p><p>is just<br> the residue of the last thing that passed through you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=793f032adc0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Between Humans and ML/AI: analogies from both worlds decoded (2)]]></title>
            <link>https://raffanimir.medium.com/in-my-previous-article-i-dedicated-considerable-effort-to-elucidating-the-parallels-between-the-239cde5588aa?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/239cde5588aa</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[simulated-reality]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Tue, 19 Sep 2023 03:50:11 GMT</pubDate>
            <atom:updated>2023-09-19T04:13:33.838Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dTqMs_nGpy6ADrtrEV0eXw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@possessedphotography?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Possessed Photography</a> on <a href="https://unsplash.com/photos/YKW0JjP7rlU?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In my previous article, I dedicated considerable effort to elucidating the parallels between the human experience and that of artificial intelligence. While I’m quite certain that the initial intention behind technological advancement wasn’t to replicate the human experience, it appears that somewhere along the way, the trajectory took a different turn.</p><p>From my perspective, if we were to dissect the human operating system, we would discover that our decisions and actions result from a fusion of two fundamental systems: a stable, knowledge-based system and a dynamic, ever-evolving system. The stable system is related to our knowledge, serving as the backend database for all our actions. We act based on what we know. The dynamic system encompasses the constantly shifting thoughts in our minds, which remain in a perpetual state of flux. This intricate decision-making process seamlessly navigates everything from the simplest to the most complex tasks in our daily lives, inputting thoughts into the knowledge system and yielding refined, reformulated thoughts as output.</p><p>In the realm of artificial intelligence, this exchange of thoughts with knowledge has been simulated through machine learning and algorithms actively teaching machines how to make decisions by leveraging previously acquired data and a predetermined sequence of arithmetic or algebraic operations that culminate in an accurate decision, all within a predefined confidence level. It seems like a perfect emulation, doesn’t it? Well, almost perfect!</p><p>This perfection falters when we consider the predominant role of human emotions and instincts in decision-making. In Daniel Kahneman’s “Thinking, Fast and Slow,” we discern that humans engage in two distinct thought processes when arriving at decisions. System 1 operates swiftly, relying on instincts and quick responses, while System 2 operates slowly, guided by wisdom and informed judgment. Up to this point, machine learning and AI systems have excelled in simulating System 2, but what about System 1? It was approached in a rather crude manner, using the same logical and algorithmic methods as System 2. Can we genuinely replicate System 1? Is it rooted in logic? How does it interact with knowledge, logic, and the decision-making cycle? Or is it something entirely different?</p><p>The questions abound: Can we simulate instincts, emotions, and feelings in the digital realm, or is this a futile endeavor? Is there a way to orchestrate responses that aren’t rigidly controlled? Could autonomous systems or game theory offer insights into achieving this? I’m deeply intrigued by the prospects of delving into and exploring this fascinating domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jhkMb-jgqti-yT8wPraBjg.png" /><figcaption>Produced by Canva</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=239cde5588aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Forecasting : Concept Summary]]></title>
            <link>https://raffanimir.medium.com/time-series-forecasting-concept-summary-7f5a2b734899?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f5a2b734899</guid>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[time-series-data]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[time-series-model]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Wed, 23 Aug 2023 01:59:57 GMT</pubDate>
            <atom:updated>2023-08-23T02:00:39.404Z</atom:updated>
            <content:encoded><![CDATA[<h3>Time Series Forecasting in R: Concept Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAWD_atBKkQ7pyBEbzooJg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@chrisliverani?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Chris Liverani</a> on <a href="https://unsplash.com/photos/dBI_My696Rk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Recently, as part of my ongoing efforts to expand my data science skill set, I took DataCamp’s course focused on time series forecasting in R. I won’t lie; the subject matter was complex and took some time to wrap my head around. But after putting in the work, I’ve managed to distill what I’ve learned into this article. In the following sections, we’ll get into the nitty-gritty of time series forecasting, paying special attention to techniques like exponential smoothing and ARIMA models.</p><h4>The First Step: Visualization</h4><p>When you’re working with a time series dataset, the first thing you should do is plot the data. Visualizing it this way can give you critical insights, helping you identify if the data shows trends, seasonality, cycles, or if it’s just random noise.</p><blockquote>Understanding Autocorrelation and Stationarity</blockquote><p>Upon plotting, two pivotal aspects demand our attention:</p><p><strong>Autocorrelation</strong>: This quantifies the degree of similarity between a time series and its lagged versions. In more layman terms, we assess whether data points in the series are independent or exhibit dependence on preceding points.</p><p><strong>Stationarity</strong>: A stationary time series is one wherein its statistical properties remain invariant over time(mean/variance). It signifies that the series’ mean, variance and autocorrelation structure do not change with time shifts.</p><h3><strong>A Dive into Forecasting Models</strong></h3><p>At the heart of forecasting models lies a regressive mechanism that models a time series against its previous lags. Thus, discerning autocorrelation is indispensable.</p><p>If the data is devoid of autocorrelation, it essentially represents white noise, forming an ideal basis for ARIMA models.</p><p>For autocorrelated data, there are two potential avenues:</p><p><strong>Transformative Methods</strong>: Techniques such as differencing or log transformation can be employed to stabilize the variance and render the series stationary.</p><p><strong>Exponential Smoothing Models</strong>: Suited for capturing attributes like trendlines and seasonality. Within trendlines, distinctions arise between damped and multiplicative trends. A salient feature of these models is the incorporation of exponentially decaying weights, where recent points carry greater significance compared to older ones.</p><p><strong>Simple Exponential Smoothing (SES)</strong>: An elementary model ideal for time series without trend or seasonality components.<br><strong>Holt Linear Correction</strong>: Comes into play when the series exhibits a trend.<br><strong>Holt’s Winter Correction</strong>: Employed when the series displays seasonality.<br><strong>ETS Model</strong>: A comprehensive model that encapsulates most time series characteristics.<br><strong>Evaluating Stationarity</strong>: The Augmented Dickey-Fuller (ADF) Test</p><p>To rigorously ascertain whether a time series is stationary, the Augmented Dickey-Fuller (ADF) test is employed. If the series is found stationary, ARIMA models can be aptly applied. Non-stationary data, on the other hand, either undergoes transformative methods for compatibility with ARIMA or leverages exponential smoothing models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wVDYOCxHlLcnaqrajAgPiw.png" /><figcaption>Chart by RaffaNimir Produced on LucidChart</figcaption></figure><p>This article serves as an introductory survey of time series forecasting techniques in R. Like any area in applied mathematics or statistics, the landscape of time series analysis is both deep and wide. Mastery in this domain hinges not just on theoretical understanding but also on applied experience and iterative learning. To aid in grasping the core concepts and rationale, I’ve provided a graphical representation below, which offers a visual guide through the key steps in time series forecasting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f5a2b734899" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploratory Analysis of Yelp Data Using NLP Techniques in R]]></title>
            <link>https://raffanimir.medium.com/exploratory-analysis-of-yelp-data-using-nlp-techniques-in-r-da1fc6fabb4b?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/da1fc6fabb4b</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[yelp-review]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[eda]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Tue, 13 Jun 2023 15:23:04 GMT</pubDate>
            <atom:updated>2023-06-13T15:23:04.261Z</atom:updated>
            <content:encoded><![CDATA[<p>Extracting Valuable Feedback for Businesses to Improve Services</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YUKjdxQua2fmAk3Cc33BHg.png" /><figcaption><a href="https://thesocialmediamonthly.com/five-easy-ways-to-get-customer-reviews-that-boost-sales-for-local-business/">https://thesocialmediamonthly.com/five-easy-ways-to-get-customer-reviews-that-boost-sales-for-local-business/</a></figcaption></figure><h4>Introduction:</h4><p>In today’s competitive market, I strive to deliver exceptional customer experiences. One valuable source of feedback is online reviews. In this project, I leverage Natural Language Processing (NLP) techniques in R to gain insights from textual Yelp reviews. By analyzing the reviews, I can identify areas for improvement and enhance my services accordingly. Let’s dive into the details of this exploratory analysis.</p><h4>Tokenization and NLP Cleaning:</h4><p>To begin, I tokenize the text data, breaking it down into individual words or tokens. I then apply various preprocessing techniques to clean the text data, such as removing punctuation, converting to lowercase, and eliminating special characters. By preparing the data in this way, I ensure a more accurate analysis.</p><h4>Stopword Removal:</h4><p>Next, I remove common stopwords that do not contribute significantly to the analysis, such as “the,” “and,” and “is.” By eliminating these irrelevant words, I can focus on the more meaningful content of the reviews.</p><h4>Lemmatization and Stemming:</h4><p>To further improve text normalization, I utilize lemmatization and stemming algorithms. These techniques reduce words to their base forms, enabling better analysis and interpretation of the text.</p><pre>corpus &lt;- tm_map(corpus,removeNumbers)<br>corpus &lt;- tm_map(corpus, removePunctuation)<br>corpus &lt;- tm_map(corpus, stripWhitespace)<br>corpus &lt;- tm_map(corpus, content_transformer(tolower))<br>corpus &lt;- tm_map(corpus, stemDocument)<br>corpus &lt;- tm_map(corpus, removeWords, stopwords_en)</pre><h4>Top Mentioned Words:</h4><p>My analysis identifies the top 70 most frequently mentioned words in Yelp reviews. By uncovering these commonly used words, I gain a deeper understanding of what customers emphasize in their feedback.</p><pre>### term document matrix <br>tdm &lt;- TermDocumentMatrix(corpus)<br>tdm_m &lt;- as.matrix((tdm))<br>tdm_v &lt;- sort(rowSums(as.matrix(tdm)), decreasing = TRUE)<br>tdm_d &lt;- data.frame(word = names(tdm_v), freq = tdm_v)<br><br># displaying the top 50 frequent terms <br><br>head(tdm_d, 50)<br>d &lt;- head(tdm_d, 50)</pre><h4>Plot:</h4><pre>p &lt;- ggplot(d, aes(x = reorder(word, freq), y = freq))+<br>  geom_col( width = 0.6, fill = &#39;#0000FF&#39;)+<br>  coord_flip()+<br>  ylab(&quot;words&quot;)+<br>  xlab (&quot;frequency&quot;)+<br>  ggtitle(&quot;top mentioned words in customer reviews&quot;)+<br>  theme_classic()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TFRvZHbXu-cl2TFHdZvdCg.png" /></figure><h4>Bigrams and Trigrams:</h4><p>In addition to individual words, I explore meaningful phrases or expressions by generating 50 bigrams (pairs of words) and 50 trigrams (sequences of three words). These combinations provide valuable context and allow me to extract more nuanced insights from the reviews.</p><pre>p2 &lt;-  text %&gt;% <br>    unnest_tokens(word,txt,token = &quot;ngrams&quot;, n = 2)%&gt;%<br>    separate(word, c(&quot;word1&quot;, &quot;word2&quot;), sep= &quot; &quot;)%&gt;%<br>    filter(!word1 %in% stop_words$word)%&gt;%<br>    filter(!word2 %in% stop_words$word)%&gt;%<br>    unite(word, word1, word2, sep = &quot; &quot;)%&gt;%<br>    count(word, sort = TRUE) %&gt;%<br>    slice(1:50)%&gt;%<br>    ggplot()+<br>    geom_bar(aes(reorder(word,n), y =n), stat =&quot;identity&quot;, fill = &quot;#de5833&quot;)+<br>    theme_minimal()+<br>    coord_flip()+<br>    labs(title = &quot;Top Biagrams of Customers Reviews&quot;, <br>         subtitle = &quot;using tidytext in R&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sJNJxfFnwBh9C6Kmk-0FwQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fVJWc8HEXy320O-2hsx5RA.png" /></figure><h4>Wordclouds:</h4><pre>dev.new(width= 3000, height = 1500, unit =&#39;px&#39;)<br>wc &lt;- wordcloud(words = tdm_d$word, tdm_d$freq, min.freq = 5 , max.words = 500, scale = c(2,.1), random.order = FALSE , rot.per = .1, colors = brewer.pal(8,&quot;Dark2&quot;))</pre><p>To provide a visual representation of the feedback extracted from the reviews, I utilize the R sentiment package to create wordclouds representing positive, negative, and neutral sentiment. These wordclouds serve as powerful visual tools, enabling me to quickly grasp customer sentiment and identify key themes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*e6QGU351Qj1oUCZqLs9lyA.png" /></figure><h4>Packages Used for NLP:</h4><p>To perform the NLP analysis, I employed packages such as tm, tidytext, and tidyr. These libraries provide essential functions for text manipulation and analysis.</p><h4>Packages Used for Plotting:</h4><p>For generating plots and visualizations, I utilized packages like wordcloud and ggplot2. These tools allow me to present my findings in an easily interpretable and visually appealing manner.</p><h4>Packages Used for Data Wrangling:</h4><p>For efficient data wrangling tasks, I relied on packages such as dplyr and tidyverse. These libraries streamline the process of manipulating and transforming data, ensuring a smooth analysis.</p><h4>Conclusion:</h4><p>By conducting this exploratory analysis of Yelp data using NLP techniques in R, I have demonstrated how I can extract valuable feedback from customer reviews. By analyzing the most mentioned words, and meaningful phrases, I gain insights to enhance my services and address customer concerns effectively. The generated word clouds serve as powerful visual tools, enabling me to understand customer sentiment at a glance. With these insights, I can make data-driven decisions and continuously improve my offerings to meet customer expectations.</p><h4>Visit my Github:</h4><p><a href="https://github.com/RaffaNimir">https://github.com/RaffaNimir</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da1fc6fabb4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Match Up: The Story of Combining Doc2Vec and Similarity to Create a Recruiting Engine]]></title>
            <link>https://raffanimir.medium.com/building-match-up-the-story-of-combining-doc2vec-and-similarity-to-create-a-recruiting-engine-22d8bcfe9e88?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/22d8bcfe9e88</guid>
            <category><![CDATA[job-matching-technology]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[similarity-search]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Mon, 08 May 2023 16:54:05 GMT</pubDate>
            <atom:updated>2023-05-08T21:59:09.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQT7UMbmwnLQNoV4Z0Uqzg.jpeg" /><figcaption>photo source: <a href="https://unsplash.com/photos/n8Qb1ZAkK88">https://unsplash.com/photos/n8Qb1ZAkK88</a></figcaption></figure><p>During my final semester as a data analytics student and job seeker, I became frustrated with the job search process. It was tedious having to personally filter through job postings, and I was left unsure about how I was being evaluated for the positions I applied for. Rather than accepting the flawed job recommendation systems available, I decided to take matters into my own hands and build my own data science job recommender engine as my capstone project.</p><p>My goal was to develop a fair and impartial recommender system that could match job seekers with their most suitable jobs without human intervention. To accomplish this, I combined natural language processing techniques, word embeddings, and similarity to build a recruiting engine. I collected job openings data from multiple sources, including Kaggle, data. world, and other job portals. I wanted to capture the most from a job posting. Therefore the core of my analysis was around the narrated job description field, which outlines the responsibilities and requirements of a particular job. I merged datasets from various sources to create a consolidated data source containing over 16,000 records of data science job openings from different companies in the United States and covering all levels (junior-mid and senior).</p><pre>import pandas as pd <br>jobs = pd.read_csv(&quot;jobs.csv&quot;)<br>jobs.sample(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h7yVjdlswfBKXKXP6DvxNw.png" /><figcaption>dataset sample as generated by the above code</figcaption></figure><h3>Exploratory Data Analysis</h3><p>Prior to building my engine and as a typical data scientist I wanted to extract as much information as possible from the dataset, I conducted exploratory data analysis on all fields, and here’s what I learned.</p><h4>1. Top Companies</h4><p>When it came to the top companies offering data science/analytics jobs, my findings indicated that Cyber Coders had the most job openings, followed by Paypal and Jefferson Frank. However, Booz Allen ranked sixth in terms of job openings within the dataset.</p><pre># plotting the top 30 companies <br>fig, ax = plt.subplots(figsize=(20, 14))<br>ax.barh(top_companies[&quot;company_name&quot;], top_companies[&quot;counts&quot;], color=&#39;#ff5733&#39;)<br>ax.set_title(&quot; top 30 Nation&#39;s Highest Recruiting Companies for Data science/analytics jobs&quot;)<br>ax.set_xlabel(&quot;Number of openings&quot;, fontdict ={&#39;fontsize&#39;:18 })<br>ax.set_ylabel(&quot;Company_name&quot;,fontdict ={&#39;fontsize&#39;:18 })<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ow5WCM082xxP1iW5dxoWbw.png" /><figcaption>Top 30 Top Companies for Data Science/ Analytics Openings.</figcaption></figure><h4>2. Top industries/sectors</h4><p>Additionally, I was curious to find out which sectors were the most prominent in the analytics job market, and here is what I discovered: The top sector, was an unnamed field or others, followed by the computer/internet, or information technology sector, following were physical and social sciences, then engineering or architecture and business services.</p><pre>#plotting domain representation for top 20 domains<br>fig = plt.figure(figsize=(30, 18))<br>grouped = jobs.groupby(&#39;domain&#39;).size().reset_index(name=&#39;count&#39;)<br>sorted_grouped = grouped.sort_values(by=&#39;count&#39;, ascending=False)<br>top_20 = sorted_grouped.head(20)<br>others = pd.DataFrame({<br>    &#39;domain&#39;: [&#39;Others&#39;],<br>    &#39;count&#39;: [sorted_grouped[&#39;count&#39;][20:].sum()]<br>})<br>top_20 = pd.concat([top_20, others])<br># Create a pie chart using the top 20 values and &quot;Others&quot;<br>plt.pie(top_20[&#39;count&#39;], labels=top_20[&#39;domain&#39;], textprops={&#39;fontsize&#39;: 13},autopct=&#39;%1.1f%%&#39; ,startangle=90, wedgeprops={&#39;width&#39;: 0.5})<br>plt.title(&#39;Top 20 data science job domains representation  &#39;)<br>plt.show()<br>plt.savefig(&#39;Sectors.png&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JfEaVtt5ummaRDJPprBeeQ.png" /><figcaption>Top 20 Data Science job domains representations</figcaption></figure><h4>3. Top Analytics Job titles</h4><p>I was interested in finding the most commonly advertised analytics jobs so that I, personally, could refine my job search in the future using more effective keywords. Although there were some duplicates in the dataset, I decided to present the information as it appears on job posting platforms. According to my analysis, the top job title was Data Scientist, followed by Data Analyst. Other roles that made an appearance included Data Visualization, Big Data Engineer, and SQL Data Scientist.</p><pre>#grouping by titles and counting <br>titles_count = jobs.groupby(&quot;job_title&quot;).size().reset_index(name =&quot;counts&quot;)<br># sorting top 30 titles<br>top_titles = titles_count.sort_values(&quot;counts&quot;, ascending= False).head(30)<br>#reordering in a descending order<br>top_titles = top_titles.iloc[::-1]<br># plotting the top 30 companies <br>fig, ax = plt.subplots(figsize=(20, 14))<br>ax.barh(top_titles[&quot;job_title&quot;], top_titles[&quot;counts&quot;], color=&#39;#e9399e&#39;)<br>ax.set_title(&quot; top 30 job titles for Data science/analytics jobs&quot;)<br>ax.set_xlabel(&quot;Number of openings&quot;, fontdict ={&#39;fontsize&#39;:18 })<br>ax.set_ylabel(&quot;job_title&quot;,fontdict ={&#39;fontsize&#39;:18 })</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCiIaRbSTJE9VTpIqy70EQ.png" /><figcaption>Top 30 titles for Data science/analytics jobs</figcaption></figure><h4>4. Salaries</h4><ul><li>The Salary column in the merged dataset was inconsistent due to the use of different syntaxes in different data sources.</li><li>Some values were reported in ranges, while others contained symbols like “K” or “$”.</li><li>To clean the data, I used the regex package to eliminate all symbols or extra expressions and only kept numbers.</li><li>For data reported in ranges, I calculated the mean salary.</li><li>I defined salary bins starting from less than <strong>$50,000 up to $600,000</strong>, salaries less than <strong>50,000$ </strong>constituted <strong>70.24%</strong> of the data.</li></ul><pre><br>bins = [0,50000,70000,90000,110000,130000,150000,170000,190000,210000,250000,300000,350000,400000,500000,600000]<br>binned_data = pd.cut(jobs[&#39;mean&#39;], bins=bins, include_lowest=True)<br># Count the number of values in each bucket using `value_counts`<br>counts = binned_data.value_counts(sort=False)<br># Create a bar chart of the counts<br>ax = counts.plot(kind=&#39;bar&#39;, edgecolor=&#39;black&#39;)<br># Set axis labels and title<br>ax.set_xlabel(&#39;Range&#39;)<br>ax.set_ylabel(&#39;Count&#39;)<br>ax.set_title(&#39;Salary Ranges&#39;)<br># Show the plot<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*S4u_p5Ag5zwMg8qTeRpEEA.png" /><figcaption>Salary ranges plot as generated from python</figcaption></figure><h4>5. Top Openings Cities</h4><p>Following the same approach as before, I sought to identify the cities with the greatest number of Data Science job openings. I included duplicates in my analysis in order to reflect the way they are presented on job portals. Based on my findings, the cities with the highest frequency of job postings were New York, followed by Chicago, San Francisco, Austin, San Diego, Boston, and Seattle.</p><pre># plotting the top 30 companies <br>fig, ax = plt.subplots(figsize=(20, 14))<br>ax.barh(top_cities[&quot;location&quot;], top_cities[&quot;counts&quot;], color=&#39;#800000&#39;)<br>ax.set_title(&quot; top 40 job cities for Data science/analytics jobs&quot;)<br>ax.set_xlabel(&quot;Number of openings&quot;, fontdict ={&#39;fontsize&#39;:18 })<br>ax.set_ylabel(&quot;location&quot;,fontdict ={&#39;fontsize&#39;:18 })</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A74YcdJ_mEKRIzuHPJJChA.png" /></figure><h4>6. Basic Natural Language Processing (NLP)</h4><p>To construct this engine, I relied heavily on natural language processing. I have taken a few simple measures to extract the most relevant information from the [‘job_description’] field, which gives primary glimpses for the matching. Basic NLP techniques were employed to generate the word cloud presented below. I followed the following steps in this sequence to achieve these results:</p><ul><li>Converting all text to lowercase</li><li>Removing whitespace</li><li>Eliminating punctuations</li><li>Tokenizing the document</li><li>Eliminating stop words</li><li>Applying lemmatization</li><li>Implementing stemming</li></ul><pre>lemmatizer = WordNetLemmatizer()<br>stemmer = PorterStemmer()<br><br># defining a function to preprocess text<br>def preprocess_text(text):<br>    # convert to lowercase<br>    text = text.lower()<br>    # remove whitespace<br>    text = text.strip()<br>    # remove punctuation<br>    text = text.translate(str.maketrans(&#39;&#39;, &#39;&#39;, string.punctuation))<br>    # tokenize the text<br>    tokens = nltk.word_tokenize(text)<br>    # remove stop words<br>    tokens = [token for token in tokens if token not in stopwords.words(&#39;english&#39;)]<br>    # apply lemmatization to each token<br>    tokens = [lemmatizer.lemmatize(token) for token in tokens]<br>    # apply stemming to each token<br>    tokens = [stemmer.stem(token) for token in tokens]<br>    return tokens<br><br># apply preprocessing to the job_description column<br>preprocessed_text = []<br>for doc in jobs[&#39;job_description&#39;]:<br>    preprocessed_text.append(preprocess_text(doc))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*QFCnGgO237zdHTto-xyOUA.png" /><figcaption>job_description field word cloud as generated by python word cloud package</figcaption></figure><p>The visualization accurately depicts the textual field and presents a visually appealing representation. Words that appear with higher frequency, such as “data science,” “communication skills,” “data analyst,” “years of experience,” and “computer science,” are expected and common in job postings. Additionally, other demographic information, such as race, color, sexual orientation, and religion, is also typical. We can gain insight into the qualities and qualifications that companies are seeking in potential candidates by just looking deeply into this plot.</p><p>Following I conducted bigram and trigram analysis.</p><p><strong>Bigram &amp; Trigram Analysis Graph</strong></p><ul><li>Bigram and Trigram analysis did not reveal any significant findings, results were mostly related to standard application prerequisites like demographics and veteran status.</li><li>The analysis revealed interesting insights like the prevalence of natural language processing terms (1000+), large data sets, and machine learning models.</li><li>Written communication skills were found to be a top soft skill.</li><li>Computer science degrees had a frequency of just over 2000, indicating a preference for data science job candidates with a computer science background.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XO-6-cdbGsnbQDe0R8OYwA.png" /><figcaption>Bigram Graph for job_description field as generated by python</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*cwom3qVOtrvQ_yIDlV0WAw.png" /><figcaption>Trigram Graph for job_description field as generated by python</figcaption></figure><h3>The Recommender Engine</h3><blockquote><strong><em>The rationale</em></strong></blockquote><blockquote>1. The central concept of the engine involves taking a PDF-formatted resume as input and transforming it into a textual field using PyPDF2.</blockquote><blockquote>2. The text is then subjected to basic natural languages processing operations such as punctuation and stopword removal to enhance semantic capture.</blockquote><blockquote>3. Transformation to the vector space: Next, the tokenized text file is input into a pre-trained doc2vec model featuring tagged documents drawn from a reference dataset(our dataset).</blockquote><blockquote>4. The model is used to map the textual field onto the vector space, followed by the computation of various similarity measures between the model vectors and the recently vectorized field.</blockquote><blockquote>5. Based on the similarity scores, the algorithm selects the top five ranked jobs and displays them in a data frame.</blockquote><p><strong>The Math</strong></p><ul><li>This model is centered on the use of doc2vec, which is an extension of the word2vec model.</li><li>Word2Vec is a neural network model that represents words as vectors in a high-dimensional space.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hELlVp9hmZbDZVFstS61pg.png" /></figure><ul><li>This approach aims to place semantically similar words closer together in the vector space, based on the idea that similar words have similar vector representations.</li><li>Word embeddings, which are vectors of real numbers used to represent words, can be generated using either the skip-gram or the continuous bag-of-words (CBOW) algorithm.</li><li>Doc2Vec is an extension of Word2Vec that incorporates a unique document ID or tag into the input of the model, in addition to the words, enabling the model to learn document-level embeddings.</li><li>The trained Doc2Vec model can be leveraged to compute document similarity, which is exactly what has been implemented in this project.</li></ul><h4><strong>Fine-tuning the code</strong></h4><p>The model was developed using the gensim library’s tagged documents and the doc2vec model. To obtain the most accurate similarity measures, I had to fine-tune the model’s parameters several times before identifying issues with my dataset, which I will discuss later. The doc2vec model function includes the following parameters:</p><ul><li><strong>Tagged docs</strong>: <em>This is the argument passed from the tagged documents parameter, which was created by iterating through the processed job description field after applying a basic NLP cleaning function.</em></li><li><strong>Vector size</strong>: <em>This parameter represents the degree to which we want our text to be represented. Higher vector sizes provide more representation but can result in underfitting while the opposite is true for overfitting.</em></li><li><strong>Window</strong>: <em>This parameter specifies the maximum distance between the current and predicted word within a sentence, and is used by the model to predict words based on their contexts.</em></li><li><strong>Min_count:</strong> <em>This parameter specifies the minimum number of words a vector must share to be considered similar. Higher min_count values can result in underfitting, while lower values can lead to overfitting. Therefore, finding a balance is key.</em></li><li><strong>Workers</strong>: <em>This parameter specifies the number of worker threads to use for training the model.</em></li></ul><h3>Doc2Vec Code</h3><pre># Convert preprocessed job postings text into TaggedDocument objects<br>tagged_docs = [TaggedDocument(words=text.split(), tags=[i]) for i, text in enumerate(jobs[&#39;processed_text&#39;])]<br># Train a Doc2Vec model using the TaggedDocument objects<br>model = Doc2Vec(tagged_docs, vector_size=70, window=5, min_count=7, workers=3)<br># Infer vector representations for preprocessed job postings text<br>jobs[&#39;vector&#39;] = jobs[&#39;job_description&#39;].apply(lambda x: model.infer_vector(x.split()))</pre><h3>The Visualizations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHy7mdhOXHRunywCmAvZ8g.png" /><figcaption>A Visualization of 3D Projection of the vectors using t-SNE</figcaption></figure><h4>Similarity Measures</h4><p><strong>Cosine similarity</strong> is a mathematical measure of similarity between two non-zero vectors in space. It is calculated by determining the cosine of the angle between the vectors. The resulting values fall between -1 and 1, with -1 indicating completely dissimilar vectors and 1 indicating identical vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*mJwpCspK1atgt0PJO66NTQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*TvJ6APyET_L14O3POr8ryA.png" /></figure><p>Similarly, <strong>Euclidean similarity</strong> is a distance-based measure that computes the similarity between two vectors based on the distance between them in a multi-dimensional space. It uses the Pythagorean theorem to calculate the distance between the vectors represented as points in that space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*hw3R2qC8wtybFLnaDy8qhQ.png" /></figure><p>Furthermore, The <strong>Manhattan similarity</strong> is a distance-based similarity measure used to calculate the similarity between two vectors in a given space. It determines the distance between the two vectors by adding up the absolute differences of their coordinates along each dimension. This approach is similar to measuring the distance between two points on a map with a grid-like structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*vU2BciJOoQmiqSnvW6aRtg.jpeg" /></figure><p>To check if the code was functional, I submitted one resume to the system. The resume was in PDF format, which was converted to text and then subjected to NLP processing before being vectorized. To identify the top 5 job matches, the system measured the similarity between the resulting vector and all the job vectors. For my project, I employed all three similarity metrics — cosine, Euclidean, and Manhattan.</p><pre>from sklearn.metrics.pairwise import cosine_similarity<br># Calculate the cosine similarity between my text vector and job vectors<br>cosine_similarities = cosine_similarity(text_vector.reshape(1, -1), jobs[&#39;vector&#39;].tolist())[0]<br><br># I will normalize the cosine similarity scores between 0 and 1 using min-max normalization<br>normalized_similarities = (cosine_similarities - np.min(cosine_similarities)) / (np.max(cosine_similarities) - np.min(cosine_similarities))<br><br># I will retrieve the indices of the top 5 most similar jobs<br>top_job_indices = np.argsort(cosine_similarities)[-5:]<br><br># I will create a DataFrame of the top 5 most similar jobs for my resume<br>top_jobs_df = pd.DataFrame({<br>    &#39;job_ID&#39;: jobs.iloc[top_job_indices][&#39;job_ID&#39;].tolist(),<br>    &#39;job_title&#39;: jobs.iloc[top_job_indices][&#39;job_title&#39;].tolist(),<br>    &#39;job_description&#39;: jobs.iloc[top_job_indices][&#39;job_description&#39;].tolist(),<br>    &#39;similarity_rank&#39;: [1, 2, 3, 4, 5],<br>    &#39;similarity_score&#39;: cosine_similarities[top_job_indices],<br>    &#39;normalized_score&#39;: normalized_similarities[top_job_indices]<br>})<br>#show results<br>top_jobs_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WcQiNIu0dYg9_oC1GOrPVg.png" /><figcaption>results data frame as generated by the above code for cosine similarity</figcaption></figure><p>I repeated this code for cosine, euclidian, and Manhattan similarity respectively.</p><h3><strong>Code and Methods of Evaluation</strong></h3><p>Once I confirmed that the concept worked, I extended it to a larger dataset consisting of 30 resumes stored in a CSV file. To create the file, I used the PyPDF2 package and the same code that I used to process the initial resume. I then performed basic natural language processing (NLP) cleansing on the text in the CSV file and vectorized it using the model_infer parameter with the Doc2Vec model. After the conversion, the data appeared as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*vnj2yfmkd-UC_S1BiJOKkQ.png" /><figcaption>resumes file after nlp processing and vectorizing</figcaption></figure><p>I categorized my set of 30 resumes into three groups based on job type: analytics jobs (resumes 0–10 and 20–30) and intentionally biased chef jobs (resumes 11–20) for later evaluation of precision, recall, and F1 scores. Using a threshold of 0.8, I assigned a value of 1 to the “matched” column if the normalized similarities exceeded 0.8, otherwise, 0 was assigned. As my engine was trained against analytics jobs, any resumes between 11 and 20 that were matched were considered irrelevant and classified as such in the “status” column. For resumes, 0–10 and 20–30, a value of 0 indicated irrelevance, and a value of 1 indicated relevance. Using these criteria, I computed the precision, recall, and F1 score of the engine.</p><p>The codes used are as follows:</p><pre>import pandas as pd<br>import numpy as np<br>from sklearn.metrics.pairwise import cosine_similarity<br><br># Step 1: Create an empty dataframe to store the results<br>result_df = pd.DataFrame(columns=[&#39;resume_index&#39;, &#39;resume_title&#39;, &#39;resume_text&#39;, &#39;job_id&#39;, &#39;job_title&#39;, &#39;job_description&#39;, &#39;similarity&#39;, &#39;normalized_similarity&#39;, &#39;similarity_rank&#39;, &#39;match&#39;, &#39;status&#39;])<br><br># Step 2: Iterate over each row in the resumes dataframe<br>for resume_index in range(len(resumes)):<br>    <br>    # Get the current resume&#39;s vector representation<br>    resume_vector = resumes[&#39;resume_vector&#39;][resume_index]<br>    <br>    # Calculate the cosine similarity between the current resume and all jobs<br>    similarities = cosine_similarity(resume_vector.reshape(1, -1), jobs[&#39;vector&#39;].tolist())[0]<br>    <br>    # Get the indices of the top 5 highest similarity scores<br>    top_indices = similarities.argsort()[-5:][::-1]<br>    <br>    # Step 3: Iterate over the top 5 similar jobs and add them to the result dataframe<br>    for rank, job_index in enumerate(top_indices):<br>        similarity = similarities[job_index]<br>        normalized_similarity = (similarity - np.min(similarities)) / (np.max(similarities) - np.min(similarities))<br>     <br>        if (normalized_similarity &gt; 0.80).any():<br>            match = 1<br>        else:<br>            match = 0<br><br>        if resume_index &lt; 10:<br>            if match == 1:<br>                status = &#39;relevant&#39;<br>            else:<br>                status = &#39;irrelevant&#39;<br>        elif resume_index &lt; 20:<br>            if match == 1:<br>                status = &#39;irrelevant&#39;<br>            else:<br>                status = &#39;relevant&#39;<br>        else:<br>            if match == 1:<br>                status = &#39;relevant&#39;<br>            else:<br>                status = &#39;irrelevant&#39;   <br>        <br>        job = jobs.iloc[job_index]<br>        result_df = result_df.append({<br>            &#39;resume_index&#39;: resume_index,<br>            &#39;resume_title&#39;: resumes[&#39;resume_title &#39;][resume_index],<br>            &#39;resume_text&#39;: resumes[&#39;resume_text&#39;][resume_index],<br>            &#39;job_id&#39;: job[&#39;job_ID&#39;],<br>            &#39;job_title&#39;: job[&#39;job_title&#39;],<br>            &#39;job_description&#39;: job[&#39;job_description&#39;],<br>            &#39;similarity&#39;: similarity,<br>            &#39;normalized_similarity&#39;: normalized_similarity,<br>            &#39;similarity_rank&#39;: rank+1,<br>            &#39;match&#39;: match,<br>            &#39;status&#39;: status<br>        }, ignore_index=True)<br><br># Step 6: Print the result dataframe<br>result = result_df.groupby(&#39;resume_index&#39;).apply(lambda x: x[[&#39;resume_title&#39;, &#39;resume_text&#39;, &#39;job_id&#39;, &#39;job_title&#39;, &#39;job_description&#39;, &#39;similarity&#39;, &#39;normalized_similarity&#39;, &#39;similarity_rank&#39;, &#39;match&#39;,&#39;status&#39;]].sort_values(&#39;similarity_rank&#39;, ascending=False))</pre><pre>precision = len(result[result[&#39;status&#39;]==&quot;relevant&quot;]) / len(result)<br>recall =  len(result[result[&#39;status&#39;]==&quot;relevant&quot;])/ 100 <br>f1_score = 2 * (precision * recall) / (precision + recall)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uQlVXe_bqyhagz9wxdVqYw.png" /><figcaption>The final layout of result groups -grouped by indexes numbers</figcaption></figure><h3><strong>Results, Discussion, and Limitations</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*Qj-vyVobvW4c1n99OvwVTQ.png" /><figcaption>Evaluation Metrics as generated by Python</figcaption></figure><p>Although the recommendation engine demonstrated promising results, there were some limitations and potential errors that must be acknowledged. Cosine similarity performed better than Euclidian and Manhattan similarity for F1-score. However, obtaining a true measurement of model performance without ground truth was a significant obstacle.</p><p>One potential issue is that the algorithm may force-fit data that is far from the feed, leading to Type I errors. Moreover, due to the data’s biased nature, where it leans towards analytics jobs, the model may be overfitting, resulting in potential errors. The volume of data used was also considered small compared to models trained on textual data.</p><p>To mitigate these limitations and potential errors, gathering more diverse and representative data is recommended, and other evaluation metrics should be used to assess model performance. Moreover, implementing a human-in-the-loop approach for verifying the model’s output could help reduce the risks of false positives or false negatives. Despite these limitations, I believe, personally, this was a good effort, a great educational experience, and a proof of concept, and future enhancements could potentially improve the recommendation engine’s performance.</p><p><strong>GitHub Link:</strong></p><p><a href="https://github.com/RaffaNimir/Match-Up-recruiter-"><strong>https://github.com/RaffaNimir/Match-Up-recruiter-</strong></a></p><p><strong>References:</strong></p><ul><li><strong>Desku, Astrit &amp; Raufi, Bujar &amp; Luma, Artan &amp; Selimi, Besnik. (2022). Recommender Model for Secure Software Engineering Using Cosine Similarity Measures. International Journal of Engineering and Advanced Technology. 11. 10.35940/ijeat.E3628.0611522. </strong><a href="https://www.researchgate.net/publication/361208661_Recommender_Model_for_Secure_Software_Engineering_Using_Cosine_Similarity_Measures/citation/download"><strong>https://www.researchgate.net/publication/361208661_Recommender_Model_for_Secure_Software_Engineering_Using_Cosine_Similarity_Measures/citation/download</strong></a></li><li><strong>Doc2Vec — Computing Similarity between Documents: </strong><a href="https://medium.com/red-buffer/doc2vec-computing-similarity-between-the-documents-47daf6c828cd"><strong>https://medium.com/red-buffer/doc2vec-computing-similarity-between-the-documents-47daf6c828cd</strong></a></li><li><strong>Introduction to Word Embedding and Word2Vec : </strong><a href="https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa"><strong>https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa</strong></a></li><li><strong>V. U. Thompson, C. Panchev and M. Oakes, “Performance evaluation of similarity measures on similar and dissimilar text retrieval,” <em>2015 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K)</em>, Lisbon, Portugal, 2015, pp. 577–584. </strong><a href="https://ieeexplore.ieee.org/document/7526978"><strong>https://ieeexplore.ieee.org/document/7526978</strong></a></li><li><strong>Proofread by: ChatGPT [Computer software]. (2023). OpenAI. </strong><a href="https://openai.com/blog/chatting-with-ai/"><strong>https://openai.com/blog/chatting-with-ai/</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22d8bcfe9e88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Between Humans and ML/AI: analogies from both worlds decoded (1)]]></title>
            <link>https://raffanimir.medium.com/between-humans-and-ml-ai-analogies-from-both-worlds-decoded-b7990e13427e?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7990e13427e</guid>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Thu, 20 Oct 2022 06:01:33 GMT</pubDate>
            <atom:updated>2022-10-28T01:41:13.712Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*sSm6v8J9HGDF95UnRoFMeQ.png" /><figcaption><a href="https://www.callcentrehelper.com/finding-harmony-between-human-and-artificial-intelligence-117508.htm">https://www.callcentrehelper.com/finding-harmony-between-human-and-artificial-intelligence-117508.htm</a></figcaption></figure><p>Machine learning and AI have taught me a lot about people. I see ML/AI as an attempt to simulate the human experience through machines. In doing so, ML focused on capturing two essential constructs: knowledge and judgment. ML considers data as the raw essence of learning on the “knowledge construct,” meeting in the human analogy what is known as “experience.” Data in its broad meaning extend to include all sorts of structured and unstructured data; numbers, texts, images, voice, or whatever; coming closer day by day to a perfect simulated analogy of our reality. But data without training/analysis means absolutely nothing, doesn’t it?</p><p>Here comes Judgment. The machine’s analogy was simulated through algorithms, code, decision trees, and much more, Trying to capture informed decisions in the human analogy and, most frankly, exceeding it in most parts, especially in computational analysis.</p><p>In my analysis, I took a reverse route, as dissembling the human experience in the analogy of ML made me understand the human experience better. I began to view humans differently; we are very likely to these machines with little distinctions. They exceed us on the computational part, and we exceed them on the emotional intelligence we acquire. But for the most part, the analogy is quite similar; how we make decisions, how we react to the “data/past knowledge” that we acquired; how that affects our decision-making process. We are, to an extent, very similar.</p><p>Let’s weigh this into our everyday decisions; we are faced with situations that require us to decide: for example, picking a drink from a variety of drinks at a supermarket. Every decision we make in our day, whether simple or complex, to the extent that decision is a result of an analysis we run on our previous knowledge of life. You avoid yogurt cause you are lactose intolerant, and so on. The critical point here is that you make decisions based on your previous knowledge. For the most part.</p><p>This means that we refer to the data system inside of us, project, and react to that system more than what’s in front of us — having less to deal with the situation itself-but really more to do with our accumulated knowledge through the years. Having this understanding has helped me soothe and confront many life incidents involving other people and distress. Understanding that whoever standing in front of you is walking with a unique set of “knowledge/data” and reacting 70% based on them and 30% to what is really happening. (I stand by this argument people tend to patter-nize things and generalize “familiarity biases”)</p><p>This makes all humans a black box that I only know what I thought I knew and most definitely lack the knowledge for the most part. This, for me, takes out judgment, perception, and prediction outside of the equation. Again, you only know what you think you do, and -again-that’s not enough knowledge for the most part.</p><p>Simulating that is opening up more and more to me every day. Making me exchange visits from both worlds (humans/machines) and be highly curious about both.</p><p>It only takes trying to simulate a system to fully capture all its complexities. Anything can seem simple until you try to build it or create it yourself.</p><p><a href="https://www.facebook.com/hashtag/machinelearning?__eep__=6&amp;__cft__[0]=AZVxo9wrSGSvt5Q04nvYeOSlwcntmltCbbO7DfhG8TcyjzRwZA89o4TsO_I8AZDGEOqbe7lz_jV1vzVSuubkDQEIzgwp5aUBT9VmnPZidRG1hUugLRpzcyKonaxbbRc6EHcieLHyQkl4GKm9zuFxqlunbnPdXuVTYMGBqUYpDl0jUw&amp;__tn__=*NK-R">#machinelearning</a> <a href="https://www.facebook.com/hashtag/artificalintelligence?__eep__=6&amp;__cft__[0]=AZVxo9wrSGSvt5Q04nvYeOSlwcntmltCbbO7DfhG8TcyjzRwZA89o4TsO_I8AZDGEOqbe7lz_jV1vzVSuubkDQEIzgwp5aUBT9VmnPZidRG1hUugLRpzcyKonaxbbRc6EHcieLHyQkl4GKm9zuFxqlunbnPdXuVTYMGBqUYpDl0jUw&amp;__tn__=*NK-R">#artificalintelligence</a> <a href="https://www.facebook.com/hashtag/simulation?__eep__=6&amp;__cft__[0]=AZVxo9wrSGSvt5Q04nvYeOSlwcntmltCbbO7DfhG8TcyjzRwZA89o4TsO_I8AZDGEOqbe7lz_jV1vzVSuubkDQEIzgwp5aUBT9VmnPZidRG1hUugLRpzcyKonaxbbRc6EHcieLHyQkl4GKm9zuFxqlunbnPdXuVTYMGBqUYpDl0jUw&amp;__tn__=*NK-R">#simulation</a> <a href="https://www.facebook.com/hashtag/humans?__eep__=6&amp;__cft__[0]=AZVxo9wrSGSvt5Q04nvYeOSlwcntmltCbbO7DfhG8TcyjzRwZA89o4TsO_I8AZDGEOqbe7lz_jV1vzVSuubkDQEIzgwp5aUBT9VmnPZidRG1hUugLRpzcyKonaxbbRc6EHcieLHyQkl4GKm9zuFxqlunbnPdXuVTYMGBqUYpDl0jUw&amp;__tn__=*NK-R">#humans</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7990e13427e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RMarkdown feedback from a newbie]]></title>
            <link>https://raffanimir.medium.com/rmarkdown-feedback-from-a-newbie-524f663370b9?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/524f663370b9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[r-markdown]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Wed, 28 Sep 2022 04:10:12 GMT</pubDate>
            <atom:updated>2022-09-28T04:42:58.999Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/211/1*LRW4C-s2PT5EeX4-pJAH2g.png" /><figcaption>photo source: <a href="https://drizzt.home.amu.edu.pl/images/RProg/W4.pdf">https://drizzt.home.amu.edu.pl/images/RProg/W4.pdf</a></figcaption></figure><p>I recently was exposed to Rmarkdown through a course I am taking for my master’s. And man, was it fascinating to learn!</p><p>Rmarkdown is a code-text interface with a fantastic package that builds HTML pages. Now switching from the back-end to the front-end is done through “Knitting.” Constructed primarily in R, Rmarkdown has an amicable and comprehendible “error language” that makes learning enjoyable as you run through mistakes or errors.</p><p>When I started Rmarkdown, I was both exposed to R and python by that time. Using Rmarkdown, I had the flexibility to alter them as I worked. Which I found very useful! Though, that’s not the reason why I love Rmarkdown. Being exposed to a high volume of new knowledge in a compressed time has grown inside of me a massive appreciation for documentation. I now try to document all I have learned for two purposes: to shift knowledge from short-term to long-term memory. Second is that through articulating my knowledge, I, myself, get a quick assessment of my level of understanding of it. And this is something R markdown allows me to exercise freely; I can keep text and code in the same window without having to (#) Hash everything up! LOL</p><p>In Rmarkdown, you can add images, figures, and plots and style your webpage with many themes that will render beautifully on your HTML page. Styling your webpage could be done through straightforward and biggener-friendly codes.</p><p>Rmarkdown is probably the best transit point to get exposure to webpage coding. Rmarkdown is HTML friendly, and thus you might find yourself writing a few codes here and there in HTML; but as you are so wrapped up in building your webpage, you will slowly start to notice that you understand HTML syntax, which is a lovely calm way to introduce yourself to HTML.</p><p>For all that, I found Rmarkdown an integrated comfortable, and exciting tool to combine all coding knowledge with a little web page, HTML, and design exposure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=524f663370b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Talk statistics: Hypothesis Testing Cheatsheet #2]]></title>
            <link>https://raffanimir.medium.com/lets-talk-statistics-hypothesis-testing-cheatsheet-2-cbe19eb8c24e?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbe19eb8c24e</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Sun, 06 Mar 2022 04:56:49 GMT</pubDate>
            <atom:updated>2022-03-07T04:15:14.835Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week I have posted about how to think, and the steps to take when conducting a hypothesis test. If you have missed last week post review it <a href="https://medium.com/@rafaanimir/lets-talk-statistics-hypothesis-testing-cheatsheet-1-5115d4f89e8c">here</a>.</p><p>A quick review is that when we are conducting a hypothesis test, we have a parameter in question and we want to verify whether the parameter value is true (Null Hypothesis), or otherwise the parameter takes another value (alternative Hypothesis).</p><p>There are 2 ways to test the claim, one using the test statistics. When doing so the general rule is that:</p><ol><li>A z test is used when the population variance is known.</li><li>A t-test is used when the population variance is unknown but the sample variance is known.</li></ol><p>Last week we explained how we conduct the test using the test statistics.</p><p>This week I will discuss the second method usually used to conduct a hypothesis test.</p><p><strong>Method 2: The p-value.</strong></p><p>The p-value is the second method that is often used when conducting hypothesis tests. The p-value is the probability related to the test statistics so if you can imagine this on a bell curve a test statistic will lie on the x-axes and the p-value is the correlated probability.</p><p>To solve the hypothesis test using the p-value method follow the following steps:</p><ol><li>Determine your significance value( which is usually given in the problem).</li><li>Sketch the normal distribution curve and determine where the critical values lie in the plot. [(alpha, (1-alpha) — one-tailed test][(alpha/2, (1-alpha/2) — two-tailed test)].</li><li>Rationally understand where are your reject and non-reject regions are, and have them clearly dashed and identifiable in your plot.</li><li>Calculate your test statistics whether it&#39;s Z or T.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/1*bwl0-Yb39gj4JsIjfSXMmg.png" /><figcaption><a href="https://medium.com/@rafaanimir/lets-talk-statistics-hypothesis-testing-cheatsheet-1-5115d4f89e8c">https://medium.com/@rafaanimir/lets-talk-statistics-hypothesis-testing-cheatsheet-1-5115d4f89e8c</a></figcaption></figure><p>where x bar = sample average</p><p>M = Population mean</p><p>Sigma = standard deviation</p><p>n = sample size</p><p>When the population variance is unknown we use a The T value following this formula :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*qo85nB9G8TzeaFVjkjAwSA.png" /></figure><p>6. After computing these Z or T values, get the probability for them. Using Excel functions <strong>Norm.S.Dist</strong> in case of Z test statistic or <strong>T.Dist</strong> in terms of the T statistic.</p><p>7. Plot the value obtained in the sketch, determine whether or not it falls in your critical regions. (Alpha regions)</p><p>8. Decide on whether to Reject or fail to Reject based on where your P-value lies at the plot rejection and nonrejection regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*egc8hBHvn4UnKqWhdcpU-g.png" /><figcaption>Image source: <a href="https://www.researchgate.net/figure/Rejection-and-non-rejection-regions-of-the-null-hypothesis_fig1_333914148">https://www.researchgate.net/figure/Rejection-and-non-rejection-regions-of-the-null-hypothesis_fig1_333914148</a></figcaption></figure><p>An important aspect to notify is that when the critical region for the p-value is not pre-determined in your problem, there is a general rule of thumb to believe that it is 0.05. So solved with this conception.</p><p>So that’s it for today’s blog post. If you found this blog post helpful, show some support by clapping, following, or commenting. I will be back with more on statistics and Data Science topics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbe19eb8c24e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Talk statistics: Hypothesis Testing Cheatsheet #1]]></title>
            <link>https://raffanimir.medium.com/lets-talk-statistics-hypothesis-testing-cheatsheet-1-5115d4f89e8c?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/5115d4f89e8c</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[calculations]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Sat, 26 Feb 2022 20:45:55 GMT</pubDate>
            <atom:updated>2022-02-26T20:45:55.792Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*ZoNNILqqV0HFeQtiNSPHYA.jpeg" /><figcaption><a href="https://www.analyticssteps.com/blogs/what-hypothesis-testing-types-and-methods">https://www.analyticssteps.com/blogs/what-hypothesis-testing-types-and-methods</a></figcaption></figure><p>If you find yourself confused and too overwhelmed with statistical terms, well, This post is for you. Although I enjoy statistics very much, I often struggle to build an algorithm for myself for solving statistical problems.</p><p>In this blog post, we would be discussing hypothesis testing in a simple language that will ensure you solve it right each time. Hypothesis Testing is often done to test a claim about a parameter value. When we have a population distributed (almost always normally), we often face a situation where the mean or variance of this population is not definitively known. But estimated with a certain confidence level. And the problem always lies in verifying this estimation through Hypothesis testing. We often get a problem that speaks about a certain population or a random sample with X population mean values, x sample average, and Y standard deviation or variance, then we are told that the person conducting this experiment is perse 90% sure that the mean of this population is 27. Our aim here is to verify this claim. We follow these steps to answer the problem.</p><h4>1. <strong>Formulate the problem statement</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Te3ewQ6ImdhmUzUfi9upw.png" /><figcaption><a href="https://www.thoughtco.com/null-hypothesis-vs-alternative-hypothesis-3126413">https://www.thoughtco.com/null-hypothesis-vs-alternative-hypothesis-3126413</a></figcaption></figure><p><em>That is extract the null hypothesis and the alternative hypothesis, the null hypothesis is always the thing we want to verify, the ( status quo ) and the alternative hypothesis is where we are speculating otherwise.</em></p><p><em>The null hypothesis should always be an equality statement.</em></p><p><em>The alternative hypothesis is an inequality statement.</em></p><p><em>When we have smaller than or larger than statements this generates the so-called “One tail test”.</em></p><p>Where basically our reject region is bounded in one direction of the curve.</p><p><em>Right tail test if we are discussing bigger than boundaries and left tail test when we are discussing smaller than boundaries.</em></p><p><em>When we have a not equal alternative hypothesis then we acquire the so-called “two-tail test”.</em></p><p>The two-tailed test is when we are bounded with 2 critical values in where both the bigger than region , and the smaller than region constitute the boundaries of our system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*vVEk-0ePMSjKQICTQeIh9w.jpeg" /><figcaption><a href="https://www.fromthegenesis.com/difference-between-one-tail-test-and-two-tail-test/">https://www.fromthegenesis.com/difference-between-one-tail-test-and-two-tail-test/</a></figcaption></figure><p>2. <strong>Determine rejection and non-rejection region</strong></p><p>Start by roughly sketching the normal distribution bell curve then take a look back at the confidence level(CL) given in the problem. The (CL) will determine our significance level.</p><p><strong><em>The significance level</em></strong></p><p>The significance level alpha is calculated by the formula (1- CL) If we are dealing with a one-tail test. If we are dealing with the two-tailed situation then the alpha is divided by 2.</p><p>After determining our alpha regions we compute the values of x corresponding to these probabilities using the statistic variable(T or Z) and the inverse function in excel (<em>T.INV), (Z.INV)</em> suitable for the test and mark them on our plot and this now represents our <strong>critical values</strong>. The Z statistic is used when the variance is known and the T- student test statistic is used when we are oblivious of the variance value.</p><h4><strong>3. Define solution Methods</strong></h4><p>We often have two approaches when resolving a hypothesis test.</p><p><strong><em>1/ Method one: With the implication of Test-statistics</em></strong></p><p>Let’s assume that our situation requires a solution with the t-test. We bring two values of T . One is estimated using the<em> alpha value and the inverse function (T.INV), </em>and one is calculated given the parameters of the problem. Then we often compare the value calculated and where it falls in the hypothesis test regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*qo85nB9G8TzeaFVjkjAwSA.png" /></figure><p><strong>4. Decide</strong></p><p>if T computed falls within our reject zone then we reject the null hypothesis and if it is not within the reject region we conclude that there is not enough evidence to reject the null hypothesis. (hence we accept the null hypothesis ).</p><p><strong>5. Errors</strong></p><p>Now because we are deciding on hypothesis values it&#39;s reasonable to think that errors might occur. In hypothesis testing, we are subject to two types of error.</p><p><strong><em>Type 1 error or alpha error</em></strong></p><p>The Alpha error occurs when we reject the null hypothesis when it is true</p><p><strong><em>Type 2 error or Beta error</em></strong></p><p>The Beta error occurs when we don&#39;t reject the hypothesis when it’s wrong, to begin with.</p><p>The following Table would help you see the difference:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*AjzXgHUrh4NKY9RdMBUNhA.jpeg" /><figcaption><a href="https://www.simplypsychology.org/type_I_and_type_II_errors.html">https://www.simplypsychology.org/type_I_and_type_II_errors.html</a></figcaption></figure><p>if you liked this blog post follow me for more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5115d4f89e8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Being a NEW data analyst mean: the utter truth]]></title>
            <link>https://raffanimir.medium.com/what-being-a-new-data-analyst-mean-ba27ed101c30?source=rss-de23824a4dab------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba27ed101c30</guid>
            <category><![CDATA[career-paths]]></category>
            <category><![CDATA[career-change]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Raffa Nimir]]></dc:creator>
            <pubDate>Sat, 15 Jan 2022 21:42:55 GMT</pubDate>
            <atom:updated>2022-01-15T21:55:23.857Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5rXejhcKLorK7JiXnQH-tg.jpeg" /></figure><p>I come from an engineering background, and I sort of just jumped into the field of Data analytics. The major excites me for what it can do, the thought of being able to interpret and take decisions on enormous datasets, in addition to the ability to visualize data in a cool storytelling matter. That and many more reasons make data analysis a very hot topic and a cool major to pursue. <br>But to reach this level, the path is not easy and is often an unclear one. I started my journey by enrolling in a data analytics master’s program. The program and most programs out there are meant to be beginner-friendly, but the truth is that you will find yourself completely burned out.<br> I often felt lost and confused. I had to research and learn so much more because I was already missing a lot of the prerequisites. I was trying to adopt the basics and self-teach myself in a time I was also supposed to be mastering the skills I am just reviewing their acronyms. I did double work, I was racing with time and my ability to learn and recall stuff. <br>Above all, I had to keep my mental health sane in times of high self-doubt and fear. I had to believe in myself and keep focused. <br>the truth is data analytics is indeed a highly rewarding career path but the back-end work that you need to do to reach the levels of a professional is not easy. It involves heavy mathematics, high coding comfort and ability. In addition to a focused mindset and positive learning attitude because it will challenge you and frustrate you at many times.</p><p>I am writing this piece for all the people who are shifting their major to data analytics or computer science and they are swamped with the many skills they are to absorb in a short time. You are not alone.</p><p>The positive side to this is that we are in an era of digital learning, so literally, the resources are endless. With little extra work, we can reach and accomplish the skills of professionals.</p><p>Set the right expectations and be ready to do a lot of work before reaching the level that we all visualize and daydream about. Let’s support each other. Feel free to connect or leave comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba27ed101c30" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>