<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Elissa Irhamy on Medium]]></title>
        <description><![CDATA[Stories by Elissa Irhamy on Medium]]></description>
        <link>https://medium.com/@eairhamy?source=rss-bd65d19f5616------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kvFvKqtMiavClBkHKkOJvg.jpeg</url>
            <title>Stories by Elissa Irhamy on Medium</title>
            <link>https://medium.com/@eairhamy?source=rss-bd65d19f5616------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:33:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eairhamy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What I learned after working for 3 years as an entry level Data Analyst]]></title>
            <link>https://medium.com/@eairhamy/what-i-learned-after-working-for-3-years-as-an-entry-level-data-analyst-7fb4971faa0f?source=rss-bd65d19f5616------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fb4971faa0f</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-analyst]]></category>
            <category><![CDATA[entry-level-jobs]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[workplace]]></category>
            <dc:creator><![CDATA[Elissa Irhamy]]></dc:creator>
            <pubDate>Mon, 14 Feb 2022 02:25:33 GMT</pubDate>
            <atom:updated>2022-02-14T02:25:33.034Z</atom:updated>
            <content:encoded><![CDATA[<h4>(Hopefully) Some helpful lessons for fresh graduates and entry level data analysts</h4><p>Data analyst — it seems like every company needs at least one or two in their team nowadays. I graduated with a Masters in Data Analytics in May 2020 (yes, in the height of the pandemic) and have been working in this area as an intern in 2019 until now, as a full time Data Analyst in 2022. I have learned a lot in these 3 years and I’m going to tell you what they are!</p><h3>Lesson 1 — Interviews are scary but it doesn’t have to be</h3><p>I know it is nerve racking, I have done quite a few myself and the nervousness is sometimes the worst thing in the world. HOWEVER, if you have done your research, prepare some answers for possible questions and know your life inside out (which you do because uhh it’s your life?) then take a <strong>*deep breath*</strong> and <em>~relax~</em>. Remember, the interviewers are also human! It is okay to take your time thinking for an answer. Try to make it into a conversation as much as possible.</p><p>I interviewed candidates for a data analyst position recently and I could not tell you how important it is to be comfortable. They are testing your technical skills but they are also trying to see if your personality matches their group. Be yourself. TAKE A DEEP BREATH AND RELAX.</p><p><strong><em>Insider tip: Your introduction does not have to encompass your entire resume and it is fine to ask for clarifications on questions.</em></strong></p><h3>Lesson 2 — Yes skills are great but what’s better is understanding the system your company uses and the databases first</h3><p>You might think you have to know all the ways to clean a dataset or write a stored procedure on your first day but fear not, once you get the job, you will code every second of every hour that you are working. So don’t worry, those skills will come and improve with the million hours of practice. What is more important during the first few weeks is to ask questions on the data, the structure of the databases and how they are used. Write down as many notes as you need to and pay attention during the meetings.</p><h3>Lesson 3 — Is the “great attention to detail” requirement REALLY necessary? The short answer is yes.</h3><p>The long answer is also yes. The longer answer is, attention to detail is necessary when you are dealing with a multitude of data. Data is inconsistent. There will always be something unusual that pops up even when you use a particular data source all the time because you can’t possibly check each row in a dataset with one million rows, for example. I learned that triple and quadruple checks are good practice.</p><p>For example, I was joining two datasets from two different sources to one main dataset to complete the missing information. The size of the main dataset increases after the join. This can definitely happen if you are expecting many-to-many joins so I had to check if this is the case. It turns out that one of the external data sources had some rows with wrong information. If I hadn’t checked the size of the data frame after joining, this instance will cause problems later on. So, checking and testing the details are necessary!</p><h3>Lesson 4 — Document, document, document EVERYTHING</h3><p>This is THE MOST important and helpful thing I did while working. It started with me trying to jot down everything I learn so I don’t forget them but it turned into a running document of my daily work. I often work with multiple projects at a time, therefore, writing down what I did at the end of the day helped a lot when picking up my work the next day, the next week or even 2 months after. I use Google Docs and put the date as the header and project name as sub headers but there are a lot of apps you can use and style it to your liking.</p><p>Commenting what my code does WHILE WORKING ON IT also helps when you document your work at the completion of the project. I sometimes don’t even wait until I finish the project to document. I use almost every Friday to organize my code and write down the process so I can start the next week with ease. Documentation is also key when someone else has to read and use your code for future work.</p><h3>Lesson 5 – 80% of your work is the prep work with the data.</h3><p>I might underestimate the percentage of this but truly, after 3 years of doing data analysis work, I am confident in saying that the prep work takes the most time. Understanding the data from sources, reading the data dictionary, making sure the data is accessible, checking if we have enough data to use, cleaning them, understanding why certain columns are empty and then finally, after 5000 tests, checking if we can join them to our dataset to be used in our products. This grunt work is necessary to make sure your final result is up to par so expect these types of tasks in your daily work!</p><h3>Lesson 6 — Projects not getting made are normal and expected.</h3><p>This is sad but true. Sometimes you spend months conceptualizing a project, coming up with the work needed and researching methods to use to make it work, but in the end, the result is not good enough for publishing or for next steps. It is okay if your work doesn’t make it to the end, there is always more to work on :)</p><h3>Conclusion</h3><p>Data analysis can be messy, tedious and overwhelming at times but it is a work that can possibly solve new problems and open new opportunities with technology. So learn as much as you can and keep an open mind. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fb4971faa0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Bechdel Test Score using Machine Learning]]></title>
            <link>https://ai.plainenglish.io/predicting-bechdel-test-score-using-machine-learning-7253618a3f8?source=rss-bd65d19f5616------2</link>
            <guid isPermaLink="false">https://medium.com/p/7253618a3f8</guid>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[bechdel-test]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[seaborn]]></category>
            <dc:creator><![CDATA[Elissa Irhamy]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 02:21:40 GMT</pubDate>
            <atom:updated>2020-07-15T06:36:13.914Z</atom:updated>
            <content:encoded><![CDATA[<p>Can we use the most basic Supervised Machine Learning methods to predict Bechdel Test Score using movie metadata?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iSumWPpFPmGYltjmML-LDA.png" /><figcaption>Credit: <a href="https://www.themarysue.com/rick-morty-tackles-meta-narrative-bechdel-test/">https://www.themarysue.com/rick-morty-tackles-meta-narrative-bechdel-test/</a></figcaption></figure><p>Have you ever heard of the <strong>Bechdel Test</strong>? If you are a Rick and Morty’s fan, season 4 episode 4, <em>Never Ricking Morty</em>, referenced this test. <em>The Simpsons</em> also had Alison Bechdel, the founder of the Bechdel Test appear in season 29 episode 2 back in 2017. This test was founded in 1985 when a cartoonist Alison Bechdel asked in her comic strip, <em>Dykes to Watch Out For</em>, almost jokingly if movies have at least two women that talk to each other, not about a man. This is the premise of the Bechdel Test, to test the utmost minimum effort to include women in movies. <strong>Movies will pass the Bechdel Test if these three criteria are met:</strong></p><ol><li>at least two women characters</li><li>that talks to each other</li><li>not about men</li></ol><p>It sounds simple and was never intended to be a test <em>(O’Meara, 2016)</em>. Even so, only half of movies pass this test <em>(Chart 2)</em>.</p><p>Later on, various variants of this test were proposed. One example of the variant is: the two women should be named characters and the conversations should be more than 60 seconds long.</p><p><strong>Overview</strong></p><p>Looking at how this bare-minimum-effort-to-include-women-in-movies test is failed by many movies, <strong>we wanted to see if there is other information about a movie that can indicate if they will pass the Bechdel Test</strong>. In the technical terms, can movie metadata predict the Bechdel Test passing of a movie?</p><p><strong>Data</strong></p><p>To start this project we gathered the dataset needed:</p><ul><li><strong>Amazon reviews</strong>: <a href="http://jmcauley.ucsd.edu/data/amazon/">http://jmcauley.ucsd.edu/data/amazon/</a>, we selected the Movies and TV data set. The data contains information about movie reviews, it contains 1.7 million Movies and TV reviews spanning May 1996 — July 2014. This data set includes reviews (ratings, text, helpfulness votes) for a product ID.</li><li><strong>Bechdel Test API</strong>: This API contains the Bechdel scores for 8000+ movies from 1970 to 2019. We obtained all of the movies ever scored and took the movie title, their IMDb ID and Bechdel score for our project.</li><li><strong>TMDb API</strong>: We searched this API for movie title, IMDb ID, budget, revenue, profit, genre, cast and crew name, role and gender and character names and gender.</li><li><strong>Script data set from Cornell University</strong>: This script data set contains scripts from 600+ movies. It contains the movie title, movie id, character names, character gender, lines from each character and who they are conversing with.</li></ul><p>We <strong>cleaned and</strong> <strong>merged these datasets on the movie title</strong> to make sure we have all of the features needed for all movies. Although each dataset is huge, we ended up with a pretty small merge, about <strong>253 unique movies</strong>.</p><p><strong>Bechdel Test API movies statistics</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDsdi9_4ZPDuNEr1YJW7GQ.png" /><figcaption>Chart 1. Number of movies in the Bechdel API dataset by year</figcaption></figure><p>There are about 50% of movies in the Bechdel Test API that pass the Bechdel Test every year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9HoqF1MQdP5vJ4M1gUkEbw.png" /><figcaption>Chart 2. Fraction of movies that passes the Bechdel Test per year. *Disclaimer: 2019 has less than 20 movies at the time of project.</figcaption></figure><p>On average, Bechdel passing movies <strong>received less budget</strong> than non-passing Bechdel movies. Interestingly, when we look at their profit, both categories seem to <strong>gain similar profit.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*fXDxFBDb8UlmUoPRFXKEeg.png" /><figcaption>Chart 3. Budget for Bechdel passing and Bechdel non-passing movies by year. 1 represents passed movies while 0 represents failed movies.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*ZhmdrVW8crHMymv9AUfy5A.png" /><figcaption>Chart 4. Profit for Bechdel passing and Bechdel non-passing movies by year. 1 represents passed movies while 0 represent failed movies.</figcaption></figure><p>These graphs are made using Seaborn. Below is an example code with the dataset used for Chart 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*y0qsFhqnLNrCV-MgmodRNA.png" /><figcaption>Sample dataset to produce the graph above</figcaption></figure><pre>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><pre>plt.figure(figsize=(10,5))</pre><pre>chart = sns.barplot(x = ‘year’, y = ‘all_count’, data = bechdel_count_all)</pre><pre>chart.set_xticklabels(chart.get_xticklabels(), rotation=90)</pre><pre>plt.show()</pre><p>If you’re curious which celebrity works in Bechdel passing movies the most in our dataset <em>(because I certainly am)</em>, here you go!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*TogBKnm5T6HMYqBhRBUcqQ.png" /><figcaption>Hi, Meryl Streep, I see ya.</figcaption></figure><p><strong>Feature engineering</strong></p><p>Initially we thought to just use the conversations in the script dataset to imitate how movies are scored for the Bechdel Test, however, this will only mean we are producing an automated Bechdel Test. Instead, we wanted to discover another way to predict the Bechdel Test. These are the features we decided to use:</p><ol><li>Number of men, women, unknown gender characters</li><li>Gender of the director</li><li>Genre</li><li>Budget</li><li>Revenue</li><li>Profit</li><li>Average IMDb vote</li><li>Number of lines in the script</li><li>Fraction of conversation between two women</li><li>Fraction of conversation between two men</li><li>Fraction of conversation between two of the opposite genders</li></ol><p>A lot of these features are straightforward, they are there on the dataset, however, we needed to do a little more work on the movie script database to get feature 8, 9, 10 and 11. Besides that, features with gender gave us a different kind of challenge. A lot of the gender of the characters are not included in the dataset we used, therefore, we needed to do a gender analysis to figure this out.</p><p><strong>Gender Analysis</strong></p><p>We gathered the gender of characters and cast and crews from the TMDb API, however, there are a lot of missing genders. Therefore, we used a gender detector API to determine the missing genders using their name. An example code to run the gender_guesser library in Python:</p><pre>import gender_guesser.detector as gender</pre><pre>d = gender.Detector(case_sensitive = False)</pre><pre>print(d.get_gender(‘Bandaged Reaper’))</pre><p>For the small portion of the dataset with unknown gender, we manually determined their gender.</p><p>Another snag along the way is character names in movies that don’t match up perfectly. For example Lewis in the script dataset and Major Lewis Jr in another dataset. We used character matching in this case like the fuzzywuzzy library in Python and also some manual character matching. All of these cleaning is done at length before fitting any model to the dataset.</p><p><strong>Supervised Machine Learning</strong></p><p>After all of that is squared away, we get to the most anticipated part! The prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y14c87yUj2GsfGShDGNZ0Q.png" /><figcaption>A sample of the features used, omitting the movie_id and movie_title. The number columns are genres changed using one hot encoding.</figcaption></figure><p>The target is the Bechdel score for each movie, 0 — Failed Bechdel Test and 1 — Passed Bechdel Test.</p><p>We ran three supervised machine learning models, Logistic Regression, Random Forest Classifier and Support Vector Machine (SVM) while splitting the train and test 70–30. Below is the example code used.</p><pre>X = features253_full.drop([‘movie_id’, ‘movie_title’, ‘bechdel_score’], axis = 1)</pre><pre>y = features253_full[‘bechdel_score’]</pre><pre>#split train and test</pre><pre>from sklearn.model_selection import train_test_split</pre><pre>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)</pre><p>Logistic Regression — our baseline model</p><pre>from sklearn.linear_model import LogisticRegression</pre><pre>lr = LogisticRegression()</pre><pre>lr.fit(X_train, y_train)</pre><pre>lr.score(X_train, y_train)</pre><pre>from sklearn.metrics import classification_report</pre><pre>y_pred = lr.predict(X_test)</pre><pre>print(classification_report(y_pred, y_test))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*P65IHlA_ZjIwMvq-FS4q5g.png" /></figure><p>Random Forest Regression</p><pre>import sklearn.ensemble as skens</pre><pre>rf_model = skens.RandomForestClassifier(n_estimators=10,oob_score=True, criterion=’entropy’)</pre><pre>rf_model.fit(X_train, y_train)</pre><pre>rf_model.score(X_train, y_train)</pre><pre>y_pred = rf_model.predict(X_test)</pre><pre>print(classification_report(y_pred, y_test))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*Cz4_RQ6hVpRi71zs3Iq_jA.png" /></figure><pre>feat_importance = rf_model.feature_importances_</pre><pre>pd.DataFrame({‘Feature Importance’:feat_importance},</pre><pre>index = X_train.columns).nlargest(n = 10,columns = [‘Feature Importance’]).plot(kind=’barh’)</pre><pre>top5 = pd.DataFrame({‘Feature Importance’:feat_importance},</pre><pre>index = X_train.columns).nlargest(n = 5,columns = [‘Feature Importance’])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*XaQpBDXBPE4DmLQwh1OKSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*OA5MQ8EBLYI9r37IvfKVAA.png" /></figure><p>Support Vector Machine (SVM)</p><pre>from sklearn import svm</pre><pre>svm_model = svm.SVC() # Linear Kernel</pre><pre>svm_model.fit(X_train, y_train)</pre><pre>svm_model.score(X_train, y_train)</pre><pre>y_pred = svm_model.predict(X_test)</pre><pre>print(classification_report(y_pred, y_test))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*BCHb1eePTLtmvG2W86BrMg.png" /></figure><p><strong>Results</strong></p><p>We can see from the classification report above, <strong>Random Forest Regression fare better than our baseline Logistic Regression model with weight avg F1 score of 0.72</strong>. We can also look at the features importance using the Random Forest Regression and in this case, <strong>fraction of conversation between just women and fraction of conversation between just men as well as the number of men working on a show are the top three features that drive the prediction.</strong></p><p>And tada! Our very simple model managed to come up with a good F-1 score. Imagine what a bit more complexity can do to the prediction model. This simple project could be a starting point for more efforts to include more representation in movies. The Bechdel Test is in no means a perfect test for representation, however, if we can come up with a better representation test and predict if a particular movie will pass the test using their movie metadata, the movie industry is one step towards the right path.</p><p><strong>References</strong></p><p><a href="https://www.google.com/url?q=https://en.wikipedia.org/wiki/Bechdel_test&amp;sa=D&amp;ust=1594291815445000&amp;usg=AFQjCNEYJN9vtwUn-WnpVtZ6-1Cv4CSDCw">https://en.wikipedia.org/wiki/Bechdel_test</a></p><p><a href="https://www.google.com/url?q=https://www-tandfonline-com.proxy.lib.umich.edu/doi/full/10.1080/14680777.2016.1234239&amp;sa=D&amp;ust=1594291815444000&amp;usg=AFQjCNGWmtWKGR2ylvg81OsMMvrfeji-1Q">https://www-tandfonline-com.proxy.lib.umich.edu/doi/full/10.1080/14680777.2016.1234239</a></p><p><em>Acknowledgment</em></p><p><em>This project is done with Aljoharah Alfayez.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7253618a3f8" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/predicting-bechdel-test-score-using-machine-learning-7253618a3f8">Predicting Bechdel Test Score using Machine Learning</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>