<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ahmed Omar on Medium]]></title>
        <description><![CDATA[Stories by Ahmed Omar on Medium]]></description>
        <link>https://medium.com/@hmedOmar?source=rss-cffadd6bb9e6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IMCbRb3sc9b-nlKBOVB1lw.jpeg</url>
            <title>Stories by Ahmed Omar on Medium</title>
            <link>https://medium.com/@hmedOmar?source=rss-cffadd6bb9e6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:35:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hmedOmar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reflections from PyData Amsterdam: Embracing Simplicity in a Complex Field]]></title>
            <link>https://medium.com/@hmedOmar/reflections-from-pydata-amsterdam-embracing-simplicity-in-a-complex-field-f7ddc26c15b4?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7ddc26c15b4</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Sat, 21 Sep 2024 02:48:17 GMT</pubDate>
            <atom:updated>2024-09-21T02:48:17.940Z</atom:updated>
            <content:encoded><![CDATA[<p>This week, I had the opportunity to attend PyData Amsterdam, and it was an incredible experience — full of informative sessions and, more importantly, amazing people who were eager to share their knowledge and help others. With so many sessions happening simultaneously, it felt impossible to catch them all. Thankfully, the talks were recorded, so I’ll be adding quite a few new links to my bookmarks.</p><h3>Overwhelming Variety</h3><p>There were sessions on topics I used to know well, some I’m fairly comfortable with, and others I’ve always wanted to explore but never had the time. And then, of course, there were areas I’d never even heard of. Even after 10 years in the data field, it’s humbling to realize just how much there is still to learn.</p><h3>“Data Scientist” — But What Do You Actually Do?</h3><p>One thing that became clearer to me during the event is how broad the term “data scientist” really is. It essentially means you work with data, but it doesn’t specify much about <em>what</em> you actually do. Keeping up with the latest in data science is exciting, but let’s be honest — it’s nearly impossible to stay on top of everything. The field is just too vast.</p><h3>Common Challenges, Different Tools</h3><p>A recurring theme across many sessions was how much we all grapple with similar challenges — whether it’s finding the right data, ensuring data quality, or selecting the best tools and methodologies. Every time a new topic emerges, we see a divergence of solutions, but over time, the best practices tend to converge. Take machine learning on tabular data, for instance. Five years ago, everyone debated the “best” algorithm, but now there’s a consensus: use boosted trees for most problems.</p><p>I noticed this same pattern starting to happen with causal inference and large language models (LLMs). Right now, everyone is debating the best approaches, but in time, I expect this space to settle too.</p><h3>Simplicity Over Complexity</h3><p>Another key takeaway is that simple solutions and methodologies are often the most impactful. The smaller the problem you’re addressing, the easier it is to develop a clear, straightforward solution. After all, as the saying goes, “All models are wrong, but some are useful.” The most useful solutions don’t necessarily provide a definitive answer — they give an answer that’s “good enough” to make progress. Often, that answer paves the way for even better solutions later on.</p><p>In a world where everyone seems to be turning to LLMs for everything, it’s refreshing to see how more traditional approaches, like network analysis, can still deliver valuable insights for many analytical problems.</p><h3>Start With the Problem</h3><p>One key lesson I’m taking away is the importance of starting with the problem itself, rather than jumping to a solution like machine learning. It’s tempting to get excited about the latest technology and look for ways to use it, but often the simplest solution is the best. By focusing on the problem, you may find that a less complex approach can yield solid results — just by thinking differently.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7ddc26c15b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Package Installation Analysis: Comparing MoM downloads]]></title>
            <link>https://medium.com/@hmedOmar/python-package-installation-analysis-comparing-mom-downloads-edf3cb0f7f4b?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/edf3cb0f7f4b</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Sun, 28 Jul 2024 18:01:42 GMT</pubDate>
            <atom:updated>2024-07-28T18:01:42.363Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://ahmedomareissa.github.io/pypi-MoM">https://ahmedomareissa.github.io/pypi-MoM</a></p><p>Last month, I wrote an<a href="https://ahmedomareissa.github.io/pypi-eda"> article about a project to analyze</a> the monthly downloads of Pypi packages. Since I only had one month’s data, I analyzed the distribution package installs, and we found that ~500 packages contribute to around 80% of installations.</p><p>This month, I downloaded June’s dataset, so I now have two months of data: May and June 24. I will use these two datasets to identify June 24 trending packages. In this article, I’ll explain the methodology and process used to identify the trending packages.</p><p><strong>Goal</strong></p><p>There are two crucial questions I want to answer using the two months’ data.</p><ol><li>What are the top <strong>new</strong> packages?</li><li>What are the top trending packages?</li></ol><p>In this article, I’ll focus on answering question #1. However, we can answer both questions using the same methodology, comparing the package installs between May and June. If the number of installs in May was 0, this is a new package.</p><p><strong>How can we visualize the data?</strong></p><p>I used scatter plots for this purpose, where each point represents a package, the X axis represents May installs, and the Y axis represents June installs.</p><p><strong>Trial #1: Not great, but you need to start somewhere.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jv-EVifwWy2DXvyM" /></figure><p>On the x-axis, you can see May and June installations on the y-axis. But you can barely see any dots in this scatter plot because of Outliers; this one dot on the top right is a package with 10⁹ that makes the axis so huge that almost all the other packages fit 1 line between 0 and this package May and June downloads.</p><p>How can we solve this? Look at trial #2</p><p><strong>Trial #2: Log scale charts.</strong></p><p>Log scale is handy for drawing data with skewness and outliers instead of having linear (normal) steps, i.e., 1,2,3,4. <strong>The logarithmic scale has exponential steps of (usually 10s), i.e., 1, 10, 100,1000. This means we zoom a bit more on the small numbers, but we also keep the big numbers on the photo frame, perfect, no?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7N59h28kxcKsua8U" /></figure><p>Now, it is a different picture. By changing the scale, we can see more variations in the bottom left packages, with 1000 and 10000 package downloads per month. However, <strong>we still see too many points </strong>which is normal because we have too many packages, now it is the time to start group things together. Which will lead us to trial # 3</p><p><strong>Trial #3 Grouping things together</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WXjd-5U0o5GZbc6e" /></figure><p><strong>WTH we did to move from trial 2 to trial 3? <br></strong>Simple, we rounded things to the closest exponential. This means that there are a lot of overlapping points, but I can now start thinking of the things that I’m very interested in:</p><ol><li><strong>Packages above the redline:</strong> those packages got more downloads in June than in May, with a special focus on the packages that showed 0 downloads on May 24, but in June, we see 10k+ downloads (let’s call them New &amp; Trending packages).</li><li><strong>Packages below the redline</strong>, those packages got less downloads in June than in May, they might be dying projects, but for the sake of focusing on the positive side of things, we will not be looking much into those.</li><li><strong>Packages on the redline</strong>, I like to call it the boring line, those packages maintain the same level of downloads between May and June. from MoM comparison, those ones are not very interesting to us. In the previous article, we talked about the most downloaded packages (the top right dots)</li></ol><p>Things became more interesting, no? Now it is time to share the last version. The goal now is to answer the question of <strong>how many packages fall in each bucket outside the boring line. </strong>Now, let’s look at the final version.</p><p><strong>Final Trial. Wrapping things together</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OSAnPbDtAc2mYq4T" /></figure><p>In this chart, the size of each dot represents the number of packages, while the color represents how far it is from the boring line. <strong>The farther the point from the boring line, the more interesting it is. </strong>I purposely removed the details, i.e., how many packages fall into each bucket. However, from the look and feel, it seems that we have more new packages than dying packages.</p><p>Now, can you answer the question I asked earlier: What are the top <strong>new</strong> packages?</p><p>If we deep dive into the three dots in the top left. We can easily identify those packages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*JLWK_Voh2Grw3RIZs6HRdA.png" /><figcaption>Top 10 new PyPI packages June 24</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edf3cb0f7f4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Package Installation Analysis: May 2024]]></title>
            <link>https://medium.com/@hmedOmar/python-package-installation-analysis-may-2024-baca76116a64?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/baca76116a64</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pypi]]></category>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Sat, 15 Jun 2024 22:33:58 GMT</pubDate>
            <atom:updated>2024-06-15T22:33:58.643Z</atom:updated>
            <content:encoded><![CDATA[<p>originally posted on <a href="https://ahmedomareissa.github.io/pypi-eda">my website</a></p><h4>Overview:</h4><p>I’ve recently started a new project to track package installations in Python. My goal is to create a monthly report or newsletter with the trending packages and general trends and behaviors in package installs. With this new project, we have the opportunity to gain valuable insights into the Python package landscape.</p><h3>Data Source:</h3><p>For this project, I’m utilizing the powerful Google BigQuery public dataset. <a href="https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/">Read more about the dataset here</a>. I’m currently running a query on Google Cloud, downloading the data as a CSV, and then analyzing it in Python in this <a href="https://github.com/AhmedOmarEissa/pypi_analysis">GitHub repository</a>. The use of these robust tools ensures the accuracy and efficiency of our data analysis.</p><h3>May 24: EDA Findings:</h3><p>Find the analysis notebook <a href="https://github.com/AhmedOmarEissa/pypi_analysis/blob/main/notebooks/0_EDA.ipynb">here</a></p><p>Since May was the first month I queried the data and had no insights on the MoM trends, I initiated an Exploratory Data Analysis (EDA) on the dataset. This involved analyzing the distribution of package installs, which led to some interesting findings about the most used packages.</p><h3>The Dataset</h3><p>Below is a sample of the dataset used to analyze the data. There are 2 columns:</p><ul><li><strong>Project</strong>: the name of the package</li><li><strong>Installs</strong>: How many times this package was downloaded in May</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*buVGnPeuXO31bWf-ZHvenA.png" /></figure><h3>Summarizing the Data</h3><p>I’ve split the projects into buckets based on the number of installs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-azy17JntyyWAUDdckAYEA.png" /></figure><p>This table presents data on projects and their corresponding installation counts categorized into different bins based on the number of installations. Here’s a description of the table:</p><ul><li><strong>Bin</strong>: This column represents the range of installations per project. Each bin range is represented as an interval, such as (0, 1], (1, 10], (10, 100], and so on.</li><li><strong>projects_count</strong>: This column shows the number of projects falling within each bin range. For instance, there are 11,043 projects in the (0, 1] range, 9,507 projects in the (1, 10] range, and so forth.</li><li><strong>projects_installs</strong>: This column indicates the number of installations corresponding to the projects within each bin range. For example, there are 11,043 installations in the (0, 1] range, 38,229 installations in the (1, 10] range, and so on.</li><li><strong>projects_installs_pct</strong>: This column displays the percentage of total installations represented by the projects within each bin range. For instance, projects within the (0, 1] range represent 0.00% of the total installations, projects within the (1, 10] range represent 0.00%, and so forth.</li></ul><h4>Concentration of Installations</h4><p>While most projects fall into the lower installation ranges, a significant portion of the total installations is concentrated in projects with higher installation counts. For instance, projects with over 1,000,000 installations represent 94.90% of the total installations despite being only 2,058 projects.</p><h4>Long Tail Distribution</h4><p>The distribution follows a long tail pattern, where a few projects with extremely high installation counts contribute to a large portion of the total installations. Most projects have relatively low installation counts, while a small number have very high installation counts.</p><h3>Pareto Analysis</h3><p>Pareto Analysis, often referred to as the 80/20 rule or the principle of factor sparsity, is a technique used to prioritize efforts. It states that, for many outcomes, roughly 80% of the consequences come from 20% of the causes. We can apply this analysis to the provided data to identify which installation ranges contribute to most installations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*c232jle8Cc-PzCMB.png" /></figure><p>This chart illustrates a Pareto Analysis, which is a graphical representation showing how many PyPI projects contribute to 80% of the total installs. The x-axis represents the percentage of PyPI projects, plotted on a logarithmic scale ranging from 0.001% to 100%. The y-axis shows the cumulative percentage of total installs, spanning from 0% to 100%.</p><p>Key insights from the chart include:</p><p>The analysis reveals a significant imbalance in the distribution of installs across projects. A small fraction of packages (0.1% or 507 packages) accounts for a disproportionately large share (80%) of total installs. This highlights the dominance of a few highly popular packages in the PyPI ecosystem, while the majority of projects contribute to a smaller proportion of total installs.</p><p>Stay tuned for more insights and detailed reports on Python package trends and behaviors. Your feedback and suggestions are always welcome as we continue to explore the fascinating world of Python packages.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=baca76116a64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Impact of Remote Work on Tech Working Hours: A Data-Driven Analysis]]></title>
            <link>https://medium.com/@hmedOmar/the-impact-of-remote-work-on-tech-working-hours-a-data-driven-analysis-96b2a4daeed7?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/96b2a4daeed7</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Mon, 20 Feb 2023 19:01:18 GMT</pubDate>
            <atom:updated>2023-02-20T19:30:27.007Z</atom:updated>
            <content:encoded><![CDATA[<p>One day I asked ChatGPT to suggest an article to write about data analysis, That was the answer from ChatGPT:</p><p>“The Impact of Remote Work on Employee Productivity: A Data-Driven Analysis</p><p>As a result of the COVID-19 pandemic, many employees have shifted to working remotely. This has led to ongoing discussions about the effects of remote work on employee productivity. By conducting a data-driven analysis, this article could provide valuable insights into this topic.”</p><h4><strong>Seems interesting, can you give me some data sources?</strong></h4><p>Well, that seems a very interesting topic, I could potentially dig deeper here. But of course, I need data to start my data-driven analysis, so I asked ChatGPT about what can be potential data sources to do such analysis, Summary answer was:</p><ol><li>Government Statistics</li><li>Surveys</li><li>Company data</li><li>Academic research</li><li>Publicly available datasets</li></ol><p>Hummm, seems quite a generic answer, and I couldn’t find relevant datasets to work with. But I still like the direction of the analysis, can I find something else?</p><h4>The research questions reformulation.</h4><p>The Impact of Remote Work on Employee Productivity: A Data-Driven Analysis, <em>seems very vague</em>, especially when it comes to how we define productivity. productivity can mean different things to different people based on their titles, jobs, and seniority. it is also hard to measure from a quantitative point of view.</p><p>one good measure I can think of is <strong>the working hours</strong>, it is something I can measure, and compare before and after 2020, so I reformulated the research question to <em>Are we working more or fewer hours after the majority of people started to work remotely in 2020?</em></p><h4>In search of a suitable dataset.</h4><p>Personally, I know the answer by looking at my working hours before and after 2020, but it could be just me, so I need to find some data that I can use to answer the question I asked. I did some research on Kaggle, Data.world, and data.gov but unfortunately, I couldn&#39;t find something useful given that not everyone is logging their working hours anymore and there’s no check-in and check-out in the remote working. what can be a good alternative to check-in and check-out?</p><p><strong>Stackoverflow: <br></strong>One good approximation I can think of is views of websites that we use at work, one very famous example is <strong>stackoverflow.com</strong> software engineers usually use this website to search for code and fix bugs, and my hypothesis is: <em>if you’re working more hours you will be open </em><strong><em>StackOverflow </em></strong><em>in late hours.</em></p><p>The<em> </em>approach of StackOverflow views seems logical (to me at least) but unfortunately, I couldn’t find data on StackOverflow views, What should I do then?</p><p><strong>BigQuery Public datasets, Stackoverflow questions, and answers.<br></strong>Luckily, <a href="https://cloud.google.com/blog/topics/public-datasets/google-bigquery-public-datasets-now-include-stack-overflow-q-a">Google big query</a> has many public datasets and one of them is StackOverflow questions, replies, and comments:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GKSLEK4957GggORWudozhg.png" /><figcaption>BigQuery Stackoverflow Public Dataset</figcaption></figure><p>The dataset was last updated in September 2022, which gives me ~ 2.5 after COVID remote work policy started.</p><h4>Fitting the research question to the available data.</h4><p>Given that StackOverflow is only relevant to tech workers (e.g. engineers) so the question of research here is: <em>what is the impact of remote work on tech working hours? </em>In other words, <em>do we work fewer or more hours after we started working remotely in 2022?</em></p><h4>Data Challenges</h4><p><strong>Time Zones challenge: </strong>Working hours are something that differs from one country to another, due to timezones, I needed to select one or a few countries to limit my analysis for them, but first I need to see the user distribution across the countries,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OZGWqB2OD9VbwFxIfLVHMw.png" /><figcaption>Number of users per location.</figcaption></figure><p>it seems like a huge portion of users didn’t specify their locations, and India is the top country in terms of # of users.</p><p><strong>Non Active Users</strong>, the number of users doesn&#39;t always mean that these users are interacting with the site, therefore I decided to look on the number of comments per location</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZQVm0YzspNpRShL-UyV85A.png" /><figcaption>Number of comments per location.</figcaption></figure><p>We now see a slightly different view from the users view.</p><h4>What should the final output look like?</h4><p><strong>Distribution of comments/posts across the day. </strong>The final output should be in a version of the number of comments and posts per hour for users in a specific country.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BD03AtUajz4z_A_NzG4SnA.png" /><figcaption>Number of comments per hour in the netherlands</figcaption></figure><p>If we are staying late hours working remotely I should see more comments and posts happen after the working hours if we compare the results from 2018 to 2022.</p><h4>Results</h4><p>First, let&#39;s look at the number of comments/posts that happens outside work hours in both India and the Netherlands</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5UsHlSj-EyChBw2QtKqIzQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V55AIBlRsj6VHXXFMheYXw.png" /></figure><p>In both India and Netherlands, we see a spike in the % of comments / posts that happen outside working hours in India 36% pre-2020 and 43 post 2020. In the Netherlands 29% pre-2020 and 31% post 2020.</p><p><strong>Then let’s explore the distribution of comments/questions per hour.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-UPkYGzFQWt9kCNN_-AXmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wr3Cl4dTe32OsWUd5eG5ww.png" /></figure><p>In both India and Netherlands, we see more questions and comments happening in later hours compared to pre-2020 levels.</p><h4>Conclusion</h4><p>As shown in the deep-dive above it seems like coders are posting more questions and comments in StackOverflow during late hours in the remote working years (2020,2021,2022) this indicates later working hours for coders. The real cause of this shift is beyond the scope of this analysis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96b2a4daeed7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving the birthday paradox using python]]></title>
            <link>https://medium.com/@hmedOmar/solving-the-birthday-paradox-using-python-bbdfa6297a3c?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbdfa6297a3c</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Sat, 28 Jan 2023 20:49:15 GMT</pubDate>
            <atom:updated>2023-01-28T20:49:15.972Z</atom:updated>
            <content:encoded><![CDATA[<p>How many people do you need in a room in order to have a 50% chance that at least two will have the same birthday?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*oMwYM0UCFwE2QvxZ.jpg" /></figure><p>I’m sure you came up with a number much higher than 23. You’re not alone, and that’s why it is called <strong>The birthday paradox. </strong>The birthday paradox is counterintuitive and there’re many trials to explain <a href="https://statisticsbyjim.com/fun/birthday-problem/">how math works</a>.</p><h3>So, can we investigate the problem using python?</h3><p>The answer is YES!</p><p>In this article, I want to validate if the 23 is the correct answer by</p><ol><li>Running 1000 trials of rooms with different sizes (from 1 to 365).</li><li>Count how many of the 1000 trials have at least two people with the same birthday = probability of having at least two people with the same birthday.</li><li>Plot the probability vs the room size from 1 to 365.</li></ol><p><strong>Example</strong></p><p>Imagine we want to generate a room with 5 people, here is the code you can use:</p><pre>np.random.randint(low=0, high=365, size=(5,))</pre><p>The output should be something like with each item representing the day of the year:</p><pre>array([168, 345,  29, 247,  31])</pre><p>So person 1 was born on the 168th day and person 2 was born on the 345th day… etc.</p><h4>Now we need to generate 1000 rooms.</h4><p>Imagine now we have 1000 rooms with each having 23 persons inside, if 500 rooms have 2 persons or more with the same birthday, then the answer is true.</p><p>Here is how the code look</p><pre>import numpy as np <br>import pandas as pd <br>import matplotlib.pyplot as plt<br><br>room_size = 23<br>room_with_same_bd = 0 <br>number_of_rooms = 1000<br>for _ in range(number_of_rooms):<br><br>  room = np.random.randint(low=0, high=365, size=(room_size,))<br><br>  if room_size &gt; len(np.unique(room)): ## Number of people &gt; Number of unique birthdays. <br>    room_with_same_bd += 1 <br><br>print(&#39;Rooms with 2 or more persons share same BD: &#39;,  room_with_same_bd)<br>print(&#39;Probability of the room with 2 or more persons share same BD: &#39;,  room_with_same_bd/number_of_rooms)</pre><pre>Rooms with 2 or more persons share same BD:  512<br>Probability of the room with 2 or more persons share same BD:  0.512</pre><p>Ok, the answer seems correct, if we have 1000 rooms, and each room has 23 persons, 512 rooms will have at least two persons sharing the same birthday.</p><h3>Ok, now can we explore different room sizes?</h3><p>of course, we need to add another for loop, but this time to replace the <strong>room_size </strong>variable.</p><p>The code can be shown as follow:</p><pre>import numpy as np <br>import pandas as pd <br>import matplotlib.pyplot as plt<br><br>for room_size in range(20,30):<br>  room_with_same_bd = 0 <br>  number_of_rooms = 1000<br>  <br>  for _ in range(number_of_rooms):<br><br>    room = np.random.randint(low=0, high=365, size=(room_size,))<br><br>    if room_size &gt; len(np.unique(room)): ## Number of people &gt; Number of unique birthdays. <br>      room_with_same_bd += 1 <br>  print(&#39;Room size: &#39;,room_size , &#39; Rooms with shared BD: &#39;,room_with_same_bd)</pre><pre>Room size:  20  Rooms with shared BD:  419<br>Room size:  21  Rooms with shared BD:  437<br>Room size:  22  Rooms with shared BD:  477<br>Room size:  23  Rooms with shared BD:  513<br>Room size:  24  Rooms with shared BD:  545<br>Room size:  25  Rooms with shared BD:  577<br>Room size:  26  Rooms with shared BD:  581<br>Room size:  27  Rooms with shared BD:  625<br>Room size:  28  Rooms with shared BD:  643<br>Room size:  29  Rooms with shared BD:  665</pre><p>Ok, the more people in the room the higher the probability of having shared birthdays.</p><h3>Finally let’s put the findings in a chart.</h3><p>Using the code below to generate a chart:</p><pre>fig,ax = plt.subplots(1)<br>fig.set_figwidth(10)<br>fig.set_figheight(5)<br><br>x = np.log10(prcnt_series.index); <br>y = prcnt_series.values * 100 <br><br># plot the data<br>ax.plot(x,y)<br>ax.set_xticklabels(np.round(10 ** (ax.get_xticks()), 0 ).astype(int)) <br>for i, j in  enumerate(x):<br>  if i%10 == 0 and i &lt;= 50 : <br>    ax.text( .05 , y[i]+1.5 ,<br>            &#39;N = &#39; + str(int(round(10**(j),0)) ) + &#39; &#39; + &#39;| P = &#39; +str(y[i] ));<br>    ax.axhline(y[i] ,color = &#39;black&#39;, linestyle = &#39;-&#39;, linewidth = .5)<br>plt.xlabel(&#39;Room Size (N)&#39;) <br>plt.ylabel(&#39;Probability of shared birthday (P) &#39;);</pre><p>We can see how the probability of shared birthday is plotted vs the room size (e.g. with 40 people in the room, the probability of 2 or more people share the same birthday is 89.8%)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*LjgAt-vdKWawq_En__4zxg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbdfa6297a3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My roadmap for re-visiting Statistics in 2023]]></title>
            <link>https://medium.com/@hmedOmar/my-roadmap-for-re-visiting-statistics-in-2023-8eb2774fa65?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8eb2774fa65</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Sun, 22 Jan 2023 08:45:17 GMT</pubDate>
            <atom:updated>2023-01-22T08:45:17.747Z</atom:updated>
            <content:encoded><![CDATA[<p>Or how I want to be confident about my confidence intervals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YwWtXK11NKVLEEXe" /><figcaption>Photo by <a href="https://unsplash.com/@bdchu614?utm_source=medium&amp;utm_medium=referral">Brendan Church</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I have been working as a data analyst for almost 8 years now, through my career I’ve learned a lot of skills that I’m very proud of, including coding, data visualization, business acumen, and finally communication skills. but still, Statistics is one of the topics I want to be more confident about in the next 5 years. here is how I plan to do it.</p><h3><strong>Where do I stand with my statistical knowledge?</strong></h3><p>I consider my knowledge of statistics to be “good enough”, I mean .. I’m a data analyst, right? jokes aside, I have a very good understanding of mathematical topics, and the basics of statistics, combining them together, with some courses/tutorials I could build tons of visualization, run a few experiments, and deploy a handful of machine learning models. However, I’ve always felt that something is missing and I feel that I’ll never master statistics unless I have at least a Ph.D. (this is where things start to get super complex for me).</p><h3>Where statistics is hard?</h3><p>Compared to other data science skills, statistical is the most challenging technical skill, there’re multiple reasons for that, for example:</p><p><strong>The competing theories: </strong>There’s no one best way of doing things, there are always competing theories (Frequentist vs Bayesian , Parametric Vs Nonparametric … etc.) as a user of statistical methods, this add additional complexity on choosing which path I should choose.</p><p><strong>The bad naming conventions: </strong>with<strong> </strong>full respect to all amazing contributions in Statistics, it feels extra challenging to remember the variable names if they are named after someone, or it is a very lengthy acronym that no one tells me what it stands for.</p><p><strong>The “how I can apply this?” question: </strong>Some times it is hard to draw a line between theory and application, especially with real life scenarios.</p><h3>What I plan to do to re-visit statistics in 2023?</h3><p>This is my plan to be more confident working with statistics until I can receive proper academic degree:</p><ol><li><strong>Focus on the applications.</strong></li><li><strong>Keep a note with all the statistical terms I learn.</strong></li></ol><h3>What courses / books do I intend to read in 2023?</h3><p>Even when learning statistical courses and reading statistical books, I’ve decided to focus on the application of statistics, based on my initial findings <strong>I want to read 2 books and complete 2 Courseara specialization:</strong></p><h3>Book1: Applied Survey Data Analysis</h3><p>To get deeper on how to analyze sample data, I want to read the book Applied Survey Data Analysis by Brady T. West, Patricia A. Berglund, and Steven G. Heeringa. The book seems very interesting with already some applications on survey and census datasets. There’re also some applications on R, SASS.. etc. <em>Maybe</em> I can create a python tutorials for this examples (no promises lol)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/0*NKP5F0MMQHmIfnes.jpg" /><figcaption><a href="https://websites.umich.edu/~surveymethod/asda/">Book Site</a></figcaption></figure><h3>Book 2 : Trustworthy online controlled experiments</h3><p>My goal is to get more deeper knowledge on experimentation and that is why I plan to read trustworthy online controlled experiments by Diane Tang, Ron Kohavi, and Ya Xu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/0*TxChWu_0NG4s9C92.jpg" /><figcaption><a href="https://experimentguide.com/">Book Site</a></figcaption></figure><h3>Specialization 1: Statistics with Python</h3><p><a href="https://www.coursera.org/specializations/statistics-with-python">Provided by University of Michigan</a>, I plan to enroll in this specialization, the main goal from this track is to understand If I’m missing any python statistical tools. also, the third course of this specialization( Fitting Statistical Models to Data with Python) seems very interesting.</p><h3><strong>Specialization 2: </strong>Survey Data Collection and Analytics Specialization</h3><p><a href="https://www.coursera.org/specializations/data-collection">Also, provided by University of Michigan</a> I intend to build on the knowledge gained from Applied Survey Data Analysis book and complete this course on coursera.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8eb2774fa65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 online tools you need to know and bookmark.]]></title>
            <link>https://medium.com/@hmedOmar/4-online-tools-you-need-to-know-and-bookmark-428523cf1895?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/428523cf1895</guid>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regex]]></category>
            <category><![CDATA[shell]]></category>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Fri, 29 Jul 2022 09:26:08 GMT</pubDate>
            <atom:updated>2022-07-29T09:26:08.366Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TLDR</strong>; the article below lists 4 tools I use to quickly remind me of Cron, Shell, Regex, and probability distribution. These topics easily slip from your mind as I don’t use them on daily basis, but they are important concepts.</p><p>In this article , I want to share some of the online tools that I use from time to time. As I don’t use some technologies on daily basis, my memory doesn’t always help me to find the right commands I need to use, but these tools help me to quickly identify the right commands. I found it useful to bookmark them as you will need to use them at some point and will save you a ton of searching time if you just stick to them.</p><h3><a href="https://crontab.guru/">crontab guru</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eKBPW-zNkzqsegU39iSk2Q.png" /></figure><p>This tool is handy in translating the <a href="https://en.wikipedia.org/wiki/Cron">cron</a> schedule to human-readable format. <br>for example; 5 4 * * * in cron schedule means at 4:05 every day. I use this website from time to time whenever I need to automate any task using corn jobs, I test the schedule here so that I don’t need to memorize it.</p><h3><a href="https://regex101.com/">regex101</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5JWz0gfxoh77DVOE2_E6ew.png" /></figure><p><a href="https://en.wikipedia.org/wiki/Regular_expression">Regular expressions</a> can be tricky and it is almost impossible to memorize the different commands and patterns. regex101 website helps me a lot whenever I need to test a regex that I’ve created or create a new one.</p><p>It goes even beyond testing your regex as it has a part to explain the regex pattern you created and it also has many patterns for quick reference.</p><h3>Distribution Calculator</h3><p><a href="https://gallery.shinyapps.io/dist_calc/">https://gallery.shinyapps.io/dist_calc/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rhwJtnMsosE5WTQB_Y-amw.png" /></figure><p>I found this shiny app is very useful to visualize the different distributions and determine the area under the probability distribution curve. I use it to quickly determine the p-value in one-tail and two-tail tests.</p><h3><a href="https://explainshell.com/">explainshell.com</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EPERtAAuHDzzgMAJmhJNZA.png" /></figure><p>As I need to work with shell from time to time, it is useful to bookmark this handy website, it simply explains the shell command you write in a simple language, also very useful to help you remember the code you have written before if you are not expert in shell.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=428523cf1895" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Technical tasks of data science managers.]]></title>
            <link>https://medium.com/data-science/technical-tasks-of-data-science-managers-b2e80b797602?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2e80b797602</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[career-development]]></category>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Sun, 10 Jan 2021 14:30:36 GMT</pubDate>
            <atom:updated>2021-01-10T14:30:36.252Z</atom:updated>
            <content:encoded><![CDATA[<h4>How not to lose your technical skills while being a manager.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4MMghd6w63uIgTG5" /><figcaption>Photo by <a href="https://unsplash.com/@kaleidico?utm_source=medium&amp;utm_medium=referral">Kaleidico</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>TLDR; you can be a data science manager and still get things done by yourself.</p><p>It is a big decision for you to change your role and become a manager. driven by your natural curiosity, you don’t want to stop learning new skills and of course, you don’t want to lose the skills you developed as an individual contributor. I assume also you have a great sense of ownership, however, you don’t want to micro-manage your team writing your code line by line. In this article, I suggest a few ideas to help you keep updated and motivated to do data science as a manager.</p><h4><strong>How can my technical skills help my team?</strong></h4><p>As a manager, you should have faced tons of errors and bugs, from your experience you are able to find creative solutions to save the team’s time and expand their horizon. The Data science world is changing every day, if you can be your company’s eyes to the outer data science world, everyone will appreciate that, especially your team members.</p><p>Also, there are a lot of tasks <strong>no one has time for</strong> however it is very important for teamwork. In general, there’re many unfilled gaps in the data science teams as the analytics area has seen more focus recently, these gaps need new jobs to be created within the team to be filled, for example, data scientists had to do a lot of data engineering work a few years ago until the title was defined and adopted by many companies. You may need to step in to fill one of those gaps and become a business analyst, data architect, or product owner in one project.</p><h4><strong>What can you do to help?</strong></h4><ol><li><strong>Explore new technologies: </strong>Recommending new ways for doing tasks is one of the great managers’ qualities. It shows both creativity and deep technical knowledge when done right, this is one of the easiest things to do to gain the respect of both your team and your stakeholders. It is also a fun thing to do and helps you to keep yourself updated and learning.</li><li><strong>Solve team challenges. </strong>Understanding your team’s challenges and working on them yourself helps you to keep on top of your team’s work as it helps you to understand the details without writing the whole code yourself. You will not have the time to work on every problem yourself, but it is great to focus on the main challenges and start working on long term solutions.</li><li><strong>Explore what other teams do. </strong>Explore what other data science teams do inside and outside your organization. Don’t fall in love with your ideas, also, don’t fall in love with your teams’ ideas. Read technical blogs, ask questions, or reach out to your old teammates. This can be a win-win situation; see what problems they are facing, maybe you already have solutions, and ask about possible solutions to your challenges.</li><li><strong>Organize work:</strong> while everyone’s head is down coding, there are many tools that can help organize work no one will have time to explore, maintain and ensure that everyone in the team is using them. Version controlling is one of many examples, how the team projects should look like, what is the process of updating the repo… etc.</li><li><strong>Ensure scalability &amp; reliability</strong>: No one likes slow and unstable solutions, as a manager you like to increase the adoption of solutions built by your teams, this can’t be done without building high-quality solutions, your role is to predict what issues solutions built by your team may face proactively.</li><li><strong>Automate manual tasks: </strong>What you can learn from data is limitless but our team’s bandwidth is limited. You should be able to identify what should be automated either: <strong>Internally: </strong>By creating templates for slides or dashboards, or by creating code and packages to be used by your team. <strong>Externally: </strong>By automating tasks when it is possible to automate, like self-service tools for dashboarding, querying data, and generating high-level insights</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2e80b797602" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/technical-tasks-of-data-science-managers-b2e80b797602">Technical tasks of data science managers.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Takeaways from learning Cloud Data Engineering]]></title>
            <link>https://medium.com/@hmedOmar/takeaways-from-learning-cloud-data-engineering-83cf18116c9c?source=rss-cffadd6bb9e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/83cf18116c9c</guid>
            <dc:creator><![CDATA[Ahmed Omar]]></dc:creator>
            <pubDate>Mon, 29 Jun 2020 19:12:31 GMT</pubDate>
            <atom:updated>2020-06-29T19:34:40.210Z</atom:updated>
            <content:encoded><![CDATA[<h3>Takeaways from learning cloud data engineering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nRe3hl0OyjliCRJ6" /></figure><p><em>Over the last few months, I’ve been enjoying the learning journey of Coursera’s specialization: </em><a href="https://www.coursera.org/specializations/gcp-data-machine-learning#courses"><strong><em>Data Engineering, Big Data, and Machine Learning on GCP</em></strong></a><em>, the specialization consists of 5 courses, the first course is a general introduction about analytics and machine learning applications and some use cases for GCP, the following 3 courses focus more on data engineering, data lakes, batch processing and data streaming on GCP. The last course is about different machine learning models setup on GCP and how to deploy your model to production.</em></p><p>I wanted to share the top 3 takeaways from the specialization and the experience of working with GCP during the courses exercises:</p><p>1 <strong>Integration is key: </strong>The journey from data to insights and applications is a long multi-step process, cloud platforms trying to provide analytics one-stop-shop where all the steps, tools, and services are gathered in one place, however, as shown below, there are many services and each service has its own function. It is crucial for all the services to be integrated and able to connect and transfer data to each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RoqirAH4QDMvWNXk" /><figcaption>Source: <a href="https://cloud.google.com/ai-platform/">https://cloud.google.com/ai-platform/</a></figcaption></figure><p><strong>2 ML code is not everything: </strong>One of my regrets is spending too much time in the past trying to understand everything about machine learning algorithms, once I started to get hands-on and started running some models, I realized how important data cleaning and feature extraction are. What I liked about this course is providing even a bigger picture on how to deal with end to end data analytics application, therefore I started to realize the importance of many concepts that play a pivotal role in the data journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NSukC9m_KDQPaofh" /><figcaption><strong>Source: </strong><a href="https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf">Hidden Technical Debt in Machine Learning Systems</a></figcaption></figure><blockquote>Only a small fraction of real-world ML systems are composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex.</blockquote><p>3 <strong>We still don’t know how the world will look like in 5 years</strong>: Innovation in the field of analytics is happening so fast and I believe there’s still a lot of innovation needed and will happen in the next few years, one of the examples that excited me the most is BigQuery ML. SQL is the most used language for data and business analysts, learning python or R was always something extra for those who want to go the extra mile, applying ML models using SQL will change the way we think about ML going forward and help in democratizing ML applications allowing all SQL users to automate insights generation using SQL ML.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83cf18116c9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>