<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Wael Dimassi on Medium]]></title>
        <description><![CDATA[Stories by Wael Dimassi on Medium]]></description>
        <link>https://medium.com/@dim.wael?source=rss-684f4f39050c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*pDCCLLjmfaTRNcu0</url>
            <title>Stories by Wael Dimassi on Medium</title>
            <link>https://medium.com/@dim.wael?source=rss-684f4f39050c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:41:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dim.wael/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to deal with AWS Certified Machine Learning Specialty]]></title>
            <link>https://medium.com/@dim.wael/how-to-deal-with-aws-certified-machine-learning-specialty-9da3f8b03a10?source=rss-684f4f39050c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9da3f8b03a10</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[aws-certification]]></category>
            <dc:creator><![CDATA[Wael Dimassi]]></dc:creator>
            <pubDate>Thu, 05 Aug 2021 11:59:30 GMT</pubDate>
            <atom:updated>2021-08-05T12:16:45.765Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3PTrJei3zFAylve2qnUglg.png" /></figure><p>Still happy with the positive result, I want to share with you my learning journey. First of all, I want to thank <a href="https://www.think-it.io/">Think-it</a> that made this possible for me. Believe me, you will enjoy the experience and learn a lot of tricks and make some details stick on your mind.</p><h3>Do I need this ?</h3><p>It depends on your career goals. But still, this is a great opportunity nonetheless to prove and test your knowledge. What is special about this exam is that it won’t only test your familiarity with AWS services (I know S in AWS stands for services 😛 ) but will also test your knowledge on machine learning and deep learning.</p><p><strong>What am I supposed to get from this ?</strong> It may be easier to prove that you know what you’re doing in this field and have an extra credibility. Whether you aim for a raise, new job, promotion or even a personal challenge, this is a great way to do that. Also, for recruiters, this is supposed to give them more confidence in your profile.</p><p><strong>Why AWS ?</strong> Actually, AWS is relatively the most mature cloud computing platform and the most used worldwide. So your chances are higher to get a very decent experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*POqwBG5zluP28SJ8E2fvEw.jpeg" /></figure><p><strong>How it looks like ?</strong> <br>3 hours, 65 questions and 75% to pass. Simple ! But not too simple. For the majority of the questions, you will have 4 answers and others are multi choices questions. The pattern I saw on the questions is that you will have 1–2 answers that you can discard easily, but you will end up with 2 similar choices. Unless you really have the answer for that, you will need to focus a lot and read the whole question again and again very carefully (believe me !). Remember this, detect the keywords to help you link it with the answer (I will try to give some examples of this in the end).</p><p>To summarize, read the whole question and all answers to have a general idea about the topic, then get into details, discard false answers and focus on keywords until you get the clear answer. It doesn’t matter how many times you do this, you will still have plenty of time. What is important is to select the right answer. If you felt that you are stuck, just mark the question for review and carry on.</p><h3>How to prepare ?</h3><p>I would say that taking courses is not enough ! You must diversify your resources, and taking multiple practice exams ; repeating the same ones is also a good idea, actually. I will share with you some resources I used during the preparation :</p><ul><li><strong>Courses (Moocs)</strong></li></ul><p>I took mainly 2 courses, and they were more than enough for me. I must say that I do have a previous background in ML, so I’ve focused more on the AWS services.</p><p>First one, <strong>Stephane Maarek </strong>and<strong> Frank Kane </strong><a href="https://www.udemy.com/course/aws-machine-learning/"><strong>course</strong></a> which is a great one ! You can find it on Udemy. I highly recommend it. They went through almost all the curriculum. The services has been well explained and in details. But in my opinion, some details were not well covered like SageMaker parameters. You can buy their <a href="https://www.udemy.com/course/aws-machine-learning-practice-exam/?referralCode=6D783CF357FC6D7BF8B2"><strong>practice exam</strong></a> as a separate course.</p><p>Second one, it is also a really nice <a href="https://www.udemy.com/course/amazon-web-services-machine-learning/"><strong>course</strong></a>. Great teaching methodology, and the content is very fitting for the AWS ML Certification Exam preparation. The presentation of the course material is very well organized and enjoyable.</p><ul><li><strong>Hands-on</strong></li></ul><p>If you are like me, you learn by doing, then this <a href="https://acloud.guru/overview/aws-certified-machine-learning-specialty"><strong>one</strong></a> is perfect for you. In this course, you’ll learn:</p><ul><li>The domains of knowledge for the AWS Certified Machine Learning Specialty exam.</li><li>Best practices for using the tools and platforms of AWS for data engineering, data analysis, machine learning modeling, model evaluation and deployment.</li><li>Hands-on labs designed to challenge your intuition, creativity and knowledge of the AWS platform.</li></ul><p>With this course you’ll get a solid understanding of the services and platforms available on AWS for Machine Learning projects, build a foundation to pass the certification exam and feel equipped to use the AWS ML portfolio in your own real-world applications.</p><p>Don’t just sit idly by, watching as robotic overlords take over the world. Create your own army of sentient machines and beat them at their own game! And keep being awesome.</p><ul><li><strong>Books</strong></li></ul><p>One underestimated tip : RTFM (Read The Fabulous Manual 😉 ). There will be many questions from <a href="https://docs.aws.amazon.com/"><strong>this documentation</strong></a>, including very specific ones. For example, what type of input a certain SageMaker algorithm accepts. You have to know this documentation <strong>very well</strong>.</p><p>Make sure you do not only focus on the ML services but also the other services under<strong> Analytics</strong> and <strong>Application Integration </strong>section.</p><ul><li><strong>Tips for preparation</strong></li></ul><p>Prepare a plan ! Elaborate a clear schedule for learning and keep track of your progress. A tool I can recommend for this job is <strong>Notion</strong>.</p><p>Take practice exams before starting the course, so your mind can catch the real answer afterwards during the learning phase. You can get bored easily when you are just watching a dozen of videos. You may think of this as an active learning technique. Don’t hesitate to take the practical hands-on lab on <strong>Cloud Guru</strong> or sign up for a free AWS account.</p><h3><strong>Extra points</strong> ; technical tips</h3><ul><li><strong>Link services with keywords</strong> <br>ETL can highly correlate with AWS Glue<br>Cleaning a huge data, may require AWS EMR<br>Protobuf is a good input type for AWS Sagemaker models<br>Streaming tasks require AWS Kinesis</li><li><strong>Understand how services can be linked together for a pipeline</strong><br>AWS Glue can be a good fit with AWS Athena<br>S3 is really valuable for SageMaker (input data, output models)<br>AWS CloudWatch can be helpful with SageMaker<br>Apache Spark library can be used in Sagemaker</li><li><strong>Models input</strong><br>Protobuf is a good input type for AWS SageMaker models<br>AWS SageMaker pipe mode is generally the best approach<br>AWS BlazingText require a special input format</li><li><strong>Over-Under fitting</strong><br>Detect by difference of error train-validation dataset<br>Overfitting require less complex model<br>L1 and L2 are really helpful for model training<br>Early-stopping and dropout can fix overfitting</li><li><strong>Learning rate / batch size<br></strong>Learning rate too Large leads to overshooting true minima<br>Learning rate too Small leads to slows down convergence, takes more time<br>batch size<strong> </strong>too Large leads to stuck at local minima<br>batch size<strong> </strong>less Size leads to true minima</li><li><strong>Services with least development or maintenance</strong><br>Mainly serverless services : S3, Glue, Athena, and Quicksight</li><li><strong>Anomaly detection on a stream</strong><br>Kinesis Data Analytics use Random Cut Forest that AWS is proud of.</li></ul><p>I see you get until here, so I would wish you good luck and hope this with was helpful ! Another good article that helped me during preparation : <a href="https://simpledataflow.com/machine-learning-specialty-exam/">Link</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9da3f8b03a10" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DistilBert for Question-Answer System]]></title>
            <link>https://medium.com/analytics-vidhya/distilbert-for-question-answer-system-e00954c37f13?source=rss-684f4f39050c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e00954c37f13</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[questions-answers]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Wael Dimassi]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 13:26:31 GMT</pubDate>
            <atom:updated>2020-07-10T13:55:14.908Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article I will present how to create an Open Domain Question Answering bot using bm25 and DistilBert.</p><p>As anyone searching for a piece of information, he will enter the question or the keywords and make a Google search about that topic. He will start by clicking on the first links provided. Then will find himself facing several long paragraphs where the needed information is hidden. It could be a tiring and unappealing task to do especially for a person in a hurry. That’s why our project will handle all that by taking your question and going through the whole pipeline for you. Thanks to Natural Language Processing revolutionary techniques and Machine Learning models that now can give way much better results than ever, this is possible and can facilitate your life.</p><p><strong>How does that work ?</strong></p><p>Let’s get back to the person in a hurry for the information. First, he needs to read pretty much all paragraphs to locate the best one that contains the relevant information, then he will focus on understanding it more and then extracting the exact answer. It is pretty much the same thing here that is happening in our pipeline: the <strong>Ranker </strong>will be responsible for searching for the best paragraph and the <strong>Reader </strong>is the one that will extract the exact answer. We can clearly see the analogy between human interaction with finding the right answer and our pipeline. But let me explain more.</p><p><strong>Question Answering Bot :</strong></p><p>The three main components that the Question Answering bot mechanism is built on are the <strong>Retriever</strong>, the <strong>Ranker </strong>and the <strong>Reader</strong>. The architecture of our pipeline is represented in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o14LDUDNidCvhqGGF8e2oQ.png" /><figcaption>High level architecture</figcaption></figure><p>In more explicit words, we need to make 3 kinds of sorting ; search the best Wikipedia article then the best paragraph and finally the best sentence or paragraph chunk.</p><p>So let’s dig deeper into each one :</p><p><strong>The retriever : </strong>It acts at the early stage, when a question is asked and then sent to the system, it mainly outsources that to google search engine to find the best Wikipedia article and usually get this step returns the required link for the rest of the pipeline. <br>Once there, we will scrape that web page and extract all paragraphs, store them in a list that will be cleaned afterwards. Because text format is very crucial we really need to focus on how to delete noise and HTML tags, non-useful special characters or Unicode errors.</p><p><strong>The ranker:</strong> Actually in this phase, we have the list of paragraphs, that could be enormous in certain cases, and the question that we need to match. Such a task is the hardest part since the final output depends a lot on this. We used a bag-of-words retrieval function, called BM25, that ranks a set of documents based on the query terms appearing in each document, regardless of their proximity within the document.</p><p><strong>The reader:</strong> Final step is getting the final part from the paragraph where the answer of our question is. This kind of task may require a language model with NLU and trained on SQuaD. In fact, even with using big model, skipping the ranking phase is error prone and will affect enormously the results because such models can get lost in big text data but are very powerful when it comes to average / small paragraphs.</p><p>As you may know there are many OD QA solutions that tackle the same problem. The accuracy of the results depends on the chosen models and NLP tasks used. But our motivation for this project was testing the DistilBERT as a question answering model to obtain a high accuracy and relevant results. We have also taken into consideration building a web interface for our solution that we will present at the end of the article.</p><p><strong>Ranker using bm25 what is it? Why did we use it?</strong></p><p><strong>Bm25 </strong>is<strong> </strong>a ranking function. It represents state-of-the-art <a href="https://en.wikipedia.org/wiki/TF-IDF">TF-IDF</a>-like retrieval functions used in document retrieval. It is more than a term scoring method, but rather a method for scoring documents with relation to a query. <br>We have chosen to use it since it has better results than TF-IDF when used in ranking and it’s more robust. (2)</p><p>Actually BM25 stands for “Best Match 25”. Released in<a href="http://trec.nist.gov/pubs/trec3/t3_proceedings.html"> 1994</a>, it’s the 25th iteration of tweaking the relevance computation. BM25 has its roots in<a href="http://nlp.stanford.edu/IR-book/html/htmledition/probabilistic-information-retrieval-1.html"> probabilistic information retrieval</a>. Probabilistic information retrieval is a fascinating field unto itself. Basically, it casts relevance as a probability problem. A relevance score, according to probabilistic information retrieval, ought to reflect the probability a user will consider the result relevant.</p><p><strong>Ranking Function :</strong></p><p>Ranking function will calculate the score for each paragraph and the question asked and then sort it and returns the highest score.</p><p>For Question Q, Documents as D we use this formula :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/0*Y9Eb0NmYA7GCC03e" /></figure><p>Where IDF : Inverse Term Frequency, where f (qi , D ) is qi ‘s<a href="https://en.wikipedia.org/wiki/Term_frequency"> term frequency</a> in the document D. k1 and <em>b</em> are free parameters, usually chosen, in absence of an advanced optimization, as k1 ∈ [ 1.2 , 2.0 ] and b = 0.75.</p><p>Well don’t get intimidated by the math behind this. What we wanted to show is how this overpass TF-IDF and as far as we used this it gave promising results. For general knowledge, this method is used in O’Reilly library project for searching book chunks.</p><p><strong>Why Distilbert as a QA model?</strong></p><p>DistilBERT is a reference to distillation technique for making Bert models smaller thus faster. In fact, <strong>distillation</strong> is a technique used for compressing a large model, called the teacher, into a smaller model, called the student to reproduce the same behavior. Since the other models are very large ; today we talk about models with <a href="https://venturebeat.com/2019/08/13/nvidia-trains-worlds-largest-transformer-based-language-model/">8.3 billion parameters</a>: 24 times larger than BERT-large, 5 times larger than GPT-2, while<a href="https://arxiv.org/abs/1907.11692"> RoBERTa</a>, the latest work from Facebook AI, was trained on 160 GB of text. Which is huge and furthermore not a good choice for production, even with GPU, this is still hard to implement regarding size and response time.</p><p>So the question was how can we tame these big models ?</p><p>Actually, several papers were concerned toward this. The majority of the papers are not academic ones, since the models are becoming so big that require huge computational power and also the environmental cost of training those models is expensive : They require a lot of energy thus a lot of carbon dioxide emission.</p><p>For making smaller models there are 3 main approaches :</p><ul><li><strong>Pruning</strong> :</li></ul><p>Model pruning in a nutshell is about eliminating superficial values in the weight tensors. Setting the neural network parameters’ values to zero to remove what is estimated as unnecessary connections between the layers of a neural network. <br>The process of pruning happens during the training process to allow the neural network to adapt to the changes. In fact, this allows us to reduce 90% of the sparse model for MNIST 12 MB to 2 MB. Also, this technique is applicable to any NN type across distinct tasks.</p><ul><li><strong>Quantization</strong>:</li></ul><p>Quantization is also a simple technique for performing computations and storing tensors at lower bit widths. Actually a quantized model has changed some or all of the values on tensors to integers rather than floating-point values. This allows for a more compact model representation and the use of high performance vectorized operations on many hardware platforms. The quantized model compared to the normal models can get to a 4x reduction in the size and a 4x reduction in memory bandwidth requirements. Thus, simpler calculation less amount of energy consumption. But, with quantized weights during the post training, there could be an accuracy loss, particularly for smaller networks which is the price to pay.</p><ul><li><strong>Distillation</strong> :</li></ul><p>Knowledge distillation is a model compression method in which a small model is trained to mimic a pre-trained, larger model. This training setting is sometimes referred to as “teacher-student”, where the large model is the teacher and the small model is the student.</p><p>As a good example of distillation, we have DistilBert, if we compare its performance against: BERT base which is not one but DistilBERT’s teacher we can obviously say that DistilBERT <strong>compares surprisingly well to BERT</strong>: we are able to <strong>retain more than 95% of the performance</strong> with 40% fewer parameters. All this while in terms of inference time, DistilBERT is <strong>60% faster and smaller </strong>than BERT and<strong> 120% faster and smaller </strong>than ELMo+BiLSTM.</p><p>Question answering System interface :</p><p>After some extra work, I managed to wrap-up my Question Answering System into an API. And with some luck, I created a nice HTML page that consume that API in a way to send the query and receive the main answer, paragraph where the answer was retrieved and the link of the article from Wikipedia.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_eqimiAc1lQ6qZWz" /><figcaption>Web interface of the QA System</figcaption></figure><p><strong>Conclusion :</strong></p><p>This project was initially for fun only and for the sake of investing the power of the DistilBert. You may check the whole project in <a href="https://github.com/dimwael/QuestionAnsweringDistilbert/tree/master">here</a>. Please note that pull requests are welcome.<br>Feel free to reach out to me in case you have a comment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e00954c37f13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/distilbert-for-question-answer-system-e00954c37f13">DistilBert for Question-Answer System</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dash for data science]]></title>
            <link>https://medium.com/@dim.wael/dash-for-data-science-10669ab4aa5f?source=rss-684f4f39050c------2</link>
            <guid isPermaLink="false">https://medium.com/p/10669ab4aa5f</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[dash]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Wael Dimassi]]></dc:creator>
            <pubDate>Fri, 04 Oct 2019 14:12:31 GMT</pubDate>
            <atom:updated>2019-10-04T14:12:31.739Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Often, the output of a project is a overlook or graphs to that explains the whole work done. That is why we want to reach a high quality of data visualization that showcase our efforts. That’s why picking up the right tool is a high priority especially if we don’t have lot of time for extra development. In this brief tutorial, we will show how easy it is t create dynamic dash-boards<strong>.</strong></p><h3><strong>What is Dash ?</strong></h3><p><a href="https://plot.ly/dash/">Plotly Dash </a>an python package that enables you to build interactive web apps using both the statistical power of python and the interactivity of the modern web. An excellent and efficient alternative to spreadsheets and printed visualizations, it also saves space and time in the construction, automation, and distribution of data visualizations and statistical analyses.</p><p>Dash is a user interface library for creating analytical web applications. Those who use Python for data analysis, data exploration, visualization, modelling, instrument control, and reporting will find immediate use for Dash.</p><h3><strong>What can Dash provide ?</strong></h3><p>Basically, Dash contains various amount of elements starting from simple HTML components to dynamic charts like Sankey diagrams, 3D and maps.</p><h3><strong>How to start with Dash Plotly then ?</strong></h3><p>Well it is quite easy to begin with this. But first some installation must be done : So, let’s get some useful packages.</p><ol><li>The core dash backend.</li><li>Dash front-end</li><li>Dash HTML components</li><li>Dash core components</li><li>Plotly</li></ol><pre>pip install dash==0.21.1  <br>pip install dash-renderer==0.13.0  <br>pip install dash-html-components==0.11.0<br>pip install dash-core-components==0.23.0  <br>pip install plotly --upgrade</pre><p>Once our environment is ready, we can start coding !!<br>Generally a Dash application is usually composed of two parts. The first part is the layout and describes how the app will look like and the second part describes the interactivity of the application. First part is where we will initiate the html components and charts while second part is where we will tell them how to behave or more precisely how to update. After user interactivity the component should react to that and keep itself always updated. And that’s why we need the Callbacks !</p><h3><strong>Simple App !</strong></h3><p>To show how easy it is to create a nice graphs, we will start with a nice simple application. For that we will import our tools ;)</p><pre>import dash<br>import dash_html_components as html<br>import dash_core_components as dcc<br>import plotly.graph_objects as go</pre><p>Then we will randomly generate Inputs :</p><pre># Import numpy library<br>import numpy as np</pre><pre>random_x = np.random.randint(1,101,100) # Randomly generate x vector<br>random_y = np.random.randint(1,101,100) # Randomly generate yvector</pre><p>In case we want a pseudo-random vector we can still use np.random.seed() for that ;) .. <br>For now, we already created our inputs , we can now get into the creation of the application :</p><pre>app = dash.Dash()<br>app.layout = html.Div([dcc.Graph(id=&#39;scatterplot&#39;,<br>    figure = {<br>           &#39;data&#39;:[go.Scatter(<br>                  x=random_x,<br>                  y=random_y,<br>                  mode=&#39;markers&#39;,<br>                  marker={<br>                           &#39;size&#39;:12,<br>                           &#39;color&#39;:&#39;rgb(51,204,153)&#39;,<br>                           &#39;symbol&#39;:&#39;pentagon&#39;,<br>                           &#39;line&#39;:{&#39;width&#39;:2}})]<br>,&#39;layout&#39;:go.Layout(title=&#39;My Graph&#39;)}<br>)])</pre><p>So what’s in that code ? <br>First we created the layout of the app, then we initiated with a Div that contains our graph. The graph is a figure with the id ‘scatterplot’. The id is very important in case you want to call your figure or component from a function. <br>Our graph is a Scatter plot that takes our random vectors as x and y input.</p><p>At this point, this is enough but we want to make it more personalized. We want to precise the markers looks, like size,color, symbol and line width. Because dash is about simplicity and beauty !! And finally added the title to describe our chart :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xo2zGtVqLVBhiYctX5cYQ.png" /></figure><h3>Our first dynamic app !</h3><p>For this part things will be more interesting ! If we want to only show a percentage of the data depending on the user demands, we need to put a slider. Slider that can determine the amount of points should be shown in the graph.</p><p>In this part, our code will be divided into 2 parts, first where the components and layout are presented then , the theirs behaviors. <br>So for the first part , we just create a graph ,slider and a div :</p><pre>app = dash.Dash()</pre><pre>app.layout = html.Div([<br>dcc.Graph(id=&#39;scatterplot&#39;),           # Our graph<br>dcc.Slider(                            # The Slider<br>        id=&#39;my-slider&#39;,<br>        min=0,<br>        max=100,<br>        step=1,<br>        value=10,<br>    ),<br>html.Div(id=&#39;slider-output-container&#39;) # Text <br>])</pre><p>To wire things together with the behavior we will need the callback functions and the id as we mentioned earlier to identify the input and output. Each function can take as many inputs as you want and returns the output which is mainly the graph :</p><pre><a href="http://twitter.com/app">@app</a>.callback(<br>dash.dependencies.Output(&quot;scatterplot&quot;, &quot;figure&quot;),<br>[dash.dependencies.Input(&#39;my-slider&#39;, &#39;value&#39;)])<br>def update_graph(slider_value):<br>    new_random_x = random_x[:slider_value]<br>    new_random_y = random_y[:slider_value]<br>    return {<br>           &#39;data&#39;:[go.Scatter(<br>                 x=new_random_x.tolist(),<br>                 y=new_random_y.tolist(),<br>                  mode=&#39;markers&#39;,<br>                  marker={<br>                           &#39;size&#39;:12,<br>                           &#39;color&#39;:&#39;rgb(51,204,153)&#39;,<br>                           &#39;symbol&#39;:&#39;pentagon&#39;,<br>                           &#39;line&#39;:{&#39;width&#39;:2}})]<br>,&#39;layout&#39;:go.Layout(title=&#39;My Graph&#39;)}</pre><p>The function takes as a parameters the variable coming from the input which is the slider value.</p><h3>The Map !</h3><p>In this final part, we will work on a map project. Because creating a map is very important to see the distribution and detect more classification potential. With dash, such a task is quite easy. You can create that , customize it and deployed without any external API because dash is already provide mapbox services. You only need the <em>mapbox_access_token </em>for that.</p><p>As a first step we need to set the layout of our map :</p><pre><em>layout = dict(<br> autosize=True,<br> automargin=True,<br> margin=dict(<br> l=30,<br> r=30,<br> b=20,<br> t=40<br> ),<br> hovermode=”closest”,<br> plot_bgcolor=”#F9F9F9&quot;,<br> paper_bgcolor=”#F9F9F9&quot;,<br> legend=dict(text=’opo’,font=dict(size=19), orientation=’h’),<br> title=’map title‘,<br> mapbox=dict(<br> accesstoken=mapbox_access_token,<br> style=”light”,   # Can be changed to &quot;dark&quot;<br> center=dict(     # Default center of the map<br> lon=-14.05,<br> lat=15.54<br> ),<br> zoom=5,          # Default zoom value<br> )<br>)</em></pre><p>Once the layout is specified , we can now focus more on the implementation of the callback function :</p><pre><a href="http://twitter.com/app">@app</a>.callback(Output(&#39;mapbox&#39;, &#39;figure&#39;),[Input(&#39;year-picker&#39;,&#39;value&#39;)])<br>def make_main_figure(selected_year):<br>   <br>    df1 = df[df[&#39;year&#39;]==selected_year]<br>    traces = []<br>    for dff in df1.groupby(&#39;year&#39;):<br>        trace = dict(<br>            type=&#39;scattermapbox&#39;,<br>            lon=dff[1][&#39;longitude&#39;],<br>            lat=dff[1][&#39;latitude&#39;],<br>            ids = dff[1][&#39;location&#39;],<br>            #text=&quot;rrrr&quot;,<br>            textposition = &quot;middle center&quot;,</pre><pre>title =&#39;title&#39;,<br>            #hovertemplate =&quot;district: %{dff[1][&#39;location&#39;]}&quot;,<br>            #hoverinfo = &#39;text + name&#39;,<br>            customdata=dff[1][&#39;fatalities&#39;],<br>            name= &#39;Event&#39;,#dff[1][&#39;location&#39;],<br>            legend= &#39;Event&#39;,<br>            legendgroup =&#39;fofo&#39;,</pre><pre>showlegend = True,<br>            #color_continuous_scale=px.colors.cyclical.IceFire,<br>            marker=dict(<br>                size=12,<br>                opacity=0.6,<br>            ),<br>            text = dff[1][&#39;location&#39;],<br>            hoverinfo = &quot;lat+lat+text&quot;,<br>            hovertemplate = &quot;District: %{text} &lt;br&gt;Fatalities: %{customdata}&quot;,<br>            #hovertext = &quot;all&quot;, #dff[1][&#39;location&#39;]+dff[1][&#39;fatalities&#39;],<br>        )<br>        traces.append(trace)<br>    figure = dict(data=traces, layout=layout1)<br>    return figure</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10669ab4aa5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build a Genetic Algorithm in Unity3D]]></title>
            <link>https://medium.com/@dim.wael/how-to-build-a-genetic-algorithm-in-unity3d-77d336da14de?source=rss-684f4f39050c------2</link>
            <guid isPermaLink="false">https://medium.com/p/77d336da14de</guid>
            <category><![CDATA[genetic-algorithm]]></category>
            <category><![CDATA[unity3d]]></category>
            <category><![CDATA[game-development]]></category>
            <dc:creator><![CDATA[Wael Dimassi]]></dc:creator>
            <pubDate>Wed, 25 Jul 2018 10:45:48 GMT</pubDate>
            <atom:updated>2018-07-25T10:46:37.334Z</atom:updated>
            <content:encoded><![CDATA[<h4>Tutorial for beginners</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*7D-eQuIqW_m5HxWrEqUdgw.jpeg" /></figure><p>With the right process in place, it will not be difficult to find state-of-the-art Genetic Algorithm for a given prediction task. Out of the three approaches — manual, machine-assisted, and algorithmic — this article will focus on ‘Gamification’ of AI since a huge number of Deep learning -specially Reinforcement learning- thesis and reaserches are oriented to games the apparition of <a href="https://blog.openai.com/openai-five/">OpenAI Five</a> and <a href="https://github.com/openai/universe">Universe</a>. The article will cover how I did it, get to the proof that the method works, and provide the understanding of why it works. The main principle is simplicity.</p><h3>Let’s get started shall we ?</h3><p>To begin properly we need to introduce some simple definitions :</p><ul><li><strong>Initial population </strong>: as its name refer to, a genetic algorithm is biomimicry of how genes and chromosomes evolve from generation to the next one. like Darwin said about the natural selection only keeps the best ,in our case, the most fitted genes. So short story long, the initial population is the set of individuals who are characterized by a set of parameters (variables) known as <strong>Genes</strong>. Genes are joined into a string to form a <strong>Chromosome</strong> (solution).</li><li><strong>Fitness Function : </strong>shows how fit the individual is (the ability of an individual to compete with other individuals). It gives a <strong>fitness score</strong> calculated on his potential in that very environment. The probability that an individual will be selected for reproduction is based on its fitness score.</li><li><strong>Selection </strong>: this<strong> </strong>phase consist on selectting the fittest individuals and let them pass their genes to the next generation. Two pairs of individuals (<strong>parents</strong>) are selected based on their fitness scores. Individuals with high fitness have more chance to be selected for reproduction.</li><li><strong>Crossover</strong> : The main idea of the genetic algorithm. In this phase where happen the whole process of reproduction and creation of new generation based on the best fitted individuals of the previous population.</li><li><strong>Mutation</strong> : If it happens in nature, it should happen in the algorithm. Thus, we need to introduce it with in low random probabilites to maintain diversity within the population and prevent premature convergence.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1TDD1O5SFlvwjI5pFdqIOg.png" /></figure><h3>Sounds good but… I want an <a href="https://genetic-algo.herokuapp.com/">application</a>!</h3><p>Well ! we will translate all that into code to create a simple application about colors picking game : the main idea is to select colors so finally we have a popultaion with a dominant gene.</p><p>First Step : Create our initial population in a random way</p><pre>for (int i = 0; i &lt; populationSize; i++)<br>{<br>    Vector3 pos = new Vector3(Random.Range(-9, 9), Random.Range(-4.5f, 4.5f), 0);<br>    GameObject go = Instantiate(personPrefab, pos, Quaternion.identity);<br>    go.GetComponent&lt;DNA&gt;().r = Random.Range(0.0f, 1.0f);<br>    go.GetComponent&lt;DNA&gt;().g = Random.Range(0.0f, 1.0f);<br>    go.GetComponent&lt;DNA&gt;().b = Random.Range(0.0f, 1.0f);<br>    population.Add(go);<br>        }</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WBZqCZPXlXNgCDpPx24blw.png" /></figure><p>the life time of the current population will be set to 10 seconds , and at that time, we will be choosing our prefab with our chosen color.</p><p>After that amount of time we should breed the new generation from our chosen prefab of previous population.</p><pre>void BreedNewPopulation()<br>{<br>    List&lt;GameObject&gt; newPopulation = new List&lt;GameObject&gt;();<br>    //get rid of unfit individuals<br>    List&lt;GameObject&gt; sortedList = population.OrderByDescending(o =&gt; o.GetComponent&lt;DNA&gt;().timeToDie).ToList();<br> <br>     population.Clear();<br>     //breed upper half of sorted list<br>     for (int i = (int)(sortedList.Count / 2.0f) - 1; i &lt; sortedList.Count - 1; i++)<br>     {<br>         population.Add(Breed(sortedList[i], sortedList[i + 1]));<br>         population.Add(Breed(sortedList[i + 1], sortedList[i]));<br>      }<br> <br>      //destroy all parents and previous population<br>      for (int i = 0; i &lt; sortedList.Count; i++)<br>      {<br>         Destroy(sortedList[i]);<br>       }<br>       generation++;<br>    }</pre><p>The breed function will be set as this :</p><pre>GameObject Breed(GameObject parent1, GameObject parent2)<br>{<br>   Vector3 pos = new Vector3(Random.Range(-9, 9), Random.Range(-4.5f, 4.5f), 0);<br>   GameObject offspring = Instantiate(personPrefab, pos, Quaternion.identity);<br>   DNA dna1 = parent1.GetComponent&lt;DNA&gt;();<br>   DNA dna2 = parent2.GetComponent&lt;DNA&gt;();<br>   //swap parent dna<br>   if (Random.Range(0, 10) &lt; 5)<br>   {<br>      offspring.GetComponent&lt;DNA&gt;().r = Random.Range(0, 10) &lt; 5 ? dna1.r : dna2.r;<br>      offspring.GetComponent&lt;DNA&gt;().g = Random.Range(0, 10) &lt; 5 ? dna1.g : dna2.g;<br>      offspring.GetComponent&lt;DNA&gt;().b = Random.Range(0, 10) &lt; 5 ? dna1.b : dna2.b;<br>    }<br>    else<br>    {<br>       offspring.GetComponent&lt;DNA&gt;().r = Random.Range(0.0f, 1.0f);<br>       offspring.GetComponent&lt;DNA&gt;().g = Random.Range(0.0f, 1.0f);<br>       offspring.GetComponent&lt;DNA&gt;().b = Random.Range(0.0f, 1.0f);<br>    }<br>    return offspring;<br>}</pre><p>The convergence is impressive. After 25–35 generation, we can have a homogenous population with a dominant color.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yv7-hzk4UpuI9NcJyjX2Cg.png" /></figure><h3>Finally !</h3><p>I hope this was fun and helpful, hands-on way to learn how to build your own GA. Try it for yourself and enjoy tweeking to get better results. Or go further and try to implement a GA on another problem set; see how you would change the breed functions to handle other types of chromosomes. We’re just scratching the surface here!</p><p>To enjoy this project please visit it on heroku : <a href="https://genetic-algo.herokuapp.com/">here!</a></p><p>Please comment what do you think about this project .<br>Any feedback or suggestions for improvement will be really appreciated!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77d336da14de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build your own Twitter sentiment analysis system in Python]]></title>
            <link>https://medium.com/@dim.wael/how-to-build-your-own-twitter-sentiment-analysis-system-in-python-e5139b440445?source=rss-684f4f39050c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5139b440445</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Wael Dimassi]]></dc:creator>
            <pubDate>Sun, 22 Jul 2018 11:09:30 GMT</pubDate>
            <atom:updated>2018-07-22T11:09:30.496Z</atom:updated>
            <content:encoded><![CDATA[<h4>Beginner Tutorial</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DU5VNuMw8ewQbY4KbFNdAA.jpeg" /></figure><p><strong>Motivation: </strong>As part of my personal journey to gain a better understanding of NLP , I’ve decided to build a Twitter scroller from scratch . I believe that such project may help a lot in different fields.</p><p>This article contains what I’ve learned, and hopefully it’ll be useful for you as well!</p><h3>What’s NLP?</h3><p><strong>Natural language processing (NLP) </strong>refers to the way we, humans, communicate with each other.</p><p>Namely, speech and text.<em> is the ability of computers to analyze, understand and generate human language, including speech.</em></p><p>But basically,NLP is the ability of a computer program to understand human language as it is spoken or written. <a href="http://searchbusinessanalytics.techtarget.com/definition/natural-language-processing-NLP"><strong><em>NLP</em></strong></a><strong><em> is a component of artificial intelligence (AI) where many studies have been made to give the tools to interact in a normal way with the humains.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*i5D2GDiHykPnjCk9l5OhRg.jpeg" /><figcaption>NLP</figcaption></figure><p><strong><em>Areas where NLP is useful:</em></strong></p><ul><li><em>enterprise search is the organized retrieval of structured and unstructured data within an organization.</em></li><li><em>interpret free text and make it analyzable there is a huge amount of information stored in free text files. For example, patients’ medical records.</em></li><li><em>opinion mining (sentiment mining) opinion mining is a type of NLP for tracking the mood of the public about a certain product and to figure out the right location in the market.</em></li></ul><h3>What are the NLP tools ?</h3><p>Since NLP is a large domain in AI, it wouldn’t be surprizing to find a lot of API and such a huge amount of articles.</p><p>That’s why i only choosed these elite packages who handle a wide range of tasks such as part-of-speech (POS) tagging, sentiment analysis, document classification, topic modeling, and much more.</p><ul><li>NLTK : t’s the most famous Python NLP library, and it’s led to incredible breakthroughs in the field. NLTK is also popular for education and research. NLTK has over 50 corpora and lexicons, 9 stemmers, and dozens of algorithms to choose from. Yet, this is also one of NLTK’s major downsides. It’s heavy and slippery, and it has a steep learning curve. The second major weakness is that it’s slow and not production-ready.</li><li>TextBlob : TextBlob makes text processing simple by providing an intuitive interface to NLTK. It’s a welcome addition to an already solid lineup of Python NLP libraries because it has a gentle learning curve while boasting a surprising amount of functionality. That’s why i have choosen this very Library in my project.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*nT7vG_XHGdziMXv7xfFoZA.png" /><figcaption>TextBLob Sentiment Analysis</figcaption></figure><ul><li>spaCy : Because it’s built on Cython, it’s also lightning-fast. Folks have called spaCy “state-of-the-art,” and it’s hard to disagree. Its main weakness is that it currently only supports English.</li></ul><h3>Sounds good but… I want an application!</h3><p>The introduced application will be done in 3 phases :</p><ul><li>Collecting Data : Using the Twitter API to get a list of tweets about a specific searched term</li></ul><pre>from tweepy import Stream<br>from tweepy import OAuthHandler<br>import tweepy<br>from tweepy.streaming import StreamListener<br><br>consumer_key=&quot;&quot;<br>consumer_secret=&quot;&quot;<br>access_token=&quot;&quot;<br>access_secret=&quot;&quot;<br><br>auth = OAuthHandler(consumer_key, consumer_secret)<br>auth.set_access_token(access_token, access_secret)<br>api = tweepy.API(auth)<br></pre><pre>#Get a list of tweets:<br>public_tweets = api.search(query)</pre><ul><li>Clean Data : Since the data are collected in a crude status we will need to put them in a certain format to begin the sentiment analysis; for that we have to introduce some changements :</li></ul><pre>import re</pre><pre>pattern1 = re.compile(&#39;http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&amp;+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+&#39;)<br>    pattern2 = re.compile(&#39;RT @:?(?:[a-zA-Z]|[0-9]|[$-_@.&amp;+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+&#39;)</pre><pre>#Foreach Element we should introduce our cleaning Process<br>for tweet in public_tweets:<br><br>        tweet.text = pattern1.sub(&#39;&#39;, tweet.text)<br>        tweet.text = pattern2.sub(&#39;&#39;, tweet.text)<br>        str(tweet.text)</pre><ul><li>Here we are now ready to extract and apply the sentiment analysis algorithm : In fact the TextBlob will provide us the Polarity and the subjectivity of each tweet in our list.</li><li>Finally i have decided to put the cherry on the cake with <a href="https://plot.ly/products/dash/">dash </a>— a web based interface for python</li></ul><h3>Results :</h3><p>NLP is very promissing field of AI , yet challenging but will introduce a new era for the humanity. This project might be an intoduction to extract , treat and visualize your ‘hashtag’ or your trademark through the web.</p><p>Lastly, I have to say that these observations are true for any research at any time. However, one thing which is true in general is that there is not enough specefic informations to explain Polarity.</p><p>Testing the other NLP Library will be so fun : So which one is your favorite? Please comment with the reasons.<br>Any feedback or suggestions for improvement will be really appreciated!</p><p>Check out the whole project in my GitHub <a href="https://github.com/dimwael/TwitterScroller">here!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5139b440445" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>