<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ZAIDI Houda on Medium]]></title>
        <description><![CDATA[Stories by ZAIDI Houda on Medium]]></description>
        <link>https://medium.com/@zaidi.houd?source=rss-3978cb98aacd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*13AOteyoaSiA68V830fFRw.png</url>
            <title>Stories by ZAIDI Houda on Medium</title>
            <link>https://medium.com/@zaidi.houd?source=rss-3978cb98aacd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:21:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zaidi.houd/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Smart data for helping in investment choice — Part III: Cleaning & exploration data with pyspark &…]]></title>
            <link>https://medium.com/@zaidi.houd/smart-data-for-helping-in-investment-choice-part-iii-cleaning-exploration-data-with-pyspark-fa494f87df58?source=rss-3978cb98aacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa494f87df58</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[cleaning]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[ZAIDI Houda]]></dc:creator>
            <pubDate>Thu, 25 Jan 2018 10:02:28 GMT</pubDate>
            <atom:updated>2018-01-25T10:02:28.108Z</atom:updated>
            <content:encoded><![CDATA[<h3>Smart data for helping in investment choice — Part III: Cleaning &amp; exploration data with pyspark &amp; cassandra</h3><p>This is the third part of my project that I started several weeks ago. The objective of my project is to develop a predictive model that allows to estimate the price of a real estate or an automobile. The summary of this project is below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6H7Td7Xb4tTNiJ-A9EH3rQ.png" /></figure><ol><li><a href="https://medium.com/@zaidi.houd/smart-data-for-helping-in-investment-choice-part-i-collect-data-scrapping-55946ef7ad21"><strong>Part I: Collect data — scraping</strong></a></li><li><a href="https://medium.com/p/d179fcc8ee7/edit"><strong>Part II: Store data — cassandra</strong></a></li><li><strong>Part III: Cleaning &amp; exploration data</strong></li><li><strong>Part IV: Web App that use a simple model in Python to estimate the properties price depending on its location or condition.</strong></li><li><strong>Part V: Improving the model with additional features and a better algorithm</strong></li><li><strong>Part VI: Setup of a chatbot to chat with this App in natural language</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6H7Td7Xb4tTNiJ-A9EH3rQ.png" /></figure><p>In this part of my project i’ll explain how explore my dataset and clain it with pyspark.</p><p>Firstly, I’ll explain the different step to set up a pyspark developement environement in virtual ubuntu VM. I’ll use the Jupyter Notebook for the this tutorian, but if you prefer .py files with an IDE or Text Editor that is no problem!</p><p>Before we begin the setup and coding with Pyspark, let’s discuss what Spark is in the BigData context and why python is very suitable for our needs. We’ll begin with a general explanation of what spark is:</p><ol><li><strong>Spark introduction concept:</strong></li></ol><p>Spark is one of the latest technologies being used to quickly and easily handle Big Data. It is an open source project on Apache. It was first released in February 2013 and has exploded in popularity due to it’s ease of use and speed. It was created at the AMPLab at UC Berkeley</p><p>Spark is a fast and general engine for large scale data processing. It is optimised to run in memory, allowing it to process data far more quickly than alternative approaches like Hadoop’s MapReduce. In this tutorial I guess the people who are interested in it are familiar with these technologies (like hadoop) so I will not dwell on them.</p><p>Hadoop MapReduce requires files to be stored in HDFS, Spark does not, it can use data stored in a variety of formats : Cassandra/ AWS S3/HDFS/ And more. Spark also can perform operations up to 100x faster than Hadoop MapReduce. So how does it achieve this speed? MapReduce writes most data to disk after each map and reduce operation. Spark keeps most of the data in memory after each transformation. Spark can spill over to disk if the memory is filled</p><p>in the next figure we see the different components of spark :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*j--s4OeurgZt8IOZ_uzINg.png" /></figure><ul><li><strong><em>Spark Core</em></strong> contains the basic functionality of Spark, including components for task scheduling, memory management, fault recovery, interacting with storage systems, and more. Spark Core is also home to the API that defines resilient distributed datasets (RDDs), which are Spark’s main programming abstraction. An RDD in Spark is simply an immutable distributed collection of objects. Each RDD is split into multiple partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes. Spark Core provides many APIs for building and manipulating these collections.</li><li><strong><em>Spark SQL :</em></strong> is Spark’s package for working with structured data. It allows querying data via SQL as well as the Apache Hive variant of SQL. Spark SQL supports many sources of data, including Hive tables, Parquet, and JSON. Beyond providing a SQL interface to Spark, Spark SQL allows developers to intermix SQL queries with the programmatic data manipulations supported by RDDs in Python, Java, and Scala, all within a single application, thus combining SQL with complex analytics.</li><li><strong><em>Spark Streaming</em></strong> : is a Spark component that enables processing of live streams of data. Examples of data streams include logfiles generated by production web servers, or queues of messages containing status updates posted by users of a web service. Spark Streaming provides an API for manipulating data streams that closely matches the Spark Core’s RDD API, making it easy for programmers to learn the project and move between applications that manipulate data stored in memory, on disk, or arriving in real time. Underneath its API, Spark Streaming was designed to provide the same degree of fault tolerance, throughput, and scalability as Spark Core.</li><li><strong><em>Spark MLLib:</em></strong> spark comes with a library containing common machine learning (ML) functionality, called MLlib. MLlib provides multiple types of machine learning algorithms, including classification, regression, clustering, and collaborative filtering, as well as supporting functionality such as model evaluation and data import. It also provides some lower-level ML primitives, including a generic gradient descent optimization algorithm. All of these methods are designed to scale out across a cluster.</li><li><strong><em>GraphX</em></strong> : is a library for manipulating graphs (e.g., a social network’s friend graph) and performing graph-parallel computations. Like Spark Streaming and Spark SQL, GraphX extends the Spark RDD API, allowing us to create a directed graph with arbitrary properties attached to each vertex and edge. GraphX also provides various operators for manipulating graphs (e.g., subgraph and mapVertices) and a library of common graph algorithms (e.g., PageRank and triangle counting).</li><li><strong><em>Spark resources manager:</em></strong> Spark is designed to efficiently scale up from one to many thousands of compute nodes. To achieve this while maximizing flexibility, Spark can run over a variety of cluster managers, including Hadoop YARN, Apache Mesos, and a simple cluster manager included in Spark itself called the Standalone Scheduler. If you are just installing Spark on an empty set of machines, the Standalone Scheduler provides an easy way to get started; if you already have a Hadoop YARN or Mesos cluster, however, Spark’s support for these cluster managers allows your applications to also run on them.</li></ul><p><strong>2. Local VirtualBox pyspark Set-up : </strong>In this part i’ll explain how install spark, python and jupyter-notebook onto our machine. It’ s important to know that spark 2.1 is incompatible with python 3.6. In this tutorial we’ll work with python 3.5. If we don’t have python 3 and also jupyter and for that you will sometimes need to install pip3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*_CPkhOVBMQCKlducZz28kQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*etKWjEktzJzfHbS3-UfDPw.png" /></figure><p>make sure we have a java version more than 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*htIN24AqpGq6LS-1OWJaVQ.png" /></figure><p>Download the spark zip file from this <a href="https://spark.apache.org/downloads.html"><strong><em>link</em></strong></a><strong><em> </em></strong>or with wget command line.</p><pre>$ wget <a href="https://www.apache.org/dyn/closer.lua/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz">https://www.apache.org/dyn/closer.lua/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz</a></pre><p>Copy it to home folder and dezip it with this command line</p><pre>$ sudo tar -zxvf sparkfolder.tgz</pre><p>Then add the following environment variable (SPARK_HOME) to your bash profile (~/.bash_profile or ~/.bashrc):</p><pre>export SPARK_HOME=&#39;/path/to/spark/folder&#39;<br>export PATH=$SPARK_HOME/bin:$PATH</pre><p>To make the variable available. Now you have Spark! We still need to take a few more steps to get PySpark work.</p><pre>export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH<br>export PYSPARK_DRIVER_PYTHON=jupyter<br>export PYSPARK_DRIVER_PYTHON_OPTS=&#39;notebook&#39;<br>export PYSPARK_PYTHON=python3</pre><p>and run the following:</p><pre>$ source ~/.bashrc </pre><p>Restart your terminal and launch PySpark again:</p><pre>$ pyspark</pre><p>Now, this command should start a Jupyter Notebook in your web browser. <strong>great!</strong> You are now able to run PySpark in a Jupyter Notebook :)</p><p>If you want to use anather python IDE, there is is another and more generalized way to use PySpark with a python package <strong>findSpark.</strong> This package , note spacefic to jupiter notebook, able us to make a Spark Context available in our code.</p><p>To install findspark:</p><pre>$ pip install findspark</pre><p>Launch a regular Jupyter Notebook:</p><pre>$ jupyter notebook</pre><p>Create a new Python [default] notebook and write the following script:</p><pre>import findspark<br>findspark.init(path/spark/folder)</pre><pre>import pyspark</pre><p>Now our development environment is ready we just have to connect it with cassandra. Pyspark can access to the Cassandra data source by the pyspark-cassandra connector. Datastax has created a convenient and easy-to-use connector. This does not require DataStax Enterprise. To enable Cassandra access the Spark Cassandra Connector assembly jar must be included on both the driver and executor classpath for the PySpark Java Gateway. This can be done by starting the PySpark shell similarly to how the spark shell is started. The preferred method is now to use the Spark Packages <a href="http://spark-packages.org/package/datastax/spark-cassandra-connector"><strong><em>website</em></strong></a><strong><em> </em></strong>as<strong><em> :</em></strong></p><pre>$ pyspark --packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.1</pre><p>We can then uses this connector to load Cassandra data source as a DataFrame by specifiying keyspace and table where keyspace = project_invest_decision and table = properties_ads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SmzkKCdDikVD_TuNU6aw3w.png" /></figure><p>Now we have the cassandra table loaded as pypark dataframe. Now I will explain what are the differents tools available to us to clean up and</p><p><strong>3. Cleaning data with pyspark dataframe :</strong></p><p>The reality is, much of your time as a data scientist will be spent preparing and “cleaning” your data.</p><ul><li><strong>Outliers </strong>: Sometimes it’s appropriate to remove outliers from your training data. It’s important but do this responsibly! and understand why you are doing this. For example: in collaborative filtering, a single user who rates thousands of movies could have a big effect on everyone else’s ratings. That may not be desirable. Another example: in web log data, outliers may represent bots or other agents that should be discarded. But if someone really wants the mean income of US citizens for example, don’t toss out Donald Trump just because you want to. Our old friend standard deviation provides a principled way to classify outliers. Find data points more than some multiple of a standard deviation in your training data. What multiple? You just have to use common sense</li><li><strong>Missing Data :</strong> Some information are missing in your data, do not worry you can adjust it with several ways: by removing the associated samples, mean of others samples, last sample, next sample, min sample, max sample, more closly sample, … You can do it but responsably because it can influate your model</li><li><strong>Malicious Data : </strong>This type of data is dangerous because sometimes these data have a very familiar behavior and it becomes difficult to detect their existence. So, before building the model you need to make sure the data is compatible</li><li><strong>Erroneous Data :</strong> you need business knowledge to recognize this type of data. Once detected you should correct or delete them since these values will surely have an impact on the moidele</li><li><strong>Irrelevant Data :</strong> this kind of data is redundant in an analytic model and it may falsify the results, it is also necessary to identify and remove them from the model. This can be bo by business knowledge or also thanks to our old friend covariance.</li><li><strong>Formatting : </strong>If your model is based on several numerical attributes — are they comparable? Example: ages may range from 0–100, and incomes from 0-billions. Some models may not perform well when different attributes are on very different scales. It can result in some attributes counting more than others. Bias in the attributes can also be a problem.</li></ul><p>In order to clean your data, you need to look at your data! Examine it! Question your results! And always do this — not just when you don’t get a result that you like! For the treatment and cleaning of my data I will explain and use three concepts:<br> <br><strong> → Dataframe and spark SQL</strong></p><p><strong>→ Regular expressions</strong></p><p><strong>→ Lambda Function</strong></p><p><strong>→ Pipeline</strong></p><p>In the following I will introduce these concepts and give examples for each element.</p><p><strong>a. DataFrame and spark SQL:</strong></p><p>A Spark DataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R or Pandas. They can be constructed from a wide array of sources such as an existing RDD / database ( SQL or NoSQL) … If you’ve worked with pandas in Python, R, SQL or even Excel, a DataFrame will feel very familiar! Spark DataFrames hold data in a column and row format. Each column represents some feature or variable. Each row represents an individual data point.</p><p>Spark began with something known as the “RDD” syntax which was a little ugly and tricky to learn. Later Spark 2.0 and higher has shifted towards a DataFrame syntax which is much cleaner and easier to work with! Spark DataFrames are able to input and output data from a wide variety of sources. We can then use these DataFrames to apply various transformations on the data. At the end of the transformation calls, we can either show or collect the results to display or for some final processing. In this section we’ll cover all the main features of working with DataFrames that you need to know.</p><ul><li><strong>Showing the data :</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xEmHIvnhNSBr-PYvMwUnmA.png" /></figure><p>In order to avoid truncating the display you can add the option (truncate = false) as :</p><pre>$ dataset.show(truncate=False)</pre><p>To see the schema of our data, we can do it with :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OiyTqf-Knig5-MTXOxYHgg.png" /></figure><p>Also display the different columns or describe a data frame:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZyBv7_7YxloZ949yEgVYCQ.png" /></figure><p>Some data types make it easier to infer schema (like tabular formats such as csv which we will show later). However you often have to set the schema yourself if you aren’t dealing with a .read method that doesn’t have inferSchema() built-in. Spark has all the tools you need for this, it just requires a very specific structure:</p><pre>from pyspark.sql.types import StructField,StringType,IntegerType,StructType</pre><p>Next we need to create the list of Structure fields<br> * :param name: string, name of the field.<br> * :param dataType: :class:`DataType` of the field.<br> * :param nullable: boolean, whether the field can be null (None) or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Il-yLwzzsEhTN-IerkExA.png" /></figure><ul><li><strong>Grabbing the data : </strong>I’m showing some simple dataframe grabbing data operations :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D75d3h7AliHxyfnCvesedg.png" /></figure><ul><li><strong>Creating new columns : </strong>In order to enrich and convert the columns of a dataframe, we will surely need to add new columnes :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*jwYctP3ehOQVSVbjH2HhEQ.png" /></figure><ul><li><strong>Using SQL :</strong> To use SQL queries directly with the dataframe, you will need to register it to a temporary view. With this view we can do all sql request like select, joins …. the request result will be dataframe</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*fxWPANVaA9Y0U5uriGEQ5w.png" /></figure><ul><li><strong>Basic Operations</strong></li></ul><p>This lecture will cover some basic operations with Spark DataFrames like :</p><p>→ Filtering Data: A large part of working with DataFrames is the ability to quickly filter out data based on conditions. Spark DataFrames are built on top of the Spark SQL platform, which means that is you already know SQL, you can quickly and easily grab that data using SQL commands, or using the DataFram methods (which is what we focus on in this course).</p><p><strong>b. Regular expressions:</strong></p><p>Regular expressions represent a toolbox that is extremely useful for retrieving information, manipulating and processing text such as code, log files, spreadsheets, or even documents. Python supports regular expressions through the standard python library <strong><em>re </em></strong>which is bundled with every Python installation. In this <a href="https://docs.python.org/2/library/re.html"><strong><em>link </em></strong></a>you’ll find the most important regular expression pattern. In the following I will explain the main elements of regulars expressions :</p><ul><li><strong>Matching a string</strong></li></ul><p>To test whether a regular expression matches a specific string in Python, you can use re.search(). This method either returns None if the pattern doesn’t match, or a re.MatchObject with additional information about which part of the string the match was found. Note that this method stops after the first match, so this is best suited for testing a regular expression more than extracting data. In this link we</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*fvu1LYrGoGqmcWr0e0_zLQ.png" /></figure><ul><li><strong>Capturing groups :</strong></li></ul><p>Inlike the re.search(), if you want to find all pattern occurrences, we can use re.findall() to perform a global search over the whole input string. If there are capture groups in the pattern, then it will return a list of all the captured data, but otherwise, it will just return a list of the matches themselves, or an empty list if no matches are found. If you need additional context for each match, you can use re.finditer() which instead returns an iterator of re.MatchObjects to walk through. Both methods take the same parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*t1DgoiJuIHRAk0fLm1kzJA.png" /></figure><ul><li><strong>Finding and replacing strings</strong></li></ul><p>Another common task is to find and replace a part of a string using regular expressions, for example, to replace all instances of an old email domain, or to swap the order of some text. You can do this in Python with the re.sub() method. The optional count argument is the exact number of replacements to make in the input string, and if this is value is less than or equal to zero, then every match in the string is replaced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*bCxa95sqQNUmrYjkxbb-TA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*b6Rk6gOnQ-KiRK_W4oRmBQ.png" /></figure><p><strong>c. Functions :</strong><br>There are a variety of functions you can import from pyspark.sql.functions. Check out the <a href="http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#module-pyspark.sql.functions"><strong><em>documentation</em></strong></a> for the full list available:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*b6Rk6gOnQ-KiRK_W4oRmBQ.png" /></figure><p>Most basic functions you would expect to be available are, so make sure to check out the documentation.</p><p>Nevertheless, if there is no pre-developed spark native function as needed, there is a concept that allows to instantiate a function using the concept of user defined function (UDF).</p><p>User-defined functions (UDFs) are a key feature of most SQL environments to extend the system’s built-in functionality. UDFs allow developers to enable new functions in higher level languages such as SQL by abstracting their lower level language implementations. Apache Spark is no exception, and offers a wide range of options for integrating UDFs with Spark SQL workflows. UDFs transform values from a single row within a table to produce a single corresponding output value per row. For example, most SQL environments provide an UPPER function returning an uppercase version of the string provided as input.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa494f87df58" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Supervised Deep Learning with keras, TensorFlow and Theano]]></title>
            <link>https://medium.com/@zaidi.houd/supervised-deep-learning-with-keras-tensorflow-and-theano-9dfd4fa17358?source=rss-3978cb98aacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9dfd4fa17358</guid>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[theano]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[ZAIDI Houda]]></dc:creator>
            <pubDate>Wed, 24 Jan 2018 14:50:55 GMT</pubDate>
            <atom:updated>2018-01-26T13:08:33.135Z</atom:updated>
            <content:encoded><![CDATA[<p>At the beginning of this article I would like to dwell on the definition of “deep learning”, what is this revolution to bring to traditional computing. In fact, before this revolution, learning something about a computer is like making a code. But this requires an explicit knowledge of the instructions to be given to this code. However, this information is not always easy to provide on certain complex subjects, for example, to recognize the content of an image or to understand the spoken language. The deep learning revolution lies here in learning through examples. For example, by showing the algorithm images with a cat, then others a dog, and by signaling each time the correct answer expected, the computer learns to recognize cats and dogs from a library of images. After a certain number of images the system starts to generalize, and knows how to recognize the animal on photos that he has never seen.</p><p>Before starting, let’s explain what’s the difference between deep learning and machine learning. In fact with machine learning, workloads are dominated by communication’s data and a lot of Input/Output. Then reduces the amount of communication is the first concern of a developer. But deep learning is typically done on a much smaller cluster. So it needs more computing power and less memory. It’s recommended using <strong>GPU </strong>(graphics processing units) accelerators to accelerate deep learning ‘training.’ When you have to make a choice between Machine Learning or Deep Learning, ask yourself whether you can use a high-performance GPU and have a large volume of labeled data. If you do not have any of these elements in your possession, it is probably better to use Machine Learning instead of Deep Learning.</p><p>The fields of application are vast and varied, we can list as examples: Automated Driving, Aerospace and Defense, Medical Research, Industrial Automation, Electronics, … Although the first theories concerning Deep Learning date back to the 1980s, it is only recently that it has become exploitable. The reasons are obvious:</p><ol><li>Deep Learning requires a <strong>large set</strong> of <strong>labeled data</strong>.</li><li>Deep Learning requires considerable <strong>computing power</strong>.</li></ol><p>With the emergence of new technologies related to <strong>cloud computing</strong>, <strong>dig data</strong> and <strong>distributed architecture</strong>, it has become possible to reduce the training time of deep learning networks to a few hours or less. There is three ways to use Deep Learning :</p><ol><li><strong>Training from scratch </strong>: you need to gather a very large amount of labeled data and design a network architecture that will learn the features and the model. It is a method suitable for new applications, or those that will obtain many categories of results. This is not a widespread approach because, because of the large volume of data and the pace of learning, network training can easily be spread over several days or even weeks.</li><li><strong>Transfer Learning</strong> : Most Deep Learning applications use Transfer Learning, or Transfer Learning, which involves developing a pre-trained model. The process starts with an existing network, such as AlexNet or GoogLeNet, that needs to be enriched with new data containing classes previously unknown to the network. Once you have made some adjustments to the network, you can perform a new task, such as categorizing dogs or cats only, instead of 1,000 different objects. This technique also has the advantage of requiring a much smaller amount of data (thousands of images must be processed, rather than millions), making calculation a matter of hours or minutes.</li><li><strong>Feature extraction</strong> : A less frequent and specialized approach to Deep Learning is to use the network as a feature extractor. Since all layers are responsible for learning certain characteristics from images, we can retrieve these characteristics from the network at any point in the training process. It is then possible to use these characteristics as input data for a Machine Learning model such as SVMs.</li></ol><p>In this tutorial we’ll talk about how to train a deep learning model from scratch. 2017 has seen an emergence of many works concerning the deep learning and distribution of these treatments in a distributed environment. Tensorflow and Pytorch are the two most popular open-source libraries for Deep Learning. In this tutorial i’ll show you how to use both of them ! Let’s introduce these two frameworks.</p><p><strong>TensorFlow </strong>was developed by Google and is used in their speech recognition system, in the new google photos product, gmail, google search and much more. Companies using Tensorflow include AirBnb, Airbus, Ebay, Intel, Uber and dozens more. <strong>Theano </strong>is another open source deep learning library. It’s very similar to Tensorflow in its functionality, but nevertheless we will still cover it. <strong>Keras </strong>is an incredible library to implement Deep Learning models. It acts as a wrapper for Theano and Tensorflow. Thanks to Keras we can create powerful and complex Deep Learning models with only a few lines of code. This is what will allow us to have a global vision of what you are creating. Everything we make will look so clear and structured thanks to this library, that you will really get the intuition and understanding of what we are doing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*dl-uHA0FYfJ4CI9AboldMw.png" /><figcaption>Keras based on theano and tensorflow</figcaption></figure><p>To install these tools :</p><pre>pip install theano</pre><pre>pip install tensorflow</pre><pre>pip install keras</pre><pre>conda update — all</pre><p>Like machine learning, deep learning can learn whith two ways :</p><ol><li><strong>supervised deep Learning</strong></li><li><strong>unsupervised deep learning</strong></li></ol><p>In this tutorial i’ll explain the supervised deep learning algorithms. I’ll keep the unsupervised deep learning tpo the next tutorial.</p><h3>Section I : supervised deep Learning :</h3><p>Most Deep Learning methods use neural network architectures, which is why <strong>deep neural networks</strong> are often used to designate models of <strong>Deep Learning</strong>. In this article I will explain the three types of neuron networks most used and most known:</p><ul><li><strong>Artificial Neural Networks</strong></li><li><strong>Convolutional Neural Networks</strong></li><li><strong>Recurrent Neural Networks</strong></li></ul><p>In the following of this section I will explain the intuition of each of these methods and show you how to make a training deep learning models with TensorFrames (TensorFlow on Spark Dataframes) library in spark cluster.</p><ol><li><strong>Artificial Neural Networks (ANN) :</strong></li></ol><p>The basic principles of ANN are detailed in this university <a href="https://www.sciencedirect.com/science/article/pii/S0731708599002721"><strong><em>article</em></strong></a>. The inventor of the first neurocomputer, Dr. Robert Hecht-Nielsen, defines a neural network as <em>“…a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs.”</em></p><p>The idea of ANNs is based on the belief that working of human brain, can be imitated using silicon and wires as living <strong>neurons</strong> and <strong>dendrites</strong>. The human brain is composed of 86 billion nerve cells called <strong>neurons.</strong> They are connected to other thousand cells by <strong>Axons.</strong> Stimuli from external environment or inputs from sensory organs are accepted by dendrites. These inputs create electric impulses, which quickly travel through the neural network. A neuron can then send the message to other neuron to handle the issue or does not send it forward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*t9BFGaZa6FCnHLQZsDrtAw.png" /><figcaption>Diagram of Neuron [Wikipedia]</figcaption></figure><p>ANNs are composed of multiple <strong>nodes</strong>, which imitate biological <strong>neurons</strong> of human brain. The neurons are connected by links and they interact with each other. The nodes can take input data and perform simple operations on the data. The result of these operations is passed to other neurons. The output at each node is called its <strong>activation</strong> or <strong>node value. </strong>Each link is associated with <strong>weight.</strong> ANNs are capable of learning, which takes place by altering weight values. The following illustration shows a simple ANN</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*cHi7IdNcPPHzatSlgmbCuw.png" /><figcaption>ANN principle</figcaption></figure><p>Backpropagation is a common method for training a neural network. The backpropogation is maded to update the ANN weight to minimise the cost function (the error between ANN output value and actual valu). To minimise this function, gradient Descent can be used but it request absolutly the cost function to be convexe (the function have only 1 global minimum). When the cost function is not convexe, the <strong><em>stochastic gradient </em></strong>descent could be the good shoose. The difference between the betch gradient descent and the stochastic gradient descent is very intuitive: For batch gradient descent we take all the data to calculate and minimase the cost function. On the other the stochastic gradient descent we take the data row by row.</p><p>The construction of ANN model is very easy with keras, the templat is as usualy :</p><ul><li>Stage 1 - data preprocessing : In this step, whole process of data preparing for model consumption is accomplished here. This involves importing data and librairies, cleaning data (missing, Outliers, Malicious, …), encoding categorical data, splitting the dataset into the Training set and Test set and Feature Scaling</li><li>Stage 2 : Ones the data is prepared, it’s time to make the ANN model. Let’s do it.</li></ul><p>The first step is to importing the Keras libraries and packages</p><pre>import keras<br>from keras.models import Sequential<br>from keras.layers import Dense</pre><p>Next step is to Initialising the ANN and adding the ANN layers</p><pre># Initialising the ANN<br>classifier = Sequential()</pre><pre># Input layer and the first hidden layer<br>classifier.add(Dense(units = 6, kernel_initializer = ‘uniform’, activation = ‘relu’, input_dim = 11))</pre><pre># Then Adding the second hidden layer<br>classifier.add(Dense(units = 6, kernel_initializer = ‘uniform’, activation = ‘relu’))</pre><pre># and the output layer<br>classifier.add(Dense(units = 6, kernel_initializer = ‘uniform’, activation = ‘relu’))</pre><p>The finally stepis to compiling the ANN and Fitting the ANN to the Training set</p><pre>classifier.compile(optimizer = ‘adam’, loss = ‘binary_crossentropy’, metrics = [‘accuracy’])<br>classifier.fit(X_train, y_train, batch_size = 10, epochs = 100)</pre><ul><li>Stage3 : the last stage is to making prediction with the model and to evaluate it</li></ul><pre># Predicting the Test set results<br>y_pred = classifier.predict(X_test)<br>y_pred = (y_pred &gt; 0.5)</pre><pre># Making the Confusion Matrix<br>from sklearn.metrics import confusion_matrix<br>cm = confusion_matrix(y_test, y_pred)</pre><p>2. <strong>Convolutional Neural Networks :</strong></p><p>Convolutional Neural Networks (<strong>CNNs</strong>) are a category of Neural Networks that are widely used in areas such as image recognition and classification. CNNs have proven very effective in identifying faces, objects and traffic signs apart from powering vision in robots and self driving cars.</p><p>CNNs are very similar to ordinary ANN (previously explain) : they are made up of neurons, based on the update of weights. Each neuron receives some inputs, performs a set of operation and give a result.</p><p>The CNNs architecture is illustrate in the follow image and composed from 4 concepts. Befor that we need to remember that every image can be represented as a matrix of pixel values. An image from a standard digital camera will have three channels — red, green and blue — you can imagine those as three 2d-matrices stacked over each other (one for each color), each having pixel values in the range 0 to 255. The next picture introduce CNNs’s stages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wA-NB2YRBIx-rbgdieSn4g.png" /><figcaption>CNNs steps</figcaption></figure><p><strong><em>i. Convolution :</em></strong> The Conv layer is the core building block of a CNNs Network that does most of the computational heavy lifting. The principal purpose of the convolution stage is to extract features from the image. It is useful to preserves the spatial relationship between pixels by learning image features using small squares of input data. I’ll not go to mathematical detail of convolution concept here but i’ll try to explain how it work. The objectif of convolution is to reduce the matrix size and in the same hand to preserve the spatial features by doing a conv product of this matrix by another 3 x 3 matrix. An additional operation called ReLU has been used after every Convolution operation. ReLU stands for Rectified Linear Unit and is a non-linear operation.</p><p><strong><em>ii. Pooling :</em></strong> Spatial Pooling (also called subsampling or downsampling) consist to reduces the dimensionality of each feature map but retains the most important information. Spatial Pooling can be of different types: Max, Average, Sum etc.</p><p><strong><em>iii. Flatened : </em></strong>consist to convert the matrix results of the previous layer to a vector. This last will be the input of the neural network ‘like ANN’ layer</p><p><strong><em>iv. Fully connected : </em></strong>The term “Fully Connected” implies that every neuron in the previous layer is connected to every neuron on the next layer. I recommend <a href="https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/">reading this post</a> if you are unfamiliar with Multi Layer Perceptrons.</p><p>To built a CNNs model, the first stage after the data preprocessing step (it’s a common stage between all the data models) is to built the CNNs architecture. by this steps</p><p>The first step is to import the Keras libraries and packages</p><pre>from keras.models import Sequential<br>from keras.layers import Conv2D<br>from keras.layers import MaxPooling2D<br>from keras.layers import Flatten<br>from keras.layers import Dense</pre><p>Next step is to Initialising the CNN and adding the layers :</p><pre># Initialising the CNN :<br>classifier = Sequential()</pre><p>Then built the CNNs architecture</p><pre># Step 1 — Convolution<br>classifier.add(Conv2D(32, (3, 3), input_shape = (64, 64, 3), activation = ‘relu’))</pre><pre># Step 2 — Pooling<br>classifier.add(MaxPooling2D(pool_size = (2, 2)))</pre><pre># Adding a second convolutional layer<br>classifier.add(Conv2D(32, (3, 3), activation = ‘relu’))<br>classifier.add(MaxPooling2D(pool_size = (2, 2)))</pre><pre># Step 3 — Flattening<br>classifier.add(Flatten())</pre><pre># Step 4 — Full connection<br>classifier.add(Dense(units = 128, activation = ‘relu’))<br>classifier.add(Dense(units = 1, activation = ‘sigmoid’))</pre><p>Now the CNNs model is ready to be fitted</p><pre># Compiling the CNN<br>classifier.compile(optimizer = ‘adam’, loss = ‘binary_crossentropy’, metrics = [‘accuracy’])</pre><pre># Fitting the CNN to the images</pre><pre>classifier.fit_generator(training_set,<br> steps_per_epoch = 8000,<br> epochs = 25,<br> validation_data = test_set,<br> validation_steps = 2000)</pre><p><strong>3. Recurrent Neural Networks (RNNs)</strong></p><p>RNNs are popular models that have shown great promise in many NLP tasks. RRNs are based on a simple idea that is to make use of sequential information. In a traditional neural network we assume that all inputs (and outputs) are independent of each other. But for many tasks (like a time series data) that’s a very bad idea. If you want to predict the next word in a sentence you better know which words came before it. RNNs are called <em>recurrent</em> because they perform the same task for every element of a sequence, with the output being depended on the previous computations. Another way to think about RNNs is that they have a “memory” which captures information about what has been calculated so far. In theory RNNs can make use of information in arbitrarily long sequences, but in practice they are limited to looking back only a few steps (more on this later).</p><p>The idea behind the RNNs is to consider a classic ANNs and we squashed them together. all the neurones still they but imagine that we see it through, to simplify the representation I chose to make a vertical presentation with a loop in blue to adjust the temporal dimension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r8VRXZQQCHCpuZnonHcI1g.png" /><figcaption>Idea behind RNNs</figcaption></figure><p>Training a RNN is similar to work with traditional ANN but with a little twise. We also use the backpropagation algorithm to update the parameters. As it’s explained in the folowing picture RNN parameters are shared by all time steps in the network, the gradient at each output depends not only on the calculations of the current time step, but also the previous time steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oWmnqSqo29jLE9kjZQgBaA.png" /><figcaption>Vanishing Gradient problem</figcaption></figure><p>For now I should mention that classic RNN have difficulties learning long-term dependencies (e.g. dependencies between steps that are far apart) due to what is called the vanishing gradient problem. Let’s then explain what’s this mean. The vanishing gradient problem was <a href="http://people.idsia.ch/~juergen/fundamentaldeeplearningproblem.html">originally discovered by Sepp Hochreiter in 1991</a> and has been receiving attention again recently due to the increased application of deep architectures.</p><p>Remember that our goal is to calculate the gradients of the error with respect to our parameters and then learn good parameters using Stochastic Gradient Descent. A simple differential drivation show that the gradient of the error at the instant <em>t </em>depend from the value of <strong>Wrec</strong>.<em> </em>The mathematical formula of this gradient is written in the previous figure. This explains the reason of occuration of the vanishing/exploding problem. In fact if we start learning with small <strong>Wrec </strong>coefficients Vanishing problem will occurate because the gradient will tend to zero very quickly and if we start learning with large <strong>Wrec </strong>the gradient will tend to infiny so exploding problem will occurate. More details for Vanishing problem are in this <a href="http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf"><strong><em>link</em></strong></a>.</p><p>A popular solution for this problem is to use Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) architectures. LSTMs were <a href="http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf">first proposed in 1997</a> and are the perhaps most widely used models in NLP today. GRUs, <a href="http://arxiv.org/pdf/1406.1078v3.pdf">first proposed in 2014</a>, are simplified versions of LSTMs. Both of these RNN architectures were explicitly designed to deal with vanishing gradients and efficiently learn long-range dependencies. In this tutorial i’ll explain the LSTMs.</p><p>LSTMs don’t have a fundamentally different architecture from RNNs, but they use a different function to compute the hidden state (x_t). The memory in LSTMs are called <em>cells</em> and you can think of them as black boxes that take as input the previous and the current state. Internally these cells decide what to keep in (and what to erase from) memory. They then combine the previous state, the current memory, and the input. It turns out that these types of units are very efficient at capturing long-term dependencies. This <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/"><strong><em>pos</em></strong></a><strong><em>t</em></strong><a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/"> is an excellent explanation</a> for LSMTs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r0hvVD3Ndxk7u4Ynoi7FPw.png" /><figcaption>LSTM architecture</figcaption></figure><p>Let’s now explain how to build the RNNs :</p><p>We start always by importing the needed libraries and packages</p><pre>from keras.models import Sequential<br>from keras.layers import Dense<br>from keras.layers import LSTM<br>from keras.layers import Dropout</pre><p>Then initialising the RNN :</p><pre>regressor = Sequential()</pre><p>Then build the LSTM architecture :</p><pre># Adding the first LSTM layer and some Dropout regularisation<br>regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))<br>regressor.add(Dropout(0.2))</pre><pre># Adding a second LSTM layer and some Dropout regularisation<br>regressor.add(LSTM(units = 50, return_sequences = True))<br>regressor.add(Dropout(0.2))</pre><pre># Adding a third LSTM layer and some Dropout regularisation<br>regressor.add(LSTM(units = 50, return_sequences = True))<br>regressor.add(Dropout(0.2))</pre><pre># Adding a fourth LSTM layer and some Dropout regularisation<br>regressor.add(LSTM(units = 50))<br>regressor.add(Dropout(0.2))</pre><pre># Adding the output layer<br>regressor.add(Dense(units = 1))</pre><p>Once the architecture is ready we will need to compile and fit the RNN model</p><pre># compile the RNN model<br>regressor.compile(optimizer = ‘adam’, loss = ‘mean_squared_error’)</pre><pre>#Finaly fitting the RNN to the training set<br>regressor.fit(X_train, y_train, epochs = 100, batch_size = 32)</pre><p>To summarize, in this article we reviewed the three main models of supervised deep learning (ANN, CNN, RNN) and a tutorial to explain how to build from scratch these three models based on keras and tensorflow &amp; theano. I hope this tutorial has been helpful and see you in the las section :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dfd4fa17358" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Smart data for helping in investment choice - Part II: Store data — cassandra]]></title>
            <link>https://medium.com/@zaidi.houd/smart-data-for-helping-in-investment-choice-part-ii-store-data-cassandra-d179fcc8ee7?source=rss-3978cb98aacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d179fcc8ee7</guid>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[scrapy]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[datastax]]></category>
            <dc:creator><![CDATA[ZAIDI Houda]]></dc:creator>
            <pubDate>Mon, 23 Oct 2017 19:28:40 GMT</pubDate>
            <atom:updated>2017-11-08T16:47:11.331Z</atom:updated>
            <content:encoded><![CDATA[<p>This is the second part of my project that I started several weeks ago. The objective of my project is to develop a predictive model that allows to estimate the price of a real estate or an automobile. The summary of this project is below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6H7Td7Xb4tTNiJ-A9EH3rQ.png" /></figure><ol><li><a href="https://medium.com/@zaidi.houd/smart-data-for-helping-in-investment-choice-part-i-collect-data-scrapping-55946ef7ad21"><strong>Part I: Collect data — scraping</strong></a></li><li><strong>Part II: Store data — cassandra</strong></li><li><strong>Part III: Cleaning &amp; exploration data</strong></li><li><strong>Part IV: Web App that use a simple model in Python to estimate the properties price depending on its location or condition.</strong></li><li><strong>Part V: Improving the model with additional features and a better algorithm</strong></li><li><strong>Part VI: Setup of a chatbot to chat with this App in natural language</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6H7Td7Xb4tTNiJ-A9EH3rQ.png" /></figure><p>The last part, available via <a href="https://medium.com/p/d179fcc8ee7/edit"><strong><em>this link</em></strong></a><strong><em>, </em></strong>we talk about the data collection with scraping ‘<a href="https://www.leboncoin.fr/"><strong><em>leboncoin.fr</em></strong></a>’ website. In this part this data will be store using the NoSQL database cassandra.</p><p>In a first time the main concepts of Cassandra, like data modeling, clusters, monitoring tool, query language, etc…, will be presented. Then we’ll see how this database is used as a spyder pipeline to avoid local data storage. In this way the scrapy python code will be used as a connector so that the scraped data will be stored directly in a Cassandra table.</p><ol><li><strong>Getting Started With Apache Cassandra</strong></li></ol><p>Apache Cassandra is highly scalable, distributed and high-performance NoSQL database, designed to handle a huge amount of data. Apache Cassandra is :</p><ul><li>Open source</li><li>NoSQL database technology</li><li>Distributed database technology</li><li>Bigdata technology which provides massive scalability</li><li>Commonly used to creat a database that is spread accross nodes in more than one data center, for high availability</li><li>Based on Amazon Dynamo and Google Big Table</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*z6H6Ij6GlDofL0D4z_wkJA.png" /></figure><p>The system requirements to Canssandra hands-on are :</p><ul><li>64-bit operating system</li><li>8 GB (or more) of RAM</li><li>30 GB (or more) of free hard drive space</li></ul><p>Download Cassandra from apache site <a href="http://cassandra.apache.org/"><strong><em>here</em></strong></a><strong><em>. </em></strong>Before you install assure that you have a java (jdk or jre and not openjdk) version more than 7. To check that use this command :</p><pre>$ java -version</pre><p>To install Cassandra, we must extract the cassandra apache file using:</p><pre>$ tar -xvzf apache-cassandra-x.x.x-bin.tar.gz</pre><p>with x.x.x is the Cassandra version and apache-cassandra-x.x.x-bin.tar.gz is the folder downloded from apache cassandra website. The main configuration of cassandra is on the folder conf situated on the Cassandra’s intallation folder. For example cassandra.yaml is one of the most important configuration file of cassandra, it contains the name of cluster, number of tokens, partitioner, nodes ip adresses … All this parameters will be studied later. Let’s now study the differents concept of Cassandra.</p><ul><li><strong>Write data: </strong>when a client requests to write data in a Cassandra node, the request is persisted on a commit log file on disk and then the data is written in a memory table called <em>memtable</em>. When the memtable is full, after reaching a pre-configured threshold, it is flushed to disk in an immutable structure called <em>SSTable</em>. Each table on Cassandra has a respective memtable and SSTable.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*2SLbBzLHhpqC-K_B.jpg" /></figure><ul><li><strong>Snitch :</strong> There is a way for the nodes in a cluster to know about the topology of the cluster named <strong>Snitch</strong>. Ways to define Snitch: Dynamic Snitching / SimpleSnitch / RackInferingSnitch / PropertyFileSnitch / EC2Snitch / EC2MultiRegionSnitch</li></ul><p>For example PropertyFileSnitch :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/1*iwRHk-566MJVidZk-jg5Fw.png" /></figure><ul><li><strong>Gossip :</strong> There is also a way used with cluster’s nodes to communicate with each others named <strong>Gossip</strong>. Every one second, each node communicate with up to three other nodes, exchanging information about itself and all the other nodes that it has information about. <strong>Gossip </strong>is the internal communication method for nodes in a cluster. For external communication, such as from an application to a cassandra database, <strong>CQL </strong>or <strong>Thrift </strong>are used.</li><li><strong>Partitionner: </strong>Data distribution is done throught consistent haching, to strive for even distribution of data across the nodes in a cluster. Rather than all of the rows of a table existing on only one node, the rows are distributed across the nodes in the cluster, in an attempt to evently spread out the load of the table’s data. For example, notice the dollowing rowss of data, to be inserted in a table within a Cassandra database (which is spread across the nodes of a cluster). To distribute the rows across the nodes, a <strong>partitioner </strong>is used. The partitioner uses an algorithm to determine which node a given row of data will go to. The default partitioner in Cassandra is <strong>Murmur3</strong>. Murmur3 takes the value in the first column of the row to generate a unique number between -2**63 and 2**63. Each node in a cluster is assigned one token range (or multiple ranges with virtual nodes). Calculate the token ranges like :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*diGCKjNKGoT937cXniZECQ.png" /></figure><p>or with a Murmur3 calculator like with <a href="https://www.geroba.com/cassandra/cassandra-token-calculator/"><strong><em>this site</em></strong></a><strong><em>.</em></strong> Once the hash of the identifier is calculated, this data will be assigned to the node whose hadh belongs to its range of tokens.</p><ul><li><strong>Replication factor:</strong> must specified whenever a database is defined. The replication factor specifies how many instances of the data there be within a given database. Although I can be specified, it is common to specify 2, 3, or more, so that if a node goes down, there is at least one other replica of the data, so that the data is not lost with the down node.</li><li><strong>Virtual nodes</strong>: There is another alternative way to assign token ranges to nodes’s cluster. This way is named <strong>Virtual nodes</strong> and are now the default in Cassandra. With virtual node, instead of a node being responsible for just one token range, it is instead responsible for many small token ranges (by default, 256 of them). They allow for assigning a high number of ranges to a powerful computer (e.g 512) and a lower number of ranges (e.g 128) to a less powerful computer. Virtuel nodes (aka vnodes) were created to make it easier to add new nodes to a cluster while keeping the cluster balanced. When a new node isadded, it receives many small token range slices from the existing nodes, to maintain a balanced cluster. With the old way, the best was to double the number of nodes, to maintain a balanced cluster without having to change the end-point values of the existing nodes.</li><li><strong>CQL (Cassandra Query Language):</strong> is a SQL-Like query language for communicating with Cassandra, created to make it easy for people familiar with SQL to work with Cassandra. Like SQL, CQL commands are not case-sensitive. Although CQL looks similar to SQL, it does not have all the same options as SQL, due to the distributed nature of Cassandra database. CQL offers a model close to SQL in the sense that data is put in tables containing rows of columns. The most popullar CQL operations like creating a table, creating a keyspace, inserting data are described in the Apache cassandra documentations <a href="http://cassandra.apache.org/doc/latest/cql/#"><strong><em>here</em></strong></a>.</li><li><strong>Thrifl: </strong>is a low-level API, currently still supported in Cassandra (support may be phased out in future release of Cassandra)</li><li><strong>JMX (Java Management Extensions):</strong> For administrative activities, such as cluster monitoring and management taskes, tools built on <strong>JMX </strong>(Java Management Extensions) are commonly used.</li><li><strong>Data Modeling:</strong> Before loading or inserting data into a Cassandra database, it is necessary to establish a model that best represents our dataset. When data medelling in Cassandra, it is important to understand the implications of working with a distributed database versus a relational one. The fact that j<strong>oins do not exist in Cassandra</strong> is perhaps the most significant data modeling difference between a distributed database versus a relational database. As a result, data modeling in a Cassandra database needs to be done in away so that all the data for a query is available in one table.</li><li><strong>Secondary indexe:</strong> The other difference of Cassandra is about the WHERE clause quesries. As Cassandra is a distributed database with the data organized by partition key, WHERE clause queries generally need to incluse a partition key. This mean that when we use the clause WHERE the variable used after should be a partition key. By creating an indes for columns beyoud the partitions and clustering clolumns, values in these other columns can be referenced in WHERE clauses. We call therse <strong>secondary indexes</strong>. For eaches secondary index, Cassandra creates a hidden table on each node in the cluster. It is important to know that, contrary to indexes in relational databases, creating secondary indexes does not increase the speed of queries in Cassandra. Secondary indexes simply make it so that WHERE clauses that reference values in column beyond the primary and clustering columns can run. For increasing the speed of a query, instead of defining a secondary index, you could create a table specifically for the query.</li><li><strong>update data:</strong> Note that the update command can be used to update data stored in cassandra and also that with this command we can update more than one item. When doing an update in Cassandra, unlike relational databases, there is no disk seek to find a record, update its data, and save it. Instead, an update is essentially another write. An update is simply a write to a memtable, to eventuelly get flushed to a new SSTable once the memtable is full.</li><li><strong>delete data:</strong> There are a number of CQL commands that can be used to delete data in Cassandra, including DELETE, TRUNCATE and DROP</li></ul><pre>DELETE can be used to delete a value in a column, or te delete a row or rows<br>TRUNCATE can be used to delete all of the rows in a table<br>DROP can be used to delete a table or a keyspace.</pre><ul><li><strong>Trombstone:</strong> When a delete is written to Cassandra, a trombstone is created, marking the data for deletion. The reason that data is not immediately deleted is so that there is time for all of nodes with a replica of the data (like any down nodes) to learn about the mark for deletion. If the deletion were happen immediately, while a replica node is down, it would be possible for the down node to come back up and replicate the data to the node that had previously deleted it, unaware of the delete. The minimum amount of time that must pass before a delete can occur (and its corresponding tombstone) is set in the <strong>gc_grace_seconds </strong>property for the table. By default, the value of gc_grace_seconds is 864000 (10 days). Once the gc_grace_seconds amount of time has passed, the data is eligible for deletion.</li><li><strong>Compaction:</strong> For the data to actually delete, <strong>compaction </strong>needs to happen. Compaction is when SSTables are combined, ti improve the performance of reads due to the fewer number of SSTables, and to reclaim disk from deleted data. Compaction is normally done automatically, although it can also be started, by using the nodetool compact command.</li><li><strong>TTL:</strong> TTL stands for Time To Live, and is a way to specify an expiration date for data that is being inserted. TTL values are marked with tombstones one they have expired. Then, once the gc_grace_period of time has passed, the data can be deleted from disk through compaction. A TTL value can be updated via the UPDATE command, withh the USING TTL clause.</li></ul><p>Now it will be interesting to present some useful Cassandra command to get started.</p><p>to start Cassandra, simply we do :</p><pre>$ cassandra -f</pre><p>To check the statue of cassandra’s nodes we use nodetool to have all the needed informations about cluster nodes :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jT3NiibRhrnJrRbookFUIw.png" /></figure><p>To know more about Cassandra I advise you the website of datastax academy present via this <a href="https://academy.datastax.com/"><strong><em>link</em></strong></a>.</p><p><strong>2. Cassandra connection with scrapy to store the data</strong></p><p>With the python project developed in the first part of this project and presented in my <a href="https://github.com/Zaidihouda/InvestmentDecisionHelp"><strong><em>github</em></strong></a>, we can configurate the pipeline of scrapy to store the data with cassandra. Before starting, we need to install python cassandra driver.</p><pre>$ pip install cassandra-driver # python cassandra driver</pre><p>We will create a KEYSPACE cassandra that we’ll use to store the scraped column family like :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p9SGSZG2cp4IO3zXGw0M2Q.png" /></figure><p>update the parameters associated with the pipeline model that we want to put in place</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*mVnHtBSCuBPpJds9bheGOQ.png" /></figure><p>Then what you have to do is develop the Cassandra-pipeline class in the pipeline.py file that will allow you to:</p><ul><li>open a connection to cassandra and</li><li>use the keyspace we have already created</li><li>create a family column cassandra</li><li>insert each item extract with scrapy in this table cassandra</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CTscDBMl94Cdm0Es5tcFcw.png" /></figure><p>The development set of this project are available on my github via this <a href="https://github.com/Zaidihouda/InvestmentDecisionHelp"><strong><em>link</em></strong></a>. In the next chapter of this project I will take care of the cleaning of this data with pyspark &amp; cassandra.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d179fcc8ee7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Smart data for helping in investment choice: Part I : Collect data — scrapping]]></title>
            <link>https://medium.com/@zaidi.houd/smart-data-for-helping-in-investment-choice-part-i-collect-data-scrapping-55946ef7ad21?source=rss-3978cb98aacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/55946ef7ad21</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[scrapy]]></category>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[ZAIDI Houda]]></dc:creator>
            <pubDate>Fri, 20 Oct 2017 12:53:08 GMT</pubDate>
            <atom:updated>2017-12-20T09:52:47.444Z</atom:updated>
            <content:encoded><![CDATA[<p>There is some time I’m looking for a personal project that allows me to improve my expertise along the data chain value. I wanted to develop a project encompassing different skills and tools from the end to end bigdata treatment chain.</p><p>I’ve seen all those people who want to buy / sell automobiles or properties and I thought I could do something fun and useful about it. My idea was to collect many properties and data from auto sales listings. In this way, I can provide many analyzes on this data. For example, I can give the estimate of the price of your car or your home, the price per square meter by region, offer some recommendations for investors.</p><p>I will divide this post in a series of articles, with the following plan (subject to change along the road):</p><ol><li><strong>Part I: Collect data — scraping</strong></li><li><strong>Part II: Store data — cassandra</strong></li><li><strong>Part III: Cleaning &amp; exploration data</strong></li><li><strong>Part IV: Web App that use a simple model in Python to estimate the properties price depending on its location or condition.</strong></li><li><strong>Part V: Improving the model with additional features and a better algorithm</strong></li><li><strong>Part VI: Setup of a chatbot to chat with this App in natural language</strong></li></ol><h3><strong>Part I: Collect data — scraping</strong></h3><p>Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting, etc.) is a technique for extracting large amounts of data from websites and save the extracted data to a local file or to a database.</p><p>Scrapping is a very interesting tool for parsing and retrieving data hibergated in a website. A small visit on the internet shows that python is a very powerful tool to do scrapping but also that there are two python packages allowing to do this very efficiently:</p><ol><li>BeautifulSoup</li><li>Scrapy</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*hENeekhZAc_I9mWw6fJeCA.png" /><figcaption>BeautifuSoupe vs Scrapy</figcaption></figure><p>I will talk about the features of Scrapy and BeautifulSoup, compare them, and decide which one is better for my project.</p><ol><li><strong>BeautifulSoup</strong></li></ol><p>BeautifulSoup is a tool which help programmer quickly extract valid data from web pages, its API is very friendly to newbie developer, and it can also handle malformed markup very well. The doc of BeautifulSoup is very comprehensive you can get a lot of examples there and quickly learn how to use it. BeautifulSoup works fine on Python 2 and Python 3, so compatibility will not be a problem.</p><p>We will use lxml, which is an extensive library for parsing XML and HTML documents very quickly; it can even handle messed up tags. However, in most cases, BeautifulSoup alone cannot get the job done, you need use another package such as requests to help you download the web page and then you can use BeautifulSoup to parse the HTML source code. I will try in this article to scrapp the “<a href="https://www.leboncoin.fr/">leboncoin.fr</a>” website to collect the sale ads of properties and vehicles.</p><p>The first step to scrap a webpage is to obtain the html contains of the website. Actualy, when we open a browser what will be happen? In fact the steps are: <br> USER → open google on browse → browser send request to google server → server receives request — process it → server returns response to browse → browse display the response server sent → google homepage opens. <br> Then the operations which happens are USER — -&gt; Request — -&gt; Server — -&gt; Response — -&gt; USER.<br> <br> Request is a library that allows us to query the server of a Web site and return the response to the user. To install requests use pip install requests. The most commonly use of requests are : POST and GET.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZjkeIbU6-p4aKdHMndeizg.png" /></figure><p>To display the content of the html page, we use :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L40BJ2lqmKzSXEfCCwyKLg.png" /></figure><p>The statue of the request response will be :</p><ul><li>1xx informational</li><li>2xx success</li><li>3xx Redirection</li><li>4xx Client Error</li><li>5xx Server Error</li></ul><p>More détails =&gt; see <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html"><strong><em>this article</em></strong></a></p><p>With request we can choose our prefer browser. This mean when we open your website we want to use for example google-chrome or firefox… For that we can use useragent to fix some header parameters. If we don’t indicate this parameter, the servers know that a robot or programme is sending this request and will return the response. But there is some site which have a very strict policy and don’t allow programme to request the servers and fair the request. To resolve that we can fake a useragent to show the server that we are basiqualy a browser and not juste a piece of code. Then let’s do that:</p><p>To install <strong><em>fake_useragent </em></strong>Python package, you can simply use pip:</p><p><strong><em>$ pip install fake-useragent</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VXIupXM_cLRwoiQ6U-z40Q.png" /></figure><p>As you can see above, we have downloaded an HTML document. Beautiful soup is a veru powerful python package to parse this document, and extract the text from the html tags. You can think of a page of a html as having a tree structure. We first have to import the library, and create an instance of the BeautifulSoup class to parse our document. We need before install bs4 and lxml Python package, you can simply use pip:</p><pre>$ pip install bs4<br>$ pip install lxml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IOMSaIua4skCier3kUURuw.png" /></figure><p>So BeautifulSoupe makes simple parsing and access to data on a website by some operations like :</p><ul><li>formatted html data using the prettify method</li><li>select the elements at the top level of the page</li><li>get the <em>p</em> tag by finding the children of the <em>body </em>tag</li><li>extract a single tag, we can instead use the <em>find_all </em>method</li><li>extract text or <em>href </em>from tags</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXvKuWdMukgNvyPkuR35mg.png" /></figure><p>To take a look at the different features <a href="https://github.com/Zaidihouda/InvestmentDecisionHelp/blob/master/Beautiful_Webscrapping_Notebook.ipynb"><strong><em>here </em></strong></a>is a tutorial for the webscrapping using beautifulsoupe and Requests. Although this tool is very rich but this is very limited for achievement of complex processing and exploration html data. In the next part of this section we’ll discuss scrapy which is based on xpath adress, which makes the parsing more fun</p><p><strong>2</strong>. <strong>Scrapy</strong></p><p>Scrapy is a Python package for large scale web data extraction. It represent a set of powerful tool s that we need to extract, process ans structured data from websites.</p><p>Scrapy installation can be made with simply using <strong>pip</strong> with the following command:</p><pre>$ pip install scrapy</pre><p>Now, you need to create a Scrapy project. In your Terminal/CMD navigate to the folder in which you want to save your project and then run the following command (project_scrapy is the name we want to offer to our project) :</p><pre>$ scrapy startproject project_scrapy</pre><p>In your Terminal, navigate to the folder of the Scrapy project we have created in the previous step. As we called it <strong><em>project_scrapy</em></strong>, the folder would be with the same name and the command should simply be:</p><pre>$ cd project_scrapy</pre><p>After that, create the spider using the genspider command and give it any name you like; here we will call it <strong><em>boncoinimmo</em></strong>. Then, it should be followed by the URL.</p><pre>$ scrapy genspider boncoinimmo <a href="https://www.leboncoin.fr/ventes_immobilieres/offres/">https://www.leboncoin.fr/ventes_immobilieres/offres/</a></pre><p>Now, manually navigate to the scrapy project. You can see that it is structured as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*PWBcC7DBV58KsXQRUIqaww.png" /></figure><p>Let’s check the parts of the main class of the file automatically generated for our <strong><em>boncoinimmo </em></strong>Scrapy spider:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*VhWDH7c8pynrPiA4rt2Huw.png" /></figure><ol><li>name of the spider.</li><li>allowed_domains the list of the domains that the spider is allowed scrape.</li><li>start_urls the list of one or more URL(s) with which the spider starts crawling.</li><li>parse the main function of the spider. Do NOT change its name; however, you may add extra functions if needed.</li></ol><p>To parse our site it remains to develop this function (parse), instead of pass, to the parse() function. Now we need to focus on how to navigate the website. To find an HTML element you need to use the Chrome developer tools. Right click on the website and select “Inspect”. This will open a box on the right side of the Chrome browser. Then click on the inspect icon (highlighted in blue).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3AuuBKKwE3mpOcYGaTdAFw.png" /></figure><p>Next use the inspector cursor to click on a section of the website that you want to control. When you have clicked, the HTML that creates that section will be highlighted on the right. In the photo below, I have clicked on the bar of all ads by regions located by region all over france. Next right click on the HTML element, and select under “Copy” -&gt; “Copy XPath”. Then if i want to extract the information of the furst ad i search the tag that contain the information in the html elements ( at the bottom of the previous page). if you select an element at the bottom of the page, its color changes to blue (like the next print screen. The next step is to make an xpath copy like the next print Screen. That XPath is the most important piece of information!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zq1Xk9uTV849x-ZGaBthfA.png" /></figure><p>To return this information we just use this command</p><pre>$ <em>ad= response.xpath(search_xpath).extract()</em></pre><p>Then we can print this information or process it. <strong><em>search_xpath </em></strong>is the extracted google xpath. <strong><em>response </em></strong>is simply the whole html source code retrieved from the page. Actually, “response” has a deeper meaning because ; however (response.body) will get the whole source code. Anyhow, when you use XPath expressions to extract HTML nodes, you should directly use response.xpath(). You can use this command to run your spider and store the scraped data to a CSV file.</p><pre>$ scrapy crawl boncoinimmo -o result_one.csv</pre><p>As we agreed, you first need to scrape all the wrappers from the page. So under the parse() function, write the following. Note thate selector is a powerful API that can be used for quickly selecting nested data :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*f4bxdVpCmhIHje86MmzNFA.png" /></figure><p>Note that here you will not use extract() because it is the wrapper from which you will extract other HTML nodes. You can now extract the ads information from the wrappers using a for loop. You can extract the ad address and URL from the wrappers using the same for loop as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*v0yVjoz2RKGa-Yb5tcdPPQ.png" /></figure><p>The function parse_item will return the ad element from the webpage. Then we notice that we need to browse all the pages for that we have to make a recursion to the same function for the next page as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*8ckaN9w0i5yGS6QMOdJfDA.png" /></figure><p>If you are using a regular browser to navigate a website, your web browser will send what is known as a “User Agent” for every page you access. So it is recommended to use the Scrapy <strong><em>USER_AGENT</em></strong> option while web scraping to make it look more natural. The option is already in Scrapy <em>settings.py</em> so you can enable it by deleting the # sign. You can find lists of user agents online; select a recent one for <a href="http://www.useragentstring.com/pages/useragentstring.php?name=Chrome">Chrome</a> or <a href="http://www.useragentstring.com/pages/useragentstring.php?name=Firefox">Firefox</a>.</p><p>The option DOWNLOAD_DELAY is already there in Scrapy settings.py so you can just enable it by deleting the # sign. According to Scrapy documentation, “this can be used to throttle the crawling speed to avoid hitting servers too hard.” You can see that the offered number is 3 seconds; however, you can make them less or more. Still, this makes sense because there is another option that is activated by default which is RANDOMIZE_DOWNLOAD_DELAY and it is set from 0.5 to 1.5 seconds.</p><p>The final code is present in my github <a href="https://github.com/Zaidihouda/InvestmentDecisionHelp"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><p>If everything works you now have a lot of data to handle. We will firstly store this tada with cassandra NoSQL database. The tutorial will be available in my next post — the Part II !</p><p>Thank you very much for reading, and do not hesitate to post anything in order to improve this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55946ef7ad21" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>