<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by BigData Republic on Medium]]></title>
        <description><![CDATA[Stories by BigData Republic on Medium]]></description>
        <link>https://medium.com/@admin_10849?source=rss-12eb83dfb55b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*C8a3XXK0p1XwyNB99ut-uw.png</url>
            <title>Stories by BigData Republic on Medium</title>
            <link>https://medium.com/@admin_10849?source=rss-12eb83dfb55b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:10:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@admin_10849/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[RE-WORK Deep Learning Summit London]]></title>
            <link>https://medium.com/bigdatarepublic/re-work-deep-learning-summit-london-c2ca8abfda68?source=rss-12eb83dfb55b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2ca8abfda68</guid>
            <dc:creator><![CDATA[BigData Republic]]></dc:creator>
            <pubDate>Thu, 12 Oct 2017 09:01:44 GMT</pubDate>
            <atom:updated>2018-03-06T08:10:09.564Z</atom:updated>
            <content:encoded><![CDATA[<p>On the 21th and 22nd of September we attended the RE-WORK Deep Learning Summit in London. This event was focused on the development of deep learning techniques and business applications based on deep learning. Facebook and Amazon presented their Natural Language Processing research using neural networks, which was a recurring theme in the summit. Bayesian Neural Networks and Generative models were other main topics discussed at the summit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pmomLtzyxc4Nsx7O.png" /></figure><p>We saw many interesting start-ups based on innovative applications using deep learning and also some bigger companies with specialized deep learning teams. An overview of some of these companies:</p><ul><li>DigitalGlobe: analyzes satellite images for various applications. For instance, classifying swimming pools for a cleaning company to determine the best settlement, or determining a flooded area for an insurance company.</li><li>Alpha-i: Provides a Time Series API which can do forecasting, failure prevention, risk monitoring and anomaly detection on presented data. They use Bayesian Neural Networks of which the certainty of a prediction, for instance predicting stock-market prizes, can be very valuable.</li><li>Echobox: Generate and post social media messages. Already many companies (mostly news websites, for example “The Guardian”, “Le Monde” and “NewScientist”) use this system. It provides the text, selects an optimal picture and posts the message at the best time of the day. Echobox also has a side project which analyzes trends in social media and use it to predict elections.</li><li>Jukedeck: an AI that can compose original music. They provided examples of electronic music and even created a Bach-like composition.</li><li>Uizard.io: Automatic front-end development. The model classifies an image of a front-end layout and builds working code for it.</li><li>Deep Instinct: Prevent cyber-attacks using deep learning. Many anti-virus protectors cannot detect new malware, however many viruses are variations of existing malware with minor modifications. Their model is able to detect these variations with great accuracy.</li></ul><p>The variation of use-cases for deep learning is huge. But how is it any different from using other machine learning tools, such as random forests or generalized linear models? The difference is that deep learning works extremely well on unstructured media, such as images, video and sound. Using deep learning we can build learned feature extractors instead of handcrafting a set of features from our input. This requires less work, is more robust for future new data and requires less domain knowledge.</p><p>At BigData Republic we have several experts in the deep learning field with many scientific publications. We have worked on various deep learning projects amongst which one was a ProRail case where we built an OCR system to analyze train wagon labels from video footage. Please feel free to contact us if you want to know more about deep learning or have questions about processing your unstructured datasets!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2ca8abfda68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigdatarepublic/re-work-deep-learning-summit-london-c2ca8abfda68">RE-WORK Deep Learning Summit London</a> was originally published in <a href="https://medium.com/bigdatarepublic">bigdatarepublic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine learning for predictive maintenance: where to start?]]></title>
            <link>https://medium.com/bigdatarepublic/machine-learning-for-predictive-maintenance-where-to-start-5f3b7586acfb?source=rss-12eb83dfb55b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f3b7586acfb</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[BigData Republic]]></dc:creator>
            <pubDate>Tue, 29 Aug 2017 10:55:29 GMT</pubDate>
            <atom:updated>2018-03-06T08:10:08.893Z</atom:updated>
            <content:encoded><![CDATA[<p>Think about all the machines you use during a year, all of them, from a toaster every morning to an airplane every summer holiday. Now imagine that, from now on, one of them would fail every day. What impact would that have? The truth is that we are surrounded by machines that make our life easier, but we also get more and more dependent on them. Therefore, the quality of a machine is not only based on how useful and efficient it is, but also on how reliable it is. And together with reliability comes maintenance.</p><p>When the impact of a failure cannot be afforded, such as a malfunctioning airplane engine for example, the machine is subjected to preventive maintenance, which involves periodic inspection and repair, often scheduled based on time-in-service. The challenge of proper scheduling grows with the complexity of machines: in a system with many components working together and influencing each other’s lifetime, how can we find the right moment when maintenance should be performed so that components are not prematurely replaced but the whole system still stays functioning reliably? Providing an answer to this question is the aim of predictive maintenance, where we seek to build models that quantify the risk of failure for a machine in any moment in time and use this information to improve scheduling of maintenance.</p><p>The success of predictive maintenance models depend on three main components: having the right data available, framing the problem appropriately and evaluating the predictions properly.</p><p><strong>In this post, we will elaborate the first two points and give insights on how to choose the modelling technique that best fits the question you are trying to answer and the data you have at hand.</strong></p><h4>DATA COLLECTION</h4><p>To build a failure model, we require enough historical data that allows us to capture information about events leading to failure. In addition to that, general “static” features of the system can also provide valuable information, such as mechanical properties, average usage and operating conditions. However, more data is not always better. When collecting data to support a failure model, it is important to make an inventory the following:</p><ul><li>What are the types of failure that can occur? Which ones will we try to predict?</li><li>How does the “failure process” look like? Is it a slow degradation process or an acute one?</li><li>Which parts of the machine/system could be related to each type of failure? What can be measured about each of them that reflect their state? How often and with which accuracy do these measurements need to be performed?</li></ul><p>The life span of machines is usually in the order of years, which means that data has to be collected for an extended period of time in order to observe the system throughout its degradation process.</p><p>In an ideal scenario both data scientists and domain experts would be involved in the data collection plan to ensure that the data gathered is suitable for the model to be built. However, what mostly happens in real life is that the data has already been collected before the data scientist arrives and he/she must try to make the best of what is available.</p><p>Depending on the characteristics of the system and on the data available, a proper framing of the model to be built is essential: which question do we want the model to answer and is it possible with the data we have at hand?</p><h4>PROBLEM FRAMING</h4><p>When thinking about how to frame a predictive maintenance model, it is important to keep a couple of questions in mind:</p><ul><li>What kind of output should the model give?</li><li>Is enough historical data available or just static data?</li><li>Is every recorded event labelled, i.e. which measurements correspond to good functioning and which ones correspond to failure? Or at least, is it known when each machine failed (if at all)?</li><li>When labelled events are available, what is the proportion of the number of events of each type of failure and events of well functioning?</li><li>How long in advance should the model be able to indicate that a failure will occur?</li><li>What are the performance targets that the model should be optimized for? High precision, high sensitivity/recall, high accuracy? What is the consequence of not predicting a failure or predicting a failure that will not happen?</li></ul><p>With all this information at hand, we can now decide which modelling strategy fits best to the available data and the desired output, or at least which one is the best candidate to start with. There are multiple modelling strategies for predictive maintenance and we will describe four of them in relation to the question they aim to answer and which kind of data they require:</p><ol><li>Regression models to predict remaining useful lifetime (RUL)</li><li>Classification models to predict failure within a given time window</li><li>Flagging anomalous behaviour</li><li>Survival models for the prediction of failure probability over time</li></ol><h4>STRATEGY 1: Regression models to predict remaining useful lifetime (RUL)</h4><p>OUTPUT: How many days/cycles are left before the system fails?</p><p>DATA CHARACTERISTICS: Static and historical data are available, and every event is labelled. Several events of each type of failure are present in the dataset.</p><p>BASIC ASSUMPTIONS/REQUIREMENTS:</p><ul><li>Based on static characteristics of the system and on how it behaves now, the remaining useful time can be predicted, which implies that both static and historical data are required and that the degradation process is smooth.</li><li>Just one type of “path to failure” is being modelled: if many types of failure are possible and the system’s behaviour preceding each one of them differs, one dedicated model should be made for each of them.</li><li>Labelled data is available and measurements were taken at different moments during the system’s lifetime.</li></ul><h4><strong>STRATEGY 2:</strong> Classification models to predict failure within a given time window</h4><p>Creating a model which can predict lifetimes very accurate can be very challenging. In practice however, one usually does not need to predict the lifetime very accurate far in the future. Often the maintenance team only needs to know if the machine will fail ‘soon’. This results in the next strategy:</p><p>QUESTION: Will a machine fail in the next N days/cycles?</p><p>DATA CHARACTERISTICS: Same as for strategy 1</p><p>BASIC ASSUMPTIONS/REQUIREMENTS: The assumptions of a classification model are very similar to those of regression models. They mostly differ on:</p><ul><li>Since we are defining a failure in a time window instead of an exact time, the requirement of smoothness of the degradation process is relaxed.</li><li>Classification models can deal with multiple types of failure, as long as they are framed as a multi-class problem, e.g.: class = 0 corresponding to no failure in the next n days, class = 1 for failure type 1 in the next n days, class = 2 for failure type 2 in the next n days and so forth.</li><li>Labelled data is available and there are “enough” cases of each type of failure to train and evaluate the model.</li></ul><p>In general, what regression and classification models are doing is modelling the relationship between features and the degradation path of the system. That means that if the model is applied to a system that will exhibit a different type of failure not present in the training data, the model will fail to predict it.</p><h4><strong>STRATEGY 3:</strong> Flagging anomalous behaviour</h4><p>Both previous strategies require a lot of examples of both normal behaviour (of which we often have a lot of) and examples of failures. However, how many planes will you let crash to collect data? If you have mission critical systems, in which acute repairs are difficult, there are often only limited, or no examples of failures at all. In this case, a different strategy is necessary:</p><p>QUESTION: Is the behaviour shown normal?</p><p>DATA CHARACTERISTICS: Static and historical data are available, but either labels are unknown or too few failure events were observed or there are too many types of failure</p><p>BASIC ASSUMPTIONS/REQUIREMENTS: It is possible to define what normal behaviour is and the difference between current and “normal” behaviour is related to degradation leading to failure.</p><p>The generality of an anomaly detection model is both its biggest advantage and pitfall: the model should be able to flag every type of failure, despite of not having any previous knowledge about them. Anomalous behaviour, however, does not necessarily lead to failure. And if it does, the model does not give information about the time span it should occur.</p><p>The evaluation of an anomaly detection model is also challenging due to the lack of labelled data. If at least some labelled data of failure events is available, it can and should be used for evaluating the algorithm. When no labelled data is available, the model is usually made available and domain experts provide feedback on the quality of its anomaly flagging ability.</p><h4><strong>STRATEGY 4: </strong>Survival models for the prediction of failure probability over time</h4><p>The previous three approaches focus on prediction, giving you enough information to apply maintenance before failure. If you however are interested in the degradation process itself and the resulting failure probability, this last strategy suits you best.</p><p>QUESTION: Given a set of characteristics, how does the risk of failure change in time?</p><p>DATA CHARACTERISTICS: Static data available, information on the reported failure time of each machine or recorded date of when a given machine became unobservable for failure.</p><p>A survival model estimates the probability of failure for a given type of machine given static features and is also useful to analyse the impact of certain features on lifetime. It provides, therefore, estimates for a group of machines of similar characteristics. Therefore, for a specific machine under investigation it does not take its specific current status into account.</p><h4>Bottom line:</h4><p>What is the most suitable approach for a predictive maintenance model? As for all other data science problems, there is no free lunch! The advice here is to start by understanding which types of failure you are trying to model, which type of output you would like the model to give and which kind of data is available. Having put all this put together with the advice given above, I hope you now know from where to start!</p><h4>Some useful links:</h4><ul><li>Survival analysis in scikit-learn: <a href="https://github.com/sebp/scikit-survival">https://github.com/sebp/scikit-survival</a></li><li>Imbalanced classes: <a href="https://svds.com/learning-imbalanced-classes/">https://svds.com/learning-imbalanced-classes/</a></li><li>Novelty and outlier detection on scikit-learn: <a href="http://scikit-learn.org/stable/modules/outlier_detection.html">http://scikit-learn.org/stable/modules/outlier_detection.html</a></li></ul><p><a href="https://bigdatarepublic.nl/"><em>BigData Republic</em></a><em> provides these type of Big Data Solutions. We are experienced in advanced predictive modeling and deploying large-scale big data pipelines. Interested in what we can do for you? Feel free to </em><a href="https://www.bigdatarepublic.nl/about-us#contact-us"><em>contact us</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f3b7586acfb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigdatarepublic/machine-learning-for-predictive-maintenance-where-to-start-5f3b7586acfb">Machine learning for predictive maintenance: where to start?</a> was originally published in <a href="https://medium.com/bigdatarepublic">bigdatarepublic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Genetic Algorithms in Practice]]></title>
            <link>https://medium.com/bigdatarepublic/genetic-algorithms-in-practice-63bcdc552fbf?source=rss-12eb83dfb55b------2</link>
            <guid isPermaLink="false">https://medium.com/p/63bcdc552fbf</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[evolutionary-computing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[BigData Republic]]></dc:creator>
            <pubDate>Fri, 30 Jun 2017 10:15:40 GMT</pubDate>
            <atom:updated>2018-03-06T08:09:58.700Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/234/0*OcrRjPJ0C7IrMHFC.jpg" /></figure><p>A Genetic Algorithm (GA) is part of a family of algorithms known as Evolutionary Computing. These algorithms are all inspired by biological evolution, used for various optimizations. In this post we discuss what type of projects used Genetic Algorithms and provide a step-by-step example.</p><p>The basics of a GA are as follows: candidate solutions to a problem are randomly initialized and become part of the first population. Each individual (candidate solution) receives a fitness score, based on how well it solves the problem. A selection step follows, for which the fittest individuals have a better chance of survival. Via crossover and mutations new individuals are created and form the new population. This step can be repeated numerous times, until an individual is found that meets the acceptation criteria.</p><p>As an example, NASA created a space antenna using a GA, which had to meet a difficult set of requirements. They claimed that: “The current practice of designing and optimizing antennas by hand is limited in its ability to develop new and better antenna designs because it requires signi cant domain expertise and is both time and labor intensive”. The individuals (random antennas) were represented as a list of operations, such as <em>forward(length, radius) </em>or <em>rotate-x(angle). </em>Both the operations and their parameters were mutatable. The fitness score was mostly a combination of wide beamwidth for a circularly polarized wave and wide impedance bandwidth.</p><p>To illustrate in depth the working of a Genetic Algorithm, we use a simpler case than NASA’s antenna: The knapsack problem. For a given a set of items, that each have a certain weight and value, the total value of items that fit in the fixed size knapsack has to be optimized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*Um3SJ8TMyxZSRZjY.png" /></figure><p>The optimization of this problem is NP-hard, meaning that there is no known algorithm that determines whether a solution is optimal in polynomial time. Genetic algorithms typically tackle NP-hard problems and approximate the optimal solution.</p><p>The first step in creating a GA is representing the problem in a way that is suitable for applying genetic operators. For the knapsack problem it is rather easy, the knapsack is represented as a flexible sized array with item identities. The fitness score is the summed value of the items in the backpack. If a backpack exceeds the carrying limit, its fitness score is 0.</p><p>Step 2 is creating an initial population. <em>N </em>backpacks will be filled with random items until the weight limit is reached.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Z2jzvRhZ6bptfCGF.png" /></figure><p>Step 3 is determining the fitness score and transition to the next population. Two individuals are chosen pseudo-randomly (higher fitness score = higher probability). The first applied genetic operator is crossover, which relates to giving birth. With the combined genes of the parents a new individual is formed. It is difficult however to determine what parts of the parents are used, since there are two constraints that need to be satisfied: the backpack cannot contain duplicate items and it should not exceed the weight limit. The fact that we stated “should” instead of “cannot” is because it is a choice whether new individuals can be “falsely” born into the world, in which case new random individuals will take their place in the new population, similar to the first population.</p><p>The second operator is mutation. Mutation can be broken down to three operators of which one is randomly chosen to apply: adding an (valid) item, removing an item or replacing an item.</p><p>After the creation of a new population the steps repeat itself. For the knapsack problem a single iteration computes in only a few milliseconds, allowing the number of repetitions to be enormous and therefore guaranteeing an approximation to the optimal solution.</p><p>Recent developments in deep learning and especially Convolutional Neural Networks (CNN) have caused an increased interest for Evolutionary Computing. Google wrote a <a href="https://research.googleblog.com/2017/05/using-machine-learning-to-explore.html">blogpost</a> showing their research in this field. The “architecture” of a CNN is the specification of the layers and their parameters. The search space of the architectures is immense, a 10 layer network can have up to ~1010 variations. Many researchers study new types of architectures and the optimization of architectures, however there is no known architecture that is proven to be optimal. Evolutionary Computing can optimize these architectures by exploring the amount, types and order of the layers. The parameters of the layers and the global parameters of the architecture, such as learning rate and learning function can also be optimized. An issue is the computing power required to optimize these architectures. The fitness score of an architecture is determined by training and evaluating on a certain task, which for a typical GA has to be repeated thousands or millions of times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ePIZ2-4gqGeoeC9o.png" /></figure><p>To summarize: Evolutionary Computing uses techniques derived from natural processes and optimizes problems that have no clear optimal solution. It has been applied in various projects and has recently made an uprising in deep learning research, of which we will definitely see more of in the upcoming years.</p><p>Students from the Radboud University used an Evolutionary Algorithm to form a target image using small pieces of a base image. A live demo can be found <a href="http://tanjacrijns.com/naturalcomputing/">here</a>!</p><p><a href="https://bigdatarepublic.nl/"><em>BigData Republic</em></a><em> provides these type of Big Data Solutions. We are experienced in advanced predictive modeling and deploying large-scale big data pipelines. Interested in what we can do for you? Feel free to </em><a href="https://www.bigdatarepublic.nl/about-us#contact-us"><em>contact us</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63bcdc552fbf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigdatarepublic/genetic-algorithms-in-practice-63bcdc552fbf">Genetic Algorithms in Practice</a> was originally published in <a href="https://medium.com/bigdatarepublic">bigdatarepublic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Key Topics of Spark Summit Europe 2016]]></title>
            <link>https://medium.com/bigdatarepublic/5-key-topics-of-spark-summit-europe-2016-51ef8bf14b6b?source=rss-12eb83dfb55b------2</link>
            <guid isPermaLink="false">https://medium.com/p/51ef8bf14b6b</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[BigData Republic]]></dc:creator>
            <pubDate>Wed, 02 Nov 2016 07:00:56 GMT</pubDate>
            <atom:updated>2018-03-06T08:09:54.598Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the tools <a href="https://www.bigdatarepublic.nl/">BigData Republic</a> uses for its large scale data crunching is <a href="https://spark.apache.org/">Apache Spark</a>. While Spark at first was mainly focused on processing large amounts of data in an efficient way, it is continuing to increase support for major machine learning algorithms. This way, it is becoming the defacto standard for data science and big data engineering.</p><p>Together with my colleagues Gerben Oostra and Alexander Backus, I attended the Spark Summit in Brussels last week. In this blog post, I will discuss the (in my opinion) 5 main topics to keep you updated with the state-of-the-art developments.<br><br>Note that all of the talks from this Spark Summit will be freely available on the <a href="https://spark-summit.org/eu-2016/schedule/">official site</a>, expected the 4th of November.</p><p><strong>1. Deep Learning support</strong></p><p>This is one of the coolest additions for Spark. Using <a href="https://github.com/databricks/tensorframes">Tensorframes</a>, which are basically a wrapper around the <a href="https://www.tensorflow.org/">TensorFlow framework</a>, it is possible to utilize the GPUs that might be available on worker nodes for the purpose of training deep neural networks. In fact, all machine learning algorithms implemented in TensorFlow should become available this way.</p><p>The idea is to utilize Spark’s distributed features to (horizontally) distribute relevant input parameters over the worker nodes, and then use TensorFlow to (vertically) optimise computations for specific node hardware including GPUs. This is especially handy if you want to run compute intensive parameter tuning jobs for your machine learning model.</p><p>One of the key features that enables data passing between TensorFlow and Spark efficiently is the new Spark columnar in-memory storage format. This helps porting between TensorFlow as well as other machine learning frameworks who use vectorized (column-wise) data structures to operate on, as opposed to Spark’s default row-based format.</p><figure><img alt="Deep Learning on Spark with TensorFrames" src="https://cdn-images-1.medium.com/max/300/0*Z7A-tOdfzXV5XJ8R.jpg" /><figcaption>Deep Learning on Spark with TensorFrames</figcaption></figure><p>Related recommended talks to watch:<a href="https://spark-summit.org/eu-2016/events/tensorframes-deep-learning-with-tensorflow-on-apache-spark/"> TensorFrames: Deep Learning with TensorFlow on Apache Spark</a></p><p><strong>2. Spark Performance improvements</strong></p><p>Since Spark was first released many new features have been introduced, but run-time performance optimizations were not yet touched upon. Most of the code responsible for the actual execution of computations in a Spark job consisted of many virtual function calls (a by-product of Object Oriented programming in Scala). Many standard operations in Spark code are highly generic, but therefore hard to optimize for the Scala compiler and at run-time by the CPU (e.g. branch-prediction is hard). To solve these problems, Tungsten was developed. This is a whole-stage at run-time code generation component that first inspects the whole Spark job end-to-end, then generates code that is optimised for the particular job instead of the job being highly generic.</p><figure><img alt="Tungsten CodeGen Spark 2.0" src="https://cdn-images-1.medium.com/max/300/0*6Cd6WUNsPg6CccrF.jpg" /><figcaption>Tungsten CodeGen Spark 2.0</figcaption></figure><p>Specifically for Spark SQL, Spark 2.0 also comes with improvements to the component that selects which operations are actually selected to be part of the job. This is the Catalyst SQL query plan optimizer, which defines strategies on which operations to use. For example, for joining two tables, multiple types of JOIN operations could be used ranging from a map-side join using broadcasts to fully distributed cartesian product joins. Tungsten and Catalyst are complementary components. Catalyst first optimises the query plan for a job, then Tungsten generates highly optimized code for each of the operations inside that job.</p><figure><img alt="Catalyst Optimizer Spark 2.0" src="https://cdn-images-1.medium.com/max/300/0*ihoA0y73zEIJTw2O.jpg" /><figcaption>Catalyst Optimizer Spark 2.0</figcaption></figure><p>Related recommended talks to watch: <a href="https://spark-summit.org/eu-2016/events/sparks-performance-the-past-present-and-future/">Spark’s Performance: The Past, Present, and Future</a> (Tungsten), <a href="https://spark-summit.org/eu-2016/events/a-deep-dive-into-the-catalyst-optimizer/">A Deep Dive into the Catalyst Optimizer</a> (Catalyst)</p><p><strong>3. Productionizing your model</strong></p><p>As more companies start implementing data science for their business, they realise they need to integrate their machine learning models with their existing workflows and infrastructure to get a return on their investment. However, most of the models data scientists create are not suitable to be put into production directly. Using software libraries like MLeap or H2O allows data scientists to export their machine learning models to a standardised format that can be easily integrated in (existing) big data pipelines. In addition the companies backing these libraries try to provide users with a platform that makes deployment to production easier, even going as far as providing a one-click production deployment solution.</p><figure><img alt="MLeap Vision" src="https://cdn-images-1.medium.com/max/300/0*vey8ZMbeulKWnj6k.jpg" /><figcaption>MLeap Vision</figcaption></figure><p>Related recommended talks to watch: <a href="https://spark-summit.org/eu-2016/events/mleap-and-combustml-deploying-machine-learning-models-to-production/">MLeap and Combust.ML: Deploying Machine Learning Models to Production</a> (MLeap), <a href="https://spark-summit.org/eu-2016/events/sparkling-water-20-the-next-generation-of-machine-learning-on-apache-spark/">Sparkling Water 2.0: The Next Generation of Machine Learning on Apache Spark</a> (H2O)</p><p><strong>4. Online machine learning with structured streaming</strong></p><p>Sadly I was not fast enough to make a screen for this topic. But nevertheless this is one of the nicest talks I have attended at the Spark Summit. Most of the machine learning libraries are focused on training models in batch. This limits models that are already implemented in real-time big data processing pipelines, because they cannot adapt to fresh data in near real-time.</p><p>However, more effort is put in to make online training possible using Spark Streaming. This means a model can be updated in near real-time whenever the big data pipeline feeds it new data.</p><p>The technical implementation in Structured Spark Streaming is roughly as follows:</p><ol><li>A distributed cache (such as Redis or Glint, see point 5) is used to store the most recent model</li><li>Whenever new data comes in this model is distributed to all Spark Streaming workers.</li><li>Each of the workers performs one training step using the model and their own input data, resulting in N (number of workers) new models.</li><li>In the reduce step these N new models are aggregated (using a user defined function) to one new model that is stored in the cache again.</li></ol><p>This looks very promising, but take note that the caveat is in the details of choosing a good aggregation function.</p><p>Related recommended talks to watch: <a href="https://spark-summit.org/eu-2016/events/online-learning-with-structured-streaming/">Online Learning with Structured Streaming</a></p><p><strong>5. Factorization Machines and Parameter servers</strong></p><p>One of the more recent development in machine learning is <a href="http://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf">Factorization Machines</a>. This machine learning algorithm is especially suitable for very sparse datasets, but no distributed implementation was yet available. In one of the talks this was introduced, as well as the Glint parameter server.</p><p>The Glint parameter server is a Spark-optimized distributed cache for storing common machine learning parameter data structures. This component should prove useful for making more embedded distributed machine learning implementation possible in the future.</p><figure><img alt="Factorization Machines on Spark with GlintFM" src="https://cdn-images-1.medium.com/max/300/0*cmm42MS01-Z_5_N-.jpg" /><figcaption>Factorization Machines on Spark with GlintFM</figcaption></figure><p>Related recommended talks to watch: <a href="https://spark-summit.org/eu-2016/events/scaling-factorization-machines-on-spark-using-parameter-servers/">Scaling Factorization Machines on Spark Using Parameter Servers</a> , <a href="https://spark-summit.org/eu-2016/events/glint-an-asynchronous-parameter-server-for-spark/">Glint: An Asynchronous Parameter Server for Spark</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51ef8bf14b6b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigdatarepublic/5-key-topics-of-spark-summit-europe-2016-51ef8bf14b6b">5 Key Topics of Spark Summit Europe 2016</a> was originally published in <a href="https://medium.com/bigdatarepublic">bigdatarepublic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Will big data replace physicians?]]></title>
            <link>https://medium.com/bigdatarepublic/will-big-data-replace-physicians-f6d2d0f4ccb8?source=rss-12eb83dfb55b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6d2d0f4ccb8</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[BigData Republic]]></dc:creator>
            <pubDate>Wed, 17 Aug 2016 10:51:08 GMT</pubDate>
            <atom:updated>2018-03-06T08:09:55.667Z</atom:updated>
            <content:encoded><![CDATA[<p>Self-diagnosing an illness is becoming increasingly popular. Online there are <a href="http://symptoms.webmd.com/#introView">sites</a> that make this very easy for you. The quality however tends to be very low, as so much information is provided. In contrast to these websites, a physician is able to ask questions and to do additional tests. But could you state that providing the right information to smart algorithms would be better, in order to eliminate human error from the equation?</p><h4><strong>Decision tree diagnosis</strong></h4><p>What a physician does in general is following a decision tree based on questions for each of the presented symptoms, until enough confidence is acquired to make a diagnosis. How the branches and leafs are weighted and evaluated is dependent on knowledge, skill and experience of the physician. The society to improve diagnosis in medicine (SIDM) <a href="https://improvediagnosis.site-ym.com/page/Facts">claims</a> that 5–15% of all diagnosis are wrong (USA), indicating that some of these physician’s decision trees were trained poorly.</p><p>Like normal decision trees it is quite easy to overfit the train data, which increases the test error, or in this case increases the risk of misdiagnosing a patient. Why not combining these diagnosis steps (trees) of multiple physicians into a bigger general system (forest) thus improving the accuracy?</p><figure><img alt="Afbeelding1" src="https://cdn-images-1.medium.com/max/300/0*4axwfVTiROo_9xoq.png" /></figure><p><strong>Diagnostic tests and Imaging</strong></p><p>Diagnostic tests like <a href="https://www.youtube.com/watch?v=IpdTeGPruFA">drawing blood</a> and analyzing it, is a process that can be carried out by smart algorithms. Even medical imaging like X-ray or MRI, for which <a href="http://www.diagnijmegen.nl/index.php/Home">some institutions</a> already have smart machine learning to read and interpret them. As more of these techniques evolve and more data is available, machine learning techniques can gain far more experience than any physician could achieve in multiple lifetimes.</p><h4><strong>Data collection and storage</strong></h4><p>It will be a challenge to collect features that can capture behavior of deceases. Most behavior features that increase decease understanding invade personal lives extensively because you need to consistently monitor all patients even when they are in seemingly full health. Checkups only when people are sick would need to move to a more continuous data collection about personal health, like the use of <a href="http://www.informationweek.com/healthcare/mobile-and-wireless/10-medical-device-wearables-to-improve-patients-lives/d/d-id/1323544">medical-device wearables</a>.</p><p>Data generated by the continuously monitoring needs to be stored somewhere or at least transformed into forms that can be used to analyze. The need of storing data of many people will only be possible with a big data infrastructure that can guarantee privacy.</p><h4><strong>Accepted machine learning take over</strong></h4><p>Information-power companies like Google and Netflix apply machine learning to predict what search results are most relevant to you or what kind of movie you like to watch. Their algorithms work well as a vast amount of people use their services. Clearly it doesn’t bother people that their movie-watch-behavior is used to predict what movie they want to watch next, probably because it helps them a great deal in finding the right movies to watch. <a href="http://www.mckinsey.com/industries/pharmaceuticals-and-medical-products/our-insights/the-role-of-big-data-in-medicine">McKinsey</a> states these accepted ways of using big data can also be applied in medicine.</p><p>It might not be long before big data techniques could take over the work of physicians. Techniques we are already using, such as recommender systems, could probably be adapted to predict optimal treatments for deceases, though in my opinion it could still take a while before these data-driven decisions will be accepted. What do you think?</p><p><a href="https://bigdatarepublic.nl/"><em>BigData Republic</em></a><em> provides Big Data Solutions. We are experienced in advanced predictive modeling and deploying large-scale big data pipelines that take security and privacy into account. Interested in what we can do for you? Feel free to </em><a href="https://www.bigdatarepublic.nl/about-us#contact-us"><em>contact us</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6d2d0f4ccb8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigdatarepublic/will-big-data-replace-physicians-f6d2d0f4ccb8">Will big data replace physicians?</a> was originally published in <a href="https://medium.com/bigdatarepublic">bigdatarepublic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Security & Data Privacy (part 2 of 2)]]></title>
            <link>https://medium.com/bigdatarepublic/data-security-data-privacy-part-2-of-2-39b082646f11?source=rss-12eb83dfb55b------2</link>
            <guid isPermaLink="false">https://medium.com/p/39b082646f11</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[privacy]]></category>
            <dc:creator><![CDATA[BigData Republic]]></dc:creator>
            <pubDate>Fri, 10 Jun 2016 17:20:51 GMT</pubDate>
            <atom:updated>2018-03-06T08:10:14.562Z</atom:updated>
            <content:encoded><![CDATA[<p>A growing number of companies are on their way to become data driven organizations. In order to take personal privacy into account, additional measures need to be taken when building data processing infrastructure. Data should be stored and processed in a secure way, and privacy controls should be in place before processing sensitive information.</p><p>In <a href="https://www.bigdatarepublic.nl/data-security-and-privacy-part-1/">part one</a> of these series, we focused on securing your big data infrastructure. In the second part of this blog post, we will discuss data privacy. We look at various measures that can be put in place to address to some extent privacy issues in your processing pipeline.</p><h4>Part 2: Data Privacy in the Data Lake</h4><p>Although many legislative reports exist on the topic of data privacy, <a href="https://www.ipc.on.ca/images/resources/privacybydesign.pdf">privacy by design</a>, and privacy in big data, not many discuss how to actually implement measures into big data processing pipelines. In this blogpost, we will provide you with practical measures that can be applied to increase data privacy in your data lake. These measures are based upon recommendations by the European Union Agency for Network and Information Security (ENISA), as published in their late 2015 <a href="http://arxiv.org/abs/1512.06000">report</a>.</p><h4>Data Minimization</h4><p>A first step towards implementing data privacy into a data pipeline would be to make sure only relevant personal information is stored. A top-down big data approach greatly helps with this. Instead of first building a data lake, and afterwards doing data science, you would reverse this process. Start working from a business value perspective, where you first perform machine learning analysis on smaller datasets that are already available, instead of starting to gather lots and lots of data without knowing if you will ever need it. This will give a good first indication on what data is relevant for your analysis. You can use this to only store personal information that was actually relevant.</p><p>A second step would be to determine how long certain personal records should be retained. This requires a similar top-down approach to big data. It might be your machine learning model only has 1% better accuracy if you use a year of data, compared to only using a couple of months. This gives you actual facts to base your data retention policy on.</p><p>After you have gone through this process, you should inform your users which personal information you will actually be storing and processing, and for how long. This can be done in a public privacy statement for example. Even though you need certain explicit values, it might turn out that you can anonymize certain fields. This brings us to the second privacy enhancing technique (PET): Anonymization.</p><h4>Anonymization</h4><p>Anonymization is the process of altering/masking personal data in such a way that individuals cannot be re-identified and no information about them can be learned. One simple technique to do this is using hashing. A hashing algorithm maps the original value (e.g. a person’s name) to a string of seemingly “random” characters. Hashing algorithms have the property that for the same input, they always have identical output. Also, when you only know the output, you cannot retrieve back the input. This can keep both data utility and data privacy in the datasets, because data scientists will still be able to link different data records based on hash identifiers, while at the same time not directly being able to identify which person’s information they are processing.</p><p>However, although anonymization techniques work great on individual fields in the data, it might still be possible to identify a person when combining fields and/or records from different datasets. This remains a trade-off. In addition, although many personal data fields might be good candidates for anonymization (e.g. e-mail, name, etc.), it might be required to use the original values for other personal data such as age in order to retain utility of the dataset. Good techniques to still get to a reasonable level of privacy would be to use aggregation in combination with data separation.</p><h4>Aggregation &amp; Separation</h4><p>Data privacy can still be assured to a certain extent in case of non-anonymizable personal data. One way to do this is to only report data analysis results at a higher aggregate level. This could be, for example, first clustering personal data by age or by geographical region, such that at least 10 people are in each group. This way, no individual’s privacy is at stake, while still being able to do proper analysis. This technique is actually easier to perform on big data than on small data. Simply because the sheer numbers will still show patterns after grouping.</p><p>Another accompanying technique is to use data separation in a distributed environment. This means storing different personal data fields (columns) across different servers. When used in combination with aggregation and anonymization techniques, this will prevent people from re-identifying an individual, since fields belonging to one person are completely separated. Separation can be accomplished by using technology that uses columnar storage across different storage servers. Examples of this are Apache Spark, Apache Cassandra, and HDFS storage formats such as Parquet.</p><h4>Auditing</h4><p>Just like for data security, auditing/recording who has access to the data and when is of utmost importance. Although this might raise privacy concerns for the data scientists, this would allow companies to pro-actively signal when a data leak is happening, and privacy of potentially thousands of people is at stake. The Hadoop eco-system provides tools like Apache Ranger to perform audit logging. This detailed logging provides the information about which user or person has accessed (or tried to access) which part or service of your data platform. Applying anomaly detection algorithms on this data could enable the data lake to monitor its users’ actions all by itself. The combination of these tools and techniques help the cause-analysis of security breaches or data leaks. This additional layer of security ensures that data leaks, can be traced even when it was caused by authorized users.</p><h4>Control / right to be forgotten</h4><p>As a last measure, one should take into account that an individual should have control over his personal data. This means that the individual should be able to request its personal data to be removed from the system when he or she wants. Large corporations such as Google already offer such options. Links to personal information can be removed from Google’s search results upon legitimate request.</p><p>Implementing this measure might pose one of the hardest as it directly opposes many data engineering best-practices. For example, storing all the data in an append-only format such that new data is added, but data is considered read-only until it hits retention limits. However, there are ways to implement this. One could for example write a batch job that will remove all references for a specific person from all the datalake’s datasets, and re-compacts them to prevent data fragmentation (which degrades performance).</p><h4>Conclusion</h4><p>Privacy, like security, should be incorporated by design, and not be an afterthought. If we are able to deal with data privacy challenges by applying appropriate measures, this could build more trust in the big data ecosystem. However, current state-of-the-art big data technology is not at all privacy focused. Thus, applying these privacy measures still requires in-depth analysis and careful planning.</p><p><a href="https://bigdatarepublic.nl/"><em>BigData Republic</em></a><em> provides Integrated Data Solutions. We are experienced in deploying large-scale big data pipelines that take security and privacy into account. Interested in what we can do for you? Feel free to </em><a href="https://www.bigdatarepublic.nl/about-us#contact-us"><em>contact us</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39b082646f11" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigdatarepublic/data-security-data-privacy-part-2-of-2-39b082646f11">Data Security &amp; Data Privacy (part 2 of 2)</a> was originally published in <a href="https://medium.com/bigdatarepublic">bigdatarepublic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Security & Data Privacy (part 1 of 2)]]></title>
            <link>https://medium.com/bigdatarepublic/data-security-data-privacy-part-1-of-2-d8a467d3bf75?source=rss-12eb83dfb55b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8a467d3bf75</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[BigData Republic]]></dc:creator>
            <pubDate>Fri, 27 May 2016 16:23:40 GMT</pubDate>
            <atom:updated>2018-03-06T08:10:17.514Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*aJuBu4ROGNbsA44h.jpg" /><figcaption>Photo credit to perspecsys.com</figcaption></figure><p>In 2014 the European Union court of justice ruled (<a href="http://ec.europa.eu/justice/data-protection/files/factsheets/factsheet_data_protection_en.pdf">EU Data protection factsheet</a>) that individuals have the right to ask search engines to remove links with personal information about them. In the Netherlands, since first of January 2016, companies and authorities are required to report cases of data leaks (<a href="https://autoriteitpersoonsgegevens.nl/nl/melden/meldplicht-datalekken">website in Dutch</a>) to an authority and in some case to involved individuals as well.</p><p>On the other hand, a growing number of companies are on their way to become data driven organizations. In order to comply with mentioned legal developments, additional measures need to be taken when building data processing infrastructure. These measures, however, might clash with data engineering best-practices. A simple example is the preferred append-only approach for data gathering. Stored events could easily involve personal information about your costumers. What happens to your system when someone requests to remove some of his personal data, and as a consequence breaks the append-only assumption?</p><p>In this two-part blog post, we look at the implications of these legal developments for your big data infra structure and analytics pipeline. In part one we focus on securing your big data infrastructure. In part two we look at various measures that can be put in place to address to some extent privacy issues in your processing pipeline.</p><h4>Part 1: Data Lake Security</h4><p>Good data lake security is like an onion: the core is surrounded by multiple layers of security measures. This way, reaching the core means that all those layers must be breached to achieve the valuable information: your customers’ data and the algorithms that encode how you drive your business towards a competitive edge. Important layers in data lake security are: fire-walling, authentication / authorization, encryption, and auditing. We will address all of these layers in this blogpost.</p><h4>Fire-walling / access control</h4><p>A first layer of defense around your data lake. A firewall is a system which protects a computer, or network of computers against abuse from outside sources. What outside means, depends on the level of security you want to put in place. Unfortunately, security breaches and data leaks are not always caused by external sources. In large organizations it is therefore good practice to place sensitive infrastructure, such as a data lake, behind a dedicated firewall. This way access to a data lake can be tightly controlled on the network level. With a firewall you can secure with relative simple measures such as packet filtering, or more advanced measures such as state-full filtering and application layer filtering.</p><p>To further enhance the security enforced by a firewall, you can put your data lake behind a proxy. A proxy hides the network topology of the cluster that is behind it. Besides this security advantage, it also helps the scalability of your architecture. Furthermore, proxies can be integrated with authentication systems. In the Hadoop eco-system, <a href="https://knox.apache.org/">Apache Knox</a> is a well known example of such a proxy. It integrates tightly with components from the Hadoop stack and can connect with identity management systems such as Active Directory or LDAP. This brings us to the topic of authentication.</p><h4>Authentication / Authorization</h4><p>The next layers of security are authentication and authorization. Authentication is the process of ensuring that people or services who access data or algorithms on the data lake, really are who they claim to be. A widely used method for this process is the Kerberos protocol which is integrated in enterprise identity management systems such as Active Directory. For human end-users, security can be further enhanced by requiring two-factor authentication. This requires end-users to first provide their username/password combination, followed by a code that is generated on external device, usually by a smartphone app.</p><p>Knowing who is requesting access to data or a service is not enough. It may well be that certain persons or services should only access a limited subset of the available data and predictive services on your infrastructure. In most organizations which take security serious, the least privileged principle is applied. Meaning, both users and applications, only get access to the minimal amount of data and services that is required for them to operate properly. This is what is meant by authorization, describing who gets access to what. In the Hadoop eco-system <a href="http://ranger.apache.org/">Apache Ranger</a> offers centralized security administration and let you control access to data and services in your data platform.</p><h4>Auditing</h4><p>Knowing who is using the infrastructure, and trusting that this person or service is really who it tells to be is not enough. In case of security breaches or data leaks it is essential to understand who has had access to what. The Hadoop eco-system provides tools like Apache Ranger to perform audit logging. This detailed logging provides the information about which user or person has accessed (or tried to access) which part or service of your data platform. Applying anomaly detection algorithms on this data could enable the data lake to monitor its users’ actions all by itself. The combination of these tools and techniques help the cause-analysis of security breaches or data leaks. This additional layer of security ensures that data leaks, can be traced even when it was caused by authorized users.</p><h4>Encryption</h4><p>Finally, data encryption is key for securing the data in a data lake. To secure your data, it is imperative to encrypt it both in transit and at rest. Data encryption in transit makes sure the data communication between different servers is secured. Eavesdroppers will not be able to make sense of the data transferred between different components in the system. Data encryption at rest encrypts the data on physical hard disks. This renders the data useless when physical data storage disks are compromised.</p><p>Many components in the Hadoop eco-system allow for encryption to be configured. Configuring encryption correctly is crucial for securing the data in the cluster.</p><h4>Conclusion</h4><p>Security is as good as the weakest link, or as the thinnest layer to stay in terms of the onion. Stacking multiple layers of security measures will strengthen the overall defense of your valuable data and intellectual property. In any case, data platform security is something to take in account from the beginning and not something to add as an afterthought. Many of above mentioned security measures need analysis and careful planning to get properly implemented.</p><p><a href="https://bigdatarepublic.nl/"><em>BigData Republic</em></a><em> provides Integrated Data Solutions. We are experienced in deploying large-scale big data pipelines that take security and privacy into account. Interested in what we can do for you? Feel free to </em><a href="https://www.bigdatarepublic.nl/about-us#contact-us"><em>contact us</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8a467d3bf75" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigdatarepublic/data-security-data-privacy-part-1-of-2-d8a467d3bf75">Data Security &amp; Data Privacy (part 1 of 2)</a> was originally published in <a href="https://medium.com/bigdatarepublic">bigdatarepublic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>