<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Well Red - Medium]]></title>
        <description><![CDATA[Exploring the tech industry one bit at a time. Brought to you by redspace.com. - Medium]]></description>
        <link>https://medium.com/well-red?source=rss----f81bf41391f2---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Well Red - Medium</title>
            <link>https://medium.com/well-red?source=rss----f81bf41391f2---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:26:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/well-red" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Finding Value in Unlabeled Data]]></title>
            <link>https://medium.com/well-red/finding-value-in-unlabeled-data-5799e113ecf2?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/5799e113ecf2</guid>
            <category><![CDATA[unlabeled-data]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[string-matching]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Zack Miller]]></dc:creator>
            <pubDate>Fri, 18 Sep 2020 12:09:11 GMT</pubDate>
            <atom:updated>2020-09-18T12:09:11.879Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A person holding a magnifying glass up to a laptop, examining the keys." src="https://cdn-images-1.medium.com/max/1024/0*TxEGWFqtco2qi70T" /><figcaption>Photo by <a href="https://unsplash.com/@olloweb?utm_source=medium&amp;utm_medium=referral">Agence Olloweb</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>“Machine learning” conjures up images of complex neural networks and massive data warehouses, but you probably already have all you need to start reaping the benefits.</blockquote><p>For businesses that are new to machine learning, it can sound like an overly complex buzzword that requires massive amounts of specialized data, and even more talent and expertise. In reality, machine learning and data science are just about generating insight by recognizing patterns in all kinds of data. With the right approach, companies can ease into things by extracting value from data and assets that already exist within their organization. In this post, I’ll discuss how the application of data science and machine learning techniques helped transform metadata from an existing content management system into a new QA process.</p><h3>The Situation</h3><p>A client was looking to automate some quality controls for their content pipeline. In this instance, the concern was a portal where users could upload a file and include a short description. If the description did not match the contents of the file, they wanted to raise a warning so that the error could be corrected. The portal had been in operation for quite some time, which meant that the client had access to plenty of existing descriptions and file content.</p><p>The initial request was to find a string matching algorithm that would take in the file contents and descriptions and return a confidence score for whether or not the inputs were a match. This would involve selecting several candidate algorithms, applying them across the dataset, and comparing their results. However, because the pipeline previously had no quality controls, there was nothing in the dataset to indicate whether the description for each file did, in fact, match the content. This type of data is referred to as <strong>unlabeled data</strong>.</p><p>Unlabeled data is a common snag in many machine learning projects. One remedy is to manually review the data and create labels for each row, but this can quickly become expensive and time-consuming. There are also heuristic tools, <a href="https://www.snorkel.org/">like Snorkel</a>, that allow developers to write custom labeling code to enrich their dataset — but even this process can require a high level of effort and domain expertise. Labeled data can be used to train supervised learning models, but it is also possible to train certain models without ever labeling the data. This is called <strong>unsupervised learning</strong>.</p><h3>The Data</h3><p>The format of the existing dataset contained a unique identifier for each file, along with a string description and a list of strings extracted from the file contents via ASR (automatic speech recognition) and OCR (optical character recognition).</p><figure><img alt="A table describing the database contents. Each row contains an ID, text description, and list of file contents." src="https://cdn-images-1.medium.com/max/1024/1*6dLs0JTfj1MHuSaGTyOqrA.png" /><figcaption>Sample rows in the client’s media database</figcaption></figure><p>In the example above, row 1 should match because the description is paired well to the contents of the file. The description for row 2 is unlikely to pass QA because it does not relate to the contents.</p><h3>The Solution</h3><p>Six different fuzzy string matching algorithms were selected as candidates and applied to the dataset using <a href="https://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/">the fuzzywuzzy Python package</a> to find the best scoring match among the extracted file contents. The results looked like this:</p><figure><img alt="A table describing the algorithm results. Each row contains the file ID, description, content, and six scores." src="https://cdn-images-1.medium.com/max/1024/1*edu98LSAN76chX-oeuBX3w.png" /><figcaption>Sample scores generated by several string matching algorithms</figcaption></figure><p>By examining the difference in score distribution between each of the candidate functions, we can see that there is not an immediately obvious consensus about which rows do or do not match. Heuristic labeling alone does not appear to be enough to label our dataset.</p><figure><img alt="A collection of six histograms, each displaying the distribution of scores generated by an individual matching algorithm." src="https://cdn-images-1.medium.com/max/1024/1*fRZb0Y61MbZs7HdUdE3naA.png" /><figcaption>Distributions of scores from each string matching algorithm</figcaption></figure><p>This is where unsupervised learning can help. If we treat the output scores of each algorithm as an input feature, we can train a classifier model to divide the input rows into several possible classes or categories. One such model is called a <a href="https://towardsdatascience.com/kmeans-clustering-for-classification-74b992405d0a"><strong>k-means classifier</strong></a> and allows us to choose the number of clusters we will generate. Since we are trying to create a binary pass-fail label for the data, we can train a k-means model to find 2 clusters in the model and review the distribution of the scores in each. If we do that, the scores look something like this:</p><figure><img alt="A collection of two score histograms. One is labelled “potential match” and the other is labeled “potential reject”." src="https://cdn-images-1.medium.com/max/523/1*KbfwFfL_xd5juU_dMfN6iQ.png" /><figcaption>Score distribution for two clusters generated by k-means classification</figcaption></figure><p>The scores in the above clusters are approximately <a href="https://www.statisticshowto.com/what-is-a-bimodal-distribution">bimodal</a>, with the mean of the potential match cluster averaging a score of 0.8 and the potential reject cluster averaging 0.47. This is a meaningful result, but deploying such a classifier in a production pipeline would mean running 6 separate matching functions against every upload and passing the results to a seventh process to determine whether to accept or reject the file description. It would be less cumbersome to apply a single scoring method that best represents the distinction between the two labeled classes.</p><p>The <a href="https://shap.readthedocs.io/en/latest/">SHAP package in Python</a> allows us to analyze the inputs and outputs of various machine learning models using an “optimal credit allocation” approach. This effectively tells us which inputs most greatly influence the output of a model. Applied to the labeled clusters above, we obtain the following:</p><figure><img alt="A scatter plot which shows the total impact of six matching algorithms on the horizontal axis with color-coded input values." src="https://cdn-images-1.medium.com/max/1024/0*13daTbY8tM72TcUO" /><figcaption>Influence of each matching algorithm, as determined by SHAP</figcaption></figure><p>Method 6 is the most discriminating scoring function, and gives us enough confidence to include it as a standalone part of the quality control pipeline. It is important to note that this post does not touch on several critical elements of the data science process, such as data cleaning, centering, model tuning, and model evaluation. In applied machine learning, like with many other software projects, a number of other contextual factors can inform a final decision.</p><p>For example, this particular client planned to use a human-augmented approach which would automatically approve matches above a very high threshold, reject matches below a moderately low threshold, and refer the remaining records to a manual review process. When enough data has been manually reviewed, the pipeline will be analyzed all over again using a supervised approach with the newly available labeled data. For this reason, we were comfortable choosing a matching algorithm that was sufficiently discriminatory without further investigating confusion metrics and other parameters that you’d expect in a more comprehensive analysis.</p><h3>Conclusion</h3><p>Machine learning may carry the magical aura of a modern business buzzword, but following the powerful and proven fundamentals of data science can set almost any project on the right track. I hope this post has introduced you to some new tools and concepts for dealing with unlabeled data, and I encourage you to reach out to REDspace to see what unrealized opportunities your data might be hiding.</p><p><a href="https://bit.ly/35YgimT"><em>REDspace</em></a><em> is a software development company that builds innovative, custom, and flexible platforms for global enterprises. We combine well-built backends with engaging front-end user experiences. </em><a href="https://bit.ly/35YgimT"><em>redspace.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5799e113ecf2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/finding-value-in-unlabeled-data-5799e113ecf2">Finding Value in Unlabeled Data</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[State Machines for Everyone — Part 1 Introduction]]></title>
            <link>https://medium.com/well-red/state-machines-for-everyone-part-1-introduction-b7ac9aaf482e?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/b7ac9aaf482e</guid>
            <category><![CDATA[abstraction]]></category>
            <category><![CDATA[software-model]]></category>
            <category><![CDATA[finite-state-machine]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Alex Dodge]]></dc:creator>
            <pubDate>Tue, 08 Sep 2020 14:12:50 GMT</pubDate>
            <atom:updated>2020-09-09T16:25:21.700Z</atom:updated>
            <content:encoded><![CDATA[<h3>State Machines for Everyone — Part 1 Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AMr2pvPg-aQXfFpfrEL9ag.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@iniguez?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Carlos Alberto Gómez Iñiguez</a> on <a href="https://unsplash.com/s/photos/traffic-light?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>You pull up to an intersection after having just missed the green light. It’s a light you’ve driven through many times, so you know you’ve got about a minute before it changes. Without fail, a minute passes and the light changes from red to green, and you continue on your way! On to the next one….</p><p>There’s a lot of things in life that are predictable like stoplights. Kettles and coffee machines providing your morning dose of energy, starting your car and putting it into gear, the elevator in your apartment or office building. Alright maybe the car example is debatable depending on how old your car is. But the point is that there are a ton of things around us in our everyday life that always move through known steps or phases, and always take the same actions to get them there. Building and maintaining these systems would also be quite hard if these phases and actions weren’t clear. For instance if every morning you went to turn on your coffee machine, and it randomly chose between three buttons, but only one of those buttons would work on a given day.</p><p>In computer science we can represent these systems in a more formal way, as something called <strong>State Machines</strong> or in our case above and in this article <strong>Finite State Machines</strong>. It sounds fancy, but they really just help us to describe and work with any system in a way that makes it easier on our brains and our software.</p><p>In this article we’ll dig into <strong>Finite State Machines (FSMs) </strong>at an introductory level. By the end of the article you will understand what they are, how to represent them, and some tools that will help you define and visualize them.</p><p>This article will provide some code in JavaScript and Typescript, but could be generalized to any language, and a lot of the introduction just deals with understanding from a visual perspective. So feel free to read on no matter your level!</p><p>This article is also part of a series and is broken down as follows:</p><ul><li><strong>Part 1 </strong>— Introduction and how to represent specific systems as <strong>Finite State Machines</strong> through tables, state diagrams, and JavaScript / TypeScript code, along with some tools to help you visualize your own system.</li><li><strong>Part 2</strong> — Exploring the process of building a <strong>Finite State Machine</strong> abstraction, and use it in both the sample we discussed in this first article, as well as some more real to life ones.</li><li><strong>Part 3 </strong>— Popular libraries and variations on <strong>Finite State Machine</strong>, and different ways to use them.</li></ul><h3>State Machines and Traffic Lights</h3><p>Traffic lights are a nice example when working with state machines, as they’re something we’re<strong> </strong>all familiar with.</p><p>Let’s start with a common traffic light you might find in North America,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*C85bOFa0r5aoGNdi3TUvbg.png" /><figcaption>Traffic Lights — All States</figcaption></figure><p>As everyone knows <strong>Green</strong> means go, <strong>Red</strong> means stop, and <strong>Amber</strong> to stop if you can safely do so (or just drive faster to make the light). A <strong>Flashing Red</strong> is also sometimes shown which means to treat an intersection like four way with stop signs.</p><p>If we put ourselves in the shoes of the people who had to build these, we could envision using a <strong>Finite State Machine</strong> to represent the stoplight. We stick finite on the front state machine because there are a <strong>finite (set number) of states</strong> <strong>(colours) </strong>that the traffic light will transition between. There are also <strong>a finite number of actions or triggers (traffic light timers) </strong>which are then used to transition between the states.</p><p>Let’s go back to the image and fill it out explicitly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t03qBGv_S_Iz7dxnOnwt_w.png" /><figcaption>Traffic Lights — States and Actions</figcaption></figure><p>Now you’ve learned about your first <strong>Finite State Machine</strong>! We’re going to go a lot deeper down the rabbit hole, but this will serve as a nice stepping off point.</p><h3>The Potential of State Machines</h3><p>Before we dig in further I will outline some of the benefits state machines can give to you, your team, or your codebase. You can even use state machine representations to better understand systems within a product or even a UI design.</p><ul><li>Remember the finite part? Knowing the states and actions up front will force you to have a better understanding of the systems you’re creating. So <strong>clarity</strong> in terms of <strong>structure </strong>and<strong> scope </strong>are two things state machines will give you.</li><li>For those who are programmers, I’m sure you can envision a codebase that had far to many <strong>isDisabled</strong>, <strong>isOn</strong>, <strong>isLoading</strong>, <strong>isErrored </strong>flags littered throughout levels of nested if statements, and to pick a certain path you had to traverse the same minefield of conditionals every time. State machines help to dramatically lessen or remove this top down approach of decision making. So <strong>less nesting</strong> <strong>and</strong> <strong>complexity of decision making </strong>in<strong> </strong>your codebase.</li><li>State machines exposed as public properties also serve as great mechanisms for context and testing. They <strong>simplify</strong> and provide a <strong>cleaner interface</strong> for <strong>users and testing mechanisms</strong> to determine what’s going on under the hood of your API or broader system.</li></ul><p>At this point we covered the what and the why, so let’s dig into the how and start seeing FSMs in practice.</p><h3>Modelling Our Own State Machines</h3><p>There are a lot of finite state machine libraries which give you access to ready made and well tested APIs. This process is nice because they’re a great concept to break down, and building your own isn’t too complicated.</p><p>We’ll walk through how to model our stoplight example with an <strong>Adjacency List, State Transition Diagram, </strong>and a <strong>JavaScript / TypeScript interface</strong>. By doing this we will have a better idea of what the code looks like that drives the logic behind these systems, and gives us a clear visual of our system.</p><h4>Adjacency list</h4><p>To start, let’s write all the possible ways our lights will transition from one colour to another. A simple way to do this is to put them in a table which has the available states of our system in one column, and the adjacent states in the other.</p><p>For instance, because a <strong>Red</strong> light will transition to a <strong>Green</strong> light, we would say it’s adjacent to the <strong>Green</strong> light state and the <strong>Green</strong> light is adjacent to <strong>Red</strong>.</p><p>The table below is the adjacency list for our traffic light system:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68b4c3b56e4c593807e56a7785d0e429/href">https://medium.com/media/68b4c3b56e4c593807e56a7785d0e429/href</a></iframe><p>This list is harder to look at, because it doesn’t convey direction. So let’s make another which specifies just the states that are transitioned to.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b9e96cc3a66f515803e04b60fe69af2/href">https://medium.com/media/0b9e96cc3a66f515803e04b60fe69af2/href</a></iframe><p>Note that for the <strong>FLASHING_RED </strong>colour state, we’re making a broad assumption that when the system is restarted it will first transition to a <strong>RED </strong>light. We’ll consider this the <strong>starting state of our system</strong>.</p><h4>State Transition Diagram</h4><p>For simple state machines we could just skip writing out the adjacency lists, but for more complex systems you will probably end up writing out a version of it anyways, so it’s a useful practice.</p><p>This leads us to the <strong>State Transition Diagram,</strong> a great tool for visualizing and designing finite state machines.</p><p>For the <strong>State Transition Diagram</strong> we’re going to turn our lists above into a visual diagram, with circles representing our states, arrows between them indicating the transitions, and labels for the actions. For stoplights, the action that causes the lights to transition is often a timer, or in the case of the flashing red, an error signal caused by a power outage. We’ll put all this info into the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4STuGzPGmPCMSriRKqck9w.png" /><figcaption>Traffic Lights — State Transition Diagram</figcaption></figure><p><strong>State Transition Diagrams</strong> are nice because they give us a clear way to visually look at a system, and help reveal which parts are overly complex, or where there is redundancy.</p><p>In this step we also decided on the names of the actions which cause the transitions between states. In the next step we’ll turn this into code!</p><h4>State Map Using JavaScript and TypeScript</h4><p>Now that we have a state transition diagram, we will build out an object which represents it. We’ll then type that object using a TypeScript interface, and in <strong>Part 2 </strong>we’ll use the interface to help us when designing our own custom state machine abstraction.</p><p>First up, let’s take a look at the JavaScript FSM (finite state machine) map object equivalent to above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c282cda79be2ee617a6222aeb65a7b5/href">https://medium.com/media/1c282cda79be2ee617a6222aeb65a7b5/href</a></iframe><p>Please notice that instead of having an action that specifies what timer we’re dealing with, we can generalize to have only a single <strong>TIMER_EXPIRES </strong>action.</p><p>Next we will type the object in TypeScript:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d487fb8e71bdf9093531a57f992a188/href">https://medium.com/media/6d487fb8e71bdf9093531a57f992a188/href</a></iframe><p>Given everything above, we now have the tools we need to represent real life systems as <strong>Finite State Machines</strong>, and the tools to visualize them in a number of ways.</p><p>In <strong>Part 2</strong> we’ll investigate how we can use these objects to build an actual <strong>FSM abstraction</strong> which we can use in our codebases.</p><h4>Final Resources and Tools</h4><p>I’m leaving you with two helpful links to explore before reading<strong> Part 2</strong>: A talk by the creator of <strong>XState</strong>- a finite state machine library for JavaScript and Typescript, and the visualization tool <strong>XViz</strong> which lets you take a JavaScript object very similar to the one we described above and dynamically create the state transition diagram that goes along with it.</p><p><a href="https://slides.com/davidkhourshid/finite-state-machines">David </a>Khourshid<a href="https://slides.com/davidkhourshid/finite-state-machines"> — Finite State Machines</a></p><p>Using our JS object above, we will modify it slightly, and copy in into the following <strong>XViz</strong> site. You’ll see the state diagram get generated automatically, which should look very similar to the state transition diagram we created above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d272b177a539f9e9c146e197adad880/href">https://medium.com/media/1d272b177a539f9e9c146e197adad880/href</a></iframe><p><a href="https://xstate.js.org/viz/">XState Visualizer</a></p><h3>Conclusion</h3><p><strong>Finite State Machines </strong>are a way of representing systems with a finite number of states, and a finite number of actions to move between those states. They help make the modelled systems clearer in terms of structure and scope, and generally help to simplify and reduce nesting when used in codebases.</p><p>In the next article we’ll use the above configuration and TypeScript definitions to build out a basic FSM abstraction, and use it to generate a running example of our traffic light state machine, along with some practical UI and API examples!</p><p><a href="http://redspace.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=finitestatemachines"><em>REDspace</em></a><em> is a software development company that builds innovative, custom, and flexible platforms for global enterprises. We combine well-built backends with engaging front-end user experiences. </em><a href="http://redspace.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=finitestatemachines"><em>redspace.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7ac9aaf482e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/state-machines-for-everyone-part-1-introduction-b7ac9aaf482e">State Machines for Everyone — Part 1 Introduction</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to gRPC and Protocol Buffers]]></title>
            <link>https://medium.com/well-red/intro-to-grpc-and-protocol-buffers-c21054ef579c?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/c21054ef579c</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[c-sharp-programming]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Trevor Kendrick]]></dc:creator>
            <pubDate>Wed, 29 Jul 2020 14:22:13 GMT</pubDate>
            <atom:updated>2020-07-29T16:09:32.922Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P4yRnjxSJID9rep5.jpg" /><figcaption>RPCs are dead! Long Live RPCs!</figcaption></figure><p>gRPC is a burgeoning piece of technology that allows you to build web services where you can write code in your codebase’s native language; without writing your 50th custom HTTP Client to conform to some Swagger doc that you read on your company’s ill-maintained documentation hub. In this article, I will outline some of the basic concepts of gRPC and Protocol Buffers, some use cases for these technologies and finish with a simple demo of some cross-language gRPC services in action. I will also outline some common concerns at the end to consider. For official documentation, check out <a href="https://grpc.io/">grpc.io</a>.</p><h3>What are Protocol Buffers?</h3><p>Protocol buffers are a mechanism for serializing structured data. It is language-independent and uses an Interface Definition Language (IDL) for defining the shape of a given message and the properties within that message. Protobuf message definitions have values, but no methods; they are data-holders. Protobuf messages are defined in .proto files.</p><p>Protocol buffers has a compilation tool called protoc that can compile your code in many supported languages. The current list of officially-supported as of the writing of this article includes:</p><ul><li>C / C++</li><li>Objective C</li><li>C#</li><li>Dart</li><li>Go</li><li>Java</li><li>Kotlin (JVM)</li><li>Javascript (Node)</li><li>PHP</li><li>Python</li><li>Ruby</li><li>Javascript (Web)</li></ul><p>There are other languages that have implementations but are not part of the “official” list yet, such as Swift and Haskell. You can find some of them on the <a href="https://github.com/grpc">grpc GitHub account</a>.</p><p>Protobufs use default values for all properties, so if you do not define an int, for example, it will default to 0. Strings will default to empty strings, booleans to false, etc.</p><p>You can define a protobuf message in a .proto file like so:</p><pre>message MovieItem {<br>   <em>//1 is the index used when serializing; do not reuse if the type of data changes!</em><br>  string name = 1;</pre><pre>  double price = 2;<br>  bool inStock = 3;<br>}</pre><h3>What is gRPC, then?</h3><p>gRPC is a group of technologies that allow you to communicate with other applications using Remote procedure calls (RPCs). Remote procedure calls are function calls that you make in your own codebase, but they execute on another machine, then return in your app. This abstracts the network from you, letting you call methods as if they were local code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*oToGAk3r72KlNJoWgDFHkQ.png" /><figcaption>Sourced from: <a href="https://grpc.io/docs/what-is-grpc/introduction/">https://grpc.io/docs/what-is-grpc/introduction/</a></figcaption></figure><p>gRPC uses Protocol Buffers to serialize data between clients and servers, and has extra .proto syntax for defining services used in your applications. With the help of a gRPC plugin for protoc (or a plugin as part of your build script for your given platform), you can generate code that will give you the methods needed to call a given service, all with native typing in your language of choice (if applicable)!</p><p>You can define a gRPC service in your .proto file like this (we will dissect this .proto file later to explain how it works):</p><pre>syntax = &quot;proto3&quot;;</pre><pre>package moviecatalog;</pre><pre>service MovieCatalog {</pre><pre>  rpc SaveNewMovie (MovieItem) returns (AddMovieResponse) {}<br>  rpc FetchExistingMovie (FetchMovieRequest) returns (MovieItem) {}<br>}</pre><pre>message MovieItem {<br>    string name = 1;<br>    double price = 2;<br>    bool inStock = 3;<br>}</pre><pre>message AddMovieResponse {<br>    bool wasSaved = 1;<br>    int32 itemId = 2;<br>}</pre><pre>message FetchMovieRequest {<br>    string name = 1;<br>}</pre><p>gRPC supports multiple data transfer methods: Request + Response (like your typical REST service), Server-side streaming (out of scope for this article), Client-side streaming (out of scope for this article), Bidirectional streaming (out of scope for this article).</p><p>Examples of these other transfer methods can be explored in a future post.</p><h3>Why should I use gRPC instead of REST (HTTP+JSON)?</h3><p>When using Protocol Buffers for sending messages over the network, your payloads are serialized in binary, so they are much smaller than XML or JSON on the wire. This can save you bandwidth ($$$) and improve network performance.</p><p>NOTE: There is a slight overhead for serializing and deserializing binary packets (and a couple of bytes in the headers), so if you have thousands of super tiny messages from multiple clients, you may not save as much in terms of milliseconds compared to REST. However, in most cases, especially long-lasting connections, gRPC will beat out REST most of the time. For benchmarks, check out <a href="https://www.yonego.com/nl/why-milliseconds-matter">this article</a>.</p><p>Since gRPC uses code generators that give you native code for your given language, this saves a lot of time in overhead when writing custom serializers for JSON or XML. Because gRPC gives you method stubs for service calls, this also means you do not have to use other tools (or manual effort) to make sure your calls include the appropriate payload structure, since they are enforced by actual code.</p><h3>What kind of things should I use gRPC for?</h3><p>Anything you use REST for today can be done with gRPC. gRPC also has support for byte streaming, so you can use it for sending things like audio and video data, or geolocation data over time, without polling for data over time with individual requests. Since gRPC uses HTTP/2, it also has support for multiple simultaneous streams multiplexed over the same connection.</p><h3>Okay, show me the code!</h3><p>The following sample service will be a simple request+response service, similar to your typical REST call. Streaming will not be covered in this article, but may be addressed in a future one.</p><p>We will build a simple service for saving and fetching movies from a movie store. We will define the structure of a movie, the requests and responses, and the service contract. We will then implement the business logic for the service and create a server to host the service, as well as a few clients to interact with the service to demonstrate the various implementations in different programming languages. Let’s get started!</p><p>Defining your service contract in .proto files Message payloads and gRPC service stubs are defined in .proto files. There is a decent amount of typing support in protobuf that will convert to the compiled language. You can see a full list of types in the <a href="https://developers.google.com/protocol-buffers/docs/proto3">Protocol Buffer language guide</a>.</p><p>Let’s break down what’s going on in the .proto file we showed before:</p><pre>(movie-service.proto)</pre><pre>//Use this to set the proto language version. As of this article, proto3 is the current version and the syntax here will adhere to its rules<br>syntax = &quot;proto3&quot;;</pre><pre>//These are additional options you can provide to the proto compiler. We are using them to make our generated Java code a little bit more organized; this is not required.<br>option java_package = &quot;grpc.demo.java.moviecatalog&quot;;<br>option java_outer_classname = &quot;MovieCatalogProto&quot;;</pre><pre>//Define the package/namespace that will be used in the generated code<br>package moviecatalog;</pre><pre>//Defines our gRPC service name<br>service MovieCatalog {</pre><pre>  <em>//This has a few things going on, so let&#39;s dissect this line:</em><br>  <em>// `rpc` is a keyword that defines the contract (ie function/method name) of this remote procedure call</em><br>  <em>// `SaveNewMovie` is the name of the method that will be generated</em><br>  <em>// `(MovieItem)` is the message type we will pass as an argument to `SaveNewMovie()` in our code</em><br>  <em>// `returns (AddMovieResponse)` will send a message with the `AddMovieResponse` format back to the caller once it completes successfully</em><br>  <em>// `{}` is used for additional custom options that are out of scope for this article</em><br>  rpc SaveNewMovie (MovieItem) returns (AddMovieResponse) {}</pre><pre>  rpc FetchExistingMovie (FetchMovieRequest) returns (MovieItem) {}<br>}</pre><pre>//Defines a message (similar to a struct or data object) with the given name<br>message MovieItem {</pre><pre>  //Each property has a type that will be converted the appropriate equivalent in the compiled language<br>  //`string` is the data type. *This must NOT change after it has been used in production or you WILL break clients*<br>  //`name` is the label given to that data type. This can change later if you decide on a better name.<br>  //`= 1;` is the interesting bit. The number here is an index used by the generated code to understand which value to serialize/deserialize between clients and servers. *This must NOT change* If you no longer wish to use this field in the future, you MUST create a new field and leave this one unused in your implementation but you can NOT remove it.<br>  string name = 1;</pre><pre>  double price = 2;<br>  bool inStock = 3;<br>}</pre><pre>message AddMovieResponse {<br>  bool wasSaved = 1;<br>  int32 itemId = 2;<br>}</pre><pre>message FetchMovieRequest {<br>  string name = 1;<br>}</pre><h3>Generating gRPC code in Java (Gradle)</h3><p>You could use the protoc compiler via the command line if you wish, but if you need to build your proto files on a regular basis, you will want to generate it as part of your build to ensure you&#39;re up to date. Thankfully there are plugins for your build tool of choice to let you automate this process.</p><p><a href="https://github.com/google/protobuf-gradle-plugin">https://github.com/google/protobuf-gradle-plugin</a></p><p>This example shows a Groovy-style Gradle file with the plugins you will need:</p><pre>(build.gradle)</pre><pre>buildscript {<br>    repositories {<br>        mavenCentral()<br>    }<br>    dependencies {<br>        classpath &#39;com.google.protobuf:protobuf-gradle-plugin:0.8.12&#39;<br>    }<br>}</pre><pre>plugins {<br>    <em>// Apply the java plugin to add support for Java</em><br>    id &#39;java&#39;</pre><pre>    <em>//Required for protobuf</em><br>    id &quot;com.google.protobuf&quot; version &quot;0.8.12&quot;</pre><pre>    <em>// Apply the application plugin to add support for building a CLI application.</em><br>    id &#39;application&#39;<br>}</pre><pre><em>//Update these versions as necessary; these are the current versions as of the writing of this article.</em><br>def grpcVersion = &#39;1.29.0&#39;;<br>def protobufVersion = &#39;3.12.2&#39;;</pre><pre><em>//Search these repos for dependencies</em><br>repositories {<br>    mavenCentral()<br>}</pre><pre>dependencies {<br>    <em>//grpc dependencies</em></pre><pre>    <em>//The annotations api is used by the generated gRPC code</em><br>    implementation &#39;javax.annotation:javax.annotation-api:1.3.2&#39;</pre><pre>    <em>//This string interpolation ${} lets us reuse the version variables we defined above</em><br>    implementation &quot;io.grpc:grpc-all:${grpcVersion}&quot;<br>    implementation &quot;com.google.protobuf:protobuf-java-util:${protobufVersion}&quot;<br>    runtimeOnly &quot;io.grpc:grpc-netty-shaded:${grpcVersion}&quot;</pre><pre>    <em>//You can use the testing libraries for unit testing your gRPC services (out of scope for this article)</em><br>    testImplementation &quot;io.grpc:grpc-testing:${grpcVersion}&quot;</pre><pre>}</pre><pre>application {<br>    <em>// Define the main class for the application.</em><br>    mainClassName = &#39;grpc.demo.java.App&#39;<br>}</pre><pre>test {<br>    <em>// Use junit platform for unit tests</em><br>    useJUnitPlatform()<br>}</pre><pre><em>//This holds the configuration for the protocol buffers compilation step</em><br>protobuf {<br>    <em>// Configure the protoc executable</em><br>    protoc {<br>        <em>// Download the proto compiler from maven central</em><br>        artifact = &#39;com.google.protobuf:protoc:3.12.3&#39;<br>    }</pre><pre>    <em>//In order to get our gRPC service stubs and client/server code, we need to include the gRPC plugin for protobuf</em><br>    plugins {<br>        grpc {<br>            artifact = &quot;io.grpc:protoc-gen-grpc-java:${grpcVersion}&quot;<br>        }<br>    }</pre><pre>    <em>//This is needed to apply the gRPC plugin to protoc</em><br>    generateProtoTasks {<br>        all()*.plugins {<br>            grpc {<br>                <em>//This optional override for the output directory can be used to place generated code where you need it, in case you do not want generated code mixed with your hand-written code</em><br>                outputSubDir = &#39;java&#39;<br>            }<br>        }<br>    }<br>    <em>//This changes the base directory where your protobuf and gRPC generated Java code will be dropped</em><br>    generatedFilesBaseDir = &quot;$projectDir/src/&quot;<br>}</pre><h3>Implementing your service in Java</h3><pre>(MovieCatalogService.java)</pre><pre>package grpc.demo.java.moviecatalog.server;</pre><pre>import static io.grpc.Status.ALREADY_EXISTS;<br>import static io.grpc.Status.NOT_FOUND;</pre><pre>import grpc.demo.java.moviecatalog.MovieCatalogGrpc;<br>import grpc.demo.java.moviecatalog.MovieCatalogProto.AddMovieResponse;<br>import grpc.demo.java.moviecatalog.MovieCatalogProto.FetchMovieRequest;<br>import grpc.demo.java.moviecatalog.MovieCatalogProto.MovieItem;<br>import io.grpc.StatusException;<br>import io.grpc.stub.StreamObserver;<br>import java.util.ArrayList;<br>import java.util.List;</pre><pre><em>//We need to extend the generated gRPC Service Stubs so that we can create true implementations for them here</em><br>public class MovieCatalogService extends MovieCatalogGrpc.MovieCatalogImplBase {</pre><pre>  private final List&lt;MovieItem&gt; moviesInCatalog;</pre><pre>  public MovieCatalogService() {<br>    <em>//Populate our collection of movies available to query for. To keep things simple, we will just store data in memory.</em></pre><pre>    var schindlersList = MovieItem.newBuilder().setName(&quot;Schindler&#39;s List&quot;).setPrice(12.99d).setInStock(true).build();<br>    var piratesOfSiliconValley = MovieItem.newBuilder().setName(&quot;The Pirates of Silicon Valley&quot;).setPrice(10.99d)<br>        .setInStock(true).build();<br>    var warGames = MovieItem.newBuilder().setName(&quot;War Games&quot;).setPrice(22.99d).setInStock(true).build();<br>    var princessBride = MovieItem.newBuilder().setName(&quot;The Princess Bride&quot;).setPrice(15.99d).setInStock(true).build();<br>    var lionKing = MovieItem.newBuilder().setName(&quot;The Lion King&quot;).setPrice(24.99d).setInStock(true).build();<br>    var aKnightsTale = MovieItem.newBuilder().setName(&quot;A Knight&#39;s Tale&quot;).setPrice(8.99d).setInStock(true).build();</pre><pre>    moviesInCatalog = new ArrayList&lt;&gt;();<br>    moviesInCatalog.add(schindlersList);<br>    moviesInCatalog.add(piratesOfSiliconValley);<br>    moviesInCatalog.add(warGames);<br>    moviesInCatalog.add(princessBride);<br>    moviesInCatalog.add(lionKing);<br>    moviesInCatalog.add(aKnightsTale);<br>  }</pre><pre>  <em>//We need to override each of the protobuf `rpc` methods we defined in our `.proto` file</em><br>  @Override<br>  public void fetchExistingMovie(FetchMovieRequest request, StreamObserver&lt;MovieItem&gt; responseObserver) {<br>    MovieItem fetchedMovie;</pre><pre>    try {<br>      <em>//Grab the first movie that has a substring of the requested movie&#39;s name</em><br>      fetchedMovie = moviesInCatalog.stream()<br>          .filter(movie -&gt; movie.getName().contains(request.getName()))<br>          .findFirst()<br>          .orElseThrow(() -&gt; new MovieNotFoundException(request.getName()));</pre><pre>    } catch (MovieNotFoundException e) {<br>      e.printStackTrace();<br>      <em>//Throw an exception visible to the client (NOT_FOUND is a constant defined in the generated gRPC error handling code. We use StatusException for reporting gRPC call exceptions back over the wire.)</em><br>      responseObserver.onError(new StatusException(NOT_FOUND));<br>      return;<br>    }</pre><pre>    System.out.println(&quot;Found movie: &quot; + fetchedMovie.getName());<br>    <em>//Send response (this needs to be called in order to return the value we fetched back to the client)</em><br>    responseObserver.onNext(fetchedMovie);<br>    <em>//Close request gracefully</em><br>    responseObserver.onCompleted();<br>  }</pre><pre>  <em>//This follows basically the same flow as the previous method as far as gRPC-related things</em><br>  @Override<br>  public void saveNewMovie(MovieItem request, StreamObserver&lt;AddMovieResponse&gt; responseObserver) {</pre><pre>    boolean movieAlreadyExists = moviesInCatalog.stream()<br>        .anyMatch(movie -&gt; movie.getName().equalsIgnoreCase(request.getName()));</pre><pre>    if (movieAlreadyExists) {<br>      System.err.println(&quot;Movie &quot; + request.getName() + &quot; already exists!&quot;);<br>      responseObserver.onError(new StatusException(ALREADY_EXISTS));<br>    } else {</pre><pre>      moviesInCatalog.add(request);<br>      System.out.println(&quot;Saved movie: &quot; + request.getName());</pre><pre>      <em>//We will treat the id as just the index of the item in our in-memory list</em><br>      var response = AddMovieResponse.newBuilder()<br>          .setWasSaved(true)<br>          .setItemId(moviesInCatalog.indexOf(request))<br>          .build();</pre><pre>      responseObserver.onNext(response);<br>      responseObserver.onCompleted();</pre><pre>    }<br>  }<br>}</pre><h3>Setting up a server in Java</h3><pre>(MovieServer.java)</pre><pre>package grpc.demo.java.moviecatalog.server;</pre><pre>import io.grpc.Server;<br>import io.grpc.ServerBuilder;<br>import java.io.IOException;<br>import java.util.concurrent.TimeUnit;</pre><pre>public class MovieServer {</pre><pre>  public static void main(String[] args) throws IOException, InterruptedException {<br>    var service = new MovieCatalogService(); <em>//Instantiate our service that we defined above</em></pre><pre>    System.out.println(&quot;Starting movie server...&quot;);</pre><pre>    final int port = 9090;<br>    var server = ServerBuilder.forPort(port)<br>    .addService(service) <em>//Here is where we register the service implementation we just completed to our server</em><br>    .build()<br>    .start(); <em>//Start the server</em></pre><pre>    System.out.println(&quot;Server started on port &quot; + port + &quot;!&quot;);</pre><pre>    server.awaitTermination(); <em>//Wait until the app is killed so that our server doesn&#39;t stop immediately.</em></pre><pre>    <em>//This is some magic to make sure we don&#39;t keep the server running if the app is terminated externally</em><br>    Runtime.getRuntime().addShutdownHook(new Thread(() -&gt; {<br>      <em>// Use stderr here since the logger may have been reset by its JVM shutdown hook.</em><br>      System.err.println(&quot;*** shutting down gRPC server since JVM is shutting down&quot;);<br>      try {<br>        stopServer(server);<br>      } catch (InterruptedException e) {<br>        e.printStackTrace(System.err);<br>      }<br>      System.err.println(&quot;*** server shut down&quot;);<br>    }));</pre><pre>  }</pre><pre>  private static void stopServer(Server server) throws InterruptedException {<br>    if (server != null) {<br>      server.shutdown().awaitTermination(30, TimeUnit.SECONDS);<br>    }<br>  }<br>}</pre><h3>Setting up a client in Java</h3><pre>(MovieClient.java)</pre><pre>package grpc.demo.java.moviecatalog.client;</pre><pre>import grpc.demo.java.moviecatalog.MovieCatalogGrpc;<br>import grpc.demo.java.moviecatalog.MovieCatalogProto.AddMovieResponse;<br>import grpc.demo.java.moviecatalog.MovieCatalogProto.FetchMovieRequest;<br>import grpc.demo.java.moviecatalog.MovieCatalogProto.MovieItem;<br>import io.grpc.ManagedChannelBuilder;</pre><pre>public class MovieClient {</pre><pre>  public static void main(String[] args) {<br>    <em>//Create our client channel. Since we are running on localhost, and to keep things simple, we will not be implementing security for this demo.</em><br>    var channel = ManagedChannelBuilder<br>        .forAddress(&quot;localhost&quot;, 9090)<br>        .usePlaintext() <em>//No encryption</em><br>        .build();</pre><pre>    <em>//Create a blocking (synchronous) client to work with. This allows us to treat method calls like local calls. Alternatively, you could use a FutureStub if you wanted to use multithreading and asynchronous calls to the service.</em><br>    var client = MovieCatalogGrpc.newBlockingStub(channel);</pre><pre>    <em>//Create a movie object based on what we defined in our `proto` file to save it to the server</em><br>    var theMatrix = MovieItem.newBuilder().setName(&quot;The Matrix&quot;).setPrice(12.99).setInStock(true).build();</pre><pre>    <em>//Make the save call to the server with our movie</em><br>    AddMovieResponse saveResponse = client.saveNewMovie(theMatrix);<br>    System.out.println(&quot;Movie was saved: &quot; + saveResponse.getWasSaved() + &quot; / item id: &quot; + saveResponse.getItemId());</pre><pre>    <em>//Try to fetch the movie we just saved to the server by constructing a request object</em><br>    var fetchRequest = FetchMovieRequest.newBuilder().setName(&quot;The Matrix&quot;).build();<br>    var fetchResponse = client.fetchExistingMovie(fetchRequest);</pre><pre>    System.out.println(String.format(<br>        &quot;Fetch movie %s at price $%f. In stock?: %s&quot;,<br>        fetchResponse.getName(),<br>        fetchResponse.getPrice(),<br>        fetchResponse.getInStock()<br>    ));<br>  }<br>}</pre><h3>Generating gRPC code in .NET Core</h3><p>There are currently (as of the writing of this article) two different main implementations of gRPC for .NET. The one we will be using here is the gRPC for .NET implementation, which you can find <a href="https://github.com/grpc/grpc-dotnet">here</a>. This example is given using a .NET Core 3.0 C# app.</p><p>We will use NuGet to install the following packages (in this demo we are using version 2.29):</p><ul><li>Grpc</li><li>Grpc.AspNetCore</li><li>Grpc.Net.Client</li><li>Grpc.Net.ClientFactory</li><li>Grpc.Tools</li><li>Google.Protobuf (in this example, 3.12.3)</li></ul><pre>(grpc-demo-dotnet.csproj)</pre><pre>&lt;Project Sdk=&quot;Microsoft.NET.Sdk&quot;&gt;</pre><pre>    &lt;PropertyGroup&gt;<br>        &lt;OutputType&gt;Exe&lt;/OutputType&gt;<br>        &lt;TargetFramework&gt;netcoreapp3.0&lt;/TargetFramework&gt;<br>        &lt;RootNamespace&gt;grpc_demo_dotnet&lt;/RootNamespace&gt;<br>    &lt;/PropertyGroup&gt;</pre><pre>    &lt;ItemGroup&gt;<br>      &lt;PackageReference Include=&quot;Google.Protobuf&quot; Version=&quot;3.12.3&quot; /&gt;<br>      &lt;PackageReference Include=&quot;Grpc&quot; Version=&quot;2.29.0&quot; /&gt;<br>      &lt;PackageReference Include=&quot;Grpc.AspNetCore&quot; Version=&quot;2.29.0&quot; /&gt;<br>      &lt;PackageReference Include=&quot;Grpc.Net.Client&quot; Version=&quot;2.29.0&quot; /&gt;<br>      &lt;PackageReference Include=&quot;Grpc.Net.ClientFactory&quot; Version=&quot;2.29.0&quot; /&gt;<br>      &lt;PackageReference Include=&quot;Grpc.Tools&quot; Version=&quot;2.29.0&quot; PrivateAssets=&quot;All&quot; /&gt;<br>    &lt;/ItemGroup&gt;</pre><pre>    <em>&lt;!-- This item group needs to be added manually so that the gRPC plugins can generate the gRPC client code --&gt;</em><br>    &lt;ItemGroup&gt;<br>        &lt;Protobuf Include=&quot;Protobuf/movie-service.proto&quot; GrpcServices=&quot;MovieCatalog&quot; /&gt;<br>    &lt;/ItemGroup&gt;</pre><pre>&lt;/Project&gt;</pre><h3>Setting up a client in .NET Core (C#)</h3><pre>(Client.cs)</pre><pre>using System;<br>using Grpc.Core;<br>using Moviecatalog;</pre><pre>namespace grpc_demo_dotnet<br>{<br>    internal static class Client<br>    {<br>        private static void Main(string[] args)<br>        {<br>            Console.WriteLine(&quot;Starting gRPC client...&quot;);</pre><pre>            <em>//Create a channel with details about the server we want to connect to</em><br>            var channel = new Channel(&quot;localhost&quot;, 9090, ChannelCredentials.Insecure);<br>            <em>//Pass our channel to the client</em><br>            var client = new MovieCatalog.MovieCatalogClient(channel);</pre><pre>            <em>//Create a fetch request for a movie we know exists in the default data</em><br>            var fetchRequest = new FetchMovieRequest<br>            {<br>                Name = &quot;War Games&quot;<br>            };</pre><pre>            <em>//Call the client method like a normal method</em><br>            var response = client.FetchExistingMovie(fetchRequest);<br>            Console.WriteLine($&quot;Fetched movie from server: {response}&quot;);</pre><pre>            <em>//Construct a new movie (with strong native C# typing)</em><br>            var createRequest = new MovieItem<br>            {<br>                Name = &quot;Toy Story&quot;,<br>                Price = 29.99d,<br>                InStock = false<br>            };</pre><pre>            <em>//Save our movie like you would expect!</em><br>            var saveResponse = client.SaveNewMovie(createRequest);<br>            Console.WriteLine($&quot;Saved new movie: {saveResponse}&quot;);<br>        }<br>    }<br>}</pre><h3>Generating gRPC code in Node (JS)</h3><p>Node.js has a handy set of syntax that will dynamically generate your gRPC service code for you at runtime:</p><h3>Setting up a client in Node.js</h3><pre><em>//Boilerplate for generating gRPC code</em><br>const PROTO_PATH = &#39;../protobuf/movie-service.proto&#39;;<br>const grpc = require(&#39;grpc&#39;);<br>const protoLoader = require(&#39;@grpc/proto-loader&#39;);<br>const packageDefinition = protoLoader.loadSync(<br>    PROTO_PATH,<br>    {<br>        keepCase: true,<br>        longs: String,<br>        enums: String,<br>        defaults: true,<br>        oneofs: true<br>    });<br>const movie_proto = grpc.loadPackageDefinition(packageDefinition).moviecatalog;</pre><pre><em>//Our actual client code</em><br>function main() {<br>    <em>//Start the client</em><br>    let client = new movie_proto.MovieCatalog(&#39;localhost:9090&#39;, grpc.credentials.createInsecure());</pre><pre>    <em>//Create a movie (any unassigned values in JS will default to the protobuf defaults. Extras can be captured by the server, but are out of scope for this article)</em><br>    let terminator2 = {<br>        name: &#39;Terminator 2&#39;,<br>        price: 20.95,<br>        inStock: true<br>    };</pre><pre>    console.log(&#39;Saving movie: &#39;, terminator2);</pre><pre>    <em>//Call our save rpc and log a success or failure</em><br>    client.SaveNewMovie(terminator2, function (err, response) {<br>        if (response) console.log(&#39;Response: &#39;, response);<br>        if (err) console.error(err);<br>    });</pre><pre>    let fetchRequest = {<br>        name: &#39;The Lion King&#39;<br>    }</pre><pre>    console.log(&#39;Fetching movie: &#39;, fetchRequest);</pre><pre>    <em>//Call our fetch rpc and log a success or failure</em><br>    client.FetchExistingMovie(fetchRequest, function (err, response) {<br>        if (response) console.log(&#39;Response: &#39;, response);<br>        if (err) console.error(err);<br>    });<br>}</pre><pre><em>//Run our main method</em><br>main();</pre><h3>Let’s See It Run!</h3><h4>Java Server (starting up)</h4><pre>Starting movie server...<br>Server started on port 9090!</pre><h4>Java Client</h4><pre>Movie was saved: true / item id: 6<br>Fetch movie The Matrix at price $12.990000. In stock?: true</pre><pre>Process finished with exit code 0</pre><h4>C# Client</h4><pre>Starting gRPC client...<br>Fetched movie from server: { &quot;name&quot;: &quot;War Games&quot;, &quot;price&quot;: 22.99, &quot;inStock&quot;: true }<br>Saved new movie: { &quot;wasSaved&quot;: true, &quot;itemId&quot;: 7 }</pre><pre>Process finished with exit code 0.</pre><h4>Node Client</h4><pre>$ node index.js<br>Saving movie:  { name: &#39;Terminator 2&#39;, price: 20.95, inStock: true }<br>Fetching movie:  { name: &#39;The Lion King&#39; }<br>Response:  { name: &#39;The Lion King&#39;, price: 24.99, inStock: true }<br>Response:  { wasSaved: true, itemId: 8 }</pre><h4>Java Server (after clients have run)</h4><pre>Starting movie server...<br>Server started on port 9090!<br>Saved movie: The Matrix<br>Found movie: The Matrix<br>Found movie: War Games<br>Saved movie: Toy Story<br>Found movie: The Lion King<br>Saved movie: Terminator 2</pre><p>As you can see, each of these languages are capable of talking to a Java server as if it were regular source code in their native language. This has been a fairly simple example that doesn’t really show off some of the really neat stuff you can do, but hopefully this gives you an idea of the basics so that you can start experimenting yourself.</p><h3>What about my language of choice?</h3><p>gRPC supports many different languages, so check out the documentation to see if your language is supported, and how to get started. Since every target language converts their payloads to protobuf as part of the service call, any language can interop with any other.</p><h3>Okay, what’s the catch?</h3><h4>Infrastructure</h4><p>gRPC has a bit of an up-front cost in terms of setting up your infrastructure to use it. Your DevOps infrastructure has to support HTTP/2 and there are some tricky things to get load balancing to work properly since gRPC uses long-lived socket connections as opposed to polling. You will also need to implement a health check service and update some configurations in order for Kubernetes, or whatever other platform you’re using, to ensure the service is running properly.</p><h4>Security</h4><p>gRPC security typically requires a certificate and a private key for all clients if you want an encrypted connection. The setup for this is out of scope for this article, but the <a href="https://grpc.io/docs/guides/auth/">documentation</a> can help you a bit.</p><h4>Testing</h4><p>Since gRPC requires you to have a client library to talk to a server, manually testing requests the way you would with a REST service (via Postman or Insomnia, etc.) can be done with a tool called <a href="https://github.com/uw-labs/bloomrpc">BloomRPC</a>. Since the data is binary over the wire, it may be difficult to diagnose data issues if you can’t deserialize it, but you can test your protobufs using this tool while developing your server.</p><h4>Browser-based RPCs</h4><p>Working with gRPC in the browser is not as straight-forward. You will need to use gRPC Web to communicate with a gRPC service, which uses a client and a server-side proxy to create gRPC request on the server side. Streaming support is also fairly limited with gRPC Web as of now. See this<a href="https://grpc.io/blog/state-of-grpc-web/"> article on the gRPC blog</a> for info.</p><h3>In Conclusion</h3><p>Whether gRPC is appropriate for your team could depend on a number of things. If you are a small group of 5 developers working on a website with the same stack, the overhead of setting up gRPC may not be worth it. On the other hand, if you are a large organization that uses a lot of different tech stacks and languages that need to interact with each other, then the benefits of code generation and allowing developers to work in their native language could be worth the initial cost.</p><h3>Sources and Related Links</h3><ul><li><a href="https://grpc.io/">gRPC Documentation</a></li><li><a href="https://developers.google.com/protocol-buffers">Protocol Buffers Documentation</a></li><li><a href="https://github.com/uw-labs/bloomrpc">BloomRPC</a></li><li><a href="https://www.youtube.com/watch?v=RoXT_Rkg8LA">Intro to gRPC: A Modern Toolkit for Microservice Communication</a></li><li><a href="https://www.yonego.com/nl/why-milliseconds-matter/">Why Milliseconds Matter: Comparing REST performance with gRPC</a></li><li><a href="https://grpc.io/blog/grpc-with-json/">Using gRPC with JSON instead of Protobuf</a></li><li><a href="https://grpc.io/blog/state-of-grpc-web/">State of gRPC Web</a></li></ul><p><a href="https://redspace.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=gRPC"><em>REDspac</em></a><em>e is a software development company that builds innovative, custom, and flexible platforms for global enterprises. We combine well-built backends with engaging front-end user experiences. </em><a href="https://redspace.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=gRPC"><em>redspace.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c21054ef579c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/intro-to-grpc-and-protocol-buffers-c21054ef579c">Intro to gRPC and Protocol Buffers</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[React Suspense & Concurrent Mode]]></title>
            <link>https://medium.com/well-red/react-suspense-concurrent-mode-cb92e99d95e2?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/cb92e99d95e2</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[react-suspense]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[asynchronous]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Lauchlan Chisholm]]></dc:creator>
            <pubDate>Tue, 12 May 2020 16:50:37 GMT</pubDate>
            <atom:updated>2020-05-12T16:50:37.619Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qe5rC9YCPHP7TGlF" /><figcaption>Photo by <a href="https://unsplash.com/@arlandscape?utm_source=medium&amp;utm_medium=referral">Harald Arlander</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The React Team has been working hard on the upcoming Concurrent Mode features. What will these features mean for the React Community?</p><p>The front end of web has been overwhelmed by JavaScript Frameworks. Just as you’ve learned one, another pops up and we start to question how long this game of “whack-a-mole” can go on for. Or at least, this has been the case for the last few years.</p><p>There have been a few that have stood from the crowd — often backed by tech giants — and they’ve all got great features to offer. Google maintained <a href="https://angular.io/">Angular</a> is a fantastic tool that shines on large projects by utilizing <a href="https://www.typescriptlang.org/">typescript</a> out-of-the-box, and treating <a href="https://github.com/tc39/proposal-observable">Observables</a> as a first class citizen. <a href="https://vuejs.org/">Vue</a> — started and maintained entirely in the Open-Source community — has incredible documentation, their own video courses, and has been adopted by the PHP framework <a href="https://laravel.com/">Laravel</a> for it’s go to front-end framework.</p><p>However, this post is about <a href="https://reactjs.org/">React</a>, and more specifically its horizon. Started and maintained at Facebook, React has a history of maintaining stable releases while also pushing out game changing features. In February of 2019, the React Team made another step toward implementing their <a href="https://github.com/acdlite/react-fiber-architecture">Fiber Architecture</a> with the release of <a href="https://reactjs.org/blog/2019/02/06/react-v16.8.0.html">“hooks”</a>. This was a fundamental shift in the way React developers produced their apps, with everything moving closer to the functional paradigm.</p><p>In a strict, academic-Computer-Science sense, JavaScript doesn’t support functional programming, but it does have the <a href="https://www.bytesized.xyz/an-introduction-to-the-basic-principles-of-functional-programming-in-javascript">properties of a dynamically typed functional language</a>. And React has been steadily moving away from the object-oriented paradigm, and towards this functional approach. It’s important to note here that React doesn’t advertise itself as a framework, in fact, the heading on their homepage is “A JavaScript library for building user interfaces”.</p><p>So, React is a <strong>library</strong> for building UI, and is becoming more functional. What does that mean from the perspective of a developer? It means that every piece of your view is just the return value of a function. It means that every data value we calculate, and every piece of business logic we derive, can be the result of a function. Since functions are just JavaScript, it means React is moving closer to its tagline of being a <strong>JavaScript</strong> library. It means: <strong>If you’re good at JavaScript, you’re good at React.</strong></p><p>This leads to the meat of this article — what is on the horizon for the React Community? <strong>Asynchronous — interruptible — rendering</strong>.</p><p>Concurrent Mode is the set of new, still experimental, features being introduced to React. These features are the next logical step in the functional, JavaScript-first, approach the React Team has been taking.</p><p>Asynchronous functions have long been a part of JavaScript, but up until this time, React has only supported synchronous rendering. When we get the release of the new Concurrent Mode features, we will see another fundamental shift in the way developers write React apps. We will be able to use asynchronous functions, and promises, to drive our rendering: unresolved promises will have a fallback, and once the promise returns a value, we’ll render the component as expected. This release will also see a dramatic improvement in the way users respond to these apps.</p><p>If you’ll recall, the React Team is sponsored by Facebook, and thus has access to the research and development Facebook performs. Having the world’s two most popular social media sites as a playground provides a lot of opportunity to learn about user interaction. One of the results of this research is outlined in the <a href="https://reactjs.org/docs/concurrent-mode-intro.html">Concurrent Mode docs</a>: “displaying too many intermediate loading states when transitioning between screens makes a transition feel <em>slower”.</em></p><p>This problem will be addressed in the new Concurrent Mode features. As mentioned before, we can use Promises to drive our rendering, but the base behaviour will ensure the user doesn’t experience too many loading states, or more importantly, that the user experiences “intentional loading sequences”.</p><p>As JavaScript grows, the spirit of that growth is engendered in the community. The move toward Concurrent Mode, and React Fiber in general, is a reflection of the React Team aligning with the JavaScript community. By integrating closer with the fundamentals of JavaScript, React is continually improving the experience of development. This allows React developers to pay that experience forward in their applications and ultimately, their users. It’s an exciting time to be a part of the JavaScript community, as we move our horizon forward.</p><p>Lauchlan did a talk on React Suspense at the <a href="https://www.meetup.com/Halifax-ReactJS-Meetup">Halifax React Meetup</a> on April 21, 2020. Check out the <a href="https://www.youtube.com/watch?v=JhEIeRN4QWo">video recording here</a>, the <a href="https://docs.google.com/presentation/d/1BvMAlK2sasAu8yD3PgPwu_NJrb2rfdgPWYSQIdsE3ok/edit#slide=id.g74c554e55b_0_0">slides used here</a>, and the <a href="https://csb-c4i54-9ccbwbs0a.now.sh/">App used in the demo here</a> (<a href="https://codesandbox.io/s/react-suspense-intro-c4i54">codesandbox</a> if you want to see the code).</p><p><a href="https://redspace.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=concurrent-react"><em>REDspace</em></a><em> is a software development company that builds innovative, custom, and flexible platforms for global enterprises. We combine well-built backends with engaging front-end user experiences. </em><a href="https://redspace.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=concurrent-react"><em>redspace.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb92e99d95e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/react-suspense-concurrent-mode-cb92e99d95e2">React Suspense &amp; Concurrent Mode</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The (Not So) Big Upgrade to Drupal 9]]></title>
            <link>https://medium.com/well-red/the-not-so-big-upgrade-to-drupal-9-d1660dfd8f59?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/d1660dfd8f59</guid>
            <category><![CDATA[php]]></category>
            <category><![CDATA[composer]]></category>
            <category><![CDATA[drupal-9]]></category>
            <category><![CDATA[drupal]]></category>
            <category><![CDATA[drupal-8]]></category>
            <dc:creator><![CDATA[Kyle Cogswell]]></dc:creator>
            <pubDate>Mon, 20 Apr 2020 17:47:36 GMT</pubDate>
            <atom:updated>2020-04-21T17:07:11.766Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*YhcbIfzMRfOZBoXvurJAdQ.jpeg" /></figure><p>A new major version of Drupal is on the way! So I guess we should all start planning for the major upgrade effort early right?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3o6fIUOohyg4yegu1G%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fsport-corso-3o6fIUOohyg4yegu1G&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2F3o6fIUOohyg4yegu1G%2Fgiphy.gif%3Fcid%3D790b7611387f122ed8f6b9505bd7646b7e1408f5208bbd8a%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/1afb7dd1f2cb7305ea3ce397f49bfcca/href">https://medium.com/media/1afb7dd1f2cb7305ea3ce397f49bfcca/href</a></iframe><p>Drupal 8 introduced many changes in how we develop on the platform:</p><p>Integration of Symfony with <strong>services</strong> and the <strong>dependency injection container</strong>, a new<strong> Twig theming engine</strong>, and <strong>configuration management</strong> were just a few of the major changes we needed to learn with its release.</p><p>With this newest major version, the Drupal community is overhauling the way new releases are done. Major rewrites of modules and themes won’t be necessary this time because 9.0 will be completely backwards compatible with your existing D8 project!</p><p>Drupal 9 will essentially be the same as the last major 8.x release with only a few changes.</p><p><em>Note: Before continuing with these steps, make sure you have updated to the latest minor release of Drupal (8.8.x or later) and updated all of your contributed modules to their latest versions.</em></p><h3>Deprecations are now removed!</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FhFXwY4lER3oBO%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fjustin-timberlake-gif-singing-bye-hFXwY4lER3oBO&amp;image=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FhFXwY4lER3oBO%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="262" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d388bc0c8524f40efe25a393bf80c55/href">https://medium.com/media/4d388bc0c8524f40efe25a393bf80c55/href</a></iframe><p>The most significant change you will need to make to your project is updating any deprecated code with their new implementations — something we all should have been staying aware of anyway, right?</p><p>There are plenty of functions and services that will be going away forever, so I’ll just cover a couple of what I think would be the most common:</p><p><em>drupal_set_message () has been moved to a service:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6ayNpIP4mhn7yFhj5M60g.png" /><figcaption><a href="https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Messenger!Messenger.php/class/Messenger/9.0.x">Messenger Service API Documentation</a></figcaption></figure><p><em>entity.manager service </em><a href="https://www.drupal.org/node/2549139"><em>has been split into many different services</em></a><em>:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h9hOyD7QLZIZLcuD1drmeQ.png" /><figcaption><a href="https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Entity%21EntityTypeManagerInterface.php/interface/EntityTypeManagerInterface/9.0.x">Entity Type Manager API Documentation</a></figcaption></figure><p>Again, those are just the two most common updates that I see in my day-to-day work — but there are many more.</p><p>View the full list of deprecations at <a href="https://api.drupal.org/api/drupal/deprecated/8.9.x">https://api.drupal.org/api/drupal/deprecated/8.9.x</a>.</p><p>Yeah … the list of deprecations is over twenty pages long. That’s a lot.</p><p>How will I ever find every instance of these changes in my project?</p><h3>The Upgrade Status Module!</h3><p><a href="https://www.drupal.org/project/upgrade_status">The Upgrade Status module</a> is going to make your upgrade to Drupal 9 a breeze. A simple user interface will make it easy to find the required changes in your project.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3og0IzCY7KZAyaEGfm%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FBounce-TV-bounce-saints-and-sinners-3og0IzCY7KZAyaEGfm&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExMTk3YTBlOGRlZWU1OTg1MzMyMjY0ZWI3OTYwZmY5MjllYTA3Yzk0ZSZlcD12MV9pbnRlcm5hbF9naWZzX2dpZklkJmN0PWc%2F3og0IzCY7KZAyaEGfm%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="243" frameborder="0" scrolling="no"><a href="https://medium.com/media/82c814262a60c9053e3ecc7b997beaf0/href">https://medium.com/media/82c814262a60c9053e3ecc7b997beaf0/href</a></iframe><p>This module will scan the code, any installed contributed or custom modules in your project, then report on any deprecated code that must be replaced before the next major version release.</p><p><a href="https://www.drupal.org/project/upgrade_status">Upgrade Status</a> does require Composer to be installed, but if your site hasn’t made that jump yet (please ... for me … start using composer), you’re not out of luck.</p><p><a href="https://www.drupal.org/project/upgrade_status">Upgrade Status</a> depends on the <a href="https://github.com/mglaman/drupal-check">drupal-check analysis tool</a> so you can still get the same functionality through the command line — if necessary.</p><h3>Okay, so all of my code is updated … what now?</h3><p>Now that you’ve done all the groundwork of updating your custom code, updated all of your contributed modules, and updated to the latest minor release of Drupal 8, you should be ready for the easiest part.</p><ol><li>Put your site into Maintenance Mode and clear cache with Drush</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_uxmZfb-SMCtsbg1HWIDg.png" /></figure><p>2. Use Composer to update Drupal core and its dependencies</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pq6ywjs1vm1MrWVUu3cssA.png" /></figure><p>3. Update your database and clear cache with Drush</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YumEDmcwOFtTpAWGxIsx7g.png" /></figure><p>Voila — your project is updated to Drupal 9!</p><p>Stay tuned for more information on the new features coming over the life of Drupal 9, when it becomes available. It’s an exciting time to be a Drupal developer!</p><p><a href="https://redspace.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=durpal9"><em>REDspace</em></a><em> is a full-service digital studio specializing in web, mobile, gaming, and video solutions. We craft digital experiences that empower our clients, helping deliver their content to their audiences. </em><a href="https://redspace.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=durpal9"><em>redspace.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1660dfd8f59" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/the-not-so-big-upgrade-to-drupal-9-d1660dfd8f59">The (Not So) Big Upgrade to Drupal 9</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Git Therapy (best practises)]]></title>
            <link>https://medium.com/well-red/git-therapy-best-practises-32e928587975?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/32e928587975</guid>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Mark McKenna]]></dc:creator>
            <pubDate>Wed, 19 Feb 2020 17:34:52 GMT</pubDate>
            <atom:updated>2020-11-26T17:50:38.009Z</atom:updated>
            <content:encoded><![CDATA[<h3>Git Therapy (best practices)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aoBY9adV08PyQ-Vt" /><figcaption>Photo by <a href="https://unsplash.com/@yancymin?utm_source=medium&amp;utm_medium=referral">Yancy Min</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A lot of people use Git on a regular basis to track their software. In this blog post, I want to talk about some ways to make that experience as good as possible for everyone involved.</p><p>Here’s a quick summary of my recommendations, which I’ll explore in more detail below:</p><ul><li>Commit often</li><li>Describe your commits well</li><li>Make each commit mean something</li><li>Use small, logical pull requests</li><li>Resolve pull requests quickly</li></ul><p>I’ll be assuming here that you use a workflow that includes pull requests. If you don’t, you probably should, but most of the recommendations will still apply anyway.</p><p>Now without further ado, on to the recommendations! The easiest change I can think of that you could implement today, would be to…</p><h3>Commit Often</h3><p>The simplest way to use Git is to write a bunch of code, commit, and create a pull request. It’s not wrong; it’ll get the job done just fine. But committing (and pushing) more frequently offers a bunch of benefits:</p><ul><li>If something goes wrong with your machine, you don’t lose everything; you just lose everything since your last push.</li><li>If you write a bunch of code you decide later that you don’t want, you can easily roll back to an earlier checkpoint.</li><li>If you want to get input from another developer, you have something already available for them to look at. You can even keep going on your own machine while they read your stuff on theirs.</li><li>If you figure out that your feature was too broadly defined, you can use the commits you already wrote to help understand how to split up the work.</li><li>When people are reviewing your work, they can select individual commits and get a more chronological view of how your work evolved.</li></ul><p>I’m sure there are other reasons too. It’s just generally a good idea to keep tabs on where you’ve been; it gives everyone options.</p><p>My habit here is to commit every time I reach a logical stopping point for my task; and also every time I need to put the task down and move on to something else (like at the end of the day, or if I’m working on multiple projects).</p><p>But when you commit, what do you say? You should…</p><h3>Describe Your Commits Well</h3><p>What’s the world’s most common commit message? I bet it’s either “WIP” or “PR clean up.” Totally understandable… if you’re committing frequently and addressing pull request feedback, you’ll have a lot of commits that don’t have a well-defined purpose.</p><p>But at some point, you should find yourself reaching a threshold where you’ve written code you understand and that you feel is complete: it implements a solid portion of the feature you’re working towards. When you get there, take a minute to come up with a good description of what you wrote, and commit with that description.</p><p>There are various blog posts around that do a good job of explaining what a good commit message should look like. <a href="https://chris.beams.io/posts/git-commit/">Here’s one that I really like</a>. The upshot is to write a short one line summary of what the change is for, leave a blank line, and then write a longer description adding some more details.</p><p>You shouldn’t need to describe the change itself — that’s what clean code is for. Instead, explain <em>why</em> the change was made. The two main reasons people read your commit messages are because something went wrong and they need to know why; or because your changes were good, and they want to get them into a release. In both cases, what they need to understand is your motivations.</p><p>Unlike the other recommendations, #1 tends to make following this one a bit harder: the more you commit, the harder it is to meaningfully describe each commit, and the more time it takes to do so. A fun trick that helps here is to make sure your first commit on the branch is a nice, well-described ‘framework’ commit, outlining the intent of the change; then use the following command for work-in-progress commits:</p><pre>git commit --fixup head</pre><p>This will produce a commit that says !fixup ab9cd34 &lt;description&gt;, referring back to the last commit on the branch and repeating its summary. The idea here is that this commit doesn’t really mean anything on its own; it’s just an addendum to something meaningful you did before. Better still, if you do use tools like rebase, the resulting commit can be automatically folded into the work before, giving you the best of both worlds: frequent commits <em>and</em> meaningful commit messages.</p><p>But in order to write a meaningful commit message, you will need to…</p><h3>Make Each Commit Mean Something</h3><p>This is a bit more tricky than the others; it requires planning. It dovetails pretty well with recommendation #2. The idea here is that, ideally, your commits shouldn’t just mark off work you’ve done; they should actually encapsulate logical changes.</p><p>Say you’re writing a simple customer management system, and you’re working on a ticket to build a customer entry form. Before starting to write code, think the job through, and develop a plan of attack. Maybe something like this:</p><ol><li>Create a module containing the form (blank) and its associated functionality (not implemented)</li><li>Lay out the controls on the form with submit and cancel buttons</li><li>Implement+test submission and cancellation logic</li><li>Add in-page validation</li><li>Add animations and other bells and whistles</li><li>Bind the form into the larger customer management component</li></ol><p>Each of these steps results in something meaningful, and makes a good commit. It’s easy to describe each one in terms of why you wrote it, and they might even be separately useful — a working input form without validation can be tested at a basic level, maybe even released as part of a beta product.</p><p>Which leads into the next point…</p><h3>Use Small, Logical Pull Requests</h3><p>You don’t need to save all your changes associated with a feature, and submit them for review all at once. As long as you’ve organized them well, you will often find that a subset of changes are worth reviewing and merging back to master as a standalone component.</p><p>Notice above that we don’t actually connect our form to the rest of the system until step 6? There’s a good reason for that: none of this code will break anything in production until that happens. That makes all of the other commits totally safe to merge into master (provided your code is encapsulated properly, of course), even if they haven’t been QA’d yet.</p><p>One of the more painful parts of development on a team is resolving merge conflicts. Merge conflicts happen, generally when you and another developer make competing changes on two separate branches. Two things increase risk of conflicts: large branches, and long-lived branches. The latter can be addressed by merging or rebasing your branch against master frequently; the former can only be addressed by merging your branch back into master.</p><p>As long as your changes are bite-sized, well described, self-contained, and sandboxed, you can merge them into master as often as you want. This means branches are both small, <em>and</em> short-lived — avoiding both sources of conflict at once.</p><p>In the extreme, each commit can be a pull request. This isn’t always necessary or desirable, but it has one nice little benefit: most PR software will use the commit description as the PR description, if there is only one commit in the PR. That saves you writing the same thing twice (provided your commit messages are good, of course). Then:</p><h3>Resolve Pull Requests Quickly</h3><p>Obviously this dovetails with the above, but it reaches beyond yourself. You want your PR to get merged before conflicts happen, but you need others to approve it first; in order to get their approval you’ll need to address their feedback. At the same time, everyone else on the team wants the same thing.</p><p>This becomes a teamwork issue, then: review people’s pull requests promptly; and address their feedback on yours promptly as well. When they address <em>your</em> feedback, approve right away, and when your PR is approved, merge it right away.</p><p>You need to cultivate a relationship of respect and responsiveness within the team, which is something that requires everyone’s participation. The team lead plays a key role here obviously, but I’ve found that anyone on the team who chooses to move quickly, communicate clearly, be positive and respectful, and so on, will bring the level of the team up in general. People react to the “social tone” of the group, and one person (even the lead) can’t set that unilaterally — but every little bit will move the needle.</p><h3>The Upshot</h3><p>Once you’ve got yourself committing more often, and describing those commits better; being more logical in what commits you make, and getting them accepted and merged quickly; you should find things working out considerably better for yourself and your teammates.</p><p>These things reinforce one another, and dovetail with other good programming practises that you’re probably familiar with. Writing good commit messages requires writing good commits, which in turn pushes you to plan your features more; this encourages you to organize your thoughts and your code better. Organizing your thoughts helps you to understand what change you’re making, and why; which in turn helps you to write better commit messages, and probably make fewer mistakes along the way. Making PRs more frequently means you’ll accumulate feedback earlier in development, when it’s easier to address; and also means you’ll get better feedback, because the stuff they’re commenting on will be easier to understand.</p><p>Of course, all of this assumes a pretty clean context, where you have a good understanding of what you’re supposed to be building, well-groomed tickets, reasonable deadlines, good leadership, and so on. More often than not, this isn’t true; which means you’ll need ways to deal with this stuff after the fact.</p><p>I’ll address all of that in a future post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32e928587975" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/git-therapy-best-practises-32e928587975">Git Therapy (best practises)</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Phaser 2 and Flump Animations]]></title>
            <link>https://medium.com/well-red/phaser-2-and-flump-animations-63c4cfd0dc39?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/63c4cfd0dc39</guid>
            <category><![CDATA[phaserjs]]></category>
            <category><![CDATA[adobe-animate]]></category>
            <category><![CDATA[animation]]></category>
            <category><![CDATA[flump]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Aaron Malley]]></dc:creator>
            <pubDate>Wed, 11 Dec 2019 15:46:32 GMT</pubDate>
            <atom:updated>2019-12-11T15:47:15.579Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5AHDT764PaDIk1JlN2I4Xw.png" /></figure><h3>Introduction</h3><p>For HTML5 game development, there are a few options developers have to create animations. We use Adobe Animate (formerly known as Flash) and Flump to produce high quality keyframe-based animations. Animate provides our artists with an industry standard animation tool. Flump takes animations created in Animate and serialized timeline data to JSON or XML, and rasterize vector art into texture atlases.</p><p>Unfortunately, Flump is not very wide spread within the game development community. It is because of this that it does not have a lot of support within some of the more popular HTML5 game engines. We use Phaser 2 for our HTML5 games, which doesn’t support Flump. To solve this, we wrote a plugin for Phaser that will load and play Flump animations.</p><h3>Flump vs. Other Options</h3><p>So, why Flump? By default, Phaser 2 supports two types of animations: flipbook sprite animations, and Spine. Flipbook animations are standard sprite animations where you swap images to give the illusion of motion. Spine is a skeletal animation tool that allows animators to animate individual parts of a character by attaching them to bones.</p><p>We use Flump instead of flipbook animations because of quality and memory consumption. Flump provides smooth television quality animations that use less texture memory.</p><p>When it comes to choosing Flump over Spine, it came down to cost and workflow. Spine is an excellent animation tool and popular within game development. You can get a similar level of quality with Spine as you can in Adobe Animate; and it is natively supported in Phaser. So why use Flump over Spine? Simple, Spine is expensive. It is free to use within Phaser 2, however the Spine animation tools have extremely high licensing fees for medium to large businesses. The license costs increase based on company revenue. While Adobe Animate does have license fee, it is far cheaper for individuals and companies that are already using Adobe Creative Cloud.</p><p>We also do a lot of work with world-class media brands, and often adapt television animations to game content. Being able to directly use TV animations created with Adobe Animate in our games allows for a more authentic experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gq-17P_9UY56NY8aExm_vg.png" /><figcaption>Example of a character setup for Flump in Adobe Animate.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*Dm4mvM0fvfEVtjTqluBqKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*cEQaXgGOwusGTUrJOXr6uQ.png" /><figcaption><em>Example of the character previewed in the Flump exporter.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*XWT-6ki6rWzm3ww9jKin5Q.png" /><figcaption><em>Exported Flump atlas of the character.</em></figcaption></figure><h3>Technical Breakdown</h3><h3>The original runtime</h3><p>The plugin we wrote is based off the runtime written for the Flump exporter tool. Flump is open source and is licensed under the MIT license. This meant we were free to use what we needed from Flump to get animations working in Phaser, however, it wasn’t a straight port. Since Flump is written in ActionScript 3 for the Starling framework, bringing it over to Phaser 2 meant integrating Flump so it worked fluidly with the engine.</p><h3>Asset Loading</h3><p>Our first step was implementing a custom loader that we could use with Phaser’s load manager. We wanted to be able to use the standard Phaser pipeline to load our Flump assets. With our new Flump atlas loader, we can call `game.load.flumpAtlas(…)`.</p><pre>preload() {<br>    game.load.flumpAtlas(“demoLibrary”, “assets/library.json”, “assets/atlas0.png”);<br>}</pre><p>To load the Flump assets, we provide a key that we will use to identify those assets within the cache object. In this case, our key is “demoLibrary”. The loader will store the `library.json` data using the key, and add the key as a prefix to the image file name. In our example, the file “assets/atlas0.png” is stored as “demoLibrary/atlas0.png”. We need to prefix the library key onto our images because the Flump exporter does not allow us to change the naming scheme it uses when exporting the atlas images. Without the prefixed library key, we would run into naming collisions if the assets for more than one library were loaded.</p><h3>Library Creation</h3><p>After we have the Flump assets loading into Phaser’s asset cache, we initialize a Flump library to use its resources. A Flump library is a collection of textures and movie symbols that were exported from Animate. Each symbol is unique to the library it belongs to, and this was something we wanted to make sure remained the case within the plugin.</p><pre>create() {<br>    game.flump.add(“demoLibrary”);<br>}</pre><p>When a library is added to the plugin, it uses the key we provide to pull all the resources it needs from Phaser’s asset cache. This includes “library.json” and any atlas images that were prefixed by our custom loader. Once the data has been retrieved, it will parse “library.json” and create the frame data for each atlas image. With the atlas images paired with frame data, the Flump atlas can be used as if it were any other image atlas.</p><pre>game.image.add(0, 0, “demoLibrary/atlas0.png”, “orangeSquare”);</pre><h3>Porting the Animation Player</h3><p>Having our Flump assets integrated into the standard Phaser workflow allowed us to easily begin porting over the animation player from the Flump exporter.</p><p>We started by creating a generic Symbol object. A symbol is any object that Flump uses to play an animation. This includes empty placeholder objects, textures, and movies. In practice, our Symbol is just an extended Phaser Image. This allows us to assign any symbol texture data from a Flump atlas. It also allows pooling of texture symbols when they are not being used by a movie.</p><p>When Flump exports an animation with rotations, it converts the rotation values into a scale and skew. This means that the plugin needed support for skewing images. Our Symbol object allowed us to add skewing support into Phaser.</p><p>Symbol pooling is one of the optimizations we added. We wanted the ability to reuse a movie symbol to play different animations, which meant that when changing animations some symbols would no longer be needed. Instead of allocating and deallocating symbols, we pool and reuse them by assigning new texture data to it. Any symbol not being used remains in the pool until one is needed.</p><h3>Frame Labels</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*jvQMJSS_1pc_wZk3qkB-mQ.png" /></figure><p>One of the more interesting features of Flump is the ability to export frame labels from the timeline in Animate. A frame label is just a string given to a keyframe. This is useful because it allows developers to listen for frame labels to trigger in-game events. For example, if our animators create a cutscene animation, they can use frame labels to tell the game when it should begin playing voice over audio or sound effects. Frame labels can also be used in code to tell an animation to play until it reaches some label, or to go to and stop at a specific label. This was an important feature to make sure we carried over into the plugin.</p><pre>movie.labelEvents.add((label) =&gt; {<br>    if (label === “playSfx”) {<br>        sound.play();<br>    }<br>});</pre><p>With our Symbol object and symbol pooling in place, we were able to port the Flump runtime over into Phaser. Overall, it went smoothly. We came across a couple bugs that we fixed, and we did not run into any major issues. After everything was done, we had a Flump animation plugin for Phaser 2 that accurately played content created in Adobe Animate.</p><h3>Speed Control</h3><p>After the Flump implementation was complete, we found that we wanted to have a greater level of control over playback. While animations did play and animate at the correct frame rate, we wanted the ability to control the playback speed of animations. This would allow us to speed up and slow animations down based on game logic. Adding the feature was not difficult, because it only required scaling the delta time used to advance an animation.</p><p>This did introduce an issue when animations were slowed down. They looked clunky; as if the game was running at a lower frame rate even though it was still running at 60 fps. When Flump advances an animation, it rounds its frame position. It does this so it does not run into errors when pulling keyframes from an array. The issue this causes is that when it came to calculating the interpolation between one frame to the next, it did so in whole steps. So frame one to frame two, two to three, and so on. This is good for figuring out which keyframe data to use, however, it does not work for scaling the speed of an animation. To fix this, we allowed the interpolation to use fractions of frames without allowing the animation to play further than the keyframe it is currently on. So if our last frame was one and our next frame was two, then if game time determined we are actually at frame position 1.5, then the animation would use 1.5 to interpolate between the two frames. We still used whole numbers to determine keyframe positions, however now animations can use fraction of frames to interpolate. The result is far smoother animations. They still animate at the frame rate that was set in Adobe Animate, however they play smoother in our plugin. These smoother animations allowed us to scale the playback speed without slower speeds looking clunky.</p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*7tDgsYnT5DwXl4MFsuOnlQ.gif" /><figcaption><em>A GIF of the Flump mascot playing in Phaser 2</em></figcaption></figure><p>After we were able to smooth out animations, the plugin was complete. We were able to create a Flump player for Phaser 2 that allows developers to reuse movies for multiple animations as well as implemented optimizations and improvements.</p><p>You can find the source for the Phaser 2 Flump plugin on GitHub <a href="https://github.com/theREDspace/phaser2-flump-plugin">here</a>. The plugin is also available on NPM <a href="https://www.npmjs.com/package/@redspace/phaser2-flump-plugin">here</a>. We also provide a demo project that shows examples of how to use the plugin <a href="https://github.com/theREDspace/phaser2-flump-plugin-demo">here</a>.</p><h3>What’s Next?</h3><p>We are currently bringing this plugin over to Phaser 3. Although the Phaser 2 port of Flump was straight forward, Phaser 3 does not support the ability to skew an image. Since Phaser 2 is built on top of PIXI, we were able to easily use Flump’s scale and skew value to calculate a matrix to figure out the rotation of an object. In Phaser 3, the matrix calculations have been decoupled from the renderers and game objects making Flump integration more difficult. When we finally do port Flump to Phaser 3, we want to make sure it is seamlessly integrated into the engine to ensure maximum stability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63c4cfd0dc39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/phaser-2-and-flump-animations-63c4cfd0dc39">Phaser 2 and Flump Animations</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Github actions + Fastlane + iOS = ❤]]></title>
            <link>https://medium.com/well-red/github-actions-fastlane-ios-1f6d43cce726?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/1f6d43cce726</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[fastlane]]></category>
            <dc:creator><![CDATA[TJ Gillis]]></dc:creator>
            <pubDate>Thu, 07 Nov 2019 17:09:32 GMT</pubDate>
            <atom:updated>2019-11-07T17:09:31.935Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ah9IBHyl9mlxtUu-zN6cog.png" /></figure><p>There is a lot to cover here so we will jump right in by looking at what our ideal build process would be:</p><ul><li>xcodegen project¹</li><li>fastlane build/sign</li><li>upload to hockey app²</li></ul><p>Straightforward, eh?</p><p>Let’s see what this looks like when we put this into an action workflow.</p><pre>name: CI<br> <br>on: [push]<br> <br>jobs:<br>  build:<br>    runs-on: macOS-latest<br>    steps:<br>    - uses: actions/checkout@v1<br>    - name: Install and run xcodegen<br>      run: |<br>        brew install xcodegen<br>        xcodegen generate<br>    - name: Run fastlane setup<br>      env:<br>        APPLE_ACCOUNT: ${{ secrets.APPLE_ACCOUNT }}<br>        TEAM_ID: ${{ secrets.TEAM_ID }}<br>      run: |<br>        fastlane setup --verbose<br>    - name: Run fastlane build<br>      env:<br>        MATCH_REPO: ${{ secrets.MATCH_REPO }}<br>        MATCH_PASSWORD: ${{ secrets.MATCH_PASS }}<br>        SIGNING_ID: ${{ secrets.SIGNING_ID }}<br>        APPLE_ACCOUNT: ${{ secrets.APPLE_ACCOUNT }}<br>        TEAM_ID: ${{ secrets.TEAM_ID }}<br>      run: |<br>        fastlane compile --verbose<br>    - name: Run fastlane deploy<br>      env:<br>        HOCKEY_TOKEN: ${{ secrets.HOCKEY_TOKEN}}<br>        APPLE_ACCOUNT: ${{ secrets.APPLE_ACCOUNT }}<br>        TEAM_ID: ${{ secrets.TEAM_ID }}<br>      run: fastlane deploy --verbose</pre><p>That’s a clean build script. Let’s step through it and see what’s happening here.</p><pre>name: CI<br>on: [push]</pre><p>Here we have a name and an on field. Name is exactly what it sounds like. This is the name of your workflow. The next field is more interesting. This is an array of different webhooks you can listen to for when to run the job. For now, we’re just interested in push but <a href="https://help.github.com/en/articles/workflow-syntax-for-github-actions#on">here</a> are the docs if your case requires others.</p><pre>jobs:<br>  build:<br>    runs-on: macOS-latest</pre><p>Following this, we have jobs and runs-on. <a href="https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobs">Jobs</a> are what they sound like. A unit of work. Here, our job is called build. Within that build job we can choose what OS we want to work on by setting runs-on. See <a href="https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idruns-on">here</a> for the different platforms but in our case, we’re attempting to build for iOS so macOS it is.</p><pre>steps:<br>- uses: actions/checkout@v1</pre><p>Next, we have our steps. The first step is mandatory if you want to actually check out the code from your repo. You can read more about it <a href="https://help.github.com/en/articles/configuring-a-workflow#using-the-checkout-action">here</a> but essentially without it you aren’t able to perform any other tasks, which would make this exercise rather pointless.</p><pre>- name: Install and run xcodegen<br>  run: |<br>    brew install xcodegen<br>    xcodegen generate</pre><p>Now we’re starting to see our code performing what we described at the top as our ideal process.</p><p>This here is a step named “Install and run xcodegen”. Luckily, the macOS image comes pre-installed with homebrew so we will use that to install xcodegen and generate a project with it.</p><pre>- name: Run fastlane setup<br>  env:<br>    APPLE_ACCOUNT: ${{ secrets.APPLE_ACCOUNT }}<br>    TEAM_ID: ${{ secrets.TEAM_ID }}<br>  run: <em>|<br>    </em>fastlane setup --verbose</pre><p>Our next step, “Run fastlane setup”, is another element that is concisely named. But this is where things get tricky, at least for me. For match and fastlane to work how I wanted, I required access to the keychain. This is not something that we seem to have permissions to when using actions.</p><p>Thankfully, there is a fastlane action (this is not the same as a Github action) that will create a keychain and set it as the default, which makes up the content of our setup_project lane.</p><pre>lane :setup do<br>  setup_project<br>end<br><br>private_lane :setup_project do<br>  create_keychain(<br>    name: &quot;actiontest_keychain&quot;,<br>    password: &quot;meow&quot;,<br>    default_keychain: true,<br>    unlock: true,<br>    timeout: 3600,<br>    lock_when_sleeps: false<br>  )<br>end</pre><p>Above we use <a href="https://docs.fastlane.tools/actions/create_keychain/">create_keychain</a> to generate a new keychain that we’ve called actiontest_keychain, with a password of meow.</p><p>Now that we have our keychain made, let’s download our certs and build the app.</p><pre>- name: Run fastlane build<br>  env:<br>    MATCH_REPO: ${{ secrets.MATCH_REPO }}<br>    MATCH_PASSWORD: ${{ secrets.MATCH_PASS }}<br>    SIGNING_ID: ${{ secrets.SIGNING_ID }}<br>    APPLE_ACCOUNT: ${{ secrets.APPLE_ACCOUNT }}<br>    TEAM_ID: ${{ secrets.TEAM_ID }}<br>  run: <em>|<br>    </em>fastlane compile --verbose</pre><p>Here you see an example of environment vars and secrets. Secrets are in your GitHub repo settings and allow for removing passwords and other sensitive info from source code. Very useful. But Fastlane does not understand how to access secrets, so we need to assign it to an environment var so we can ensure access.</p><pre>private_lane :build do<br>match(<br>  type: &quot;appstore&quot;,<br>  readonly: is_ci,<br>  keychain_name: &quot;actiontest_keychain&quot;,<br>  keychain_password: &quot;meow&quot;<br>)<br><br>update_project_provisioning(<br>  xcodeproj: ENV[&quot;XCODE_PROJ&quot;],<br>  profile: ENV[&quot;sigh_com.redspace.actionstest_appstore_profile-path&quot;],<br>target_filter: &quot;actionstest&quot;, <br>  build_configuration: &quot;Release&quot;,<br>  code_signing_identity: &quot;iPhone Distribution: REDspace Inc.&quot;<br>)<br><br>build_app(<br>  scheme: &quot;actionstest&quot;,<br>  project: ENV[&quot;XCODE_PROJ&quot;],<br>)<br><br>end</pre><p>Our build lane is simple for how much trouble it gave me.</p><p>First, we run our fastlane match action. The special part of this match is that we are telling it what keychain to add the certs to. Another thing to keep in mind is at the moment it is difficult to check out repos on another host (such as Stash) with ssl. Your best bet would be to use http check out.</p><p>Since we’re not using automatic code signing, we must run the update_project_provisioning fastlane action with our profile, target, config, and signing identity. In terminal match will print out a list of environment variables that it has created and within them you’ll find an environment variable with your profile already set.</p><p>Finally, call build_app.</p><p>After that’s completed, we have one final step, which is to upload to hockey app.</p><pre>- name: Run fastlane deploy<br>  env:<br>    HOCKEY_TOKEN: ${{ secrets.HOCKEY_TOKEN}}<br>    APPLE_ACCOUNT: ${{ secrets.APPLE_ACCOUNT }}<br>    TEAM_ID: ${{ secrets.TEAM_ID }}<br>  run: fastlane deploy --verbose</pre><pre>content_copy<br>private_lane :hockey_upload do<br>  hockey(<br>    api_token: ENV[&quot;HOCKEY_TOKEN&quot;],<br>    create_status: &quot;2&quot;,<br>    ipa: &quot;actionstest.ipa&quot;,<br>    notes: ENV[&quot;RELEASENOTES&quot;]<br>  )<br>end</pre><p>No secret sauce here.</p><p>Congratulations! You’ve created an action capable of generating, signing, building, and deploying your iOS project. Yay!</p><p>If you’d like to see this is as completed project we have made an example available on our <a href="https://github.com/theREDspace/github_actions_ios_example">GitHub</a>.</p><p>Footnotes:</p><p>¹ <a href="https://github.com/yonaskolb/XcodeGen">XcodeGen</a> is a command line tool written in Swift that generates your Xcode project using your folder structure and a project spec.</p><p>² <a href="https://www.hockeyapp.net/">HockeyApp</a> is a service that allows developers to recruit and manage testers, distribute apps, and collect crash reports, among other things.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f6d43cce726" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/github-actions-fastlane-ios-1f6d43cce726">Github actions + Fastlane + iOS = ❤</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Preliminary analysis on IMDB dataset with Python]]></title>
            <link>https://medium.com/well-red/preliminary-analysis-on-imdb-dataset-with-python-c34df19fe6ed?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/c34df19fe6ed</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[inside-ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Reza Rajabi]]></dc:creator>
            <pubDate>Tue, 30 Jul 2019 12:35:57 GMT</pubDate>
            <atom:updated>2019-07-30T12:35:57.477Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*p3CHKjeSgW4IauU_pf-4hA.jpeg" /><figcaption><a href="https://easieraccounting.com/mid-year-financial-analysis-far/">Picture reference</a></figcaption></figure><h3>A Preliminary Analysis with Python</h3><h4>Understand your data before applying the machine learning techniques</h4><p>In the <a href="https://medium.com/well-red/cleaning-a-messy-dataset-using-python-7d7ab0bf199b">previous article</a>, I showed how we can clean a messy dataset (I used a subset of IMDB movies dataset). The next phase of the cleaning process is analyzing data with the hope of reaching your goal or even discovery. In fact, the type of analysis that you want to do highly depends on the goal of your analysis. Do you want to classify the data or just cluster them? Is your data labeled or not? Do you want to predict a value in your data or just perform a statistical analysis? There are some other factors that you should take into your consideration including number of features (columns) your dataset has, the size of your dataset, type of features, and so on.</p><h3>Running a Python program in the cloud</h3><p>This may not relate to the analysis, but for running a Python program at any time and at any place you need to have your code on a cloud. What about running your program on Google cloud? If you have a Google account, you just need to create your notebook in <a href="https://colab.research.google.com/">Colaboratory</a>. It is just like a Jupyter notebook, but in a cloud. It is beneficial to run your examples or your simple projects on the cloud, however, note that the program is not running on your high speed local machine. Another thing, installing python libraries on a cloud is a bit different from installing it on a local Jupyter notebook, as it requires a few additional steps or sometimes adding additional commands! All in all, it is GOOD to run your code in Colaboratory when:</p><ul><li>you want to share your Python code with someone else</li><li>you want to run a code at any time on any machine</li><li>your projects and data are allowed to be published on a cloud (privacy!)</li></ul><p>Another plus of running your code on a cloud service is having access to your data on the cloud. For example, if you want to read a CSV file, you just need to copy it on your Google drive. Although an authentication step is needed to access your Google drive, it is worthwhile. As you can see in the figure below, you need to mount your drive to access the main folder of your Google drive. The only thing you should do is authorize Colaboratory to access your Google drive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*8s5HPYy4lHQo0zn3WeSIMQ.jpeg" /><figcaption>Google authentication for accessing google drive</figcaption></figure><p>Then you can access your CSV file!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*vv8FIiaIm9uR6_qqgRnSug.png" /><figcaption>Access to data in Google drive</figcaption></figure><h3>Dataset analysis</h3><p>First of all, let’s see what we have in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*NGfko1cd5ExCx0YNEHWVQw.png" /><figcaption>dataset columns</figcaption></figure><p>It is a sample of IMDB dataset including a movie’s director name, duration, gross amount, genre, title, the year it was produced, the country, budget, its IMDB score, number of its Facebook likes, the actors’ names (only first three main actors), and its GOB (this is gross over budget metric that we created in the previous article).</p><h3>Analysis questions</h3><p>To perform a preliminary analysis, we should outline a set of questions that we want to answer as a result of our analysis.</p><blockquote>Note: Our data is a subset of IMDB dataset. The reports show the result on this sample, not all the movies!</blockquote><p>You can comment on this article, if you have more interesting questions. My questions are:</p><ul><li>Who are the directors of the top GOB movies?</li><li>Which year had the highest GOB?</li><li>Can we cluster the movies based on their GOB/IMDB scores?</li><li>Is there any relationship between IMDB score and facebook likes?</li></ul><h3>Who are the directors of the top GOB movies?</h3><p>To get the highest GOB scores, first, we sort the dataset based on GOB descending. In the following code, I get the top 15 highest GOB scores and store it in top_GOB dataframe.</p><pre>top_GOB = dataset_imdb.sort_values(‘GOB’,ascending=False).head(15)</pre><p>As we have the directors’ full names, I add another column to shorten their names to better visualize them in a graph. To do that, I just take the second part of the name, which might be their family name.</p><pre>top_GOB[&#39;director_familyName&#39;] = dataset_imdb[&quot;director_name&quot;].str.split(&quot; &quot;, n = 2, expand = True) [1]</pre><p>The next step is simply visualizing them in a simple bar chart, as follows:</p><pre>fig,ax = plt.subplots(figsize=(7, 5))</pre><pre># Draw a bar graph<br>ax = sns.barplot(x=”director_familyName”, y=”GOB”, data=top_GOB,ci=None)</pre><pre># Rotate the directors&#39; name 45 degrees <br>ax.set_xticklabels(ax.get_xticklabels(), rotation=45)</pre><pre># Title the graph<br>fig.suptitle(‘Top movie directors with highest GOB’, fontsize=12)</pre><pre># Set font size of axis label<br>ax.set_xlabel(&#39;Direcot name&#39;,fontsize=20)<br>ax.set_ylabel(&#39;Gross over Budget&#39;,fontsize=20)</pre><pre># Set tick size of axis <br>ax.tick_params(axis=&#39;x&#39;, labelsize=15)<br>ax.tick_params(axis=&#39;y&#39;, labelsize=15)</pre><pre># Show the graph<br>plt.show();</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*2zfdrYLw0PSxFKoIfqpBfA.png" /><figcaption>Top GOB movie directors</figcaption></figure><h3>Which year had the highest GOB?</h3><p>The next report is getting the years with high gross over budget scores for all the movies. To do that, we should group the records based on title_year. Pivot table in python is used to group columns and aggregate them based on numeric columns, as shown in the following code.</p><pre># Group data based on movie year<br>dataset_pivot=dataset_imdb.pivot_table(index=[&#39;title_year&#39;], <br>                      values=[&#39;GOB&#39;], aggfunc=np.mean ,margins=True)</pre><p>The year column in the dataset includes some missing values, we will ignore it.</p><pre># Some cells are empty in year column<br>dataset_pivot =dataset_pivot.loc[dataset_pivot[&#39;title_year&#39;]&gt;0]</pre><p>Finally, we will draw a bar chart that simply shows the year and average value of GOB for each year.</p><pre># We do not need last record which includes the summary metric<br>dataset_pivot = dataset_pivot[:-1]</pre><pre># As index changed after creating a pivot table, we reset the index<br>dataset_pivot.reset_index(inplace=True)</pre><pre># Some cells are empty in year column<br>dataset_pivot =dataset_pivot.loc[dataset_pivot[&#39;title_year&#39;]&gt;0]</pre><pre>fig,ax = plt.subplots(figsize=(9, 8))<br>ax=sns.barplot(x=&quot;title_year&quot;, y=&quot;GOB&quot;, data=dataset_pivot)</pre><pre>fig.suptitle(&#39;GOB over the years&#39;, fontsize=12)<br>plt.xlabel(&#39;Movie year&#39;)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*1-3K28O8tEsMoHPNpu3rsw.png" /><figcaption>Top GOB over the year</figcaption></figure><h3>Can we cluster the movies based on their GOB/ IMDB scores?</h3><p>As our data is not labeled,(meaning that they are not in different categories or classes) we can use an unsupervised learning algorithm to cluster them. We would like to know if the machine can find groups or categories of data for us, instead of adding the labels by a user? To this light, we can choose a clustering algorithm to find groups of data. the algorithm we use, depends on several factors including the nature of data, data size, number and types of features in our dataset, etc. I use the K-Means algorithm,one of the popular clustering algorithms, that works iteratively to assign each data point to one of <em>K</em> groups based on the features that are provided. Below, you can see how this algorithm works in the figure. Also, you can find more information about clustering at <a href="https://www.datascience.com/blog/k-means-clustering">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*pwCAVEzTCi22Ihto234ZlA.gif" /><figcaption>K-Means clustering algorithm (picture from <a href="https://commons.wikimedia.org/wiki/File:K-means_convergence.gif">here</a>)</figcaption></figure><p>In our dataset, we are looking for a relation between a GOB and IMDB score to see whether or not we can cluster movies based on these two metrics. First, we need to specify the number of clusters we are looking for. I chose three. If you visualize your data in a scatter plot, you may guess about the number of groups you might get from the algorithm, but if not, do not worry. You can change the number of clusters afterwards and run the algorithm again to see if you get a better result or not! I use a simple scatter plot in Python to visualize these two metrics (the GOB and IMDB score) on a graph.</p><pre>plt.scatter(x=dataset[&#39;imdb_score&#39;], y=dataset[&#39;GOB&#39;])<br>plt.show()</pre><p>And get this as the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*lR3fZuDJNallgLEUpNm6aw.png" /><figcaption>IMDB score and GOB in a scatter plot</figcaption></figure><p>Now, I ask the algorithm to cluster my points into three groups.</p><pre># Exclude the missing values and selecy only GOB and IMDB score<br>selected_dataset=dataset.loc[<br>(dataset[&#39;GOB&#39;]&gt;0) &amp; dataset[&#39;imdb_score&#39;]&gt;0][[&#39;<strong>imdb_score</strong>&#39;,&#39;<strong>GOB</strong>&#39;]]</pre><pre># Clustering the dataset usinig K-Means algorithm<br>cls = KMeans(n_clusters=<strong>3</strong>)</pre><pre># Fit the model into the algorithm<br>cls.fit(selected_dataset)</pre><pre># Give me the center point and label of each group<br>centroids=cls.cluster_centers_<br>labels = cls.labels_</pre><p>As the next step, I would like to color the points based on their groups and specify the center of each group with a marker (x).</p><pre># We have three cluster, one color for each cluster<br>colors = [&quot;g.&quot;,&quot;r.&quot;,&quot;b.&quot;]</pre><pre># Create an array of each dataset to traverse over the points<br>dataset_array = np.array(selected_dataset)</pre><pre># ploting the point on a scatter plot<br>for i in range(len(dataset_array)):<br>    plt.plot(dataset_array[i][0], <br>dataset_array[i][1], colors [labels[i]], markersize =25);</pre><pre><br>plt.scatter(centroids[:,0], centroids[:,1], marker=&#39;x&#39;, s=150);</pre><p>The output looks like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*w0qUQsQAsk-aWOkTxlj2bg.png" /><figcaption>K-Means clustering algorithm visualization</figcaption></figure><p>We see three clusters in the figure. First cluster (blue), is the movies with the low GOB score (not sold well) and low IMDB score (not ranked well by IMDB users). The second cluster includes the movies with a very good IMDB score, but not very high GOB score. The third cluster includes few movies with IMDB scores between 7 and 8 and very high GOB. Like I said, this is a subset of movies, you can have a better interpretation when you apply the algorithm over all the movies.</p><h3>Is there any relationship between IMDB score and Facebook likes of movies?</h3><p>To find a relationship between an IMDB score and Facebook likes, we can use regression algorithm. A simple Linear Regression can be used to model the relationship between a single input independent variable (in our case Facebook likes) and an output variable (e.g., IMDB score) using a linear model i.e a line. This is just an example. In reality we may not find a linear relationship between two variables in a dataset. Find more information about the regression algorithms <a href="https://towardsdatascience.com/selecting-the-best-machine-learning-algorithm-for-your-regression-problem-20c330bad4ef">here</a>.</p><p>First of all, we make the Facebook likes in a low scale, as they all have more than 10,000 likes, to better visualize the number in the graph.</p><pre>selected_dataset[&#39;movie_facebook_likes&#39;]=dataset[&#39;movie_facebook_likes&#39;].apply(lambda row: row/10000)</pre><p>Then we specify the X and Y axis to be used the regression graph:</p><pre>X = selected_dataset[&#39;budget&#39;].values[:,np.newaxis]<br>y = selected_dataset[&#39;movie_facebook_likes&#39;].values</pre><p><a href="https://seaborn.pydata.org/">Seaborn library</a> has a regression plot, which is a very nice and powerful tool to plot a linear regression graph. You just need to specify the X and Y axis along with the actual dataset, as followed:</p><pre>ax=sns.regplot(&#39;imdb_score&#39;, &#39;movie_facebook_likes&#39;,  selected_dataset);</pre><p>Then we set the other configurations to make the graph more readable and fancy:</p><pre>fig.suptitle(&#39; Relation between IMDB score and facebook likes &#39;, fontsize=15)<br># Set x-axis label<br>plt.xlabel(&#39;IMDB score&#39;);<br># Set y-axis label<br>plt.ylabel(&#39;Facebook like (10K)&#39;);</pre><pre>fig.set_size_inches(9,8)<br>font = {&#39;family&#39; : &#39;Arial&#39;,<br>        &#39;size&#39;   : 14}<br>plt.rc(&#39;font&#39;, **font);<br>plt.show();</pre><p>And get this as the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*g8Jk4D684ms-qusOpWABgg.png" /><figcaption>Linear regression using Seaborn regplot</figcaption></figure><p>You can set the graph setting in the function to change the marker’s shape (marker=&#39;+&#39;) or change the color with color=’g’or change the markers’ size with scatter_kws={&#39;s&#39;:100}. If you do not have a large dataset, we can specify ci in the configuration as well. ci<em> </em>is <em>s</em>ize of the confidence interval for the regression estimate, and it will be drawn using translucent bands around the regression line. We can also set fit_reg=True to estimate and plot a regression model relating the x and y variables. There are other parameters in <a href="https://seaborn.pydata.org/generated/seaborn.regplot.html">the documentation</a> that you can set, based on the type of analysis you want.</p><pre>ax=sns.regplot(&#39;imdb_score&#39;, &#39;movie_facebook_likes&#39;,  selected_dataset, ci=70, scatter_kws={&quot;s&quot;:100}, order=2, fit_reg=True,color=&quot;g&quot;);</pre><p>After using the Python code above, I redrew the graph, and this is what I got:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*tIc1sbDBvueN5OQAXbkWvQ.png" /><figcaption>linear regression between IMDB score and Facebook likes</figcaption></figure><p>Based on this graph, there is a linear relationship between movies’ IMDB score and their Facebook likes, which means a higher IMDB score results higher Facebook likes.</p><p>Having said that, we can use a simple linear regression algorithm to predict Facebook likes of a movie based on its IMDB score. In the following code, a movie with IMDB score 8.3 has an estimation of 111,000 likes.</p><pre># Assign imdb_score to X as one of our variables<br>X = selected_dataset[&#39;imdb_score&#39;].values[:,np.newaxis]</pre><pre># Assign Facebook likes to Y<br>y = selected_dataset[&#39;movie_facebook_likes&#39;].values</pre><pre># Specify the type of mode<br>model = LinearRegression()</pre><pre># Fit the model with X and y<br>model.fit(X, y)</pre><pre># IMDB score input for our prediction<br>IMDB_score=8.3<br>X_test=np.array([IMDB_score]).reshape(1,-1)</pre><pre># make a prediction based on linear regression<br>y_predict = model.predict(X_test)</pre><pre>print(&#39;Facebook likes estimation (10K):&#39;,y_predict)</pre><pre>fig,ax = plt.subplots(figsize=(9, 8))</pre><pre># Set x-axis label<br>plt.xlabel(&#39;IMDB score&#39;);<br># Set y-axis label<br>plt.ylabel(&#39;Facebook like (10K)&#39;);</pre><pre># Visualize data and the linear regression line<br>plt.scatter(X, y,color=&#39;b&#39;)<br>plt.plot(X, model.predict(X),color=&#39;g&#39;)</pre><pre># Plot the prediction point with a color and size<br>plt.scatter(X_test, y_predict, color=&#39;r&#39;, s=100) <br>plt.show()</pre><p>The scatter plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*hnucN-rtCR6LaRv1SGoGmA.png" /><figcaption>Linear regression with prediction</figcaption></figure><p>I’m wrapping up the second part of article here. The type of analysis depends strongly on what goal(s) you want to achieve. Setting the goal before preforming the actual analysis is crucial. In any case, I always start with a preliminary analysis using some visualizations such as a scatter plot, bar chart or line graph over my dataset to see what is going on my data. If you do not have any goals and want to publish a paper or start a new project, this approach gives you an idea about WHAT you can do with your dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c34df19fe6ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/preliminary-analysis-on-imdb-dataset-with-python-c34df19fe6ed">Preliminary analysis on IMDB dataset with Python</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cleaning a messy dataset using Python]]></title>
            <link>https://medium.com/well-red/cleaning-a-messy-dataset-using-python-7d7ab0bf199b?source=rss----f81bf41391f2---4</link>
            <guid isPermaLink="false">https://medium.com/p/7d7ab0bf199b</guid>
            <category><![CDATA[data-cleansing]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Reza Rajabi]]></dc:creator>
            <pubDate>Fri, 14 Jun 2019 13:01:35 GMT</pubDate>
            <atom:updated>2020-01-30T23:41:02.158Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*gKPsu_jnY8TRb74iZRzs-A.jpeg" /></figure><p>According to a survey conducted by <a href="https://www.figure-eight.com/">Figure Eight</a> in 2016, almost 60% of Data Scientists’ time is spent on cleaning and organizing data. You can find the survey results at <a href="https://visit.figure-eight.com/rs/416-ZBE-142/images/CrowdFlower_DataScienceReport_2016.pdf">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*d3GTouqzjqQN3gYHBXvwyg.png" /><figcaption><a href="https://visit.figure-eight.com/rs/416-ZBE-142/images/CrowdFlower_DataScienceReport_2016.pdf">Spent time by data scientists in 2016 survey</a></figcaption></figure><p>Data in real world comes from different sources and can be incomplete, noisy, and inconsistent. Dirty data can be produced as a result of human errors, integration of separated systems, or changing requirements. Poor data quality can lead to <strong><em>inaccurate results </em></strong>in data analytics and drive misguided decision making. In this post, I intend to go through a data cleaning process to prepare a messy dataset for the final analysis.</p><p>The following figure depicts a set of IMDB records, that I downloaded from IMDB data source at <a href="https://www.imdb.com/interfaces/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0BJpvWssNjhaQEAXl0nZHQ.jpeg" /><figcaption>A sample messy dataset</figcaption></figure><p>As you can see, there might be different types of errors in a dataset:</p><ul><li><strong>Outliers: </strong>outliers are the numbers in a column that are out of range. In fact, an outlier lies outside of most of the other values in a dataset (red cells in the figure).</li><li><strong>Duplicates: </strong>A dataset can contain repetitive rows or records (purple rows in the figure).</li><li><strong>Missing values: </strong>We might lose data as the result of human error or missing information (orange cells in the figure).</li><li><strong>Bad character or NULL values: </strong>Some values may contain bad characters like <strong><em>₮ </em></strong>or<strong><em> NULL </em></strong><em>values. Sometimes null data may be specified with different values like </em><strong><em>N/A</em></strong><em> or </em><strong><em>NA </em></strong><em>or </em><strong><em>NAN </em></strong>(light or dark cells in the figure).</li></ul><h3><strong>Data cleaning</strong></h3><p>Data cleaning is a scientific process to explore and analyze data, handle the errors, standardize data, normalize data, and finally validate it against the actual and original dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*6A5kcvlPEVNnnwLsC99zBg.png" /><figcaption>Data cleaning tasks</figcaption></figure><h4>Sample dataset</h4><p>To perform data cleaning, I selected a subset of 100 records from IMDB movie dataset. It included around 20 attributes, which was reduced to 12 for our analysis. The dataset is available at <a href="https://github.com/erajabi/Python_examples/blob/master/movie_sample_dataset.csv">here</a>. I used Python for the analysis, as a powerful, flexible, and open-source language. Python has a set of libraries for data manipulation, analysis and visualization. However, there are other data analytics tool that you can choose for data cleaning such as: <em>Tableau, R, QlickView, SAP, Excel, Apache SPARK</em>, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t3kOFe-bogoZ1tlqkxSAEQ.jpeg" /><figcaption>Data cleaning process</figcaption></figure><h4><strong>Data exploration</strong></h4><p>First step in data cleaning is understanding data by exploring the dataset and its attributes. The type of analysis might be different for each attribute type. The following table shows 12 attributes we consider for data cleaning with their data types. I categorized the attributes into different groups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*gQGBTCalrT1K3QBk0X8mTA.png" /><figcaption>dataset’ attributes</figcaption></figure><ul><li><strong>Nominal: </strong>an attribute can be categorical like color that can contain a set of restricted values (e.g., blue,red, brown, ... )</li><li><strong>Text:</strong> an attribute can be a free text or string.</li><li><strong>Numeric: </strong>an attribute can be numeric (e.g.,currency, scores).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*neao78z2BpAI7ajETxJUTg.png" /><figcaption>Missing or repetitive values in columns</figcaption></figure><h4><strong>Handling errors</strong></h4><p>Based on the type of error, we choose a specific strategy to handle it. If we have missing or repetitive values for one attribute we may decide to drop the column from our analysis. Based on experience, if more than 60% of a column contain <em>null </em>values, we can drop that column for our analysis. This is true for repetitive values of one column. If all the values of a column are the same, it is definitely useless. In our example, all the movies in our selected dataset is in English language, that can be dropped for our analysis.</p><p>Duplicate records also mislead us to perform a precise analysis. Let’s assume that we have the following numbers: 4,6,8,10 . The average of these numbers 7 . However, if we wrongly add another 8 to this set (4,6,8,8,10), the average will be 7.2 which is not equal to the previous average. Removing the duplicate records in a dataset is an essential step in an analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yH1LPjztJA0CIFwJXAVFMg.png" /><figcaption>Duplicate records</figcaption></figure><p>Identifying invalid outliers in our dataset is also very important, as it impacts on the analysis’ result. There are different ways to identify the outliers in a dataset. One is simply finding minimum and maximum of the numbers. Another approach is visualizing the numeric data in a box-plot to see if there is an outlier in the data or not. Some outliers might be valid and should be precisely treated in the dataset (like health data that may include some numbers out of range and show a specific behavior of a medical test result). Filtering out invalid outliers or filling them with a default or appropriate value are two other solutions to face this issue in data analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*eUBl4n3NstsnvTNeeorHtQ.png" /><figcaption>Outliers can be depicted with box-plots</figcaption></figure><h4><strong>Standardize values</strong></h4><p>Another data cleaning step is making data standard. First of all, data should be placed in the right column and values are in correct data type. Also, the values in columns should be unified. For example, if there is an attribute in <em>datetime</em> format, all the values in that column should be unified in same format ( e.g., YYYY-MM-dd ). Text values should be also unified, and bad character within a text should be identified and fixed. For example, if we have a country column, all canada, CANADA, and Canada should be converted to Canada or canada.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*BXgwF70PXZyc6lhnUGIpdA.png" /></figure><h4>Normalize values</h4><p>To minimize anomaly, avoid data modification issues, and simplify queries, we should normalize data in a dataset. An attribute should contain atomic values and NOT a combination of several values. In our example, there are three actors are stored in one column. To normalize this column, we should split them up into three attributes (e.g., Actor1, Actor2, and Actor3).</p><p>Another step towards data normalization can be adding new metric(s) to a dataset. For example, we might want to add a metric named gross over budget to the table to have a calculated metric for our final analysis.</p><h4>Data validation</h4><p>After cleaning data, the final dataset should be compared to the original dataset to assure data accuracy. This step is essential, as we want to know that whether or not we lose data as the result of data cleaning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7AFfg9qlftM2zyrsZsg1Gg.png" /></figure><h3>Python code for data cleaning our example</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94b8cb1c353fab08834098a4d53dd20b/href">https://medium.com/media/94b8cb1c353fab08834098a4d53dd20b/href</a></iframe><p><strong>Read CSV file in python</strong></p><p>In the following line, we read an IMDB sub-dataset using read_csv command.</p><pre>dataset = pd.read_csv(‘movie_sample_dataset.csv’, encoding=’utf-8&#39;)</pre><p>First of all, we drop the useless columns:</p><pre>dataset.drop([‘color’,’language’], axis=1, inplace=True)</pre><p>In the next step, we fill up the empty values of two numeric columns: gross and budget with 0.</p><pre>dataset[‘gross’].fillna(0, inplace=True)<br>dataset[‘budget’].fillna(0, inplace=True)</pre><p>Then, we unify the values for other columns like country, director name, or movie title:</p><pre># Uppercase all the country values<br>dataset[&#39;country&#39;]=dataset[&#39;country&#39;].str.upper()<br>dataset[&#39;country&#39;] = np.where(dataset[&#39;country&#39;]==&#39;UNITED STATES&#39;,&#39;USA&#39;, dataset[&#39;country&#39;])</pre><pre># Bad data entry<br>dataset[&#39;director_name&#39;] = np.where(dataset[&#39;director_name&#39;]==&#39;N/A&#39;,&#39;&#39;, dataset[&#39;director_name&#39;])<br>dataset[&#39;director_name&#39;] = np.where(dataset[&#39;director_name&#39;]==&#39;Nan&#39;,&#39;&#39;, dataset[&#39;director_name&#39;])<br>dataset[&#39;director_name&#39;] = np.where(dataset[&#39;director_name&#39;]==&#39;Null&#39;,&#39;&#39;, dataset[&#39;director_name&#39;])<br>dataset[&#39;movie_title&#39;] = dataset[&#39;movie_title&#39;].str.replace(&#39;Â&#39;, &#39;&#39;)</pre><p>As I mentioned above, one of the solutions to handle the outliers, is filling the correspondent rows with appropriate numbers. For example, movie duration in our dataset cannot be less than 10 or more than 300 minutes. For those cases, we fill up the cells with zero values.</p><pre>dataset[&#39;duration&#39;] = np.where(dataset[&#39;duration&#39;]&lt;=10,0, dataset[&#39;duration&#39;])<br>dataset[&#39;duration&#39;] = np.where(dataset[&#39;duration&#39;]&gt;300,0, dataset[&#39;duration&#39;])<br>dataset[&#39;imdb_score&#39;] = np.where(dataset[&#39;imdb_score&#39;]&lt;=0,0, dataset[&#39;imdb_score&#39;])<br>dataset[&#39;title_year&#39;] = np.where(dataset[&#39;title_year&#39;]&lt;2010,0, dataset[&#39;title_year&#39;])</pre><p>To normalize data, we split up actors into three different attributes and drop the actual column as follows:</p><pre>actor_list = dataset[&quot;actors&quot;].str.split(&quot;,&quot;, n = 2, expand = True) <br>dataset[&quot;actor1&quot;]= actor_list[0] <br>dataset[&quot;actor2&quot;]= actor_list[1] <br>dataset[&quot;actor3&quot;]= actor_list[2] <br>dataset.drop(columns=[&#39;actors&#39;], inplace=True)</pre><p>And finally, we define a new metric called GOB that shows gross over budget for each movie:</p><pre># Add a new metric GOB(Gross over Budget)<br>dataset[&#39;GOB&#39;] = dataset.apply(lambda row: row[&#39;gross&#39;]/row[&#39;budget&#39;] if row[&#39;budget&#39;]!=0 else 0, axis=1)<br>top_GOB=dataset.sort_values(&#39;GOB&#39;,ascending=False).head(15)</pre><p>and save the result in another csv file to validate:</p><pre>dataset.to_csv(&#39;output_IMDB.csv&#39;)</pre><p>Thank you for reading this article! I will probably write another article about the analyzing as well as visualizing this clean dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d7ab0bf199b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/well-red/cleaning-a-messy-dataset-using-python-7d7ab0bf199b">Cleaning a messy dataset using Python</a> was originally published in <a href="https://medium.com/well-red">Well Red</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>