<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Code Chalet - Medium]]></title>
        <description><![CDATA[Design patterns and implementations for production software. - Medium]]></description>
        <link>https://medium.com/code-chalet?source=rss----db59dc0dfce5---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Code Chalet - Medium</title>
            <link>https://medium.com/code-chalet?source=rss----db59dc0dfce5---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:52:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/code-chalet" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Key Value Stores]]></title>
            <link>https://medium.com/code-chalet/key-value-stores-a5148047d586?source=rss----db59dc0dfce5---4</link>
            <guid isPermaLink="false">https://medium.com/p/a5148047d586</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Ben Johnson]]></dc:creator>
            <pubDate>Fri, 25 Feb 2022 20:03:27 GMT</pubDate>
            <atom:updated>2022-02-25T20:03:14.934Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*iYEXAA6OgLYrFu4X-O_Bsg.png" /></figure><p>Databases are generally divided into two categories : relational databases, and non-relational databases. Relational databases were invented in 1970 to model financial transactions [1]. A relational database (RDB) organizes data into rows identified by primary keys. Tables can reference records in other tables, using their primary keys. A primary key embedded in a foreign table is called a foreign key.</p><p>RDBs can model complex systems in the real world, including business transactions, analytics and scientific experiments. One of the advantages of RDBs is the ease of joining records in one table with records in another. Consider the data model represented by the diagram below — the `Users` table can be compressed by storing the name of the country and continent in a different table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*x0LvXmGSVBMtjfdgcvesKg.png" /></figure><p>If relational databases are so versatile, what are non-relational databases and why are they useful? To answer this question, let’s start by revisting a well-known data structure.</p><h3>Hash Tables</h3><p><a href="http://codechalet.com/docs/algorithms/arrays/hash-tables.html">Hash tables</a> are a data structure consisting of a key and a value. A hash table users a hash function which accepts a key as an input, and returns the address in memory where the value is stored. Compared to an array, a hash table has a better lookup time because computing the hash function, and accessing a pointer both take constant time O(1).</p><p>The most basic implementation of a key value store is a hash table. For example, you might have a data store, <strong>users</strong>, which stores user information. Each user is uniquely identified by a key, <strong>user_id.</strong> The hash function maps the key <strong>user_id</strong> to a block of bytes in memory. The block of bytes is used to store the value, in this case, the name of the user.</p><p>Now let’s say you want to overwrite the names of one of the users. At first glance, updating the block of memory directly seems like the most sensible option — you could look up the user with a <strong>user_id = 1</strong> and replace the name which is stored at that address.</p><p>However, this approach has several disadvantages. Consider the case when a user tries to update the database, and the database crashes during the middle of the write. The blocks in memory in this case will contain part of the old value, and part of the new value. Concurrency also becomes difficult — how does the database handle the case when two users try to update the same value at the same time? [3]</p><p>Databases often solve this problem using transaction logs. Write operations are appended to a log, and records which appear later in the log take precedence. If two users update the same value, the transaction log will contain two records with updates to that value.</p><h3>SSTables</h3><p>SSTables are a special implementation of transaction logs. Originally introduced by Google [2], SSTables are a core component of non-relational DBs including <strong>Google BigTable</strong>, <strong>Amazon DynamoDB</strong> and <strong>Apache Cassandra</strong>.</p><p>An SSTable is a segment of a transaction log which is sorted by key. This data structure offers several performance advantages over an unsorted log. First, it is much faster to combine sorted log segments as opposed to unsorted segments. The combination algorithm is similar to the <a href="http://www.codechalet.com/docs/algorithms/sorting/merge-sort.html">merge sort algorithm</a>. Second, since the keys are sorted you can store a sparse index and infer the location of keys which are not contained within the index.</p><p>Now that we have taken a look at the data structures which power non-relational databases, let’s answer a different and perhaps more important question. Why do people use NoSQL databases?</p><h3>Replication</h3><p>Let’s say your company has gotten a lot of traction in Europe, and you need to reduce the time of requests which are made by the European user base. In order to support low latency, you want requests in Europe to be routed to a data center in Europe. You need to <strong>replicate</strong> your data across different machines.</p><p>You might start by writing a script to copy data from your data center in North America to your data center in Europe. However, consider the following scenario : a user in North America and a user in Europe try to update the same piece of data, at the same time. How can you replicate a database, and guarantee consistency?</p><p>The long answer is that there is always a tradeoff between consistency and availability. On the one hand, you can have a totally consistent system, meaning clients always read the same values when accessing the database at the same time. On the other, you can have a totally available system meaning that the system is available for all clients in the event of a node outage.</p><p><strong>Dynamo</strong> and <strong>Cassandra</strong> implement eventual consistency — clients will always have access to the system, but clients may read different values when accessing the DB at the same time [4]. For most applications, this tradeoff is acceptable. One notable exception might be financial applications where it is critical that all users have access to consistent data.</p><p>Returning to our original question — non-relational databases are useful when you need to replicate data across a distributed system. If you were to implement replication with a SQL database you would need to write code which enforces a consistency model. Fortunately, there are plenty of NoSQL databases which have already solved this problem.</p><h3>Sharding</h3><p>Let’s say we are building Youtube, and we need to create a data store for user videos. To start, you might store all the videos in a single database <strong>videos</strong>, which is hosted on a single machine. As you scale your user base, you will quickly run out of storage space on your machine.</p><p>To solve the problem, you purchase <strong>n</strong> machines, partition the <strong>videos</strong> database into <strong>n</strong> segments, and store one segment on each machine. This works for a while, but what happens when one of the machines goes down?</p><p>Now, you start replicating each partition across 3 different machines. When a node experiences an outage, you can reference one of the backups. This preserves the availability of data stored in your system. Handling node outages causes the system to grow in complexity — we also need nodes to be aware of which nodes are alive and which are dead.</p><p>Some of these distributed system problems are solved by non-relational databases. To solve the sharding problem with a relational database, you might need to write code which coordinates communication between nodes. Fortunately, distributed communication is already implemented in databases like <strong>BigTable</strong> and <strong>Cassandra.</strong></p><h3>Conclusion</h3><p>We have examined the data structures which power non-relational databases — specifically, hash tables and SSTables. We took a look at some problems which engineers face when building a distributed data store, and also some of the reasons an engineer might choose to use a non-relational database. This article hardly scratches the surface — I encourage any interested reader to continue their research!</p><h3>References</h3><p>[1] E.F. Codd. A Relational Model of Data for Large Shared Data Banks.<br>[2] Google. BigTable, A Distributed Storage System for Structured Data.<br>[3] Martin Kleppmann. Designing Data Intensive Applications, Chapter 3.<br>[4] Alex Wu. System Design Interview.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5148047d586" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-chalet/key-value-stores-a5148047d586">Key Value Stores</a> was originally published in <a href="https://medium.com/code-chalet">Code Chalet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Heap Sort in Python]]></title>
            <link>https://medium.com/code-chalet/heap-sort-in-python-1ea19433819c?source=rss----db59dc0dfce5---4</link>
            <guid isPermaLink="false">https://medium.com/p/1ea19433819c</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[sorting-algorithms]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[heapsort]]></category>
            <dc:creator><![CDATA[Ben Johnson]]></dc:creator>
            <pubDate>Sun, 09 Jan 2022 22:38:33 GMT</pubDate>
            <atom:updated>2022-01-09T22:37:31.624Z</atom:updated>
            <content:encoded><![CDATA[<p>A <em>heap</em> is a complete binary tree, where each node is less than or equal to its parent. The binary tree below is a heap, because each parent node is greater than or equal to both of its child nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*LBfUsVBjZ16bce5pWA1YIQ.jpeg" /></figure><p>Heaps can be represented as arrays, where the position of each array element corresponds to its position in the binary tree. Tree nodes are labeled from top to bottom, and left to right. For example, the third element of the heap pictured above is 35.</p><p>We can represent a heap in Python as a list. Because the heap is a binary tree, the position of the parent node is related to the position of its children. The children nodes of any parent can be found using the index of the parent. If the parent has index <em>i</em>, the left and right children can be found at indices 2<em>i</em>+1 and 2<em>i</em>+2.</p><p>Convince yourself of this relation using the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c573237639b714fd87a3ab5f9c0861a/href">https://medium.com/media/4c573237639b714fd87a3ab5f9c0861a/href</a></iframe><h3>Heapify</h3><p>The first algorithm we will build is the <strong>heapify </strong>algorithm. Heapify floats a node which is out of place, down the heap. In order for heapify to work properly, the selected node must be the only node which is out-of-place in the heap. More specifically, the two child trees of the selected node must both be heaps in order for <strong>heapify</strong> to work.</p><p>Inspect the binary tree below, and you will see that node 2, which has a value of 26, violates the heap property, since its value is less than the values of its children. Let’s build an algorithm to float 26 down and restore the heap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Bq1-2B9qQ0zmCeXdpiu34g.jpeg" /></figure><p><strong>Heapify </strong>will move an out-of-place node down the tree by comparing the node to its children. If the selected node is greater than its children, <strong>heapify</strong> will exchange the node with its child.</p><p>The algorithm iterates down the tree using a recursive call onto itself. The following is an implementation of the <strong>heapify</strong> algorithm, as defined in <a href="https://en.wikipedia.org/wiki/Introduction_to_Algorithms">Introduction to Algorithms</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d6904194c4cf18491bd6d5a94e789e2/href">https://medium.com/media/1d6904194c4cf18491bd6d5a94e789e2/href</a></iframe><p>The heapify function has two steps - a computation, and a recursive call to itself. The computation has runtime <strong>O</strong>(1), since their execution does not depend on the input size. In the worst case, the recursion operates on a tree of size 2<em>n</em>/3. Combine the two, and we get the upper bound on the execution time of heapify.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*cIFItUr8Gup7TnPnYyEz7g.png" /></figure><p>Soving for the recursion, we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*bdjxz4F-BBaHsSmBS23qHA.png" /></figure><h3>Heap Build</h3><p>Heapify can float one element down an existing heap, but how can we create a heap from scratch? If we called heapify on the <em>A[n/2] </em>element, we could construct a heap with the three elements <em>A[n/2]</em>, <em>A[n-1], A[n].</em></p><p>Now, we have a 3 element heap, and we can call heapify on <em>A[n/2–1]. </em>In this manner we can apply heapify in a bottom up manner so that the child trees are guaranteed to be heaps before heapify is run at a node.</p><p>We don’t need to run the algorithm on the set of nodes with no children, since these nodes are already one-element heaps. The last element of a binary tree which has children, is the<em> n</em>/2 element. So, we only need to call heapify on the set of elements with children.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*2Ngltm5cySIznLt43jF5-A.png" /></figure><p>Starting with the <em>A</em>[<em>n</em>/2] element, we will call heapify <em>n</em>/2 times. Therefore, the upper bound on the execution time of heap build is the product of <em>n</em>/2 and the execution time of heapify , which we solved for above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*iJmlaCmsUOQNOfnfiTnS_g.png" /></figure><p>Let’s neglect the constant factor and simplify.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/206/1*jbhf2rwWGFjssn9B3HUaMw.png" /></figure><h3>Heap Sort</h3><p>Finally, we can write the algorithm we set out to write. <strong>Heap sort</strong> will accept an unsorted array, build a heap, and convert the heap to a sorted array.</p><p>In a heap, the maximum element is stored at the root node. However, in a sorted array, the maximum element will be the last element of the array. Let’s remove the first element of the heap and place it at the end of the array. We will also reduce the heap size by 1, so that the last element in the array is not included in the heap.</p><p>If we remove the first element of the heap, we need to replace it with another element to retain the shape of the binary tree. What if we replaced it with the last element of the heap? The root node of the heap would violate the heap property, but all other nodes would be heap-like. Therefore, we only need to call heapify to float the root node down the heap!</p><p>In this manner we can reduce the size of the heap by replacing the first element, and calling heapify to restore the heap. Ater (<em>n</em>−1) iterations, we will reduce the size of the heap to 0.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e9186bc5ce49d2dd3808c53153a4613/href">https://medium.com/media/6e9186bc5ce49d2dd3808c53153a4613/href</a></iframe><p>The upper bound on heap_sort can be shown to be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/206/1*yxwdxY1l73JPdt74cnWrOw.png" /></figure><p>The original article can be viewed on <a href="http://codechalet.com/docs/algorithms/sort/heap-sort.html">Codechalet.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ea19433819c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-chalet/heap-sort-in-python-1ea19433819c">Heap Sort in Python</a> was originally published in <a href="https://medium.com/code-chalet">Code Chalet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML architecture in PySpark for rapid development]]></title>
            <link>https://medium.com/code-chalet/ml-architecture-in-pyspark-for-rapid-development-3338e285afdc?source=rss----db59dc0dfce5---4</link>
            <guid isPermaLink="false">https://medium.com/p/3338e285afdc</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Ben Johnson]]></dc:creator>
            <pubDate>Fri, 03 Dec 2021 20:25:56 GMT</pubDate>
            <atom:updated>2021-12-03T20:25:23.658Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*tMIohUNl_5BVb6WffQjEMg.png" /></figure><p>If you have developed an ML pipeline to execute within a data warehouse, you might have spent more than a few hours waiting for your code to execute. One of the challenges of developing programs which execute on massive datasets is the latency between executing a program and receiving feedback. I can’t count the number of times I have executed a program, only to realize hours later that there was some bug in the code.</p><p>To solve this problem, I developed an architecture which allows ML pipelines to execute on local datasets, as well as datasets in the cloud. The flexible architecture allows me to iterate quickly when I am developing, while still executing my code in production against datasets in the cloud.</p><p>The crux of the concept is using Pandas to execute the pipeline on local data, and using PySpark to execute the pipeline on data in the cloud. Both Pandas and PySpark include groupBy functions which implement the <a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html">split-apply-combine</a> concept. The functions have slightly different implementations in the two languages, which leads to the first design constraint.</p><h3>Adapters</h3><p>Since the Pandas <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.groupby.html">groupBy</a> function has a different interface than the PySpark <a href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.groupBy.html">groupBy</a> function, I used the <a href="https://refactoring.guru/design-patterns/adapter">adapter</a> design pattern to build two functions which execute the ML pipeline. In this way, the ML pipeline can remain consistent while adapting to the specific interface requirements of each framework.</p><p>The <strong>spark_pipeline_adapter</strong> executes the pipeline via Spark, and <strong>pandas_pipeline_adapter</strong> executes the pipeline via Pandas. The Spark analysis runs against data stored in the data warehouse, and the Pandas analysis runs against data stored locally on the machine. The Pandas adapter can be used during development, and the Spark adapter can be used in production.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f0311189f4f157c74a89b4c1cd9adb3/href">https://medium.com/media/6f0311189f4f157c74a89b4c1cd9adb3/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b79872801a6451cc5dff047bb51ed01a/href">https://medium.com/media/b79872801a6451cc5dff047bb51ed01a/href</a></iframe><h3>Pipelines</h3><p>The pipeline itself will be specific to your use case. It could be a regression, an ML model, or any sequence of transformations with a set of input parameters, and a set of output values. I factored my pipeline to fit the functional programming paradigm, where an operation can be expressed as a function which accepts input values and returns output values.</p><h3>Models</h3><p>The pipeline is performed on an individual partition of data, which is split from the main dataset as part of the split-apply-combine framework. I wanted to generalize the pipelines so they can be performed on all of the partitions in our data warehouse. To generalize the <strong>pipeline </strong>described in the previous section, we need to enforce some properties of the input parameters. For example, you might have a threshold for usage data completeness — if the partition is missing some amount of data, you may want to avoid executing the pipeline.</p><p>To enforce properties on the partition, I created a class to represent the partition. The partition could be a User, a House, or any other object. I found it helpful to construct the class using a builder pattern — the class is first instantiated, and then loaded with data. The builder pattern allows us to perform data validation on each of the class attributes, without overloading the constructor. Tldr; using a builder pattern will lead to cleaner code.</p><h3>Implementation</h3><p>The <strong>pandas_pipeline_adapter</strong> function will look like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef581fd260a4042290434e9a2365ce0c/href">https://medium.com/media/ef581fd260a4042290434e9a2365ce0c/href</a></iframe><p>The <strong>spark_pipeline_adapter</strong> function will be similar, with some slight modifications to the interface:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce7975d0f6bd51f7e07990e6b693df93/href">https://medium.com/media/ce7975d0f6bd51f7e07990e6b693df93/href</a></iframe><p>Notice that parameters which are the same across an entire partition (such as timezone) are broadcast across the entire partition. It is up to the adapter function to interpret broadcast variables by de-duplicating them — see lines 5–7.</p><p>Also notice the difference in the return value of each function. The groupBy function in PySpark expects a dataframe to be returned from the aggregation function, while the groupBy function in Pandas expects a series.</p><p>Hopefully this helps! This architecture has certainly helped me to cut down on development time, while ensuring that code will be compatible with both local data and data stored in the cloud.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3338e285afdc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-chalet/ml-architecture-in-pyspark-for-rapid-development-3338e285afdc">ML architecture in PySpark for rapid development</a> was originally published in <a href="https://medium.com/code-chalet">Code Chalet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayesian Bicycles and Cold Weather]]></title>
            <link>https://medium.com/code-chalet/bicycles-in-cold-weather-13fb62282f26?source=rss----db59dc0dfce5---4</link>
            <guid isPermaLink="false">https://medium.com/p/13fb62282f26</guid>
            <category><![CDATA[bayesian]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Ben Johnson]]></dc:creator>
            <pubDate>Wed, 10 Nov 2021 20:12:05 GMT</pubDate>
            <atom:updated>2021-11-10T20:15:23.375Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*DvxPM3R-R2fZgNcOjMkX8A.png" /></figure><p>Are people less likely to ride bicycles during a stint of cold weather? This analysis sets out to answer that question, using a data from the <a href="https://www.capitalbikeshare.com/system-data">Capital Bikeshare</a> system in Washington DC. The operators of the Capital Bikeshare system have published their system data for research, and we are going to use a small sample of that data collected during 2011 and 2012. The dataset is available on the <a href="https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset">UCI website</a>.</p><p>The hourly datset includes 17,379 observations, representing approximately 8760 hourly observations recorded over a 2 year period. Each observation records the timestamp, normalized temperature and a count of total rental bikes. I will focus on these features to keep our model simple.</p><p>My hypothesis is that temperature can be used to predict the count of rental bikes. When the temperature increases, the count of rental bikes will also increase. This hypothesis is motivated by my own experience as a bike rider who does not particularly enjoy biking in cold weather.</p><h3>Linear Model</h3><p>I will use a linear Bayesian model to represent the relationship between an observed temperature <strong><em>t_i</em></strong>, and and observed bike count <strong><em>x_i</em></strong>. Notice that each of our parameters is defined as a stochastic distribution — this is a feature of Bayesian analyses. The only parameter which is not defined as a distribution is <strong><em>μ</em></strong>, and that is because <strong><em>μ</em></strong> is defined entirely in terms of other stochastic parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*PbL5_EArRsB75WkBZxNE-w.png" /></figure><h3>Likelihood</h3><p>The first equation in the model is the <strong>likelihood function</strong> — it is the probability of the data <em>xi</em>, given our prior beliefs. I constructed the likelihood function as a normal distribution, because I did not have a good reason to use a different type of distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*H5AL4SRF6Xs1Ek5XewbUew.png" /></figure><p>As Richard McElreath points out in his book <em>Statistical Rethinking</em>, the normal distribution is a good first order approximation to many stochastic processes because of the <a href="https://en.wikipedia.org/wiki/Central_limit_theorem">central limit theorem</a>, which states that observations of independent random variables tend towards a normal distribution.</p><p>Our likelihood function is defined by two parameters, the mean <strong><em>μ</em></strong> and the variance <strong><em>σ</em></strong>. The parameters <strong><em>μ</em></strong> and <strong><em>σ</em></strong> are not observed, and we will need to construct a prior belief about their distribution. First, let’s define <strong><em>μ</em></strong> in terms of the other observable parameter, temperature. This relation is our <strong>linear model</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*YtrhdlEkHmC_bm2iffvXNQ.png" /></figure><p>Now we have three unobserved parameters <strong>(<em>α</em>,<em>β</em>,<em>σ</em>)</strong>, in addition to our three observable parameters <strong>(<em>x_i</em>,<em>t_i</em>,<em>t_b</em>ar)</strong>. Notice I did not include<strong> <em>μ</em></strong> in the list of unobserved parameters, because <strong><em>μ</em> </strong>is a joint distribution, completely defined by the other parameters.</p><h3>Priors</h3><p>Let’s construct <strong>prior distributions</strong> for each of the unobserved parameters. This step requires some intuition, so we will think it through. The plots are generated using code which I have <a href="https://github.com/bdjohnson529/statistics/blob/master/bike_sharing/scripts/prior-prediction.R">uploaded to Github</a>.</p><p>A good place to start is by thinking about extreme values. When the observed temperature <strong><em>t_i</em></strong> equals the average observed temperature <strong><em>t_bar</em></strong>, the equation for <strong><em>μ</em></strong> will simplify.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*DPVSPmXBcU52_XLEiGPR2w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*02YD6kMAZbHJhLe2aalQ1A.png" /></figure><p>Let’s try to understand the rate of change between normalized temperature <em>ti</em> and bike counts <em>xi</em>. The maximum temperature ever recorded in Washington D.C. is 41 Celsius, and the minimum is -21 Celsius. This gives us a maximum range of 62 Celsius. The normalized temperature ranges from 0 to 1. One-tenth the normalized temperature range represents approximately 6.2 degrees.</p><p>Let’s take a look at rate of change of our linear equation. Suppose a 6.2 degree difference resulted in 100 additional riders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*Bvkvxi4o3ZshU-NzddtEOQ.png" /></figure><p>We will take a more conservative estimate, and define <em>β</em> as a normal distribution centered around 500, with a variance of 200.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*s66xfw1DajDaZdUs_sL8Bg.png" /></figure><p>Finally, we need to construct a prior for the variance of the likelihood function. Variance must be positive, so we can once again bound it at zero. Recall that our bike counts range from 1 to 1000. I will use a uniform distribution between 0 and 200 as the initial prior, so that the mean plus or minus 2 standard deviations covers 80% of the observed range of bike counts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*NEXJfG7dVhI9HXjuQrbFQA.png" /></figure><h3>Prior Prediction</h3><p>Now let’s sample from the likelihood function, and see how our model predicts bike counts. To produce the prior prediction, I will take 100 samples from <strong><em>α</em></strong> and <strong><em>β</em></strong>, substitute the observed mean in the dataset for <strong><em>t_bar</em></strong>, and plot each of the 100 lines which are predicted by our likelihood function. The R code to produce this plot can be <a href="https://github.com/bdjohnson529/statistics/blob/master/bike_sharing/scripts/prior-prediction.R">found on Github.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*mIdacAnaUO0cm56rS1FrWw.png" /></figure><p>Our prior prediction does cover the range of the observed values from 1 to 1000. It could certainly be improved by eliminating the priors which predict negative values for <strong><em>x_i</em>,</strong> since negative bike counts are impossible. For now, we will accept the weaknesses of our priors and move on to the next step of our analysis.</p><h3>Posterior Distribution</h3><p>I will use the quadratic approximation to find the posterior distribution. The quadratic approximation approximates the posterior as a Gaussian distribution, and finds the peak and curvature near the peak.</p><p>The figure below displays several different pieces of information. The blue points represent the observed data, collected by the Capital Bikeshare System. The solid black line represents the average predicted count, at each temperature value. The shaded region represents the 89% prediction interval, calculated at each temperature value.</p><p>The code which generates this plot can be <a href="https://github.com/bdjohnson529/statistics/blob/master/bike_sharing/scripts/posterior-quap.R">found on Github.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*zUp_5DSSGzyCx9Xnii4yzA.png" /></figure><h3>Model Evaluation</h3><p>Our model is entirely deficient in some ways, and useful in other ways. The first weakness of the model is its failure to describe bike counts about 600. The 89% prediction interval does not include the largest bike counts, and fails to describe a large portion of the data.</p><p>The model does do a good job describing the lower bound of bike counts when it is warm outside. The 89% prediction interval excludes the bottom right of the graph, which represents low bike counts when the temperature is high. The data also does not include any values in this region, which indicates that our model accurately represents this aspect of the data we have collected. Of course, the sample we collected may not represent the world at large.</p><p>By examining the data, it is apparent that a linear model may not describe the data with accuracy. An exponential or higher order polynomial may do a better job. Future analyses could explore the predictions of a polynomial model in this context.</p><p>Original article can be found on the <a href="http://codechalet.com/docs/bayesian/analyses/bikes.html">Code Chalet website.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13fb62282f26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-chalet/bicycles-in-cold-weather-13fb62282f26">Bayesian Bicycles and Cold Weather</a> was originally published in <a href="https://medium.com/code-chalet">Code Chalet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayesian Coin Flips]]></title>
            <link>https://medium.com/code-chalet/bayesian-coin-flips-d195f23500cb?source=rss----db59dc0dfce5---4</link>
            <guid isPermaLink="false">https://medium.com/p/d195f23500cb</guid>
            <category><![CDATA[bayesian]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Ben Johnson]]></dc:creator>
            <pubDate>Sun, 07 Nov 2021 22:17:05 GMT</pubDate>
            <atom:updated>2021-11-08T04:05:29.958Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*i_sUxOjjg_Y5GbnJYHkcOQ.jpeg" /></figure><p>Let’s do a simple Bayesian analysis of a weighted coin flip. For the purposes of the experiment, suppose you are having Thanksgiving dinner with your friends, and one of your friends brings a coin. Your friend suggests a game — she will flip a coin 100 times, and the person who predicts the number of heads most accurately wins.</p><p>Note that in this article, I will be using R code to demonstrate the analysis. You can understand the analysis without understanding the code, but it does help to follow along with the calculations. The functions introduced in this article are documented well on the <a href="https://www.rdocumentation.org/">RDocumentation</a> site.</p><p>We will model the coin flip as a random experiment with two possible outcomes: heads, and tails. The probability that a coin flip will result in heads can be represented by a binomial distribution. Probability in this case is the parameter which we would like to understand. If we improve our understanding of the probability that the coin will land on heads, we could improve our prediction.</p><p>The binomial distribution measures the number of successes in an experiment which has two possible outcomes. We can define the two outcomes of flipping a coin as <em>success</em> and <em>failure</em> — in our case we will define success as heads, and failure as tails. Mathematically, the probability of <strong>X</strong> successes, given a likelihood <strong>p</strong> and number of experiments <strong>x</strong> can be expressed like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*bklqBVl0RZv4YcKFZcNxww.png" /></figure><h3>Prior</h3><p>This particular friend is known to be shady. You suspect that the coin is not a fair coin. You have reason to believe the coin is weighted, but you don’t have a good reason to suspect it is weighted towards heads rather than tails. Your <em>prior </em>knowledge of the parameter <strong>p</strong> is therefore a uniform distribution — from your perspective, each of the parameter values is equally likely.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d5f4ce21cbd00e3f35ff667dfbcbaf7/href">https://medium.com/media/5d5f4ce21cbd00e3f35ff667dfbcbaf7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xOeapGYAKFUMpsP4ynFKXw.png" /></figure><h3>Likelihood</h3><p>Your friend flips the coin, and out of 100 coin flips, 77 are heads. This is the experiment data, and we can use it to update the prior. In Bayesian analysis, we use the experiment data to compute the likelihood of each prior prediction. Essentially, we are using data to update our model of the world.</p><p>Let’s use a grid analysis to perform a numerical analysis. First, instantiate a vector with 1000 evenly-spaced values between 0 and 1, to represent the possible values of parameter <strong>p. </strong>Recall <strong>p</strong> represents the probability that the coin will land on heads.</p><p>Once we have a discrete vector representing the possible values of <strong>p, </strong>we will use that vector to compute the <em>likelihood</em> of each value of <strong>p,</strong> using the binomial probability mass function. Note the code makes use of R’s vectorized computations — since we pass a vector into <strong>dbinom()</strong>, the result is also computed as a vector.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/647a1b1a3b52c2f36960a95094a293fc/href">https://medium.com/media/647a1b1a3b52c2f36960a95094a293fc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEyV8yMLZwBTD6rQwsi_WA.png" /></figure><h3>Posterior</h3><p>Now that we have a <em>prior</em> and a <em>likelihood</em>, we can compute the <em>posterior</em> <em>distribution</em>. In Bayesian analysis, the <em>posterior </em>is the <em>prior </em>multiplied by the <em>likelihood</em> of the data. Since the prior is a uniform distribution, our posterior is simply identity multiplied by the likelihood function (so the posterior will be identical to the likelihood function).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62c0825f5bdcd7eb97523c3f80dc3751/href">https://medium.com/media/62c0825f5bdcd7eb97523c3f80dc3751/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Te3o33sSobspX2B0TG_Cw.png" /></figure><h3>Sampling the Posterior</h3><p>In our case, the posterior distribution has an analytic solution (at this point the posterior can be represented by the binomial probability mass function, since the posterior was formed by multiplying a uniform prior and the likelihood). However, in more complex Bayesian models, the posterior does not have a simple solution. It can be computationally expensive to even calculate the posterior distribution. In this case we can approximate the posterior distribution by sampling it.</p><p>Let’s take 10,000 samples from the posterior distribution. We are sampling the values of the parameter <strong>p</strong>, which represents the probability that the coin will land on heads. Recall the vector <strong>p_grid</strong> represents 1000 discrete parameter values, and the vector <strong>posterior</strong> represents the plausibility of each parameter value.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cc2e43680c9c8532491905a92e5d27d/href">https://medium.com/media/2cc2e43680c9c8532491905a92e5d27d/href</a></iframe><p>Each value in the <strong>samples</strong> vector is one of the discrete parameter values specified in <strong>p_grid</strong>. Let’s create a histogram to represent the frequency of each parameter value in our set of samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKQu88yzhkdn3HfT2OHlAw.png" /></figure><h3>Interpreting the Model</h3><p>The posterior distribution gives us the likelihood of different parameter values. If we were to predict the results of the next 100 coin flips, we could use the posterior distribution to select a parameter value to use in our calculations. In this simple experiment, the most likely parameter value is 0.77, as the posterior distribution is entirely characterized by the experiment data (recall the prior was a uniform distribution).</p><p>There are definite limitations to our model. The model assumes independence between events, but we know that the physical world is deterministic and a past flip of the coin could influence future coin flips. In his book <em>Statistical Rethinking,</em> Richard McElreath points out that physical events are deterministic and therefore past events can influence future events. This could inspire a debate on free will but for our purposes, suffice it to say that the assumption of independence is questionable.</p><p>The prior distribution could also be improved. Physics makes it extremely unlikely that the coin would have a 99% probability of landing on heads. Extreme parameter values are unlikely, but our prior distribution treats 0.99 and 0.70 as equally plausible.</p><p>The next time your friend wants to play the game, you can use your posterior distribution to make a prediction! After the game, you can use the new experiment data to update the model. The current posterior would become your new prior, and it can be updated using new experiment data.</p><p>The original article can be accessed on the <a href="http://codechalet.com/docs/bayesian/binomial.html">Code Chalet website.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d195f23500cb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-chalet/bayesian-coin-flips-d195f23500cb">Bayesian Coin Flips</a> was originally published in <a href="https://medium.com/code-chalet">Code Chalet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Production Spark via Databricks]]></title>
            <link>https://medium.com/code-chalet/production-spark-via-databricks-307f58f34045?source=rss----db59dc0dfce5---4</link>
            <guid isPermaLink="false">https://medium.com/p/307f58f34045</guid>
            <category><![CDATA[production]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Ben Johnson]]></dc:creator>
            <pubDate>Tue, 02 Nov 2021 01:07:02 GMT</pubDate>
            <atom:updated>2021-11-02T01:06:47.696Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2KnQ9EiINGNOURoQ" /><figcaption>Photo by <a href="https://unsplash.com/@erikvandijk?utm_source=medium&amp;utm_medium=referral">Erik van Dijk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you use Databricks to execute your Spark jobs, you may wonder how to put these jobs into production. Databricks provides a nice UI for creating and editing notebooks, but what if you want to refactor these notebooks into production software? This article will explain how to do exactly that.</p><h3>Databricks Connect</h3><p>Typically, Databricks code will exist in a series of notebooks. Notebooks are web-based documents which contain code, visualizations and markdown. In production, we want to avoid notebooks and execute our code in the traditional way — as files.</p><blockquote>How can you execute a Spark script on a Databricks cluster?</blockquote><p><a href="https://docs.databricks.com/dev-tools/databricks-connect.html">Databricks Connect</a> allows you to execute Spark scripts on a Databricks cluster. Using DB Connect, you can execute Spark jobs from a machine external to the Databricks cluster, be it your local machine, or a production server. DB Connect makes it possible to run Spark in client mode. In client mode, your local machine acts as the cluster manager and performs the parsing and planning of the job before submitting an instruction set to the Spark cluster.</p><p>Client mode is distinguished from cluster mode, where the cluster manager exists within the cluster. Cluster mode is actually preferable for a production environment: since the cluster manager is co-located with the cluster nodes, latency between the cluster manager and the worker nodes is minimized. However, cluster mode is currently not supported for standalone Python applications, and it seems that DB Connect only supports client mode.</p><p>To install Databricks Connect, follow the instructions on the <a href="https://docs.databricks.com/dev-tools/databricks-connect.html">Databricks website</a>, or the abbreviated instructions saved on the <a href="http://localhost:4000/docs/analytics/databricks/databricks-connect.html">Code Chalet website</a>. You will receive warnings if Hadoop is not installed on your local machine -to fix these errors, install the <a href="http://localhost:4000/docs/analytics/databricks/databricks-connect.html">Hadoop binaries</a> and add <strong>winutils.exe </strong>to the binaries.</p><h3>Test the Connection</h3><p>To test the connection, we will write a simple script which creates a table in the Databricks data warehouse, queries the table, and then drops it. Our program will create temperature data for several airports cross the United States, following the format of many other introductory coding problems.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c95a378b443df0badb3f5f29fdbb9005/href">https://medium.com/media/c95a378b443df0badb3f5f29fdbb9005/href</a></iframe><p>If Databricks connect is set up properly, you should be able to run this script and verify the output of the SELECT statement in the console. Configuring the Databricks client is the hardest part of this exercise, so it may take several iterations until this code executes successfully.</p><h3>Structuring Application Code</h3><p>I’ve found it helpful to organize all the application code within a folder isolated from the runtime. Within the <strong>application </strong>folder, you can separate your code into folders which are more specific. For example, I usually create a folder for models, another one for helper functions, and another for pipelines.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6965f956a3d5942edd5a479cca36410b/href">https://medium.com/media/6965f956a3d5942edd5a479cca36410b/href</a></iframe><h3>Runtime Code</h3><p>The file run.py contains the runtime program. This program has several responsibilities:<br> 1. Instantiate a connection to the Spark cluster<br> 2. Ship the application code to the Spark cluster<br> 3. Plan the Spark job.<br> 4. Distribute the Spark job to the cluster nodes</p><p>Before you execute the job, you’ll need to create a <a href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.SparkSession.html">Spark session</a>. The Spark session is the entry point for Pyspark. Keep in mind that this step depends on the successful configuration of Databricks Connect.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06cb49541fbcb74123d5424fc9220293/href">https://medium.com/media/06cb49541fbcb74123d5424fc9220293/href</a></iframe><p>So far our program is quite simple. The next thing we need to do is ship our application code to the Spark cluster. We will use <a href="https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.SparkContext.html">SparkContext</a> to accomplish this goal. The SparkContext API includes a function <a href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.addPyFile.html">addPyFile</a>, which can be used to add a .py or .zip dependency to all tasks executed within the context.</p><p>When you add a file to the SparkContext, it is shipped to the Databricks cluster and each of the worker nodes. We could add each individual Python file to the Spark context, or we could create a .zip archive of the application code and ship that. The .zip archive will be compressed, so in theory the upload time will be faster. Make sure to add application.zip to the .gitignore so it is not inadvertently added to source control.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/88f1248eae00bbebc5a17487a678505e/href">https://medium.com/media/88f1248eae00bbebc5a17487a678505e/href</a></iframe><p>Now we can write our code which executes within the SparkSession! Since we have shipped the application code to the cluster, the cluster nodes each have a copy of the functions and modules in the application folder.</p><p>Follow <a href="https://medium.com/code-chalet">Code Chalet</a> for more publications like this one!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=307f58f34045" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-chalet/production-spark-via-databricks-307f58f34045">Production Spark via Databricks</a> was originally published in <a href="https://medium.com/code-chalet">Code Chalet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Encryption for Production Credentials]]></title>
            <link>https://medium.com/code-chalet/encryption-for-production-credentials-e063a95e747f?source=rss----db59dc0dfce5---4</link>
            <guid isPermaLink="false">https://medium.com/p/e063a95e747f</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[encryption]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Ben Johnson]]></dc:creator>
            <pubDate>Mon, 25 Oct 2021 00:24:52 GMT</pubDate>
            <atom:updated>2021-10-24T22:25:40.226Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6D5PoaKgkBRZqx0u" /><figcaption>Photo by <a href="https://unsplash.com/@bergerteam?utm_source=medium&amp;utm_medium=referral">Florian Berger</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’ll confess: encryption is not the first thing I think about when I start to build a piece of software. When it comes to secrets, I would rather let other people do the worrying. My sisters will eagerly tell you that I have never been good at keeping secrets of any sort, and I’m not particularly invested in developing that skill.</p><p>If you’re deploying software, at some point you will need to communicate a secret between two machines. I usually solve this problem using an encrypted communication channel, such as Teams, Outlook or Slack. In the past, I’ve created a file containing all the secrets for a piece of software, and sent those secrets to the production machine via encrypted email.</p><blockquote>How can a team share secrets?</blockquote><p>What about a team that needs to share the responsibility of deployment? How can they share the secrets, while keeping them secure?</p><h3><strong>Design 1</strong></h3><p>The easiest implementation would involve sharing secrets via an encrypted channel. For example, your team could send a secrets file in a Teams or Slack conversation. Each each team member would download the secrets file and save it to the proper location in the repository.</p><p>Let’s pretend Bob is building a service which uses the Google Maps API. Bob could save the API key to a file named <strong>maps_api_key.txt</strong>. His service would import <strong>maps_api_key.txt</strong>, and read the API key from the file. When his service is ready for QA, Bob would send an encrypted email containing <strong>maps_api_key.txt</strong> to his teammates. Each team member would download the file, and place it in the repository at the appropriate path.</p><blockquote>The secrets are isolated from source control.</blockquote><p>What are the scale implications of this decision? The next time a team member wants to add a credential, they will follow the same process: create a file with the credential, and email the file to the entire team. As your architecture scales, it will quickly become more difficult to manage all the credentials. This design scores low on maintainability, as well, because once Bob leaves it is unclear who will be responsible for saving his API key and sharing it with the team. There are also reliability issues : if a team member saves the file to the wrong location, the software will fail.</p><p><strong>Scalability </strong>: 1/5<br><strong>Maintainability </strong>: 1/5<br><strong>Reliability </strong>: 1/5</p><h3>Design 2</h3><p>Another solution would involve saving secrets in a centralized location. A first order implementation of this design might be saving secrets in an encrypted, shared document (such as Sharepoint). When a team member needs to add a new secret, they log into the shared document and add a new line. Each team member would need to download the newest version of the secrets file to ensure the software would run properly.</p><blockquote>Secrets are still isolated from source control.</blockquote><p>Maintainability is better; now Bob is not the only one who is keeping track of the Google API key. Scalability is better, as well. As the list of secrets grow, the shared document will grow, but the secrets are limited to a single document. Unfortunately, reliability is still poor: the developers still need to move the secrets from the shared document to a file within the repository, and there is a possibility that the file will become corrupted and unusable.</p><p><strong>Scalability </strong>: 3/5<br><strong>Maintainability </strong>: 3/5<br><strong>Reliability </strong>: 1/5</p><h3>Design 3</h3><p>A third solution would consist of saving the secrets directly to source control. How is this possible? If we encrypt the secrets before we upload them to source control the secrets will be, well, <em>secret </em>from anyone who does not have the encryption key. <a href="https://www.ibm.com/docs/en/ztpf/2020?topic=concepts-symmetric-cryptography">Symmetric encryption</a> uses the same key for encryption and decryption, so the team will only need to keep track of one key.</p><blockquote>In this design, secrets are part of source control.</blockquote><p>Maintainability is even better now; all secrets are saved directly to version control and no one developer is responsible for communicating or maintaining the secrets. Scalability is improved; the team only needs to keep track of a single encryption key so as the number of secrets grow, the complexity of our solution remains the same. Reliability is good as well; the encryption key is a single string so there are fewer opportunities for it to be corrupted when team members copy the key to their repositories.</p><p><strong>Scalability </strong>: 5/5<br><strong>Maintainability </strong>: 5/5<br><strong>Reliability </strong>: 5/5</p><h3>Python Implementation</h3><p>I’ll demonstrate a simple Python implementation. This approach was inspired by <a href="https://edgeguides.rubyonrails.org/security.html">encrypted credentials</a> in Ruby on Rails, so filenames will be similar.</p><p>Our user will have two files: an encrypted file containing secrets (<strong>secrets.yml.enc</strong>), and the encryption key (<strong>master.key</strong>). The encrypted file will be uploaded to source control, while the encryption key will not.</p><p>The interface has two responsibilities:<br> • Decrypt the secrets file using the encryption key<br> • Encrypt a file using the encryption key</p><p>We will use the <a href="https://cryptography.io/en/latest/fernet/">Fernet module</a> for symmetric encryption. Our encryption function will accept three arguments: the filename of the file to encrypt, the desired filename of the encrypted file, and the filename of the master key.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/baf1fb0384c64a885e4368a9bec3ba7b/href">https://medium.com/media/baf1fb0384c64a885e4368a9bec3ba7b/href</a></iframe><p>The decryption function is similar.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/910386119449015aa35175136b14db46/href">https://medium.com/media/910386119449015aa35175136b14db46/href</a></iframe><p>Make sure you add the encryption key to your .gitignore , so that it is not committed to source control! You’ll may want to modify the decryption function so that it returns an object which you can use in your program.</p><blockquote>“Excellent”, I cried. “Elementary”, said he.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hmPvhj2Xid9Co9-j" /><figcaption>Photo by <a href="https://unsplash.com/@sammywilliams?utm_source=medium&amp;utm_medium=referral">Sammy Williams</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e063a95e747f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-chalet/encryption-for-production-credentials-e063a95e747f">Encryption for Production Credentials</a> was originally published in <a href="https://medium.com/code-chalet">Code Chalet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>