<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jerry Jackson on Medium]]></title>
        <description><![CDATA[Stories by Jerry Jackson on Medium]]></description>
        <link>https://medium.com/@jrryjcksn?source=rss-8846ba570c8b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*bsbbUAaXb8P1KTaw.</url>
            <title>Stories by Jerry Jackson on Medium</title>
            <link>https://medium.com/@jrryjcksn?source=rss-8846ba570c8b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 11:21:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jrryjcksn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Forgotten Programming Paradigm]]></title>
            <link>https://medium.com/@jrryjcksn/the-forgotten-programming-paradigm-5f020e576c2a?source=rss-8846ba570c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f020e576c2a</guid>
            <category><![CDATA[clojure]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[rules]]></category>
            <category><![CDATA[programming-paradigms]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Jerry Jackson]]></dc:creator>
            <pubDate>Fri, 02 Nov 2018 15:41:17 GMT</pubDate>
            <atom:updated>2018-11-02T15:41:17.003Z</atom:updated>
            <content:encoded><![CDATA[<p>Most people who have any familiarity with Computer Science are at least familiar with the concepts of Turing Machines and the Lambda Calculus. This isn’t too surprising since every common programming language can be viewed as an application of one or the other (Turing Machines for imperative languages and Lambda Calculus for functional ones). However, there is a third equivalent formalism that is less well-known outside of the parsing world: the Production System. Production systems repeatedly apply rewrite rules to a “working memory” until a desired state is reached. Production systems are closely related to grammars and parsing systems are the main place most programmers have been exposed to the concept. Not surprisingly, though, the paradigm has other applications besides parsing.</p><h3>Rule Engines</h3><p>Everyone has heard of rule engines but most people don’t have a real sense of what they do or where they’re valuable. It doesn’t help the confusion that many systems called “rule engines” amount to nothing more than tools to let business analysts write simple constraints like:</p><p><strong>IF</strong> user.age &gt; 58 AND user.retirementSavings &lt; 100,000<br><strong>THEN</strong> notify(“UH-OH!”)</p><p>Not that such systems aren’t useful but they don’t inspire programmers to take them on; they’re basically simplified, impoverished versions of normal straight line code. The rule engines I’m talking about are the ones that implement an efficient match algorithm like RETE, TREAT, or LEAPS. The efficient matching opens the door to new coding approaches that would not otherwise be reasonable. Let’s look at a few examples.</p><h4>Reactive Systems</h4><p>The Cassatt Corporation built a product in the early 2000s called Active Response. Cassatt Active Response (CAR) managed computer clusters. It imaged hardware, replaced failed servers and provided load-based allocation. When we started to build CAR, we initially approached it like a traditional application with straight line code and transactions. What we discovered, though, is that it was very difficult to make something like that work since the system needed to respond to high priority events immediately, no matter what else it might be doing. Also, sequences of actions taken by the system to achieve some goal might need to be altered if, for example, one of the servers failed while the action was ongoing. Some systems deal with this sort of problem via priority interrupts but that means that there isn’t really any good way for the two code streams to cooperate. They might even work at cross purposes.</p><p><strong>Enter a Rule Engine<br></strong>We eventually realized we could handle this by using rules. The great advantage of efficient matching means that the entire state of the system can be evaluated prior to taking any action (even in a system with thousands of servers). So, if a server fails while constructing an application tier, it can be taken into account in deciding the next action.</p><p>We also obtained another big advantage from using rules. Cluster management is a complex problem and not really “solvable” in the sense that there’s a well-defined model that just needs to be implemented. We “learned as we went”. As we did so, we could easily add special case rules to handle them and fold them into the core over time which helped accelerate development immensely.</p><h4>Constraint Processing</h4><p>Kubernetes is widely used to model and deploy modern applications. The pieces of a Kubernetes definition, however, are deployed independently despite having implied relationships between them based on naming and labels. This means it’s difficult to determine if a configuration is valid without running it. At Yipee.io, rules are used to define such constraints which makes the code far more clear and concise. Without rules, each constraint check would need to navigate all the pieces of the app looking for relationships and that navigation code would be brittle under future changes to the definition of Kubernetes.</p><h4>Prototyping</h4><p>Recently, at Yipee.io, we developed a “diff” facility for Kubernetes manifests. It works by turning objects “inside-out” into fields with paths. For example, the object:</p><p><strong>{ “foo”: “bar”, “baz”: {“quux”: 6}}</strong></p><p>would turn into the following set of fields:</p><p><strong>[“foo”, “bar”]<br>[“baz”, “quux”]<br>[“baz”, “quux”, 6]</strong></p><p>It’s probably clear from a few moments thought that a diff at the level of [“baz”, “quux”] overrides any at [“baz”, “quux”, 6]. That is, if we’re diffing:</p><p><strong>{ “foo”: “bar”, “baz”: {“quux”: 6}}<br>{“foo”: “bar”, “baz”: {“quux”: 28}}<br>{“foo”: “bar”, “baz”: “yow”}</strong></p><p>we want to generate:</p><p><strong>{“foo”: “bar”, “baz”: &lt;hole&gt;}</strong></p><p>with three different values for the hole. To achieve this during prototyping, we defined a simple rule that said:</p><p><strong>IF</strong> subseq(diffPath1, diffPath2)<br><strong>THEN</strong> delete(diffPath2)</p><p>Not a very efficient approach but fantastic for prototyping. We just dropped every diff path into working memory and Voila! As it turned out, we didn’t even need to rewrite this since we were able to cut out non-diff paths before invoking the rules so that was the entire implementation of the path pruning.</p><h3>Summary</h3><p>Production systems in general, and rule engines in particular should be in every programmer’s toolbox. There are many use cases for which they are extremely well-suited. If you’re a Clojure programmer, check out the rule engine that was recently open-sourced by Yipee.io <a href="https://github.com/yipeeio/arete">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f020e576c2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Future-Proofing Backend Services with GraphQL, PostgreSQL and Docker (Part 2)]]></title>
            <link>https://medium.com/@jrryjcksn/future-proofing-backend-services-with-graphql-postgresql-and-docker-part-2-f78a41185c05?source=rss-8846ba570c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f78a41185c05</guid>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Jerry Jackson]]></dc:creator>
            <pubDate>Thu, 01 Nov 2018 21:39:13 GMT</pubDate>
            <atom:updated>2018-11-01T21:39:13.973Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/@jrryjcksn/future-proofing-backend-services-with-graphql-postgresql-and-docker-part-1-527bd6b1a59a">Part 1</a>, I covered how clients interact with GraphQL. In this post, we’ll look at the implementation of a <a href="http://graphql.org/">GraphQL</a> server using <a href="https://golang.org/">Go</a> and <a href="https://www.postgresql.org/">PostgreSQL</a>.</p><h3>Implementing the Server Side of a GraphQL Application</h3><p>We wrote our server code in Go but the basic model for server-side GraphQL is language-independent. GraphQL servers are built around the idea of a <em>resolver</em>. Resolvers provide the implementations for accessing fields of our GraphQL types. In the excellent Go server we’re using <a href="http://github.com/neelance/graphql-go">(github.com/neelance/graphql-go)</a>, a root resolver has methods for each of the queries and mutations defined in our schema. Here is the resolver for our simplest query:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe443a97e0e17f366b871d469e0f6667/href">https://medium.com/media/fe443a97e0e17f366b871d469e0f6667/href</a></iframe><p>There’s not much going on here because a log level is just a string. Things get more interesting when we’re resolving an instance of a structured type:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7180eef72d2aeba01e6ec95e23f1fbc6/href">https://medium.com/media/7180eef72d2aeba01e6ec95e23f1fbc6/href</a></iframe><p>In this example, a “Feature” is an aspect of the application that can be turned on or off for A/B testing. Features are simple named values so the feature type just has two fields: <em>ID</em> and <em>Name</em>. The Feature method of the root resolver looks up the feature by id. The lookup function GetFeatureById does a database query and constructs a sub-resolver for feature. This resolver is used to retrieve fields of the feature object:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/074855126b4588167a8a1b5679a1d9a5/href">https://medium.com/media/074855126b4588167a8a1b5679a1d9a5/href</a></iframe><p>When a feature resolver is returned to the GraphQL server, it invokes these methods as required to fetch fields mentioned in a query or mutation. Since features are simple objects, the buck stops here. What happens though, you might ask, when a sub-resolver has references to other structured types (or lists of them)? As you likely suspect, more of the same. The methods for those fields return other resolver objects, etc. Given this one-level-at-a-time fetch-as-needed model, the most direct way to implement these resolvers is to have the methods for reference fields perform queries on demand. This is, in fact, how we initially implemented our server. As you can probably imagine, however, this leads to a <em>lot</em> of queries. Consider the problem of user authorization. In order to authorize users, we need to know the set of organizations a user can access, along with whether or not they are admins or writers for said organizations. We also need to know about the plan associated with each organization, etc. Here is the basic query for establishing context:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c2d3a504272a57b2fedcf9a468c1284/href">https://medium.com/media/5c2d3a504272a57b2fedcf9a468c1284/href</a></iframe><p>In the trivial case we started with above (remembering that we now call him “snake”) we get:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f9c7b40f85017b9f53e0768b41abe353/href">https://medium.com/media/f9c7b40f85017b9f53e0768b41abe353/href</a></iframe><p>This is the simplest case, in which the user is only associated with a single organization: his freemium individual organization. Even in this case, though, the number of queries in a naive approach is relatively large: nine queries are needed for a user with only one organization and no other users associated with the organization. The numbers grow rapidly as the number of organizations and org members increase. It would be nice to get more of the data up front and reduce the number of queries. How should we do that, though? Relational queries are not generally well suited to deeply nested structures as the result tables are flat (leaving aside group queries which really wouldn’t help here). One possibility for optimizing queries that comes to mind is to cache fetched values. This is certainly possible and even fairly easy in a single server case. A cache could simply map (id, fieldname) pairs to values and flush them when such fields are updated. Of course, in a multi-server scenario, this would require a distributed cache. This is still not necessarily a deal breaker. However, it turns out that PostgreSQL has our backs here. Some extensions to the relational model supported by PostgreSQL make it a uniquely powerful fit to GraphQL. You may have heard that PostgreSQL now has JSON support. If so, you&#39;d be forgiven for assuming that just means having a JSON column type (possibly with query support for selecting based on fields within JSON documents). Well, it certainly does have this sort of functionality, but that&#39;s only the beginning. In fact, PostgreSQL&#39;s JSON support combined with its array support is the answer we&#39;re looking for. Let&#39;s look at a simple example. The definition of a &quot;Plan&quot; in the schema for our organizational model is:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb610fd923b4e90de56bd0f1b152dd95/href">https://medium.com/media/eb610fd923b4e90de56bd0f1b152dd95/href</a></iframe><p>Our initial implementation of a plan involved one query that returned the simple values (id, name, maxWriters) and three other follow-on queries to obtain the related instances. By far, the most commonly queried connected set is the elements (it’s even in our basic context query above). Is there an easy way to retrieve the elements for a plan in the initial query? Yes, there is! Here’s a PostgreSQL query to select a plan <em>with</em> its elements:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0a515b0b5cbdbb2593ae12b9583e38f/href">https://medium.com/media/b0a515b0b5cbdbb2593ae12b9583e38f/href</a></iframe><p>Let’s unpack what’s going on here; we’re selecting rows containing three columns each: planname, maxwriters, and a JSON array containing plan element JSON objects. PostgreSQL allows us to turn the results of a query into an array and then turn the array into JSON. Nice. Let’s run the query in psql for a particular plan:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02b8fddb3047f7efc5534b42e12e3590/href">https://medium.com/media/02b8fddb3047f7efc5534b42e12e3590/href</a></iframe><pre>Freemium 1 [{&quot;ID&quot; : &quot;ec810d52-de19-11e8-aede-23b1e5582497&quot;, &quot;Name&quot; : &quot;downloadComposeFiles&quot;}, {&quot;ID&quot; : &quot;ec810d52-de19-11e8-aede-23b1e5582497&quot;, &quot;Name&quot; : &quot;downloadComposeFiles&quot;}]</pre><p>Fantastic! With a bit of work, this can be extended to arbitrary nesting levels. We’re able to obtain all the data in the context GraphQL query above in a single database query; so much for latency! I won’t include that full query as it looks a bit intimidating. Rest assured, however, that it’s just a straightforward extension of the same approach. Problem solved.</p><h3>Summary</h3><ul><li>GraphQL is a new approach to JSON-based web apis that provides significant flexibility advantages over REST.</li><li>GraphQL apis are driven by schemas that define types and entry points for <em>queries</em> and <em>mutations</em>.</li><li>A GraphQL api is exposed through a single HTTP end point.</li><li>GraphQL allows the specification of arbitrarily nested queries.</li><li>Relational databases are a good match to GraphQL schemas.</li><li>PostgreSQL provides array and JSON extensions that make it easy to process nested queries for optimization of complex GraphQL requests.</li><li>Docker helps future proof service development by minimizing cycle times and reducing friction in deployment.</li></ul><p>Thanks for reading!</p><p><em>Originally published at </em><a href="https://gist.github.com/9947ed9872fca5bf6b1af2d7e9d8a7b9"><em>gist.github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f78a41185c05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Future-Proofing Backend Services with GraphQL, PostgreSQL and Docker (Part 1)]]></title>
            <link>https://medium.com/@jrryjcksn/future-proofing-backend-services-with-graphql-postgresql-and-docker-part-1-527bd6b1a59a?source=rss-8846ba570c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/527bd6b1a59a</guid>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Jerry Jackson]]></dc:creator>
            <pubDate>Thu, 01 Nov 2018 20:48:32 GMT</pubDate>
            <atom:updated>2018-11-01T20:48:32.954Z</atom:updated>
            <content:encoded><![CDATA[<p>At Yipee.io, we recently added team functionality to our modeling tool. Teams provide support for sharing applications and are tied into our pricing, enabling teams to purchase different plans with different sets of associated features and increasing complexity for us. We agreed on the relationships between users, teams, plans and features but it wasn’t necessarily clear up front exactly how the UI would need to navigate the relationships. We might want to find the features defined for a user but, alternatively, we might want to find all users, teams1, and plans that possess a particular feature. At the time, for historical reasons, we had a backend system driven by <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">REST</a> APIs that stored data in <a href="https://www.mongodb.com/">MongoDB</a>. MongoDB is an excellent choice for certain use cases (particularly very high data volumes) but a complicated graph of relations, which can be navigated from multiple endpoints along a wide variety of paths, is not a particularly good fit. We decided to switch our backend to a relational database as we didn’t anticipate ever reaching a volume of data that would make a relational implementation problematic. Now that multiple vendors support <a href="https://en.wikipedia.org/wiki/Relational_database_management_system">RDBMS</a> as a service, the decision to use a relational database has been made even easier. Having had positive experiences with <a href="https://www.postgresql.org/">PostgreSQL</a> in the past, we decided to give it a try. Our DaaS vendor supported PostgreSQL in addition to MongoDB, so the switch could be made fairly easily. As it turned out, PostgreSQL was quite possibly the ideal choice for this application. I’ll discuss its unique qualifications later in the post. Once we had decided on a relational database, a similar thought process ensued around our REST APIs. The document model behind REST is a somewhat poor fit to a relationship graph with properties on the associations. The model we were managing looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xhdcSKruUF8NDg8U.png" /></figure><p>We could have mapped out REST endpoints for our immediate operational needs, but we might easily have been chasing use cases for some time as the UI or another API user needed different collections of data. An initial dataset for a user identity needed by the UI was:</p><ul><li>The user’s id, terms and features</li><li>Each org1 containing the user with its’ id, name, owner (with id and identity), set of users (with id and identity), set of admins (with id and identity), set of writers (with id and identity), plan elements (with id and name), set of features (with id and name) and the set of features for the plan (with id and name)</li></ul><p>The backend API needed a similar but somewhat different dataset to establish a user context. These sorts of queries make for an odd REST interface. So we decided to move away from REST for our team model and use <a href="http://graphql.org/">GraphQL</a>. The rest of this discussion will talk about our experience using GraphQL with PostgreSQL to implement our team model. 1*Teams are called orgs in the schema since there are orgs for individuals and we wanted to talk about true team plans separately. Single-person orgs work just like orgs associated with larger teams.*</p><h3>GraphQL</h3><p>There are many excellent sources for information about GraphQL online. In particular, <a href="http://graphql.org/">this site</a> has a wealth of useful material. That said, I’ll give a brief overview here for context. GraphQL is an alternative to REST that allows clients to design their own queries instead of requiring that all requests be defined by the server. This works because GraphQL is schema-based. A GraphQL server defines:</p><ul><li>a <em>schema</em> which specifies all the meaningful relationships between the elements of a data model</li><li>a set of <em>queries</em> which are entry points into the schema for retrieving data</li><li>a set of <em>mutations</em> which are the operations that may be performed on model data</li></ul><p>Unlike a REST api, a query entry point does not itself define what data may be retrieved. For example, here is a query from our team model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08bfc0c63fd9222dea7990967aa9ae03/href">https://medium.com/media/08bfc0c63fd9222dea7990967aa9ae03/href</a></iframe><p>This says that a client can retrieve a user given a service (e.g. “GitHub”) and an identity (e.g. GitHub id). However, the user is just the beginning. Here are the descriptions of the user and org types in our schema:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e26c25c729ad0023e045db7f9aff6d0b/href">https://medium.com/media/e26c25c729ad0023e045db7f9aff6d0b/href</a></iframe><p>An exclamation point (!) means that the item may not be null. The user definition says that a user consists of three string values, one boolean, and three lists containing instances of associated types (similar for an organization). A GraphQL schema contains type descriptions like this plus definitions for the queries and updates (called “Mutations”) supported by the server. Our queries and mutations are:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/09b28815209ae5b64f232576bc5b1cf8/href">https://medium.com/media/09b28815209ae5b64f232576bc5b1cf8/href</a></iframe><p>To me, this looks a lot like some sort of <a href="https://en.wikipedia.org/wiki/Remote_procedure_call">RPC</a> definition. The special sauce that makes GraphQL powerful, however, is the client’s ability to select which data she wants returned. Let’s go back to that first query:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32c047abf8ce6ea6f8e3ee89ba792576/href">https://medium.com/media/32c047abf8ce6ea6f8e3ee89ba792576/href</a></iframe><p>Here is a use of it that requests the value of the “terms” field:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/529546ba47ab19de27ff82b99c8d9989/href">https://medium.com/media/529546ba47ab19de27ff82b99c8d9989/href</a></iframe><p>This is passed to a GraphQL server via an HTTP POST. All operations for the API go through the same HTTP end point. In curl, this would look like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e979a064f7236f29c67ccfb59d4197f5/href">https://medium.com/media/e979a064f7236f29c67ccfb59d4197f5/href</a></iframe><p>The returned payload is:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bea271052a1f02b80eb9d52ac6ce25af/href">https://medium.com/media/bea271052a1f02b80eb9d52ac6ce25af/href</a></iframe><p>Notice that we received the value for the “terms” field. What if we ask for more?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5edb361b9868ff67a2a76546f57d9e5d/href">https://medium.com/media/5edb361b9868ff67a2a76546f57d9e5d/href</a></iframe><p>Now we get:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/038c96cd1254e5fa87e051eae64934e6/href">https://medium.com/media/038c96cd1254e5fa87e051eae64934e6/href</a></iframe><p>So we can specify the fields we want for our types and even the fields for referenced objects recursively. Here we said we wanted the “name” and “maxWriters” field for each org owned by plissken. As one final example, let’s ask for the admins for each owned org:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f58607770101c94c2e7c5179f3ef5ff6/href">https://medium.com/media/f58607770101c94c2e7c5179f3ef5ff6/href</a></iframe><p>We get:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8217d88b2c89bf37ff21d083802dc020/href">https://medium.com/media/8217d88b2c89bf37ff21d083802dc020/href</a></iframe><p>Clearly, this is very powerful. Once you establish a schema, a GraphQL client can select the information required for a particular task as needed. Also, if we decide later that we need more data for a particular type, we can add it and all existing calls will remain unaffected.</p><h3>Mutations</h3><p><em>Mutations</em> are very much like RPC calls except that they allow specification of the data to be returned, just like queries. If we set out to update our user, we can use our updateUserByIdentity mutation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8785b1ba040eb3d1db6af6ad4c945aa/href">https://medium.com/media/c8785b1ba040eb3d1db6af6ad4c945aa/href</a></iframe><p>This says that we can update a user by specifying her service and identity values and an instance of UserUpdateInput for the new data. The update returns a UserChangeResponse which has fields for the user and any errors that were generated. Just like in a query, the returned user data can be expanded by specifying fields of interest. Here&#39;s an example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1aa4d286c32c586ac70e5ee6888e2dee/href">https://medium.com/media/1aa4d286c32c586ac70e5ee6888e2dee/href</a></iframe><p>This call returns:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91a3ee62efacb303c11c0f89eb81d1b4/href">https://medium.com/media/91a3ee62efacb303c11c0f89eb81d1b4/href</a></iframe><h3>PostgreSQL</h3><p>We picked up substantial benefits right away for this kind of highly connected model just by switching to a relational database. Supporting navigation from multiple directions and sets of users for organizations with different properties (e.g. admins, writers) would require redundancies in a document model and would make it impossible to perform updates transactionally. Additionally, the integrity constraints provided by an RDBMS remove an entire class of errors we would otherwise need to handle. One simple example is if a client tries to add a user to an organization but presents an invalid user id. This is caught for us by a constraint that the user field in the table connecting orgs and users reference an entry in the users table. Done! None of this is probably news to anyone reading this, but I feel like some of the benefits of an RDBMS are often forgotten today with the popularity of NoSQL databases. In any case, we were able to easily map our GraphQL schema onto a relational model. We’ll see where PostgreSQL really shines when we talk about optimizations in Part 2.</p><h3>Docker</h3><p>I believe it’s valuable to discuss <a href="https://www.docker.com/">Docker</a> in the context of future proofing services because we gained great leverage from our user of Docker. Changing out underlying database technology could have easily caused a significant impact to development for the following reasons:</p><ul><li>Our UI developers were working concurrently on unrelated features.</li><li>We didn’t want them to wait until our transition was complete to get started.</li><li>Deploying a database and tying it together with related services can be complicated and time-consuming.</li><li>Swapping out environments to work on different tasks often produces a lot of development friction.</li></ul><p>We got around these issues through judicious use of Docker. Here at Yipee.io, we model our own application using our modeler and run it using Docker. This means that our UI developers could simply use docker-compose to bring up or shut down our entire set of services (including the database). They could bring up one set of services to do unrelated development and switch to the new database version in about 30 seconds. Also, as we added features or fixed bugs, we could push a new version of our team functionality as a Docker image and they could incorporate it with little to no effort. They, in turn, could generate a new UI image to hand off to our UX team for review. Such short cycle times radically change the time and resources required to react to user feedback and new requirements.</p><h3>Part 2</h3><p>In part 2, I’ll discuss how GraphQL servers are built and how PostgreSQL offers significant leverage for future proofing their construction.</p><p><em>Originally published at </em><a href="https://gist.github.com/6551beeceaba8a066eebc7b893e661da"><em>gist.github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=527bd6b1a59a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Software is Harder Than You Let Yourself Think]]></title>
            <link>https://medium.com/@jrryjcksn/building-software-is-harder-than-you-let-yourself-think-14b04f537b9a?source=rss-8846ba570c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/14b04f537b9a</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[agile-methodology]]></category>
            <dc:creator><![CDATA[Jerry Jackson]]></dc:creator>
            <pubDate>Thu, 01 Nov 2018 14:22:29 GMT</pubDate>
            <atom:updated>2018-11-01T14:22:29.626Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F2WE4O7gQPLkuNJf.jpg" /></figure><p>Large software systems are by far the most complicated artifacts ever constructed by humans. Think about that for a second. Submarines, Rockets, Televisions, etc. are simple by comparison (except where they incorporate large software systems, of course). As software developers, we generally repress that knowledge on a day-to-day basis so we can go about our jobs without dissolving into quivering wrecks. In this post, however, I’d like to examine the results of doing the opposite. Let’s see what happens if we confront this reality head on.</p><h3>The Confident Engineer</h3><p>I think a typical engineer enters the field feeling pretty confident. Each of the programming tasks she has taken on prior to her first real job has either been successful or failed because of some clear mistake recognizable in hindsight. If you combine that with a history of being the smartest person in the room for most of her life, you can understand how this happens. Based on past successes, she will likely even take pride in writing “clever” code, disregarding <a href="https://www.cs.princeton.edu/~bwk/">Brian Kernighan’s</a> excellent observation:</p><blockquote><em>Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.</em></blockquote><p>Even if she ends up on a string of failed projects, each one can usually be hand-waved away as the result of some error that seems obvious in retrospect. In my experience, it takes a long time to see past this. Until you’ve worked on a variety of successful and unsuccessful projects, there isn’t enough data on which to base any kind of understanding as to what might lead to either outcome. It can take years or even decades to develop sufficient perspective. The problem is made worse by the fact that any reasonably well-designed large system will be constructed in a modular fashion. The small subsystem assigned to a hypothetical young engineer will look a lot like a small standalone program that she has had success with in the past. To a degree, this will <em>always</em> be the case since humans can only process a certain amount of complexity and any amount beyond what she can handle will typically fade into the background when she focuses on a particular task. To summarize, for a significant portion of an engineer’s career, her mindset is typically something like: “Writing software is challenging but well within my abilities; however, it <em>is</em> necessary to be careful.” This is patently false. Writing high quality, large-scale software is not “well within” anyone’s abilities. If it weren’t for the extreme malleability of software, few large systems would ever succeed. This is even more true for any kind of concurrent system. The anecdote in the next section should make this point more clear.</p><h3>Everyone Gets it Wrong</h3><p>What follows is an extended quote from a paper called <a href="https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf">The Problem With Threads</a> by Edward A. Lee from U.C. Berkeley. I believe it speaks for itself:</p><blockquote><em>In the early part of the year 2000, my group began developing the kernel of Ptolemy II [20], a modeling environment supporting concurrent models of computation. An early objective was to permit modification of concurrent programs via a graphical user interface while those concurrent programs were executing. The challenge was to ensure that no thread could ever see an inconsistent view of the program structure. The strategy was to use Java threads with monitors. A part of the Ptolemy Project experiment was to see whether effective software engineering practices could be developed for an academic research setting. We developed a process that included a code maturity rating system (with four levels, red, yellow, green, and blue), design reviews, code reviews, nightly builds, regression tests, and automated code coverage metrics [43]. The portion of the kernel that ensured a consistent view of the program structure was written in early 2000, design reviewed to yellow, and code reviewed to green. The reviewers included concurrency experts, not just inexperienced graduate students (Christopher Hylands (now Brooks), Bart Kienhuis, John Reekie, and myself were all reviewers). We wrote regression tests that achieved 100 percent code coverage. The nightly build and regression tests ran on a two processor SMP machine, which exhibited different thread behavior than the development machines, which all had a single processor. The Ptolemy II system itself began to be widely used, and every use of the system exercised this code. No problems were observed until the code deadlocked on April 26, 2004, four years later. It is certainly true that our relatively rigorous software engineering practice identified and fixed many concurrency bugs. But the fact that a problem as serious as a deadlock that locked up the system could go undetected for four years despite this practice is alarming. How many more such problems remain? How long do we need test before we can be sure to have discovered all such problems? Regrettably, I have to conclude that testing may never reveal all the problems in nontrivial multithreaded code.</em></blockquote><p>Here we see recognized experts in the field follow every best practice in constructing a concurrent system only to have it fail <em>four years</em> into production. What are the odds that <em>we’re</em> the ones who will get everything right given the typically tighter time constraints on development in industry?</p><h3>Estimates are Notoriously Bad</h3><p>Anyone who has ever worked on a software project knows that people habitually underestimate the time required for even relatively simple tasks. They continue to do so in spite of prior personal experience — often by wide margins. Why is this? In my mind, this is a result of three factors:</p><ul><li>The belief that developing software is “challenging but well within my abilities <strong>(CBWWMA)</strong>”</li><li>The difficulty of predicting the unknown</li><li>“Imposter syndrome”</li></ul><p>If asked to justify an estimate, an engineer can describe the work involved in implementing a solution but she cannot describe the work involved in dealing with the unexpected issues that will arise. Since writing software is something she is ostensibly good at, she is likely to balk at suggesting that “something unforeseen might come up”. She is even less likely to allot extra time for the unexpected if she suffers from imposter syndrome (as many/most of us do). For anyone unfamiliar with the term, it refers to the feeling that one is unqualified or out of one’s depth and, hence, an “imposter” of sorts. Perhaps surprisingly, I’ve never found an engineer who didn’t admit to feeling this way on occasion — even those who always seem from the outside to be incredibly confident and competent. When you feel this way, you don’t want to expose yourself as a fraud so you keep your doubts to yourself. Unfortunately, the other members of your team are likely doing the same. The result is that an engineer almost always estimates software as if the development effort will follow a “happy path” that aligns closely with the relatively vague conception of the process she holds in her head. The fact that development occasionally <em>does</em> follow a happy path reinforces the behavior. Instead of seeing the successes as lucky breaks, she sees the failures as bad luck.</p><h3>State is the Biggest Culprit</h3><p>Though there are a vast number of ways in which software development can go wrong, the area that produces the greatest difficulties is the management of state. Ben Moseley and Peter Marks present a very clear and lucid description of the issue in their excellent paper: <a href="https://github.com/papers-we-love/papers-we-love/blob/master/design/out-of-the-tar-pit.pdf">Out of the Tar Pit</a>. I heartily recommend reading it. I’ll just mention a few key points they discuss:</p><ul><li>The number of possible states of a typical object-oriented software application is astronomically large.</li><li>In general, the behavior of a system in one state tells you very little about it’s behavior in a different state.</li><li>As a practical matter, tests can only be performed in a small subset of the possible system states.</li></ul><p>Sounds bad, doesn’t it? Minimizing and isolating state in a software system are the two biggest contributors to software quality and reliability — particularly in the presence of concurrency. The extreme case of minimizing state occurs in a <em>functional</em> program. A fully functional program can have its parts tested in isolation and each one will continue to work correctly regardless of the context in which it’s deployed. Think about this the next time someone dismisses functional programming as “not practical” or “unnecessarily restrictive”. In developing successful software, we need to take advantage of any leverage available to reduce the prevalence and severity of errors. Minimizing state via functional programming and immutable data is a significant lever. That said, please don’t think I’m claiming that functional programming is a silver bullet. It’s not. There are plenty of other ways in which our development can go wrong; however, functional programming <em>can</em> remove whole classes of possible errors. Who wouldn’t want that?</p><h3>Facing the Problem</h3><p>So, if <strong>CBWWMA</strong> is wrong, what is the right perspective? I suggest something along the lines of: “Writing complex software is at or beyond the limits of human capacity. Good software is possible, but not guaranteed. When building any significant system, I will <em>undoubtedly</em> get many things wrong and must be prepared to deal with the results.” That’s the most important point… I <em>will</em> get many things wrong. Not “might” — “will”. It’s this realization that has led to most of (all of?) the improvements in software development over the last several decades. For example:</p><ul><li>Assuming we’ll get something wrong in our design leads us to solicit design and code reviews or do pair development.</li><li>Assuming that specifications (or our understanding of them) will be wrong encourages us to construct good regression tests so that we can make changes without breaking things.</li><li>Assuming that our understanding of the interfaces between our software and other parts of a system will be wrong motivates continuous integration.</li><li>Assuming that our estimates will be wrong encourages shorter development cycles so that errors will cost less and be found sooner.</li><li>Assuming that we will be unable to understand or manage the complexity of a system with significant mutable state leads us to adopt immutable data and functional programming where possible.</li><li>Assuming that any or all of these things will go wrong encourages us to develop systems with strong introspection and debuggability built in. Our mindset needs to change from one that treats errors as exceptional conditions to one that emphasizes limiting their scope and maximizing the ability to address them when they occur. Along with the practices mentioned above, two recent related software trends can help with this.</li></ul><h3>Microservices and Containers</h3><p>It’s impossible to work in the software field and not be aware of <a href="http://microservices.io/">microservice architectures</a> and container technology. Microservices and containers are the two biggest buzzwords in the industry today. Unlike many “hot” technology trends in the past, there are good reasons for their popularity. Let’s see how they can help with the issues we’ve been discussing.</p><h4>Problems with Monolithic Applications</h4><p>What can be done when a large, monolithic application encounters a significant error (or even a crash)? The entire functionality of the application may be lost and, once the cause is discovered, correcting it can involve a large-scale retesting effort and a complex deployment process. The deployment may even fail if the runtime environment has changed since the last update. Since we’re assuming there will be errors, this is a less than ideal situation.</p><h4>Fixing the Problems</h4><p>What we would like is an application for which:</p><ul><li>Unrelated subsystems continue to function in the presence of the error.</li><li>We only need to retest the subsystem containing the error.</li><li>We can independently deploy a corrected subsystem simply and quickly.</li></ul><p>We can get all of these benefits via a microservice architecture using containers. The use of microservices <em>does</em> add complexity (though <a href="http://yipee.io/">tools</a> can help), but if done correctly can isolate errors within subsystems. These subsystems can be tested and deployed independently, avoiding the loss of the entire application’s functionality. Containers allow us to apply subsystem updates rapidly and consistently across a variety of underlying environments without special case code for different deployments. Additionally, containers often allow developers to test against a complete application rather than just small pieces, allowing integration issues to be discovered and dealt with earlier in the development process. Together, microservices and containers can assist in discovering errors before deployment and drastically reduce the cost of errors discovered after deployment.</p><h3>State of the Art</h3><p>So, what would an ideal software development model look like? We would construct an application from isolated microservices in which each microservice:</p><ul><li>Would be deployed in a container.</li><li>Would be developed in a test-driven fashion with design and code reviews.</li><li>Would have multiple developers who possess a deep understanding (possibly through pair programming).</li><li>Would go through a complete regression test suite on each code submission which exercises the full integrated application.</li><li>Would be written in a functional style with immutable data where possible.</li><li>Would be instrumented for easy debugging and tunable logging. And we would <em>still</em> get it wrong… but we might have a prayer of fixing it.</li></ul><p><em>Originally published at </em><a href="https://gist.github.com/a392b2ccb5f1b2cd8084c14c6e73fe11"><em>gist.github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14b04f537b9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DRY Kubernetes]]></title>
            <link>https://medium.com/@jrryjcksn/dry-kubernetes-1a11d7fa0054?source=rss-8846ba570c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a11d7fa0054</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[inheritance]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[modeling]]></category>
            <dc:creator><![CDATA[Jerry Jackson]]></dc:creator>
            <pubDate>Fri, 17 Aug 2018 21:09:52 GMT</pubDate>
            <atom:updated>2018-08-20T13:44:13.797Z</atom:updated>
            <content:encoded><![CDATA[<p>If you had to pick one property of a program that makes it easier to maintain, what would it be? Good documentation is certainly a candidate, but if the code itself is bad enough, that probably won’t help you. How about encapsulation? It can be liberating to know that the changes you’re making to a piece of code will only have local effects. However, what if there are other versions of that same code encapsulated elsewhere? Oops. To my mind, the most important property is lack of redundancy. The more places in the code that perform the same function or embody the same implied understanding, the harder it is to maintain without breaking something. This has been well-understood in the database world for decades; normalization is nearly a religious practice in relational database circles. The principle is important enough that it has a well-known acronym: DRY (Don’t Repeat Yourself). How might we apply this principle to Kubernetes manifests?</p><h3>Kubernetes Redundancy</h3><p>First of all, is this even a problem for Kubernetes manifests? For a single application, maybe not. The services, deployments, and other Kubernetes constructs will likely all be unique since any common behavior will be supported by replicated pods at runtime, not by replicated controllers in manifests. However… the ease of container-based deployment has made it common practice to run an application in a variety of environments. At Yipee.io, we run our own application in production, staging, and development environments, from public clouds to developer laptops. This ability to easily replicate a significant application is extremely powerful but brings with it the problem of maintaining several closely related but non-identical application configurations.</p><h3>Existing Approaches</h3><p>There have already been several attempts to address this problem within the industry. Here are some of the most prominent:</p><ul><li><a href="https://docs.docker.com/compose/">docker-compose</a> provides the option to use multiple configuration files (for Swarm applications). Each additional configuration file overrides any previous ones for singular values and merges sets of values with those from earlier files.</li><li><a href="https://helm.sh/">Helm</a> supports manifest <em>templates</em> that can be used to parameterize deployments. Sections of a YAML manifest can be marked as variables which are instantiated via separately specified values.</li><li><a href="https://ksonnet.io/">Ksonnet</a> supports programmatic configuration of Kubernetes apps via code in the extended JSON dialect <em>Jsonnet</em>.</li><li><a href="https://github.com/kubernetes-sigs/kustomize">Kustomize</a> has <em>overlays</em> that can be applied to base YAML manifests to produce specific configurations.</li></ul><p>All of these work (with varying levels of convenience and power) but have issues:</p><ul><li>docker-compose’s multiple configuration files only allow certain values to be changed and only in specific ways.</li><li>Helm templates require the base configuration to anticipate what parts might be changed by later overrides so that template variables can be inserted. Also, they require the user to learn an additional templating language in order to construct shared configurations.</li><li>Ksonnet uses a unique configuration language based on JSON. This forces users to learn a new formalism <em>and</em> makes configurations opaque for new users.</li><li>Kustomize’s overlays are just YAML which means users don’t have to learn a new language. However, the division of a model into many separate related files makes it hard to figure out just what a model contains and the overlays themselves cannot be read in isolation since they are updates to existing files.</li></ul><h3>The Yipee Approach</h3><p>Yipee.io started out as a SaaS-based graphical modeling tool for building Kubernetes manifests and Docker Compose files. It was clear from the start that YAML, though capable of expressing complex configurations, left something to be desired as the sole representation of an application. In particular, though it’s relatively easy to modify an existing configuration for someone who knows how the application works, it’s difficult to see the big picture when looking at a large configuration in YAML. Yipee.io models, which represent Kubernetes objects graphically, make it far easier to understand the structure of an application. We believe the same holds true for managing multiple related configurations. We’re currently building an on-premise version of Yipee.io that runs in a user’s own Kubernetes cluster. Models in the on-premise Yipee.io can be derived from “parent” models and inherit their configurations. The derived models can then be modified but maintain their connections to their parents so that parent-level changes can be automatically propagated. The advantages of this approach include:</p><ul><li>The model you’re working on is always complete. You don’t have to piece it together from chunks spread around the filesystem.</li><li>Graphical diffs between models (and versions of a single model) make it easy to see how a configuration has changed, even if large-scale changes have been made.</li><li>Upstream changes can be accepted/rejected in bulk or they can be addressed on a per-change basis. The system remembers when you change or remove a parent model object and won’t ask you to reprocess a change if you compare the same two models again.</li><li>Changes to ancestor models don’t accidentally break descendant models. Unless a user explicitly accepts upstream changes, the descendant models remain unchanged. If a user <em>does</em> want the changes, though, it’s trivial to accept them.</li><li>Subsystems can be included <em>or</em> embedded. If the subsystem is best viewed as a black box, it can be treated as if it were itself a Kubernetes object. If it’s more naturally seen as a set of top-level parts, it can be unpacked directly into the descendant model.</li></ul><p>Contrast two representations of the same application. First, the Kubernetes manifests:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36be1e8c299d0d6bc6c67eb93dc79021/href">https://medium.com/media/36be1e8c299d0d6bc6c67eb93dc79021/href</a></iframe><p>Here’s the Yipee.io version:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*SGrC6tay8Akbetvt.png" /></figure><p>I think it’s clear that the graphical view helps make the big picture evident. You can see instantly that the application is built from three container types, one deployment, one stateful set, and two services. You can also see how they’re related. If the structure of the application in an ancestor model changes, a graphical diff will make it obvious what changed and what depends on the change.</p><h3>Summary</h3><p>The rise of container-based deployment has led many companies to construct multiple variants of their application configurations. These variants have significant commonality and generate the same issues around redundancy that software developers and database administrators have dealt with for decades. The issues are serious enough that, even though Kubernetes is a relatively young technology, multiple players in the industry have attempted to address them with varying levels of success. Yipee.io is developing a graphical approach to managing Kubernetes models that supports embedded models, model inheritance, and model versioning with an intuitive visual editor and graphical diff capability. The graphical diff makes it easy to manage derived models and propagate changes from ancestors to descendants.</p><p>If anything you’ve read here sounds interesting to you, please visit our <a href="https://yipee.io/dry_apps/">DRY Kubernetes</a> page and fill out the form to let us know. We’re looking for early adopters to help ensure that we’re solving everyone’s problems and not just our own. Also, if you’re a fan of GraphQL, you might want to check out our Kubernetes GraphQL interface that we’re building in support of our on-premise product: <a href="https://github.com/yipeeio/kubeiql">Kubeiql</a>.</p><p>Thanks for reading!</p><p><em>Originally published at </em><a href="https://gist.github.com/8dafe49b15b282c1eb3525611805b4f7"><em>gist.github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a11d7fa0054" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>