<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Brigade Engineering - Medium]]></title>
        <description><![CDATA[Restoring our democracy, one line of code at a time. - Medium]]></description>
        <link>https://medium.com/brigade-engineering?source=rss----41e6ebfe4a48---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Brigade Engineering - Medium</title>
            <link>https://medium.com/brigade-engineering?source=rss----41e6ebfe4a48---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:30:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/brigade-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reloading UITableView while Animating Scroll in iOS 11]]></title>
            <link>https://medium.com/brigade-engineering/reloading-uitableview-while-animating-scroll-in-ios-11-f89910c958b6?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/f89910c958b6</guid>
            <category><![CDATA[uitableview]]></category>
            <category><![CDATA[scroll-animation]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[ios-11]]></category>
            <dc:creator><![CDATA[Ryan Quan]]></dc:creator>
            <pubDate>Thu, 08 Mar 2018 23:50:55 GMT</pubDate>
            <atom:updated>2018-03-08T23:50:55.053Z</atom:updated>
            <content:encoded><![CDATA[<p>Calling reloadData on UITableView may not be the most efficient way to update your cells, but sometimes it’s easier to ensure the data you are storing is in sync with what your UITableView is showing. In iOS 10 reloadData could be called at any time and it would not affect the scrolling UI of UITableView. However, in iOS 11 calling reloadData while your UITableView is animating scrolling causes the UITableView to stop its scroll animation and not complete. We noticed this is only true for scroll animations triggered via one of the UITableView methods (such as scrollToRow(at:at:animated:)) and not for scroll animations caused by user interaction. This can be an issue when server responses trigger a reloadData call since they can happen at any moment, possibly when scroll animation is occurring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*_CWQFyVQMy90G5FDZFjjUw.gif" /><figcaption>Example of scroll animation getting interrupted by a `reloadData` call</figcaption></figure><h4>Solution</h4><p>If you don’t want to switch to updating cells individually, one solution is to defer your reloadData call until all scroll animations are complete. We use a utility class to encapsulate interactions with aUITableView.</p><pre>class TableViewUtility {</pre><pre>    let tableView: UITableView<br>    var data: Data?<br>    <br>    init(tableView: UITableView) {<br>        self.tableView = tableView<br>    }</pre><pre>    func showData(_ data: Data) {<br>        self.data = data<br>        tableView.reloadData()<br>    }</pre><pre>    // Other UITableViewDataSource and UITableViewDelegate methods<br>}</pre><p>Now we add:</p><ul><li>A method to animate scrolling (here we use scrolling an index path to the bottom of the UITableView as an example)</li></ul><pre>func scrollIndexPathToBottom(indexPath: IndexPath) {<br>   isAnimatingScroll = true<br>   tableView.scrollToRow(at: indexPath, at: .bottom, animated: true)<br>}</pre><ul><li>A variable to keep track of whether or not scroll is being animated</li></ul><pre>var isAnimatingScroll = false</pre><ul><li>A variable to store any Data we need to load once scroll animations have completed</li></ul><pre>var pendingData: Data?</pre><ul><li>Update showData to store Data if scroll animation is occurring when we try to reload the table</li></ul><pre>func showData(_ data: Data) {<br>    guard !isAnimatingScroll else {<br>        pendingData = data<br>        return<br>    }</pre><pre>    self.data = data<br>    tableView.reloadData()<br>}</pre><p>Altogether it looks like:</p><pre>class TableViewUtility {<br>    ...</pre><pre>    var isAnimatingScroll = false<br>    var pendingData: Data?</pre><pre>    func scrollIndexPathToBottom(indexPath: IndexPath) {<br>        isAnimatingScroll = true<br>        tableView.scrollToRow(at: indexPath, at: .bottom, animated: true)<br>    }</pre><pre>    func showData(_ data: Data) {<br>        guard !isAnimatingScroll else {<br>            pendingData = data<br>            return<br>        }</pre><pre>        self.data = data<br>        tableView.reloadData()<br>    }<br>}</pre><p>Now we need to figure out when scroll animations are finished so we can show any pendingData. Unfortunately the scroll methods that give us scroll animations for UITableView don’t take completion blocks. We found that putting these calls into a CATransaction and setting a completion block didn’t work either, so we used the UIScrollViewDelegate method scrollViewDidEndScrollingAnimation(_ scrollView: UIScrollView)</p><pre>func scrollViewDidEndScrollingAnimation(_ scroll: UIScrollView) {<br>    isAnimatingScroll = false<br>    if let pendingData = pendingData {<br>        showData(pendingData)<br>        pendingData = nil<br>    }<br>}</pre><p>There is one gotcha with this. If tableView.scrollToRow(at:at:animated:) is called when the given IndexPath is already at the specified position, scrollViewDidEndScrollingAnimation does not get triggered since no scrolling animation occurred. To make sure our isAnimatingScroll flag doesn’t get stuck with a value of true , we have to make sure the IndexPath we’re scrolling to is not already in position:</p><pre>func scrollIndexPathToBottom(indexPath: IndexPath) {<br>    let scrollPadding: CGFloat = 1.0<br>    let indexRect = tableView.rectForRow(at: indexPath)<br>    let targetScrollYOffset = indexRect.origin.y - tableView.frame.height + indexRect.size.height + tableView.contentInset.bottom</pre><pre>    guard fabs(tableView.contentOffset.y - targetScrollYOffset) &gt; scrollPadding else {<br>        return<br>    }    </pre><pre>    isAnimatingScroll = true<br>    tableView.scrollToRow(at: indexPath, at: .bottom, animated: true)<br>    }</pre><p>We use a scrollPadding of 1.0 here to account for small inconsistencies in view placements.</p><p>The final thing to watch out for is if you are updating cells individually as well, make sure your pendingData is updated appropriately otherwise you may get stale data being loaded after scroll animations. This means <strong>all</strong> potential properties of your data should be up to date. For example, if a scroll animation gets triggered and two separate properties of your data get updates, the pendingData should have both of these changes. This may seem obvious, but it can be subtle depending on your app architecture.</p><h4>The Result</h4><p>With this, we get smooth scroll animations even when we try to reload the data mid-animation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*vk9tIjCNaPxtXfjTNUbTcg.gif" /></figure><p>If you’ve encountered this problem as well, feel free to leave a comment on how you solved it or if this method was helpful!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f89910c958b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/reloading-uitableview-while-animating-scroll-in-ios-11-f89910c958b6">Reloading UITableView while Animating Scroll in iOS 11</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BigQuery — one store to rule them all]]></title>
            <link>https://medium.com/brigade-engineering/bigquery-one-store-to-rule-them-all-222429788d4?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/222429788d4</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Mark Cooper]]></dc:creator>
            <pubDate>Thu, 20 Apr 2017 14:37:28 GMT</pubDate>
            <atom:updated>2017-04-23T16:29:11.148Z</atom:updated>
            <content:encoded><![CDATA[<p>At Brigade we are building tools to help empower voters — specifically activists and organizers. We believe one of the keys to accomplishing this lies in data — data about voters, politicians, as well as various other civic datasets.</p><p>We love data here. So much so that we’ve got it stashed in all sorts of places and in various formats. This is of course a problem, as we often want to join across disparate datasets. Before Brigade moved to BigQuery, we used Spark SQL to do cross database joins — <a href="https://brigade.engineering/how-to-radically-streamline-complex-analytics-f50901c433e3#.sba7wpe72">as discussed here</a> in a previous post. By way of Apache Zeppelin and Spark SQL we gave much of our company the ability to do really sophisticated queries. It worked well for a while — until our datasets got big. And then it broke. It broke a lot.</p><p>Not that we don’t still love these tools but we quickly realized they were never going to be stable enough to just work — not given our cluster and dataset sizes. We were either going to have to lock down what our internal customers could do or move to a different tool.</p><h3>Enter BigQuery</h3><p>BigQuery is a curious animal. It‘s a public implementation of <a href="https://cloud.google.com/files/BigQueryTechnicalWP.pdf">Google’s Dremel system</a>. Data is stored and columnar format and parallelized across a very large number of nodes.</p><p>You pay (primarily) for the bytes you read — at a rate of $5/terabyte for each byte queried but very little for data at rest ($20/terabyte for storage).</p><p>We are now replicating data from all of our datastores (Cassandra, Mysql, Parquet and CSV files) into BigQuery. While we are early in our use, our usage costs — something that may cost a $1000s in Redshift are close to zero:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*7JYKC2kplswp2ITBarHGGQ.png" /></figure><p>To some this may sounds unattractive, paying per query, but in reality you may not end up paying much depending on your usage. BigQuery is a column oriented database and you only pay for the bytes you read in a query — so if you only hit two of ten columns in a table you only get charged for reading those columns. Moreover, the columns are compressed, often taking up much less space than they would in a traditional RDBMS.</p><h3>Pulling It Together</h3><p>The tooling around BigQuery definitely lags compared to similar stores such as Redshift so we had to build much of what we needed ourselves.</p><p>To really make use of BigQuery, we really needed to get all our important data into it. To do this, we started simple by writing a few Spark jobs which pull data from our Rackspace hosted cluster and write to BigQuery (via Google Cloud Files). Two two libraries we tried out to help us with this were: <a href="https://github.com/appsflyer-dev/spark-bigquery">this one</a> from Appsflyer and <a href="https://github.com/spotify/spark-bigquery/">this one</a> from Spotify.</p><p>Each library has its own merits. The Appsflyer library uses JSON encoding to push data to BigQuery, which mean we can create tables with a wider set of types (namely DataTime fields). The Spotify client uses Avro behind the scenes which is far more compact — this means we can copy tables far more quickly. The downside though is that Avro doesn’t support a DateTime type so the tables that get created ended up turning our user friendly date types into unix timestamps. We ended up going with Spotify’s library because, in addition to the performance benefits we found it could handle a wider array of source inputs (read: quirky MySQL column types).</p><p>To get around the lack of DateTime types we end up creating views on top of the tables. The views simply expose the existing fields and convert the timestamps back to something our analyst can work with. For example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1cfb8581630467806397c16af2e083f2/href">https://medium.com/media/1cfb8581630467806397c16af2e083f2/href</a></iframe><p>Our data ingestion flow is pretty straightforward — a set of Spark jobs that run nightly and push full snapshots into BigQuery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Va6SdEB5Rhmw8OqWBJNi1w.png" /></figure><p>In the simplest case — say copying data from Parquet to BigQuery, the meat of your code is:</p><blockquote>val sourceFileDF = sqlContext.read.parquet(“/prod/data/myfile/”)<br>val sourceFileDF.saveAsBigQueryTable(“my-project:my_dataset.my_table”)</blockquote><p>We built a simple tool to help us with RDBMS replication specifically. We’ve open sourced it and you can get it here (it’s still very rough):</p><p><a href="https://github.com/markncooper/mysql-bigquery-replicator">https://github.com/markncooper/mysql-bigquery-replicator</a></p><p>It takes a database, credentials and a white or black list. After that it does the rest of the magic of discovering your tables and writing them to BigQuery (via Cloud Files). Big tables automatically get copied a chunk at a time in parallel. Parallelization is done using the primary key of the row and only happens if a given table exceeds a certain size threshold.</p><h3>Getting to the data</h3><p>BigQuery gives you a pretty barebones UI through which to load data and run queries. The real power comes from integration with other tools. In our case, we choose to plug in Chartio. Plugging Chartio into BigQuery takes only a few minutes. It’s slick UI lets you put together complex reports and dashboards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*c9Tu4hlx0G5eydBdj1Gryg.png" /></figure><p>One restriction of Chartio is that it doesn’t allow big joins across tables stored in different BigQuery datasets. To solve this, we load all our data into the same dataset. To fix the potential name-space collision issue we ended up prefixing all our tables with “store_database_” i.e. mysql_analyticsdb_customers. A little clunky but it did the trick.</p><h3>Query Plans and Stats</h3><p>After that our analysts, product managers and engineers can now write complex queries and visualizations to their hearts content.</p><p>Below is the results of the explain tab you get when running one of our basic join and aggregation queries. The query itself took around 50 second — not bad for a join with 5 billion rows on one side and 500+ million on the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AILH316Fg8jXeEvHI92b_g.png" /><figcaption>Query Explanation</figcaption></figure><p>This query also ended up touching ~200 gigabytes of data — which ended up costing around $1 to run.</p><h3>The Rub</h3><p>BigQuery isn’t right for all problems. Queries take at least a second or two to execute. There are also no indexes so all of your queries do full table scans. On the plus side, no pesky indexes.</p><p>Tables are more or less immutable and append only. While rows have keys, they are only used to guarantee exactly-once streaming semantics within one minute of a write. If you write two updates with the same row key in the same minute BigQuery will do the de-duplication. If an hour later you write a second row with the same row key you’ll end up with two rows in your store. In our case we rewrite tables in their entirety so this is not a problem, but if it is for you — Google recommends putting a view on top of your table to do the de-duplication and then cleaning up the old rows periodically.</p><p>While there are no indexes, you can though organize your date using a date partitioner. Say you are only querying the last week of data but have a year’s worth in your store, the partitioner will only hit rows for the days you need. This will speed up the query time and dramatically lowering your costs.</p><h3>Similar Tools</h3><p>Other tools that we also considered:</p><ul><li><a href="https://prestodb.io/">Presto</a>: A distributed SQL query engine Facebook released back in 2013. It sits on top of your existing datastores (HDFS, Cassandra, RDBMSs) and allows for similar cross DB joins.</li><li><a href="https://aws.amazon.com/athena/">Athena</a>. A hosted offering from Amazon, Athena is a hosted version of Presto. It pricing is very similar to that of BigQuery.</li><li><a href="https://aws.amazon.com/redshift/">Redshift</a> — Amazon’s very popular Postgres-like distributed analytics database. It scales really well, though in our case it would be much more expensive as we would have to pay for the Redshift cluster even when no queries are being run.</li><li><a href="https://impala.incubator.apache.org/">Impala</a> — a fast open source distributed database from Cloudera that sits on top of HDFS. It has the downside that, much like Redshift, it would have cost us much more because there is no pay-per-query model for Impala.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=222429788d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/bigquery-one-store-to-rule-them-all-222429788d4">BigQuery — one store to rule them all</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dock: A tool to define, build, and run development environments with Docker]]></title>
            <link>https://medium.com/brigade-engineering/dock-a-tool-to-define-build-and-run-development-environments-with-docker-84509458703?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/84509458703</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Shane da Silva]]></dc:creator>
            <pubDate>Tue, 15 Nov 2016 16:00:30 GMT</pubDate>
            <atom:updated>2016-11-15T16:00:29.882Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://www.brigade.com/">Brigade</a>, we’ve been using Docker both for development and production with great success for the past two years. While a post about how we use Docker in production is forthcoming, the tools we’ve built using Docker to make our development lives easier and more effective is something particularly worth sharing.</p><h3>Problems we were trying to solve</h3><p>Not everyone has the same problems, and so what may work for our team may not work for yours.</p><p>The high-level problems were:</p><h4>Engineers weren’t able to quickly get a working development environment for our different services written in different languages requiring different runtime dependencies</h4><p>We have a variety of projects written in Ruby, Scala, and Python, each with slightly different dependencies (and potentially conflicting ones). Ideally, installing all software dependencies and having a working instance of the project would be as simple as running one command.</p><h4>Discrepancies between development and testing environments made reproducing failures difficult</h4><p>We specifically had issues where a test would fail on our Jenkins build machines but not locally, since most developers were on macOS whereas our build machines were CentOS. Also, we wanted to enforce the particular versions of software that developers were using (e.g. MySQL, Redis), which is difficult on macOS, since most developers use <a href="http://brew.sh/">Homebrew</a> which always pulls the latest versions by default (there are some workarounds such as <a href="https://github.com/Homebrew/homebrew-versions">homebrew-versions</a>, but they are far from perfect).</p><p>Furthermore, due to the variety of projects we have, we needed to support multiple “types” of build machines. We wanted to treat these machines like cattle rather than pets and make them homogeneous to avoid dealing with installing and managing conflicting dependencies of different projects.</p><h4>Testing major system changes in integration tests was time-consuming and potentially disruptive</h4><p>Upgrading MySQL or Redis can be a daunting and stressful task. We have a staging environment with production-like data, but in a company with teams developing native apps against our staging environment, testing anything too risky in staging is a recipe for blocking many engineers across the entire company!</p><p>If we wanted to try out the upgrade in Jenkins builds first, upgrading backend services on our Jenkins workers would involve modifying one of the workers (making it a unique snowflake), running the tests on it and reverting the machine back to the original state if the tests failed. This is a time-consuming ordeal.</p><p>We wanted a solution that would ideally let us change a version in a [Docker]file, have Jenkins test the commit, and get feedback on whether anything broke.</p><h3>How Docker helped solve some of these problems</h3><p>A common theme of all of these problems is the need to make something reproducible, and to isolate environments from each other. These are both major advantages offered by using Docker containers.</p><h4>Dock: Vagrant for Docker containers</h4><p>Our solution was to create a tool called <a href="https://github.com/brigade/dock"><strong>dock</strong></a>. It loads a <strong>.dock</strong> configuration file from the root of a repository and uses it to create a development environment with all the necessary dependencies available inside the container, mounting the repository within it. If you’ve used <a href="https://www.vagrantup.com/">Vagrant</a>, this surely sounds familiar. We avoided Vagrant so we wouldn’t require Ruby as a global dependency, keeping the tool as simple as possible and easy to run in a wide variety of Docker images without adding to the image size by including the Ruby runtime. Using containers is also way faster that spinning up and down full virtual machines.</p><p>At a high level, <strong>dock</strong> uses the <strong>.dock</strong> configuration file to create a <a href="https://docs.docker.com/engine/reference/run/"><strong>docker run</strong> command</a> which it then executes, resulting in the developer being dropped into a container with all the tools and services ready for development. If there is a Dockerfile it needs to build to create the image, it builds that Dockerfile first. Here’s an example of the resulting <strong>docker run</strong> command that is created for one of our projects:</p><pre>docker run \<br>  --name project-dock --hostname project-dock \<br>  --rm --interactive --tty --privileged \<br>  --workdir /workspace \<br>  --volume ~/src/project:/workspace:rw \<br>  --volume /usr/local/bin/dock:/usr/local/bin/dock \<br>  --volume project-dock_gems:/usr/local/bundle \<br>  --volume ~/src/project/script/dock-start-docker-daemon:/entrypoint.d/start-docker-daemon:ro \<br>  --volume project-dock_docker:/var/lib/docker:rw \<br>  --volume project_cache:/home/app/.cache \<br>  --volume project_local-storage:/home/app/.local \<br>  --volume ~/src/project/.my.cnf:/home/app/.my.cnf \<br>  --env INSIDE_DOCK=1 -p 3306:3306 -p 6379:6379 -p 9292:9292 \<br>  project:dock script/start-everything</pre><p>Look how long this command is! It’s complex and difficult to grok at first glance. The <strong>.dock</strong> configuration file responsible for this command is quite long, but it’s well-documented and easier to maintain than a long list of flags broken up over multiple lines. Here’s an excerpt of what the configuration looks like with many lines removed for brevity (it’s just a Bash script that <strong>dock</strong> <a href="http://www.tldp.org/HOWTO/Bash-Prompt-HOWTO/x237.html">sources</a>):</p><pre># File: /path/to/project/.dock</pre><pre>dockerfile Dockerfile # Use Dockerfile from the repository</pre><pre># Store gems in a persistent volume<br>volume &quot;$(container_name)_gems:/usr/local/bundle&quot;</pre><pre>publish 3306:3306 # MySQL<br>publish 6379:6379 # Redis<br>publish 9292:9292 # Sinatra</pre><pre># Command to execute when no explicit command passed to Dock<br>default_command script/start-everything</pre><p><em>Note: some of the commands (e.g. </em><strong><em>user_id</em></strong><em> and </em><strong><em>container_name</em></strong><em>) are helpers provided by Dock.</em></p><p>Here’s the corresponding Dockerfile that’s referenced in the configuration above:</p><pre># File: /path/to/project/Dockerfile</pre><pre>FROM ruby:2.3.1</pre><pre># Install Docker-related software<br>RUN curl -L <a href="https://get.docker.com/builds/Linux/x86_64/docker-1.12.3.tgz">https://get.docker.com/builds/Linux/x86_64/docker-1.12.3.tgz</a> \<br>    | tar -xzf - -C /usr/local/bin --strip-components=1 \<br>    &amp;&amp; curl -L <a href="https://github.com/docker/compose/releases/download/1.8.1/docker-compose-Linux-x86_64">https://github.com/docker/compose/releases/download/1.8.1/docker-compose-Linux-x86_64</a> --output /usr/local/bin/docker-compose \<br>    &amp;&amp; chmod +x /usr/local/bin/docker-compose</pre><pre>RUN yum install -y \<br>  # Allow us to compile mysql2 gem native extensions<br>  Percona-Server-client-56 Percona-Server-devel-56 \<br>  # Redis client<br>  redis \<br>  &amp;&amp; yum clean all</pre><p>Here’s an example of the output a user would see running <strong>dock</strong>:</p><pre>$ cd src/project<br>$ dock<br>Building Dockerfile into image project:dock...<br>Sending build context to Docker daemon 3.584 kB<br>Step 1 : FROM ruby:2.3.1<br>2.3.1-latest: Pulling from library/ruby</pre><pre>Digest: sha256:068ed8e5b7674ff899c1869b996498f6bef71ebcc6d30f56db510cc2505f2250<br>Status: Image is up to date for ruby:2.3.1<br> ---&gt; 813e5b0bd370<br>Step 2 : RUN curl -L <a href="https://get.docker.com/builds/Linux/x86_64/docker-1.12.3.tgz">https://get.docker.com/builds/Linux/x86_64/docker-1.12.3.tgz</a> | tar -xzf - -C /usr/local/bin --strip-components=1     &amp;&amp; curl -L <a href="https://github.com/docker/compose/releases/download/1.8.1/docker-compose-Linux-x86_64">https://github.com/docker/compose/releases/download/1.8.1/docker-compose-Linux-x86_64</a> --output /usr/local/bin/docker-compose &amp;&amp; chmod +x /usr/local/bin/docker-compose<br> ---&gt; Using cache<br> ---&gt; cf874968fcdb<br>Step 3 : RUN yum install -y Percona-Server-client-56 Percona-Server-devel-56 redis &amp;&amp; yum clean all<br> ---&gt; Using cache<br> ---&gt; 3cf899677bf6<br>Successfully built 3cf899677bf6<br>Dockerfile built into project:dock!<br>Starting container project-dock from image project:dock</pre><pre>...&lt;script/start-everything command drops user into a shell here&gt;...</pre><p>The key takeaway here is that an engineer is able to get a working development environment with no dependencies other than Docker and Bash by just running <strong>dock</strong> in the project’s repository.</p><h4>Running scripts within a Dock environment</h4><p>You might be saying “neat trick, but you’re still just executing <strong>docker build</strong> and <strong>docker run</strong> commands,” and you’d be correct. However, another problem Dock solves is making it easy to reproduce Jenkins builds exactly as they would be on build machines, regardless of your host OS.</p><p>Dock accomplishes this by allowing you to add it to a script’s <a href="https://en.wikipedia.org/wiki/Shebang_(Unix)">shebang line</a> so it is used as the interpreter. Here’s an example of a script that runs RSpec tests:</p><pre>#!/usr/local/bin/dock bash</pre><pre># Ensure we shut down MySQL cleanly before terminating<br>trap &quot;docker-compose stop&quot; EXIT INT QUIT TERM</pre><pre>docker-compose pull mysql<br>docker-compose up -d mysql</pre><pre>bundle install<br>bundle exec rake db:migrate<br>bundle exec rspec</pre><p>Thanks to the shebang line, executing this script will result in it automatically being run with Bash inside a Dock container (if we aren’t already in one; if we are it just executes the script with Bash). This makes it simple to check out a commit locally and run a script to quickly reproduce build failures. A developer doesn’t even need to know that a script runs inside a Dock container, as the shebang line automatically takes care of that for them.</p><p>You’ll also notice we’re using <a href="https://docs.docker.com/compose/"><strong>docker-compose</strong></a> to manage the MySQL service that RSpec tests hit during the test run. This saves us having to reinvent the wheel, as <strong>docker-compose</strong> does a fantastic job of allowing you to define services and start/stop them as needed. This requires having a Docker daemon running inside the container (a.k.a. “Docker-in-Docker”), which comes with a <a href="https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/">number of gotchas</a>, but we’ve used it with great success.</p><h4>Running services inside the Dock container</h4><p>Running a Docker daemon inside the container gives us a lot of power, since we can now stand up services that are isolated to that container. This allows you to run multiple projects simultaneously even if they run services that listen on the same port (e.g. MySQL or Redis). You could configure these to use different ports for each of your projects, but then you need to keep track of all of these ports. Using well-known ports is easier since it requires less configuration, and eases the cognitive load on engineers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*iyjnFEqUpHd7-kSOCQ2ROg.png" /><figcaption>Diagram of Docker daemon running services in a Dock container via docker-compose</figcaption></figure><p>Inside our Dock containers, we use <strong>docker-compose</strong> to spin up the various services needed by a project. Here’s a <strong>docker-compose.yml</strong> for a project that depends on Redis:</p><pre>version: &#39;2&#39;</pre><pre>services:<br>  redis:<br>    image: &#39;redis:2.8&#39;<br>    network_mode: host<br>    volumes:<br>      - &#39;redis:/data&#39;</pre><pre># Define names of volumes that you want to be preserved between<br># container restarts.<br># These are referenced above in the `volumes` option of a service.<br>volumes:<br>  redis:</pre><p>If you run <strong>docker-compose up -d</strong> in the Dock container, Redis will be started in a container in the background. Notice that we specify <strong>network_mode: host</strong> — since we’re running the Redis container with the Docker daemon running <em>inside</em> the Dock container, the network namespace is already isolated. This removes the need to specify the <strong>port</strong> option for services since we don’t need to proxy from outside the container to inside the container.</p><p>We can automate this so that when you run dock inside the project we automatically start up all the needed background services. Recall the last line of our <strong>.dock</strong> configuration file:</p><pre># Command to execute when no explicit command passed to Dock<br>default_command script/start-everything</pre><p>We can have <strong>script/start-everything</strong> run <strong>docker-compose up -d</strong> for us. A simplified example:</p><pre># File: /path/to/project/script/start-everything</pre><pre>#!/usr/local/bin/dock bash</pre><pre># Ensure we shut down Redis cleanly before terminating<br>trap &quot;docker-compose stop&quot; EXIT INT QUIT TERM</pre><pre>docker-compose pull redis<br>docker-compose up -d redis</pre><pre># Drop into a shell<br>bash -il</pre><p>The above script allows us to add custom logic for starting up our development environment, using nothing but Bash and the Docker tool set.</p><p>This has proven incredibly useful at Brigade. It means engineers can get up and running on their laptop by installing Docker and running a single command, <strong>dock</strong>, without any additional setup.</p><h3>Problems we haven’t solved yet</h3><p>We won’t lie to you and say everything is now sunshine and rainbows. While Docker helped solve many of the problems above, we still face challenges. However, we think the net result is a win for our team’s productivity and ability to iterate quickly, as well as lowering the barrier for engineers to jump between different projects.</p><p>Examples of issues we still have:</p><h4>Problem 1: Different Docker images between development and production</h4><p>Since development environments often include a number of additional tools used in debugging and regular development which you wouldn’t include in production, we still maintain different images for deploying our applications to production. This hasn’t been a huge issue but it would be nice to know that the software we test locally uses the exact same image as what we deploy to production, as this ensures fewer possible ways for our code to work in development but not production. At the end of the day, it feels somewhat impractical given the importance of developer time, and so having a development/testing-specific environment that optimizes for developer time is a pragmatic tradeoff here.</p><h4>Problem 2: Starting up dependent services in different repos</h4><p>When developing microservices, you’ll inevitably have one service make a call to another. While we usually test these calls against our staging environment, sometimes one needs to make changes to both services, which makes this difficult to test unless you can run them both locally. Since we run the service inside the Dock container, this makes it difficult to link with other services running inside a different Dock container, since they are both isolated from the host network.</p><p>We don’t yet have a good solution for this outside of deploying the change to staging and testing it out, risking a break in staging in the process. We haven’t bothered solving this problem because needing to run both services locally is rare for us.</p><h3>Let us know what you think</h3><p>How are you using Docker to optimize your development and testing processes? If you’re interested in giving Dock a try or suggesting improvements, feel free to <a href="https://github.com/brigade/dock">check out the project!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84509458703" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/dock-a-tool-to-define-build-and-run-development-environments-with-docker-84509458703">Dock: A tool to define, build, and run development environments with Docker</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting how you’ll vote: a follow-up]]></title>
            <link>https://medium.com/brigade-engineering/predicting-how-youll-vote-a-follow-up-3eaf5452c4f6?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/3eaf5452c4f6</guid>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Dave Kincade]]></dc:creator>
            <pubDate>Mon, 07 Nov 2016 17:43:29 GMT</pubDate>
            <atom:updated>2016-11-07T17:43:29.268Z</atom:updated>
            <content:encoded><![CDATA[<p>I <a href="https://brigade.engineering/predicting-vote-pledges-via-social-alignment-8ffefa3d645#.4z7o1wasx">recently wrote</a> about how Brigade is able to help people decide how they should vote on Election Day. In that post, I talked about our <a href="https://www.brigade.com/ballot-guide">social ballot guide</a>, which contains thousands of state and federal races as well as every statewide ballot measure in the country. Our tool compares your views on a range of issues to users who’ve pledged to vote for particular politicians or ballot measures in order to arrive at a voting recommendation for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*pZYsn1lWL0vuj8gpgUWZrg.png" /><figcaption>Personalized recommendations for president</figcaption></figure><p>We calculate alignment by having users agree or disagree with short opinion statements we call positions and comparing responses to those positions. It is our main indicator for whether two people may be like-minded and it has a major presence throughout the product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*kBEpfgA3uaL1Keb4dUTiFQ.png" /><figcaption>A position on Brigade</figcaption></figure><p>Since that blog post was published, <strong>more than a quarter million people have pledged their vote for a presidential candidate</strong>. We’ve added more races, ballot measures, dramatically improved the ability to share your pledges and attempt to sway others to vote with you. We’ve also reworked the system to incorporate more information to better calculate alignment.</p><p>What we haven’t changed is our calculation for alignment. In the screenshot above, the personalized recommendations for Hillary Clinton and Jill Stein are very close to each other, and the issues that have greatest alignment are the same. This was something that we’d identified as a weakness, and it gets more difficult to determine as candidates are more similar to each other:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*XVqzkBG_mnkcApq7vBRSUQ.png" /></figure><h3>Results of experimentation</h3><p>Over the course of a few weeks, our data team experimented with everything from <a href="https://en.wikipedia.org/wiki/Logistic_regression">logistic regression</a> to <a href="https://en.wikipedia.org/wiki/Random_forest">random forest</a>, up to <a href="https://en.wikipedia.org/wiki/Deep_learning">deep learning</a>. We focused largely on the presidential race, where we have the most data.</p><p>Most algorithms behaved similarly; random forest and deep learning in particular were very close. In general, for Hillary Clinton and Donald Trump we saw <a href="https://en.wikipedia.org/wiki/Precision_and_recall">precision</a> of ~0.85 (where the predictions that we made were for the right candidate and recall of ~0.6 (where the predictions that we made were for the candidate that users had actually chosen). As a rule of thumb, precision of 0.8 is roughly what one can expect with an unoptimized application of a model in most uses. Performing the same analysis against social alignment, we found precision of 0.6, implying that most machine learning models could be better at predicting who someone will vote for compared to what we are doing.</p><p>Looking at Jill Stein and Gary Johnson, we found that most models tended to optimize them out of the system — any vote pledges for them were mostly ignored. Recall for them dropped way down as a result. This is not true for social alignment, where we don’t take the number of pledges of each group into account. The more sophisticated models tended to have better recall for all candidates, especially when they took some factors into consideration that tend to sway Stein supporters from Clinton supporters.</p><h3>The objective</h3><p>This leads us to the question that we should have been asking from the beginning (and indeed was asked one or two members of our team): What should we be solving for? It’s useful to tell our users who we think they will vote for, which is what we get with our machine learning models. However, we built social alignment to tell users who they <strong>should</strong> vote for, based on how similar their views are to others who are pledging to vote. “Should” is a nebulous term, and we grappled with the difference for a time.</p><p>In the end, we decided that the distinction between the two is significant enough that identity of our recommendation being based on alignment is important. Once we start applying ML to this problem, we start to lose our ability to explain “why” someone should pledge to vote for a candidate. One of the pillars of Brigade is that by acting together as a community, we can have greater effect than being alone. By removing the feeling of the community from our recommendations, we risk losing a bit of that purpose.</p><p>Of course, that doesn’t mean that our analysis is done. In the previous post, we laid out a bunch of issues with alignment as it stands today, and those should be corrected. There is a lot that we can do to figure out the significance of each agree and disagree action that a user takes, identifying what positions unify groups of people, and what positions aren’t particularly important to a group.</p><p>Improving these can improve alignment across our product, and we’re eager to get to it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eaf5452c4f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/predicting-how-youll-vote-a-follow-up-3eaf5452c4f6">Predicting how you’ll vote: a follow-up</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting vote pledges via social alignment]]></title>
            <link>https://medium.com/brigade-engineering/predicting-vote-pledges-via-social-alignment-8ffefa3d645?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/8ffefa3d645</guid>
            <category><![CDATA[2016-election]]></category>
            <category><![CDATA[politics]]></category>
            <dc:creator><![CDATA[Dave Kincade]]></dc:creator>
            <pubDate>Sat, 24 Sep 2016 00:20:49 GMT</pubDate>
            <atom:updated>2016-09-24T00:20:49.104Z</atom:updated>
            <content:encoded><![CDATA[<p>Brigade has a big focus on connecting voters with varying viewpoints, showing them their similarities and differences, and helping them influence the policies and elections that affect their lives. It’s fundamental in our mission to reenergize public participation in our democracy. One significant tool at our disposal is the concept of alignment.</p><p>Alignment is a measure that helps users and Brigade as a platform establish a sense of similarity between a single user and another user or some portion of the platform in aggregate. For instance, look at this position:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*UVd8MaWtdemD2jynJSXInQ.png" /></figure><p>Taking the “agree” side on this position will increase a user’s alignment with anyone who agrees, while decreasing alignment with anyone who disagrees. Here’s a simple model to help you reason how alignment on Brigade works:</p><p>Alice and Bob are taking positions on immigration. When a position is taken, they can agree with it (✓), disagree with it (✗), or leave it as unsure. Brigade has seven immigration-specific positions that address key themes related to the issue and we require that a user take at least five of them to begin to show alignment with other users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*ET1t9_Z4PmnvWQWM0T8rVg.png" /></figure><p>In this example, Alice and Bob have three overlapping positions (3, 4, and 5) and share the same response on only one of those (5). Thus you could conclude that they agree 33% of the time. That’s a really simple way to think about similarity.</p><h3>Aligning with candidates</h3><p>An important tool that Brigade is rolling out for the 2016 election is our personalized <a href="https://www.brigade.com/ballot-guide">Ballot</a>. It provides state and national voting recommendations from Brigade users who share your views and in so doing helps you understand which candidates align with you. The tool also allows you to recruit others to join you in voting for a particular candidate. That action is called a <strong>pledge</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*PKPk7s_-4Z3L7QaZRrUcTA.png" /><figcaption>Brigade’s ballot guide, after pledging</figcaption></figure><p>To date, we’ve calculated how you align with politicians in the same way we calculate alignment between users: you take a number of positions on a range of issues and establish similarity based on how much you agree on those issues. This has been an effective approach for presidential candidates and others on the national stage. We’ve painstakingly defined what we think represents many politicians’ platforms on a variety of issues.</p><p>However, the further down-ballot you go, the less likely the Democratic or Republican Party platforms apply and the harder it is for us to determine a politician’s views. For instance, San Francisco Supervisor Scott Wiener may be a Democrat but as a candidate for California State Senate, he has different concerns than his party at a federal level.</p><h3>Enter social alignment</h3><p>What if we applied one level of indirection? We might not know what Scott Wiener thinks, but we have a rich set of data about what his supporters (and supporters of his opposition, challenger and fellow San Francisco Supervisor Jane Kim) think. We have a mechanism to establish similarity for a user’s beliefs vs a group of people. If we apply similarity analysis against each candidate in a race (eg California State Senate District 11), we can come up with a recommendation for which candidate best represents voters that have similar views to a user.</p><p>Now we don’t need to curate a corpus of politicians’ likely responses to our positions. We can provide recommendations for any race or against any ballot measure as long as enough people have pledged for candidates (or sides, for measures).</p><p>You might spot a few problems with this approach:</p><ol><li>In the alignment model laid out earlier, each position is considered equally. In the Wiener-Kim race, views on defense are not as relevant as views on transportation and housing. Transportation and housing issues tend to be resolved at the state and local level, while defense issues are mostly a federal concern. Between districts at the same level, not all issues will be equally relevant (eg, California may have considerations for water conservation that don’t apply to Florida). Even within a single district, each candidate may be focusing on different positions and issues that rally their supporters to different degrees.</li><li>Not all (or even most) supporters of a candidate will agree on individual positions. There are a few reasons for this, not least of which is that candidates may not have published their stances on all positions such that there is no standard to rally behind. Even if all positions were known, in many cases a voter chooses between a small handful of candidates and likely none align with voters on all of the issues that they care about.</li><li>Having multiple candidates in a race can accentuate the differences between candidates in a way that doesn’t really fit reality (a phenomenon known as <a href="https://en.wikipedia.org/wiki/Distinction_bias">distinction bias</a>). For instance, the presence of the Green Party’s Jill Stein as a presidential candidate on Brigade shifts the average pledger for Hillary Clinton to be more centrist while the two candidates (as a whole) have a relatively similar platform.</li></ol><p>Nevertheless, the approach to similarity that we’ve built will result in a powerful tool to expose the supporting base for each candidate and give some immediate feedback to voters as to what sorts of issues they can expect candidates to be concerned with.</p><p>It ultimately results in a prediction, where the inputs are your responses to core positions on Brigade (as well as anybody else participating in your elections) and the output is a measure of how much each candidate represents your views.</p><h3>What next?</h3><p>Some of the challenges to our approach to social alignment are endemic to our approach to alignment in general. Similarity and prediction are incredibly hard problems, and we’re just scratching the surface so far. The specifics will have to wait for another post, but we’re excited for how we can better solve this issue of candidate vote pledge prediction.</p><p>Candidate vote pledging ends up being a classic problem for deep learning. The fact that we’ve got vote pledge data on Brigade means that we can train models very effectively. We believe that deep learning eliminates several issues that we have with our current approach — not all positions are equally important for each candidate, and deep learning highlights the positions that correlate with pledges. We’re particularly excited to compare this against our existing “direct” alignment approach where we attempt to divine the positions that each candidate would take, if they were on Brigade.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ffefa3d645" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/predicting-vote-pledges-via-social-alignment-8ffefa3d645">Predicting vote pledges via social alignment</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Screen Flows with VIPER on iOS]]></title>
            <link>https://medium.com/brigade-engineering/building-screen-flows-with-viper-on-ios-3459353d3a10?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/3459353d3a10</guid>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Nebs Petrovic]]></dc:creator>
            <pubDate>Tue, 15 Mar 2016 00:17:17 GMT</pubDate>
            <atom:updated>2016-03-15T00:17:17.001Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*8YmcwjCTptOAcPzR8CKhug.jpeg" /><figcaption>Image: <a href="https://www.flickr.com/photos/pagedooley/">Kevin Dooley</a></figcaption></figure><p>We use the <a href="https://medium.com/brigade-engineering/brigades-experience-using-an-mvc-alternative-36ef1601a41f">VIPER architecture pattern</a> in the Brigade iOS app. We’ve had to build a few screen flows with VIPER so I thought I’d share some of our learnings.</p><p>We haven’t found a one-size-fits-all design pattern so we’ve identified three: <strong>Single-Module Flows</strong>, <strong>Multi-Module Flows</strong>, and <strong>Linked-List Flows</strong>.</p><h4>Single-Module Flows</h4><p>In this pattern, the whole flow is contained in a single VIPER module. Each step is a separate view controller. There’s a single presenter whose user interface is dynamically set to the current view controller.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*86aJqBRPC6zYi5NI6plTBA.png" /></figure><p><strong>Where and why we use single-module flows</strong></p><p>We use this pattern in the <a href="https://medium.com/bridge-collection/how-design-can-encourage-unusual-but-good-user-behavior-c80b4f245f6c">voter verification flow</a> because we don’t have to reuse views outside of the flow. If you don’t need to use existing modules inside your flow then the Single-Module pattern is pretty convenient and practical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*lnbGOInwAiowW4X2WdE5UQ.gif" /></figure><p><strong>How it works</strong></p><p>The wireframe is the orchestrator. It initializes the view controllers and pushes them onto the navigation controller as needed.</p><p>When a view controller reports that it’s “done”, it notifies the wireframe. At this point the wireframe pushes the next view controller onto the stack and the process continues until the last step.</p><p><strong>Pros</strong></p><ul><li>Minimum overhead and VIPER boilerplate. <em>There’s no need to create an entire module for each step in the flow.</em></li><li>Simple and easy to manage. <em>No need to jump around or context switch to different modules.</em></li></ul><p><strong>Cons</strong></p><ul><li>Bloated application logic stack. <em>The presenter, interactor, and data manager handle all the business logic for the entire flow so this can start to get heavy depending on the flow.</em></li></ul><h4>Multi-Module Flows</h4><p>Unlike the Single-Module pattern, in this pattern the flow is composed of many VIPER modules (one per step). There’s also a coordinator module responsible for orchestrating the flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*sUucBfrBL1ETlN0CQBQ_HQ.png" /></figure><p><strong>Where and why we use multi-module flows</strong></p><p>We use this pattern in the onboarding flow because it requires a little more flexibility from a product standpoint. For example, we had to include our existing inviter into this flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gb1uaIPL4ZJFjIqfmMAK5Q.png" /></figure><p><strong>How it works</strong></p><p>The coordinator wireframe’s job is to initialize everything and manage the steps in the flow. It does this by using a <a href="https://en.wikipedia.org/wiki/Finite-state_machine">finite state machine</a>.</p><p>When a step module tells the coordinator “Hey, I’m done!”, the coordinator grabs the next module’s view controller and pushes it onto the navigation controller.</p><p><strong>Pros</strong></p><ul><li>Ability to inject existing modules into the flow.</li><li><a href="https://en.wikipedia.org/wiki/Single_responsibility_principle">Single Responsibility Principle</a> / <a href="https://en.wikipedia.org/wiki/Separation_of_concerns">Separation of concerns</a>. <em>The coordinator doesn’t need to know about the details of each step.</em></li><li>Managing the step order is easy. <em>There’s only one file to check.</em></li></ul><p><strong>Cons</strong></p><ul><li>Adding new steps is costly. <em>You have to create a new VIPER module for each one.</em></li><li>The coordinator wireframe implements a lot of protocols.</li></ul><h4>Linked-List Flows</h4><p>This last pattern is the most “compact” in terms of code. The “flow” here is implicit and is controlled by the modules in the flow directly. When module 1 is done, it just routes onto module 2, etc..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7VtgX67vRTqgYu9RBpaqw.png" /></figure><p><strong>Where and why we use linked-list flows</strong></p><p>We use this pattern in the flow where after signing a petition we ask the user to invite others to sign. This flow only had two steps so the Linked-List flow pattern was most applicable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M-ToEjBBlIapz86VtQZ45w.png" /></figure><p><strong>How it works</strong></p><p>When one module is done, its wireframe initializes and routes to the next module. This routing happens either directly or using a central router (depending on how your app is architected overall).</p><p><strong>Pros</strong></p><ul><li>No need for a coordinator module. <em>The flow is implicitly controlled by each module.</em></li></ul><p><strong>Cons</strong></p><ul><li>Modules in the flow have to be dedicated to the flow. <em>If a module needs to work both inside and outside a flow then it must be configurable as such, perhaps via a state flag during initialization, which is probably not the best pattern.</em></li><li>Changing the order of the flow can be difficult.</li><li>Reasoning about the flow is challenging. <em>There is no birds eye view of the steps in the flow. Instead, you have to follow the breadcrumbs from module to module.</em></li></ul><h3>Final Thoughts</h3><p>There is no perfect solution for managing and orchestrating flows with VIPER. While each pattern has its strengths and weaknesses, I prefer the Single-Module flow due to its flexibility. Of course, the type of flow you choose will depend on your feature requirements.</p><p>In general I find that short and simple flows make for better user experiences, simpler code, and preserved sanity.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3459353d3a10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/building-screen-flows-with-viper-on-ios-3459353d3a10">Building Screen Flows with VIPER on iOS</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to radically streamline complex analytics]]></title>
            <link>https://medium.com/brigade-engineering/how-to-radically-streamline-complex-analytics-f50901c433e3?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/f50901c433e3</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[cassandra]]></category>
            <dc:creator><![CDATA[Mark Cooper]]></dc:creator>
            <pubDate>Fri, 19 Feb 2016 23:17:57 GMT</pubDate>
            <atom:updated>2016-10-31T19:11:09.799Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://www.brigade.com/">Brigade</a> we are building a platform to revolutionize civic and social engagement. For us, data is king. We’ve used a number of off-the-shelf analytics products in our short existence as a company: <a href="https://www.google.com/analytics/">Google Analytics</a>, <a href="https://www.kissmetrics.com/">Kissmetrics</a>, <a href="http://branch.io/">Branch.io</a> and <a href="https://www.tune.com/">Tune</a>, to name a few.</p><p>Off-the-shelf analytics tools are amazing — they are cheap, require little engineering work to integrate and can do a lot of powerful things. Sometimes, you just want to do something that the off-the-shelf tools can’t.</p><p>We hit a point where we wanted to query our data in a lot of complicated ways and to join datasets unavailable in the standard analytics tools. We felt it was worth building our own custom solution. Our first step into the world of custom analytics: start collecting activity data with as much contextual information as possible. Our initial architecture was pretty straightforward:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*kKozDN3w_Gu0xm0bw730tw.png" /></figure><p>First we built out end-to-end data collection: apps and our website write to a simple collection service which we built using <a href="https://finagle.github.io/">Finagle</a>. The service then forwards everything to <a href="http://kafka.apache.org/">Kafka</a>. On the other end a <a href="https://github.com/linkedin/camus">Camus</a> job pulls the events out of various Kafka queues and stores them in HDFS. To analyze the data we coded up Spark jobs which ingest the events from HDFS. We quickly discovered that, as beautiful and succinct as Spark is, writing a job for every new pivot of our data was just too time consuming. Moreover the Camus layer didn’t allow for real-time access to events as they flowed in and our own custom data compaction mechanisms added complexity and brittleness to the system.</p><h4>It’s a real-time world</h4><p>We decide to rework our design a bit and instead of using HDFS, write the event data to Cassandra.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*Jsg_p72MP3l8LPhTSrsEpg.png" /><figcaption>Cassandra added to our architecture</figcaption></figure><p>Presto — real-time access to our data and no need to worry about pesky data compaction issues. But writing the Spark job was still a hassle. Wouldn’t it be great if we could do the bulk of the analysis work in SQL?</p><h4>SQL — saving time since 1992</h4><p>Now that we have our event data in Cassandra, why not use the excellent Spark SQL bindings the fine people over at Datastax have implemented:</p><p><a href="https://github.com/datastax/spark-cassandra-connector">datastax/spark-cassandra-connector</a></p><p>Now if we just had a simple UI to write and run our queries with. Enter the Zeppelin project:</p><p><a href="https://github.com/apache/incubator-zeppelin/">apache/incubator-zeppelin</a></p><p>Unfortunately Zeppelin didn’t entirely work out of the box for us. It handles Spark SQL queries fine. It handles Cassandra queries fine. But Spark + Cassandra was a no go. Undeterred, we leveraged the extendability of Zeppelin to whip up a beautifully simple custom interpreter. Behind the scenes our interpreter simply hands queries off to a globally available CassandraSQLContext.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*2nLYr-HMwSt3hbUmlt6yKg.png" /></figure><p>We chose to bind our interpreter to a new “%csql” tag. So now we can execute queries as such:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*RpERuTMN9ANStm-fHF1RbA.png" /></figure><p>The Datastax Spark Connector even understands how to unpack Cassandra UDTs and maps (as seen when calling “client.platform”).</p><p>We also went nuts and hacked in an “INTO” clause which lets us easily chain operations. Here’s an example of a pair of chained queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*QxfbDc0UiU0OTUYSoGg8sQ.png" /></figure><p>Each statement is separated by a colon — the last statement in the chain is displayed in the table.</p><h4>Not just SQL</h4><p>Another awesome feature of Spark SQL — it’s insanely easy to write User Defined Functions (UDFs). Here’s an example UDF definition followed by a usage of it (our interpreter allows the user to group UDFs with queries). In this case our UDF is implemented in Scala.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S-lTYLAli0aaiePlf_CnjA.png" /></figure><h4>Datastore Polygamy</h4><p>Another extremely powerful feature of Spark SQL — cross datastore joins. While we are writing event log data to Cassandra, the bulk of our state-full user data is in Mysql. Exposing it in Zeppelin is simple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd2cbe756c331c34e4e99cfce28273e6/href">https://medium.com/media/dd2cbe756c331c34e4e99cfce28273e6/href</a></iframe><p>Now we have a Spark SQL accessible table (“analytics_users”) backed by the actual Mysql table that we can join against our Cassandra event log tables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*_t4HdW0TDSTbDpc6vVBwKg.png" /></figure><p>In Summary</p><p>Hurray, powerful fast SQL driven analytics. Our analysts and product managers are going to love us.</p><p>Users of our new system can get now get access in real-time to all of the event data we collect by writing straightforward SQL queries. Gone are the disparate data silos and the need to involve the data team in writing cumbersome Spark jobs.</p><p>In the coming months we plan to give back to the community by open-sourcing our Zeppelin modifications. In our next blog post we’ll cover a few more Zeppelin features we’ve add as well as how we’ve decided to organize and query our data in Cassandra.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f50901c433e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/how-to-radically-streamline-complex-analytics-f50901c433e3">How to radically streamline complex analytics</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10,000 times faster than MySQL?]]></title>
            <link>https://medium.com/brigade-engineering/10-000-times-faster-than-mysql-7296389e74bb?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/7296389e74bb</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Mark Cooper]]></dc:creator>
            <pubDate>Thu, 05 Nov 2015 21:44:01 GMT</pubDate>
            <atom:updated>2016-10-31T19:13:31.100Z</atom:updated>
            <content:encoded><![CDATA[<h4>Keep it simple</h4><p>At <a href="https://www.brigade.com/">Brigade</a> we are building easy and effective ways for people to declare their beliefs, organize with others, and take action to shape the policies affecting their lives. Our initial tool allows users to express whether they agree or disagree with opinion statements about a range of topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EJn8NnZvJdnmMKGAqwrWkg.jpeg" /></figure><p>Since we released earlier this summer, users have taken millions of such positions on everything from immigration and Internet policy to the environment and the economy.</p><p>Our architecture started off like many new companies — a basic Rails app backed by a MySQL database. Keep it simple and don’t optimize prematurely: great mantras to live by.</p><p>One of our core early offerings is to help users find alignment with other users on the basis of opinions they’ve shared. We do this to determine if you and your friends, family, and neighbors agree on, say, campaign reform. We compare your answers to a series of opinion statements (which we call positions) to a set of other users. We did this via a somewhat complex database join. A simplified version of this join looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e864b40cd7c5984dc8bbf45b32b011c/href">https://medium.com/media/5e864b40cd7c5984dc8bbf45b32b011c/href</a></iframe><p>This query worked fine when we started out, but as we grew, it didn’t scale. While the median latency for this query was reasonable, the 99th percentile could take 10 seconds or more. The Brigade Engineering team strives to build fast responsive apps. This was clearly too slow.</p><p>Instead of doing with a database we decided to move this computation into a separate service storing a continuously updated snapshot of the user data in RAM. For the actual computation we decided to keep the data in memory and store them in <a href="https://en.wikipedia.org/wiki/Bit_array">bitsets</a>. If we were to use Scala bitsets we might represent our data as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a2b4ddc4097051b31e6f78794c4874b/href">https://medium.com/media/5a2b4ddc4097051b31e6f78794c4874b/href</a></iframe><p>While this gives us a fast way to find out how many positions a pair of users have in common, BitSets have one major problem — they are very inefficient for our use because of the number of distinct opinions. As the Scala docs explain:</p><blockquote>Bitsets are sets of non-negative integers which are represented as variable-size arrays of bits packed into 64-bit words. The memory footprint of a bitset is determined by the largest number stored in it.</blockquote><p>Fortunately clever people have already solved this problem through the creation of compressed bitsets. We came across this fantastic implementation from Daniel Lemire called a <a href="https://github.com/lemire/RoaringBitmap">Roaring Bitmap</a>:</p><blockquote>Roaring bitmaps are compressed bitmaps which tend to outperform conventional compressed bitmaps such as <a href="http://dl.acm.org/citation.cfm?doid=502585.502689">WAH</a>, <a href="https://github.com/lemire/javaewah">EWAH</a> or <a href="http://druid.io/blog/2012/09/21/druid-bitmap-compression.html">Concise</a>. In some instances, roaring bitmaps can be hundreds of times faster and they often offer significantly better compression. They can even be faster than uncompressed bitmaps.</blockquote><p>After loading our production dataset into these structures we were pleasantly impressed to find that storing data in a Roaring Bitmap only used around 2.5 bytes of memory / entry. Nice! <strong>Storing a few million records only requires around 10 megabytes of RAM.</strong></p><h4>The core operation</h4><p>Here is a simplified version of our code that computes overlaps for a set of users:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9213c7d7d475853463729ffcc404f11/href">https://medium.com/media/e9213c7d7d475853463729ffcc404f11/href</a></iframe><h4>Performance Gains</h4><p>But how fast are they? We found that with a single thread on a reasonably fast machine we can overlap around 1M bitsets / second each containing a few thousand items. <strong>This brought our 99th-percentile latency down to around 2ms!</strong></p><p>To see the performance improvement take a look at this graph below. Below is the latency for this operation before and after rolling out the new service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*MmLbOZaoALcf2SoEvQipqQ.png" /></figure><h4>Higher Up the Stack</h4><p>We were already using Spark, Scala, Kafka at Brigade for analytics work. We’ve found these three technologies helped us innovate quickly. Contemplating how to roll our own service we decided to leverage our Scala expertise by using Twitter’s amazing <a href="https://twitter.github.io/finagle/">Finagle</a> framework and we exposed our API into our service via <a href="https://thrift.apache.org/">Thrift</a>. Thrift’s code generator gives us a set of language independent client libraries that were easy to use. As our service is written in Scala, we use the <a href="https://github.com/twitter/scrooge">Scrooge</a> compiler to produce our backend service. The client for this service is currently a Rails application so we use the standard Thrift compiler to generate a <a href="https://thrift.apache.org/tutorial/rb">Ruby library</a>. A simplified version of the Thrift definition for our service looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/427180e7ffadc4f55a7de13a009f787f/href">https://medium.com/media/427180e7ffadc4f55a7de13a009f787f/href</a></iframe><p>To push data into our new service we used <a href="http://kafka.apache.org/">Kafka</a> as the backbone. Kafka is a high-throughput distributed messaging queue. All updates to user records flow through Kafka and are read by a consumer in our service. Kafka is fault tolerant and most messages arrive in our service within a second of occurring.</p><p>And that’s it!</p><p>In summary:</p><ul><li>Start with simple solutions and don’t prematurely optimize until it’s needed. — Leverage the rich ecosystem of available tools and technologies — because we used so many great open source libraries our final service is only around thousand lines of code.</li><li>Don’t be afraid to stray away from conventional databases.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7296389e74bb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/10-000-times-faster-than-mysql-7296389e74bb">10,000 times faster than MySQL?</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reduce Chef infrastructure integration test times by 75% with test-kitchen and Docker]]></title>
            <link>https://medium.com/brigade-engineering/reduce-chef-infrastructure-integration-test-times-by-75-with-test-kitchen-and-docker-bf638ab95a0a?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/bf638ab95a0a</guid>
            <category><![CDATA[chefs]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Shane da Silva]]></dc:creator>
            <pubDate>Mon, 16 Mar 2015 21:05:10 GMT</pubDate>
            <atom:updated>2016-10-31T19:16:29.090Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*992euKF2IXed0P1SvJOCSg.jpeg" /></figure><h2>Reduce Chef infrastructure integration test times by 75%</h2><h4>with test-kitchen and Docker</h4><p>Infrastructure automation tools like <a href="https://www.chef.io/">Chef</a> and <a href="https://puppetlabs.com/">Puppet</a> are becoming more and more common as organizations look to code their infrastructure like they code their applications.</p><p>At <a href="https://www.brigade.com/">Brigade</a>, we use Chef to build our infrastructure. One of the challenges we’ve faced is the tradeoff between thorough testing and the time it takes to run those tests. With every Chef cookbook we write, we want to be able to deploy changes with reasonable confidence that they will work in production. At the same time, we don’t want to wait hours for automated scripts to spin up, converge, and spin down nodes in order to verify that our code works. We’ve created <a href="https://medium.com/brigade-engineering/smarter-relevant-chef-tests-ac998e957955">some tools</a> to help reduce the amount of time our tests take to run, but there is always room for improvement.</p><p>One of the most powerful tools available in the Chef ecosystem is <a href="http://kitchen.ci/">test-kitchen</a>, a project that simplifies the process of converging your cookbooks in an isolated test environment. Out of the box, it works by spinning up a <a href="https://www.virtualbox.org/">VirtualBox</a> VM for each test suite, converging the VM with Chef, and then running some tests to verify the VM is in a desired state.</p><h3>The Problem: Long Test Times</h3><p>Where this solution has its limitations is that creating and destroying virtual machines is time-consuming and resource-intensive. For our collection of over 20 test suites representing various core components of our infrastructure, testing all of them took almost 2 hours. Converging and testing our base cookbook (the cookbook included on all nodes in our infrastructure which contains common global configuration) took almost 15 minutes alone, so even running tests in parallel would still take significant time.</p><p>These long test times were not ideal because:</p><ul><li>We weren’t getting quick feedback on whether a commit broke our infrastructure in some way, which slowed down our development speed.</li><li>Adding more test suites made our integration tests take even longer, so as our infrastructure grew the time it took to test grew at an unsustainable rate.</li></ul><p>We clearly needed to explore other solutions to this problem.</p><h3>Identifying Bottlenecks</h3><p>Digging into where we were spending significant amounts of time in our tests, we realized that:</p><ul><li>Creating and destroying VMs took a minimum of 1–2 minutes for each test suite.</li><li>There was noticeable CPU and disk I/O overhead running within a VM (5–10% in some cases).</li><li>VMs limited our ability to parallelize, as each VM took a minimum amount of memory, even if it didn’t use all of it. This limited the utilization of our resources as a lot of memory would effectively go unused during the course of a test run—memory that could have been used to run other test suites.</li></ul><p>While there certainly were ways we could have addressed some of these issues while continuing to use VMs, it seemed like we needed to drastically rethink how we were going about solving the problem, as the fundamental issue was the overhead of running VMs.</p><h3>Solution: Containers as VMs</h3><p>We realized that containers might be a viable alternative. Containers are quick to provision, incur less overhead than a full VM, and make better use of resources as they don’t reserve memory on startup, but share whatever system memory is available. We decided to explore using <a href="https://www.docker.com/">Docker</a> to provision VM-like containers that we could run our tests within.</p><h4>kitchen-docker</h4><p>Fortunately, there was already a <a href="https://github.com/portertech/kitchen-docker">plugin</a> for test-kitchen that added support for provisioning Docker containers. However, Docker containers are usually very lightweight—they <a href="https://jpetazzo.github.io/2014/06/23/docker-ssh-considered-evil/">typically aren’t running SSH</a>. The test-kitchen runtime however, relies on the existence of SSH in order to connect to and run commands on a machine.</p><p>The kitchen-docker plugin handles this by generating a Dockerfile that runs an SSH daemon within the container. This allows test-kitchen to access the container via SSH and execute commands like running a Chef client and tests. However, this only allows you to work with very simple Chef cookbooks—using resource providers like <a href="https://docs.chef.io/resource_service.html"><strong>service</strong></a> will cause Chef runs to fail since there is no <a href="https://en.wikipedia.org/wiki/Init">init process</a> running in the container.</p><h4>Thick Containers</h4><p>In order to test our cookbooks that used the <strong>service</strong> resource provider, we needed to come up with a way to get an init process running as the root process in our containers. It seemed like we were trying to fit a square peg into a round hole, and in many ways we were—Docker isn’t really intended to be used for so-called “thick” containers, but it is perfectly capable of doing so.</p><p>Our infrastructure is built entirely on CentOS 7, whose init process is <a href="http://www.freedesktop.org/wiki/Software/systemd/">systemd</a>. Building on Jim Perrin’s <a href="https://jperrin.github.io/centos/2014/09/25/centos-docker-and-systemd/">Dockerfile</a> for a CentOS container with systemd, we came up with the following Dockerfile:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c62a28bcf319a99ade90c8a58fc295c/href">https://medium.com/media/7c62a28bcf319a99ade90c8a58fc295c/href</a></iframe><p>Much of the above code is commented, but here are the following high-level takeaways:</p><ul><li>We swap the fakesystemd dummy package included in the base CentOS 7 image with the actual systemd package, and remove some unnecessary unit files that only make sense for a “real” machine.</li><li>We run SSH as a service under systemd.</li><li>We finally install some basic utilities including Chef itself so that we don’t need to install them each test run, saving us time.</li></ul><p>The kitchen-docker configuration in our <strong>.kitchen.yml</strong> file is pretty straightforward:</p><pre>driver:<br>  name: docker<br><br>platforms: <br>  - name: centos-7 <br>    driver_config: <br>      dockerfile: test/Dockerfile<br>      privileged: true # Needed by systemd to access cgroups<br>      run_command: /usr/sbin/init # Start systemd as root process<br>      use_sudo: false</pre><h4>Loose Ends</h4><p>This got us 95% of the way towards our goal. However, two problems still remained:</p><ul><li>If our SSH daemon wasn’t working for whatever reason, we could not log into the machine via the <strong>kitchen login</strong> command.</li><li>We noticed that destroying containers would timeout, resulting in a forced shutdown which slowly <a href="https://github.com/jpetazzo/dind/issues/19">leaked loopback devices</a>.</li></ul><p>The first problem was solved by monkey-patching the kitchen-docker driver to execute <strong>docker exec -it %{container_id} bash</strong> instead of SSH when using <strong>kitchen login</strong>. Regardless of the state of the SSH daemon we could now always run a bash shell within the container to poke around and debug.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22aaeb5f76c2ad7dbe8dbb60b0e8f7cf/href">https://medium.com/media/22aaeb5f76c2ad7dbe8dbb60b0e8f7cf/href</a></iframe><p>The second problem turned out to be a bit thornier. When Docker shuts down a container, it sends the <a href="https://en.wikipedia.org/wiki/Unix_signal">SIGTERM</a> signal to the root process. Apparently, systemd simply <a href="http://unix.stackexchange.com/a/7469">re-execs itself</a> when receiving SIGTERM, and there is no other signal that causes it to shut down gracefully.</p><p>Thus we needed to monkey-patch the kitchen-docker driver to execute <strong>shutdown now</strong> in the container to trigger a proper shutdown of systemd.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f071fa674e16af0d8c0c777901ec597/href">https://medium.com/media/9f071fa674e16af0d8c0c777901ec597/href</a></iframe><p>This fixed our leaky loopback device woes.</p><p>In order to get these monkey-patches to work, we needed to include the following line at the top of our <strong>.kitchen.yml</strong> file:</p><pre># &lt;% load &quot;#{File.dirname(__FILE__)}/test/kitchen_docker.rb&quot; %&gt;</pre><p>While it looks like it’s in a comment, this actually gets executed since the <strong>.kitchen.yml</strong> file is interpreted as ERB.</p><h3>The Results</h3><p>The results of these efforts were substantial. Using the same test hardware as before, our overall test time dropped from almost 2 hours to ~30 minutes, and our base cookbook test suite dropped from 15 minutes to 3.</p><p>This was due to the containers being faster to create and destroy, as well as having lower CPU overhead. We were also able to run more test suites in parallel, since containers didn’t reserve memory all at once, but rather only as needed.</p><h4>Gotchas</h4><p>While this approach works for our infrastructure needs, this is not a silver bullet. Some important notes:</p><ul><li>The host machine running the container must also be CentOS 7 (or at least be running the same Linux kernel version).</li><li>The containers need to be run with root privileges in order for the systemd within the container to access cgroups. We’re ok with this in our test infrastructure, but others might not like the idea of containers with root privileges on the host system.</li><li>Perhaps obvious, but we’re not actually converging our cookbooks on a real machine. For our purposes this is fine, but there may be certain use cases for which you really need an isolated machine. In cases where you need true isolation, you can set the individual test suite to use the default test-kitchen Vagrant/VirtualBox integration.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf638ab95a0a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/reduce-chef-infrastructure-integration-test-times-by-75-with-test-kitchen-and-docker-bf638ab95a0a">Reduce Chef infrastructure integration test times by 75% with test-kitchen and Docker</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wait, that’s how it works?!]]></title>
            <link>https://medium.com/brigade-engineering/wait-thats-how-it-works-f04125878f5c?source=rss----41e6ebfe4a48---4</link>
            <guid isPermaLink="false">https://medium.com/p/f04125878f5c</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[civictech]]></category>
            <dc:creator><![CDATA[Hao Su]]></dc:creator>
            <pubDate>Thu, 12 Feb 2015 16:46:33 GMT</pubDate>
            <atom:updated>2015-04-01T05:02:21.225Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-d3mzckq9KcyCelJj_6oJg.jpeg" /></figure><h4>What we’ve learned about our representation through Open Civic Data</h4><p>At Brigade, we want anyone and everyone to better understand their own political beliefs and how to act on these opinions. Part of this is figuring out who actually represents you. To date, <a href="https://medium.com/@noahsilas">@noahsilas</a> and I have been working hard to integrate our not-yet-launched product with the Google Civic Information API and other sources of civic data. We’ve been learning about how our government’s quirks are modeled digitally and here are some interesting things we’ve picked up along the way.</p><p><a href="http://opencivicdata.org/">Open Civic Data</a> identifiers (or OCD IDs) identify different parts of our government. At this time, it covers people, organizations, divisions, and jurisdictions. The majority of our work has been understanding Open Civic Data’s political divisions and how they’re represented. A division is a politically relevant geographic area that maps to by a set of addresses (possibly by GPS geometries) and is identified by an OCD Division ID. Identifiers are specified in this format:</p><pre>ocd-division/country:&lt;country_code&gt;(/&lt;type&gt;:&lt;type_id&gt;)* </pre><p>and here’s a OCD ID for reference:</p><pre>ocd-division/country:us/state:ca/cd:12</pre><p><a href="https://developers.google.com/civic-information/">Google’s Civic Data API</a> provides us with a means to look up elected official information, political office information, and even election information through OCD Divisions and address. Since it would be more than a bit cumbersome to iterate through all known US addresses, it’s natural to use OCD IDs. Open Civic Data has been kind enough to post all current OCD IDs in <a href="https://github.com/opencivicdata/ocd-division-ids/">a GitHub repository</a> for easy access. Having all the data at once makes it much easier to see patterns and inconsistencies in what first appears to be the OCD ID naming scheme but turns out to be quirks in our political system.</p><p><strong>Let’s start off small: </strong>Nebraska is unicameral while all other states’ legislatures and the U.S. Congress are bicameral. In the Cornhusker State, this body is referred to as the ‘Legislature’ or sometimes just the ‘Unicameral’. In 1934 Nebraska citizens voted to abolish the lower house due to budgetary constraints during the Great Depression. How did we find out? The Google Civic API response for all other states includes divisions for both upper and lower legislative bodies, but Nebraska only has one. Ahhh!</p><p>State upper legislatures as ‘sldu’<em> (ocd-division/country:us/state:ca/sldu:1) </em>while lower houses are presented with ‘sldl<em> ‘ (ocd-division/country:us/state:ca/sldl:1). </em>We noticed Nebraska’s lack of a lower legislature through the missing ‘sldl’ identifiers.</p><p>Not too bad so far right? Unlike for other states, we just won’t have extra OCD IDs to keep track of. Man, why can’t we all be unicameral?</p><p><strong>Warming up now:</strong> All federal lower body divisions (House of Representatives) are labeled by number (<em>ocd-division/country:us/state:ca/cd:12</em>), and the upper body (Senate) divisions are just the state (<em>ocd-division/country:us/state:ca). </em>Each state is then broken down in to divisions for their upper and lower legislative houses. It shouldn’t be too crazy to assume that all states would name their divisions numerically right? Unfortunately, Massachusetts and Vermont use names (<em>ocd-division/country:us/state:vt/sldu:chittenden</em>) for their upper house divisions, and more than a few states use either alphanumeric combinations or names to denote their lower houses. This really isn’t a technical pain but it is surprising. C’mon state governments, can’t we get some consistency over here?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*W4bEGjGqWsddCoBBRoQRZA.jpeg" /><figcaption><a href="http://en.wikipedia.org/wiki/Boston_Tea_Party">Boston’s epic tea party of 1773.</a></figcaption></figure><p><strong>On to the next one:</strong> Keeping in line with state legislature shenanigans, many states have the same boundaries drawn for their upper and lower house divisions. So what are the differences? The lower houses usually have shorter election cycles (every 2 years) while <em>most </em>upper houses have longer cycles. Now New Jersey is a curious example. The members of New Jersey’s upper house serve four year terms except after redistricting, where they’ll only serve two years. Also known as the “2–4–4&quot; cycle. We came across this one while researching why all of New Jersey’s lower house OCD IDs were aliased to their upper house OCD IDs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*REDtcIg1oZzuO6xuybfM1w.jpeg" /><figcaption><a href="http://blog.nj.com/njv_editorial_page/2013/02/in_defense_of_jughandles_edito.html">New Jersey is the land of jughandles and the “2–4–4&quot; election cycle.</a></figcaption></figure><p>Modeling all of the aliased OCD IDs adds complexity. Aliased OCD IDs are stored in a separate file (<a href="https://github.com/opencivicdata/ocd-division-ids/blob/master/identifiers/country-us/aliases.csv">https://github.com/opencivicdata/ocd-division-ids/blob/master/identifiers/country-us/aliases.csv</a>) rather than in the canonical OCD ID files. Unfortunately at the time of writing, aliased OCD IDs are actually not resolving in the Google Civic API.</p><p><strong>And through that we found this: </strong>The New Jersey legislature was <a href="https://www.google.com/webhp?q=prorogue#q=prorogue">prorogued</a> (I had to look this up too) by then Governor William Franklin on December 6, 1775. An entirely new legislature was elected almost a year later on August 13, 1776. There isn’t any technical aspect, but it’s a fun bit of trivia.</p><p><strong>Last, we narrowly avoided this modeling headache:</strong> The California Senate redistricted in 2011! The story is best told by… <a href="http://www.legislature.ca.gov/legislators_and_districts/districts/senatedistricts.html">the official California legislative website</a>:</p><blockquote>Due to redistricting, Senate districts have a unique issue that Assembly and Congressional districts do not have. Of the Senate districts established by the Citizens Redistricting Commission in 2011, only the odd-numbered districts went into effect for the 2012 election cycle. The new even-numbered districts will go into effect for the 2014 election cycle, and the even-numbered districts previously established by the 2001 redistricting will continue to exist until 2014. These unique circumstances <strong>create some areas of overlap</strong> between the old and new districts (“accelerated areas”) and <strong>some areas without coverage</strong> (“deferred areas”). For the 2013–14 Regular Session of the Legislature, <strong>each accelerated area essentially has two Senators representing the area and each deferred area has none</strong>. The Senate Committee on Rules will assign a Senator to provide appropriate constituent services to each deferred area. This is a normal consequence of the redistricting process.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*ZSNSGpFuuuBDwUU0uWQYuw.png" /><figcaption><a href="http://www.legislature.ca.gov/legislators_and_districts/districts/senatedistricts.html">http://www.legislature.ca.gov/legislators_and_districts/districts/senatedistricts.html</a></figcaption></figure><p>So for that sticky interim period we had fun temporary OCD IDs like this: <em>ocd-division/country:us/state:ca/sldu:assigned-29. </em>Lucky for us, these will only last through 2014 and not any further. So whew, no need to model two sets of divisions for a single legislative body… yet…</p><p>The work that Open Civic Data and Google have done with making data about our crazy federal and <em>state</em> governments accessible through code is thankless. Much of the information they are provided with is inconsistent and must be updated manually. The herculean effort of tracking state information to <em>county precincts</em> is mind-blowing, especially after redistricting shenanigans. These are some of the few quirks we’ve learned about just through reading the data, and we’ll be sure to post about the others that we find as our work continues.</p><p>Citations:</p><ul><li><a href="http://en.wikipedia.org/wiki/Nebraska#Legislative_branch">http://en.wikipedia.org/wiki/Nebraska#Legislative_branch</a></li><li><a href="https://github.com/opencivicdata/ocd-division-ids/blob/master/identifiers/country-us/openstates/state-ma-openstates.csv">https://github.com/opencivicdata/ocd-division-ids/blob/master/identifiers/country-us/openstates/state-ma-openstates.csv</a></li><li><a href="http://en.wikipedia.org/wiki/New_Jersey_Legislature#Elections_and_terms">http://en.wikipedia.org/wiki/New_Jersey_Legislature#Elections_and_terms</a></li><li><a href="http://en.wikipedia.org/wiki/New_Jersey_Legislature#Colonial_period">http://en.wikipedia.org/wiki/New_Jersey_Legislature#Colonial_period</a></li><li><a href="http://www.legislature.ca.gov/legislators_and_districts/districts/senatedistricts.html">http://www.legislature.ca.gov/legislators_and_districts/districts/senatedistricts.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f04125878f5c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brigade-engineering/wait-thats-how-it-works-f04125878f5c">Wait, that’s how it works?!</a> was originally published in <a href="https://medium.com/brigade-engineering">Brigade Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>