<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ryan Barrett on Medium]]></title>
        <description><![CDATA[Stories by Ryan Barrett on Medium]]></description>
        <link>https://medium.com/@snarfed?source=rss-4f279603e35e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1wStlDtNpG5ur5Fr1mMW7A.png</url>
            <title>Stories by Ryan Barrett on Medium</title>
            <link>https://medium.com/@snarfed?source=rss-4f279603e35e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:06:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@snarfed/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Engineering bioinformatics in seconds, not hours]]></title>
            <link>https://medium.com/color-genomics/engineering-bioinformatics-in-seconds-not-hours-b17af684f682?source=rss-4f279603e35e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b17af684f682</guid>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[local-development]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Ryan Barrett]]></dc:creator>
            <pubDate>Tue, 12 Mar 2019 16:57:44 GMT</pubDate>
            <atom:updated>2019-03-12T17:20:10.285Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksOWM7JKK0Zy1JhbJmxbYw.jpeg" /><figcaption><a href="http://longnow.org/clock/">10,000 Year Clock, Long Now</a></figcaption></figure><p>It was winter 2014. Pharrell had just dropped <a href="http://www.billboard.com/charts/2014/year-end"><em>Happy</em></a>, the <a href="http://www.bbc.com/news/science-environment-30026398">Rosetta probe landed on a comet</a>, President Obama was <a href="http://www.bbc.co.uk/news/world-us-canada-30516740">opening diplomatic relations with Cuba</a>…</p><p>…and here at Color, the bioinformatics team had a problem. Our <a href="https://blog.color.com/building-a-high-quality-reliable-and-efficient-bioinformatics-pipeline-aa4a01e182ba">pipeline </a>— the data processing system that crunches raw DNA data from our lab into the variants we report to patients — was slow. 12 to 24 hours slow.</p><p>This wasn’t a problem in and of itself — bioinformatics pipelines routinely run for hours or even days — but it was a royal pain for development. We’d write new pipeline code, start it running, go home, and return the next morning to find it had crashed halfway through because we’d <a href="https://www.quora.com/Why-is-the-error-of-missing-a-semicolon-seemingly-so-difficult-to-fix-for-many-people">missed a semicolon</a>. Argh. Or worse, since we hadn’t launched yet, our live pipeline would hit similar bugs in production R&amp;D samples, which would delay them until we could debug, test, and deploy the fix. No good.</p><h3>No quick fixes</h3><p>We tried all the standard solutions. We wrote unit tests, which helped, but couldn’t easily cover the many parts that shell out to external tools or libraries in other languages. We tested parts manually, in both the command line and the Python REPL, but that quickly became tedious and didn’t catch regressions. We checked out an automated integration test that ran the whole pipeline, but it was much too expensive and slow to run on every commit, and running it nightly didn’t help our development cycle.</p><p>I was undeterred. I’ve seen the impact of short development cycles and feedback loops firsthand, and have spent much of my career identifying and ruthlessly cutting them — for data science here at Color, and elsewhere for <a href="https://snarfed.org/resume#App+Engine">web apps</a>, <a href="https://snarfed.org/resume#Quip">Android apps</a>, <a href="https://snarfed.org/resume#network">network ops</a>, <a href="https://granary-demo.appspot.com/">social network data</a> <a href="https://snarfed.org/2011-11-09_anouncing_mockfacebook">and APIs</a>, and <a href="https://snarfed.org/resume#IBM">small molecule visualization</a>. Bioinformatics was new to me, but so what? I rolled up my sleeves and dove in.</p><h3>s/dream/reality/</h3><p>My goal was to come up with a tiny input dataset that the entire pipeline could run on, including all external packages and dependencies. I wanted it to run both locally and in our continuous integration environment. I wanted to generate a “golden” file with expected data that I could compare to the pipeline’s output data. And most importantly, I wanted it to run in seconds to minutes at most. I hoped that a small input DNA sample — just a few regions and 10k or so base pairs — would do the trick and still exercise most of the pipeline code paths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nyPLn4B6KUYLcEqJiyYoYA.jpeg" /><figcaption><a href="https://www.gamesmediapro.co.uk/80s-smash-hit-game-simon-to-launch-35th-year-anniversary-special/">“Simon Says,” GamesMediaPro</a></figcaption></figure><p>I started with getting the full pipeline to run on my dev machine. Like many tech companies today, Color engineers use Linux in production but develop on Macs, so I had to find and build all the corresponding packages for Mac OS X with the same configurations and versions: R, Java, <a href="https://s3.amazonaws.com/color-static-prod/pdfs/validationWhitePaper.pdf">BWA, samtools, Picard, GATK</a>, and many more. I also needed all of our datasets and related files in place, with the same directory structure as in prod.</p><pre>brew install r<br># if that didn’t install 3.1.3, and you don’t already have it installed, use<br># this to install it: <a href="http://stackoverflow.com/a/9832084/186123">http://stackoverflow.com/a/9832084/186123</a><br>brew switch r 3.1.3<br>brew install homebrew/versions/gcc49 — with-fortran # takes 40m!<br>brew link r<br>brew pin r</pre><p><em>Example: build and install R 3.1.3 with GCC 4.9 and Fortran support</em></p><p>I’ll spare you the details, but suffice it to say, matching our exact toolchain and environment took a lot of trial and error, and more than a little language inappropriate to a workplace environment.</p><h3>Toy data in, toy data out</h3><p>Once I got the full pipeline running on my laptop, I moved to the test dataset. I started with a well known test DNA sample from the NHS, and extracted one region in BRCA1 and one in BRCA2:</p><pre>samtools view -b AH9TKMADXX_DS-152355_ACAAGCTA.bam 17:41255000–41260000 &gt; chr17.bam<br>samtools view -b AH9TKMADXX_DS-152355_ACAAGCTA.bam 13:32895000–32905000 &gt; chr13.bam<br>samtools cat -o pipeline_test.bam chr17.bam chr13.bam</pre><pre># separate paired reads. <a href="https://github.com/samtools/samtools/issues/402">https://github.com/samtools/samtools/issues/402</a><br>samtools view -uf64 pipeline_test.bam | samtools bam2fq → fwd.fastq<br>samtools view -uf128 pipeline_test.bam | samtools bam2fq → rev.fastq</pre><pre># watch out for unpaired reads here, which BWA will die on with<br># “paired reads have different names” errors.<br># <a href="https://github.com/alexdobin/STAR/issues/45#issuecomment-108522238">https://github.com/alexdobin/STAR/issues/45#issuecomment-108522238</a></pre><pre>java -jar /resources/picard/SamToFastq.jar INPUT=pipeline_test.bam FASTQ=fwd2.fastq SECOND_END_FASTQ=rev2.fastq</pre><p><em>Extract two small test regions from a BAM file, then generate a FASTQ file</em></p><p>(Coordinates in <a href="https://en.wikipedia.org/wiki/FASTQ_format">FASTQ files</a> always start from 0, so I also had to shuffle them down in the test FASTQ file to match.)</p><pre>;; Open each VCF file, put point at first char of location (second column) on<br>;; first line of each chromosome, 13 and 17, and run this for each line as a<br>;; keyboard macro.<br>(defun shuffle-location ()<br>  (let* ((bounds (bounds-of-thing-at-point ‘word))<br>      (pos (apply ‘delete-and-extract-region (car bounds) (cdr bounds) nil)))<br>      (save-excursion<br>      (insert (int-to-string (- (string-to-int pos)<br>                              32895000 ; for chr 13<br>                              ;41255000 ; for chr 17<br>      ))))<br> (let ((line-move-visual nil))<br> (next-line))))</pre><p><em>Emacs Lisp for shuffling down DNA locations in VCFs</em></p><p>Finally, I made the golden <a href="https://en.wikipedia.org/wiki/VCF">VCF file</a> with the test sample’s variants in just these two regions:</p><pre>echo “chrom\tchromStart\tchromEnd\n17\t41255000\t41260000\n13\t32895000\t32905000” &gt; positions.bed<br>for f in *.vcf; do<br> vcftools — bed positions.bed — recode — vcf $f<br> mv out.recode.vcf minimal/$f<br>done</pre><p><em>bash snippet for extracting test regions from VCF files</em></p><h3>Data, data everywhere, and way too much to drink</h3><p>Now I could run the pipeline end to end on my laptop, and I had a tiny piece of input DNA to give it. Many of the pipeline stages used their own reference datasets, though, which could be gigabytes or more. And I hadn’t touched any of them yet.</p><p>Sure enough, I ran the pipeline locally on my test data and it took almost an hour. Definitely better than 12–24h for a full-sized input sample, but still way too long for a decent iteration loop.</p><p>The task was daunting. The pipeline had dozens of stages, and each one did a different combination of tasks. Some used entirely homegrown code and internal data; others used multiple third party packages and datasets from across research, open source, and proprietary companies. They came in different languages and compiled binaries, and the data files ranged from CSV to XML to compressed and binary files with no identifying magic header bytes that file(1) threw up its hands on. The low hanging fruit was long gone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R5ve0zfwlhomk506" /><figcaption>Our pipeline four years ago. It’s changed a lot since then!</figcaption></figure><p>Nonetheless, I dove into each stage, identified the packages, data files and other dependencies, and figured out how to cut them down and then reconstitute them so they were still usable.</p><p>Lots of different things used the reference genome, so I started there…</p><pre>cd ../hg19/sequence<br>samtools faidx chr13.fa chr13:32895000–32905000 &gt; chr13.fa.minimal<br>samtools faidx chr17.fa chr17:41255000–41260000 &gt; chr17.fa.minimal<br># remove location indices from headers in those output files …<br># remove offsets …</pre><p>…and moved on to the rest, one by one, debugging line ending compatibility issues:</p><pre># CaptureRegions.bed has CRLF line terminators. ugh.<br>tr ‘\n\r’ ‘\0\n’ &lt; ../../r.full/PlatformInput/CaptureRegions.bed \<br> | grep -E ‘^(chr13.3289[5–9]|chr17.4125[5–9])’ | tr ‘\0\n’ ‘\n\r’ \<br> &gt; CaptureRegions.bed<br># shuffle locations down with Emacs Lisp snippet above</pre><p>I decompressed, split, extracted, modified, and recompressed. Endlessly.</p><pre>gunzip gencode.v18.annotation.gtf.gz<br>(grep ^”#” gencode.v18.annotation.gtf ; grep -v ^”#” gencode.v18.annotation.gtf | sort -k1,1 -k4,4n) | bgzip &gt; sorted.gff.gz<br>tabix sorted.gff.gz<br>grep -E ^”#” gencode.v18.annotation.gtf &gt; min.gtf<br>tabix sorted.gff.gz chr13:32895000–32905000 chr17:41255000–41260000 &gt;&gt; min.gtf<br># shuffle down locations in min.gtf<br>bgzip -c min.gtf &gt; src/datasets/etc/pipeline_test/gencode.v18.annotation.gtf.gz<br>git add src/datasets/etc/pipeline_test/gencode.v18.annotation.gtf.gz</pre><p>I trial-and-errored, and trial-and-errored, and trial-and-errored:</p><pre>grep -E ‘(chr13.3289[5–9]|chr17.4125[5–9])’ chainSelfLink.txt \<br> &gt; ../../../../r/UCSC/hg19/annotation/chainSelfLink.txt<br># expand a little since none were in our exact regions<br>grep -E ‘(chr13.[+-].328|chr17.[+-].412)’ refFlat.txt \<br> &gt; ../../../../r/UCSC/hg19/annotation/refFlat.txt<br># expand a little more…<br>grep -E ‘(chr13.[+-].32|chr17.[+-].41)’ refFlat.txt \<br> &gt; ../../../../r/UCSC/hg19/annotation/refFlat.txt<br># shuffle locations with emacs snippet above</pre><p>I lsofed and straced, reverse engineered mystery file formats, decompiled and modified the incomprehensible resulting code, then recompiled. I examined each stage, each tool and file, learning its data and cutting down as small as possible. I ran the pipeline over and over and watched its running time drop from over an hour, to under an hour, to 30 minutes, 15 minutes, 10, 5, and finally just over 2 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*YlgwUjii9M9zs8hKm38odw.jpeg" /><figcaption><a href="https://www.youtube.com/watch?v=NpghkBoYgcY">“Dragon egg hatches!” JesseAndMike</a></figcaption></figure><p>Now, I must say, I was ecstatic. I wrapped it up in a test harness that used the test data, compared it against the golden output VCF, added a couple commands to our toolchain to install the dependencies, set up the environment, and ran the test. I helped our other bioinformatics engineers get it running on their laptops and added it to our continuous integration server. It started running on every single commit to our codebase. Victory.</p><h3>Blood, sweat and tears</h3><p>Our pipeline has grown substantially since I first put this together. We’ve switched out some dependencies, added some new ones, and <a href="https://blog.color.com/cancer-meets-precision-medicine-the-color-test-for-risk-of-common-hereditary-cancers-in-women-and-3ac62dc2a2ff">expanded the pipeline to handle many more genes</a> and genetic features. We’ve added many more unit tests and tools.</p><p>Even so, on every change to the pipeline we’ve continued to run this test both locally while developing and in our continuous integration environment. It has sped up our development cycle substantially, prevented countless bugs and regressions, and freed us to iterate and explore in real time, without fear.</p><p>This project was one of the most unusual, challenging, and rewarding things I’ve done in my career. If it sounds interesting to you too — frankly, if you’re even still reading this — <a href="https://www.color.com/careers">let’s talk</a>. We’re always looking for engineers who want to use technology to help improve our healthcare system. Reverse engineering strictly optional.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b17af684f682" width="1" height="1" alt=""><hr><p><a href="https://medium.com/color-genomics/engineering-bioinformatics-in-seconds-not-hours-b17af684f682">Engineering bioinformatics in seconds, not hours</a> was originally published in <a href="https://medium.com/color-genomics">Color</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a public research database out of spare parts]]></title>
            <link>https://medium.com/color-genomics/building-a-public-research-database-out-of-spare-parts-1fd8f996b733?source=rss-4f279603e35e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fd8f996b733</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[genetics]]></category>
            <category><![CDATA[data-sharing]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Ryan Barrett]]></dc:creator>
            <pubDate>Fri, 08 Feb 2019 17:21:47 GMT</pubDate>
            <atom:updated>2019-02-08T18:54:00.992Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*68_FVm_ZKoi4RFsrniW-0A@2x.png" /><figcaption><a href="https://data.color.com/">Color Data</a></figcaption></figure><p>A couple months ago, we launched a <a href="https://data.color.com/">public research database with DNA, health history, and more from 50,000 of our clients</a>. You might be surprised at how little work it took us: under four person-months total. Read on to hear how we designed and built it, went above and beyond the usual privacy safeguards, and did it all in the blink of an eye.</p><p>At first glance, Color Data may seem far from unique. <a href="https://www.ncbi.nlm.nih.gov/clinvar/">ClinVar</a>, <a href="http://gnomad.broadinstitute.org/">gnomAD</a>, <a href="https://cancergenome.nih.gov/">TCGA</a>, <a href="http://www.internationalgenome.org/">1000 Genomes</a>, and others all address similar goals: sharing anonymized genotype and phenotype data with academic researchers to help them advance science and knowledge. We’re in an unusual position at Color, though, in that we have a large population with both sequenced DNA and self-reported phenotype that has opted to share it with researchers. Even better, our population is a bit more diverse across ethnicity, age, health history, and other characteristics than many other research datasets.</p><p>We feel strongly that participating and sharing with the genomics research community should be one of our primary goals as a company. Given that, we’ve had the idea to publish a dataset like this for a long time. We had a clear vision in our heads: elegant visualizations of genes and classifications, demographics and health history, with simple tools to filter and drill down into subpopulations.</p><p>We knew it was a substantial project, though, and as a startup, we run lean and, so we never quite had the resources to spare. Extracting and preparing the data, crafting the charts and visualizations, designing and implementing an interface to drill into it…we’d need a team of multiple engineers for a minimum of a quarter, probably more like half a year.</p><h3>An ounce of prevention</h3><p>I was dismayed. I like to joke that I’m a <a href="https://snarfed.org/2017-09-12_25492">Software Prevention Engineer</a>, but I’m at least partly serious. I’m a strong believer in <a href="http://norvig.com/">Peter Norvig</a>’s maxim that <em>All code is liability</em>. Every line of code takes work to write, maintain, and eventually replace. Every line can have a bug, a security hole, or worse. It’s easy to think our job is to write code, but in practice, I believe our mandate is to <em>create results.</em> Code is our usual tool, but if we can get the same results with less code, or even none, so much the better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNFSQx7ujAQYAZIs4jfaGw.jpeg" /><figcaption><a href="https://www.flickr.com/photos/puroticorico/6355586033">“Empty Pockets” by Richie Diesterheft</a></figcaption></figure><p>So I looked around. What could we do instead? What could we reuse? It didn’t take long for <a href="https://www.metabase.com/">Metabase</a> to come to mind. It’s a simple, self serve BI tool that we use heavily internally at Color. It lets engineers and non-engineers alike query data, design visualizations and dashboards, and share them. We already used it to build many similar data analysis tools internally; why not this too?</p><p>Well, lots of reasons. Metabase could <a href="https://www.metabase.com/docs/latest/administration-guide/13-embedding.html">embed dashboards in web pages and serve them to the public</a>, in read only mode, but we’d never tried it. We’d definitely never used it to build something with interactivity quite this deep. It had no way to implement the privacy safeguards we needed, and we didn’t know if we could add them ourselves. We had no idea how it would scale. Overall, it seemed unlikely.</p><p>Still, I was determined that we could and should spend a few engineer-<em>months</em> on this, cumulatively, not years. So I dove in.</p><p>I tackled the embedding first. My dream was to write a bit of static HTML, just enough to put our logo and maybe some links at the top, and then use Metabase to generate the bulk of the page. We’d avoid running a dedicated web server altogether by serving the HTML entirely from a CDN <a href="https://aws.amazon.com/cloudfront/">like AWS’s CloudFront</a>. Happily, embedding Metabase was refreshingly quick and simple; I had this up and running in half an hour or so. Phew.</p><h3>Spy vs spy</h3><p>The next task was a big one: de-identify the data. This meant more than just removing names and other identifying details. We needed to ensure that no matter how skilled an attacker was, they’d still find it difficult or impossible to identify any individual client in the database and learn anything new about them.</p><p>Fortunately, the academic community has studied this problem deeply, and came up with techniques like <a href="https://en.wikipedia.org/wiki/Statistical_database">statistical databases</a>, <a href="http://infolab.stanford.edu/~dilys/dbprivacy/utkarsh/hippocratic_databases.html">hippocratic databases</a>, and <a href="https://en.wikipedia.org/wiki/Differential_Privacy">differential privacy</a>. I’d even followed it myself in the past, on and off. It’s a fascinating field, and a truly hard problem. I was rubbing my hands together in anticipation at this point. Tackling a hard problem is always fun.</p><p>The simplest place to start is to allow only aggregate queries, not row level results at all. You then also need to prevent filtering down to a single row. You can do that by imposing a minimum bound on the number of rows incorporated into any aggregate operation — say, 10 — but that can be easily defeated with a query (A) that matches your target row and 10 other rows, a query (B) that matches the same 10 rows but <em>not</em> your target, and then simply subtracting B’s results from A.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gFCIJ5bjHBEPo__RNQ-XCw.jpeg" /><figcaption><a href="https://snarfed.org/2012-07-07_venice_florence_tuscany#Venice">Carnival masks in Venice, Ryan Barrett</a></figcaption></figure><p>The complexity ramps up from there. I was dismayed to find that everyone concluded it’s more or less impossible to prevent re-identification, in general, with a number of queries that scales sublinearly with both the dataset size and query restrictions. Differential privacy was a major step forward, showing adding random noise could guarantee minimal privacy levels and require a higher number of queries to determine a given amount of new information, but reduced the accuracy of results, especially for smaller datasets. Plus, when I mentioned the idea of modifying query results by small amounts to our researchers, they laughed me out of the room.</p><p>Regardless, while perfect de-identification may be impossible, there were still lots of useful safeguards in the literature that were entirely doable and provided industrial strength defense against virtually all attacks. Even better, I was able to implement almost all of them in SQL, in the Metabase queries themselves. Only a couple required me to dive into <a href="https://github.com/metabase/metabase">Metabase’s open source code</a> itself.</p><h3>A simple matter of programming</h3><p>Next, I jumped into the <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a>, the code to extract and prepare the DNA and phenotype data from the subpopulation of our clients who had opted in. We use Google’s <a href="https://cloud.google.com/bigquery/">BigQuery</a> liberally as our data warehouse, and we’d already pointed Metabase at it, so I worked with our fearless head of data science, <a href="https://twitter.com/gilad">Gilad</a>, to write a bit of SQL and code to find data from research-consented clients, fully de-identify their data, clean up and normalize the data, write the results to CSV files, and upload them to a new set of tables in BigQuery. It took some careful thought to ensure we applied the right filters and quality checks to the self-reported data, but fortunately not much code.</p><p>BigQuery is designed for large scale data analysis. We knew this database would fall squarely on the “small data” end of the spectrum, totaling millions of rows at most, not billions or trillions. We also hoped for this database to be highly interactive and low latency, more like an end user application, which BigQuery is definitely <em>not</em> designed for. Still, BigQuery was a great place to prototype. We already had an ETL harness set up to write to it.</p><p>We also made liberal use of one of my favorite BigQuery features, <a href="https://cloud.google.com/bigquery/docs/schema-detect">schema auto-detection</a>. We changed the tables and columns and re-uploaded them over and over, and it happily re-detected the new schemas. We didn’t have to waste time updating an explicit DDL over and over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*5eu-gdCNDiLQWj1lGpBGzg.jpeg" /><figcaption><a href="https://www.flickr.com/photos/48336728@N06/4817594763">Butt-Kicking Machine, LisaGenius</a></figcaption></figure><h3>Delegate like a boss</h3><p>I moved on to the dashboard itself, the visualizations and data navigation that would comprise the meat of the tool. I threw together a handful of simple tables and graphs, added parameters to let users filter by DNA and phenotype features, then showed it to our <a href="https://www.color.com/research">research team</a>. They had <em>lots</em> of feedback. Great. I started to implement them, then quickly thought of a better option. They should implement them on their own!</p><p>One of the best ways to build something is to iterate. Try something, see how it works, identify weak parts, improve them, try again. It can take dozens of these cycles, often hundreds, to move from bad to mediocre to good to great. When the cycles are fast, that means they’re cheap, so you’re more likely to make it all the way to great before a deadline makes you stop and ship something merely good, or even mediocre.</p><p>I’ve spent much of my career identifying, analyzing, and speeding up these kinds of feedback loops. Every time I manage to do it, it pays off in spades, and this was no exception. Metabase is extremely accessible. Most of our researchers were already using it to do their own work, in various ways. They were immediately able to jump in, edit my queries and visualizations, and iterate on their own. No communication overhead, no waiting for me, just immediate gratification and <a href="https://en.wikipedia.org/wiki/Flow_(psychology)">flow</a>. It worked great, and made me feel all warm and fuzzy inside.</p><h3>The road to launch</h3><p>I came up for air and looked around. What next? Production architecture. This would be one of our first foray into serving an end user application based on code we hadn’t written ourselves. How should we set it up and secure it?</p><p>Unlike de-identification, this was a well understood problem, and our design gave us some lucky breaks. Our ETL system meant we could put the final dataset anywhere we wanted, as long as Metabase could get to it. We originally expected we’d have to migrate from BigQuery to a more conventional relational database like PostgreSQL, but when our final Metabase dashboard threw dozens of synchronous queries at BigQuery, it responded in a few seconds at most, which felt snappy enough to end users. We happily crossed that migration off our list.</p><p>Next, security. The name of the game was mitigating risk, and the hammer I had was separating and limiting access, so I began hammering everything in sight. I moved the dashboard to its own Metabase instance on a separate EC2 instance, security group, and VPC. I moved the BigQuery tables to their own Google Cloud project, created a dedicated service account for Metabase to use, and made sure that service account couldn’t access any of our other Google Cloud resources or data. Lastly, I put the HTML files in a dedicated S3 bucket and CloudFront distribution, separate from our other buckets and distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*iDXk2liDgz-kD5d0LN_bnQ.png" /><figcaption><a href="https://data.color.com/v1/">Color Data: cancer onset age histogram</a></figcaption></figure><p>At this point, we had a working site and database. We shopped it around internally, gathered feedback, and iterated. I threw together some ugly HTML, CSS, and social sharing buttons. I even <a href="https://github.com/metabase/metabase/pull/7817">wrote a few bits of JavaScript</a> to get individual queries into Google Analytics so we could see how people used the database.</p><h3>Victory lap</h3><p>Everyone loved it. Business development people started using it to pitch potential customers. Genetic counselors drilled into it to understand specific niches in our population. Our PR people especially liked it; they pitched it to GenomeWeb, which ended up publishing <a href="https://www.genomeweb.com/cancer/color-unveils-public-repository-variant-clinical-data-50k-customers-ashg-meeting">a thoughtful, clear-eyed take on it</a> that got a lot of traction. We even wrote a research paper and submitted it to <a href="https://academic.oup.com/database"><em>Database: The Journal of Biological Databases and Curation</em></a>, and good news: they’ve accepted it!</p><p>Our research team <a href="https://blog.color.com/color-at-ashg-a-year-of-large-scale-genomics-research-84885646cd4d">announced the database publicly at ASHG this year</a> to a standing room only crowd. We’ve heard a lot of interest and positive feedback from all over. More importantly, we’re seeing a growing amount of usage by real people using the database to get real insights out of our population.</p><p>My favorite result of all, though, is the software we prevented and the work we avoided. I worked on the database for four months or so, spending roughly half of my time on it. Our intrepid genetics researcher <a href="https://www.linkedin.com/in/cynthianeben02">Cynthia Neben</a> also worked hard on it for much of that time, spending maybe a cumulative a month or so total, including at least 10–20 hours politely nodding at my bad jokes. Gilad spent another week or two on data normalization and ETL.</p><p>A number of other people helped in invaluable ways, but when you add it all up, we still invested well under two person-quarters total. That’s way less than the multiple person-years we originally estimated it would take to build it from scratch. Even better, the only code we have to maintain is some static HTML and CSS and <a href="https://github.com/metabase/metabase/pulls?q=is%3Apr+author%3Asnarfed">a few small Metabase pull requests</a>. Score one for software prevention!</p><p>If it’s not already obvious, I had a ball building <a href="https://data.color.com/">Color Data</a>. I got to prototype an unconventional idea, iterate quickly with amazingly smart people, dive deep into a hard research problem, and deliver something that’s valuable to the research community at large. And as a bonus, my family and friends can try it too!</p><p><em>If this sounds interesting, </em><a href="https://www.color.com/careers"><em>we’re hiring</em></a><em>! </em><a href="/service/mailto:ryan@color.com"><em>Drop me a line any time</em></a><em>, I’m always happy to talk about Color.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fd8f996b733" width="1" height="1" alt=""><hr><p><a href="https://medium.com/color-genomics/building-a-public-research-database-out-of-spare-parts-1fd8f996b733">Building a public research database out of spare parts</a> was originally published in <a href="https://medium.com/color-genomics">Color</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop paying the ETL tax]]></title>
            <link>https://medium.com/color-genomics/stop-paying-the-etl-tax-385f08a0ffa7?source=rss-4f279603e35e------2</link>
            <guid isPermaLink="false">https://medium.com/p/385f08a0ffa7</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ryan Barrett]]></dc:creator>
            <pubDate>Thu, 31 Jan 2019 19:11:03 GMT</pubDate>
            <atom:updated>2019-01-31T19:11:03.147Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPSxihhBzCjk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPSxihhBzCjk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPSxihhBzCjk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a420c1f223d34ca70e5e7b6fb4b2976/href">https://medium.com/media/0a420c1f223d34ca70e5e7b6fb4b2976/href</a></iframe><blockquote><a href="https://en.wikiquote.org/wiki/The_Graduate#Dialogue">I want to say one word to you. Just one word. Are you listening? … ETL.</a></blockquote><p>Let me guess: that didn’t set your imagination on fire. Even in software engineering and data science, it’s not exactly a household term. Nor are the more modern terms <em>data platform</em> or <em>data engineering</em>. If you do know what they are, chances are you don’t have strong opinions. You know they’re out there, people do them, and that may be the end of it.</p><p>ETL stands for <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">Extract, Transform, Load</a>. It’s how you get your data from your primary <a href="https://en.wikipedia.org/wiki/OLTP">OLTP</a> database, which serves your application, into an <a href="https://en.wikipedia.org/wiki/OLAP">OLAP</a> <a href="https://en.wikipedia.org/wiki/Data_warehouse">data warehouse</a> designed for analysis, business intelligence, and data science.</p><p>Whatever your product is, it’s hopefully a core competence for your company. It’s a key differentiator. For many of us, data science and analysis are also key differentiators. ETL, however, is not. It looks basically the same everywhere, and does basically the same thing. These are all signposts that generally point toward buying or reusing, not building from scratch. Doing this kind of thing yourself just won’t move the needle.</p><p>Sadly, it’s exceedingly difficult to bolt an existing ETL onto the average database. You’ll use an existing data warehouse, and sometimes even tools that help with ETL, but rarely an existing ETL itself. Why? It’s entirely dependent on your data model, application, and needs. The input is your schema, the output is another schema driven entirely by your analysts and data scientists, and in the middle, a bunch of custom code that applies your business logic and stitches together your data in custom ways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*LePQVvefG08VZCtDDcZZZw.png" /><figcaption><a href="https://www.videoblocks.com/video/slow-motion-smashing-piggy-bank-xmqtytq">Slow Motion Smashing Piggy Bank, Storyblocks Video</a></figcaption></figure><p>You can find open source libraries and commercial tools that help read and write the data efficiently, the E and L in ETL. There are even GUI schema mapping tools that let you drag and drop columns and tables to make the T, transform, easier. There aren’t many, though, and they only support the simplest and smallest needs. What’s worse, transform is the most time intensive of the three, and takes the most work to maintain and update over time as your data model and needs evolve.</p><p>Modern alternatives like <a href="https://en.wikipedia.org/wiki/Extract,_load,_transform">ELT</a> and <a href="https://www.techopedia.com/definition/30153/schema-on-read">schema-on-read</a>, which move the transform into the data warehouse, and <a href="https://en.wikipedia.org/wiki/Data_virtualization">data virtualization</a>, which remove the data warehouse entirely, can be compelling but don’t fundamentally change the problem. Regardless of where or how, you’ll eventually need a data model for analysis that’s different from the one serving your application. You can construct it implicitly or explicitly, once inside your ETL/ELT or many times over inside individual queries, with engineers or analysts or other people entirely, but you’ll still need to do it.</p><p>So, we have this thing you have to build yourself, but won’t give you any meaningful advantage. Ugh.</p><p>We’ve felt this pain at <a href="https://www.color.com/">Color</a> just like everyone else. We avoided ETL as long as we could, but we eventually admitted that analyzing our data directly inside our primary schema, at scale, had become infeasible. We wrote and deployed our own ETL, and then another, and then another. We lamented that we had to pay this tax, and accepted it as a fact of life. Render unto the Lord what is the Lord’s, unto Caesar what is Caesar’s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/0*HWCVUmaVAV5-PRYI.jpg" /><figcaption><a href="https://www.imdb.com/title/tt0055683/">The Jetsons</a></figcaption></figure><p>Fast forward to last week. I’m complaining about ETL to <a href="http://hellowill.makeloft.org/">Will Stedden</a>, one of our amazing data scientists. After the usual whining and groaning, an idea occurs to us. We have a wealth of analysis queries in tools like <a href="https://www.metabase.com/">Metabase</a>, numbering in the thousands, many written in SQL. We want a good OLAP schema, but we don’t feel strongly about the exact tables and columns. We definitely don’t care about ETL code itself.</p><p>So we want good output, we have lots of training data, and we don’t need tight control over the implementation. Sounds like a task for…machine learning? It’s easy to think ML is the answer to everything these days, but in this case, it just might work.</p><p>We plunged ahead into a hand wavy, straw man design for ML-based ETL. Auto generating real code is a Hard Problem, so we started by envisioning it in pure SQL: one query to generate each individual output OLAP table.</p><ol><li>We start where most companies start, with our hodgepodge of BI and analysis queries written directly against the OLTP schema, in SQL. No ETL at all. It will generally take months to years to build up a large enough set of these queries to run a model on, so plan for it ahead of time. There’s still no such thing as a free lunch.</li><li>Once we have a comprehensive set of queries, we train a model to generate a core OLAP schema. The fitness function is the number of tables: fewer is generally better. This is somewhat analogous to principle component analysis: we want the smallest set of orthogonal tables that we can find, within reason.</li><li>For each candidate OLAP schema, we use a SQL query rewriter to rewrite all input queries onto the new OLAP tables. We then add in a second fitness function: simpler rewritten queries are better.</li><li>When our model finds the best OLAP schema, we again use our query rewriter to generate a SQL query to create each of its tables from the OLTP tables. We might initially install these as views in a read replica of our primary db, to test. Later, once we’re happy with them, we’d probably switch to a real ETL harness to run them regularly and populate a traditional data warehouse.</li><li>We update the existing SQL queries in BI tools like Metabase to use the OLAP schema. Hopefully we use APIs to edit them in place, programmatically.</li><li>As our application, OLTP schema, and analysis needs grow and evolve, we’ll need to update our ETL. To do this, we’ll need to update and write new BI queries on the primary OLTP schema to retrain the ETL model. Those queries can also include the existing OLAP queries. We’d then rerun the model, generate a new OLAP schema, and update all of the BI queries.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PtiFyZTX2ThemomE.jpg" /><figcaption><a href="https://en.wikipedia.org/wiki/The_Lord_of_the_Rings">The Lord of the Rings</a></figcaption></figure><p>This idea is flawed in all sorts of ways. For example, the UX isn’t great. It requires you to write lots of BI queries on your primary OLTP schema first, which we’ve already admitted is a poor fit. I also hand waved through a few nontrivial problems. Some are <a href="https://en.wikipedia.org/wiki/Small_matter_of_programming">simple matters of programming</a>, but others, like using a model to generate a relational database schema, are significant ML challenges in their own right. As our <a href="https://twitter.com/gilad">fearless head of data science</a> likes to say, that’s why they pay us the medium sized bucks. More importantly, that’s what makes them fun!</p><p>We don’t currently plan to do this ourselves at Color. It’s probably a boondoggle in all sorts of ways. Still, it seems like it just might work, and while we’ve found <a href="https://unifisoftware.com/blog/2017/03/unifis-advanced-machine-learning-artificial-intelligence-automates-etl/">people with somewhat similar ideas</a>, <a href="https://www.google.com/search?q=machine+learning+ETL">we haven’t yet found anyone who’s actually tried it</a>. If you’re interested, we’d happily be your first guinea pig!</p><p><em>If this sounds interesting, </em><a href="https://www.color.com/careers"><em>we’re hiring</em></a><em>! </em><a href="/service/mailto:ryan@color.com"><em>Drop me a line any time</em></a><em>, I’m always happy to talk about Color.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=385f08a0ffa7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/color-genomics/stop-paying-the-etl-tax-385f08a0ffa7">Stop paying the ETL tax</a> was originally published in <a href="https://medium.com/color-genomics">Color</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cosmic Serpent: DNA and the great Python 3 Migration]]></title>
            <link>https://medium.com/color-genomics/cosmic-serpent-dna-and-the-great-python-3-migration-52c7dcc39ce9?source=rss-4f279603e35e------2</link>
            <guid isPermaLink="false">https://medium.com/p/52c7dcc39ce9</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[Ryan Barrett]]></dc:creator>
            <pubDate>Tue, 08 May 2018 20:29:16 GMT</pubDate>
            <atom:updated>2018-06-06T16:24:46.737Z</atom:updated>
            <content:encoded><![CDATA[<p><em>(</em><a href="https://www.amazon.com/Cosmic-Serpent-DNA-Origins-Knowledge/dp/0874779642"><em>with apologies to </em></a><a href="https://www.amazon.com/kindle-dbs/entity/author/B001K8G6DQ/ref=dp_byline_cont_book_1"><em>Jeremy Narby</em></a><em>)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*m_7tUvc8MWAtmxZ_Ajbzzw.jpeg" /><figcaption>Ouroborus, the mythical snake that eats its own tail</figcaption></figure><p>Like any good software engineering team, we’re always updating our dependencies to make sure we’re running the most secure, bug-free versions. Recently, we tackled our oldest dependency: Python 2. Python 3 was released nearly 10 years ago, in December 2008 — long before Color was even founded! — but has been slow to unseat Python 2 as the industry standard. As of 2017, an estimated <a href="https://twitter.com/pycharm/status/865659029460209664">half of all Python projects</a> still hadn’t made the switch.</p><p>We’ve been thinking about Python 3 since the early days of Color, hoping to become relatively early adopters. Dependencies are always the first obstacle: back in 2014, common bioinformatics packages had no plan for Python 3 compatibility. We launched Color on Python 2, and over the years we periodically checked in on the compatibility of our dependencies, dreaming of Unicode strings, sane division, type annotations, and more. We’d inevitably get stuck when we learned that some small but important dependency still wasn’t Python 3 compatible. Plus, time is precious, so we deferred upgrading in favor of more pressing and immediate goals.</p><p>During our <a href="https://blog.color.com/fixing-things-up-in-color-engineering-e799442de38">August 2017 Fixit Week</a>, we took another pass through our dependencies and were pleasantly surprised to find that of the 130+ packages we relied on, all but a handful were Python 3 compatible. Fabric, Ansible, and Supervisor were important exceptions, but even those now had a path forward: we found a Python 3 fork of Fabric, and could install Ansible and Supervisor outside of our runtime virtualenv. So, with our dependencies in good shape, we began planning a full migration of our main repo.</p><p>According to <a href="http://cloc.sourceforge.net/">cloc</a>, we’d need to migrate more of 100,000 lines of non-test code, with roughly the same again in tests. We studied previous migration stories, advice, and popular tools for automating much of the work of updating, like <a href="https://docs.python.org/2/library/2to3.html">2to3</a> (links to recommended reading: [<a href="https://medium.com/@boxed/moving-a-large-and-old-codebase-to-python3-33a5a13f8c99">1</a>, <a href="http://blog.pyspoken.com/wp-content/uploads/2018/02/python2_to_3_migration_guide.pdf">2</a>, <a href="https://thenewstack.io/instagram-makes-smooth-move-python-3/">3</a>]). We appreciated benefitting from wisdom shared by others, so we want to share our own story, along with a few additional tips and notes that we haven’t seen elsewhere.</p><p>Broadly speaking, there are two approaches to upgrading a Python 2 codebase to Python 3:</p><ul><li><strong>Upgrade incrementally,</strong> maintaining Python 2 compatibility while gradually adding Python 3 support. Eventually you switch your interpreter, and your code continues to work as usual. This approach relies on libraries such as <a href="https://python-modernize.readthedocs.io/">modernize</a>, <a href="http://six.rtfd.org/">six</a>, and <a href="http://python-future.org/">future</a> to bridge the gaps.</li><li><strong>Upgrade the entire codebase, all at once.</strong> This approach relies on running 2to3, which upgrades your code without attempting to maintain compatibility.</li></ul><p>For the most part we chose the second option. Many organizations spend months (or years!) on painfully slow migrations, an ongoing tax we sought to avoid. With our next <a href="https://blog.color.com/fixing-fixit-week-b7591208b0a1">Fixit Week</a> just around the corner, we decided it would be more expedient to start a separate branch for Python 3, using Fixit Week as an opportunity to institute a brief code freeze so we could safely merge the Python 3 branch into master and test it extensively before deploying to production.</p><p>A few weeks ahead of the planned switchover, we worked on getting all of our unit tests passing in the Python 3 branch, backporting as many changes as possible into master to minimize the difference between the branches. Our workflow was:</p><ol><li>At the beginning of the week, checkout a new branch based off latest master.</li><li>Apply 2to3 fixers one at a time.</li><li>Rebase all changes from previous branch onto new branch.</li><li>Run tests and fix things.</li><li>Backport safe fixes into master.</li><li>Goto 1 and repeat.</li></ol><p>Thanks to relatively thorough test coverage, we found that we were able to safely backport many changes to master (still running Python), either directly or via limited/judicious use of six:</p><ul><li>The types, tuple_params, reduce, numliterals, next, funcattrs, except, and asserts 2to3 fixers, which resulted in Python 2 compatible code in our codebase.</li><li>Exception.message, which was already deprecated as of Python 2.7.</li><li>Converting print keywords to function calls, by adding from __future__ import print_function and a precommit hook to enforce using the function.</li><li>The import fixer, along with switching to absolute imports with from __future__ import absolute import.</li><li>Upgrading libraries to Python 3 compatible versions.</li><li>Comparators, i.e. the cmp keyword.</li><li>Using iterators directly where possible, notably dict.iteritems instead of items. Much of this was just removing unnecessary list() calls added by the iteritems fixer.</li><li>The division operator: we took a simple approach, grepping for /, filtering out false positives (lots of them), confirming that a float return value was acceptable or changing the operator to //, and finally adding from __future__ import division.</li></ul><p>These backported changes ran in production, in advance of our switchover, long enough that we were able to iron out the few wrinkles we had initially overlooked. Smooth sailing so far.</p><p>As Fixit Week began, we deployed our release candidate Python 3 branch to our internal staging environment and began aggressive manual QA together with our colleagues. We hoped that our automated testing was sufficient to catch any non-trivial regressions, but of course, hope is not a strategy. We were pleased to discover zero regressions in the most sensitive and carefully validated parts of our codebase, especially the <a href="https://blog.color.com/building-a-high-quality-reliable-and-efficient-bioinformatics-pipeline-aa4a01e182ba">bioinformatics pipeline</a> and our genetic report/result generator, both of which we backtest extensively against historic data.</p><p>We did uncover two minor regressions in rarely exercised non-clinical codepaths, but somewhat gratifyingly, these weren’t caused by the Python 3 migration: they had been introduced the previous week. That’s a successful milestone, in our book. (And of course, we fixed and updated test coverage accordingly!)</p><p>We were ready. With bated breath, we deployed our initial Python 3 release on Wednesday. Thanks to our test plan, things went quite smoothly, with no external-facing customer impact. Woo!</p><p>In the spirit of Color’s <a href="https://www.shrm.org/hr-today/news/hr-magazine/pages/0515-learning-culture.aspx">learning culture</a>, we note that we did encounter a few character encoding issues with asynchronous background tasks that lacked sufficient test coverage, which were easy to diagnose and fix. The most significant issue, and a tip which we haven’t seen shared elsewhere, is that we didn’t test running Python 2 and Python 3 side by side (as happens naturally when multiple instances are being updated mid-deploy).</p><p>Specifically, we cache heavily in production, and Python 2 was unable to read some cached objects stored by Python 3 because they were serialized with a newer version of pickle. We also saw this in other places that our code communicated across processes, notably arguments to Celery tasks that were pickled by an application server and unpickled by a Celery worker.</p><p>The migration’s biggest operational cost was, perhaps surprisingly, troubleshooting a few of our colleagues’ local development environments as they made the switch. We engineers are often particular about how we like our environment configured, and as usual, <a href="https://xkcd.com/1987/">XKCD hits a little too close to home here</a>. Not a big deal at Color — we’re a small team and can afford to help our colleagues individually get up and running in the new environment — but larger engineering organizations should plan accordingly.</p><p>Overall, the migration was a sound investment for Color engineering, one which we hope will benefit our productivity and the onboarding of our future colleagues — the next generation of Color engineers <a href="http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_answers.html#why-is-python-3-considered-a-better-language-to-teach-beginning-programmers">don’t need to learn</a> legacy Python quirks! Coding with Python 3 might not feel like living in the future, but we’re now at least squarely in the present.</p><p>Have you attempted a Python 3 migration? Leave a response and share your team’s experience — tips, tricks, or pitfalls to be avoided.</p><p><em>Thanks to TP Wong, Zach Langley, Jeremy Ginsberg, and everyone else in Color’s engineering and product teams who went above and beyond the call of duty to make this happen.</em></p><p><em>We’re hiring! If you’re a software engineer hoping to apply your skills in the service of a greater mission, while embracing modern software development practices and technologies, drop us a line! Check out our </em><a href="https://blog.color.com/building-a-high-quality-reliable-and-efficient-bioinformatics-pipeline-aa4a01e182ba"><em>other</em></a><em> </em><a href="https://blog.color.com/how-do-you-set-a-bioinformatics-speed-record-9d1d8986ad61"><em>recent</em></a><em> </em><a href="https://blog.color.com/fixing-fixit-week-b7591208b0a1"><em>engineering</em></a><em> </em><a href="https://blog.color.com/hacking-for-healthy-happy-humans-46c9980bfac6"><em>posts</em></a><em> to get a sense of how our team operates, or visit </em><a href="https://color.com/careers"><em>color.com/careers</em></a><em> for more information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52c7dcc39ce9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/color-genomics/cosmic-serpent-dna-and-the-great-python-3-migration-52c7dcc39ce9">Cosmic Serpent: DNA and the great Python 3 Migration</a> was originally published in <a href="https://medium.com/color-genomics">Color</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fixing things up in Color Engineering]]></title>
            <link>https://medium.com/color-genomics/fixing-things-up-in-color-engineering-e799442de38?source=rss-4f279603e35e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e799442de38</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ryan Barrett]]></dc:creator>
            <pubDate>Fri, 01 Sep 2017 21:23:45 GMT</pubDate>
            <atom:updated>2017-09-01T21:23:45.525Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AwWruqkw5O1ZfKPrFQIkeg.png" /></figure><p>Many startups are familiar with the idea of Hack Days and Hack Weeks. We’ve had a <a href="https://blog.color.com/channeling-the-hacker-way-7a4d8894cc79">few successful ones</a> over the years at Color. Fixit Weeks are less common, but just as important! Even the best engineering teams gradually accumulate <a href="https://en.wikipedia.org/wiki/Technical_debt">tech debt</a> over time that needs to be fixed.</p><p>Kim Scott <a href="https://books.google.com/books?id=IkLjDAAAQBAJ&amp;lpg=PA136&amp;ots=W9rJbWDpxs&amp;dq=fixit%20week%20vs%20hack%20week&amp;pg=PA136#v=onepage&amp;q&amp;f=false">notes</a> in <em>Radical Candor</em> that Fixit Week is a chance “to fix old and annoying problems that have been bothering people for months. It’s like cleaning out the utensil drawer where you spilled honey three months ago, but never quite found the time to take all the knives and forks out and scrub properly.”</p><p>We do routine maintenance and cleanup on an ongoing basis, alongside our core product roadmap development. Fixit Week helps us focus on bigger cleanup projects and sand-in-the-gears fixes that we haven’t quite managed to tackle yet. It might sound boring, but for engineers, it’s actually a lot of fun.</p><p>Last week was Color’s third Fixit Week. We kept a list of all the ideas and bugs we hoped to fix, voted on the ones we wanted most, and kept a running tally of who’d fixed what. At the end of the week, we held a ceremony and awarded prizes based on judges’ votes, popular choice, and special categories. Prizes came from — where else? — the local hardware store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*86bIFE24lyNvnTcRZWV2GQ.jpeg" /><figcaption>Katsuya Noguchi, one of the Color engineers who participated in Fixit Week.</figcaption></figure><p>We made the most of the opportunity:</p><ul><li>Added <a href="http://www.onebigfluke.com/2012/06/i-showed-how-perceptual-diffs-are-core.html">perceptual diffs</a> to our <a href="https://en.wikipedia.org/wiki/Continuous_integration">continuous integration</a> to catch unintentional UI changes.</li><li>Redesigned our engineering interview process to be more hands on and better aligned with our actual daily work.</li><li>Prototyped an OCR integration that will let clients <a href="https://support.color.com/hc/en-us/sections/200717340-Activating-Your-Kit">activate kits</a> by taking a picture instead of typing in long numbers and barcodes.</li><li>Automated a number of processes for <a href="http://color.com/benefits">benefits providers</a> that we’d been doing by hand, slowly and laboriously.</li><li>Automatically generate invoices that our support team was creating manually.</li><li>Automated employee onboarding and account creation across dozens of services.</li><li>Switched one of our analytics tools, <a href="http://www.metabase.com/">Metabase</a>, to use single-sign-on instead of VPN, making it more convenient to use outside of the office.</li><li>Added more important information to the tools used by our genetic counselors, so it’s at their fingertips when they talk with clients and they don’t have to interrupt conversations.</li><li>Upgraded a number of big libraries and frameworks to new major versions: Python, Angular, and more.</li></ul><p>One of my favorite parts of Fixit Week isn’t even the week at all, but afterward. We’ve been living with most of these rough edges for so long, we’ve gotten used to them. We only notice them again after they’re gone, and our lives are a little better, just a bit, in dozens of ways. It’s therapeutic, even liberating.</p><p>It almost makes me want to try it at home. Robotic kitty litter box, anyone?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e799442de38" width="1" height="1" alt=""><hr><p><a href="https://medium.com/color-genomics/fixing-things-up-in-color-engineering-e799442de38">Fixing things up in Color Engineering</a> was originally published in <a href="https://medium.com/color-genomics">Color</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Channeling The Hacker Way]]></title>
            <link>https://medium.com/color-genomics/channeling-the-hacker-way-7a4d8894cc79?source=rss-4f279603e35e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a4d8894cc79</guid>
            <category><![CDATA[genetic-testing]]></category>
            <category><![CDATA[color-genomics]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[hackathons]]></category>
            <dc:creator><![CDATA[Ryan Barrett]]></dc:creator>
            <pubDate>Wed, 31 Aug 2016 19:58:30 GMT</pubDate>
            <atom:updated>2016-11-30T15:33:26.239Z</atom:updated>
            <content:encoded><![CDATA[<p>We have big dreams and ambitious plans. We want to push the state of the art in health and genetics, and we need broad, crazy moonshot product ideas to get us there. How do we find those ideas?</p><p>We’ve always been inspired by <a href="http://www.businessinsider.com/google-20-percent-time-policy-2015-4">20% time</a> at Google and 3M as well as <a href="https://blog.twitter.com/2012/hack-week-twitter">hack weeks at Twitter</a>, so we decided to do our own hack week. We invited everyone to put normal work on hold for a full week to try out new ideas, no matter how crazy or tangential. We didn’t know how many people would participate, or whether we could ship any of the results, but it’s always worth a shot!</p><h3>benji shine on Twitter</h3><p>Get comfortable with uncertainty; it leaves room for inspiration. #hackweek</p><p>Our first task was to pick a theme. We chose <em>engagement</em>: how can we help people continue to engage with their genetics and health over time? We dragged everyone into a few big rooms and brainstormed project ideas. No judgment, no constraints, no requirements, no analysis or pro/con lists. Just ideas. By the end of these sessions, we had over 50 candidates to explore in the week to come.</p><p>We kicked off hack week proper on Monday morning with an example project: reincarnation detection. Could we identify who you’d been in past lives? With tongue firmly in cheek, we described how to recruit teammates with good karma, work with researchers to identify genetic markers across lives, partner with Buddhist monks, and design a “Talk to the Other Side” messaging UX.</p><p>We set everyone loose, and they hit the ground running. People wrote up pitches, formed teams with great names — <em>Pink Duck</em> was a personal favorite — and jumped on company mailing lists to recruit team members. Many people joined two or three teams each.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TeOqwu5ScpKM-GOZapMntw.jpeg" /></figure><p>We emphasized that we wanted <em>everyone</em> to participate, across job roles. Design mockups and prototype code were nice, but not required. We encouraged people to start broad and high level, then find a <a href="https://en.wikipedia.org/wiki/Vertical_slice">vertical slice</a> that they could flesh out in a single week. We put up screens around the office that rotated between team Slack channels, project docs, and mockups and prototypes as they came together.</p><p>The week culminated in an all hands <a href="https://techcrunch.com/2016/08/22/y-combinator-demo-day-summer-2016/">Demo Day</a> where each team got a few minutes to present their project. We saw live demos, skits, theme songs, and even a custom video. We voted on yearbook-style awards like Best Dressed, Most Likely to Succeed, and Crazy like a Fox. Everyone had a blast.</p><p>After the excitement wound down and the dust settled, we surveyed the results. Over half the company participated, working on 11 projects that all made significant progress toward proofs of concept.</p><p>Hack week infused us with new ideas to help our clients understand and act on their genetic data. We look forward to implementing some of the best ideas, and we hope they’ll help our clients lead longer, healthier lives. That’s the real prize.</p><p><em>We’re always looking for talented engineers. </em><a href="https://getcolor.com/careers"><em>Join us</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a4d8894cc79" width="1" height="1" alt=""><hr><p><a href="https://medium.com/color-genomics/channeling-the-hacker-way-7a4d8894cc79">Channeling The Hacker Way</a> was originally published in <a href="https://medium.com/color-genomics">Color</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>