<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samuel Lampa on Medium]]></title>
        <description><![CDATA[Stories by Samuel Lampa on Medium]]></description>
        <link>https://medium.com/@saml?source=rss-1f14ac874b71------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*piNo5WitPMBXTQC_u0vLBg.jpeg</url>
            <title>Stories by Samuel Lampa on Medium</title>
            <link>https://medium.com/@saml?source=rss-1f14ac874b71------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:06:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saml/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Parsing DrugBank XML (or any large XML file) in streaming mode in Go]]></title>
            <link>https://medium.com/@saml/parsing-drugbank-xml-or-any-large-xml-file-in-streaming-mode-in-go-ea9a1d570ca8?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea9a1d570ca8</guid>
            <category><![CDATA[xml-processing]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[drug-discovery]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Thu, 20 Aug 2026 13:23:10 GMT</pubDate>
            <atom:updated>2026-08-20T13:23:10.172Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rchfRyPBdne_FdNB0c564A.png" /><figcaption>Image by ChatGPT. Gopher by René French.</figcaption></figure><p>I had a problem in which I thought I needed to parse the full <a href="https://www.drugbank.ca/">DrugBank</a> dataset, which comes as a <a href="https://www.drugbank.ca/releases/5-0-11/downloads/all-full-database">(670MB) XML file</a></p><p>(For open access papers describing DrugBank, see: [1], [2], [3] and [4]). It turned out what I needed was available as CSV files under “<a href="https://www.drugbank.ca/releases/latest#structures">Structure External Links</a> ”. There is probably still some other uses of this approach though, as the XML version of DrugBank seems to contain a lot more information in a single format. And in any case, this forced me to figure out how to parse large XML files in a streaming fashion in Go, as older tools like <a href="http://xmlstar.sourceforge.net/">XMLStarlet</a> chokes for many minutes upon the DrugBank file (trying to read it all into memory?), killing any attempt at an iterative development cycle. And, it turns out Go’s support for streaming XML parsing is just great!</p><p>While Go’s XML stream-parsing support is great, the details of how to do that in a streaming fashion was not immediately clear from the docs, and I was thus saved by <a href="http://blog.davidsingleton.org/parsing-huge-xml-files-with-go/">this blog post</a> by <a href="http://twitter.com/dps">David Singleton</a> . Basically, you could use his blog post as a starting point, but I wanted to write up my own post to document some specifics and peculiarities I figured out.</p><h3>Idea: Parse DrugBank XML to TSV</h3><p>So in short, we want to parse the DrugBank XML, which contains tons of hierarchical information about each drug in the dataset, and extract just a few fields, and output that into a nicely formatted tab-separated (.tsv) file.</p><p>The fields which we want to extract in this example (which is based on my real world problem mentioned above) for each drug are:</p><ul><li>InchiKey (a hashed ID representing a chemical structure)</li><li>Approved/Withdrawn status</li><li>ChEMBL ID (for the compound)</li><li>PubChem Compound ID (CID)</li><li>PubChem Substance ID (SID)</li></ul><h3>The DrugBank XML format</h3><p>The DrugBank XML format is a pretty simple one on its highest level: It is basically a lot of elements thrown into a enclosing tag. The relative complexity comes within the tag. But because of how Go parses XML to structs using tags, we can just skip most of that info, and only focus on the parts we are interested in.</p><p>A schematic example of the DrugBank XML containing only the fields we are interested could look like this:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;drugbank xmlns=&quot;http://www.drugbank.ca&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;http://www.drugbank.ca http://www.drugbank.ca/docs/drugbank.xsd&quot; version=&quot;5.0&quot; exported-on=&quot;2017-12-20&quot;&gt;<br>&lt;drug type=&quot;small molecule&quot; created=&quot;2005-06-13&quot; updated=&quot;2017-12-19&quot;&gt;<br>  &lt;name&gt;Bivalirudin&lt;/name&gt;<br>  &lt;groups&gt;<br>    &lt;group&gt;approved&lt;/group&gt;<br>    &lt;group&gt;investigational&lt;/group&gt;<br>  &lt;/groups&gt;<br>  &lt;external-identifiers&gt;<br>    &lt;external-identifier&gt;<br>      &lt;resource&gt;PubChem Compound&lt;/resource&gt;<br>      &lt;identifier&gt;16129704&lt;/identifier&gt;<br>    &lt;/external-identifier&gt;<br>    &lt;external-identifier&gt;<br>      &lt;resource&gt;PubChem Substance&lt;/resource&gt;<br>      &lt;identifier&gt;46507415&lt;/identifier&gt;<br>    &lt;/external-identifier&gt;<br>    &lt;external-identifier&gt;<br>      &lt;resource&gt;ChEMBL&lt;/resource&gt;<br>      &lt;identifier&gt;CHEMBL2103749&lt;/identifier&gt;<br>    &lt;/external-identifier&gt;<br>  &lt;/external-identifiers&gt;<br>  &lt;calculated-properties&gt;<br>    &lt;property&gt;<br>      &lt;kind&gt;InChIKey&lt;/kind&gt;<br>      &lt;value&gt;OIRCOABEOLEUMC-GEJPAHFPSA-N&lt;/value&gt;<br>      &lt;source&gt;ChemAxon&lt;/source&gt;<br>    &lt;/property&gt;<br>  &lt;/calculated-properties&gt;<br>&lt;/drug&gt;<br>&lt;/drugbank&gt;</pre><p>(In reality, each element is far far many more lines than this, which really necessitates proper XML parsing tools).</p><h3>Mapping XML to Go Structs</h3><p>The parsing strategy for XML in Go, is the same as for other formats such as JSON: Define one or more structs into which we parse certain XML elements and attributes. The mapping between XML (or JSON) and the struct fields are done using so called “tags” which are added within back-quotes after the fields in the struct definition. Thus, defining sensible structs and XML element-to-struct field mappings is the core of the work, and will directly influence how easy your code will be to work with.</p><p>Below you can see the struct-structure (sounds just right, right?) I defined to be able to parse out the data I’m interested in:</p><pre>type Drugbank struct {<br>    XMLName xml.Name `xml:&quot;drugbank&quot;`<br>    Drugs   []Drug   `xml:&quot;drug&quot;`<br>}<br><br>type Drug struct {<br>    XMLName              xml.Name             `xml:&quot;drug&quot;`<br>    Name                 string               `xml:&quot;name&quot;`<br>    Groups               []string             `xml:&quot;groups&gt;group&quot;`<br>    CalculatedProperties []Property           `xml:&quot;calculated-properties&gt;property&quot;`<br>    ExternalIdentifiers  []ExternalIdentifier `xml:&quot;external-identifiers&gt;external-identifier&quot;`<br>}<br>type Property struct {<br>    XMLName xml.Name `xml:&quot;property&quot;`<br>    Kind    string   `xml:&quot;kind&quot;`<br>    Value   string   `xml:&quot;value&quot;`<br>    Source  string   `xml:&quot;source&quot;`<br>}<br>type ExternalIdentifier struct {<br>    XMLName    xml.Name `xml:&quot;external-identifier&quot;`<br>    Resource   string   `xml:&quot;resource&quot;`<br>    Identifier string   `xml:&quot;identifier&quot;`<br>}</pre><p>We can note the following:</p><ul><li>As said, the stuff within back-quotes represent the structure in the XML to be mapped to a certain field.</li><li>Note how, for nested hierarchies, we need multiple struct types, such as the “Property” and “ExternalIdentifier” ones … which are then linked to from the main “Drug” struct”.</li><li>We also need a struct for the highest level element, .</li><li>Each struct needs to have a field of xml.Name type (named XMLName for simplicity), that defines its name in the XML, so that we have somewhere to add our XML-mapping tag.</li><li>Note, when we have a slice (“list”) of things, such as the “CalculatedProperties” field in the “Drug” struct, how we need to specify a two level path (xml:&quot;calculated-properties**&gt;**property&quot;) into the XML structure, so that we get down to the individual “property” XML elements which are placed inside a grouping “calculated-properties” element.</li></ul><p>With that set up, we can create our Go code to loop over our XML file in a streaming fashion, along the lines of David’s <a href="http://blog.davidsingleton.org/parsing-huge-xml-files-with-go/">blog post</a> , while also creating a TSV writer, which we use to stream-write our extracted output into a new file, drugbank_extracted.tsv (imports and main function left out for brevity):</p><pre>xmlFile, err := os.Open(&quot;drugbank.xml&quot;)<br>if err != nil {<br>    panic(&quot;Could not open file: drugbank.xml&quot;)<br>}<br><br>tsvFile, err := os.Create(&quot;drugbank_extracted.tsv&quot;)<br>if err != nil {<br>    panic(&quot;Could not create file: drugbank_extracted.tsv&quot;)<br>}<br><br>tsvWrt := csv.NewWriter(tsvFile)<br>tsvWrt.Comma = &#39;\t&#39;<br>tsvHeader := []string{&quot;inchikey&quot;, &quot;status&quot;, &quot;chembl_id&quot;, &quot;pubchem_sid&quot;, &quot;pubchem_cid&quot;}<br>tsvWrt.Write(tsvHeader)<br><br>// Implement a streaming XML parser according to guide in<br>// http://blog.davidsingleton.org/parsing-huge-xml-files-with-go<br>xmlDec := xml.NewDecoder(xmlFile)<br>for {<br>    t, tokenErr := xmlDec.Token()<br>    if tokenErr != nil {<br>        if tokenErr == io.EOF {<br>            break<br>        } else {<br>            panic(&quot;Failed to read token:&quot; + tokenErr.Error())<br>        }<br>    }<br>    switch startElem := t.(type) {<br>    case xml.StartElement:<br>        if startElem.Name.Local == &quot;drug&quot; {<br>            var status string<br>            var inchiKey string<br>            var chemblID string<br>            var pubchemSID string<br>            var pubchemCID string<br><br>            drug := &amp;Drug{}<br>            decErr := xmlDec.DecodeElement(drug, &amp;startElem)<br>            if err != nil {<br>                panic(&quot;Could not decode element&quot; + decErr.Error())<br>            }<br>            for _, g := range drug.Groups {<br>                if g == &quot;approved&quot; {<br>                    status = &quot;A&quot;<br>                }<br>                // Withdrawn till &quot;shadow&quot; (what&#39;s the correct term?) approved status<br>                if g == &quot;withdrawn&quot; {<br>                    status = &quot;W&quot;<br>                }<br>            }<br>            for _, p := range drug.CalculatedProperties {<br>                if p.Kind == &quot;InChIKey&quot; {<br>                    inchiKey = p.Value<br>                }<br>            }<br><br>            for _, eid := range drug.ExternalIdentifiers {<br>                if eid.Resource == &quot;ChEMBL&quot; {<br>                    chemblID = eid.Identifier<br>                } else if eid.Resource == &quot;PubChem Substance&quot; {<br>                    pubchemSID = eid.Identifier<br>                } else if eid.Resource == &quot;PubChem Compound&quot; {<br>                    pubchemCID = eid.Identifier<br>                }<br>            }<br><br>            tsvWrt.Write([]string{inchiKey, status, chemblID, pubchemSID, pubchemCID})<br>        }<br>    case xml.EndElement:<br>        continue<br>    }<br>}<br>tsvWrt.Flush()<br>xmlFile.Close()<br>tsvFile.Close()</pre><h3>Turn into a reproducible workflow with SciPipe</h3><p>Now, we could use SciPipe (Go-based workflow library I’m developing) to put this into a little workflow, where we also automaticalyl download the DrugBank data, Unzip it, and run the XML-to-TSV code. See <a href="https://gist.github.com/samuell/fc82fad39e7efda7987fc18173777f7f">this gist</a> for the full workflow code.</p><p>To run the Go file in the gist, in short, what you need to do is:</p><ul><li>Create a file drugbank_userinfo.txt, containing your DrugBank website username and password, on the form: USERNAME:PASSWORD</li><li>Install Go</li><li>Install scipipe with go get github.com/scipipe/scipipe/...</li><li>Make sure you have curl installed. On Ubuntu: sudo apt-get install curl</li></ul><p>Then, you should be able to run it, with:</p><pre>go run drugbank_xml_to_tsv_with_scipipe.go</pre><h3>Full SciPipe workflow code example</h3><p>I’m including the full SciPipe workflow code below as well, for the day when Github is down ;):</p><pre>package main<br><br>import (<br>    &quot;encoding/csv&quot;<br>    &quot;encoding/xml&quot;<br>    &quot;io&quot;<br>    &quot;os&quot;<br>    sp &quot;github.com/scipipe/scipipe&quot;<br>)<br><br>// ------------------------------------------------------------------------<br>// Workflow definition<br>// ------------------------------------------------------------------------<br><br>func main() {<br>    wf := sp.NewWorkflow(&quot;exvsdb&quot;, 2)<br>    // DrugBank XML<br>    // Note: We need to prepend &#39;../&#39; to the file drugbank_userinfo.txt, because the workflow will be run in a subfolder<br>    download := wf.NewProc(&quot;download&quot;, &quot;curl -Lfv -o {o:zip} -u $(cat ../drugbank_userinfo.txt) https://www.drugbank.ca/releases/5-0-11/downloads/all-full-database&quot;)<br>    download.SetOut(&quot;zip&quot;, &quot;dat/drugbank.zip&quot;)<br>    unzip := wf.NewProc(&quot;unzip&quot;, `unzip -d dat/ {i:zip}; mv &quot;dat/full database.xml&quot; {o:xml}`)<br>    unzip.SetOut(&quot;xml&quot;, &quot;dat/drugbank.xml&quot;)<br>    unzip.In(&quot;zip&quot;).From(download.Out(&quot;zip&quot;))<br>    xmlToTSV := wf.NewProc(&quot;xml2tsv&quot;, &quot;# Custom Go code with input: {i:xml} and output: {o:tsv}&quot;)<br>    xmlToTSV.SetOut(&quot;tsv&quot;, &quot;{i:xml}.extr.tsv&quot;)<br>    xmlToTSV.In(&quot;xml&quot;).From(unzip.Out(&quot;xml&quot;))<br>    xmlToTSV.CustomExecute = NewXMLToTSVFunc() // Getting the custom Go function in a factory method for readability<br>    wf.Run()<br>}<br>// --------------------------------------------------------------------------------<br>// DrugBank struct definitions<br>// --------------------------------------------------------------------------------<br>type Drugbank struct {<br>    XMLName xml.Name `xml:&quot;drugbank&quot;`<br>    Drugs   []Drug   `xml:&quot;drug&quot;`<br>}<br>type Drug struct {<br>    XMLName              xml.Name             `xml:&quot;drug&quot;`<br>    Name                 string               `xml:&quot;name&quot;`<br>    Groups               []string             `xml:&quot;groups&gt;group&quot;`<br>    CalculatedProperties []Property           `xml:&quot;calculated-properties&gt;property&quot;`<br>    ExternalIdentifiers  []ExternalIdentifier `xml:&quot;external-identifiers&gt;external-identifier&quot;`<br>}<br>type Property struct {<br>    XMLName xml.Name `xml:&quot;property&quot;`<br>    Kind    string   `xml:&quot;kind&quot;`<br>    Value   string   `xml:&quot;value&quot;`<br>    Source  string   `xml:&quot;source&quot;`<br>}<br>type ExternalIdentifier struct {<br>    XMLName    xml.Name `xml:&quot;external-identifier&quot;`<br>    Resource   string   `xml:&quot;resource&quot;`<br>    Identifier string   `xml:&quot;identifier&quot;`<br>}<br>// --------------------------------------------------------------------------------<br>// Components<br>// --------------------------------------------------------------------------------<br>// NewXMLToTSVFunc returns a CustomExecute function to be used by the XML to TSV<br>// component in the workflow above<br>func NewXMLToTSVFunc() func(t *sp.Task) {<br>    return func(t *sp.Task) {<br>        fh, err := os.Open(t.InPath(&quot;xml&quot;))<br>        if err != nil {<br>            sp.Fail(&quot;Could not open file&quot;, t.InPath(&quot;xml&quot;))<br>        }<br>        tsvWrt := csv.NewWriter(t.OutIP(&quot;tsv&quot;).OpenWriteTemp())<br>        tsvWrt.Comma = &#39;\t&#39;<br>        tsvHeader := []string{&quot;inchikey&quot;, &quot;status&quot;, &quot;chembl_id&quot;, &quot;pubchem_sid&quot;, &quot;pubchem_cid&quot;}<br>        tsvWrt.Write(tsvHeader)<br>        // Implement a streaming XML parser according to guide in<br>        // http://blog.davidsingleton.org/parsing-huge-xml-files-with-go<br>        xmlDec := xml.NewDecoder(fh)<br>        for {<br>            t, tokenErr := xmlDec.Token()<br>            if tokenErr != nil {<br>                if tokenErr == io.EOF {<br>                    break<br>                } else {<br>                    sp.Fail(&quot;Failed to read token:&quot;, tokenErr)<br>                }<br>            }<br>            switch startElem := t.(type) {<br>            case xml.StartElement:<br>                if startElem.Name.Local == &quot;drug&quot; {<br>                    var status string<br>                    var inchiKey string<br>                    var chemblID string<br>                    var pubchemSID string<br>                    var pubchemCID string<br>                    drug := &amp;Drug{}<br>                    decErr := xmlDec.DecodeElement(drug, &amp;startElem)<br>                    if err != nil {<br>                        sp.Fail(&quot;Could not decode element&quot;, decErr)<br>                    }<br>                    for _, g := range drug.Groups {<br>                        if g == &quot;approved&quot; {<br>                            status = &quot;A&quot;<br>                        }<br>                        // Withdrawn till &quot;shadow&quot; (what&#39;s the correct term?) approved status<br>                        if g == &quot;withdrawn&quot; {<br>                            status = &quot;W&quot;<br>                        }<br>                    }<br>                    for _, p := range drug.CalculatedProperties {<br>                        if p.Kind == &quot;InChIKey&quot; {<br>                            inchiKey = p.Value<br>                        }<br>                    }<br>                    for _, eid := range drug.ExternalIdentifiers {<br>                        if eid.Resource == &quot;ChEMBL&quot; {<br>                            chemblID = eid.Identifier<br>                        } else if eid.Resource == &quot;PubChem Substance&quot; {<br>                            pubchemSID = eid.Identifier<br>                        } else if eid.Resource == &quot;PubChem Compound&quot; {<br>                            pubchemCID = eid.Identifier<br>                        }<br>                    }<br>                    tsvWrt.Write([]string{inchiKey, status, chemblID, pubchemSID, pubchemCID})<br>                }<br>            case xml.EndElement:<br>                continue<br>            }<br>        }<br>        tsvWrt.Flush()<br>        fh.Close()<br>    }<br>}</pre><p>(Code license: <a href="https://unlicense.org/">Public Domain</a> )</p><ul><li><strong>Note I (2018–03–21):</strong> <a href="https://twitter.com/yokofakun">Pierre Lindenbaum</a> <a href="https://twitter.com/yokofakun/status/976488405654736896">kindly suggested</a> an <a href="https://gist.github.com/lindenb/5a76d95397a86b860386cdf3976726a2">alternative approach</a> using the <a href="https://docs.oracle.com/javase/8/docs/technotes/tools/unix/xjc.html">xjc</a> tool to generate Java code, and a Makefile to provide for workflow functionality (<a href="https://gist.github.com/lindenb/5a76d95397a86b860386cdf3976726a2">see code</a> ). He also <a href="https://twitter.com/yokofakun/status/976515166002204673">provided a version</a> implemented with XSLT (<a href="https://gist.github.com/lindenb/63c85ee0e8c21b6cc3e7d44b77dd93db">see code</a> ).</li><li><strong>Note II (2018–03–21):</strong> <a href="https://www.reddit.com/r/golang/comments/854zl8/data_science_parsing_drugbank_xml_or_any_large">Some comments on the post on reddit</a> .</li><li><strong>Edit 2018–03–20:</strong> Clarification of general usefulness of parsing the DrugBank XML in the intro.</li><li><strong>Edit 2018–06–16:</strong> Update code examples to new <a href="https://github.com/scipipe/scipipe/releases/tag/v0.7">SciPipe 0.7 API</a> (Use From() instead of Connect() for connections)</li><li><strong>Edit 2019–03–02:</strong> Update code examples to new <a href="https://github.com/scipipe/scipipe/releases/tag/v0.8.0">SciPipe 0.8 API</a> (Use SetOut() instead of SetPathStatic(), SetPathExtend() etc)</li></ul><p><em>This post was originally </em><a href="https://livingsystems.cc/posts/parsing-drugbank-xml-or-any-large-xml-file-in-streaming-mode-in-go/"><em>posted to livingsystems.cc</em></a><em> (in 2018)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea9a1d570ca8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Equation-centric dataflow programming in Go]]></title>
            <link>https://medium.com/@saml/equation-centric-dataflow-programming-in-go-dae783037df3?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/dae783037df3</guid>
            <category><![CDATA[functional-programming]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Sat, 20 Jul 2019 23:16:56 GMT</pubDate>
            <atom:updated>2019-07-20T23:27:21.626Z</atom:updated>
            <content:encoded><![CDATA[<p>Even though computations done on computers are very often based on some type of math, it is striking that the notation used in math to express equations and relations is not always very readily converted into programming code. Outside of purely <a href="https://en.wikipedia.org/wiki/Symbolic_programming">symbolic programming</a> languages like <a href="http://www.sagemath.org/">sage math</a> or the (proprietary) <a href="https://www.wolfram.com/language">wolfram language</a>, there seems to always be quite a divide between the mathematical notation and the numerical implementation.</p><p>I’m generelly interested in how we can go towards a more declarative way of specifying relations between entities, to minimize the amount of coding we have to do when modelling systems. I find it especially interesting with approaches that are easy to implement in mainstream programming environments, so that existing knowledge, 3rd party library ecosystems and performance optimizations gone into these systems can be re-used.</p><p>Optimal is if we can fluently convert such declarative coding approaches into <a href="https://en.wikipedia.org/wiki/Dataflow">dataflow</a>-, or even <a href="https://en.wikipedia.org/wiki/Flow-based_programming">flow-based programming</a> programs, since these approaches are a pretty good fit for today’s multicore CPU chips, where the inherent pipeline parallelism makes good use of multiple computing cores without too much hoops, except we express the computations as pipelines of asynchronously running processing components.</p><h3>Enter Go generator functions</h3><p>Pondering the fact that Go’s concurrency primitives (go-routines and channels) combined into the generator pattern (<a href="https://talks.golang.org/2012/concurrency.slide#25">see Rob Pike’s slides here</a>), is pretty much dataflow in a pretty functional style, it struck me that Go generator functions might be useful for expressing computation in a way that resembles mathematical equations.</p><p>If you haven’t seen Go generator functions, check the link above to Rob’s slides. Otherwise, in short, they are functions which, instead of returning a finished value, return a channel, from which one or more values can be retrieved. Internally, the generator function will start a go-routine which will produce the values for you <em>lazily</em>, or, <em>concurrently</em> while you read from the channel (or it might pre-compute some values if the channel is buffered).</p><p>Said and done. This turned out to be easier than expected. With the help of one or two helper functions, I could easily implement various numeric operators like “Add” and “Div” in the form of generator functions that operate on a stream of numbers, and in turn returns a stream of integers or numbers. Since the returned channel represents the output of each operator function, it becomes possible to nest these operator function calls in a way that somehow mimics a mathematical formula chosen to implement.</p><p>As example formula for my experiment, I wanted something with simple operators (mostly “Add”, “Sub”, “Mul” and “Div), but complex enough that it is relevant to make the formula as readable as possible. The monthly payment formula from <a href="http://www.math.utah.edu/~pa/math/equations/equations.html">this page</a> seemed like a good fit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/0*V3lqVSm3F5Q6l869.gif" /><figcaption>Image source: <a href="http://www.math.utah.edu/~pa/math/equations/equations.html?source=post_page---------------------------">www.math.utah.edu/~pa/math/equations/equations.html</a></figcaption></figure><p>It expresses what your monthly payment, m, will be, given that you borrow an amount L of dollars and pay it back over N months, with an annual interest rate of p percent.</p><p>With a few generator functions for division, addition, multiplication and exponentiation, we can now implement this formula as a nested tree of such generation functions. The formula expressed in this way looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a966ba8c8d38afeb0525be073106fd23/href">https://medium.com/media/a966ba8c8d38afeb0525be073106fd23/href</a></iframe><p>As you can see, there are some peculiarities, like that we have to wrap constants in “Val()” functions, to make these values into a stream of their values, rather than just constants (since all of the generator functions take streams of values rather than pure values).</p><p>The full Go code can be found below, and in the bottom you will find the implementaiton of the operator functions, and the Val functions. In this example, they all build on the Apply2() function further below, which takes two input streams of values, and a function, and applies that function on the stream of values, returning a single stream of values. (For more complex functions, we might need Apply1, Apply3, etc):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8169ed9a6975c3922c961f3ed073cb9/href">https://medium.com/media/e8169ed9a6975c3922c961f3ed073cb9/href</a></iframe><p>You can view and run an experiment demonstrating this <a href="https://play.golang.org/p/C3OovxjcMU">on the go playground</a>.</p><h3>Limitations and concluding remarks</h3><p>So, as you see, it is indeed possible to express dataflow computations in a quite an equation-centric way in Go.</p><p>Now, there are some limits to this approach. For example, we have had to decide in which direction the computation goes: From inputs to outputs. Perhaps it is possible to express this in a more generic way, since Go channels are after all bi-directional, unless they are explicitly made uni-directional?</p><p>We also get into territory where Go’s lack of generics is giving some consequences. In order to keep the above code example reasonably short, we have limited outselves to streams of float values. Extending this to arbitrary combinations of floats, integers, and of all the sizes, would lead to quite an explosion in code duplication, and be far less fun to implement.</p><p>Anyways, hope this little experiement can spur some ideas on ways to merge declarative, relation-centric programming into mainstream programming environments.</p><p><em>Originally published at </em><a href="http://bionics.it/posts/equation-centric-dataflow-programming-in-go"><em>bionics.it</em></a><em>. Tags: Dataflow, Go, Golang, Mathematics</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dae783037df3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structured Go-routines or framework-less Flow-Based Programming in Go]]></title>
            <link>https://medium.com/@saml/structured-go-routines-or-framework-less-flow-based-programming-in-go-2c7ab200e332?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c7ab200e332</guid>
            <category><![CDATA[design]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Sat, 02 Mar 2019 13:08:23 GMT</pubDate>
            <atom:updated>2019-03-02T13:08:23.422Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post has </em><a href="https://bionics.it/posts/structured-go-routines-or-framework-less-flow-based-programming-in-go"><em>previously been published</em></a><em> at bionics.it.</em></p><p>I was so happy the other day to find someone else who found the great benefits of a little pattern for how to structure pipeline-heavy programs in Go, which I described in a <a href="https://blog.gopheracademy.com/composable-pipelines-pattern/">post</a>, and a <a href="https://blog.gopheracademy.com/advent-2015/composable-pipelines-improvements/">follow-up post</a> on GopherAcademy in 2015, and later packaged up in a helper library I’ve called <a href="http://flowbase.org/">FlowBase</a>.</p><p>I have been surprised to not find more people using this kind of pattern, which has been so extremely helpful to us, so I thought to take this opportunity to re-iterate it again, in the hopes that more people might get aware of it.</p><p>The pattern is very pedagogically explained by <a href="https://github.com/eapache">Evan Huus</a> (twitter handle anyone?) in his talk about complex pipeline patterns in Go (also from 2015), <a href="https://youtu.be/2HOO5gIgyMg?t=1855">starting from 30:55</a> and just 1–2 minutes onward. He explains it very neatly in just a few slides, so I urge you to have a look.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F2HOO5gIgyMg%3Fstart%3D1855%26feature%3Doembed%26start%3D1855&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2HOO5gIgyMg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F2HOO5gIgyMg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b02750769ae23b0649690cddcc8b7042/href">https://medium.com/media/b02750769ae23b0649690cddcc8b7042/href</a></iframe><p>One of the main slides summarizing the idea is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*3hs54pljs_ep8chFGBzWkA.png" /><figcaption>Slide by <a href="https://github.com/eapache">Evan Huus</a>, summarizing the idea to package up go-routines in Go structs.</figcaption></figure><p>Btw, this exact idea is also the basis of the <a href="http://scipipe.org/">SciPipe</a> (<a href="https://github.com/scipipe/scipipe">Github</a>) batch workflow / pipeline library, for which we recently published a <a href="https://www.biorxiv.org/content/10.1101/380808v1">technical preprint paper</a>. Take a look if you want!</p><p>If you want a real world use case of how SciPipe can be used, also <a href="https://bionics.it/posts/parsing-drugbank-xml-or-any-large-xml-file-in-streaming-mode-in-go">check out my post about parsing DrugBank XML in streaming mode using SciPipe</a>.</p><p>Samuel (<a href="http://twitter.com/smllmp">@smllmp</a>)</p><p><em>This post has </em><a href="https://bionics.it/posts/structured-go-routines-or-framework-less-flow-based-programming-in-go"><em>previously been published</em></a><em> at bionics.it.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c7ab200e332" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a light-weight Linux-like non-WSL terminal environment on Windows]]></title>
            <link>https://medium.com/@saml/setting-up-a-light-weight-linux-like-non-wsl-terminal-environment-on-windows-59a583c958ed?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/59a583c958ed</guid>
            <category><![CDATA[terminal]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[msy]]></category>
            <category><![CDATA[windows]]></category>
            <category><![CDATA[bash]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Sun, 30 Dec 2018 00:07:58 GMT</pubDate>
            <atom:updated>2019-02-21T06:48:07.818Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up a light-weight Linux-like (non-WSL) terminal environment on Windows</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*IUz0eQDR5g_a-lug.png" /><figcaption>An appetizer about the end result …</figcaption></figure><p>In my new job as a consultant and developer in Machine Learning and AI at <a href="http://www.savanticab.com/en">Savantic AB</a>, although we internally have a “choose your favorite OS” policy, we’re from time to time running into situations where we need to do development on or for Windows. Being a heavy user of Bash on Xubuntu Linux this quickly had me frustrated with how to do things quickly and efficiently on the Windows commandline. The default terminal on windows (aka cmd.exe) is not the nicest thing to have to deal with, and the much more powerful PowerShell is different enough from bash that it will cause some frustration and grief.</p><p>So, what I was looking for was a no-fuss, lightweight, robust and as simple as possible solution to running my normal Bash-based workflow, but inside the main Windows filesystem, interacting with the Windows world. Turns out there are some solutions (hint: look at the thumbnail picture to the right). Read on for more info on that.</p><h3>Windows Subsystem for Linux too heavy</h3><p>First, I must mention the impressive work by Microsoft on the <a href="https://docs.microsoft.com/en-us/windows/wsl/about">Windows Subsystem for Linux (aka. WSL)</a>, which more or less lets you run a full-blown (at least GUI-less, and not even completely GUI-less) distribution of popular Linux distros like Ubuntu and Fedora. WSL is awesome, but also kind of heavy to install, easily taking something closer to an hour to install (or that was my impression). It also has some odd behaviours like opening files in Windows from WSL will give you troubles with line endings (as you might know Windows uses \n\r for line endings while Linux uses just \n). Finally, the default bash terminal for WSL did not even have zoomable text. All in all, this made it clear that this is not the light-weight simple solution I was looking for. WSL is awesome, but too heavy that I will be comfortable to quickly install it on “any Windows machine” I need to spend time working on.</p><h3>Enter Git-Bash and MSYS2</h3><p>Thus, I continued my search. I found Git-Bash, which is part of the <a href="https://git-scm.com/downloads">Git package for Windows</a>. It provides a simple bash environment with git pre-installed and configured, and works extremely well for simple tasks.</p><p>The main drawback I found with Git-Bash was that it does not contain or allow an easy route to install my favorite terminal multiplexer / window manager: <a href="https://en.wikipedia.org/wiki/Tmux">tmux</a>. <a href="https://blog.pjsen.eu/?p=440">One nice blog post</a> mentioned the system that Git-Bash builds upon, and that it sports a package manager which can install a lot more software: MSYS2. Long story short: I tried MSYS2, and now I’m (figuratively) in love. With a few (fixable) caveats I now can run pretty much my full command-line bash+tmux+vim+git(+etc etc) workflow from Xubuntu, IN WINDOWS, right inside the Windows filesystem … and things just work!</p><p>So, for my own and others’ documentation, I below provide documentation of the procedure I have so far followed to enable a comfortable very Linux-like bash environment in Windows.</p><h3>How to set up a nice environment based on MSYS2</h3><ul><li>Install MSYS2 by following the instructions <a href="https://www.msys2.org/">here</a>.</li><li>Start MSYS2 from the start menu in Windows.</li><li>Run upgrades of packages with the pacman packager:pacman -Syu</li><li>Hard restart terminal<br>pacman -Su</li><li>Install packages with<br>pacman -S &lt;package&gt;</li><li>Some packages I installed:<br>vim<br>git<br>tig (The awesome text-mode interface to git)<br>tmux<br>man-db (for manual pages)<br>cgdb</li><li>Install the python 3.7 Windows package from python.org</li><li>Configure git to check out files with Windows style line endings, but check in with Linux style ditto:<br>git config --global core.autocrlf true</li><li>Make python scripts properly print output continuously:<br>alias python=&#39;winpty python.exe&#39;</li></ul><p>For Python development,</p><ul><li>Install the <a href="https://chocolatey.org/">Chocolatey package manager</a>, for installing many open source programs via the commandline, by following <a href="https://chocolatey.org/install">these instructions.</a></li><li>If you run Windows in VirtualBox, make sure to enable bidirectional clipboard in VirtualBox, by right clicking on the virtual machine and choosing: “Settings &gt; General &gt; Advanced” and setting “Shared clipboard” to “bidirectional”.</li></ul><h3>End result</h3><p>If you follow the above instructions roughly, you might end up with an environment like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*IUz0eQDR5g_a-lug.png" /></figure><p>So far, I’m super pleased with this setup.</p><h3>Some caveats</h3><p>I noticed the following caveats:</p><ul><li>For pasting in terminals in Windows:You use Shift + Insert instead of the normal Ctrl + Shift + V from Linux.</li><li>At least for Git-Bash, you need to name you ssh-keys ~/.ssh/id_rsa, not some custom name like ~/.ssh/.id_rsa_gitlab, or it won&#39;t be picked up (haven&#39;t tried with MSYS2 yet). There should be a way to change this, but I haven&#39;t had the time to check it up just yet.</li><li>Some commands, such as dotnet, the commandline tool for <a href="https://dotnet.github.io/">.NET Core</a>, are not cancelleable unless you run them via winpty. So, you&#39;d run: winpty dotnet</li><li>Running commands via winpty on the other hand means that you can’t pipe or redirect the outputs like normal linux commands. To solve that, run winpty -Xallow-non-tty</li><li>For dotnet, I created an alias definition alias dn=&#39;winpty -Xallow-non-tty dotnet&#39; which I put in my ~/.bash_aliases file.</li></ul><p><em>This post has previously been </em><a href="http://bionics.it/posts/linux-like-non-wsl-terminal-env-on-windows"><em>posted at bionics.it</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59a583c958ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linked Data Science — For improved understandability of computer-aided research]]></title>
            <link>https://medium.com/@saml/linked-data-science-for-improved-understandability-of-computer-aided-research-2e70539fc36f?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e70539fc36f</guid>
            <category><![CDATA[linked-data]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[semanticweb]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Tue, 30 Oct 2018 13:20:17 GMT</pubDate>
            <atom:updated>2018-10-30T13:20:17.014Z</atom:updated>
            <content:encoded><![CDATA[<h3>Linked Data Science — For improved understandability of computer-aided research</h3><p><em>Originally </em><a href="http://bionics.it/posts/linked-data-science"><em>posted at bionics.it</em></a><em> — This is an excerpt from the “future outlook” section of my thesis, titled </em><a href="http://uu.diva-portal.org/smash/record.jsf?pid=diva2%3A1242336&amp;dswid=2522"><em>“Reproducible Data Analysis in Drug Discovery with Scientific Workflows and the Semantic Web”</em></a><em> (click for open access full text), which aim to provide various putative ways forward towards improved reproducibility, understandability and verifiability of computer-aided research.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*h9ToBBArhSRdWvfg.png" /></figure><p>Historically, something of a divide has developed between the metadata rich datasets and approaches in the world of Semantic Web/Ontologies/Linked Data, versus in the Big Data field in particular, which has been at least initially mostly focused on large scale unstructured datasets.</p><p>Even today’s focus on data science and development of predictive models using technologies like deep learning, is to a large extent focusing on data sets with a relatively non-complex structure (2D-images), and not so much on merging knowledge stored in opaque deep learning models with explicit knowledge stored in ontologies and knowledge bases of different kinds.</p><p>In order to ripe the fruits of the recent boom in developments of machine learning technologies also in other fields with much more complex and rich data but without large numbers of very similar examples, it seems we will need to link Big Data science with <em>smart</em> data science. That is, state-of-the-art methods and tools from data science, Big Data and machine learning, with semantic data modelling, such as with technologies in Semantic Web, Linked Data and Logic Programming.</p><p>Personally, I think it is particularly the <em>reasoning</em> capabilities around linked/semantic data, that need further development. Based on <a href="http://uu.diva-portal.org/smash/record.jsf?pid=diva2:398839">very positive experience of using SWI-Prolog to implement lookup methods in cheminformatics</a>, I am personally a strong believer in using logic programming approaches as a very practical approach to implement extremely intelligent reasoning systems that can still be easy to understand.</p><p>At the <a href="https://lankadedata.se/LDSV/2018/index.html">Linked Data Sweden symposium</a> in Uppsala in April 2018, <a href="http://bionics.it/posts/semantic-web-data-science-my-talk-at-linked-data-sweden-2018">I presented these ideas</a> and suggested that <a href="http://www.swi-prolog.org/">SWI-Prolog</a> in particular might be a suitable integration platform for what we could call <em>linked data science</em>, or <em>linked big data science</em> (<a href="http://bionics.it/posts/semantic-web-data-science-my-talk-at-linked-data-sweden-2018">slides and video for the presentation here</a>).</p><p><a href="http://www.swi-prolog.org/">SWI-Prolog</a> is a mature, well-maintained and high quality open source Prolog implementation, with an extensive set of libraries, including top-class support for interacting with RDF datasets and SPARQL endpoints. The main drawback of SWI-Prolog for large datasets has been that it has lacked a generic disk based storage solution. Instead, full datasets have had to be loaded into memory before doing complex queries upon them. A recently developed plugin for the <a href="http://rdfhdt.org/">RDF-HDT</a> format seems to provide an interesting way to solve this problem, at least for read-only access to large datasets. Since RDF-HDT is an indexed format, it allows doing queries against its data by only loading the index into memory, not the full dataset. Since SWI-Prolog also can fluently incorporate SPARQL queries into logic programming queries, this also provides an ability to interact with many large-scale datasets such as <a href="https://www.wikidata.org/">Wikidata</a>, not to mention <a href="https://www.semantic-mediawiki.org/">Semantic MediaWiki</a>s with the <a href="https://github.com/rdfio/RDFIO">RDFIO</a> extension that we developed in paper III. In the release notes for RDFIO <a href="https://github.com/rdfio/RDFIO/releases/tag/v3.0.1">version 3.0.1</a> and <a href="https://github.com/rdfio/RDFIO/releases/tag/v3.0.2">3.0.2</a> we provided screenshots demonstrating the ability to query Semantic MediaWiki data via the SPARQL endpoint provided with RDFIO, from SWI-Prolog. SWI-Prolog also provides a web based notebook quite similar to Jupyter notebooks called <a href="https://swish.swi-prolog.org/">SWISH</a>, allowing to use the notebook paradigm for interactive data exploration.</p><p>The described vision is illustrated in the figure above, which illustrates how SWI-Prolog can be used as an interactive integration workbench for accessing and fluently integrating knowledge from external linked datasets. Data sources that can be accessed include SPARQL endpoints (e.g. RDFIO powered Semantic MediaWikis) or RDF-HDT data files. These can be fluently integrated into the logic programming environment, via rules or queries, which can be used as building blocks for further rules or queries. While the SWISH screenshot is here used only for illustration purposes and shows toy example code, the terminal screenshot in the background is a real-world one, showing how SWI-Prolog is interacting with an external SPARQL endpoint of an RDFIO-powered Semantic MediaWiki.</p><p><em>Originally published at </em><a href="http://bionics.it/posts/linked-data-science"><em>bionics.it</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e70539fc36f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is a scientific (batch) workflow?]]></title>
            <link>https://medium.com/@saml/what-is-a-scientific-batch-workflow-a05d195dadd0?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/a05d195dadd0</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[workflow-tool]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Tue, 25 Sep 2018 15:37:11 GMT</pubDate>
            <atom:updated>2026-08-20T12:51:38.407Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This story was originally </em><a href="https://livingsystems.cc/posts/what-is-a-scientific-batch-workflow/"><em>posted on livingsystems.cc</em></a></p><h3>Workflows and DAGs — Confusion about the concepts</h3><p><a href="https://twitter.com/joergenbr">Jörgen Brandt</a> <a href="https://twitter.com/joergenbr/status/907626987333746688">tweeted a comment</a> that got me thinking again on something I’ve pondered a lot lately:</p><blockquote>“A workflow is a DAG.” is really a weak definition. That’s like saying “A love letter is a sequence of characters.” representation ≠ meaning<br>- <a href="https://twitter.com/joergenbr/status/907626987333746688">@joergenbr</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I0wC9Vv601Ai82Hyjec_sA.png" /></figure><p>Jörgen makes a good point. A <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">Directed Acyclic Graph (DAG)</a> does not by any means capture the full semantic content included in a computational workflow. I think <a href="http://www.worldscientific.com/doi/pdf/10.1142/9789814508728_0001">Werner Gitt’s <em>universal information</em> model</a> is highly relevant here, suggesting that information comes in at least five abstraction layers: statistics (signals, number of symbols), syntax (set of symbols, grammar), semantics (meaning), pragmatics (action), apobetics (purpose, result). A DAG seems to cover the syntax and semantics layers, leaving out three layers out of five.</p><p>Anyways, this also points to another fact: A lot of workflow users have a less than solid understanding of what workflows actually are. This leads to further confusion about how different types of workflow tools actually differ. As a small example, in many workflows-as-DAG discussions, there is often not even a common understanding of what the nodes and edges of the DAGs should represent. Are nodes processing components? tasks? task invocations? … or data inputs and outputs? Are the edges data? … or just dependencies?</p><p>It seems the concept of a scientific workflow might be just slightly more complicated than many realize. The ones who seem to truly understand the differences are the ones who tried building a workflow tool themselves (which is one good reason to not always refrain from trying to build your own tool, despite all the “reinventing the wheel” complaints :)).</p><p>Anyways, all this confusion is why I’m writing this post, hoping to clarify a little bit what scientific batch workflow systems are at their core, as far as my experience goes.</p><p>There are many ways to describe workflows, but specifically, I’m trying to define <em>the smallest set of information that is always encoded in workflow definitions, across the most popular shell-based scientific batch workflows used in bioinformatics today</em>.</p><h3>Some context</h3><p>Firstly, just some assumptions. In this context, what I mean with a scientific workflow, or <a href="https://en.wikipedia.org/wiki/Scientific_workflow_system">scientific workflow system or tool</a> is a set of shell commands producing outputs, where the commands are depending on each other in such a way that one command can only start after some other command has finished. This is how a lot of bioinformatics in high-throughput disciplines such as Next-Generation Sequencing is done, at least here at <a href="https://www.scilifelab.se/">scilifelab</a> or <a href="https://pharmb.io/">pharmb.io</a> for example. This definition leaves out a lot of details though, so we will need to dig a little bit deeper below.</p><p>Secondly we also have to distinguish between the <em>workflow definition</em>, and <em>concrete workflow invocations</em> based on this definition. We will in this post mostly talk about the former one.</p><h3>The smallest constituent of a workflow</h3><p>One good way to define something, is generally by identifying its smallest common denominator constituent, so let’s do that.</p><p>Here again I think lies some confusion. When people think of the smallest common constituent in a workflow, it is common to hear suggestions such as the workflow “steps”, “processes”, “components”, “tasks” and similar. I argue that not all of these concepts are actually referring to the smallest common part of workflows.</p><p>I argue that the smallest common constituent of a scientific batch workflow definition is always <strong><em>unique program invocation definitions</em></strong>. That is, a definition specifying a specific program (with the version fixed) coupled with specific input data (e.g. specific files with specific content, and specific parameter values). Note that if anything changes in these constituting parts, even just a single bit of an input file, this will result in creating a <strong>new</strong> unique program invocation definition.</p><p>Unique program invocation definitions are mostly not what is referred to when talking about workflow <strong><em>steps</em></strong>, <strong><em>components</em></strong> or <strong><em>processes</em></strong>. It is, on the other hand, mostly what is meant with the term <strong><em>workflow task</em></strong>. Thus, that is the term we’ll use throughout this post.</p><h3>Building on the smallest constituting part</h3><p>Now that we have defined the smallest constituent of a workflow definition, it is much easier to try to identify various larger constructs that builds upon this concept.</p><p>It turns out that terms such as <strong>workflow step</strong>, <strong>workflow process</strong> and <strong>workflow component</strong>, typically are all variations of the same thing, namely <strong><em>templates for generating unique program invocation definitions. </em></strong>In other words, <strong><em>templates for generating tasks</em></strong><em>.</em></p><p>It turns out that such vastly different approaches to workflow design as <a href="https://www.gnu.org/software/make/">GNU Make</a> and its successors such as <a href="http://snakemake.readthedocs.io/">Snakemake</a>, dataflow-centric tools like <a href="http://nextflow.io/">Nextflow</a> or <a href="http://scipipe.org/">SciPipe</a>, other push-based but non-dataflow-based tools and paradigms like <a href="http://www.cuneiform-lang.org/">Galaxy</a>, <a href="http://pachyderm.io/">Pachyderm</a> or the <a href="http://www.commonwl.org/">Common Workflow Language (CWL)</a>, or tools wrapping commandline execution in a functional language like <a href="http://www.cuneiform-lang.org/">Cuneiform</a>, are all “just” variations on the same thing: A templating mechanism for concrete program invocation definitions, augmented with mechanisms for populating those templates.</p><p>And this last part — the mechanism for populating the templates — is where the inherent differences between all the mentioned tool approaches and concrete tools lie. In make-like tools, workflow steps use interpolation of file names to figure out concrete parameter values and input file names, to populate the task template. In dataflow-centric and other push-based systems, the templates are instead augmented with dataflow connections pointing to the source from which concrete values will be received once produced. In functional programming environments, the inputs to the templates are instead generated from the function call graph generated when calling a nested chain of functions depending on each other.</p><h3>One more ingredient: (Data) Dependencies</h3><p>We have seen that most workflow tools are basically template-filling machinery, built upon different paradigms. In the template-filling machinery, we can identify one more common and re-occurring ingredient though: dependencies. All tools provide some way of specifying how tasks are dependent on other tasks.</p><p>In make-like systems, this is again done through file name patterns: Dependencies are specified via the input file name patterns, that are made concrete by interpolating file name patterns. In dataflow systems, channels between components provide a kind of “template” for dependencies: It provides a static connection which will serve as the mechanism to generate any concrete dependencies on upstream files for specific tasks. The other push-based systems work somewhat similarly to dataflow systems, except with a more restricted model of how the concrete dependencies are generated, as they typically lack <a href="http://bionics.it/posts/dynamic-workflow-scheduling">dynamic scheduling</a> (one component does not typically produce more than one output per set of inputs, for example, as could be the case for a dataflow-component). Finally, in functional languages, the function call graph itself is the dependency definition.</p><p>One caveat to look out for here: Some tools tend to define dependencies only in terms of the finishing of tasks, not on the availability of the outputs of that task, thus leaving out data flow specification from the workflow definition. This is often problematic as it requires task definitions to include internals of upstream tasks, in order to navigate to the right output files. <a href="http://bionics.it/posts/dynamic-workflow-scheduling">I have blogged about it here</a>, but in summary, it is generally recommendable to specify the dependencies in terms of data, not just task finish status.</p><h3>And the sum of all this is?</h3><p>Ok, so let’s conclude with a summary of this little foray into what defines a scientific batch workflow:</p><p>A scientific batch workflow is:</p><ul><li>A set of concrete tasks (program invocation definitions)<br><strong>-</strong> With a specific program and version<br><strong>-</strong> With specific input data (such as files and parameter values)</li><li>A set of dependencies between the inputs and outputs of the aforementioned tasks</li></ul><p><strong>That’s it</strong>, as far as I can see and understand. With this definition in mind, it is — as said — much more interesting to go out and study how different workflow tools are representing this information, and how they make it easier for the user to generate the concrete set of tasks (which can often be a very large number), from some more compact definition.</p><h3>So what about the DAGs?</h3><p>I think the question of how to best represent workflows as DAGs is a bigger topic that needs a separate post. But at least we can make some notes based on the clarifications made here:</p><ul><li>There are multiple ways of representing workflows as DAGs<br><strong>-</strong> Nodes can mean anything between tasks, concrete task invocations (concrete events in time), or some kind of task template<br><strong>-</strong> Edges can mean either data, or just dependency on task finish in general</li><li>The DAGs representing task templates (processes, steps or the like) will be very different from DAGs representing tasks, or even worse, concrete task invocations.<br><strong>-</strong> The DAGs representing tasks or task invocations are less likely to be very intelligible, as they can easily contain thousands of nodes.</li></ul><p>So, if coming back to Jörgen’s <a href="https://twitter.com/joergenbr/status/907626987333746688">tweet</a>, I think it is fair to say that “a workflow is <em>just</em> a DAG” is a bit of a over-simplification. An over-simplification in more than one way, it seems :)</p><p><em>This story was originally </em><a href="https://livingsystems.cc/posts/what-is-a-scientific-batch-workflow/"><em>posted on livingsystems.cc</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a05d195dadd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combining the best of Go, D and Rust?]]></title>
            <link>https://medium.com/@saml/combining-the-best-of-go-d-and-rust-1c6a42e3f4c8?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c6a42e3f4c8</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[sd]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Mon, 09 Jul 2018 11:07:57 GMT</pubDate>
            <atom:updated>2018-07-09T11:07:57.211Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post was originally published in 2016 at </em><a href="http://bionics.it/posts/the-best-of-go-d-and-rust"><em>bionics.it</em></a><em>.</em></p><p>I have for years been looking for a replacement for Python and Java for developments of various data processing tools in bioinformatics / cheminformatics, which happens to be my field of study. That is, something more performant with better concurrency and parallelism than Python but without the ecosystem complexity, JVM startup times and JRE dependency of Java. Something that compiles to static binaries for easy distribution and high performance while being productive enough to use as a replacement for a scripting language.</p><p>I’ve been following the development of <a href="http://dlang.org/">D</a>, <a href="http://golang.org/">Go</a> and <a href="https://www.rust-lang.org/">Rust</a> (and also <a href="http://www.freepascal.org/">FreePascal</a> for <a href="https://github.com/NBISweden/mdc-file-export">some use cases</a>) for some years (<a href="http://saml.rilspace.org/moar-languagez-gc-content-in-python-d-fpc-c-and-c">been into some benchmarking for bioinfo tasks</a>), and now we finally have three (four, with fpc) stable statically compiled languages with some momentum behind them, meaning they all are open source languages, past 1.0.</p><p>While I have went with Go for <a href="http://scipipe.org/">current projects</a>, I still have a hard time “totally falling in love” with any single of these languages. They all fulfill different subsets of my wishlist for an optimal compiled data munging language.</p><p>So, in my backhead I’m now starting to ask: Who will create a language that combines the best parts of each, into the super-language that everybody can use for everything? :)</p><p>Let me start off this with my own wishlist: Let’s have a language with:</p><ul><li>The performance of D and Rust</li><li>The smallness of executables from D and Rust</li><li>The concurrency primitives from Go (channels and lightweight threads multiplexed on cpu cores)</li><li>The simple, pseudo-code-like code (and thus productivity) of D<br>(Compare e.g. this <a href="https://gist.github.com/samuell/5589994">D-code</a> and this <a href="https://gist.github.com/samuell/5591369">Go-code</a>, with <a href="http://saml.rilspace.org/moar-languagez-gc-content-in-python-d-fpc-c-and-c">similar performance</a>)</li><li>The smallness of the language from Go</li><li>The <a href="http://www.cs.loyola.edu/~binkley/papers/icpc09-clouds.pdf">objectively more readable</a> snake_case syntax of Rust</li><li>The order of variable name and type from Go and FreePascal (name first)</li><li>The great tooling from Go (The go and go tool subcommands, etc)</li><li>The great standard library of Go</li><li>The amazing cross-platform GUI builder and IDE of FreePascal: <a href="http://www.lazarus-ide.org/">Lazarus</a> (Not a language feature, but worth a shout-out either way).</li></ul><p>Bonus points if it can also include:</p><ul><li>A powerful REPL, like Scala or Julia, or Python of course.</li><li>Non-indentation based syntax (not fun in some use cases, when wrapping a lot of tools in tools, as happens).</li></ul><p>Can we haz something like this? :) Or is there some existing language that’s already getting close?</p><p><em>This post was originally published in 2016 at </em><a href="http://bionics.it/posts/the-best-of-go-d-and-rust"><em>bionics.it</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c6a42e3f4c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The problem with GNU Make for scientific workflows]]></title>
            <link>https://medium.com/@saml/the-problem-with-gnu-make-for-scientific-workflows-7552f6044a5d?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/7552f6044a5d</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[make]]></category>
            <category><![CDATA[scientific-workflows]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Mon, 09 Jul 2018 10:50:53 GMT</pubDate>
            <atom:updated>2018-07-09T10:54:19.946Z</atom:updated>
            <content:encoded><![CDATA[<p><em>(originally published in 2015 at </em><a href="http://bionics.it/posts/the-problem-with-make-for-scientific-workflows"><em>bionics.it</em></a><em>)</em></p><p>As soon as the topic of scientific workflows is brought up, there are always a few Make fans fervently insisting that the problem of workflows is solved once and for all with <a href="http://www.gnu.org/software/make/">GNU make</a>, written first in the 70’s :)</p><p>Personally I haven’t been so sure. On the one hand, I know the tool solves a lot of problems for many people. Also, there is something very attractive about buildling on a tool that you can be sure will be available on more or less every unix-like operating system, for decades to come.</p><p>On the other hand, the tool seems less suited for tasks where you have a dynamic number of outputs, such as in machine learning cross validation tasks with variable number of folds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*lk4FUPPyZE6EmcmrlDWQKw.png" /><figcaption>Screenshot of an example Makefile</figcaption></figure><p>This is due to the way you specify dependencies: Rather than just specifying the order of the rules <em>by their names</em>, you specify the target <em>file name pattern</em> of the desired output, and let make figure out what rules it takes to create those. Thus when you have a variable or dynamically computed number of outputs anywhere in the workflow, it becomes hard — if not impossible — to know what these resulting file patterns will be. You simply can’t know before you started executing the workflow.</p><p>Because this was just my “intuitive understanding”, I was quite interested when I stumbled upon a <a href="http://www.aclweb.org/anthology/W/W08/W08-0503.pdf">paper that discusses this exact problem in some detail</a>. In section 2.2 the authors explain a number of problems with it for the kind of use cases I have been thinking of.</p><p>They list the problems under three main categories:</p><ul><li>Files are represented by strings</li><li>[Problems with] combinatorial dependencies</li><li>Non-representable dependency structures.</li></ul><p>I will quote some selected parts under each of the categories, below.</p><h3>Files are represented by strings</h3><blockquote>“… from our perspective, make is a declarative language for specifying dependencies. This seems to be exactly what we want, and indeed it does solve some of the problems detailed above. make has several new problems, though, which result in its being not an ideal solution to our problem”</blockquote><blockquote>“… there are three serious problems …”</blockquote><blockquote><em>“The basic problem here is that to make, a file is represented by a string, its filename. For machine learning and natural language processing experiments, it is much more natural to represent a file as a set of key-value pairs. …”</em></blockquote><h3>[Problems with] combinatorial dependencies</h3><blockquote>“The second problem with make is that it is very difficult to specify combinatorial dependencies. If one continued to write the makefile above, one would eventually need to write a final all target to specify all the files which would need to be built. There are 60 such files […] There is no easy way in make of listing these 60 files in a natural manner. One can escape to a shell script, or use GNU make’s foreach function, but both ways are messy.”</blockquote><h3>Non-representable dependency structures</h3><blockquote>“The final problem with make also relates to dependencies. It is more subtle, but it turns out that there are some sorts of dependency structures which cannot be represented in make. Suppose I want to compare the effect of using one of three parsers, one of three part-of-speech-taggers and one of three chunkers for a summarization experiment. This involves three separate three-way distinctions in the makefile, where for each, there are three different commands that might be run. A non-working example is in Figure 3. The problem is that make pattern rules (rules using the % character) can only match the suffix or prefix of a filename. This makefile does not work because it requires the parser, chunker, and tagger to all be the last part of the filename before the type suffix.”</blockquote><p>But by all means, have a look at <a href="http://www.aclweb.org/anthology/W/W08/W08-0503.pdf">the paper</a> for yourself! :)</p><h3>Some comments</h3><p>I find this paper interesting in that it provides concrete use cases to a problem that I had only an intuitive understanding of before.</p><p>It might not completely hit the nail at its head with all the problem description. E.g, I don’t think being “filename based” versus “parameter based” is a problem, if you just have a powerful file name pattern mechanism that allows you to freely and flexible incorporate the parameter names and values at any desired place in the filename pattern. That is the case .e.g in <a href="https://github.com/spotify/luigi">Spotify’s Luigi workflow tool</a>, which we have been using in our work at <a href="https://pharmb.io/">pharmb.io</a>.</p><p>The other two points, about some dependency graph patterns being impractical to work with, seem to touch close to the heart of the problem though. like I mentioned earlier that in Make you have to specify file name pattern of <em>the resulting dataset</em>, to drive the <em>upstream</em> rules/tasks.</p><p>In cases where outputs of some tasks are either variable or when the whole dependency graph is branching into a large number of final targets, then specifying the final target file patterns can be extremely impractical to specify manually.</p><p>For these kind of use cases it seems it would be nicer to just specify the tasks, the indata, and how the data flows from output of one task to input of another one, without needing to know the final resulting file name pattern in before hand. Thus a bit more “push-based” approach compared to “pull-based”, and also with explicit wiring of data dependencies, rather than implicit ones based upon file name pattern rules (That is, more “dataflow” or “flow-based programming” like).</p><p>Interestingly, this approach is taken in tools such as <a href="http://www.nextflow.io/">NextFlow</a> and seemingly to a slighlty more limited extent in tools like <a href="http://bpipe.org/">BPipe</a>. (EDIT: And definitely in our new Go-based library <a href="http://scipipe.org/">SciPipe</a>).</p><p>It should be mentioned though that python-based <a href="https://bitbucket.org/johanneskoester/snakemake">SnakeMake</a> provides some constructs that makes some of the problems described above a lot more tractable to implement even though the basic execution mechanism is “pull-based” just like in make itself (you specify the desired target file pattern to drive upstream tasks) but it has a much better handling of named inputs and outputs than GNU make, and also a more powerful file pattern interpolation mechanism, where you can use python code to interpolate the file pattern in any way.</p><p>Let’s hope to see more dataflow inspired workflow tools in the future, but in the meanwhile, keep an eye on NextFlow, BPipe, and (also SnakeMake if you can live with its pull-based mechanism)!</p><p>// <a href="http://twitter.com/smllmp">Samuel</a></p><p><strong><em>UPDATE:</em></strong><em> I should clarify that the problems discussed below do not apply to some ways of working with make such as this interesting method by </em><a href="https://twitter.com/yokofakun"><em>Pierre Lindenbaum</em></a><em>, where the makefile is </em><a href="https://github.com/lindenb/ngsxml"><em>generated from an XML description</em></a><em>, thus using make as a kind of “assembly language”, to which higher level workflow definitions are “compiled”. Pierre </em><a href="https://twitter.com/yokofakun/status/576878752673693696"><em>mentions a few other problems with make here though</em></a><em>.</em></p><p><em>(originally published in 2015 at </em><a href="http://bionics.it/posts/the-problem-with-make-for-scientific-workflows"><em>bionics.it</em></a><em>)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7552f6044a5d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Launching kubernetes jobs from the Go API — notes from a beginner]]></title>
            <link>https://medium.com/@saml/launching-kubernetes-jobs-from-the-go-api-notes-from-a-beginner-2b34fbc502c0?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b34fbc502c0</guid>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[hpc]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Tue, 14 Feb 2017 22:42:21 GMT</pubDate>
            <atom:updated>2017-02-14T22:57:27.266Z</atom:updated>
            <content:encoded><![CDATA[<p>My current work at <a href="http://pharmb.io/">pharmb.io</a> entails adding <a href="http://kubernetes.io/">kubernetes</a> support to my light-weight Go-based scientific workflow engine, <a href="https://github.com/scipipe/scipipe">scipipe</a> (kubernetes, or <em>k8s</em> for short, is Google’s open source project for orchestrating container based compute clusters), which should take scipipe from a simple “run it on your laptop” workflow system with HPC support still in the work, to something that can power scientific workflows on any set of networked computers that can run kubernetes, which is quite a few (AWS, GCE, Azure, your Raspberry Phi cluster etc etc).</p><p>As scipipe is itself written in Go it seemed natural to go with the officially supported <a href="https://github.com/kubernetes/client-go">programmatic Go API</a> to kubernetes. This turned out to be less than a smooth ride though, at least to start with. Thus, I thought it worthwhile to jot down a few of the observations I did as a complete newcomer to kubernetes and this style of writing programming APIs, in the hopes that it might help some other newcomer, and maybe give kubernetes developers a few hints on where to focus efforts in documenting the API.</p><h3>Documentation — or not</h3><p>The first thing I noted, when looking for documentation for the programmatic access to the API, is that there wasn’t much, except for <a href="https://kubernetes.io/docs/user-guide/accessing-the-cluster/#programmatic-access-to-the-api">a minimal section in the user guide</a>, and <a href="https://github.com/kubernetes/client-go#how-to-use-it">two possibly even more minimal code examples</a>. None of those showed specifically how to work with the (batch) jobs API though, so I was still left in the dark in that regard.</p><p>I also, somewhere down the line found the <a href="https://github.com/kubernetes/client-go/blob/master/pkg/apis/batch/types.go">code file where most of the API types for the batch job API were defined</a>. But since the types were all spread out, and depending on each other in very complex ways, without a clear starting point, it was not too helpful for a newcomer.</p><p>A few guiding words on the principles, or how to find stuff, would have helped tons here.</p><p>Well, finally, by surfing around lots and lots, I stumbled upon the <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md">API conventions document</a>, which in fact gives a few hints about the thinking behind the API structure. That helped to get some kind of idea of the thinking behind the API (link this document more prominently maybe?).</p><p>And, what finally made me get forward, was when I found an example of a 3rd party library that created kubernetes jobs, <a href="https://github.com/pachyderm/pachyderm/blob/805e63/src/server/pps/server/api_server.go#L480">in the code</a> of <a href="http://pachyderm.io/">Pachyderm</a> (Pachyderm is cool cool stuff, but more on that in another post perhaps).</p><p>I was still heavy scared by stuff like these import statements:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*s0e2L-mE2aC4NwB5HnLdzA.png" /></figure><p>Heeya! … I mean, which one do I even start with, to look for … some kind of starting point or something?</p><h3>When it clicked</h3><p>Well, following the pachyderm code example, and working backwards, jumping to the definitions of each type or function returning a type, a pattern finally started to emerge.</p><p>So, it turns out, that complex struct types such as Job, JobSpec, ImageSpec etc, are simply just complex structures of structs of structs of structs … which could essentially be defined as a large JSON structure, but by being a structure of hierarchical types, one gets automatic type safety already at the compilation level … and also it is not that hard to find the right type to use for a field: The type of the value that will populate a field, is typically named the same as the field name itself, so one can typically just assume that, and start typing SomeFieldName: api.SomeFieldType... in a Go editor with good autocompletion support (such as <a href="http://code.visualstudio.com/">VSCode</a> with Luke Hoban’s <a href="https://github.com/Microsoft/vscode-go">great great Go plugin</a>, or Vim with Fatih’s <a href="https://github.com/fatih/vim-go">awesome Vim-Go plugin</a>), and you will find it (see <a href="https://github.com/samuell/k8s-experiments/blob/0ada4a53c9e19d29f129960b5d278afe99cdb67f/k8sexp.go#L43-L90">my working code example</a> mentioned at the very end of the post, for a look at how this looks in reality).</p><p>Finally, theunversioned and batch API files seems to be used only in a few special cases, so the main api endpoint (k8s.io/kubernetes/pkg/api) seems to be where to look most of the time.</p><p>All in all, after I get the idea behind this, with the hierarchy of typed structs. It all kind of starts to make sense, as you get the safety and all. And most importantly, I finally managed to <a href="https://github.com/samuell/k8s-experiments/blob/0ada4a53c9e19d29f129960b5d278afe99cdb67f/k8sexp.go#L43-L90">start my first kubernetes job via the Go API</a>! :) Now it just remains to <a href="https://github.com/scipipe/scipipe/issues/32">get it integrated in scipipe</a> soon enough.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b34fbc502c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How time-boxing and a unified trello board solved my productivity problems]]></title>
            <link>https://medium.com/@saml/i-used-to-have-a-very-hard-time-getting-an-overview-of-my-current-work-and-prioritizing-and-4d65917a1f5f?source=rss-1f14ac874b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d65917a1f5f</guid>
            <category><![CDATA[trello]]></category>
            <category><![CDATA[concentration]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Samuel Lampa]]></dc:creator>
            <pubDate>Fri, 26 Feb 2016 12:36:07 GMT</pubDate>
            <atom:updated>2018-04-11T18:19:31.987Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YVwA6lIKY8SP8Fm1Yf_BGA.png" /><figcaption><em>Sketchy screenshot of how my current board looks. Notice especially the “Now” stack, marked in yellow, where you are only allowed to put </em><strong><em>one single</em></strong><em> card.</em></figcaption></figure><p>I used to have a very hard time getting an overview of my current work, and prioritizing and concentrating on any single task for too long. I always felt there might be something else that might be more important than what I were currently doing. And in fact, how would I know if I didn’t have the overview?</p><p>Juggling two part-time jobs, each with multiple parallel on-going projects, as well as many hobby interests in similar areas, didn’t exactly help.</p><p>I had tried organizing all my projects into separate <a href="https://trello.com/">trello-boards</a>, with detailed tasks and check-lists and the like, but that didn’t work.</p><p>All of this changed after I read <a href="https://pragprog.com/book/snfocus/pomodoro-technique-illustrated">“Pomodoro Technique Illustrated”, by Staffan Nöteberg</a>. The book revolves around one simple idea: Divide all your work into 25 minute work chunks and continuously during the day prioritize and select one work chunk at a time and work on it 100% exclusively for 25 minutes (you will set a timer that rings after that time … I just use the timer function in my smartphone). After that you allow yourself a few minutes to goof out, check mail, facebook, hackernews or whatever urges your attention … and importantly, select the next work chunk to work on exclusively for 25 minutes.</p><p>I find the whole idea to be brilliant. Knowing that you will be able to goof out a bit after the 25 minutes enables you to suppress any urges to check up various things that hit your mind, such as social media and other channels.</p><p>Nöteberg’s book also goes through practical tips on how to constantly plan and re-prioritize your current batch of work items for a few selected time periods, such as “today”, and “this week” etc.</p><p>Anyhow, these ideas inspired me to create a much better way to organize my trello board as well, which I’m describing below.</p><p>Instead of having separate boards for each high level project, I created one single board on which I keep all my work tasks. I simply call the board “time-boxing” (another common term for the “pomodoro technique”).</p><p>In this board, inspired by Nöteberg’s book, I have created the following stacks:</p><ul><li>On hold</li><li>To do this week</li><li>To do Today</li><li>To do Now</li><li>Done [Month name] (Archived monthly)</li></ul><p>(I use labels to mark which of my part-time jobs a task belongs to … which is sometimes more than one).</p><p>This new structure has done wonders for me! Slightly surprising, the most important stack of these were in fact the “Doing right now” stack in which, you are only allowed to have <strong><em>one single</em></strong> card! (The yellow one in the screenshot above).</p><p>I hadn’t realized the power of this. The “right now” stack is now what really helps me remember what I’m currently doing, even after goofing out those 5 minutes and getting dozens of other threads of thought starting in my head, which would normally make me lose track of even what I was currently on. Now I can just glance on that card and immediately be on track with what I was supposed to do for the next 25 minute chunk of concentrated work.</p><p>In summary I can highly recommend the time-boxing technique for anyone having problems concentrating amidst too many current projects and tasks. If you want some more reading on the topic before staring, <a href="https://pragprog.com/book/snfocus/pomodoro-technique-illustrated">Nöteberg’s book</a> is a good resource.</p><p><em>Full disclosure: I don’t know Staffan Nöteberg, and I’m not paid to write this.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d65917a1f5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>