<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Greg Dingle on Medium]]></title>
        <description><![CDATA[Stories by Greg Dingle on Medium]]></description>
        <link>https://medium.com/@gregdingle?source=rss-2b97954b9e73------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VaTd55Rkt7XDmF3GxfRyXg.jpeg</url>
            <title>Stories by Greg Dingle on Medium</title>
            <link>https://medium.com/@gregdingle?source=rss-2b97954b9e73------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:40:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gregdingle/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to make fast, reliable APIs out of slow, unreliable websites]]></title>
            <link>https://medium.com/czi-technology/how-to-make-fast-reliable-apis-out-of-slow-unreliable-websites-edf2be94358d?source=rss-2b97954b9e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/edf2be94358d</guid>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[crispr]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Greg Dingle]]></dc:creator>
            <pubDate>Wed, 13 Feb 2019 20:52:57 GMT</pubDate>
            <atom:updated>2019-02-15T18:55:26.413Z</atom:updated>
            <content:encoded><![CDATA[<p>For a recent project, we had to automate a workflow out of several different bioinformatics web sites that were never intended to be automated. This posed some interesting problems that we managed to solve quite nicely in the end. I’ll detail the problems and solutions here so that if you’re ever in a similar situation, you can avoid repeating our efforts.</p><h3>Background</h3><p>The finished product was to be a web tool that scientists in the <a href="https://www.czbiohub.org/">Chan Zuckerberg Biohub</a> and elsewhere could use to design and analyze CRISPR edits in batch (now live at <a href="https://crispycrunch.czbiohub.org/">crispycrunch.czbiohub.org</a>). With rapid advances in the field of genome engineering, scientists are devising ever-larger experiments that require automation to be practical. At the Biohub, <a href="https://www.czbiohub.org/people3/cz-biohub-staff/#l-start">Manuel Leonetti’s</a> team is inserting fluorescent proteins into all 22,000 protein-coding genes in the human genome. Similarly, <a href="https://directory.ucsf.edu/people/search/id/147363">Ryan Leenay</a> edited 1,521 different genome locations in order to train a model that predicts edit outcomes in immune system T-cells. In order to do experiments of this scale, the scientists needed to automate their manual workflows. Part of CZI’s mission is to apply modern engineering and product methods to accelerate biological research.</p><h3>Software architecture</h3><p>The scientists we worked with were already using two key web tools in their CRISPR work: <a href="http://crispor.tefor.net/">Crispor</a> and <a href="http://crispresso.rocks/">Crispresso</a>. A scientist would design a CRISPR edit in Crispor, run their experiment in the wet lab, then analyze the results with Crispresso. Our software, named <a href="http://crispycrunch.czbiohub.org/">CrispyCrunch</a> (notice a theme here?), unites the two functions of design and analysis, and it enables batch processing — up to 96 edits in one experiment, the standard size of a cell culture plate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*bMH0st1aoI1jYiMIHBGhbA.png" /><figcaption>How CrispyCrunch depends on outside tools</figcaption></figure><p>We decided to build CrispyCrunch on top of the existing tools of Crispor and Crispresso because they were already trusted by our scientists, we wanted to have a working prototype ASAP, and we could replace them with our own components later. The risk was in integration. These tools were not designed to be part of a larger system.</p><h3>Asking permission</h3><p>As preface to all the technical hacks below, it’s worth saying that we asked the owners of Crispor and Crispresso early on whether they would be okay with us treating their sites as APIs. While we didn’t expect them to do any work for us, we hoped they would not block CrispyCrunch at the first sign of trouble. Gladly <a href="http://pinellolab.org/">Luca</a> and <a href="https://hgwdev.gi.ucsc.edu/~max/">Max</a> agreed.</p><h3>Problem #1: How to make an API out of a website</h3><p>This is the classic problem of web scraping which nearly every developer runs into sooner or later. The problem is not so much getting the data out — this is easy with libraries such as <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup</a>, which extract data from web pages by CSS selectors — but in making the results reliable and errors meaningful.<br> <br> To that end, we employed the following techniques:</p><ul><li>Use HTML selectors that are <strong>least likely to change</strong>. For example, prefer HTML IDs over element types.</li><li>Select the data you want from the DOM <strong>directly by classnames or IDs</strong>. Avoid selecting child or parent elements which may easily change location in the document hierarchy.</li><li>When POSTing, <strong>don’t rely on default values</strong>, which may change. Provide explicit values for all fields.</li><li><strong>Learn all</strong> possible types of <strong>errors</strong> by systematically varying GET and POST params.</li><li>Promote string errors to <strong>exceptions</strong>. A HTTP request may return 200 OK while the web page text contains a blatant “ERROR!!!” .</li><li>Use <strong>assertions</strong> to codify assumptions. For example, assert that some HTML list has a non-zero number of elements.</li><li>Expect the unexpected. <strong>Log</strong> copiously and <strong>get notified</strong> of exceptions by email or by a service such as <a href="https://sentry.io/">Sentry</a>.</li><li><strong>Retry</strong> to workaround intermittent failures. Use <strong>backoff</strong> as explained below.</li></ul><h3>Problem #2: How to increase an API client’s throughput</h3><p>Having got the base functionality we needed, the next problem was to make it fast. The obvious solution was to send independent requests in parallel. <br> <br> Using python and the ubiquitous <a href="http://docs.python-requests.org/en/master/">requests</a> library, you could do something as simple as this:</p><pre>import requests<br>from concurrent.futures import ThreadPoolExecutor</pre><pre>paths = [<br>    &#39;/view_report/mNGplate1&#39;,<br>    &#39;/view_report/mNGplate2&#39;,<br>    &#39;/view_report/mNGplate3&#39;,<br>    # ...more<br>]</pre><pre>def _request(path):<br>    return requests.get(&#39;http://crispresso.rocks&#39; + path)</pre><pre>with ThreadPoolExecutor() as pool:<br>    results = pool.map(_request, paths)</pre><p><em>Note:</em> <a href="https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor">ThreadPoolExecutor</a> is typically more efficient than <a href="https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor">ProcessPoolExecutor</a> for web requests because a thread uses less memory than a process, and the <a href="https://wiki.python.org/moin/GlobalInterpreterLock">python GIL</a> is not relevant when blocking on I/O.</p><h3>Problem #3: How to make an API client resume a partially succeeded batch</h3><p>When a batch of requests takes more than a few seconds, it’s desirable to be able to restart without losing progress so far. This is desirable in several common situations:</p><ul><li>Intermittent failure</li><li>Debugging</li><li>The user needs to go back and change something</li></ul><p>The solution here again is simple: cache HTTP responses so that a request for a previously downloaded resource returns a cached copy. When restarting a batch, only incomplete requests will be re-executed. <br> <br> For caching, we used the handy <a href="https://github.com/reclosedev/requests-cache">requests_cache</a> module. Some tips in its usage:</p><ul><li>Unlike the tutorial, construct an explicit <strong>cache session object</strong> to avoid side-effects.</li><li>You should <strong>explicitly delete</strong> the cache for a URL if you determine that the content is somehow invalid, so you can try again with a fresh request. In other words, errors that are not HTTP errors need be handled specially.</li><li>requests_cache will <strong>not</strong> cache POST requests that include <strong>files</strong>, because the file boundary will be a <strong>random</strong> string. You can fix that by monkey-patching <a href="https://urllib3.readthedocs.io/en/latest/">urllib3</a>. (This took a while to figure out!)</li></ul><pre>urllib3.filepost.choose_boundary = lambda: ‘crispycrunch_super_special_form_boundary’</pre><ul><li>The <strong>remote server</strong> may cache responses–sometimes in a way you don’t want. For example, one of the APIs we used cached error responses 🙁 The fix was to append a random string to the URL to make it always new from the server point of view, and to tell requests_cache to ignore the random param, so the URL is the same from the client point of view.</li></ul><h3>Problem #4: How to make an API client testable in isolation</h3><p>If you have complex response parsing code, you should have unit tests for it that you can run isolated from the internet. The problem is that constructing test data can be time-consuming and awkward. HTML pages can be hundreds of KBs in <a href="https://www.keycdn.com/support/the-growth-of-web-page-size">size</a>.<br> <br> Caching comes to the rescue again. You can use <a href="https://github.com/reclosedev/requests-cache">requests_cache</a> to store a full response in <a href="https://en.wikipedia.org/wiki/SQLite">SQLite</a>, then commit the SQLite file to version control. Instant test isolation! Vary the name of the file in test mode, and add it as an exception to your gitignore file (or equivalent). <em>Note</em>: you can verify that a request came from the cache with the from_cache attribute.</p><h3>Problem #5: How to make your client not crash the API</h3><p>Even official web APIs may be brought to their knees by surprisingly little traffic. The only solution to this problem is to <a href="https://en.wikipedia.org/wiki/Stress_testing_(software)">stress test</a> the API so that you discover traffic limits before your users do and rate limit accordingly–see #7 below. <em>Note</em>: you may want to alert the API owner before you do so!<br> <br> For stress testing, you can use <a href="https://httpd.apache.org/docs/2.4/programs/ab.html">Apache Bench</a> (ab) command line tool. <em>Note</em>: servers themselves may cache in various ways so be careful to make your load test realistic.</p><h3>Problem #6: How to avoid race conditions in API calls</h3><p>A nasty problem we discovered when chaining API calls was intermittent 404s. This is symptomatic of a <a href="https://en.wikipedia.org/wiki/Race_condition">race condition</a> in which a resource is not ready although preceding <a href="https://en.wikipedia.org/wiki/List_of_HTTP_status_codes">HTTP response codes</a> imply that it is. Such race conditions may not be revealed on a website until is used in a programmatic way because a human user cannot act as fast as a computer client. <br> <br> The solution to race conditions in a system you don’t control is either:</p><ul><li>Introduce an <strong>artificial delay</strong> such as</li></ul><pre>import time; time.sleep(1)</pre><ul><li><strong>Block</strong> on the resource becoming available</li></ul><pre>import requests</pre><pre>def _request(path):<br> return requests.get(‘http://crispresso.rocks&#39; + path)</pre><pre>while _request(path).status == 404:<br> print(f’Waiting on {path}’)<br> time.sleep(1)</pre><p><em>Note</em>: if you are caching responses as described above, you won’t pay the round-trip cost of an extra request in the case of no waiting.</p><h3>Problem #7: How to keep just under API rate limits</h3><p>Some APIs might be nice enough to report when you are stressing them too much. If you want to optimize within the limit, you can use a client-side rate limiter such as <a href="https://github.com/tomasbasham/ratelimit">ratelimit</a>. Combined with <a href="https://en.wikipedia.org/wiki/Exponential_backoff">exponential backoff</a>, you can have a respectful and performant client.</p><pre>import requests<br>from ratelimit import limits, RateLimitException<br>from backoff import on_exception, expo</pre><pre>@on_exception(expo, RateLimitException, max_tries=8)<br>@limits(calls=15, period=900) # 15min<br>def call_api(url):<br>    return requests.get(url)</pre><p>Match calls and period to the observed or stated rate limit of the API.</p><h3>Conclusion</h3><p>These are the problems we had to solve to make <a href="https://crispycrunch.czbiohub.org/">CrispyCrunch</a> work smoothly with the <a href="http://crispor.tefor.net/">Crispor</a> and <a href="http://crispresso.rocks/">Crispresso</a> web tools which were never intended to be web services. We hope you find the details helpful. If you’d like to learn more about CZI Engineering, check out <a href="https://chanzuckerberg.com/technology/">our Technology page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edf2be94358d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/czi-technology/how-to-make-fast-reliable-apis-out-of-slow-unreliable-websites-edf2be94358d">How to make fast, reliable APIs out of slow, unreliable websites</a> was originally published in <a href="https://medium.com/czi-technology">CZI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>