<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nathan Sutton on Medium]]></title>
        <description><![CDATA[Stories by Nathan Sutton on Medium]]></description>
        <link>https://medium.com/@nasutton?source=rss-cffc89afabd0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*rXYSciPSpAHfWa0S</url>
            <title>Stories by Nathan Sutton on Medium</title>
            <link>https://medium.com/@nasutton?source=rss-cffc89afabd0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:56:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nasutton/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nine coding harnesses vs. your laptop]]></title>
            <link>https://nasutton.medium.com/nine-coding-harnesses-vs-your-laptop-8b7bf5e8fbae?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b7bf5e8fbae</guid>
            <category><![CDATA[mlx]]></category>
            <category><![CDATA[coding-agents]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 21:11:06 GMT</pubDate>
            <atom:updated>2026-09-13T21:11:06.001Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have ever considered swapping out the API calls your coding harness is making for a local model it probably ended in disappointment. You ran a quick llama-bench and thought you were going to see X tokens/second. What that benchmark doesn’t tell you is how it actually feels to use these harnesses. The development experience is extremely variable. You might have sat there for minutes before seeing any response. Maybe you were getting somewhere and halfway through it started stalling. ‘Who is this prefill, taking all my time?’ It’s not your fault: most coding harnesses weren’t built with a local model in mind.</p><p>This is not a hit job on the excellent work of other teams. I’m trying to reliably measure what happens when you swap out the data center for localhost. A certain level of wrong tool for the problem is to be expected. Please add salt: I have been tinkering with <a href="https://github.com/nathansutton/chad">chad</a>, a coding harness optimized specifically for Qwen 3.8 27B on Apple silicon.</p><h4>Laptop Physics</h4><p>Most harnesses conspire against localhost in three ways.</p><p><strong>Large system prompts &amp; tool schemas.</strong> Let’s assume your laptop reads at 90 tokens per second and writes about 10. These represent the prefill &amp; generation parts <em>of</em> every loop in your coding harness. At these speeds every 1,000 tokens of reading a prompt translates to ~ 11 seconds of staring at the cursor before the model begins to write. Before your LLM does any work it will read the entire system prompt along with any loaded tool schemas. In the pi harness this combination was 2,008 tokens for Qwen 3.8 27B, but 18,046 all in for Opencode. You won’t notice much difference when you have a data center GPU with prefill rates averaging 10k+ tokens/second. This collapses to 0.2 versus 1.8 seconds. On my laptop that was 22 seconds for pi versus 226 for Opencode. Unbearable!</p><p><strong>Smaller context windows.</strong> After your LLM finishes the system prompt you have a finite context window left in memory to do work. It is smaller than you think, and your harness just spent part of it. How much context you have depends on how much memory you are starting with vs. how big the model’s weights are. There are a lot of variables here, but 32,000 tokens is a reasonable guess at how much room you’ll have left with a reasonably good model on a reasonably good laptop. How much of that 32,000 token budget is left for the pi harness? 94% — seems manageable. Opencode? With 18,046 tokens out the door already only 44% of your context is left for actually doing work.</p><p><strong>A habit of side requests.</strong> In the traditional local client / remote server pattern the harness can make as many side requests to the data center servers as it likes. Your laptop is both the client and the server. In the best case those side requests cause the local model to queue and wait. In the worst case they cause repeated long prefills. Over 24 tasks opencode fired 33 of them, crush 51 and dsh 24 (session titles and summaries), almost every one overlapping an agent turn. The model was “busy” 125% and 114% of wall clock for opencode and crush: two requests in flight on one GPU.</p><h3>Harness Outcomes</h3><p>I put 9 harnesses through a series of 8 Exercism exercises, each in its own auto-approve mode, with one identical one-sentence prompt. Each task leveraged the same M4 MacBook Pro (24GB, macOS 26.6.2) with a 3 bit quant of <a href="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF">Qwen 3.8 27B</a> model served via llama.cpp (build 10470). The same llama-server was shared by every harness and a common proxy enforced the same recommended sampling regime for Qwen (temperature=1.0, top_k=20, top_p=0.95, min_p=0.05). Every session had the same 32,768 unified cache served across four slots. Every number reported below is llama-server’s own accounting as read through the proxy, never a harness self-report, except the two rows marked * (chad on its in-process MLX engine, where there is no server to observe, so they come from chad’s own prefill trace with the same definitions).</p><ul><li><strong>wait before 1st token</strong> is the server’s prefill time for the system prompt, tool schemas, and the first request.</li><li><strong>wait / later turn</strong> is the same pause on every turn after the first, side requests excluded.</li><li><strong>cache reuse</strong> is the share of each later turn served from the prefix cache.</li><li><strong>experienced tokens/second</strong> is generated tokens divided by wall clock for the whole task, prefill and tool time included. The same harness varied by up to 50% on this column from one rep to the next (n=3), so small differences between the lean harnesses are noise.</li><li><strong>pass</strong> is a gate for the Exercism task. Please do not interpret as a ranking, as these are trivial python tasks. T counts timeouts at the 1,200 second cap.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23364e72b2ab467991575136d7088f46/href">https://medium.com/media/23364e72b2ab467991575136d7088f46/href</a></iframe><p>In my experience the harnesses fell into three groups.</p><p><strong>Lean and stable</strong> (pi, mini-swe-agent, chad). These harnesses have a trim system prompt and 96–99% cache reuse. They largely work out of the box with either a data center or local model backend. The only caveat here is that mini-swe-agent only passed 11/24 of the Exercism tasks and had the most timeouts.</p><p><strong>Heavy but disciplined</strong> (dsh, cline, codex, goose). The combined effect of either long system prompts or a high tool schema count. Their prefix is byte stable so once you get started it feels tolerable. (goose earned this group at 1.50.0: earlier releases re-rendered a minute-resolution timestamp into the first user message every turn. This tanked cache reuse to 78%)</p><p><strong>Heavy to start</strong> (crush, opencode). You’ll wait 3–4 minutes before you see any activity.</p><h3>The case for chad</h3><p>None of these harnesses are badly engineered. Opencode’s 18k system prompt is there because it helps frontier models deployed behind an API. The 26 tool schemas in crush are fine when you have 200k context and prefill is instantaneous. All of these choices were made in an environment where the prefill is virtually free. At local speeds this all falls apart.</p><p>chad is minimal on purpose. There are only <a href="https://github.com/nathansutton/chad/blob/main/docs/design.md#why-the-tool-surface-is-five-tools">5 tools available</a>, all shapes the model already learned during pretraining. A disk check-pointed system prompt, persistent prefix cache, and a fitted drafter are all extensions of this scarcity mindset. This discipline alone doesn’t explain the gap you’d actually feel. With the standard client/server model the chad/llama.cpp interface isn’t even the fastest of the lean three. You could probably write most of the loop in an afternoon.</p><p>chad’s real number shows when you have the liberty to couple the server and the agent loop in the same process. The harness now owns the cache. With the same harness and same prompts, the in-process MLX engine experienced 7.9 → 17.4 tokens/second (or 12.4 tokens/second with the drafter off). The only advantage of the MLX process was decode and not chad’s request pattern: the wait per later turn is the same one second on either engine. Paired against its own control inside generation over three more nights, the DFlash2 drafter reads 23.3 vs. 15.9 tokens/second, 1.47x pooled and a win on every one of the eight tasks.</p><h3>Reproduce it</h3><p>Everything runs from a checkout of <a href="https://github.com/nathansutton/chad">chad</a>: the runner, forcing proxy, and scorecard for the eight tasks are under <a href="https://github.com/nathansutton/chad/tree/main/benchmarks/matrix">benchmarks/matrix</a>. Each arms raw rows (grid.json, one turns.jsonl row per request, the sampler audit, provenance) are committed beside the pooled tables with the intention that so every number above traces to a row.</p><pre>uv run python benchmarks/matrix/run.py setup|smoke|llama|mlx|table</pre><p>Thanks to the llama.cpp maintainers for a server-side prefix cache and a timings object good enough to build the whole instrument on, and to z-lab for DFlash2.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b7bf5e8fbae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A new generative model for radiology]]></title>
            <link>https://nasutton.medium.com/a-new-generative-model-for-radiology-b687a993cbb?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b687a993cbb</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[radiology]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Thu, 02 Feb 2023 21:09:38 GMT</pubDate>
            <atom:updated>2023-02-03T21:33:14.615Z</atom:updated>
            <content:encoded><![CDATA[<h4>you should include text inputs along with images</h4><p>Machine learning in radiology has come a long way. For a long time the goal was simply to make a probability estimate of different conditions available to the radiologist at the time of interpretation. As evidence, see any of the hundreds of AI vendors that have commercialized computer vision algorithms. On the academic frontier, recent advances have made it possible to generate realistic sounding radiology reports directly from an image. The first paper I found describing such a model was from 2017, but there have been many more recently with the onset of transformers.</p><p>However, every example architecture I have found suffers from the same structural problem.</p><p><strong>They aren’t answering a clinical question.</strong></p><p>When a provider orders an imaging study the ordering provider is asking the radiologist to combine their education and experience to answer a clinical question. Unlike most specialist consultations, the format is asynchronous. Occasional telephone or face-to-face consults do occur for time-sensitive findings, but this is not the norm. Instead, the radiology report is the primary product of this provider-radiologist specialist consultation. This document’s purpose to answer the clinical question, or indication. It usually has a succinct ‘impression’ section at the end. For example…</p><blockquote>INDICATION: 36yo M with hypoxia // ?pna, aspiration.</blockquote><blockquote>FINDINGS: PA and lateral views of the chest provided. The lungs are adequately aerated. There is a focal consolidation at the left lung base adjacent to the lateral hemidiaphragm. There is mild vascular engorgement. There is bilateral apical pleural thickening. The cardiomediastinal silhouette is remarkable for aortic arch calcifications. The heart is top normal in size.</blockquote><blockquote>IMPRESSION: Focal consolidation at the left lung base, possibly representing aspiration or pneumonia. Central vascular engorgement.</blockquote><h3>Incorrect Architectures</h3><p>Medical images alone are not sufficient to answer why an imaging study was ordered by a provider. For example, a provider might order a chest x-ray because their patient is presenting with shortness of breath and they suspect pneumonia. In another case, they might suspect a fracture after a motor vehicle collision and order a chest x-ray to rule out a broken rib. The same image could answer either of these clinical questions.</p><p>In my search I found over ten studies since 2017 describing model architectures that could conditionally generate entire radiology reports from an image (<a href="https://arxiv.org/abs/1711.08195">Jing et al. 2017</a>, <a href="https://proceedings.neurips.cc/paper/2018/hash/e07413354875be01a996dc560274708e-Abstract.html">Li et al. 2018</a>, <a href="https://link.springer.com/chapter/10.1007/978-3-030-00928-1_52">Xue et al. 2018</a>, <a href="https://arxiv.org/abs/2102.09777">Singh et al. 2019</a>, <a href="https://arxiv.org/pdf/1907.09085.pdf">Yuan et al. 2019</a>, <a href="https://arxiv.org/abs/2010.16056">Chen et al. 202</a>0, <a href="https://arxiv.org/abs/2010.10042">Miura et al. 2020</a>, <a href="https://arxiv.org/abs/2106.06963">Fenglin et al. 2021</a>, <a href="https://arxiv.org/abs/2102.09777">Nooralahzadeh et al. 2021</a>, <a href="https://scholar.google.com/scholar?hl=en&amp;as_sdt=0%2C34&amp;q=Attention+based+automated+radiology+report+generation+using+CNN+and+LSTM+%28&amp;btnG=">Sirshar et al. 2022</a>, <a href="https://arxiv.org/abs/2204.13258">Chen et al. 2022</a>, <a href="https://doi.org/10.1016/j.media.2022.102510">Yang et al. 2022</a>). Unfortunately, none had text inputs. The one exception was a recent paper included the full-text radiology report as a textual input alongside the image, but its goal was to remove hallucinated references by cleaning up the data used for model training (<a href="https://proceedings.mlr.press/v193/ramesh22a.html">Ramesh et al. 2022</a>). These oversights are a problem because all the variation in the generated reports will come from the images. They cannot answer a clinical question posed in text by the ordering provider.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*D61_OwgUwtXl7H0j5my6ag.png" /><figcaption>Look at the architecture diagrams of any of the listed studies and you’ll find variations of the same pattern. Only visual features are used to generate the radiology reports. Image by author.</figcaption></figure><h3>Better Architectures</h3><p>To realistically describe what a radiologist is doing when they write a report a deep learning model needs to accept the same inputs. This means the conditional generation of radiology reports should include both image and text inputs and have a text output. This year a new transformer architecture particularly suited for this type of multi-modal problem was just released by SalesForce (<a href="https://proceedings.mlr.press/v162/li22n.html">Li et al. 2022</a>). BLIP has a dual text and vision encoder paired with a text decoder. This allows it to continue generating new text for a radiology report from a given prompt’s starting point. Lucky for us, the first paragraph of most radiology reports is the clinical question!</p><p>This makes conditionally generating a radiology report possible in couple of lines of code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40716bba1546b70ce6810d909a28a943/href">https://medium.com/media/40716bba1546b70ce6810d909a28a943/href</a></iframe><h3>Simplified Application</h3><p>Starting from the <a href="https://huggingface.co/Salesforce/blip-image-captioning-large">base BLIP image captioning model</a>, I fine-tuned a <a href="https://huggingface.co/nathansutton/generate-cxr">causal language model</a> to generate radiology reports from a chest x-ray and a clinical prompt. The data used to fine-tune these assessments were derived from the MIMIC critical care database. Specifically, I cross referenced the original radiology reports in the <a href="https://physionet.org/content/mimic-cxr/2.0.0/">MIMIC-CXR</a> project with the JPG images available in the <a href="https://physionet.org/content/mimic-cxr-jpg/2.0.0/">MIMIC-CXR-JPG</a> project.</p><p>More information on how to reproduce these labels can be found in the corresponding <a href="https://github.com/nathansutton/prerad"><strong>Github repository.</strong></a></p><p>Does it work? Let’s go back to our original radiology report and perturb it with two different clinical indications. On the left we show the original question for this image (‘question pneumonia’) and on the right a fictitious concern (‘question pneumothorax’). The original reference report is in quotes above. You can play around with your own de-identified images in this interactive web application hosted graciously by huggingface spaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zB6e8pCH1wqgEY5LKRRHEA.png" /><figcaption>Available @ <a href="https://nathansutton-prerad.hf.space/">https://nathansutton-prerad.hf.space/</a>. The same image with two different clinical prompts. This shows that by changing the prompt to ‘chest pain, history of pneumothorax’ the model successfully changed its answer in response to a different clinical question.</figcaption></figure><p>This is a simplified example was meant to demonstrate one concept. Conditionally generated radiology reports should include text inputs alongside the medical images to answer a clinical question. Obviously this kind of automation is not intended to replace radiologists, but it could help them quickly template their reports so they aren’t starting from scratch.</p><p>In the bigger scheme of things, I have seen this kind of disparity between the question technologists are answering and providers are asking repeated across healthcare. <strong>Bring a provider on board to collaborate, and you’ll be rewarded with useful models</strong>.</p><p>Many thanks to @GarySorcher for his invaluable feedback.</p><p>Note: these opinions are my own and are not reflective of any work @ Inflo Health.</p><h3>References</h3><p>Chen, Zhihong, et al. “Generating radiology reports via memory-driven transformer.” <em>arXiv preprint arXiv:2010.16056</em> (2020).</p><p>Chen, Zhihong, et al. “Cross-modal memory networks for radiology report generation.” <em>arXiv preprint arXiv:2204.13258</em> (2022).</p><p>Jing, Baoyu, Pengtao Xie, and Eric Xing. “On the automatic generation of medical imaging reports.” <em>arXiv preprint arXiv:1711.08195</em> (2017).</p><p>Li, Junnan, et al. “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.” <em>International Conference on Machine Learning</em>. PMLR, 2022.</p><p>Li, Yuan, et al. “Hybrid retrieval-generation reinforced agent for medical image report generation.” <em>Advances in neural information processing systems</em> 31 (2018).</p><p>Liu, Fenglin, et al. “Exploring and distilling posterior and prior knowledge for radiology report generation.” <em>Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>. 2021.</p><p>Miura, Yasuhide, et al. “Improving factual completeness and consistency of image-to-text radiology report generation.” <em>arXiv preprint arXiv:2010.10042</em> (2020).</p><p>Nooralahzadeh, Farhad, et al. “Progressive transformer-based generation of radiology reports.” <em>arXiv preprint arXiv:2102.09777</em> (2021).</p><p>Ramesh, Vignav, Nathan A. Chi, and Pranav Rajpurkar. “Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors.” <em>Machine Learning for Health</em>. PMLR, 2022.</p><p>Sirshar, Mehreen, et al. “Attention based automated radiology report generation using CNN and LSTM.” <em>Plos one</em> 17.1 (2022): e0262209.</p><p>Singh, Sonit, et al. “From chest x-rays to radiology reports: a multimodal machine learning approach.” <em>2019 Digital Image Computing: Techniques and Applications (DICTA)</em>. IEEE, 2019.</p><p>Yang, Shuxin, et al. “Knowledge matters: Chest radiology report generation with general and specific knowledge.” <em>Medical Image Analysis</em> 80 (2022): 102510.</p><p>Xue, Yuan, et al. “Multimodal recurrent model with attention for automated radiology report generation.” <em>Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16–20, 2018, Proceedings, Part I</em>. Springer International Publishing, 2018.</p><p>Yuan, Jianbo, et al. “Automatic radiology report generation based on multi-view image fusion and medical concept enrichment.” <em>Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part VI 22</em>. Springer International Publishing, 2019.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b687a993cbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A template for your ML take home]]></title>
            <link>https://nasutton.medium.com/a-template-for-your-ml-take-home-e59395034027?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e59395034027</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Fri, 12 Aug 2022 19:40:36 GMT</pubDate>
            <atom:updated>2022-08-12T19:40:36.195Z</atom:updated>
            <content:encoded><![CDATA[<h4>Dockerized with Jupyter, Postgres, FastAPI, and Streamlit</h4><p>One common interview practice for machine learning roles is to send the candidate a small take home problem. They usually are framed like the following.</p><blockquote>Here are some data and instructions</blockquote><blockquote>Build a model for us using a Jupyter notebook</blockquote><blockquote>Send us the notebook and/or some slides and we’ll talk about it</blockquote><p>Years ago when we gave this problem to a candidate he went further to actually deploy a little model inside a web application. <strong>We all really appreciated his desire to take a vision end to end, and this </strong><a href="https://github.com/nathansutton/ml-api-template"><strong>template repository</strong></a><strong> is meant to give a similar flavor. </strong>It will allow you focus on the problem at hand with less boilerplate code and a much better deliverable!</p><p>With this repository you can be sure that you work is reproduced by a hiring committee because <strong>it’s dockerized</strong>! In my experience it pays dividends to actually walk through a notebook instead of just looking at the output.</p><p>With this repository you can take your model <strong>deploy it as a REST API</strong>. This shows you can think beyond your interactive work in notebooks.</p><p>With this repository your <strong>users can interact with your model </strong>in a web application (albeit, a crude one). This pay dividends as your hiring committee can inspect your model’s outputs and see your latency.</p><h3>Services</h3><p>This repository exposes four components that are useful in a take home assignment.</p><ul><li>A container running Jupyter notebooks with common machine learning libraries (port:8888). Any notebooks will persist in a mounted volume (./volumes/notebooks).</li><li>A container running Postgres in the event a relational database is useful (port:5432). Any transformations will persist between containers in a mounted volume (./volumes/postgres).</li><li>A container running FastAPI to serve predictions from a scikit-learn model (port:8080)</li><li>A container running Streamlit allows a user to access the predictions from their scikit-learn model based on user inputs (port:8501)</li></ul><h3>Usage</h3><p>This template is completely dockerized, and so to turn on the individual components just involves a single command.</p><pre>docker-compose up </pre><h3>Data</h3><p>This template includes an example dataset that I created based on iteratively querying the <a href="https://www-doh.state.nj.us/doh-shad/query/builder/birth/PretermAllCnty/Preterm.html">New Jersey State Health Assessment Data</a>, and these data are also available as an extract on <a href="https://www.kaggle.com/datasets/natesutton/njpretermbirths">Kaggle</a>. Full caveats there is virtually no signal in these data to predict premature birth outcomes, it is merely meant to illustrate a problem.</p><h3>REST API Endpoint</h3><p>The model is available as a REST API endpoint on port 8080. It accepts JSON data that look like 1 row of the dataframe it as trained on.</p><pre>curl --request POST http://127.0.0.1:8080/predict \<br>    -H &#39;Content-Type: application/json&#39; \<br>    -d &#39;{&quot;age_group&quot;: &quot;Under 15 yrs&quot;,&quot;reported_race_ethnicity&quot;:  &quot;White, non-Hispanic&quot;, &quot;previous_births&quot;: &quot;None&quot;,&quot;tobacco_use_during_pregnancy&quot;: &quot;Yes&quot;,&quot;adequate_prenatal_care&quot;: &quot;Inadequate&quot;}&#39; </pre><h3>Streamlit User Interface</h3><p>A small web application can take features used to drive your model, then return a prediction from the REST API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/0*e0nqAFSveKXnhCqO.png" /><figcaption>Sample User Interface — all images by author.</figcaption></figure><h3>Let’s Connect!</h3><p>If you’ll benefit from this work, clone the repository and chug away at your problem! If this is all a little daunting and you’d like help breaking into ML roles in industry, reach out to me or other mentors @ <a href="https://sharpestminds.com/?r=nathan-sutton">SharpestMinds</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e59395034027" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 excellent healthcare data sources]]></title>
            <link>https://nasutton.medium.com/10-excellent-healthcare-data-sources-531468e3dee1?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/531468e3dee1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Wed, 30 Mar 2022 16:09:15 GMT</pubDate>
            <atom:updated>2022-04-11T20:05:03.200Z</atom:updated>
            <content:encoded><![CDATA[<p>Spanning surveys, electronic records, and medical images.</p><p>I have taken on a number of mentees interested in the applications of data to the healthcare space over the years. Their biggest roadblock to developing useful products is the <strong>lack of available healthcare data</strong>. This equally applies to startups — the sales cycle of actually convincing a covered entity under <a href="https://www.cdc.gov/phlp/publications/topic/hipaa.html#:~:text=The%20Health%20Insurance%20Portability%20and,the%20patient&#39;s%20consent%20or%20knowledge.">HIPAA</a> to share data in exchange for a new product can take months, years!</p><p>Fortunately, depending on your needs there is a reasonable diversity of healthcare data available for free. You just have to be willing to jump through some hoops.</p><p>tl;dr — you can find the entire list and contribute your own on my <a href="https://github.com/nathansutton/healthcare-data-sources/blob/main/README.md">Github</a>.</p><h3><strong>Electronic Health Records</strong></h3><p><strong>MIMIC — </strong>EHRs are the golden goose of data sources in healthcare, but good luck getting even read-only access to one. My first recommendation for people interested in healthcare data is the Multiparameter Intelligent Monitoring in Intensive Care, or MIMIC database. In particular, focus on the Observational Medical Outcomes Partnership (OMOP) <a href="https://physionet.org/content/mimic-iv-demo-omop/0.9/">translation of MIMIC</a> into a harmonized relational database. That will at least give you a fighting chance not to have to rewrite every single pipeline you build when you land a real client.</p><ul><li><em>Cost</em>: Free</li><li><em>Size</em>: 49,785 hospital admissions of 38,597 patients</li><li><em>Requires</em>: Application, human subjects research course, several week turnaround time</li><li><em>Includes</em>: demographics, vitals, labs, procedures, medications, nursing notes, imaging reports, mortality events</li><li><em>Reference</em>: <a href="https://doi.org/10.13026/C2XW26">https://doi.org/10.13026/C2XW26</a></li><li><em>Access</em>: <a href="https://mimic.mit.edu/docs/gettingstarted">https://mimic.mit.edu/docs/gettingstarted</a></li><li><em>Temporal coverage</em>: 2001 through 2012</li><li><em>Source</em>: Single Institution — Beth Israel Deaconess Medical Center</li><li><em>Owner</em>: MIT Laboratory for Computational Physiology</li></ul><p><strong>SYNTHEA — </strong>If you don’t have time for a full application process, you can quickly spin up synthetic data from the SYNTHEA project that realistically models patient healthcare pathways.</p><ul><li><em>Cost</em>: Free</li><li><em>Size</em>: variable</li><li><em>Requires</em>: no application or restrictions</li><li><em>Includes</em>: demographics, vitals, labs, procedures, medications, imaging</li><li><em>Reference</em>: <a href="https://doi.org/10.1093/jamia/ocx147">https://doi.org/10.1093/jamia/ocx147</a></li><li><em>Access</em>: <a href="https://synthetichealth.github.io/synthea/">https://synthetichealth.github.io/synthea/</a></li><li><em>Source</em>: synthetic data</li><li><em>Owner</em>: MITRE Corporation</li></ul><h3>Medical Images</h3><p>There are a number of open datasets for medical images given how much easier it is to anonymize images than electronic health records. This list is by no means exhaustive. You’ll notice a common theme — chest x-rays are the dominant data available.</p><p><strong>MIDRC —</strong>The<strong> </strong>Medical Imaging and Data Resource Center have an interesting project that is meant to solve many of the problems startups have faced in taking a deep-learning model through FDA clearance. I especially appreciate their emphasis on multi-institution data with a stratified sampling scheme to ensure the validity of any results across diverse populations.</p><ul><li><em>Cost</em>: Free</li><li><em>Size</em>: 29,630 imaging studies of 10,835 patients (and growing)</li><li><em>Labels</em>: COVID</li><li><em>Requires</em>: No application</li><li><em>Includes</em>: imaging, demographics</li><li><em>Reference</em>: <a href="https://doi.org/10.1117/1.JMI.8.S1.010902">https://doi.org/10.1117/1.JMI.8.S1.010902</a></li><li><em>Access</em>: <a href="https://data.midrc.org/DD">https://data.midrc.org/DD</a></li><li><em>Temporal coverage</em>: 2019 onwards</li><li><em>Source</em>: Multiple Insitutions</li><li><em>Owner</em>: Center for Translational Data Science @ University of Chicago</li></ul><p><strong>NIH — </strong>The National Institute of Health chest x-ray data was the catalyst for many deep learning applications in radiology.</p><ul><li><em>Cost</em>: Free</li><li><em>Size</em>: 108,948 chest radiographs of 32,717 patients</li><li><em>Labels</em>: 9 different pathologies</li><li><em>Requires</em>: No application</li><li><em>Includes</em>: imaging</li><li><em>Reference</em>: Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, Ronald M. Summers; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2097–2106</li><li><em>Access</em>: <a href="https://stanfordmlgroup.github.io/competitions/chexpert/">https://stanfordmlgroup.github.io/competitions/chexpert/</a></li><li><em>Source</em>: Multiple Institutions</li><li><em>Owner</em>: National Institute of Health</li></ul><p><strong>CheXpert — </strong>more recently Stanford released CheXpert with more pathologies and radiographs than the NIH data.</p><ul><li><em>Cost</em>: Free</li><li><em>Size</em>: 224,316 chest radiographs of 65,240 patients</li><li><em>Labels</em>: 14 different pathologies</li><li><em>Requires</em>: No application</li><li><em>Includes</em>: imaging</li><li><em>Reference</em>: <a href="https://arxiv.org/abs/1901.07031">https://arxiv.org/abs/1901.07031</a></li><li><em>Access</em>: <a href="https://stanfordmlgroup.github.io/competitions/chexpert/">https://stanfordmlgroup.github.io/competitions/chexpert/</a></li><li><em>Temporal coverage</em>: 2002 through 2017</li><li><em>Source</em>: Single Insitution — Stanford Hospital</li><li><em>Owner</em>: Stanford Machine Learning Group</li></ul><h3>Medical Surveys</h3><p>NHIS — The National Health Interview Survey is a great place to start if you’re looking for anonymous healthcare data on individuals outside of an electronic health record. A grain of salt, it’s a survey so the values are self-reported. These surveys go back to the 1960s, but practically they are more consistent in the last decade. You can go get the data directly yourself from the government, but I would skip the conforming and go straight to the <a href="https://nhis.ipums.org/">IPUMS</a> project at the University of Minnesota. They maintain a slick interactive web application that lets you add data to your cart to generate an extract.</p><ul><li><em>Cost</em>: Free</li><li><em>Size</em>: Variable, 50–100k responses annually</li><li><em>Requires</em>: No application</li><li><em>Includes</em>: self-reported health outcomes, demographics, socioeconomic data</li><li><em>Reference</em>: <a href="https://doi.org/10.18128/D070.V6.4">https://doi.org/10.18128/D070.V6.4</a></li><li><em>Access</em>: <a href="https://nhis.ipums.org/">https://nhis.ipums.org/</a></li><li><em>Temporal coverage</em>: 1963 through present</li><li><em>Source</em>: Census Microdata</li><li><em>Owner</em>: IPUMS</li></ul><h3>Medical Claims Databases</h3><p>The most diversity in healthcare data can be found in the data trail of the payments infrastructure.</p><p><strong>NIS — </strong>The National Inpatient Sample represents ~20% of all hospital admissions, and so it is verifiably huge. Students can access most of these data for a reasonable cost, at least relative to other claims databases.</p><ul><li><em>Cost</em>: $150 to $1000 per year depending on your situation and data selected</li><li><em>Size</em>: Huge, this approximates a 20-percent stratified sample of all hospital discharges</li><li><em>Requires</em>: Application and 15m training course</li><li><em>Includes</em>: fine-grain billing details, hospital reported health outcomes, demographics</li><li><em>Access</em>: <a href="https://www.distributor.hcup-us.ahrq.gov/Databases.aspx">https://www.distributor.hcup-us.ahrq.gov/Databases.aspx</a></li><li><em>Temporal coverage</em>: 2012 through present</li><li><em>Source</em>: Multiple Institutions</li><li><em>Owner</em>: Healthcare Cost and Utilization Project</li></ul><p><strong>MarketScan — </strong>if you have a pile of cash or are supported by an institution, you can look into IBM’s all-payer claims database.</p><ul><li><em>Cost</em>: $$$$$</li><li><em>Requires</em>: talking to a sales rep</li><li><em>Includes</em>: demographic data, geographic location, inpatient and outpatient claims, medical claims of over 250 million patients</li><li><em>Access</em>: <a href="https://www.ibm.com/products/marketscan-research-databases/databases">https://www.ibm.com/products/marketscan-research-databases/databases</a></li><li><em>Temporal coverage</em>: 2010 through 2019</li><li><em>Source</em>: Multiple institutions</li><li><em>Owner</em>: IBM</li></ul><p><strong>SynPUF</strong> — Years ago the Centers for Medicare and Medicaid services created a synthetic extract that mimics their limited claims dataset (available for purchase). This would make sense if Medicare plans are your primary endpoint.</p><ul><li><em>Cost</em>: Free</li><li><em>Requires</em>: No application</li><li><em>Includes</em>: synthetic billing data that mirrors the limited CMS datasets</li><li><em>Reference</em>: Borton, J. M., et al. “Data Entrepreneurs’ Synthetic PUF: A Working PUF as an Alternative to Traditional Synthetic and Non-synthetic PUFs.” JSM Proceedings, Survey Research Methods Section (2010).</li><li><em>Access</em>: <a href="https://www.cms.gov/Research-Statistics-Data-and-Systems/Downloadable-Public-Use-Files/SynPUFs/DE_Syn_PUF">https://www.cms.gov/Research-Statistics-Data-and-Systems/Downloadable-Public-Use-Files/SynPUFs/DE_Syn_PUF</a></li><li><em>Temporal coverage</em>: 2008 through 2010</li><li><em>Source</em>: Multiple institutions</li><li><em>Owner</em>: Centers for Medicare and Medicaid Servicee</li></ul><h3>Summary statistics</h3><p><strong>Dartmouth Health Atlas</strong> — If you are just looking for statistics relating to healthcare cost and utilization in a geographic region, look no further than the Dartmouth Health Atlas. I have had a lot of success tieing these in as features in ML models when my data is already spatially coded.</p><ul><li><em>Cost</em>: Free</li><li><em>Requires</em>: no application or restrictions</li><li><em>Includes</em>: medical reimbursement rates, access to care, mortality, hospital capacity</li><li><em>Access</em>: <a href="https://data.dartmouthatlas.org/#rates">https://data.dartmouthatlas.org/#rates</a></li><li><em>Temporal coverage</em>: 2011 through 2019</li><li><em>Source</em>: Multiple institutions</li><li><em>Owner</em>: Dartmouth</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=531468e3dee1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which tool should you use for database migrations?]]></title>
            <link>https://medium.com/data-science/which-tool-should-you-use-for-database-migrations-4e0b9c44b790?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e0b9c44b790</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Wed, 09 Feb 2022 18:13:48 GMT</pubDate>
            <atom:updated>2023-01-06T14:10:50.162Z</atom:updated>
            <content:encoded><![CDATA[<p>Choose a paradigm based on your needs.</p><p>All my data teams have had a relational database at the core of our operations. When you have a relational database, you will inevitably have to change it! This post will help you decide between three useful frameworks for versioned database migrations — <a href="https://sqitch.org/"><strong>sqitch</strong></a>, <a href="https://flywaydb.org/"><strong>flyway</strong></a>, and <a href="https://www.liquibase.com/pricing?utm_term=liquibase&amp;utm_campaign=brandsearch&amp;utm_source=google&amp;utm_medium=ppc&amp;hsa_acc=7217304035&amp;hsa_cam=12808646352&amp;hsa_grp=119923141903&amp;hsa_ad=517707365974&amp;hsa_src=g&amp;hsa_tgt=kwd-370520217004&amp;hsa_kw=liquibase&amp;hsa_mt=e&amp;hsa_net=adwords&amp;hsa_ver=3&amp;gclid=Cj0KCQiAxoiQBhCRARIsAPsvo-wjD1EMdIGF00wKitSh_7kJhUUUIsxq6bHntP0Q8DzrpU3yq0iZI7YaAqDIEALw_wcB"><strong>liquibase</strong></a>. There is an associated <a href="https://github.com/nathansutton/database-migrations">repository</a> in GitHub that walks through each implementation of the same tables in a dockerized PostgreSQL database.</p><h3>tl;dr</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXTcL3QGjGg2c9YIhcaqAw.png" /><figcaption>A quick decision tree to help you decide (Image by author)</figcaption></figure><h3><strong>Why these three?</strong></h3><p>I focused this guide on sqitch, flyway, and liquibase because they all share two important characteristics. First, they all work with virtually any relational database — this cuts out Microsoft’s excellent <a href="https://docs.microsoft.com/en-us/sql/relational-databases/data-tier-applications/data-tier-applications?view=sql-server-ver15">DACPACs</a>. Second, they all allow users to write migrations in plain SQL — this cuts out ORM implementations for migrations like <a href="https://alembic.sqlalchemy.org/en/latest/">Alembic</a> and <a href="https://orator-orm.com/">Orator</a>.</p><h3>Does your data come from a CRUD application?</h3><blockquote>you have options</blockquote><p>The defining feature of many OLTP workloads is that the backend receives and stores data from the application layer. All three of these migration tools excel at the basic functionality of keeping a database and its application in sync. Migrations are atomically applied to a database and the data remains at rest in the same tables.</p><h3>Does your data come from cloud storage?</h3><blockquote>consider flyway</blockquote><p>In contrast, many OLAP workloads are easily reproduced by loading all the staging files again from cloud storage like S3 or Azure Blob Storage. Flyway shines here as a SQL executor to reliably reproduce the state of a given database. In particular, it is easy to separate the DML from the DDL in different types of plain SQL files. This allows you to run the same transformations again, but load in the latest data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*4NlWzAUjZtR3aWaLYAsHAw.png" /><figcaption>Flyway excels at repeating the same task with new data in ‘callbacks’ (Image by author)</figcaption></figure><h3>Are there many ordered DML transformations?</h3><blockquote>consider flyway</blockquote><p>If your data manipulation language moves data between multiple tables before being ready to use downstream, then flyway has a unique advantage over sqitch because y<em>ou can easily infer the order of execution from the file names. </em>Flyway is broken down into ‘versioned migrations’ (V* prefix, most commonly your DDL) and ‘callbacks’ (afterMigrate* prefix, most commonly your DML). Most of my teams have used XXXX numerical scripting as a naming convention. In the simplified example below, it is easy to see that the gross table is loaded <a href="https://github.com/nathansutton/database-migrations/tree/main/src/migrations/flyway">AFTER the price table.</a></p><p>This kind of interpretability is super helpful for developers who are responsible for batch analytical workloads. [<em>Warning: biased by my focus in data science</em>]. Virtually all statistical models I have deployed depend on pipelines that iteratively cleanse and aggregate data until they are ready for the model. If you are not constrained by operating in real-time, this translates to a lot of ‘please execute this SQL in order’ at some interval. Flyway is a sure-fire way to get the same result every time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EgL1XrKzMYEBvyyzt-MbHA.png" /><figcaption>The order of execution is clear from the file system (Image by author)</figcaption></figure><p>In contrast, the same ordering logic is hidden with sqitch in a sqitch.plan file. This is made exponentially more difficult when teams use cross-reference ticket numbers as their file names. Now you have to look up in three places to figure out what (file system), when (sqitch.plan), and why (JIRA et al.) anything was done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugNhJ2w9JC9sbpBIqyE6Cw.png" /><figcaption>The order of execution is apparent in the sqitch.plan, but not the file system (Image by author)</figcaption></figure><h3>Do you need to revert migrations?</h3><blockquote>consider sqitch</blockquote><p>Sqitch has more overhead in development in that you have to write logic to make a change, logic to test if it worked, and logic to revert the change if your test fails. This is more work at the outset, but it comes with the excellent feature that you can roll back migrations to any given point in history. <strong>Unfortunately, </strong><a href="https://flywaydb.org/documentation/command/undo"><strong>rolling back migrations is a paid feature in flyway</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YNOcvIZvUyTIMCnn06vmDA.png" /><figcaption>trust but verify with sqitch (Image by author)</figcaption></figure><h3>Do you use environment variables to parameterize your deployments?</h3><blockquote>avoid liquibase</blockquote><p>I would be surprised if you weren’t doing this already. Keeping database secrets out of version control is security 101. <strong>Unfortunately, using environment variables is a </strong><a href="https://docs.liquibase.com/commands/liquibase-environment-variables.html"><strong>paid feature in liquibase</strong></a>. Both flyway and sqitch support parameterization out of the box. Of course, you can dive down and modify their docker entrypoint, but WHY is this necessary?!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubLIwW2GvHa9KrcmGPM1Ww.png" /><figcaption>Please keep your secrets out of version control (Image by author)</figcaption></figure><h3>Still not sure?</h3><p><a href="https://github.com/nathansutton/database-migrations"><strong>Fork this repository</strong></a> to try building out some of your core table structures. All the painful things like configuration files and volume mappings are taken care of, and you can simply swap in your migrations into any of one of the three tools.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e0b9c44b790" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/which-tool-should-you-use-for-database-migrations-4e0b9c44b790">Which tool should you use for database migrations?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing distributions of hospital prices]]></title>
            <link>https://nasutton.medium.com/comparing-distributions-of-hospital-prices-bc59cad48759?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc59cad48759</guid>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Mon, 18 Oct 2021 11:33:06 GMT</pubDate>
            <atom:updated>2022-04-11T20:05:27.970Z</atom:updated>
            <content:encoded><![CDATA[<p>Kolmogorov and Smirnov to the rescue.</p><p>I just received an unfortunate out-of-network bill for the physician services portion of an emergency room visit (in-network facility), and so hospital prices are on my mind again. I’ll spend a moment exploring how an <a href="https://nasutton.medium.com/an-open-database-of-hospital-prices-e07aa8120fa1">open database of hospital prices</a> can help answer the following question.</p><blockquote><strong>Which hospital system in North Carolina has the least costly distribution of prices relative to other hospitals?</strong></blockquote><p>These data begin with obscure chargemaster files hidden on hospital websites, but I went through the exercise to migrate them into a handy <a href="https://github.com/nathansutton/hospital-price-transparency">dockerized database</a>. In this world, they are represented as a normalized price table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*HyxQbehsOeiLg2jrdxATKA.png" /></figure><p>In the first iteration of these files, one consistent type of charge reported by hospitals is <strong>gross</strong> prices. These aren’t actually paid by anyone but can be thought of as the starting point a hospital takes in its price negotiations with insurers. Here is a small sample of the table. The column <strong>hospital_id</strong> links to the <a href="https://github.com/nathansutton/hospital-price-transparency/blob/gross/volumes/data/dim/hospital.csv">dimension table</a> for each hospital, and <strong>concept_id </strong>links to the appropriate CPT or HCPCS code in the <a href="https://athena.ohdsi.org/search-terms/start">Athena standard vocabulary</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*nd8sVDLG-nDxkzQeBwwmfw.png" /></figure><p>These prices can then be pivoted into distinct columns for each hospital system. Every row in this view represents a billing concept. Every column in this view represents a hospital group. For example, Transylvania Regional Hospital is part of the Mission Healthcare system (now owned by HCA). Note that there can be many prices for one billing concept in this table. In particular, one distinct price for each hospital in a system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*s6mq_bqqJwjCfNDQJmDTeQ.png" /></figure><p>Using this matrix we can compare the hospital prices of one system to the rest of the systems using their empirical cumulative distribution functions. In each plot below a hospital system (red) is compared to the rest of the combined systems (grey). <strong>Higher values mean that a system has relatively lower gross prices than others</strong>. The y-axis represents the quantile of the empirical cumulative distribution. Given the incredible skew in prices, I chose to log10 transform these prices (x-axis).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Z2SvH3bhS1AuXA4jFGx6NQ.png" /></figure><p>One non-parametric test to evaluate of two distributions are equivalent is the <a href="https://www.tandfonline.com/doi/abs/10.1080/01621459.1951.10500769">Kolmogorov-Smirnov test</a>. This returns a test statistic D that is larger when the two distributions diverge. Most of these differences are small, with only First Health not differing significantly from the rest of the hospital systems.</p><blockquote><strong>Lucikly for me, my favorite hospital Pardee UNC has substantially lower gross charges (D=0.26) for all billing codes in their cumulative distribution.</strong></blockquote><p>Caveats — This is meant to be an illustrative and light-hearted look at hospital pricing data. It does not account for contracted rates or self-pay discounts at any hospital system. In no way use this to determine where you should seek care.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc59cad48759" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An open database of hospital prices]]></title>
            <link>https://nasutton.medium.com/an-open-database-of-hospital-prices-e07aa8120fa1?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e07aa8120fa1</guid>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Tue, 07 Sep 2021 20:56:20 GMT</pubDate>
            <atom:updated>2021-09-07T20:56:20.362Z</atom:updated>
            <content:encoded><![CDATA[<p>Normalized data for healthcare research</p><h3>tl;dr</h3><p>You can visit this <a href="https://github.com/nathansutton/hospital-price-transparency">Github repository</a> or <a href="https://www.kaggle.com/natesutton/hospitalpricetransparency">Kaggle data</a> to find a open source database of reported hospital prices. Currently I covered most hospitals in North Carolina, but I plan to expand across the southeast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*C6JrJBFFQCoWnPdQ4UnPHg.png" /></figure><h3>Motivation</h3><p>The Centers for Medicare and Medicaid Services recently required hospitals under <a href="https://www.federalregister.gov/d/2019-24931/p-1010">45 CFR §180.50</a> to publish a <a href="https://www.cms.gov/hospital-price-transparency">list of prices</a> on their websites. They specifically instruct hospitals to make these lists…</p><ul><li>As a comprehensive machine-readable file with all items and services.</li><li>In a display of shoppable services in a consumer-friendly format.</li></ul><p>There is a lot of variation in adherence to these policies. Without strong guidance on formatting from CMS, it is no wonder hospitals are all over the map on formatting. Many hospitals have complied with the new rules but in ways that are not consumer friendly. <a href="https://chsprodrsrcegrporg385.blob.core.windows.net/cms-price-transparency/56-0529945_AtriumHealthCabarrus_StandardCharges.json">500 Megabytes of JSON</a> data is not a strong start!</p><p><a href="https://turquoise.health/">Turquoise Health</a> has created a consumer-friendly lookup tool to interactively look up reported prices in different hospital systems. However, my guess is that they would not be happy to sharing the underlying data they have monetized (but I will ask). <strong>This repository fills the gap with open data for health researchers and data people.</strong></p><h3>Supplied Data</h3><p>If you don’t have the proclivity to transform these data yourself with docker, there are CSV extracts available in ./volumes/data/extracts or directly on Kaggle. They are broken down into four distinct groups.</p><ul><li>gross: this is often the top line item that the hospital never actually charges</li><li>cash: this is the self-pay discounted price you would pay without insurance</li><li>max: this is the maximum negotiated rate by an insurance company in the hospital network.</li><li>min: this is the minimum negotiated rate by an insurance company in the hospital net</li></ul><p>A minority of hospitals included the payer and plan specific charges as their own column. I found that hospitals much more frequently reported the de-identified maximum and minimum negotiated prices, and so I started there.</p><h3>Ontology</h3><p>Using standardized codes will ensure that we are talking about the same thing in different hospitals. To accomplish this goal I built on the excellent work of the <a href="https://athena.ohdsi.org/">Athena</a> vocabulary to define the ontology of healthcare procedures. This maps <a href="https://www.ama-assn.org/practice-management/cpt">CPT</a> and <a href="https://www.cms.gov/Medicare/Coding/MedHCPCSGenInfo">HCPCS</a> codes into a <a href="https://github.com/OHDSI/CommonDataModel">common data model</a>.</p><p>The disadvantage with this normalization is that we exclude the hospital-specific items such as their room and board charges. For example, ‘deluxe single’ does not confer the same charge in different hospital systems.</p><h3>A word of caution</h3><p>I took great care to make the data generation process reproducible inside of a docker container. However, I sacrificed some scalability for the name of speed of development. There are some <a href="https://github.com/vsoch/hospital-chargemaster/blob/master/hospitals.tsv">excellent examples</a> how you could scrape your way through this to complete automation. I introduced a manual step of downloading a file and naming it by my generated hospital ID. All other transformations are codified and reproducible in the container.</p><h3>Where to go from here</h3><p>I will use these data as a launching point to investigate questions around the economics of hospital prices. The New York Times had a great quote in their recent article ‘<a href="https://www.nytimes.com/interactive/2021/08/22/upshot/hospital-prices.html?smid=url-share">Hospitals and Insurers Didn’t Want You to See These Prices. Here’s Why</a>’.</p><blockquote>The trade association for insurers said it was “an anomaly” that some insured patients got worse prices than those paying cash.</blockquote><p><strong>This seems like an excellent question for data people to answer.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e07aa8120fa1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Step into OMOP with a single line]]></title>
            <link>https://nasutton.medium.com/step-into-omop-with-a-single-line-3b3188cc19e2?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b3188cc19e2</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Fri, 19 Feb 2021 16:21:51 GMT</pubDate>
            <atom:updated>2021-02-19T16:21:51.981Z</atom:updated>
            <content:encoded><![CDATA[<h3>tl;dr</h3><p>You can visit our <a href="https://github.com/nathansutton/omop">Github repository</a> to easily spin up a containerized version of OMOP’s common data model with flyway migrations.</p><h3>What is OMOP?</h3><p>I am continually impressed by the offerings from the <a href="http://www.ohdsi.org/">Observational Health Data Sciences and Informatics</a> program. In particular, their Observational Medical Outcomes Partnership’s <a href="https://github.com/OHDSI/CommonDataModel">Common Data Model</a> (CDM) promises to standardize the pain we feel in dealing with different healthcare source data systems. I have worked with data from electronic medical records, health level seven feeds, and almost always some form of medical claims.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*3MShuNajv9CV6T7x_h9zaQ.png" /><figcaption>The CDM promises to harmonize all disparate healthcare data sources into a single relational database.</figcaption></figure><p>Harmony sounds well and good, but why is it useful?</p><blockquote>Most of my recent projects required training a machine learning model on a different source system than it needed to be deployed on.</blockquote><p>For example, at <a href="https://medium.com/@nasutton/machine-learning-supports-clinicians-to-safely-discharge-chest-pain-patients-from-the-emergency-1285a3b4d13e">Vituity</a> we build a model to estimate the probability a patient presenting with chest pain in the ER was experiencing acute coronary syndrome. This started with a direct feed from the electronic medical record but had to be deployed on top of HL7 messages in order to be useful at the point of care. The common data model makes this transition seamless.</p><h3>A single line?</h3><p>OMOP maintains a thorough repository of its common data model with every flavor of SQL you could need. I wrapped their PostgreSQL flavor into a containerized database with Docker. This means to get started you just execute a single command.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13f37c54cf533cedd90076950a292fd1/href">https://medium.com/media/13f37c54cf533cedd90076950a292fd1/href</a></iframe><p>Voila! You have a common data model to start playing with. In the backend, all the SQL is structured specifically so <a href="https://flywaydb.org/">Flyway</a> can easily execute all the commands.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f406c226c95b2ef986c13ae1c6104a31/href">https://medium.com/media/f406c226c95b2ef986c13ae1c6104a31/href</a></iframe><p>If you want to shut down your database, don’t worry! All your data will be persisted outside of your container.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b898f0f054a632105b1d26cb4432b162/href">https://medium.com/media/b898f0f054a632105b1d26cb4432b162/href</a></iframe><h3><strong>Future Work</strong></h3><p>We’ll use this infrastructure as a launching pad for several more posts. Next, we’ll demonstrate how you can banish mapping tables for medical concepts from your database by deriving them from the <a href="https://athena.ohdsi.org/search-terms/start">Athena vocabulary</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b3188cc19e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It’s not you, it’s them.]]></title>
            <link>https://nasutton.medium.com/its-not-you-it-s-them-9ea9fc8f0d34?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ea9fc8f0d34</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[remote-working]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[recruiting]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Wed, 13 Mar 2019 16:42:00 GMT</pubDate>
            <atom:updated>2019-03-13T16:44:48.114Z</atom:updated>
            <content:encoded><![CDATA[<h3>It’s not you, it’s them: Larger companies take longer to call you back</h3><h3><strong>tl;dr</strong></h3><p>I am about 30 days into a job search for a <a href="https://yanirseroussi.com/2018/11/03/reflections-on-remote-data-science-work/">remote data science position</a>, and I thought I’d report some evidence from the field. <strong>Increasing a company’s size by 10% will increase the number of days to set up a screening call by 3%.</strong></p><h3><strong>Data Collection</strong></h3><p>I applied to 24 companies in this round, and at that moment in time I was shopping for a home among medium to larger teams. I kept track of my search in a spreadsheet, but here it is crudely put in a data.frame in in R.</p><p>Of these 24 companies, I was called to set up an interview from 10 of them.</p><pre>#####<br># R #<br>##### </pre><pre>df &lt;- data.frame(<br>  <em># company names</em><br>  company = c(&#39;Catalyst&#39;, &#39;Ascenscion&#39;, &#39;Aetna&#39;, &#39;Cigna&#39;, &#39;Clover&#39;, &#39;Genpact&#39;, &#39;CVS&#39;, &#39;IQVIA&#39;, &#39;Novant&#39;, &#39;Oscar&#39;, &#39;Mission&#39;, &#39;United&#39;, &#39;Medtronic&#39;, &#39;Siemens&#39;, &#39;Publicis&#39;, &#39;Dexcom&#39;, &#39;Oschner&#39;, &#39;Matrix&#39;, &#39;Rally&#39;, &#39;Invitae&#39;, &#39;GE&#39;, &#39;Renown&#39;, &#39;Virginia Premier&#39;, &#39;Atrium&#39;),<br>  <em># employee population on linkedin</em><br>  size = c(700, 56194, 41439, 28388, 518, 112857, 68250, 45142, 10333, 836, 3527, 115727, 79230, 32491, 2818, 2370, 9208, 1149, 850, 720, 51990, 2990, 830, 4368),<br>  <em># number of days it took them to call me back, NA if no callback</em><br>  lag = c(8, NA, 8, 9, 2, 8, NA, NA, 3, NA, NA, 24, NA, NA, NA, NA, 7, 5, NA, NA, NA, NA, 1, NA),<br>  stringsAsFactors = FALSE<br>)</pre><p><strong>Company size </strong>— I quickly scanned the LinkedIn page to get the number of active employees on their platform. This is a reasonable estimate of the employee population. My intuition is that this estimate will be biased downwards for larger companies that have more diverse positions, but for the moment let’s take it as given.</p><p><strong>Lag until callback</strong> — I measured the difference in days between my application date and my first point of contact with the company (e.g. ‘hello from X, we’d like to set up a phone screen). Note: this does not account for the number of days later the first call was actually scheduled.</p><h3><strong>Model Specification</strong></h3><p>In this particular application I am interested in the elasticity between company size and the lag until callback. To accomplish this I’ll use a transformation to build a ‘log-log’ regression model.</p><pre>fit &lt;- lm(<br>  log(lag) ~ log(size),<br>  data = df<br>)</pre><p>I’ll quickly test to make sure this model specification is valid with the excellent <a href="https://amstat.tandfonline.com/doi/abs/10.1198/016214505000000637">gvlma package</a>, or global validation of linear model assumptions.</p><pre>summary(gvlma::gvlma(fit))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*OIr3m4p-gke3cqUd0E7BBw.png" /></figure><p>Remember that this is a log-log specification, and so the coefficient can be interpreted as an elasticity. In this case, for every 10% increase in company size we would expect a 3% increase in the number of days it takes them to call you back.</p><pre>summary(fit)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*oYYU9_DBZ6oRyXZaYFcAzQ.png" /></figure><p>With all the assumptions satisfied, we can now add the regression line and confidence intervals to our plot assess the fit visually. This seems reasonable enough, but I would caveat in this sample I don’t have any companies that fall in the range of 1000–8000 employees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*PRXEIbN0UATNLCKvPatyCQ.png" /></figure><h3>Do you have more data?</h3><p>I would love to flesh out this analysis by increasing the sample size from a paltry 10 companies. <a href="https://www.linkedin.com/in/nasutton">Reach out</a> via LinkedIn and tell me your experience so we can build a more robust regression together!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ea9fc8f0d34" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy your H2O model without hassle in plain SQL]]></title>
            <link>https://nasutton.medium.com/deploy-your-h2o-model-without-hassle-in-plain-sql-66cfc5110ff3?source=rss-cffc89afabd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/66cfc5110ff3</guid>
            <category><![CDATA[h2o]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[production]]></category>
            <dc:creator><![CDATA[Nathan Sutton]]></dc:creator>
            <pubDate>Wed, 06 Mar 2019 21:35:28 GMT</pubDate>
            <atom:updated>2019-03-06T21:35:28.387Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*sK69K7yPr8D2oeJYX2gIOA.png" /><figcaption>or is SQL the new ML?</figcaption></figure><p>I consistently feel (warranted) pressure to improve my model with more elaborate feature engineering only to later facepalm when I need to put that model into production. The path forwards to deploy a complex model is clear, just make an API! This post represents the other end of the spectrum. <strong>Let’s do as much in the database as possible.</strong></p><h4>tl;dr</h4><p>You can still leverage all the power of Python/R to build and validate a generalized linear model. These functions will allow you to quickly deploy your model in a database as raw SQL code.</p><h4>Who this is for</h4><p>I have written this especially for</p><ul><li>Data Science practitioners</li><li>who are asked to deliver interpretable linear models quickly</li><li>whose data starts in a SQL database</li><li>whose predictions end up in a SQL database</li><li>who want to keep is simple</li></ul><h4>Functions</h4><p>In this post we’ll use the R version.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d488100bd5404d00ea170e37222f19ef/href">https://medium.com/media/d488100bd5404d00ea170e37222f19ef/href</a></iframe><p>But you could just as easily use the equivalent Python.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9663fa0adfaa2937abe4543fcba9765/href">https://medium.com/media/b9663fa0adfaa2937abe4543fcba9765/href</a></iframe><p><strong>Walkthrough</strong></p><p>In this post I’ll build a purposefully simple model for all-cause readmissions based on a demo sample of the <a href="https://mimic.physionet.org/">MIMIC-III</a> critical care database. This is a relational database that keeps track of all of a patient’s medical information (labs, diagnoses, vitals, etc) during a single hospital stay.</p><p>I purposefully did all the feature engineering / denormalization in SQL so that my deployment script would be this simple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*dsVFpjyaTItvfkGgKoGi1g.png" /></figure><p>In another post I’ll speak about how I got to this view from the MIMIC source tables. The important thing to know here is that the rows represent a cross-section of individuals across time. This is a really simple model, and so each of the columns represent a binary indicator (encoded as 1=present, 0=absent).</p><p>Now that we’ve inspected the data, let’s bring it into a dataframe.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75ad45feb1ee26ccd5fff7c678edb2a8/href">https://medium.com/media/75ad45feb1ee26ccd5fff7c678edb2a8/href</a></iframe><p>With our data in hand we can build a simple logistic regression model with H2O.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25982d9d97c223d07af48126d5a51f71/href">https://medium.com/media/25982d9d97c223d07af48126d5a51f71/href</a></iframe><p>Then translate it into SQL.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acfae38c6d0023b965aadfe5541b02de/href">https://medium.com/media/acfae38c6d0023b965aadfe5541b02de/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*JRjNDPZOWs1ntiSAqlPaYg.png" /></figure><p>The power here is that we can cut R (or Python) out of the loop entirely. All that we need to do is put this into an appropriate SQL structure, like a view.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd55fee8d39b008ef6da4dabe09b3809/href">https://medium.com/media/dd55fee8d39b008ef6da4dabe09b3809/href</a></iframe><p>This function is only useful if it faithfully returns the same predictions. Hooray for the 1:1 line!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*BdbvJ1xL7IpalRneHlSYgw.png" /></figure><h4>Make your life simple</h4><p>Some of my most successful deployments have followed this pattern. Without any complications from R or Python code to mess anything up, these views silently did their job and never failed.</p><p>Feel free to <a href="https://www.linkedin.com/in/nasutton">reach out</a> and connect if this is helpful to you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66cfc5110ff3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>