<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[CollectiveMD - Medium]]></title>
        <description><![CDATA[Improving online diagnosis with collective medical intelligence. - Medium]]></description>
        <link>https://medium.com/collectivemd?source=rss----4004eddadb80---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>CollectiveMD - Medium</title>
            <link>https://medium.com/collectivemd?source=rss----4004eddadb80---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:06:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/collectivemd" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Combining Multiple Diagnoses]]></title>
            <link>https://medium.com/collectivemd/collectivemd-bf1a8d0a923a?source=rss----4004eddadb80---4</link>
            <guid isPermaLink="false">https://medium.com/p/bf1a8d0a923a</guid>
            <category><![CDATA[insight-data-science]]></category>
            <category><![CDATA[insight-health-data]]></category>
            <category><![CDATA[health]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vivek Rajasekharan]]></dc:creator>
            <pubDate>Thu, 28 Sep 2017 18:08:19 GMT</pubDate>
            <atom:updated>2017-10-10T16:48:28.783Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Improving diagnosis with collective intelligence.</em></p><h3>Motivation</h3><p>As part of the Insight Data Science Fellowship, I worked on a project with an organization I will refer to as WC. The organization’s aim is to provide medical expertise to doctors. Such an initiative would be vastly beneficial to clinicians in need of second opinions, and ultimately the general public.</p><h3>Background</h3><p>Each medical case on WC is uploaded by a clinician and has details (e.g., the age, sex of the patient, chief complaint for the doctor’s visit) that help other clinicians solve the case. A case is typically solved by 10 clinicians. Each solving clinician picks their top 3 diagnoses for the case of interest.</p><h3>Goal</h3><p>Some disorders such as <a href="http://www.webmd.com/migraines-headaches/default.htm">migraine</a> (<strong><em>Figure 1, within the leftmost bar</em></strong>) are straightforward to diagnose from the symptoms, and have all 10 clinicians agreeing on the diagnosis, and therefore there is a single diagnosis. An example of the other extreme <a href="http://emedicine.medscape.com/article/1123821-overview">is pyoderma gangrenosum</a> (<strong><em>Figure 1, within the rightmost bar</em></strong>), an uncommon skin ulcer that shares symptoms with other conditions such as malignancy, infection, diabetes and trauma and is typically diagnosed by excluding these other conditions. It is not a surprise that attempting to diagnose this condition online leads to different diagnoses from all 10 solving clinicians. Most conditions lie in between and have 5 (mean) ± 2 (std) different diagnoses (<strong><em>Figure 1</em></strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*Xzs-P0N99doxOwukbHGlAw.png" /><figcaption><strong>Figure 1:</strong> The number of different diagnoses among clinicians solving a case follows a normal distribution with a mean of 5.</figcaption></figure><p><em>My goal is to use machine learning to predict the right diagnosis from different diagnoses provided by the clinicians. </em>I also aim to provide a metric that reveals the confidence in a prediction as this would allow recruitment of further clinicians to solve cases of low confidence.</p><h3>Data Exploration</h3><p>The dataset (<strong><em>Figure 2</em></strong>) explored consists of 721 cases, each solved by 10 clinicians (7210 samples). Every sample in addition to the case details, and predicted diagnoses also contains information regarding the doctors creating and solving the case (e.g., tenure, training institution, time the solver spent on the case).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*42JceRoEl8BiG_Yf7jkGxg.png" /><figcaption><strong>Figure 2: </strong>The dataset for a sample case containing case information and information regarding the 10 doctors solving the case. The true diagnosis for this case is malaria, which is accurately predicted by Doc 1 and Doc 3, Doc 2 and 10 get it wrong (Note: Case shown is fictional).</figcaption></figure><p>The cases can be divided into three categories based on whether the true diagnosis is known and what the confidence is in the true diagnosis: (i) Expert Reviewed Cases (ERC)are educational in nature and have been reviewed by multiple expert physicians. (ii) Non-ERC cases are also created by clinicians but the credibility of the diagnosis listed is unknown, and needs to be determined. (iii) The third type of cases are uploaded by clinicians that are interested in but not aware of the true diagnosis. The dataset available to me contains 15% ERC cases, 85% non-ERC cases and none of the cases with unknown diagnosis.</p><p>The raw data is hand-labeled by experts to indicate when there is a match between the true diagnosis and predicted diagnosis (<strong><em>Figure 3, left panel</em></strong>). If there is a match between the true diagnosis and the first, second or third predicted diagnosis, the label is set to ‘1’, ‘2’ or ‘3’ respectively. If none of the predicted diagnoses match, the label is set to ‘-’. About 90% of solvers either get the diagnosis right in their first attempt(‘1’), or get it wrong (‘-’). For this reason, I only take the first prediction into account; if the solvers get the diagnosis right in their first attempt I set the label to 1 as before, else I set the label to 0 (<strong><em>Figure 3, right panel</em></strong>). Each solving doctor has a diagnostic accuracy of 49.5% — the baseline to beat for the machine learning model!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TjwVeT1JXhdBvkLxDAxurg.png" /><figcaption><strong>Figure 3: </strong>Plot showing counts for the labels in the raw data (<strong>left</strong>) and after relabeling (<strong>right</strong>).</figcaption></figure><p>Two case features, patient sex and patient age, show interesting trends (Figure 4). Female and male patients have slightly lower and higher than average diagnostic accuracy, respectively, and middle aged patients are easier to diagnose than either young or old patients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6WIxiA6WSBtBL2Sdkgt5Q.png" /><figcaption><strong>Figure 4:</strong> <strong>Left:</strong> Sex vs count. Males have a greater chance of successful diagnosis while females have a greater chance at a failed diagnosis. <strong>Right:</strong> Middle aged patients have highest diagnostic accuracy.</figcaption></figure><h3>Data Cleaning</h3><p>Predicted diagnoses are hand entered by solvers. Therefore, duplicate diagnoses have typos, variations (e.g.: British vs American spelling, abbreviations), and include or exclude details to appear non-identical. I used fuzzy matching to detect and replace non-identical duplicates. Some examples are shown in <strong><em>Table 1</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*gWwlNgkBaf4537-r_0GZ2g.png" /><figcaption><strong>Table 1</strong>: Examples of non-identical duplicates identified by fuzzy matching</figcaption></figure><h3>Feature Engineering</h3><p>As I am interested in predicting the accurate diagnosis for a case from the set of diagnoses inputted by the clinicians solving the case rather than the accuracy of any individual clinician, I create a new feature to utilize ‘wisdom of the crowd’ — I determine the number of doctors agreeing on each predicted diagnosis, ‘# doctors in agreement’(<strong><em>Figure 5</em></strong>). Fuzzy matching helps merge duplicate diagnoses that would’ve previously appeared non-identical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*lJ0H9MPhaEPHgB7vmsopiw.png" /><figcaption><strong>Figure 5: </strong>New feature # doctors in agreement created with the sample case from Figure 2. Six doctors predict the disorder is malaria, while three predict it is dengue fever and one doctor thinks it is influenza.</figcaption></figure><p>‘# doctors in agreement’ appears to have great predictive power (<strong><em>Figure 6</em></strong>). Cases with all 10 different opinions , i.e. a maximum of only one doctor agreeing on a diagnosis have a &lt; 20% diagnostic accuracy (<strong><em>Figure 6, leftmost bar</em></strong>), and cases where all 10 doctors agree on one diagnosis have a 100% diagnostic accuracy (<strong><em>Figure 6, rightmost bar</em></strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFXKx18UYUUIiOSwmWRsvg.png" /><figcaption><strong>Figure 6: </strong>More doctors in agreement, more accurate the diagnosis</figcaption></figure><p>I use this feature, ‘# doctors in agreement’ along with two features of the case, ‘patient age’ and ‘patient sex’, and one feature of the solving doctor, ‘the time the doctor takes to solve the case’ to train my model as these features correlate well with diagnostic accuracy. First I show in <strong><em>Figure 7</em></strong> that these features are not correlated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*7pG8Hyw7N_8nE3_4q5QN1Q.png" /><figcaption><strong>Figure 7: </strong>Features I use in the model are not correlated.</figcaption></figure><h3>Machine Learning</h3><p>I use only the ERC data (15%) for training and testing because of the high confidence in the quality of these cases. I use an 80:20 train:test split (<strong><em>Figure 8</em></strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*WIpisJU7aKfONWw2YM6Y_w.png" /><figcaption><strong>Figure 8: </strong>ERC (verified) cases (yellow + red) are 15% of the dataset and used to train (yellow) and test (red) the model.</figcaption></figure><p>I use logistic regression because: (i) I am interested in answering a binary question — is a predicted diagnosis the true diagnosis? (ii) logistic regression works with sparse features; and (iii) I am interested in an easily interpretable model and using logistic regression allows me to determine the probability the predicted diagnosis is the true diagnosis. This would allow me to seek out more solvers for cases where all predicted diagnoses have low probabilities of being the true diagnosis. The model initially achieved an accuracy of 73 ± 2%; fuzzy matching boosts the accuracy by another 6% to reach 79 ± 2% (<strong><em>Figure 9</em></strong>). This is 29% greater than the baseline, the prediction accuracy of a single doctor. I check the non-ERC data against my trained model to find the model achieves an accuracy of 80%, suggesting the credibility of the non-ERC cases is on par with the ERC cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q6dq-01xMTszPbQMYVX1IA.png" /><figcaption><strong>Figure 9: </strong>ERC (verified) cases are 15% of the dataset and used to train and test the model.</figcaption></figure><h3>Deliverables</h3><p>I built a tool that can:</p><ul><li>select the right diagnosis from the different diagnoses for new cases of unknown diagnosis &amp; also output the accuracy of the prediction; and</li><li>detect cases that have low diagnostic accuracy so the company can seek out more solvers and boost the accuracy.</li></ul><h3>Future Directions</h3><p>Although fuzzy matching identifies a lot of non-identical duplicate strings based on their similar patterns, because it is not based on the meaning, it misses synonyms (e.g., lung cancer and pulmonary cancer) and acronyms (e.g.: STEMI or myocardial infarction) that lack any pattern similarity. I propose the use of <a href="https://nlp.stanford.edu/projects/glove/">GloVe</a>. GloVe is meaning based and is trained on co-occurrence statistics in a corpus and can determine how related two words are. GloVe trained on wikipedia or a medical database should be able to better identify non-identical duplicates, increasing the accuracy of the model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf1a8d0a923a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/collectivemd/collectivemd-bf1a8d0a923a">Combining Multiple Diagnoses</a> was originally published in <a href="https://medium.com/collectivemd">CollectiveMD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>