<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by karan sindwani on Medium]]></title>
        <description><![CDATA[Stories by karan sindwani on Medium]]></description>
        <link>https://medium.com/@kskaran94?source=rss-d64890dbc998------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*wIRWRe5ffwhNFq2S-eDR6w.jpeg</url>
            <title>Stories by karan sindwani on Medium</title>
            <link>https://medium.com/@kskaran94?source=rss-d64890dbc998------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 14:33:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kskaran94/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to train an Image Classifier on TFRecord files]]></title>
            <link>https://medium.com/@kskaran94/how-to-train-an-image-classifier-on-tfrecord-files-97a98a6a1d7a?source=rss-d64890dbc998------2</link>
            <guid isPermaLink="false">https://medium.com/p/97a98a6a1d7a</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[karan sindwani]]></dc:creator>
            <pubDate>Thu, 24 Sep 2020 16:26:25 GMT</pubDate>
            <atom:updated>2020-09-24T16:26:25.977Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Si6WjJz7dxBsQD5Dl0dVhg.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@magda-ehlers-pexels?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Magda Ehlers</a> from <a href="https://www.pexels.com/photo/orange-cube-1340185/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><h4>Improving the performance of pipelines using TFRecords</h4><h3>Introduction to TFRecords</h3><p>TFRecords store a sequence of binary records, which are read linearly. They are useful format for storing data because they can be read efficiently. You can learn more about TFRecords <a href="https://www.tensorflow.org/tutorials/load_data/tfrecord">here</a>.</p><p>A binary file format for storage while working with voluminous data can have a significant impact on the performance of import pipeline and as a consequence on the training time of your model. Binary data takes up less space on disk, takes less time to copy and can be read much more efficiently from disk. This is especially true if your data is stored on spinning disks, due to the much lower read/write performance in comparison with SSDs.</p><p>Apart from performance, TFRecords are optimized for use with Tensorflow in multiple ways. Firstly, it makes it easy to combine multiple datasets and integrates seamlessly with the data import and preprocessing functionality provided by the library. Especially for datasets that are too large to be stored fully in memory this is an advantage as only the data that is required at the time (e.g. a batch) is loaded from disk and then processed. Another major advantage of TFRecords is that it is possible to store sequence data — for instance, a time series or word encodings — in a way that allows for very efficient and (from a coding perspective) convenient import of this type of data.</p><p>The next few sections focus on converting an image dataset to tfrecords, loading the data, model training and prediction.</p><h3>Data Download</h3><p>For this illustration, I shall be using a ubiquitous image dataset CIFAR-10.More details on the same can be found <a href="https://www.cs.toronto.edu/~kriz/cifar.html">here</a>. The code snippet below loads the CIFAR10 dataset directly from tensorflow.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a753edccd41872fe6310533792a4c6cc/href">https://medium.com/media/a753edccd41872fe6310533792a4c6cc/href</a></iframe><h3>Convert dataset to TFRecords</h3><p>A TFRecord file stores the data as a sequence of binary strings. We need to specify the structure of our data before writing it to the file.</p><p>We shall be using tf.train.Example for the same. While writing an image to a TFRecord, we need the image itself and the corresponding label . In the code snippet below , we are defining two features namely image and label within the tf.train.Example.</p><p>The image feature stores the image array as bytes and the label feature stores the label as int. You may choose to store the label as string also or you could store additional information such as height, width and depth.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0dfd0c5d8b22f75b37a237953564e2df/href">https://medium.com/media/0dfd0c5d8b22f75b37a237953564e2df/href</a></iframe><h3>Load the dataset</h3><p>The process of reading TFRecords is straightforward:</p><ol><li>Read the TFRecord using a tf.data.TFRecordDataset</li><li>Define the features you expect in the TFRecord by using <a href="https://www.tensorflow.org/api_docs/python/tf/FixedLenFeature">tf.FixedLenFeature</a> and <a href="https://www.tensorflow.org/api_docs/python/tf/VarLenFeature">tf.VarLenFeature</a>, depending on what has been defined during the definition of tf.train.Example.</li><li>Parse one tf.train.Example (one file) a time using <a href="https://www.tensorflow.org/api_docs/python/tf/parse_single_example">tf.parse_single_example</a>.</li><li>Shuffle the dataset and extract by batch_size</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42d84cb915bbcd9c3519c5f1c497bbb2/href">https://medium.com/media/42d84cb915bbcd9c3519c5f1c497bbb2/href</a></iframe><p><strong>Visualize input images</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2d365d5b108a9798f08e5579d2618bc/href">https://medium.com/media/f2d365d5b108a9798f08e5579d2618bc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u29fCw0uCEt7-ZiP1TmnsA.png" /><figcaption>Photo from CIFAR10 dataset (<a href="https://www.cs.toronto.edu/~kriz/cifar.html">https://www.cs.toronto.edu/~kriz/cifar.html</a>)</figcaption></figure><h3>Model Training and Prediction</h3><p>Since we are using a relatively small and balanced dataset with 10 classes, we decided to write a custom model architecture similar to Lenet.</p><p>Our evaluation metric will be accuracy, but one may choose to have other metrics like average precision or area under roc curve</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a4200bc037e654d77edbe114ee9f582/href">https://medium.com/media/7a4200bc037e654d77edbe114ee9f582/href</a></iframe><p>The next step would be fitting the model , saving it and predicting for test set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/430a563f8a50533216465d03dcf17c8e/href">https://medium.com/media/430a563f8a50533216465d03dcf17c8e/href</a></iframe><h3>Conclusion</h3><p>Though the task of converting raw data to TFRecords may seem arduous but it is worth the effort. Few notable advantages are:</p><ul><li>TFRecords occupy less disk space than raw jpegs/pngs/csvs.</li><li>TFRecords improves I/O performance as it takes less time to copy and can be read much more efficiently from disk</li><li>Tensorflow provides a seemless integration of TFRecords with import pipelines</li><li>It has the ability to store sequential data</li><li>There is no change required in the model fit and definition statements while working with TFRecords</li><li>As a consequence of improved I/O performance, TFRecords reduces the training time.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97a98a6a1d7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unsupervised Entity Resolution using Graphs]]></title>
            <link>https://medium.com/@kskaran94/unsupervised-entity-resolution-using-graphs-e12f5abae0ae?source=rss-d64890dbc998------2</link>
            <guid isPermaLink="false">https://medium.com/p/e12f5abae0ae</guid>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[karan sindwani]]></dc:creator>
            <pubDate>Thu, 26 Sep 2019 17:51:32 GMT</pubDate>
            <atom:updated>2021-06-26T07:01:58.568Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>What is Entity Resolution ?</strong></h3><p>Entity Resolution refers to the task of finding all mentions of same -real world entity within a knowledge base or across multiple knowledge bases. This can be achieved through linking and grouping.For example, there could be different ways of addressing the same person in text, different addresses for businesses, or photos of a particular object.</p><h3>Motivation for Entity Resolution</h3><p>In the modern world, the speed and volume of data has increased exponentially. Thus making inference across networks and semantic relationships between entities a greater challenge to overcome. Entity resolution can reduce the complexity by proposing canonicalised references to entities and deduplicating and linking entities. The applications of entity resolution are tremendous, particularly for public sector and federal datasets related to health, transportation, finance, law enforcement, and antiterrorism</p><h3>Tasks in Entity Resolution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*6Vm_6I9dB3VPZeUrq8QUpQ.jpeg" /><figcaption>Entity Resolution Tasks</figcaption></figure><p><strong>Deduplication</strong>: The process of clustering/grouping records or mentions that represent the same real world entity .</p><p><strong>Record Linkage</strong>: The process of matching records from one deduplicated knowledge base to another.</p><p><strong>Canonicalisation:</strong> The process of converting data with more than one possible representation into a standard form.</p><p><strong>Reference Matching</strong>: The process of matching noisy records to clean ones in a deduplicated reference table.</p><h3>Challenges For Entity Resolution</h3><p>One might face many challenges while performing entity resolution on a data set. The first one being due to poor quality of data available. This is mostly to presence of unstructured data or human maintained data such as such as different spellings (cancel and cancelling), missing values (e.g., missing price ) and ambiguity . The traditional pair-wise distance approaches are less effective on noisy datasets.</p><p>Additionally, the presence of one-to-many and many-to-many relation between entities creates a challenge. For instance, a product might be associated with many prices (normal or discount), and each manufacturer is associated with many products. The heterogeneous nature of relationships brings in an additional challenge when performing collective entity resolution : to determine which kind of relationship is best suited for resolving a particular type of entity.</p><h3><strong>Graphical Approach to Entity Resolution</strong></h3><p>In this section an unsupervised graphical solution is proposed to solve entity resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*5jkFbNmJjLji5wEu2UQ7Mg.png" /><figcaption>Illustration</figcaption></figure><h3>Problem Formulation</h3><p>Firstly, The original dataset is transformed into multiple K-type graphs , with nodes representing fields of the row entries of the dataset. Nodes are connected by weighted edges representing relationships between entities themselves as well as between entities and their attributes. The weights on edges within layer represent the similarity scores while edges between two layers are calculated using TFIDF score.</p><p>The goal is to reduce the entity resolution problem to a graph summarisation problem where the nodes representing different mentions of the same hidden entity are summarised</p><ul><li>k sets of <strong>super nodes</strong>, where each super node denotes a cluster of same type of vertices in the original graph (S)</li><li>(k C 2 ) sets of links where each weighted edge denotes the expected probability that one type of super node is connected with another type of super node (L)</li><li>k sets of “zoom-in” mappings such that each mapping denotes probabilistic mapping between vertices and super nodes.( C )</li></ul><p>One can postulate that super nodes in summary graph need to be coherent, in other words the nodes which are clustered in a super node should be similar to each other. The explanation behind this assumption is that different mentions of the same hidden entity needs to share some similarities otherwise the problem is infeasible.</p><p><strong>Inherent Similarity</strong> : Similarity between content of nodes</p><p><strong>Structural Similarity</strong> : Similar connectivity patterns in the multi-type graph</p><p>To accommodate for the first type of similarity, we define the following optimisation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*oNDkjRHwgJoVB3uJPQabNA.png" /><figcaption>equation_1</figcaption></figure><p>This objective function ensures that any summary graph in which two highly similar vertices (x; y) are not mapped to the same super node, incurs a penalty.</p><p>To accommodate for structural similarity, we define the following optimisation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*MNmph9rR2YrA4pB7r4RFTw.png" /><figcaption>equation_2</figcaption></figure><p>Next step would be to combine equation 1 and 2 to form equation 3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*APd3EoCt1zW6MSXwpXlj7g.png" /><figcaption>equation_3</figcaption></figure><p>Equation 3 tends to produce more super nodes. The first term becomes irrelevant if two nodes are very dissimilar (sim(x; y) 0), whereas the second term will tend to assign structurally dissimilar nodes to different super nodes. Furthermore, the second term favours a larger number of super nodes, whereas the first term tends to condense similar nodes as much as possible. These differences introduce some nontrivial tradeoffs in the optimisation process, which allow us to arrive at good summary graphs</p><h3>Methodology</h3><p><strong>Solution Overview</strong></p><p>In this section , a solution to equation 3 is introduced. The overview of the solution is as follows :</p><ul><li>Start with a random summary graph</li><li>Search for an improved summary graph with fewer super nodes, by crossing out one or many super nodes</li><li>The next step is to fix the number of super nodes and compute the vertex-to-clustering mapping C(Equation 4) and super links L (Equation 5)</li><li>These two procedures are performed alternately, until they reach a locally optimal summary graph</li></ul><p><strong>Graphs Summarisation with super nodes</strong></p><p>We assume that the number of super nodes in the summary graph is given. With this assumption, the vertex to super nodes mapping C and the connections among super nodes L can be computed with a standard multiplicative update.The intuition of the multiplicative rule is that whenever the solution is smaller than the local optimum, it multiplies with a larger value; otherwise, it multiplies with a smaller value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*h2Y-Giw99kP65LNEiYUb2g.png" /><figcaption>Equation_4</figcaption></figure><p>Equation 4 calculates the vertex to super nodes mapping.</p><p>Similarly equation 5 calculates connections among super nodes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*37YOAG4UHDc6hmbL2FnQSw.png" /><figcaption>Equation 5</figcaption></figure><p><strong>Searching for Optimal Number of Super Nodes</strong></p><p>A greedy local search algorithm is proposed that can automatically determine the number of super nodes. The intuition of the approach is to use a backward search procedure : starting with an initialisation of a summary graph, where each type of vertices is assigned to a maximum number of clusters, it repeatedly removes one or many super nodes from a summary graph with the lowest information.</p><h3>Limitations</h3><p>In this blog, a multi graph co-summarisation based method was proposed that simultaneously identifies entities and their connections.</p><p>The proposed method has some limitations. First, the quality of entity-resolution solution depends on the quality of the user-supplied same-type vertex similarity.</p><p>Second, if the same-type vertex similarity matrices and the observed graphs are very dense, the proposed algorithm is not scalable.</p><h3>References</h3><p>Unsupervised Entity Resolution on Multi-type Graphs. ISWC 2016–15th International Semantic Web Conference. 2016. Linhong Zhu and Majid Ghasemi-Gol and Pedro Szekely and Aram Galstyan and Knoblock, Craig A. <a href="http://usc-isi-i2.github.io/papers/zhu16-iswc.pdf">http://usc-isi-i2.github.io/papers/zhu16-iswc.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e12f5abae0ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Early Detection of Sepsis Using Physiological Data]]></title>
            <link>https://medium.com/@kskaran94/early-detection-of-sepsis-using-physiological-data-78d5f31fab9d?source=rss-d64890dbc998------2</link>
            <guid isPermaLink="false">https://medium.com/p/78d5f31fab9d</guid>
            <category><![CDATA[health-analytics]]></category>
            <category><![CDATA[sepsis]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[karan sindwani]]></dc:creator>
            <pubDate>Fri, 05 Jul 2019 20:25:44 GMT</pubDate>
            <atom:updated>2019-07-05T20:25:44.902Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Sh4bzDrvxVMIZl5l5qAy3w.jpeg" /></figure><h3><strong>What is Sepsis ?</strong></h3><p>Sepsis is a potentially life-threatening condition caused by the body’s response to an infection. In a usual case, the body releases chemicals into bloodstream to neutralise an infection. Sepsis occurs when the body’s response to these chemicals is out of balance, triggering changes that can damage multiple organ systems.</p><p>Sepsis is caused by infection and can happen to anyone. Sepsis is most common and most dangerous in:</p><ul><li>Older adults</li><li>Pregnant women</li><li>Children younger than 1</li><li>People who have chronic conditions, such as diabetes, kidney or lung disease, or cancer</li><li>People who have weakened immune systems</li></ul><h4><strong>Statistics</strong></h4><ul><li>In USA, 270,000 people die from sepsis each year</li><li>Internationally , 6 Million people die from sepsis each year</li><li>US hospitals spend 24 Billion each year on sepsis (13 % of Health Budget)</li><li>Each hour of delay in treatment can roughly increase mortality by 4–8 %</li></ul><blockquote>Source : <a href="https://www.mayoclinic.org/diseases-conditions/sepsis/symptoms-causes/syc-20351214">https://www.mayoclinic.org/diseases-conditions/sepsis/symptoms-causes/syc-20351214</a></blockquote><h3>Objective</h3><p>The goal of this blog is the early detection of sepsis using physiological data. The early prediction of sepsis is potentially life-saving, and we aim to predict sepsis 6 hours before the clinical prediction of sepsis. Conversely, the late prediction of sepsis is potentially life-threatening, and predicting sepsis in non-sepsis patients (or predicting sepsis very early in sepsis patients) consumes limited hospital resources.</p><h3>Challenge Data</h3><p>The Challenge data repository contains one file per patient (e.g., training/p00101.psv ).</p><p>Each training data file provides a table with measurements over time. Each column of the table provides a sequence of measurements over time (e.g., heart rate over several hours), where the header of the column describes the measurement. Each row of the table provides a collection of measurements at the same time (e.g., heart rate and oxygen level at the same time).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*Xvl9y9ogTdDF5VsnTNucNQ.png" /></figure><h4>Features:</h4><ul><li><strong>Vital Signs</strong> : Heart Rate, Temperature , Blood Pressure, Respiratory rate,</li><li><strong>Laboratory Values : </strong>Platelet Count, Glucose , Calcium etc</li><li><strong>Demographics : </strong>Age, Gender, Time in ICU , Hospital Admit time</li></ul><h4><strong>Label :</strong></h4><p>0 (Non-sepsis) and 1 (Sepsis)</p><blockquote>Note :Feature description and data can be downloaded from <a href="https://physionet.org/challenge/2019/">https://physionet.org/challenge/2019/</a></blockquote><h3>Plan Of Action</h3><p>The data for the problem is an hourly time sequence record for each patient. But the records do not have a time-label associated with them, so that opens the scope of interpreting it as a non-temporal problem (ignoring the time component)</p><p>There are two ways in which one can approach this problem:</p><ol><li><strong>Temporal Approach</strong> : Take into the account the time component for the data. Sepsis is diagnosed for each patient at each hour using the past data.</li><li><strong>Non-temporal Approach</strong> : Ignore the time component and treat record as independently and identically distributed. This approach would help in predicting Sepsis at each hour for any patient(with or without patient past data)</li></ol><p>For this blog I am going to talk about only the Non-temporal approach .</p><h3>Non-Temporal Approach</h3><p>In this approach we ignore the time component associated with each patient hourly record and treat them as independently and identically distributed.</p><h4>Train-Validation-Test -Split</h4><p>The data repository has data from two hospitals and a total of 40 thousand patients. The actual number of records would be higher as a patient could have stayed in the hospital for a variable amount of time.</p><p>Splitting these records to train , validation and test. While splitting I have made sure that each patient is fully contained in exactly one of the splits.</p><ul><li><strong>Train</strong> : 30K Patients</li><li><strong>Test</strong> : 5K Patients</li><li><strong>Validation</strong> : 5K Patients</li></ul><blockquote>Note : The script to divide the data to train -test-validation split can be found here <a href="https://github.com/kskaran94/Sepsis_Identification/blob/master/Data_Preprocessing.ipynb">https://github.com/kskaran94/Sepsis_Identification</a></blockquote><h3>Exploratory Data Analysis</h3><p>After performing descriptive data analysis on the train data, these were the concerns that highlighted</p><h4>Concerns</h4><p><strong>Extremely Imbalance data : </strong>As we can see from the bar plot, the records are extremely imbalanced (Less than 1 % vs 99 %+) with the minority class being Sepsis (1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/0*ljH0I24-0FrB1b_8" /><figcaption>Label Distribution</figcaption></figure><p><strong>Missing Data : </strong>High Percentage of missing data in most of the features</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/0*dX1CzolLIvfle4zH" /></figure><blockquote>Note : Detailed EDA and baseline can be found on <a href="https://github.com/kskaran94/Sepsis_Identification/blob/master/Data_Preprocessing.ipynb">https://github.com/kskaran94/Sepsis_Identification</a></blockquote><h4>Handling Class Imbalance</h4><p>There are various pre-defined ways of handling class imbalance in machine learning, which have proven to be successful in many scenarios.</p><p>However most of them can not be applied in this problem. As we have an extreme rare imbalance , undersampling the majority class would lead to 99% data loss.</p><p>Oversampling or SMOTE(Synthetic Minority Over-sampling Technique) can applied to the minority class, but it would not be a conceptually correct idea as we are dealing with real world health care records and we would want to preserve the original distribution of data .</p><p>For Sepsis Identification non-temporal approach, we are going ahead with the original distribution of data and choosing an appropriate evaluation metric for modeling the data.</p><h4>Handling Missing Data</h4><p>There are various pre-defined ways of imputing continuous data such as Median , Mean etc, which have proven to be successful in many scenarios.</p><p>However continuous data imputation can not be applied in this problem. It would not be a conceptually correct idea as we are dealing with real world health care records and we would want to preserve the original distribution of data .</p><p>For Sepsis Identification non-temporal approach, instead of imputing continuous features. We engineer categorical features out of existing continuous features and impute the missing with a new category.</p><h3>Feature Selection</h3><p>The data has 40 features which can broadly be classified into</p><ul><li><strong>Demographics</strong></li><li><strong>Vital Signs</strong></li><li><strong>Laboratory</strong> <strong>values</strong></li></ul><p>After doing research on Sepsis from credible sources like <a href="http://www.cdc.gov,/">www.cdc.gov,</a> Symptoms of Sepsis are High Fever, Abnormal Blood pressure, High respiratory rate. These symptoms give us a direction that features like Heart Rate, Temperature and Blood pressure may be important while predicting sepsis.</p><p>Also sepsis is mostly prevalent either in infant or Old patients. This makes age an important feature.</p><p>As pointed in the previous section, we cannot handle missing data in the usual way. And the features in the category of Laboratory values have 90% or more missing data. Imputing these features even after engineering them as categorical would lead to features with low variance. Hence they may not add much information to the model</p><p>Therefore the features with more than 80% of missing data are ignored.</p><h3>Feature Engineering</h3><p>The Feature engineering for the selected features has been described below</p><ul><li><strong>Heart Rate</strong> : Converted to Categorical (Normal, Abnormal, Missing)</li><li><strong>O2Stat</strong> : Converted to Categorical (Normal, Abnormal, Missing)</li><li><strong>Temperature</strong> : Converted to Categorical (Normal, Abnormal, Missing)</li><li><strong>SBP</strong>/<strong>DBP</strong> : Converted to Categorical (Normal, Abnormal, Missing)</li><li><strong>Respiratory_Rate</strong> : Converted to Categorical (Normal, Abnormal, Missing)</li><li><strong>Age</strong> : Converted to Categorical (Old,Infant, Child/Adult)</li><li><strong>Gender</strong> : Unchanged</li><li><strong>HospAdmTime(Hospital Admission Time)</strong> : Unchanged</li><li><strong>ICULOS(ICU Length of Stay)</strong>: Unchanged</li></ul><blockquote>Note : Code for Feature Selection and Engineering can be found on <a href="https://github.com/kskaran94/Sepsis_Identification/blob/master/Data_Preprocessing.ipynb">https://github.com/kskaran94/Sepsis_Identification</a></blockquote><h3>Model Training and Evaluation</h3><h4>Evaluation Metric</h4><p>Choosing a suitable/useful evaluation metric is important than we think. In this case choosing a metric like accuracy would not be useful, as a model which predicts majority class would have high accuracy.</p><p>For imbalance data problems one can choose either Average_precision or F1_weighted. Since they give a complete representation of the confusion matrix.</p><p>We will be going ahead with average precision as the primary metric but still look at other metric like precision and recall.</p><h4>Machine Learning Models</h4><p>Converting the Categorical features to OnehotEncoding and scaling the Continuous features.</p><p>Performance of different models can be seen in the table below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*LUazmaCc68o99jmK116N_A.png" /></figure><h4>Deep Learning Models</h4><p>Auto-encoders have proven to be useful for anomaly detection use-cases which involves high class imbalance.</p><p>This blog post offers a comprehensive approach for using auto-encoders for extremely rare classification</p><blockquote>Blog link : <a href="https://towardsdatascience.com/extreme-rare-event-classification-using-autoencoders-in-keras-a565b386f098">https://towardsdatascience.com/extreme-rare-event-classification-using-autoencoders-in-keras-a565b386f098</a></blockquote><p>Auto-encoders were expected to perform better than traditional Machine Learning models as they are modeling the behavior of positive class and treating the negative class as an anomaly.</p><p>Auto-encoders increased the average precision to 7 percent, which is a good number for the health care domain. Considering the case we are not compromising neither false positives nor false negatives.</p><p>Plotting a precision-recall curve shows that over any threshold , both precision and recall don’t take higher values. This explains our relatively low average precision value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*-8B5C7GBaVTWLWx9.png" /><figcaption>Precision-recall curve</figcaption></figure><p>We can confirm our thinking by plotting the reconstruction error for a single threshold. As we can see the red line (threshold) cannot perfectly divide the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*FfAFRJVyk4Tf_Mrm.png" /></figure><blockquote>Note : Code for Model Training and Evaluation can be found on <a href="https://github.com/kskaran94/Sepsis_Identification/blob/master/Data_Preprocessing.ipynb">https://github.com/kskaran94/Sepsis_Identification</a></blockquote><h3>Conclusion</h3><p>After looking at both Machine and Deep Learning models we can conclude that we need to add more features or data , for a better model performance. Or even switch to temporal approach for better performance.</p><blockquote>Connect with me on Linked-In : <a href="https://www.linkedin.com/in/karansindwani/">https://www.linkedin.com/in/karansindwani/</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78d5f31fab9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>