<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jill Cates on Medium]]></title>
        <description><![CDATA[Stories by Jill Cates on Medium]]></description>
        <link>https://medium.com/@topspinj?source=rss-f40e9e268985------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*4BHoFHF3XNp_avRD_dAdFw.jpeg</url>
            <title>Stories by Jill Cates on Medium</title>
            <link>https://medium.com/@topspinj?source=rss-f40e9e268985------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:49:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@topspinj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Feature Engineering of Electronic Medical Records]]></title>
            <link>https://medium.com/@topspinj/feature-engineering-of-electronic-medical-records-7447ee1c47b4?source=rss-f40e9e268985------2</link>
            <guid isPermaLink="false">https://medium.com/p/7447ee1c47b4</guid>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[electronic-medical-record]]></category>
            <dc:creator><![CDATA[Jill Cates]]></dc:creator>
            <pubDate>Fri, 04 Oct 2019 18:37:35 GMT</pubDate>
            <atom:updated>2019-10-04T18:37:35.610Z</atom:updated>
            <content:encoded><![CDATA[<h4>A comprehensive overview of data cleaning and feature engineering techniques for clinical data</h4><blockquote>This article is cross-posted on the BioSymetrics website. To see the original post, check out the BioSymetrics <a href="https://www.biosymetrics.com/#biosym-blog-hp">blog</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*n-A6H0eJsOAYuh_qfiyCzw.png" /></figure><p>The electronic medical record (EMR) is a digital version of a patient’s chart that collects data related to a patient’s visit such as past medical history, lab results, prescriptions, diagnosis, and patient reported outcomes. EMR data are notorious for being messy, incomplete, and inconsistent. Part of the “messiness” is due to the diverse nature of clinical data. A typical EMR consists of disparate datatypes including:</p><ul><li>unstructured text (e.g., clinical notes)</li><li>2D and 3D images (e.g., X-ray, MRI, fMRI)</li><li>time-series signals (e.g., ECG and EEG traces)</li><li>PDFs with text (e.g., lab reports)</li></ul><p>In this post, we will walk through several techniques for cleaning clinical data and engineering features from an EMR database.</p><h3>Data Standardization</h3><p>The first step of the data cleaning process is data standardization. Let’s say we are working with data stored on multiple EMRs. While interoperability standards ensure that there is consistency in patient record content, there is no official standard for how this content gets stored, which means that each EMR can have its own data schema. A recent report has estimated that a single hospital network will have on average 16 different EMR platforms in use at its affiliate institutions [<a href="https://www.healthcareitnews.com/news/why-ehr-data-interoperability-such-mess-3-charts">1</a>]. In order to analyze data across disparate EMR platforms, we need to transform it into a common format.</p><p>The Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM) is a framework that can help harmonize disparate data into a standardized format [<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3240764/">2</a>]. OMOP-CDM was developed by a group of international researchers coordinated through Columbia University, with the intention of developing a standard to facilitate reproducible research. It can be broken down into two steps:</p><ol><li><strong>Transforming the data schema into a standard format:</strong> Data should be wrangled into OMOP-CDM which consists of 15 clinical data tables. Some of these tables include PERSON (containing demographic information about the patient), OBSERVATION_PERIOD (containing timestamp of patient’s visit to the physician), and DRUG_EXPOSURE (containing prescription and over-the-counter medicines, vaccines, and large-molecule biologic therapies). For a detailed outline of the OMOP-CDM schema, you can check out the official wiki <a href="https://github.com/OHDSI/CommonDataModel/wiki">here</a>.</li><li><strong>Mapping content to standardized vocabularies:</strong> There are numerous ways to describe a clinical event. To map medical jargon to standardized vocabularies, the OMOP-CDM makes use of the U.S. National Library of Medicine’s <a href="https://www.nlm.nih.gov/research/umls/index.html">Unified Medical Language System (UMLS)</a> which contains medical ontologies such as SNOMED, RxNorm, ICD10, and LOINC.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*SYjqiQvnLdJfhKlwS519cQ.png" /></figure><p>This process can involve extensive manual data wrangling depending on how diverse the data structures are. To learn more about how to standardize EMR data using OMOP, check out <a href="https://github.com/OHDSI/CommonDataModel">this Github repo</a>.</p><h3>Diagnosis Codes</h3><p>International Statistical Classification of Diseases, also known as ICD, is a classification system developed by the World Health Organization that is used for describing patient diagnosis. At the end of a patient’s hospital stay, a list of ICD codes are assigned to that particular hospital admission, which ultimately gets used for reimbursement purposes. There are two versions of ICD:</p><ol><li><strong>ICD-9</strong>, which is now deprecated, contains 13,000 diagnosis codes.</li><li><strong>ICD-10</strong> contains 68,000 diagnosis codes and is considered the gold standard for classifying patient diagnoses. There are <a href="https://www.aapc.com/icd-10/codes/">resources</a> available for converting ICD-9 codes to ICD-10.</li></ol><p>Let’s say we’re building a clinical diagnosis model and we want to include “diagnosis” as a feature. Using the raw format of ICD codes, we would have a very sparse high-dimensional dataset, with each column representing a unique ICD code. To reduce data dimensionality, we can categorize ICD codes into a smaller subset of well-defined comorbidities using the Charlson and Elixhauser comorbidity mappings. Charlson collapses ICD codes into 17 comorbidites, while Elixhauser groups codes into 30 comorbidities. A complete list of Elixhauser and Charlson comorbidity categories can be found below:</p><ul><li><strong>Charlson: </strong>myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, chronic pulmonary disease, rheumatic disease, peptic ulcer disease, mild liver disease, diabetes without chronic complication, diabetes with chronic complication, Hemiplegia/paraplegia, renal disease, any malignancy, moderate/severe liver disease, metastatic solid tumour, AIDS/HIV</li><li><strong>Elixhauser: </strong>congestive heart failure, cardiac arrhythmia, valvular disease, pulmonary circulation disorders, peripheral vascular disorders, hypertension (uncomplicated), hypertension (complicated), paralysis, other neurological disorders, chronic pulmonary disease, diabetes (uncomplicated), diabetes (complicated), hypothyroidism, renal failure, liver disease, peptic ulcer disease, AIDS/HIV, lymphoma, metastatic cancer, solid tutor without metastasis, rheumatoid arthritis, coagulopathy, obesity, weight loss, fluid and electrolyte disorders, blood loss anemia, deficiency anemia, alcohol abuse, drug abuse, psychoses, depression</li></ul><p>Here’s an example of ICD-10 codes that fall under “peptic ulcer disease” as defined by Elixhauser’s comorbidity mapping:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*H-gfBTEPxvgslIl76ov5FQ.png" /><figcaption>ICD-10 codes for “peptic ulcer disease”</figcaption></figure><p>By mapping hyper-specific ICD diagnosis codes into well-defined comorbidities, we are reducing the dimensions of our dataset which effectively reduces noise and complexity. Several studies have suggested that the Elixhauser comorbidity measure has better predictive ability of patient outcome than Charlson [<a href="https://journals.lww.com/lww-medicalcare/Abstract/2004/04000/Comparison_of_the_Elixhauser_and_Charlson_Deyo.8.aspx">3</a>], [<a href="https://link.springer.com/article/10.1007/s11999-014-3686-7">4</a>], [<a href="https://onlinelibrary.wiley.com/doi/full/10.1002/cncr.25653">5</a>].</p><h3>Medication Data</h3><p>Free-text fields allow for rapid data entry by the physician. However, this type of unstructured data entry can lead to spelling errors and variations in abbreviations, which makes it difficult to analyze patient data. Prescription drugs are not always recorded in a standardized format in a patient’s record. There can be variability in naming convention across clinicians. For example, the blood thinning drug <em>Warfarin</em> can also be referred to as <em>Coumadin</em>. <em>Acetaminophen</em>, a common pain reliever drug, is also known by its brand name <em>Tylenol</em>. Intravenous normal saline, which is often administered to patients in the ICU, may be recorded as <em>NS</em> which stands for normal saline, or by its chemical description <em>0.9% sodium chloride</em>. An example of the many variations of drug names is illustrated in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*4k_mV-MYtHbCu4b0128w-w.png" /><figcaption>Spelling variations of acetaminophen and normal saline</figcaption></figure><h4>Normalizing Drug Names</h4><p>If we want to incorporate prescription drugs as a feature in our model, we need to find a way to map drugs into a universal nomenclature. Two options for standard drug nomenclature include:</p><ol><li><strong>National Drug Codes (NDC):</strong> NDC is a coding system consisting of 10 or 11 digits that uniquely identifies drugs approved by the U.S. Food and Drug Administration. All drugs in commercial distribution in the U.S. have a unique NDC. The NDC for 100 tablets of acetaminophen (Tylenol) is 69618–010–[01], while the NDC for 1000 tablets is 69618–010–[10]. The two NDCs are almost identical except for the last two digits.</li><li><strong>RxNorm:</strong> RxNorm represents medications by 3 properties: 1) ingredients, 2) strength, and 3) dose form. It was developed by the U.S. National Library of Medicine because there was a need for standard representation of medications [<a href="https://ieeexplore.ieee.org/abstract/document/1516084">6</a>]. Each unique combination of ingredient, strength, and dose form has a common unique identifier called RxCUI. For example, Acetaminophen 500 MG Oral Tablet (Tylenol) is represented by RxCUI 209459. The RxCUI for the same medication, but in 325 MG strength, is 209387.</li></ol><p>Though both nomenclatures are valid options for normalizing drug names, RxNorm appears to be more widely used than NDC for representing medications in a clinical context. Nelson et al. explains that NDC is less appealing because it contains very granular information about the manufacturer/packager which is generally not needed for clinical research [<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3128404/">7</a>]. Whichever nomenclature you decide to use, it’s important to make sure that drug names are normalized before proceeding to the feature engineering and modelling phase of your pipeline. To learn more about these drug nomenclatures, you can check out the <a href="https://rxnav.nlm.nih.gov/RxNormAPIs.html#">RxNorm API</a> which provides the RxCUI and NDC for a given drug, and also has other functionality such as returning spelling suggestions for a misspelled drug name.</p><h4>Extracting Features from Drugs</h4><p>As part of our data mining step, we may want to reduce the dimensions of our drug dataset. To cluster drugs, we can extract properties using the National Institute of Health’s medical subject headings (MeSH). Let’s say we want to know whether the patient was prescribed an antibiotic, but we only have the drug name. We can get relevant attributes of a drug by looking at its MeSH terms. For example, the MeSH terms for Tylenol are “Anti-Inflammatory Agents, Non-Steroidal” and “Analgesics, Non-Narcotic”, while amoxicillin’s MeSH terms are “Anti-Bacterial Agents” and “beta-Lactamase Inhibitors”. With this approach, we can identify antibiotic drugs by filtering those with MeSH term “Anti-Bacterial Agents”.</p><p>Another strategy for grouping drugs is the World Health Organization’s Anatomical Therapeutic Chemical (ATC) drug classification system. The ATC classifies drugs at 5 different levels, including the organ or system on which the drug acts, their therapeutic use, and the chemical properties of the drug. Each component of a drug’s ATC code represents a specific property of that drug. For example, the ATC code for amoxicillin is J01CR02 which can be broken down into the following levels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*D4UeYFXY2Sd_ZQVmwCDADA.png" /><figcaption>ATC code breakdown for amoxicillin (5 levels)</figcaption></figure><p>With the ATC approach, we can identify antibiotic drugs by filtering those that start with “J01”. There are open-source APIs available, such as <a href="https://rxnav.nlm.nih.gov/RxNormAPIs.html#">RxNorm</a> and <a href="https://open.fda.gov/apis/">OpenFDA</a>, that you can use to supplement your drug data with MeSH terms and ATC properties.</p><h3>Clinical Notes</h3><p>Clinical notes are a valuable source of information that can significantly improve the quality of a predictive diagnostic model. Ghassemi et al. found that clinical notes were the most meaningful data for improving prediction accuracy of in-hospital mortality [<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4185189/">8</a>]. Other studies have also reported a considerable improvement in model performance when using narrative text from clinical notes.</p><p>Since clinical notes are a form of unstructured data, we need to process this narrative text into structured features. Here, we will describe two natural language processing (NLP) techniques to generate structured data from unstructured text.</p><h4>Topic Modelling</h4><p>Topic modelling is a type of unsupervised learning that clusters similar groups of documents together based on shared attributes. Its goal is to discover “hidden topics” from a collection of documents. Prior to applying topic modelling to your clinical notes dataset, it’s a good idea to do some text pre-processing using a technique called term frequency-inverse document frequency (<a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a>). Here, we consider a “word” to be a “term”. TF-IDF is a term-weighting metric that evaluates the importance of a word based on its frequency within in a document. The more often a word appears in a document, the higher the TF-IDF score. Words that appear more frequently in general get down-weighted based on the number of documents they appear in. Once you apply TF-IDF to your clinical notes dataset, you will get a document-term matrix of shape (n_docs, n_terms) where each cell represents the TF-IDF weighted score of a given term in a document. We can use this document-term matrix to train our topic model.</p><p>There are two different techniques that perform topic modelling:</p><ul><li><strong>Latent Dirichlet Allocation (LDA): </strong>This probabilistic model is based on the assumption that 1) every document can be described by a mixture of topics weighted by probabilities, and 2) every topic is defined by a mixture of words [<a href="http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf">9</a>]. LDA uses hierarchical Bayesian modelling to simultaneously estimate the mixture of words associated with each document, and find the mixture of topics that describes each document.</li><li><strong>Non-negative Matrix Factorization (NMF): </strong>This linear algebraic algorithm uses matrix factorization to generate latent topics from a document-term matrix. With NMF, the original matrix (n_docs, n_terms) gets factorized into two factor matrices: a document-topic matrix (n_docs, n_topics) and a topic-term matrix (n_topics, n_terms).</li></ul><p>In both LDA and NMF, you need to define the number of topics that you want to generate. This hyperparameter can be tuned using metrics such as model perplexity and topic coherence to find the optimal number of topics that best suits the dataset at hand. If you’re using Python, take a look at <a href="https://radimrehurek.com/gensim/">gensim</a> and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html">scikit-learn</a> libraries which both have modules for performing TF-IDF, LDA, and NMF.</p><h4>Named Entity Recognition Model</h4><p>Named Entity Recognition (NER) classifies named entities from unstructured text into pre-defined categories such as persons, locations, organizations, and products. Some examples of relevant clinical named entities are symptom, diagnosis, medication, treatment, procedure, and anatomy. A patient’s medical history and presenting symptoms are typically described in the form of clinical notes, so NER is a useful tool for extracting patient-related features from this text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*I0lgXdeJawNeOoqynrPH4g.png" /><figcaption>Named Entity Recognition on a clinical note using AWS Comprehend Medical</figcaption></figure><p>If you’re looking to apply NER to unstructured text, there are several approaches to choose from: a rule-based approach (e.g., regular expressions), unsupervised learning (e.g., clustering), supervised learning (e.g., <a href="https://en.wikipedia.org/wiki/Conditional_random_field">conditional random fields</a>), or deep learning (e.g., <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">long short-term memory</a>). <a href="https://spacy.io/">spaCy</a> is a useful open-source NLP package that implements NER using a deep learning technique involving residual convolutional neural networks (CNNs). Amazon (AWS) recently launched <a href="https://aws.amazon.com/comprehend/medical/">Comprehend Medical</a>, which highlights key medical terms from clinical text using a biomedical-specific NER model. These off-the-shelf solutions are a great start to applying NER to your own dataset.</p><h3>Wrap Up</h3><p>Machine learning has massive potential to enhance clinical decision-making and improve patient outcomes. A major barrier to applying machine learning in healthcare is the complex, unstructured nature of patient data. This post covers various tools and techniques for standardizing, cleaning, and extracting features from electronic medical records. These steps are a crucial part of the machine learning pipeline, and play an important role in building robust predictive diagnostic models.</p><h3><strong>Take-Aways</strong></h3><ul><li>Electronic medical records are messy and need to be standardized, cleaned, and processed before being fed into a machine learning model.</li><li>The OMOP Common Data Model is a framework for standardizing data from multiple EMRs into a common format.</li><li>Charlson and Elixhauser comorbidity mappings help reduce diagnosis codes into well-defined comorbidity categories.</li><li>RxNorm and NDC are two nomenclatures for normalizing drug names.</li><li>MeSH and ATC are classification systems that can be used to extract properties from a given drug.</li><li>Clinical notes contain valuable information about the patient but requires processing to convert unstructured text into structured features.</li><li>Topic modelling is a text-mining technique that can be implemented in two ways (LDA and NMF) to help extract hidden topics from unstructured text.</li></ul><h3>References</h3><ol><li>“Why EHR Data Interoperability Is Such a Mess in 3 Charts | Healthcare IT News.” n.d. Accessed September 30, 2019. <a href="https://www.healthcareitnews.com/news/why-ehr-data-interoperability-such-mess-3-charts">https://www.healthcareitnews.com/news/why-ehr-data-interoperability-such-mess-3-charts</a>.</li><li>Overhage, J Marc, Patrick B Ryan, Christian G Reich, Abraham G Hartzema, and Paul E Stang. 2012. “Validation of a Common Data Model for Active Safety Surveillance Research.” <em>Journal of the American Medical Informatics Association : JAMIA</em> 19 (1): 54–60.</li><li>Southern, Danielle A., Hude Quan, and William A. Ghali. 2004. “Comparison of the Elixhauser and Charlson/Deyo Methods of Comorbidity Measurement in Administrative Data.” <em>Medical Care</em> 42 (4): 355.</li><li>“The Elixhauser Comorbidity Method Outperforms the Charlson Index in Predicting Inpatient Death After Orthopaedic Surgery | SpringerLink.” n.d. Accessed September 30, 2019. <a href="https://link.springer.com/article/10.1007/s11999-014-3686-7">https://link.springer.com/article/10.1007/s11999-014-3686-7</a>.</li><li>Lieffers, Jessica R., Vickie E. Baracos, Marcy Winget, and Konrad Fassbender. 2011. “A Comparison of Charlson and Elixhauser Comorbidity Measures to Predict Colorectal Cancer Survival Using Administrative Health Data.” <em>Cancer</em> 117 (9): 1957–65.</li><li>Liu, S., Wei Ma, R. Moore, V. Ganesan, and S. Nelson. 2005. “RxNorm: Prescription for Electronic Drug Information Exchange.” <em>IT Professional</em> 7 (5): 17–23.</li><li>Nelson, Stuart J, Kelly Zeng, John Kilbourne, Tammy Powell, and Robin Moore. 2011. “Normalized Names for Clinical Drugs: RxNorm at 6 Years.” <em>Journal of the American Medical Informatics Association : JAMIA</em> 18 (4): 441–48.</li><li>Ghassemi, Marzyeh, Tristan Naumann, Finale Doshi-Velez, Nicole Brimmer, Rohit Joshi, Anna Rumshisky, and Peter Szolovits. 2014. “Unfolding Physiological State: Mortality Modelling in Intensive Care Units.” <em>KDD : Proceedings / International Conference on Knowledge Discovery &amp; Data Mining. International Conference on Knowledge Discovery &amp; Data Mining</em> 2014 (August): 75–84.</li><li>Blei, David M., Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. <em>Journal of Machine Learning Research</em> 3 (2003): 993–1022.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7447ee1c47b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build a Recommender Engine for Medical Research Papers]]></title>
            <link>https://medium.com/@topspinj/how-to-build-a-recommender-engine-for-medical-research-papers-a9dc1bad12f2?source=rss-f40e9e268985------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9dc1bad12f2</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[collaborative-filtering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jill Cates]]></dc:creator>
            <pubDate>Fri, 16 Aug 2019 15:10:17 GMT</pubDate>
            <atom:updated>2019-08-16T20:26:50.267Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step guide to building a recommender pipeline, from data wrangling to model evaluation</h4><p>In 2006, Netflix, which was then a DVD rental service, announced a data science competition for movie rating predictions. The company would offer a $1 million grand prize to the team that could improve their existing recommender system’s prediction accuracy by 10%. The competition garnered much interest from researchers and engineers in both academia and industry. Within the first year of the competition, over 40,000 teams from more than 100 countries had entered the competition [<a href="https://en.wikipedia.org/wiki/Netflix_Prize#2007_Progress_Prize">1</a>].</p><p>In June 2009, the prize was awarded to BellKor’s Pragmatic Chaos, a team of AT&amp;T engineers, who submitted the winning algorithm a few minutes earlier than the second-place team [<a href="https://www.wired.com/2009/09/bellkors-pragmatic-chaos-wins-1-million-netflix-prize/">2</a>]. The paper describing their solution can be found <a href="https://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKor.pdf">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*pHlsMYXIqn6NerW-rzD_pQ.jpeg" /><figcaption>BellKor’s Pragmatic Chaos, the winning team of the Netflix Challenge (2009)</figcaption></figure><p>A decade later, recommender systems have become an active topic of research. It’s now being taught in computer science courses [<a href="https://www.coursera.org/specializations/recommender-systems">3</a>] and discussed in international conferences [<a href="https://recsys.acm.org/">4</a>]. A quick Google search of “recommender systems” will give you an overwhelming list of tutorials, papers, book chapters, and videos about the topic. However, much of this content focuses on the theory around recommenders, rather than practical application. This post is a step-by-step practical guide on how to build a recommender pipeline, using a case study of medical research papers.</p><h3>Overview</h3><ol><li>The Problem and Dataset</li><li>Data Pre-Processing</li><li>Model Selection</li><li>Hyperparameter Tuning</li><li>Model Training and Evaluation</li><li>Data Post-Processing</li></ol><h3>The Problem and Dataset</h3><p>In 2018, I collaborated with a team of data scientists to build a recommender system for <a href="https://qxmd.com/read-by-qxmd">QxMD Read</a> — a mobile application that lets healthcare professionals stay up-to-date with the latest research in medicine. We were presented with the challenge of generating personalized recommendations for community-curated collections. A “collection” can be thought of as a Spotify “playlist” for research papers. It contains a selection of research papers hand-picked by a user of the QxMD platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*WmAiZ9rkBpi0mb-MUB9VPw.png" /><figcaption>QxMD’s “collection” is analogous to Spotify’s “playlist”</figcaption></figure><p>This problem was more complex than a traditional recommendation system because we were dealing with user-curated collections (i.e., playlists for papers), which could be updated at any time. We had data on which collections a user had subscribed to, and which papers a user had opened and read. While we knew when a paper was read and for how long, we did not have information on how the paper was accessed (i.e., via a collection, user’s home page, or search result). Our challenge was to identify relevant collections for users using their interaction data at the paper-level (rather than collection-level).</p><h3><strong>Step 1: Data Pre-Processing</strong></h3><p>Our core recommender system was a collaborative filtering model, which requires data to be in the form of a user-item or “utility” matrix. In a utility matrix, each cell represents a user’s degree of preference towards a given item. User preference can be represented as explicit feedback (e.g., movie ratings) or implicit feedback (e.g., number of times a song was replayed). Though implicit feedback is inherently noisy, its quantity tends to outweigh the quantity of explicit feedback, which helps in building a more robust recommender system.</p><p>We were working with an implicit feedback dataset, so each cell in our utility matrix represented a user’s interaction with a given collection. This can be thought of as our response variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*zNuPDMLKthKb3UCnsHxXUA.png" /></figure><p>We tested different types of preference metrics such as:</p><ul><li>whether a user proceeded to read the full-text paper from the abstract</li><li>whether a user shared a paper</li><li>whether a user “favourited” a paper</li><li>time spent reading a paper (dwell time) [<a href="http://www.hongliangjie.com/publications/recsys2014.pdf">5</a>]</li></ul><p>You can also define your own “preference score” based on what type of activity you want to optimize. If the goal is to get users to read more paper abstracts and share more frequently, you can construct a score that takes into account a user’s abstract views and sharing activity. To incorporate temporal information, you can modify for your score to give more weight to more recent interactions [<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.84.5471&amp;rep=rep1&amp;type=pdf">6</a>].</p><p>As part of our initial solution, we used number of abstract and full-text paper reads as our preference score. If a user has read many papers within a collection, we assumed that this was a strong signal of user preference. We identified outliers in our dataset as users with unusual bot-like behaviour, having thousands of “shares” and paper reads in a single day. These users were removed from the utility matrix. Our resulting utility matrix consisted of around 500,000 users and 10,000 collections, with a density of ~1%. As the matrix becomes more dense, recommendations tend to improve in quality.</p><h3>Step 2: Model Selection</h3><p>We considered two different models that handle implicit feedback data: 1) Alternating Least Squares, and 2) Bayesian Personalized Ranking.</p><h4>1. Alternating Least Squares</h4><p>Alternating Least Squares (ALS) uses a technique called “matrix factorization” to generate item recommendations for a set of users [<a href="http://Collaborative Filtering for Implicit Feedback Datasets">7</a>]. Matrix factorization involves decomposing a matrix, R, into two lower dimensional factor matrices, U and V, such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*Q-sE6rZ-asiYNb7kTI8ZaQ.png" /></figure><p>where “k” represents latent factors. By multiplying latent factor matrices <em>U</em> and <em>V</em>, we get a reconstructed matrix<em> R</em> with populated values representing estimated user ratings (or interaction score since we’re dealing with implicit feedback). In traditional matrix factorization, we solve for U and V simultaneously. With ALS, we fix U and compute V, then proceed to fix V and compute U. This process is repeated until the loss function converges. What makes ALS appealing is that it performs extremely well given sufficient data and it’s parallelizable, which makes it relatively inexpensive to tune hyperparameters.</p><h4>2. Bayesian Personalized Ranking</h4><p>Bayesian Personalized Ranking (BPR) uses a technique called “learning to rank” (LTR) to generate a personalized ranked list of items [<a href="https://arxiv.org/pdf/1205.2618.pdf">8</a>]. LTR aims to optimize the order of a list of items. It doesn’t care as much about the predicted preference score (e.g., rating) of a given item, but rather the relative ordering among a set of items. LTR can be used in any application that involves ranking — for example, search engines, online advertising, and recommender systems. BPR is a type of LTR algorithm that was developed specifically for recommender systems that use implicit feedback. BPR is based on the assumption that a user prefers positive items (i.e., observed items) over non-observed items. The goal is to rank items in order of preference by maximizing the likelihood that user <em>u</em> prefers item <em>i </em>over item <em>j</em>. For more information on the technical implementation of BPR, check out <a href="https://arxiv.org/pdf/1205.2618.pdf">this paper</a> by Rendle et al. [<a href="https://arxiv.org/pdf/1205.2618.pdf">8</a>]. BPR can be implemented using matrix factorization, which means that it can be run over a distributed system like Spark [<a href="https://stanford.edu/~rezab/classes/cme323/S16/projects_reports/rodrigo_oliveira.pdf">9</a>].</p><p>We ultimately chose <strong>Alternating Least Squares</strong> as our model for the recommender pipeline because it was 1) easier to implement, 2) computationally efficient, and 3) generated impressive results based on a smaller proof-of-concept prototype.</p><h3>Step 3: Hyperparameter Tuning</h3><p>Hyperparameter tuning is a crucial step in the machine learning pipeline that seeks to optimize a model’s performance by choosing optimal hyperparameter values. As part of this step, you need to decide which evaluation metric you want to optimize. There are several evaluation metrics that can be used for recommender systems including precision@k, recall@k, and root mean squared error (RMSE). We will use precision@10, which measures the proportion of relevant items that are found in a user’s top 10 recommendations. Here, we define “relevance” as whether a user had previously interacted with a recommended item. If a user did interact with that item and it appeared in their top 10 recommendations, we assumed that this was a good indicator that the recommendation was “relevant” to the user. Our goal is to find the optimal combination of hyperparameters that maximizes the precision of a user’s top 10 recommendations.</p><p>We used a Python package called <a href="https://scikit-optimize.github.io/">scikit-optimize</a> to tune the hyperparameters of our ALS model. If you take a look at scikit-optimize’s <a href="https://scikit-optimize.github.io/">documentation</a>, you’ll learn that it’s based on sequential model-based optimization (SMBO), which is more efficient and less exhaustive than other optimization techniques such as grid search. With SMBO, results from the previous iteration are used to improve the sampling method of the next iteration over a defined search space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*kYrvl-j3-HNqprld7caeqg.png" /><figcaption>Sequential model-based optimization using scikit-optimize</figcaption></figure><p>The plot above illustrates how <em>scikit-optimize</em> works. In this example, there are two hyperparameters that are being optimized:</p><ol><li>number of latent factors (int)</li><li>regularization factor (float, ranging from 0 to 1)</li></ol><p>Each dot represents one iteration of the optimization process. Earlier iterations are represented by purple dots, while later iterations are represented by green and yellow dots. The red dot represents the optimal value of a given hyperparameter. As shown above, yellow dots are clustered around the red dot which suggests that as the number of iterations increases, we converge toward the optimal hyperparameter value.</p><p>Hyperparameter tuning is a dynamic process that needs to be re-run as more data enters the pipeline. Constantly tuning hyperparameters on the fly can get computationally expensive, especially if you’re dealing with a large dataset. To reduce the computation costs of hyperparameter tuning, you can set up a cron job that runs the tuning process at a reasonable time interval (e.g., hourly, daily, weekly). You can also use the “warm-start” technique which lets you initialize the current tuning process with input points calculated from a previous tuning process [<a href="https://arxiv.org/abs/1710.06219">10</a>]. In other words, with warm-starting, you can continue the tuning process from where it left off.</p><h3>Step 4: Model Training and Evaluation</h3><p>After identifying the optimal number of latent factors and regularization factor, we plugged these values into Alternating Least Squares and ran it on the entire dataset. We evaluated the recommendations generated by this model both quantitatively and qualitatively.</p><h4>Quantitative Evaluation</h4><p>We first calculated precision@k and recall@k where k=10. Our average precision@10 across all users was around 0.65, which means that approximately 65% of recommendations were considered to be “relevant” to the user. This is a positive indicator of a well-performing recommender system. Recall@10 was harder to interpret because some users were relatively new to the system and therefore had fewer “relevant” collections, which resulted in lower recall@10 scores due to limited data.</p><h4>Qualitative Evaluation</h4><p>While precision@k and recall@k are good metrics for quantitative evaluation, it is difficult to truly assess the quality of recommendations. The ultimate metric for evaluating the quality of a recommender system is A/B testing. We built a dashboard using <a href="https://bokeh.pydata.org/en/latest/">Bokeh</a> that renders a user’s profile with their professional and specialty, along with their top recommendations. With this dashboard, we performed usability testing to ensure that recommendations made sense and weren’t completely off. For example, it would be worrisome if an orthopaedic surgeon whose reading history was focused on sports medicine was recommended papers in the field of gynaecology.</p><h3>Step 5: Data Post-Processing</h3><p>The last step in our pipeline is to clean up our results. We filtered out collections based on the following criteria: 1) whether a user already subscribed to a collection, 2) whether a user had previously subscribed but subsequently unsubscribed to a collection, and 3) whether a user had access privileges to papers within a collection. For example, a user with limited journal access would be recommended collections with a higher proportion of open-source papers.</p><p>Another post-processing task was to generate more descriptive titles of our collections. Since collections are curated by users of QxMD, a title may not accurately represent the content of that collection. For example, let’s say a paediatric resident’s recommended collections are titled “Aa”, “Neo”, and “Pediatrià”. While the content of these collections may be highly relevant, the ambiguous titles may discourage the user from opening the collection and taking a look at the papers within it. To solve this issue, we used a technique called “term frequency–inverse document frequency” (TF-IDF) to extract the most relevant topics from a given collection. To solve this issue, we used medical subject headings (MeSH) to extract the most relevant topics from a given collection.</p><p>Upon publication, a peer-reviewed medical research paper will get assigned a set of medical subject headings (MeSH). These MeSH terms are essentially keywords that represent the main topics of a given paper. A collection can contain anywhere from a few to over 100 papers, with each paper having several MeSH terms. We can collect these MeSH terms and perform a technique called “term frequency–inverse document frequency” (<a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a>) to identify the most pertinent MeSH terms that describe the collection.</p><p>TF-IDF is a term-weighting schema that is often used for information retrieval and text mining. It works by looking at the frequency of a term in a given collection, and comparing it to the frequency across all collections. As the frequency of a terms increases within a given collection, its TF-IDF value increases which indicates that the term of interest is highly relevant to that collection. A term’s TF-IDF value is offset by the number of other collections that also contain this term, which ensures that we don’t overvalue terms that appear more frequently in general. We used <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html">scikit-learn</a> to implement TF-IDF and chose the top 5 most relevant terms to describe a collection. The figure below is an example of the results from this process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*gwxsHlfouxnm1BY7QpyNdQ.png" /></figure><p>This MeSH feature will be deployed to production once appropriate changes to the UI have been made.</p><h3>Wrap Up</h3><p>This post outlines the process of building a collaborative filtering recommender engine for medical research papers. We walked through how to create a “preference score” using implicit feedback, how to transform this data into a user-item matrix, hyperparameter tuning using sequential-based model optimization, and the process of describing collections of papers with MeSH terms using TF-IDF. We used medical research papers as a case study for building a recommender engine, but most of the techniques described in this post can be applied to any dataset.</p><h3>References</h3><ol><li><a href="https://en.wikipedia.org/wiki/Netflix_Prize#2007_Progress_Prize">Netflix Prize Wikipedia Page</a></li><li><a href="https://www.wired.com/2009/09/bellkors-pragmatic-chaos-wins-1-million-netflix-prize/">BellKors Pragmatic Chaos Wins $1 Million Netflix Prize by Mere Minutes. Wired 2009.</a></li><li><a href="https://www.coursera.org/specializations/recommender-systems">Recommender Systems on Coursera</a></li><li><a href="https://recsys.acm.org/">ACM RecSys Conference</a></li><li><a href="http://www.hongliangjie.com/publications/recsys2014.pdf">Yi et al. 2014. Beyond Clicks: Dwell Time for Personalization.</a></li><li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.84.5471&amp;rep=rep1&amp;type=pdf">Lee et al. 2006. A Time-based Recommender System using Implicit Feedback.</a></li><li><a href="http://yifanhu.net/PUB/cf.pdf">Hu et al. 2008. Collaborative Filtering for Implicit Feedback Datasets</a></li><li><a href="https://arxiv.org/pdf/1205.2618.pdf">Rendle et al. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback</a></li><li><a href="https://stanford.edu/~rezab/classes/cme323/S16/projects_reports/rodrigo_oliveira.pdf">Rodrigo et al. 2016. Distributed Bayesian Personalized Ranking in Spark</a></li><li><a href="https://arxiv.org/abs/1710.06219">Kim et al. 2017. Learning to Warm-Start Bayesian Hyperparameter Optimization</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9dc1bad12f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recommender Systems 101]]></title>
            <link>https://medium.com/@topspinj/recommender-systems-101-bcbdfbe1e6e7?source=rss-f40e9e268985------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcbdfbe1e6e7</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[collaborative-filtering]]></category>
            <dc:creator><![CDATA[Jill Cates]]></dc:creator>
            <pubDate>Tue, 06 Aug 2019 14:36:18 GMT</pubDate>
            <atom:updated>2019-08-06T19:54:57.493Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This is a general introduction to recommender systems. For more information on recommenders, check out my <a href="https://github.com/topspinj/recommender-tutorial">tutorial series</a> which walks through how to implement a recommender system of your own using Python.</blockquote><p>If you’ve ever had a social media account (Facebook, Twitter), purchased something online (Amazon), or consumed content from a streaming platform (Spotify, Netflix), there’s a good chance that you’ve been influenced by a recommender system. Recommender systems help drive user engagement on these platforms by generating personalized recommendations based on a user’s past behaviour. Some recommendations that we may encounter in real-life include:</p><ul><li>“Customers who bought this item also bought…” (Amazon)</li><li>“Because you watched this show…” (Netflix)</li><li>Made for You: Discover Weekly playlist (Spotify)</li><li>“Based on your reading history…” (News article platform)</li><li>“You and Bob are 99% compatible” (OkCupid)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*2XEHaME4lxNI5V8aUWR-CQ.png" /></figure><p>Behind the scenes, a recommender system is simply a machine learning algorithm that predicts future behaviour based on a user’s historical preferences. It looks at demographic data, behavioural data, and item attributes to predict which items a user will find most relevant.</p><p>In this post, we will cover the motivation behind recommender systems, the difference between implicit and explicit feedback, collaborative filtering and content-based filtering, and techniques used to evaluate recommender systems.</p><h3>Motivation Behind Recommender Systems</h3><p>In order to understand the motivation behind recommender systems, we need to look back at the history of e-commerce.</p><p>Before the rise of e-commerce, things were sold exclusively in brick-and-mortar stores. A store’s inventory was limited to the physical space of the store and products that didn’t sell well were unprofitable. With a fixed inventory, merchants were motivated to sell only the most popular mainstream products.</p><p>In the mid 1990s, the introduction of the online marketplace revolutionized the retail industry. This new type of digital marketplace allowed for unlimited inventory, which meant that merchants could expand their product offerings to less mainstream, more niche items. As Chris Anderson describes in The Long Tail, “the mass market is turning into a mass of niches”. By overcoming the limitations of inventory, niche products are able to outsell the bestsellers. As a result, e-commerce businesses are seeing more value in niche products than before.</p><p>That being said, having a massive selection of offerings doesn’t necessarily mean that users will buy more stuff. In 2000, two psychologists from Columbia and Stanford University conducted a study at a supermarket tasting booth (see paper<a href="https://www.researchgate.net/profile/Mark_Lepper/publication/12189991_When_Choice_is_Demotivating_Can_One_Desire_Too_Much_of_a_Good_Thing/links/56107d7d08ae6b29b49c75fa/When-Choice-is-Demotivating-Can-One-Desire-Too-Much-of-a-Good-Thing.pdf"> here</a>). One booth had 6 samples of jam; the other had 24.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*7KsVSkzV5c07bzHzSLvWMQ.png" /><figcaption>The Tasting Booth Experiment (Iyengar and Lepper, 2000)</figcaption></figure><p>While the booth with more samples attracted more customers, the booth with fewer samples had a higher conversion rate. In fact, people who stopped at the booth with fewer samples were 10 times more likely to follow through with a jam purchase as compared to people who sampled from the booth with more options. This observation is known as “choice overload”: if a person is presented with too many options, they are less likely to buy. Choice overload extends beyond grocery shopping and can affect the decisions people make in online retail too. Recommender systems alleviate choice overload by reducing the search space and identifying the most relevant, high-quality items for users.</p><h3>Implicit vs. Explicit Feedback</h3><p>Recommender systems are fuelled by user feedback. As we collect information on what a user likes and dislikes, we are able generate more relevant recommendations tailored to the tastes of that user.</p><p>User feedback comes in two forms: 1) explicit feedback, and 2) implicit feedback. Explicit feedback is when a user directly likes or rates an item. Examples of explicit feedback include rating a movie on a 1–5 Likert scale, “liking” a friend’s Facebook post, or writing a positive review for a restaurant. Implicit feedback is more subtle and attempts to infer a user’s preferences based on their indirect behaviour. Examples of implicit feedback include a user’s purchase history, how many times they replayed a song, how quickly they binge-watched a series, or how long they read an article for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*9Wzd9H2cg11_BtGJybTYXQ.png" /><figcaption>User feedback comes in two forms: explicit and implicit feedback</figcaption></figure><p>By feeding user preferences into a recommender system, we get predictions about a user’s future behaviour. Which songs will they like next? Which movies will they binge watch? Which products will they buy next? These predictions are called recommendations.</p><h3>Collaborative vs. Content-Based Filtering</h3><p>The two most common recommender system techniques are: 1) collaborative filtering, and 2) content-based filtering.<br> <br><strong>Collaborative filtering</strong> is based on the concept of “homophily” — similar people like similar things. The goal is to predict a user’s preferences based on the feedback of similar users.</p><p>Collaborative filtering uses a user-item matrix (also known as a “utility” matrix) to generate recommendations. This matrix is populated with values that indicate a user’s degree of preference towards a given item. These values can represent explicit feedback, implicit feedback, or a hybrid of both. It’s very unlikely that a user has interacted with every item, so in most cases, the user-item matrix is very sparse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*I0gJu52xkxXwgaZWYl8aLw.png" /><figcaption>User-item (utility) Matrix</figcaption></figure><p>In the user-item matrix above, we see that Anne and Jim have very similar tastes in movies. For example, they both like “Titanic” and “A Star Is Born” but dislike “Harry Potter” and “Shawshank Redemption”. Anne hasn’t watched “King Kong” yet but Jim has and he rated it highly. Since Anne and Jim have similar rating patterns, we can assume that Anne will also rate King Kong highly and thus, we should recommend this movie to her. This process of comparing rating patterns among users is called collaborative filtering.</p><p>Given sufficient data, collaborative filtering performs extremely well. However, in cases where we have a lot of new users and new items, we run into an issue called the <a href="https://en.wikipedia.org/wiki/Cold_start_(computing)">cold-start problem</a>. Collaborative filtering won’t work on users and items that don’t have any interaction data. To overcome this problem, we can use a technique called content-based filtering</p><p><strong>Content-based filtering </strong>handles the cold-start problem because it doesn’t need any interaction data to generate recommendations. Instead, it looks at user features (e.g., age, gender, spoken language, employment, etc.) and item features (e.g., movie genre, release date, cast).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*ZFzdYev6sTRsMDEZco_jPw.png" /><figcaption>User and Item Features</figcaption></figure><p>Let’s say a new movie is out and doesn’t have any ratings yet. We can look at its features such as genre and cast to find similar movies with those features. To measure similarity, we first need to represent movies in a n-dimensional space where <em>n </em>represents number of movie features. We can then measure the distance between movies in this n-dimensional space using similarity metrics such as Euclidean distance, cosine similarity, Pearson correlation, and Jaccard distance. The smaller the distance between two movies, the more similar they are to one another.</p><p>Suggesting similar movies based on a movie you recently watched is called an item-item recommendation. Examples of this type of recommendation include “Because you watched this movie”, “More Like This”, “Customers who bought this also bought these”, etc. With content-based filtering, which leverages movie features, new movies can still show up in their recommendations despite not having any rating data.</p><h3>Evaluation of Recommendations</h3><p>The best approach to evaluating a recommender system is to test it in the wild using a strategy called <a href="https://en.wikipedia.org/wiki/A/B_testing">A/B testing</a>. With A/B testing, users are randomly split into two groups. For example, one group of users are served by <em>Recommender System A</em> and another group by <em>Recommender System B</em>. The group that exhibits more user engagement is assumed to have higher quality recommendations. So if users in group A watch more movies and give higher ratings than those in group B, we can conclude that <em>Recommender System A</em> is more effective than <em>Recommender System B</em>. Testing recommendations in a live production setting is called “online” evaluation.</p><p>In cases where we want to evaluate a recommender system that is still under development, we resort to “offline” evaluation in which a recommender system is tested in a sandbox environment unexposed to users. To evaluate a recommender model offline, we hold out a proportion of user-item interaction data in our user-item matrix. This user-item matrix, which has masked values, is fed into our recommender system to generate predictions for each user-item pair. Let’s say we’re dealing with a user-item matrix that is populated with movie ratings. We can compare the predicted ratings generated from the recommender system against the actual ratings that were held out during training. There are several evaluation metrics that we can use to quantify recommendation performance such as:</p><ul><li><strong>Precision@K</strong>: Proportion of top-K recommendations that are relevant to the user.</li><li><strong>Recall@K</strong>: Proportion of relevant items found in the top-K recommendations.</li><li><strong>Root Mean Squared Error (RMSE)</strong>: Looks at the difference between ratings predicted by the recommender system and actual ratings given by the users. A low RMSE indicates that the recommender system is able to accurately predict a user’s ratings.</li></ul><p>When we talk about precision and recall, a “relevant” item is one that a user has shown a high degree of preference towards, either through explicit feedback (5-star rating) or implicit feedback (binge-watching the show). Recall aims to minimize the number of false negatives (number of relevant recommendations that didn’t make it to the top-K recommendations), while precision aims to minimize the number of false positives (number of recommendations that were incorrectly identified as relevant). In general, a good recommender system is expected to have a high precision@k and recall@k.</p><p>Recent efforts in the field of recommender evaluation have shifted towards new concepts such as novelty, serendipity, and diversity. Research has shown that users are more engaged in recommendations that include novel items, that are serendipitous (i.e., an “accidental discovery”), and have a diverse selection. These are also important factors to take into consideration when building a recommender system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcbdfbe1e6e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to update a forked repo with git rebase]]></title>
            <link>https://medium.com/@topspinj/how-to-git-rebase-into-a-forked-repo-c9f05e821c8a?source=rss-f40e9e268985------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9f05e821c8a</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[rebase]]></category>
            <dc:creator><![CDATA[Jill Cates]]></dc:creator>
            <pubDate>Wed, 11 Apr 2018 04:26:51 GMT</pubDate>
            <atom:updated>2018-04-11T04:27:36.646Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Step 1: </strong>Add the remote (original repo that you forked) and call it “upstream”</p><blockquote>git remote add upstream https://github.com/original-repo/goes-here.git</blockquote><p><strong>Step 2: </strong>Fetch all branches of remote upstream</p><blockquote><em>git fetch upstream</em></blockquote><p><strong>Step 3:</strong> Rewrite your master with upstream’s master using git rebase.</p><blockquote><em>git rebase upstream/master</em></blockquote><p><strong>Step 4: </strong>Push your updates to master. You may need to force the push with “<em>--force</em>”.</p><blockquote><em>git push origin master --force</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9f05e821c8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>