<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Murali Gandhirajan on Medium]]></title>
        <description><![CDATA[Stories by Murali Gandhirajan on Medium]]></description>
        <link>https://medium.com/@murali.gandhirajan?source=rss-c8947e4f8bf7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7iBqoFXj55k0gSRlRlgp0Q.jpeg</url>
            <title>Stories by Murali Gandhirajan on Medium</title>
            <link>https://medium.com/@murali.gandhirajan?source=rss-c8947e4f8bf7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:53:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@murali.gandhirajan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reimagining Population Health Management with Advanced Data and AI]]></title>
            <link>https://medium.com/snowflake/reimagining-population-health-management-with-advanced-data-and-ai-2de55ff5320d?source=rss-c8947e4f8bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2de55ff5320d</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[population-health]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Murali Gandhirajan]]></dc:creator>
            <pubDate>Thu, 19 Jun 2025 14:01:48 GMT</pubDate>
            <atom:updated>2025-06-19T14:01:48.437Z</atom:updated>
            <content:encoded><![CDATA[<p>In collaboration with <a href="https://medium.com/u/eca5b343129d">Jeevan Rag</a></p><p>Healthcare organizations are under increasing pressure to improve patient outcomes, reduce costs, and deliver personalized care at scale. Population health dashboards have long been critical tools for aggregating data to identify trends, manage chronic diseases, and allocate resources. However, as healthcare complexity grows — fueled by diverse patient populations, multi-modal data sources, and evolving care models — traditional dashboards are struggling to keep pace. To drive meaningful impact, organizations must reimagine these tools by integrating advanced data and AI capabilities.</p><h4>The Business Imperative: Goals of Population Health Dashboards</h4><p>The primary goal of population health management (PHM) is to improve health outcomes for <em>groups</em> of individuals while lowering per-capita costs. Dashboards play a pivotal role in this by enabling:</p><ol><li><strong>Risk Stratification</strong>: Identifying high-risk patients for proactive interventions.</li><li><strong>Trend Analysis</strong>: Spotting disease outbreaks, care gaps, or disparities across populations.</li><li><strong>Resource Optimization</strong>: Aligning clinical, financial, and operational efforts to prioritize needs.</li><li><strong>Performance Tracking</strong>: Measuring the impact of interventions against quality metrics (e.g., reduced hospital readmissions).</li></ol><p>However, success hinges on balancing <strong>patient-level insights</strong> (e.g., personalized risk scores, treatment adherence) with <strong>population-level trends</strong> (e.g., community health disparities, vaccine uptake rates). Without this duality, organizations risk either overlooking individual needs or missing systemic challenges.</p><h4>Challenges Holding Back Traditional Dashboards</h4><p>Despite their potential, legacy dashboards face critical limitations:</p><ol><li><strong>Inability to Unlock Multi-Modal Data</strong>: Patient data now spans EHRs, wearables, genomic databases, social determinants of health (SDOH), and patient-generated inputs. Most tools fail to harmonize these unstructured, siloed datasets.</li><li><strong>Limited Data Access</strong>: Privacy regulations, interoperability gaps, and technical debt restrict access to real-time, holistic patient and population data.</li><li><strong>Static, Non-Actionable Insights</strong>: Dashboards often display historical data without AI-driven predictions, recommendations, or drill-down capabilities to guide decisions.</li></ol><p>For example, a health system might recognize rising diabetes rates in a population but lack the tools to pinpoint at-risk individuals or prescribe targeted lifestyle interventions.</p><h4>Three Pillars of a Modern Population Health Analytics</h4><p>To overcome these challenges, healthcare organizations must focus on three transformative pillars:</p><p><strong><em>Understand Populations Better with Comprehensive Data Access</em></strong></p><p>Break down data silos by integrating EHRs, claims, wearables, SDOH, and patient-reported outcomes into a unified repository. Modern platforms leverage APIs, FHIR standards, and patient consent frameworks to ensure secure, real-time access. For instance, combining SDOH data (e.g., housing instability) with clinical records can reveal why certain populations have higher ER utilization — enabling tailored outreach programs.</p><p><strong><em>Modernize with a Scalable Data &amp; AI Platform</em></strong></p><p>A cloud-native, scalable platform is essential to process and analyze massive datasets. Machine learning models can:</p><ul><li>Predict disease outbreaks or readmission risks using historical and real-time data</li><li>Generate patient risk scores by analyzing lab results, medications, and behavioral patterns</li><li>Automate administrative tasks (e.g., risk-based care gap alerts) to reduce clinician burnout</li></ul><p><strong><em>Reimagine Dashboards with AI-Driven Intelligence</em></strong></p><p>Next-gen dashboards must move beyond static charts to deliver:</p><ul><li><strong>Predictive Analytics</strong>: “What’s likely to happen?” (e.g., forecasting COPD exacerbations during flu season)</li><li><strong>Prescriptive Insights</strong>: “What should we do?” (e.g., prioritizing telehealth visits for high-risk cardiac patients)</li><li><strong>Personalization</strong>: Tailoring views for administrators, clinicians, and care coordinators to align with their roles</li></ul><p>Imagine a dashboard that not only flags a cluster of uncontrolled hypertension cases but also recommends culturally relevant education materials and tracks community health worker follow-ups.</p><h4>Population Health Dashboards Powered by Snowflake</h4><p>Snowflake data and AI platform provides a robust solution for population health management (PHM) by providing a flexible, scalable, and secure environment for managing and analyzing multi-modal data and layer in AI capabilities that helps to modernize population health dashboards</p><p>Snowflake’s ability to handle vast amounts of structured and unstructured data from various sources, including EHRs, claims, wearables, SDOH, and patient-reported outcomes, enables organizations to create a unified data repository. With Snowflake’s data integration capabilities, healthcare organizations can seamlessly connect and harmonize data from disparate systems, ensuring a comprehensive view of patient populations.</p><figure><img alt="Comphrehensive Multimodal Patient data + AI Infusion" src="https://cdn-images-1.medium.com/max/1024/0*VXQfR6hM_Ko-VG4V" /><figcaption>Comprehensive Multimodal Patient data + AI Infusion</figcaption></figure><p>Snowflake with its advanced Cortex AI capabilities can help reimagine dashboards with AI-driven Intelligence. Below are key AI/ML Infusion opportunities:</p><ul><li><strong>AI Powered Data Extraction from unstructured health data</strong> — Leverage AI techniques such as Visual LLM, OCR to extract critical information from unstructured health documents in a structured output and organize it side by side to structured health data</li><li><strong>Agentic AI Powered Pop Health Assistant </strong>— Provide natural language based chatbot experience for population health managers to ask questions and get relevant information from population health data</li><li><strong>Patient Health Event/Clinical Data Abstraction using AI</strong> — Leverage AI to parse through unstructured clinical/medical notes data to extract health concepts</li><li><strong>AI Powered Summary and Patient Insights</strong> — AI as Co-pilot and assistant summarizes key information and patient insights providing easy to consume information in population health dashboard</li><li><strong>ML Models for Risk Prediction and Population Stratification</strong> — Leverage power of traditional ML models with extracted feature information from unstructured documents to predict risks and stratify patient/members based on their health risks and design better interventions</li><li><strong>Aggregate population/patient level insights with AI </strong>— Aggregate Patient Level and population level insights from multiple encounter/health records (unstructured notes) with AI, such as SDoH insights at patient level summarized across multiple health records</li></ul><p>Here are some screenshots with art of possible view on how AI can be infused across population health management analytics:</p><figure><img alt="Holistic patient health insights by integrating unstructured data" src="https://cdn-images-1.medium.com/max/800/0*vg5_PB0yf8hdjznp" /><figcaption><strong>Holistic patient health insights by integrating unstructured data</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Q1HNbeOD5O0NAw6n" /><figcaption><strong>Augment traditional population analytics with actionable insights</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*dK0tqpinSZNYHulV" /><figcaption><strong>Talk to your Patient Data with Cortex Agent(Agentic AI)</strong></figcaption></figure><h4>Conclusion: From Insights to Impact</h4><p>By embracing <strong>data accessibility</strong>, <strong>scalable AI platforms</strong>, and <strong>intelligent dashboards</strong>, healthcare organizations can transform population health management from reactive to proactive. These pillars enable:</p><ul><li><strong>Better Outcomes</strong>: Early interventions for high-risk patients and communities.</li><li><strong>Lower Costs</strong>: Reduced hospitalizations through predictive care.</li><li><strong>Equitable Care</strong>: Addressing disparities by integrating SDOH and behavioral data.</li></ul><p>The future of population health lies in dashboards that don’t just report data — they <em>drive action</em>. With AI as a copilot, healthcare leaders can turn insights into interventions, ensuring no patient or population is left behind.</p><p><em>The time to act is now. By reimagining PHM tools, we can build healthier communities — one data point at a time.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2de55ff5320d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/reimagining-population-health-management-with-advanced-data-and-ai-2de55ff5320d">Reimagining Population Health Management with Advanced Data and AI</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GenAI assisting with healthcare coding extraction to calculate appropriate patient risk scores for…]]></title>
            <link>https://medium.com/snowflake/genai-assisting-with-healthcare-coding-extraction-to-calculate-appropriate-patient-risk-scores-for-a7e067231749?source=rss-c8947e4f8bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7e067231749</guid>
            <dc:creator><![CDATA[Murali Gandhirajan]]></dc:creator>
            <pubDate>Mon, 20 May 2024 14:01:53 GMT</pubDate>
            <atom:updated>2024-05-20T14:01:53.034Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>GenAI assisting with healthcare coding extraction to calculate appropriate patient risk scores for better reimbursements</strong></h3><p><em>Snowflake Cortex AI to uncover healthcare coding insights</em></p><p>In collaboration with Mihir Bhojani, Senior Solutions Engineer-Snowflake &amp; <a href="https://medium.com/u/f91707f39e96">Brian Baillod</a>, Senior Sales Engineer-Snowflake</p><p><strong>Business Context</strong><br>In healthcare, accurately calculating patient risk scores is critical for better reimbursements. Providers must balance high-quality care with financial stability, making precise risk scores essential. These scores, derived from demographic and clinical factors, determine a patient’s financial risk. Accurate assessment leads to better decisions on resource allocation, treatment plans, and care management, improving patient outcomes and resource efficiency.</p><p>Coding extraction plays a vital role by ensuring that all relevant patient data is captured and properly coded. With widespread electronic health records (EHRs) and a focus on data interoperability, providers and payers have access to vast information. However, effective data extraction and interpretation require specialized knowledge and tools. Accurate coding helps healthcare organizations identify key risk factors and assign appropriate risk scores.</p><p><strong>Opportunity</strong></p><p>By leveraging GenAI techniques, healthcare organizations can streamline the coding extraction process and improve the accuracy of patient risk score calculations.</p><p>This blog explores the use of GenAI for coding extraction from unstructured patient transcripts data and presents missing diagnosis codes and corresponding HCC coding for humans to consider and appropriately qualify for patient risk score calculations <strong>using Snowflake Cortex AI and Streamlit capabilities.</strong></p><p><strong>Implementation Overview</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATreCHTvBcUcdiy23NuO2Q.png" /><figcaption>Figure1: GenAI assisting in Coding Extraction: Process Flow</figcaption></figure><p><strong><em>Step 1: Parse and extract medical transcripts data from patient transcripts and load in Snowflake table</em></strong></p><p>As a first step, we need to get the medical transcripts parsed, chunked into smaller chunks and loaded into a Snowflake base table. Here are the detailed steps involved.</p><p>Load medical transcripts into a cloud storage bucket. In this case, scrubbed medical transcripts downloaded from publicly available sources on the internet are loaded into S3 bucket and then storage integration for that s3 bucket is created in Snowflake. Please refer to Snowflake documentation on <a href="https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration">how to create s3 storage integration.</a></p><p>Create an external stage in Snowflake — <a href="https://docs.snowflake.com/en/sql-reference/sql/create-stage">steps here</a>. We can now access these PDFs in Snowflake as shown in Figure2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PDS6CCnTUardvN0i" /><figcaption>Figure2: PDF Documents in External Storage accessed in Snowflake</figcaption></figure><p>Now that medical transcripts are loaded into an external stage and accessible within Snowflake, we have to read those PDFs, extract contents and break the contents into smaller chunks if necessary. For that we can create a python function that uses<strong> pyPDF2 to extract the contents </strong>and use langchain function to chunk as needed<strong>. </strong>Deploy this python function as User Defined Table Function (UDTF) on Snowflake which takes document URL as input and returns document URL and extracted text.</p><p>As a next step, create a table docs_and_text and run the UDTF for all the documents in the external stage and load the output into that table docs_and_text. Figure3 shows the output of that table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NoaTtTvviwvRvbDd" /><figcaption>Figure3: Medical Transcript PDFs extracted into a column in Snowflake table</figcaption></figure><p><em>Please note that your unstructured patient data is typically gathered from multiple sources and having a data product approach to collate all patient data including unstructured data is foundational to deploying AI/ML capabilities at scale. What we covered here as medical transcripts is one of the many sources for patient’s unstructured health data where meaningful diagnosis code for risk scoring can be extracted.</em></p><p><strong><em>Step 2: Use Snowflake Cortex AI to extract diagnosis codes and suggest HCC codes</em></strong></p><p>Let us put AI to use now! Pass the extracted text content from medical transcripts as input to the Snowflake Cortex Complete function and ask LLM to identify ICD10 diagnosis codes. Few considerations:</p><ol><li>Pick the right model size that performs fairly well for coding extraction</li><li>Context window length: Oftentimes, the unstructured clinical notes and transcripts run multiple pages, choosing a model that has better context window helps to avoid chunking the transcripts</li><li>Play around with prompt engineering to get the codes in the format that requires less parsing post extraction of codes from the LLM model</li></ol><p>For list of models available in Snowflake cortex and its context window, please visit this <a href="https://docs.snowflake.com/user-guide/snowflake-cortex/llm-functions#model-restrictions">page.</a> In this case, we have picked “reka-flash” which has context window of 100K tokens and found to be performing fairly well in extracting ICD10 codes and HCC codes.</p><p>Create a new table with the identified ICD10 codes and HCC codes using a SQL like below. Here, we wanted to push the LLM and ask it to identify HHS based HCC codes as well as CMS based HCC codes based on latest version.</p><pre>create or replace table docs_hcc_coding as <br>select relative_path,<br>doc_text,<br>SNOWFLAKE.CORTEX.COMPLETE(&#39;reka-flash&#39;, concat(doc_text||&#39;Given this medical transcript, list major ICD10-CM diagnosis code in this format ONLY: X##.# (Diagnosis description). Dont provide explanation&#39;)) as AI_ICD10_Code,<br>SNOWFLAKE.CORTEX.COMPLETE(&#39;reka-flash&#39;, concat(AI_ICD10_Code||&#39;List the CMS HCC code version 28 for this ICD10 code in format: HCC-XYZ (HCC Category Description). Dont provide explanation&#39;)) as AI_CMS_HCC_Code,<br>SNOWFLAKE.CORTEX.COMPLETE(&#39;reka-flash&#39;, concat(AI_ICD10_Code||&#39;List the HHS HCC code version 28 for this ICD10 code in format: HCC-XYZ (HCC Category Description). Dont provide explanation&#39;)) as AI_HHS_HCC_Code, <br>from docs_and_text<br>;</pre><p>Output of the table looks like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EsuaHka_OW_dozG1xoMkyg.png" /><figcaption>Figure4: Output of extracted codes using GenAI in Snowflake Table</figcaption></figure><p>Please note that accuracy of these extracted codes are not verified by coders. Intention here is to have GenAI heavy lift the extraction of codes and present it for coders to verify and qualify the right fit ones for risk score calculation. This is humans-in-loop application of GenAI geared towards productivity increase of coders who are involved in identifying these missing codes from patient data</p><p><strong><em>Step 3: Build and deploy streamlit app that presents these identified ICD10 diagnosis codes/HCC codes for coders review and qualify missing diagnosis codes for risk score calculation</em></strong></p><p>Once we have the extracted codes loaded into Snowflake table as described in above step, next step is to create a streamlit app that lets coders visualize the extracted ICD10 codes and HCC codes and do additional analysis as required to validate the codes. As part of the validation process, coders can be given access to ask further questions to LLM with the context of a particular patient transcript that helps them to make a decision. Once a decision is made, coders can be allowed to submit the validated codes as input for risk scoring model or reject them and document the reason which can be captured in a table in the backend for future reference and auditing.</p><p>Please refer to documentation <a href="https://www.snowflake.com/en/data-cloud/overview/streamlit-in-snowflake/">here</a> for further information about Streamlit and how to stand up a simple streamlit application like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rUmZRx6cHsClHo7J4fq_Yw.png" /><figcaption>Figure5: Streamlit App showing the extracted ICD10 codes using Snowflake Cortex &amp; reka-flash LLM</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MJIENP-jGPsq9bW-gMgMOA.jpeg" /><figcaption>Figure6: Streamlit app showing HCC codes for extracted ICD10 codes</figcaption></figure><p><strong>Conclusion</strong></p><p>Integrating GenAI in healthcare coding extraction can revolutionize patient risk score calculations. By identifying and extracting ICD10 and HCC codes from unstructured patient transcripts with human-in-the-loop validation, healthcare providers can ensure more precise reimbursements and better patient care. This innovative approach streamlines the coding process, reduces errors, and benefits both providers and patients. As GenAI continues to evolve, its impact on healthcare will lead to improved efficiency, cost savings, and enhanced patient outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7e067231749" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/genai-assisting-with-healthcare-coding-extraction-to-calculate-appropriate-patient-risk-scores-for-a7e067231749">GenAI assisting with healthcare coding extraction to calculate appropriate patient risk scores for…</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interact with LLM to search and uncover insights from medical transcript documents]]></title>
            <link>https://medium.com/snowflake/interact-with-llm-to-search-and-uncover-insights-from-medical-transcript-documents-ab5d694e70e4?source=rss-c8947e4f8bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab5d694e70e4</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[healthcare-innovations]]></category>
            <category><![CDATA[snowflake-cortex]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Murali Gandhirajan]]></dc:creator>
            <pubDate>Mon, 06 May 2024 15:55:02 GMT</pubDate>
            <atom:updated>2024-05-06T15:55:02.489Z</atom:updated>
            <content:encoded><![CDATA[<p>Retrieval Augmented Generation (RAG) Using Snowflake Cortex LLM</p><p>Author: <a href="https://medium.com/u/c8947e4f8bf7">Murali Gandhirajan</a> Co-Author: <a href="https://medium.com/u/3c71786c28d8">Venkat Sekar</a></p><p>Roughly about 80% of healthcare data is unstructured. A typical healthcare organization deals with different types of unstructured healthcare data such as discharge summaries, medical transcripts, clinical notes, pathology reports, radiology reports, medical charts and many more. Extracting rich patient information that exists within these documents is crucial to understand the patient as a whole and generate insights that drive patient health outcomes.</p><p>In this blog, we will focus on one particular type of unstructured health document which is medical transcripts. Oftentimes care managers and clinicians struggle to get to patients’ history of medical transcripts based on various encounters and are not able to search for medical transcripts with certain health scenarios for reference. For a long time now, technology challenges have been a bottleneck to enable such use cases for healthcare professionals. But with the rise of Large Language Models, embedding models that create vector embeddings of unstructured data, ability to search the embedding to get to nearest match to the content being searched in corpus of documents and Retrieval Augmented Generation (RAG) approach, now we can leverage technology to make such use cases a reality.</p><p>RAG is a popular framework in which an LLM gets access to a specific knowledge base with the most up-to-date, accurate information available in this case patient’s medical transcripts before generating a response. There is no need to retrain the model and this extends the capability of any LLM to specific domains such as healthcare in a relatively simple and cost-effective way</p><p>Implementing RAG using Snowflake Cortex to search and uncover insights from medical transcripts can be done in few simple steps:</p><p><strong><em>Step 1:</em></strong><em> Parse Medical transcripts in PDF format, create smaller chunks and load into Snowflake table</em></p><p><strong><em>Step 2:</em></strong><em> Augment chunked medical transcripts with summary and specialty using Snowflake Cortex LLM functions and create vector embeddings</em></p><p><strong><em>Step 3: </em></strong><em>Interact with LLM to search and uncover insights from medical transcripts using RAG approach and Streamlit</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Lx2FhKHn9YhA0cw" /><figcaption>Figure1: Pre-processing steps to prepare medical transcript data and augment with insights from LLM for RAG Efficiency</figcaption></figure><p>You can refer to Snowflake quick start <a href="https://quickstarts.snowflake.com/guide/asking_questions_to_your_own_documents_with_snowflake_cortex/index.html#0">build your own document chatbot with Snowflake Streamlit and Corte</a>x to jump start implementation of below steps</p><p><strong><em>Step 1: Parse Medical transcripts in PDF format, create smaller chunks and load into Snowflake table</em></strong></p><p>As a first step, we need to get the medical transcripts parsed, chunked into smaller chunks and loaded into a Snowflake base table. Here are the detailed steps involved.</p><p>Load medical transcripts into a cloud storage bucket. In this case, scrubbed medical transcripts downloaded from publicly available sources on the internet are loaded into S3 bucket and then storage integration for that s3 bucket is created in Snowflake. Please refer to Snowflake documentation on <a href="https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration">how to create s3 storage integration.</a></p><p>Create an external stage in Snowflake — <a href="https://docs.snowflake.com/en/sql-reference/sql/create-stage">steps here</a>. We can now access these PDFs in Snowflake as shown in Figure2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oYtXXebLcodi-NvV" /><figcaption>Figure2: PDF Documents in External Storage accessed in Snowflake</figcaption></figure><p>Now that medical transcripts are loaded into an external stage and accessible within Snowflake, we have to read those PDFs, extract contents and break the contents into smaller chunks. For that we can create a python function that uses<strong> pyPDF2 to extract the contents</strong> and <strong>langchain RecursiveCharacterTextSplitter to split the contents into chunks (chunk size 4000 and overlap 400)</strong>. Deploy this python function as User Defined Table Function (UDTF) on Snowflake which takes document URL as input and returns a table of document URL and associated chunks. Please note there may be more than one chunk returned for each of the document that was processed</p><p>As a next step, create a table doc_chunked and run the UDTF for all the documents in the external stage and load the output into that table doc_chunked. Figure3 shows the output of that table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*76jBj6FpJckw0BeM" /><figcaption>Figure3: Output of parsed and chunked med transcripts</figcaption></figure><p><strong><em>Step 2: Augment chunked medical transcripts with summary and specialty using Snowflake Cortex LLM functions and create vector embeddings</em></strong></p><p>Once document contents are parsed and chunked, we will concatenate all the chunks for a given document using listagg function in Snowflake. Store the output in a temporary table as shown in Figure4</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OGt3WIAOmiODcFHU" /><figcaption>Figure4: Med Transcripts content extracted and concatenated into single column</figcaption></figure><p>Pass that full extracted text as input to two Snowflake Cortex LLM functions. First function COMPLETE, uses mixtral-8x7b model to have LLM summarize the report contents in less than 5 words and the second function EXTRACT_ANSWER to extract specialty information from the text corpus and update the temporary table with the function outputs as additional columns</p><pre>SNOWFLAKE.CORTEX.Complete (&#39;mixtral-8x7b&#39;, concat(doc_text||<br>        &#39; In less than 5 words, how would you best describe the type of the document content? Do not provide explanation. Remove special characters&#39;)) as report, <br>SNOWFLAKE.CORTEX.EXTRACT_ANSWER (doc_text, <br>        &#39;What is the medical specialty?&#39;)[0]:answer::varchar as specialty</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YXBxl8cNK66roQ2-" /><figcaption>Figure5: Output of Cortex LLM summarizing report content and extracting medical specialty information</figcaption></figure><p>Now that we have used LLM to get additional summary information, we will add it to chunks in the docs_chunked base table so that each of the chunks is appended with report summary and specialty information. <em>This step is done to make sure rich summary and specialty information is available in all chunks to ensure the RAG approach picks up the relevant chunks based on prompt questions which would most likely list out keywords from the summary and/or specialty. This significantly improves RAG accuracy.</em></p><p>Pass the updated chunks to Snowflake CORTEX function EMBED_TEXT to create vector embedding and load the embeddings into VECTOR data type column in the table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X5mA4Ad-Bd7kBKsi" /><figcaption>Figure6: Output of Pre-Processing steps that holds embeddings of chunked and augmented med transcripts</figcaption></figure><p>All pre-processing steps are done now, final table looks like above and has all the information needed to implement RAG approach</p><p><strong><em>Step 3: Interact with LLM to search and uncover insights from medical transcripts using RAG approach and Streamlit</em></strong></p><p>Next step is to create a Streamlit app that lets us interact with LLM and get context rich responses to our prompt question and help us uncover insights from medical transcripts. Please refer to <a href="https://www.snowflake.com/en/data-cloud/overview/streamlit-in-snowflake/">streamlit in Snowflake for more information</a>.</p><p>Streamlit app can be designed with three simple widgets title, dataframe to list all the docs in the external stage and chat message input and render output from LLM message back</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MVuf2LbvsRTFsb6T" /><figcaption>Figure7: Streamlit app to interact with LLM</figcaption></figure><p>Most of the LLMs are stateless which means they don’t remember the previous interaction. In order to have the LLM remember previous chat, we preserve the previous prompts and responses and concatenate that with prompt questions every time as chat history to ensure LLMs are responding back in an interactive chat fashion</p><p>In the streamlit, we can provide an option to select the model we want to use for LLM interaction. Right now, Snowflake Cortex COMPLETE function supports multiple models — <a href="https://docs.snowflake.com/user-guide/snowflake-cortex/llm-functions">refer to documentation here for full list</a></p><p>Now, we can interact with LLM in chat fashion and explore insights from patient medical transcripts. Let us ask question to get patients with chronic glossitis condition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VtBH2PV-_xgLWvfE" /><figcaption>Figure8: Exploring Insights from Medical Transcripts</figcaption></figure><p>Let us now ask LLM to get the top transcript from the context for the patient with chronic glossitis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WcsCqb9BJFqdGZbM" /><figcaption>Figure9: RAG in action giving contextual response</figcaption></figure><p>Finally let us ask LLM to summarize the treatment plan given to that patient</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b9frrcegXFckftJs" /><figcaption>Figure10: LLM in action to extract and summarize treatment plan information found in medical transcripts</figcaption></figure><p><strong>Conclusion</strong></p><p>A RAG based approach is an effective way to leverage foundational models to provide relevant and contextual response to prompts and helps to explore unstructured patient health documents that exist within health care organizations. Snowflake platform has a rich set of features and capabilities around GenAI/LLM. Snowflake abstracts not just the complexities of setting up infrastructure needed to host LLM but also the deployment and setup of LLMs in itself with its CORTEX LLM functions and enables healthcare organizations to focus more on generating insights from the data that truly helps drive patient health outcomes and improve operational efficiency</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab5d694e70e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/interact-with-llm-to-search-and-uncover-insights-from-medical-transcript-documents-ab5d694e70e4">Interact with LLM to search and uncover insights from medical transcript documents</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging Snowflake Native App Framework to build and deploy data apps that ingest and parse…]]></title>
            <link>https://medium.com/snowflake/leveraging-snowflake-native-app-framework-to-build-and-deploy-data-apps-that-ingest-and-parse-3a52677f2eb1?source=rss-c8947e4f8bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a52677f2eb1</guid>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[hl7]]></category>
            <category><![CDATA[interoperability]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Murali Gandhirajan]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 21:20:01 GMT</pubDate>
            <atom:updated>2023-08-09T21:53:27.038Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Snowflake Native App Framework to build and deploy data apps that ingest and parse healthcare data formats (HL7) for analytics</strong></h3><p>The U.S. healthcare industry is a network of various entities such as health insurance firms, physician offices, hospital systems, medical device manufacturers, pharmaceuticals, community-based organizations, social care service organizations, and so on, collaborating to serve the health needs of a patient and deliver successful health outcomes. In the context of healthcare service providers or hospitals, data exchange between healthcare professionals is recognized as a catalyst to ensure care continuum, improve patient outcomes, reduce hospital length of stay (LOS) and hospitalization costs, and higher patient satisfaction.</p><p>To enable and promote data exchange between different healthcare entities, the Office of the National Coordinator for Health Information Technology (ONC) has published standards and formats. Health Level 7 (HL7) v2 is the most widely used data interchange standard that allows hospital information systems to communicate clinical and administrative data to one another. HL7 Consolidated Clinical Document Architecture (C-CDA) is another XML-based data format. Recently, ONC introduced the Fast Healthcare Interoperability Resources (FHIR) format and organizations are rapidly transitioning to it.</p><p>Healthcare organizations onboard data received in these formats not just to integrate with clinical systems (systems of records) for clinical decisioning but also to integrate into their data platform for analytics purposes, which provides valuable holistic insights about patients. Data in these formats often needs to be ingested, parsed to extract health data elements, and stored in a way that is conducive for analytics. There is a great opportunity to templatize this parsing and publish it as a “Health Data Adapter” (aka data application) that can then be installed and configured for a given healthcare organization’s need.</p><p>The <a href="https://www.snowflake.com/blog/native-app-framework-available-developers-aws/">Snowflake Native App Framework</a> enables healthcare organizations or healthcare technology firms to build data applications leveraging native Snowflake technologies. In this blog, we will talk about how to create such data applications for healthcare organizations using Snowflake Native App Framework with HL7 v2 data format processing as an example.</p><p>Before we begin talking about specifics of how to create HL7 v2 data file format parsing as a native app, we’ll start with the generic building blocks for a Snowflake Native Application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lLpZ538RQI9WeRH-" /><figcaption>Snowflake Native App Development Framework</figcaption></figure><p><strong><em>Steps to Build and Deploy a Snowflake Native App</em></strong></p><p><strong><em>Develop &amp; Test</em></strong></p><p>Organizationally, publishers who want to develop and publish Snowflake native apps will start with a development DB. The development DB will ideally contain at least two schemas; one to hold the application’s logic, SQL scripts and other resources, and another to house the unit tests for the application. The application logic will be core data app functionality that caters to a particular business use case.</p><p><strong><em>Package &amp; Version</em></strong></p><p>When this application code is built and tested, it is moved to a code stage DB to prepare for conversion into an application package. The code stage DB can be thought of purely as a staging DB for code that will eventually be turned into an application package. This code stage should contain only the artifacts pertinent to the application itself. The application package is created from the code stage DB. Publishers will include version and patch information to the app pack definition. This model supports the rollout of application upgrades (and rollbacks) by the publisher to all consumers using the application. Publishers can test the application package by creating a local application from it directly.</p><p><strong><em>Publish</em></strong></p><p>Once the application package is prepared, the publisher is ready to offer it as a listing in <a href="https://www.snowflake.com/en/data-cloud/marketplace/">Snowflake Marketplace</a> (either publicly or privately) with optional support for monetization. Previous steps can all be accomplished through SQL scripts, however, the publication of a “listing” is a manual process within the Snowflake UI. Developers navigate to the “Publishers Studio” within the Snowsight UI to complete the listing process. At that point, your target audience would see your listing in the Marketplace.</p><p><strong><em>Deploy</em></strong></p><p>Depending on whether the application listing was defined as a public Marketplace listing or a private listing to a specific customer account, the consumer will find the offering in the Marketplace or apps sub menu in the Snowflake UI. Locate the application, name it, and select the warehouse you want to install the application. Once the installation is complete, the functionality (UDFs, stored procs, shared data sources, etc.) is accessible as any other first-class object within your Snowflake account.</p><p><strong><em>Creating, Publishing and consuming a sample Snowflake Native App — HL7 v2 Health Data Adapter app</em></strong></p><p>Let us take HL7 v2 data format parsing as an example and use the above building blocks to create, publish, and consume HL7 v2 file format parsing as a native app. We will take a Python library for parsing HL7 v2 files, wrap it as a user defined table function, and turn it into a native application we could publish for other Snowflake users. The published application is available for installation in the consumer account under “HL7ParserApp.”</p><p>Let’s assume we already have our development DB in place. We will build all of the artifacts out of the PUBLIC schema, however, you can use a custom schema if you prefer. We need to create a STAGE to house the artifacts that will be used by this application.</p><pre>--add a stage for application artifacts<br>create or replace stage lib_stg;<br>--create a TEST schema &amp; stage to house our test files<br>create or replace schema test<br>create or replace stage test_stg;</pre><p>We then add the artifacts to each respective stage. We have the Python third-party library that contains the parsing logic (hl7apy-1.3.4.tar.gz; please refer to <a href="https://pypi.org/project/hl7apy/">https://pypi.org/project/hl7apy/</a> for additional details) and custom Python script that we have developed to act as the interface to this library (hl7parserUDTF.py). We will also upload a manifest and a setup script as required by the native app framework to create an application. The manifest contains information about the characteristics that define the app and the path to the setup script that will execute on the consumer side when the app is installed.</p><pre>-add the artifacts to the lib stage<br>put file://./src/native_app/scripts/hl7_setup.sql @public.lib_stg<br>overwrite=true auto_compress=false;<br>put file://./src/python/hl7pyparserUDTF.py @public.lib_stg<br>overwrite=true auto_compress=false;<br>put file://./src/python/libs/hl7apy-1.3.4.tar.gz @public.lib_stg<br>overwrite=true auto_compress=false;<br>put file://./src/native_app/manifest.yml @public.lib_stg<br>overwrite=true auto_compress=false;<br>put file://./src/native_app/readme.md @public.lib_stg<br>overwrite=true auto_compress=false;<br>–add the test file to test schema<br>put file://./data/hl7_2–3_samples.txt @test.data_stg<br>overwrite=true auto_compress=false;</pre><p>Create a setup script that contains SQL commands to run when the app is installed. The contents of the example setup script is as follows</p><pre>create application role hl7_app_public;<br>create or alter versioned schema public;<br>grant usage on schema public to application role hl7_app_public;<br>--create a HL7 message parsing udtf function<br>CREATE OR REPLACE FUNCTION public.hl7parserUDTF(p_stagefile string)<br>RETURNS TABLE (seqno int ,data_file varchar ,raw_msg varchar ,parsed_status boolean ,parsed_msg variant)<br>LANGUAGE python<br>runtime_version = 3.8<br>packages = (&#39;snowflake-snowpark-python&#39;)<br>imports=(&#39;/hl7apy-1.3.4.tar.gz&#39;, &#39;/hl7pyparserUDTF.py&#39;)<br>HANDLER = &#39;hl7pyparserUDTF.HL7FlParser&#39;<br>COMMENT = &#39;Python based UDTF hl7v2 message parser&#39;;<br>GRANT USAGE ON FUNCTION public.hl7parserUDTF(string) TO application ROLE hl7_app_public;</pre><p>An application role is specifically designed for use with applications. These roles have privileges so the admin in the consumer account can assign ROLEs or USERs in their account.</p><p>Once the code is unit tested and ready to package as an application, it’s a best practice to recreate our dev DB as a staging DB (see the conceptual steps above) prior to creating the app package (stage_DB recreation steps not shown as they are mostly copies of the above). Once this is complete, we can create the app package from this DB.</p><pre>Create application package hl7app_pkg;<br>--add the version information and resources to the app package<br>Alter application package hl7app_pkg add version v_0_0_0_1A using &#39;@stage_db.public.lib_stg&#39;;<br>Alter application package hl7app_pkg set default release directive version = v_0_0_0_1A patch = 0;</pre><p>The app package is now complete.</p><p>Before we create a listing to share this application with other accounts, let’s test it locally by creating the application from the app package.</p><pre>Create application hl7app from application package hl7app_pkg using version v_0_0_0_1A debug_mode=true;</pre><p>Running this will create a new application artifact in the UI and will show up as a DB (with a different icon to indicate it’s an app). The function defined as part of the startup script will have created the UDTF definition pointing to the resources loaded in our lib_stg.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/0*j68iZgmHo6JLaRSk" /><figcaption>Snowsight UI showing UDTF function</figcaption></figure><p>The function is now retrievable by its fully qualified path name and we can run necessary tests to validate the parsing logic in the local DB.</p><p>Let’s now create a listing. In the Snowsight UI, navigate to Data -&gt; Provider Studio then select Listings from the top of the screen on the right. You will need to be in the accountadmin role or one that has been granted CREATE DATA EXCHANGE LISTING permission to create the listing. Click the + Listing button to start the process. Name the listing and specify whether it will be shared in the Marketplace across the entire Snowflake customer base or with only a specific customer account. For this example, we will go with the direct share. The Marketplace option requires details about yourself, your organization, and about the app being offered. It also kicks off an automated security review of your offering before the listing is approved. (For more details on security reviews, please refer to <a href="https://docs.snowflake.com/en/developer-guide/native-apps/running-security-scan">Running the Automated Security Scan</a>.) Choose whether this listing will be free or require payment for use. The paid version requires app providers to create a provider profile, so for our demo, we will choose free. Next, we specify the HL7_app_pkg we previously created and then specify the customer(s) we will offer this to by using their account identifier. Click on publish to complete the process. Your app now appears as live. You can add/remove customers from this listing or unpublish it to completely disable it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L30cfBQ2FFbLZnig" /><figcaption>Snowsight UI — Create Listing</figcaption></figure><p>The customer account you have shared the app with will now be able to find the offering in their Snowsight UI by navigating to the Apps menu item. From there, they will see the listing to add to their account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aOBi5NM2A4X76eRz" /><figcaption>Consumer Account — Snowsight UI showing the shared native app</figcaption></figure><p>Select the Get button to start the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tRvCWoHGd-dcqEUv" /><figcaption>Consumer Account — Installing Native App</figcaption></figure><p>Consumers can set the name of the application as they want it, and the app will appear as a database in their account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*2egG_d4K7IBnJRg2" /><figcaption>Consumer Account Snowsight UI showing installed App</figcaption></figure><p>The parser function is now callable from within the consumer account using its fully qualified path name hl7parser.public.hl7parserudtf(&lt;filepath&gt;)</p><p>With this native app, consumers can drop HL7 message files into cloud storage buckets and have the above native app function parse and load messages into Snowflake for further analytics, greatly simplifying the effort to build and deploy parsers for health data formats.</p><p><strong><em>Key Benefits</em></strong></p><p>Creating and deploying repeatable industry-specific business functionalities as Snowflake’s Native App comes with plenty of benefits, including the following:</p><ul><li>Native applications bring processing logic to data, making it efficient and simplifying the tech stack</li><li>Logic is more repeatable and helps jump-start implementation of business use cases</li><li>Native app developers can confidently deploy the logic of their proprietary/intellectual property without exposing details of the logic</li><li>Sensitive data need not be shared across organizational entities</li><li>Data app creators are empowered to monetize their IP using Snowflake Marketplace</li></ul><p><strong><em>Conclusion</em></strong></p><p>As different healthcare organizations or healthcare technology firms implement the Snowflake Native App Framework to build and deploy data applications, we foresee the formation of a data app mesh that will open up data app economies of scale. The Health Data Adapter mentioned in this blog is one such example. Opportunities are endless!</p><p>Learn more about<a href="https://www.snowflake.com/blog/native-app-framework-available-developers-aws/"> Snowflake’s Native App framework</a>. At the time of this blog release, please note that Snowflake Native App functionality is in public preview and currently only available in AWS.</p><p>Special thanks to <a href="https://medium.com/u/d0aa1cbfdded">Ryan Templeton</a> (<a href="https://www.linkedin.com/in/ryantempleton/">https://www.linkedin.com/in/ryantempleton/</a>) for co-authoring this blog with me and for his valuable contributions in building this demo.</p><p>Want to read more, here are some more useful resources:</p><ol><li><strong>Blog</strong>: <a href="https://medium.com/snowflake/hl7-ingestion-and-processing-architectural-patterns-with-snowflake-3703b8c08ea4">https://medium.com/snowflake/hl7-ingestion-and-processing-architectural-patterns-with-snowflake-3703b8c08ea4</a></li><li><strong>Blog</strong>: <a href="https://medium.com/snowflake/processing-hl7-v2-x-messages-using-snowflake-571441ed3b23">https://medium.com/snowflake/processing-hl7-v2-x-messages-using-snowflake-571441ed3b23</a></li><li><strong>Snowflake Quickstart:</strong> <a href="https://quickstarts.snowflake.com/guide/processing_hl7_v2_messages_with_snowflake/index.html?index=..%2F..index#0">https://quickstarts.snowflake.com/guide/processing_hl7_v2_messages_with_snowflake/index.html?index=..%2F..index#0</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a52677f2eb1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/leveraging-snowflake-native-app-framework-to-build-and-deploy-data-apps-that-ingest-and-parse-3a52677f2eb1">Leveraging Snowflake Native App Framework to build and deploy data apps that ingest and parse…</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>