<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ojal Bhandarkar on Medium]]></title>
        <description><![CDATA[Stories by Ojal Bhandarkar on Medium]]></description>
        <link>https://medium.com/@ojal.bhandarkar?source=rss-5529260323da------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*l2Q9STZgmiUicfkC</url>
            <title>Stories by Ojal Bhandarkar on Medium</title>
            <link>https://medium.com/@ojal.bhandarkar?source=rss-5529260323da------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:10:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ojal.bhandarkar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Profiling Using Dataplex]]></title>
            <link>https://medium.com/clairvoyantblog/data-profiling-using-dataplex-d8a1076c4ddb?source=rss-5529260323da------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8a1076c4ddb</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[dataplex]]></category>
            <category><![CDATA[data-profiling]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[data-engineer]]></category>
            <dc:creator><![CDATA[Ojal Bhandarkar]]></dc:creator>
            <pubDate>Thu, 27 Apr 2023 10:04:46 GMT</pubDate>
            <atom:updated>2023-04-27T10:04:46.385Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Profiling using Dataplex</h3><h4>It’s your data but profiler knows it better. Let’s find out how?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E4afPYZsKy29vGVy.jpg" /><figcaption>source: <a href="http://www.ataccama.com/">ataccama</a></figcaption></figure><p>Data profiling is the process of examining the data available from an existing information source and collecting statistics or informative summaries about that data. <a href="https://en.wikipedia.org/wiki/Data_profiling">Wikipedia</a>.</p><p>In other terms, Data profiling is used to know the content, structure, pattern, statistical metrics, frequency, accuracy and quality of your data. This information leads to the best use of data and directly proportional to organizational growth.</p><p>Google Cloud Platform provide service Dataplex which is the data fabric that unifies distributed data and automates data management and governance for that data.(Source:<a href="https://cloud.google.com/dataplex/docs/introduction">Google Document</a>)</p><h3><strong>Let’s see how we can do data profiling using Dataplex</strong></h3><p>Dataplex-<em>data profile scan</em> is associated with one BigQuery table and scans the table to generate the data profile results. You can configure the scan to run on the entire table or incrementally on the newer data and you can choose to run the scan on a schedule or on demand. The results of the scan are available as part of each scan execution. (Source:<a href="https://cloud.google.com/dataplex/docs/data-profiling-overview">Google Document</a>)</p><p>As an example, I am using covid19_summary data to perform profiling. It has state-wise total positive/negative covid cases, number of icu beds available on the date, death count, recovered count, current hospitalizations count. This is very crucial data and can be used by organisation to decide next step. But before handing it over to the analyst team,data should be profiled, cleaned to shorten the implementation cycle and improve understanding of data.</p><blockquote>Step 1: In <a href="https://console.cloud.google.com/dataplex/govern/profile">Dataplex Profile</a> section create a profile to scan.Provide fully qualified name of the table.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TtWRjnx2wa2hlUVLtoNdcA.png" /><figcaption>Create Profile</figcaption></figure><blockquote>Step 2: Schedule or run the scan to get result immediately.In Job history tab you will be able to see running the job running.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CvjHTq27GKdeaNiH_R12Vw.png" /><figcaption><em>Job History</em></figcaption></figure><blockquote>Step 3: Once job status updated to completed. Click on job id to see data profiling result.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SGwhshvSIb41afcifscBRg.png" /><figcaption><em>Profiling Result</em></figcaption></figure><h3>Know the Profile of your data</h3><p>If you have observed in<em> Img:Profiling Result in step 3</em>, result is varying as per the data type.</p><blockquote>Summary : Record scan- 420</blockquote><p>Job has done profiling on entire data of table on given date.(<em>Img: Job History in step 2</em>)</p><blockquote>state : Text data (2-letter postal abbreviation for the state)</blockquote><p>State data is non-null but not unique.Value 56 have highest number of duplicate entries. 2-letter postal abbreviation but contain some data values with 1-letter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L7IkXlzQFOHuYeDIs5Xugw.png" /></figure><blockquote>cases_positive_total : Numeric data (total number of positive cases)</blockquote><p>Few of the rows has null data that means there is no positive cases found in state for the date. Data is almost unique. Now instead of whole data, the user can check 0.24% of the data to check why it is null, or why are there no positives? Was covid test done on that day? Was there some technical issue on that day? Do we have entry on paper for that day? There can be the different reasons.</p><p>Let’s observe data in terms of numeric value. Here 2 has highest entry in table. Quartile show 25% below this point , 50% of data below this point, 25% upper quartile of data below this point(Learn about <a href="https://en.wikipedia.org/wiki/Quartile">quartile</a>). 28756489 is the maximum value present. Standard deviation shows values are spread out over a wider range. Average shows average of all non-null values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sk2wLLiU_Z1j5gGzDPIKJw.png" /></figure><blockquote>last_checked_ts : Date along with time data (datetime of the last update from state department health sites)</blockquote><p>Data is non-null and unique. There are no parallel entries on same data-time. Data is unique so count is 1 for all values and hence Histogram is uniform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sv-aC6hjhyuyelPW0kKa7w.png" /></figure><h3>Summary</h3><p>Profiling helps you measure the quality of your data, detect problems like missing or duplicate values and gives insights. Organizations can accordingly plan the strategy to monitor and possibly fix their data using such insights.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8a1076c4ddb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clairvoyantblog/data-profiling-using-dataplex-d8a1076c4ddb">Data Profiling Using Dataplex</a> was originally published in <a href="https://medium.com/clairvoyantblog">Clairvoyant Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top 5 FAQs on Cache answered with Examples]]></title>
            <link>https://medium.com/clairvoyantblog/top-5-faqs-on-cache-answered-with-examples-3f1894393edf?source=rss-5529260323da------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f1894393edf</guid>
            <category><![CDATA[cache]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[system]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Ojal Bhandarkar]]></dc:creator>
            <pubDate>Mon, 07 Mar 2022 19:55:10 GMT</pubDate>
            <atom:updated>2022-03-07T19:55:10.421Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding basic concepts of caching which will help you to decide which cache strategy is appropriate while designing a system</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/0*lyBQE3fiCrSKgz53" /><figcaption>Cache and Debug (source: <a href="https://blog.filestack.com/tutorials/cache-and-debug/">link</a>)</figcaption></figure><blockquote><strong><em>What is Cache?</em></strong></blockquote><p>In computing, a cache is a hardware or software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere. (Ref: <a href="https://en.wikipedia.org/wiki/Cache_(computing)#:~:text=listen)%20KASH)%20is%20a%20hardware,copy%20of%20data%20stored%20elsewhere.">Cache (computing) — Wikipedia</a>)</p><p>I will explain this using the below real-time example:</p><blockquote><em>In morning, </em><strong><em>Bob</em></strong><em> and </em><strong><em>Alice</em></strong><em> were chatting.</em></blockquote><blockquote><strong><em>Alice: </em></strong><em>Hi Bob, I need to give some gifts to my team tomorrow. There are 79 members in my team and our budget is $1,299 for each member. Please help me to calculate the final budget.</em></blockquote><blockquote><strong><em>Bob:</em></strong><em> I am not that good with math. Let me use a calculator. It’s $102,621!</em></blockquote><blockquote><em>In evening </em><strong><em>Bob</em></strong><em> met </em><strong><em>Jay.</em></strong></blockquote><blockquote><strong><em>Jay:</em></strong><em> Hi Bob, I have some good news. Just now, I got a contract for 79 parks where I need to install 1299 benches in each. Do you have a calculator handy please help me to find total benches I need to order by today.</em></blockquote><blockquote><strong><em>Bob:</em></strong><em> Wow what a coincidence, here we don’t need a calculator, it’s 102621.</em></blockquote><blockquote><strong><em>Jay: </em></strong><em>Thanks Bob, that was quick.</em></blockquote><p>Well, the story is not that interesting, but enough to give a basic idea about cache. Here if we noticed Bob has an answer somewhere in his memory that’s why he was able to answer Jay quickly without any calculations that memory is nothing but cache.</p><p>The below diagram will help you understand more about caching.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VA_ZWgLd_DbKRxsf.png" /><figcaption>Fig1-<a href="https://cdn-images-1.medium.com/max/1600/0*VA_ZWgLd_DbKRxsf.png">Cache</a></figcaption></figure><blockquote><strong><em>What are Cache Events?</em></strong></blockquote><p>Now that we understand what a cache is, let&#39;s understand cache events in more detail.</p><h4>Cache Hit</h4><p>If the requested data <strong>is</strong> present in the cache, then it is a cache hit. The cache will return the response and the database can relax and have coffee.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*NYeuLLVwRekTmn7G.png" /><figcaption>Fig2-<a href="https://cdn-images-1.medium.com/max/1600/0*NYeuLLVwRekTmn7G.png">Cache Hit as Req #909 is present in cache</a></figcaption></figure><h4>Cache Miss</h4><p>If the requested data <strong>is not</strong> present in the cache, then it is a cache miss. The database will return the response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*G7J2ItGV2GiL78nC.png" /><figcaption>Fig3-<a href="https://cdn-images-1.medium.com/max/1600/0*G7J2ItGV2GiL78nC.png">Cache Miss as Req #800 is not present in cache</a></figcaption></figure><blockquote><strong><em>Where can you add Cache?</em></strong></blockquote><p>You can add cache at any stage of your system. It can be at the front-end or back-end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5Z8jBs3QeUflsfBf.png" /><figcaption>Fig4-<a href="https://cdn-images-1.medium.com/max/1600/0*5Z8jBs3QeUflsfBf.png">Cache can be added at any level.</a></figcaption></figure><blockquote><strong><em>How to keep the cache updated?</em></strong></blockquote><p>It is very important to keep the cache updated and synced with the database. Otherwise, we will end up with inconsistent responses. To understand the problem please refer to the below example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*l_VJpUu8e-iLoPdR.png" /><figcaption>Fig5-<a href="https://cdn-images-1.medium.com/max/1600/0*l_VJpUu8e-iLoPdR.png">Response for request #102 should be #800 but there is no communication between DB and Cache resulting to older response #107 because of cache hit.</a></figcaption></figure><p>To avoid the above problem we have a concept called <strong>Cache Invalidation</strong>. This provides a different pattern to keep the cache updated.</p><p>Below are the patterns to keep the cache updated and in sync with the database:</p><h4>Write through Cache</h4><p>Data will be written in the cache than in the database. Data modeling of cache and database should be the same. The cache layer adds an extra layer of latency while writing to the database. Cache failure can cause a system failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cFU2h1g-I3uIRgGt.png" /><figcaption>Fig6-<a href="https://cdn-images-1.medium.com/max/1600/0*cFU2h1g-I3uIRgGt.png">Write through cache</a></figcaption></figure><h4>Write around Cache</h4><p>Data will be written in the database than in the cache. Data modeling of cache and database should be the same. It will be helpful to reduce cache latency as we are directly writing in the database first. The system will work even if the cache goes down.<br>Application code should be responsible to keep cache and database in sync.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IY09O5JHNJy_VLGg.png" /><figcaption>Fig7-<a href="https://cdn-images-1.medium.com/max/1600/0*IY09O5JHNJy_VLGg.png">Write around cache</a></figcaption></figure><h4>Write back Cache</h4><p>Data will be written in the cache then a batch of cached data will be written in the database in a provided interval. Data modeling of cache and database should be the same. Database downtime can be handled by the cache for the time as the cache has data in bulk. Cache failure can cause a system failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*glFEPRxGum9Ym1D7.png" /><figcaption>Fig8-<a href="https://cdn-images-1.medium.com/max/1600/0*glFEPRxGum9Ym1D7.png">Write back cache</a></figcaption></figure><blockquote><strong><em>What is Eviction Policy in Cache?</em></strong></blockquote><p>An eviction policy helps the cache to decide which data needs to be deleted when the cache is full.</p><h4>Least Recently Used (LRU)</h4><p>The “least recently used” data is nothing but older data that needs to be purged in this cache pattern. Let’s take an example of recipes asked by clients in the recipe list.</p><blockquote><em>Users asked for a ice-cream and burger recipe, so these two recipes are in demand. Nowadays, users are more interested in pizza recipes, so we need to add this in cache, but the cache is full. So, we will delete the salad recipe as users are not interested. Here, the salad recipe is least recently used.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*sFmnHVSwKrYi-_Ad.png" /><figcaption>Fig9-<a href="https://cdn-images-1.medium.com/max/1600/0*sFmnHVSwKrYi-_Ad.png">LRU cache</a></figcaption></figure><h4>Least Frequently Used (LFU)</h4><p>The “least frequently used” data is cold data which is not in demand that needs to be purged when the cache is full. LFU data can be old data or new data. Let’s understand this with a real-time example of multiple word suggestions on a keypad.</p><blockquote><em>Whenever a user types “I am at” into the keypad, the suggestion shows different most frequently used options. Nowadays, the user is completing a sentence with “I am at hotel”, which is the most frequently used data. So, cache will delete “airport” from the cache, as it is full and it is the least frequently used data for user.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*wthdcnBYCbv7CUzl.png" /><figcaption>Fig10-<a href="https://cdn-images-1.medium.com/max/1600/0*wthdcnBYCbv7CUzl.png">LFU cache</a></figcaption></figure><h4>Most Recently Used (MRU)</h4><p>The “most recently used” is new data for the user which will be evicted from the cache. MRU is used in situations where, instead of new data, older data is much more accessible. Let’s understand this with a real-time example of the Tinder dating application.</p><blockquote><em>Tinder generally caches all the potential matches of a user. When a user either swipes left or right on a profile, the app shouldn’t recommend the same profile to the user again. So here the most recently viewed profile is MRU, which will be deleted from the cache.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*R-IBT6HAKlcGj6vH.png" /><figcaption>Fig11-<a href="https://cdn-images-1.medium.com/max/1600/0*R-IBT6HAKlcGj6vH.png">MRU cache</a></figcaption></figure><h4>Random Replacement</h4><p>The random cache replacement policy selects random data from the cache to be deleted. It doesn’t require storing any previous or new data tracking information. Let’s understand this with a real-time example of fetching student data from a school system.</p><blockquote><em>When there is new student id entry or a request for fetching the details of a student, the cache policy will randomly select one student id from the cache and delete it to make room for a new student id.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*SwTdx4bqfUjXyVnr.png" /><figcaption>Fig12-<a href="https://cdn-images-1.medium.com/max/1600/0*SwTdx4bqfUjXyVnr.png">Random replacement</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f1894393edf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clairvoyantblog/top-5-faqs-on-cache-answered-with-examples-3f1894393edf">Top 5 FAQs on Cache answered with Examples</a> was originally published in <a href="https://medium.com/clairvoyantblog">Clairvoyant Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>