<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Murari Ramuka on Medium]]></title>
        <description><![CDATA[Stories by Murari Ramuka on Medium]]></description>
        <link>https://medium.com/@murari045?source=rss-c7d6ba262ba9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YgZ-P5DHFjJRqqmlw_7vsw.png</url>
            <title>Stories by Murari Ramuka on Medium</title>
            <link>https://medium.com/@murari045?source=rss-c7d6ba262ba9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:42:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@murari045/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Governance with Dataplex]]></title>
            <link>https://medium.com/google-cloud/data-governance-with-dataplex-f8c0d4b1c82e?source=rss-c7d6ba262ba9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8c0d4b1c82e</guid>
            <category><![CDATA[data-fabric]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[dataplex]]></category>
            <dc:creator><![CDATA[Murari Ramuka]]></dc:creator>
            <pubDate>Wed, 19 Apr 2023 16:27:14 GMT</pubDate>
            <atom:updated>2023-04-24T08:37:59.901Z</atom:updated>
            <content:encoded><![CDATA[<p>Gone are those days when Data governance process was good to have, now it is mandatory for all the enterprise organization. Robust Data governance process enable data driven culture in the organisation by providing right set of data to right user at right time.</p><p><strong>What is Dataplex?</strong></p><p><strong>Dataplex’s Intelligent Data Fabric, available on Google Cloud</strong>, enables organizations to centrally discover, manage, monitor, and manage data across <strong>data lakes, data warehouses, and data stores</strong> with consistent controls that enable access data and perform analyses.</p><p><strong>What are the Benefits of Dataplex?</strong></p><p><strong>Simplified data discovery</strong></p><p>· Automates data discovery, classification, and metadata enrichment of structured, semi-structured, and unstructured data, stored in Google Cloud and beyond, with built-in data intelligence.</p><p><strong>Data organization and life cycle management</strong></p><p>· Logically organize your data that spans multiple storage services into business-specific domains using Dataplex lakes and data zones.</p><p><strong>Centralized security and governance</strong></p><p>· Enable central policy management, monitoring, and auditing for data authorization and classification, across data silos.</p><p><strong>Built-in data quality and lineage</strong></p><p>· Automate data quality across distributed data and enable access to data you can trust. Use automatically captured data lineage to better understand your data, trace dependencies, and effectively troubleshoot data issues.</p><p><strong>Serverless data exploration</strong></p><p>· Interactively query fully governed, high-quality data using a serverless data exploration workbench with one-click access to Spark SQL scripts and Jupyter notebooks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1iiQxwVIv9DPrq2D.jpg" /><figcaption>What is Dataplex</figcaption></figure><p>Let’s take a use case where company XYZ has a vision to setup a datamesh landscape for their organization. They want to enable business and operational user with right set of data information to imbibe the data driven culture.</p><p>In order to solve this use case a domain based architecture for each functions (i.e. Sales, Finance, Marketing, and Operation etc) is needed. Dataplex will enable domain based architecture required by XYZ organization and then adding respective assets to the domain. With Dataplex business user can view required data set and take the respective decision. So if we segregate our entire database into particular domains such as Sales, Production, Manufacturing. It makes our visibility to our database clearer and helps us to make more effective insights over it .Under the hood, Dataplex take care of central access control along with right set of template tagging, cataloguing and data masking. With this approach any organization would be able to enable the data for their end user by applying right governance and eliminating the data deduplicacy.</p><p>One of the important value add for Enterprises offered by Dataplex is to ensure that high-quality data is easily discoverable and accessible for analysis, across multiple silos, to a growing number of people and tools within their organization. They are often forced to make trade-offs — moving and duplicating data across silos to enable different analytic use cases, or keeping their data distributed but limiting decision-making flexibility.</p><p><strong>With Dataplex</strong>, an intelligent data fabric that provides a way to centrally manage, monitor, and control your data across data lakes, data warehouses, and data marts, and make that data available to a variety of analytics and data science tools.</p><p><strong>Dataplex</strong> provides an integrated analytics environment that combines the best of Google Cloud and open source tools so you can quickly manage, secure, integrate and analyze data at scale. With built-in data intelligence using Google’s artificial intelligence (AI) and machine learning (ML) capabilities, and a flexible consumption model, you can now spend less time fighting infrastructure and more time focused on driving business results.</p><p><strong>Dataplex</strong>, have a built in feature named as Data<strong> </strong>Catalog. <strong>Data Catalog</strong> is a fully managed, scalable metadata service. Dataplex provides the organization with all the data governance as they to handle their issues related to data.</p><p>Data governance is a critical aspect of data management. It ensures that data is accurate, secure, and used appropriately. <strong>Data Catalog</strong> provide a central location for managing data governance policies and procedures, ensuring that data is managed in accordance with relevant regulations and best practices.</p><p><strong>Advanced data catalog features:</strong></p><p>Advanced data catalog does much more than manage data or make data available. Modern data catalog provides all of the above features and more, making them even more powerful tools for any business aiming to become a truly data-driven organization.</p><p><strong>Data lineage:</strong></p><p>Data lineage is another important aspect of data management and one for which a modern data catalog should enable a seamless user experience. A data lineage refers to the historical record of a data asset, from its origin to its current state, any transformations and processes it has undergone.</p><p>A modern data catalog helps trace the data lineage and enables organizations to understand where their data came from, how it was transformed, and who used it. Data pipelines are important for data quality and compliance purposes because they allow data professionals to understand how the data was processed and identify potential problems.</p><p>Enterprises have data that is distributed across data lakes, data warehouses, and data marts. Dataplex allows you to discover, manage and unify this data without any data movement, organize it according to your business needs and centrally manage, monitor and control it. Dataplex helps you standardize and unify metadata, security policies, governance, classification and data lifecycle management across this distributed data.</p><p><strong>Securing your Data Lake with Dataplex:</strong></p><p>The Dataplex security model allows you to manage who has access to perform the following tasks:</p><p>· Administering a lake (creating and attaching assets, zones, and additional lakes)</p><p>· Accessing data connected to a lake through the mapping asset (Google Cloud resources such as Cloud Storage buckets and BigQuery datasets)</p><p>· Accessing metadata about the data connected to a lake</p><p>· An administrator for a lake controls access to Dataplex resources (lake, zone, and assets) by granting the following basic and predefined roles.</p><p>Dataplex Labs : <a href="https://github.com/GoogleCloudPlatform/dataplex-labs/tree/main/dataplex-quickstart-labs">https://github.com/GoogleCloudPlatform/dataplex-labs/tree/main/dataplex-quickstart-labs</a></p><p>Thanks for your time to read this blog. Stay tuned for part # 2.</p><p>Follow me at Linkedin : <a href="https://www.linkedin.com/in/murari-ramuka-98a440a/">https://www.linkedin.com/in/murari-ramuka-98a440a/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8c0d4b1c82e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/data-governance-with-dataplex-f8c0d4b1c82e">Data Governance with Dataplex</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging Redis Platform to optimize search for Fraud Detection]]></title>
            <link>https://medium.com/@murari045/leveraging-redis-platform-to-optimize-search-for-fraud-detection-256c0c75140c?source=rss-c7d6ba262ba9------2</link>
            <guid isPermaLink="false">https://medium.com/p/256c0c75140c</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[fraud]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Murari Ramuka]]></dc:creator>
            <pubDate>Sat, 20 Aug 2022 06:33:16 GMT</pubDate>
            <atom:updated>2022-08-22T15:36:00.293Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Leveraging Redis Platform to optimize search for Fraud Detection</strong></p><p><strong>#What is Fraud Detection</strong></p><p>Fraud detection is a set of activities performed to prevent money or property from being obtained under false pretences. Fraud detection is used in many industries such as banking or insurance. In banking, fraud can involve forging checks or using stolen credit cards. Other forms of fraud may include exaggerating losses or causing an accident with the sole intention of a payout.</p><p>With the unlimited and growing number of ways someone can commit fraud, detection can be difficult. Activities such as reorganization, downsizing, transitioning to new information systems, or encountering a cyber-security breach could weaken an organization’s ability to detect fraud. Techniques such as real-time fraud monitoring are recommended. Organizations should look for fraud in financial transactions, locations, devices used, sessions initiated and authentication systems.</p><p><strong><em>An</em></strong> <strong><em>Example</em></strong>,<br> Imagine that your ads are generating a lot of traffic, but you’re not getting the desired results from your ad spend. This may not be a coincidence — fraudsters often try to steal digital advertising marketing budgets using a variety of sophisticated mechanisms. Fake clicks may look like a real user is interacting with the ad, but in reality, when these fake clicks lead to an install, the cost of the install goes into the fraudster’s pocket. As companies’ willingness to spend more on digital advertising grows, so do the number of fraudsters in the ad markets.</p><p>#<strong>Fraud Detection Techniques:</strong></p><p>Fraud usually involves multiple repeated methods, so looking for patterns is a general focus for fraud detection. For example, data analysts can prevent insurance fraud by creating algorithms to detect patterns and anomalies. Fraud detection can be separated using statistical data analysis or artificial intelligence (AI) techniques.</p><p>Statistical data analysis techniques include:</p><p>1. Calculation of statistical parameters</p><p>2. Regression analysis</p><p>3. Probability distributions and models</p><p>4. Data matching</p><p><strong>Artificial intelligence</strong> techniques used to detect fraud include:</p><p>§ Data mining classifies, groups and segments data to search through millions of transactions, find patterns and detect fraud.</p><p>§ Neural networks learn suspicious-looking patterns and use them for further detection.</p><p>§ Machine learning automatically identifies characteristics found in fraud.</p><p>§ Pattern recognition detects classes, clusters, and patterns of suspicious behaviour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*dYdKApSzko1io6sV9YcerQ.png" /><figcaption>Image 1</figcaption></figure><p><strong>#Types of Fraud</strong></p><p>I. A common type of bank fraud is the takeover of a customer account. This is when someone illegally gains access to a victim’s bank account using bots. Other examples of banking fraud include the use of malicious apps, use of false identities, money laundering, credit card fraud, and mobile fraud.</p><p>II. Insurance fraud includes premium diversion fraud, which is the misappropriation of premiums, or free churning, which is excessive trading by a stockbroker to maximize commissions. Other forms of insurance fraud include property diversion, workers’ compensation fraud, auto accident fraud, stolen or damaged car fraud, and house fire fraud. The motive behind all insurance fraud is financial gain.</p><p>III. Government fraud commits fraud against federal agencies such as the US Department of Health and Human Services, the Department of Transportation, the Department of Education, or the Department of Energy. Types of government fraud include billing for unnecessary procedures, overcharging for items that cost less, providing old equipment while billing for new equipment, and reporting hours worked for a worker who does not exist.</p><p>IV. Healthcare fraud includes drug and health fraud as well as insurance fraud. Health care fraud is committed when someone defrauds an insurance company or a government health care program.</p><p><strong>#What is Redis?</strong></p><p>Redis is an open source (BSD licensed), in-memory data structure store used as a database, cache, message broker, and streaming engine.</p><p>Redis Cache delivers sub-millisecond response times, allowing fast and powerful real-time applications in industries such as games, fintech, ad-tech, social media, healthcare, and IoT. This tutorial shows a basic mathematical dashboard application written in NodeJS (JavaScript) that uses the Redis Bitmap data type.</p><p>Redis provides data structures such as <a href="https://redis.io/topics/data-types-intro#strings">strings</a>, <a href="https://redis.io/topics/data-types-intro#hashes">hashes</a>, <a href="https://redis.io/topics/data-types-intro#lists">lists</a>, <a href="https://redis.io/topics/data-types-intro#sets">sets</a>, <a href="https://redis.io/topics/data-types-intro#sorted-sets">sorted sets</a> with range queries, <a href="https://redis.io/topics/data-types-intro#bitmaps">bitmaps</a>, <a href="https://redis.io/topics/data-types-intro#hyperloglogs">hyperlog logs</a>, <a href="https://redis.io/commands/geoadd">geospatial indexes</a>, and <a href="https://redis.io/topics/streams-intro">streams</a>. Redis has built-in <a href="https://redis.io/topics/replication">replication</a>, <a href="https://redis.io/commands/eval">Lua scripting</a>, <a href="https://redis.io/topics/lru-cache">LRU eviction</a>, <a href="https://redis.io/topics/transactions">transactions</a>, and different levels of <a href="https://redis.io/topics/persistence">on-disk persistence</a>, and provides high availability via <a href="https://redis.io/topics/sentinel">Redis Sentinel</a> and automatic partitioning with <a href="https://redis.io/topics/cluster-tutorial">Redis Cluster</a>.</p><p>Let’s look towards an example that helps us to understand the concept of Fraud Detection using the Redis Enterprise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*fgVoj_4vEq04wuzlretTPA.png" /><figcaption>Image 2</figcaption></figure><p>Following are the steps that can help to create the above implementations:</p><p>1. We can deploy our Python-Flask application on GCP using multiple ways such as:</p><p>a. Kubernetes Engine</p><p>b. Containers</p><p>c. From Cloud Shell</p><p>2. For easy deployment we can choose cloud shell, that can be deployed using the python scripting.</p><p>3. In the application we must create a Connection with our Redis Enterprise.</p><p>4. We import the redis package for package. All the core Redis commands are available in this Redis package.</p><p>5. We import the redisbloom package. Since RedisBloom is a module, the clients used to interact with this module are also different.</p><p>6. <strong>Click Spamming: </strong>The app is configured to allow two events in a window of 10 seconds from the same device. To verify, make more than two curl requests within 10 seconds and the result will be click_spam.</p><p>7. We can use Looker to plot the Fraud Detection by connecting the Redis to Looker.</p><p><strong>Conclusion:</strong></p><p>• If we consider the entire flow starting from fraud control, from event streaming to data processing to visualization (using reports), all this would require multiple components and extensive orchestration. With the Redis Ecosystem, most of this is removed.</p><p>• This is just the beginning of the additional checks that can be performed on events. Many other checks can be done using modules and data structures. For example, Redis provides geospatial data structures built on sorted sets. Since latitude and longitude can be derived from IP using IP to location conversion providers, a lot of information can be derived about whether an event may be fraudulent or not.</p><p>• The redis-cell key limit request evaluation module can be used to completely deny service requests.</p><ul><li>RedisAI and neural-redis can be used to improve fraud identification efficiency.</li></ul><p><strong>Learn more:</strong></p><ul><li><a href="https://redis.info/3NBGJRT">Try Redis Cloud for free</a></li><li><a href="https://redis.info/3Ga9YII">Watch this video on the benefits of Redis Cloud over other Redis providers</a></li><li><a href="https://redis.info/3LC4GqB">Redis Developer Hub — tools, guides, and tutorials about Redis</a></li><li><a href="https://redis.info/3wMR7PR">RedisInsight Desktop GUI</a></li><li><a href="https://cloud.google.com/free">Try GCP for Free</a></li><li><a href="https://cloud.google.com/sdk/docs/cheatsheet">GCloud CLI command</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=256c0c75140c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a real time analytics with Redis]]></title>
            <link>https://medium.com/google-cloud/build-a-real-time-analytics-with-redis-b94e502864f9?source=rss-c7d6ba262ba9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b94e502864f9</guid>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Murari Ramuka]]></dc:creator>
            <pubDate>Sat, 25 Jun 2022 09:58:12 GMT</pubDate>
            <atom:updated>2022-06-26T14:45:22.729Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>1.</strong> <strong>What is real-time analytics?</strong></p><p>Real-time analytics allow businesses to retrieve data and do something about the data immediately or after the data enters their system. Real-time app statistics answer questions in seconds. They handle large amounts of data at high speeds and low response times. Real-time statistics refer to the process of preparing and measuring data as soon as it enters a website. In other words, users get the data or they can reach conclusions sooner (or sooner) the data enters their system. Real-time statistics allow businesses to respond without delay. They can take advantage of opportunities or prevent problems before they occur.</p><p>Real-time math is a discipline that uses intelligence and statistics in data to provide information to make better decisions faster. In some usage cases, real time means that the calculations are completed within a few seconds or minutes after the new data arrives. The required real-time statistics wait for users or systems to request a query and submit statistical results. Ongoing real-time statistics are very effective and alert users or trigger responses as events occur.</p><p><strong>2.</strong> <strong>What is the importance of real-time analytics?</strong></p><p>Real-time analytics enable businesses to respond without delay, quickly detect and respond to patterns in users’ behaviour, exploit potential opportunities and prevent problems before they arise.</p><p>Entities that use real-time analysis significantly reduce risk across their entire company as the system uses data to predict results and suggest alternatives rather than relying on predictions based on past events or recent scans — as is the case with historical data analysis. Real-time statistics provide details about what is happening right now.</p><p><strong>Other benefits of real-time statistics include:</strong></p><p><strong>Viewing data: </strong>Real-time data can be visualized and reflect events across the company as they occur, while historical data can only be entered into a chart to convey a complete perspective.</p><p><strong>Improved competition:</strong> Businesses that use real-time statistics can identify trends and benchmarks faster than their competitors who still use historical data. Real-time statistics also allow businesses to review performance reports from partners and competitors quickly.</p><p><strong>Specific information:</strong> Real-time analysis focuses on quick analysis that is always useful for creating focused results, which helps ensure that time is not wasted on useless data collection.</p><p><strong>Low cost</strong>: While real-time technology can be expensive, their many and continuous benefits make them a great advantage if used long-term. In addition, technology helps to prevent delays in using resources or accessing information.</p><p><strong>Quick results:</strong></p><p>The ability to quickly separate raw data allows queries to collect relevant data and process it quickly. This, in turn, allows for faster and better trend predictions and decision-making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*i38ZUT3ad5FnHvpoaD5HXg.png" /></figure><p>Speed ​​is a key advantage of real-time data analysis. The less time a business has to wait for data to reach the time it arrives and is processed, the more quickly an entity can use data to make changes and work with important decisions. For example, analyzing the monitoring data from the production line can help to intervene early before equipment malfunctions.</p><p>Similarly, real-time data analysis tools allow companies to see how users interact with a product when it is released, which means there is no delay in understanding user behavior in making the necessary changes.</p><p>Real-time analysis offers the following advantages over standard calculations:</p><p>1. Create interactive analysis tools.</p><p>2. Share information on transparent dashboards.</p><p>3. Customize your behaviour.</p><p>4. Make adjustments as needed.</p><p>5. Use machine learning.</p><p>Other benefits and uses include:</p><p><strong>Local data management</strong> — Helps determine the appropriate local data sets and what needs to be updated to get the full local intelligence.</p><p><strong>Confusing discovery </strong>— Identifies statistics caused by security breaches and technical failures.</p><p><strong>Better Marketing </strong>— Gets information on demographics and customer behavior to improve the effectiveness of advertising and marketing campaigns. It helps determine the best pricing strategies and target audience.</p><p><strong>3.</strong> <strong>Why to use Redis?</strong></p><p>Functional statistics dashboards serve a number of purposes. They allow you to share data and provide you with all that important information to make decisions that change the game very quickly. Creating a real-time dynamic dashboard using a standard social networking site may require a set of complex questions. By using NoSQL website like <strong>Redis</strong>, you can create a powerful interactive and dynamic dashboard with a small number of <strong>Redis</strong> commands.</p><p><strong>#What is Redis?</strong></p><p>Redis is an open source (BSD licensed), in-memory <strong>data structure store</strong> used as a database, cache, message broker, and streaming engine.</p><p>Redis Cache delivers sub-millisecond response times, allowing fast and powerful real-time applications in industries such as games, fintech, ad-tech, social media, healthcare, and IoT. This tutorial shows a basic mathematical dashboard application written in NodeJS (JavaScript) that uses the Redis Bitmap data type.</p><p>Redis provides data structures such as <a href="https://redis.io/topics/data-types-intro#strings">strings</a>, <a href="https://redis.io/topics/data-types-intro#hashes">hashes</a>, <a href="https://redis.io/topics/data-types-intro#lists">lists</a>, <a href="https://redis.io/topics/data-types-intro#sets">sets</a>, <a href="https://redis.io/topics/data-types-intro#sorted-sets">sorted sets</a> with range queries, <a href="https://redis.io/topics/data-types-intro#bitmaps">bitmaps</a>, <a href="https://redis.io/topics/data-types-intro#hyperloglogs">hyperlog logs</a>, <a href="https://redis.io/commands/geoadd">geospatial indexes</a>, and <a href="https://redis.io/topics/streams-intro">streams</a>. Redis has built-in <a href="https://redis.io/topics/replication">replication</a>, <a href="https://redis.io/commands/eval">Lua scripting</a>, <a href="https://redis.io/topics/lru-cache">LRU eviction</a>, <a href="https://redis.io/topics/transactions">transactions</a>, and different levels of <a href="https://redis.io/topics/persistence">on-disk persistence</a>, and provides high availability via <a href="https://redis.io/topics/sentinel">Redis Sentinel</a> and automatic partitioning with <a href="https://redis.io/topics/cluster-tutorial">Redis Cluster</a>.</p><p><strong>TAKING A POINT OF VIEW TO IMPLEMENT THIS IN GOOGLE CLOUD PLATFORM</strong></p><p>In today’s fast-paced world, the emphasis is on instant gratification. Common operating conditions include SaaS operators who provide real-time metrics for their KPIs or marketers’ need for quick data on the performance of their offerings or website testing.</p><p>This solution will show you how to build a real-time website statistics dashboard in GCP.</p><p><strong>Google Cloud Components used:</strong></p><p><strong>1.</strong> <strong>Pub/Sub</strong></p><p><strong>2.</strong> <strong>Dataflow</strong></p><p><strong>3.</strong> <strong>Redis Enterprise</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*j58KIk8WP1fwl7Lp5-HndA.png" /></figure><p><strong>Pub/Sub</strong> enables you to create systems of event producers and consumers, called <strong>publishers</strong> and <strong>subscribers</strong>. Publishers communicate with subscribers asynchronously by broadcasting events, rather than by synchronous remote procedure calls (RPCs).</p><p><strong>Dataflow</strong> is a managed service for executing a wide variety of data processing patterns. The documentation on this site shows you how to deploy your batch and streaming data processing pipelines using Dataflow, including directions for using service features.</p><p><strong>Redis Enterprise is</strong> integrated with Google Cloud to provide a best-in-class <a href="https://redis.com/customers/inovonics/">customer experience</a> in two major deployment form factors, a fully managed service, and a self-managed software. Redis operates as a key value store, allowing for many popular usage situations such as cache, session management and works with confidence as the main site for real-time applications. It enables many different verticals such as marketing, financial services, online games, social media, and much more to use your important apps on Google Cloud.</p><p><strong>#How to access Redis Enterprise?</strong></p><p>Customers have the options to subscribe to fully managed service through Google Cloud Marketplace or directly on Redis Enterprise Cloud at <a href="https://app.redislabs.com/">https://app.redislabs.com</a>. If a customer purchases through Google Cloud Marketplace, then following are the benefits:</p><p>1. Google Cloud commit drawdown</p><p>2. Unified billing within Google Cloud</p><p><strong>Comparison of Google Cloud Marketplace and Redis Enterprise Cloud</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*4zsK1pHDzt2iSK5189f9xw.png" /></figure><p>Here comes the solution,</p><p>We are going to start with pub/sub which is an event bus or a message bus which takes all the real-time events from whenever they are being generated. Assume that we have a web application or a web site which is generating all of these events and we are running experiments on that website so first we have a pub sub which takes all the events from the website behind it we have a data flow which is actually going to process that events or process those events generate any sort of metrics. Then instead of putting that into a traditional database we are going to put that into Redis Enterprise which is our managed version of database. Now, why memory store the reason for memory store is Redis allows us to do operations inbuilt which can do all these kinds of cardinality and set operations I don’t have to code for it.</p><p>You’re going to have these events going into pub sub a messaging bus. It’s then going to go into data flow which is going to provide a real-time streaming data pipeline. Which will then go into Redis memory store and then you’ll be able to have a real-time dashboard that’s sitting on top of that effectively.</p><p>Steps to Perform:</p><p>• <strong>Create Pub/Sub Topic</strong>: Pub/Sub is a global message bus enabling easy message consumption in a decoupled fashion. Create a Pub/Sub topic to receive application instrumentation messages.</p><p>• <strong>Create VPC Network</strong>: Creating a separate VPC network with external ingress blocked by a firewall provides basic security for the instance.</p><p>• <strong>Start analytics pipeline:</strong> The analytic metrics pipeline will read click-stream messages from Pub/Sub and update metrics in the Redis database in real-time. The visualization application can then use the Redis database for the dashboard.</p><p>• <strong>Start the website event generator: </strong>Event generator is a Python executable, which needs to keep running, this can be achieved by launching the generator in a <strong>separate shell session</strong>.</p><p>• <strong>Run the Visualization Engine: </strong>Use the simple reporting application located in dashboard/ folder, built using SpringBoot and simple HTML+JS based UI. The application reads the metrics from the Redis database and makes it available to the dashboard UI. The Application server needs to be on the same VPC network as the Redis server, to achieve this for demo purposes, we will use a Proxy VM to tunnel the ports to Cloud Shell VM, as its not on the same network.</p><p><strong>References:</strong></p><p><strong>1.</strong> <a href="https://redis.com/blog/redis-enterprise-on-google-cloud-five-deployment-scenarios/"><strong>https://redis.com/blog/redis-enterprise-on-google-cloud-five-deployment-scenarios/</strong></a></p><p><strong>2.</strong> <a href="https://github.com/GoogleCloudPlatform/redis-dataflow-realtime-analytics"><strong>https://github.com/GoogleCloudPlatform/redis-dataflow-realtime-analytics</strong></a></p><p><strong>Learn More about Redis:</strong></p><p>· <a href="https://redis.info/3NBGJRT">Try Redis Cloud for free</a></p><p>· <a href="https://redis.info/3Ga9YII">Watch this video on the benefits of Redis Cloud over other Redis providers</a></p><p>· Embed this video if possible</p><p>· <a href="https://redis.info/3LC4GqB">Redis Developer Hub — tools, guides, and tutorials about Redis</a></p><p>· <a href="https://redis.info/3wMR7PR">RedisInsight Desktop GUI</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b94e502864f9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/build-a-real-time-analytics-with-redis-b94e502864f9">Build a real time analytics with Redis</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self Healing Data Pipeline]]></title>
            <link>https://medium.com/google-cloud/self-healing-data-pipeline-d54f78983bc8?source=rss-c7d6ba262ba9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d54f78983bc8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Murari Ramuka]]></dc:creator>
            <pubDate>Sun, 13 Mar 2022 08:12:16 GMT</pubDate>
            <atom:updated>2022-03-14T04:28:17.752Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 1 : <a href="https://medium.com/@murari045/self-healing-data-pipeline-3148074057a0">https://medium.com/@murari045/self-healing-data-pipeline-3148074057a0</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/0*hTHjL7poEzRD-rgZ.png" /></figure><p><strong>Problem Statement:</strong></p><p>A company named XYZ Pvt. Ltd., has an employee named Joe Philip, who works as Data Engineer in the organization. The organization has many data pipelines to process/transform the data at large scale, to enable end-to-end analytics use cases. Most of the Data Pipelines take the data from google cloud storage in CSV format and apply business transformation rules over it and then store that processed data into GCP data warehouse service which is BigQuery. Due to multiple Data Quality issues, Joe needs to spend a lot of time in debugging the failed data pipeline and trace the rejected records. Joe thought to automating this process to auto correct the rejected records by leveraging supervised learning of machine learning.</p><p>This corrupt data or this faulty data could cost millions of dollars and wastage of qualified Data Engineers time in the organization. It is not just about one organization. Many organizations face similar types of problems that needs to be handled in a similar way.</p><p>Due to data quality issues(mentioned below), any Data Ingestion Pipeline<strong> </strong>can either <strong><em>fail</em> </strong>or push these records to the <strong><em>DLQ/Reject</em> </strong>directory which leads to <strong><em>loss of important data</em></strong> and <strong><em>business opportunities.</em></strong></p><p>To recover this lost data, it requires <strong><em>manual intervention by Data engineers/stewards</em></strong> which will be multiple iterations of <strong><em>root-cause-analysis </em></strong>for invalid records, manually <strong><em>correcting</em></strong> the data quality issues, and again pushing it back into the <strong><em>data ingestion</em></strong> pipeline/source system.</p><p>How does it looks like in actual environment:</p><p>Source System: GCS Bucket as shown below in the screenshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*0rFMNGJ8eSTu4ld7tHISgQ.png" /></figure><p>Let’s have a look over the source data received from the front-end into GCS bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*cC1ChoMOdFPRH3ErMFKwnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*yEl74AuLzqRHwGo8-3u0jw.png" /></figure><p>When Joe goes share this data with the stackholders, he is been asked to restore all this faulty data to specific format and make the changes manually, which is time consuming and mundane.</p><p>Let’s have a look over the graphical representation of dataflow that discards the erroneous data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*jRYI_q3HDgFBP_fSxd8IOw.png" /></figure><p><strong><em>Dataflow Graph(Data Ingestion Pipeline):</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*0zwbBQIgW0b4jRzQk0zD1Q.png" /></figure><p>There are three steps present in the dataflow job. Following are the steps:</p><p>• <strong>Read CSV File: </strong>In this step select the csv file which needs to be uploaded into the warehouse.</p><p>• <strong>Transformation: </strong>In this step, all the data validations rules are applied. So that any corrupt data doesn’t affect data representation or future calculations.</p><p>• <strong>Store to BigQuery: </strong>In the last step, all the transformed data is uploaded in the bigquery.</p><p>By incorporating <strong><em>Self-Healing pipeline </em></strong>in Data ingestion/Engineering process Joe can reduce all these efforts and utilize his skills for other important tasks.</p><p><strong>Proposed Solution:</strong></p><p>Let’s add two bifurcations in unloading the data in BigQuery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*psxfcwwA0sV_Q29MhC2-BQ.png" /></figure><p>Just before uploading the data into final target layer, source data can be separate out the faulty/rejected records and store them in another dataset or table and correct records will be successfully stored in the another <strong>Bigquery</strong> table as shown above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*tWptAquFpUfddy8Aae7R2w.png" /></figure><p>We can pick up the faulty/rejected records stored in bigquery dataset and apply our ML Algorithms which is built in Vertex AI to correct those records based on past pattern and store them back to the source raw_data table or GCS bucket.</p><p>This can also be implemented into GCP enviornment with the native services like GCS, Dataflow, BigQuery, Vertex AI and Looker (for visualization) to understand data more accurately.</p><p>Let’s take a deeper look on how a data can be transformed with the implementation of Self-Healing Pipeline in the GCP environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*Cf0NGxnNDSz5uDMHWYTtVA.png" /></figure><p><strong>Some of the data quality issues shown in above picture are listed below:</strong></p><p><strong>1.</strong> <strong>Gender Column:</strong></p><p>From the above figure, we can see that there is data discrepancy in the <strong>Gender</strong> Column.</p><p>In dataflow, there is logic that confirms whether the person is A Male or Female by the keyword’s “M” and “F”. But one user’s gender is written “male”. That will make the dataflow to ignore that record and store into rejected/other records.</p><p>This records get rejected, analytics performed on customer might not have 100% accurate result.</p><p>Hence, to improve customer rejected records, Self-Healing Pipeline can help to solve this immense problem.</p><p><strong>2.</strong> <strong>Missing Values:</strong></p><p>- If some values are missing in the dataset, they can be recovered by using imputations and ML algorithms.</p><p>- Developing ML models with the precision of filling in blanks and incorrect data will result in more insights from the source data.</p><p>- As the training data increases, the accuracy of the model increases as well. This can provide consistency, reliability, and data completeness.</p><p><em>Please Note:</em> <em>If a Unique identifier is missing, it can’t be recovered.</em></p><p>We can implement GCP <strong>Vertex AI</strong> in code snippet that can help us predict <strong>zip code </strong>from the combination of state, city and past address pattern as shown below in the dataset.</p><p>§ From Vertex API we can build, develop and scale ML models faster, with pre-defined and custom tooling within a unified artificial intelligence platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*x-NFUg5GimM8SMjxFGnh3g.png" /></figure><p>There could be other data quality problem which can be solved based on the use cases via Vertex AI (Auto ML/Custom Model). This can help us in many ways quickly by understanding the data quality problem and make logical changes to them.</p><p>§ Vertex AI Model can be retrained regularly to detect new data errors. With this implementation over a period Vertex AI model will be able to handle almost all the data quality errors.</p><p>§ These Vertex AI trained models, then further can utilized into different organisations as pretrained model for easy detection of data quality errors.</p><p>§ ML models can learn from a large amount of dataset and figure out patterns</p><p>§ Evolution of models as data evolves</p><p>We can take over the problems that happen because of a DQ (<em>Data Quality</em>) issue. Following is a diagram that explains the Data Utilization difficulty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*16quLsPFKZ3ss-0riEbA_g.png" /></figure><p>We can implement a custom supervised learning model that can understand the data and observe the causes of data quality issues are and make predictions over it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*Qsi6fG6d5gli80i4asIBdQ.png" /></figure><p><em>Please Note:</em> <em>In this section generic ML algorithm has been explained which can vary based on the use cases and sometime can be interpreted as Rule based search.</em></p><p><strong>3.</strong> <strong>Date field and Format:</strong></p><p>This is another type of data quality probelm. If the format of the date field is different for different records it needs to be changed to a single consistent expected format. This can be a rule based formatting which can be applied into the source data to make the date field consistent.</p><p>E.g. mm-dd-yy, yy-mm-dd, dd-mm-yyyy</p><p>E.g. String to Numeric data-type, 21-Jan-2001 to 21–01–2001</p><p><strong>4.</strong> <strong>Data type errors:</strong></p><p>One of the common data quality problem, which impact most of the data pipeline. Expecting a different data type for a particular input attribute in the source data but got a different data type.</p><p>E.g. String value for numeric values.</p><p><strong>Sample Code Explaination with Limited Scenario:</strong></p><p><em>Please Note: Below code example is just for demostration purpose and comparing both the files (Correct and Rejected) for sample use cases.</em></p><p>○ <strong>User Interface Backend:</strong> Self-healing Pipeline is based on a Flask Interface hosted in a Python environment. Where user needs to feed their inputs/source data in the form of data files.</p><p>○ Let’s look into a code example here user needs to upload their files to perform data quality correction operations using self-healing pipeline . In the below python code, you can see we are securing the file and storing it into a File Pointer.</p><p>○ We are also providing the flexibility to specify the delimiter in the source data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*9fgdP0EwRMMMT9BvSpW2rw.png" /></figure><p>○ After reading the input file, the code checks for the columns which are to be corrected as requested by the user.</p><p>○ We are comparing the values which can be corrected from the sample data provided.</p><p>○ We are also providing the user to select specific operations to be performed on their data on each column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*sl7rBPu8DGiRlJDz1lkJXA.png" /></figure><p>○ <strong>Backend Package:</strong> This package is built to execute all the functionalities that are to be offered in the self-healing pipeline. It stores all the ML logics behind the correction operation being implemented on the corrupt data.</p><p>○ Here is some code logic which has been implemented for this demo/article in the below image:</p><p>■ Capitalization</p><p>■ Gender Correction</p><p>■ Indentation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*QCRW60_PFKvKMwY_Yi9vBw.png" /></figure><p>Post completion of self-healing pipeline execution we are providing the two files, named as recoverable file and non-recoverable file.</p><p>• <em>Recoverable: It contains all the corrected records.</em></p><p>• <em>Non-recoverable: It contains the records which cannot be recovered because of its undefined nature or technical restrictions.</em></p><p><strong>User Interface of the tool:</strong></p><ul><li>Here, user has to provide two types of data files. Such as Sample data and corrupt data. As shown in below screenshot.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*emNkn49mzWTS3QnuEg5ofA.png" /></figure><ul><li>Code will read the file metadata and show below screen to apply the data quality checks such as Indentation/Capitalization/Title/Gender. User can select what corrections they want on their data and implement those changes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*Q3RBlS56Biofjy_mAqpbeg.png" /></figure><ul><li>Post selecting the required data quality correction, user can click run button which will take rejected data file and apply all the selected corrections in the rejected data. This saves a lot of manual interventions and records will be auto corrected.</li><li>There will be two output files which will get generated Recoverable and Non-Recoverable files.</li><li>Recoverable file is the data quality issues corrected data file which can be used in next day/run data ingestion cycle.</li><li>Non recoverable file need to go to source system data steward to have a look into those rejected records.</li></ul><p><strong>Closing Notes:</strong></p><p>This is just a high level demonstration of the thought via a simple UI/code which can be further extended to solve more complex data Quality use cases.</p><p>Disclaimer: This is to inform readers that the views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author’s employer, organization, committee or other group or individual.</p><p><em>Interested in learning more about how self healing data pipeline can be implemented ? Reach out to </em><a href="https://www.linkedin.com/in/murari-ramuka-98a440a/"><em>Murari Ramuka</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d54f78983bc8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/self-healing-data-pipeline-d54f78983bc8">Self Healing Data Pipeline</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self Healing Data Pipeline]]></title>
            <link>https://medium.com/google-cloud/self-healing-data-pipeline-3148074057a0?source=rss-c7d6ba262ba9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3148074057a0</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Murari Ramuka]]></dc:creator>
            <pubDate>Sat, 12 Feb 2022 16:26:11 GMT</pubDate>
            <atom:updated>2023-04-24T08:37:33.019Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*ZTjbC590imot4U0ET8kW7A.png" /><figcaption>Self Healing Data Pipeline</figcaption></figure><p>Data Quality is always been a major concern in data engineering process. Below statistics will help to understand the depth of Data Quality issues:</p><ol><li><strong>On an average, it costs about $1 to prevent a duplicate, $10 to correct a duplicate, and $100 to store a duplicate if left untreated.</strong></li><li><strong>25–30% of data becomes inaccurate leading to less effective sales and marketing campaigns.</strong></li><li><strong>Businesses lose as much as ~ 20% of revenue due to poor data quality.</strong></li></ol><p>Now since we understood that Data Quality is a major problem and need immediate attention.</p><p>Let me come to “Self Healing Data Pipeline”. Self-Healing pipeline is a mechanism which has ability to auto correct the erroneous data in the data engineering pipeline based on past patterns. This implementation is a unique combination of Data Engineering and Machine learning.</p><p>•Self-Healing pipeline is a tool/application/pipeline which can be developed into Cloud environment.</p><p>•It is an automated Self Validation Quality Control Data Pipeline process.</p><p>•Self Healing Data pipeline can increase the accuracy of data by filling in the missed or incorrect data. It can also take care of some data abnormalities.</p><h4>How it Works:</h4><ul><li>Based on past occurrences of data patterns, the pipeline has<strong> ability of Automatic Correcting the Data</strong>.</li><li>On the path of Creating Data Quality Assurance “Self-Healing Pipeline” will create <strong>ML Training Models </strong>from Good Set of data and replace Error Data with Accurate data so that no re-processing is required.</li><li>It can also be clubbed together to improvize automated <strong>Modernisation </strong>and <strong>Data Governance.</strong></li><li>With Self Healing Data Pipeline, Data Validity and Data Profiling Analysis for complete dataset and even frequency analysis for Data Completeness can be done.</li><li>It helps in improving <strong>Data Accuracy </strong>and <strong>Data Lineage</strong>.</li></ul><p>There will be more article coming up for Self Healing Data Pipeline implementation soon.<strong> Stay Tuned….</strong></p><p>Disclaimer: This is to inform readers that the views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author’s employer, organization, committee or other group or individual.</p><p><em>Interested in learning more about how self healing data pipeline can be implemented ? Reach out to </em><a href="https://www.linkedin.com/in/murari-ramuka-98a440a/"><em>Murari Ramuka</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3148074057a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/self-healing-data-pipeline-3148074057a0">Self Healing Data Pipeline</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>