<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bertrand Cariou on Medium]]></title>
        <description><![CDATA[Stories by Bertrand Cariou on Medium]]></description>
        <link>https://medium.com/@bcariou?source=rss-614f607ae6d8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Kdpl86Q8Mk5lRCNfQVL-yQ.jpeg</url>
            <title>Stories by Bertrand Cariou on Medium</title>
            <link>https://medium.com/@bcariou?source=rss-614f607ae6d8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:54:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bcariou/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pricing Optimization During The Crisis: Data Is Key]]></title>
            <link>https://medium.com/data-science/pricing-optimization-during-the-crisis-data-is-key-a4d4f6c678b?source=rss-614f607ae6d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4d4f6c678b</guid>
            <category><![CDATA[data-preparation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[pricing-optimization]]></category>
            <category><![CDATA[revenue-management]]></category>
            <dc:creator><![CDATA[Bertrand Cariou]]></dc:creator>
            <pubDate>Thu, 04 Jun 2020 17:39:34 GMT</pubDate>
            <atom:updated>2020-06-05T06:40:35.045Z</atom:updated>
            <content:encoded><![CDATA[<p>The crisis caused by the pandemic is putting unprecedented pressure on customer demand. Organizations, when not massively impacted by these circumstances, must adapt accordingly and quickly to the market shift to sustain or even grow their business and surpass the competition. Optimizing their pricing strategy is then essential to navigating this new normal.</p><p>However, even in ordinary times, rarely does a company correctly price their product on the first try — nor the second, third or fourth try, either. According to a <a href="https://www.bain.com/insights/is-pricing-killing-your-profits">Bain &amp; Company study</a>, a whopping 85% of B2B companies “have significant room for improvement in pricing.” But even if a company does happen to land close to the mark, continuing to inch toward optimal pricing can make a huge difference. <a href="https://hbr.org/1992/09/managing-price-gaining-profit">Harvard Business Review</a> demonstrated that a mere 1% improvement in price generates a 7–11% increase in profits, and the study has since been replicated many times over to become a reliable benchmark.</p><p>I have recently been working on this particular subject with my colleagues from <a href="http://www.thetoplinelab.com/">The Top Line Lab</a>, and especially his founder <a href="https://www.linkedin.com/in/manu-carricano-phd-906455/">Manu Carricano</a> and we put together <a href="https://www.trifacta.com/gated-form/pricing-optimization-whitepaper/"><em>this whitepaper</em></a><em> </em>and delivered a <a href="https://www.trifacta.com/resource-library/pricing-optimization-webinar/"><em>webinar</em></a><em> </em>sharing some of our findings and how to put together a solid and agile framework to monitor and adapt a pricing strategy.</p><h3>Data is Key</h3><p>A fundamental element of pricing optimization is data. Traditionally, companies have anchored their pricing strategy by selecting a handful of relevant data streams to create a fixed price. Increasingly, however, today’s organizations are incorporating more and more elements into a pricing model that is dynamic, not fixed. As pricing becomes a central lever for differentiation, and with these improbable times, organizations need to be able to respond to market changes faster than ever, which means optimizing for a continuum of decisions (such as list price or promotions) based on context (such as localization or special occasion) to serve the function of multiple business objectives (such as net revenue growth or cross-selling).</p><h3>Common Data Model: The First Step Toward Trustworthy Pricing Models</h3><p>A dynamic pricing model has the potential to significantly move the needle on an organization’s bottom line. But it can also test the limits of specific data architectures and data management practices. When incorporating more and more data streams, many organizations have faced repeated “trust” issues due to a lack of data quality, poor data governance across multiple geographies or business units, and extremely long and inefficient deployments.</p><p>One of the first steps towards resolving these issues is developing a Common Data Model (CDM). Having a centralized model for all required data is essential to ensure that users have a single source of truth. A CDM defines standards for the key components that are influential to pricing optimization, including transactions, products, prices, and customers. From there, the standards are applied through a blended dataflow and serve the downstream systems to leverage pricing data (business applications, dashboards, microservices, etc.) with one homogeneous data model. The CDM also offers a consistent way for the various teams involved in the initiative to better collaborate using a common language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*83ycYo_-1fZdo62nB_GkHg.png" /></figure><h3>Proper Data Preparation for Pricing Optimization</h3><p>Having a Common Data Model in place is an important foundational component. But the true test of pricing optimization efficiency will be measured in how quickly pricing and revenue specialists can take action on this data. In other words, how easily can users prepare data, the first and most critical step in data analysis? Historically, the answer has been “not very.” Preparing data, which can include everything from addressing null values to standardizing mismatched data to splitting or joining columns is a time-consuming process that requires a great attention to detail and, often, a large amount of technical skills. It can take up to 80% of the overall analytics process. But by and large, it’s worth the extra time spent — properly prepared data can make the difference between a faulty and accurate final analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D8hHNZ0qSKVs8yzZ.png" /></figure><p>To resolve the data aspect of pricing optimization, I am a big fan of Google Cloud’s answer to the data preparation problem with <a href="https://www.trifacta.com/cloud-dataprep/">Cloud Dataprep</a>, a fully-managed service that leverages <a href="http://trifacta.com/">Trifacta</a> technology and allows users to access, standardize, and unify required data sources. Its machine-learning driven engine and visual interface accelerate the overall data preparation process by up to 90%. Some of the critical steps involved in leveraging Cloud Dataprep for pricing optimization are:</p><ol><li><strong>Assess Data Sources</strong>: Upon ingesting the necessary data for pricing optimization, its contents must be assessed. Each source system will have its own way of describing and storing data. Each source system will also have a different level of accuracy. In this first step, building an inventory is essential in order to get a clear idea of the quality of each source, which will later inform how they should be cleaned and standardized.</li><li><strong>Standardize Data</strong>: After identifying the source systems and assessing their data quality, the next step is actually resolving those data quality issues to achieve data accuracy, integrity, consistency, and completeness. Eventually, this data will be normalized and mapped to its corresponding CDM class of data.</li><li><strong>Unify in One Structure</strong>: Unifying this data into a singular structure consists of joining all individual CDM data classes with attributes from each individual class at the finest granular level. This is a critical step because it creates one source of trusted data for all pricing optimization work.</li><li><strong>Deliver Analytics &amp; ML/AI</strong>: Once data is clean and ready for analysis, analysts can begin to run use cases and scenarios that explore the impacts of pricing changes. It is during this stage where the organization will begin to see dramatic changes to their bottom line — but not without all of the hard upfront work of preparing that data.</li></ol><h3>Learning More</h3><p>Want to learn more about the keys to successfully implement a pricing analytics framework in the cloud? You can <a href="https://www.trifacta.com/gated-form/pricing-optimization-whitepaper/">read the whitepaper</a> we put together with <a href="https://thetoplinelab.com/">TheTopLineLab</a> specialized in the practice of Pricing Optimization. You can also <a href="https://www.trifacta.com/resource-library/pricing-optimization-webinar/">watch the webinar</a> we delivered on that particular subject so you can identify your organization’s weaknesses, better understand the data sources needed on your pricing optimization journey, and review a step-by-step demo of how the Google Cloud Smart Analytics suite enables an essential Common Data Model (CDM).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4d4f6c678b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pricing-optimization-during-the-crisis-data-is-key-a4d4f6c678b">Pricing Optimization During The Crisis: Data Is Key</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get A Data Studio Data Warehouse Ready In Minutes]]></title>
            <link>https://medium.com/@bcariou/get-a-data-studio-data-warehouse-ready-in-minutes-a4511b07d10f?source=rss-614f607ae6d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4511b07d10f</guid>
            <category><![CDATA[google-data-studio]]></category>
            <category><![CDATA[data-preparation]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Bertrand Cariou]]></dc:creator>
            <pubDate>Tue, 14 Apr 2020 09:59:18 GMT</pubDate>
            <atom:updated>2020-04-14T10:10:46.694Z</atom:updated>
            <content:encoded><![CDATA[<h3>Get Your Data Studio Data Warehouse Ready In Minutes</h3><p>This article outlines the basic concepts I learned while exploring Google Cloud Platform to set up an analytic framework for my own reporting needs. I wanted to have the ability to build quickly ad-hoc and recurring reports on large data sets to get the necessary insight to inform some business decisions (and share it with the team) or feed data for some marketing campaigns I was running.</p><p>I am pretty happy with the result and want to share here a summary of what I learned so others can get to similar results faster. I hope it can be useful to you too.</p><h3>What is self-service analytics?</h3><p>For me, the objective of self-service analytics is to empower data-driven professionals like me to create their own end-to-end solution for their analytics, respond to their decision-making requirements, and research for insight. Self-service means that there are no necessary dependencies on others to get the job done and that the technology in use is simple yet powerful and scalable enough to be easily adopted by anyone to deliver results quickly.</p><h3>What are the ingredients for self-service analytics on the Google Cloud Platform (GCP)?</h3><p>To establish a self-service analytics practice, you need to rely on three fundamental constituents: data storage, data preparation, and data visualization.</p><p><strong>Data Storage.</strong> You need a space that your data can live and grow within. This data storage is the foundation to construct your analytics on top of. Within the GCP, the easiest way to store and retrieve your data for analytics at scale is BigQuery.</p><p><strong>Data Preparation.</strong> From the data storage layer, you can prepare your data for reporting and dashboarding. You’ll be using Cloud Dataprep to clean, standardize, combine, and create various calculations and metrics stored in BigQuery so that you can design your visualization layer on top of it. Data</p><p><strong>Visualization.</strong> The visualization layer shows your data in the form of various reports and dashboards containing graphs and tables to visually represent data for your insight and to help your data-driven decision process. You’ll be using Data Studio consuming BigQuery data to create your reports and dashboards. With these fundamental elements in place, you’re ready to implement a scalable and comprehensive self-service analytics solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*ZpxZ1pDL9oQQI1Eagw2Kew.png" /></figure><h3>Understand GCP self-service analytics concepts</h3><p>Here are the basic and minimum concepts for an end-to-end self-service analytics solution built on the GCP. These concepts are the foundation you need to understand and iterate on for your self-service analytics solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yWfRQ-Ht2JngMZVA1zLO0w.jpeg" /><figcaption>GCP Concepts for self-service analytics</figcaption></figure><p>Here is a visual representation of these concepts and their relations to create a self-service analytics solution within the GCP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PpKZEKHPdB6GqF6HXBz1hQ.png" /><figcaption>flow between concepts</figcaption></figure><h3>Moving from theory to practice</h3><p>Still a bit too theoretical? No problem! You can experience it for yourself following <a href="https://medium.com/@bcariou/self-service-analytics-on-bigquery-live-in-30-min-stopwatch-in-hand-b106fc223652">this step-by-step guide to build your first self-service Data Warehouse</a>. You will establish a scalable self-service analytics solution that leverages BigQuery to store and retrieve data; Cloud Dataprep to clean, combine, and create metrics; and Google Data Studio to visually report on top of your data. By exploring each GCP service deeper and iterating on these principles, you will be able to solve any requirements for your analytics in a self-service manner.</p><p><em>Originally published on </em><a href="https://www.trifacta.com/resource-library/getting-started-with-self-service-analytics-on-bigquery-with-cloud-dataprep/"><em>www.trifacta.com</em></a><em> on July 24, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4511b07d10f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get A BigQuery Data Warehouse Ready In Minutes]]></title>
            <link>https://medium.com/@bcariou/get-a-bigquery-data-warehouse-ready-in-minutes-5be513e6bdc7?source=rss-614f607ae6d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5be513e6bdc7</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-preparation]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Bertrand Cariou]]></dc:creator>
            <pubDate>Tue, 14 Apr 2020 09:54:38 GMT</pubDate>
            <atom:updated>2020-04-14T10:09:32.626Z</atom:updated>
            <content:encoded><![CDATA[<h3>Get Your BigQuery Data Warehouse Ready In Minutes</h3><p>This article outlines the basic concepts I learned while exploring Google Cloud Platform to set up an analytic framework for my own reporting needs. I wanted to have the ability to build quickly ad-hoc and recurring reports on large data sets to get the necessary insight to inform some business decisions (and share it with the team) or feed data for some marketing campaigns I was running.</p><p>I am pretty happy with the result and want to share here a summary of what I learned so others can get to similar results faster. I hope it can be useful to you too.</p><h3>What is self-service analytics?</h3><p>For me, the objective of self-service analytics is to empower data-driven professionals like me to create their own end-to-end solution for their analytics, respond to their decision-making requirements, and research for insight. Self-service means that there are no necessary dependencies on others to get the job done and that the technology in use is simple yet powerful and scalable enough to be easily adopted by anyone to deliver results quickly.</p><h3>What are the ingredients for self-service analytics on the Google Cloud Platform (GCP)?</h3><p>To establish a self-service analytics practice, you need to rely on three fundamental constituents: data storage, data preparation, and data visualization.</p><p><strong>Data Storage.</strong> You need a space that your data can live and grow within. This data storage is the foundation to construct your analytics on top of. Within the GCP, the easiest way to store and retrieve your data for analytics at scale is BigQuery.</p><p><strong>Data Preparation.</strong> From the data storage layer, you can prepare your data for reporting and dashboarding. You’ll be using Cloud Dataprep to clean, standardize, combine, and create various calculations and metrics stored in BigQuery so that you can design your visualization layer on top of it. Data</p><p><strong>Visualization.</strong> The visualization layer shows your data in the form of various reports and dashboards containing graphs and tables to visually represent data for your insight and to help your data-driven decision process. You’ll be using Data Studio consuming BigQuery data to create your reports and dashboards. With these fundamental elements in place, you’re ready to implement a scalable and comprehensive self-service analytics solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*ZpxZ1pDL9oQQI1Eagw2Kew.png" /></figure><h3>Understand GCP self-service analytics concepts</h3><p>Here are the basic and minimum concepts for an end-to-end self-service analytics solution built on the GCP. These concepts are the foundation you need to understand and iterate on for your self-service analytics solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yWfRQ-Ht2JngMZVA1zLO0w.jpeg" /><figcaption>GCP Concepts for self-service analytics</figcaption></figure><p>Here is a visual representation of these concepts and their relations to create a self-service analytics solution within the GCP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PpKZEKHPdB6GqF6HXBz1hQ.png" /><figcaption>flow between concepts</figcaption></figure><h3>Moving from theory to practice</h3><p>Still a bit too theoretical? No problem! You can experience it for yourself following <a href="https://medium.com/@bcariou/self-service-analytics-on-bigquery-live-in-30-min-stopwatch-in-hand-b106fc223652">this step-by-step guide to build your first end-to-end self-service data Warehouse</a>. You will establish a scalable self-service analytics solution that leverages BigQuery to store and retrieve data; Cloud Dataprep to clean, combine, and create metrics; and Google Data Studio to visually report on top of your data. By exploring each GCP service deeper and iterating on these principles, you will be able to solve any requirements for your analytics in a self-service manner.</p><p><em>Originally published on </em><a href="https://www.trifacta.com/resource-library/getting-started-with-self-service-analytics-on-bigquery-with-cloud-dataprep/"><em>www.trifacta.com</em></a><em> on July 24, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5be513e6bdc7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Check-list: selecting the right data transformation service for Google BigQuery]]></title>
            <link>https://medium.com/data-science/check-list-selecting-the-right-data-transformation-service-for-google-bigquery-bd99307da581?source=rss-614f607ae6d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd99307da581</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-preparation]]></category>
            <dc:creator><![CDATA[Bertrand Cariou]]></dc:creator>
            <pubDate>Mon, 13 Jan 2020 20:52:29 GMT</pubDate>
            <atom:updated>2020-04-09T16:22:49.268Z</atom:updated>
            <content:encoded><![CDATA[<h4>Which Google Cloud service can get your data ready for analytics?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDLSqbCrytpAsQrMXHqcWA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@kylejglenn?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Kyle Glenn</a> on <a href="https://unsplash.com/s/photos/choice?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The good news is, when it comes to moving and transforming data for analytics and AI/ML built on top of BigQuery, Google Cloud offers a plethora of native choices. The bad news is that you have to make a choice among many options to ensure the technology will fit your needs and prove to be sustainable over time.</p><p>This article is a check-list to help you choose the best technology (or combination of technologies) that is right for you. If you’re planning to leverage BigQuery for your analytics and data science projects, it means that you will need to clean, transform, and combine data in batches and/or micro-batches. For this reason, I have ruled out <a href="https://cloud.google.com/bigquery/transfer/">Google BigQuery Data Transfer Service</a>, which can only move data into BigQuery but can’t transform it. Also, I limited my research to only native Google Cloud data transformation services, including <a href="https://cloud.google.com/dataproc/">Cloud Dataproc</a>, <a href="https://cloud.google.com/dataflow/">Cloud Dataflow</a>, <a href="https://cloud.google.com/data-fusion/">Cloud Data Fusion</a>, and <a href="https://www.trifacta.com/cloud-dataprep/?utm_source=medium">Cloud Dataprep</a>. Sure, transformations can also happen in BigQuery using SQL, which is quite a powerful language (I have spent years building applications leveraging SQL and stored procedures); however, SQL does not make a sustainable solution for enterprise data transformation services for many reasons and there are also good reasons why there are so many technologies that have been invented to transform data (this would need a complete article on its own).</p><p>So what are the questions you should ask yourself for selecting the right technology?</p><h3>Question 1: Are you looking to write code in a programming language for your solution?</h3><p>If software engineers or developers have to develop and integrate various components to create an application that includes data transformation at scale, then you should look at Cloud Dataflow or Cloud Dataproc. These are code-based solutions that require programming skills such as Java, Python, RestAPI, or Apache Spark. These require fairly technical knowledge. With these services, you already know which transformations need to be created, and you need a development tool to code these transformations. To figure out whether to choose Cloud Dataflow or Cloud Dataproc, <a href="https://stackoverflow.com/questions/46436794/what-is-the-difference-between-google-cloud-dataflow-and-google-cloud-dataproc">read this article</a>, which provides good arguments to use one or the other.</p><h3>Question 2: is your data already in Google Cloud?</h3><p>If your data is not yet in Google Cloud, Cloud Data Fusion, the Extract, Transform, and Load (ETL) solution from Google built on the open-source project <a href="https://cdap.io/">CDAP</a>, provides numerous connectors to bring the data into Google Cloud and BigQuery. Alternatively, you could leverage other ETL solutions you already have in-house that can bring the data into Google Cloud. Google recommends <a href="https://cloud.google.com/bigquery/?utm_source=google&amp;utm_medium=cpc&amp;utm_campaign=emea-gb-all-en-dr-bkws-all-solutions-trial-e-gcp-1007176&amp;utm_content=text-ad-none-any-DEV_c-CRE_335630920200-ADGP_Hybrid+%7C+AW+SEM+%7C+BKWS+~+EXA_1:1_GB_EN_Data+Warehousing_google+bigquery-KWID_43700041731167839-kwd-63326440124-userloc_1006094&amp;utm_term=KW_google%20bigqueryg&amp;ds_rl=1245734&amp;gclid=EAIaIQobChMIqLGjg-OT5AIVhkPTCh0MqwN_EAAYASAAEgL3QPD_BwE#partners">these ETL vendors, including Matillion, Fivetran, Informatica, and many more</a>. ETL solutions such as Cloud Data Fusion are best to develop large-scale data pipelines to move data from on-prem and cloud sources for “hydrating” a data lake or a Data Warehouse (DWH). Data engineers, ETL developers, and data architects will benefit most from Cloud Data Fusion. They will be able to develop solid real-time and batch-data transformation pipelines. Cloud Data Fusion and ETLs are excellent in building stable data pipelines to move data routinely to a DWH or a data lake. They are not meant to create pipelines that evolve often or to put in the hands of some business users.</p><h3>Question 3: does your data initiative need the flexibility to adapt to business requirements often? Does your project need, in addition to data engineers, self-service data transformation by business users or less technically savvy professionals?</h3><p>Data preparation is the most recent evolution of data transformation technologies, bringing these capabilities in the hands of data-driven professionals who are less technical and often reside in a line of business (in addition to the usual technical data specialists such as data engineers). These professionals know the data better than anyone else however, they may lack the technical skills to clean, combine, and enrich the data to make it usable for their needs. Typically they would know Excel, Google Sheets, and SQL. Cloud Dataprep (developed in collaboration with <a href="https://www.trifacta.com/cloud-dataprep/?utm_source=medium">Trifacta</a>) offers a grid-based, machine learning-guided visual interface to enable agile data exploration and assessment to refine, standardize, and combine data for BigQuery for analytics. Users interact with the content of data to iteratively refine and bring together data for their downstream business-driven analytics.</p><h3>Bonus question for Cloud Dataprep or Cloud Data Fusion: are you leaning more toward Cloud Dataproc or Cloud Dataflow?</h3><p>Cloud Data Fusion generates Cloud Dataproc code to transform the data, while Cloud Dataprep generates some Dataflow code to transform the data. Both have their advantages based on your use case and your former experience. This could also be an element to weigh in favor of a particular technology to deliver analytics on top of BigQuery.</p><p>Ultimately, you may need to combine multiple solutions to create end-to-end analytics or machine learning solutions for BigQuery. This is quite often that we see an ETL to bring the data in a Cloud DWH or Data Lake and then Data Preparation technology to expose the data to the consumers.</p><p>The following table summarizes these technologies’ characteristics to guide your choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYsHvunsT0J5PymbLd5O3A.png" /></figure><p><em>Originally published at </em><a href="https://www.trifacta.com/blog/google-fusion-for-bigquery/"><em>www.trifacta.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd99307da581" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/check-list-selecting-the-right-data-transformation-service-for-google-bigquery-bd99307da581">Check-list: selecting the right data transformation service for Google BigQuery</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving data quality for machine learning and analytics with Cloud Dataprep]]></title>
            <link>https://medium.com/google-cloud/improving-data-quality-for-machine-learning-and-analytics-with-cloud-dataprep-c55a47580fd8?source=rss-614f607ae6d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c55a47580fd8</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-preparation]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Bertrand Cariou]]></dc:creator>
            <pubDate>Fri, 06 Sep 2019 17:37:48 GMT</pubDate>
            <atom:updated>2019-09-06T17:37:48.820Z</atom:updated>
            <content:encoded><![CDATA[<p>Data quality is a critical component of any analytics and machine learning initiative, and unless you’re working with pristine, highly-controlled data, you’ll likely face data quality issues. To illustrate the process of turning unknown, inconsistent data into trustworthy assets, we will leverage the example of a forecast analyst in the retail (consumer packaged goods) industry. Forecast analysts must be extremely accurate in planning the right quantities to order. Supplying too much product results in wasted resources, whereas supplying too little means that they risk losing profit. On top of that, an empty shelf also risks consumers choosing a competitor’s product, which can have a harmful, long-term impact on the brand.</p><p>To strike the right balance between appropriate product stocking levels and razor-thin margins, forecast analysts must continually refine their analysis and predictions, leveraging their own internal data as well as third-party data, over which they have no control.</p><p>Every business partner, including suppliers, distributors, warehouses and other retail stores, may provide data (e.g. inventory, forecast, promotions, or past transactions) in various shapes and level of quality. One company may use palettes instead of boxes as a unit of storage, pounds versus kilograms, may have different categories nomenclature and namings, may use a different date format, or will most likely have product SKUs that are a combination of internal and other supplier IDs. Furthermore, some data may be missing or may have been incorrectly entered.</p><p>Each of these data issues represents an important risk to reliable forecasting. Forecast analysts must absolutely clean, standardize, and gain trust in the data before they can report and model on it accurately. This post reviews key techniques for cleaning data with <a href="https://cloud.google.com/dataprep/">Cloud Dataprep</a> and covers new features that may help improve your data quality with minimal effort.</p><p><strong>Basic concepts</strong></p><p>Cleaning data with Cloud Dataprep corresponds to a three-step iterative process:</p><ol><li>Assessing your data quality</li><li>Resolving or remediating any issues uncovered</li><li>Validating cleaned data, at scale</li></ol><p>Cloud Dataprep constantly profiles the data you’re working on, from the moment you open the grid interface and start preparing data. With Dataprep’s real-time Active Profiling, you can see the impact of each data cleaning step on your data.</p><p>The profile result is summarized at the column header with basic data points to point out key characteristics in your data, in the form of an interactive visual profile. By clicking one of these profile column header bars, Cloud Dataprep suggests some transformations to remediate mismatched or missing values. You can always try a transformation, preview its impact, select it or tweak it. At any point, you can always revert to a specific previous step if you don’t like the result.</p><p>With these basic concepts in mind, let’s cover Cloud Dataprep data quality capabilities.</p><h3><strong>1. Assessing your data quality</strong></h3><p>As soon as you open a dataset in the grid interface, you can access to data quality signals that help you assess data issues and guide your work in cleaning the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Halt5KRTg7ynsOR4.png" /><figcaption>Data Quality Header</figcaption></figure><p><strong>Rapid profiling</strong></p><p>You’ll likely scan over your column headers and identify the potential quality issues to understand which columns may need your attention. Mismatched values (red bar) based on the inferred data types, missing values (black) and uneven value distribution (bars) can help you quickly identify which columns need your attention.</p><p>In this particular case, our forecast analyst knows she’ll have to drill down on the `material` field that includes some mismatched and missing values. How should these data defaults impact her forecast and replenishment models?</p><p><strong>Intermediary data profiling</strong></p><p>If you click on a column header, you’ll see some extra statistics in the right panel of Dataprep. This is particularly useful if you expect a specific format standard for a field and want to identify the values that don’t comply to the standard. In the example below, you can see that Cloud Dataprep discovered three different format patterns for the order_date. You might have follow-up questions: can empty order dates be leveraged in the forecast? Can mismatched dates be corrected and how can you correct them?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*iErGmGPBMRkSTQlQ.png" /><figcaption>Data Profiling Preview</figcaption></figure><p><strong>Advanced profiling</strong></p><p>If you click “Show more”, or click the column header menu and “column details” in the main grid, you’ll land on a comprehensive data profiling page with some details about mismatched values, value distribution, or outliers. You can also navigate to the pattern tab to explore the data structure within a specific column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9nWKv-gSZfixmvhi.png" /><figcaption>Data Profiling Details</figcaption></figure><p>These three data profiling capabilities are dynamic by nature in the sense that Cloud Dataprep reprofiles the data in real-time at each step of a transformation, to always present you with the latest information. This helps you clean your data faster and more effectively.</p><p>The value for the forecast analyst is that she can immediately validate as she goes through the process of cleaning and transforming the data so that it fits the format she expects for her downstream modeling and reporting.</p><h3><strong>2. Resolving data quality issues</strong></h3><p>Dynamic profiling helps you assess the data quality at hand, and it is also the point of entry to start cleaning the data. Graph profiles are interactive and offer transformation suggestions as soon as you interact with them. For example, clicking the missing value space in the column header displays transformation suggestions such as deleting the values or setting the values to a default one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yYKsHqlFdFg32LIa.png" /><figcaption>Transformation Suggestions</figcaption></figure><p><strong>Resolving incorrect patterns</strong></p><p>You can efficiently resolve incorrect patterns in a column (such as the recurrent date formatting issue in the order_data column) by accessing the <em>pattern</em> tab in the column details screen. Cloud Dataprep shows you the most frequent patterns. Once you select a target conversion format, Cloud Dataprep displays some transformation suggestions on the right panel, to convert all the data to fit the selected pattern. Watch the animation below, and try it for yourself:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QmxsYnm9cmy4SUq-.gif" /><figcaption>Pattern-Based Transformation</figcaption></figure><p><strong>Highlight over data content</strong></p><p>Another interactive way to clean your data is to highlight over some portion of a value in a cell. Cloud Dataprep will suggest a set of transformations based on your selection, and you can refine the selection by highlighting over some additional content from another cell. Here is an example that extracts the month from the order date in order to calculate the volume per month:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LTzD4-OMHUeCzmdv.gif" /><figcaption>Highlight Over Based Transformations</figcaption></figure><p><strong>Format, replace, conditional functions, and more</strong></p><p>You can find most of the functions you’ll use to clean up data in the Column menu from the <em>format</em> or replace sections, or in the conditional formulas in the icon bar as shown below. These can be useful to convert all product or category names into uppercase or trim the names that have often quotes after import from a CSV or Excel file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ww1KB8A6p7meknfr.png" /><figcaption>Format Functions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1_y3LkGmz4TxM95F.png" /><figcaption>Extract Functions</figcaption></figure><p>The extract functions can be particularly useful to extract a subset of a value within a column. For example, you may want to extract from the product_id “Item: ACME_66979905111536979300 - PASTA RONI FETTUCINE ALFR” each individual component by splitting it on the “ - ” value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Y6Q8clgkE_CVBiK.png" /><figcaption>Delimiter-Based Splitting</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*11uKOGFio7veZCNv.png" /><figcaption>Conditional Functions</figcaption></figure><p>Conditional functions are useful for tagging values that are out of scope. For example, you can write a formula that will tag records when a quantity is over 10,000, which wouldn’t be valid for the order sizes you typically encounter.</p><p>If none of the visual suggestions give you what you require for cleaning your data, you can always edit a suggestion or manually adding a new step in a Dataprep recipe. Type in a search box what you want to do and Cloud Dataprep will suggest some transformations you can then edit and apply to the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3hFvQohPUMCcx8lc.png" /><figcaption>Function Search</figcaption></figure><p><strong>Standardization</strong></p><p>Standardizing values is a way to group similar values into a single, consistent format. This problem is especially prevalent with free-form entries like a product, product categories, company names. You can access the standardization feature from the Column menu. Additionally, Cloud Dataprep can group similar values together by string similarities or by pronunciation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S1mZOTZ--RejZloS.png" /><figcaption>Cluster-Based Standardization</figcaption></figure><p><em>Tip: You can mix-and-match standardization algorithms. Some values may be standardized using spelling, while others are more sensibly standardized based on international pronunciation standards.</em></p><h3><strong>3. Validation at scale</strong></h3><p>The last, critical step of a typical data quality workflow in Cloud Dataprep is to validate that no single data quality issue remains in the dataset, at scale.</p><p><strong>Leveraging sampling to clean data</strong></p><p>Sometimes, the full volume of a dataset won’t fit into Cloud Dataprep via your browser tab (especially when leveraging BigQuery tables with hundreds of millions of records or more). In that case, Cloud Dataprep automatically samples the data from BigQuery to fit it in your local computer’s memory. That might lead you to question: how can you ensure you’ve standardized all the data from one column (e.g. product name, category, region, etc.) or you have cleaned all the date formats from another?</p><p>You can adjust your sampling settings by clicking the sampling icon at the top right and choosing the sampling technique that fits your requirements.</p><ul><li>Select <strong>anomaly-based</strong> to keep all the data mismatched or missing for one of multiple columns</li><li>Select <strong>stratified</strong> to retrieve every distinct value for a particular column (particularly useful for standardization)</li><li>Select <strong>filter-based</strong> to retrieve all the data based on a particular formula (i.e format does not match dd/mm/yyyy)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MXGuavyVTnREbVJE.png" /><figcaption>Sampling Options</figcaption></figure><p><strong>Profiling the data at scale</strong></p><p>At this point, hopefully, you’re happy and confident that your recipe will produce a clean dataset, but until you run it at scale across the whole data set, you can’t ensure all your data is valid. To do so, click the ‘Run Job’ button and check that <strong>Profile Results</strong> is enabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6W1uzq2yMKXvrUYo.png" /><figcaption>Running a Job</figcaption></figure><p>If in the job results you still see some red, this most likely means you need to adjust your data quality rules and try again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w4SINZBxhvYRoKeW.png" /><figcaption>Profiling Result</figcaption></figure><p><strong>Scheduling</strong></p><p>To ensure that the data quality rules you create are applied on a recurring basis schedule your recipes to run automatically. In the case of forecasting, data may change on a weekly basis, so users must run the job every week to validate that all the profile results stay green over time. If not, you can simply reopen and adapt the recipe to address the new data inconsistencies you discovered.</p><p>In the flow view, select <em>Schedule Flow</em> to define the parameters to run the job on a recurring basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B88cHvOlQEEz-fwF.png" /><figcaption>Scheduling a Flow</figcaption></figure><h3><strong>Conclusion</strong></h3><p>Our example here is retail-specific, but regardless of your area of expertise or industry, you may encounter similar data issues. Following this process and leveraging Cloud Dataprep, you can become more effective and faster at cleaning up your data for analytics or feature engineering.</p><p>We hope you that by using Cloud Dataprep, the toil of cleaning up your data and improving your data quality is, well, not so messy. If you’re ready to start, log in to Dataprep via <a href="https://console.cloud.google.com/dataprep">Google Cloud Console</a> to start using this three-step data quality workflow on your data.</p><p><em>Originally published at </em><a href="https://cloud.google.com/blog/products/gcp/improving-data-quality-for-machine-learning-and-analytics-with-cloud-dataprep"><em>https://cloud.google.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c55a47580fd8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/improving-data-quality-for-machine-learning-and-analytics-with-cloud-dataprep-c55a47580fd8">Improving data quality for machine learning and analytics with Cloud Dataprep</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Service Analytics on BigQuery live in 30 min, stopwatch in hand!]]></title>
            <link>https://medium.com/google-cloud/self-service-analytics-on-bigquery-live-in-30-min-stopwatch-in-hand-b106fc223652?source=rss-614f607ae6d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b106fc223652</guid>
            <category><![CDATA[dataprep]]></category>
            <category><![CDATA[google-data-studio]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Bertrand Cariou]]></dc:creator>
            <pubDate>Mon, 02 Sep 2019 10:35:35 GMT</pubDate>
            <atom:updated>2019-09-02T10:40:56.232Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Or how to leverage BigQuery, Cloud DataPrep, and Data Studio to turn your raw data into a beautiful report without the help of anyone but you. And all in less than 30 minutes thanks to Google Cloud Platform smart analytics suite.</em></p><p>But, before you start your timer, make sure you are familiar with the basic concepts of self-service analytics on Google Cloud Platform. This blog <a href="https://medium.com/@bcariou/understanding-self-service-analytics-on-bigquery-with-cloud-dataprep-7f1ed729a42a">“Getting Started with Self-Service Analytics on BigQuery with Cloud Dataprep”</a> will give you this knowledge.</p><p>Then, you need to have at your disposal a GCP project. We will refer to “my-self-service-analytics” project in this blog. Please follow <a href="https://cloud.google.com/bigquery/docs/quickstarts/quickstart-web-ui">these instructions</a> to create your GCP project if you do not have one yet.</p><p>For your first self-service analytics project, your objective is to create a Data Studio dashboard reporting on your eCommerce website performance and the best web channels driving revenue. It will tell you where to invest more and which products need more focus.</p><p>You will have to clean and combine multiple data sources within Cloud Dataprep, output it to BigQuery to make it consumable within Google Data Studio and create your dashboard.</p><p>3…2…1 you can start your timer now. 30 min stopwatch in hand!</p><h3>Setting up BigQuery, Cloud Dataprep, and Google Data Studio</h3><h3>BigQuery Setup</h3><p>You have to create a dataset, the top-level containers to organize your tables and views within BigQuery.</p><ul><li>Go to the <a href="https://console.cloud.google.com/bigquery">BigQuery console</a>. If asked, accept the terms and conditions to activate the service. It can take a few minutes.</li><li>Select your GCP project “my-self-service-analytics” on the left side.</li><li>Select “CREATE DATASET and name it “MyAnalytics”</li></ul><h3>Cloud Dataprep Setup</h3><ul><li>Go to <a href="https://clouddataprep.com/">Cloud Dataprep</a>. Accept the terms and conditions to activate the service. It can take a few minutes.</li><li>Create a “My Analytics” Flow</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*X0iDNbdRCQTXbf6O" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1gly1V2wMdxppM3g" /></figure><ul><li>Add the data to your flow by importing the files <a href="https://drive.google.com/open?id=1P3Rb5_QLDPUXWByhxua_LWBPSS9NPNmf">Sales_Data.csv and Ad_Clicks.csv</a> which contain respectively the e-Commerce transactions and the clickstream ads log file capturing website traffic information. Select “Add Datasets”, then “Import Datasets”, “Chose a file”, and “Import &amp; Add to Flow”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*PBkHYyeRfiOAgwC7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5fvtRRoKnngpcG5H" /></figure><h3>Google Data Studio Setup</h3><ul><li>Go to <a href="https://datastudio.google.com/">Data Studio</a>. Accept the terms and conditions.</li></ul><p>You’re now ready for preparing the data and designing your reports. Keep the 3 tabs open, you will reuse them.</p><h3>Prepare the Data</h3><p>We want to understand which are the domains driving the most traffic to the eCommerce website and which ones are converting best. To achieve this, we need to combine the Ad_Clicks and Sales_Data together. However, both data sets can’t be combined as is and are missing some metrics. We need to prepare the data so it can be combined and enriched for reporting. Here are the data manipulations we need to go through:</p><p>In the Ad_Clicks dataset, we will need to understand which product is advertised, so we will need to extract the product_id from the URL as such:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*Zv5F1-OpAdT5HcyDlDWvzw.png" /></figure><p>In the Sales_Data we will need to calculate the profit by product to identify the most performing web channels.</p><p>Then we can combine the two datasets together and load it into BigQuery to further report on it.</p><h3>Steps to prepare the data</h3><ul><li>Go to <a href="https://clouddataprep.com/">Cloud Dataprep</a></li><li>Select “My Analytics” flow. It should look like this.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dvTn_W38nosdnvMf" /></figure><ul><li>Add and edit a new recipe to Ad_click.csv</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/0*7i0LZMH-xIGnPM1B" /></figure><ul><li>In the grid interface, expand the <strong>column URL</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*2TaAu7BWPfKxwdl4" /></figure><ul><li>highlight the value next to pid=</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/0*o10FFlQOy0kjwQBK" /></figure><ul><li>Select the first suggestion to extract the product id</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*_jgawg5hvQp3S0cY" /></figure><ul><li>Rename the newly created column into product_id by clicking on the column header and selecting the “rename”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/0*mji_oj8uDWnVzAVP" /></figure><p>You have now extracted the product_id which is necessary to combine the Ad_Clicks and Sales_Data datasets together.</p><ul><li>Switch back to the flow view by clicking “MY ANALYTICS &gt;”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/0*2_BKS6TvlbqOGYaa" /></figure><ul><li>Add and edit a new recipe for Sales_Data.csv</li><li>In the grid interface when the dataset has opened, on the far right select Recipe</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/0*N6kOjkx5s46Ini04" /></figure><ul><li>Add a “new step” and type “New Formula”, then enter (or copy/past) “NUMFORMAT(totalSale — totalCost, ‘###.##’)” in the Formula field and “profit” in the New column name</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*hdPJS62AXc56ntX6" /></figure><p>You have now created a new metric representing your profit for each transaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8sT5psum6fQhcjSg" /></figure><ul><li>To combine (join) Sales_data with Ad_clicks, use the join tool and select Ad_Clicks dataset to join with, and accept</li><li>Edit/Add the join conditions to reflect these characteristics</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/0*1FouLgO5peTwg5Pu" /></figure><ul><li>then select Next and check these output columns:</li><li>ordermonth</li><li>orderyear</li><li>productkey</li><li>customerstate</li><li>customercountry</li><li>totalSale</li><li>profit</li><li>domain</li><li>device_type</li><li>Then Review and Save to Recipe.</li></ul><p>You have now created a combined dataset you can report on with DataStudio, but before this step we need to load the dataset result into BigQuery.</p><ul><li>Run the job to publish into BigQuery.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/0*OtBhnGgnACQdg2ER" /></figure><ul><li>In the Run Job on Dataflow screen, edit the publishing action to replace the default file publishing to a BigQuery table. <br>Select BigQuery, MyAnalytics dataset, and select Create a New Table</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8tVFCfrF7oPQr0oX" /></figure><ul><li>Select Append to this table every run and Update</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*apomMgjxsN13Gcsf" /></figure><ul><li>And select Run Job</li><li>The data is now processed and the result will be loaded in a BigQuery table</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OMpVb1s0OrQTixi9" /></figure><h3>Visualize the Data</h3><p>Now that the data has been prepared in Cloud Dataprep and loaded into a BigQuery table, you are ready to create a report with Data Studio on top of it.</p><ul><li>Go to <a href="https://datastudio.google.com/">Google Data Studio</a></li><li>Select DATA SOURCES and select</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/0*NpK8Jq0x_QkzCoHd" /></figure><ul><li>to add a new source</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*7PS8O3UGdx6TSq69" /></figure><ul><li>Select BigQuery and authorize Data Studio for BigQuery if requested</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/0*h4CYagvlpIDCuyh_" /></figure><ul><li>Navigate BigQuery datasets to connect to Sales_Data; you might need to authorize Bigquery access</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*k-gNzQZls7r4hI0U" /></figure><ul><li>Change customerstate to a Geo.Region type and customercountry to a Geo.country type. It will permit using specific geographical chart to represent the data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cNP9IVkXfI-KgXnO" /></figure><ul><li>Change orderyear to a Year (YYYY) format. That will be the default format displayed in your charts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oFTRycEzTH2ynlOR" /></figure><ul><li>Select Create a Report when done</li><li>Add a new Geo Chart in the report</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VFqICXN2GXeXwIHA" /></figure><ul><li>Drag and drop the customerstate field to replace the customer country dimension section.<br>Drag and drop the totalSales field to replace Record Count in the metric section<br>It should look like this. This is a nice representation to quickly know in which states your products sell well.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ypNl9WHM6yH3cUZp" /></figure><ul><li>Add a bar chart to the report</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MIdFR5y_l2on2kxv" /></figure><ul><li>Drag and drop the domain field to replace the productKey dimension section<br>Drag and drop the profit field to the metric section and remove Record Count<br>It should look like this. This addition graph gives you an idea of the web sites that are diving the most profit (when you remove all the costs associated to advertise your products).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oVorPfOA-gn8oP7I" /></figure><ul><li>You can now switch to the view mode to visualize and interact with the report.</li></ul><p>Stop your timer! How much did you do? You can post in the comment to let me know ;-)</p><h3>Conclusion</h3><p>While this is a rudimentary example, you have now achieved the foundational steps to establish a scalable self-service analytics solution leveraging:</p><ul><li>BigQuery for storing and retrieving the data</li><li>Cloud Dataprep to clean, combine, and create metrics</li><li>Google Data Studio to visually report on top of your data.</li></ul><p>By exploring each GCP service deeper and iterating on these principles, you will be able to solve any requirements for your analytics in a self-service manner. Maybe in more than 30 minutes though, but you will have the total control of the end to end process, which will empower you with the agility you need for faster, more accurate and personalized data insight to drive your business.</p><p><em>Originally published at </em><a href="http://www.trifacta.com/blogs"><em>www.trifacta.com/blogs</em></a><em> on July 25, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b106fc223652" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/self-service-analytics-on-bigquery-live-in-30-min-stopwatch-in-hand-b106fc223652">Self-Service Analytics on BigQuery live in 30 min, stopwatch in hand!</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Self-Service Analytics on BigQuery with Cloud Dataprep]]></title>
            <link>https://medium.com/google-cloud/understanding-self-service-analytics-on-bigquery-with-cloud-dataprep-7f1ed729a42a?source=rss-614f607ae6d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f1ed729a42a</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[google-data-studio]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-preparation]]></category>
            <dc:creator><![CDATA[Bertrand Cariou]]></dc:creator>
            <pubDate>Mon, 02 Sep 2019 10:30:03 GMT</pubDate>
            <atom:updated>2019-09-03T23:03:54.153Z</atom:updated>
            <content:encoded><![CDATA[<p>This article outlines the basic concepts I learned while exploring Google Cloud Platform to set up an analytic framework for my own reporting needs. I wanted to have the ability to build quickly ad-hoc and recurring reports on large data sets to get the necessary insight to inform some business decisions (and share it with the team) or feed data for some marketing campaigns I was running.</p><p>I am pretty happy with the result and want to share here a summary of what I learned so others can get to similar results faster. Hope it can be useful to you too.</p><h3><strong>What is self-service analytics?</strong></h3><p>For me, the objective of self-service analytics is to empower data-driven professionals like me to create their own end-to-end solution for their analytics, respond to their decision-making requirements, and research for insight. Self-service means that there are no necessary dependencies on others to get the job done and that the technology in use is simple yet powerful and scalable enough to be easily adopted by anyone to deliver results quickly.</p><h3>What are the ingredients for self-service analytics on the Google Cloud Platform (GCP)?</h3><p>To establish a self-service analytics practice, you need to rely on three fundamental constituents: data storage, data preparation, and data visualization.</p><p><strong>Data Storage.</strong> You need a space that your data can live and grow within. This data storage is the foundation to construct your analytics on top of. Within the GCP, the easiest way to store and retrieve your data for analytics at scale is BigQuery.</p><p><strong>Data Preparation.</strong> From the data storage layer, you can prepare your data for reporting and dashboarding. You’ll be using Cloud Dataprep to clean, standardize, combine, and create various calculations and metrics stored in BigQuery so that you can design your visualization layer on top of it. Data</p><p><strong>Visualization.</strong> The visualization layer shows your data in the form of various reports and dashboards containing graphs and tables to visually represent data for your insight and to help your data-driven decision process. You’ll be using Data Studio consuming BigQuery data to create your reports and dashboards. With these fundamental elements in place, you’re ready to implement a scalable and comprehensive self-service analytics solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*ZpxZ1pDL9oQQI1Eagw2Kew.png" /></figure><h3>Understand GCP self-service analytics concepts</h3><p>Here are the basic and minimum concepts for an end-to-end self-service analytics solution built on the GCP. These concepts are the foundation you need to understand and iterate on for your self-service analytics solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yWfRQ-Ht2JngMZVA1zLO0w.jpeg" /><figcaption>GCP Concepts for self-service analytics</figcaption></figure><p>Here is a visual representation of these concepts and their relations to create a self-service analytics solution within the GCP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PpKZEKHPdB6GqF6HXBz1hQ.png" /><figcaption>flow between concepts</figcaption></figure><h3>Moving from theory to practice</h3><p>Still a bit too theoretical? No problem! You can experience it for yourself following <a href="https://medium.com/@bcariou/self-service-analytics-on-bigquery-live-in-30-min-stopwatch-in-hand-b106fc223652">this step-by-step guide to build your first end-to-end analytics solution</a>. You will establish a scalable self-service analytics solution that leverages BigQuery to store and retrieve data; Cloud Dataprep to clean, combine, and create metrics; and Google Data Studio to visually report on top of your data. By exploring each GCP service deeper and iterating on these principles, you will be able to solve any requirements for your analytics in a self-service manner.</p><p><em>Originally published on </em><a href="https://www.trifacta.com/resource-library/getting-started-with-self-service-analytics-on-bigquery-with-cloud-dataprep/"><em>www.trifacta.com</em></a><em> on July 24, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f1ed729a42a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/understanding-self-service-analytics-on-bigquery-with-cloud-dataprep-7f1ed729a42a">Understanding Self-Service Analytics on BigQuery with Cloud Dataprep</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>