<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Soda on Medium]]></title>
        <description><![CDATA[Stories by Soda on Medium]]></description>
        <link>https://medium.com/@sodadata?source=rss-2e82189e60b8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*D8CjoXbuxZbKqLbCy6tf8w.png</url>
            <title>Stories by Soda on Medium</title>
            <link>https://medium.com/@sodadata?source=rss-2e82189e60b8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:05:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sodadata/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unlocking Data Quality on Google Cloud Platform: Integrating Soda for Enhanced Data Integrity]]></title>
            <link>https://sodadata.medium.com/unlocking-data-quality-on-google-cloud-platform-integrating-soda-for-enhanced-data-integrity-6d2e51ec4602?source=rss-2e82189e60b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d2e51ec4602</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[data-monitoring]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-integrity]]></category>
            <dc:creator><![CDATA[Soda]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 22:12:57 GMT</pubDate>
            <atom:updated>2024-06-27T22:12:57.583Z</atom:updated>
            <content:encoded><![CDATA[<h4>By Yang Li, Staff Cloud Solutions Architect at Google Cloud and Janet Revell, Technical Writer at Soda</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4HjML-07zJaS9rXaM3JHlw.jpeg" /><figcaption>Photo credit: Janet Revell</figcaption></figure><p>If you and your team know that now is the time to tackle data quality in your GCP data sources, you can set up and connect to Soda to finally get that project rolling.</p><p><a href="https://www.soda.io/">Soda</a> is a tool that enables Data Engineers, Data Scientists, and Data Analysts to test data for quality where and when they need to.</p><ul><li>Is your data fresh?</li><li>Is it complete or missing values?</li><li>Are there unexpected duplicate values?</li><li>Did something go wrong during transformation?</li><li>Are all the data values valid?</li><li>Are anomalous values disrupting downstream reports?</li></ul><p>These are the questions that Soda answers.</p><p>Soda works by taking the data quality checks that you prepare and using them to run a scan of datasets in a data source. A scan is a command which instructs Soda to execute data quality checks on your data source to find invalid, missing, or unexpected data. When data quality checks fail, they surface bad-quality data and present check results that help you investigate and address quality issues.</p><p>Working together, a Soda Agent, Soda Cloud, and <a href="https://go.soda.io/sodacl">Soda Checks Language (SodaCL)</a> empower you and your colleagues to collaborate on data quality testing.</p><p>The Soda Agent is a tool that you deploy as a helm chart in a Kubernetes cluster which empowers Soda Cloud users to securely access data sources to scan for data quality. This setup enables Soda Cloud users to securely connect to data sources, such as BigQuery, from within the Soda Cloud web application. Any user in your Soda Cloud account can then write their own no-code checks to check for data quality in the new data source.</p><p>This blog outlines the steps and procedures to sign up for Soda, deploy an agent, and connect to your data source so you and your team can begin monitoring for data quality. For reference, what you’ll set up will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SYwd-_s9YdC5EU50SvCqlA.png" /></figure><h4>Step 1: Create a Google Kubernetes Engine (GKE) cluster</h4><p>Though you can absolutely use an existing GKE cluster into which to deploy a Soda Agent, be sure to use a Standard Regional GKE cluster, as opposed to Autopilot. The agent will work with an Autopilot cluster, but the scan performance is slower because the profile provisions new nodes for each scan.</p><p>In your Cloud Console, select an existing project, or create a new one. Then, go to Navigation menu &gt; Kubernetes Engine and Enable the project’s API.</p><p>Create a new cluster in your Cloud Console, or via the command line using the following command, replacing the value of the region to match your own.</p><pre>gcloud container clusters create-auto soda-agent-gke \<br>--region us-west1</pre><p>Use the following command to add configuration to connect the new cluster to your local kubectl configuration.</p><pre>gcloud container clusters get-credentials soda-agent-gke</pre><p>Output :</p><pre>Fetching cluster endpoint and auth data. <br>kubeconfig entry generated for soda-agent-gke.</pre><p>Create a new namespace in your cluster, use the following command.</p><pre>kubectl create ns soda-agent</pre><p>Change the context to associate the current namespace to the one you just created.</p><pre>kubectl config set-context - current - namespace=soda-agent</pre><p>Run the following command to verify that the cluster kubectl recognizes soda-agent as the current namespace.</p><pre>kubectl config get-contexts</pre><p>Output:</p><pre>CURRENT   NAME   CLUSTER  AUTHINFO  NAMESPACE<br>* gke_soda-agent-gke_us-west1… gke_soda-agent-gke_us-west1… <br>gke_soda-agent-gke_us-west1*** soda-agent</pre><h4>Step 2: Create a Soda Cloud account</h4><p>The Soda Agent communicates with your Soda Cloud account using API public and private keys. Soda Cloud is available for free for a 45-day trial period.</p><p>To validate your account license or free trial, a Soda Agent must communicate with a Soda Cloud account via API keys. You create a set of API keys in your Soda Cloud account, then use them to configure the connection to Soda Library or a Soda Agent.</p><p>If you have not already done so, create a Soda Cloud account at <a href="https://cloud.soda.io/signup">cloud.soda.io</a>. If you already have a Soda account, log in.</p><p>In your Soda Cloud account, navigate to your avatar &gt; Data Sources, then navigate to the Agents tab. Click New Soda Agent.</p><p>The dialog box that appears offers abridged instructions to set up a new Soda Agent from the command-line. For now, copy and paste the values for both the API Key ID and API Key Secret to a temporary, secure place in your local environment. You will need these values when you deploy the agent in your Kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*sfoGhbMtM93naU9PhqeVAQ.png" /></figure><h4>Step 3: Deploy a Soda Agent in the GKE cluster</h4><p>It’s a good idea to prepare a values YAML file as a way of deploying the Soda Agent helm chart on a cluster while keeping sensitive values secure.</p><p>Using a text or code editor, create a new YAML file called values.yml. In that file, copy+paste the content below, replacing the following values:</p><ul><li>id and secret with the values you copy+pasted from the Deploy a Soda Agent dialog box in your Soda Cloud account. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets.</li><li>name with a custom name for your agent, if you wish.</li><li>endpoint according to your local region: <a href="https://cloud.us.soda.io/">https://cloud.us.soda.io</a> for the United States, or <a href="https://cloud.soda.io/">https://cloud.soda.io</a> for all else.</li><li>(optional) the format for log output: rawfor plain text, or json for JSON format.</li><li>(optional) the level of log information you wish to see when deploying the agent: ERROR WARN INFO DEBUG TRACE</li><li>(optional) key-value pairs for the access credentials to the Google Cloud Platform data source you want to scan for data quality.</li></ul><pre>soda:<br>  apikey:<br>    id: &quot;***&quot;<br>    secret: &quot;***&quot;<br>  agent:<br>    name: &quot;myuniqueagent&quot;<br>    logformat: &quot;raw&quot;<br>    loglevel: &quot;ERROR&quot;<br>  cloud:<br>    endpoint: &quot;https://cloud.soda.io&quot;<br>  env:<br>    BIGQUERY_USER: &quot;sodalibrary&quot;<br>    BIGQUERY_PASS: &quot;sodalibrary&quot;</pre><p>Save the file. Then, in the same directory in which the values.yml file exists, use the following command to install the Soda Agent helm chart.</p><pre>helm install soda-agent soda-agent/soda-agent \ --values values.yml \<br>--namespace soda-agent</pre><p>(Optional) Validate the Soda Agent deployment by running the following command:</p><pre>kubectl describe pods</pre><p>In your Soda Cloud account, navigate to your avatar &gt; Agents. Refresh the page to verify that you see the agent you just created in the list of Agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5uwoNx5Ydgm2_u97" /></figure><p>Be aware that this may take several minutes to appear in your list of Soda Agents. Use the describe podscommand in step four to check the status of the deployment. When Status: Running, then you can refresh and see the agent in Soda Cloud.</p><p>Output:</p><pre><br>Name:             soda-agent-orchestrator-66-snip<br>Namespace:      soda-agent<br>Priority:         0<br>Service Account:  soda-agent<br>Node:             &lt;none&gt;<br>Labels:           agent.soda.io/component=orchestrator<br>                  agent.soda.io/service=queue<br>                  app.kubernetes.io/instance=soda-agent<br>                  app.kubernetes.io/name=soda-agent<br>                  pod-template-hash=669snip<br>Annotations:    seccomp.security.alpha.kubernetes.io/pod: runtime/default<br>Status:           Running<br>...</pre><h4>Step 4: Add a BigQuery data source</h4><p>After you have a Soda Agent deployed and running in your GKE cluster, you use the Soda Cloud user interface to connect to your data source.</p><p>In your Soda Cloud account, navigate to your avatar &gt; Data Sources. Click New Data Source, then follow the guided steps to create a new data source.</p><p>In the editing panel in step 2, provide the connection configurations Soda Cloud needs to be able to access your data source. Connection configurations are data source-specific and include values for things such as a database’s host and access credentials. An example of a <a href="https://docs.soda.io/soda/connect-bigquery.html">BigQuery connection configuration</a> with a Service Account Key authentication method follows.</p><pre>data_source my_datasource_name:<br>  type: bigquery<br>  account_info_json: &#39;{<br>      &quot;type&quot;: &quot;service_account&quot;,<br>      &quot;project_id&quot;: &quot;gold-platform-67883&quot;,<br>      &quot;private_key_id&quot;: &quot;d0121d000000870xxx&quot;,<br>      &quot;private_key&quot;: &quot;-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n&quot;,<br>      &quot;client_email&quot;: &quot;abc333@project.iam.gserviceaccount.com&quot;,<br>      &quot;client_id&quot;: &quot;XXXXXXXXX.apps.googleusercontent.com&quot;,<br>      &quot;auth_uri&quot;: &quot;https://accounts.google.com/o/oauth2/auth&quot;,<br>      &quot;token_uri&quot;: &quot;https://accounts.google.com/o/oauth2/token&quot;,<br>      &quot;auth_provider_x509_cert_url&quot;: &quot;https://www.googleapis.com/oauth2/v1/certs&quot;,<br>      &quot;client_x509_cert_url&quot;: &quot;https://www.googleapis.com/robot/v1/metadata/x509/...&quot;<br>    }&#39;<br>  auth_scopes:<br>  - https://www.googleapis.com/auth/bigquery<br>  - https://www.googleapis.com/auth/cloud-platform<br>  - https://www.googleapis.com/auth/drive<br>  project_id: &quot;platinum-platform-67883&quot;<br>  dataset: sodacore</pre><p>During its initial scan of your datasource, Soda Cloud discovers all the datasets the data source contains. It captures basic information about each dataset, including dataset names and the columns each contains. In the editing panel, specify the datasets that Soda Cloud must include or exclude from this basic discovery activity.</p><p>To gather more detailed profile information about datasets in your data source, you can configure Soda Cloud to profile the columns in datasets. Column profile information includes details such as the calculated mean value of data in a column, the maximum and minimum values in a column, and the number of rows with missing data. In the editing panel, provide details that Soda Cloud uses to determine which datasets to include or exclude when it profiles the columns in a dataset.</p><p>When Soda Cloud automatically discovers the datasets in a data source, it prepares automated monitoring checks for each dataset. These checks detect anomalies and monitor schema evolution, corresponding to the SodaCL anomaly detection and schema checks, respectively. In the editing panel, specify the datasets that Soda Cloud must include or exclude when preparing automated monitoring checks.</p><p>In the final step, assign an Owner for the data source. By default, your own Soda Cloud user account is assigned.</p><h4>Let the checks begin!</h4><p>With Soda deployed and connected to your data source, you’re all set to begin writing checks for data quality.</p><pre>checks for retail_products:<br>  - freshness(updated_on) &lt; 2d:<br>    name: Data is up to date<br>  - missing_percent(stock_count) &lt; 5%:<br>    name: Stock count acceptable<br>  - duplicate_count(product_id) = 0:<br>    name: No duplicate product IDs<br>  - invalid_percent(product_id) &lt;= 2%:<br>    valid max length: 10<br>    name: Product IDs are valid<br>  - anomaly detection for row_count:<br>    name: Expected volume of added rows </pre><p><a href="https://go.soda.io/checks">Learn more</a> about how to get started with SodaCL and how to set up scheduled data quality scans that automatically run as often as you need and notify you when there are issues.</p><p>Need help? Join the <a href="https://community.soda.io/slack">Soda community on Slack</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d2e51ec4602" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Incident Management for dbt, Powered by Soda]]></title>
            <link>https://sodadata.medium.com/introducing-incident-management-for-dbt-powered-by-soda-8abee4251ff5?source=rss-2e82189e60b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8abee4251ff5</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Soda]]></dc:creator>
            <pubDate>Wed, 09 Mar 2022 15:43:04 GMT</pubDate>
            <atom:updated>2022-03-09T15:49:51.390Z</atom:updated>
            <content:encoded><![CDATA[<h4>By Bastien Boutonnet, Insights and Automation</h4><p><em>Original </em><a href="https://www.soda.io/resources/incident-management-for-dbt"><em>article</em></a><em> published January 18, 2022</em></p><p><strong><em>In this article:</em></strong></p><ul><li><strong><em>Powering the Modern Data Stack</em></strong></li><li><strong><em>Build or Buy</em></strong></li><li><strong><em>Free the Analytics Engineer</em></strong></li><li><strong><em>First Things First</em></strong></li><li><strong><em>dbt + Soda</em></strong></li><li><strong><em>Getting Started with 24/7 Data Availability</em></strong></li></ul><p>Let’s begin on a bold note: <strong>dbt</strong> is the de facto tool for analytical data transformations and we — like hundreds of thousands of others — love it.</p><p>As a data scientist, avid user, fan, and contributor (both <a href="https://github.com/dbt-labs/dbt-core"><strong>dbt-core</strong></a> but also <a href="https://github.com/bitpicky/dbt-sugar"><strong>dbt-sugar</strong></a>), I can confidently testify to our bold statement. I’ve used dbt in all of my past data science roles, and in my last role at TripActions, I was involved in making the data analysts and data scientists super efficient using dbt, supercharging the data team, and getting everyone to work like a data engineer (without them realizing it, of course!). I’ve also got two <a href="https://coalesce.getdbt.com/"><strong>Coalesce</strong></a> speaker badges, a keyring, a hat, and a supersoft sweater.</p><h3><strong>Powering the Modern Data Stack</strong></h3><p>When we spoke to the Soda community about which tools they use the most to get insights from raw data, dbt was certainly at the top of the list. Its transformation workflow helps build robust data pipelines and enforces the execution of data validation tests at the point the transformations are coded. dbt gets the data flowing.</p><h3><strong>Build or Buy</strong></h3><p>I’ve said it twice, and I’ll say it again, dbt is the best tool to write transformations. Because the data doesn’t stop flowing and organizations need end-to-end data observability, there’s a demand for additional tools that work across the entire data product lifecycle, to help teams better manage their data products.</p><p>It’s common for organizations to build a solution that parses failures at the end of a test run and creates an alert in Slack to trigger incident resolution. I know, because that’s exactly what my team and I did at TripActions — we built a self-serve, low-code workflow so that anyone in the organization could access all the insights they need to make a decision. Our goal was to enable any team member to do more with data.</p><p>I learned though, that if an existing solution can free up your time and allow you to focus on what you enjoy doing, then you should buy the tool that does most of the work.</p><p>And that’s why, at Soda, we’re integrating the transformational power of dbt with the observability and incident management of Soda.</p><h3>Free the Analytics Engineer</h3><p>At Soda, we’re building data reliability tools and an observability platform to help data teams discover, prioritize, and resolve data quality issues. We&#39;ve simplified a cumbersome process with a comprehensive, end-to-end workflow to detect and resolve issues, and automatically alert the right people at the right time.</p><p>We want to set the Analytics Engineers free! To this end, I’m delighted to announce our integration with dbt.</p><h3><strong>First Things First</strong></h3><p>Yes, Soda has out-of-the-box rules-based testing, anomaly detection, and distribution checks. Our ethos with respect to testing is the same as dbt — our tools are Open Source and use YAML, too — but we would be insane to encourage you to abandon the hundreds of tests that run in dbt and expect that you rewrite them in Soda. We strongly believe that you should leverage the tools that you know and love, which is why we prioritized this integration.</p><p>We know that there’s massive benefit in ingesting your dbt test results into Soda Cloud. That’s the beauty of the modern data stack: it’s extensible and flexible enough that you can use the tools that are best-suited for the job at hand. Data observability is a key part of the modern data stack and, at Soda, we believe that we’re building the most robust platform for data teams.</p><h3>dbt + Soda</h3><p>With dbt + Soda Cloud, data teams can augment the tests that they run in dbt. Watch the video to see dbt + Soda in action, ingesting dbt test results into Soda and using Soda Incidents.</p><p>We love it so much, we made two videos! Choose your flavor based on what you use: dbt core, or dbt Cloud.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FV4cGCniBpMg%3Flist%3DPLFhNacHzuYSIiYbmt_X-9wwMRwEqYWg4J%26start%3D1&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DV4cGCniBpMg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FV4cGCniBpMg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/922d21b5ea53e47e1fbbcd9e1ef35814/href">https://medium.com/media/922d21b5ea53e47e1fbbcd9e1ef35814/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FlcGHJxVLOLI%3Flist%3DPLFhNacHzuYSIiYbmt_X-9wwMRwEqYWg4J%26start%3D10&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DlcGHJxVLOLI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FlcGHJxVLOLI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bbe0d3e8f83b771f788327a62786518a/href">https://medium.com/media/bbe0d3e8f83b771f788327a62786518a/href</a></iframe><p><strong>Here are the highlights of each video:</strong></p><ul><li>Add more testing capabilities to your dbt test results, including automated anomaly detection and schema evolution monitoring.</li><li>Alert necessary stakeholders of test failures. Set up an integration between Soda Cloud and Slack so that your data teams are the first to know when data quality issues arise, and can fix the problem before there is downstream damage.</li><li>Manage data reliability and quality incidents, whether at the dataset or record level. Soda Incidents simplifies the process to detect, triage, diagnose, and resolve data issues across the entire data product lifecycle.</li></ul><h3>Piecing It All Together</h3><p>Data management in the modern data architecture is computationally tied into every step of the data flow and product lifecycle. Most data teams today are organized by domain and are composed of people in different roles such as an analytics engineer, an analyst, and a data product manager. Given the cross-functional nature of data, teams are dependent on each other to provide reliable high-quality data, every day.</p><p>We’re so proud to be able to bring all of the goodness of dbt to Soda Cloud, making it possible to support the entire workflow, end-to-end, for data management, data reliability, and data quality. Our integration helps dbt users connect to their data sources, conduct root cause analysis, and manage bad data quickly before it has a downstream impact.</p><h3>Get Started with 24/7 Data Availability</h3><p>We’re fortunate to count Disney, HelloFresh, Servier, and Udemy among those who have deployed Soda’s data reliability tools in production. And we love their contributions! Now it’s time for you to put Soda to the (dbt) test.</p><p>You can <a href="https://docs.soda.io/soda/integrate-dbt.html">read our comprehensive docs</a> on integrating dbt with Soda, or quickly get started with the following three steps.</p><p>1. Run some tests using dbt’s `<a href="https://docs.getdbt.com/docs/building-a-dbt-project/tests">test</a>` or `<a href="https://docs.getdbt.com/reference/commands/build">build</a>` commands, to capture test results. If you’re using dbt Cloud to schedule your jobs, make sure you run the `soda ingest` command somewhere near the end of that job. (Currently dbt Cloud cannot trigger actions in other tools, but as soon as a more integrated option appears on the scene, we’ll be on it!)</p><p>2. Call `soda ingest`. First, make sure you have configure <a href="https://docs.soda.io/soda-cloud/connect_to_cloud.html">Soda SQL to connect to Soda Cloud</a>. Then, `pip install soda-sql-dbt` and ingest dbt test results using the following command:</p><p>`soda ingest dbt — warehouse-yml-file &lt;path to warehouse.yml&gt; — dbt-artifacts &lt;path to dbt artifact jsons&gt;`</p><p>If you use dbt Cloud, we’ve got you covered! Check out <a href="https://docs.soda.io/soda/integrate-dbt.html">our docs on how to set it up</a>.</p><p>3. View, collaborate, and manage data quality incidents in Soda Cloud. Once ingested, Soda Cloud displays your dbt test results. You can immediately start <a href="https://docs.soda.io/soda-cloud/monitors.html">setting up alerts</a> or <a href="https://docs.soda.io/soda-cloud/incidents.html">creating incidents</a>.</p><p>If you’re new to Soda, you can quickly (and easily!) get started for free. You need to <a href="https://docs.soda.io/soda-sql/installation.html">install Soda Tools</a>, available as Open Source, and connect it to Soda Cloud, a free account available as a trial version. If you need help, contact the Soda Team in our <a href="https://community.soda.io/slack">Soda Community on Slack</a>.</p><h3>What’s Next?</h3><p>The entire team and I are excited by the possibilities that dbt + Soda offers dbt users, establishing a true end-to-end data quality workflow, from detection right through to resolution.</p><p>Give it a try and let us know what you would like to see as a deeper integration with dbt. Here’s what we are working on:</p><ul><li>Closer integration, where a run of dbt test may be able to trigger a run of `soda ingest`.</li><li>dbt allows users to `<a href="https://docs.getdbt.com/reference/resource-configs/store_failures">store failures</a>`. Very soon, `soda ingest` will be able to also get that information from dbt and show it to you. If you do not use `store_failures`, then we’ll show you the compiled SQL so that you can get to the failing records in a jiffy!</li><li>Ingest table-level lineage to help you see how datasets that are modeled using dbt connect together to help you with root cause analysis.</li><li>And just in case you use Great Expectations for testing, we’re looking at providing a similar ingestion approach.</li></ul><p>We’re also working on <a href="https://www.soda.io/resources/introducing-a-new-domain-specific-language-for-data-reliability">SodaCL</a>, a new data reliability language, built to be included in a data pipeline. It will be freely available as Open Source and promises to change how we test and validate data, as-code.</p><h4>Recommended for you:</h4><ul><li><a href="https://www.soda.io/resources/data-availability-incident-management">Blog: 24/7 Data Availability with Incident Management for Data Teams</a></li><li><a href="https://docs.soda.io/soda-sql/interactive-demo.html">Soda OSS in Action: Interactive Demo</a></li><li><a href="https://docs.soda.io/soda-cloud/quick-start.html">Quick Start for Soda Cloud</a></li><li><a href="https://www.soda.io/resources/cloud-metrics-store">Blog Feature Deep-Dive: Cloud Metrics Store</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8abee4251ff5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Eliminating Data Downtime with Incident Management for Data Teams]]></title>
            <link>https://sodadata.medium.com/eliminating-data-downtime-with-incident-management-for-data-teams-379a1afe4629?source=rss-2e82189e60b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/379a1afe4629</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Soda]]></dc:creator>
            <pubDate>Fri, 21 Jan 2022 21:00:50 GMT</pubDate>
            <atom:updated>2022-01-25T08:18:04.444Z</atom:updated>
            <content:encoded><![CDATA[<h3>Eliminating Data Downtime with Data Incident Management</h3><h4>By Jonas Marien, Cloud Infrastructure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnCCTeZYJWc81Ny6BVhbQA.png" /></figure><p><em>Original </em><a href="https://www.soda.io/resources/data-availability-incident-management"><em>article</em></a><em> published December 22, 2021</em></p><p><strong><em>In this article:</em></strong></p><ul><li><strong><em>From Data Downtime to Data Availability</em></strong></li><li><strong><em>Introducing Soda Incidents</em></strong></li><li><strong><em>The Complete Data Quality Workflow</em></strong></li><li><strong><em>Assign and Conquer</em></strong></li><li><strong><em>Teamwork Makes the Data Work</em></strong></li><li><strong><em>Measuring Time to Respond &amp; Resolve (MTTR)</em></strong></li><li><strong><em>(Almost) Every Data Issue Has A Silver Lining</em></strong></li></ul><p>Data products are transforming the way every business operates. Built using ever-increasing volumes of data from different sources, they are accelerating the shift towards digital automation. Once in production, data products need grooming and maintenance to address inevitable changes to data schemas and structures, broken transformation logic, and concept drift, all of which impact reliability, quality, and ultimately trust in data.</p><p>As they work to maintain the quality of data products, data and analytics engineers often find themselves firefighting data issues when reports or machine learning models break. When an alarm bell rings, it’s a mad scramble to find out what has broken and what has been impacted.</p><p>When “data downtime” strikes, we empathize with data engineering teams fighting the fire and their downstream data consumers who are desperate for resolution. Not only is data downtime frustrating, it’s a waste of everyone’s time and money to constantly cycle through a break-fix loop instead of focusing on work that adds value to the business.‍</p><h3>From Data Downtime to Data Availability‍</h3><p>Have you ever wondered about the full cost and impact of data downtime? When we’ve asked this question of our community, we hear about costs such as disruption to the business, loss of revenue, productivity loss, and increased skepticism in the trust of data. Depending on the type of organization, data downtime can even impact regulatory or compliance mandates, customer retention, and employee satisfaction.</p><p>For every organization that relies on data to drive revenue through confident decision-making, that nurtures happy customers and employees, every minute of “data uptime” — when good-quality data is accurate and available — matters.</p><p>Soda was built to address data availability, to help data teams discover, prioritize, and resolve data quality issues. We’ve been eager to simplify this cumbersome process with a comprehensive end-to-end workflow to detect and resolve issues, and automatically alert the right people at the right time. Thus, we introduce Soda Incidents.‍</p><h3>Welcome, Soda Incidents</h3><p>At <a href="https://www.soda.io/">Soda</a>, we’re really pleased to announce the latest feature of the Soda Cloud platform: Soda Incidents, a workflow to streamline your data operations.‍</p><p>Soda Incidents enables data teams to effectively manage data reliability and quality incidents, whether those are in a dataset or at the record level. It alleviates the fear of not knowing, and the pain of finding out too late that a data quality issue has had a downstream impact.</p><p>Soda Incidents simplifies the process to detect, triage, diagnose, and resolve data issues across the entire data product lifecycle. Our data reliability and observability tools are built by data engineers and product owners, all of whom have first-hand experience with building reliable systems that produce high-quality data. In collaboration with Soda Cloud early adopters and in reviewing industry best practices, our team identified three key pillars of good incident management: defined roles and responsibilities, coordination between team mates; and clear communication. Further, good incident management must be efficient and operate at scale: when a data quality issue occurs, everyone must be able to efficiently respond under pressure. Another key component to the process is to carry out a retrospective or post-mortem to discover what went wrong, understand why it went wrong, and the lessons learned to avoid the incident happening again in the future.</p><h3>‍The Complete Data Quality Workflow‍</h3><p>We built an operations workflow for incident management that incorporates best practices from incident management in the infrastructure and application layer and enables organizations to configure Soda to meet their specific business requirements.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.loom.com%2Fembed%2Fa535d99520fc4524a8594c779f6e45a8&amp;display_name=Loom&amp;url=https%3A%2F%2Fwww.loom.com%2Fshare%2Fa535d99520fc4524a8594c779f6e45a8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=loom" width="1664" height="1248" frameborder="0" scrolling="no"><a href="https://medium.com/media/1172c9fe09c8ec299919895f6fb9facd/href">https://medium.com/media/1172c9fe09c8ec299919895f6fb9facd/href</a></iframe><p>It all begins when you create a monitor in Soda Cloud and add an alert to notify you when a data quality test fails. When your team receives an alert via Slack or email, you can log in to review the failed test and click to open a new Incident. The built-in workflow enables you to assign a Lead Investigator and Status to the Incident to track your team’s progress in investigating, and resolving the data quality issue.</p><h3>‍Assign and Conquer‍</h3><p>Good incident management and resolution depends on everyone understanding their role. By making sure everyone’s role is clear, you can manage who is doing what and in turn, get more done. During the scramble of data downtime, knowing who is responsible for what determines how quickly you can get back to business as usual.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*5OgPz5rUulJ2YolF.png" /></figure><p>Soda creates an Incident and starts a Slack channel that includes the Lead and Reporter.</p><p>For each Incident, a Lead Investigator takes ownership of the Incident and ensures that the team takes action to diagnose, analyze, and resolve the issue. Notably, the Reporter of an Incident is not assigned as the Lead Investigator by default. We did this on purpose to empower any team member to spin up an Incident without having to own the investigation.</p><p>At Soda we’re really good at building tools that help data teams create trusted data products; we are not in the business of building issue trackers. We know that tools like Jira and OpsGenie are widely used and exhaustively address end-to-end issue tracking. It would have been absurd to go to the extreme of building yet another issue tracker so instead, we built Soda Incidents to integrate with existing tools. In the coming months, we’ll be releasing more integrations with popular issue trackers — <a href="https://community.soda.io/slack">let us know</a> if you have a favorite!</p><h3>‍Teamwork Makes the Data Work‍</h3><p>Data quality is a team sport: if everyone monitors data quality, Incidents need to be accessible by everyone in your organization. It needs to be simple for people to get involved and bring everyone closer to data they know they can trust.</p><p>With this in mind, we purposefully selected Slack as the first tool to integrate with Soda Incidents. Soda Cloud already integrates with Slack to send notifications, and we know it is a tool-of-choice for many organizations’ internal communication and collaboration. Our research shows that data teams are already using Slack to work through technical issues so extending Slack to investigate data quality issues felt right. From within an Incident, click to open a new Slack channel so your team can immediately begin collaborating to resolve a specific data quality issue. Working in Slack, it’s easy for team members to ask questions, trigger ideas, and surface intrinsic technical information that only one person knows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/0*rAmT1TeRkne3gdMq.png" /></figure><p>Add more monitors to the Incident | Status update posted to Slack within the ongoing conversation</p><h3>Measuring Time To Respond &amp; Resolve (MTTR)‍</h3><p>As your team works their way through the investigation, you can update the Status of an Incident, change the Lead Investigator, and add or remove monitor results that are related to the Incident. Soda Cloud pushes each change to the Incident’s Slack channel. This is how everyone stays on the same page and stakeholders stay informed on progress.</p><p>And when the fire is out, when the team is able to breathe easy and close the Incident, Soda Cloud automatically archives the channel in Slack. There it stays, awaiting the day when a similar issue arises and a team can access the details that will help them fix an issue even faster. What starts with a data failure at source, ends with a community of specialists, such as analytics engineers, data stewards, and data managers, who work together to keep the data in pristine condition.</p><p>After incorporating Soda Incidents into your team’s data quality practices, consider leveraging <a href="https://www.soda.io/resources/soda-introduces-new-reporting-api">Soda’s Reporting API</a> to build dashboards using Key Performance Indicators that help you understand Soda’s impact on data quality in your organization. Use the API to:</p><ul><li>assess data quality test coverage in your organization</li><li>gauge the overall “Data quality health” of datasets</li><li>find out how often your team uses Soda Cloud to check on data quality‍</li></ul><h3>(Almost) Every Data Issue Has A Silver Lining‍</h3><p>With every investigation and every issue resolved, there are lessons to be learned for next time. When you set the Status of an Incident to Resolved, Soda Cloud prompts you to record Resolution Notes, encouraging you to review your incident data and capture any associated intrinsic knowledge that helped resolve the issue.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.loom.com%2Fembed%2F81277f8723b647868df35e986fb19a30&amp;display_name=Loom&amp;url=https%3A%2F%2Fwww.loom.com%2Fshare%2F81277f8723b647868df35e986fb19a30&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=loom" width="1664" height="1248" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5dbc17b7676b5a44f998d234543623f/href">https://medium.com/media/b5dbc17b7676b5a44f998d234543623f/href</a></iframe><p>This input can help you identify trends and anomalies, or any common causality for your data incidents, leading to insights that enable you to improve your systems and operations even more.</p><h3>‍What’s Next?‍</h3><p>We are excited by the possibilities that Soda Incidents offers our users, helping them manage data quality incidents with a true end-to-end workflow, from detection right through to resolution. We’re working adding the capability to update the Status of an Incident in Slack.</p><p>Soon, we’ll be announcing our newest integration that enables you to leverage all the data tests you created in dbt. Soda Cloud can ingest all those dbt test results and display them alongside the data quality monitors you created in Soda Cloud. Run your dbt pipeline, schedule Soda SQL to run soda ingest, and instantly see all your test results in one place. Bet you can’t wait to see the time-series charts from your dbt test results in Soda Cloud!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=379a1afe4629" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modern Data Testing for PySpark DataFrames]]></title>
            <link>https://sodadata.medium.com/modern-data-testing-for-pyspark-dataframes-207d4da9adbb?source=rss-2e82189e60b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/207d4da9adbb</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[spark-sql]]></category>
            <dc:creator><![CDATA[Soda]]></dc:creator>
            <pubDate>Fri, 21 Jan 2022 20:37:07 GMT</pubDate>
            <atom:updated>2022-01-25T08:16:47.620Z</atom:updated>
            <content:encoded><![CDATA[<h4>By Vijay Kiran, Data Engineering Lead</h4><p><em>Original </em><a href="https://www.soda.io/resources/data-quality-testing-spark-pipelines"><em>article</em></a><em> posted on December 22, 2021</em></p><p><strong><em>In this article:</em></strong></p><ul><li><strong><em>Modern Data Testing for PySpark DataFrames</em></strong></li><li><strong><em>Introducing Soda Spark</em></strong></li><li><strong><em>Tutorial: Data Quality Testing for Spark Pipelines</em></strong></li><li><strong><em>How Soda Adds the Spark to Your Pipelines</em></strong></li></ul><p>As part of the mission to bring everyone closer to their data, Soda has introduced <a href="https://github.com/sodadata/soda-spark">Soda Spark</a>, a modern data testing, monitoring, and reliability tool for engineering teams that use PySpark DataFrames. The latest open source tool in Soda’s Data Reliability Toolset, Soda Spark was built for data and analytics engineers working in data-intensive environments, where reliable, high-quality data is of paramount importance.</p><p><a href="https://github.com/sodadata/soda-spark">Soda Spark</a> alleviates the pain of maintaining a high level of trust in data in the Spark ecosystem, where engineers typically spend a lot of time building frameworks for data quality checks. Data and analytics engineers can control the tests that screen for bad data — or as we like to call them, silent data issues– and the metrics that evaluate the results. By using the Spark Dialect already available in Soda SQL, the Soda Spark library provides an API to extract data metrics and column profiles according to the queries you write in YAML configuration files.</p><p>Data engineers use Soda Spark to write declarative data quality tests to Spark DataFrames and detect problematic data. When you connect Soda Spark to a Soda Cloud account, you can configure alerts that notify your team when your Soda Spark tests fail. Teams get the right information right away to begin triaging, investigating, and quickly resolving issues.</p><h3><strong>Tutorial: Data Quality Testing for Spark Pipelines</strong></h3><p><a href="https://github.com/sodadata">Soda’s Open Source Data Reliability Tools</a> can operate across multiple data workloads, engines, and environments, including Kafka, Spark, AWS S3, Azure Blob Storage, Google Cloud Datastore, Presto, Snowflake, Azure Synapse, Google BigQuery, and Amazon Redshift.</p><p>Using Soda Spark is simple: install it via pip and start using the API directly in your PySpark programs. Here’s a small video on how to install, use, and test your Spark DataFrames using Soda Spark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*Q-4hGkkWTbpFZWzryjwlbg.gif" /></figure><p>Alternatively, if you are using a popular Spark execution environment like Databricks, you can also easily use Soda Spark in your Notebooks, and embed the testing directly into your pipelines or schedule testing.</p><p>Follow this five-step tutorial to see how you can run tests for data quality of Apache Spark DataFrames running on a Databricks cluster. If you do not already have a Databricks cluster up and running, you can create one from the Databricks Community.</p><h4>1. To install Soda Spark in your Databricks Cluster, run the following command directly from your notebook:</h4><p>%pip install soda-spark</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/0*28BNpsp7URYVz-TS.png" /></figure><h4>2. Load the data into a DataFrame, then create a scan definition with tests for the DataFrame.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ntvxo84EME9n7UKU.png" /></figure><h4>3. Run a Soda scan to execute the tests you defined in the scan definition (scan YAML configuration file).</h4><p>You can define tests in one of two places: 1) in your code as a string or 2) in an external YAML file.</p><p>In this tutorial we define tests in an external YAML configuration file and use that file to test a DataFrame’s data quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cQk1WZ0lh9dYLuRs.png" /></figure><p>When the Soda scan is complete, you can review the test results and measurements in the command-line (see image below, top), and the output in the Databricks Notebook (see image below, bottom).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3xIIjXKCor74ZIDI.png" /></figure><h4>4. Check the test results for tests that failed during the scan. In the output example below, no tests failed.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AJ84cQaiCeI-07vQ.png" /></figure><p>This output displays the metrics against which Soda tested your data, the resulting measurements, and each test’s pass or fail status.</p><h4>5. And now, you can:</h4><ul><li><a href="https://docs.soda.io/soda-spark/install-and-use.html#send-scan-results-to-soda-cloud">configure Soda SQL</a> to send the metrics and test results to Soda Cloud.</li><li>join our <a href="https://soda-community.slack.com/">Soda Slack Community</a> to ask questions and make feature requests.</li><li><a href="https://github.com/sodadata/soda-spark/blob/main/CONTRIBUTING.md">become a contributor</a>: we are always looking to improve our library. All contributions are welcome!</li></ul><p>Our biggest thanks to Cor Zuurmond (<a href="https://godatadriven.com/">GoDataDriven</a>), the first contributor who helped us develop Soda Spark, and Abhishek Khare (<a href="https://www.hellofreshgroup.com/en/">HelloFresh</a>) and Anil Kulkarni (<a href="https://shop.lululemon.com/">Lululemon</a>), for their early testing and valuable feedback! Check out the Soda Spark repo <a href="https://github.com/sodadata/soda-spark">here</a>.</p><h3>How Soda Adds the Spark to Your Pipelines</h3><p>Soda is the data reliability company that provides Open Source (OSS) and Software as a Service (SaaS) tools that enable data teams to discover, prioritize, and resolve data issues. Our OSS Data Reliability Tools and SaaS cloud platform bring everyone closer to their data, resulting in data products and analytics that everyone can trust. The Soda global data community already counts Disney, HelloFresh, and Udemy as major contributors to have deployed Soda’s data reliability tools.</p><p>Here are 8 good reasons we think that your team should use Soda to test data quality across your entire data product lifecycle.</p><h3>1. Out-of-the-Box</h3><p>Out of the box, Soda supports multiple data sources, provides anomaly detection, visualizations of historical test results, and automated incident management. It is easy to get started and simple to define your tests and expectations.</p><h3>2. Installation</h3><p>Easily install Soda Spark via pip, then use the command-line to connect to a data source, discover datasets, and run a scan within a few minutes. You can follow documented examples to set up programmatic scans from the command-line and integrate with data pipeline tools. Soda’s documentation is easy-to-follow and highly rated by the community and stargazers. Using SQL as its Domain Specific Language, there is no need for Admin help and no requirement for familiarity with writing high-level programming languages.</p><h3>3. Library of Tests</h3><p>Soda docs provide several examples for using built-in metrics, which are easy to maintain and simplify the complexity of writing data quality tests.</p><h3>4. Sharing and Collaboration</h3><p>Use Soda Cloud to invite team members, send notifications to Slack, share data test results, and create and track data quality Incidents. Team members can review scan results and create new monitors in Soda Cloud. Soda Cloud is more inclusive of a non-coder user base so it makes collaboration easier across an organization.</p><h3>5. Get Started with Good Documentation</h3><p>Soda has invested in writing and publishing comprehensive, concise, up-to-date documentation for both the Open Source Data Reliability Tools and cloud platform. Soda’s documentation enables data engineers to immediately discover and monitor data issues through a series of fully configurable tests.</p><h3>6. Cloud Metrics Store</h3><p>You can define historic metrics so that you can write tests in scan YAML files that test data relative to the historic measurements contained in the Soda <a href="https://www.soda.io/press/soda-releases-data-health-metrics-store">Cloud Metric Store</a>. Essentially, this type of metric allows you to access the historic measurements in the Soda Cloud Metric Store and write tests that use those historic measurements.</p><h3>7. Automated Monitoring</h3><p>Soda Cloud offers time-series anomaly detection and schema evolution monitoring. In fact, when you onboard a dataset, Soda Cloud automatically creates both an anomaly detection monitor for row count to start learning patterns about your data right away, and a schema evolution monitor to start tracking changes to columns in your dataset.</p><h3>8. Visualization of Data Monitoring</h3><p>In Soda Cloud, the full history of metrics and tests can be published and visualized on a Soda Cloud account.</p><p>Ready? Get started with Soda Spark <a href="https://docs.soda.io/soda-spark/install-and-use.html">here</a>.</p><p>You can get started straight away and be testing your dataframes in just a few steps. Get Soda Spark <a href="https://docs.soda.io/soda-spark/install-and-use.html">here</a>. We’d love to hear how Soda Spark works for you. Join our <a href="http://community.soda.io/slack">Soda Community on Slack</a> and let us know what you think.</p><p>Check out Cor and Anil’s Soda Spark write-ups to learn even more:</p><ul><li><a href="https://anilkulkarni.com/2021/12/soda-data-quality/">Soda — Data Quality</a>, Anil Kulkarni</li><li><a href="https://www.demystifyingdata.io/2021/09/27/soda-get-ahead-of-silent-data-issues.html">Soda — get ahead of silent data issues</a>, Cor Zuurmond</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=207d4da9adbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build a Dream Team Around Data]]></title>
            <link>https://sodadata.medium.com/how-to-build-a-dream-team-around-data-41b1fc8921a0?source=rss-2e82189e60b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/41b1fc8921a0</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-team]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Soda]]></dc:creator>
            <pubDate>Fri, 21 Jan 2022 15:53:57 GMT</pubDate>
            <atom:updated>2022-01-25T08:21:01.544Z</atom:updated>
            <content:encoded><![CDATA[<h4>By Jesse Anderson — Big Data Institute</h4><p><em>Original </em><a href="https://www.soda.io/resources/building-the-data-dream-team"><em>article</em></a><em> posted on October 28, 2021</em></p><p><strong><em>In this article:</em></strong></p><ul><li><strong><em>Introducing the Data Dream Team</em></strong></li><li><strong><em>The Analogy: Data + Football</em></strong></li><li><strong><em>The Three Teams for Data</em></strong></li><li><strong><em>The Operations Team</em></strong></li><li><strong><em>The Data Engineering Team</em></strong></li><li><strong><em>The Data Science Team</em></strong></li><li><strong><em>The Manager</em></strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SXU0oHz-Y-pmTc2_8-nUKg.jpeg" /></figure><p>Organizations are making a fundamental shift in how they think about data, as the modern data value chain evolves. And at the same time, they need to make a shift in how they build data teams around data.</p><p>In partnership with <a href="https://www.soda.io/">Soda</a>, a Belgian-born software company who is establishing itself as the data observability platform to test, monitor, and manage data quality, we’ve embarked on a series to explore the ‘<a href="https://dreamteam.soda.io/">Data Dream Team</a>’.</p><p>We think the topic of the data team is really important. Not only because we believe that data is a team sport, but because with the seismic shift to making data front and centre in so many organizations, there’s a new approach needed to align how the organization, the team, the people, are structured and organized around data. New roles, shifted accountability, breaking silos, and forging new channels of collaboration. We’ll be breaking it down, and building it up, to understand what “good” looks like.</p><p>To get us started on the Data Dream Team series, we’ve chosen football as our analogy, for a few good reasons (many of which I am sure you will not guess). The first reason is that Belgium is currently ranked by FIFA (Federation of International Football Associations) as the <a href="https://www.fifa.com/fifa-world-ranking">number one football team in the world</a>. The second reason is that 2021 was the summer of football in Europe, with the UEFA Euro 2020 taking place in June and July, and whilst unfortunately Belgium did not take the championship (congratulations Italy), the importance of football on this continent could not be missed.The third reason was inspiration from Michael Cox’s book Zonal Marketing, in which he explores the making of modern European football defined by team formations, game strategies, and culture.</p><p>Analogy complete and link to Belgium justified.</p><h3><strong>The Three Teams for Data</strong></h3><p>In my book, <a href="https://tiny.datateams.io/amazon"><em>Data Teams</em></a>, I talk often about the importance of getting the balance of a team right, between people and skills, experience, and education — and then getting everyone to collaborate. From my research into successful and failing teams, I have come to the conclusion that to manage data as a product, you need three different overarching teams, with each team doing something very specific. Each team represents different skills, strengths, and approaches that are needed to create value from data.</p><p>Let me introduce you to each team briefly. The data engineering team creates the data products and makes sure they’re delivered through reliable data pipelines. The data science team uses these data products and combines them with advanced analytics to create new business value. The operations team keeps everything running smoothly so that data consumers are not relying on something that isn’t dependable.</p><p>So where do you begin and where do you invest first? It’s important to first of all understand what skills and strengths you currently have in your team, and identify the gaps in experience.</p><h3>The Operations Team</h3><p><em>In football, this is the defence. The world’s top ranked team (Belgium) is packed with attacking creativity, and they have formed one of the most trustworthy and dependable protective backlines in world football for the best part of a decade.</em></p><p>The operations team is responsible for putting off-the-shelf software and custom in-house build software into production and making sure it runs smoothly. The skills needed on an operations team are wide-ranging: hardware, software/operating systems, distributed systems, troubleshooting, security, data structures and formats, scripting/programming, operationalization best practices, monitoring and instrumentation, and disaster recovery.</p><p>The operations team has to be familiar with the data being used. This data includes information like the expected amount or size of the data itself, the type of data being sent, and the correct data format. Such information helps the team plan resource allocation.</p><h3>The Data Engineering Team</h3><p><em>On the football field this is the midfield. For Belgium, this is the strongest area of the team because they can call on an array of supporting talent, all of whom understand the role they need to play to support the team. It sets them apart — the balance of skills used to nullify an opponent or build an attack is unrivalled.</em></p><p>The data engineering team gets the data into the right place and ensures it is usable. A data engineering team is responsible for creating data products and the architecture to build data products, data pipelines. The data engineering team must choose the right technologies for the data and the use cases to accomplish this task.</p><p>Because writing and testing that code is a part of the data engineering team’s job, a software engineering background is highly recommended. The data engineers are tasked with writing the code that executes the use case on the big data framework, so they need to be skilled programmers. The data engineers are also responsible for continuous integration, unit tests, and engineering processes. These needs are often misunderstood and left missing from teams. Data engineers also have an interest in data itself. Some would go so far as to say they have a love of data — and I would agree.</p><h3>The Data Science Team</h3><p><em>On the football field these are the forwards. Belgian’s forwards operate at the sharp end, using the solid foundations built by the rest of the team to do the most important thing in football: score goals.</em></p><p>The goal of the data science team is to create advanced analytics. On the high end of the spectrum, these could be generated by artificial intelligence (AI), where machine learning (ML) is currently the most common technique. On the low end, these analytics are built from advanced statistics and mathematics. Other data team members may know statistics and mathematics too, but not at the sophisticated level required for data science.</p><p>Thus, a data scientist combines an advanced mathematics and statistics background with programming, domain knowledge, and communication skills to analyze data, create applied mathematical models, and present results in a form useful to the organization.</p><p>The data science team shouldn’t be asked to create the data infrastructure or software architectures they use to create the data products, do discovery, train the models, or deploy these models. These infrastructure tasks are something more in the domain of the data engineering team. Furthermore, data scientists are rare assets who have plenty to do besides creating infrastructure. Thus, the data science team should depend on the data engineering team’s data infrastructure and software architecture.</p><h3>The Manager</h3><p><em>In football, the manager isn’t on the field. You can see them pacing on the sidelines hoping their (data first) strategy and training is going to pay off. They’re responsible for the big picture decisions. It’s their job to make sure the right people are in the right positions and no skill is missing.</em></p><p>In data teams, this executive manager varies by company. At some companies, there are two or three different executives. For example, data science could be under a Chief Financial Officer and data engineering/operations is under the Chief Technical Officer. With two different executives, come different goals that can translate into challenges for the teams to align. With one executive, such as a Chief Analytics Officer or Chief Data Officer, there can be much better alignment around objectives and responsibilities. It’s my opinion and experience that data teams flourish under a single executive, just like there is only one manager on the football team.</p><h3>Conclusion</h3><p>Now that we’ve seen the various teams and their football counterparts, we can understand a data team better. We indeed see that no single part of the team is more important than the others. What would happen if one of our parts was missing?</p><p>From <a href="https://www.jesse-anderson.com/2020/12/data-teams-survey-results/">my research</a>, I’ve found that many organizations are missing at least one of the three teams. Depending on which team they’re missing, they’re either having all kinds of goals scored on them (no operations), can’t move the ball to a place to score (no data engineering), or can’t score a goal (no data science). Perhaps the most common team issue I’ve seen has been to hire a data team solely composed of data scientists and wonder why they can’t score any goals.</p><p>An effective team has to cover just about all the skills in the computer field, including a few that are relatively rare. So not only is it essential to assign clear roles to teams but some differentiation within each team is also needed to get the range of expertise you need. That said, there is some overlap between the roles of the different teams. Hence the importance of soft skills, especially communications, to round out the jobs. It’s only after extensive teamwork that we can score the goal!</p><p>As part of the Data Dream Team, there are a <a href="https://dreamteam.soda.io/">series of podcasts</a> that delve into growing and nurturing your team and their skills — with guests including Paco Nathan, Jordan Morrow, Zhamak Dehgahni, and Holden Karau. I encourage you to <a href="https://dreamteam.soda.io/">join the conversation</a> and hear how data leaders and practitioners are putting their data dream teams into play.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41b1fc8921a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Get Started Managing Data Quality With SQL and Scale]]></title>
            <link>https://sodadata.medium.com/how-to-get-started-managing-data-quality-with-sql-and-scale-d4108e4684aa?source=rss-2e82189e60b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4108e4684aa</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Soda]]></dc:creator>
            <pubDate>Wed, 12 May 2021 17:05:21 GMT</pubDate>
            <atom:updated>2022-01-21T21:08:42.249Z</atom:updated>
            <content:encoded><![CDATA[<h4>By Tom Baeyens, CTO &amp; Co-Founder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*UDeHU1B2LmTJUm_RBHmjNQ@2x.png" /><figcaption>Unsplash Contributor: Ricardo Gomez Angel</figcaption></figure><h3>Silent data quality issues are the biggest problem facing data teams today, who are flying blind with no systems or processes in place to monitor and detect bad data before it has a downstream impact.</h3><h3>Why data management?</h3><p>In the last three years I’ve transitioned from being a software engineer to a data engineer. I fell into the area of data management when Maarten Masschelein, my fellow co-founder at Soda, and I started working together to solve the problem of data issues that are silent and undetected. Coming from a software engineering background, writing unit tests and monitoring applications in production is a given but in data, it’s quite different. Whilst most organizations are aware they should test, there is no strategy in place and they just don’t know how to start addressing the problem which leaves their systems exposed and can result in serious downstream issues for the data products they are building.</p><p>With more and more products being built using data as the core input, it’s never been more important to test and monitor the quality of data being used. And so we set about building a data observability platform that enables organizations to discover, prioritize and resolve data issues.</p><h3>Define good data quality</h3><p>We started with <a href="https://github.com/sodadata/soda-sql">Soda SQL</a>, made available in February 2021. It’s our first open source data testing, monitoring and profiling tool for data-intensive environments. It works with your existing data engineering workflows to create a quick and easy way to define what good quality data means to your business. This enables data engineers to define tests and protect against the silent data issues that go undetected in datasets, data lakes, and data warehouses.</p><h3>Open source to the rescue</h3><p>Soda SQL is an open source tool with simple Command Line Interface (CLI) and Python library to test your data through metric collection. It utilizes YAML config files as input to prepare SQL queries that run tests on tables in a database to compute a wide range of metrics and tests. It’s super easy to find invalid, missing, or unexpected data. Because Soda SQL leverages — you guessed it — SQL, the data can stay where it is and existing compute engines can be leveraged.</p><p>If tests fail, Soda SQL allows you to stop the pipeline and prevent bad data from causing damage. As metrics are computed, diagnostic information is captured as well to help with the analysis if a data issue is detected. Steps can then be taken to prioritize and collaboratively resolve issues as one data team. Soda SQL can be used manually on its own or integrated with a data orchestration tool to schedule scans and automate actions based on scan results.</p><p>You can check out the <a href="https://docs.soda.io/soda-sql/getting-started/5_min_tutorial.html?utm_source=blog&amp;utm_medium=blogpost&amp;utm_campaign=KD%20Nuggets%20promo">5-minute tutorial</a> on how to get started but here’s a quick example:</p><ol><li>Simple metrics and tests can be configured in scan YAML configuration files. An example of the contents of such a file is as follows:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*sLcUfyb_49cqY4Kl" /></figure><p>2. Based on these configuration files, Soda SQL will scan your data each time new data arrived like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*5TqXZgXVp5yDiJGd" /></figure><h3>Bring everyone closer to the data</h3><p>We have just released <a href="https://www.soda.io/platform">Soda Cloud</a>, which is a web application where the Soda SQL metrics and test results can be monitored over time. Soda Cloud creates transparency from engineers to other people in the data team. With this collaboration data teams get ahead of the silent data issues. Soda Cloud extends Soda SQL and the two work together seamlessly.</p><p>First of all Soda Cloud extends Soda SQL with a metrics database so that measurements and test results can be visualized over time. This enables monitoring change over time and anomaly detection on all of the metrics.</p><p>These visualizations and data profiles already create transparency between different people in the larger data team. All people in the data team get to see what data is actually present, what tests are performed.</p><p>But the Soda Cloud goes one step further. It enables non-technical people to build and maintain their own monitors in a simple UI with a 3 step wizard. This is important because it removes the bottleneck to monitoring the domain knowledge that Subject Matter Experts have. If they don’t need to involve data engineers to get their domain logic tested, that means a lot more of that domain knowledge will be used to define what good data looks like. And as a result a lot more bad data will be captured preventing various kinds of damages.</p><p>Soda Cloud prescriptively solves the problem of discovering the silent data issues, by giving data teams a central platform to track and score the health of data across core quality dimensions.</p><p>Data and analytics engineers are equipped with a way to test data each and every time it transforms to ensure data pipelines are reliable. Via Soda SQL, data production can be stopped and quarantined. Soda Cloud visualizes the health of data sets and acts as a communication hub for data issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q6SjlfeFIUuIK4Kq" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*DC42fIbJg6IROzhS" /></figure><p>Data consumers and producers can now easily align on what’s important, what’s expected, and what to measure so that data remains fit for purpose. We’ve also built integrations with email and Slack to ensure the right people are alerted, at the right time to diagnose, prioritize and resolve the data issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9fplC75i96GG3M7H" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kab6yq76WL7zjetM" /></figure><p>We’re on a mission to bring everyone closer to the data, as we believe that data quality is a team sport. Everyone who has a stake in the data (and we think that’s everyone in the business nowadays), needs to understand it, trust it, and stay on top of it.</p><p>My main responsibility at Soda is to ensure data engineers love using our products and help them solve real problems quickly. We help solve the problem with a combination of a cloud platform and a set of open source developer tools, that give data teams the configurability they need to create end-to-end observability.</p><p>Good quality data is for everyone. Access <a href="https://github.com/sodadata/soda-sql">Soda SQL on GitHub</a> and Soda Starter, our free trial, on <a href="https://www.soda.io/platform">Soda.io</a> (extended to June 30, 2021). Our <a href="http://community.soda.io/slack">Slack Community</a> and <a href="https://docs.soda.io/soda-sql/">Docs</a> contain best practices and helpful resources.</p><p>Get ahead of the silent data issues. Good luck!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/1*ombt6XiCFlMY9nmdYQLSVA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4108e4684aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HelloFresh Hackathon: Trust in the Data!]]></title>
            <link>https://sodadata.medium.com/hellofresh-hackathon-trust-in-the-data-da6668c7fd05?source=rss-2e82189e60b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/da6668c7fd05</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-monitoring]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[hellofresh]]></category>
            <dc:creator><![CDATA[Soda]]></dc:creator>
            <pubDate>Wed, 12 May 2021 16:50:28 GMT</pubDate>
            <atom:updated>2022-01-21T21:09:53.199Z</atom:updated>
            <content:encoded><![CDATA[<h4>By Maarten Masschelein, CEO &amp; Co-Founder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*gy5cnD2GZ9l9hYL--oCenA.png" /><figcaption>© HelloFresh</figcaption></figure><p>We were thrilled to sponsor a globally organized hackathon at HelloFresh, which challenged cross-market (global) teams of HelloFresh employees to build a ‘Standard Data Quality Dashboard’.</p><p>Subtitled “In Data We Trust?!”, the hackathon was part of HelloFresh’s efforts to improve data literacy and data quality across the organization in an engaging and fun way. Whilst the organization has been building its data capabilities, maintaining trust in the data quality and integrity is always top of mind. With the data governance department, led by <a href="https://www.linkedin.com/in/dr-kinda-el-maarry-a27a13163/">Dr. Kinda El Maarry</a>, looking to ensure a standard way of reporting on data quality whilst ensuring high adoption rates, hackathon teams were tasked with creating a dashboard that decreased the mental load needed to understand various formats of data quality reporting, enabled easy comparisons of data assets and freed up resources and time spent on reacting to data quality issues. Now that time and resource is devoted to adding value through deeper insight and analysis.<br> <br>Soda was delighted to be able to support and sponsor this hackathon — we believe that data quality is a team sport and everyone who has a stake in the data, should be able to trust it and understand it. It was fascinating to see what the teams came up with! So, with six teams battling it out for a fantastic prize for each team member, plus some very cool HelloFresh/Soda swag up for grabs, what did our contestants come up with?<br> <br>The teams all identified the six dimensions that are used to assess the quality of data within a dataset; accuracy, completeness, consistency, timeliness, uniqueness and validity. The teams understood that the dashboard should be able to give a high-level executive summary of how selected datasets performed against each of these dimensions, highlighting at a glance whether and where data was falling short of the approved quality threshold.<br> <br>Being able to understand in simple terms where data quality is falling short is a powerful way of enabling an organization to boost data awareness amongst those not working within a specialised data team. Team “Fruit Salad” identified that its dashboard should be comprehensible to a Head of Production or Operations, offering high level information that wasn’t so granular that only data engineers and analysts could understand, but also wasn’t so topline that it was more suited for a senior executive level audience. <br> <br>Being able to delve deeper into data quality and see exactly where datasets are failing was another key functionality all of the teams built into their dashboards. Team “Data Chefs” suggested building the ability to pinpoint exact dates and times for when data that did not meet the 6 pillars was added to a dataset, enabling streamlined tracing and quarantine of bad data. “Two Nordics Guys” suggested a dashboard which allowed data engineers to view the rules of each data quality dimension to test and validate which specific rules and metrics caused the majority of data quality issues.<br> <br>Team “Tricolour Kiwis” took this a step further, and suggested the ability to weight the importance of each data quality dimension specific to each data asset. The ability to delve down into this level of granular detail, and zoom all the way out to a top level overview, is instrumental in encouraging meaningful interactions with data across all levels and functions of an organization.</p><p>A round of applause for all of the participants and congratulations to the winning team, <a href="https://www.linkedin.com/in/lauren-yiu/">Lauren Yiu</a> and <a href="https://www.linkedin.com/in/julien-varin/">Julien Varin</a> aka team “Tricolour Kiwis”.</p><p>What was absolutely clear in all of these entries was the importance of making data monitoring accessible to every part of an organization, creating complete transparency and providing end-to-end observability — that’s a sentiment we can get behind!<br> <br>Underpinning the data quality metrics being displayed by these dashboards is <a href="https://www.soda.io/platform">Soda’s Data Observability Platform</a>. I was pleased to be able to present my thoughts on best practices and tips on linking metrics to data quality dimensions, based on my experience in providing data monitoring solutions to organizations just like HelloFresh. I also shared our product design and ideations around data quality reporting and visualizations, a feature that we are releasing later this year. <br> <br>The ability to monitor data in order to ensure good data quality is critical to making good business decisions. My presentation at the hackathon touched on our first open-source project, <a href="https://github.com/sodadata/soda-sql">Soda SQL</a>, which is helping us to champion the engineering principles of Test Driven Development and apply them to the principles and framework of our data monitoring platform.<br> <br>Being able to monitor these test results couldn’t be easier, thanks to the ability to view and collaborate through a web application enabled by a <a href="https://cloud.soda.io/signup">free Soda Cloud account</a>.<br> <br>At Soda, we believe the foundation of data driven decision making is good quality data. If you want to get hands on with your data and leverage the Soda Hackathon-in-a-Box like the HelloFresh Team, please <a href="https://www.soda.io/company#Contact">get in touch with our events team</a> who will be happy to set you up (swag included). To see what a difference it could make for your organization, go ahead and visit our <a href="https://github.com/sodadata/soda-sql">Soda SQL Github Page</a>, and sign up for your <a href="https://cloud.soda.io/signup">free Soda Cloud account</a> today!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*6H-lJ69jUtHIxxWNbhMmwg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da6668c7fd05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Welcome Soda Cloud: End-to-End Data Observability and Collaboration]]></title>
            <link>https://sodadata.medium.com/welcome-soda-cloud-end-to-end-data-observability-and-collaboration-a70109ab0668?source=rss-2e82189e60b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a70109ab0668</guid>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-monitoring]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-transparency]]></category>
            <dc:creator><![CDATA[Soda]]></dc:creator>
            <pubDate>Wed, 12 May 2021 16:29:30 GMT</pubDate>
            <atom:updated>2021-06-10T08:30:04.337Z</atom:updated>
            <content:encoded><![CDATA[<p>By Maarten Masschelein, CEO &amp; Co-Founder</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T1aYALbqyo_Og5OLTKyayQ.png" /><figcaption>© Soda, created by Dahlia Ramirez</figcaption></figure><p>I got involved in the data space when I joined Collibra, a platform for data governance, cataloging, and discovery back in 2010. At that time, the role of the Chief Data Officer / Head of Data didn’t exist, and organizations were struggling to understand what was going on with their data, let alone manage data quality issues as the tooling that was available, was unwieldy and antiquated, and did not support the needs of the business (still does not in most cases). When I was introduced to Tom Baeyens, my fellow co-founder, it didn’t take long to connect the dots — and the data — and realize that there was a real world problem that we could solve combining Tom’s open source background and my data experience.</p><h3>The real world problem? Silent data issues.</h3><p>The issue we identified, and stand by today, is silent data issues. Silent data issues are at the core of what needs to be solved. Most data teams today are flying blind without systems or processes to detect problems with data. And as a result, data issues remain silent. Data issues are hidden across an organization’s entire data stack and largely impact data products — software that uses data to drive outcomes such as automated decision making, algorithms, derived data, raw data and decision support. Not only does the volume of data continue to grow exponentially, so does the number of data products that need to be managed on an ongoing basis. The data of these products can be compromised anywhere, at any time, as it moves from source to decision.</p><p>The issue continues to get bigger as systems keep going and processing bad data, with uncontrolled consequences including producing unexpected or erroneous results. This is why we call them<em> silent</em> data issues — data quality issues that are identified only once datasets are put to use in reports, campaigns, models, and used for decision making.</p><p>This can, and often does, result in data engineers spending too much time fire-fighting a data issue; data consumers having no confidence to trust the data; and the business spending too much time trying to resolve the far-reaching effects and consequences.</p><p>Let me be clear: data quality is an age-old problem. You’ll find it referenced in DAMA’s Data Management Body of Knowledge, whose first edition was published in 2009. The overall principles are solid; it’s the process, and technology that need to be hit with a refresh.</p><p>Our big idea was to empower data teams with tools to detect and resolve issues that matter <em>earlier upstream</em> to enable data confidence. And so Tom, the team and I, set to work to remove the fear of not knowing and the pain of finding out too late that a silent data issue has had a downstream impact.</p><p>Introducing Soda Cloud.</p><p>Early in our market research, it struck us that there was a general lack of observability into data systems (commonly called Data Observability). Many organizations had a way to index and discover data (catalog), but very few organizations could automatically discover and track the data problems across and within data sources that surface in the context of building data <em>products</em>. As a result, most data issues remained silent.</p><p>When teams were implementing a system to detect issues, it was mostly a rules-based testing service. To achieve this, they were relying on homegrown frameworks, most often using YAML files to configure the checks that needed to run each time new data arrived. At runtime, the DSL translates into a set of compute instructions to calculate metrics — typically in SQL or Spark — and evaluate checks.</p><p>The biggest problem data teams have with this set-up is that it doesn’t scale. It doesn’t scale because rules are hard to set-up and maintain, and it doesn’t scale because there are no automated issue discovery systems (e.g. tables are not refreshing based on the historical refresh schedule we’ve inferred from the query logs).</p><p>Another, even more important, scalability problem that we identified had <strong>nothing to do with technology, but with people and process</strong>. The majority of high-value checks were defined by subject matter experts (SMEs) of data, who are often not comfortable with Git + a YAML DSL. This limits adoption of both the new process and technology tremendously.</p><p>Data teams require a common framework to define and manage <strong>expectations for data behavior</strong>. Setting “Service Level Agreements” between data engineers and data consumers would bring clarity and consistency to teams when creating data products, eliminating (unwanted) assumptions and improving data quality on a continuous (automated) basis.</p><p><strong>Soda Cloud is a new, prescriptive approach to get ahead of silent data issues and manage data quality. It combines predictive capabilities with a super simple, yet powerful rules-based system. This allows data teams to create coverage very quickly, and get end-to-end observability.</strong></p><p>Soda Cloud is designed for a broad range of team members to get involved, from data platform engineers, to analytics engineers, product managers, and analysts. The goal is to help them to <strong>discover, prioritize, and resolve data issues collaboratively — and sooner</strong>. This way, the CDO / Head of Data can keep the oversight they need to help remove bottlenecks and ensure governance.</p><h4>Discover issues with data products intelligently.</h4><p>Through automated monitoring, <strong>metadata is collected, tracked, and monitored across core data quality dimensions</strong> including timeliness, completeness, consistency, and validity. <strong>By tracking datasets over-time, Soda learns</strong> the data refreshes interval, the typical volume of data processed, as well as any changes in the table schema (including changes inferred types). By doing this automatically for all datasets, a large portion of data issues can already be discovered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dnStTcn9q64b7Z3E-Wncsg.png" /><figcaption><em>This Soda Cloud Dashboard shows the results of automated monitoring. Key aspects of your data, such as freshness, data volumes, schema changes, user defined tests, and column level metrics are reported.</em></figcaption></figure><p>Next to this automated anomaly detection for “known unknowns”, data issues can also be discovered by testing and validating data. For this, our team has developed a simple, yet very powerful low-code, Domain Specific Language (sometimes called DSL) that allows you to do a wide range of checks. These include, but are not limited to consistency-over-time checks, reconciliations, reference data checks, and virtually any business logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Js9vxx3jnM5oHjkv" /><figcaption><em>This Soda Cloud Data Validation user experience allows non-SQL savvy users to set-up complex data validations to catch issues with a known exception scenario.</em></figcaption></figure><h4>Prioritize the data issues that matter most.</h4><p>Triaging and prioritizing data issues today has become an ever greater challenge, largely because of the volume of data that’s being amassed — and the variety of different data owners, consumers and stakeholders across the business. Data teams are realizing that <strong>not all data is created equal</strong>, and in order to keep quality high, an ongoing process needs to be put in place where data owners take responsibility for both prioritizing and resolving data issues. This thinking has most recently been outlined in the so-called <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">data mesh</a>.</p><p><strong><em>Data Freshness Dashboards — a mechanism to manage product performance<br></em></strong>Soda Cloud gives teams a tangible framework to define, visualize and track data behavior through <strong>Data Freshness Dashboards</strong>. Whether it’s for a BI report for Finance, or machine-learning instructions set up by Operations, This approach enables the data owners to manage their data as a product, and understand what every team expects of the data downstream. These dashboards show the data owner the discovered data issues from across the organization to accelerate both discovery and resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*8hyqDj3XePRMDHygrOf5Zw.png" /><figcaption><em>In this area of the Soda Cloud, scoring the health and fitness of data using data quality dimensions, can trigger an alert if for example a particular column is missing values, or a critical alert is missing more than 5%. This functionality defines “what good data looks like” based on the users’ expectations and helps eliminate alert fatigue.</em></figcaption></figure><p>Further analysis on the root cause of a data issue is often also needed. This can be achieved by, for example, exploring the data lineage in your data transformation, orchestration, and/or data cataloging tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1C0Lb2UcJBWC8uY3" /><figcaption>Lineage graph in the DBT Data Transformation tool.</figcaption></figure><h4>Resolve data issues collaboratively at the source.</h4><p>There’s no point in discovering and prioritizing data issues if there isn’t a robust follow up process to resolve them. <strong>Behind our mission is the ethos that the right people need to be brought together, at the right time</strong>. In this part of the process, it is to collaboratively resolve issues, and ultimately prevent issues.</p><p>At Soda, we believe that data quality is a team sport, however, you need to make sure that it’s the right people getting involved through role based alerting. The Soda platform facilitates collaboration by creating a shared context, and a clear resolution workflow for prioritizing and resolving issues and assigning tasks that matter most to the business.</p><p>Our approach and workflows take into consideration and recognize that it is the data owner that can ultimately make decisions on where to invest time and effort to improve the quality of data. Data owners should be able to easily ask both data engineers and data SMEs in the business to help analyze and fix issues. To streamline communications, integration options are available for the most-used channels such as e-mail, Chat, Slack or ServiceNow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j23NBWr-OaFzhcy4G9CKFg.png" /><figcaption><em>Resolution and root cause analysis begins with Annotation functionality. Here a data owner or business users can comment on an observation about the data and assign a Task for follow-up.</em></figcaption></figure><h3>Data Quality is a Team Sport: Soda Cloud Brings Everyone Together</h3><p>Strongly driven by the community that is forming, we are building and motivated by our mantra that data quality is a team sport. The moment that an organization can bring everyone (and we believe that is every single person in the business) closer to the data, that is when — well, the magic happens!</p><p><strong>Soda’s approach is different.</strong> Whilst there are a number of offerings in the market (with more and more joining this growing data space every day), they tend to focus on only one part of the problem, or solve the challenges of only one role in the data team.</p><p>With Soda Cloud, the entire end-to-end data quality process is brought together in a single platform, integrated and centralized. <strong>Each user can work with the right tools and workflows, in an environment that best suits their needs and expertise</strong>. This is what we mean when we say that <em>Soda brings everyone closer to the data</em>.</p><p>We solve the problem with a combination of a cloud platform and a set of open source developer tools.</p><h3>This is the Modern World of Data Quality</h3><p>Soda Cloud prescriptively solves the problem of discovering the silent data issues that matter, by <strong>giving data teams a central platform to track and score the health of data</strong> across core quality dimensions. For us, end-to-end observability and real-time collaboration will only occur when:</p><ul><li>Data platform engineers have a central infrastructure to enable monitoring-as-a-service, so that others can build data products on it.</li><li>Data and analytics engineers are equipped with a simple way to test data each and every time it is transformed, to ensure data pipelines are more reliable. From the developer toolkit, they can use Soda SQL to easily stop data production and quarantine bad data. Soda Cloud will visualize the health of datasets and act as a communication hub for the data issues.</li><li>Analysts, stewards and subject matter experts can validate data in a self-service way.</li><li>Data owners and data product managers can get an overview of all data expectations so they can easily prioritize issues for resolution.</li><li>Heads of data and analytics, or Chief Data Officers, are able to implement a culture of data ownership and a community of good data practices.</li></ul><h3>Join the Community</h3><p>For us, the goal of our community is to simplify the effort and share best practices. Everyone is looking to solve common problems with common solutions. And to help, we’re being open and reducing the friction to quickly get started with testing and monitoring data. We also want to reduce the fear, pain and sleepless nights caused by no solution or homegrown solutions that are not solving the problem.</p><p>Soda SQL is delivered in a white box approach that puts engineers in control. <a href="https://github.com/sodadata/soda-sql">The Soda Developer Tools kit, available on GitHub</a>, is built to fit naturally into the data engineer’s workflow.</p><p><a href="https://www.soda.io/licensing-options">Soda Cloud</a> is available as a free trial version (now extended until June 30 2021). Users can get monitoring in just minutes and realize the power of Soda on their own data.</p><p>There is power in the community and there is power in good data.</p><p><strong>I invite you to access our </strong><a href="https://github.com/sodadata/soda-sql"><strong>Developer Tools</strong></a> <strong>or </strong><a href="https://www.soda.io/platform"><strong>Soda Cloud</strong></a><strong> today. It’s time for every modern data team to create trust in the data.</strong></p><p>If I can help, <a href="https://www.soda.io/company#Contact">let me know</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*6H-lJ69jUtHIxxWNbhMmwg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a70109ab0668" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>