<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikita Dementev on Medium]]></title>
        <description><![CDATA[Stories by Nikita Dementev on Medium]]></description>
        <link>https://medium.com/@ndementev?source=rss-3f41e41fa638------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*59i_477XANMIBwzh-GVKSg.jpeg</url>
            <title>Stories by Nikita Dementev on Medium</title>
            <link>https://medium.com/@ndementev?source=rss-3f41e41fa638------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:59:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ndementev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Enabling Data Discovery and Data Observability for Apache Spark]]></title>
            <link>https://medium.com/opendatadiscovery/enabling-data-discovery-and-data-observability-for-apache-spark-bb807869abd1?source=rss-3f41e41fa638------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb807869abd1</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-discovery]]></category>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-catalog]]></category>
            <dc:creator><![CDATA[Nikita Dementev]]></dc:creator>
            <pubDate>Mon, 20 Mar 2023 12:18:00 GMT</pubDate>
            <atom:updated>2023-03-20T12:18:00.681Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qF0w5Bj8-qS5BDNahxfESA.jpeg" /><figcaption><a href="https://www.freepik.com/free-photo/amazing-people-abstract-flaming-dance_1089929.htm#query=etl%20spark&amp;position=9&amp;from_view=search&amp;track=ais">Image by jigsawstocker</a> on Freepik</figcaption></figure><p>Data catalogs are essential parts of modern data infrastructures. They offer a centralized and standardized way to keep track of all data assets in an organization, including location, format, and source. By using a data catalog, data practitioners can easily find and understand the data they need for their work. This boosts efficiency, productivity, and data-driven decision making. Data catalogs also help ensure data quality and compliance by providing clarity on data sources and usage.</p><p>At OpenDataDiscovery, we are developing a data discovery and observability solution to help you efficiently find and understand the data you need for your work. However, we recognize that integrating ETL tools into data catalogs can be a challenging task, particularly for large organizations with complex data ecosystems. In the following paragraphs, we will discuss this issue in more detail and present a solution that we have developed to simplify this process.</p><h3>The Challenge of integrating ETL Tools into Data Catalogs</h3><p>Although products such as databases, data storage systems, and data lakes provide APIs to retrieve and process metadata, ETL tools often lack such APIs, making it difficult to fully integrate them into data catalogs automatically.</p><p>This problem is especially difficult for big companies with complicated data systems. In these environments, there may be hundreds or even thousands of sources of data, each with its own ETL tool and metadata format. This makes it virtually impossible to manually integrate all of the metadata into a single data catalog.</p><p>As a result, organizations may need to invest significant resources into developing their own tools or working with third-party vendors to create custom solutions. This can be time-consuming and expensive and may require specialized expertise that is not readily available in-house.</p><p>However, there is an ETL tool that is not affected by this problem and can be relatively easily integrated with data discovery solutions: <strong>Apache Spark™</strong>!</p><p>Spark’s built-in API can provide a wealth of metadata such as information on inputs, outputs, runs, and metrics. This API makes it especially convenient to integrate Spark ETL jobs with data discovery solutions.</p><h3>Integrating with OpenDataDiscovery Platform</h3><p>The OpenDataDiscovery solution covers data discovery and observability processes. Integrating Spark can improve these processes, keeping our users satisfied. Spark’s popularity in data companies makes it an appealing target for integration, as it can enhance the data exploration and visibility provided by the OpenDataDiscovery infrastructure.</p><p>So we did! Check this out: <a href="https://github.com/opendatadiscovery/odd-spark-adapter">https://github.com/opendatadiscovery/odd-spark-adapter</a></p><h4>OpenDataDiscovery Spark Adapter</h4><p>We are pleased to introduce the release of the OpenDataDiscovery Spark Adapter!</p><p>OpenDataDiscovery Spark Adapter simplifies the integration of Spark ETL jobs with the OpenDataDiscovery Platform, without requiring users to rewrite ETL jobs. Instead, the adapter leverages an internal Spark Listener interface to provide comprehensive metadata information for data discovery and observability processes.</p><p>The OpenDataDiscovery Spark Adapter version 0.0.1 currently supports the following:</p><ul><li>JDBC data sources</li><li>Kafka topics (batch mode only)</li><li>Snowflake</li><li>Delta tables on S3 or S3-compatible backend</li><li>low-level RDD jobs</li></ul><h4>Configuration and Use</h4><p>To use the OpenDataDiscovery Spark Adapter, it is necessary to configure the following settings:</p><ol><li>spark.odd.host.url — URL of OpenDataDiscovery Platform deployment</li><li>spark.odd.oddrn.key — The ODDRN key for a Spark cluster must uniquely identify it in the user’s data environment. This value can be randomly generated.</li></ol><p>As an optional configuration, you can enable or disable the integrated Spark Adapter using the spark.odd.enabled setting. By default, it is set to true.</p><p>The OpenDataDiscovery Spark Adapter is currently delivered in the form of a JAR file via GitHub. To use it, download the file and add it to the additional JAR classpath of the Spark ETL Job.</p><p>The complete command for running the Spark ETL job with OpenDataDiscovery Spark Adapter would be:</p><pre>./spark-submit \<br> --packages &lt;needed packages for the Spark jobs&gt; \<br> --jars &lt;path to the ODD Spark adapter JAR&gt; \<br> --conf &quot;spark.odd.host.url=&lt;ODD Platform URL&gt;&quot; \<br> --conf &quot;spark.odd.oddrn.key=&lt;ODDRN key&gt;&quot; \<br>   &lt;path_to_etl_job_code&gt;</pre><h3>Known Limitations</h3><p>In addition to adding new integrations to the OpenDataDiscovery Spark Adapter, there are currently two limitations:</p><ol><li>It does not support Spark structured streaming.</li><li>It only supports Spark version 3.3.x.</li></ol><p>These limitations are not dead ends. We are continuously improving the OpenDataDiscovery Spark Adapter by fixing them and adding support for additional Spark features and data sources based on user feedback.</p><h3>Next Steps for Development</h3><p>In addition to integrating more data sources that Spark can work with, we are planning to add support for various Spark versions and their features, such as job metrics or structured streaming</p><h3>Examples</h3><p><a href="https://github.com/opendatadiscovery/odd-spark-adapter/tree/main/docker">Follow this link</a> to find a demo project, complete with a README file that will guide you through the example.</p><h3>Conclusion</h3><p>We have developed the OpenDataDiscovery Spark Adapter to simplify the integration of Spark ETL jobs with the OpenDataDiscovery Platform, without requiring users to rewrite their ETL jobs. While we’ve made significant progress, there is always room for improvement.</p><p>We would greatly appreciate your support in advancing our project by giving us a star on GitHub and becoming an active participant in its development. We’re excited to hear your ideas and feedback! Please feel free to join the conversation on our Slack channel. We look forward to discussing with you!</p><p>Also, check out our Contribution Guide to learn about more ways to contribute to our mission of expanding data discovery and observability.</p><h3>Resources</h3><ul><li>OpenDataDiscovery — <a href="https://opendatadiscovery.org/">https://opendatadiscovery.org/</a></li><li>Apache Spark™ — <a href="https://spark.apache.org/">https://spark.apache.org/</a></li><li>Source code for OpenDataDiscovery Spark Adapter — <a href="https://github.com/opendatadiscovery/odd-spark-adapter">https://github.com/opendatadiscovery/odd-spark-adapter</a></li><li>Demo project — <a href="https://github.com/opendatadiscovery/odd-spark-adapter/tree/main/docker">https://github.com/opendatadiscovery/odd-spark-adapter/tree/main/docker</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb807869abd1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendatadiscovery/enabling-data-discovery-and-data-observability-for-apache-spark-bb807869abd1">Enabling Data Discovery and Data Observability for Apache Spark</a> was originally published in <a href="https://medium.com/opendatadiscovery">Open Data Discovery</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>