<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ziang Jia on Medium]]></title>
        <description><![CDATA[Stories by Ziang Jia on Medium]]></description>
        <link>https://medium.com/@ziangjia?source=rss-801dc1f7519a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MRH7WY1WYYXNgPphw7iBPw@2x.jpeg</url>
            <title>Stories by Ziang Jia on Medium</title>
            <link>https://medium.com/@ziangjia?source=rss-801dc1f7519a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:00:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ziangjia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Challenges for Enterprise GenAI App Design]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@ziangjia/things-like-change-data-capture-or-single-sign-on-matter-to-genai-application-d63858f29c7b?source=rss-801dc1f7519a------2"><img src="https://cdn-images-1.medium.com/max/1050/1*JoA7xkmY_wYK6YKfcwT70g.png" width="1050"></a></p><p class="medium-feed-snippet">If you enter this question into your ChatGPT, you get quite a few good bullet points, and it might provide you with a link to where it&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@ziangjia/things-like-change-data-capture-or-single-sign-on-matter-to-genai-application-d63858f29c7b?source=rss-801dc1f7519a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@ziangjia/things-like-change-data-capture-or-single-sign-on-matter-to-genai-application-d63858f29c7b?source=rss-801dc1f7519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d63858f29c7b</guid>
            <category><![CDATA[enterprise-technology]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ziang Jia]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 20:53:53 GMT</pubDate>
            <atom:updated>2024-10-30T14:57:00.835Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Key Considerations When Procure Data Warehouse Solution]]></title>
            <link>https://medium.com/@ziangjia/key-considerations-when-procure-data-warehouse-solution-ea4cde67052f?source=rss-801dc1f7519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea4cde67052f</guid>
            <category><![CDATA[procurement]]></category>
            <category><![CDATA[pricing-strategy]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[budget]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Ziang Jia]]></dc:creator>
            <pubDate>Wed, 25 Sep 2024 19:01:58 GMT</pubDate>
            <atom:updated>2024-09-25T19:01:58.099Z</atom:updated>
            <content:encoded><![CDATA[<p>Most organizations adapt two-tier or three-tier architecture when building data warehouse. Although it is already a common practice, limitations or performance bottlenecks could reduce efficiency in either reporting or advanced data analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*eMXOAtts0koDZ8Ky0wpAFA.png" /><figcaption>Three-Tier Data Architecture</figcaption></figure><p>When implementing a typical three-tier data architecture with cloud services, engineers often choose object store for staging area, follow by a licensed SQL warehouse to process and store raw data, finally expose the data models to analyst through a data mart. This implementation often requires multiple ETL or EL processes between the tiers, such as one ETL to load data from source into object store, one ETL from object store to SQL data warehouse, another ETL to move data from SQL data warehouse to analytic server where analysts often use Tableau or Python to generate reports. The ETL process can be time-consuming when dealing large volume of data so our analysts has to wait hours and hours to get the data they need. Moreover, as long running ETL tends to be buggy mostly due to connection issue, it needs significant budget and effort to maintain. This is where a Data Lakehouse architecture can do a better job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*sulLCSQLsM9YAu2w5Tkg_w.png" /><figcaption>Data Lakehouse Architecture in Azure Databricks</figcaption></figure><p>The most well-known data lakehouse architecture was brought to the market by Databricks in 2020. Other vendors are all developing their go-to-market data lakehouse platforms, such as Snowflake, Oracle OCI, Cloudera CDP, Azure Synapse, Google BigLake, etc. Nevertheless, the fundamental open-source projects behind the scenes are mainly these three — <a href="https://www.databricks.com/session_na20/a-thorough-comparison-of-delta-lake-iceberg-and-hudi">Delta Lake, Apache Iceberg, and Apache Hudi</a>. As a result, platforms that support the same open-source tools are more “natively interchangeable” and much easier to integrate.</p><h3>Human Centered Approach</h3><p>Once knowing what functionalities are needed, it is time to consider return on investment metrics. Data lakehouses are often considered an “accelerator” to improve human efficiency when they are working on delivering data insights. Therefore, as part of the standard procurement process when onboarding new technology, decision-makers should consider “human” as a significant dependency:</p><ol><li>What challenges must the team resolve after onboarding to this new platform?</li><li>Does my team currently hold the skillsets to use the lakehouse efficiently? What skill sets are required? How much training would my team need to support it long-term?</li><li>How much does the advanced feature would cost? Does the team need these features?</li><li>How does the team integrate transaction systems with the Lakehouse platform? Can they do it themselves, or would we need a Service Integration vendor?</li></ol><p>For example, if most of your analysts are familiar with Apache Spark, then adopting a platform that supports Delta-Lake is a better choice; Otherwise, choose a comprehensive SQL-like solution, such as Snowflakes. Similarly, if the organization mainly uses Google products, then choosing BigLake could make integration easier, etc; Otherwise, if everything is on Azure, adopt Synapse Analytics wherever possible.</p><h3>Pilot</h3><p>It is never easy to select the right platform, given the many similar offerings on the market. To find out which one really works, conducting pilot projects always delivers the correct answer. Since the big data analytic platforms market is so competitive, the organization may pilot two or three similar platforms simultaneously.</p><p>This is especially useful in the discovery phase when the organization is trying to decide the return on investment (ROI) of building and supporting a data lakehouse in the long term. A barebones PoC allows one to shop “necessaries” based on minimum needs instead of shopping “deals” from numbers of “fancy but might not be useful” functionalities. Some must-to-have features in a data lakehouse architecture might include but are not limited to:</p><ol><li>Network topology support, especially PaaS products.</li><li>High-performance object storage layer.</li><li>Support of standard file formats, such as parquet, csv, json, etc.</li><li>Economic computing layer.</li><li>Portable data governance layer, such as metadata, catalog, tags, etc</li><li>Upstream/Downstream integrated tools like ETL, ML, BI, etc.</li></ol><h3>Comprehensive Cost Estimate</h3><p>Let&#39;s assume one of the pilots goes well and we identify a lakehouse solution that can be seamlessly integrated with all our existing tools. Now it is time to estimate how much it would cost us once we put all our analytics workload on it.</p><p>We love sales engineers. They are smart and straight to the point, but sometimes, they purposefully leave ambiguity. The IT environment within an organization is mostly complicated, and it is hard to communicate everything clearly to an external account representative. Therefore, it should be the customers’ responsibility to determine the most accurate total cost ownership before signing any commitment agreement. The following questions cover some key aspects.</p><ol><li>What are the use cases/analytic products that need to run on the new lakehouse? How often do they run?</li><li>How many users will be using the lakehouse?</li><li>Pricing model. License by headcount, consumption-based, etc?</li><li>Is infrastructure cost included? This is usually in addition to the license cost.</li><li>Do we need to hire a professional service for the initial installation/configuration? or pro-serve is covered by the vendor?</li><li>Support Language Agreement. Who to call if the product is down and where to request a ticket if a promised feature doesn’t work?</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea4cde67052f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stream CDC Event to Databricks In Real-Time With Debezium]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@ziangjia/stream-cdc-event-to-databricks-in-real-time-with-debezium-7f46193e053e?source=rss-801dc1f7519a------2"><img src="https://cdn-images-1.medium.com/max/2571/1*ApiAR5EUQaVJPMXmkXFpTw.png" width="2571"></a></p><p class="medium-feed-link"><a href="https://medium.com/@ziangjia/stream-cdc-event-to-databricks-in-real-time-with-debezium-7f46193e053e?source=rss-801dc1f7519a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@ziangjia/stream-cdc-event-to-databricks-in-real-time-with-debezium-7f46193e053e?source=rss-801dc1f7519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f46193e053e</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Ziang Jia]]></dc:creator>
            <pubDate>Wed, 25 Sep 2024 02:29:37 GMT</pubDate>
            <atom:updated>2024-09-27T14:17:47.463Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Sharing External Hive-Metastore Across Jupyter, Databricks and Azure Synapse]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@ziangjia/sharing-external-hive-metastore-across-jupyter-databricks-and-azure-synapse-885e525bf7e5?source=rss-801dc1f7519a------2"><img src="https://cdn-images-1.medium.com/max/1200/1*CWc142ftoRYyHg3zUklNLA.png" width="1200"></a></p><p class="medium-feed-snippet">Learn how to setup a shared external Hive metastore to be used across platforms, JupyterHub on Kubernetes, Databricks and Synapse Analytics</p><p class="medium-feed-link"><a href="https://medium.com/@ziangjia/sharing-external-hive-metastore-across-jupyter-databricks-and-azure-synapse-885e525bf7e5?source=rss-801dc1f7519a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@ziangjia/sharing-external-hive-metastore-across-jupyter-databricks-and-azure-synapse-885e525bf7e5?source=rss-801dc1f7519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/885e525bf7e5</guid>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[azure-synapse-analytics]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <dc:creator><![CDATA[Ziang Jia]]></dc:creator>
            <pubDate>Wed, 14 Sep 2022 02:09:57 GMT</pubDate>
            <atom:updated>2022-09-14T02:10:45.593Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Secure Data Quality with Great Expectations in Databricks]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@ziangjia/secure-data-quality-with-great-expectations-in-databricks-d93a2e12da66?source=rss-801dc1f7519a------2"><img src="https://cdn-images-1.medium.com/max/1500/1*r2ClOlQwMxGCOHel6bZDbg.jpeg" width="1500"></a></p><p class="medium-feed-snippet">Validating data quality has became one of the top priorities for both data engineers and data scientists in their day-to-day data analysis&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@ziangjia/secure-data-quality-with-great-expectations-in-databricks-d93a2e12da66?source=rss-801dc1f7519a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@ziangjia/secure-data-quality-with-great-expectations-in-databricks-d93a2e12da66?source=rss-801dc1f7519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d93a2e12da66</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[great-expectations]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Ziang Jia]]></dc:creator>
            <pubDate>Tue, 23 Aug 2022 01:07:11 GMT</pubDate>
            <atom:updated>2022-08-23T02:42:39.648Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[DIY MPP Platform: Integrate JupyterHub with Apache Spark on Kubernetes]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@ziangjia/run-apache-spark-jobs-with-jupyter-notebook-on-kubernetes-cluster-the-open-source-solution-b41631bbb80b?source=rss-801dc1f7519a------2"><img src="https://cdn-images-1.medium.com/max/2344/1*fJ2yV7AGjo_XWe3Z1npguQ.png" width="2344"></a></p><p class="medium-feed-snippet">Build yourself an IDE to massively parallel process data at scale.</p><p class="medium-feed-link"><a href="https://medium.com/@ziangjia/run-apache-spark-jobs-with-jupyter-notebook-on-kubernetes-cluster-the-open-source-solution-b41631bbb80b?source=rss-801dc1f7519a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@ziangjia/run-apache-spark-jobs-with-jupyter-notebook-on-kubernetes-cluster-the-open-source-solution-b41631bbb80b?source=rss-801dc1f7519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b41631bbb80b</guid>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[jupyterhub]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Ziang Jia]]></dc:creator>
            <pubDate>Sun, 10 Jul 2022 03:19:55 GMT</pubDate>
            <atom:updated>2022-10-16T01:53:33.989Z</atom:updated>
        </item>
    </channel>
</rss>