<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeff Zhang on Medium]]></title>
        <description><![CDATA[Stories by Jeff Zhang on Medium]]></description>
        <link>https://medium.com/@zjffdu?source=rss-4a87d907e2ca------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QoTJnqdfnQCzDHpHv0Cb2A.png</url>
            <title>Stories by Jeff Zhang on Medium</title>
            <link>https://medium.com/@zjffdu?source=rss-4a87d907e2ca------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:57:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zjffdu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Pipeline Abstraction]]></title>
            <link>https://zjffdu.medium.com/data-pipeline-abstraction-7cd0d71bb457?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cd0d71bb457</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Mon, 28 Apr 2025 13:52:43 GMT</pubDate>
            <atom:updated>2025-04-28T13:52:43.111Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KnSeCELNKicAhSZ_" /></figure><h3>Introduction</h3><p>In ETL world, data engineers/data analysts often talk about data pipeline, data flow, workflow which are very similar. Many people misunderstand those concepts or are confused by those concepts. Today I’d like to talk about their differences from abstraction perspective. In my opinion, we can classify data pipelines into 2 categories:</p><ul><li><strong>Operator Level Abstraction</strong> ( e.g. Spark pipeline / Apache beam pipeline, each node in DAG is a model/dataset. Building this kind of data pipeline is just like <strong>object oriented programming</strong>)</li><li><strong>Task Level Abstraction</strong> (e.g. Airflow pipeline, each node in DAG is a task. Building this kind of data pipeline is just like <strong>procedural programing</strong>)</li></ul><h3><strong>Operator Level Abstraction</strong></h3><p>This abstraction is <strong>data oriented</strong> which means the abstraction is on the dataset. Each node represents a dataset after applying one operator. The operator type could be categorized as classical Source/Transform/Sink pattern which corresponds to <strong>read/process/write</strong> data. For this kind of pipeline:</p><ul><li>All the nodes logic is running <strong>under the same runtime</strong>, this kind of pipeline is bounded with one underlying<strong> computing engine</strong> (e.g. Spark, Flink)</li><li>Data would move between nodes, and the data movement is controlled by the underlying computing engine(e.g. Spark, Flink).</li></ul><p>Here’s one example pipeline which reads data from Kafka, then does some filter processing and finally writes to Hdfs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8jzZ8gUqWP9SAxeEl-LCA.png" /></figure><h3>Task Level Abstraction</h3><p>This abstraction is <strong>task oriented</strong>. Each task could be <strong>heterogeneous</strong> and <strong>running under different runtime</strong> (e.g. SparkSql, Shell, Python).</p><ul><li>This kind of pipeline is bounded with a <strong>workflow engine</strong> (e.g. Airflow)</li><li>Data movement between nodes is not mandatory and is under user’s control, the pipeline runtime framework (Airflow) doesn’t control that, pipeline runtime framework only controls the execution order.</li></ul><p>Here’s one example pipeline which first runs one Spark sql job, then runs a shell script, and finally runs one Python script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jPJggZCEO2D-i7jK8umlwg.png" /></figure><p>Here’s a summary table to compare those 2 kinds of pipelines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*pjsYQwsrIEDb6Hpn5J796w.png" /></figure><h3>DBT &amp; SqlMesh</h3><p>At the first glance, you may think dbt &amp; sqlmesh is task level abstraction. But actually they are operator level abstraction. Each node of dbt/sqlmesh represent a dataset. It is not a arbitrary sql script in each node of db/sqlmesh, it is a speical sql script with select statement as the last statement. This select statement represents a dataset. So they are data oriented rather than task oriented.</p><h3>Mixture Those 2 Kinds of Pipelines</h3><p>Users can not mix those 2 kinds of pipelines arbitrarily. E.g. The following pipeline is invalid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IGPCQoAWiG96Q2n_hHFhuw.png" /></figure><p>But we can still mix those 2 kinds of pipelines by following the specific rule (Put operator level pipeline as a sub node of task level pipeline).e g.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EdkRoD80_6_xXkrPLo3JNA.png" /></figure><h3>Summary</h3><p>In conclusion, understanding the distinctions between Operator Level and Task Level abstractions is crucial for designing effective data pipelines in ETL processes. Operator Level pipelines, with their data-centric approach and reliance on a unified computing engine like Spark or Flink, offer streamlined data movement and processing for cohesive workflows. In contrast, Task Level pipelines, orchestrated by workflow engines like Airflow, provide flexibility by supporting heterogeneous tasks and user-controlled data movement, catering to diverse runtime environments. By recognizing these abstraction models — akin to object-oriented versus procedural programming — data engineers and analysts can better select and implement pipelines that align with their specific project requirements and computational needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cd0d71bb457" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to represent knowledge in LLM prompt]]></title>
            <link>https://zjffdu.medium.com/how-to-represent-knowledge-in-llm-prompt-f5f3e8036b37?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5f3e8036b37</guid>
            <category><![CDATA[knowledge-representation]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[knowledge]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 01:58:55 GMT</pubDate>
            <atom:updated>2025-04-25T03:19:47.152Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sP91BBq143-7PTur" /></figure><p>We all know that we need to tell LLM external knowledge in prompt. But how to represent knowledge, here’s my thoughts on that.</p><p>In my options, there’re 3 ways to represent knowledge in LLM prompt.</p><ul><li>Unstructured Plain Text</li><li>Structured Text (e.g. YAML, JSON)</li><li>Code (SQL, Python, etc.)</li></ul><p>Let’s deep dive into each approach.</p><h3>1. Unstructured Plain Text</h3><p><strong>Best for:</strong></p><ul><li><strong>Contextual knowledge</strong> (narrative, explanations, descriptions)</li><li><strong>Policies</strong>, guidelines, FAQs</li><li><strong>Business logic</strong> that isn’t formalized</li><li><strong>Background knowledge</strong> like domain-specific context</li></ul><p><strong>Strengths:</strong></p><ul><li>Natural to read and write</li><li>LLMs excel at reading between the lines</li><li>Great for nuanced or fuzzy information</li></ul><p><strong>Limitations:</strong></p><ul><li>Harder for the model to “anchor” or extract structured info from</li><li>Risk of ambiguity, repetition, or inconsistency</li></ul><p><strong>Examples:</strong></p><ul><li>“Our company defines an active user as someone who logs in at least once per week.”</li><li>“The table ‘orders’ includes transactions from all regions except internal testing regions.”</li></ul><h3>2. Structured Text (e.g. YAML, JSON)</h3><p><strong>Best for:</strong></p><ul><li><strong>Configurations</strong>, <strong>mappings</strong>, and <strong>schemas</strong></li><li><strong>Glossaries</strong> or definitions</li><li><strong>Entity-relationship info</strong> (e.g., table → columns → types)</li><li><strong>Prompt scaffolding</strong> or few-shot examples</li></ul><p><strong>Strengths:</strong></p><ul><li>LLMs parse it easily and reliably</li><li>Enforces clarity and structure</li><li>Easy to extract, transform, and reuse</li></ul><p><strong>Limitations:</strong></p><ul><li>Not expressive for complex logic</li><li>Hard to convey context or reasoning</li></ul><p><strong>Examples:</strong></p><pre>glossary:<br>  - term: active_user<br>    definition: Logs in at least once per week<br>  - term: churned_user<br>    definition: No logins in the last 30 days</pre><h3>3. Code (SQL, Python, etc.)</h3><p><strong>Best for:</strong></p><ul><li><strong>Executable knowledge</strong></li><li><strong>Transformations</strong> and <strong>queries</strong></li><li><strong>Data logic</strong> (joins, filters, conditions)</li><li><strong>Automation rules</strong> or <strong>calculations</strong></li></ul><p><strong>Strengths:</strong></p><ul><li>Very precise — less room for misinterpretation</li><li>Often easier to test and debug</li><li>Reusable across pipelines, workflows</li></ul><p><strong>Limitations:</strong></p><ul><li>May be abstract or brittle without explanation</li><li>Not human-readable for all audiences</li></ul><p><strong>Examples:</strong></p><pre>SELECT user_id<br>FROM logins<br>WHERE login_date &gt;= CURRENT_DATE - INTERVAL &#39;7 days&#39;</pre><p>Here’s a comparison table of these 3 knowledge representation</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3720f920010fd4d942b0e237ca5d6c2/href">https://medium.com/media/d3720f920010fd4d942b0e237ca5d6c2/href</a></iframe><h3>Comparison in Information Theory</h3><p>In information theory, more constrains means less entropy which means more information. For those knowledge representation, plain has the least constraint which means high entropy, structured text has the medium constrains which means medium entropy, while code has the most constraints which means low entropy. So in LLM prompt, you should use code as much as possible if it is suitable to use code to represent this kind of knowledge according the above comparison. Because code has the most information density, which cost less token also means cost less money.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97d44a01595b8ce42a5ad4fe925f2fbf/href">https://medium.com/media/97d44a01595b8ce42a5ad4fe925f2fbf/href</a></iframe><h3>Summary</h3><p>In LLM prompts, knowledge can be represented in three main formats: plain text, structured text (like YAML or JSON), and code (such as SQL or Python). Each format serves a different purpose based on how constrained and precise the information needs to be. The most effective prompts often combine all three formats to balance precision with interpretability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5f3e8036b37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knowledge Graph & Attention Mechanism.]]></title>
            <link>https://zjffdu.medium.com/knowledge-graph-attention-mechanism-134a0f7c4cec?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/134a0f7c4cec</guid>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[self-attention]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Tue, 22 Apr 2025 01:37:30 GMT</pubDate>
            <atom:updated>2025-05-18T13:34:40.770Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hFaYutzOSU3j9HPG" /></figure><p><strong>What if two powerful ideas in AI — attention and knowledge graphs — weren’t so different after all?</strong> One helps models like ChatGPT understand language by focusing on the right words in a sentence. The other stores real-world facts and connections, like a smart map of knowledge. At first glance, they seem like opposites — one is soft and learned, the other is structured and fixed. But when you look closer, they work like a perfect team. In this article, we’ll explore how attention and knowledge graphs each understand relationships in their own way — and why putting them together could lead to smarter, more reliable AI.</p><h4>Similarities between Knowledge Graph &amp; Attention Mechanism</h4><ol><li><strong>Relational Modeling</strong>:</li></ol><ul><li><strong>Attention Mechanism (in Transformers)</strong>: Models the <em>implicit</em> relationships between tokens. The attention scores tell you <em>how much</em> one token should “pay attention to” another, which can be interpreted as weighted edges in a graph.</li><li><strong>Knowledge Graph</strong>: Explicitly encodes relationships between entities (e.g., “Barack Obama” — <em>born_in</em> → “Hawaii”) as directed edges in a graph. These are human-curated or extracted from structured data.</li></ul><p><strong>2. Graph Perspective</strong>:</p><ul><li>A sentence <strong>can be seen as a graph</strong> where tokens are nodes and attention weights are edges. This is especially true in self-attention where each token attends to all others — forming a fully connected weighted graph.</li></ul><p><strong>3. Information Propagation</strong>:</p><ul><li>Both systems <strong>propagate information</strong> through connections (edges), whether they are learned (attention) or predefined (knowledge graph edges).</li></ul><h4>Differences between Knowledge Graph &amp; Attention Mechanism</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60c7a494442586568ca8fe2604de2a5e/href">https://medium.com/media/60c7a494442586568ca8fe2604de2a5e/href</a></iframe><h4>Knowledge graphs and attention mechanisms are highly complementary</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5282915a50218acd1891ab89361a9e2c/href">https://medium.com/media/5282915a50218acd1891ab89361a9e2c/href</a></iframe><h4>Summary</h4><p>We can think of attention as inducing a <em>latent</em> graph per sentence, which adapts dynamically depending on context. Knowledge graphs, in contrast, are more like a static <strong>semantic memory</strong> — a world model that encodes facts.</p><p>Combining them gives you:</p><ul><li>The <strong>flexibility and reasoning power</strong> of LLMs, and</li><li>The <strong>factual precision and structure</strong> of KGs.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=134a0f7c4cec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive into Apache Iceberg via Apache Zeppelin]]></title>
            <link>https://zjffdu.medium.com/deep-dive-into-apache-iceberg-via-apache-zeppelin-441ac79e0e01?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/441ac79e0e01</guid>
            <category><![CDATA[apache-zeppelin]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Fri, 15 Jul 2022 03:40:22 GMT</pubDate>
            <atom:updated>2022-07-17T14:34:18.506Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn the internal mechanism of Apache Iceberg in an interactive way</h4><p>Apache Iceberg is a high-performance format for huge analytic tables. There’re a lot of tutorials on the internet about how to use Iceberg. This post is a little different, it is for those people who are curious to know the internal mechanism of Iceberg. In this post, I will use Spark sql to create/insert/delete/update Iceberg table in Apache Zeppelin and will explain what happens underneath for each operation.</p><h3>Start Zeppelin Docker Container</h3><p>To demonstrate the internal mechanism more intuitively, I use Apache Zeppelin to run all the example code. You can reproduce what I did easily via Zeppelin docker. You can check this <a href="https://zjffdu.medium.com/learn-spark-on-zeppelin-in-docker-container-9f3f7b2db230">article</a> for how to <a href="https://zjffdu.medium.com/learn-spark-on-zeppelin-in-docker-container-9f3f7b2db230">play Spark in Zeppelin docker</a>. Here I just summarize it as following steps:</p><ul><li>Step 1. git clone <a href="https://github.com/zjffdu/zeppelin-notebook.git">https://github.com/zjffdu/zeppelin-notebook.git</a></li><li>Step 2. Download Spark 3.2.1</li><li>Step 3. Run the following command to start the Zeppelin docker container. ${zeppelin_notebook}is the notebook folder you cloned in Step 1, ${spark_location} is the Spark folder you downloaded in Step 2.</li></ul><pre>docker run -u <strong>$(</strong>id -u<strong>)</strong> -p 8080:8080 -p 4040:4040 --rm -v ${spark_location}:/opt/spark -v ${zeppelin_notebook}:/opt/notebook -e ZEPPELIN_NOTEBOOK_DIR=/opt/notebook -e SPARK_HOME<strong>=</strong>/opt/spark -e ZEPPELIN_LOCAL_IP=0.0.0.0 --name zeppelin apache/zeppelin:0.10.1</pre><p>Then open <a href="http://localhost:8080/">http://localhost:8080</a> in browser, and open the notebook Spark/Deep Dive into Iceberg which contains all the code in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lDfrUijSZKllUAw3RWPtKw.png" /><figcaption>Image by author</figcaption></figure><h3>Architecture of Iceberg</h3><p>Basically, there’re 3 layers for Iceberg:</p><ul><li>Catalog layer</li><li>Metadata layer</li><li>Data Layer</li></ul><h4>Catalog Layer</h4><p>Catalog layer has 2 implementations:</p><ul><li>Hive catalog which uses hive metastore. Hive metastore uses relational database to store where’s current version’s snapshot file.</li><li>Path based catalog which is based on file system. This tutorial uses path based catalog. It uses files to store where’s the current version’s metadata file. ( version-hint.text is the pointer which point to each version’s metadata file v[x].metadata.jsonin the below examples)</li></ul><h4>Metadata Layer</h4><p>In metadata layer, there’re 3 kinds of files:</p><ul><li>Metadata file. Each CRUD operation will generate a new metadata file which contains all the metadata info of table, including the schema of table, all the historical snapshots until now and etc. Each snapshot is associated with one manifest list file.</li><li>Manifest list file. Each version of snapshot has one manifest list file. Manifest list file contains a collection of manifest files.</li><li>Manifest file. Manifest file can be shared cross snapshot files. It contains a collection of data files which store the table data. Besides that it also contains other meta info for potential optimization, e.g. row-count, lower-bound, upper-bound and etc.</li></ul><h4>Data Layer</h4><p>Data layer is a bunch of parquet files which contain all the historical data, including newly added records, updated record and deleted records. A subset of these data files compose one version of snapshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-zpTw9puOcX0ucqXVgD6w.png" /><figcaption>Image by author</figcaption></figure><p>The diagram above is the architecture of Iceberg and also demonstrates what we did in this tutorial</p><ul><li>S1 means the version after we insert 3 records</li><li>S2 means the version after we update one record</li><li>S3 means the version after we delete one record</li></ul><h3>Preparation</h3><h4>Download jq and avro tools jar</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cTiabGsElmu24bBp-gZ3NA.png" /><figcaption>Image by author</figcaption></figure><p>jqis used for display json , avro tools jar is used to read iceberg metadata files (avro format) and display it in plain text.</p><h4>Configure Spark</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7vawz9mJm3CyGDGxfx9_zw.png" /><figcaption>Image by author</figcaption></figure><p>%spark.conf is a special interpreter to configure Spark interpreter in Zeppelin. Here I configure the Spark interpreter as described in this <a href="https://iceberg.apache.org/docs/latest/getting-started/">quick start</a>. Besides that, I specify the warehouse folder spark.sql.catalog.local.warehouse explicitly so that I can check the table folder easily later in this tutorial. Now let’s start to use Spark and play Iceberg in Zeppelin.</p><h3>Create Iceberg Table</h3><p>First Let’s create an Iceberg table events with 2 fields: idand data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ue5Z9okl-60DCj0xVhF2xw.png" /><figcaption>Image by author</figcaption></figure><p>Then describe this table to check its details</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S6Cot0lO6DkuOnxD1IcgCw.png" /><figcaption>Image by author</figcaption></figure><h4>Check Table Folder</h4><p>So what does Iceberg do underneath for this create sql statement? Actually, Iceberg did 2 things：</p><ul><li>Create a directory events under the warehouse folder /tmp/warehouse</li><li>Add a metadata folder which contains all the metadata info</li></ul><p>Since this is a newly created table, no data is in this table. There’s only one metadata folder under the table folder (/tmp/warehouse/db/events ). There’re 2 files under this folder:</p><ul><li>version-hint.text. This file only contains one number which point to the current metadata file v[n].medata.json)</li><li>v1.metadata.json. This file contains the metadata of this table, such as the schema, location, snapshots and etc. For now, this table has no data, so there’s no snapshots in this metadata file.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYWk8cpYM-Qsx4iII3gz1A.png" /><figcaption>Image by author</figcaption></figure><h3>Insert 3 Records (S1)</h3><p>Now let’s insert 3 new records (1, `a`), (2, `b`), (3, `c`)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dlDOipNLfTI7p3a7hxH5eQ.png" /><figcaption>Image by author</figcaption></figure><p>Then use select statement to verify the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCJ1IOQzsKtBmBgsv8egbg.png" /><figcaption>Image by author</figcaption></figure><h4>Check Table Folder</h4><p>Actually there’re 2 things happened underneath for this insert operation.</p><ul><li>In data folder, 3 parquet files are created. One record per parquet file.</li><li>In metadata folder, the content ofversion-hint.text is changed to 2, v2.metadata.jsonis created which has one newly created snapshot which point to one manifest list file. This manifest list file points to one manifest file which points to the 3 parquet files.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LtINcW6SAtaCb9RS1it6ag.png" /><figcaption>Image by author</figcaption></figure><p>We can use the avro tools jar to read the manifest list file which is avro format. And we find that it stores the location of manifest file and other meta info like added_data_files_count, deleted_data_files_count and etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OcOSYBYfe9fIbxm6-96ZbA.png" /><figcaption>Image by author</figcaption></figure><p>Then use the avro tools jar to read the manifest file which contains the path of the data files and other related meta info.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*taJYFdzkl4DribiWxS1bqg.png" /><figcaption>Image by author</figcaption></figure><p>We can use spark api to read raw parquet data files, and we can find there’s one record in each parquet file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKtjadVf_xD58nAz1ShVGQ.png" /><figcaption>Image by author</figcaption></figure><h3>Update Record (S2)</h3><p>Now, let’s use update statement to update one record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-RtO8r4b9Zit6RgPiOKtRA.png" /><figcaption>Image by author</figcaption></figure><h4>Check result after update</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_nPx1I4Wh1aPU7HKhauzA.png" /><figcaption>Image by author</figcaption></figure><h4>Check Table Folder</h4><ul><li>In data folder, the existing parquet files are not changed. But one new parquet file is generated.(3, `c_updated`)</li><li>In metadata folder, the content ofversion-hint.text is changed to 3, v3.metadata.jsonis created which has 2 snapshots. One snapshot is the first snapshot in above step, another new snapshot is created which has a new manifest list file.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fl5JhQ0nQjUKdv4MTyKlsw.png" /><figcaption>Image by author</figcaption></figure><p>You might be curious to know how Iceberg implements the update operation without changing existing data. This magic happens in Iceberg metadata layer. If you check this version’s metadata file, you will find now it contains 2 snapshots, and each snapshot is associated with one manifest list file. The first snapshot is the same as above, while the second snapshot is associated with a new manifest list file. In this manifest list file, there’re 2 manifest files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v5iPDyFRXWPo0jdalBTB-g.png" /><figcaption>Image by author</figcaption></figure><p>The first manifest file point to newly added data file (3, `c_updated`). While in the second manifest file, you will find that it still contains 3 data files that contains (1, `a`), (2, `b`), (3, `c`), but the status of the third data file(3, `c`) is 2 which means this data file is deleted, so when Iceberg read this version of table, it would skip this data file. So only (1,`a`), (2, `b`) will be read.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tG8YYxf06hfgJd--v6acA.png" /><figcaption>Image by author</figcaption></figure><h3>Delete Record (S3)</h3><p>Now, let’s delete record (2, `b`)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yFJAghhcN1exjT0ETBkDHA.png" /><figcaption>Image by author</figcaption></figure><p>Use select statement to verify the result</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-zlkxpTQvKJGmRdsV-ifdw.png" /><figcaption>Image by author</figcaption></figure><h4>Check Table Folder</h4><ul><li>In data folder, nothing changed.</li><li>In metadata folder, the content ofversion-hint.text is changed to 4, v4.metadata.jsonis created which has one more snapshots (totally 3 snapthots).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOGtyjmFP79MR0wmt7PeXA.png" /><figcaption>Image by author</figcaption></figure><p>The manifest list file associated with the new snapshot contains 2 manifest files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UF6F0Q8fpAwtv999oos2ng.png" /><figcaption>Image by author</figcaption></figure><p>The first manifest point to 1 data files (3, `c_updated`), , the second manifest file point to data file (1, `a`), (2, `b`). But the status of data file (2, `b`) is 2, which means it has been deleted, so when Iceberg read this version of table, it would just skip this data file. So only (1, `a`) will be read.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rc8tqnipCWw7wLaf0VZR-w.png" /><figcaption>Image by author</figcaption></figure><p>Use spark api to read these data files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1L6b3YxJTd1QRv2eUU3hdQ.png" /><figcaption>Image by author</figcaption></figure><h3>Inspect Metadata</h3><p>You can also read metadata tables to inspect a table’s history, snapshots, and other metadata.</p><h4>Inspect history metadata</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kB54FHRB6JvY8LND9n5zTA.png" /><figcaption>Image by author</figcaption></figure><h4>Inspect snapshot metadata</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*np9PThzBUbx5v41GVq7SOg.png" /><figcaption>Image by author</figcaption></figure><h4>Inspect manifest metadata</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LXB63m82KZAF1eUxsvkrVg.png" /><figcaption>Image by author</figcaption></figure><h4>Inspect file meta table</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6_ao_i2_wogGdO2NqefCAA.png" /><figcaption>Image by author</figcaption></figure><h3>Summary</h3><p>In this article, I do 4 main steps to play Apache Iceberg:</p><ul><li>Create Table</li><li>Insert Data</li><li>Update Data</li><li>Delete Data</li></ul><p>At each step, I check what is changed under the table folder. All the steps are done in Apache Zeppelin docker container, you can reproduce them easily. Just one thing to remember, because the file names are randomly generated (snapshot file, manifest file, parquet file), so you need to update code to use the correct file name. Hope this article is useful for you to understand the internal mechanism of Apache Iceberg.</p><h3>References</h3><ul><li><a href="https://zeppelin.apache.org/docs/0.10.1/interpreter/spark.html">Spark Interpreter for Apache Zeppelin</a></li><li><a href="https://iceberg.apache.org/">Apache Iceberg™</a></li><li><a href="https://www.dremio.com/resources/guides/apache-iceberg-an-architectural-look-under-the-covers/">Apache Iceberg: Architectural Insights | Dremio</a></li><li><a href="https://www.starburst.io/blog/trino-on-ice-iv-deep-dive-into-iceberg-internals/">Trino on Ice IV: Deep Dive Into Iceberg Internals</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=441ac79e0e01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn PySpark in Apache Zeppelin Notebook (Part-1)]]></title>
            <link>https://zjffdu.medium.com/learn-pyspark-in-apache-zeppelin-notebook-part-1-3214abede98a?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/3214abede98a</guid>
            <category><![CDATA[notebook]]></category>
            <category><![CDATA[apache-zeppelin]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 14:51:54 GMT</pubDate>
            <atom:updated>2021-12-06T14:51:54.213Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn PySpark in an interactive way</h4><p>There’re many tutorials on the internet about how to learn PySpark in the Jupyter notebook. But most of the tutorials you find are telling you how to run PySpark in local mode in the Jupyter notebook. If you want to run PySpark in distributed mode (yarn or k8s), it would be a big pain point to do that in the Jupyter notebook. In this series of articles, I’d like to introduce Apache Zeppelin Notebook where you can learn PySpark without an extra complex setup. Only a few configurations are needed if you want to run PySpark in yarn or k8s. I would split this series of tutorials into 2 parts:</p><ul><li>Learn PySpark in Zeppelin Notebook docker (Part-1)</li><li>Setup production PySpark environment (on yarn) in Zeppelin (Part-2)</li></ul><p>The easiest way to learn PySpark on Zeppelin is via its docker. It is super easy to learn PySpark in Zeppelin docker.</p><h3>Setup PySpark environment via Zeppelin docker</h3><ul><li>Step 1</li></ul><p>In this article, I would use Spark 3.1.2. You can download it here, <a href="https://spark.apache.org/downloads.html">https://spark.apache.org/downloads.html</a>，after downloading please untar it like following:</p><pre>tar -xvf <a href="https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop2.7.tgz">spark-3.1.2-bin-hadoop2.7.tgz</a></pre><ul><li>Step 2</li></ul><p>Start Zeppelin in docker via the following command. To be noticed, /Users/jzhang/Java/lib/spark-3.1.2-bin-hadoop3.2 is my spark location, use your own spark location that you download in Step 1</p><pre>docker run -u $(id -u) -p 8080:8080 -p 4040:4040 --rm -v /Users/jzhang/Java/lib/spark-3.1.2-bin-hadoop3.2:/opt/spark -e SPARK_HOME=/opt/spark  --name zeppelin apache/zeppelin:0.10.0</pre><ul><li>Step 3</li></ul><p>Open <a href="http://localhost:8080/">http://localhost:8080</a> in your browser, you can see the Zeppelin home page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y96YYEKzaJFzn8l5QJ3M_g.png" /></figure><p>You can see there’re many Spark tutorials shipped in Zeppelin, since we are learning PySpark, just open note: <strong>3. Spark SQL (PySpark)</strong></p><p>SparkSession is the entry point of Spark SQL, you need to use SparkSession to create DataFrame/Dataset, register UDF, query table and etc. Zeppelin has created SparkSession(spark) for you, so don’t create it by yourself.</p><h3>PySpark API</h3><p>PySpark API is very similar to pandas api which has so many APIs, I won’t suggest you remember every api of PySpark. Instead, you can put them into the following 3 main categories, and just learn some often-used api, for these not often-used api, you can search the doc when needed.</p><ul><li>Create DataFrame</li><li>Transmation on single DataFrame</li><li>Transformation on multiple DataFrames</li></ul><h3>Create DataFrame</h3><p>Overall, there’re 2 ways to create DataFrame</p><ul><li>Create a dataframe from python objects</li><li>Create a dataframe from files</li></ul><h3><strong>Transformation on single DataFrame</strong></h3><h4>Add column</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lJpGX120SuJmRD9ozKYStQ.png" /></figure><h4>Remove column</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wu_6NW8tDPykxF5G5LsuVA.png" /></figure><h4>Select subset of columns</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wRBQyGEJy_p2vHo9J3473A.png" /></figure><h4>Filter rows</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0cOXFyVMVYdBK-PU7oWEA.png" /></figure><h4>Group by</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZbR3w1ScyoKqULcDbgfDNQ.png" /></figure><h3>Transformation on Multiple DataFrames</h3><h4>Join on a single column</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_KPxk-vPINEdZJgaZJ61Jg.png" /></figure><h4>Join on multiple columns</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nhC36DWrzYXlu44Q3hSrLQ.png" /></figure><h3>Use SQL on DataFrame</h3><p>One advantage of Zeppelin is that you can collaborate with multiple languages, e.g. you can use both Python and SQL together in one notebook, and they share the same SparkContext/SparkSession.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9H34WkoyXTmbsRcBUjtWA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uIKd398GV3kkbbiST-qETw.png" /></figure><h3>Visualization</h3><p>Besides the above Zeppelin built-in visualization capability, you can also use Python’s visualization libraries, such as matplotlib,</p><h4>Matplotlib</h4><p>Matplotlib is a Python 2D plotting library that produces publication-quality figures in a variety of hardcopy formats and interactive environments across platforms. The usage of Matplotlib in Zeppelin is the same as Jupyter Notebook. The key is to put %matplotlib inline before using Matplotlig. Below is one simple example, for more usage of Matplotlib, you can refer to this <a href="https://matplotlib.org/">link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrxcCsKTefLrh9uEB0D8lA.png" /></figure><h4>Seaborn</h4><p>Seaborn is a Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics. Its usage in Zeppelin is the same as in Jupyter. For seaborn usage please refer to this <a href="https://seaborn.pydata.org/">link</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q_MLVtM352QCjLNnDnSuUA.png" /></figure><ul><li>Plotnine</li></ul><p>plotnine is an implementation of a grammar of graphics in Python, it is based on ggplot2. The grammar allows users to compose plots by explicitly mapping data to the visual objects that make up the plot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qb4LGcY48aSI9eho61ROZw.png" /></figure><h4>Altair</h4><p><a href="https://altair-viz.github.io/">Altair</a> is a declarative statistical visualization library for Python, based on Vega and Vega-Lite, and the source is available on GitHub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7uLvEl6HtK7m5MPjzeumFQ.png" /></figure><h4>Plotly</h4><p><a href="https://plotly.com/python/">plotly.py</a> is an interactive, open-source, and browser-based graphing library for Python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LIFdEKTTQXKKmYaPwBWqtQ.png" /></figure><h3>Summary</h3><p>This is part-1 of the PySpark tutorial series in Zeppelin, just mention the basic features of PySpark in Zeppelin. In the next part, I would take more such as the configuration of PySpark in yarn cluster, how to customize Python runtime environment and etc.</p><h3>References</h3><ul><li><a href="https://zeppelin.apache.org/docs/0.10.0/interpreter/spark.html">Spark Interpreter for Apache Zeppelin</a></li><li><a href="https://zeppelin.apache.org/docs/0.10.0/interpreter/python.html">Python 2 &amp; 3 Interpreter for Apache Zeppelin</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3214abede98a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive into Delta Lake via Apache Zeppelin]]></title>
            <link>https://medium.com/data-science/deep-dive-into-delta-lake-via-apache-zeppelin-d59db1673584?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/d59db1673584</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[apache-zeppelin]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Sun, 28 Nov 2021 11:23:57 GMT</pubDate>
            <atom:updated>2021-12-25T16:01:41.977Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn the internal mechanism of Delta Lake in an interactive way</h4><p>Delta Lake is an open-source project that enables building a Lakehouse architecture on top of data lakes. There’re a lot of tutorials on the internet about how to use Delta Lake. This post is a little different, it is for those people who are curious to know the internal mechanism of Delta Lake, especially the transaction log.</p><h3><strong>Start Zeppelin Docker Container</strong></h3><p>To demonstrate the internal mechanism more intuitively, I use Apache Zeppelin to run all the example code. You can reproduce what I did easily via Zeppelin docker. You can check this <a href="https://zjffdu.medium.com/learn-spark-on-zeppelin-in-docker-container-9f3f7b2db230">article</a> for how to <a href="https://zjffdu.medium.com/learn-spark-on-zeppelin-in-docker-container-9f3f7b2db230">play Spark in Zeppelin docker</a>. Here I just summarize it as following steps:</p><ul><li>Step 1. git clone <a href="https://github.com/zjffdu/zeppelin-notebook.git">https://github.com/zjffdu/zeppelin-notebook.git</a></li><li>Step 2. Download Spark 3.1.2 (This is what I used in this tutorial, don’t use Spark 3.2.0, it is not supported yet)</li><li>Step 3. Run the following command to start the Zeppelin docker container. ${zeppelin_notebook}is the notebook folder you cloned in Step 1, ${spark_location} is the Spark folder you downloaded in Step 2.</li></ul><pre>docker run -u <strong>$(</strong>id -u<strong>)</strong> -p 8080:8080 -p 4040:4040 --rm -v ${spark_location}:/opt/spark -v ${zeppelin_notebook}:/opt/notebook -e ZEPPELIN_NOTEBOOK_DIR=/opt/notebook -e SPARK_HOME<strong>=</strong>/opt/spark -e ZEPPELIN_LOCAL_IP=0.0.0.0 --name zeppelin apache/zeppelin:0.10.0</pre><p>Then open <a href="http://localhost:8080/">http://localhost:8080</a>, and open the notebook Spark/Deep Dive into Delta Lake which contains all the code in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6tLe7BmjvU2EOIWGs0Ww_g.png" /><figcaption><em>Zeppelin Home Page (Image by author)</em></figcaption></figure><h3>Configure Spark</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_OAhT-TuPJezTpSpuihG1g.png" /><figcaption><em>Spark Configuration (Image by author)</em></figcaption></figure><p>This is the first paragraph of Deep Dive into Delta Lake, which is to configure Spark interpreter to use Delta Lake.</p><p>%spark.conf is a special interpreter to configure Spark interpreter in Zeppelin. Here I configure the Spark interpreter as described in this <a href="https://docs.delta.io/latest/quick-start.html#">quick start</a>. Besides that, I specify spark.sql.warehouse.dir the warehouse folder explicitly so that I can check the table folder easily later in this tutorial. Now let’s start to use Spark and play Delta Lake in Zeppelin.</p><h3>Create Delta Table</h3><p>First Let’s create a Delta table events with 2 fields: id and data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IIsiNnBFl6GfAH-rrsUlKg.png" /><figcaption><em>Create Delta Table (Image by author)</em></figcaption></figure><p>So what does Delta do underneath for this create SQL statement? Actually, Delta did 2 things：</p><ul><li>Create a directory events under the warehouse folder /tmp/warehouse</li><li>Add a transaction log that contains the schema of this table</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D2_AQ0gjMA2nyAC4geuAaw.png" /><figcaption>Image by author</figcaption></figure><h3>Insert data</h3><p>Now let’s insert some data into this Delta table. Here I just only insert only 2 records: (1, data_1), (2, data_2)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vCySdCvif_3rItgu3OqUqA.png" /><figcaption>Insert data (Image by author)</figcaption></figure><p>Then let’s run a select SQL statement to verify the result of this insert statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fDU_U0HM_my7wmslMLoH7Q.png" /><figcaption>Query table (Image by author)</figcaption></figure><p>So what does Delta do underneath for this insert SQL statement? Let&#39;s check the table folder /tmp/warehouse/events , there’re 2 changes</p><ul><li>Another new transaction log file is generated.</li><li>2 parquet files are generated</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bqCc6HkZBJHQ1KH9_omRVg.png" /><figcaption>Image by author</figcaption></figure><p>First, let’s take a look at the new generated transaction file (00000000000000000001.json). This JSON file is very readable, it contains the operation of this insert SQL statement: Add 2 parquet files that contain the 2 records. To be noticed, there’s no table schema info in this new transaction log file, because it is already in the first transaction log file (00000000000000000000.json). When Delta read the table, it would merge all the historical transaction files since then to get all the information of this table (including the schema of this table and what data files are included)</p><p>Since we only insert 2 records, it is natural to guess that each parquet contains one record. We can read these 2 parquet files directly to verify that. As the following code shows, our guess is correct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*po2Suu9pcYiMrPNKIYmNzg.png" /><figcaption>Image by author</figcaption></figure><h3>Update Data</h3><p>The most important feature of Delta is ACID support, you can update the table at any time without affecting others who also read/write the same table simultaneously. Now let’s update this eventstable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gm96vECS5ry4_RttUjrbwA.png" /><figcaption>Update data (Image by author)</figcaption></figure><p>Then run a select statement to verify the result of this update statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ejzqWlVh6ZPEkLEPuBgkA.png" /><figcaption>Image by author</figcaption></figure><p>So what does this update statement do underneath? We can check the events table folder and would find 2 changes:</p><ul><li>Another new transaction log file is generated</li><li>Another parquet file is added (the previous 2 parquet files are still there)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yS9R1xKUiWKJavWnQapMfw.png" /><figcaption>Image by author</figcaption></figure><p>First, let’s take a look at the new transaction log file content, there’re 2 operations:</p><ul><li>Remove one parquet file</li><li>Add a new parquet file</li></ul><p>It is natural for us to guess that the removed file contains the records (2, data_2), while the newly added file contains the record (2, data_2_updated). Let’s read these 2 parquet files directly to verify our guess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gDyr21GUqFw6c67cElaUqg.png" /><figcaption>Image by author</figcaption></figure><p>Now let’s use the time travel feature of Delta. We would like to use the last version of this table before this update operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kdL9ZKGsKNwSqkpG2ue0ZA.png" /><figcaption>Image by author</figcaption></figure><p>The time travel feature works just because Delta doesn’t delete the data file, it only records all the operations in the transaction logs. When you read version 1 of this table, Delta Lake would only read the first 2 transactions logs: 00000000000000000000.json &amp; 00000000000000000001.json.</p><h3>Delete Data</h3><p>Now let’s do the delete operation on this events table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPSlTIi4IEBJra6kooPu6g.png" /><figcaption>Delete data (Image by author)</figcaption></figure><p>And then run a select statement to verify the result of this delete statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yWJTwc9h234_Rkvn8jMHKQ.png" /><figcaption>Image by author</figcaption></figure><p>So what does Delta do for this delete operation underneath? We can still check the eventstable folder and would find 2 changes:</p><ul><li>A new metadata transaction log file is generated</li><li>A new parquet file is added</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xPgZ5LVQNRZM5UEbFk1h0g.png" /><figcaption>Image by author</figcaption></figure><p>In the new transaction log file we still see 2 operations: remove and add</p><p>It is natural to guess that the remove operation just removes the file which contains record (1, data_1), so what does this new add operation do? Actually, the newly added parquet file is empty which contains nothing, we can read these 2 parquet files directly to verify that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*In3zdot_0vVepQ972HXhgw.png" /><figcaption>Image by author</figcaption></figure><h3>Summary</h3><p>In this article, I do 4 main steps to play Delta Lake:</p><ul><li>Create Table</li><li>Insert Data</li><li>Update Data</li><li>Delete Data</li></ul><p>At each step, I check what is changed in the transaction log and data file. All the steps are done in Apache Zeppelin docker container, you can reproduce them easily, hope this article is useful for you to understand the internal mechanism of Delta Lake.</p><h3>Reference</h3><ul><li><a href="http://zeppelin.apache.org/">Zeppelin</a></li><li><a href="https://databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html">Understanding the Delta Lake Transaction Log - Databricks Blog</a></li><li><a href="https://databricks.com/blog/2020/09/29/diving-into-delta-lake-dml-internals-update-delete-merge.html">Delta Lake Tutorial: How to Easily Delete, Update, and Merge Using DML</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d59db1673584" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deep-dive-into-delta-lake-via-apache-zeppelin-d59db1673584">Deep Dive into Delta Lake via Apache Zeppelin</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn Spark on Zeppelin in Docker container]]></title>
            <link>https://zjffdu.medium.com/learn-spark-on-zeppelin-in-docker-container-9f3f7b2db230?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f3f7b2db230</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[apache-zeppelin]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[notebook]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Mon, 25 Oct 2021 09:42:37 GMT</pubDate>
            <atom:updated>2021-10-25T09:42:37.699Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Zeppelin is web-based notebook that enables data-driven,<br>interactive data analytics and collaborative documents with SQL, Scala, Python, R and more.</p><p>This article is to guide you how to play Spark on Zeppelin in docker container without any manual setting.</p><h3>Step 1. Download Spark</h3><p>Zeppelin (0.10.0) supports many versions of Spark (from 1.6 to 3.1), you can download whatever version you’d like to use from <a href="https://spark.apache.org/downloads.html">apache spark site</a>.</p><h3>Step 2. Start Zeppelin in Docker</h3><p>Then run the following command to start Zeppelin in Docker container:</p><pre>docker run -u <strong>$(</strong>id -u<strong>)</strong> -p 8080:8080 -p 4040:4040 -p 6789:6789 --rm -v ${spark_location}:/opt/spark -e SPARK_HOME<strong>=</strong>/opt/spark -e ZEPPELIN_LOCAL_IP=0.0.0.0 --name zeppelin apache/zeppelin:0.10.0</pre><p>Just replace ${spark_location} with the Spark location that you downloaded in step 1.</p><p>Then open <a href="http://localhost:8080/">http://localhost:8080</a>, you can see the following page. You can also run the above command on another machine, and access Zeppelin remotely via url like <a href="http://my-zeppelin-server:8080/">http://my-zeppelin-server:8080</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QIkPcU1mvQGq5BqoMDxQdg.png" /></figure><p>Zeppelin is shipped with some built-in tutorial notes. Open folder Spark Tutorial , you can see 8 Spark tutorial notes.</p><p>Here’s one short video which demonstrate how to Play Spark in Zeppelin docker container. You can run most of them directly without any setting. In the following video, I showed you how to run the following 3 tutorial note.</p><ul><li>2. Spark Basic Features</li><li>3. Spark SQL (PySpark)</li><li>5. SparkR Basics</li><li>6. SparkR Shiny App</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FxxMNWL1Yq0g%3Fstart%3D3%26feature%3Doembed%26start%3D3&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxxMNWL1Yq0g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FxxMNWL1Yq0g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dcf359a5d9f2f3badc06b9ea2f86c735/href">https://medium.com/media/dcf359a5d9f2f3badc06b9ea2f86c735/href</a></iframe><p>Besides that, there’re more features in Zeppelin’s Spark interpreter, for more details, please check out Zeppelin’s official document. <a href="https://zeppelin.apache.org/docs/0.10.0/interpreter/spark.html,">https://zeppelin.apache.org/docs/0.10.0/interpreter/spark.html,</a> and Join our community here. <a href="http://zeppelin.apache.org/community.html">http://zeppelin.apache.org/community.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f3f7b2db230" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Play Python on Zeppelin in Docker container]]></title>
            <link>https://zjffdu.medium.com/play-python-on-zeppelin-in-docker-container-b6877bbbfa97?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6877bbbfa97</guid>
            <category><![CDATA[apache-zeppelin]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[notebook]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Thu, 23 Sep 2021 09:03:48 GMT</pubDate>
            <atom:updated>2021-09-23T09:03:48.878Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Zeppelin is web-based notebook that enables data-driven,<br>interactive data analytics and collaborative documents with SQL, Scala, Python, R and more.</p><p>This article is to guide you how to play Python on Zeppelin in docker container without any manual setting.</p><p>First run the following command to start Zeppelin in Docker container:</p><pre>docker run -u <strong>$(</strong>id -u<strong>)</strong> -p 8080:8080 --rm --name zeppelin apache/zeppelin:0.10.0</pre><p>Then open <a href="http://localhost:8080/">http://localhost:8080</a>, you can see the following page. You can also run the above command on another machine, and access Zeppelin remotely via url like <a href="http://my-zeppelin-server:8080/">http://my-zeppelin-server:8080</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l76PP0NBdrElRsJWZVh8yQ.png" /></figure><p>Zeppelin is shipped with some built-in tutorial notes. Open folder Python Tutorial , you can see 4 Python tutorial notes. The first 3 notes can run directly in docker container.</p><ul><li>1. IPython Basics</li><li>2. IPython Visualization Tutorial</li><li>3. Keras Binary Classification (IMDB)</li></ul><p>Here’s one short video which demonstrate how to Play Python in Zeppelin docker container.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKWJvc6l_g1w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKWJvc6l_g1w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKWJvc6l_g1w%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f325ece9d8495f9f5b3921790bbe4034/href">https://medium.com/media/f325ece9d8495f9f5b3921790bbe4034/href</a></iframe><p>Besides these feature, there’re more features in Zeppelin’s Python interpreter, for more details, please check out Zeppelin’s official document. <a href="https://zeppelin.apache.org/docs/0.10.0/interpreter/python.html">https://zeppelin.apache.org/docs/0.10.0/interpreter/python.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6877bbbfa97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use IPython in Apache Zeppelin Notebook]]></title>
            <link>https://zjffdu.medium.com/how-to-use-ipython-in-apache-zeppelin-notebook-c0f4b7ccc4a0?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0f4b7ccc4a0</guid>
            <category><![CDATA[apache-zeppelin]]></category>
            <category><![CDATA[ipython]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[notebook]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Sun, 27 Jun 2021 05:47:02 GMT</pubDate>
            <atom:updated>2021-06-27T05:47:02.250Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="http://zeppelin.apache.org/">Apache Zeppelin</a> notebook is web-based notebook that enables data-driven,<br>interactive data analytics and collaborative documents with SQL, Scala, Python and more.</p><p>In this post, I will talk about how to use IPython in Apache Zeppelin Notebook (Although Zeppelin support vanilla Python, it is strongly recommended to use IPython). This would make the python development experience in Zeppelin notebook almost the same as Jupyter notebook. All the contents in this notebook can be found in these 2 example notebook</p><ul><li><a href="http://zeppelin-notebook.com/#/notebook/2EYDJKFFY">IPython Basic Tutorial</a></li><li><a href="http://zeppelin-notebook.com/#/notebook/2F1S9ZY8Z">IPython Visualization Tutorial</a></li></ul><h3>How to configure</h3><p>Enable IPython in Zeppelin is pretty straightforward, first you need to install the following 3 python packages</p><pre>pip install jupyter<br>pip install grpcio<br>pip install protobuf</pre><p>Besides manually installing these packages, you can also use conda to create a customized python environment which has these installed. You can check <a href="https://zjffdu.medium.com/use-customized-and-isolated-python-environment-in-apache-zeppelin-notebook-afbd7cf5eb4f">this post</a> for more details.</p><p>Then configure Python Interpreter，the most important configuration is zeppelin.python which need to point to the correct python executable in case you have multiple python installed in your machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0j2LwKBOi-pSlKHrf7a0XA.png" /></figure><h3>IPython Features</h3><p>Now, you can experience almost the same python development experience as Jupyter notebook. Here’s a list of feature that I’d like to highlight.</p><ul><li>Support ipython magic function</li><li>Better code completion</li><li>Rich visualization libraries support</li></ul><h4>IPython magic function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xqSuHG09u4pL7kp2VDIjJQ.png" /></figure><h4>Better code completion</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F9ArQnKQ11wkQQWx.gif" /></figure><h4>Rich visualization libraries support</h4><p>Visualization libraries are a big family of python ecosystem. Like jupyter notebook, you can use most of the popular python visualization libraries in Zeppelin notebook.</p><p>Here’re a list of examples of how to use popular python visualization libraries in Zeppelin:</p><ul><li>Matplotlib</li><li>Pandas</li><li>Seaborn</li><li>Plotnine</li><li>Bokeh</li><li>Holoviews</li><li>Altair</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y7eDPkuhnifoeI-EKLQ1Gw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oK7m8fK0XOEFkudju9Qtsg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RztBq4Svc_zb9iBP0S_SIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tGdVKQF_NX-w0DvX-JSzGA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9zinRcwcKseFrhzGIQxBxQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sn0XaBOURKIEEDNE9DtQWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rLJLq4tWRv1ICGK5ugHGqA.png" /></figure><h3>Summary</h3><p>This is just a brief introduction of how to use IPython in Zeppelin notebook. If you have any question, you can ask in zeppelin user mail list or slack channel (<a href="http://zeppelin.apache.org/community.html">http://zeppelin.apache.org/community.html</a>)</p><h3>References</h3><p>[1] Apache Zeppelin <a href="http://zeppelin.apache.org/">http://zeppelin.apache.org/</a></p><p>[2] Zeppelin notebooks website <a href="http://zeppelin-notebook.com/">http://zeppelin-notebook.com/</a>.</p><p>[3] Zeppelin notebooks git repo <a href="https://github.com/zjffdu/zeppelin-notebook">https://github.com/zjffdu/zeppelin-notebook</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0f4b7ccc4a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use customized and isolated python environment in Apache Zeppelin notebook]]></title>
            <link>https://zjffdu.medium.com/use-customized-and-isolated-python-environment-in-apache-zeppelin-notebook-afbd7cf5eb4f?source=rss-4a87d907e2ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/afbd7cf5eb4f</guid>
            <category><![CDATA[apache-zeppelin]]></category>
            <category><![CDATA[notebook]]></category>
            <category><![CDATA[ipython]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[conda]]></category>
            <dc:creator><![CDATA[Jeff Zhang]]></dc:creator>
            <pubDate>Mon, 21 Jun 2021 14:57:53 GMT</pubDate>
            <atom:updated>2021-06-27T13:57:14.359Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="http://zeppelin.apache.org/">Apache Zeppelin</a> notebook is web-based notebook that enables data-driven,<br>interactive data analytics and collaborative documents with SQL, Scala, Python and more.</p><p>For Python developers, using a customized and isolated Python runtime environment is an indispensable requirement. You and your colleagues may want to use different versions of python and python packages and don’t want to affect each others’ environment. In this article, I’d like to introduce to you how to use customized and isolated Python environment in hadoop yarn cluster. Regarding how to achieve this for PySpark, I will leave it to another article. (All the features in this article is done in this jira <a href="https://issues.apache.org/jira/browse/ZEPPELIN-5330">ZEPPELIN-5330</a>). And you can reproduce all the steps here via downloading this note:</p><ul><li><a href="http://zeppelin-notebook.com/#/notebook/2G7RDR415">Python Conda Env in Yarn Mode</a></li></ul><h3>Step 1. Create your customized conda env</h3><p>First, let’s create a yaml file which define the python conda env including:</p><ul><li>env name</li><li>channels which is used to install packages</li><li>python version</li><li>other third party python packages</li></ul><pre>name: python_env<br>channels:<br>  - conda-forge<br>  - defaults<br>dependencies:<br>  - python=3.7 <br>  - pycodestyle<br>  - numpy<br>  - pandas<br>  - scipy<br>  - grpcio<br>  - protobuf<br>  - pandasql<br>  - ipython<br>  - ipykernel<br>  - jupyter_client<br>  - panel<br>  - pyyaml<br>  - seaborn<br>  - plotnine<br>  - hvplot<br>  - intake<br>  - intake-parquet<br>  - intake-xarray<br>  - altair<br>  - vega_datasets<br>  - pyarrow&quot;</pre><p>Then run the using the following commands to create conda env tar and upload it to hdfs.</p><pre>conda pack -n python_env<br>hadoop fs -put python_env.tar.gz /tmp<br># The python conda tar should be publicly accessible, so need to change permission here.<br>hadoop fs -chmod 644 /tmp/pyspark_env.tar.gz</pre><h3>Step 2. Configure Python Interpreter</h3><pre>%python.conf</pre><pre># set zeppelin.interpreter.launcher to be yarn, so that python interpreter run in yarn container, <br># otherwise python interpreter run as local process in the zeppelin server host.<br>zeppelin.interpreter.launcher yarn</pre><pre># zeppelin.yarn.dist.archives can be either local file or hdfs file<br>zeppelin.yarn.dist.archives hdfs:///tmp/python_env.tar.gz#environment</pre><pre># conda environment name, aka the folder name in the working directory of yarn container<br>zeppelin.interpreter.conda.env.name environment</pre><h3>Step 3. Run Python Interpreter in this customized conda env</h3><pre>%python</pre><pre>%matplotlib inline</pre><pre>import matplotlib.pyplot as plt</pre><pre>plt.plot([1,2,3,4])<br>plt.ylabel(&#39;some numbers&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qYW8yu8yA-8xznBlmDlg1w.png" /></figure><h3>Summary</h3><p>The this feature is not released yet when this article is published, you can build Zeppelin master branch by yourself and import <a href="http://zeppelin-notebook.com/#/notebook/2G7RDR415">this note here</a> to try this feature. If you have any question, you can ask in zeppelin user mail list or slack channel (<a href="http://zeppelin.apache.org/community.html">http://zeppelin.apache.org/community.html</a>)</p><h3>References</h3><p>[1] Apache Zeppelin <a href="http://zeppelin.apache.org/">http://zeppelin.apache.org/</a></p><p>[2] Zeppelin notebooks website <a href="http://zeppelin-notebook.com/">http://zeppelin-notebook.com/</a>.</p><p>[3] Zeppelin notebooks git repo <a href="https://github.com/zjffdu/zeppelin-notebook">https://github.com/zjffdu/zeppelin-notebook</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afbd7cf5eb4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>