<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marcellinus Aditya Witarsah on Medium]]></title>
        <description><![CDATA[Stories by Marcellinus Aditya Witarsah on Medium]]></description>
        <link>https://medium.com/@aw_marcell?source=rss-2856fdb1b852------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*apPCFQ5vGUjEBtwTjzLPqg.png</url>
            <title>Stories by Marcellinus Aditya Witarsah on Medium</title>
            <link>https://medium.com/@aw_marcell?source=rss-2856fdb1b852------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:35:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aw_marcell/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Indonesia Financial Market Data Platform, Part 2: Implementation]]></title>
            <link>https://medium.com/@aw_marcell/indonesia-financial-market-data-platform-part-2-implementation-3d8943a2f019?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d8943a2f019</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[medallion-architecture]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Sat, 21 Feb 2026 07:44:25 GMT</pubDate>
            <atom:updated>2026-02-21T07:44:25.141Z</atom:updated>
            <content:encoded><![CDATA[<p>Previously, I talked about the scope and data architecture that will be used for this project.</p><p><a href="https://medium.com/@aw_marcell/stock-market-data-platform-project-1-279ffded8e99">Stock Market Data Platform Project, Part 1: Scope &amp; Data Architecture Design</a></p><p>In this blog, I will share <strong>how I implemented it.</strong></p><h3>Project Folder and Code Structure</h3><p>In this section, I will show the project folder structure, which consists of:</p><ol><li>infrastructures: contains all of the Dockerfiles and configurations needed for setting up data lakehouse locally. See this <a href="https://medium.com/@aw_marcell/how-to-build-your-own-local-data-lakehouse-using-open-source-tools-c5a8626d10f7">blog post</a> to know how it works.</li><li>pipelines: contains all of the data pipelines, broken down into three folders based on the medallion architecture.</li><li>scripts : contains scripts used in this project for example SQL script for creating schema and tables.</li><li>src/utils : contains generanl functions that is commonly used in this project, such as iceberg table operations and loggers.</li><li>docker-compose.yml : YAML file configuration for deploying multiple containers.</li><li>requirements.txt : contains Python libraries used in this project.</li></ol><pre>indonesia-capital-market-data-platform/<br>│<br>├── infrastructures/<br>│<br>├── pipelines/<br>│   ├── bronze/<br>│   │   ├── market_data_1m.py<br>│   │   └── ticker_info.py<br>│   │<br>│   ├── silver/<br>│   │   ├── ticker_ohlcv_1m.py<br>│   │   └── ticker_profile.py<br>│   │<br>│   └── gold/<br>│       └── ticker_daily_metrics.py<br>├── scripts/<br>│   ├── create_schema_and_tables.sql<br>├── src/<br>│   ├── utils/<br>│   │   ├── common_functions.py<br>│   │   ├── iceberg_table_operations.py<br>│   │   └── logger.py<br>├── .env.example<br>├── .gitignore<br>├── docker-compose.yml<br>└── requirements.txt</pre><p>On top of that, I would like to share with code structure that I used for all data pipelines. It can be broken down to three parts:</p><ol><li>Data Load</li><li>Data Transformation</li><li>Data Insertion</li></ol><p>In all of the .py files under pipelines folder, you will find this specific structure.</p><h3>Bronze Layer Pipelines</h3><p>The main idea is simple <strong>bring data as-is from the source</strong>.<strong> </strong>For this project, the source is <a href="https://pypi.org/project/yfinance/">yfinance</a> and I extracted two data:</p><ol><li><a href="https://ranaroussi.github.io/yfinance/reference/api/yfinance.Ticker.history.html">Open High Low Close Volume (OHLCV)</a> in form of <em>Pandas DataFrame</em>.</li><li><a href="https://ranaroussi.github.io/yfinance/reference/api/yfinance.Ticker.info.html">Company profile information</a> in form of JSON string.</li></ol><p>The table structures will follow whatever the structure from the source, with two additional metadata columns load datetime and load partition date:</p><ol><li><strong>load datetime</strong> is timestamp when the record is inserted into the bronze layer table. Example: 2026–02–07 14:31:24.686509 UTC.</li><li><strong>load partition date</strong> is date when the record is inserted into the bronze layer table. Example: 2026–02–07.</li></ol><p>I just want to remind that we two data types: <strong><em>Pandas DataFrame</em></strong> and <strong><em>JSON string</em></strong>.</p><p>If the data is already in form of a table like <em>Pandas DataFrame</em>, we only need to convert it to Spark DataFrame and then add metadata columns.</p><pre># -----------------------------------------------------------<br># From pipelines/bronze/market_data_1m.py<br># Get data form source<br># -----------------------------------------------------------<br>data = pd.DataFrame()<br>for ticker in tickers:<br>    stock_data = (<br>        yf.Ticker(ticker)<br>        .history(period=period, interval=interval)<br>        .reset_index()<br>    )<br>    stock_data[&quot;Ticker&quot;] = ticker<br>    data = pd.concat([data, stock_data], ignore_index=True)<br>logger.info(&quot;Data retrieved from API successfully.&quot;)<br><br># -----------------------------------------------------------<br># Create Spark Dataframe<br># -----------------------------------------------------------<br>df = spark.createDataFrame(data)<br>df = df.withColumn(&quot;load_dttm&quot;, lit(datetime.now()).cast(TimestampType()))<br>df = df.withColumn(&quot;load_prdt&quot;, lit(datetime.now().date()).cast(DateType()))<br>logger.info(&quot;Spark Table Create Successfully.&quot;)</pre><p>But what if the data is in form of a JSON String. For this project I created my own table that has two columns:</p><ol><li>ticker (for the ticker name)</li><li>info (JSON string)</li></ol><p>Then I add the metadata columns.</p><pre># -----------------------------------------------------------<br># From pipelines/bronze/ticker_info.py<br># Define Spark Table Schema<br># -----------------------------------------------------------<br>schema = StructType(<br>    [<br>        StructField(&quot;ticker&quot;, StringType(), False),<br>        StructField(&quot;info&quot;, StringType(), True),<br>    ]<br>)<br><br># # -----------------------------------------------------------<br># # Get data form source<br># # -----------------------------------------------------------<br>data = []<br>for ticker in tickers:<br>    data.append(<br>        (<br>            ticker,<br>            json.dumps(yf.Ticker(ticker).info),<br>        )<br>    )<br>logger.info(&quot;Data retrieved from API successfully.&quot;)<br><br># -----------------------------------------------------------<br># Create Dataframe<br># -----------------------------------------------------------<br>df = spark.createDataFrame(data=data, schema=schema)<br>df = df.withColumn(&quot;load_dttm&quot;, lit(datetime.now()).cast(TimestampType()))<br>df = df.withColumn(&quot;load_prdt&quot;, lit(datetime.now().date()).cast(DateType()))<br>logger.info(&quot;Spark DataFrame created successfully.&quot;)</pre><p>After this step, all of the data are in form of Spark DataFrame. The next step is inserting it into bronze layer tables. In this layer, I simply append all records as-is because the bronze layer acts as a <strong>proxy of the source system’s raw data</strong>.</p><pre># -----------------------------------------------------------<br># Write to Iceberg Table<br># -----------------------------------------------------------<br>df.write.format(&quot;iceberg&quot;).partitionBy(&quot;load_prdt&quot;).mode(&quot;append&quot;).option(<br>    &quot;mergeSchema&quot;, &quot;true&quot;<br>).saveAsTable(<br>    f&quot;{os.getenv(&#39;CATALOG_NAME&#39;)}.bronze.{__file__.split(&#39;/&#39;)[-1].split(&#39;.&#39;)[0]}&quot;<br>)</pre><h3>Silver Layer Pipelines</h3><p>In the <strong>silver layer</strong>, the goal is to make the data <strong>clean and standardized</strong>. All silver tables perform:</p><ol><li>Data cleaning: remove duplicates, extra spaces, and replace empty string with NULL.</li><li>Data standardization: enforce the columns related to prices, dividends, and volumes to be FLOAT data type. Columns related to time must be in form of TIMESTAMP data type. Columns related to date must be in a DATE data type.</li></ol><p>For me, silver layer tables should maintain the <strong>same level of granularity as the raw tables</strong>, but in a cleaned and standardized format.</p><p>Here’s the implementation on the project</p><pre># -----------------------------------------------------------<br># From pipelines/silver/ticker_ohlcv_1m.py<br># Data Transformation<br># -----------------------------------------------------------<br>df = (<br>    df.alias(&quot;market_data&quot;)<br>    .select(<br>        col(&quot;market_data.Ticker&quot;).astype(StringType()).alias(&quot;ticker&quot;),<br>        col(&quot;market_data.Datetime&quot;).astype(TimestampType()).alias(&quot;datetime&quot;),<br>        col(&quot;market_data.Open&quot;).astype(FloatType()).alias(&quot;open&quot;),<br>        col(&quot;market_data.High&quot;).astype(FloatType()).alias(&quot;high&quot;),<br>        col(&quot;market_data.Low&quot;).astype(FloatType()).alias(&quot;low&quot;),<br>        col(&quot;market_data.Close&quot;).astype(FloatType()).alias(&quot;close&quot;),<br>        col(&quot;market_data.Volume&quot;).astype(FloatType()).alias(&quot;volume&quot;),<br>        col(&quot;market_data.Dividends&quot;).astype(FloatType()).alias(&quot;dividends&quot;),<br>        col(&quot;market_data.`Stock Splits`&quot;)<br>        .astype(FloatType())<br>        .alias(&quot;stock_splits&quot;),<br>    )<br>    .distinct()<br>)<br>df = df.withColumn(&quot;load_dttm&quot;, lit(datetime.now()).cast(TimestampType()))<br>logger.info(f&quot;Data transformation completed.&quot;)</pre><pre># -----------------------------------------------------------<br># From pipelines/silver/ticker_profile.py<br># Data Transformation<br># -----------------------------------------------------------<br>df = (<br>    df.alias(&quot;ticker_info&quot;)<br>    .withColumn(&quot;parsed_info&quot;, from_json(col(&quot;ticker_info.info&quot;), schema))<br>    .select(<br>        string_cleanser(col(&quot;ticker_info.ticker&quot;))<br>        .astype(StringType())<br>        .alias(&quot;ticker&quot;),<br>        string_cleanser(col(&quot;parsed_info.longName&quot;))<br>        .astype(StringType())<br>        .alias(&quot;company_name&quot;),<br>        string_cleanser(col(&quot;parsed_info.longBusinessSummary&quot;))<br>        .astype(StringType())<br>        .alias(&quot;business_summary&quot;),<br>        string_cleanser(col(&quot;parsed_info.currency&quot;))<br>        .astype(StringType())<br>        .alias(&quot;currency&quot;),<br>        col(&quot;parsed_info.marketCap&quot;).astype(FloatType()).alias(&quot;market_cap&quot;),<br>        string_cleanser(col(&quot;parsed_info.industry&quot;))<br>        .astype(StringType())<br>        .alias(&quot;industry&quot;),<br>    )<br>    .distinct()<br>)<br>df = df.withColumn(&quot;load_dttm&quot;, lit(datetime.now()).cast(TimestampType()))<br>logger.info(&quot;Data transformation completed.&quot;)</pre><h3>Gold Layer Pipelines</h3><p>The main idea of the gold layer tables is to specifically process silver layer tables for a specific business use case.</p><p>There are no strict limitations as long as the transformation help answer business problems.</p><p>For example in this project, we don’t need to maintain granularity because the metrics that we want to calculate are daily metrics. This means that the data must be aggregated into 1-day interval.</p><p>Here are some of the example metrics I computed for this project.</p><h4>Daily Returns</h4><p>Daily return measures the <strong>percentage change between the opening price and the closing price of a stock on the same day</strong>.</p><p>The return can be positive or negative:</p><ul><li>Positive → closing price is higher than opening price</li><li>Negative → closing price is lower than opening price</li></ul><p>Here’s the formula</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fmath.embed.fun%2FhHPacikFnCDi1e3cQuh8fU&amp;type=text%2Fhtml&amp;schema=embed&amp;display_name=Embed+Fun&amp;src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FhHPacikFnCDi1e3cQuh8fU" width="200" height="96" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fef1cbeb971902cbae1941e263a30ca/href">https://medium.com/media/7fef1cbeb971902cbae1941e263a30ca/href</a></iframe><p>Steps:</p><ol><li>Create a column called datederived from the datetime column that will be used as a grouping key.</li><li>Since the data is in a 1-minute intervals, we need to get the first open price and the last close price.</li><li>Use row_number() spark sql function partitioned by <strong>ticker</strong> and <strong>date</strong>, ordered by <strong>datetime</strong>.</li><li>Join two of those tables using ticker and date column as the join key.</li><li>Calculate the daily returns.</li></ol><p>Here’t the code snippet</p><pre>...<br><br># -----------------------------------------------------------<br># Data Load<br># -----------------------------------------------------------<br>ticker_ohlcv_1m_df = spark.table(<br>    f&quot;{os.getenv(&#39;CATALOG_NAME&#39;)}.silver.ticker_ohlcv_1m&quot;<br>).filter(col(&quot;datetime&quot;).between(start_date, end_date))<br>ticker_ohlcv_1m_df = ticker_ohlcv_1m_df.withColumn(<br>    &quot;date&quot;, to_date(col(&quot;datetime&quot;))<br>)<br>logger.info(&quot;Spark Table loaded successfully.&quot;)<br><br># -----------------------------------------------------------<br># Data Transformation<br># -----------------------------------------------------------<br># 1. Daily Returns: percentage change between the closing price and opening price of the day<br>df_daily_first_record = iceberg_table_ops.get_partitioned_row_number(<br>    ticker_ohlcv_1m_df, [&quot;ticker&quot;, &quot;date&quot;], [&quot;datetime&quot;], order=&quot;asc&quot;<br>).filter(col(&quot;row_number&quot;) == 1)<br>df_daily_last_record = iceberg_table_ops.get_partitioned_row_number(<br>    ticker_ohlcv_1m_df, [&quot;ticker&quot;, &quot;date&quot;], [&quot;datetime&quot;], order=&quot;desc&quot;<br>).filter(col(&quot;row_number&quot;) == 1)<br><br>df_daily_returns = (<br>    df_daily_last_record.alias(&quot;daily_last_record&quot;)<br>    .join(<br>        df_daily_first_record.alias(&quot;daily_first_record&quot;),<br>        (col(&quot;daily_last_record.ticker&quot;) == col(&quot;daily_first_record.ticker&quot;))<br>        &amp; (col(&quot;daily_last_record.date&quot;) == col(&quot;daily_first_record.date&quot;)),<br>        &quot;inner&quot;,<br>    )<br>    .select(<br>        col(&quot;daily_last_record.ticker&quot;).alias(&quot;ticker&quot;),<br>        col(&quot;daily_last_record.date&quot;).alias(&quot;date&quot;),<br>        (<br>            (col(&quot;daily_last_record.close&quot;) - col(&quot;daily_first_record.open&quot;))<br>            / col(&quot;daily_first_record.open&quot;)<br>        ).alias(&quot;daily_returns&quot;),<br>    )<br>)<br><br>...</pre><h4>Daily Volatility</h4><p>Daily volatility measures <strong>how much the stock price fluctuates around the average price within a day</strong>.</p><p>For example:<br>If volatility is <strong>500</strong> and the average price is <strong>5000</strong>, it means the price typically moves around that range during the day.</p><p>This uses the standard deviation formula</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fmath.embed.fun%2Fe2TYkmvM6RCxQbc7ZVChMD&amp;type=text%2Fhtml&amp;schema=embed&amp;display_name=Embed+Fun&amp;src=https%3A%2F%2Fmath.embed.fun%2Fembed%2Fe2TYkmvM6RCxQbc7ZVChMD" width="298" height="207" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f447ef98faaa9d329a585084492fbe0/href">https://medium.com/media/2f447ef98faaa9d329a585084492fbe0/href</a></iframe><p>Steps:</p><ol><li>Calculate the average close price for each ticker per day.</li><li>Compute the <strong>squared differences</strong> between stock prices and its average price.</li><li>Aggregate sum the squared differences.</li><li>Calculate the daily volatility.</li></ol><p>Here’t the code snippet</p><pre>...<br><br># 2. Daily Volatility: standard deviation of the closing prices throughout the day<br>df_daily_average_price_by_ticker_date = (<br>    ticker_ohlcv_1m_df.alias(&quot;ticker_ohlcv_1m&quot;)<br>    .groupBy(col(&quot;ticker_ohlcv_1m.ticker&quot;), col(&quot;date&quot;))<br>    .agg(<br>        avg(col(&quot;ticker_ohlcv_1m.close&quot;)).alias(&quot;avg_close&quot;),<br>    )<br>)<br>df_sse = (<br>    ticker_ohlcv_1m_df.alias(&quot;ticker_ohlcv_1m&quot;)<br>    .join(<br>        df_daily_average_price_by_ticker_date.alias(<br>            &quot;df_daily_average_price_by_ticker_date&quot;<br>        ),<br>        (<br>            col(&quot;ticker_ohlcv_1m.ticker&quot;)<br>            == col(&quot;df_daily_average_price_by_ticker_date.ticker&quot;)<br>        )<br>        &amp; (<br>            col(&quot;ticker_ohlcv_1m.date&quot;)<br>            == col(&quot;df_daily_average_price_by_ticker_date.date&quot;)<br>        ),<br>        &quot;left&quot;,<br>    )<br>    .withColumn(<br>        &quot;squared_differences&quot;,<br>        pow(<br>            col(&quot;ticker_ohlcv_1m.close&quot;)<br>            - col(&quot;df_daily_average_price_by_ticker_date.avg_close&quot;),<br>            2,<br>        ),<br>    )<br>    .groupBy(col(&quot;ticker_ohlcv_1m.ticker&quot;), col(&quot;ticker_ohlcv_1m.date&quot;))<br>    .agg(<br>        sum(col(&quot;squared_differences&quot;)).alias(&quot;sum_squared_errors&quot;),<br>        count(col(&quot;ticker_ohlcv_1m.ticker&quot;)).alias(&quot;n&quot;),<br>    )<br>)<br>df_daily_volatility = df_sse.select(<br>    col(&quot;ticker&quot;),<br>    col(&quot;date&quot;),<br>    sqrt(col(&quot;sum_squared_errors&quot;) / col(&quot;n&quot;)).alias(&quot;daily_volatility&quot;),<br>)<br><br>...</pre><h3>Utility Functions</h3><h4>Common Functions</h4><p>This is where I put a general function that is used accross multiple pipelines. Currently, there’s only one function that I created which is a string_cleanser() that:</p><ol><li>removes extra white space at beginning and end of a string</li><li>converts empty string into <strong>NULL</strong> values</li></ol><h4>Iceberg Table Operations Functions</h4><p>This contains iceberg table functions that I frequently used in data pipelines. There are two functions that I commonly used:</p><ol><li>get_partitioned_row_number() : performs a row number operation based on partition keys and ordering keys.</li><li>get_latest_record() : retrieves the latest record based on partition keys and ordering keys. It can also filter records by date. This is useful when source systems are prone to duplicate records.</li></ol><h4>Logger Functions</h4><p>Logger is a very important because it allows you to format output in a consistent way.</p><h3>GitHub Link</h3><p><a href="https://github.com/marcellinus-witarsah/indonesia-capital-market-data-platform">GitHub - marcellinus-witarsah/indonesia-capital-market-data-platform</a></p><h3>Final Words</h3><p>These implementations are mainly focused on demonstrating how the <strong>Medallion Architecture</strong> can be applied in a real project. However, there are still many areas that can be improved in the future, such as:</p><ul><li>Adding an orchestrator like Airflow to manage and schedule the pipelines.</li><li>Adding unit tests to ensure the reliability of each data pipeline.</li><li>Implementing data quality checks to validate the data before it moves to the next layer.</li><li>Setting up CI/CD pipelines for automated testing and deployment.</li><li>Improving monitoring and logging for better observability.</li><li>Optimizing performance as the data volume grows.</li></ul><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d8943a2f019" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Built a Data Pipeline — Then I Forgot Why It Existed]]></title>
            <link>https://medium.com/@aw_marcell/i-built-a-data-pipeline-then-i-forgot-why-it-existed-9dc6a1ddc543?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/9dc6a1ddc543</guid>
            <category><![CDATA[experience]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[building-data-pipelines]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Thu, 05 Feb 2026 15:47:27 GMT</pubDate>
            <atom:updated>2026-02-05T15:47:27.802Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>I Built a Data Pipeline — Then I Forgot Why It Existed</strong></h3><p>Have you ever experienced this?</p><ul><li>Someone asks you about a data pipeline you built, but you can’t remember the details.</li><li>You read your own code and wonder, <em>“Why did I build it this way?”</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*skTLOvPPjfrW59VztuKMRA.png" /><figcaption>Image Generated by ChatGPT</figcaption></figure><p>Yeah — I’ve been there too.</p><p>And honestly, it’s such a waste of time. I had to drop whatever I was working on just to review my own data pipeline again. The reason was simple: <strong>I was too focused on delivering results</strong>. I jumped straight into coding without really thinking about the table design, or without asking enough questions at the beginning.</p><p>This kind of habit causes problems later on:</p><ul><li>It takes longer to fix bugs because you’ve forgotten what was built and why.</li><li>When someone challenges your data pipeline, you struggle to explain your decisions.</li><li>Other developers lose productivity when they want to improve, or update your pipeline.</li></ul><h3>How do I solve this issue?</h3><p>I did the thing most developers are lazy to do — <strong>documentation</strong>.</p><p>I know, it sounds boring. But I can guarantee that this will save you a lot of time in the future. It doesn’t have to be fancy. What worked for me was just a simple table with six columns:</p><ul><li>Target Table</li><li>Target Column</li><li>Source Table</li><li>Source Column</li><li>Transformation</li><li>Reason</li></ul><p>That’s it.</p><p>I’ll show an example of the documentation template I use — just a simple Excel file — and how I fill it in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VWfXk5xyUwCA-psfiX8sCw.png" /><figcaption>An Example of Excel Table Containing The Table Columns and How to Fill it.</figcaption></figure><h3>How It Impacted My Work</h3><p>There are 3 positive impacts that I felt for until know:</p><ul><li>I feel more confident in discussions, especially when someone questions my data pipeline.</li><li>I get more clarity on what needs to be done before jumping into coding. It also speeds up the development process because you don’t need to juggle between coding and figuring out how to fill the target columns.</li><li>I can switch context between pipelines much faster, especially when something breaks in production.</li></ul><h3>Final Words</h3><p>It took me quite some time to realize this but it’s okay, it is part of the learning process. But it would be a shame not to share it.</p><p>In the end, data pipelines are decisions not just a code. A code is used for machines to execute, but documentation gives contexts and freezes your decisions. I hoped this article will help my fellow data engineers who experience the same thing.</p><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dc6a1ddc543" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stock Market Data Platform Project, Part 1: Scope & Data Architecture Design]]></title>
            <link>https://medium.com/@aw_marcell/stock-market-data-platform-project-1-279ffded8e99?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/279ffded8e99</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[medallion-architecture]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Sun, 25 Jan 2026 14:02:37 GMT</pubDate>
            <atom:updated>2026-02-14T05:41:39.925Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>Inside every investment firm, trading firm, hedge fund, and similar organization, there is a data platform that stores data for various needs. In financial markets, firms that have better access to data tend to gain an advantage over others. This is why firms invest significant amounts of money in building data platforms. This data platform can be used by for reporting, analyzing, and testing their strategies for making money in the stock market.</p><h3>Scope</h3><p>The scope of this project includes:</p><ul><li>Ingesting historical <strong>OHLCV data for Top 10 Indonesian Stocks by market capital </strong>from <a href="https://pypi.org/project/yfinance/">yfinance</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7VHdbZS7Z9BwjB3LE0YnA.png" /><figcaption>Figure 1: Top 10 Indonesian Stocks by Market Capital per December 2025 (Source: <a href="https://idx.co.id/id/data-pasar/laporan-statistik/digital-statistic/monthly/biggest-market-capitalization-most-active-stocks/biggest-market-capitalization?filter=eyJ5ZWFyIjoiMjAyMyIsIm1vbnRoIjoiMyIsInF1YXJ0ZXIiOjAsInR5cGUiOiJtb250aGx5In0%3D">https://idx.co.id/id/data-pasar/laporan-statistik/digital-statistic/monthly/biggest-market-capitalization-most-active-stocks/biggest-market-capitalization?filter=eyJ5ZWFyIjoiMjAyMyIsIm1vbnRoIjoiMyIsInF1YXJ0ZXIiOjAsInR5cGUiOiJtb250aGx5In0%3D</a>).</figcaption></figure><ul><li>Users will be <strong>stock</strong> <strong>daily returns</strong>, and <strong>stock daily volatility</strong>.</li></ul><p>I intentionally keep the scope as small because to focus on learning and demonstrating how data flows through the architecture.</p><h3>Data Architecture</h3><p>The data architecture used in this project is data lakehouse, which combines elements data lake and data warehouse. Figure 2 shows the architecture diagram for this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*XYtr5llCe8iak8izUZug2w.png" /><figcaption>Figure 2: Data Lakehouse Architecture Diagram (Image by Author).</figcaption></figure><p>This architecture consists of:</p><ol><li>Apache Gravitino as data catalog.</li><li>MinIO as data lake.</li><li>Spark as distributed compute engine.</li><li>Apache Iceberg as the open table format.</li></ol><p>The tech stack is similar with the one I already written which you can visit it below.</p><p><a href="https://medium.com/@aw_marcell/how-to-build-your-own-local-data-lakehouse-using-open-source-tools-c5a8626d10f7">How to Build Your Own Local Data Lakehouse using Open Source Tools</a></p><h3>Data Design Pattern</h3><p>I chose the <strong>medallion architecture,</strong> a data design pattern that organizes data into multiple layers where data is progressively processed and enriched as it movees through the layer.</p><p>Those layers are:</p><ol><li>Bronze layer that receives and stores data from external source systems as it is. This provides historical archive that can be revisited for various purpose without fetching again from the source.</li><li>Silver layer that performs data cleaning and standardization. This provide data that is already deduplicated, cleaned, and schema conformed while keeping its original granularity from the raw layer.</li><li>Gold layer that further process data from silver layer for specific use cases and ready to consume by business users.</li></ol><p>The reason why I chose this data design patterns is because it gives me flexibility of creating new tables in the future due to data from systems are kept in bronze layer. It also gives me indempotency if I want to rerun a data pipeline.</p><h3>Data Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*3huqWcayGBFFMpmRvS1BjQ.png" /><figcaption>Figure 3: Data Flow through The Medallion Architecture.</figcaption></figure><p>The data flow is self-explanatory, as illustrated in Figure 3.</p><ol><li>Source to Bronze: Data is ingested from the source system and stored as it is.</li><li>Bronze to Silver: The data is cleaned, deduplicated, and conformed based on the schema design.</li><li>Silver to Gold: The data will be aggregated for specific use cases.</li></ol><h3>Data Model</h3><p>I only create a data model for silver layer and gold layer tables because these layers are processed internally, while the bronze layer stores data exactly as it is. These data model design that I created are based on the data that I will use from yfinance, which are:</p><ol><li><a href="https://ranaroussi.github.io/yfinance/reference/api/yfinance.Ticker.html">Company info</a></li><li><a href="https://ranaroussi.github.io/yfinance/reference/api/yfinance.Ticker.history.html">Stock OHLCV data</a></li></ol><h4>Silver Layer Data Model</h4><p>Here is the data model for the silver layer which is shown in Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5lCiN2ZY7voHWcR2Y44Tw.png" /><figcaption>Figure 4: Silver layer data model</figcaption></figure><h4>Gold Layer Data Model</h4><p>Here is the data model for the silver layer which is shown in Figure 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*omAf-LPKiWBGpfd0i7M-EQ.png" /><figcaption>Figure 5: Silver layer data model</figcaption></figure><h3>Final Words</h3><p>You might feel odd that I am publishing a plan for my own data engineering project. Honestly, there are two reasons:</p><ol><li>It gives me a clear path and end goal of the project.</li><li>It keeps me focus on achieving the end goal while minimizing scope creep which I learned this a lot from my workplace.</li></ol><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=279ffded8e99" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Here’s What I Learned about “Data Products” in Financial Market]]></title>
            <link>https://medium.com/@aw_marcell/heres-what-i-learned-about-data-products-in-financial-market-c61992447d7c?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/c61992447d7c</guid>
            <category><![CDATA[financial-markets]]></category>
            <category><![CDATA[market-data]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Sun, 18 Jan 2026 04:30:10 GMT</pubDate>
            <atom:updated>2026-01-18T04:30:10.215Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IkeQM-AjuDbDk6K167ZbXg.png" /><figcaption>Image Generated by ChatGPT</figcaption></figure><p>Naively, I thought financial market data products are bunch of prices with timestamp. Turns out, it is much more than that and through this article I want to share with you guys what are some of the data products and how I think we can leverage these data.</p><p>Before we jump in, some of you who might still newbie like me — my own definition of data product it is just a curated data by data provider or vendors for external consumers (like you and me).</p><p>There are so many data products that I found in the internet. But don’t be afraid, I will simplify it for you by categorizing them into four categories:</p><ul><li>Trade data,</li><li>News data,</li><li>Economic data, and</li><li>Company data.</li></ul><h3>Trade Data</h3><p>By definition, it is a data product containing about trade transactions which varies from price, quantity, datetime, action, currency, asset being traded, bid and ask, etc. These are all collected from financial market trading activities happened inside exchanges.</p><h4>Quote data</h4><p>It is a data containing the most recent information of a financial instrument such current price, bid and ask price, volume, etc. We can use this information to know:</p><ol><li>The price of people that are willing to buy (<strong>bid price</strong>) and willing to sell (<strong>ask price</strong>).</li><li>The latest traded price of a financial instrument (<strong>current price</strong>).</li><li>The latest traded volume of financial instrument (<strong>volume</strong>).</li></ol><h4>Order Book</h4><p>It is a collection of bid and ask orders (price and volume) at multiple price points. In simple terms it is a list of order prices and quantity where people willing to buy or sell. The bid prices will be in <strong>descending</strong> order while the ask prices will be in <strong>ascending</strong> order — just like shown in Figure 1. This is logical because in every type of market, people are willing to buy at a current price or lower than that, same as selling — people want to sell at a higher price to maximize gains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omHon0-c9ArrPj4In3_ACA.png" /><figcaption>Figure 1: Example of What an Order Book Looks Like (Image Generated by ChatGPT).</figcaption></figure><p>Order book usually contains:</p><ol><li>Prices, and quantity of people who perform a bid order.</li><li>Prices, and quantity of people who perform a ask order.</li><li>History of trades that already happened.</li></ol><p>Order Book usually has two levels:</p><ol><li><strong>L1 Data</strong> shows the best bid and ask price currently. This information can be used to set a price of a trade. If you are passive buying you can set at the best bid price, but if you are aggresive buying then set it to best ask price.</li><li><strong>L2 Data </strong>contains bid and offer prices at each price points (just like shown in Figure 1). This information can be used to determine market depth. Market depth shows how well a financial instrument can withstand large transaction without moving the price significantly. This information usually are used for entities who usually enters with large amount of trades.</li></ol><h4>Tick</h4><p>It is an information of a specific trade usually contains price, quantity, financial instrument name, timestamp, trade id, etc. The lowest granular of market data.</p><h4>OHLCV</h4><p>It is a data contain open, high, low, close price of a financial instrument within specified period. This is draw a candlestick chart which is mainly used for technical analysis.</p><h3>News Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GqZTmmE3jW5bTK4Y8MhtRQ.png" /><figcaption>Figure 2: News Can Sometimes Influence Market Moves (Image Generated Using ChatGPT).</figcaption></figure><p>It can be any news that relate or influence a move on financial instruments that can be used as an input for financial decisions to take inside the market. This can be used to know about the market feels about a specific financial instruments. Good news might infer market interests on a specific financial instrument.</p><h3>Economic Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UUkw6FA-Zw80Ut5sBy-_gQ.png" /><figcaption>Figure 3: Example of Economic Data (Image Generated by ChatGPT).</figcaption></figure><p>Usually refers to macroeconomy data that gives an overall view of how the economy is doing as a whole — usually at a country level. Most of the data are produced by <strong>government</strong> and <strong>non-profit organization</strong> or even <strong>universities</strong>. This is good for neutrality — if a hedge fund publish an economic data, public will be skeptical of possible manipulation that will benefit financial instruments that they are currently holding.</p><p>Some of the most famous economic indicator are:</p><h4>Gross Domestic Product (GDP)</h4><p>It measures the market value of goods and services produced inside a country. It is calculated by summing consumer spendings, government spendings, investments, and net exports. This information can be used to know whether the country economy is growing because you sure don’t want to invest into a company where the country itself is not doing well economically.</p><h4>Nonfarm Payroll Report (NFP)</h4><p>It measures how well a country’s citizens having a job and how much is getting paid (excluding farmers, government workers, self-employeed workers, and workers for non-profit organization). It gives you a view whether its citizens are doing well financiall which gives you a hint whether the economy will be growing or not. If more people are getting paid, the economy will grow due to spending.</p><h4>Consumer Price Index (CPI)</h4><p>It is an average cost of goods and services that people have to pay. The CPI measures the average cost of — <strong>food and beverages</strong>, <strong>housing</strong>, <strong>transportation</strong>, <strong>apparel</strong>, <strong>medical care</strong>, <strong>education</strong>, <strong>recreation</strong>, <strong>communication</strong> , and <strong>services</strong>. This information can be used to track inflation and predict what the Central Bank going to do with interest rates.</p><h4>Money Supply</h4><p>It is about how much money existed inside the financial system. This information can be used to predict whether risky assets price is going to go up or not because more money in the financial system means more buying power while the supply of risky assets is limited.</p><h3>Company Data</h3><p>It is an information that ties to a company stock.</p><h4>Corporate Actions</h4><p>It is an activity done by a company that brings material changes and also impact its share holders (usually in a good way).</p><p>Some of common the corporate actions are:</p><ol><li>Dividends: payment to the shareholders based on the the amount of shares they had. Usually dividends are from the net income of the company.</li><li>Stock-split: reduce the nominal value of shares by increasing its amount. This usually makes the shares affordable by other investors.</li><li>Merger and Acquisition: multiple companies to become a new entity (merger) or one company acquiring another company (acquisition).</li><li>Right Issue: company increasing more shares and the current shareholders usually are given the right to purchase the stock before it’s released to public. Usually right-issue are used to raise a capital for company’s initiatives. It might also use the money for anothercorporate action as well.</li><li>Buyback: a company buy it’s own shares through public market using their own money. Usually gotten from its revenue.</li></ol><h4><strong>Company Profiles</strong></h4><p>It is just the profile of the stock company. This information can be used to found out who own the company and usually people invested based on the reputation of the owner.</p><h4>Company Financials</h4><p>It tells about how the company is doing financially. These information can be used by investors to tell how well the company in managing its cashes. Some of the important components are:</p><ol><li>Income Statement that talks about whether the company is making any money or not.</li><li>Balance Sheet that talks about what assets and debt that the company has.</li><li>Cash Flow Statement talks about how the money move in and out of the business.</li></ol><h3>Final Words</h3><p>There’s a lot of more to learn but I think these are common ones that investor or traders might be analyzing at before deciding on a trade or investment. It changed my mental model that investing and trading benefits those who has a data and analyze it well.</p><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c61992447d7c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Wish I Knew These “terms” Before Navigating into The Financial Market]]></title>
            <link>https://medium.com/@aw_marcell/i-wish-i-knew-these-terms-before-navigating-into-the-financial-market-c3901d4ee718?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3901d4ee718</guid>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[financial-freedom]]></category>
            <category><![CDATA[personal-finance]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 14:03:04 GMT</pubDate>
            <atom:updated>2026-01-16T10:58:15.940Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/705/0*I1FfYlWH-oUs98de.jpg" /><figcaption>Image Source: <a href="https://www.caixinglobal.com/2019-09-11/in-depth-stricter-regulations-push-chinas-money-market-funds-for-major-makeover-101461436.html">https://www.caixinglobal.com/2019-09-11/in-depth-stricter-regulations-push-chinas-money-market-funds-for-major-makeover-101461436.html</a>.</figcaption></figure><p>Personally, I am on a journey to becoming a data engineer in financial market domain. As I went through studying these terms I realized that my life would been easier if I had known it earlier.</p><p>When I invested my money in a company’s stock in 2020 for the first time, I didn’t bother to learn about common terms in financial market. As I wanted to get serious of becoming data engineer with financial market domain expertise, I realised that I didn’t understand many of these terms, especially when exploring the financial market dataset examples.</p><p>I took a step back and learned all of those terms and understanding it. Through this article I want to share these terms in a simple manner so that whoever you are — whether you are just starting to navigate into the financial market as a data engineer, data scientist, machine learning engineers, or even an investor or trader — I hope this article helps you in kickstarting the journey.</p><h3>Financial Market</h3><p>If you break it down, the term <strong>`financial`</strong> relates to managing funds, assets, money of an individual or organization. While <strong>`market` </strong>is a place for a buyer and seller to trade. If you combine them those words you get an financial market which is simply a place for buyers and sellers to trade a financial instruments such as currencies, stocks, bonds, commodities, and many more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*2d9dJOo1Ixwosmj08eXLBg.png" /><figcaption>Image by Author.</figcaption></figure><p>From what I learned, I think there are <strong>three main components</strong> that makes a financial market:</p><ol><li>Markets</li><li>Instruments</li><li>Participants</li></ol><p>This is logical because without these three, financial market doesn’t exists or can’t function. Without <strong>markets</strong>, there is no place for buyers and sellers to trade. Without <strong>instruments</strong>, there will is nothing to trade inside the market. Without <strong>participants</strong>, the market would be an empty place.</p><p>Let’s dig deeper on each of those components.</p><h3>Markets</h3><p>You already know what markets is, let’s jump into the common places where financial instruments can be traded:</p><h4>Exchanges</h4><p><em>A place for buyers and sellers trade financial instruments</em>.<br>All of the transactions are centralized through exchanges to ensure a fair price during transaction. Example of exchanges are: NYSE (New York Stock Exchanges), Nasdaq, IDX (Indonesia Stock Exchanges), etc.</p><h4>Over The Counter (OTC) Market</h4><p><em>A place where buyers and sellers trade financial products directly without going through exchanges.<br></em>These kind of transactions usually involve financial instruments that isn’t publicly listed in exchanges.</p><h4>Electronic Communication Network</h4><p><em>It is how modern exchanges works these days where they use </em><strong><em>technology information systems</em></strong><em> to automatically matched buyers and sellers orders transactions.</em> <br>Back then the exchanges is a place full of brokers shouting bid and offer for their client’s purchase.</p><h3>Intruments</h3><p>The “goods” that is being traded in the market, here are so common instruments traded:</p><h4>Foreign Currencies</h4><p>Some examples are USD, GBP, Euro, Yen, and many more.</p><h4>Commodities</h4><p>Some examples are precious metals, oil, wheat, coal, etc. Transactions can happen in two ways: buying the actual commodity and store it or buying contracts with the commodities as the underlying asset.</p><h4>Stocks</h4><p><em>A financial instrument that represents an ownership of a company.</em> <br>There are two ways to purchase a stock: <strong>you can buy a specific company’s stock </strong>or <strong>you can</strong> <strong>buy ETF (Exchange Traded Funds) stocks</strong>. ETF Stocks are basically a pool of stocks from multiple companies bundeld into one tradeable financial instrument.</p><h4>Fixed Income</h4><p><em>A financial instrument with a predefined returns based on the amount of money invested.<br></em>Most of the instruments are based on debt, which involves investors lending money to the borrowers. The borrower is obligated to pay interest or fees to the investor and after a maturity date, they are expected to return the original amount of to the investor. This is why it is called <strong>fixed income </strong>—<strong> </strong>because returns are known in advance by the investors.</p><h4>Derivative</h4><p><em>A financial instrument that is derived from underlaying asset (from another financial instrument).<br></em>Derivative instruments are mostly used to speculate on the price movement of a financial instrument. Some examples include options, futures, forwards, and swaps.</p><h3>Participants</h3><p>The entities that actually performing the transactions. Here are some of the common participants.</p><h4>Retail Investors</h4><p><em>Inviduals like you and me.</em></p><h4>Investment Banks</h4><p><em>Intermediaries for large and complex financial instruments (stock, bonds) transactions.</em><br>They also have a dedicated teams for managing assets for entities. Most of the clients they served are institutions with massive amount money, such as pension funds, endowment funds, corporations, family offices, and others.</p><h4>Hedge Funds</h4><p><em>Institutions that pool money from the investors — usually the wealthy individuals — to be trade and take percentage of the total gains as the fee.</em></p><h4>Central Banks</h4><p><em>Independent institution with a mandate to stabilize the currency value.</em> <br>They have variety of ways to do this, so such as raising or lowering interest rates, buying bonds, and others.</p><h3>Final Words</h3><p>I truly believe that I have only scratch the surface of what financial market is all about. There’s still a lot more to learn, but I hope this article gives you a head start in navigating into financial market.</p><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3901d4ee718" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Your Own Local Data Lakehouse using Open Source Tools]]></title>
            <link>https://medium.com/@aw_marcell/how-to-build-your-own-local-data-lakehouse-using-open-source-tools-c5a8626d10f7?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5a8626d10f7</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-lake]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Sun, 28 Dec 2025 11:47:43 GMT</pubDate>
            <atom:updated>2026-02-25T13:27:37.516Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n--0gvR-l7QrZs5x8R1bhQ.png" /><figcaption>Data Lakehouse Illustration (Source: <a href="https://www.oracle.com/a/ocom/img/rc24-data-lakehouse.jpg">https://www.oracle.com/a/ocom/img/rc24-data-lakehouse.jpg</a>)</figcaption></figure><p>The first time I heard about “Data Lakehouse” was back in September 2024. I just embarked on a new journey of becoming a data engineer — transitioning from being a data scientist. Throughout my data career, I had heard terms of data warehouses, data lakes, object storages, and databases, but never a data lakehouse.</p><p>Curious of how it works, it led me to create a local data lakehouse. Through this writing, I would like to share my experience and knowledge that will help you in building your own data lakehouse locally using free and open source tools.</p><h3>But, why Data Lakehouse even existed in the first place …</h3><p>From the 1980s, <strong>data warehouse</strong> was a go to solution for enabling business reports. Data are stored, cleansed, and transformed in a single location. Back then all of the data are generated from a software systems that writes to OLTP — so you would expect all data were already in structured format.</p><p>As time goes by, new technologies emerged and produces all kinds of data starting from: logs, texts, audios, videos, and many more. Data warehouse can’t handle such variety because data engineers need to process it before insert it. Also, due to the rise of data science and machine learning, it needs raw data for analysis and model training. This is where <strong>data lake</strong> became an important role because it can stores data in any file format: parquet, csv, json, mp3, mp4, txt, etc. But data lake has its own limitation when it comes to data discoverability — it will be very hard to find the data inside of it as the data inside the company starts to grow (data swamp).</p><p>So main problem statement is …</p><blockquote>How can you store data regardless of its file format while still maintaining the discoverability of that data?</blockquote><p>Data lakehouse is a solution for those problems. Data lakehouse is simply a combination of data warehouse and data lake. It takes takes advantages from both architecture:</p><ol><li>Data Lake — is a centralized storage that <strong>stores any kind of data from structured, semi-structured and unstructured format</strong> with <strong>low cost</strong>.</li><li>Data Warehouse — is a centralized storage that <strong>stores a data in a well-defined and structured format</strong>. Also it enforces data management and governance allowing for data discoverability eventhough it is stored in a data lake.</li></ol><h3>Enough theory, this is how you can build your own local data lakehouse</h3><p>A typical data lakehouse architecture pattern will always include:</p><ol><li>Data lake for storing all kinds of data.</li><li>Data catalog for navigating through the data inside data lake.</li><li>Open table format for enabling files to be treated as a table.</li><li>File format for telling what format of your actual data will be stored.</li><li>Distributed compute engine for general data processing.</li><li>Distributed query engine for quering data in efficiently.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*PkWINX51YFcG7_GnGSfajQ.png" /><figcaption>Data lakehouse architecture.</figcaption></figure><p>The open source tools that will be used are:</p><ol><li><strong>Apache Spark</strong> for general distributed compute engine.</li><li><strong>Trino</strong> for distributed query engine.</li><li><strong>Apache Gravitino</strong> for REST catalog. Also I will be using <strong>PostgreSQL</strong> as the data catalog backend, <strong>Iceberg</strong> open table format, and <strong>parquet </strong>as the file format.</li><li><strong>Apache Iceberg</strong> for the open table format.</li><li><strong>Parquet</strong> for the file format.</li><li><strong>MinIO</strong> for object storage as data lake.</li></ol><h3><strong>Minio: Data Lake</strong></h3><p>In order to set up a data lake using Minio you need to do two things. <strong>First</strong>, create a MinIO server container. Here’s the Docker compose yaml script for MinIO server.</p><pre>minio:<br>  platform: linux/amd64<br>  image: minio/minio:${MINIO_IMAGE_TAG}<br>  container_name: minio<br>  ports:<br>    - &quot;9000:9000&quot;<br>    - &quot;9002:9002&quot;<br>  env_file:<br>    - .env<br>  environment:<br>    MINIO_ROOT_USER: minio<br>    MINIO_ROOT_PASSWORD: miniopass<br>  volumes:<br>    - ./infrastructure/minio/volumes/data/:/data<br>  command: server /data --address :9000 --console-address &#39;:9002&#39;<br>  healthcheck:<br>    test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;-f&quot;, &quot;http://localhost:9000/minio/health/ready&quot;]<br>    interval: 5s<br>    timeout: 60s<br>    retries: 5<br>    start_period: 20s<br>  networks:<br>    - local-data-lakehouse-iceberg</pre><p><strong>Second</strong>, create a MinIO Client container to create a bucket and the initial objects inside of it. Here’s the Docker compose yaml script for MinIO client.</p><pre>minio-client:<br>  platform: linux/amd64<br>  image: &quot;minio/mc:${MINIO_CLIENT_IMAGE_TAG}&quot;<br>  container_name: minio-client<br>  env_file:<br>    - .env<br>  depends_on:<br>    minio: <br>      condition: service_healthy<br>  entrypoint: &gt;<br>    /bin/sh -c &quot;<br>    /usr/bin/mc alias set myminio http://minio:9000 minio miniopass;<br>    /usr/bin/mc mb myminio/iceberg --ignore-existing;<br>    /usr/bin/mc mb myminio/iceberg/warehouse --ignore-existing;<br>    /usr/bin/mc anonymous set public myminio/iceberg/warehouse;<br>    &quot;<br>  networks:<br>    - local-data-lakehouse-iceberg</pre><h3>Gravitino: REST Catalog Iceberg</h3><p>Next will be setting up REST data catalog where you need to do<strong> two things</strong>.</p><p><strong>First</strong>, you need create a <strong>metastore</strong> server container. Metastore is a storage system for storing your metadata. For this, we will use PostgreSQL as the metastore. Here’s the Docker compose yaml script for PostgreSQL.</p><pre>postgres:<br>  platform: linux/amd64<br>  image: postgres:${POSTGRESQL_IMAGE_TAG}<br>  container_name: postgres<br>  ports:<br>    - &quot;5432:5432&quot;<br>  env_file:<br>    - .env<br>  environment:<br>    POSTGRES_USER: postgres<br>    POSTGRES_PASSWORD: postgres<br>    POSTGRES_DB: catalog_metastore_db<br>  volumes:<br>    - ./infrastructure/postgres/volumes/data:/var/lib/postgresql/data<br>  healthcheck:<br>    test: [&quot;CMD-SHELL&quot;, &quot;pg_isready -U postgres -d catalog_metastore_db&quot;]<br>    interval: 5s<br>    timeout: 60s<br>    retries: 5<br>    start_period: 20s<br>  networks:<br>    - local-data-lakehouse-iceberg</pre><p><strong>Second</strong>, create a Gravitino server container that runs a REST Catalog Iceberg service inside of it.</p><p>For this you need to create a custom Dockerfile because of the custom configuration needed in order to run it. These configurations are:</p><ol><li>gravitino.conf for running the REST Catalog Iceberg service</li><li>core-site.xml for Gravitino to interact with MinIO</li><li>Shell script for downloading JAR Files</li><li>Shell script for starting Gravitino Server</li></ol><h4>1. gravitino.conf for running the REST Catalog Iceberg service</h4><p>It is located inside infrastructure/gravitino/conf/gravitino.conf. The configurations helps you in:</p><ol><li>Setting up service that you want to run inside the Gravitino Server container which is <strong>Iceberg REST Catalog</strong> service. It also sets up the host and port for other container to access this service.</li><li>Setting up what storage system that will be used as the metastore alongside with its credentials. In this case we use PostgreSQL.</li><li>Setting up where the location of the data lake and how to write and access to the data lake. In this case we use Hadoop File IO implementation.</li></ol><pre># =======================================================================<br># Enable Iceberg REST service in Gravitino<br># =======================================================================<br>gravitino.auxService.names=iceberg-rest<br>gravitino.iceberg-rest.classpath=iceberg-rest-server/libs, iceberg-rest-server/conf<br><br># =======================================================================<br># Set up Iceberg REST service host and port<br># =======================================================================<br>gravitino.iceberg-rest.host = 0.0.0.0<br>gravitino.iceberg-rest.httpPort = 9001<br><br># =======================================================================<br># Use PostgreSQL as REST Catalog backend<br># =======================================================================<br>gravitino.iceberg-rest.catalog-backend=jdbc<br>gravitino.iceberg-rest.uri=jdbc:postgresql://postgres:5432/catalog_metastore_db<br>gravitino.iceberg-rest.warehouse=s3a://iceberg/warehouse<br>gravitino.iceberg-rest.catalog-backend-name=jdbc<br>gravitino.iceberg-rest.jdbc-user=postgres<br>gravitino.iceberg-rest.jdbc-password=postgres<br>gravitino.iceberg-rest.jdbc-initialize=true<br>gravitino.iceberg-rest.jdbc-driver=org.postgresql.Driver<br><br># =======================================================================<br># Set up Hadoop FileIO for S3 access in Iceberg REST Catalog<br># =======================================================================<br>gravitino.iceberg-rest.io-impl=org.apache.iceberg.hadoop.HadoopFileIO</pre><h4>2. core-site.xml for Gravitino to interact with MinIO</h4><p>It is located inside infrastructure/gravitino/conf/core-site.xml. These configurations helps you in:</p><ol><li>Finding the location of the data lake.</li><li>Providing the credentials to access it.</li><li>Enforcing path style access (like this <a href="http://http//[MinIO_Endpoint]/[bucket-name]/[object-key]">http://[MinIO_Endpoint]/[bucket-name]/[object-key</a>]) because this is the default access method to access MinIO.</li><li>Disabling SSL security because this is just an experiment.</li></ol><pre>&lt;!-- core-site.xml --&gt;<br>&lt;configuration&gt;<br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.impl&lt;/name&gt;<br>    &lt;value&gt;org.apache.hadoop.fs.s3a.S3AFileSystem&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.endpoint&lt;/name&gt;<br>    &lt;value&gt;http://minio:9000&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.access.key&lt;/name&gt;<br>    &lt;value&gt;minio&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.secret.key&lt;/name&gt;<br>    &lt;value&gt;miniopass&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.path.style.access&lt;/name&gt;<br>    &lt;value&gt;true&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.connection.ssl.enabled&lt;/name&gt;<br>    &lt;value&gt;false&lt;/value&gt;<br>  &lt;/property&gt;<br>&lt;/configuration&gt;</pre><h4>3. Shell script for downloading JAR Files</h4><p>It is located inside infrastructure/gravitino/init/dependency.sh. This script is used for downloading the necessary JAR files for Gravitino to interact with MinIO.</p><pre>#!/bin/bash<br># =======================================================================<br># Load common functions<br># =======================================================================<br>. &quot;/root/gravitino/common/common.sh&quot;<br><br># =======================================================================<br># Prepare target directory for packages<br># =======================================================================<br>target_dir=&quot;/root/gravitino&quot;<br>if [[ ! -d &quot;${target_dir}/packages&quot; ]]; then<br>  mkdir -p &quot;${target_dir}/packages&quot;<br>fi<br><br># =======================================================================<br># Download Hadoop AWS JAR and MD5<br># Why?: This is needed for Gravitino to connect to S3 compatible storage like MinIO<br># =======================================================================<br>HADOOP_AWS_JAR=&quot;https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_AWS_JAR_VERSION}/hadoop-aws-${HADOOP_AWS_JAR_VERSION}.jar&quot;<br>echo &quot;Downloading ${HADOOP_AWS_JAR_VERSION}&quot;<br>HADOOP_AWS_MD5=&quot;${HADOOP_AWS_JAR}.md5&quot;<br>download_and_verify &quot;${HADOOP_AWS_JAR}&quot; &quot;${HADOOP_AWS_MD5}&quot; ${target_dir}<br><br><br># =======================================================================<br># Download AWS Java SDK Bundle JAR and MD5<br># Why?: This is needed for Gravitino to connect to S3 compatible storage like MinIO<br># =======================================================================<br>AWS_JAVA_SDK_BUNDLE_JAR=&quot;https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_JAVA_SDK_BUNDLE_JAR_VERSION}/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_JAR_VERSION}.jar&quot;<br>AWS_JAVA_SDK_BUNDLE_MD5=&quot;${AWS_JAVA_SDK_BUNDLE_JAR}.md5&quot;<br>download_and_verify &quot;${AWS_JAVA_SDK_BUNDLE_JAR}&quot; &quot;${AWS_JAVA_SDK_BUNDLE_MD5}&quot; ${target_dir}</pre><h4>4. Shell script for starting Gravitino Server</h4><p>It is located inside infrastructure/gravitino/init/init.sh. This is used for moving the JAR files and configuration files (core-site.xml and gravitino.conf) into a location that can be noticed by the Gravitino Server. Next we can just start the Gravitino server.</p><pre>#!/bin/bash<br>echo &quot;Start to download the jar packages&quot;<br><br># =======================================================================<br># Copy all of the JAR files to their respective folders<br># =======================================================================<br>cp &quot;/root/gravitino/packages/hadoop-aws-${HADOOP_AWS_JAR_VERSION}.jar&quot; /root/gravitino/iceberg-rest-server/libs<br>cp &quot;/root/gravitino/packages/hadoop-aws-${HADOOP_AWS_JAR_VERSION}.jar&quot; /root/gravitino/catalogs/lakehouse-iceberg/libs<br>cp &quot;/root/gravitino/packages/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_JAR_VERSION}.jar&quot; /root/gravitino/iceberg-rest-server/libs<br>cp &quot;/root/gravitino/packages/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_JAR_VERSION}.jar&quot; /root/gravitino/catalogs/lakehouse-iceberg/libs<br><br><br># =======================================================================<br># Copy all of the configuration in their respective folders<br># =======================================================================<br>cp /tmp/conf/gravitino.conf /root/gravitino/conf/gravitino.conf<br>cp /tmp/conf/core-site.xml /root/gravitino/conf/core-site.xml<br>cp /tmp/conf/core-site.xml /root/gravitino/iceberg-rest-server/conf/core-site.xml<br><br># =======================================================================<br># Start Gravitino server<br># =======================================================================<br>echo &quot;Finish downloading&quot;<br>echo &quot;Start the Gravitino Server&quot;<br>/bin/bash /root/gravitino/bin/gravitino.sh start &amp;<br>sleep 3<br>tail -f /root/gravitino/logs/gravitino-server.log</pre><p>Next, all of the configurations will be included during image Gravitino server image build process using the Dockefile located inside infrastructure/gravitino/Dockerfile.</p><pre># =======================================================================<br># Build layer image from apache/gravitino base image<br># =======================================================================<br>ARG GRAVITINO_IMAGE_TAG<br>FROM apache/gravitino:${GRAVITINO_IMAGE_TAG} AS gravitino-base<br>RUN apt-get update<br><br># =======================================================================<br># Build layer image from gravitino-base image<br># =======================================================================<br>ARG HADOOP_AWS_JAR_VERSION<br>ARG AWS_JAVA_SDK_BUNDLE_JAR_VERSION<br>FROM gravitino-base AS gravitino<br><br># Copy all initialization scripts<br># Commmon initialization scripts<br>COPY ./infrastructure/common/ ./common/<br># Gravitino initialization scripts<br>COPY ./infrastructure/gravitino/init/ ./init/<br><br># Install Gravitino Dependencies<br>RUN chmod u+x ./init/dependency.sh<br>RUN ./init/dependency.sh<br><br># Run Gravitino Initialization Script<br>RUN chmod u+x ./init/init.sh<br>ENTRYPOINT [ &quot;./init/init.sh&quot; ]</pre><h3>Spark: Distributed Compute Engine</h3><p>Next will be creating a Spark server container. There are five things that needs to be prepared:</p><ol><li>Shell script for downloading JAR files</li><li>Shell script for starting the Spark server</li><li>core-site.xml for Spark to interact with MinIO</li><li>spark-defaults.conf for Spark default configuration</li></ol><h4>1. Shell script for downloading JAR files</h4><p>It is located inside infrastructure/spark/init/dependency.sh. This script is used for downloading the necessary JAR files for Spark to:</p><ol><li>Working with Iceberg open table format</li><li>Interact with Iceberg REST Catalog</li><li>Interact with MinIO</li></ol><pre>#!/bin/bash<br><br># =======================================================================<br># Load common functions<br># =======================================================================<br>. &quot;/root/spark/common/common.sh&quot;<br><br># =======================================================================<br># Prepare target directory for packages<br># =======================================================================<br>target_dir=&quot;/root/spark&quot;<br>if [[ ! -d &quot;${target_dir}/packages&quot; ]]; then<br>  mkdir -p &quot;${target_dir}/packages&quot;<br>fi<br><br># =======================================================================<br># Download Iceberg Spark Runtime JAR and MD5<br># Why?: This is needed for Spark to work with Iceberg tables<br># =======================================================================<br>ICEBERG_SPARK_RUNTIME_JAR=&quot;https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-${SPARK_SCALA_VERSION}/${ICEBERG_VERSION}/iceberg-spark-runtime-${SPARK_SCALA_VERSION}-${ICEBERG_VERSION}.jar&quot;<br>ICEBERG_SPARK_RUNTIME_MD5=&quot;${ICEBERG_SPARK_RUNTIME_JAR}.md5&quot;<br>download_and_verify &quot;${ICEBERG_SPARK_RUNTIME_JAR}&quot; &quot;${ICEBERG_SPARK_RUNTIME_MD5}&quot; ${target_dir}<br><br># =======================================================================<br># Download PostgreSQL JAR and MD5<br># Why?: This is needed for Spark to connect to REST Catalog that uses PostgreSQL as backend<br># =======================================================================<br>POSTGRESQL_JAR=&quot;https://repo1.maven.org/maven2/org/postgresql/postgresql/${POSTGRESQL_JAR_VERSION}/postgresql-${POSTGRESQL_JAR_VERSION}.jar&quot;<br>POSTGRESQL_MD5=&quot;${POSTGRESQL_JAR}.md5&quot;<br>download_and_verify &quot;${POSTGRESQL_JAR}&quot; &quot;${POSTGRESQL_MD5}&quot; ${target_dir}<br><br># =======================================================================<br># Download Hadoop AWS JAR and MD5<br># Why?: This is needed for Spark to connect to S3 compatible storage like MinIO<br># =======================================================================<br>HADOOP_AWS_JAR=&quot;https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_AWS_JAR_VERSION}/hadoop-aws-${HADOOP_AWS_JAR_VERSION}.jar&quot;<br>HADOOP_AWS_MD5=&quot;${HADOOP_AWS_JAR}.md5&quot;<br>download_and_verify &quot;${HADOOP_AWS_JAR}&quot; &quot;${HADOOP_AWS_MD5}&quot; ${target_dir}<br><br># =======================================================================<br># Download AWS Java SDK Bundle JAR and MD5<br># Why?: This is needed for Gravitino to connect to S3 compatible storage like MinIO<br># =======================================================================<br>AWS_JAVA_SDK_BUNDLE_JAR=&quot;https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_JAVA_SDK_BUNDLE_JAR_VERSION}/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_JAR_VERSION}.jar&quot;<br>AWS_JAVA_SDK_BUNDLE_MD5=&quot;${AWS_JAVA_SDK_BUNDLE_JAR}.md5&quot;<br>download_and_verify &quot;${AWS_JAVA_SDK_BUNDLE_JAR}&quot; &quot;${AWS_JAVA_SDK_BUNDLE_MD5}&quot; ${target_dir}</pre><h4>2. Shell script for starting the Spark server</h4><p>It is located inside infrastructure/spark/init/init.sh. This is used for moving the JAR files and configuration files (core-site.xml and spark-defaults.conf) into a location that can be noticed by the Spark server. Next we can just start the Spark server.</p><pre>#!/bin/bash<br><br># =======================================================================<br># Copy all of the JAR files to their respective folders<br># =======================================================================<br>cp ./packages/iceberg-spark-runtime-${SPARK_SCALA_VERSION}-${ICEBERG_VERSION}.jar /root/spark/jars/iceberg-spark-runtime-${SPARK_SCALA_VERSION}-${ICEBERG_VERSION}<br>cp ./packages/postgresql-${POSTGRESQL_JAR_VERSION}.jar /root/spark/jars/postgresql-${POSTGRESQL_JAR_VERSION}.jar<br>cp ./packages/hadoop-aws-${HADOOP_AWS_JAR_VERSION}.jar /root/spark/jars/hadoop-aws-${HADOOP_AWS_JAR_VERSION}.jar<br>cp ./packages/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_JAR_VERSION}.jar /root/spark/jars/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_JAR_VERSION}.jar<br><br># =======================================================================<br># Copy all of the configuration in their respective folders<br># =======================================================================<br>cp /tmp/conf/spark-defaults.conf ${SPARK_HOME}/conf/spark-defaults.conf<br>cp /tmp/conf/core-site.xml ${SPARK_HOME}/conf/core-site.xml<br><br># =======================================================================<br># Create metalake and catalog in Gravitino<br># =======================================================================<br>sh /tmp/common/init_metalake_catalog.sh<br><br># Run spark based on the workload type<br>SPARK_WORKLOAD=$1<br>echo &quot;SPARK_WORKLOAD: $SPARK_WORKLOAD&quot;<br>if [ &quot;$SPARK_WORKLOAD&quot; == &quot;master&quot; ];<br>then<br>    start-master.sh -p 7077<br>elif [ &quot;$SPARK_WORKLOAD&quot; == &quot;worker&quot; ];<br>then<br>    start-worker.sh spark://spark-master:7077<br>elif [ &quot;$SPARK_WORKLOAD&quot; == &quot;history&quot; ]<br>then<br>    start-history-server.sh<br>fi</pre><h4>3. core-site.xml for Spark to interact with MinIO</h4><p>It is located inside infrastructure/spark/conf/core-site.xml. This xml file is for Spark to connect to and interact with MinIO.</p><pre>&lt;configuration&gt;<br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.impl&lt;/name&gt;<br>    &lt;value&gt;org.apache.hadoop.fs.s3a.S3AFileSystem&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.endpoint&lt;/name&gt;<br>    &lt;value&gt;http://minio:9000&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.access.key&lt;/name&gt;<br>    &lt;value&gt;minio&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.secret.key&lt;/name&gt;<br>    &lt;value&gt;miniopass&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.path.style.access&lt;/name&gt;<br>    &lt;value&gt;true&lt;/value&gt;<br>  &lt;/property&gt;<br><br>  &lt;property&gt;<br>    &lt;name&gt;fs.s3a.connection.ssl.enabled&lt;/name&gt;<br>    &lt;value&gt;false&lt;/value&gt;<br>  &lt;/property&gt;<br>&lt;/configuration&gt;</pre><h4>4. spark-defaults.conf for Spark default configuration</h4><p>It is located inside infrastructure/spark/conf/spark-defaults.conf. This will be the default configuration when you run a Spark session inside the container. You only need to worry about the configuration that helps Spark in working with Iceberg and accessing Gravitino Iceberg REST Catalog.</p><pre># =======================================================================<br># Set up Spark Configuration<br># =======================================================================<br>spark.master=spark://spark-master:7077<br>spark.eventLog.enabled=true<br>spark.eventLog.dir=/opt/spark/spark-events<br>spark.history.fs.logDirectory=/opt/spark/spark-events<br><br># =======================================================================<br># Enable data catalog access from Gravitino Iceberg REST Catalog<br># =======================================================================<br>spark.jars.packages=org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.0<br>spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions<br>spark.sql.catalog.catalog_iceberg=org.apache.iceberg.spark.SparkCatalog<br>spark.sql.catalog.catalog_iceberg.type=rest<br>spark.sql.catalog.catalog_iceberg.uri=http://gravitino:9001/iceberg/</pre><p>Next, all of the configurations will be included during image Spark server image build process using the Dockefile located inside infrastructure/spark/Dockerfile.</p><pre># Install Image with Python Version 3.10.11<br>ARG PYTHON_VERSION<br>FROM python:${PYTHON_VERSION}-bullseye AS spark-base<br><br>RUN apt-get update &amp;&amp; \<br>    apt-get install -y --no-install-recommends \<br>    sudo \<br>    curl \<br>    vim \<br>    unzip \<br>    rsync \<br>    software-properties-common \<br>    openjdk-11-jdk \<br>    build-essential \<br>    ssh &amp;&amp; \<br>    apt-get clean &amp;&amp; \<br>    rm -rf /var/lib/apt/lists/*do<br><br>FROM spark-base AS spark<br><br># =======================================================================<br># Create folder for the Hadoop and Spark folders<br># =======================================================================<br>ARG SPARK_VERSION<br># RUN echo $SPARK_VERSION<br>ENV SPARK_HOME=&quot;/root/spark&quot;<br>RUN mkdir -p ${SPARK_HOME}<br>WORKDIR ${SPARK_HOME}<br><br># =======================================================================<br># Install Standalone Spark<br># =======================================================================<br>RUN curl https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz -o ${SPARK_HOME}/spark-${SPARK_VERSION}-bin-hadoop3.tgz \<br>    &amp;&amp; tar xvzf ${SPARK_HOME}/spark-${SPARK_VERSION}-bin-hadoop3.tgz --directory ${SPARK_HOME} --strip-components 1 \<br>    &amp;&amp; rm -rf ${SPARK_HOME}/spark-${SPARK_VERSION}-bin-hadoop3.tgz<br><br>ARG SPARK_SCALA_VERSION<br>ARG ICEBERG_VERSION<br>ARG POSTGRESQL_JAR_VERSION<br>ARG HADOOP_AWS_JAR_VERSION<br>ARG AWS_JAVA_SDK_BUNDLE_JAR_VERSION<br><br>FROM spark AS spark-deps<br># =======================================================================<br># Copy all initialization scripts<br># =======================================================================<br># Commmon initialization scripts<br>COPY ./infrastructure/common/ ./common/<br><br># Spark initialization scripts<br>COPY ./infrastructure/spark/init/ ./init/<br># =======================================================================<br># Install Spark Dependencies <br># =======================================================================<br>RUN chmod u+x ./init/dependency.sh<br>RUN ./init/dependency.sh<br><br># =======================================================================<br># Install Python Libraries <br># =======================================================================<br>ARG SPARK_VERSION<br>ARG SCALA_VERSION<br>ARG ICEBERG_VERSION<br>ARG POSTGRESQL_JAR_VERSION<br>ARG HADOOP_AWS_JAR_VERSION<br>ARG AWS_JAVA_SDK_BUNDLE_JAR_VERSION<br><br>FROM spark-deps AS spark-python<br># Install python dependencies<br>COPY ./infrastructure/spark/requirements.txt .<br>RUN pip3 install -r requirements.txt<br><br># =======================================================================<br># Set Environment Variables for Spark <br># =======================================================================<br>ENV PATH=&quot;${SPARK_HOME}/sbin:${SPARK_HOME}/bin:${PATH}&quot;<br>ENV SPARK_MASTER=&quot;spark://spark-master:7077&quot;<br>ENV SPARK_MASTER_HOST=spark-master<br>ENV SPARK_MASTER_PORT=7077<br>ENV PYSPARK_PYTHON=python3<br><br># =======================================================================<br># Set The sbin and bin to be Executable<br># =======================================================================<br>RUN chmod u+x ./sbin/* &amp;&amp; \<br>    chmod u+x ./bin/*<br><br>ENV PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH<br><br># =======================================================================<br># Run init script<br># =======================================================================<br>RUN chmod u+x ./init/init.sh<br>ENTRYPOINT [ &quot;./init/init.sh&quot; ]</pre><h3>Trino: Distributed Compute Engine</h3><p>Setting up Trino is simpler. You only need to set up Catalog properties, like:</p><ol><li>Connector that you want to use, in this case we use iceberg connector.</li><li>Type and URI for REST Catalog.</li><li>Location and credentials for Trino to interact with MinIO.</li></ol><p>All of these configurations will be stored in /etc/catalog/&lt;catalog_name&gt;.properties file. The &lt;catalog_name&gt; will be filled according to the catalog name created in data lakehouse. It can be anything.</p><p>Here is the properties that I included for setting up Trino.</p><pre>connector.name=iceberg<br>iceberg.catalog.type=rest<br>iceberg.rest-catalog.uri=http://gravitino:9001/iceberg/<br>fs.hadoop.enabled=true<br><br>hive.s3.endpoint=http://minio:9000<br>hive.s3.aws-access-key=minio<br>hive.s3.aws-secret-key=miniopass<br>hive.s3.path-style-access=true</pre><h3>Finally …</h3><p>We will wrap all of the containers initialization under docker-compose.yml file.</p><h3>Common Problems and How to Solve it.</h3><p>The most common problem that I dealt with was an error relating to JAR packages. Eventhough it didn’t stated that it is due to incompatible version, turns out it is really all about the JAR version. To solve this, you have to make sure that all of your JAR file versions must be compatible with each other.</p><h3>Final Words</h3><p>In the end it is not all about the data tools because new ones will arise in the future but the architecture pattern for data lakehouse remainse the same: <strong>data lake</strong> for storage, <strong>distributed</strong> <strong>compute engine </strong>for data processing, <strong>distributed query engine </strong>for efficient querying especially for dashboards, and <strong>data catalog</strong> for navigating inside data lake</p><h3>Code Repository</h3><p>You can access the code repository through this link: <a href="https://github.com/marcellinus-witarsah/local-data-lakehouse-iceberg">https://github.com/marcellinus-witarsah/local-data-lakehouse-iceberg</a></p><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5a8626d10f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Data Streaming Project: Bitcoin Live Price Dashboard]]></title>
            <link>https://medium.com/@aw_marcell/real-time-data-streaming-project-bitcoin-live-price-dashboard-dda614c28177?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/dda614c28177</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[timescaledb]]></category>
            <category><![CDATA[data-streaming]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Mon, 02 Jun 2025 15:00:35 GMT</pubDate>
            <atom:updated>2025-06-02T15:00:35.306Z</atom:updated>
            <content:encoded><![CDATA[<h4>Built using Docker, Kafka, TimescaleDB, and Dash</h4><h3>Intro</h3><p>In the financial world, Bitcoin has emerged as a revolutionary digital asset. Created in 2009 by the anonymous figure known as <em>Satoshi Nakamoto.</em> Some view him as a hero because he gave Bitcoin to the world — a decentralized cryptocurrency, designed to operate without any central authority — not even the government. While Satoshi’s identity remains a mystery, his contribution to decentralized finance is undeniably historic.</p><p>This project is both a learning experience in <strong>real-time data streaming</strong> and a tribute for Satoshi. I’ve built a dashboard that displays live Bitcoin prices with candlestick using modern data engineering tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*Hz9JEE6oQtNbCSIL.jpg" /><figcaption>Status of Satoshi Nakamoto (Image by Wikipedia)</figcaption></figure><h3>About The Project</h3><p>The goal is simple.</p><p>Build a <strong>web application dashboard</strong> that visualizes <strong>Bitcoin live price data in USD</strong>, using candlestick charts at three intervals: 10 seconds, 30 second, and 1 minute.</p><blockquote>A candlestick chart is a graph that displays the high, low, open, and closing prices of a stock or other security over time. Technical analysts use candlestick charts to track pricing trends that may be repeated.<br> — <a href="https://www.investopedia.com/terms/c/candlestick.asp">Investopedia</a> —</blockquote><p>A single candlestick represents prices within defined unit measure of time. It could be a candlestick of a second, an hour, a day, and so on. Here are the components of a candlestick:</p><ol><li><strong>Open</strong>: the first price in the interval</li><li><strong>High</strong>: the highest price traded in the interval</li><li><strong>Low</strong>: the lowest price in the interval</li><li><strong>Close</strong>: the last price in the interval</li><li><strong>Volume</strong>: the volum traded in the interval</li></ol><p><em>*However for drawing the candlestick OHLC (Open High Low Close) is enough.</em></p><p>The color inside the body will be determined by the open and close prices:</p><ol><li>🟩Green if the color of the <strong>open price &lt; close price</strong>.</li><li>🟥Red if the color of the <strong>open price &gt; close price</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*_XMa5nddywxCLf9sVH-yXw.png" /><figcaption>Image of OHLC Inside the Candlestick (Image by Author)</figcaption></figure><h3>Data</h3><p>I use trade data from <strong>Kraken </strong>which is a crypto exchange application. The trade data can be accessed through a websocket API. For more information please visit the Kraken API Center documentation on their website.</p><p><a href="https://docs.kraken.com/api/docs/websocket-v2/trade">Trades | Kraken API Center</a></p><h3>Architecture &amp; Tools</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*MUgMI_4WEHGsCLxp73QcxQ.png" /></figure><p>This system follows a <strong>Kappa-inspired architecture</strong> — a design pattern where data flows through a single stream pipeline. In this project, I simplified it by pushing transformations into the time-series database itself.</p><p>The project follows a <strong>Kappa-inspired Architecture</strong> — a design pattern where data flows through a single stream pipeline. However, I simplified it by pushing transformations into the time-series database itself. I designed it this way because of project scope is simple: stream Bitcoin trade data in USD currency and perform an aggregation for 10 seconds, 30 second, and 1 minute. A timeseries database like is sufficient for the aggregation tasks.</p><p>Some of the tools were used are:</p><ul><li><a href="https://www.docker.com/">Docker</a> for containerizing Kafka and TimescaleDB.</li><li><a href="https://kafka.apache.org/">Kafka</a> for streaming data.</li><li><a href="https://www.timescale.com/">TimescaleDB</a> for storing and processing time series data.</li><li><a href="https://dash.plotly.com/">Dash</a> for displaying trade data through a web application dashboard.</li></ul><h3>Stages</h3><p>In this project, There are four stages traversed by the data:</p><ol><li><strong>Ingestion</strong></li><li><strong>Loading and Transformation</strong></li><li><strong>Serving</strong></li></ol><h4>Ingestion</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*n0oHcaF9DKVngUgjfVXcVg.png" /></figure><ul><li>Websocket connection is created between client and <strong>Kraken API Server</strong>. Through this persisted open connection trade data will be pushed from the server to the client, so that the client won’t need to continuously poll for data.</li><li><strong>Kafka producer</strong> receive the trade data in form of JSON and push it to <strong>Kafka topic </strong>in serialized format.</li><li>A <strong>Kafka Consumer</strong> listens to the topic, deserializes the messages, and prepares them for storage.</li></ul><p><em>*A Kafka topic will be created inside Kafka Broker which is responsible for storing Kafka topics and managing read from Kafka producer and write requests from Kafka consumer.</em></p><h4>Loading and Transformation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*XcQa7lcL9pM-pG5l8L_htA.png" /></figure><ul><li>Incoming trade data will be loaded into <strong>TimescaleDB </strong>which an enhanced PostgreSQL for processing time series data.</li><li>TimescaleDb stores the data inside a <strong>hypertable </strong>that automatically create chunks of data inside a table and partition it by time. Whenever you perform a query, <strong>TimescaleDB </strong>can identify the correct chunk and runs the query on it. This speeds up the query process without reading the whole table.</li><li>Under the hood, <strong>hypertable</strong><em> </em>utilized a storage engine called <strong>hypercore </strong><em>— a </em>hybrid between <em>row-oriented </em>and <em>columnar-oriented </em>data storage. A <em>row-oriented </em>data storage optimized for inserting and updating new <em>hot </em>records. As the data <em>cools </em>or older, it will be converted into and <em>columnar-oriented </em>format optimized for analytics.</li></ul><p>To generate candlestick charts, we need OHLCV data aggregated at multiple intervals (10 seconds, 30 second, and 1 minute). TimescaleDB helps by:</p><ul><li>Defining <strong>continuous aggregates</strong> — pre-computed materialized views that update automatically as new data arrives using <strong>automatic refresh policy.</strong></li><li>Storing these aggregates in <strong>materialized views</strong>, so queries can be without recomputing data.</li></ul><h4>Serving</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*z6DO2v9JrBCGT7xFeaMyWA.png" /></figure><p>I built a <strong>web dashboard using Dash</strong>, a Python framework for building interactive UIs.</p><ul><li>The candlestick chart is rendered using <strong>Plotly</strong>, which Dash uses under the hood.</li><li>The UI includes a <strong>dropdown menu</strong> to choose the aggregation interval: 10 seconds, 30 second, or 1 minute.</li><li>The dashboard updates every 10 seconds to fetch and display the latest data (based on the lowest time interval).</li></ul><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ltCde-77Rjxeg04kEta4A.gif" /></figure><h3>What I learnt from this</h3><p>This project helped me in understanding:</p><ul><li><strong>Real-Time Data Ingestion</strong><br>I learned how to establish a WebSocket connection using the <strong>Kraken WebSocket API</strong> to receive live Bitcoin trade data without polling.</li><li><strong>Building a Simple Data Streaming Pipeline</strong><br>I implemented a basic real-time pipeline streaming using <strong>Kafka </strong>from creating producer, topic, and consumer.</li><li><strong>Containerizazing Infrastructure with Docker</strong> <br>I containerized the Kafka, and TimescaleDB using <strong>Docker </strong>making the setup reproducible and portable.</li><li><strong>Understanding Candlestick Charts</strong><br>I gained domain knowledge about <strong>OHLCV</strong> and how candlestick charts work, including how traders use them to interpret price movement in financial markets.</li></ul><h3>Improvements</h3><p>This is still a prototype and of course there’s still a room for improvement, such as:</p><ul><li><strong>Leverage Scalability of Kafka<br></strong>Add more trade data from multiple exchanges Like Coinbase, Binance, Bybit, etc. to leverage the scalabilility of Kafka.</li><li><strong>Leverage Distributed Realtime Processing with Apache Flink</strong><br>Leveraging Apache Flink for distributed realtime processing if we want to stream and process trade data from multiple source.</li><li><strong>Archive Data using Object Storage</strong><br>Use Object Storage for store raw trade data for historical analysis, or ML tasks in the future.</li><li><strong>Add Technical Indicators</strong><br>Use technical indicators for feature enhancement such as moving averages, exponential moving averages, RSI, and more.</li></ul><h3>GitHub Link</h3><p><a href="https://github.com/marcellinus-witarsah/btc-price-dashboard">GitHub - marcellinus-witarsah/btc-price-dashboard: Candlestick Chart Dashboard using Kafka, TimescaleDB, and Docker</a></p><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dda614c28177" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I learned from “The Data Warehouse Toolkit” about Data Warehousing]]></title>
            <link>https://medium.com/@aw_marcell/what-i-learned-from-the-data-warehouse-toolkit-471e6961f9e3?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/471e6961f9e3</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[personal-growth-journey]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Fri, 09 May 2025 13:56:15 GMT</pubDate>
            <atom:updated>2025-05-09T14:43:20.022Z</atom:updated>
            <content:encoded><![CDATA[<h3>What I learned from “The Data Warehouse Toolkit”</h3><h4>Personal Takeway from This Book</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nsYgvh5wuhrEE2HS" /><figcaption>The Data Warehouse Toolkit Third Edition — Ralph Kimball and Margy Ross</figcaption></figure><p>At work, I am working on creating data pipelines, which involve creating pipeline logic and target tables. Then I was given a document showing a massive number of tables connected to each other. As I asked my coworker, I learned that this is the data model that shows how data is going to be stored in the data lakehouse. I began to question how they were able to come up with this design. I asked what this document is called, and he said “Logical Data Model.” Then I went surfing through the internet and found <em>The Data Warehouse Toolkit, Third Edition</em> by Ralph Kimball and Margy Ross. I read the book and found 5 insights that I want to share.</p><h3><strong><em>1. Data is Very Important Especially in The 21st Century</em></strong></h3><p>Personally, data is anything that contains facts of measurements and descriptions about events that happened in the world. Data come in mainly two types:</p><ul><li><strong>Analog Data </strong>which is<strong> </strong>any data that is produced in the world whether it’s a voice, writing, etc.</li><li><strong>Digital Data </strong>which is any data stored in a from of digital, whether its obtained from transforming analog data to digital data, or generated from a system, etc.</li></ul><p>I think it’s a no-brainer to say that data is important, given the growing amount of data being created every day. Back in the early days, data didn’t get much attention until the rise of software applications. Nowadays, we spend most of our time in the digital world — social media, web browsing, shopping, reading, you name it. Software applications are generating a massive amount of data from these digital activities. The image below shows a projection of global data creation in 2030 and 2035, expected to reach 612 and 2,142 zettabytes, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*l_O_9d5aA0M059ng.jpeg" /><figcaption>Image from <a href="https://www.statista.com/chart/17727/global-data-creation-forecasts/">https://www.statista.com/chart/17727/global-data-creation-forecasts/</a></figcaption></figure><p>I think Ralph Kimball had a vision about the rise of data in the future and that must be the reason that he led him to invent <strong>dimensional modeling </strong>technique.</p><h3><strong><em>2. There are Two Sides of Data Systems — OLTP vs OLAP</em></strong></h3><p>Data are stored in data systems, there are two of the most common data systems:</p><ul><li><strong>Online Transaction Processing (OLTP)</strong><br>Data system used for day-to-day operations. This system is optimized for fast reads and writes on small amounts of data — like inserting or updating or even deleting a single record in a table. Examples of this process will be sales transaction, customer account creation, booking order, order tracking, and many more.</li><li><strong>Online Analytical Processing (OLAP)</strong><br>Data system used analytical queries. This system is optimized for processing a millions or billions records.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VwxzuBZI8vRwgAPq" /><figcaption>Illustration of How OLTP and OLAP is Used (Image By Author)</figcaption></figure><p>Businesses always start with OLTP systems for their operations. In the early stages, getting insights and reports on business performance is still possible using OLTP systems since the number of records produced is manageable. However, as the business grows, they will struggle to handle an increasing volume of data. That’s where OLAP systems come into play. One example of an OLAP system is a <strong>data warehouse</strong>.</p><h3>3. <strong><em>Data Warehousing It’s a Design Process</em></strong></h3><p>From this book, I learned that it’s not about creating a data warehouse but it’s designing a data warehouse. The challenge is how to design it so that we can represent events in business operations into a structured table inside data warehouse for business users. If we could succeeded in doing this, our business users can evaluate their business performances. Designing a data warehouse requires condiferation from different aspects, such as:</p><ul><li><strong>Simplicity</strong><br>Having a simple and understandable data willl win the hearts of many business users. This can be achieved by designing schema that represents the business process and common terms used in organization.</li><li><strong>Consistency</strong><br>The role of the data warehouse is to be the single source of truth for business users. All elements inside the data warehouse such as dimensions, columns, labels, etc. must have the same definition across organization. Also ensure that the formula being used is the same.</li><li><strong>Adaptability</strong><br>Design a table structure that is robust to ever changing business questions. In order to do so, we have to capture the business events in the lowest granularity that we can.</li><li><strong>Security</strong><br>Data is treated as an important asset for the company as it holds confidential information and trade secrets of an organization. Security is also being considered.</li></ul><h3><strong><em>4. Dimensional Modeling Technique</em></strong></h3><p>Of course the main star from this book — <strong>dimensional modeling</strong>. It also has been widely accepted as a go-to method for schema design. Dimensional modeling is a technique that represnts you data in form of facts and dimensions. <strong>Facts</strong> are data containing the results of business process while the <strong>dimensions </strong>are data containing descriptiosn about facts.</p><p>Dimensional modeling technique can be implemented in both OLTP and OLAP. In OLTP system, it is known as <strong>Star Schema</strong>. While in OLAP system, it is known as <strong>OLAP Cubes</strong>. Eventhough they have different names, but the design implementation is similar. More about dimensional modeling will be explained in separate article.</p><h3>5. Building The Best Data Warehouse Solution isn’t The Main Objective</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OuS5HsPZtmST4A2c" /><figcaption>Image by Author</figcaption></figure><p>Building the best data warehouse with the best tech stack isn’t the primary goal — it’s about getting buy-in from business users. These are the people who will support your project. Don’t be a perfectionist; ship something simple that creates impact for the business. Generating quick wins will attract more support from business users.</p><p>I also had a conversation with my manager about this topic. He explained that, in order to gain support from business users, you have to be a “salesperson” inside the organization. What benefits does your data warehouse solution bring to their table?</p><p>Also, Involving business users in the dimensional modeling process is also important. They help clarify what really happens in day-to-day business operations.</p><h3>Final Thoughts</h3><p>This book opened my eyes because I used to think that a data warehouse was all about technical stuff. However, it shows the crucial part of data warehouse which is <strong>dimensional modeling</strong>. The technique itself is customer-centric — it helps in designing tables that represents business operations and comfortable for business users to work with. It’s all about enabling them to navigate their business data easily — it’s not just about using cutting-edge data tools to build the data warehouse.</p><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=471e6961f9e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Here’s What “Fundamentals of Data Engineering” Book Taught Me About Data Engineering]]></title>
            <link>https://medium.com/@aw_marcell/heres-what-fundamentals-of-data-engineering-book-taught-me-about-data-engineering-335657331c48?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/335657331c48</guid>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Thu, 01 May 2025 14:34:20 GMT</pubDate>
            <atom:updated>2025-05-02T05:31:56.271Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FkocjaUruj38q25XWOtjjA.png" /><figcaption>Fundamentals of Data Engineering: Plan and Build Robust Data Systems — Joe Reis and Matt Housley</figcaption></figure><p>At the start of my career as a data engineer, being new without knowing anything really bothered me, so I started to learn the basics about data engineering. I searched through the internet and found a book called <em>“Fundamentals of Data Engineering”</em> — by Joe Reis and Matt Housley. I started reading it, and this book is interesting because it gives me a glimpse of what Data Engineering really is. I want to share these insights with you.</p><h4>1. Data Engineering Lifecycle will be Remain Relevant in The Future</h4><p>The Data Engineering Lifecycle for me is a map that shows the processes and stages of data traversing from its source and being used by the business. This concept is implemented in almost every data engineering project in the industry. Below are the picture of the stages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-bw0dTof257EQHyc" /><figcaption>Data Engineering Lifecycle (Image from Fundamentals of Data Engineering: Plan and Build Robust Data Systems — Joe Reis and Matt Housley).</figcaption></figure><ol><li><strong>Generation</strong>: source system produces data</li><li><strong>Ingestion</strong>: process of bringing data for processing</li><li><strong>Transformation</strong>: standardize and enrich the data</li><li><strong>Serving</strong>: provide data for business users to consume</li><li><strong>Storage</strong>: place for storing the data during ingestion, transformation, and serving</li></ol><p>There are some undercurrents (which I won’t explain one by one) that supports the data engineering lifecycle itself. Even now in my work, I acknowledge that the way we process data for client projects is similar to the data engineering lifecycle.</p><h4>2. Understand the Concepts and Fundamentals, not The Tools</h4><p>Back in university, I always focused on learning programming languages rather than understanding the concepts. I constantly binged endless tutorials on how to program in C++, Python, Java, SQL, and so on. The way I studied was by memorizing syntax. After reading this, I realized that I had been doing it all wrong. That’s why, even after long hours of memorizing syntax, I had no idea what I was going to do next.</p><p>A similar analogy would be learning how to use a saw to cut a tree without actually understanding the tecnique of cutting a tree — and then, decades later, chainsaws were introduced, and people started using them. But the technique of cutting a tree stayed the same, even with the new tools like a chainsaw.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1er6tUK5dZ-lRSkC" /><figcaption>Analogy of Cutting Trees with Saw vs Chainsaw (Image by Author)</figcaption></figure><p>The way I learn now is by understanding the concepts first, and then choosing the tools to apply the concepts I just learned. If you’re learning a tool like dbt, don’t jump straight into it — you might get confused by concepts like dimensional modeling. Why not learn the concept first, at least briefly, before using the tool, so you understand how it helps you build the data model.</p><p>At the end of the day, tools are just tools — they can become obsolete as newer, more efficient tools emerge. But fundamentals and concepts last much longer — often for at least 10+ years. For example, Hadoop MapReduce as a big data processing software has faded away, but the MapReduce concept is still highly relevant in modern data platforms like Spark, Snowflake, and Google BigQuery.</p><h4>3. Choosing the Right Technology for Data Engineering Projects</h4><p>Data engineering is easier than ever before, thanks to the rise of modern data tools. But the number of emerging data tools is out of control, leading to a <strong><em>Paradox of Choices </em></strong>where too many options can hinder clear decision-making. That’s why selecting the right technology is an underrated skill in data engineering. In this book, they recommend:</p><ol><li>Identifying project requirements before selecting technologies.</li><li>Not chasing the hype of implementing cutting-edge technologies.</li><li>Choosing technologies that are familiar to the data team and speed up development.</li><li>Selecting tools that are likely to remain relevant in the current and near future. For safe options, go with tools that have been tested by time, such as object storage, ETL/ELT, distributed computing, etc.</li></ol><h4>4. Be Curious and Always Improve Yourself</h4><p>I think the title is self-explanatory. We always have to reinvent ourselves in order to survive in a the tech world. Some of the advice includes:</p><ol><li>Learning new concepts and data tools.</li><li>Attending seminars.</li><li>Joining and getting involved in communities.</li><li>Reading papers.</li></ol><h3>Final Words</h3><p>Those are some insights that I found to be a profound mindset for getting into data engineering. I don’t know about you guys, but if you think there’s more, I’d love to see it in the comment section.</p><blockquote>If you enjoyed this article, feel free to follow me on Medium! You can also connect with me through <a href="https://www.linkedin.com/in/marcellaw/">LinkedIn</a></blockquote><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=335657331c48" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[About Me — Marcellinus AW]]></title>
            <link>https://medium.com/@aw_marcell/about-me-marcellinus-aw-8ee36efc923b?source=rss-2856fdb1b852------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ee36efc923b</guid>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[about-me]]></category>
            <category><![CDATA[introduction]]></category>
            <category><![CDATA[blogging]]></category>
            <dc:creator><![CDATA[Marcellinus Aditya Witarsah]]></dc:creator>
            <pubDate>Sun, 20 Apr 2025 09:44:55 GMT</pubDate>
            <atom:updated>2025-05-03T08:16:04.989Z</atom:updated>
            <content:encoded><![CDATA[<h3>About Me — Marcellinus AW</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H19jqwpHpGanUkbP5y8sCg.png" /><figcaption>Image Generated by ChatGPT (Credits to anotechhub for The Prompt)</figcaption></figure><p>Hi everyone, so glad that you stumble into my medium blog. First of all, allow me to introduce myself.</p><p>My name is Marcellinus Aditya Witarsah, and I’m currently based in Indonesia. Right now, I’m working as a Data Consultant at one of the Big 4 firms for almost a year now. My work mainly revolves around supporting clients in building data platforms, as well as working with machine learning, deep learning models, and other data-driven solutions.</p><p>Even though I am working at one of the world’s top consulting firms, I don’t think that makes me special. What brought me here wasn’t being the smartest in the room — it was being curious and open to learning new things. Still, after months of working the usual 9–5 and studying in my free time, I started to feel empty — like my life was on auto-pilot. So I turned to writing — to express myself, reflect, and do something that fuels me. I’ve shared a few articles before, but this time I want to make writing a consistent part of my life.</p><p>I want to share everything I’ve learned so far, and I hope that anyone on a similar journey can also drop by and share their experiences. I’d love to learn from you too, and I believe both my writing and your stories can help us grow together. And if you’re a fellow Indonesian reading this — feel free to talk to me in Bahasa! I’ll be more than happy to reply.</p><p>By the way, some of the topics that I will be writing:</p><ol><li><em>Data Engineering</em></li><li><em>Data Science</em></li><li><em>Work Experience</em></li><li><em>Self Development</em></li><li><em>Blockchain</em></li><li><em>Finance</em></li></ol><p><a href="https://medium.com/@aw_marcell">Marcellinus Aditya Witarsah - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ee36efc923b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>