<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Thanh Enc on Medium]]></title>
        <description><![CDATA[Stories by Thanh Enc on Medium]]></description>
        <link>https://medium.com/@thanhenc?source=rss-bfd74b3c7d6b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FIlVz2gS1qX3IqxZVowjnA.png</url>
            <title>Stories by Thanh Enc on Medium</title>
            <link>https://medium.com/@thanhenc?source=rss-bfd74b3c7d6b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:26:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thanhenc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to combine multiple MCP servers together]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://generativeai.pub/how-to-combine-multiple-mcp-servers-together-12a1e3d229c2?source=rss-bfd74b3c7d6b------2"><img src="https://cdn-images-1.medium.com/max/1424/1*efXyIfm1bsa6t6UeQbaffw.png" width="1424"></a></p><p class="medium-feed-snippet">Treat MCP server as a route in a big one (Parent MCP Server)</p><p class="medium-feed-link"><a href="https://generativeai.pub/how-to-combine-multiple-mcp-servers-together-12a1e3d229c2?source=rss-bfd74b3c7d6b------2">Continue reading on Generative AI »</a></p></div>]]></description>
            <link>https://generativeai.pub/how-to-combine-multiple-mcp-servers-together-12a1e3d229c2?source=rss-bfd74b3c7d6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/12a1e3d229c2</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[lifespan]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Thanh Enc]]></dc:creator>
            <pubDate>Sun, 08 Jun 2025 05:28:12 GMT</pubDate>
            <atom:updated>2025-06-08T09:12:13.200Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to Ingest 20-year-stock-trading data REST API with Apache Spark the right way (without Auto…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/how-to-ingest-20-year-stock-trading-data-rest-api-with-apache-spark-the-right-way-without-auto-50a1da303077?source=rss-bfd74b3c7d6b------2"><img src="https://cdn-images-1.medium.com/max/1502/1*rsZNVn4s2hEompTjt7I4sw.png" width="1502"></a></p><p class="medium-feed-snippet">How to ingest data with REST API in Apache Spark the right way</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/how-to-ingest-20-year-stock-trading-data-rest-api-with-apache-spark-the-right-way-without-auto-50a1da303077?source=rss-bfd74b3c7d6b------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/how-to-ingest-20-year-stock-trading-data-rest-api-with-apache-spark-the-right-way-without-auto-50a1da303077?source=rss-bfd74b3c7d6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/50a1da303077</guid>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[optimize]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Thanh Enc]]></dc:creator>
            <pubDate>Sun, 01 Jun 2025 10:08:30 GMT</pubDate>
            <atom:updated>2025-06-01T20:32:07.818Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[[Part 2] Remote SSE MCP Server: How I Built a Powerful Fintech AI Agent Integrating Realtime …]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/part-2-remote-sse-mcp-server-how-i-built-a-powerful-fintech-ai-agent-integrating-realtime-9657f4e1ffd7?source=rss-bfd74b3c7d6b------2"><img src="https://cdn-images-1.medium.com/max/2020/1*E6nPJJBTZ4PSx9RHLXH4aw.png" width="2020"></a></p><p class="medium-feed-snippet">An MCP server can be accessed by a single endpoint for production AI Agent products.</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/part-2-remote-sse-mcp-server-how-i-built-a-powerful-fintech-ai-agent-integrating-realtime-9657f4e1ffd7?source=rss-bfd74b3c7d6b------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/part-2-remote-sse-mcp-server-how-i-built-a-powerful-fintech-ai-agent-integrating-realtime-9657f4e1ffd7?source=rss-bfd74b3c7d6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9657f4e1ffd7</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[sse-mcp]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[remote-mcp-server]]></category>
            <category><![CDATA[model-context-protocol]]></category>
            <dc:creator><![CDATA[Thanh Enc]]></dc:creator>
            <pubDate>Sat, 17 May 2025 14:01:44 GMT</pubDate>
            <atom:updated>2025-05-18T11:55:39.011Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[[Part 1] Remote SSE MCP Server: How I Built a Powerful Fintech AI Agent Integrating Realtime …]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/part-1-remote-sse-mcp-server-how-i-built-a-powerful-fintech-ai-agent-integrating-realtime-ecdd11098361?source=rss-bfd74b3c7d6b------2"><img src="https://cdn-images-1.medium.com/max/2020/1*E6nPJJBTZ4PSx9RHLXH4aw.png" width="2020"></a></p><p class="medium-feed-snippet">For the easiest definition of remote SSE MCP server, it is a MCP server hosted somewhere and you are able to connect to it by just a URL&#x2026;</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/part-1-remote-sse-mcp-server-how-i-built-a-powerful-fintech-ai-agent-integrating-realtime-ecdd11098361?source=rss-bfd74b3c7d6b------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/part-1-remote-sse-mcp-server-how-i-built-a-powerful-fintech-ai-agent-integrating-realtime-ecdd11098361?source=rss-bfd74b3c7d6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecdd11098361</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[vietnam]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[model-context-protocol]]></category>
            <dc:creator><![CDATA[Thanh Enc]]></dc:creator>
            <pubDate>Sun, 04 May 2025 16:11:31 GMT</pubDate>
            <atom:updated>2025-05-22T10:52:33.709Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Production Grade K8S From Bare Metal/Workstation With Multipass, MicroK8s, MetalLB, Cloudflare…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/production-grade-k8s-from-bare-metal-workstation-with-multipass-microk8s-metallb-cloudflare-b335b4c97b7e?source=rss-bfd74b3c7d6b------2"><img src="https://cdn-images-1.medium.com/max/1920/1*BHXleDuoK1QhboQedpmA_w.png" width="1920"></a></p><p class="medium-feed-snippet">This article provides valuable insights and guidance on establishing a production-ready Kubernetes cluster for your home lab, data&#x2026;</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/production-grade-k8s-from-bare-metal-workstation-with-multipass-microk8s-metallb-cloudflare-b335b4c97b7e?source=rss-bfd74b3c7d6b------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/production-grade-k8s-from-bare-metal-workstation-with-multipass-microk8s-metallb-cloudflare-b335b4c97b7e?source=rss-bfd74b3c7d6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b335b4c97b7e</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Thanh Enc]]></dc:creator>
            <pubDate>Fri, 02 May 2025 17:19:39 GMT</pubDate>
            <atom:updated>2025-05-04T10:15:50.093Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Turn Old Laptop To Private Kubernetes Cluster — Enable Others to Connect to It]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/turn-old-laptop-to-private-kubernetes-cluster-enable-others-to-connect-to-it-c2921fcb3ff1?source=rss-bfd74b3c7d6b------2"><img src="https://cdn-images-1.medium.com/max/1398/1*vCiUvy--aD2dA5nqAWRM0A.png" width="1398"></a></p><p class="medium-feed-snippet">Imagine you have an old, rarely used laptop and want to set up a K8s cluster at home instead of renting one from cloud services with high&#x2026;</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/turn-old-laptop-to-private-kubernetes-cluster-enable-others-to-connect-to-it-c2921fcb3ff1?source=rss-bfd74b3c7d6b------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/turn-old-laptop-to-private-kubernetes-cluster-enable-others-to-connect-to-it-c2921fcb3ff1?source=rss-bfd74b3c7d6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2921fcb3ff1</guid>
            <category><![CDATA[k8s-data-project]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[private-network]]></category>
            <category><![CDATA[setup]]></category>
            <dc:creator><![CDATA[Thanh Enc]]></dc:creator>
            <pubDate>Sun, 09 Feb 2025 13:39:47 GMT</pubDate>
            <atom:updated>2025-02-11T18:15:51.209Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore…]]></title>
            <link>https://blog.dataengineerthings.org/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-dea3e012b89c?source=rss-bfd74b3c7d6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/dea3e012b89c</guid>
            <category><![CDATA[trinos]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[end-to-end]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Thanh Enc]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 12:57:58 GMT</pubDate>
            <atom:updated>2025-03-18T16:19:39.223Z</atom:updated>
            <content:encoded><![CDATA[<h3>PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8S8XP_nK-su6qjtOldWfw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3z-7GRrM85HywirX3h07Fg.png" /><figcaption>Data flow (Source: Author)</figcaption></figure><p><strong>PoC Data Platform</strong> is an innovative project designed to demonstrate how data from AdventureWorks can be extracted, loaded, and transformed within a data lake environment. Leveraging a suite of modern data tools and technologies, the platform serves as a comprehensive showcase and a practical template for individuals interested in data engineering, data system design, or simply exploring the data world. Whether you’re a professional or just beginning to learn about data systems, this project offers a clear reference point to deepen your understanding of the modern data stack.</p><p>Below is a Docker map that provides a visual overview of the interrelationships between all services integrated within the platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WDGBixsfnhKPw4E4Frjuow.png" /><figcaption>Docker map (Source: Author)</figcaption></figure><h3>I. Components/Services</h3><p>📂 Project structure</p><pre>.<br>├── adw14/                            /* AdventureWorks data folder */<br>├── airflow/                          /* Airflow folder contains DAGs, configurations and docker */<br>├── assets/                           /* Contains various project assets like pdf, images, etc. */<br>├── dbt/                              /* DBT project folder contains project files and docker */<br>├── docker/                           /* Docker configurations and related files of data product */<br>│   ├── backup/                           /* Backup files for databases or components */<br>│   ├── certbot/                          /* Certbot folder for managing SSL certificates */<br>│   ├── lightdash/                        /* Lightdash folder for business intelligence tools */<br>│   ├── minio/                            /* MinIO for object storage (S3-compatible) */<br>│   ├── nginx/                            /* Nginx web server and reverse proxy configurations */<br>│   ├── spark/                            /* Apache Spark cluster configurations */<br>│   ├── trino/                            /* Trino cluster configurations */<br>│   ├── volumes/                          /* Docker volumes for persistent data storage */<br>│   ├── .env.example                      /* Template of environment variables file */<br>│   ├── docker-compose.yaml               /* Docker Compose file to define data product services */<br>│   ├── Makefile                          /* Makefile for automating tasks and commands */<br>├── .gitignore<br>├── Author.md<br>├── LICENSE<br>├── README.md<br>└── References.md</pre><p><strong>1.1. Adventureworks OLTP</strong></p><p><a href="https://learn.microsoft.com/en-us/sql/samples/adventureworks-install-configure?view=sql-server-ver16&amp;tabs=ssms">Adventureworks</a> is utilized as the OLTP database for our project. As a comprehensive operational database published by Microsoft, Adventureworks covers a wide range of domains. For full visibility into its data schema, you can reference <a href="https://dataedo.com/samples/html/AdventureWorks/doc/AdventureWorks_2/home.html">Dataedo</a>. To align with the requirements of our data platform, we have migrated this database to PostgreSQL, simulating a real-world database environment that acts as the primary data source for our data lake.</p><blockquote><em>How to initialize the data</em></blockquote><p>After launching the OLTP service (PostgreSQL) in the data platform, you can import data into the database by running the following command</p><pre>cd docker<br>make init-db</pre><p>Or you can run the bash file directly in the <em>adw14 </em>folder</p><pre>cd adw14<br>chmod +x adw14data.sh<br>adw14data.sh</pre><p><strong>1.2. Apache Airflow</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fXXKC71p5rkwmen9XqFSUQ.gif" /><figcaption>Airflow overview (Source: Author)</figcaption></figure><p>Apache Airflow serves as data platform orchestrator, which manages and schedules all data pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5pG3XdvtBOnil7F1Ntc2gQ.png" /><figcaption>Airflow Docker map (Source: Author)</figcaption></figure><p>You can start Airflow by running this following command</p><pre>cd airflow<br>make compose-init<br># if it&#39;s the first time you run airflow, or you want to rebuild airflow image<br><br># or<br># make compose-up</pre><p>In this project, I created 2 connections in Airflow. Make sure you also create required connections before running the DAGs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4XUu1zxLe8poE-bTKOIwLQ.png" /><figcaption>Airflow connections (Source: Author)</figcaption></figure><p><strong>1.3. Apache Spark</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IhoVjKqbsZnP-SwT_lb0Zw.png" /><figcaption>Spark Master jobs history (Source: Author)</figcaption></figure><p>Apache Spark acts as the data processing engine responsible for extracting and loading data from the OLTP source to the bronze layer in the data lake. Spark’s power lies in its versatility, handling not only data engineering tasks but also data science and machine learning workloads, whether on single-node machines or distributed clusters. However, for the purposes of this showcase, Spark is used in a minimal capacity to allow other tools in the data stack to take center stage 😜</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*GU3FLeCg1HML30OpXvm9yA.png" /><figcaption>Spark Docker map (Source: Author)</figcaption></figure><p>In this project, I initialized the Spark cluster with 01 master node and 01 worker node. You can easily scale the number of worker nodes by specifying the --scale spark-worker=N parameter, where N represents the desired number of worker nodes</p><pre>cd docker<br>make compose-up-spark1 #To init 01 worker node<br># make compose-up-spark2 #To init 02 worker nodes<br><br># or you can run this command without Make to init 02 worker nodes<br>docker compose --profile spark --scale spark-worker=2 up -d</pre><p>😉 You can check and adjust my Makefile in docker folder to have more management commands</p><p><strong>1.4. MinIO</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CLvrMxtGAUWTD0eNlZ6Usg.png" /><figcaption>MinIO Overview (Source: Author)</figcaption></figure><p>MinIO serves as the storage layer in the data lake, housing both raw and transformed data. Apache Spark, Hive Metastore, and Trino interact directly with MinIO for various data operations. For a clearer understanding of how these services are connected, you can refer to the overview Docker map, Spark Docker map, and Trino Docker map, which illustrate the relationships between these components.</p><p><strong>1.5. Hive-metastore</strong></p><p>😅 Ops, I don’t know which images about Hive-metastore should be added to this article. We will dive into the docker map to know more about the relationship between Hive-metastore and other services 🫡</p><p><strong>1.6. Trino</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EwQ3bh1ZW56OQmM8CvEQKw.gif" /><figcaption>Trino overview (Source: Author)</figcaption></figure><p>Trino functions as a query engine that can access and query data from multiple sources while performing transformations within its cluster. In our setup, we utilized Trino as a data warehouse, where DBT orchestrates the transformation jobs by triggering the execution directly within Trino.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*INTN4EAqFlsX-9yYfjN-EQ.png" /><figcaption>Delta catalog in Trino after transforming data (Source: Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*Xi7fTw8_Zd66LMYC-e9h6g.png" /><figcaption>Trino Docker map (Source: Author)</figcaption></figure><p><strong>1.7. DBT</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oq7AqiVI9BZmiEPcyI6KhQ.gif" /><figcaption>DBT Docs (Source: Author)</figcaption></figure><p>DBT acts as an essential layer in our data transformation process, facilitating the process of moving data from the bronze layer to the data mart. Within the DBT project, schema information, field definitions, data quality checks, and data lineage are managed in a centralized location. Additionally, DBT documents can be hosted as a site, providing easy access to data mart’s metadata.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nxIpAGzM9NTEEavGnuoVA.png" /><figcaption>Fact sales lineage (Source: Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pawnFzn-ruT7CQblOwF7uw.png" /><figcaption>DBT Docker map (Source: Author)</figcaption></figure><p>We can use DBT with 02 options:</p><ul><li>Cloud: <a href="https://www.getdbt.com/product/dbt-cloud">dbt Cloud | dbt Labs (getdbt.com)</a></li><li>DBT-core: <a href="https://github.com/dbt-labs/dbt-core">https://github.com/dbt-labs/dbt-core</a></li></ul><p><strong>Data model</strong>: In this project, I created a simple data model with 3 dimensions. You can expand the model by adding more dimensions and metrics to the fact table, which would be a great way to enhance data analysis skills.</p><p>✌🏻 With dim customers, you can have more insightful information from XML data in the Person schema. But I’m too lazy to do it now 😜 give you this chance🫵🏻</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*64yQ0qEKU3mMiMlWk9LW-A.png" /><figcaption>Sales mart data model (Source: Author)</figcaption></figure><p><strong>1.8. Lightdash</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Im_boucKKwqD2V-Uejo6yw.gif" /><figcaption>Lightdash overview (Source: Author)</figcaption></figure><p>Lightdash is used as BI tool in this project. The reason why I chose it is Lightdash is a perfect match with DBT and it’s opensource which we can use for self-hosting.</p><blockquote><em>Daily delivery sales dashboard to stakeholders via email</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*51NmfHsrPIRiroKJIfrVmw.png" /><figcaption>Lightdash daily delivery email (Source: Author)</figcaption></figure><p>Stakeholders can download the pdf file of the sales dashboard or directly access Lightdash to view the interactive version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m_JHsoCyx86l4sPQIggwSA.png" /><figcaption>Lightdash Docker map (Source: Author)</figcaption></figure><p>We can use Lightdash with 02 options</p><ul><li>Cloud: <a href="https://www.lightdash.com/">Lightdash | Self-serve BI to 10x your data team</a></li><li>Self-host: <a href="https://docs.lightdash.com/self-host/self-host-lightdash/">Self-Host Lightdash | Documentation | Lightdash</a></li></ul><p><strong>1.9. And some other services</strong></p><p>As seen in the Docker setup, I’ve included several essential services such as Nginx, Certbot and metadata database for data services. These components serve as the foundational building blocks of our data platform, ensuring the smooth operation of the project.</p><p>However, these services can be replaced by equivalent cloud solutions if needed</p><ul><li>Metadata database (Airflow, Lightdash, Hive metastore, OLTP): <a href="https://neon.tech/">Neon Serverless Postgres</a> <a href="https://supabase.com/">Supabase</a></li><li>Minio: Amazon S3, Cloudflare R2, DigitalOcean Spaces Object Storage</li></ul><h3>II. Quick start</h3><blockquote>🍀<strong>You can find my Github repo here</strong>: <a href="https://github.com/thanhenc/e2e-data-platform">thanhENC/e2e-data-platform: End-to-end data platform: A PoC Data Platform project utilizing modern data stack (Spark, Airflow, DBT, Trino, Lightdash, Hive metastore, Minio, Postgres) (github.com)</a></blockquote><h4>Service distribution</h4><p>Due to the limitations in computing resources, I have decided to distribute data services based on their resource consumption rate and the interrelationships among all services, rather than consolidating evething on a single server 🔥</p><p>For this project, I chose to rent VPS from DigitalOcean. It’s an excellent option for running side projects beside other providers like AWS, GCP, Azure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*enUdZ3bvsNh2K8stEiiB6A.png" /><figcaption>Services distribution (Source: Author)</figcaption></figure><p>⚠️ Something I wanna inform you</p><ul><li>Lightdash only accepts secure connections from Trino using username-password; connections without authentication are not permitted</li><li>Trino must be hosted under SSL/TLS to utilize username and password authentication</li><li>There are several methods to add SSL/TLS to Trino. For this project, I used a certificate from Certbot (Let’s Encrypt). Alternatively, you can implement SSL/TLS by setting up a proxy through Cloudflare 🫡</li></ul><h4>Domain name</h4><p>In this project, we need a <strong>domain name</strong> for several important reasons, particularly to enable <strong>SSL/TLS certificates</strong> for secure <strong>user-password authentication</strong> in <strong>Trino</strong>. One great option for a <strong>free domain name</strong> is <strong>DuckDNS</strong> 🦆, which allows you to point a domain to your server easily</p><blockquote><a href="https://www.duckdns.org/"><em>https://www.duckdns.org/</em></a></blockquote><p>🤑 Alternatively, you can purchase a domain from providers like <a href="http://namecheap.pxf.io/BnyazB"><strong>Namecheap</strong></a>, <strong>Name.com</strong>, or <strong>GoDaddy.</strong></p><p>Beyond securing Trino, having a domain offers additional benefits. It simplifies access to other services, like <strong>Airflow Webserver</strong>, <strong>Lightdash</strong>, <strong>DBT documentation</strong>, and <strong>Spark UI</strong>, allowing you to use a recognizable domain instead of the less convenient &lt;ip&gt;:&lt;port&gt; format for accessing these services 🙂‍↕️</p><h4>SSL/TLS</h4><p>Two solutions you can try to implement SSL/TLS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*c67EOo09Q0xO0GA3jARMQw.png" /></figure><p>If you already have SSL/TLS from domain name provider when you purchased it, feel free to skip this section.</p><h4>Starting all services</h4><p>To start data services in this project, let’s jump to the dockerdirectory and run command in Makefile or docker command if you have more specify</p><pre>Usage: make [target]<br><br>Targets:<br>  create-network                Create docker network if not exists<br>  compose-up-all                Compose up with all services<br>  compose-up-spark1             Compose up with 1 spark worker<br>  compose-up-spark2             Compose up with 2 spark workers<br>  compose-up-spark3             Compose up with 3 spark workers<br>  compose-up-certbot            Compose up with certbot<br>  compose-up-trino              Compose up with 1 trino workers<br>  compose-down                  Compose down all services then remove network<br>  get-cert                      Get certbot certificate<br>  spark-submit                  Spark master submit<br>  backup                        Backup OLTP, Minio and Hive metastore data<br>  init-db                       Initialize adventureworks database<br>  clean                         Clean temp data<br>  clean-all                     Clean all<br>  help                          Show this help message</pre><p>For example, starting all services</p><pre>cd docker<br>cp .env.example .env<br>make compose-up-all</pre><p>If you can’t work with Makefile</p><pre>cd docker<br>cp .env.example .env<br>docker network inspect data-network &gt; /dev/null 2&gt;&amp;1 || docker network create --driver bridge data-network<br>docker compose --profile spark --profile trino --env-file .env up -d</pre><p>😉If you need to customize the configuration, please refer to the comments in our .env.example file and update the corresponding values in your .env file. Additionally, you might need to make adjustments to the docker-compose.yaml file itself, such as changing image versions, service profiles, port mappings, or volume mounts, based on your specific deployment environment and requirements. After making any changes, please re-run make compose-up-all</p><h3>Supervisors &amp; Supporters</h3><p>I came up with the idea for this side project after a meeting with my sister, <a href="https://www.linkedin.com/in/trangtth179/">Ms. Trang</a> (MSc Business Informatics at the University of Mannheim, Germany). I would like to express my heartfelt gratitude to Ms. Trang for her inspiration and encouragement throughout my journey.</p><p>Thank <a href="https://www.linkedin.com/in/trannhatnguyen2604/">@NhatNguyen</a>‘s <a href="https://github.com/trannhatnguyen2/NYC_Taxi_Data_Pipeline">work</a> and <a href="https://github.com/langgenius/dify">Dify</a> for the main references about project components and structures.</p><p>Thank <a href="https://www.linkedin.com/in/baotram237/">@BaoTram</a> for giving me foundational knowledge about DBT to apply to this project.</p><p>And a big thanks to all the authors of articles, blogs, and videos I’ve watched to finish this project. You can find them in References list here.</p><p>Thank you for reading my article. If you find it helpful, please give me a celebrate 👏🏻👏🏻👏🏻</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8S8XP_nK-su6qjtOldWfw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dea3e012b89c" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-dea3e012b89c">PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore…</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transforming a Template Repository into Your Own GitHub Project]]></title>
            <link>https://medium.com/@thanhenc/transforming-a-template-repository-into-your-own-github-project-7f123c964c7c?source=rss-bfd74b3c7d6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f123c964c7c</guid>
            <category><![CDATA[github-template-setup]]></category>
            <category><![CDATA[customizing-git-repo]]></category>
            <category><![CDATA[push-code-to-github]]></category>
            <category><![CDATA[change-git-remote-origin]]></category>
            <category><![CDATA[git-clone-to-your-repo]]></category>
            <dc:creator><![CDATA[Thanh Enc]]></dc:creator>
            <pubDate>Sun, 25 Aug 2024 08:45:38 GMT</pubDate>
            <atom:updated>2024-08-25T08:45:38.905Z</atom:updated>
            <content:encoded><![CDATA[<p>When starting a new project, using a template repository can be a quick and efficient way to get up and running. However, once you’ve cloned the template, you’ll want to customize it and make it your own by linking it to your own GitHub repository. In this guide, I’ll walk you through the simple steps to clone a template repository, replace its remote with your own, and push your customized project to GitHub.</p><h3>Step 1: Clone the Template Repository</h3><p>Start by cloning the template repository to your local machine:</p><pre>git clone &lt;template repository link&gt;<br><br># Example: git clone https://github.com/template_user/template_repo.git</pre><h3>Step 2: Remove the Template’s Remote Repository</h3><p>Once cloned, remove the connection to the template’s remote repository:</p><pre>git remote remove origin</pre><h3>Step 3: Connect Your Own GitHub Repository</h3><p>Next, link your own GitHub repository by adding it as the new remote origin:</p><pre>git remote add origin &lt;your repository link ending in **.git**&gt;<br><br># Example: git remote add origin https://github.com/your_username/your_repo.git</pre><h3>Step 4: Verify the New Remote Repository</h3><p>Ensure that your repository is correctly set up by checking the current remote URLs:</p><pre>git remote --verbose</pre><h3>Step 5: Push the Code to Your Repository</h3><p>Finally, push the code to your own GitHub repository:</p><pre>git push -u origin main</pre><p>And that’s it! You’ve successfully transformed a template repository into your own GitHub project. By following these steps, you can quickly set up new projects using templates, ensuring that your code is organized and versioned right from the start. Happy coding, and may your projects thrive on GitHub!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f123c964c7c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>