<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Albert Franzi on Medium]]></title>
        <description><![CDATA[Stories by Albert Franzi on Medium]]></description>
        <link>https://medium.com/@FranziCros?source=rss-95473c1b1c09------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jGxrvP_zRxPHn5lmRzQp7Q.jpeg</url>
            <title>Stories by Albert Franzi on Medium</title>
            <link>https://medium.com/@FranziCros?source=rss-95473c1b1c09------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:05:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@FranziCros/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[“Reducing Costs with Debezium, Kafka, and Iceberg: A Journey to Efficient Data Integration with…]]></title>
            <link>https://medium.com/albert-franzi/reducing-costs-with-debezium-kafka-and-iceberg-a-journey-to-efficient-data-integration-with-36bebaeab6e9?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/36bebaeab6e9</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Mon, 16 Sep 2024 08:16:44 GMT</pubDate>
            <atom:updated>2024-09-16T10:54:22.068Z</atom:updated>
            <content:encoded><![CDATA[<h3>“Reducing Costs with Debezium, Kafka, and Iceberg: A Journey to Efficient Data Integration with Snowflake”</h3><blockquote>In this article, we share how we transformed our data integration pipeline by forwarding data captured by Debezium from Kafka to Snowflake, leveraging <a href="https://iceberg.apache.org/">Iceberg</a> as the intermediary. This approach not only streamlined our data processing but also significantly reduced operational costs.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZBOeizPSvyyuHW20MgXhoQ.jpeg" /></figure><h3>Context</h3><p>This is the second article in our series detailing how we migrated from the MySQL Fivetran connector to Debezium — a change that resulted in a $250,000 annual cost reduction. In this installment, we focus on how we forward and persist data from Kafka to Amazon S3, structure it using Apache Iceberg with AWS Glue Catalog, and ultimately access it from Snowflake through its external table capability with Iceberg.</p><h3>The infrastructure</h3><p>This part of our data flow involves several key components:</p><ol><li><strong>Spark Operator for Streaming Jobs</strong>: We use the <a href="https://github.com/kubeflow/spark-operator">Spark Operator</a> to deploy and run streaming jobs that transfer data from Kafka to S3.</li><li><strong>PostgreSQL RDS for Checkpointing</strong>: A PostgreSQL RDS instance serves as our checkpoint database. It monitors which data has been forwarded to Iceberg and which data is pending, ensuring data consistency and robustness.</li><li><strong>Airflow Orchestrated Spark Jobs</strong>: We utilize Airflow to orchestrate Spark jobs that create or upsert partitions in Iceberg tables. These tables are then accessed as external tables in Snowflake via our DBT models.</li></ol><p>By integrating these components, we establish a robust data pipeline that efficiently moves data from Kafka to Snowflake, leveraging Apache Iceberg for scalable and performant data storage.</p><h3>The Spark Operator</h3><p>The Spark Operator, a project currently being sponsored by the Kubeflow community, enables us to run streaming Spark jobs inside K8s by defining a <a href="https://www.kubeflow.org/docs/components/spark-operator/user-guide/writing-sparkapplication/">SparkApplication</a> K8s resource.</p><p>To provide the Spark Operator with the correct code to execute, we build a Docker image containing our Spark application and push it to Amazon ECR (Elastic Container Registry) using GitHub Actions (GHA). This image includes all necessary dependencies and our custom application code.</p><p>While we chose to build our own Spark Docker image from scratch to tailor it to our specific needs, the <a href="https://github.com/kubeflow/spark-operator/tree/master/spark-docker">community-provided</a> images are also a viable option for many use cases.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd0b6852b7e8153c5e5d072fd178f60b/href">https://medium.com/media/cd0b6852b7e8153c5e5d072fd178f60b/href</a></iframe><p>We manage the deployment of our Spark jobs using Argo CD, a declarative GitOps continuous delivery tool for Kubernetes. Whenever we update our Spark application code or add a new job, Argo CD detects the changes in our Git repository and automatically deploys them to our cluster.</p><p>Ensure that your Spark Docker image includes the following packages:</p><ul><li><a href="https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html">spark-sql-kafka</a>: Allows Spark to read from and write to Kafka streams.</li><li><a href="https://iceberg.apache.org/spark-quickstart/">iceberg-spark-runtime</a>: Provides integration with Apache Iceberg for data management.</li></ul><p>Including these packages ensures that our Spark applications can interact with Kafka and Iceberg seamlessly.</p><h3>From Kafka to S3</h3><p>As explained in our previous article, we set up a Consumer ACL user with read-only permissions to allow our Spark jobs to read from Kafka securely, preventing any accidental writes.</p><p>To authenticate with Kafka using our Consumer credentials, we retrieve and set them in the Spark options. We use AWS Secrets Manager to handle all credentials, fetching them programmatically in jobs that have the appropriate IAM Role for Service Accounts (IRSA). This approach prevents unauthorized access to secrets and ensures that credentials are not hardcoded or exposed in code or other processes.</p><p>Once we have established a connection to Kafka, we use Spark Structured Streaming to read the data from the Kafka topics. We forward the data to S3 every five minutes.</p><p>Because we consume multiple topics at once by subscribing to a set of topics via a pattern, we need to construct the S3 paths dynamically based on the topic information. We extract the topic pattern from the key field and iterate over each dataset with the correct schema.</p><p>Now, let’s focus on extracting the data from the JSON records.</p><h4><strong>Retrieving Schemas with SQLAlchemy</strong></h4><p>Debezium supports including the schema in the Change Data Capture (CDC) records, but this can create excessive overhead and fill the MSK cluster faster than desired. Instead, we retrieve the source schemas using SQLAlchemy’s inspector capability. Given an SQLAlchemy engine, we programmatically retrieve the table schemas (DDL statements). We then build the Spark StructType schema from this information and properly decode the JSON fields.</p><p>Here is how we achieve this in our code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71c6018f7d7e2dcd530d32f430694b0c/href">https://medium.com/media/71c6018f7d7e2dcd530d32f430694b0c/href</a></iframe><p>In these functions:</p><ul><li><strong>get_field</strong>: Maps each column from the database schema to a Spark StructField, converting the SQL data type to a corresponding Spark data type.</li><li><strong>create_spark_schema</strong>: Constructs a StructType schema for Spark DataFrames based on the list of columns.</li><li><strong>retrieve_schema</strong>: Uses an SQLAlchemy Inspector to get the columns of a specific table and then builds the Spark schema.</li><li><strong>get_dataset_schema</strong>: Retrieves and caches the schema for a given dataset to improve performance.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5bbbbc96889d56d3ffc377bb5de5583/href">https://medium.com/media/c5bbbbc96889d56d3ffc377bb5de5583/href</a></iframe><h4><strong>Processing Debezium Records</strong></h4><p>We define a function to process the Debezium DataFrame:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a38e2809a28ea41c5db98a19e3b9454/href">https://medium.com/media/4a38e2809a28ea41c5db98a19e3b9454/href</a></iframe><p>In this function:</p><ul><li>We cast the Kafka value field to a string and parse it as JSON using the debezium_schema.</li><li>We extract relevant fields such as the database and table names, operation type, and the data before and after the change.</li><li>We determine if a record has been deleted (_debezium_deleted) and extract the timestamp of the operation.</li><li>We generate the S3 path dynamically based on the source, database, table, and event date.</li></ul><blockquote><strong>Note: </strong>Since we are moving from Fivetran to Debezium, we decided to adopt a similar approach to the _fivetran fields by defining _debezium_deleted and _debezium_updated.</blockquote><h4><strong>Extracting DataFrames with the Correct Schema</strong></h4><p>After processing the Debezium DataFrame, we need to extract the data using the correct schema:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fb73a9d03be467aa45ef3a290781aae/href">https://medium.com/media/1fb73a9d03be467aa45ef3a290781aae/href</a></iframe><p>In these methods:</p><ul><li><strong>extract_schema</strong>: Attempts to retrieve the schema for the dataset. If it cannot find the schema, it falls back to inferring the schema from the data.</li><li><strong>extract_data_df</strong>: Parses the JSON value field using the inferred schema and extracts the data fields along with any extra fields needed.</li></ul><h4><strong>Writing Data to S3</strong></h4><p>We define a batch_writer function to write each batch of data to S3:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e71e76ebdf42144729998fd88a09694e/href">https://medium.com/media/e71e76ebdf42144729998fd88a09694e/href</a></iframe><p>In this function:</p><ul><li>We cache the batch DataFrame for performance.</li><li>We identify all unique S3 paths based on the s3_field.</li><li>For each path, we filter the DataFrame and write the data to the corresponding S3 location in Parquet format.</li><li>We log the dataset information so that later we know which datasets can be forwarded from raw data to Iceberg.</li></ul><h4><strong>Configuring Kafka Credentials</strong></h4><p>To connect to Kafka securely, we define a KafkaCredentials model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bed684a3d28a085aa7141195cd2f3a7a/href">https://medium.com/media/bed684a3d28a085aa7141195cd2f3a7a/href</a></iframe><p>In this model:</p><ul><li>We use AWS Secrets Manager to retrieve the Kafka credentials securely.</li><li>The options method provides the necessary Kafka consumer options for Spark to connect.</li></ul><h4><strong>Reading from Kafka and Starting the Streaming Query</strong></h4><p>We define a KafkaDriver class to handle reading from Kafka and writing the streaming data:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/709242dbce3f37ef195b8409f8b96e97/href">https://medium.com/media/709242dbce3f37ef195b8409f8b96e97/href</a></iframe><p>In this class:</p><ul><li><strong>read_stream</strong>: Sets up the streaming DataFrame from Kafka using the provided credentials and topic pattern.</li><li><strong>write_batch_stream</strong>: Starts the streaming query, processing data every five minutes, and uses the batch_writer to handle each micro-batch.</li></ul><p><strong>Summary of the Process</strong></p><ol><li><strong>Authentication and Connection</strong>: We authenticate to Kafka using credentials retrieved securely from AWS Secrets Manager and configure Spark with the necessary options.</li><li><strong>Reading from Kafka</strong>: We use Spark Structured Streaming to read data from Kafka topics, subscribing via a pattern to consume multiple topics.</li><li><strong>Schema Retrieval</strong>: We retrieve the table schemas from the source databases using SQLAlchemy’s inspector. This allows us to parse the JSON payloads from Debezium correctly.</li><li><strong>Processing Data</strong>: We parse the Debezium messages, extract relevant fields, and prepare the data for writing to S3.</li><li><strong>Dynamic S3 Paths</strong>: We construct the S3 paths dynamically based on the source database, table, and event date, ensuring that data is organized appropriately in S3.</li><li><strong>Writing to S3</strong>: We write the data to S3 in Parquet format, organizing it into partitions for efficient storage and later processing.</li></ol><p>By following this approach, we efficiently transfer data from Kafka to S3, preparing it for further processing and integration with Apache Iceberg and Snowflake.</p><h3>Conclusion</h3><p>In this article, we detailed how we forward Debezium data from Kafka to Amazon S3 using Spark Structured Streaming. By retrieving source schemas with SQLAlchemy and dynamically constructing S3 paths, we ensure that our data is accurately processed and organized for downstream consumption.</p><p><strong>What’s Next</strong></p><p>While writing this article, I become aware that, just the Kafka to S3 was enough material to fit one post, so in favor of not making a long and boring one, I’ve decided to continue in the new one where we will cover:</p><ul><li><strong>Curating the Iceberg Files</strong>: How we organize and optimize the data stored in S3 using Iceberg.</li><li><strong>Accessing Iceberg from Snowflake</strong>: How we integrate Snowflake with our Iceberg tables, enabling seamless querying of the data through external tables and leveraging DBT models for transformations.</li></ul><p>Feedback is a gift 🎁 — I’d love to hear your thoughts! If you’ve worked with similar solutions, feel free to share your experiences. Let’s connect and explore how we can improve and optimize together.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36bebaeab6e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/reducing-costs-with-debezium-kafka-and-iceberg-a-journey-to-efficient-data-integration-with-36bebaeab6e9">“Reducing Costs with Debezium, Kafka, and Iceberg: A Journey to Efficient Data Integration with…</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MySQL: From Fivetran to Debezium]]></title>
            <link>https://medium.com/albert-franzi/mysql-from-fivetran-to-debezium-b82f1dfb91b7?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/b82f1dfb91b7</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[fivetran]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Sat, 07 Sep 2024 16:43:02 GMT</pubDate>
            <atom:updated>2024-09-10T11:24:10.011Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This post outlines how we transitioned from using MySQL Fivetran connectors to a self-hosted Debezium solution, which led to a $250,000 reduction in our annual expenses.</blockquote><h3>Context</h3><p>At <a href="https://lokalise.com/">Lokalise.com</a>, we manage multiple <a href="https://fivetran.com/docs/connectors/databases/mysql">MySQL</a> instances responsible for handling all translation processes (both l10n and i18n). As the volume of text data transferred between our MySQL databases and our Snowflake instance grew up, we relied on Fivetran’s MySQL connector for Change Data Capture (CDC). However, Fivetran’s pricing model, based on <strong><em>Monthly Active Rows (</em></strong><a href="https://fivetran.com/docs/usage-based-pricing"><em>MAR</em></a><strong>)</strong>, quickly became unsustainable as our customer base expanded. It was when we lost a significant discount in the yearly renewal, when our bill increased sixfold 💸.</p><p>In search of a cost-effective alternative, we decided to self-host <a href="https://debezium.io/"><strong>Debezium</strong></a> on our <strong>EKS (Kubernetes)</strong> cluster. This allowed us to directly capture all CDC events from MySQL binlogs, bypassing the rising costs associated with Fivetran.</p><blockquote><strong>Note</strong>: Our current costs are around $700/month.</blockquote><h3>The infrastructure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yKoOplf56UBeNGIyCSx14Q.jpeg" /><figcaption>Pipeline overview E2E</figcaption></figure><blockquote>Before diving into the details of deploying and connecting Debezium, it’s important to note that we already have an <strong>EKS</strong> (Kubernetes) cluster in place, managed by <a href="https://argo-cd.readthedocs.io/en/stable/"><strong>ArgoCD</strong></a><strong> </strong>for Helm-based deployments.</blockquote><p>This article won’t cover the setup of these components but will instead focus on:</p><ul><li>Deploying Apache Kafka (MSK) with SCRAM authentication and Access Control Lists (ACLs).</li><li>Deploying Debezium connectors using <strong>Strimzi</strong> on Kubernetes.</li><li>Storing data in <strong>S3</strong> and forwarding it to <strong>Iceberg (Glue)</strong> and <strong>Snowflake</strong>.</li></ul><h3>1. Deploying a Kafka Cluster MSK</h3><p>Over the past few years, AWS has significantly improved <strong>MSK (Managed Streaming for Apache Kafka)</strong>, making it a reliable and relatively affordable option for running Kafka in the cloud. Given its feature set and maturity, we decided to adopt it for our CDC pipeline.</p><h4><strong>Deploying MSK Using Terraform</strong></h4><p>To simplify the deployment, we leveraged the <strong>AWS-provided Terraform module</strong> (<a href="https://registry.terraform.io/modules/terraform-aws-modules/msk-kafka-cluster/aws/latest">terraform-aws-modules: msk-kafka-cluster</a>), allowing us to spin up a MSK cluster quickly and consistently.</p><blockquote><strong>Note:</strong> We highly recommend configuring the MSK cluster to <strong>not be exposed to the public internet</strong> for security reasons.</blockquote><h4><strong>Authentication and Authorization</strong></h4><p>For client authentication, we enabled both IAM and SCRAM mechanisms. The <strong>IAM authentication</strong> is used to enable the <strong>Kafka Terraform Provider</strong>, while <strong>SCRAM</strong> is used to specify user ACLs.</p><p>The <a href="https://registry.terraform.io/providers/mongey/kafka"><strong>Kafka Terraform Provider</strong></a> is configured with the necessary permissions, as we granted MSK access to our Terraform IAM role, enabling authentication and authorization for <strong>ACL resources </strong>creation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18757394b698401179dfd72c273e17e3/href">https://medium.com/media/18757394b698401179dfd72c273e17e3/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93c191de5d1cfb3710018bc043eb29ef/href">https://medium.com/media/93c191de5d1cfb3710018bc043eb29ef/href</a></iframe><h4><strong>Configuring Kafka ACLs</strong></h4><p>We set up two distinct ACL modules (<strong><em>see</em></strong><em>: </em><a href="https://docs.confluent.io/platform/current/security/authorization/acls/overview.html"><em>Kafka ACL docs</em></a>):</p><ul><li><strong>ACL Consumers:</strong> Read-only access, defined with a topic prefix (e.g., lok.mysql).</li><li><strong>ACL Producers:</strong> Write access, also defined with a topic prefix (e.g., lok.mysql).&quot;</li></ul><p>In addition to defining these ACLs, each <strong>Kafka User ACL</strong> must be stored in <strong>AWS Secrets Manager</strong>. This allows us to securely manage user credentials and associate them with the MSK cluster during instantiation (<strong><em>see</em></strong><em>: </em><a href="https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html"><em>aws docs</em></a><em>)</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e07746178f3617326f7c099167fb7ed/href">https://medium.com/media/4e07746178f3617326f7c099167fb7ed/href</a></iframe><p>The following code snippet demonstrates how to register a new Kafka user by generating a random secret in <strong>AWS Secrets Manager</strong>, which can be associated with the MSK instantiation. Refer to the AWS documentation for details on <a href="https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html#msk-password-limitations"><strong><em>MSK limitations</em></strong></a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67e36a70dd6746cc17c29beaa6edfe22/href">https://medium.com/media/67e36a70dd6746cc17c29beaa6edfe22/href</a></iframe><blockquote><strong>Note: </strong>These modules are based on topic prefixes for simplicity, but they can be adapted according to your specific business requirements.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e537afe82e941178f75aecaa0130262/href">https://medium.com/media/3e537afe82e941178f75aecaa0130262/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/776aaef3af86e3932ed89ca0d02f1dec/href">https://medium.com/media/776aaef3af86e3932ed89ca0d02f1dec/href</a></iframe><h3>2. Deploying our Strimzi Debezium connectors</h3><p>Strimzi is an excellent project that enables us to run Apache Kafka clusters on Kubernetes. To deploy Debezium connectors, we first need to deploy the <a href="https://artifacthub.io/packages/helm/strimzi/strimzi-kafka-operator">Strimzi-kafka-operator</a>, allowing us to later manage the connectors with simple Helm charts.</p><h4>Key Strimzi Components</h4><p>To enable Debezium on Strimzi, we needed to deploy the following components:</p><ul><li><strong>KafkaConnect: </strong>Manages the configuration of the Kafka connectors (<strong><em>see</em></strong><em>: </em><a href="https://github.com/strimzi/strimzi-kafka-operator/blob/0.40.0/helm-charts/helm3/strimzi-kafka-operator/crds/041-Crd-kafkaconnect.yaml"><em>schema</em></a>)</li><li><strong>KafkaConnector: </strong>Defines and manages the individual connectors themselves (<em>see: </em><a href="https://github.com/strimzi/strimzi-kafka-operator/blob/0.40.0/helm-charts/helm3/strimzi-kafka-operator/crds/047-Crd-kafkaconnector.yaml"><em>schema</em></a>)</li></ul><p>One major advantage of <strong>KafkaConnect</strong> is the ability to specify the set of <strong>plugins</strong> it supports. In our case, we needed to include the <strong>Debezium MySQL plugin</strong>. Upon deploying KafkaConnect, Strimzi automatically initiates a build pod, which downloads the necessary plugins, builds, and publishes a new <strong>ECR image</strong>. The running pod will start with these plugins pre-installed.</p><blockquote><strong>Note</strong>: The pod ServiceAccount will need to be associated with IAM permissions so it can push the image to the registry.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/246d7142d8706a01eaba064254bcc527/href">https://medium.com/media/246d7142d8706a01eaba064254bcc527/href</a></iframe><h4>Deploying Debezium Connectors</h4><p>Once the <strong>KafkaConnect</strong> service is up and running, you can deploy your Debezium connectors via Helm charts. The following is an outline of our typical deployment flow:</p><ol><li>Define your <strong>KafkaConnector</strong> configuration (e.g., specifying source databases, topics, and data serialization formats).</li><li>Apply the configuration to the cluster using the Strimzi <strong>KafkaConnector</strong> resource.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b55a5488ca3793054d8d22d2a073dbdf/href">https://medium.com/media/b55a5488ca3793054d8d22d2a073dbdf/href</a></iframe><blockquote><strong>Note</strong>: To visualize cluster resources and inspect your topics, I highly recommend deploying the <a href="https://docs.kafka-ui.provectus.io/">Kafka-UI</a> tool by <a href="https://medium.com/u/8f04d5ae8d70">Provectus</a> (<strong>see</strong>: <a href="https://github.com/provectus/kafka-ui-charts">helm chart</a>).</blockquote><h3>3. Monitor Debezium</h3><p>When working with large volumes of critical data, monitoring becomes essential to ensure the system is running smoothly and to receive alerts when issues arise.</p><p><strong>KafkaConnect</strong> includes a feature that allows exporting metrics to <strong>Prometheus</strong> via the <strong>jmx_prometheus_exporter</strong>. This makes it easy to collect and visualize important metrics about your KafkaConnect and Debezium setup.</p><p>The <strong>Strimzi</strong> GitHub project provides several useful <a href="https://github.com/strimzi/strimzi-kafka-operator/tree/main/examples/metrics">examples</a> that you can follow to implement this.</p><p>One of the most important metrics to track is the <strong>Time Behind Source</strong>, which can be visualized using <strong>Grafana</strong>. This metric shows the delay between when data is captured and when it is processed, which is critical for ensuring low-latency data streaming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDk36W0W9jUi-EAWCNBw3g.png" /><figcaption><strong>Grafana graph</strong>: Time Behind Source</figcaption></figure><p>To enable Prometheus metrics collection, you’ll need to annotate the <strong>KafkaConnect</strong> pod with <strong>Prometheus annotations</strong> so that the Prometheus server can discover and scrape the metrics.</p><p>Additionally, you will need to configure the <strong>JMX exporter</strong> by defining the <strong>jmx-exporter-config</strong>, which ensures that Kafka and Debezium metrics are exposed to Prometheus in a clean and structured format.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b019a2eb767888b17ecd12c0dac384d2/href">https://medium.com/media/b019a2eb767888b17ecd12c0dac384d2/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d48cc0bfbb2456cbf869d67f5d59be6/href">https://medium.com/media/3d48cc0bfbb2456cbf869d67f5d59be6/href</a></iframe><h3>4. Spark Streaming to S3</h3><p>In the <strong>next</strong> article, I will cover Spark Streaming and the integration with Iceberg and Snowflake. By splitting the topics across articles, I aim to keep each post focused on a specific area, making it easier to digest without becoming too lengthy 🤓.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F0NXLAm4Api4LgDFb6RJQw.jpeg" /></figure><p>Given the importance of security and performance in this area, I’ve included some key notes and recommendations to enhance these aspects in your deployment.</p><h3>🔒Security notes</h3><h4>Securing the MSK Cluster</h4><p>When deploying MSK, ensuring that your cluster is secure is paramount. Here are several best practices to consider:</p><ol><li><strong>Network Isolation</strong>:<br>It is crucial to ensure that your MSK cluster is not exposed to the public internet. Always deploy MSK within a <strong>private subnet</strong> and use <strong>VPC endpoints</strong> to control access to the cluster. This helps limit exposure and reduces the risk of unauthorized access.</li><li><strong>Encryption</strong>:<br><strong>Enable encryption at rest</strong> and <strong>in transit</strong> for all data handled by MSK. AWS MSK natively supports encryption using <strong>AWS KMS</strong>, so you can ensure that all sensitive data is securely encrypted before being written to storage. Also, ensure that <strong>TLS</strong> (Transport Layer Security) is enabled for encryption in transit between Kafka brokers, producers, and consumers.</li><li><strong>IAM Access Control</strong>:<br>AWS MSK integrates with IAM to control who can create, modify, or delete resources in your Kafka cluster. By enforcing <strong>least-privilege IAM roles</strong>, you can restrict access to MSK and its resources, ensuring that only authorized users and services can interact with Kafka topics.</li><li><strong>Monitoring and Alerts</strong>:<br>Set up <strong>AWS CloudWatch metrics</strong> and <strong>Kafka-specific alerts</strong> to monitor the health of your MSK cluster in real-time. Configure alarms for key metrics such as <strong>disk usage</strong>, <strong>consumer lag</strong>, and <strong>broker CPU utilization</strong>. Monitoring helps ensure you are alerted to performance degradation or potential security threats, such as unauthorized access attempts.</li></ol><h4>Kafka Access Control Lists (ACLs)</h4><p><strong>Kafka ACLs</strong> are vital for managing who can produce and consume messages in your Kafka topics. Here’s how you can strengthen security at the Kafka level:</p><ol><li><strong>Granular Access Control</strong>:<br>Define granular permissions using <strong>prefix-based ACLs</strong> to restrict access to specific topics. This minimizes the risk of unauthorized data access. For example, an ACL might grant read-only access to topics that begin with lok.mysql, ensuring only consumers with proper credentials can access these topics.</li><li><strong>Client Authentication</strong>:<br>Ensure that <strong>SCRAM</strong> or <strong>Mutual TLS (mTLS)</strong> is enabled for all clients interacting with your Kafka cluster. This ensures that both the client and the server mutually authenticate each other, preventing unauthorized clients from accessing your topics.</li><li><strong>Periodic Audit</strong>:<br>Regularly audit your Kafka ACL configurations to ensure that only necessary permissions are granted. Over time, permissions can accumulate, and unused ACLs may become a security risk if not revoked.</li></ol><h3>Performance Tuning for Kafka and Debezium Notes</h3><p><strong>Connector Task Scaling</strong>:<br>The <strong>tasksMax</strong> configuration in Debezium allows you to parallelize the workload by running multiple tasks for a single connector. For high-throughput environments, increase the number of tasks to distribute the load across multiple nodes in the Kafka cluster.</p><blockquote><strong>Note</strong>, because the MySQL connector always uses a single task, increasing it will has no effect, but check the documentation for other engine connectors.</blockquote><p><strong>Tuning Kafka Partitions</strong>:<br>For large datasets or when dealing with many concurrent producers and consumers, adjust the <strong>number of partitions</strong> for your Kafka topics. More partitions allow Kafka to process data in parallel, improving throughput and reducing lag in high-traffic scenarios.</p><p><strong>Connector Offsets and Retention Policies</strong>:<br>To ensure fault tolerance and data consistency, Kafka stores <strong>offsets</strong> for each connector. You can fine-tune the <strong>offset retention period</strong> by adjusting the Kafka retention policy, ensuring that connector states are maintained even in the event of a failure.</p><p><strong>Resource Allocation for KafkaConnect</strong>:<br>When running <strong>KafkaConnect</strong> on Kubernetes, allocate sufficient <strong>CPU</strong> and <strong>memory</strong> resources to avoid bottlenecks. Monitor your pod performance with tools like <strong>Prometheus</strong> and set up autoscaling for Kubernetes pods if necessary.</p><h4>Links of Interest</h4><ul><li>Great <a href="https://medium.com/@howdyservices9/cdc-using-debezium-in-kubernetes-f41448b7f5db">article</a> covering Debezium in Kubernetes by <a href="https://medium.com/u/c1fa25ca701">OSDS</a>.</li><li><a href="https://docs.confluent.io/platform/current/security/authorization/acls/overview.html">Kafka Access Control Lists</a> (ACLs).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b82f1dfb91b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/mysql-from-fivetran-to-debezium-b82f1dfb91b7">MySQL: From Fivetran to Debezium</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Democratizing the DBT Docs and the Elementary Quality Reports]]></title>
            <link>https://medium.com/albert-franzi/democratizing-the-dbt-docs-and-the-elementary-quality-reports-2f11fed68d13?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f11fed68d13</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[elementary]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Thu, 18 Apr 2024 09:23:12 GMT</pubDate>
            <atom:updated>2024-04-18T09:23:12.380Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s fast-paced digital world, having immediate access to clear, accurate data documentation and quality reports isn’t just nice to have — it’s essential. Whether you’re a data analyst looking for the latest data transformations or a team leader needing assurance on data quality, quick and secure access to this information can drastically improve decision-making and operational efficiency across the company.</p><p>In this article, we’re diving into a practical solution to automate and secure the delivery of <a href="https://docs.getdbt.com/reference/commands/cmd-docs">dbt docs</a> and <a href="https://www.elementary-data.com/">Elementary data</a> quality reports. By using Airflow for orchestration, Amazon S3 as static content storage, and Nginx hosted on EKS to serve content within our VPN, we create a robust system that not only safeguards your data but also makes it readily accessible to the entire company through a secure connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QORMGwFPQfqW797fh3wjQA.jpeg" /></figure><h4>Background and Tools Overview</h4><p>To truly democratize data within a company, it’s essential to understand and effectively utilize the tools that enable such accessibility. In this article, we focus on a toolkit that integrates <a href="https://docs.getdbt.com/reference/commands/cmd-docs">dbt</a>, <a href="https://www.elementary-data.com/">Elementary data</a> quality, Airflow, Amazon S3, and Nginx within an Amazon EKS. Here’s how each component contributes to a streamlined data reporting system:</p><ul><li><strong>dbt docs</strong>: It generates and provides a complete catalog about our DBT project, with the lineage and internal dependencies between all our models and business logic.</li><li><strong>elementary-data</strong>: In addition to being a DBT plugin that consumes all the execution and test metrics from our DBT executions, elementary also provides a set of capabilities (i.e. anomaly tests detection, performance metrics, etc), being the generation of a quality report one of them.</li><li><strong>nginx:</strong> deployed within an Amazon EKS (Elastic Kubernetes Service) cluster, acts as a web server to serve the static sites generated by dbt and Elementary.</li><li><strong>s3: </strong>provides a secure and scalable way to store and retrieve any amount of data. In our solution, it serves as the hosting platform for the static content generated by dbt and Elementary, while also it secures the access behind a set of private network policies.</li><li><strong>airflow</strong>: the orchestrator tool which we will schedule all the work to be done by dbt and Elementary relying on the <a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html#kubernetespodoperator">K8sPodOperator</a>.</li></ul><blockquote>By removing manual interventions and centralizing data access, this setup not only increases efficiency but also empowers all team members with the knowledge they need to drive the company forward.</blockquote><h3>The process</h3><h4>1. DBT Catalog</h4><p>As we mentioned above, we love the K8sPodOperator, that why encapsulate all our DBT models with our packages into an ECR image every time we merge our code to the main branch.</p><p>Therefore, we provide the following script which we will execute after our dbt build task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x89on777CuWXUaEsHNREoQ.png" /><figcaption>DBT Airflow Graph</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e9f569ab7345d330f529d3cf3f957ae/href">https://medium.com/media/6e9f569ab7345d330f529d3cf3f957ae/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1831eafb446f4abdb23559f86fb92803/href">https://medium.com/media/1831eafb446f4abdb23559f86fb92803/href</a></iframe><p>The build_dbt_operator method is just a wrapper we define on top of the KubernetesPodOperator to inject the DBT docker image, the env vars, the resource requests, and the service account.</p><h4>2. The S3 catalog hosting</h4><p>When hosting sensitive data documentation such as dbt docs on S3, it’s crucial to implement robust security measures to ensure that the data remains protected and accessible only to authorized users.</p><p>One effective security measure is to restrict access to the S3 bucket by filtering based on IP addresses, in our case by the NAT Gateway IPs to which the Nginx K8s pod is defined.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3db2458e03250bc3489deb45f99a7ec/href">https://medium.com/media/a3db2458e03250bc3489deb45f99a7ec/href</a></iframe><h4>3. Nginx routing the VPN private traffic</h4><p>Once we have our DBT catalog and the <a href="https://docs.elementary-data.com/oss/guides/generate-report-ui">Elementary Observability report</a> in S3, we will proceed with deploying an Nginx pod to proxy the internal request within the VPN to our static websites.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f76ef525e0703f4e696b7979ffa12e7d/href">https://medium.com/media/f76ef525e0703f4e696b7979ffa12e7d/href</a></iframe><p>Our Nginx helm contains the <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/">ALB</a> annotations required to provide two DNSs for our resources accessible only within the private network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dRDRevz9T6lX51BFA7sChg.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4385b13fc6f35851d45744067e306b48/href">https://medium.com/media/4385b13fc6f35851d45744067e306b48/href</a></iframe><p>Since we are using <a href="https://argo-cd.readthedocs.io/en/stable/">ArgoCD</a> for deploying our helms we will inject all the ${variable} when terraforming the <a href="https://registry.terraform.io/providers/oboukili/argocd/latest/docs/resources/application">ArgoCD Application</a>.</p><p>And that’s it, we have our DBT Catalog and Elementary reports ready to be consumed and accessible only within our magic VPN 🚀.</p><h4>Your Insights Are Invaluable</h4><p>As we continue to refine our approach and explore new tools and techniques, your feedback and insights are invaluable. We encourage you to share your experiences:</p><ul><li>Have you implemented similar solutions in your own environments?</li><li>What challenges have you faced during such implementations?</li><li>Do you have suggestions or best practices that might enhance this framework further?</li></ul><p>Please leave your comments below or reach out directly. Your input is crucial for us to learn, adapt, and improve our processes. Together, we can push the boundaries of what’s possible in data and make our data ecosystems not only more efficient but also more secure and compliant.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f11fed68d13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/democratizing-the-dbt-docs-and-the-elementary-quality-reports-2f11fed68d13">Democratizing the DBT Docs and the Elementary Quality Reports</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GA4 raw data to Snowflake]]></title>
            <link>https://medium.com/albert-franzi/ga4-raw-data-to-snowflake-cef363756ad5?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/cef363756ad5</guid>
            <category><![CDATA[ga4]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Fri, 12 Apr 2024 08:57:43 GMT</pubDate>
            <atom:updated>2024-04-12T08:57:43.879Z</atom:updated>
            <content:encoded><![CDATA[<p>From Google Analytics to the Cloud and Beyond: How we successfully integrated GA4 with BigQuery, streamlined Data into S3 and Iceberg, and made it all available in Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEYvYlwDK8xmG-Jecs2gMQ.jpeg" /><figcaption>GA4 data flow into Snowflake</figcaption></figure><h3>Context</h3><p>Welcome to our journey! As we dive into the world of data, it’s crucial to handle it well from start to finish — gathering, storing, and analyzing. We took on a challenge: making Google Analytics 4 (GA4) talk to BigQuery, then neatly tucking that data into Amazon S3 and <a href="https://iceberg.apache.org/">Iceberg</a>, and finally making sense of it all through Snowflake. This integration wasn’t just about keeping our data tidy; it was about unlocking insights that can make a difference.</p><p>In this article, I’ll walk you through how we connected these dots. We’ll explore each step — from the very first data export in GA4 to the deep dives into analysis with Snowflake. I’ll share the tricks and tools that helped us along the way and how each part of our setup contributes to smarter data handling.</p><p>Whether you’re wrestling with similar data challenges or just curious about how it’s done, I hope our story helps and inspires you. Let’s get into how we turned clicks into actionable insights!</p><h3>Journey</h3><h4>1. Connect GA4 with BigQuery</h4><p>Connecting Google Analytics 4 to BigQuery allows you to access and analyze your analytics data with greater flexibility and depth than is possible directly in GA4.</p><blockquote><strong>Note: </strong>Before setting up the export, ensure that you have admin access to both your Google Analytics and Google Cloud Platform accounts. This access is necessary to make the required configurations in GA4 and BigQuery.</blockquote><p><a href="https://support.google.com/analytics/answer/9823238">[GA4] Set up BigQuery Export</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*HvKUphshll_mkId-Sq37xg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*yOxMDI13kWj6N9gZpJpk7A.png" /></figure><p>After you’ve made your selections and reviewed the settings, submit your configuration. Google will start exporting data to the specified BigQuery dataset at the frequency you’ve selected (Daily vs Streaming).</p><blockquote><strong>Note on region selection</strong>: Since we are not using the gcp enterprise plan, we cannot use an aws-omni region, therefore we chose the europe-west3 region to comply with our data storage regulations. Remember, the chosen region should align with your project&#39;s data regulation requirements and geographic location to minimize latency and costs.</blockquote><h4>2. Creating a GCP Service Account for Cross-Platform Authentication</h4><p>Given that our Kubernetes (K8s) cluster is hosted on the AWS platform, we need a mechanism to authenticate and authorize access to the Google Cloud Platform (GCP) resources, including the newly created analytics dataset in BigQuery. By creating a GCP service account, we establish a secure method for cross-platform authentication, enabling seamless access to GCP resources from the AWS side.</p><p>Step-by-step:</p><ul><li><a href="https://cloud.google.com/iam/docs/service-accounts-create">Create service accounts | IAM Documentation | Google Cloud</a></li><li><a href="https://cloud.google.com/iam/docs/manage-access-service-accounts">Manage access to service accounts | IAM Documentation | Google Cloud</a></li></ul><p><strong>Role Assignment</strong>: Assign the ‘<strong>BigQuery User</strong>’ role to the service account to grant it the necessary permissions for executing queries on the analytics dataset.</p><ul><li><a href="https://cloud.google.com/bigquery/docs/control-access-to-resources-iam">Control access to resources with IAM | BigQuery | Google Cloud</a></li><li><a href="https://cloud.google.com/iam/docs/keys-create-delete">Create and delete service account keys | IAM Documentation | Google Cloud</a></li></ul><p>5. Upload GCP Credentials into <a href="https://aws.amazon.com/es/secrets-manager/">AWS Secrets Manager</a>. This step ensures that the key can be retrieved securely by your Kubernetes cluster without storing it in less secure configurations like Kubernetes secrets or directly in container images.</p><h4>3. Retrieving GA4 Datasets from BigQuery</h4><p>For this mission, we will use <a href="https://github.com/googleapis/python-bigquery-pandas">python-bigquery-pandas (pandas_gbq)</a>, allowing us to authenticate with the SA key and access our analytics dataset remotely by downloading it as a Pandas DataFrame.</p><pre>pip install google-api-python-client pandas-gbq</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a896ec8f96935db22192dbacb2e465ed/href">https://medium.com/media/a896ec8f96935db22192dbacb2e465ed/href</a></iframe><p>With the authentication set, connect to BigQuery to run queries. Use the credentials obtained to configure your client.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04b35bc70658fe3dd79da322f3929d1c/href">https://medium.com/media/04b35bc70658fe3dd79da322f3929d1c/href</a></iframe><blockquote><strong>Note: </strong>Since we want to be memory-conservative, we decided to partition the data using the event_timestamp as a reference.</blockquote><blockquote><strong>Alternative: </strong>If we were in <strong>enterprise</strong> plan with GCP, we could opted for creating an omni region and creating an schedule Data Transfer between Bigquery and S3 <a href="https://cloud.google.com/bigquery/docs/omni-aws-create-connection">omni-aws-create-connection</a> &amp; <a href="https://cloud.google.com/bigquery/docs/omni-aws-export-results-to-s3">omni-aws-export-results-to-s3</a>.</blockquote><h4>4. Validating GA4 Schema Using Pandera and Pydantic</h4><p>The purpose of data validation in our pipeline is crucial. It ensures that the data extracted from GA4 and processed through BigQuery meets our quality standards before it’s used for further analysis or stored in other systems. This step helps prevent issues related to data quality, such as incorrect data types, missing values, or format inconsistencies, which can significantly impact downstream processes and decision-making.</p><p>To implement robust data validation, we use Pandera in conjunction with Pydantic. <a href="https://pandera.readthedocs.io/en/stable/">Pandera</a> provides a flexible and expressive API for DataFrame validation, making it easy to define schema constraints and check data. <a href="https://pydantic.dev/">Pydantic</a> adds an additional layer of type checking and data parsing, which is particularly useful for handling complex and nested data structures typical in GA4 data.</p><blockquote><strong><em>Literature: </em></strong><a href="https://support.google.com/analytics/answer/7029846">[GA4] BigQuery Export schema</a><em> &amp; </em><a href="https://support.google.com/analytics/answer/12769371">[GA4] BigQuery Export user-data schema</a>.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b4e76c789eb9f08b199448830435dcb/href">https://medium.com/media/7b4e76c789eb9f08b199448830435dcb/href</a></iframe><blockquote><strong><em>Improvement: </em></strong><em>To make our data even nicer, we took advantage of the </em><a href="https://docs.pydantic.dev/latest/concepts/validators/">Pydantic Validators</a><em>.</em></blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b22d91387877f9eac928c59a64795270/href">https://medium.com/media/b22d91387877f9eac928c59a64795270/href</a></iframe><p>By incorporating <strong>Pydantic</strong> and <strong>Pandera</strong> for schema validation, you ensure that the data adheres to expected formats and meets the quality standards required for reliable analytics. This setup not only reinforces the integrity of your data pipeline but also automates checks that might otherwise be manual and error-prone, significantly enhancing the robustness and reliability of your data handling processes.</p><h4>5. Uploading Clean Data to S3 + AWS Glue Iceberg Tables</h4><p><a href="https://aws-sdk-pandas.readthedocs.io/en/stable/index.html"><strong>AWS Wrangler</strong></a> is a powerful utility that simplifies data operations on AWS services like S3, Glue, and Redshift. For uploading data to S3 and creating <a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html"><strong>Glue Iceberg</strong></a> tables, AWS Wrangler provides streamlined functions that handle underlying complexities.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39db903f6fc051d5d5bf943a8f65f006/href">https://medium.com/media/39db903f6fc051d5d5bf943a8f65f006/href</a></iframe><p>If the table doesn’t exist yet in the Glue Catalog, awswrangler will create it for us inheriting the schema from the DataFrame.</p><h4>6. Creating an External Iceberg Table in Snowflake</h4><p>The objective of this section is to create an external Iceberg table in Snowflake. This setup allows us to utilize Snowflake’s capabilities to perform advanced analytics on our GA4 data stored in AWS Glue Iceberg tables. By linking Snowflake with AWS Glue, we leverage the best of both platforms: the robust data handling and scalability of AWS and the advanced querying and analytics features of Snowflake.</p><p><strong>a. </strong><a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration"><strong>Create a Catalog Integration</strong></a>: This integration will connect Snowflake to the AWS Glue catalog, allowing Snowflake to understand the metadata of the Iceberg tables stored in AWS Glue.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bba7e98c9ef3c0ee00ea00a639b9498f/href">https://medium.com/media/bba7e98c9ef3c0ee00ea00a639b9498f/href</a></iframe><p><strong>b. </strong><a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-external-volume"><strong>Create an External Volume</strong></a>: This step specifies where Snowflake should look for the data files associated with the Iceberg tables.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b249d729daa206326d113e34941858e0/href">https://medium.com/media/b249d729daa206326d113e34941858e0/href</a></iframe><p><strong>c. </strong><a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-create"><strong>Creating the Iceberg Table in Snowflake</strong></a>: Once the integrations are in place, the next step is to create the Iceberg table which will point to the data stored in AWS Glue.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8484efb634a44984af731d978cf7452/href">https://medium.com/media/a8484efb634a44984af731d978cf7452/href</a></iframe><h4>7. Refreshing Snowflake Iceberg Metadata</h4><p>Iceberg tables in Snowflake store metadata about the data files and their organization. As new data is added or existing data is modified in AWS S3 through AWS Glue operations, the metadata stored in Snowflake becomes outdated. Regularly refreshing this metadata ensures that Snowflake queries return complete and up-to-date results.</p><p><strong>[Option A]: Manual Refresh</strong>: For ad-hoc refreshes or initial testing, you can manually trigger the metadata refresh using the following SQL command in Snowflake:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/381429be29542f16a44ca40773de1efe/href">https://medium.com/media/381429be29542f16a44ca40773de1efe/href</a></iframe><p><strong>[Option B]: Automated Refresh with Airflow:</strong> To automate the process and ensure that metadata is refreshed at regular intervals or after specific events (like data uploads), you can set up an Airflow job with the <a href="https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/operators/snowflake.html">SnowflakeOperator</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30940fc00e0211c0a0e257d9da306d4c/href">https://medium.com/media/30940fc00e0211c0a0e257d9da306d4c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aebpqiixkWGmFiZe4eLM1A.png" /></figure><h3>Conclusion and Invitation for Feedback</h3><p>We’ve now walked through the comprehensive process of integrating Google Analytics 4 with BigQuery, migrating data efficiently to AWS S3 using Glue Iceberg, and finally, leveraging the powerful capabilities of Snowflake to manage and analyze that data. This journey underscores our commitment to leveraging state-of-the-art technology to enhance our data workflows, providing actionable insights that drive business decisions.</p><p>As we continue to refine and evolve our data pipeline, we are keen to learn from the community. Your experiences, insights, and feedback are invaluable as they help us identify opportunities for improvement and innovation. Whether you are a data engineer, a business analyst, or just someone passionate about data, we would love to hear your thoughts:</p><ul><li>What has been your experience with similar data integrations?</li><li>Are there other tools or methods you’ve found effective?</li><li>Do you have suggestions on how we might further optimize our current setup?</li></ul><p>Please share your feedback in the comments below or reach out to us directly. Let’s collaborate to make data handling as efficient and insightful as possible.</p><p>Thank you for taking the time to explore this process with us. We look forward to your contributions and to furthering the conversation around advanced data engineering solutions!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cef363756ad5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/ga4-raw-data-to-snowflake-cef363756ad5">GA4 raw data to Snowflake</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing Airflow with AWS Shared Secrets cross-accounts]]></title>
            <link>https://medium.com/albert-franzi/enhancing-airflow-with-aws-shared-secrets-cross-accounts-3353bf263be4?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/3353bf263be4</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[secrets]]></category>
            <category><![CDATA[passwords]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Sun, 04 Feb 2024 21:04:43 GMT</pubDate>
            <atom:updated>2024-02-04T21:04:43.184Z</atom:updated>
            <content:encoded><![CDATA[<h4>Unlocking Cross-Account Secrets in Airflow</h4><p>Managing secrets in cloud environments, especially across multiple AWS accounts, presents a common challenge for developers and DevOps professionals. Apache Airflow, while robust in workflow orchestration, requires enhancements to handle secrets more efficiently in such complex setups. This post delves into integrating a custom AWS Secrets Manager backend with Apache Airflow to achieve seamless cross-account &amp; cross-region secret management.</p><blockquote>The content of this article is also provided in my <a href="https://afranzi.github.io/airflow-aws-shared-secrets/"><strong>Airflow AWS Shared Secrets project</strong></a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LpIMVO7dsnK2Z_DVZXzNGw.png" /></figure><h4>The Challenge</h4><p>Airflow’s default secret management capabilities, though effective within single-account boundaries, falter when workflows span across various AWS accounts. This limitation necessitates a solution that not only simplifies secret management but also adheres to security best practices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-TTALK1WcXygMA-zdxbyKg.jpeg" /></figure><h3>Our Solution: Custom AWS Secrets Manager Backend</h3><h4>Overview</h4><p>We propose a custom backend solution extending Airflow’s native secret management to support cross-account secret retrieval from AWS Secrets Manager. This solution leverages AWS’s robust security features to ensure secure and efficient secret handling.</p><blockquote>In essence, we enhance the <a href="https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html"><strong>SecretsManagerBackend</strong></a> to enable cross-account and cross-region secrets access, thereby providing a more versatile and secure secrets management strategy.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vEM01UoaZQogrVL1kyzsBQ.jpeg" /></figure><h3>Installation</h3><h4>Prerequisites</h4><ul><li><strong>Apache Airflow 2.x:</strong> Ensure Airflow is updated to at least version 2.0.</li><li><strong>AWS Account Access:</strong> You need access to the AWS accounts from which secrets will be fetched <em>( See: </em><a href="https://docs.aws.amazon.com/secretsmanager/latest/userguide/auth-and-access_examples_cross.html"><em>Permissions to AWS Secrets Manager secrets for users in a different account</em></a><em>)</em></li><li><strong>Permissions</strong>: Adequate permissions to manage secrets within AWS Secrets Manager and configure IAM policies.</li><li><strong>Helm:</strong> Familiarity with Helm for deploying and managing Kubernetes applications, as this guide uses the <a href="https://airflow-helm.github.io/charts/">Airflow Helm Chart (Users Community)</a>.</li></ul><h4>The Code</h4><p>Our custom backend expanding the existing SecretsManagerBackend expects additional properties within backend_kwargs in addition to the <a href="https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html#aws-secrets-manager-backend">native ones</a>.</p><ul><li><strong>shared_account</strong>: Specifies the AWS account ID where the primary secrets are stored. This facilitates cross-account secret access.</li><li><strong>aws_region</strong>: Defines the AWS region of the secrets manager, ensuring the backend can retrieve secrets from the specified geographical location.</li><li><strong>connections_prefix_shared: </strong>Prefix for the shared secrets.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/671f5aaa70948c0f5f74b3c149c967c3/href">https://medium.com/media/671f5aaa70948c0f5f74b3c149c967c3/href</a></iframe><p>When fetching secrets from a different region and account we must provide the full secret ARN, otherwise the boto library will expect to find the secret in the existing account &amp; region where the Airflow POD is being executed.</p><p>Once we have the code built into a <a href="https://pypi.org/project/airflow_aws_shared_secrets/">Pypi library</a>, we must continue with the Helm configuration.</p><h4>Helm configuration</h4><p>This documentation leverages the <a href="https://airflow-helm.github.io/charts/">Airflow Helm Chart (Users Community)</a>. The instructions should be universally applicable across different Helm deployments, thanks to the flexibility of environment variables and configuration files.</p><blockquote>Ensure the <strong>airflow-aws-shared-secrets</strong> package is included in the extraPipPackages section of your Helm values. This ensures the custom library is deployed into the Airflow worker pods, enabling them to interact with the custom backend.</blockquote><h4>Configure Airflow to use our Custom Backend</h4><p>To activate the AWS Shared Secrets Backend in Airflow, adjust either the airflow.cfg file directly or set the appropriate environment variables:</p><p>Edit airflow.cfg:</p><pre>[secrets]<br>backend = &#39;airflow_aws_shared_secrets.secret_manager.SharedSecretsManagerBackend&#39;</pre><p>Or, set an environment variable:</p><pre>### [secrets]<br>AIRFLOW__SECRETS__BACKEND: &#39;airflow_aws_shared_secrets.secret_manager.SharedSecretsManagerBackend&#39;<br>AIRFLOW__SECRETS__BACKEND_KWARGS: &#39;{&quot;connections_prefix&quot;: &quot;airflow/connections/&quot;, &quot;connections_prefix_shared&quot; : &quot;airflow/connections/shared&quot;, &quot;shared_account&quot;: &quot;123581321&quot;, &quot;aws_region&quot;: &quot;eu-central-1&quot;}&#39;</pre><h4>Specs Example</h4><pre>apiVersion: v2<br>kind: HelmRelease<br>metadata:<br>  name: airflow<br>spec:<br>  releaseName: airflow<br>  chart:<br>    spec:<br>      chart: airflow<br>      version: 8.8.0<br>  interval: 2h0m0s<br>  install:<br>    remediation:<br>      retries: 3<br>  values: # https://github.com/airflow-helm/charts/blob/main/charts/airflow/values.yaml<br>    airflow:<br>      image:<br>        repository: apache/airflow<br>        tag: 2.8.1-python3.10 # https://hub.docker.com/r/apache/airflow/tags<br>      executor: CeleryExecutor<br>      extraPipPackages:<br>        - &quot;apache-airflow-providers-amazon==8.13.0&quot;<br>        - &quot;airflow-aws-shared-secrets==0.0.5&quot; # https://github.com/afranzi/airflow-aws-shared-secrets<br>      config:<br>        ### ref: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html<br>        ### [core]<br>        ...<br><br>        ### [secrets]<br>        AIRFLOW__SECRETS__BACKEND: &#39;airflow_aws_shared_secrets.secret_manager.SharedSecretsManagerBackend&#39;<br>        AIRFLOW__SECRETS__BACKEND_KWARGS: &#39;{&quot;connections_prefix&quot;: &quot;aily/${tenant_name}/${environment}/airflow&quot;, &quot;shared_account&quot;: &quot;258781458051&quot;, &quot;connections_prefix_shared&quot; : &quot;aily/coreproduct/airflow&quot;, &quot;aws_region&quot;: &quot;eu-central-1&quot;}&#39;<br><br>      ## Extra environment variables for the airflow Pods<br>      ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-environment-variables.md<br>      extraEnv:<br>        ...<br><br>    serviceAccount:<br>      create: true<br>      annotations:<br>        eks.amazonaws.com/role-arn: ${airflow_irsa}</pre><h4>Conclusion</h4><p>By following these steps, we’ve successfully overridden the default AWS connections backend in Airflow with a custom one that allows accessing secrets from other AWS accounts.</p><h4>Implementing Best Practices</h4><p>When configuring and using the custom backend, adhere to the following best practices for security and efficiency:</p><ul><li><strong>Minimal IAM Permissions:</strong> Assign only the necessary permissions to the IAM roles used by Airflow, following the principle of least privilege.</li><li><strong>Secure Secret Storage:</strong> Ensure that all secrets stored in AWS Secrets Manager are encrypted at rest using keys managed by AWS KMS.</li><li><strong>Regular Audits:</strong> Periodically review AWS access logs and Airflow access patterns to ensure compliance with security policies.</li></ul><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;secretsmanager:ListSecretVersionIds&quot;,<br>        &quot;secretsmanager:GetSecretValue&quot;,<br>        &quot;secretsmanager:GetResourcePolicy&quot;,<br>        &quot;secretsmanager:DescribeSecret&quot;<br>      ],<br>      &quot;Resource&quot;: [<br>        &quot;arn:aws:secretsmanager:{{ REGION }}:{{ ACCOUNT_ID }}:secret:airflow/connections/*&quot;<br>      ]<br>    }<br>  ]<br>}</pre><h4>Spreading the Knowledge</h4><p>Feedback is a gift, so we encourage you to share your experiences and insights, fostering a community of knowledge-sharing and innovation around Apache Airflow and AWS Secrets Manager.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3353bf263be4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/enhancing-airflow-with-aws-shared-secrets-cross-accounts-3353bf263be4">Enhancing Airflow with AWS Shared Secrets cross-accounts</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unit-testing DBT Postgres]]></title>
            <link>https://medium.com/albert-franzi/unit-testing-dbt-postgres-26e077156f1c?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/26e077156f1c</guid>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Tue, 02 Jan 2024 16:15:16 GMT</pubDate>
            <atom:updated>2024-01-02T16:24:22.046Z</atom:updated>
            <content:encoded><![CDATA[<p>This exploration delves into a less-traversed yet critical aspect of DBT: the sophisticated art of unit-testing our models against diverse and complex scenarios.</p><p>After extensive research into <a href="https://docs.getdbt.com/docs/introduction">DBT documentation</a> and observing a gap in the developer community’s approach to unit-testing, we decided to devise a straightforward method to ensure the reliability of our DBT models.</p><p>It is widely acknowledged that a key advantage of DBT is its robust <a href="https://docs.getdbt.com/docs/build/data-tests">testing capabilities</a>. These features facilitate the enforcement of various constraints and enable the crafting of custom tests to verify specific business rules within our models.</p><p>However, it’s crucial to note that these tests run at the time of execution, indicating that our code — and potentially any hidden bugs — have already been deployed.</p><blockquote>Numerous organizations have integrated DBT testing into their CI/CD pipelines by connecting to a real database environment, using either dummy or actual production data.</blockquote><p>Drawing from my extensive experience as a Data Engineer, particularly my proficiency with Spark and Kafka, I have always prioritized writing unit tests for new developments. This practice prompted a significant inquiry when I began working with DBT: <strong>Why not apply the same rigorous unit-testing methodology to our DBT SQL models?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pMtv_xxhuZAF23uLFCIs7A.jpeg" /></figure><h4>So, how are we going to test it?</h4><p>To make it feasible, we will take advantage of the following utilities:</p><ul><li>The <a href="https://pypi.org/project/pytest-postgresql/">pytest-postgresql</a> library.</li><li><a href="https://docs.getdbt.com/reference/programmatic-invocations">dbt runner — Programmatic invocations</a>.</li><li>dbt commands — <a href="https://docs.getdbt.com/reference/commands/parse">parse</a>, <a href="https://docs.getdbt.com/reference/commands/seed">seed</a>, <a href="https://docs.getdbt.com/reference/commands/run">run</a>, <a href="https://docs.getdbt.com/reference/commands/test">test</a>.</li><li>dbt <a href="https://docs.getdbt.com/reference/node-selection/syntax">node-selection</a>.</li></ul><h4>0. Awesome, let’s start!</h4><p>We begin by leveraging the <a href="https://pypi.org/project/pytest-postgresql/">pytest-postgresql</a> library, enabling us to validate our DBT Postgres projects without relying on a pre-existing database environment.</p><p>Let’s start by providing the “inject fixture” to our code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa44017f5d2c23559903a962a3f3e104/href">https://medium.com/media/aa44017f5d2c23559903a962a3f3e104/href</a></iframe><p>Plus initializing the Postgres process we will be using in our unit tests.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa548fdd094b1f6930fbb6c401c6805e/href">https://medium.com/media/fa548fdd094b1f6930fbb6c401c6805e/href</a></iframe><h4>1. Wait! Well done, dbt-community! 🤩</h4><p>In my <a href="https://medium.com/albert-franzi/dbt-core-airflow-7d94edac9cdf">previous post</a>, we used the <a href="https://docs.python.org/3/library/subprocess.html">subprocess</a> command to run dbt from Python, in addition to parsing the logs ourselves.</p><p>Lucky us! dbt supported programmatic invocations since v1.5, allowing us to add an extra Python layer before executing our dbt code and another one after the execution, enabling us to introspect the dbt logs programmatically.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c080deba296b55656ad48c16dd115fa3/href">https://medium.com/media/c080deba296b55656ad48c16dd115fa3/href</a></iframe><h4>2. Great, so?</h4><p>Once we can execute dbt programmatically, we need to make sure we can recreate the specific dbt lineage to be tested.</p><p>After executing dbt parse, we will get the list of external sources used in our project, and only if we document them with the data types 😉 we will be able to generate the DDLs so we can populate them before running dbt.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79ad8460da8fbc886a348950ddee5bf1/href">https://medium.com/media/79ad8460da8fbc886a348950ddee5bf1/href</a></iframe><p>Therefore copy our CSVs into the sources.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46f0aa4febf041837c92c8c2e700bfbb/href">https://medium.com/media/46f0aa4febf041837c92c8c2e700bfbb/href</a></iframe><h4>3. DB ready sir! 👨‍🍳</h4><p>Great, we already recreated the external sources and injected data, so we can proceed by running dbt taking advantage of the dbt testing feature.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3a4c953d550bf019eda67b2bcd624b5/href">https://medium.com/media/c3a4c953d550bf019eda67b2bcd624b5/href</a></iframe><p>Similar to the parse outcome, we can parse the run and test executions and validate if any model is broken or any defined test is not behaving as expected.</p><h4>5. Show me an example! 👾</h4><p>So far, the code is quite simple and we are providing a way to test our DBT code in an isolated way by simulating different potential scenarios and making sure any future change in code keeps our expected outcomes and our code free of bugs.</p><p>All we will need to provide for each test is:</p><ul><li><strong>sources_to_load</strong>: CSV file plus the source table reference to copy the data to.</li><li><strong>selector</strong>: dbt select statement to validate a subset of models. Useful if you already have too many dbt models, so you can rollout this practice model by model.</li><li><strong>outputs_to_validate</strong>: CSV file plus the target table to fully validate it’s content.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ef66c26548f1a52378601dfdeb55376/href">https://medium.com/media/2ef66c26548f1a52378601dfdeb55376/href</a></iframe><p>That’s all! With some code, we enabled us to unit-test our dbt models with different input files and we can guarantee to our data consumers we keep our dbt quality on track.</p><p><strong>Feedback is a gift</strong>, we hope that by sharing our DBT testing experience we helped you improve your DBT quality too.</p><p>We encourage you to share your experiences 💌 and approaches to DBT model testing and validation. Your feedback is invaluable, and we look forward to learning from your insights.</p><p>The code is published on <a href="https://github.com/afranzi/pytest-dbt-postgres">GitHub</a> and <a href="https://pypi.org/project/pytest-dbt-postgres/">Pypi</a>.</p><p><a href="https://github.com/afranzi/pytest-dbt-postgres">GitHub - afranzi/pytest-dbt-postgres</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26e077156f1c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/unit-testing-dbt-postgres-26e077156f1c">Unit-testing DBT Postgres</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DBT Core & Airflow]]></title>
            <link>https://medium.com/albert-franzi/dbt-core-airflow-7d94edac9cdf?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d94edac9cdf</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Mon, 19 Dec 2022 10:46:40 GMT</pubDate>
            <atom:updated>2022-12-19T10:46:40.718Z</atom:updated>
            <content:encoded><![CDATA[<h4>Empowering our data organization with DBT Core, VS Code &amp; Airflow</h4><p>This article aims to explain how we are using DBT at <a href="http://superside.com/">Superside</a> and how we successfully transitioned from using DBT Cloud to using DBT Core, <a href="https://code.visualstudio.com/">VS Code</a> &amp; Airflow while improving our development experience.</p><p>Our first approach at Superside was using DBT Cloud (IDE &amp; Jobs). Which was a good starting point since we didn’t need any extra resources and it was straightforward to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Ry2PIsLonnEdEHKHr7NOQ.png" /></figure><p>The approach was composed of a Gitlab CI/CD step sending an API call to DBT Cloud Jobs on a successful Pull Request merge, plus our Daily Scheduled jobs in DBT Cloud.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3e4756196a865d4a69f2af8852fa8bf/href">https://medium.com/media/b3e4756196a865d4a69f2af8852fa8bf/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b803992ceae7358e8789f4843cc1d59e/href">https://medium.com/media/b803992ceae7358e8789f4843cc1d59e/href</a></iframe><p>However, the <a href="https://www.getdbt.com/pricing/">Team plan</a> doesn&#39;t include SSO, and all Gitlab interactions were attributed to the first user that configured the integration, so as we kept growing from 4 Data members to 10, we reached the point where we required something more secure and scalable. For this reason, we reached to the DBT team, who told us the Enterprise plan costs $300/seat (previously $50/seat for the Team plan, now $100/seat), which was more expensive than expected.</p><blockquote>In addition, we believe SSO shouldn’t be a <strong>luxury</strong> feature, therefore SSO should be provided in all plans. Reason why <a href="https://www.getdbt.com/">getDbt</a> should be listed on <a href="https://sso.tax/">the SSO Wall of Shame</a> page.</blockquote><p>So, we planned a way out of DBT Cloud to improve our DBT experience &amp; security.</p><h3>Moving away from DBT Cloud</h3><p>Our plan on moving away from DBT Cloud was quite simpler, prepare the <strong>local setup</strong> and get ready a <strong>DAG in Airflow to execute our DBT jobs</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FsAhnhR75swnrc7AB_3IDQ.png" /><figcaption>DBT Local + Airflow jobs</figcaption></figure><h4>Developing DBT locally</h4><p>Configuring DBT locally was easier than expected, replacing DBT Cloud IDE without any regrets.</p><ol><li>For the local setup we just needed:</li></ol><ul><li><a href="https://code.visualstudio.com/">Visual Studio Code</a> (IDE)</li><li><a href="https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user">dbt Power User (VS code) plugin</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=samuelcolvin.jinjahtml">Better Jinja (VS code) plugin</a></li><li><a href="https://github.com/z3z1ma/dbt-osmosis">dbt-osmosis python package</a></li></ul><p>2. Then configure the <a href="https://docs.getdbt.com/docs/get-started/connection-profiles">DBT local profile</a> in <em>~/.dbt/profiles.yml.</em></p><pre>default:<br>  target: dev<br>  outputs:<br>    dev:<br>      type: snowflake<br>      account: xxx12358<br>      role: DATA_DEV_ROLE<br>      user: &quot;{{ env_var(&#39;DBT_USER&#39;) }}&quot;<br>      password: &quot;{{ env_var(&#39;DBT_PASSWORD&#39;) }}&quot;<br>      database: ANALYTICS_DEV<br>      warehouse: DATA_DEV_WAREHOUSE<br>      schema: &quot;dbt_{{ env_var(&#39;DBT_USER&#39;) }}&quot;<br>      threads: 1<br>      client_session_keep_alive: False<br>      query_tag: local<br><br>      # optional<br>      connect_retries: 5 # default 0<br>      connect_timeout: 10 # default: 10<br>      retry_on_database_errors: True # default: false<br>      retry_all: False  # default: false</pre><p>3. Once we open our project with VS Code, it will detect automatically our profile and it will index our DBT code, allowing us to navigate, compile, run &amp; test our models with just one click.</p><p>Same for discovering parent and child models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*hLetVLbrHdL_E7hzRNGZqg.png" /></figure><p>4. Validate DBT Connection</p><p>DBT comes with a <a href="https://docs.getdbt.com/reference/commands/debug">debug</a> command that will help us validating our profile and our connection.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2d63950449b1e587797b13ad9522e8c/href">https://medium.com/media/a2d63950449b1e587797b13ad9522e8c/href</a></iframe><h4>Running DBT Jobs in Airflow</h4><p>For this section, we wanted to emulate first what we had in DBT Cloud and then start breaking it into pieces to make it more usable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AxYao2qiKjQ_91FBfNQ6hg.png" /></figure><h4>1. Publish DBT Catalog</h4><p>The first step consisted of publishing the DBT Catalog to an S3 bucket, so we could use the static content as our <a href="https://docs.getdbt.com/docs/collaborate/documentation">DBT documentation</a> page.</p><blockquote><strong>Note:</strong> we strongly recommend configuring the s3 bucket access behind a VPN, so it’s only accessible by your organization (public bucket with <a href="https://repost.aws/knowledge-center/block-s3-traffic-vpc-ip">IpAddress</a> restricted).</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/760d285fe79b68f8d36eded003d14fc3/href">https://medium.com/media/760d285fe79b68f8d36eded003d14fc3/href</a></iframe><h4>2. Build the DBT Docker</h4><p>Since we want to be able to execute our DBT code from Airflow we have two options:</p><ul><li>Push the main code to an S3 folder on each successful merge to the main branch and then provide a Docker image that downloads the S3 DBT code and executes it.</li><li>Build a new docker image with the new DBT code and dependencies.</li></ul><p>In our case, we opted for the second option, where we publish the new images to our <a href="https://aws.amazon.com/ecr/">ECR repository</a> on each successful merge.</p><blockquote>We will schedule an Airflow job to purge old images and keep only the last X images.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e65a8c77c6ee21d743adde0f57001b8/href">https://medium.com/media/2e65a8c77c6ee21d743adde0f57001b8/href</a></iframe><p><strong><em>Note:</em></strong><em> We will cover the poetry section.</em></p><p>For the Gitlab CI/CD side, it will consist of a simple stage that will execute the previous code on main merges.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/837bcab189b8a70271335def665f7753/href">https://medium.com/media/837bcab189b8a70271335def665f7753/href</a></iframe><h4>3. DBT Airflow DAG</h4><p>At Superside, we have <strong>Airflow 2.5.0</strong> running on top of K8s using the <a href="https://artifacthub.io/packages/helm/airflow-helm/airflow">Helm provided by the community</a> with <a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html">IRSA</a> on top of it to handle AWS permissions (required to interact with ECR).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fec3d40e93231bfd45f8f258142386ff/href">https://medium.com/media/fec3d40e93231bfd45f8f258142386ff/href</a></iframe><p>The first Airflow iteration consisted of K8s operators executed sequentially: <strong>seed</strong>, <strong>snapshot</strong>, <strong>run</strong> &amp; <strong>test</strong> sequentially.</p><p>On the second Airflow iteration, we split the “run&amp;tests” between BASE; STG; and DIM+FCT models, so we guarantee that in case of bugs, they don’t propagate to subsequent layers.</p><p>For the <strong>build_k8s_pod_operator</strong>, we provide predefined pod t-shirt sizes + encapsulate the default namespace &amp; service_account to use on them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/869afdf3195bb839f14d50315e9185cb/href">https://medium.com/media/869afdf3195bb839f14d50315e9185cb/href</a></iframe><p>But 🤨… wait, DBT doesn’t have the <strong>dbt_execute</strong> command, and why are you interacting with poetry?</p><p>Well, since the DBT profile is defining the Snowflake User &amp; Password with ENV vars, we didn’t want to specify them in the Airflow code, so we developed an extra layer to:</p><ul><li>Retrieve Snowflake credentials from <a href="https://aws.amazon.com/secrets-manager/">AWS Secrets Manager</a> (again IRSA rights required).</li><li>Improve the error logging Slack integration by coding our own one.</li></ul><p>DBT Cloud comes with a <a href="https://docs.getdbt.com/faqs/accounts/slack">Slack integration</a> that notifies you when an error in the DBT job happens. However, it forces you to go to their platform for seeing the issue, implying extra clicks, and scrolling all the logs down until the errors section.</p><blockquote>This means extra time to access the information, which in the end, it’s bad UX.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mvUVf5QatXvpVGgMOHBIXA.png" /><figcaption>DBT Cloud Slack Error example</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W9kugDKYkH9YiHWSmmDp7w.png" /><figcaption>Superside DBT Error Log Slack — Custom integration</figcaption></figure><p>So, since we are quite lazy, we decided to parse the dbt logs using the <a href="https://docs.getdbt.com/reference/global-configs#log-formatting">--log-format json</a> feature.</p><p>For the extra DBT layer we used:</p><ul><li><a href="https://click.palletsprojects.com/en/8.1.x/">click</a> — To handle the command line.</li><li><a href="https://slack.dev/python-slack-sdk/">Slack SDK</a> — To send the Error slack message.</li><li><a href="https://docs.python.org/3/library/subprocess.html">subprocess</a> — To execute the DBT command lines injecting the DBT env vars and the json logs format.</li><li><a href="https://python-poetry.org/">poetry</a> — To manage the packages and dependencies.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8369223a3176ceec92d372931e74d751/href">https://medium.com/media/8369223a3176ceec92d372931e74d751/href</a></iframe><h4>4. Gitlab CI/CD trigger Airflow DBT DAG</h4><p>Once, the Airflow DAG &amp; the DBT Docker were in place, we just need to move the DBT Cloud trigger to Airflow.</p><p>Since Airflow comes with a Rest API, we start using the <a href="https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_dag_run">Trigger a new DAG run</a> endpoint on successful merges to the main branch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89469f13c487cace2d83b11181547f77/href">https://medium.com/media/89469f13c487cace2d83b11181547f77/href</a></iframe><p>Since we are using Gitlab Runners in our K8s cluster, we could call the Airflow API internally without leaving the k8s cluster. For this reason, we created an <a href="https://airflow.apache.org/docs/apache-airflow/stable/security/api.html#basic-authentication">Airflow account to authenticate our calls</a> properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgW0aQKJfGcACu6Bto0drQ.png" /><figcaption>Airflow DBT DAG with Gitlab Run</figcaption></figure><p>The good thing about using Gitlab CI/CD to trigger the DBT DAG with the Airflow API Call, is that we can add who was the responsible of triggering it by just adding the <em>${CI_COMMIT_AUTHOR} </em>in the DAG Run ID.</p><h4>5. Enjoy DBT</h4><p>And finally, we could enjoy DBT without depending on an external cloud provider who doubled the price from one day to another without adding any extra value to their proposal.</p><h3>Extra: Alternatives</h3><p>If your company doesn’t have the infrastructure or resources required to run <a href="https://airflow.apache.org/">Airflow</a>, <a href="https://www.prefect.io/">Prefect</a>, <a href="https://dagster.io/">Dagster</a>, or any other orchestration solution.</p><p>I would recommend you set up DBT locally and then reduce your DBT Cloud Team seats to 1, so all the development happens locally, and then DBT Cloud only executes/orchestrates your jobs.</p><p>This would allow you better control and experience when developing DBT models while having an external orchestrator in charge of your Jobs.</p><p><strong>Feedback is a gift</strong>, we hope that by sharing our DBT experience we helped you on improving your DBT journey too.</p><p>So, please feel free to share your experience 💌 and your approach to your DBT journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFn6N-VPSwIxTZApJxMd7w.png" /><figcaption>Images provided by <a href="http://superside.com/">Superside.com</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d94edac9cdf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/dbt-core-airflow-7d94edac9cdf">DBT Core &amp; Airflow</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open-water races and our professional development]]></title>
            <link>https://medium.com/albert-franzi/open-water-races-and-our-professional-development-65b88ef589c8?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/65b88ef589c8</guid>
            <category><![CDATA[open-water-swimming]]></category>
            <category><![CDATA[swimming]]></category>
            <category><![CDATA[professional-development]]></category>
            <category><![CDATA[professional-growth]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Tue, 21 Sep 2021 06:32:31 GMT</pubDate>
            <atom:updated>2021-09-21T06:32:31.852Z</atom:updated>
            <content:encoded><![CDATA[<p>Over my <a href="https://www.marnaton.com/cadaques-6-5/?lang=en">last open-water race</a> (6.5km) in <a href="https://goo.gl/maps/H37kP2ttWYfsLRaU9">Cadaqués</a>, I had plenty of time (<em>1h 48m</em>) to reflect on how close was swimming in an open-water race to my current professional growth.</p><p>In this post, I would love to cover different aspects and strategies followed in an open-water race which could be easily translated into our professional development.</p><blockquote><strong>Disclaimer:</strong> I’m currently <a href="https://www.linkedin.com/in/albertfranzi/">Head of Data Platform</a> at <a href="https://typeform.com/"><strong>Typeform</strong></a> leading a cross-functional team composed of 5 Data Engineers, 3 Tracking Specialists, and 1 Data DevOps.</blockquote><blockquote><strong>Disclaimer’’:</strong> My longest race has been 7km in 2h 4m.</blockquote><h3>Technical details</h3><h4>Prepare yourself</h4><p>Before enrolling in any open-water race you really need to know the details. Distance, time, path, swimmers, weather conditions, etc. Each one of them will define the evolution of the race.</p><p>My first open-water race was 3km in <a href="https://goo.gl/maps/H4Jd2SEsWzTm2hjz8">Begur</a>, it wasn’t easy but it was plenty of fun and, as a first experience, really meaningful. It was the right race to introduce myself to the open-water races so I could evaluate how much I would enjoy and how many more races I would like to run.</p><p>When we start our career in the professional world, we shouldn’t race into the senior or lead role as the first shoot, we should start from the intern/junior position and evaluate how much we enjoy the new job path.</p><p>In addition, we would need to do some research about the company we are going to join. Culture, industry, workers, teammates, etc. So we would build in advance some partial view about where we are going to swim.</p><h4>Create awareness about your energy — build endurance</h4><p>Swimming 7km requires endurance, requires being completely aware of your energy levels, so you take the right decisions over the race.</p><blockquote>Going wild in the sea could end you up throwing the towel earlier than expected.</blockquote><p>There will be moments when we will sail on conservative mode and others when we will be on turbo mode boosting our progress to jump into the next level (<strong><em>see:</em></strong><em> Jumping into the next swarm</em>).</p><p>It will be just a matter of seconds or minutes, but before taking the next action, we will evaluate how much energy we still have on us and what output will be obtained from it.</p><p>When we are progressing in our job ladders, there will be moments to evaluate if we are ready to jump to the next level and what we expect from the change.</p><p>Nevertheless, there will be times where we weren’t planning to change anything, but even then, we will need to evaluate if our energy levels are the right ones to keep the velocity.</p><p>Plus, not everything is a matter of progress, sometimes we will experience contraction before expansion (<strong><em>see:</em></strong><em> Optimize energy</em>).</p><h4>Balance observation and focus</h4><p>Open-water races are usually well signposted with big buoys (<em>best case: 1 buoy every ~200m). </em>So it’s easy to keep the right direction from the beginning to the end.</p><p>So, to visualize the next big buoy/achievement, you need to pull out your head from the water and look for them. This will impact your velocity since the right posture is to be looking down.</p><blockquote>Lowering your head when you swim allows your neck to be in a more neutral position. This creates less fatigue and allows for more freedom of movement. It also puts less strain on the lower back and hips. If you look forward and have a high head position, your back will be arched when you swim, and your hips will ride low in the water. If you look down, your back and hips ride higher in the water with less effort.</blockquote><p>So, how often will we need to raise our heads looking for the next big buoy? We will need to find the right balance (which could vary over the race). Having all the time the head outside the water would allow us to know for sure where the next buoy is, but, we would swim very badly.</p><p>Our daily job needs the same dynamic balance: how much time do we need to spend observing the market, the technological advances, competence, etc. versus how much time do we focus on our current work moving forward our team, our project, our company?</p><h4>Mind the jellyfishes</h4><p>The most feared monster by the swimmers (<em>at least in the Mediterranean</em>), are the jellyfishes and their stings. That’s why the most common question before starting the race is about them.</p><p>While swimming, it is ok to be aware that we will encounter some jellyfishes over the course. We will have three options: dodge (reduce velocity), just swim over them (high sting change), or use a suncream with jellyfish protection.</p><p>In addition, we shouldn’t put all the attention into them; otherwise, it will reduce our performance and our focus on what matters, enjoying the open waters and all the beauty underwater.</p><p><strong>Jellyfishes</strong> could be the right metaphor to designate <strong>toxic employees</strong>. We will occasionally encounter them in our jobs, but we cannot allow them to degrade our performance and happiness at work.</p><p>We could dodge them, but eventually, we will run out of dodges. We could try to swim over them by having a high chance of impacting us and our team. However, we could try to work on solutions so we are ready to encounter them without reducing our speed and without impacting our team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*if9g9dO-GJjoubo7JwuxqA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@pistos?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jeffrey Hamilton</a> on <a href="https://unsplash.com/s/photos/jellyfish?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4>The sea current and the market</h4><p>Over the race, we will encounter different current directions. Swimming with the current will bring us further with less time or energy consumption. However, encountering an opposite current will force us to spend more time and energy.</p><p>Being aware of the sea currents will give us the advantage of knowing when to reduce energy consumption to prepare for a jump or when to boost our speed to jump into the next swarm.</p><p>The market has the same currents, sometimes, the company and our team could take advantage of them by being in the right niche or because the tendency follows us. However, we will encounter complicated moments where the market goes against us (i.e Covid pandemic).</p><h4>Optimize energy</h4><p>Swimming behind someone usually allows us to reduce our consumption (10-20%) while we keep the same velocity. Even sometimes, we will touch their feet if we don’t put attention to their speed changes.</p><p>Optimizing energy could help us in three situations:</p><ul><li>We are preparing ourselves for the next swarm jump (<strong><em>see</em></strong><em>: Jumping into the next swarm</em>), so we harvest energy.</li><li>We’ve just jumped from the previous swarm and we are recovering from the extra energy consumed.</li><li>We are already at our 100%, but swimming faster than we could do it alone, so we take advantage of our lead.</li></ul><p>When working on different teams or companies we really need to acknowledge which situation we are in. Are we preparing for the next jump? Are we learning from a great team whereby giving our 100% we are capable of following them?</p><p>This could apply to interns, juniors, and mid-levels when having the right swarm allows them to keep the same velocity as Seniors (<strong><em>see:</em></strong><em> Leading a swarm</em>).</p><h3>The swarm</h3><p>I consider a swarm as a group of swimmers swimming together at the same speed.</p><h4>Shoaling with the swarm</h4><p>Being in a swarm allows us to take the <a href="https://en.wikipedia.org/wiki/Shoaling_and_schooling">shoals</a> advantages such as Optimizing Energy, reducing the number of times we look for the next buoy (+focus,-observation), and going in the right direction.</p><p>However, we could encounter some issues while shoaling. If the Swarm lead goes in the wrong direction, we will too. If the swarm reduces the velocity because is running out of energy, we will too. So we will need to take action: jump in to lead the swarm or jump into the next one.</p><p>Being in the right team will grant us too many benefits plus we will be able to learn, improve, progress, evolve over the journey. However, we will need to watch where we are in each situation and how the team/company evolves. There will be moments when our leading skills will be required, or when we will really consider the jump into another role/team/company (<strong><em>see</em></strong><em>: jumping into the next swarm</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wORZLhM5-ZSxOXKCe6enw.jpeg" /><figcaption><strong>Shoal of fish</strong> — Photo by <a href="https://unsplash.com/@dorographie?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Dorothea OLDANI</a> on <a href="https://unsplash.com/s/photos/shoal?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4>Leading a swarm</h4><p>Being in front of a swarm means that we are the ones at 100% of our energy consumption and the ones behind us are counting on us to bring them forward.</p><p>We will be looking for the next buoy more often than before, and we will not have the choice of resting behind anyone. However, if we have co-leads in our current swarm we could rely on them for some observation while we keep the focus, and in case of changing directions, they could correct us in time.</p><p>At work, I’m leading my swarm at full speed while being aware of how the entire company needs us and how our alignment impacts each area.</p><p>I’m close to having my first co-lead on board who I really believe in and who will help us in keeping the right direction while I can increase my focus time.</p><p>The team believes in the lead and they follow the same direction with him. If someone in the team stops believing in us, they could opt for: jumping out of the swarm, leading us, or the last option, touching our feet 👣 making us aware of their increased speed and needs.</p><h4>Jumping into the next swarm</h4><p>Over the race, there will be times when we are ready to jump into the next swarm. We have the energy required and we are eager to keep moving forward on the race. So we observe, we prepare, and then at the right moment, we increase our energy consumption to achieve the next swarm.</p><p>Jumping into the next swarm would bring us to swim with other swimmers which could have a different sense of orientation and different speeds than we were used to in the previous one.</p><p>We will localize the swarm lead and put ourselves just behind them since it will be the best position to recover from the energy jump while we adapt to the new swarm’s velocity.</p><p>It’s quite common and natural to experience jumps and changes in our professional development. We should evaluate before each jump if we are ready and what new colleagues and challenges we will find there. Jumping to the next adventure is not going to be easy and it will require being fully aware of who we are and where we are.</p><p>If we consume too much energy on the jump in an open-water race, we could end up not being able of following our new swarm and ending up swimming alone. Jumping into another role/team/company could have the same effect if we are not fully ready for it.</p><h4>Acknowledge all swimmers</h4><p>And last but not least, acknowledge all the swimmers. They are swimming the same race as we are. They are putting all their effort into it and all their passion. So we should respect them and acknowledge their skills.</p><p>Over the race, we will be sharing swarms, leaving some swimmers behind, and finding new ones. And we will acknowledge respect, and be grateful to each one of them.</p><p>At the moment I was crossing the goal, I was thinking that I’m not only ready to jump into the next swarm but <strong>I’m also ready to jump into the next swarm with my entire swarm</strong>.</p><figure><img alt="Cadaqués — 2021–09–18" src="https://cdn-images-1.medium.com/max/1024/1*oMbavDjXyRr9R7wXOyE31w.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65b88ef589c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/open-water-races-and-our-professional-development-65b88ef589c8">Open-water races and our professional development</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Structured Streaming in K8s with Argo CD]]></title>
            <link>https://medium.com/albert-franzi/spark-structured-streaming-in-k8s-with-argo-cd-de4942846161?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/de4942846161</guid>
            <category><![CDATA[argo]]></category>
            <category><![CDATA[spark-operator]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Wed, 30 Sep 2020 09:59:54 GMT</pubDate>
            <atom:updated>2020-09-30T09:59:54.866Z</atom:updated>
            <content:encoded><![CDATA[<p>This post aims to cover how we enhanced the capability to deploy <a href="https://spark.apache.org/streaming/">Spark Streaming</a> applications in our K8s Cluster, at <a href="https://typeform.com/">Typeform</a>, using <a href="https://argoproj.github.io/argo-cd/">Argo CD</a>. Easy, practical, and reusable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*Rq9HslcmG0T5E2LV31j1eQ.png" /><figcaption>ArgoCD + K8s + Spark</figcaption></figure><p>Over the last year, we have moved from a batch processing jobs setup, with<a href="https://airflow.apache.org/"> Airflow</a> using EC2s, to a powerful &amp; scalable setup using Airflow &amp; Spark in K8s (<em>check out our previous article:</em><a href="https://medium.com/typeforms-engineering-blog/deploy-airflow-1-10-10-in-kubernetes-using-terraform-and-helm-2476f03f07d0"><em> deploying airflow 1.10.10 in Kubernetes</em></a>).</p><p>The increasing need to support the latest Data and Machine Learning community advances, together with the need to enable multidisciplinary teams, forced us to design a solution that allows running multiple Spark versions at the same time while avoiding duplicated infrastructure and simplifying deployment, maintenance, and development.</p><h3>Airflow + KubernetesPodOperator</h3><p>Our journey started by testing out Airflow with K8s. To be more specific, by integrating Spark with the<a href="https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py"> Spark Submit Hook</a> and the <a href="https://airflow.apache.org/docs/stable/kubernetes.html#kubernetespodoperator">KubernetesPodOperator</a>.</p><p>The former provides the spark-submit command, while the K8s Pod Operator facilitates launching Spark jobs in K8s (<em>code</em><a href="https://gist.github.com/afranzi/23512d0f3d5c777ba318a5e9c13f9a2b"><em> SparkK8sStandaloneOperator.py</em></a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N8YgLL7GRM1MpvnTogtx-Q.png" /></figure><p>The process starts with uploading the assembly JAR from<a href="https://jfrog.com/artifactory"> Artifactory</a> to S3 and then executing the Spark Job inside K8s using the assembly JAR from S3.</p><p>Combining the code from S3 with the right Spark Docker image, allows us to reduce the size of our assembly JARs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14734ef4dd638bcbdd073bafcddcd84a/href">https://medium.com/media/14734ef4dd638bcbdd073bafcddcd84a/href</a></iframe><p>Spark has the feature to execute Spark jobs using the <a href="https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Introducing_the_Hadoop_S3A_client"><strong>s3a</strong></a> system.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a792dc9229e5b57b7e58d95662c1b1a/href">https://medium.com/media/2a792dc9229e5b57b7e58d95662c1b1a/href</a></iframe><blockquote><strong>Note:</strong> To execute <strong>s3a</strong> JARs, the Spark Docker image needs to have at least <em>AWS_SDK_VERSION=1.11.788</em></blockquote><p>You can find our Dockerfiles at <a href="https://gist.github.com/afranzi/4685518e24fd81e07639b97c4a5a2757">hadoop.Dockerfile</a> &amp; <a href="https://gist.github.com/afranzi/85ff3bf47632fc650cec17b0cc16bbca">spark.Dockerfile</a>.</p><p>Note that the requirement of AWS SDK Version <strong>≥ 1.11.788</strong>, is due to the <a href="https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html"><strong>WebIdentityTokenCredentialsProvider</strong></a> since it’s the first version providing the build method called by Spark when authenticating on accessing S3 <em>(check how IRSA works </em><a href="https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/"><em>here</em></a><em>)</em>.</p><p>The benefit of using the WebIdentityToken is we don’t have any AWS Secret Key in our K8s cluster. Everything works with IAM Roles defined for each service account on the namespace.</p><blockquote>Our first idea was to use --master k8s, but since we were using the WebIdentityToken with <strong>IRSA</strong> as the authorization mechanism for AWS. We needed to specify the ServiceAccount for the executors, a feature that will be available in <a href="https://github.com/apache/spark/blob/c8f3bd861d96cf3f7b01cd9f864c181a57e1c77a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala#L141-L145">Spark 3.1.0</a> :(</blockquote><h3>Spark with SparkOperator</h3><p>The setup described above with the KubernetesPodOperator in Airflow works fine for batch jobs. However, we wanted to move away from EMR for testing purposes, and we also wanted to support Streaming, so using Airflow wasn’t enough.</p><p>That’s when <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator">Spark on k8s operator</a> comes to the scene, a great Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes developed by the community.</p><p>The Spark on K8s operator needs at least K8s 1.13 and <strong>Spark 2.4.5</strong>, we are lucky to be using already Spark 3.0.0 and K8s 1.16 pending to be migrated to 1.17 soon.</p><h4>So, how does it work?</h4><p>All it needs is a YAML spec with the arguments &amp; configurations to run your spark jobs successfully.</p><p>All the YAML attributes are well defined in the <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/api-docs.md">operator documentation</a>.</p><blockquote>The main key here is the <strong>mainApplicationFile</strong> property fetching the jar file from S3.</blockquote><p>Our advice is to use the Spark docker image, mentioned above (<em>remember the </em><strong><em>WebIdentityTokenCredentialsProvider</em></strong><em>, we will need it</em>) and put your jars into S3. So, the only remaining part is adding in your CD/CI pipeline a step where you upload the artifact to S3, so it can be used by the Spark Operator, and define the fs.s3a.aws.credentials.provider with the WebIdentityTokenCredentialsProvider.</p><blockquote>We are using Travis with the <a href="https://docs.travis-ci.com/user/deployment/s3/">S3 deploy provider</a> to upload the jars to S3 on merging with the master branch.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ab921c8e9813627e9655224ebc80aae/href">https://medium.com/media/2ab921c8e9813627e9655224ebc80aae/href</a></iframe><p>As we mentioned above, we had the need of moving away from EMR. One of the main reasons being reducing costs, since having an EMR without jobs running, it equals to burning money. Besides, the EMR forced us to be attached to a specific Spark version in each cluster.</p><p>And last, but not least, using EMR means we have to wait until AWS releases the latest version in Spark (<em>check </em><a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-6x.html"><em>EMR 6.x releases</em></a>). This summer we were able to start playing with Spark 3.0.0 the day after its release (18th of June, 2020) when AWS was releasing the new EMR + Spark 3.0.0 in September 2020.</p><h3>Spark + SparkOperator + ArgoCD</h3><p>When it comes to deployment speed and simplicity, we always want to give data engineers and developers full control and flexibility to deploy their code as fast as possible, allowing short iteration cycles. And there is where our <em>de facto</em> CD tool shines: <a href="https://argoproj.github.io/argo-cd/">ArgoCD</a>.<br> <br>We wanted to follow the recommended Argo’s pattern: “<a href="https://argoproj.github.io/argo-cd/operator-manual/cluster-bootstrapping/"><strong>App of apps</strong></a>” so defining one single ArgoCD’s app containing all our different streaming applications was the clear way to go, and we did it with the help of <a href="https://helm.sh/">Helm</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sk_6Cb8yUm4dLCECukyV4w.png" /><figcaption>Deployment flow with ArgoCD and Spark Operator</figcaption></figure><p>So, when we are ready to deploy the spark application to our K8s cluster, we just need to merge the master branch and update the version specified in the YAML specs. Then ArgoCD will automatically deploy the new version.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4029cfa340f5012d5deadeec2dc4e229/href">https://medium.com/media/4029cfa340f5012d5deadeec2dc4e229/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zkQfFmxcUiAHiV3LzKaesA.gif" /><figcaption>ArgoCD UI — Spark Applications running in the Spark Operator being Synchronized <em>manually</em>.</figcaption></figure><p>We have some Spark Streaming jobs running and consuming from a Kafka Cluster, by using the Spark Operator we delegate the error recovery to the operator by specifying the <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/api-docs.md#sparkoperator.k8s.io/v1beta2.RestartPolicy">RestartPolicy</a>.</p><p>In the beginning, we were using the <strong>OnFailure</strong> policy, however, we recently changed it to <strong>Always</strong> since the restart error count doesn’t get restarted (<em>it will be nice if after X amount of time the counter got initialized again</em>).</p><blockquote>Our next step is integrating the Datadog Alert with the restart policy, so we can trigger Datadog Alerts in case of having the job stuck on the restarting step.</blockquote><h4>Bonus track: Spark History Server in K8s</h4><p>Since the Spark jobs in K8s are ephemeral and logs are lost by default. To circumvent this situation, we have implemented the Spark History Server, so we were able to analyze our jobs once finished.</p><p>To be able to recover the logs and visualize them later, we configured our jobs to upload the logs to an S3 bucket and then the Spark History Server to read them from there.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c345f200bdc4cc32a0fe7375d43598e/href">https://medium.com/media/1c345f200bdc4cc32a0fe7375d43598e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06edddba3b91dedd34590ce31c75cc5b/href">https://medium.com/media/06edddba3b91dedd34590ce31c75cc5b/href</a></iframe><blockquote><strong>Note:</strong> <a href="https://medium.com/u/965f4a3bd4a8">Carlos Escura</a> will publish an article about <strong>Spark History Server in K8s</strong> soon, stay tuned.</blockquote><h3>Next steps</h3><p>Keep adding traceability with Datadog to improve and tune our jobs.</p><p>Once Spark 3.1.0 is ready, we would like to put our hands-on and test the ServiceAccount feature submitting batch jobs directly from Airflow in cluster mode.</p><p>Since we are really happy with the Streaming jobs being handled by ArgoCD, we will iterate in this direction.</p><p><strong><em>Co-authored</em></strong><em> by </em><a href="https://medium.com/u/965f4a3bd4a8"><em>Carlos Escura</em></a><em> </em>🤓<em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de4942846161" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/spark-structured-streaming-in-k8s-with-argo-cd-de4942846161">Spark Structured Streaming in K8s with Argo CD</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Catalogue — Knowing your data]]></title>
            <link>https://medium.com/albert-franzi/data-catalogue-knowing-your-data-15f7d0724900?source=rss-95473c1b1c09------2</link>
            <guid isPermaLink="false">https://medium.com/p/15f7d0724900</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Albert Franzi]]></dc:creator>
            <pubDate>Fri, 28 Feb 2020 14:36:36 GMT</pubDate>
            <atom:updated>2020-02-28T14:36:36.594Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Catalogue — Knowing your data</h3><p>This post aims to explain how we expand the knowledge about our data inside <a href="https://typeform.com/"><strong>Typeform</strong></a> by deploying the right tool to gather, document and inform all the available metadata in a Data Catalogue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zbgub8h39JOFy6GI0s6Nwg.jpeg" /><figcaption>Phare de la Pointe du Vieux-Fort, Guadeloupe, by <a href="https://medium.com/u/95473c1b1c09">Albert Franzi</a>.</figcaption></figure><p>It’s well-known data is in a continuously changing state, it keeps evolving. Then, new concepts and ways of representing reality appear on stage and, last but not least, sometimes some of these concepts get deprecated.</p><p>What is more, not only data evolve, teams &amp; organizations too. That means people joining teams with new knowledge &amp; expertise, and, also the most important thing to care about, people leaving companies with <strong>undocumented wisdom</strong>.</p><p>Besides, I’ve seen in multiple occasions teams moving forward initiatives to start documenting data-sources, triggered by data consumers experiencing some troubles using data wrongly, or by someone leaving the company with a huge amount of data wisdom centralized in one individual.</p><blockquote>Someone leaving the company with a huge amount of centralized and non-duplicated data wisdom across the team is quite common, as even in the best families, and this lost wisdom can and will affect the entire organization until somehow &amp; someone recover it.</blockquote><p>These initiatives usually start with joy and with good intentions. People start documenting in Confluence, Notion and sometimes in excels about how data is organized, structured, produced and consumed. Nevertheless, all of these initiatives used to end up with outdated &amp; legacy documentation as a result.</p><p>All the initiatives consisting of <strong>updating documentation manually are mean to fail</strong>. Data Engineers prefer coding than documenting; Data Scientists modeling than documenting and Data Analysts &amp; Data Viz playing with data than documenting. Everyone prefers being in the playground than being documenting. That’s why matters having the right process to <strong>keep data documentation alive with automated processes</strong>.</p><blockquote>keep data documentation alive with automated processes</blockquote><p>So, after reviewing some existing solutions (<a href="https://github.com/linkedin/datahub"><em>DataHub</em></a><em>, </em><a href="https://github.com/lyft/amundsen"><em>Amundsen</em></a><em> &amp; </em><a href="https://github.com/MarquezProject/marquez"><em>Marquez</em></a>) about how the metadata is represented, future integrations &amp; user experience, we decided to move forward with <a href="https://github.com/linkedin/datahub">Data Hub</a>.</p><p>By implementing DataHub as our Data Catalogue we aim to empower all our Data Consumers and Producers by having a better understanding of how all our data is represented and interconnected.</p><blockquote><strong>DataHub</strong> is LinkedIn’s generalized metadata search &amp; discovery tool.</blockquote><p>At <a href="https://typeform.com/">Typeform</a>, our datasets are mainly stored in <a href="https://aws.amazon.com/redshift/"><strong>Redshift</strong></a> (two clusters) and <a href="https://aws.amazon.com/s3/"><strong>S3</strong></a>. Then we have <a href="https://looker.com/"><strong>Looker</strong></a> for dashboarding and <a href="https://airflow.apache.org/"><strong>Airflow</strong></a> for pipelines. That means having 3 types of entities — Datasets, Dashboards, and Jobs. The first one (<em>datasets</em>) is already supported and implemented. Now, we just need to <a href="https://github.com/linkedin/datahub/blob/master/docs/how/entity-onboarding.md">extend their entity</a> to represent the missing ones (<em>already in DataHub’s </em><a href="https://github.com/linkedin/datahub/blob/master/docs/roadmap.md"><em>roadmap</em></a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PLS-fNDT_lN47jfp9uUcWQ.png" /></figure><h3>Fetching metadata</h3><p>As we mentioned before, datasets should be documented automatically without requiring a tedious and manual process. Well, how could we do that?</p><p>For gathering all the tables available in Redshift and their descriptions, we created an Airflow Operator that queries the <strong>pg_catalog</strong> tables to fetch all these metadata and then send it to the <a href="https://github.com/linkedin/datahub/tree/master/metadata-ingestion">DataHub MCE consumer</a> by Kafka.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PiLmTyVtuk2o3WkZVPiGnQ.png" /></figure><p>Everyone in the Data team has the responsibility &amp; <strong>ownership</strong> of documenting the tables they create by providing a table &amp; column level descriptions in Redshift using the <a href="https://docs.aws.amazon.com/redshift/latest/dg/r_COMMENT.html"><strong>comment statements</strong></a>, so our Airflow extractor operator can fetch this information from it later.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a41e20a80de92cbb901ae2d6562565da/href">https://medium.com/media/a41e20a80de92cbb901ae2d6562565da/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f48d74741b3e37fef8420876d18a4c6e/href">https://medium.com/media/f48d74741b3e37fef8420876d18a4c6e/href</a></iframe><p>Besides, we will register the owner of the table as a <strong><em>producer</em></strong> in the Data Catalogue and the Airflow user as an owner. From another side, once we can register the <strong>Data Jobs/Pipelines</strong> in charge of modifying &amp; ingesting data, we will add them to the owners’ property.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LfGA1zNTJhAv0su8U4Zdgg.png" /><figcaption>Dataset owners</figcaption></figure><h3>Data Lineage</h3><p>Redshift provides the <a href="https://docs.aws.amazon.com/redshift/latest/dg/r_STL_SCAN.html"><strong>Scan feature</strong></a> from where we can fetch which tables are being scanned to update the final one. From there we will generate all the downstream relationships.</p><p>The <strong>stl_scan</strong> table also provides which S3 files are being used, so we will extract these S3 paths from it to include also external datasets to the lineage dependency in the future.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15450734fc8743ce5ec28d5cbac8ae39/href">https://medium.com/media/15450734fc8743ce5ec28d5cbac8ae39/href</a></iframe><p>Once we send the relationship metadata to the Data Catalogue, we can visualize their interactions properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HsBhlRNZuy98dE_hYJmiaw.png" /><figcaption>Form Questions Dataset and its relationship with other datasets</figcaption></figure><h3>Gathering up</h3><p>There is still a long way until we have a fully operative Data Catalogue comprehending every piece of data we have. However, we are completely sure <strong>this is the way</strong> to follow unleashing the access and knowledge about data making everything visible and understandable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4IZKWh2r_EyPD-C_dJKcw.png" /></figure><h4>Next steps</h4><p>Having all Redshift datasets documented is great. Even the Lineage between them is really powerful since it allows our Data Consumers to have a better understanding of how the data is flowing in our system.</p><p>The plan is to keep adding the S3 datasets, connecting them with the Redshift ones and then start adding all our Dashboards and Jobs information too,</p><p>By adding them it will allow us to know which jobs are the owners of these datasets and which dashboards are being populated from each of them.</p><h4>Links of Interest</h4><ul><li><a href="https://towardsdatascience.com/how-linkedin-uber-lyft-airbnb-and-netflix-are-solving-data-management-and-discovery-for-machine-9b79ee9184bb">How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine Learning Solutions</a> <em>[August 19th, 2019]</em>.</li><li><a href="https://engineering.linkedin.com/blog/2019/data-hub">DataHub: A Generalized Metadata Search &amp; Discovery Tool</a> <em>[August 14th, 2019]</em>.</li><li><a href="https://speakerdeck.com/shirshanka/the-evolution-of-metadata-linkedins-journey-strata-nyc-2019">The Evolution of Metadata: LinkedIn’s Journey</a> <em>[Strata NYC September 25th, 2019].</em></li><li><a href="https://engineering.linkedin.com/blog/2020/open-sourcing-datahub--linkedins-metadata-search-and-discovery-p">Open-sourcing DataHub</a> <em>[February 18th, 2020]</em>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15f7d0724900" width="1" height="1" alt=""><hr><p><a href="https://medium.com/albert-franzi/data-catalogue-knowing-your-data-15f7d0724900">Data Catalogue — Knowing your data</a> was originally published in <a href="https://medium.com/albert-franzi">Albert Franzi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>