<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mehdi Tazi on Medium]]></title>
        <description><![CDATA[Stories by Mehdi Tazi on Medium]]></description>
        <link>https://medium.com/@mehdi.tazi?source=rss-13a60478aec7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*818mHI8ovsiThZqd41ZZwg.png</url>
            <title>Stories by Mehdi Tazi on Medium</title>
            <link>https://medium.com/@mehdi.tazi?source=rss-13a60478aec7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 01:01:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mehdi.tazi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI Is Not Killing Software Engineering. It’s Changing Expectations and the Way Engineers Work.]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://levelup.gitconnected.com/ai-is-not-killing-software-engineering-its-changing-expectations-and-the-way-engineers-work-1b23da98d0b7?source=rss-13a60478aec7------2"><img src="https://cdn-images-1.medium.com/max/1536/1*9KHXumdW-pkEmAHXITkxYw.png" width="1536"></a></p><p class="medium-feed-snippet">&#x1F513; For non-Medium members:
Read the full article here</p><p class="medium-feed-link"><a href="https://levelup.gitconnected.com/ai-is-not-killing-software-engineering-its-changing-expectations-and-the-way-engineers-work-1b23da98d0b7?source=rss-13a60478aec7------2">Continue reading on Level Up Coding »</a></p></div>]]></description>
            <link>https://levelup.gitconnected.com/ai-is-not-killing-software-engineering-its-changing-expectations-and-the-way-engineers-work-1b23da98d0b7?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b23da98d0b7</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Wed, 13 May 2026 14:41:21 GMT</pubDate>
            <atom:updated>2026-05-15T14:40:19.450Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Introduction to Apache Iceberg]]></title>
            <link>https://levelup.gitconnected.com/introduction-to-apache-iceberg-71ef54be1683?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/71ef54be1683</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 01:06:33 GMT</pubDate>
            <atom:updated>2025-02-28T01:06:33.116Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Iceberg is an open-source <strong>table format</strong> designed for huge analytic datasets, providing a reliable way to manage data in a data lake as if it were in a traditional database. It was originally developed at Netflix to overcome limitations of older Hadoop table formats (like Apache Hive) and was later donated to the Apache Software Foundation​. Iceberg’s purpose is to bring <strong>database-like features</strong> — such as ACID transactions, schema evolution, partition flexibility, and time-travel queries — to petabyte-scale data lakes​. In essence, Iceberg defines <em>how</em> metadata is stored and data files are organized in a table, without replacing the underlying storage format (it works with Parquet, ORC, Avro, etc.)​ . This technology-agnostic design means you can use Iceberg with various processing engines (Spark, Hive, Impala, Flink, etc.) and on different storage systems (HDFS, cloud object stores) while enjoying consistent table semantics​</p><p><strong>Key features and differentiation:</strong> Several key features make Iceberg stand out from previous approaches and other table formats:</p><ul><li><strong>ACID and Concurrent Writes:</strong> Iceberg tables support <strong>ACID transactions</strong> with snapshot isolation, so multiple users or engines can read and write simultaneously without corrupting data​. This is a leap from older Hive-style lakes which lacked atomic commits and often suffered correctness issues under concurrent writes​</li><li><strong>Schema and Partition Evolution:</strong> Iceberg allows <strong>in-place schema changes</strong> (add/drop/rename columns) and <strong>partition evolution</strong> without costly rewrites​. Unlike static Hive partitions, you can update how data is partitioned as it grows or query patterns change, and Iceberg will adapt new data going forward while still querying old data seamlessly​</li><li><strong>Hidden Partitioning:</strong> Iceberg introduces <em>hidden partitioning</em>, so you don’t need to add artificial partition keys as columns. Partition metadata is stored within Iceberg, and query engines can automatically prune data by actual values (e.g. date or bucket) without user query hints​ . This simplifies table design for data architects.</li><li><strong>Time Travel and Version Rollback:</strong> Every change creates a new snapshot. Users can query data <strong>as of a specific snapshot or time</strong> (time-travel queries) and even roll back to earlier table versions if needed​. This is invaluable for troubleshooting, audits, or reproducible research (for example, re-running a report on last week’s data).</li><li><strong>Multiple Engines, Consistent View:</strong> Because Iceberg provides an open specification and library, it’s designed to be used by any compute engine. Engines like Hive, Impala, Spark, Trino, Flink, and others can all operate on the <strong>same Iceberg table</strong> and see the same data and schema. This differs from proprietary formats tied to one platform. Iceberg’s broad adoption (by Cloudera, Snowflake, Netflix, Apple, AWS, and others) reflects its neutrality and flexibility in modern data stacks​</li></ul><p>By addressing the above, Apache Iceberg has emerged as a critical building block for the <strong>data lakehouse</strong> architecture — combining the scale/flexibility of data lakes with the reliability and structure of data warehouses. In the next sections, we’ll explore Iceberg’s technical architecture and how it fits into a Lakehouse, with practical examples in Cloudera’s ecosystem.</p><h3>How Apache Iceberg Works</h3><p>Apache Iceberg’s architecture is built on a <strong>layered metadata approach</strong> that tracks table state and data files efficiently. Instead of relying on directory listings or a central metastore for tracking files (as Hive did), Iceberg stores metadata in a series of files (often on the same storage as the data) that together act as the <strong>table’s control plane</strong>​. This design ensures that reading or writing an Iceberg table requires only a few constant-time metadata lookups, regardless of the table’s size (avoiding costly operations that grow with number of files or partitions)​. The main components of Iceberg’s table architecture include:</p><ul><li><strong>Catalog:</strong> The catalog keeps track of all Iceberg tables and acts as an entry point for engines. It stores the location of each table’s <strong>current metadata file</strong>. In Cloudera, for example, the Hive Metastore can serve as the Iceberg catalog, storing a pointer to the latest table metadata and some high-level info​</li><li><strong>Table Metadata File:</strong> A metadata file contains the <strong>table schema</strong>, partition specification, and a list of all snapshots of the table​. Each time a table is updated (an insert, update, delete, or schema change), a new metadata file is written, referencing the previous metadata and the latest snapshot. This file effectively version-controls the table — enabling rollbacks and tracking schema history​. The metadata file also lists other metadata files (like manifests) that comprise the latest snapshot.</li><li><strong>Snapshots and Manifest Lists:</strong> A <strong>snapshot</strong> represents the state of the table at a given point in time. Technically, a snapshot is encoded by a <em>manifest list</em> file, which is a list of pointers to all manifest files that make up that snapshot​. The manifest list includes metadata about each manifest (e.g., which partitions it covers, how many files) so that the query engine can quickly decide which manifests (and thus which data files) are relevant for a given query​. Each snapshot is immutable; new snapshots are created on table changes, enabling time travel queries by selecting a specific snapshot.</li><li><strong>Manifest Files:</strong> A manifest is a detailed index of a subset of the table’s data files. Each manifest file contains a list of data file entries along with metadata like partition values and column statistics (e.g., min/max values for each column in that file)​. Manifests are the key to Iceberg’s query performance. They allow query engines to <strong>prune entire sets of files</strong> quickly: for example, if a query is filtering by date, Iceberg will only read the manifests and data files for the relevant date partitions, skipping others​. Because manifests themselves are tracked in the manifest list, Iceberg can <strong>reuse manifests between snapshots</strong> if those data files didn’t change, rather than rewrite metadata for every new snapshot​.</li><li><strong>Data Files:</strong> These are the actual data storage files (e.g., Parquet, ORC, Avro) containing the table’s rows​. Iceberg doesn’t mandate a specific file format; it works on top of columnar file formats that engines support. Data files are organized in the table’s storage directory but are not exposed directly to users — the Iceberg metadata tells engines which data files to read for a given query. When new data is written, new data files are created and added to a new snapshot; when data is removed or updated, Iceberg creates <em>delete files</em> or simply omits those data file entries in a new snapshot (in V2 tables, delete manifests can track row-level deletes).</li><li><strong>Serializable isolation and commits:</strong> All writes in Iceberg are performed via an <em>optimistic concurrency</em> model with atomic commits. Writers prepare new snapshot metadata and attempt to atomically swap the table’s current metadata pointer to the new one​. If another writer updated the table in the meantime, the commit will fail and typically retry based on the new latest metadata (this avoids conflicts). This design means readers never see partial writes — a table switch from one consistent snapshot to the next, or not at all. As Ryan Blue (Iceberg’s co-creator) noted, <em>“Without atomic commits, every change to a Hive table risks correctness errors… Iceberg tackled atomicity to make automation possible, even in cloud object stores.”</em>${29†L343-L350}$ In practice, data architects can rely on Iceberg tables to remain consistent and self-healing, even with streaming data ingestion or multiple concurrent ETL jobs.</li></ul><p><strong>Efficiency at scale:</strong> Thanks to its metadata hierarchy, Iceberg query planning is extremely efficient for large tables. It avoids the expensive directory-listing operations of Hive by maintaining an <strong>index of files</strong> in manifests. Planning a query involves reading a tiny metadata file and a few manifests instead of scanning a giant folder structure​. The Iceberg spec emphasizes O(1) metadata lookups: no matter if a table has 10 partitions or 10,000, the cost to open the table is nearly constant​. This allows even petabyte-scale tables with millions of files to be handled gracefully. As an example, one claim is that even multi-petabyte Iceberg tables can be listed or read by a single coordinator node without a special distributed service for metadata​. This level of scalability and performance is a cornerstone for using Iceberg in production Lakehouse environments.</p><h3>Apache Iceberg in a Lakehouse</h3><p>The <strong>data lakehouse</strong> is a modern architecture that blends elements of data lakes and data warehouses. In a lakehouse, you store data in low-cost storage (like a data lake) but impose a table structure and transactional guarantees (like a warehouse) so that a wide range of analytics can be done directly on the lake data. Apache Iceberg plays a pivotal role in enabling this architecture by providing the <strong>open table format “glue”</strong> between storage and compute engines​.</p><p><strong>Multi-engine support:</strong> One of the hallmarks of a lakehouse is allowing diverse analytical engines and workloads to operate on the same data. Iceberg was designed with this in mind — it’s not tied to one execution engine. Whether you are running SQL queries with Apache Impala, interactive analysis with Hive, streaming jobs with Flink, or machine learning with Spark, each can read and write the <strong>same Iceberg tables</strong> reliably​. This solves a major challenge from earlier data lake setups, where schema and data consistency were hard to maintain across different tools. With Iceberg’s table abstraction, all engines see a consistent view of data and schema, and changes made by one engine (e.g., a Spark ETL job) are immediately visible to others (e.g., an Impala BI dashboard query) once committed.</p><p><strong>Open storage and separation of compute:</strong> Lakehouse architectures emphasize storing data in open formats on a central storage accessible to all (often cloud object storage or HDFS) and allowing compute engines to scale independently. Iceberg fits perfectly here by <strong>separating table definition from any single processing framework</strong>​. The data and metadata live in the storage layer (files in object store or HDFS), and any engine that understands Iceberg’s spec can query it. This means you’re not locked into a specific vendor’s storage or proprietary format — a crucial factor for data architects who want longevity and flexibility in their data platform​. For example, if you have all data on S3 or HDFS, you could query it with Spark today and switch to Trino or Dremio tomorrow with minimal friction, thanks to Iceberg’s open standards.</p><p><strong>Advantages over traditional data lakes:</strong> Compared to a traditional Hadoop data lake (e.g., Hive tables on HDFS), an Iceberg-based lakehouse brings many improvements:</p><ul><li><em>Reliable Data Updates:</em> In old data lakes, updating or deleting records was cumbersome (often requiring rewriting entire partitions or using Hive ACID which had heavy overhead). Iceberg allows <strong>direct INSERT, UPDATE, DELETE</strong> operations with ACID guarantees, making data mutation manageable on the lake​.</li><li><em>Unified Batch and Streaming:</em> Lakehouse needs to support both batch analytics and real-time data. Iceberg tables can be incrementally updated by streaming pipelines (e.g., using Flink or Spark Structured Streaming) while simultaneously queried by batch jobs. Its snapshot isolation ensures streaming writes don’t disrupt running queries.</li><li><em>Metadata scalability:</em> Hive metastore-based lakes often choke on millions of partitions or files (the well-known “metastore scalability” problem). Iceberg’s design shifts partition tracking to the file metadata, <strong>dramatically reducing load on the central metastore</strong> and enabling tables with tens of thousands of partitions to operate smoothly​. Planning is faster and the “small files problem” can be managed with Iceberg’s metadata (through partitioning and file compaction strategies).</li><li><em>Auditability and Time Travel:</em> Because every change in Iceberg is captured as a new snapshot, you inherently get <strong>audit logs</strong> of data changes. Analysts or compliance officers can query data <em>as of</em> last week or last month easily, something not feasible in a raw data lake without maintaining separate copies. This built-in time travel supports reproducibility in analytics and science use cases (e.g., re-running a report on prior data to compare trends).</li></ul><p>In summary, Iceberg brings the <strong>“warehouse” capabilities to the “lake”</strong>. Cloudera aptly calls Iceberg “the secret sauce of the open lakehouse” because it allows a unified platform where structured and unstructured data can coexist and be queried with high performance​. Data architects choosing a lakehouse architecture benefit from Iceberg’s open ecosystem and robust design, avoiding vendor lock-in while meeting enterprise requirements for data integrity.</p><h3>Integration with Cloudera Private Cloud Base Edition</h3><p>Cloudera’s platform (particularly <strong>Cloudera Data Platform Private Cloud Base</strong>, the on-premises edition) integrates Apache Iceberg to provide an enterprise-ready lakehouse solution on your own infrastructure. In this environment, Iceberg works with familiar Cloudera components to store data in HDFS or Ozone and to query it with the Hadoop ecosystem tools. Here are key points on how Iceberg is leveraged within Cloudera:</p><ul><li><strong>Multiple Engine Integration:</strong> Cloudera has added full support for using Iceberg tables in its tools. For instance, as of Cloudera Private Cloud Base 7.1.9 and above, you can query Iceberg tables from <strong>Apache Hive</strong> (for ETL or analytics with Hive SQL) and <strong>Apache Impala</strong> (for high-performance BI queries) transparently​. In addition, Cloudera supports Iceberg with <strong>Apache Spark 3</strong> for batch or ML jobs, <strong>Apache Flink</strong> for streaming ingestion, and even <strong>Apache NiFi</strong> for data flow ingestion​. This means a data architect can establish a single Iceberg data lake and have all these engines read/write the data without duplication. For example, you might ingest streaming event data into an Iceberg table via Flink, then run Impala queries for reporting on the fresh data, all in Cloudera’s stack.</li><li><strong>Catalog and Metadata Management:</strong> In Cloudera’s implementation, the Hive Metastore (HMS) often serves as the Iceberg catalog. However, unlike traditional Hive tables, HMS does not store partition metadata for Iceberg tables — Iceberg’s own metadata files carry that info. This <strong>lightweight role of HMS</strong> eliminates the scalability bottlenecks of Hive’s metastore​. Essentially, HMS knows an Iceberg table exists and where its metadata is located, but the heavy lifting of tracking files is offloaded to Iceberg. Data architects using Cloudera don’t need to drastically change their metastore setup; they just gain better performance at scale.</li><li><strong>Iceberg Table Formats (V1 vs V2):</strong> Cloudera supports both Iceberg format v1 and v2. The <strong>Iceberg V2</strong> format introduces additional capabilities like <strong>row-level delete and update</strong> support (through delete files and position delete markers), which make tables fully mutable and ACID-compliant​. In Cloudera’s private cloud, Hive and Impala can both work with V2 tables, allowing DELETE statements or updates to be executed and read correctly. This is a huge win for data governance (no more orphaned data on deletes) and simplifies pipelines that require data corrections.</li><li><strong>Security and Governance:</strong> Cloudera integrates Apache Ranger security policies with Iceberg tables​. That means you can apply fine-grained access control (down to column or row level restrictions) on Iceberg data through Ranger, just as you would for Hive tables. This integration is important for compliance, ensuring that sensitive data in your lakehouse is protected by central governance rules. Additionally, Cloudera’s lineage and metadata tools (Atlas, Navigator) can track Iceberg table usage, and Cloudera’s <strong>Data Visualization</strong> can directly work with Iceberg tables for creating dashboards​– demonstrating that Iceberg is treated as a first-class citizen in the platform.</li><li><strong>Operational Considerations:</strong> Cloudera provides utilities and documentation for managing Iceberg tables, such as <strong>snapshot expiration</strong>, compaction, and <strong>Hive-Impala interoperability</strong>. You can even convert existing Hive tables to Iceberg tables using Impala syntax (ALTER TABLE ... CONVERT TO ICEBERG)​. For data architects, this means it’s feasible to gradually <strong>migrate a legacy data lake</strong> to an Iceberg-backed lakehouse without massive re-ingestion — you can convert tables in place. Cloudera’s management interfaces (Cloudera Manager, etc.) also expose configuration for Iceberg, making it easier to tune settings like split sizes or to monitor table metrics.</li></ul><p>Overall, using Apache Iceberg in Cloudera Private Cloud Base allows organizations to keep their data on-prem (for security or compliance) while gaining the benefits of a lakehouse architecture. You get the performance and reliability improvements of Iceberg with the robust ecosystem of Cloudera — a combination that enables advanced analytics and data services on a unified platform​</p><h3>Key Use Cases for Iceberg in Cloudera</h3><p>Iceberg’s capabilities unlock a variety of use cases in a Cloudera environment. Below are some key scenarios particularly relevant to data architects, and how Iceberg addresses them:</p><h3>Real-Time Analytics</h3><p>Modern analytics often demands fresh data with low latency. In Cloudera, Apache Iceberg enables <strong>real-time or near-real-time analytics</strong> by allowing continuous data ingestion and immediate queryability. For example, a streaming job (with Spark Structured Streaming or Flink) can ingest event data into an Iceberg table as small batches or micro-batches. Because Iceberg commits are atomic, readers (like Impala BI dashboards or Hive queries) can run at the same time without seeing partial writes — each query sees a consistent snapshot of the data up to the latest commit. This ensures that even as data is flowing in, the analytics results remain correct and up-to-date.</p><p>Iceberg also improves <strong>query performance</strong> for analytics. Its manifest files and partition pruning dramatically speed up scans on large datasets. Only relevant subsets of files are read for a given query, which is crucial for interactive analytics on tables that might be multi-terabyte or petabyte in size. Cloudera’s use of Impala with Iceberg benefits from this design — Impala can perform blazingly fast SQL queries since Iceberg provides it with pre-pruned file sets and column stats to optimize execution. In one Cloudera example, a fact table with tens of billions of rows can be queried in seconds for aggregate results, thanks to Iceberg avoiding the need to read unrelated partitions. Moreover, Iceberg’s ability to handle small incremental writes (from real-time pipelines) without fragmenting the table into thousands of tiny files (through its metadata tracking and possible compaction jobs) means <strong>consistent performance over time</strong>, which is critical for 24/7 analytics systems. Cloudera themselves highlight that Iceberg can handle tables of “tens of petabytes” while still delivering responsive reads– a necessity for real-time analytics on big data.</p><h3>Cost Optimization</h3><p>Iceberg contributes to cost efficiency in a lakehouse in multiple ways. First, by improving query planning (skipping irrelevant files quickly), it <strong>reduces the CPU and I/O overhead</strong> for analytics jobs. Engines spend less time scanning data that doesn’t matter, which translates to lower compute costs — whether that’s cloud VM hours or on-prem cluster CPU time. In object storage scenarios, Iceberg’s metadata avoidance of listing operations and its ability to do <strong>O(1) file lookups</strong> means you incur far fewer expensive storage API calls (like S3 LIST requests) compared to naive data lake approaches​. This can significantly lower costs for cloud deployments.</p><p>Second, Iceberg makes it easier to <strong>optimize storage layouts</strong>. Data architects can define partition strategies and sort orders that balance query performance with file sizes. Over time, if the data distribution changes, Iceberg allows you to evolve the partition scheme without reloading all old data, avoiding a huge recompute cost​. For example, you might start partitioning by date, and later decide to partition by region as well; Iceberg lets you add that new partition transform for new data while still keeping old files accessible — no massive backfill needed. This means you can continuously tune your tables for performance <em>and</em> cost (smaller partitions where it matters, larger ones where it doesn’t) in an agile way.</p><p>Within Cloudera, another cost angle is <strong>simplification of ETL pipelines</strong>. Because one Iceberg table can serve many purposes (instead of maintaining separate ETL outputs for various query types), you store fewer redundant copies of data. A well-designed Iceberg table can serve both detailed queries and high-level rollups, especially with features like <strong>metadata tables</strong> or <strong>materialized views</strong> that Cloudera supports​. By not having to create multiple derived datasets, you save storage space and maintenance effort. Iceberg’s capability to do in-place updates also means you don’t need to periodically dump and reload tables to incorporate changes, saving costly reprocessing runs. In summary, Iceberg helps <em>do more with less</em>: less data scanned, less data duplicated, and more efficient use of computing resources.</p><h3>Data Governance and Compliance</h3><p>In enterprise settings, data governance is paramount — you need to ensure data is correct, auditable, and secure. Iceberg provides strong guarantees that help in this domain. Its ACID transaction support means that <strong>data integrity is maintained</strong> even as multiple jobs modify the data. There are no partial updates — which is critical for compliance (e.g., financial records won’t be partially updated if a job fails halfway). This atomicity and isolation simplify <strong>audit trails</strong> since each snapshot is a complete point-in-time image of the table. Data engineers or auditors can trace what changed and when by looking at the snapshot history. In Cloudera’s tooling, you can even list the snapshots of an Iceberg table and examine the metadata to see operations over time.</p><p>Iceberg also enables <strong>schema evolution with governance</strong>. As data regulations or business needs change, you might need to add or rename columns (for example, marking a column as sensitive PII, or splitting a field into more granular columns). With Iceberg, you can do this evolution without losing historical data or creating a mess of datasets​. Every schema change is tracked, and old data is still queryable through the new schema (Iceberg handles the mapping). This is invaluable for compliance because you don’t have to migrate all old data when rules change — you evolve the schema going forward, saving time and avoiding errors that might come from complex migrations.</p><p>On the security front, as noted, Cloudera integrates Iceberg with Apache Ranger​. This means you can enforce <strong>fine-grained access control</strong> on Iceberg tables: for example, masking or denying access to certain columns for specific user roles. All engines (Hive, Impala, etc.) will respect those rules when accessing Iceberg data. So, your data lakehouse can be governed under a <strong>single security policy framework</strong>, crucial for compliance with laws like GDPR or HIPAA. Finally, features like <strong>time travel</strong> also support compliance use cases — e.g., proving what data existed in the system at a past date, or rolling back a table to a clean state if some erroneous data was introduced (which might be important for correcting a compliance violation or data breach). In Cloudera’s Iceberg implementation, one can use time-travel queries in Spark or Impala to access historical snapshots for such purposes.</p><h3>Machine Learning Pipelines</h3><p>Machine learning workflows benefit greatly from the capabilities of Apache Iceberg. A common challenge in ML is managing <strong>feature data and training datasets</strong> over time — ensuring that models are trained on the correct version of data and that you can reproduce those training datasets months or years later. Iceberg’s versioning and time-travel come to the rescue here. Data scientists can snapshot the exact state of an Iceberg table used for model training and later use the snapshot ID or timestamp to consistently retrieve that same data for debugging or re-training. For instance, if an ML model trained in January shows unexpected behavior in March, you can query the feature table as of the January snapshot and compare it to the current data to investigate what changed.</p><p>Iceberg is also a strong foundation for building a <strong>feature store</strong> in Cloudera’s platform. A feature store is essentially a curated warehouse of features (attributes) for ML models. With Iceberg, features can be stored in tables that are updated continuously but accessed concurrently in a reliable way. You might have a pipeline that computes aggregate features (say user engagement metrics) and upserts them into an Iceberg table daily. Downstream, multiple ML experiments can read from this feature table without worrying about inconsistent reads. If needed, each team can even create a <strong>branch or separate snapshot</strong> of the table for their experiments, then merge or compare results. Major cloud providers have embraced this pattern: for example, Amazon SageMaker Feature Store uses Iceberg as the table format for its offline store to leverage Iceberg’s fast point-in-time access to feature data​.</p><p>Another ML aspect is <strong>data volume and variety</strong>. Iceberg can handle large append-only datasets (like event logs) as well as slowly changing dimensions, which are common in ML pipelines. The ability to update and delete specific records helps when you need to correct label data or remove outliers from training sets. Also, Iceberg’s support for multiple file types means you could store raw training data in one format and derived features in another, but manage both through one consistent table interface. Cloudera’s support for using Spark with Iceberg is key for ML: data scientists can use PySpark or Spark MLlib to directly query Iceberg tables, perform feature engineering, and write results back, benefiting from the snapshot isolation (no interference between training jobs) and having <strong>one source of truth</strong> for features. And as mentioned earlier, <strong>time travel is great for ML</strong> — you can easily generate training sets for a specific past period and even feed that into model validation to simulate how a model would have performed with past data​.</p><p>In summary, Apache Iceberg provides the data reliability, consistency, and management features that accelerate machine learning pipelines — from feature storage to reproducible model training — all within the scalable Cloudera platform.</p><h3>Conclusion</h3><p>Apache Iceberg has proven itself to be a <strong>game-changer for data lakehouse architectures</strong>, enabling data architects to design systems that are both scalable and reliable. In this article, we covered how Iceberg serves as a powerful table format that brings ACID transactions, schema flexibility, and time-travel querying to big data environments​. Its architecture (with metadata and manifest files) ensures that even as datasets grow into the petabytes, the data remains manageable and performant to query​. These traits differentiate Iceberg from earlier data lake solutions and even other table formats, making it a popular choice in open data ecosystems.</p><p>Within the <strong>lakehouse paradigm</strong>, Iceberg allows a unified platform where multiple analytics engines can share data transparently, combining the best of data lakes and data warehouses​. Cloudera’s incorporation of Iceberg into its Private Cloud Base Edition highlights the practical value of Iceberg in enterprise settings — it integrates with existing Hadoop tooling (Hive/Impala/Spark) and adds the robustness needed for modern analytics and AI use cases. We explored how real-time analytics, cost optimization, data governance, and ML pipelines all benefit from Iceberg’s capabilities in a Cloudera environment. From faster queries due to intelligent metadata pruning to safer data management with ACID guarantees and historical reproducibility, Iceberg addresses many pain points that data architects have faced for years.</p><p>In conclusion, Apache Iceberg provides a <strong>strong foundation for building a lakehouse architecture</strong> that is open, scalable, and ready for diverse analytics. For data architects, adopting Iceberg means you can architect data lakes with the confidence that they will behave like well-tuned data warehouses when needed — without sacrificing the flexibility to use the tools and storage of your choice. As the data ecosystem continues to evolve, Iceberg’s open specification and community-driven development ensure it will keep adapting (with features like improved performance, new data types, and even better streaming support on the horizon). Embracing Apache Iceberg today sets up your data architecture for success, balancing innovation with the governance and efficiency that enterprises demand. With Iceberg in your lakehouse, you truly can have <strong>the best of both worlds</strong>: a data lake that’s as reliable and query-friendly as a warehouse​, positioning your organization to derive maximum value from its data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71ef54be1683" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/introduction-to-apache-iceberg-71ef54be1683">Introduction to Apache Iceberg</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prompting to AI Systems Engineering: How Enterprise GenAI Actually Evolves]]></title>
            <link>https://medium.com/towards-data-architecture/the-genai-journey-from-basic-to-advanced-ai-implementation-with-real-world-examples-3abad527c36e?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3abad527c36e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Fri, 06 Sep 2024 22:17:23 GMT</pubDate>
            <atom:updated>2026-05-12T00:13:55.484Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>The adoption of Generative AI (GenAI) in enterprises is often presented as a simple maturity ladder:</p><pre>Prompt Engineering → RAG → Fine-tuning → Pre-training</pre><p>While this progression is useful for understanding the increasing levels of customization and technical complexity, reality is far more nuanced.</p><p>In practice, organizations do not necessarily move sequentially through these stages. Many companies successfully operate with Retrieval-Augmented Generation (RAG) alone. Others combine RAG and fine-tuning. Very few organizations ever need to pre-train a model from scratch.</p><p>More importantly, the competitive advantage in 2026 increasingly comes not from the model itself, but from the surrounding AI system:</p><ul><li>data quality,</li><li>orchestration,</li><li>workflows,</li><li>governance,</li><li>evaluation,</li><li>observability,</li><li>security,</li><li>and integration with enterprise tools.</li></ul><p>This article explores how enterprise AI systems actually evolve, from simple prompting techniques to fully integrated AI architectures.</p><h3>1. Prompt Engineering</h3><h3>Overview</h3><p>Prompt engineering is the simplest and fastest way to leverage generative AI. Instead of modifying the underlying model, organizations guide the model’s behavior through carefully designed instructions.</p><p>The objective is to maximize output quality while minimizing infrastructure complexity, cost, and implementation time.</p><p>Modern prompt engineering often includes:</p><ul><li>role prompting,</li><li>chain-of-thought prompting,</li><li>few-shot examples,</li><li>structured output instructions,</li><li>context injection,</li><li>and reasoning constraints.</li></ul><p>For many business use cases, prompt engineering alone already delivers significant value.</p><h3>Example: Marketing Content Generation</h3><p>A marketing agency uses OpenAI GPT-based models to generate social media content.</p><p>Instead of training a custom model, the team designs highly specific prompts such as:</p><blockquote><em>“Write a short and engaging Instagram post promoting eco-friendly clothing for a sustainability-conscious audience aged 25–35. Use a modern and optimistic tone.”</em></blockquote><p>By iterating on prompt structure and examples, the agency generates high-quality content without investing in model training infrastructure.</p><h3>Example: Customer Support Drafting</h3><p>A support team uses prompt engineering to generate first drafts of customer replies.</p><p>The prompts include:</p><ul><li>company tone guidelines,</li><li>empathy requirements,</li><li>escalation policies,</li><li>and formatting instructions.</li></ul><p>The AI helps agents respond faster while humans remain responsible for validation and final approval.</p><h3>Limitations</h3><p>Prompt engineering is lightweight and inexpensive, but it has important limitations:</p><ul><li>limited control over factual accuracy,</li><li>no persistent knowledge acquisition,</li><li>inconsistent behavior across prompts,</li><li>sensitivity to wording changes,</li><li>and difficulty handling complex domain-specific reasoning.</li></ul><p>As enterprise requirements grow, organizations often need stronger grounding mechanisms.</p><h3>2. Retrieval-Augmented Generation (RAG)</h3><h3>Overview</h3><p>Retrieval-Augmented Generation (RAG) combines an LLM with an external retrieval system capable of searching relevant information before generating a response.</p><p>Rather than relying solely on the model’s internal training data, RAG enables AI systems to access:</p><ul><li>enterprise documents,</li><li>APIs,</li><li>databases,</li><li>vector stores,</li><li>knowledge bases,</li><li>and real-time information sources.</li></ul><p>A typical RAG pipeline includes:</p><ul><li>document ingestion,</li><li>chunking,</li><li>embeddings generation,</li><li>vector indexing,</li><li>semantic retrieval,</li><li>ranking,</li><li>and context injection into prompts.</li></ul><p>This approach significantly improves factual grounding and reduces hallucinations.</p><h3>Example: Financial Services Assistant</h3><p>A financial institution deploys an AI support assistant connected to internal knowledge systems containing:</p><ul><li>mortgage policies,</li><li>financial regulations,</li><li>product information,</li><li>and live interest rates.</li></ul><p>When a customer asks:</p><blockquote><em>“What are the current mortgage rates for first-time buyers?”</em></blockquote><p>the AI retrieves the latest approved information before generating a grounded response.</p><p>Without RAG, the model could provide outdated or hallucinated answers.</p><h3>Example: E-Commerce Recommendations</h3><p>An e-commerce platform combines user purchase history, product metadata, and live inventory data inside a RAG architecture.</p><p>Instead of generating generic recommendations, the AI produces suggestions based on:</p><ul><li>current stock availability,</li><li>customer preferences,</li><li>seasonal trends,</li><li>and pricing changes.</li></ul><p>This creates far more relevant and operationally accurate outputs.</p><h3>Why RAG Became Dominant</h3><p>RAG has become one of the dominant enterprise AI architectures because it solves several major business constraints:</p><p>ChallengeRAG AdvantageFrequently changing knowledgeReal-time retrievalPrivate enterprise documentsNo model retraining requiredRegulatory updatesEasier updatesLower infrastructure costCheaper than pre-trainingFaster implementationRapid deployment</p><p>For many enterprises, RAG delivers the best balance between cost, accuracy, flexibility, and maintainability.</p><h3>Limitations</h3><p>Despite its strengths, RAG introduces new operational complexity:</p><ul><li>retrieval quality issues,</li><li>chunking strategy problems,</li><li>latency increases,</li><li>vector database maintenance,</li><li>permission management,</li><li>context window limitations,</li><li>and observability challenges.</li></ul><p>Poor retrieval pipelines can still lead to hallucinations even with strong models.</p><h3>3. Fine-Tuning</h3><h3>Overview</h3><p>Fine-tuning adapts a pre-trained model to better perform specific tasks or behaviors using specialized datasets.</p><p>Unlike RAG, which injects external knowledge dynamically, fine-tuning modifies how the model behaves internally.</p><p>Modern fine-tuning rarely retrains the entire model from scratch. Instead, organizations typically use techniques such as:</p><ul><li>Supervised Fine-Tuning (SFT),</li><li>LoRA,</li><li>QLoRA,</li><li>adapter tuning,</li><li>and other parameter-efficient fine-tuning methods (PEFT).</li></ul><p>Fine-tuning is especially useful for:</p><ul><li>tone consistency,</li><li>structured outputs,</li><li>classification tasks,</li><li>domain-specific terminology,</li><li>reasoning style adaptation,</li><li>and workflow optimization.</li></ul><h3>Important Clarification</h3><p>Fine-tuning is not a replacement for RAG.</p><p>In practice:</p><p>NeedPreferred ApproachDynamic knowledgeRAGReal-time informationRAGBrand tone consistencyFine-tuningStructured JSON outputsFine-tuningDomain behavior adaptationFine-tuningInternal documentsRAG</p><p>Most mature enterprise AI systems combine both approaches.</p><h3>Example: Healthcare Documentation Assistance</h3><p>A healthcare company fine-tunes a medical language model using anonymized clinical documentation and structured reporting examples.</p><p>The objective is not to automate medical decisions, but to assist clinicians in:</p><ul><li>drafting structured summaries,</li><li>standardizing terminology,</li><li>extracting relevant information,</li><li>and improving documentation consistency.</li></ul><p>Human validation remains mandatory.</p><h3>Example: Retail Brand Voice</h3><p>A fashion retailer fine-tunes a language model using thousands of historical product descriptions.</p><p>The result is an AI capable of generating descriptions aligned with:</p><ul><li>brand tone,</li><li>writing style,</li><li>formatting rules,</li><li>and vocabulary standards.</li></ul><p>This significantly reduces manual content production effort.</p><h3>Limitations</h3><p>Fine-tuning introduces additional complexity:</p><ul><li>training datasets must be high quality,</li><li>poor datasets amplify biases,</li><li>evaluation becomes critical,</li><li>retraining cycles are required,</li><li>governance becomes harder,</li><li>and infrastructure costs increase.</li></ul><p>Moreover, fine-tuning does not solve knowledge freshness problems.</p><p>A fine-tuned model can still generate outdated information if the underlying knowledge changes.</p><h3>4. AI Systems Engineering and Agentic Architectures</h3><h3>Overview</h3><p>By 2026, the most advanced enterprise AI systems are no longer just “LLMs with prompts.”</p><p>They are complete AI systems composed of:</p><ul><li>orchestration layers,</li><li>tool calling,</li><li>workflow engines,</li><li>memory systems,</li><li>retrieval pipelines,</li><li>evaluation frameworks,</li><li>guardrails,</li><li>observability platforms,</li><li>and human-in-the-loop processes.</li></ul><p>This shift marks the transition from simple GenAI usage to AI Systems Engineering.</p><h3>Example: Enterprise AI Assistant</h3><p>An enterprise assistant may interact with:</p><ul><li>Salesforce,</li><li>SAP,</li><li>Atlassian Jira,</li><li>Notion,</li><li>Slack,</li><li>internal APIs,</li><li>ticketing systems,</li><li>and analytics platforms.</li></ul><p>Instead of simply answering questions, the assistant can:</p><ul><li>retrieve information,</li><li>create tickets,</li><li>summarize meetings,</li><li>trigger workflows,</li><li>validate approvals,</li><li>and orchestrate business processes.</li></ul><p>In this model, the LLM becomes only one component of a larger intelligent system.</p><h3>Why This Matters</h3><p>In modern enterprise AI, the competitive advantage increasingly comes from:</p><ul><li>workflow integration,</li><li>data quality,</li><li>orchestration,</li><li>governance,</li><li>and operational reliability.</li></ul><p>Not from the base model alone.</p><p>Two companies using the same foundation model can achieve dramatically different outcomes depending on the quality of their AI system architecture.</p><h3>5. Continued Pre-Training and Domain Adaptation</h3><h3>Overview</h3><p>Before discussing full pre-training from scratch, it is important to distinguish a more common intermediate approach: continued pre-training.</p><p>Instead of building an entirely new model, organizations further train an existing foundation model on domain-specific corpora such as:</p><ul><li>legal documents,</li><li>scientific publications,</li><li>financial filings,</li><li>or cybersecurity datasets.</li></ul><p>This approach is often more practical than full pre-training while still improving domain expertise.</p><h3>Example: Cybersecurity AI</h3><p>A cybersecurity company performs continued pre-training on:</p><ul><li>threat intelligence feeds,</li><li>malware reports,</li><li>CVE databases,</li><li>and incident response documentation.</li></ul><p>The resulting model develops stronger understanding of cybersecurity terminology and attack patterns.</p><h3>6. Full Pre-Training</h3><h3>Overview</h3><p>Full pre-training is the most resource-intensive approach.</p><p>It involves training a foundational model from scratch using enormous datasets and large-scale distributed compute infrastructure.</p><p>This requires:</p><ul><li>massive GPU clusters,</li><li>distributed training expertise,</li><li>sophisticated data pipelines,</li><li>advanced research teams,</li><li>and extremely high operational budgets.</li></ul><p>As a result, full pre-training remains relatively uncommon outside major AI labs and hyperscalers.</p><h3>Example: Search Engines and Frontier Labs</h3><p>Organizations such as Google, Microsoft, OpenAI, and Anthropic train large foundational models using vast internet-scale datasets.</p><p>These models power:</p><ul><li>search engines,</li><li>coding assistants,</li><li>enterprise copilots,</li><li>reasoning systems,</li><li>and multimodal AI products.</li></ul><h3>Example: Pharmaceutical Research</h3><p>A pharmaceutical company may pre-train specialized models using:</p><ul><li>biomedical literature,</li><li>molecular datasets,</li><li>clinical trial records,</li><li>and protein interaction data.</li></ul><p>The objective is to accelerate drug discovery and scientific research.</p><h3>Limitations</h3><p>Pre-training introduces enormous challenges:</p><ul><li>infrastructure cost,</li><li>energy consumption,</li><li>data quality management,</li><li>model alignment,</li><li>safety risks,</li><li>evaluation complexity,</li><li>and regulatory concerns.</li></ul><p>For most enterprises, pre-training from scratch is economically unjustifiable.</p><h3>Governance, Evaluation, and Observability</h3><h3>The Most Underrated Part of Enterprise AI</h3><p>In real-world deployments, the hardest part of AI is often not the model itself.</p><p>It is building trustworthy systems.</p><p>Enterprise AI requires:</p><ul><li>evaluation pipelines,</li><li>hallucination monitoring,</li><li>security controls,</li><li>access governance,</li><li>PII protection,</li><li>lineage tracking,</li><li>observability,</li><li>cost monitoring,</li><li>and compliance enforcement.</li></ul><p>Without these controls, even powerful AI systems become unreliable in production.</p><h3>Hallucinations and Reliability</h3><p>Every GenAI architecture can hallucinate:</p><ul><li>prompting,</li><li>RAG,</li><li>fine-tuning,</li><li>and even fully pre-trained systems.</li></ul><p>This is why enterprises increasingly invest in:</p><ul><li>grounded generation,</li><li>automated evaluation datasets,</li><li>confidence scoring,</li><li>retrieval validation,</li><li>and human oversight.</li></ul><h3>Comparing the Different Approaches</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dgCXoJ23-V4_XA7aWvWkbA.png" /></figure><h3>Conclusion</h3><p>The evolution of enterprise AI is not simply a progression from prompting to larger models.</p><p>The real transformation is the shift from isolated model usage to complete AI systems engineering.</p><p>Modern AI success increasingly depends on:</p><ul><li>orchestration,</li><li>data quality,</li><li>governance,</li><li>workflow integration,</li><li>evaluation,</li><li>and operational reliability.</li></ul><p>In many cases, organizations achieve far greater value by improving their AI systems architecture than by building larger or more complex models.</p><p>The future of enterprise AI will not belong only to companies with the biggest models.</p><p>It will belong to organizations capable of building the most reliable, integrated, and trustworthy AI systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3abad527c36e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-architecture/the-genai-journey-from-basic-to-advanced-ai-implementation-with-real-world-examples-3abad527c36e">From Prompting to AI Systems Engineering: How Enterprise GenAI Actually Evolves</a> was originally published in <a href="https://medium.com/towards-data-architecture">Towards Data Architecture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Skills You’ll Need For Tomorrow According to World Economic Forum’s]]></title>
            <link>https://medium.com/@mehdi.tazi/what-skills-youll-need-for-tomorrow-according-to-world-economic-forum-s-e57b23486e49?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e57b23486e49</guid>
            <category><![CDATA[life]]></category>
            <category><![CDATA[skills]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Mon, 12 Aug 2024 13:45:37 GMT</pubDate>
            <atom:updated>2024-08-12T13:45:37.792Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong></p><p>As the global labor market evolves, the skills needed to thrive in the workforce are undergoing a significant transformation. The World Economic Forum’s 2023 Future of Jobs Report provides critical insights into the skills that will be most in demand over the next five years. Understanding these changes is essential for individuals, businesses, and policymakers as they navigate the future of work.</p><p><strong>The Changing Skills Landscape</strong></p><p>In recent years, technological advancements, particularly in artificial intelligence and automation, have accelerated the pace of change in the workplace. As businesses adopt new technologies and processes, the demand for certain skills is rising, while others are becoming less critical. The report highlights the growing importance of cognitive and digital skills as the foundation for future employment.</p><p><strong>Core Skills for the Future</strong></p><p>The report identifies several key skills that will be increasingly important in the coming years:</p><ol><li><strong>Analytical Thinking:</strong> As workplaces become more data-driven, the ability to analyze complex information and make informed decisions is crucial. Analytical thinking tops the list of core skills, highlighting the need for workers who can interpret data and derive actionable insights.</li><li><strong>Creative Thinking:</strong> Alongside analytical abilities, creative thinking is gaining prominence. Businesses value employees who can innovate, solve problems creatively, and adapt to changing environments.</li><li><strong>Technological Literacy:</strong> With rapid advancements in technology, digital skills are essential for navigating modern workplaces. Understanding and effectively utilizing digital tools and platforms will be vital for success.</li><li><strong>Self-Efficacy Skills:</strong> Resilience, flexibility, and the ability to learn continuously are becoming essential as workers face dynamic and often unpredictable work environments. Companies are seeking employees who can adapt quickly and remain motivated in the face of challenges.</li><li><strong>Working with Others:</strong> Socio-emotional skills, such as empathy, active listening, and leadership, are increasingly valued as teamwork and collaboration become integral to organizational success.</li></ol><p><strong>Evolving Skill Demands</strong></p><p>The report indicates that while some skills are gaining importance, others are becoming less relevant. Physical abilities, such as manual dexterity, are declining in demand due to increased automation. Similarly, basic literacy and numeracy skills, though foundational, are no longer sufficient on their own. Instead, a focus on developing higher-order cognitive and socio-emotional skills is critical.</p><p><strong>Reskilling and Upskilling Strategies</strong></p><p>Businesses are investing heavily in reskilling and upskilling their workforce to meet the demands of the future. Key strategies include:</p><ul><li><strong>Promoting Cognitive Skills:</strong> Analytical and creative thinking are top priorities for training programs, with companies dedicating resources to enhance these abilities.</li><li><strong>Emphasizing Technological Skills:</strong> Training in AI, big data, and digital literacy is essential as technology continues to shape the workplace. Companies are focusing on equipping their employees with the skills to work alongside advanced technologies.</li><li><strong>Developing Leadership and Social Influence:</strong> As organizations recognize the importance of effective leadership, they are investing in programs to enhance leadership skills and social influence among their workforce.</li></ul><p><strong>Challenges and Opportunities</strong></p><p>While businesses are actively addressing skill gaps, challenges remain. Companies must balance the need for technical skills with the importance of fostering a supportive and inclusive work environment. Additionally, workers must embrace lifelong learning to stay competitive in a rapidly changing job market.</p><p>However, these challenges also present opportunities. As new technologies create demand for emerging roles, workers who adapt and acquire in-demand skills will find themselves well-positioned for success. Moreover, the focus on skills development can lead to more fulfilling and rewarding careers, benefiting individuals and society as a whole.</p><p><strong>Conclusion</strong></p><p>The future of work is rapidly evolving, and the skills required to succeed are shifting accordingly. By focusing on analytical and creative thinking, technological literacy, and socio-emotional skills, individuals and businesses can prepare for the opportunities and challenges that lie ahead. Embracing a culture of continuous learning and adaptation will be key to thriving in this new era of work.</p><p><strong>Please consider clapping and following the writer! 👏</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e57b23486e49" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Optimistic and Pessimistic Locking in Open Data Formats]]></title>
            <link>https://levelup.gitconnected.com/understanding-optimistic-and-pessimistic-locking-in-open-data-formats-4b4c6252f5a7?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b4c6252f5a7</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 16:50:48 GMT</pubDate>
            <atom:updated>2024-06-23T16:50:48.453Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous article “<a href="https://medium.com/towards-data-engineering/understanding-acid-properties-in-databases-a-simplified-explanation-fe48cb2681e0">Understanding ACID Properties in Databases: A Simplified Explanation</a>” we explain what are the ACID properties and how important are in ensuring data consistency and integrity in transactions.</p><p>Today, we will explore principles for managing concurrency control in distributed data systems.</p><p>Modern data platforms, such as Lakehouse architectures (eg: Databricks, Athena,etc…) relies on open table formats like DeltaLake and Apache Iceberg. These formats use efficient concurrency control mechanisms to maintain data consistency and optimize performance.</p><p>In general two primary strategies for concurrency control are widely used: optimistic locking and pessimistic locking. In this article, we will compare these mechanisms, discussing their strengths and weaknesses, and providing detailed examples to illustrate their practical applications</p><h3>Part 1: Optimistic Locking in Data Platforms</h3><p>Optimistic locking assumes that conflicts between transactions are infrequent which is more frequent on Analytics Systems (OLAP) where read operations are frequent, and the likelihood of conflicts is lower. Instead of locking resources when a transaction begins, the system proceeds without locks and only checks for conflicts at the time of commit. This approach is particularly useful in environments with low data contention, allowing for high concurrency and reduced lock management overhead.</p><h4>How Optimistic Locking Works</h4><ol><li><strong>Start Transaction:</strong> The transaction begins by reading the current state of the data without acquiring any locks.</li><li><strong>Perform Operations:</strong> Changes are made locally under the assumption that no other transaction will conflict with these changes.</li><li><strong>Intention to Commit:</strong> When the transaction is ready to commit, it compares the current state of the data with the state at the start of the transaction.</li><li><strong>Conflict Detection:</strong> If another transaction has modified the same data, a conflict is detected, and the current transaction is aborted. Otherwise, the transaction commits successfully.</li><li><strong>Retry Logic:</strong> In case of a conflict, the transaction can be retried with the updated state of the data.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bNjGd7m7msL7i75G" /></figure><h4>Advantages and Disadvantages</h4><p><strong>Advantages:</strong></p><ul><li><strong>High Concurrency:</strong> Multiple transactions can proceed in parallel, increasing throughput.</li><li><strong>Reduced Lock Overhead:</strong> Eliminates the complexity and resource cost of managing locks.</li><li><strong>Scalability:</strong> Ideal for distributed systems where maintaining locks can be challenging.</li></ul><p><strong>Disadvantages:</strong></p><ul><li><strong>Retry Overhead:</strong> High contention scenarios can lead to frequent retries, impacting performance.</li><li><strong>Complex Conflict Resolution:</strong> Requires additional logic to handle conflicts and retries.</li></ul><h3>Part 2: Pessimistic Locking in Data Platforms</h3><p>Pessimistic locking assumes that conflicts are likely and prevents them by locking resources before performing any operations which is more frequent on Operational Systems (OLTP) where transactions are short-lived and the cost of conflicts is high. This strategy ensures that once a transaction acquires a lock, no other transaction can access the locked resources until the lock is released.</p><h4>How Pessimistic Locking Works</h4><ol><li><strong>Start Transaction:</strong> Acquire locks on all resources to be modified.</li><li><strong>Perform Operations:</strong> Make changes while holding the locks.</li><li><strong>Commit:</strong> Commit changes and release locks.</li><li><strong>Conflict Handling:</strong> Conflicts are avoided by ensuring that other transactions cannot access locked resources.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r1xB_UOpeVH23G0G" /></figure><h4>Advantages and Disadvantages</h4><p><strong>Advantages:</strong></p><ul><li><strong>Conflict Prevention:</strong> Conflicts are avoided by ensuring exclusive access to resources.</li><li><strong>Simplified Commit:</strong> No need for conflict detection at commit time since resources are locked.</li></ul><p><strong>Disadvantages:</strong></p><ul><li><strong>Reduced Concurrency:</strong> Transactions can be blocked, waiting for locks to be released, leading to reduced throughput.</li><li><strong>Deadlocks:</strong> Risk of deadlocks where two or more transactions are waiting for each other to release locks.</li><li><strong>Lock Management Overhead:</strong> Managing locks can be complex, especially in distributed systems.</li></ul><h3>Part 3: Practical Examples of Optimistic and Pessimistic Locking</h3><h4>Example of Optimistic Locking in Delta Lake</h4><p>Consider a Delta Lake table where multiple users are updating records concurrently:</p><ol><li><strong>Transaction A</strong> starts and reads the current state of the table.</li><li><strong>Transaction B</strong> starts after A and reads the same state.</li><li>Both transactions make changes locally.</li><li>Transaction A commits successfully as there are no conflicts.</li><li>Transaction B attempts to commit but finds that the state has changed due to A’s commit.</li><li>Transaction B detects the conflict, aborts, and retries with the updated state.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uVjR6uMg3U0nhUME" /></figure><h4>Example of Pessimistic Locking in a Traditional Database</h4><p>Consider a traditional relational database where multiple users are updating records:</p><ol><li><strong>Transaction A</strong> starts and acquires a lock on the target records.</li><li><strong>Transaction B</strong> starts and attempts to acquire a lock on the same records.</li><li>Transaction B is blocked until A releases the lock.</li><li>Transaction A commits and releases the lock.</li><li>Transaction B acquires the lock, performs its operations, and commits.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uAAmK_udO0IRJRso" /></figure><h3>Part 4: Comparison of Optimistic and Pessimistic Locking in Open Data Formats</h3><h4>Delta Lake</h4><ul><li><strong>Optimistic Locking:</strong> Delta Lake uses optimistic concurrency control. Transactions log their changes in JSON files within the Delta log directory. Conflict detection occurs at commit time, ensuring that no conflicting changes have been made since the transaction started. If a conflict is detected, the transaction is aborted and can be retried.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h7gSzSDE7lha7yoi" /></figure><h4>Apache Iceberg</h4><ul><li><strong>Optimistic Locking:</strong> Apache Iceberg also employs optimistic concurrency control. Each transaction creates a snapshot of the table’s state, which includes a manifest list file referencing the data files. At commit time, Iceberg checks for conflicting changes and handles conflicts similarly to Delta Lake.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wj9JCGI00p96qlTw" /></figure><h4>Conclusion</h4><p>Both optimistic and pessimistic locking have their place in the world of distributed data systems. Optimistic locking, favored by modern platforms like Delta Lake and Apache Iceberg, offers high concurrency and scalability, making it ideal for big data environments with low to moderate contention. Pessimistic locking, while ensuring conflict prevention, can introduce complexities and performance bottlenecks, making it less suitable for highly distributed systems.</p><p>When considering the dimensions of OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing):</p><ul><li><strong>OLTP Systems:</strong> Pessimistic locking is often more suitable for OLTP systems where transactions are short-lived and the cost of conflicts is high. Ensuring data consistency is crucial in these environments, and the overhead of locks can be justified.</li><li><strong>OLAP Systems:</strong> Optimistic locking is more appropriate for OLAP systems where read operations are frequent, and the likelihood of conflicts is lower. These systems benefit from high concurrency and scalability, which optimistic locking provides.</li></ul><p>Understanding these concurrency control mechanisms allows data engineers and architects to choose the right strategy for their specific use cases, balancing the need for performance, scalability, and data consistency.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b4c6252f5a7" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/understanding-optimistic-and-pessimistic-locking-in-open-data-formats-4b4c6252f5a7">Understanding Optimistic and Pessimistic Locking in Open Data Formats</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding ACID Properties in Databases: A Simplified Explanation]]></title>
            <link>https://medium.com/towards-data-engineering/understanding-acid-properties-in-databases-a-simplified-explanation-fe48cb2681e0?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe48cb2681e0</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Wed, 05 Jun 2024 04:19:35 GMT</pubDate>
            <atom:updated>2024-06-05T04:19:35.831Z</atom:updated>
            <content:encoded><![CDATA[<p>Ensuring data consistency and integrity in database transactions is essential in software engineering. The ACID model provides the foundational principles necessary for reliable data handling in databases. ACID stands for Atomicity, Consistency, Isolation, and Durability. These principles are crucial for maintaining data accuracy even in cases of system crashes or network issues.</p><h3>What is ACID?</h3><p>ACID represents the properties of Atomicity, Consistency, Isolation, and Durability. These are critical for maintaining the integrity and reliability of databases, especially during concurrent transactions or system failures. Let’s break down each property to understand its role and importance.</p><h3>Atomicity</h3><p>Atomicity means that a transaction is processed as a single, indivisible unit. This means that either all operations within the transaction are completed successfully, or none are executed at all. This all-or-nothing principle is vital for preventing data inconsistencies and logical errors in the database.</p><p>Consider a bank transfer scenario. The transaction involves debiting one account and crediting another. Atomicity ensures that both actions either happen together or not at all. If the system fails after the debit but before the credit, atomicity will ensure the debit is reversed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9N-80Yh5XZgMT4GWsK3jQ.png" /></figure><h3>Consistency</h3><p>Consistency ensures that any transaction will bring the database from one valid state to another, following all the defined rules and constraints. This property helps maintain data integrity by ensuring that transactions do not violate database rules.</p><p>For instance, if a database requires that a student’s grade must be between 0 and 100, consistency ensures that any transaction attempting to insert a grade outside this range will be rejected.</p><h3>Isolation</h3><p>Isolation ensures that transactions occur independently without interference. In environments where multiple transactions happen simultaneously, isolation ensures that concurrent operations do not lead to data anomalies or corruption.</p><p>There are various isolation levels, from read uncommitted to serializable, each offering different levels of protection against issues like dirty reads and phantom reads. For example, in an e-commerce platform, isolation ensures that two customers placing orders at the same time do not affect each other’s transactions.</p><h3>Durability</h3><p>Durability guarantees that once a transaction is committed, it remains so even in the event of a system crash. This ensures that once changes are made to the database, they are permanent and protected against failures.</p><p>For example, after a bank transfer transaction is completed and the funds are transferred, durability ensures that this transaction remains recorded even if the system crashes right afterward.</p><h3>The Goal of ACID and Its Challenges</h3><p>The main objective of ACID properties is to ensure transaction reliability and data integrity in database systems. These principles enable databases to handle concurrent transactions, system failures, and unexpected scenarios without compromising data integrity.</p><p>However, implementing ACID properties presents challenges. Ensuring atomicity and durability requires sophisticated logging and recovery mechanisms, which can impact performance. High isolation levels can reduce concurrency and increase transaction times. Balancing these properties to optimize performance and reliability is a complex task for database architects and administrators.</p><h3>Implications and Limitations</h3><p>While ACID properties provide a solid framework for transaction management, they also come with limitations. High isolation levels can cause performance issues due to the locking mechanisms that prevent simultaneous data access. Additionally, ensuring durability demands significant storage for logs and backups.</p><p>In distributed database systems, maintaining consistency and isolation across multiple nodes is particularly challenging. According to the CAP theorem, a distributed system can only achieve two out of three guarantees: consistency, availability, and partition tolerance. Consequently, some databases may adopt BASE (Basically Available, Soft state, Eventual consistency) principles to enhance scalability and performance.</p><p>The CAP theorem, also known as Brewer’s theorem, states that in a distributed data store, it is impossible to simultaneously achieve Consistency, Availability, and Partition Tolerance. Here’s why:</p><ul><li>Consistency ensures that all nodes see the same data at the same time. After a write operation, any subsequent reads will return the latest written data.</li><li>Availability guarantees that every request receives a response, whether it is successful or fails.</li><li>Partition Tolerance means that the system continues to operate despite arbitrary partitioning due to network failures.</li></ul><p>To understand why achieving all three is impossible, consider a scenario where a network partition occurs, dividing the network into two parts that cannot communicate with each other. In this situation, you face a choice:</p><ol><li>Consistency over Availability: If you choose consistency, any updates in one partition will not be reflected in the other until the partition is resolved. This means some requests might not be answered until consistency is restored, sacrificing availability.</li><li>Availability over Consistency: If you choose availability, the system continues to accept and process requests even in the presence of partitions, potentially leading to inconsistencies because updates in one partition will not be immediately visible in the other.</li></ol><p>Technically, when a network partition occurs, a distributed system must decide between:</p><ul><li>Ensuring that all nodes return the same value (consistency), which might mean some nodes become unreachable (compromising availability).</li><li>Allowing all nodes to continue operating independently (availability), risking that they might diverge in their state (compromising consistency).</li></ul><p>This inherent trade-off is why no distributed system can guarantee all three properties simultaneously. Databases must make trade-offs based on the specific requirements and use cases they are designed for. Systems like NoSQL databases often opt for eventual consistency to ensure high availability and partition tolerance, whereas traditional relational databases might prioritize consistency over availability.</p><h3>Conclusion</h3><p>ACID properties are fundamental to maintaining database integrity and ensuring reliable transaction management. They are essential for maintaining data consistency across applications, but implementing them requires careful consideration of performance trade-offs and system limitations. Understanding and balancing ACID properties helps in designing robust, high-performing database systems that meet modern application requirements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe48cb2681e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/understanding-acid-properties-in-databases-a-simplified-explanation-fe48cb2681e0">Understanding ACID Properties in Databases: A Simplified Explanation</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Data Pipeline: Understanding ETL and Reverse ETL]]></title>
            <link>https://levelup.gitconnected.com/navigating-the-data-pipeline-understanding-etl-and-reverse-etl-e911bd2aa820?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e911bd2aa820</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 02:23:41 GMT</pubDate>
            <atom:updated>2024-03-18T02:23:41.191Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello, I’ve talked to many professionals who are wondering why Reverse ETL and traditional ETL processes are not adequate for all data integration tasks. It seems, on the surface, that the main difference is merely the order in which data is processed and transferred. This perspective, however, overlooks the unique roles and complexities of each process within the data management lifecycle. To shed light on these distinctions and the specific value they bring to data strategies, a fundamental review of data management basics, starting with ETL, is necessary. This chapter is dedicated to explaining the differences between ETL and Reverse ETL, aiming to clarify their individual purposes and why specialized tools have emerged to address their distinct requirements. Through exploring these processes in depth, the goal is to enhance understanding of their contributions to informed decision-making and the optimization of data ecosystems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oxcITIsa2iIaJfjk" /><figcaption>Photo by <a href="https://unsplash.com/@fabioha?utm_source=medium&amp;utm_medium=referral">fabio</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Quick reminder about ETL</h3><p>ETL, an acronym for Extract, Transform, Load, is a data pipeline process designed to consolidate data from one or multiple sources, transform it into a format suitable for analysis, and load it into a target data repository like data warehouse, data lake, etc. The purpose of ETL is multifold:</p><ul><li><strong>Extract: </strong>Data is collected from various sources, which could range from databases, SaaS platforms, to flat files and more, ensuring that data is captured in its raw form.</li><li><strong>Transform: </strong>This step involves cleaning, aggregating, and restructuring the data. Transformations ensure that data from different sources can be integrated cohesively, improving its quality and making it analysis-ready.</li><li><strong>Load: </strong>The transformed data is then loaded into a target system, typically a data warehouse or datalake, where it is stored and made accessible for analytical queries and business intelligence operations.</li></ul><h4>Overview of Typical Modern Data ETL Used For Data Integration</h4><p>Tools like Fivetran and Airbyte which are data integration technologies, have become an important pillar in building the ETL process in Modern Data Stacks, offering businesses automated solutions to manage their data pipelines efficiently. These tools are designed to minimize the manual effort involved in ETL processes, providing a range of connectors for various data sources and destinations.</p><ul><li><strong>Fivetran </strong>specializes in automating data integration, offering connectors that enable quick and reliable data extraction from hundreds of sources into a data repository. Its strength lies in its simplicity and the breadth of source connectors available.</li><li><strong>Airbyte</strong> is an open-source ETL platform that provides a customizable and extensible framework for data integration. It allows businesses to replicate data from applications, databases, and other sources into their data warehouses, lakes, or databases.</li></ul><h4>The Complexities Involved in ETL Processes</h4><p>Despite the automation offered by ETL tools, the process is fraught with complexities:</p><ul><li><strong>Data Source Variety: </strong>The sheer number of data sources, each with its own format and schema, poses a significant challenge in extracting data consistently and efficiently.</li><li><strong>Transformation Logic:</strong> Developing the logic for data transformation requires a deep understanding of both the source data and the requirements of the destination system. This often involves complex SQL queries or programming to ensure data quality and relevance.</li><li><strong>Performance and Scalability: </strong>As businesses grow, so does the volume of their data. ETL processes must be scalable and performant to handle increasing loads, necessitating sophisticated orchestration and monitoring capabilities.</li></ul><p>ETL tools must navigate these complexities, offering solutions that are not only robust and reliable but also flexible enough to adapt to the changing data landscape. The goal is to ensure that data is not just moved, but transformed into a strategic asset that can drive insights and inform business decisions.</p><p>By dissecting the ETL process and understanding the tools that facilitate it, businesses can better appreciate the critical role ETL plays in their overall data strategy. The next section will delve into Reverse ETL, highlighting how it complements ETL in closing the loop of data management and enabling actionable insights to flow back into operational systems.</p><h3>Unpacking Reverse ETL</h3><p>While ETL processes focus on moving data from various sources into a centralized data warehouse for analysis, Reverse ETL (RETL) serves the opposite function. It is the process of moving data from a centralized repository back into operational systems and tools. This capability is increasingly important as businesses seek to operationalize their insights, making data actionable in real-time across their operational workflows.</p><h4>Definition and Purpose of Reverse ETL</h4><p>Reverse ETL can be defined as the process that takes processed and enriched data from a data warehouse or data lake and pushes it into operational systems and applications like CRM (Customer Relationship Management), marketing platforms, or customer support tools. The primary purposes of Reverse ETL include:</p><ul><li><strong>Actionable Insights: </strong>By feeding analytical insights back into business operations, companies can tailor their strategies and interactions based on comprehensive, data-driven understanding.</li><li><strong>Data Activation:</strong> It allows businesses to utilize their centralized data actively, moving beyond analysis to direct application in daily operations, thus maximizing the value of their data assets.</li></ul><h4>Unique Challenges of Reverse ETL</h4><p>Implementing Reverse ETL processes comes with its own set of challenges:</p><ul><li><strong>Integration Complexity:</strong> Operational systems often have stringent requirements for data formats, requiring precise mapping and transformation. This complexity can escalate with the number of target systems.</li><li><strong>API Limitations: </strong>Many operational tools have API rate limits and data constraints, necessitating efficient data batching and error handling to ensure smooth data flow.</li><li><strong>Data Freshness and Sync Frequency:</strong> Operational decisions often require near-real-time data. Maintaining data freshness and managing synchronization frequency are critical for the effectiveness of Reverse ETL processes.</li></ul><p>By bridging the gap between analytical insights and operational execution, Reverse ETL processes play a crucial role in modern data-driven organizations. They not only extend the value of data in analytical systems but also ensure that this value is realized across the business operations. In the next sections, we will explore why businesses might choose separate tools for ETL and Reverse ETL, the challenges and opportunities of integrating these functionalities, and what the future holds for data management practices.</p><h3>Overview of Tools Specializing in Reverse ETL</h3><p>Several tools have emerged to specialize in Reverse ETL, addressing the unique challenges of pushing data back into operational systems:</p><ul><li>Census and Hightouch are leading platforms in this space, offering robust capabilities to sync data from warehouses to business applications. They provide a range of integrations with popular SaaS tools, ensuring that data moves seamlessly and securely from analysis to action.</li><li>These tools focus on simplifying the configuration process, providing user-friendly interfaces that allow non-technical users to map data from their warehouse to their operational systems without extensive coding.</li></ul><h3>Why Separate Tools for ETL and Reverse ETL?</h3><p>Businesses often encounter the decision of whether to adopt separate tools for ETL (Extract, Transform, Load) and Reverse ETL processes. While at first glance, it might seem efficient to handle both directions of data flow with a single solution, several technical and practical reasons underpin the separation of these tools.</p><h4>Technical and Practical Reasons for Separate Tools</h4><p><strong>1. Specialized Requirements: </strong>ETL and Reverse ETL serve fundamentally different purposes and, as such, have unique requirements. ETL processes are designed to ingest, clean, and prepare data for analysis, requiring robust data transformation and data quality capabilities. Reverse ETL, conversely, focuses on operationalizing insights by moving data from a warehouse/data lake back into business applications, which demands precision in data mapping, synchronization, and often real-time data updates.</p><p><strong>2. Scale of Data:</strong> The scale and volume of data typically handled by ETL processes are much larger than what is usually required for Reverse ETL. ETL tools are built to process and transform large datasets efficiently, preparing them for analysis in a data warehouse or datalake. Reverse ETL deals with a subset of this processed data, pushing it into operational systems where the emphasis is on the relevance and timeliness of data rather than volume.</p><p><strong>3. Customization and Orchestration Needs: </strong>Reverse ETL workflows often require a higher degree of customization to ensure that data fits the specific needs of various operational systems. This includes custom mappings, transformations, and even conditional logic to determine what data gets sent where and when. While ETL tools also offer customization, the nature and complexity of these customizations can differ significantly.</p><p><strong>4. Integration Capabilities:</strong> Operational systems that are targets for Reverse ETL processes (such as CRMs, marketing platforms, and customer support tools) have their own set of API limitations and data requirements. Tools specializing in Reverse ETL are designed with these considerations in mind, offering pre-built connectors and handling API rate limits efficiently, which might not be a primary focus for traditional ETL tools.</p><p><strong>5. Pricing Models : </strong>Pricing models for ETL and Reverse ETL tools often reflect their distinct uses and scales. ETL tools may charge based on the volume of data processed, given their role in handling large datasets. In contrast, Reverse ETL tools might base pricing on the number of syncs or the amount of data moved to operational systems, acknowledging the targeted and strategic nature of these operations.</p><p><strong>6. Customization and Orchestration : </strong>Reverse ETL requires a nuanced approach to data orchestration, ensuring that data is not only moved into operational systems but is done so in a way that aligns with business workflows and triggers. This level of orchestration demands a deep integration with the target systems, understanding their data models, and ensuring that the data being pushed is immediately actionable.</p><p>The decision to use separate tools for ETL and Reverse ETL is not merely a matter of convenience but is rooted in the distinct challenges and requirements of these processes. While the market offers solutions that attempt to bridge these capabilities, understanding the specific needs of your data workflows is crucial in selecting the right tools. As we move forward, we’ll delve into the overlap and integration challenges between ETL and Reverse ETL functionalities, shedding light on the complexities and considerations involved.</p><h3>Future Directions</h3><p>The data landscape is continuously evolving, with new technologies emerging and existing solutions maturing. As businesses become increasingly data-driven, the distinction between ETL (Extract, Transform, Load) and Reverse ETL processes and the tools that support them is subject to change. This section explores potential future directions for these technologies and strategic considerations for businesses looking to navigate this dynamic environment.</p><p><strong>1. Emerging Tools: </strong>New players in the data management space, such as estuary.dev, are challenging traditional boundaries by offering solutions that aim to streamline data flows in both directions. These tools seek to reduce the complexity and overhead associated with using separate systems for ETL and Reverse ETL, proposing a unified platform that can handle data extraction, transformation, loading, and reverse flows more seamlessly.</p><p><strong>2.</strong> <strong>Market Consolidation: </strong>As the industry matures, there is a potential for consolidation, where leading platforms may expand their capabilities to include both ETL and Reverse ETL functionalities. This trend could simplify the technology stack for businesses, offering a more integrated approach to data management.</p><h3>Understanding Specific Business Needs and Data Strategies</h3><p><strong>1. Assess Your Data Landscape:</strong> Begin by thoroughly understanding your current data landscape, including where your data resides, how it’s being used, and the specific processes it supports. This assessment will help you identify the critical features your ETL and Reverse ETL tools need to have.</p><p><strong>2. Identify Your Objectives:</strong> Clearly define what you aim to achieve with your data integration and analytics efforts. Whether it’s improving customer experiences, streamlining operations, or driving innovation, your objectives will significantly influence the type of tools that are best suited to your needs.</p><h3>Strategic Considerations for Businesses</h3><p><strong>1. Selecting Versatile Tools: </strong>Businesses should consider tools that offer flexibility in managing both ETL and Reverse ETL processes, especially as the lines between these operations continue to blur. Solutions that can adapt to changing data needs and integrate with a wide range of data sources and destinations will be invaluable.</p><p><strong>2. Long-term Scalability: </strong>With the volume and complexity of data only expected to increase, selecting tools that can scale efficiently with business growth is crucial. This includes not only the ability to handle larger datasets but also to maintain performance and data integrity across diverse data flows.</p><p><strong>3. Integration Capabilities: </strong>As businesses use an expanding suite of applications and data sources, the ability of ETL and Reverse ETL tools to integrate with these systems becomes a critical consideration. Tools that offer extensive pre-built connectors and robust APIs for custom integrations can significantly reduce implementation time and complexity.</p><p><strong>4. Customization and Flexibility: </strong>Look for tools that offer a high degree of customization and flexibility. Every business has unique data processes, and your tools should be able to adapt to your specific requirements, not the other way around.</p><p><strong>5. User Experience and Support: </strong>Don’t overlook the importance of a good user experience and strong customer support. These aspects can significantly impact the adoption and effective use of the tool within your organization.</p><p><strong>6. Security and Compliance:</strong> Ensure that any tool you consider has strong security features and compliance certifications that meet your industry’s regulations. Data security and privacy should always be top priorities.</p><h3>The Ongoing Evolution of Data Management Tools and Practices</h3><p><strong>1. AI and Machine Learning: </strong>The integration of AI and machine learning algorithms into ETL and Reverse ETL tools is likely to enhance automation capabilities, predictive analytics, and intelligent data mapping. These advancements could further simplify data management tasks and enable more sophisticated data insights.</p><p><strong>2. Increased Focus on Data Governance:</strong> As data privacy and compliance become increasingly important, tools that provide robust data governance and quality controls will be in high demand. This includes features for data lineage, auditability, and compliance with regulations such as GDPR and CCPA.</p><p><strong>3. Cloud-Native Solutions:</strong> The shift towards cloud-native architectures is expected to continue, with ETL and Reverse ETL tools leveraging cloud scalability, resilience, and flexibility. This aligns with the broader trend of businesses moving their data infrastructure to the cloud.</p><p>As we look to the future, businesses must stay informed about developments in ETL and Reverse ETL technologies and practices. By understanding the potential for market consolidation, evaluating tools’ flexibility and scalability, and considering the broader trends shaping the data management landscape, organizations can position themselves to make the most of their data management strategies. The evolution of these tools is not just a technical consideration but a strategic one, as the effective management of data flows is increasingly central to achieving business objectives.</p><h3>Conclusion</h3><p>The distinction between ETL (Extract, Transform, Load) and Reverse ETL processes underscores the complex and evolving landscape of data management in modern businesses. Each serves a unique role in the data lifecycle, with ETL focusing on preparing data for analysis and Reverse ETL on operationalizing insights by integrating them back into business applications. This duality reflects the growing need for agile, informed decision-making processes that leverage data effectively across all facets of an organization.</p><p>Key differences between ETL and Reverse ETL — such as their operational focus, the shape and flow of data, and the specialized tools designed for each process — highlight the importance of selecting the right technology solutions. These decisions are not merely technical but strategic, influencing an organization’s ability to respond to market changes, understand customer needs, and innovate.</p><p>As we navigate this landscape, the emergence of tools that seek to bridge the gap between ETL and Reverse ETL functionalities offers promising avenues for more integrated and efficient data management practices. However, the choice of tools requires careful consideration of factors like scalability, customization, integration capabilities, and the specific business needs they serve.</p><p>The strategic approach to selecting data management tools, therefore, involves a balance between current capabilities and future flexibility, ensuring that businesses can adapt to the ongoing evolution of data practices. Staying informed on developments within this space is crucial for leveraging data management strategies that align with organizational goals and drive value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e911bd2aa820" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/navigating-the-data-pipeline-understanding-etl-and-reverse-etl-e911bd2aa820">Navigating the Data Pipeline: Understanding ETL and Reverse ETL</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Essentials of Git: A Practical Guide]]></title>
            <link>https://medium.com/towards-data-engineering/navigating-the-essentials-of-git-a-practical-guide-aaca8363e705?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/aaca8363e705</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Tue, 05 Mar 2024 05:58:25 GMT</pubDate>
            <atom:updated>2024-03-05T05:58:25.149Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, in my journey through various projects and collaborations, I&#39;ve encountered a recurring observation: Git plays a crucial role, yet surprisingly, many people are not familiar with it. I&#39;ve seen teams, including my own, hit unnecessary roadblocks, not for lack of technical skills, but because of a gap in Git knowledge. Proposals to &quot;twist the architecture&quot; have been made, attempting to circumvent issues that, at their core, required a stronger grasp of Git fundamentals. Here, I aim to demystify Git, offering a concise guide to its most essential commands, accompanied by real-world examples, to empower you to navigate this indispensable tool with confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rXGI1vBSdKi-wx8H" /><figcaption>Photo by <a href="https://unsplash.com/@synkevych?utm_source=medium&amp;utm_medium=referral">Roman Synkevych</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Starting Your Project with Git</h4><p><strong>Git Init: </strong>Begin your project by creating a new Git repository in your project&#39;s directory. Simply run git init, and you&#39;ve laid the foundation for version control.</p><p>Example:</p><pre>cd my-project<br>git init</pre><p><strong>Git Config: </strong>Establish your user identity with Git using git config. This step is crucial for ensuring your commits are accurately attributed to you.</p><p>Example:</p><pre>git config user.name &quot;Mehdi TAZI&quot;<br>git config user.email &quot;mehdi@tazimehdi.com&quot;</pre><p><strong>Git Clone: </strong>To contribute to an existing project, use git clone to make a local copy of a remote repository.</p><p>Example:</p><pre>git clone https://github.com/example/my-project.git</pre><h4>Managing Branches</h4><p><strong>Git Branch: </strong>Create, list, or delete branches. Branches represent different versions of your project, facilitating parallel development.</p><p>Git branches allow developers to diverge from the main codebase to work on new features, fixes, or experiments in parallel, without affecting the main or production code. The purpose is to enable a collaborative, organized workflow, facilitating version control and review processes. A good branching strategy, like <a href="https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow">Git Flow</a> or <a href="https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow">Feature Branch Workflow</a>, involves clear naming conventions, timely merging of branches back to the main line after code review, and regular maintenance of obsolete branches to keep the repository clean and manageable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/0*SAkUjWzoq-hLbnX9" /><figcaption>Photo by : atlassian.com</figcaption></figure><p>Example</p><pre>git branch new-feature</pre><p><strong>Git Checkout:</strong> Switch between branches or create new ones with git checkout.</p><p>Example:</p><pre>git checkout new-feature<br>git checkout -b another-feature</pre><p><strong>Git Merge: </strong>Combine changes from different branches into your current branch.</p><p>Example:</p><pre>git merge feature-branch</pre><p><strong>Git Rebase: </strong>Reapply changes from one branch onto another, creating a linear history.</p><p>Example:</p><pre>git rebase master</pre><blockquote><strong>What is the difference between Git Rebase and Git merge?</strong></blockquote><blockquote>Git Merge lets you merge different Git branches. Git Rebase allows you to integrate the changes from one branch into another. Git Merge logs show you the complete history of commit merging. Git Rebase logs are linear</blockquote><h4>Tracking and Committing Changes</h4><p><strong>Git Add: </strong>Stage changes for commit with git add. This command tells Git which changes you want to include in the next commit.</p><p>Example:</p><pre>git add modified-file.txt</pre><p><strong>Git Status:</strong> Check the status of changes in your repository.</p><p>Example:</p><pre>git status</pre><p><strong>Git Commit: </strong>Securely record changes to the repository with git commit.</p><p>Example:</p><pre>git commit -m &quot;Add a meaningful commit message&quot;</pre><h4>Synchronizing Your Project</h4><p><strong>Git Push: </strong>Upload your local branch commits to the remote repository.</p><p>Example:</p><pre>git push origin main</pre><p><strong>Git Pull:</strong> Update your local branch with changes from its remote counterpart.</p><p>Example:</p><pre>git pull origin main</pre><h4>Reviewing and Debugging Changes</h4><p><strong>Git Log: </strong>Review the commit history.</p><p>Example:</p><pre>git log</pre><p><strong>Git Diff:</strong> Compare changes between commits, branches, or the staging area.</p><p>Example:</p><pre>git diff HEAD~1 HEAD</pre><p><strong>Git Stash: </strong>Temporarily shelve changes so you can work on a different branch.</p><p>Example:</p><pre>git stash</pre><h4>Cleaning Your Workspace</h4><p><strong>Git Rm: </strong>Remove files from your working directory and staging area.</p><p>Example:</p><pre>git rm unwanted-file.txt</pre><p><strong>Git Restore: </strong>Revert changes in your working directory.</p><p>Example:</p><pre>git restore modified-file.txt</pre><p><strong>Git Reset: </strong>Reset your current HEAD to a specified state.</p><p>Example:</p><pre>git reset --hard HEAD~1</pre><p><strong>Conclusion :</strong></p><p>Each of these commands serves as a building block in managing and collaborating on software projects with Git. By understanding and applying these fundamentals, you&#39;ll not only enhance your workflow but also contribute more effectively to your team&#39;s success. Git, with its robust features for version control and collaboration, is a skill worth mastering for any developer. Let&#39;s embrace the journey of continuous learning and improvement, leveraging Git to its full potential.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aaca8363e705" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/navigating-the-essentials-of-git-a-practical-guide-aaca8363e705">Navigating the Essentials of Git: A Practical Guide</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing Your AWS Data Lake — 9 Essential Strategies for Comprehensive Data Lake Fortification]]></title>
            <link>https://levelup.gitconnected.com/securing-your-aws-data-lake-12df5bcc6eea?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/12df5bcc6eea</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Tue, 27 Feb 2024 21:20:29 GMT</pubDate>
            <atom:updated>2024-03-08T09:36:49.358Z</atom:updated>
            <content:encoded><![CDATA[<h2>Securing Your AWS Data Lake</h2><h4>9 Essential Strategies for Comprehensive Data Lake Fortification</h4><p>In my recent role as a Data Cloud Architect for a leading European insurance firm, I had the privilege of steering the security strategy for our data lake, a critical component in safeguarding our organization’s data assets. Reflecting on this experience, I am compelled to share insights into the multifaceted security approach we employed, underscoring its significance in the current data-centric landscape. This article aims to demystify the complexities of data lake security from an architect perspective, drawing from our hands-on implementation, to illuminate the path for others in similar quests for robust data protection. The article delve into the following topics :</p><ul><li>Role-Based Access Control (RBAC) with AWS</li><li>Enhancing Data Security Through Strategic Isolation</li><li>Encryption of Data at Rest and In Transit</li><li>The Implementation of a Custom Authentication App with AWS STS</li><li>Compliance with GDPR and Local Regulations</li><li>Monitoring and Auditing with AWS CloudTrail</li><li>Advanced Data Protection Strategies : Backup, versioning, etc…</li><li>Data Classification and Protection</li><li>Disaster Recovery and Security Training</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TVMANmaVnm83LDjr" /><figcaption>Photo by <a href="https://unsplash.com/@towfiqu999999?utm_source=medium&amp;utm_medium=referral">Towfiqu barbhuiya</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The Genesis of Our Security Strategy</h3><p>Our journey began with the recognition of the inherent vulnerabilities in managing vast amounts of sensitive data. The mission was daunting: secure the integrity, confidentiality, and availability of data across a vast digital ecosystem. The strategy we devised was comprehensive, tackling security from multiple angles to fortify our data lake against an ever-evolving threat landscape.</p><h3>Role-Based Access Control (RBAC) with AWS</h3><p>At its core, RBAC is about ensuring that only authorized users have access to specific resources, based on their roles within the organization. This approach to access control not only enhances security but also simplifies the management and auditing of permissions.</p><p><strong><em>Step 1: Defining Roles and Responsibilities:</em></strong><em> </em>The first step in our RBAC implementation was to meticulously map out the various roles within our organization that interacted with our data lake. This included roles like Data Analysts, Data Scientists, Data Producers... For example Data Analysts , needed access to data for analysis purposes, Data Scientists requiring more extensive access for model training, etc.</p><p><strong><em>Step 2: Identifying Data Access Needs:</em></strong> Once roles were defined, we conducted a thorough analysis to determine the specific data access needs of each role. This process involved understanding the types of data stored in the data lake, the sensitivity of that data (as outlined in our data classification scheme), the public and private datalake zones(organized in buckets and prefixes), and the specific tasks each role needed to perform on each zone.</p><p><strong><em>Step 3: Implementing AWS IAM Policies: </em></strong>With a clear understanding of roles and their access needs, we turned to AWS Identity and Access Management (IAM) to implement our RBAC policies. AWS IAM allowed us to create highly granular policies that defined what actions each role could perform on specific AWS resources. For example, a Data Analyst role might be granted read-only access to certain S3 buckets(zones) containing data sets relevant to their work.</p><p><strong><em>Step 4: Enforcing Least Privilege: </em></strong>A key principle in our RBAC implementation was the enforcement of the least privilege access model on private zones ( see the dedicated section above ). This meant that each role was granted only the permissions necessary to perform its assigned tasks, no more, no less. This minimized the potential for unauthorized access or accidental data exposure by ensuring that users couldn’t access data or resources irrelevant to their roles. Of course, the public datalake zones remain open to all internal users.</p><p><strong><em>Step 5: Continuous Monitoring and Adjustment:</em></strong> Implementing RBAC is not a set-and-forget task. We established ongoing monitoring of role-based accesses and permissions, using tools like AWS CloudTrail for logging and auditing. This allowed us to review access patterns, detect anomalies, and refine our IAM policies over time to ensure they remained aligned with current access needs and security best practices. We implemented a data project dedicated to that, CloudTrail JSON file was sent to Datalake bronze zone, then cleaned and transformed to Parquet in the silver zone. And finally, we applied automatic audit &amp; check controls thanks to a rules engine. The results were stored in the gold layer and exposed through QuickSight.</p><blockquote><strong>Additional sharing :</strong> It worth to note that before leaving the company, we started preparing the shift to an Attribute-Based Access Control (ABAC) strategy as it was more relevant in our case. We experimented that on LakeFormation with it tag-based access control (LF-TBAC)</blockquote><h3>Enhancing Data Security Through Strategic Isolation</h3><p>In our approach to securing the data lake environment within AWS, we implemented an isolation of data in private zone, the private zone includes sensitive data and dedicated areas (S3 buckets) for external services like SaaS softwares, ensuring high levels of data isolation and protection. Here’s how these strategies played out in practice:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cqedey8P5MCAk4KCBZ-NYA.png" /></figure><h4><strong><em>Isolation of Private Zones in Dedicated Buckets</em></strong></h4><p><strong><em>Dedicated S3 Buckets for Sensitive Data:</em></strong> Recognizing the varying sensitivity levels of the data we handled, we established private zones within our data lake by allocating dedicated Amazon S3 buckets for highly sensitive or regulated data (C4 data, as we will see in a future section). This approach allowed us to apply stricter access controls and encryption settings tailored to the sensitivity of the data contained within these buckets.</p><p><strong><em>Enhanced Security Policies:</em> </strong>For these dedicated buckets, we leveraged advanced S3 bucket policies, AWS KMS for encryption, and fine-tuned IAM roles with specific permissions. This ensured that only authorized personnel, with a legitimate need, could access or manage the data, thereby significantly reducing the risk of unauthorized access or data breaches.</p><h4><strong><em>Dedicated Buckets Per External Data Provider</em></strong></h4><p><strong><em>One Bucket per Provider:</em></strong><em> </em>To manage data ingestion from various SaaS and external services effectively, we adopted a strategy of creating a separate S3 bucket for each data provider. This segregation ensured that data from different sources was kept isolated, reducing the complexity of managing access controls and mitigating the risk of cross-contamination between datasets. Within each provider-specific bucket, we further isolated the data by implementing prefixes that corresponded to internal teams. This structure allowed teams to access only the data relevant to their work, preventing unauthorized access to data outside their purview. The use of prefixes, akin to directories within a bucket, facilitated granular access control and simplified data management.</p><p><strong><em>Protection Against Misconfiguration and Data Loss:</em></strong><em> </em>A crucial benefit of this structured approach was the added protection against misconfigurations in access policies. By isolating data at both the bucket and prefix levels, we minimized the risk that a misconfiguration could inadvertently expose sensitive data to the outside world or to unauthorized internal teams. Furthermore, this strategy prevented external providers from performing actions that could lead to data deletion or alteration, as their access was strictly limited to their designated bucket and could be further controlled at the prefix level.</p><p><strong><em>In conclusion :</em></strong> the implementation of dedicated S3 buckets for private zones and structured data ingestion practices significantly bolstered our data lake’s security architecture. This strategic approach not only enhanced data isolation and protection but also provided a scalable framework that could adapt to the evolving needs and complexity of our data environment. By meticulously controlling access at every level — from the bucket down to the prefix — we ensured the integrity and confidentiality of our data, safeguarding it against unauthorized access, misconfiguration, and potential data loss scenarios.</p><h3>Encryption of Data at Rest and In Transit</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7ceZ2Ubr3vA_E2hFfAlRg.png" /></figure><h4>Encryption of Data at Rest Using AWS KMS</h4><p>Encrypting data at rest is fundamental to protecting against data breaches. It ensures that even if data storage is compromised, the information remains secure and unreadable without the encryption keys. We utilized the AWS Key Management Service (KMS) to manage encryption keys. The process involved creating new encryption keys, defining key usage policies, and applying these keys to encrypt our data at rest in S3 buckets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q_fIofpOjCtailKP59a-UA.png" /></figure><p><strong><em>Key Selection and Policies: </em></strong>Important management techniques, including audit trails, access control, and rotation, were carefully taken into account. We improved our security posture by automating many of these activities with the help of AWS KMS. One KMS Key per bucket was used in public zones, and one KMS Key per prefix (team level) was used in private zones. Every three months, KMS keys were rotated.</p><h4>Encryption of Data in Transit Using SSL</h4><p>As data moves between systems, it’s vulnerable to interception. Encrypting data in transit protects it from eavesdropping and tampering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VzYek2UBPKrxAP4zly3DKQ.png" /></figure><p>We enforced SSL/TLS encryption for all data moving to and from our data lake. This included configuring SSL certificates on all endpoints and requiring HTTPS for all data transfers. In addition, tutorials for data consumers and producers were conducted to explain how to implement that using their preferred tools.</p><h3>Custom Authentication App with AWS STS</h3><p>In our quest to improve the security framework of our data lake, we recognized the inherent vulnerabilities associated with the traditional use of AWS IAM user credentials. Standard practice leaves these credentials active until their scheduled rotation — often a quarterly event. This duration poses a significant security risk, as compromised credentials can provide unauthorized access for an extended period. Moreover, relying solely on AWS credentials for data access bypasses the robust security layers offered by corporate Identity Providers (IDPs), leading to potential authentication outside the secure perimeter of the organization’s protocols.</p><p>A particularly concerning scenario arises when considering employees who exit the company. The standard method of credential management increases the risk that former employees could retain access to Datalake without immediate and automatic revocation of their AWS access. This gap in security underscores the necessity for a more dynamic and responsive access control mechanism, one that aligns with the fluid nature of employment and the critical need for data protection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M_ENJ6lcX39Oz-M8eaHwNQ.png" /><figcaption><strong>UnAuthorized access granted</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tsbUn_3IOQI94UItuz2rNQ.png" /><figcaption><strong>UnAuthorized access rejected</strong></figcaption></figure><p>To address these challenges and mitigate the associated risks, we developed a custom authentication application. This innovative solution integrates seamlessly with our corporate IDP, ensuring that user authentication is consistently routed through a secure, centralized system. By leveraging AWS Security Token Service (STS), our app generates temporary, dynamically assigned credentials for authenticated users to act as a dedicated team on which they are authorized. These credentials grant time-limited access to the data lake, significantly reducing the risk of credential compromise. This method ensures that access rights are inherently linked to the user’s current employment status, with automatic expiration of access rights following a short duration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6e5AfTlQQKQxAJfMHTFW8w.png" /><figcaption><strong>Data Access APP Flow</strong></figcaption></figure><p>This approach not only eliminates the security vulnerabilities associated with persistent IAM user credentials but also automates the revocation of access upon employee departure. The moment an individual’s status is updated in the corporate IDP — reflecting their exit from the company — they lose the ability to authenticate and generate new temporary sessions. This mechanism ensures a swift, automated cutoff of access, reinforcing our data lake’s security against unauthorized entry.</p><p>The implementation of our custom authentication app marks a significant advancement in our data security strategy. By transcending the limitations of traditional credential management, we established a secure, efficient, and adaptable access control system. This proactive measure not only protects our data lake from potential breaches but also aligns user access management with the dynamic and evolving nature of our workforce, encapsulating our commitment to maintaining an impregnable data infrastructure.</p><h3>Compliance with GDPR and Local Regulations</h3><p>The strict guidelines for data management and privacy set out by the GDPR and local data protection legislation forced our top European insurance company to create a compliance strategy that went beyond simple legal compliance to also build client trust. In collaboration with our architecture and governance teams, we undertook comprehensive reviews of our data processing activities. This collaborative effort ensured that all aspects of our data storage, processing, and transfer mechanisms were not only compliant but also integrated with the necessary safeguards and consent mechanisms to protect customer data effectively.</p><p>A notable example of our commitment to compliance was in response to the GDPR’s right to erasure, also known as the “right to be forgotten.” Faced with the need to enable the deletion of customer data upon request, we developed a sophisticated system capable of identifying and eradicating customer information across all our databases and Datalake. This initiative required a thorough identification of our data on all systems and backups to ensure that when a customer exercised their right to be forgotten, we could comprehensively remove their data. The most challenging element was the identification of data to remove in backups and AWS S3 versions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k2BFQm9LW_AsuieOFufvUw.png" /></figure><p>Continuous monitoring and auditing are vital for detecting potential security incidents and unauthorized access attempts.<strong><em> </em></strong>We implemented AWS CloudTrail to log and monitor all actions taken on our AWS resources. This allowed us to review access patterns, detect anomalies, and investigate suspicious activities.The data gathered through CloudTrail was instrumental in refining our security policies and response strategies. It also provided evidence for compliance audits.</p><p>As explained in the first section, a data project was put into place specifically for that purpose; the CloudTrail JSON file was transferred to the Datalake Bronze Zone, cleaned, and converted to Parquet in the Silver Zone. And lastly, with the help of a rules engine, we implemented automated audit and check controls. QuickSight was used to expose the results after they had been stored in the gold layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIiEHCyeHGz2XIjVnIOW-g.png" /></figure><h3>Advanced Data Protection Strategies</h3><p>The adoption of advanced data protection strategies is not just a precaution; it’s a necessity. Among these strategies, the implementation of backup and versioning, coupled with the strategic use of separate AWS accounts for backups, stands out as a cornerstone of our approach to ensuring data integrity and resilience against loss.</p><p><strong><em>Backup and Versioning with S3:</em></strong> Our first line of defense involves leveraging Amazon S3’s versioning feature, which automatically keeps multiple iterations of an object within our buckets. This functionality is crucial for scenarios where data needs to be restored to a previous state quickly, whether due to accidental deletion or corruption. We chose to keep data for one week before deleting it. To complement this, we’ve established lifecycle policies that meticulously manage the retention and storage class transition of data. For example, data that is infrequently accessed but must be retained for compliance purposes or data versioning older than 6 days are automatically transitioned from S3 Standard to more cost-effective storage classes like S3 Deep Archive, using Amazon S3’s lifecycle management rules. This not only ensures data availability and integrity but also optimizes storage costs.</p><p><strong><em>Separate AWS Account for Backup Security: </em></strong>Enhancing our data protection strategy further, we store backups in a separate AWS account. This segregation acts as a robust safeguard, insulating our backups from potential threats to our primary account, including ransomware attacks. By isolating backups in this manner, we ensure that, in the event of a security breach, our backup data remains untouched, enabling us to restore critical operations with minimal downtime. This approach is particularly effective against ransomware, as these attacks typically seek to encrypt data for extortion purposes. With our backups securely stored in a separate account, they remain beyond the reach of such attacks, ensuring our ability to recover and maintain business continuity.</p><p>To illustrate, consider a scenario where a ransomware attack compromises our primary AWS account, encrypting data and demanding payment for decryption keys. Thanks to our backups being stored in a separate AWS account, protected by distinct security measures and access controls, we can quickly initiate a recovery process. This involves retrieving clean, unencrypted versions of our data from the backup account, effectively negating the impact of the ransomware attack and safeguarding our operational integrity.</p><p><strong><em>Configuration and Automation:</em></strong> The success of our data protection strategy relies heavily on the configuration and automation of these processes. We restrict access to the backup account, ensuring that only authorized personnel of the security team can manage and restore backups. Additionally, we automate the data movement and lifecycle transitions using AWS’s built-in tools, setting criteria based on data age, access frequency, and regulatory compliance needs. This automation ensures that our data protection measures are consistently applied, reducing the risk of human error and enhancing our overall security posture.</p><p>In conclusion, through the strategic implementation of versioning, lifecycle policies, and the use of separate AWS accounts for backups, our insurance firm has established a robust framework for data protection. This approach not only ensures the integrity and availability of our data but also provides a resilient defense against ransomware and other cyber threats, reinforcing our commitment to safeguarding client data and maintaining trust in our digital operations.</p><h3>Data Classification and Protection</h3><p>To effectively manage and protect our data, we implemented a data classification scheme ranging from C0 to C4. This scheme helped us categorize data based on its sensitivity and the potential impact of its unauthorized disclosure, alteration, or destruction.</p><ul><li><strong><em>C0</em></strong> : represented public data with no restrictions on access or distribution.</li><li><strong><em>C1 :</em></strong> covered internal data that, while not public, had low sensitivity.</li><li><strong><em>C2 :</em></strong> included confidential data that could cause moderate harm if compromised.</li><li><strong><em>C3 : </em></strong>was for sensitive data with high harm potential upon unauthorized disclosure. This data was encrypted at column level</li><li><strong><em>C4 :</em></strong> was reserved for the most sensitive data, requiring the highest protection level due to its criticality to the organization and the severe impact of any unauthorized access.</li></ul><p><strong><em>To illustrate the data protection measures, let’s take, for example, the C4 data :</em></strong></p><ul><li><strong><em>Strict Isolation:</em></strong> C4 data was stored in isolated, secure environments within our data lake. Access to these environments was tightly controlled and monitored, ensuring that only authorized processes and personnel could interact with C4 data.</li><li><strong><em>Encryption: </em></strong>All C4 data was encrypted both at rest and in transit. Utilizing AWS KMS, we implemented strong encryption standards specifically for C4 data, ensuring that even if data were somehow accessed by unauthorized individuals, it would remain indecipherable.</li><li><strong>Restricted Access: </strong>To further protect C4 data from internal threats, we ensured that even those with administrative privileges could not directly access or read the data. This was achieved through a combination of technical controls, such as encryption and access restrictions, and organizational policies mandating the separation of duties and least privilege. Instead, access was granted on a need-to-know basis and required multi-factor authentication and temporary credentials. This approach minimized the risk of insider threats and ensured that only those who absolutely needed access for their work could obtain it.</li></ul><h3>Disaster Recovery and Security Training</h3><p>In our approach to securing the AWS data lake, developing comprehensive disaster recovery plans was paramount to ensuring business continuity, even in the face of potential data breaches or losses. Recognizing the dynamic nature of threats and the critical importance of preparedness, we instituted a rigorous process to not only create but also regularly test and update these plans. This iterative process ensured that our disaster recovery strategies remained robust, relevant, and ready to deploy at a moment’s notice.</p><p><strong><em>Disaster Recovery Planning: </em></strong>Our disaster recovery plans were meticulously crafted, detailing step-by-step procedures for responding to various types of incidents, from minor data corruptions to major breaches or catastrophic data losses. These plans outlined specific roles and responsibilities, ensuring that every team member knew exactly what actions to take in the event of a disaster. Key components included data backup schedules, recovery site locations, and communication protocols to ensure a coordinated and efficient response.</p><p><strong><em>Regular Testing and Updates: </em></strong>To guarantee the effectiveness of our disaster recovery plans, we conducted regular drills simulating various disaster scenarios. These exercises allowed us to evaluate our preparedness, identify any weaknesses in our plans, and make necessary adjustments. Furthermore, we committed to periodic reviews and updates of our disaster recovery strategies to adapt to new threats, technological advancements, and changes in our business operations.</p><p><strong><em>Security Training: </em></strong>Parallel to our disaster recovery efforts, we placed a strong emphasis on regular security training for our staff. These training sessions covered a broad range of topics, including security best practices, compliance requirements, and the latest trends in cyber threats.</p><p><strong>What was missing : </strong>In addition to<strong> </strong>the regular training sessions, the company should go beyond just knowledge acquisition in order to cultivate a pervasive culture of security awareness. Employees should became proactive in identifying potential security risks, adhering to best practices in their daily activities, and understanding the critical role they played in protecting the organization’s data assets. This collective vigilance can significantly enhance the overall security posture.</p><h3>Beyond Implementation: The Uncharted Challenges</h3><p>While the above strategies significantly bolstered the firm data lake’s security posture, the landscape of cybersecurity is perpetually shifting. As I transitioned out of the company, there remained unresolved internal security challenges. The ongoing nature of cybersecurity demands continuous vigilance and adaptation, and I am curious about the evolution of these strategies in my absence.</p><h3>Conclusion</h3><p>Securing a data lake in today’s complex and threat-prone digital environment is no small feat. It requires a holistic approach, blending advanced technology solutions with stringent policy enforcement and continuous education.</p><h3>My Reflection</h3><p>Reflecting on this experience, it’s clear that the principles we applied in securing our data lake are not just technical mandates but strategic imperatives for any data-driven organization. The challenges we faced and the solutions we implemented offer valuable lessons in resilience, innovation, and the unyielding commitment to data privacy and security.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12df5bcc6eea" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/securing-your-aws-data-lake-12df5bcc6eea">Securing Your AWS Data Lake — 9 Essential Strategies for Comprehensive Data Lake Fortification</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elevating Retail Data Insights with Athena]]></title>
            <link>https://medium.com/@mehdi.tazi/elevating-retail-data-insights-with-athena-dbc197d5236f?source=rss-13a60478aec7------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbc197d5236f</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Mehdi Tazi]]></dc:creator>
            <pubDate>Mon, 26 Feb 2024 12:57:31 GMT</pubDate>
            <atom:updated>2024-02-26T12:57:31.788Z</atom:updated>
            <content:encoded><![CDATA[<h4>Unveiling a Game-Changing Strategy</h4><p>In a previous journey crafting solutions for voluminous salesdata of a leading retail entity, I had the objective to optimize data for Business Intelligence (BI) in a way that was both cost-effective and efficient. The centerpiece of this strategy? Amazon Athena.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0q3RIEBGEt7_VILm" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>The Need: A Classical BI Need</h4><p>The company’s data ecosystem was a vast expanse of sales records spread across multiple countries, updated daily. Facilitating immediate access to this data for BI teams in each country was crucial. The aim was to empower them with the ability to generate localized dashboards and reports seamlessly on a monthly basis. Analysis like the number of sold products per month was important to retrieve.</p><h4>Athena at the Core: A Strategic Pivot</h4><p><strong>Choosing Athena: A Calculated Move</strong><br>Amazon Athena emerged as a key component of our approach, chosen for its serverless querying capabilities. This decision pivoted on Athena’s cost-effectiveness, with its pricing model of approximately $5 per terabyte scanned, making it an attractive solution given our extensive data volume.</p><p><strong>Why Athena?</strong><br>Latency, while important, was not our primary concern; cost-efficiency and scalability took priority. Athena’s serverless nature meant we could execute queries directly against our data stored in S3 without managing the underlying infrastructure. This flexibility was crucial in handling our diverse and voluminous datasets.</p><p><strong>Seamless Integration with AWS Glue</strong><br>Athena was adeptly plugged into AWS Glue Data Catalog tables, which were meticulously curated atop our insight layer of data. This integration was key to providing our Tableau dashboards with direct, streamlined access to high-quality, analytical-ready data, significantly enhancing our BI reporting capabilities.</p><p><strong>Presto Over Spark</strong><br>We configured Athena to utilize Presto, prioritizing its high-speed querying capabilities over Spark. This choice was aligned with our need for efficient, ad-hoc query execution against our sales data, stored and ready for analysis in S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LC8cQdgTa45Mm9kfyxJM6Q.png" /></figure><p><strong>Optimizing with Parquet and Strategic Partitioning</strong><br>Underpinning our data strategy was the use of Parquet files, chosen for their efficiency and performance benefits. These files, sized between 256MB and 512MB, struck the perfect balance between scan efficiency and query performance. Parquet’s columnar storage format, combined with its compression and encoding schemes, significantly reduced our storage footprint and query costs. Specifically, we employed Snappy compression for its impressive balance of compression ratio and decompression speed. Snappy, known for its lightweight compression algorithm, was ideal in our context, where minimizing storage without compromising on query execution speed was critical. This choice further optimized our data querying processes, ensuring that Athena could deliver fast, cost-effective insights.</p><p>The decision to implement multi-level partitioning was deliberate and strategic. Partitioning our data by country, year, and month was not just a technical decision; it was deeply rooted in the business needs of our operation. This approach was pivotal due to the specific requirement for dashboarding and analysis on a per-country and monthly basis. By organizing our data in this manner, we could leverage Athena’s partition pruning capabilities, ensuring that queries were not only faster but also more cost-efficient. This strategy meant that BI teams could access precisely the data they needed for their localized dashboards and reports, without unnecessary data processing. The partitioning effectively minimized the data scanned during queries, thereby reducing costs and enhancing performance. It was a tailored solution that addressed the unique analytical needs of the company, enabling country-specific insights with the agility required for monthly reporting cycles.</p><h4>The Outcome: Transformative BI Insights</h4><p>The integration of Athena into our data architecture revolutionized how BI teams accessed and interacted with sales data. The ability to perform serverless queries at a fraction of the cost, coupled with the direct connection to our Tableau dashboards, facilitated a level of agility and insight previously unattainable.</p><h4>Reflecting on the Athena Experience</h4><p>Incorporating Athena into our data strategy was not without its challenges. Balancing the nuances of serverless querying with the operational demands of a large-scale retail data ecosystem required a nuanced approach. However, the payoff in terms of cost savings, efficiency, and enhanced data accessibility was undeniable.</p><h4>Conclusion: A New Benchmark in Data Strategy</h4><p>The successful deployment of Amazon Athena as a pivotal component of our retail data strategy marked a significant milestone. It underscored the power of serverless technology in transforming data management and analytics, setting a new benchmark for cost-effective, scalable BI solutions in the retail sector.</p><p>This project was a testament to the potential of innovative cloud technologies to redefine the landscape of data analytics, offering a glimpse into the future of retail data management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbc197d5236f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>