<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Millan Sanchez on Medium]]></title>
        <description><![CDATA[Stories by Millan Sanchez on Medium]]></description>
        <link>https://medium.com/@millansanchez?source=rss-dffd29cd8c06------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*wf-HCoRL2_qh33NK.</url>
            <title>Stories by Millan Sanchez on Medium</title>
            <link>https://medium.com/@millansanchez?source=rss-dffd29cd8c06------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:51:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@millansanchez/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Open Table Formats: Apache Iceberg vs. Delta Lake vs. Apache Hudi]]></title>
            <link>https://millansanchez.medium.com/open-table-formats-apache-iceberg-vs-delta-lake-vs-apache-hudi-77219dacd420?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/77219dacd420</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Mon, 19 Oct 2026 21:28:00 GMT</pubDate>
            <atom:updated>2026-10-19T21:28:00.457Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q4z_oo7XvlaOpS9QLOt4BA.jpeg" /></figure><p><em>How three projects turned the data lake into a lakehouse, and how to pick one\</em></p><h3>Introduction: The Problem With “Just Files”</h3><p>For about a decade, the data lake was a simple deal: write Parquet or ORC files to cheap object storage such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage, register them in a Hive Metastore, and query them with Spark, Presto, or Hive. It was cheap and scalable. It also broke easily.</p><p>Teams kept hitting the same problems:</p><ul><li>No atomicity. A job that failed halfway left partial files behind, and readers saw corrupted or inconsistent data.</li><li>No safe concurrency. Two writers on the same table could silently overwrite each other’s results.</li><li>Expensive metadata. Hive-style tables relied on directory listings. On object storage with millions of files, query planning alone could take minutes.</li><li>Painful schema and partition changes. Renaming a column or changing a partition scheme often meant rewriting the whole table.</li><li>No row-level changes. GDPR deletes, CDC upserts, and late-arriving corrections meant rewriting entire partitions.</li></ul><p>Open table formats fix this. They add a metadata layer on top of ordinary data files, typically Parquet. That layer defines which files make up the table at any point in time, along with the table’s schema, partitioning, and statistics. The result is database-like behavior on open storage: ACID transactions, time travel, schema evolution, and efficient row-level operations. You don’t need a proprietary engine, and you don’t hand your data to a single vendor.</p><p>Three projects lead this space: Apache Iceberg, Delta Lake, and Apache Hudi. All three solve the same core problem, but they started from different needs, and those origins still shape how each one works.</p><h3>A Shared Architecture</h3><p>It helps to see what the three formats have in common before comparing them. Each table has three layers:</p><ol><li>Data files. Immutable columnar files, almost always Parquet, stored in object storage.</li><li>Metadata / transaction log. A record of which data files belong to each version of the table, plus schema, partition, and statistics information.</li><li>Catalog. A pointer to the current metadata version. Committing a transaction means atomically moving that pointer.</li></ol><p>Each new commit produces a new snapshot. Readers always see a complete, consistent snapshot, and writers use optimistic concurrency control to detect conflicts. Where the formats differ is in how they structure the metadata layer, how they handle updates, and which ecosystem they grew up in.</p><h3>Apache Iceberg: Spec-First and Engine-Neutral</h3><p>Origin: Netflix created Iceberg around 2017 to deal with Hive tables at petabyte scale. It was donated to the Apache Software Foundation and became a top-level project in 2020.</p><p>Design philosophy: Iceberg is a specification first and a library second. Its goal is a table format that any engine can implement correctly, with no dependency on any one engine.</p><h3>How it works</h3><p>Iceberg’s metadata is a tree:</p><pre>Catalog ──► metadata.json (schema, partition specs, snapshots)<br>        └──► Manifest list (one per snapshot)<br>         └──► Manifest files (lists of data files + column stats)<br>          └──► Data files (Parquet/ORC/Avro)</pre><p>Each manifest stores per-file statistics such as min/max values and null counts. That lets query engines prune files without listing directories. Planning cost depends on how much metadata a query touches, not on how many files the table holds.</p><h3>Standout features</h3><ul><li>Hidden partitioning. You partition by a transform such as days(event_ts) or bucket(16, user_id). Users filter on event_ts and Iceberg prunes partitions automatically, so nobody has to remember to filter on a derived event_date column.</li><li>Partition evolution. You can change the partition scheme (for example, monthly to daily) without rewriting old data. Old and new layouts coexist in one table.</li><li>Safe schema evolution. Columns are tracked by unique IDs rather than names, so add, drop, rename, and reorder are all safe.</li><li>Row-level deletes. Positional and equality delete files, plus deletion vectors in format v3, enable merge-on-read updates.</li><li>Format v3 adds the VARIANT type for semi-structured data, geospatial types, default column values, and row lineage.</li><li>REST Catalog specification. A standard HTTP API for catalogs. It has become the integration point across the industry: Apache Polaris, Snowflake Horizon/Open Catalog, AWS Glue, Databricks Unity Catalog, Nessie, and others.</li></ul><h3>Ecosystem</h3><p>Iceberg has the widest multi-engine support of the three. Spark, Flink, Trino, Presto, Dremio, Snowflake, BigQuery, Athena, Redshift, StarRocks, DuckDB, and others can read it, and many can also write it. Native implementations exist in Java, Python (PyIceberg), Rust, and Go.</p><h3>Delta Lake: Simple, Log-Based, Spark-Native</h3><p>Origin: Databricks built Delta Lake and open-sourced it in 2019. It is now hosted by the Linux Foundation.</p><p>Design philosophy: Make Spark pipelines reliable with as little moving machinery as possible. Delta grew up alongside Spark and Databricks, and it shows in its strong Spark integration.</p><h3>How it works</h3><p>Every Delta table has a _delta_log/ directory of ordered JSON commit files:</p><pre>_delta_log/<br> 000000000.json ← add/remove file actions<br> 000000001.json<br> ...<br> 000000010.checkpoint.parquet ← periodic compaction of the log</pre><p>Each commit records <em>actions</em>: add a file, remove a file, change metadata. Replaying the log, starting from the latest Parquet checkpoint, reconstructs the current table state. Atomicity comes from writing the next numbered commit file with “put-if-absent” semantics.</p><h3>Standout features</h3><ul><li>A simple mental model. The transaction log is easy to inspect, debug, and reason about.</li><li>Deep Spark integration. MERGE, UPDATE, and DELETE are first-class, and unified batch and streaming with Structured Streaming is mature and well proven.</li><li>Change Data Feed (CDF). Row-level change streams for building incremental pipelines downstream.</li><li>Deletion vectors. Fast merge-on-read deletes and updates without rewriting files.</li><li>Liquid clustering. Adaptive, incremental clustering that replaces rigid Hive-style partitioning and Z-ordering for many workloads.</li><li>Delta 4.x features include the VARIANT type, type widening, row tracking, and coordinated commits for multi-writer and multi-engine scenarios.</li><li>UniForm. Writes Iceberg (and Hudi) metadata alongside the Delta log, so Iceberg clients can read Delta tables.</li></ul><h3>Ecosystem</h3><p>Delta is strongest in the Spark and Databricks world. Outside Spark, the delta-rs (Rust/Python) and Delta Kernel projects have broadened support to Trino, Flink, DuckDB, Polars, and various cloud warehouses. Feature coverage across engines can vary, though, especially for newer table features.</p><h3>Apache Hudi: Built for Streaming Upserts</h3><p>Origin: Uber created Hudi (Hadoop Upserts Deletes and Incrementals) in 2016 to ingest large volumes of rapidly changing data, such as trip records, with low latency. It became an Apache top-level project in 2020.</p><p>Design philosophy: Hudi is more than a table format. It is a data lake platform focused on incremental processing, with built-in services for ingestion, indexing, compaction, and clustering.</p><h3>How it works</h3><p>Hudi organizes data into file groups, each identified by a record key. All actions are recorded on a timeline of instants (commits, compactions, cleans, and so on) stored under .hoodie/. Hudi offers two table types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*5LGmhRJPLsnPhJb-" /></figure><h3>Standout features</h3><ul><li>Primary keys and indexing. Records have keys, and Hudi maintains indexes (bloom, simple, bucket, record-level, and secondary indexes in 1.x) to find the file containing a record quickly. This makes high-volume upserts efficient.</li><li>Incremental queries. Ask “give me every record that changed since commit X,” a natural fit for chaining incremental ETL pipelines.</li><li>Built-in table services. Compaction, clustering, cleaning, and file sizing can run inline or asynchronously, managed by Hudi itself.</li><li>Hudi Streamer (formerly DeltaStreamer). An ingestion utility for Kafka, CDC sources, and DFS.</li><li>Hudi 1.x introduced an LSM-tree-based timeline, non-blocking concurrency control for concurrent streaming writers, functional and secondary indexes, and partial updates.</li></ul><h3>Ecosystem</h3><p>Hudi works well with Spark and Flink and is readable from Trino, Presto, Hive, Athena, EMR, and others. Its community is smaller than Iceberg’s or Delta’s, and it has more configuration surface to learn. That cost buys fine-grained control over write performance.</p><h3>Head-to-Head Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eNszZXTNatXwBP1v" /></figure><h3>Interoperability</h3><p>The “format wars” have cooled. Features have converged (all three now offer deletion vectors or an equivalent, variant types, and row-level change tracking), and interoperability tooling has matured:</p><ul><li>Delta UniForm exposes Delta tables as Iceberg tables.</li><li>Apache XTable (incubating, formerly OneTable) translates metadata between Iceberg, Delta, and Hudi without copying data files.</li><li>Iceberg REST Catalog has become a common catalog interface that vendors implement, including catalogs from vendors historically aligned with other formats.</li></ul><p>In practice, Iceberg has become the de facto standard for multi-engine interoperability and Snowflake is a strong supporter, Delta remains the native choice in Databricks/Fabric environments, and Hudi keeps a strong niche in high-frequency upsert and CDC workloads. Because the data files are all Parquet, you are no longer as locked in by your choice as you would have been a few years ago.</p><h3>How to Choose</h3><p>Choose Apache Iceberg if:</p><ul><li>Several engines (Spark, Trino, Flink, Snowflake, Athena, and so on) need to read and write the same tables.</li><li>Vendor neutrality and an open catalog standard matter to you.</li><li>You have very large tables and want partition evolution and hidden partitioning.</li></ul><p>Choose Delta Lake if:</p><ul><li>Your platform is centered on Spark/Databricks/Fabric.</li><li>You want the simplest operational model with mature streaming plus batch.</li><li>You value features like liquid clustering and Change Data Feed in a tightly integrated stack.</li></ul><p>Choose Apache Hudi if:</p><ul><li>Your main workload is high-volume upserts, CDC replication, or near-real-time ingestion.</li><li>You need record-key indexing and native incremental pulls for chained pipelines.</li><li>You want built-in, managed table services and are willing to tune configuration.</li></ul><p>Whichever you choose: plan for table maintenance. Compaction, snapshot expiration, orphan file cleanup, and clustering are required in all three formats. A neglected table slows down no matter what format it uses.</p><h3>Conclusion</h3><p>Open table formats changed the data lake from a pile of files into a reliable, transactional, analytics-ready foundation, the basis of the modern lakehouse. Apache Iceberg leads on openness and engine breadth, Delta Lake with Spark integration, and Apache Hudi on streaming upserts and incremental processing.</p><p>The better question is often less “which format wins?” and more “which catalog, engines, and workloads do I need to support?” As interoperability layers like UniForm, XTable, and the Iceberg REST Catalog mature, the open table format becomes a shared contract: your data stays in open files that any engine can query.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77219dacd420" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Teradata Vantage to Fabric Migration: Tools for Migrating from Teradata to Microsoft Fabric…]]></title>
            <link>https://millansanchez.medium.com/the-teradata-vantage-to-fabric-migration-tools-for-migrating-from-teradata-to-microsoft-fabric-f0a8d4c0b497?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0a8d4c0b497</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[teradata]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Wed, 12 Nov 2025 15:23:44 GMT</pubDate>
            <atom:updated>2025-11-12T15:23:44.693Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19t0PKWkdeK-s7dEMTbo_Q.png" /><figcaption>Teradata Vantage to Microsoft Fabric Migration</figcaption></figure><h3>The Teradata Vantage to Fabric Migration: Tools for Migrating from Teradata to Microsoft Fabric Data Warehouse</h3><p>This third article covers the different tools we have in the market to help with the migration from Teradata to Microsoft Fabric. Here you can find <a href="https://millansanchez.medium.com/during-the-last-few-weeks-i-have-been-working-with-a-customer-to-evaluate-their-migration-from-415beed845f4">Article 1</a> and <a href="https://millansanchez.medium.com/this-second-article-explores-the-fundamental-differences-between-teradata-sql-with-bteq-and-07a35413550e">Article 2</a>.</p><p>Migrating a legacy Teradata data warehouse to the modern Microsoft Fabric Data Warehouse is a complex project. Teradata’s on-premises or Vantage architecture differs significantly from the cloud-native architecture of Microsoft Fabric. This means organizations must not only transfer large volumes of data but also translate SQL code, ETL processes, and analytics to run on Fabric’s platform. Fortunately, several specialized tools and solutions can accelerate this migration. Below, we present some alternatives (from Datafold, Bitwise, and Adastra), explaining what each product does, how it works, and its architecture and main components. This is not a comparison, but a description of different options available for Teradata-to-Fabric migrations.</p><h3>Datafold Migration Agent (AI-Powered Code Translation &amp; Validation)</h3><p>Datafold offers an AI-driven migration tool called the <em>Datafold Migration Agent (DMA)</em>, designed to automate two of the hardest parts of a warehouse migration: translating Teradata code into the new system and validating that the data moved correctly (<a href="http://www.datafold.com/">www.datafold.com</a>). Datafold’s approach combines advanced AI (large language models) with its proprietary data “diff” technology to ensure that code and data in Microsoft Fabric produce the same results as they did in Teradata. This focus on automated validation means teams can migrate faster while maintaining trust in the data.</p><p>How it works: Datafold’s DMA consists of three core components that work in sequence to achieve reliable migration results:</p><ul><li>Source Aligner — First, DMA takes a “snapshot” of the Teradata data and loads it into the Fabric environment, ensuring the source and target inputs are perfectly synchronized .By freezing and aligning the input datasets on both sides, the tool eliminates noise from any data discrepancies, so subsequent comparisons are meaningful. This guarantees that when code is tested, any differences in output are due to code issues and not mismatched source data.</li><li>Feedback Loop (Translate, Diff, Repeat) — Next, the agent automatically translates Teradata SQL code (including queries, scripts, or ETL logic) into the target SQL dialect(Fabric T-SQL), using AI to handle the conversion. It then runs the translated code on the aligned data in Fabric and diffs the results against the original Teradata output. If any discrepancies are found, the agent refines the code translation and repeats the process. This iterative loop continues until the Fabric output matches the Teradata output exactly, ensuring functional equivalence. The use of machine learning and differential testing greatly reduces manual rewriting and debugging of complex SQL.</li><li>Code Delivery with Validation — Once the translation is verified, DMA delivers the final migrated code along with proof of its correctness. It generates a Git repository or pull request containing all the converted SQL/ETL code for Fabric, so engineers can review and integrate it. It also provides links to detailed data diff reports — value-level comparisons of entire tables — demonstrating that the data in Fabric matches the data from Teradata after migration. These auditable diff reports serve as evidence to stakeholders that the migration is successful and no data has been lost or changed inappropriately. In this way, Datafold not only automates the migration but also builds confidence via transparent validation.</li></ul><p>Architecture and components: Under the hood, Datafold’s migration tool leverages a combination of cloud connectors, an AI/LLM-based translation engine, and the Datafold platform’s monitoring UI. The DMA can connect to Teradata and Fabric (as well as other databases) to pull schema and data, and it runs either in the cloud or in a customer’s environment depending on security needs. The key components include the AI translation engine (for SQL/code conversion), the data diff engine (for cross-database comparisons), and pipeline integration for CI/CD. This design allows Datafold to integrate with various legacy systems (Teradata, Oracle, etc.) and output to modern frameworks (including SQL-based warehouses or dbt code). By automating translation and validation, Datafold’s solution can significantly accelerate migration timelines — by up to 6× faster than manual methods, according to the company — without sacrificing data quality.</p><h3>Bitwise — Automated End-to-End Migration Suite</h3><p>Bitwise provides a migration suite and methodology, geared towards end-to-end modernization of data platforms (they are a Microsoft Fabric migration partner). Their solution is a combination of proprietary automation tools and services that cover everything from analyzing the Teradata environment to loading data into Fabric and validating the outcome (<a href="http://www.bitwiseglobal.com/">www.bitwiseglobal.com</a>). Bitwise emphasizes a code-free conversion approach — aiming to automate as much of the migration as possible — which can drastically cut down manual effort and project timelines. In fact, Bitwise reports that its automation can eliminate ~80% of manual coding and reduce overall migration time on the order of 75–80%. The Bitwise migration toolkit uses both traditional automation and generative AI to achieve these results.</p><p>How it works: Bitwise’s migration architecture comprises several key components (accelerators), each handling a crucial phase of the Teradata-to-Fabric migration:</p><ul><li>Database Object Analyzer — This tool scans and analyzes the entire Teradata source system to map out what needs to be migrated. It performs a comprehensive analysis of Teradata database objects, identifying all tables, stored procedures, macros, ETL scripts, and their interdependencies. It flags complexities or potential issues in the legacy code (for example, proprietary Teradata SQL extensions or heavy stored procs) and provides detailed and summary reports. These reports help the team understand the scope and complexity up front, allowing for better planning and effort estimation before conversion. In essence, the Analyzer builds an inventory and lineage of the Teradata environment, so nothing is missed.</li><li>Database Objects Converter — Bitwise’s conversion engine automatically translates Teradata code to the target platform’s code. Using a combination of rule-based parsing and AI (LLMs), it can convert Teradata SQL dialect (including BTEQ scripts, macros, functions, and stored procedures) into equivalent code for the Fabric environment. For example, Teradata’s proprietary functions or stored proc logic can be translated either into T-SQL for Microsoft Fabric or into PySpark code, depending on the chosen Fabric component. The converter generates platform-agnostic PySpark or SQL code that adheres to coding best practices on the cloud. By leveraging AI to handle code translation, Bitwise significantly reduces manual recoding efforts, improving developer productivity and consistency. In many cases, this code converter achieves a high percentage of automated conversion, with minimal hand-editing required.</li><li>Data Migration Framework — This is the component that handles moving the data itself from Teradata into the Microsoft Fabric environment. Bitwise’s framework is flexible: it leverages cloud-native migration tools (where appropriate) as well as Bitwise’s own automation to perform bulk data transfer and loading. For instance, depending on the scenario, it can integrate with Azure’s SQL Migration Assistant (SSMA) or Database Migration Service (DMS) to copy schema and data into. It can also use Azure Data Factory pipelines or Fabric’s Data Factory integration (Dataflow Gen2) for large-scale ETL. The framework includes support for Change Data Capture (CDC): Bitwise provides a CDC accelerator to capture ongoing changes from Teradata and apply them to the Fabric target, which is useful if the migration needs to happen with minimal downtime. This ensures that even if Teradata is still receiving new transactions during the migration period, those changes can be migrated incrementally to keep the target in sync. Overall, the Data Migration Framework orchestrates the data extraction, transformation (if needed), and loading in a scalable, controlled manner, and can adapt to different customer needs or toolsets.</li><li>Data Validation Utility — After data and code are migrated, Bitwise uses a validation tool to ensure everything in Fabric matches the original Teradata results. This utility is a Python-based testing framework that performs both schema validation (checking that all tables, columns, data types, and constraints were recreated correctly) and data validation (verifying that table row counts and contents match between Teradata and Fabric. It can run either full data comparisons or sample-based checks for very large tables and can even test the execution of migrated stored procedures, views, or macros to confirm they produce the same outcomes on Fabric. The validation tool is designed for high performance and can integrate with CI/CD pipelines or scheduling systems to automate the testing process. By certifying the accuracy of migrated data and code, it helps catch any discrepancies early and gives stakeholders confidence in the migration outcome.</li></ul><p>Architecture and approach: Bitwise’s solution is typically delivered as part of a consulting engagement, but it is powered by these proprietary software accelerators. The architecture is modular — each tool (Analyzer, Converter, etc.) can operate independently, but together they form a pipeline for migration. Bitwise often begins with an assessment phase (using the Analyzer for discovery), then uses the Converter and Framework in parallel to translate code and transfer data, and finally the Validation stage to test everything. The use of automation and AI throughout means much of the Teradata logic is migrated to Fabric without manual rewriting, and the process is faster and less error-prone. Bitwise also emphasizes best practices and optimization during conversion — for example, converting Teradata BTEQ scripts into efficient Spark jobs or Fabric-native pipelines where appropriate . As a result, the migrated solution is not only functionally equivalent, but also tuned for the Fabric environment.</p><h3>Adastra Data Migration Engine and “Adele” Platform</h3><p>Adastra, offers its own suite of tools for Teradata-to-Fabric migrations as part of their Microsoft Fabric services. Two flagship components of Adastra’s solution are the Data Migration Engine (DME) and Adele — which work in tandem to handle the migration of data and the conversion of logic, respectively (<a href="http://adastracorp.com/">adastracorp.com</a>). Adastra’s focus is on providing an automated, transparent, and flexible migration process. This includes robust planning, auditable execution, and use of AI-driven insights to reduce manual effort. Like Bitwise, Adastra’s offering is typically delivered with their expert consulting services, but it is underpinned by these specialized tools that accelerate the migration.</p><p>How it works: Data Migration Engine (DME). This is Adastra’s core framework for transferring large volumes of data from Teradata into the Fabric cloud data warehouse. DME is designed to migrate complex, disparate datasets from legacy systems to modern platforms with <em>“unparalleled precision,”</em> minimizing the risks of traditional big-bang migrations. In practice, the Data Migration Engine provides a structured, auditable process to extract data out of Teradata and load it into Microsoft Fabric. The engine emphasizes transparency — offering a single, consolidated view of the migration progress and detailed logging so that every record can be tracked. It also supports flexibility in execution, meaning migrations can be done in phases or waves to reduce business disruption. (For example, Adastra often conducts a phased migration where subsets of data are moved and validated iteratively, rather than all at once.) According to Adastra, their DME approach has achieved 100% success rates in complex projects — one case study involved unifying two banking systems and migrating hundreds of thousands of accounts with no data loss. The architecture likely makes use of an underlying data transfer tool — in fact, Adastra’s portfolio includes Adoki, an engine for automating data transfers and replication between systems. Adoki can handle both one-time bulk migrations and ongoing data synchronization, adjusting to the source/target systems’ workloads. In a Teradata migration scenario, Adoki (as part of DME) would manage extracting data from Teradata (using connectors or JDBC/ODBC) and writing into Fabric (using Fabric’s ingestion APIs or Azure Data Factory), while monitoring performance and retrying any failures. Overall, DME provides the infrastructure and monitoring needed for a reliable transfer of data at scale, with capabilities to pause/resume tasks, handle errors gracefully, and ensure data integrity throughout the process.</p><p>Adele (Migration of SQL/ETL logic): <em>Adele</em> is Adastra’s migration platform focused on translating and rebuilding Teradata’s analytical code and processes on the target Microsoft Fabric platform. It is described as a user-friendly solution to <em>“migrate your existing SQL and ETL jobs to any cloud platform with accuracy and speed.”</em> .In the context of Teradata, Adele can ingest things like Teradata SQL scripts, BTEQ scripts, stored procedures, and ETL workflows from tools such as Informatica or DataStage, and then help convert or re-platform them for Fabric. One of Adele’s strengths is its intelligent mapping and reverse-engineering capabilities. It can parse the legacy Teradata workflows to extract metadata and generate data lineage mappings — essentially documenting how data flows through the old system. By doing so, Adele provides valuable insights: it identifies dependencies between database tables, ETL jobs, and reports, and can even point out unused or redundant components in the Teradata environment. This metadata harvesting helps organizations “declutter” during migration — for example, if certain ETL jobs are no longer needed, they can be retired rather than migrated.</p><p>Once the analysis is done, Adele supports automated conversion of a large portion of the legacy logic. According to Adastra, the platform can comprehend and auto-translate up to ~80% of Teradata’s legacy code into the target form. This might involve converting Teradata SQL into T-SQL or Spark for Fabric, and converting ETL workflows into Azure Data Factory pipelines or Fabric Dataflows. For any portions that cannot be fully automated, Adele provides a guided interface where developers can manually refactor the logic with help from the lineage insights. The platform emphasizes giving users full control over the migration process — meaning you can review and adjust the mappings and conversion as needed (it’s not a black-box). It also features visualization dashboards for data lineage and impact analysis, which help in validation and testing. Engineers can easily see, for example, that a Teradata stored procedure produced certain output, and confirm the Fabric equivalent (perhaps a combination of a SQL script and a Power BI report) produces the same result. By automating much of the conversion and providing rich documentation of the old and new systems.</p><p>Architecture and components: Adastra’s migration toolset is part of their OptiSuite platform, and these components (DME, Adele, Adoki, etc.) are integrated to work together. The Data Migration Engine likely acts as the orchestrator — it manages the overall project, scheduling data extraction, overseeing the code conversion, and deploying converted jobs to Fabric. Adoki serves as the data movement engine underpinning DME, handling the connectivity and transfer between Teradata and Fabric with support for various sources, targets, and schedules. Adele operates at the application layer — it interfaces with source control and repository of Teradata scripts, performs parsing (using parsers for Teradata SQL, BTEQ, etc.), and uses a combination of rule-based algorithms and possibly AI to convert that into Fabric-compatible code. It then outputs the new code or pipeline definitions (for example, generating SQL scripts for Fabric or JSON for Azure Data Factory), which can be deployed. Adele also houses a metadata repository that stores the lineage and mapping information gleaned from the source, and a UI that allows users to visualize and verify this information. Together, these components enable a transparent migration process: users can track every step (via dashboards or reports), and the system ensures auditability (every record migrated and every piece of logic translated can be traced back to the source).</p><h3>Conclusion</h3><p>Migrating from Teradata to the Microsoft Fabric Data Warehouse involves moving not just data but an entire analytics ecosystem. The tools described above each tackle this challenge in a unique way and should be evaluated as part the methodology to migrate to Fabric.</p><p>Each solution addresses the core needs of Teradata migrations — speed, accuracy, and risk reduction — with different strengths. Depending on an organization’s priorities (be it maximum automation, in-depth lineage analysis, or seamless integration with cloud services), one of these alternatives may be the best fit. By leveraging such tools, companies can significantly de-risk and streamline the journey from a Teradata warehouse to Microsoft’s Fabric, ultimately arriving at a modern cloud data warehouse with confidence in the integrity of their migrated data and processes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0a8d4c0b497" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Data Agents: An intimate and spicy TV plot]]></title>
            <link>https://millansanchez.medium.com/ai-data-agents-an-intimate-and-spicy-tv-plot-0e9f224fe377?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e9f224fe377</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[sex]]></category>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Tue, 28 Oct 2025 01:09:53 GMT</pubDate>
            <atom:updated>2025-10-28T01:09:53.714Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8pl05XWpfMqAPLdp65i_Q.png" /><figcaption>AI Data Agents</figcaption></figure><h3>Introduction to an awkward AI Enlightenment</h3><p>So, you’re about to read a TV Plot that explains artificial intelligence using sex as the primary metaphor. Yes, really. No, it’s not what your parents think you’re reading. Well, maybe it is a little bit and by the end of these five episodes, you’ll understand AI better than most people. Your brain already knows how attraction works, how partners learn what you like, and why consent matters. Now apply that to AI and….boom: You’re an AI expert. You’re welcome, Mom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dCAJI_bilMxM-eRtWomx8w.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4opbOecdr8CqpO6PtPHL7A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s581DjbL-Zhefe_72JsUgg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W6GDEnxyGFVJecyKdu4Jvw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4DqgjVS5eQCa1xQ4iFSOrw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MfhLqC4Ip3LtvmPBEgqLuw.jpeg" /></figure><h3>The plot:</h3><h3>Episode 1: Why Good Listening Matters (In Multiple Contexts)</h3><p>The first episode teaches you that real intelligence starts with attention. Just like a partner who actually pays attention instead of just waiting for their turn to talk, an AI Agent’s superpower is <em>listening</em>. It’s not just hearing words — it’s tracking micro-signals, noticing patterns, and showing genuine interest in what makes you tick. This is the foundation of machine learning, and it turns out that the characteristics of a good listener in a relationship are literally the same characteristics of a good AI system. Whoever designed this analogy clearly understood that humans learn better when concepts are connected to things they already feel, and honestly? That’s good pedagogy disguised as spicy content. Your data teacher is basically seducing you into understanding predictive algorithms.</p><h3>Episodes 2 &amp; 3: The Magic of Pattern Recognition (And Perfect Timing)</h3><p>Here’s where things get wild: you learn that an AI Agent’s ability to anticipate your needs isn’t magic — it’s math. Beautiful, elegant, just math. Episode 2 shows how an Agent learns the intricate map of what makes you tick, while Episode 3 reveals the ultimate payoff: prediction without prompting. This is where the analogy becomes absolutely brilliant, because the best sexual partners aren’t the ones who randomly try things — they’re the ones who’ve internalized <em>your</em> patterns and respond accordingly. That’s literally what machine learning does with data. By the time you finish these episodes, you’ll understand deep learning’s core concept: give the system enough examples, and it can predict the next move before you make it. The narrative progression from “I’m nervous” to “How did you know that’s exactly what I needed?” perfectly mirrors the journey from untrained AI to sophisticated prediction models.</p><h3>Episode 4: Consent as the Actual Foundation of Everything</h3><p>This episode is where the guide gets philosophical and slightly intense. It establishes that all the magic, all the anticipation, all the personalization, all the perfect synchronization means absolutely nothing without explicit consent and respected boundaries. This is crucial, because it teaches something vital about AI that’s often glossed over: power without ethics is just surveillance. The conversation about privacy, safe words, and explicit permission is basically why we should care about how companies use our information. You’re learning about data governance, privacy, regulations, and digital ethics.</p><h3>Episode 5: Specialization and the Beauty of Coordinated Systems</h3><p>The finale reveals that one Agent is impressive, but coordinated specialized systems are where the real power lives. This is where you learn about multi-agent systems and distributed AI without any of it feeling like a computer science textbook. It is different layers and components doing different jobs but creating one seamless experience. You learn that complexity doesn’t have to feel complicated from the outside. You learn that specialization plus coordination equals elegance. And somehow, a story about intimate connection has taught you the fundamental architecture of how modern AI actually works. It’s sneaky. It’s effective. It’s kind of beautiful.</p><h3>The Real Takeaway: You’re Now Ready to Discuss AI Agents Like You Actually Understand It.</h3><p>I am not sure if my mom will want to understand about AI Agents using sex metaphors, but i will try it anyway =)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e9f224fe377" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Teradata Vantage migration to Microsoft Fabric]]></title>
            <link>https://millansanchez.medium.com/this-second-article-explores-the-fundamental-differences-between-teradata-sql-with-bteq-and-07a35413550e?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/07a35413550e</guid>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Wed, 22 Oct 2025 14:35:54 GMT</pubDate>
            <atom:updated>2025-10-28T01:11:26.963Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19t0PKWkdeK-s7dEMTbo_Q.png" /><figcaption>Teradata Vantage migration to Microsoft Fabric</figcaption></figure><h3>The Teradata Vantage to Microsoft Fabric Migration: differences between BTEQs and Fabric T-SQL</h3><p>This second article explores the fundamental differences between Teradata SQL with BTEQ and Microsoft Fabric Data Warehouse’s T-SQL implementation. This is fundamental and critical knowledge for data professionals planning migrations.</p><h3>What is Teradata SQL?</h3><p>Teradata SQL is a dialect of ANSI SQL specifically designed for Teradata. While it adheres to ANSI SQL standards, Teradata SQL incorporates powerful extensions and optimizations tailored for large-scale data warehousing, business intelligence, and advanced analytics. These enhancements enable Teradata to efficiently handle massive parallel processing (MPP) workloads across distributed architectures.</p><p>Teradata uses a Massively Parallel Processing (MPP) architecture, which spreads data across multiple nodes for concurrent processing. It closely integrates data storage with computational power, which is important to understand as you migrate to Microsoft Fabric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/0*AsqlDtJSKTYqQW-z" /></figure><p>Teradata Architecture</p><p>Teradata clusters are composed of a network of Access Module Processors (AMPs), which serve as the core drivers of its data processing prowess. Each AMP operates on an individual server instance, bearing a share of both computational and storage duties for the cluster. In this configuration, storage and compute resources are tightly coupled, with specific data segments being stored and processed on the same server.</p><p>The cornerstone of Teradata’s architecture is its Primary Index (PI) mechanism, which fundamentally differs from traditional database indexing. Every Teradata table must have a Primary Index — either a Unique Primary Index (UPI) or Non-Unique Primary Index (NUPI) — that determines how data is physically distributed across AMPs (Access Module Processors) using a hashing algorithm. This distribution mechanism enables Teradata’s legendary parallel processing capabilities and single-AMP retrieval performance when queries reference the Primary Index in WHERE clauses.</p><p>Teradata also SQL introduces several unique concepts not found in standard SQL implementations:</p><p>SET vs MULTISET Tables: Teradata distinguishes between SET tables (which automatically prevent duplicate rows) and MULTISET tables (which allow duplicates). By default, tables are created as SET tables, which perform duplicate row checking on every insert or update — potentially impacting performance on large datasets. MULTISET tables skip this overhead, making them preferable for most modern data loading scenarios.</p><p>QUALIFY Clause: One of Teradata’s most powerful extensions is the QUALIFY clause, which filters results based on window functions like ROW_NUMBER(), RANK(), or DENSE_RANK() without requiring subqueries. This eliminates nested SELECT statements and improves query readability. For example:</p><pre>SELECT employee_id, salary, department</pre><pre>FROM employees</pre><pre>QUALIFY ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) &lt;= 3;</pre><p>This query returns the top 3 highest-paid employees per department — a common analytical pattern made elegant by QUALIFY.</p><p>Volatile and Global Temporary Tables: Teradata offers sophisticated temporary table options. VOLATILE tables exist only during a session, use spool space rather than permanent space, and are automatically dropped when the session ends. GLOBAL TEMPORARY tables have their definitions stored permanently but materialize data only during sessions. These provide flexible options for intermediate result storage without consuming permanent database space.</p><h3>What is BTEQ?</h3><p>Basic Teradata Query (BTEQ) is Teradata’s primary interactive query and scripting utility. BTEQ serves multiple critical functions beyond simple query execution:</p><p>Session Control: BTEQ manages database connections through LOGON, LOGOFF, and session configuration commands. It supports multiple concurrent sessions and allows users to specify transaction modes, character sets, and response buffer settings.</p><p>Data Import and Export: BTEQ’s IMPORT and EXPORT commands enable efficient data movement. The EXPORT command can output query results in various formats (REPORT, DATA, FASTLOAD) and supports customizable formatting options. IMPORT reads data from external files for use in SQL statements with USING modifiers.</p><p>Script Control Flow: BTEQ provides robust scripting capabilities with IF/THEN/ELSE conditional logic, GOTO labels, error handling via ERRORCODE and ACTIVITYCOUNT variables, and REPEAT commands for iterative execution. This enables sophisticated ETL workflows and automated data processing.</p><p>Report Formatting: BTEQ includes extensive formatting controls for output, including headers, footers, column separators, page breaks, and field formatting options. These capabilities make BTEQ suitable for generating production reports directly from the database.</p><p>A typical BTEQ script demonstrates these capabilities:</p><pre>.LOGON 192.168.1.102/dbc,dbc;</pre><pre>.EXPORT REPORT FILE=output.txt</pre><pre>SELECT * FROM Employee WHERE DepartmentNo = 100;</pre><pre>.EXPORT RESET</pre><pre>.IF ERRORCODE &lt;&gt; 0 THEN .EXIT ERRORCODE;</pre><pre>.LOGOFF;</pre><p>While BTEQ remains widely used, it has been supplemented by more powerful utilities like Teradata Parallel Transporter (TPT), which consolidates FastLoad, MultiLoad, TPump, and BTEQ functionality into a unified framework.</p><h3>Microsoft Fabric Data Warehouse and T-SQL</h3><p>Microsoft Fabric Data Warehouse represents a modern, cloud-native approach to enterprise data warehousing. Unlike Teradata’s tightly coupled compute and storage architecture, Fabric Warehouse separates compute from storage, storing all data in Delta Lake format within OneLake — Microsoft’s unified data lake.</p><p>T-SQL Implementation: Fabric Warehouse uses T-SQL (Transact-SQL), the SQL dialect from SQL Server and Azure Synapse Analytics. However, it implements a subset of full T-SQL functionality with specific limitations tailored to the distributed, cloud-native architecture.</p><p>Supported T-SQL Features in Fabric Warehouse include:</p><ul><li>Standard DML operations (SELECT, INSERT, UPDATE, DELETE) in Warehouse mode</li><li>CREATE/DROP TABLE with support for standard data types</li><li>Views, inline table-valued functions, and stored procedures</li><li>Session-scoped distributed #temp tables (Warehouse only)</li><li>Standard, sequential, and nested CTEs</li><li>MERGE syntax (preview feature)</li><li>TRUNCATE TABLE support</li></ul><p>Data Types: Fabric Warehouse supports a focused set of data types:</p><ul><li>Exact numerics: bit, smallint, int, bigint, decimal, numeric</li><li>Approximate numerics: float, real</li><li>Date/time: date, time, datetime2 (precision limited to 6 digits)</li><li>Character strings: char, varchar (varchar(max) in preview with 1MB limit)</li><li>Binary strings: varbinary, uniqueidentifier (varbinary(max) in preview)</li></ul><p>Notably absent are nchar/nvarchar (Unicode), money types, text/ntext, image, and traditional SQL Server datetime/smalldatetime types.</p><p>More information about Data Types here: <a href="https://learn.microsoft.com/en-us/fabric/data-warehouse/data-types">Data Types in Fabric Data Warehouse — Microsoft Fabric | Microsoft Learn</a></p><h3>Migration Considerations</h3><p>When migrating from Teradata to Microsoft Fabric, organizations face several challenges:</p><p>SQL Dialect Translation: The QUALIFY clause must be rewritten using CTEs with ROW_NUMBER() in WHERE clauses. SET table behavior requires adding UNIQUE constraints or handling duplicates in application logic. Teradata-specific functions need mapping to T-SQL equivalents.</p><p>Primary Index Redesign: Since Fabric Warehouse doesn’t use Primary Indexes for distribution, migration teams must reconsider data modeling strategies. Former PI choices might migrate to partitioning strategies in Delta Lake or other decisions, but the direct mapping doesn’t exist.</p><p>Loading Process Modernization: BTEQ scripts and FastLoad/MultiLoad jobs must be reimplemented using COPY INTO statements, Data Pipelines, or Dataflows Gen2. This often represents a significant reengineering effort but can yield simpler, more maintainable code.</p><p>Stored Procedure Conversion: Teradata stored procedures require rewriting in T-SQL syntax. Macros have no direct equivalent and should be converted to stored procedures or parameterized views.</p><p>BTEQ Script Replacement: Organizations must replace BTEQ’s scripting capabilities with alternatives like Azure Data Factory pipelines, Fabric Data Pipelines, or orchestration tools. Export/import logic moves to COPY INTO or pipeline activities.</p><p>Data Distribution Philosophy: Teradata’s Primary Index-driven distribution requires careful design decisions that are permanent once a table is populated. The PI determines physical data placement and query performance characteristics. In contrast, Fabric Warehouse abstracts distribution away from users, storing data in Delta Lake format with automatic optimization. This simplifies design but offers less granular control over data placement.</p><p>Loading Mechanisms:Teradata offers specialized bulk loading utilities — FastLoad for empty tables, MultiLoad for populated tables with mixed operations, and TPump for near-real-time loading. These utilities bypass transient journals for speed but impose restrictions like requiring removal of secondary indexes. BTEQ handles smaller transactional loads. Fabric Warehouse uses the COPY INTO statement as its primary ingestion method. COPY INTO loads data from Azure Data Lake Storage Gen2 (ADLS Gen2) with support for Parquet and CSV formats. Best practices include 100MB-1GB file sizes, maximizing file count for parallelism, and using multiple concurrent COPY INTO statements for different tables.</p><h3>Conclusion</h3><p>Teradata SQL with BTEQ represents a mature, highly optimized platform built around parallel processing and explicit data distribution control. Its unique features — Primary Indexes, QUALIFY clauses, SET/MULTISET tables, VOLATILE tables, and BTEQ scripting — reflect decades of data warehousing innovation tailored for on-premises MPP architectures.</p><p>Microsoft Fabric Data Warehouse takes a fundamentally different approach, embracing cloud-native principles with separated compute and storage, Delta Lake format, and simplified T-SQL. It trades granular control for operational simplicity, automatic optimization, and seamless integration with Microsoft’s broader analytics ecosystem.</p><p>For organizations evaluating migration, the project demands rethinking data architecture, loading processes, and operational procedures for a cloud-native paradigm. The journey involves modernization opportunities alongside technical challenges, but the result can be a more agile, cost-effective, and AI-ready analytics platform</p><p><strong>PD: Article 1 is here :</strong><a href="https://medium.com/@millansanchez/during-the-last-few-weeks-i-have-been-working-with-a-customer-to-evaluate-their-migration-from-415beed845f4">https://medium.com/@millansanchez/during-the-last-few-weeks-i-have-been-working-with-a-customer-to-evaluate-their-migration-from-415beed845f4</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=07a35413550e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Teradata Vantage to Fabric Migration: Architectural Considerations and Phase planning]]></title>
            <link>https://millansanchez.medium.com/during-the-last-few-weeks-i-have-been-working-with-a-customer-to-evaluate-their-migration-from-415beed845f4?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/415beed845f4</guid>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Tue, 21 Oct 2025 15:04:21 GMT</pubDate>
            <atom:updated>2025-10-22T02:02:18.593Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19t0PKWkdeK-s7dEMTbo_Q.png" /></figure><h3>The Teradata Vantage to Microsoft Fabric Migration: Architectural Considerations and Phase planning</h3><p>During the last few weeks, I have been working with a customer to evaluate their migration from Teradata Vantage to Microsoft Fabric. It’s not an easy task, as we have several data assets to migrate, including tables, BTEQ files, reports, procedures, and others.</p><p>Migrating enterprise analytics workloads from established platforms like Teradata Vantage to solutions such as Microsoft Fabric Data Warehouse represents a significant transformation opportunity for organizations seeking enhanced scalability, integrated AI capabilities with the Microsoft AI Ecosystem, and cost optimization.</p><p>In this article, we will review the architectural considerations and phases involved in migrating from Teradata to Microsoft Fabric. You will discover how several Teradata features lack direct equivalents in Fabric and what alternative approaches are necessary to address these gaps as well as challenges we face during this type of migrations.</p><h3>Key Architectural Differences</h3><p>Understanding the fundamental architectural differences between Teradata and Fabric is crucial for successful migration planning. In terms of compute architecture, Teradata utilizes dedicated MPP nodes with fixed resource allocation, while Fabric employs serverless compute with dynamic scaling capabilities and separation of compute from storage. The storage model differs significantly, with Teradata using proprietary columnar storage optimized for its engine, whereas Fabric leverages OneLake with Delta-Parquet format, enabling open table formats and data sharing across workloads.</p><p>The query engines present notable contrasts. Teradata’s mature optimizer has been refined over decades for complex SQL operations, while Fabric’s SQL engine is optimized for cloud-native workloads with automatic query optimization. Regarding data models, Teradata traditionally emphasizes normalized schemas with strong support for temporal tables and complex data types, while Fabric favors denormalized, lake-centric models with native support for unstructured and semi-structured data.</p><p>Here are some important topics to consider:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0r_sYZ6VuYQ-54jQ" /></figure><p>Topics to consider for a migration from Teradata Vantage to Microsoft Fabric</p><h3>Migration Strategy Overview</h3><p>A successful migration is not a single event but a well-defined process. A phased approach is highly recommended to mitigate risk and deliver value incrementally.</p><p>The following are the high-level migration phases</p><h3>Assessment:</h3><p>A thorough inventory and analysis of the existing Teradata environment. This includes identifying all data models, schemas, tables, views, stored procedures, and ETL/ELT scripts (like BTEQ). Automated tools can be invaluable here to scan logs and identify object usage, dependencies, and redundancies.</p><p>Catalog Objects: Identify all databases, tables, views, macros, and stored procedures. Pay close attention to object sizes, usage frequency, and dependencies. Different tools can automate this discovery process.</p><p>Analyze Scripts and Logic: Analyze your codebase for Teradata-specific utilities and scripts. This includes:</p><ul><li>BTEQ (Basic Teradata Query): These scripts often contain not just SQL but also procedural logic, control flow (.IF, .GOTO), and environment commands that need to be translated.</li><li>FastLoad, MultiLoad, TPT: Identify all data loading scripts and their logic.</li><li>Stored Procedures: Analyze the complexity and business logic embedded within them.</li><li>Identify Unused Objects: Legacy systems often contain tables and queries that are no longer in use. A thorough assessment helps identify this “junk” that should not be migrated, reducing the scope and risk of the project.</li></ul><h3>Planning</h3><p>Developing a detailed migration plan. This involves prioritizing workloads for migration, designing the target architecture in Fabric, selecting the right tools for data movement and code translation, and establishing a comprehensive testing and validation strategy.</p><h3>Execution</h3><p>The technical implementation of the migration. This phase covers schema conversion, data movement, and the translation or rewriting of queries and business logic.</p><p>Once you have an inventory, you can begin mapping your Teradata schema to the Fabric Data Warehouse.</p><p>Data Type Mapping: Most Teradata data types have a direct or close equivalent in Fabric’s T-SQL, but others do not, so you need to make changes in your code.</p><p>Query and Procedure Translation: This is often the most complex part of migration.</p><ul><li>BTEQ (Basic Teradata Query) serves as Teradata’s command-line batch processing utility, enabling teams to execute SQL statements, procedural operations, and OS-level directives on Teradata infrastructure. Though it emerged before contemporary orchestration platforms and modern data integration frameworks became standard, it remains prevalent across large-scale corporate deployments because of its seamless Teradata compatibility and capacity to oversee full-cycle ETL operations from end to end.</li><li>BTEQ scripts demonstrate considerable power and adaptability. They unite conventional SQL with automation commands, flow management, markers, and conditional paths, regularly acting as custom-built control systems for ETL pipelines.</li><li>BTEQ extends far beyond simple SQL operations. They blend standard SQL with scripting directives, control flow, labels, and conditional branches, often functioning as hand-written orchestration engines for data pipelines.</li></ul><p>Challenges:</p><p>a. Procedural Control Flow: BTEQ contains decision-making, conditional paths, and repetition functions that lack equivalents in query languages like T-SQL.</p><p>b. Temporary Table Relationships: Job sequences often chain successive intermediate tables, with later steps relying on their form and contents.</p><p>c. Code Reusability: Numerous BTEQ implementations contain reusable segments or variable-driven sections — implying components appear repeatedly throughout multiple pipelines or receive variable configurations.</p><p>d. Exception Handling: BTEQ incorporates native error-handling through status markers and conditional routing. Matching approaches must be explicitly programmed into replacement systems.</p><p>e. Converting BTEQ to Spark cannot follow a one-to-one mapping. It demands comprehension. It demands critical analysis. It demands specialized knowledge:</p><ul><li>Manual Rewrite: For simple queries and procedures, a manual rewrite from Teradata SQL to T-SQL or Spark SQL is feasible. This is also an opportunity to refactor and optimize old logic.</li><li>Automation Tools: For large, complex codebases, automated translation is essential. Several third-party tools specialize in this: Next Pathway, DataFold, Bitwise, among others.</li></ul><p>f. Several Teradata features lack direct equivalents in Fabric, requiring alternative approaches. Identity columns aren’t supported in Fabric; organizations must implement identity column generation using sequences or GUIDs. Volatile tables used extensively in Teradata procedures need replacement with regular tables in Fabric.</p><p>g. SQL authentication in Teradata must transition to Microsoft Entra ID (formerly Azure AD) authentication, requiring user migration and permission mapping.</p><p>Data Migration: Multiple strategies exist for moving data from Teradata to Microsoft Fabric, each suited to different scenarios and constraints:</p><ul><li>Azure Data Factory / Fabric Data Pipelines: This is the primary method for orchestrating data movement. The Teradata connector in Data Factory supports parallel copying by partitioning the source data, which is crucial for performance. You can create pipelines that extract data from Teradata (using utilities like TPT for efficiency) and load it into OneLake. Fabric Pipelines are the next generation of ADF(Azure Data Factory), offering a more integrated and simplified SaaS experience within the Fabric workspace.</li><li>OneLake Shortcuts: For a less disruptive initial approach, you can use OneLake shortcuts. If your Teradata data is first exported to another cloud storage like ADLS Gen2 or Amazon S3, you can create a shortcut in OneLake that points directly to this data without physically moving it. This allows you to start working with the data in Fabric immediately while planning a full physical migration in a later phase.</li></ul><h3>Validation</h3><p>Rigorous testing to ensure data integrity, query accuracy, and performance parity or improvement. This involves running parallel tests on both the legacy and new systems to compare results.</p><p>Perform extensive data validation by comparing row counts, checksums, and aggregate values between Teradata and Fabric to ensure data integrity.</p><h3>Optimization</h3><p>Fine-tuning the Fabric environment for optimal performance, cost, and governance post-migration.</p><p>Regularly review the Fabric Capacity Metrics app to understand your consumption patterns. Set up alerts to be notified of high utilization, which could indicate inefficient operations or the need to scale.</p><p>Use the Fabric Monitoring Hub to identify long-running queries. Analyze query plans to find bottlenecks. Creating materialized views for common, complex queries can significantly boost performance.</p><h3>Conclusion</h3><p>The transition from Teradata to Fabric involves deciding from multiple architectural and operational options, each with its own set of considerations and solutions. While this article has highlighted some required adaptations, the work of modernizing your data environment extends far beyond these foundational topics. In future articles, we will dive deeper into the empirical findings from this real-world migration, identify common patterns that emerge across different organizational contexts, examine the challenges teams face during implementation, and provide key recommendations based on lessons learned. Stay tuned as once I begin working directly with the customer on the migration, I’ll share findings, challenges, day-to-day tasks, and conclusions from the field.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=415beed845f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IA, transformación digital y un corazón roto]]></title>
            <link>https://millansanchez.medium.com/ia-transformaci%C3%B3n-digital-y-un-coraz%C3%B3n-roto-ce2cff840f94?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce2cff840f94</guid>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Mon, 26 May 2025 21:20:12 GMT</pubDate>
            <atom:updated>2026-08-03T03:10:14.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HKMayuw0TZdEPzqvk-Zmkw.png" /><figcaption>AI, Digital Transformation and a broken heart</figcaption></figure><p>Las elegantes luces LED del techo proyectaban una luz cálida sobre los pisos de concreto pulido de la discoteca. El aire vibraba con ritmos electrónicos potentes, mezclándose con el tintineo de botellas de cerveza y el bullicio de conversaciones, la mayoría de ellas sin sentido. En la barra minimalista negra, una mujer de cabello rubio y ondas suaves se sentaba con dos amigos, con su mirada vagando distraídamente por el lugar. Había algo en su aura que atraía como un imán.</p><p>Mi amigo, con quien compartía yo la noche, la conocía. Siguiendo el protocolo tácito de la vida, procedió a inquirir sobre el estado de su existencia.</p><p>“Terminé con Roberto, hace 8 días”, comentó ella. Un par de frases adicionales se perdieron en el fuerte <em>beat</em> electrónico, imposibles de descifrar.</p><p>Segundos después, él, un alma experta en unir destinos, me presentó a ese ser con el corazón roto.</p><p>“Hola, mi nombre es Daniella”, dijo su voz, cargada con el peso de quien anhela una conexión humana.</p><p>“Hola”, respondí, notando la vulnerabilidad en su expresión. Sin pensarlo demasiado, y con una empatía espontánea, le susurré: “No pude evitar escucharte, lamento lo que siente tu corazón. Es uno de los sentimientos más desagradables que sentimos”.</p><p>Daniella soltó un suave suspiro. “Estoy aquí hoy porque necesitaba desconectarme por unas horas. Rompí con mi novio hace una semana. A veces, simplemente necesitas estar en un lugar donde nadie conoce tu historia, ¿sabes?”.</p><p>“Lo entiendo completamente. A veces necesitamos un nuevo comienzo, aunque sea solo por una noche”.</p><p>“¿A qué te dedicas? preguntó, agradecida por la distracción de sus pensamientos turbulentos.</p><p>“Trabajo con inteligencia artificial”, dije, mirando a sus ojos. “¿Y tú?”.</p><p>“Transformación digital. Ayudo a las empresas a cambiar la forma en que trabajan”. Soltó una risa amarga. “Es curioso cómo ambos lidiamos con el cambio, pero aquí estoy, luchando con mis propias transformaciones”.</p><p>Mi mente captó la poesía inherente de la situación: dos personas que dedicaban sus días a ayudar a otros a transformarse, encontrándose en medio de su propia metamorfosis personal.</p><p>“El cambio siempre es más difícil cuando es personal. ¿Qué pasó, si no te importa que pregunte?”.</p><p>Daniella giró su cuerpo, permitiendo que la luz de la discoteca iluminara su rostro. “Simplemente… nos distanciamos, supongo. O tal vez nunca estuvimos realmente juntos en primer lugar. Sigo preguntándome si vi lo que quería ver en lugar de lo que realmente estaba ahí”.</p><p>“Eso suena doloroso”, le dije suavemente.</p><p>“Estaba ahí físicamente a veces, pero nunca emocionalmente para mí”, me contestó.</p><p>Me incliné ligeramente hacia Daniella, buscando ofrecerle una voz cálida de comprensión. “En mi trabajo, hablamos de algo llamado ‘datos de entrenamiento’: básicamente, la información que usamos para enseñar a un sistema cómo comportarse. Si los datos de entrenamiento están incompletos o son defectuosos, el sistema aprende patrones incorrectos”. Hice una pausa, y continué: “Hay personas a las que no les enseñaron a amar, aman diferente, o se lo mostraron de forma incompleta. Sus emociones están en direcciones opuestas a las nuestras”.</p><p>La frase la golpeó suavemente. “Oh, definitivamente. Mi padre, por ejemplo, nunca estuvo presente para mí. Me abandonó desde que nací. Me tocó crecer con un amor incompleto”.</p><p>Su respuesta también me impactó, porque yo mismo crecí de una forma similar, con una madre que no pudo estar presente de la manera en que tanto ella como yo hubiéramos querido.</p><p>Ella dejó su copa, mirándome con una mezcla de sorpresa y reconocimiento. “Eso es… exactamente lo que pasa. Me encuentro persiguiendo hombres que hacen lo mismo: emocionalmente distantes, difíciles de alcanzar, siempre haciéndome adivinar si realmente les importo”.</p><p>“Es como si estuvieras ejecutando el mismo programa una y otra vez, esperando resultados diferentes”, le dije gentilmente.</p><p>Los hombros de Daniella se hundieron mientras la verdad se asentaba sobre ella. “Entonces, ¿cómo me… reprogramo, cómo me transformo?”.</p><p>“Bueno, en tecnología, cuando queremos cambiar cómo funciona algo, no solo arreglamos los problemas superficiales. Tenemos que ir al código fuente — a las bases fundamentales que hacen que todo funcione”.</p><p>“¿Cómo se vería eso para mí?”, preguntó, inclinándose hacia adelante con curiosidad genuina.</p><p>“Tal vez hay que reconocer que el amor no tiene que ser trabajo duro. Que alguien que realmente se preocupa por ti no te hará preguntarte dónde te encuentras”.</p><p>Daniella frunció el ceño, con un toque de confusión nublando sus rasgos. “¿Pero cómo sé la diferencia? Quiero decir, pensé que mi ex se preocupaba por mí. Dijo que lo hacía”.</p><p>“El amor verdadero debería ser como un correo electrónico. Simple, fácil, que logra su objetivo con solo una dirección electrónica. Pero en cambio, hoy tenemos WhatsApp, Telegram, filtros y redes sociales. Eso no nos permite conectarnos realmente porque queremos saber a qué horas se conectó, si miró o no miró el mensaje, si nos dejó en visto, si está en otro lugar al que nos dijo. No nos mostramos como realmente somos. A veces, la mejor transformación digital y personal no es poner más. Es quitar y poner menos, volver a lo simple”.</p><p>Daniella soltó un largo suspiro, sus hombros cayendo en derrota. “Sí. A veces sobre-trabajamos las cosas y nosotros mismos las complicamos. Buscamos soluciones complejas a problemas simples. Y también soluciones que tienen errores de diseño”.</p><p>“Sí, como un software confiable versus un software revolucionario, pero con errores”, le dije. “El software revolucionario podría ser emocionante porque es nuevo y novedoso, pero ¿construirías tu vida sobre eso?”.</p><p>Daniella negó con la cabeza lentamente. “No, querría algo en lo que pueda confiar. Algo que funcione bien y que crezca con mis necesidades. Quiero un software estable”.</p><p>“Pero aquí está la cosa sobre las relaciones”, dije con una voz más animada, “no son realmente software. Son más complejas. Una buena relación debería ser tanto estable como sorprendente, confiable y emocionante”.</p><p>“¿Cómo es eso posible?”, preguntó, reemplazando la tristeza en su voz por una suave brisa de emoción.</p><p>“Bueno, piénsalo de esta manera: la base debería ser sólida, pero aún hay espacio para crecimiento y descubrimiento. No deberías tener que preguntarte si tu pareja te ama. Debes sentirlo, y saber que, aunque estable, aún puedes descubrir cosas nuevas el uno del otro y crecer en base a las necesidades de ambos”.</p><p>Por primera vez esa noche, Daniella se rió genuinamente. “Cuando lo pones de esa manera, suena tan obvio”.</p><p>“¿Sabes qué es interesante sobre la inteligencia artificial?”, le dije, haciendo girar su copa pensativamente. “Aprende de patrones, pero también puede desaprenderlos cuando obtiene mejor información. Los humanos también pueden hacer eso, pero toma más esfuerzo porque tenemos emociones adjuntas “.</p><p>“¿Entonces estás diciendo que puedo aprender nuevos patrones para el amor?”, preguntó Daniella, con una esperanza filtrándose en su voz.</p><p>“Creo que sí. Pero primero podrías necesitar entender cómo se ve realmente el amor saludable. ¿Qué sería eso para ti?”.</p><p>Daniella miró fijamente su copa de vino, su ceño fruncido en concentración. “No lo sé. Honestamente no sé cómo se vería eso. ¿Alguien que… aparece? ¿Alguien que no me hace adivinar cómo se siente?”.</p><p>“Eso es un comienzo”, le dije. “¿Qué más?”.</p><p>“Alguien que no desaparece cuando las cosas se ponen difíciles. Alguien que no me hace sentir como si yo fuera muy poco”, dijo, su voz volviéndose más fuerte con cada palabra.</p><p>“Esos suenan como requisitos básicos, no expectativas irreales”, le hice saber.</p><p>La expresión de Daniella se volvió problemática otra vez. “¿Pero qué pasa si no sé cómo estar en ese tipo de relación? ¿Qué pasa si la saboteo porque se siente muy diferente?”.</p><p>Me recosté en una de las modernas sillas, haciendo girar un whisky pensativamente mientras los cubos de hielo chocaban contra el vaso de cristal. “En mi campo, cuando implementamos nuevos sistemas, no esperamos que funcionen perfectamente de inmediato. Siempre hay un período de aprendizaje, algunos ajustes necesarios. La clave es tener paciencia con el proceso y tener un constante proceso de retroalimentación”.</p><p>“¿Y estar dispuesta a probar algo diferente?”, preguntó.</p><p>“Exactamente. Y tal vez comenzar siendo paciente y amable contigo misma. No puedes amar a alguien más de manera saludable si nunca lo viste en tus padres o tu familia, o con tus amigos. Necesitas un buen punto de referencia”.</p><p>“Los corazones tardan más en actualizarse que las mentes”, le dije suavemente. “Pero pueden actualizarse. Solo toma nuevas experiencias y nueva evidencia que contradiga la programación antigua y te ayude a reprogramarte. La transformación digital se construye en base a la retroalimentación y medición permanente”.</p><p>“¿Y cómo se vería la versión actualizada?”, preguntó, inclinándose hacia adelante con interés.</p><p>“Alguien que sabe que merece amor y que no duela. Alguien que puede detectar la diferencia entre emoción y ansiedad en una relación. Alguien que no se conforma con ‘tal vez’ cuando podría tener un siempre ‘sí’. Cuando no eres una opción sino siempre la prioridad”.</p><p>Un miedo parpadeó en su rostro. “¿Qué pasa si nadie así me ama?”.</p><p>“¿Qué pasa si sí lo hacen, pero has estado muy ocupada persiguiendo personas que no? A veces tenemos que dejar de correr hacia la cosa incorrecta. Quedarnos quietos para notar la cosa correcta que está esperando que la veamos”.</p><p>“¿Y realmente crees que eso es posible? ¿Que puedo aprender a amar diferente?”.</p><p>“Creo que ya estás empezando”, respondí. “El hecho de que estés cuestionando tus patrones, que quieras algo mejor, ese es el primer paso. Eres única y ningún clon digital puede replicar lo que tienes para ofrecer a alguien que esté listo para recibirlo”.</p><p>Daniella me miró con curiosidad. “¿Como sabré que hay personas listas para recibir mi amor?”.</p><p>“Llegarán a ti personas que aparecen consistentemente. Tal vez incluso conversaciones con extraños en una discoteca, que te recuerden que vales la pena, que eres maravillosa y mereces que te amen de forma bonita. Que inspires en otros el amor alcanzable, el duradero. Uno que te lleva a ser una mejor versión de ti misma”.</p><p>Una sonrisa genuina se extendió por su rostro. “¿Es eso lo que estás haciendo?”.</p><p>Le sonreí de vuelta, mirando nuevamente sus ojos que ahora tenían un nuevo brillo. “ Muchos anhelarán verte en tu nueva versión, aunque sea en una versión de prueba por 14 días sin compromiso porque les gustó el landing page. Sin embargo, el amor real no surge de una landing page. Alguién tendrá la fortuna de tener una licencia perpetua de ti en el tiempo porque se enamorará del producto final y de sus futuras versiones. Y ese amor será real. <strong>El amor real es saber que ya lo tienes”.</strong></p><p>Daniella se rió, el sonido brillante de la discoteca pareció silenciarse por un segundo. “Esta noche me ayudaste a entender que no estoy rota, solo estoy lista para una actualización. ¿Y quién sabe? Tal vez la versión 3.7 de mí será la que encuentre el amor. Que es mío y me lo gané. Que es suyo y se lo ganó. Que ambos ganamos. <strong>Que será tan real para los dos, que sabremos que ya es nuestro</strong>.”</p><p>Levanté mi vaso en un brindis y le dije: “Creo que lo será. <strong>Y va a ser maravilloso</strong>”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce2cff840f94" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microsoft Fabric’s Oct-Nov 2023 released features]]></title>
            <link>https://millansanchez.medium.com/microsoft-fabrics-oct-nov-2023-released-features-0767c7521a05?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/0767c7521a05</guid>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Thu, 09 Nov 2023 23:41:17 GMT</pubDate>
            <atom:updated>2023-11-09T23:41:17.343Z</atom:updated>
            <content:encoded><![CDATA[<p>Microsoft Fabric is nearing its General Availability, and the Microsoft engineering team is working non-stop to add new features. Here is a summary of the main updates and enhancements made to Fabric between October 2023 and November 2023 so far. These new features are designed to improve the platform’s performance, security, and user-friendliness:</p><p>1. <strong>Column-Level &amp; Row-Level Security for Fabric Warehouse &amp; SQL Endpoint</strong>: Microsoft announced the public preview of Column-Level and Row-Level Security in Fabric Warehouse &amp; SQL Endpoint. Column-Level Security allows restricting column access within the Warehouse, thereby enhancing data security by enforcing access restrictions system-wide. Row-Level Security, on the other hand, controls access to rows in a table based on user accounts, with rules enforced database-wide to minimize security vulnerabilities. These features simplify the design and coding of security and are managed within the Warehouse or SQL Endpoint table, enforcing rules whenever data is accessed.</p><p>2. <strong>Microsoft OneLake Shortcut Support to Power Platform and Dynamics 365:</strong> Microsoft OneLake now supports creating shortcuts to Dynamics 365 and Power Platform data within Dataverse, allowing data analysis with Microsoft Fabric without needing to export data or build ETL pipelines. OneLake acts as a single unified data lake, and with this update, it virtualizes data through shortcuts that appear as if they were physically present in OneLake. This integrates the data with the lakehouse, formatted using the Delta/Parquet standard, streamlining data analysis with Spark, SQL, and Power BI reports without data transformation.</p><p>3. <strong>Data Activator Public Preview:</strong> Data Activator is now available in public preview for all Microsoft Fabric users. This feature enables automatic alerts and actions from Fabric data, eliminating the need for manual monitoring of operational dashboards. Data Activator allows users to create alerts directly from Power BI reports and is designed to be user-friendly, requiring no technical knowledge. It can trigger alerts and actions such as sending emails, Microsoft Teams messages, and trigger Power Automate flows for various conditions set by the user, such as monitoring overdue invoices or checking shipment statuses.</p><p>4. <strong>Fabric Roadmap:</strong> Responding to feedback for more transparency, Microsoft has announced the Fabric Roadmap, which will be regularly published to provide insights into upcoming features and their expected release dates. This roadmap will eventually be merged with the Power BI roadmap from the Power Platform release plans, offering a unified view of upcoming developments for Microsoft Fabric.</p><p>5. <strong>Semantic Link for BI and Data Science:</strong> The Public Preview of semantic link has been introduced, connecting Power BI datasets with Synapse Data Science within Microsoft Fabric. This enables access to Power BI data for notebooks and Python, supporting pandas and Spark APIs, and allowing the computation of Power BI measures, reading tables, and executing DAX queries. Semantic link facilitates data augmentation, validation, exploration, and an extendable set of semantic functions for a comprehensive data science environment.</p><p>6. <strong>Semantic Kernel in Microsoft Fabric:</strong> Semantic Kernel is introduced as part of Microsoft Fabric’s Lakehouse, offering an LLM-based application framework. It enhances the connection between Prompt and Code, supporting multiple programming languages and enabling seamless Copilot applications. The framework allows for managing both structured and unstructured data, utilizing traditional techniques like Python’s libraries and T-SQL for data extraction and analysis. It also provides the ability to chat with enterprise data, creating interactive data management and analysis scenarios.</p><p>7. <strong>Enhanced UX on Event Processor and Expanded Custom App Connections:</strong> Microsoft Fabric’s Eventstream has undergone updates aimed at improving efficiency and user experience. The Event Processor now features a no-code editor that simplifies the design of event processing logic, allowing for real-time processing with powerful temporal functions. The recent UX improvements facilitate adding or deleting operations with intuitive interactions. Additionally, the Custom App connection strings have been expanded to support protocols such as Kafka and AMQP, enabling seamless connection to Eventstream for applications and Kafka clients. This update emphasizes flexibility and ease of use for real-time event data ingestion and processing within Fabric.</p><p>8. <strong>Fabric Warehouse publishing full DML to Delta Lake Logs:</strong> The Data Warehouse in Microsoft Fabric now automatically publishes all Data Manipulation Language (DML) actions (Inserts, Updates, Deletes) for each table to their Delta Lake Log in OneLake. This is part of a broader vision to eliminate data silos and facilitate data sharing across teams and services without the need to duplicate data. By leveraging Delta Lake as a common format, data remains secure, easier to manage, consistent across reports, and readily shareable. These enhancements ensure that changes made in Data Warehouse tables are reflected in the Delta Lake Log within a minute, simplifying data integration for various uses, including Machine Learning and AI model training.</p><p>9. <strong>Data Warehouse SKU Guardrails for Burstable Capacity:</strong> Microsoft Fabric has introduced SKU guardrails for burstable capacity in its Data Warehouse and SQL Endpoint. Burstable capacity allows workloads to temporarily use additional resources to enhance performance, particularly useful for uneven workload patterns. The feature implements ‘smoothing’ to average out over-usage over time, thus optimizing capacity management. The new SKU guardrails ensure that this bursting ability remains within reasonable limits, preventing overconsumption of resources. With guardrails, overconsumption can be managed more effectively, maintaining performance without the risk of rapid resource depletion. This feature is automatically active for new tables created within Data Warehouses on Microsoft Fabric.</p><p>10. <strong>Time-Travelling through data:</strong> Microsoft Fabric has introduced a feature allowing users to clone data warehouse tables, capturing a snapshot of data as it existed at a specific moment in time(without data duplication), up to a seven-day retention period. This enables historical trend analysis, aids in compliance requirements, and assists in root cause analysis during data discrepancies. The cloning process is user-friendly, allowing multiple tables to be cloned at once through an intuitive interface or T-SQL, streamlining reporting, analytics, development, and testing processes.</p><p>11. <strong>Fabric Warehouse user experience updates:</strong> Fabric Warehouse rolled out several user experience enhancements designed to increase productivity and streamline user interaction. These include a new UI for cloning tables, saving queries as views or tables via the Visual query editor, an endorsement feature to highlight quality data assets, enhancements to the sample loading experience for better performance, and enabling viewers/shared recipients to save queries.</p><p>12. <strong>Automatic Data Compaction for Fabric Warehouse:</strong> Automatic data compaction has been announced for Fabric Warehouse, aiming to automate maintenance tasks to improve performance and reduce costs. The feature rewrites multiple smaller Parquet files into fewer larger ones, enhancing read performance and removing deleted rows from files. This compaction occurs as system tasks generated by the engine, working behind the scenes without additional effort from the user.</p><p>13. <strong>Create multiple shortcuts faster and easier:</strong> Microsoft OneLake, as a unified storage location, now allows users to create multiple shortcuts in a more straightforward manner. Users can select multiple targets at once and create new shortcuts in a single operation, reviewing and editing selections before creation, which significantly improves the efficiency of data management and accessibility.</p><p>14. <strong>Delta-RS introduces native support for OneLake:</strong> Delta-RS, a native Rust library for Delta Lake, has extended support to OneLake. This improvement enables direct management of lakehouses in OneLake using Delta-RS, facilitating integration with ADLS Gen2 compatible tools and libraries. The OneLake team has collaborated with the Delta-RS community to ensure seamless functionality and compatibility, promoting the integration of a broader range of data management tools.</p><p>15. <strong>Improvements to CSV data ingestion in Fabric Warehouse:</strong> The introduction of CSV file parser version 2.0 for COPY INTO commands in the Data Warehouse has been announced. Based on Microsoft Research’s innovation, the new parser significantly improves CSV file ingestion performance, with tests revealing a 38% improvement in ingestion times on average, and in some cases, speeds up to four times faster than the legacy parser. The new parser has been made the default option, simplifying the process for users and providing immediate performance benefits.</p><p>16. <strong>A New and Improved Experience for real time analytics data ingestion</strong>: Microsoft Fabric has rolled out an update that simplifies the data ingestion process for KQL databases. This is achieved through a new user interface called the Get Data experience, which reduces the number of steps and clicks needed to import data. It includes a Guided Wizard that assists users step-by-step, ensuring an intuitive process whether users are working with local files, Azure storage, Amazon S3, Event Hubs, Eventstream, or OneLake. Real-Time Analytics now supports these six different data sources, allowing for efficient data import into KQL databases with minimal effort.</p><p>17. <strong>Fabric Warehouse Support for sp_rename:</strong> Microsoft Fabric’s Data Warehouse now includes support for sp_rename, a feature that allows users to rename user objects like Tables, Stored Procedures, Functions, etc. The update ensures that when objects are renamed, their new names are reflected in OneLake within one minute, which helps maintain consistency and streamlines the data management process.</p><p>18. <strong>Dynamic Data Masking for Fabric Warehouse and Lakehouse SQL Endpoint:</strong> The introduction of Dynamic Data Masking (DDM) for Fabric Warehouse and SQL Endpoint in Lakehouse is a pivotal update for data security. DDM enables the protection of sensitive data by masking it according to user-defined rules without changing the core data structure or application logic. It provides enhanced data security by concealing sensitive information from unauthorized access, seamlessly integrates with existing applications without requiring changes, and offers fine-grained control over the masking rules.</p><p>19. <strong>Using Your Own Library with Microsoft Fabric:</strong> Microsoft Fabric has provided a comprehensive guide for creating and using personal libraries within the Lakehouse environment. This feature allows for the reuse of pre-compiled Python code, such as business logic functions, across the organization.</p><p>References:</p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-column-level-row-level-security-for-fabric-warehouse-sql-endpoint?ft=All">https://blog.fabric.microsoft.com/en-us/blog/announcing-column-level-row-level-security-for-fabric-warehouse-sql-endpoint?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/microsoft-onelake-adds-shortcut-support-to-power-platform-and-dynamics-365?ft=All">https://blog.fabric.microsoft.com/en-us/blog/microsoft-onelake-adds-shortcut-support-to-power-platform-and-dynamics-365?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-the-data-activator-public-preview?ft=All">https://blog.fabric.microsoft.com/en-us/blog/announcing-the-data-activator-public-preview?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-the-fabric-roadmap?ft=All">https://blog.fabric.microsoft.com/en-us/blog/announcing-the-fabric-roadmap?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/semantic-link-use-fabric-notebooks-and-power-bi-datasets-for-machine-learning-data-validation-and-more?ft=All">https://blog.fabric.microsoft.com/en-us/blog/semantic-link-use-fabric-notebooks-and-power-bi-datasets-for-machine-learning-data-validation-and-more?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/chat-your-data-in-microsoft-fabric-with-semantic-kernel?ft=All">https://blog.fabric.microsoft.com/en-us/blog/chat-your-data-in-microsoft-fabric-with-semantic-kernel?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/whats-new-in-eventstream-enhanced-ux-on-event-processor-and-expanded-custom-app-connections?ft=All">https://blog.fabric.microsoft.com/en-us/blog/whats-new-in-eventstream-enhanced-ux-on-event-processor-and-expanded-custom-app-connections?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-fabric-warehouse-publishing-full-dml-to-delta-lake-logs?ft=All">https://blog.fabric.microsoft.com/en-us/blog/announcing-fabric-warehouse-publishing-full-dml-to-delta-lake-logs?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/data-warehouse-sku-guardrails-for-burstable-capacity?ft=All">https://blog.fabric.microsoft.com/en-us/blog/data-warehouse-sku-guardrails-for-burstable-capacity?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/time-travelling-through-data-the-magic-of-table-clones?ft=All">https://blog.fabric.microsoft.com/en-us/blog/time-travelling-through-data-the-magic-of-table-clones?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/fabric-warehouse-user-experience-updates?ft=All">https://blog.fabric.microsoft.com/en-us/blog/fabric-warehouse-user-experience-updates?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-automatic-data-compaction-for-fabric-warehouse?ft=All">https://blog.fabric.microsoft.com/en-us/blog/announcing-automatic-data-compaction-for-fabric-warehouse?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/create-multiple-shortcuts-faster-and-easier?ft=All">https://blog.fabric.microsoft.com/en-us/blog/create-multiple-shortcuts-faster-and-easier?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/delta-rs-introduces-native-support-for-onelake?ft=All">https://blog.fabric.microsoft.com/en-us/blog/delta-rs-introduces-native-support-for-onelake?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-improvements-to-csv-data-ingestion-in-fabric-warehouse?ft=All">https://blog.fabric.microsoft.com/en-us/blog/announcing-improvements-to-csv-data-ingestion-in-fabric-warehouse?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/get-data-in-real-time-analytics-a-new-and-improved-experience?ft=All">https://blog.fabric.microsoft.com/en-us/blog/get-data-in-real-time-analytics-a-new-and-improved-experience?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-fabric-warehouse-support-for-sp_rename?ft=All">https://blog.fabric.microsoft.com/en-us/blog/announcing-fabric-warehouse-support-for-sp_rename?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-dynamic-data-masking-for-fabric-warehouse-and-lakehouse-sql-endpoint?ft=All">https://blog.fabric.microsoft.com/en-us/blog/announcing-dynamic-data-masking-for-fabric-warehouse-and-lakehouse-sql-endpoint?ft=All</a></p><p>· <a href="https://blog.fabric.microsoft.com/en-us/blog/fabric-changing-the-game-using-your-own-library-with-microsoft-fabric?ft=All">https://blog.fabric.microsoft.com/en-us/blog/fabric-changing-the-game-using-your-own-library-with-microsoft-fabric?ft=All</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0767c7521a05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling Microsoft Fabric: A Comparative Analysis Against Competing Platforms]]></title>
            <link>https://medium.com/natural-stupidity/analyzing-microsoft-fabric-vs-other-platforms-60f737b167ab?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/60f737b167ab</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Fri, 29 Sep 2023 01:42:56 GMT</pubDate>
            <atom:updated>2023-09-29T14:21:50.594Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>(Part 1: Amazon)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*DSdHz54WSsaKMwnWcI8DCQ.jpeg" /></figure><p>In the ever-evolving landscape of data analytics, Microsoft has introduced a game-changing solution — Microsoft Fabric. This innovative platform redefines the way organizations collect, store, process, and analyze data. Microsoft Fabric is designed to empower enterprises by providing a unified, integrated, and user-friendly environment for data analytics and AI-driven insights. In this series of articles, we will deep dive into Microsoft Fabric and compare it to other popular platforms from Amazon, Google, Snowflake and others. We will explore its core components, unique features, and the impact it has on the world of data engineering and analytics.</p><p><strong>Understanding the Essence of Microsoft Fabric</strong></p><p>At its core, Microsoft Fabric is a comprehensive suite of tools and services that caters to a wide range of data personas and analytical needs within an organization. Its mission is to simplify the complex world of data analytics by providing a single source of truth and a unified platform that covers the entire data lifecycle, from data ingestion to visualization and AI-based analytics.</p><p><strong>Components of Microsoft Fabric platform:</strong></p><p>Let’s delve deeper into the key components that make up the Microsoft Fabric ecosystem:</p><p>1. Integration Tools: Microsoft Fabric seamlessly brings together data from diverse sources. Whether your data lives on-premises or in the cloud, Fabric’s integration tools ensure that you can access and utilize it efficiently.</p><p>2. Spark-based Analytics Platform: Apache Spark is at the heart of Microsoft Fabric’s data processing capabilities. This powerful framework enables data engineers to perform large-scale data transformations, process data in real time and in batch, create AI and ML Models, and democratize data through a concept known as the “lakehouse.”</p><p>3. Real-time Analytics: Organizations increasingly rely on real-time data insights. With Azure Data Explorer, Microsoft Fabric offers efficient analytics for semi-structured data with high volume and shifting schemas. This real-time capability sets Fabric apart in the data analytics arena.</p><p>4. Data Warehouse: Provides industry-leading SQL performance and scale with separate compute and storage components. Data is stored in the open Delta Lake format.</p><p>5. Visualization: Microsoft’s Power BI is renowned for its business intelligence capabilities. When integrated into Fabric, it becomes an even more potent tool for data visualization and AI-driven analytics. Users can gain insights quickly and make data-driven decisions with ease.</p><p>6. Data Observability: With Data Activator, drive actions automatically from your data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*q5j-U0IYYjnct3xneAdZjw.png" /></figure><p>Microsoft Fabric unifies components from Power BI, Azure Synapse, and Azure Data Factory, offering tailored user experiences. It integrates Data Engineering, Data Factory, Data Science, Data Warehouse, Real-Time Analytics, and Power BI on a shared SaaS foundation, delivering:</p><p>1. Comprehensive, tightly integrated analytics.</p><p>2. Consistent, user-friendly experiences.</p><p>3. Easy asset access and reuse for developers.</p><p>4. Unified data lake for data preservation and analytics flexibility.</p><p>5. Centralized administration and governance.</p><p><strong>The Lakehouse Concept</strong></p><p>One of the main aspects of Microsoft Fabric is its embrace of the “lakehouse” architecture. A lakehouse combines the best of data lakes and data warehouses, allowing organizations to store and analyze both structured and unstructured data in a unified storage system. This architecture simplifies data access by automatically generating a read-only SQL endpoint and a default dataset upon creation.</p><p>The advantages of the lakehouse concept within Microsoft Fabric include:</p><p>• Single Source of Truth: With data stored in a standardized format (Delta Parquet), Fabric eliminates data duplication within the ecosystem. This ensures that all tools within the Microsoft ecosystem can access and utilize the data in a consistent manner.</p><p>• Streamlined Data Engineering: Organizations no longer need to replicate data solely to work with different technologies. This eliminates redundancies, simplifies data management, and promotes a true single source of data for analysis and decision-making.</p><p>• Compatibility Across Microsoft Tools: Microsoft Fabric’s lakehouse approach enables seamless compatibility with various Microsoft tools, including data pipelines, data flows, notebooks, SQL engines, Kusto engines, and Power BI. This consistency enhances ease of use across the entire Microsoft ecosystem.</p><p><strong>Simplified User Experience</strong></p><p>Microsoft Fabric places a strong emphasis on simplifying the user experience. The platform’s SaaS-based model integrates data and services seamlessly, allowing IT teams to configure core enterprise capabilities centrally. Permissions and data sensitivity labels are automatically applied across all services, reducing the complexity of infrastructure management.</p><p><strong>Targeting a Diverse User Base</strong></p><p>Microsoft Fabric caters to a diverse set of users, including C-level executives, department heads, data scientists, business analysts, and data analysts. By providing a single source of truth, democratized access to data, reducing duplication, removing complexity, enhancing visibility, and fostering an integrated environment, Fabric empowers users to work collaboratively and efficiently.</p><p><strong>The Impact of Microsoft Fabric</strong></p><p>Microsoft Fabric is making waves in the data analytics landscape for several reasons:</p><p>1. Rising Demand for Data Analytics: As organizations increasingly recognize the value of data-driven decision-making, the demand for robust data analytics solutions continues to grow. Microsoft Fabric’s all-encompassing platform positions it as a versatile tool for a wide range of data analytics tasks.</p><p>2. Unified Data Platform: Organizations have long sought a “Single Pane of Glass” solution for their entire data landscape. Microsoft Fabric emerges as the unified platform capable of managing data throughout its lifecycle, from collection to analysis. This unification simplifies data management, reduces costs, and enhances efficiency.</p><p>3. Integration with Microsoft Ecosystem: One of Microsoft Fabric’s compelling advantages is its tight integration with other Microsoft products and the Azure Cloud. For businesses already using Azure Cloud Services, Fabric adoption is a natural progression. This integration streamlines workflows and saves both time and money.</p><p>4. Eliminating Data Replication: Microsoft Fabric takes a bold step by addressing the issue of unnecessary data replication. By advocating for a single copy of data stored in a central lake (OneLake) using the Delta Parquet format, Microsoft aims to standardize data storage and access. This approach ensures that data is accessible to all Microsoft tools in a consistent format, promoting efficiency and reducing redundancy.</p><p>5. A Unified Data Engineering Process: With Microsoft Fabric, organizations can create a more efficient and streamlined data engineering process. The need for data replication for different technologies is eliminated, simplifying data management and fostering a true single source of data for analysis and decision-making.</p><h3><strong>Microsoft Fabric vs. Amazon: A Comparative Analysis of Data Analytics Platforms</strong></h3><p>In the rapidly evolving world of data analytics, organizations are continually searching for robust solutions to harness the power of their data. Two key players in this space, Microsoft and Amazon, offer powerful platforms tailored to meet the growing demands of data-driven enterprises. In this comparative analysis, we will explore the strengths, capabilities, and differentiating features of Microsoft Fabric and Amazon’s suite of data analytics products.</p><p><strong>Microsoft Fabric: Simplifying Data Analytics</strong></p><p>Microsoft Fabric is a comprehensive data analytics and AI platform designed to streamline the data journey for enterprises. It focuses on providing a unified and integrated environment that covers all aspects of data analytics, from data ingestion to real-time analysis and visualization. Let’s delve into the key features and strengths of Microsoft Fabric:</p><p><strong>Key Features of Microsoft Fabric</strong></p><p>1. Lakehouse Architecture: Microsoft Fabric embraces the lakehouse architecture, allowing organizations to store and analyze both structured and unstructured data in a unified storage system. This architecture simplifies data access and management by providing a read-only SQL endpoint and a default dataset.</p><p>2. Unified Data Platform: Fabric offers a centralized solution for data analytics, making it convenient for organizations to manage their data lifecycle from collection to analysis within a single platform.</p><p>3. Tight Integration: Microsoft Fabric is tightly integrated with other Microsoft products and the Azure Cloud, simplifying workflows for organizations already utilizing Azure services. This integration ensures a seamless experience and cost savings.</p><p>4. Elimination of Data Replication: Fabric’s approach to standardized data storage in the Delta Parquet format eliminates the need for data replication within the Microsoft ecosystem. This promotes efficiency and reduces redundancy.</p><p>5. Versatile User Base: Fabric caters to a diverse set of users, including C-level executives, data scientists, business analysts, and data engineers, by offering a single source of truth, enhanced collaboration, and an integrated environment.</p><h3><strong>Amazon’s Data Analytics Suite: A Multifaceted Approach</strong></h3><p>Amazon offers a suite of data analytics products and services that cater to a wide range of data needs. Let’s explore some of the key components of Amazon’s data analytics offerings:</p><p><strong>Amazon S3: Versatile Object Storage</strong></p><p>Amazon S3 is Amazon’s object storage service that serves as a foundation for many data analytics solutions. It is highly durable, available, and can store any kind of data, whether structured or unstructured.</p><p>Key Features:</p><p>• Virtually unlimited storage capacity.</p><p>• Integration with various data sources.</p><p>• Secure access control and data encryption.</p><p><strong>Amazon EMR: Managed Big Data Processing</strong></p><p>Amazon EMR (Elastic MapReduce) is a managed cluster platform designed for big data processing. It enables the execution of big data frameworks like Apache Spark and Apache Hadoop on AWS infrastructure.</p><p>Key Features:</p><p>• Scalable processing for large datasets.</p><p>• Integration with Amazon S3 for data storage.</p><p>• Support for multiple data processing frameworks.</p><p><strong>Amazon Redshift: Data Warehousing</strong></p><p>Amazon Redshift is a fully managed data warehouse solution that provides high-performance querying capabilities. It is optimized for analytical workloads and is designed to store and retrieve large amounts of data.</p><p>Key Features:</p><p>• Columnar storage for high performance.</p><p>• Integration with data sources and data lakes.</p><p>• Support for real-time data streaming and ingestion.</p><p><strong>Amazon Athena: Serverless Query Service</strong></p><p>Amazon Athena is an interactive query service that allows users to analyze data in Amazon S3 using standard SQL. It is serverless, meaning users pay only for the queries they run.</p><p>Key Features:</p><p>• Ad-hoc querying of data in Amazon S3.</p><p>• Integration with various data formats.</p><p>• Scalability and cost-effectiveness.</p><h3><strong>A Comparative Overview</strong></h3><p><strong>Integration and Ecosystem:</strong></p><p><strong>Microsoft Fabric:</strong> Fabric boasts seamless integration with the Microsoft ecosystem, making it a natural choice for organizations already using Azure services. The lakehouse architecture and Delta Parquet format enhance compatibility within the Microsoft environment and reduces vendor lock-in as technology is open source and other providers such as Databricks fully support it.</p><p><strong>Amazon:</strong> Amazon’s suite of data analytics products offers a wide range of integration options. While it may require more effort to create a unified environment, AWS services can integrate well with other AWS offerings and third-party tools.</p><p><strong>Unified Platform:</strong></p><p><strong>Microsoft Fabric:</strong> Fabric provides a unified platform that covers the entire data lifecycle(Storage, ingestion, processing, AI/ML, visualization, governance). It simplifies data management by eliminating data replication and providing a single source of truth.</p><p><strong>Amazon:</strong> Amazon’s suite comprises various services for specific data needs. While versatile, organizations may need to piece together multiple services to create a unified platform.</p><p><strong>Data Lakehouse:</strong></p><p><strong>Microsoft Fabric:</strong> Fabric’s lakehouse architecture and OneLake Foundation offer a powerful solution for storing and managing structured and unstructured data efficiently. All in a SaaS solution, simplifying the platform administration. OneLake fully embraces an open approach, harnessing the capabilities of Azure Data Lake Storage Gen2. It has the flexibility to accommodate any type of file, whether it’s structured or unstructured. In the Fabric ecosystem, all data entities, including data warehouses and lakehouses, automatically store their data in OneLake using the delta parquet format. This ensures that whether a data engineer leverages Spark to load data into a lakehouse or a SQL developer opts for a fully transactional data warehouse with T-SQL, all contributions converge to construct a unified data lake.</p><p><strong>Amazon:</strong> Amazon’s data analytics suite relies on Amazon S3 as the foundation for data storage, but it may require additional configuration to achieve the lakehouse architecture offered by Fabric.</p><p><strong>Data Warehouse:</strong></p><p><strong>Microsoft Fabric:</strong> data stored in Delta Parquet format in OneLake doesn’t need to be copied again into the Data warehouse for processing.</p><p>1. Scalability: Seamlessly adapts to changing data demands with elastic resources.</p><p>2. Performance: Lightning-fast query capabilities for extensive data processing.</p><p>3. Data Versatility: Accommodates both structured and semi-structured data.</p><p>4. Microsoft Integration: Easily integrates with Power BI and other Microsoft tools.</p><p><strong>Amazon:</strong> Amazon Redshift is a fully managed data warehouse solution that stores and queries large volumes of analytical data. It leverages machine learning and parallel processing for great performance.</p><p>Key features of Amazon Redshift include:</p><p>1. Quick Provisioning: Redshift automates administrative tasks and resource provisioning, ensuring rapid deployment.</p><p>2. Concurrency Scaling: Redshift can handle high concurrency with automatically scaling clusters.</p><p>3. Spectrum Integration: Queries on data stored in Amazon S3 can be executed without prior loading into the Redshift data warehouse.</p><p>Data usually needs to be copied into Redshift to be processed and joined.</p><p><strong>User Base and Persona Coverage:</strong></p><p><strong>Microsoft Fabric:</strong> Fabric caters to a diverse set of users, including business leaders, data scientists, and analysts. It promotes collaboration and offers an integrated environment.</p><p><strong>Amazon:</strong> Amazon’s offerings cover a wide range of data personas, from data engineers using EMR to data analysts querying data with Athena. The breadth of services accommodates different user needs. User Experience and Interface is not unified. Every role has different tools to be used.</p><p><strong>Conclusion</strong></p><p>In the realm of data analytics, both Microsoft Fabric and Amazon offer powerful solutions, each with its own strengths and capabilities. Microsoft Fabric excels in providing a unified platform with its lakehouse architecture, tight integration with the Microsoft ecosystem, and a simplified user experience. On the other hand, Amazon’s suite of data analytics products offers versatility and scalability, making it a solid choice for organizations looking to piece together tailored solutions.</p><p>Ultimately, the choice between Microsoft Fabric and Amazon’s offerings depends on an organization’s existing infrastructure, specific data analytics needs, and the level of integration and unification required. Both platforms have the potential to drive data-driven decision-making and unlock valuable insights for enterprises.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60f737b167ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/natural-stupidity/analyzing-microsoft-fabric-vs-other-platforms-60f737b167ab">Unveiling Microsoft Fabric: A Comparative Analysis Against Competing Platforms</a> was originally published in <a href="https://medium.com/natural-stupidity">Natural Stupidity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Firefox browser with fonts too small in Linux]]></title>
            <link>https://millansanchez.medium.com/firefox-browser-with-fonts-too-small-in-linux-797a4ac06602?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/797a4ac06602</guid>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[firefox]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Sun, 10 Jan 2021 18:13:42 GMT</pubDate>
            <atom:updated>2021-01-10T18:13:42.656Z</atom:updated>
            <content:encoded><![CDATA[<p>I just finished installing Mint Linux in my machine (Cinnamon Version 64 bits). I opened Firefox browser and the fonts were so small to read a Web Site.</p><p>I started my research about how to fix this, as i don&#39;t want to change the Zoom in/out configuration in Firefox continuously.</p><p>After a few minutes i was able to fix it by doing this:</p><p>Firefox has an internal setting that governs both the “chrome” area (toolbars, menus, dialog) and the content area (pages). If you want to change this, you might need to adjust the settings:</p><p>(1) In a new tab, type or paste <strong>about:config</strong> in the address bar and press Enter/Return. Click the button promising to be careful.</p><p>(2) In the filter box, type or paste <strong>pix</strong> and pause while the list is filtered</p><p>(3) Double-click <strong>layout.css.devPixelsPerPx</strong> and change its value to <strong>1.25</strong> for 25% enlargement from the classic default of 96 pixels per inch, and click OK to see whether that’s any better. I did it with 1.5 to have a 50% enlargement,</p><p>Try what suits your preferences.</p><p>Hope it helps.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=797a4ac06602" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Install TensorFlow 2 with GPU Support (Dec 2020) for Windows 10 using Conda]]></title>
            <link>https://millansanchez.medium.com/install-tensorflow-2-with-gpu-support-dec-2020-for-windows-10-using-conda-e4c0d838d497?source=rss-dffd29cd8c06------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4c0d838d497</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[windows-10]]></category>
            <dc:creator><![CDATA[Millan Sanchez]]></dc:creator>
            <pubDate>Sun, 27 Dec 2020 23:55:22 GMT</pubDate>
            <atom:updated>2020-12-27T23:55:22.406Z</atom:updated>
            <content:encoded><![CDATA[<p>Install NVIDIA Drivers for your GPU.</p><p>Install MiniConda for Python 3.8 as TensorFlow at this moment doesn’t support Python 3.9:</p><p><a href="https://docs.conda.io/en/latest/miniconda.html"></a></p><p>Once installed with default options, open the Anaconda Prompt from the applications menu in Windows 10.</p><p>First lets install Jupyter Notebooks:</p><pre>conda install -y jupyter</pre><p>Then we will create a Virtual Environment for Python 3.7 and TensorFlow:</p><pre>conda create --name tensorflow-gpu python=3.7</pre><p>(Python 3.8 is supported by TensorFlow at this moment, but as you will see it will not work to do an automated install)</p><p>and then activate the environment by using the following command:</p><pre>activate tensorflow-gpu</pre><p>Now lets install Jupyter Notebooks support to this environment:</p><pre>conda install nb_conda</pre><p>and agree to install the required components.</p><p>Theorically by running the following command the TensorFlow GPU package gets installed with all CUDA and CUDNN libraries but it is not working as Dec 2020 (don’t run this command):</p><pre>conda install tensorflow-gpu #do not run this command</pre><p>Currently <strong>conda install tensorflow-gpu</strong> installs tensorflow v2.3.0 and does NOT install the conda cudnn or cudatoolkit packages. Installing them manually (e.g. with conda install cudatoolkit=11.0) does not seem to fix the problem either.</p><p>A solution is to install an earlier version of TensorFlow, which does install cudnn and cudatoolkit, then upgrade with pip. If you follow these steps you don&#39;t need to install CUDA or CUDNN in your system using the installers from NVIDIA.</p><pre>conda install tensorflow-gpu=2.1<br>pip install tensorflow-gpu==2.3.1</pre><p>With the <strong>conda install tensorflow-gpu=2.1 </strong>command it is required to have Python 3.7 and that is the reason we didn’t install Python 3.8 when creating the environment.</p><p>Now we want to be able to select this new environment as a Kernel from Jupyter Notebooks. For this to happen run the following line in the prompt:</p><pre>python -m ipykernel install -- user -- name tensorflow -- display-name &quot;Python 3.7 (with TensorFlow GPU)&quot;</pre><p>Now lets install some basic and popular libraries to test our environment:</p><pre>conda install pandas<br>conda install scikit-learn</pre><p>That’s it. We are finished.</p><p>To test run the following code in a Jupyter notebook, and be sure to select the newly created Kernel:</p><pre>import sys</pre><pre>import tensorflow.keras<br>import pandas as pd<br>import sklearn as sk<br>import tensorflow as tf</pre><pre>print(f&quot;Tensor Flow Version: {tf.__version__}&quot;)<br>print(f&quot;Keras Version: {tensorflow.keras.__version__}&quot;)<br>print()<br>print(f&quot;Python {sys.version}&quot;)<br>print(f&quot;Pandas {pd.__version__}&quot;)<br>print(f&quot;Scikit-Learn {sk.__version__}&quot;)<br>gpu = len(tf.config.list_physical_devices(&#39;GPU&#39;))&gt;0<br>print(&quot;GPU is&quot;, &quot;available&quot; if gpu else &quot;NOT AVAILABLE&quot;)</pre><p>If all goes well you should see the following output:</p><pre>Tensor Flow Version: 2.3.1<br>Keras Version: 2.4.0<br><br>Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]<br>Pandas 1.1.5<br>Scikit-Learn 0.23.2<br>GPU is available</pre><p>as you can see GPU is detected and available.</p><p>Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4c0d838d497" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>