<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Blue Orange Digital - Medium]]></title>
        <description><![CDATA[Blue Orange is a data science consulting and machine learning development firm. Founded by engineers, we love passionate technologists and data analysts. - Medium]]></description>
        <link>https://medium.com/blue-orange-digital?source=rss----12bcb06498bc---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Blue Orange Digital - Medium</title>
            <link>https://medium.com/blue-orange-digital?source=rss----12bcb06498bc---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 11:06:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/blue-orange-digital" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unified AI Workflows: Accelerating End-to-End GenAI Projects with Databricks]]></title>
            <link>https://medium.com/blue-orange-digital/unified-ai-workflows-accelerating-end-to-end-genai-projects-with-databricks-85d9f95d8450?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/85d9f95d8450</guid>
            <dc:creator><![CDATA[Luiz]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 16:06:37 GMT</pubDate>
            <atom:updated>2025-06-23T16:06:37.109Z</atom:updated>
            <content:encoded><![CDATA[<p>Generative AI (GenAI) is rapidly transforming industries, promising unprecedented innovation and efficiency. However, building and deploying robust, production-grade GenAI applications can be a complex, multi-stage endeavor. Databricks, with its unified Data Intelligence Platform, is emerging as a key enabler, significantly accelerating end-to-end GenAI projects. By providing a cohesive environment for data management, model development, deployment, and governance, Databricks empowers organizations to move from initial concept to tangible business value at an accelerated pace.</p><p>The journey of a GenAI project typically involves several critical phases: data ingestion and preparation, model development and fine-tuning, deployment and scaling, and finally, ongoing monitoring and governance. Databricks offers a suite of tightly integrated tools and capabilities that streamline each of these stages, fostering collaboration and reducing friction for data scientists, MLOps engineers, and business stakeholders alike.</p><h3>Laying the Foundation: Accelerated Data Ingestion and Preparation</h3><p>High-quality, well-prepared data is the lifeblood of any successful GenAI application, especially for popular patterns like Retrieval Augmented Generation (RAG). Databricks simplifies and accelerates this foundational stage through its Lakehouse architecture. Built on Delta Lake, it provides a robust and scalable foundation for storing and managing the vast amounts of structured and unstructured data needed for GenAI, ensuring reliability and consistency. Databricks further streamlines this with versatile data ingestion capabilities, supporting both batch and streaming data from diverse sources. Tools like Auto Loader efficiently process new data files, ensuring GenAI applications leverage the freshest information. For RAG applications specifically, Databricks facilitates critical preprocessing steps like corpus composition, parsing, enrichment, and efficient chunking, all optimized by the high-performance Mosaic AI Vector Search with its automatic data synchronization.</p><h3>Powering Innovation: Rapid Model Development, Fine-Tuning, and Evaluation</h3><p>Once the data is ready, Databricks provides a comprehensive environment to rapidly develop, fine-tune, and evaluate GenAI models. It supports popular GenAI frameworks like LangChain and LlamaIndex within its collaborative notebook environment. Through Mosaic AI, Databricks offers access to state-of-the-art open-source Foundation Model APIs and crucial Foundation Model Fine-tuning capabilities (now part of Mosaic AI Model Training), allowing organizations to adapt pre-trained models to their specific domains for enhanced accuracy. The Mosaic AI Agent Framework further streamlines the creation of production-quality agents like RAG applications, offering tools to extend LLM capabilities and trace agent behavior. This entire process is underpinned by MLflow, which is deeply integrated for automated model tracking, versioning, and experiment management, including crucial MLflow Tracing for GenAI. Robust agent evaluation, using AI judges and custom metrics, is key to this iterative loop, helping refine agent logic and improve performance.</p><h3>Seamless Transition to Production: Accelerated Deployment and Scaling</h3><p>Getting a GenAI model into production and scaling it reliably is often a significant hurdle, a challenge Databricks addresses directly. Mosaic AI Model Serving unifies the deployment of various models — foundation, fine-tuned, or external — all managed centrally and accessible via REST API endpoints that dynamically scale. This is complemented by Databricks’ auto-scaling infrastructure, which efficiently allocates compute resources to optimize costs while ensuring performance. Furthermore, MLflow provides a standardized format for packaging models, simplifying their deployment across diverse environments.</p><h3>Ensuring Trust and Reliability: Integrated Monitoring and Governance</h3><p>For GenAI applications to be trusted and adopted, robust monitoring and governance are non-negotiable. Databricks integrates these critical aspects throughout the platform. Lakehouse Monitoring for GenAI, working closely with Agent Evaluation and MLflow Tracing, enables continuous monitoring of production applications, tracking both operational and quality metrics to quickly identify issues. Crucially, Unity Catalog delivers a unified governance solution for all data and AI assets. For GenAI, this translates to fine-grained access control, comprehensive data lineage tracking from source to production, auditing capabilities, and the ability to enforce model guardrails, ensuring security, compliance, and transparency.</p><h3>Integrating Different LLMs in Databricks</h3><p>While Databricks develops powerful models like DBRX, the platform is designed for flexibility and openness, supporting a wide ecosystem of Large Language Models (LLMs). Users are not limited to Databricks-native models; they can leverage various popular third-party models (such as those from OpenAI and Anthropic) and leading open-source options (like Meta’s Llama series) directly within the Databricks environment, often through the Model Serving capabilities. This allows organizations to choose and integrate the best models for their specific tasks and data.</p><p>DBRX, launched in March 2024, is a powerful open-source large language model (LLM) developed by Databricks’ Mosaic AI team. It employs a sophisticated Mixture-of-Experts (MoE) architecture with 132 billion total parameters, activating 36 billion parameters for any given input by dynamically selecting 4 out of 16 specialized “expert” modules. This fine-grained MoE approach, combined with techniques like RoPE, GLU, and GQA, allows DBRX to achieve state-of-the-art performance, particularly in coding and mathematical reasoning, surpassing other leading open-source models and GPT-3.5 upon its release. Furthermore, the MoE design makes DBRX highly efficient, offering faster inference speeds (up to twice as fast as Llama2–70B) and requiring less computational power for training compared to equivalent dense models.</p><p>DBRX was pre-trained on a vast, curated dataset of 12 trillion text and code tokens using Databricks’ own data processing and MLOps tools, supporting a context length up to 32,000 tokens. It’s available under an open license in two main versions: DBRX Base (the foundational model) and DBRX Instruct (fine-tuned for conversational interaction and instruction following). Users can access DBRX through Hugging Face and GitHub, integrate it via Databricks Mosaic AI Model Serving and Playground, or leverage it through partner platforms like Microsoft Azure AI, AWS, Google Cloud, and the NVIDIA API catalog..</p><h3>Quick Demos</h3><p>Databricks provides flexible ways to interact with deployed Generative AI models, enabling users to integrate LLM capabilities directly into their workflows. Two primary methods include using built-in SQL functions and leveraging the Python SDK.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XTK8g2yUFUdIjCqZ" /></figure><p>Please note that these specific features generally require an enterprise subscription and are not available in the Community Edition.</p><h4>Direct SQL Interaction with ai_query</h4><p>We can query a Large Language Model (LLM) directly within Databricks using SQL. It employs the ai_query function, specifically targeting a serving model (in the example ‘databricks-meta-llama-3–3–70b-instruct’). For there we pose a question directly as part of a SQL statement (in the example, “What are Databricks GenAI capabilities?”). The output displayed is the text generated by this LLM, which summarizes various Databricks Generative AI features like automated machine learning (AutoML), natural language processing (NLP), model serving, and its integration with the Databricks Lakehouse.</p><pre>SELECT<br>ai_query(<br>&#39;databricks-meta-llama-3–3–70b-instruct&#39;,<br>&#39;What are Databricks GenAI capabilities?&#39;<br>)</pre><p>This displays the following output:</p><blockquote>Databricks GenAI is a set of capabilities that enable businesses to build, deploy, and manage AI models at scale, using a combination of machine learning, natural language processing, and computer vision. Here are some of the key capabilities of Databricks GenAI:</blockquote><blockquote>- Automated Machine Learning (AutoML) Databricks GenAI provides automated machine learning capabilities that allow users to build and deploy models without requiring extensive machine learning expertise.</blockquote><blockquote>- Natural Language Processing (NLP): GenAI includes pre-built NLP models and tools for text analysis, sentiment analysis, entity recognition, and language translation.</blockquote><blockquote>- Computer Vision: GenAI provides pre-built models and tools for image and video analysis, object detection, and image classification.</blockquote><blockquote>- Model Serving: Databricks GenAI allows users to deploy and manage models in a scalable and secure manner, with support for real-time inference and batch scoring.</blockquote><blockquote>- Model Monitoring and Management: GenAI provides tools for monitoring model performance, data drift, and concept drift, as well as automated model retraining and updating.</blockquote><blockquote>- Explainability and Transparency: Databricks GenAI provides features for model interpretability and explainability, allowing users to understand how models are making predictions and decisions.</blockquote><blockquote>- Integration with Databricks Lakehouse: GenAI is integrated with the Databricks Lakehouse, allowing users to build and deploy AI models on top of their existing data lakehouse infrastructure.</blockquote><blockquote>- Support for Popular AI Frameworks: GenAI supports popular AI frameworks such as TensorFlow, PyTorch, and scikit-learn, allowing users to build and deploy models using their preferred frameworks.</blockquote><blockquote>- Collaboration and Governance: Databricks GenAI provides features for collaboration and governance, including model versioning, auditing, and access control.</blockquote><blockquote>- Integration with Popular Data Sources**: GenAI can integrate with popular data sources such as AWS S3, Azure Blob Storage, Google Cloud Storage, and more.</blockquote><h3>Programmatic Interaction via Python SDK</h3><p>We can also interact with a serving model endpoint programmatically using the Databricks Python SDK. We have to install the necessary databricks-sdk library, import the required WorkspaceClient and ChatMessage classes, establish a connection to the workspace, specify the target model endpoint (‘databricks-meta-llama-3–3–70b-instruct’), format the query message (“What are Databricks GenAI capabilities?”), and finally sends this message to the designated endpoint using the serving_endpoints.query method.</p><pre>%pip install — upgrade databricks-sdk<br><br>from databricks.sdk import WorkspaceClient<br>from databricks.sdk.service.serving import ChatMessage<br>endpoint_name=&lt;your_model_name&gt;<br>w = WorkspaceClient()<br>messages = [ {<br>&quot;role&quot;: &quot;user&quot;,<br>&quot;content&quot;: &quot;What are Databricks GenAI capabilities?&quot;<br>}]<br>messages = [ChatMessage.from_dict(message) for message in messages]<br>response = w.serving_endpoints.query(<br>name=endpoint_name,<br>messages=messages,<br>)<br>response.as_dict()</pre><p>The output from this SDK interaction matches the response generated earlier via the SQL ai_query function.</p><h3>Real-World Acceleration: Businesses Benefiting from Databricks for GenAI</h3><p>Numerous organizations are already leveraging Databricks to accelerate their GenAI initiatives. For instance, FactSet significantly improved accuracy by developing a text-to-code knowledge agent, while Block achieved substantial productivity gains with an AI agent system for its sellers, both built using Mosaic AI. ADP successfully built safe and accurate GenAI apps by leveraging Delta Lake and Unity Catalog for data management and governance. Companies like JetBlue and EasyJet are also innovatively using GenAI for internal operations and to empower non-technical users with natural language data insights.</p><p>The power of Databricks in accelerating end-to-end GenAI projects lies in its unified approach. By breaking down silos between data engineering, data science, and MLOps, and providing a consistent, integrated platform, Databricks eliminates friction, promotes collaboration, and enables rapid iteration. From preparing vast datasets and fine-tuning sophisticated models to deploying and governing them at scale, Databricks empowers organizations to harness the transformative potential of GenAI faster, more efficiently, and with greater confidence, turning innovative ideas into real-world impact.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85d9f95d8450" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/unified-ai-workflows-accelerating-end-to-end-genai-projects-with-databricks-85d9f95d8450">Unified AI Workflows: Accelerating End-to-End GenAI Projects with Databricks</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake AI in 2025: Making Enterprise AI Simple, Secure, and Impactful]]></title>
            <link>https://medium.com/blue-orange-digital/snowflake-ai-in-2025-making-enterprise-ai-simple-secure-and-impactful-4a4022e25fef?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/4a4022e25fef</guid>
            <category><![CDATA[copilot-programming]]></category>
            <category><![CDATA[document-ai]]></category>
            <category><![CDATA[cortex-ai]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Maximiliano Bloise]]></dc:creator>
            <pubDate>Wed, 18 Jun 2025 15:10:55 GMT</pubDate>
            <atom:updated>2025-06-18T15:10:55.452Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hJe0FySsdpTTPU40w--cXA.jpeg" /></figure><h3>Introduction</h3><p>AI adoption is no longer a future ambition — it’s a present-day necessity. In 2025, Snowflake has taken bold steps to make artificial intelligence (AI) practical for everyday business. Whether you’re a business analyst, team lead, or executive, these enhancements make it easier to use AI safely, efficiently, and with measurable value.</p><p>This article walks through:</p><ul><li>Snowflake’s newest AI tools like Cortex and Document AI</li><li>Real-life examples and SQL code</li><li>Economic results from actual businesses, and</li><li>How different teams — from HR to Finance — are using AI today.</li></ul><h3>What’s New in Snowflake AI?</h3><h4>Cortex AI — Ask Questions, Get Answers Instantly</h4><p><strong>What it is:</strong><br>Cortex AI is Snowflake’s built-in AI assistant powered by large language models (LLMs) like Meta’s Llama 3 and Mistral. It enables users to run AI queries using simple English — directly inside Snowflake’s Data Cloud — without needing separate tools or custom ML pipelines.</p><p><strong>Why it matters:</strong><br>Instead of waiting days for a data analyst to write a query, anyone can ask questions like:</p><p><em>“Who are our top-performing customers this quarter?”</em></p><p><em>… and get a complete, contextual answer instantly.</em></p><p><em>In Snowflake’s AI Predictions 2025 report, over </em><strong><em>68% of leaders said they expect every business user to interact with data using natural language within the next 12 months</em></strong><em> .</em></p><p><strong>Example:</strong></p><p>Using this in Snowflake SQL is easy thanks to their CORTEX_COMPLETE() function.</p><pre>SELECT<br>  CORTEX_COMPLETE(<br>    &#39;llama3-70b&#39;,<br>    &#39;List customers who used our app fewer than 3 times in April 2025 and might be at risk of churning.&#39;,<br>    OBJECT_CONSTRUCT(&#39;table&#39;, &#39;user_activity&#39;)<br>  ) AS ai_insight;</pre><p>This single line replaces hours of analysis and gives customer success teams immediate, actionable insights.</p><p><strong>Business Value:</strong></p><ul><li>Analysts save time on repetitive questions.</li><li>Sales and support teams become more self-sufficient.</li><li>Executives get instant, data-driven answers without dashboards.</li></ul><h4>Document AI — From PDFs to Insight</h4><p><strong>What it is:<br></strong>Document AI lets Snowflake read and interpret unstructured documents — contracts, invoices, claims, scanned forms — and convert them into clean, searchable tables.</p><p><strong>Why it matters:<br></strong>Over 80% of business data is unstructured. Most of it goes unused because extracting insights manually is time-consuming. Document AI solves that by turning scanned text into structured fields — ready for automation, analysis, or audit.</p><p><em>According to the GenAI Research Report, </em><strong><em>only 11% of enterprises said their unstructured data was AI-ready</em></strong><em>. Snowflake’s Document AI closes that gap.</em></p><p><strong>Example:</strong></p><p>Scanning documents in SQL is likewise easy with Snowflake’s <em>EXTRACT_DOCUMENT_CONTENT()</em> function.</p><pre>SELECT <br>  INVOICE_ID,<br>  AMOUNT<br>FROM <br>  TABLE(<br>    EXTRACT_DOCUMENT_CONTENT(<br>      &#39;@finance_pdf_stage/invoices/&#39;,<br>      &#39;invoice_template&#39;<br>    )<br>  );</pre><p>This lets finance teams auto-extract key data points from bulk PDFs and plug them into downstream systems — no copy-pasting required.</p><p><strong>Business Value:</strong></p><ul><li>Save hours of manual data entry per document.</li><li>Use AI to extract contract expiration dates, payment terms, and more.</li><li>Make regulatory reporting faster and more accurate.</li></ul><h4>AI That’s Ready for Production, Not Just Prototypes</h4><p><strong>What it is:<br></strong>Snowflake’s AI platform now includes tools to scale and govern models in production: infrastructure automation, cost tracking, scheduling, security controls, and integration with external ML services (like Azure, SageMaker, or Hugging Face).</p><p><strong>Why it matters:<br></strong>Many AI initiatives fail because they’re hard to maintain after the prototype stage. Snowflake’s approach removes infrastructure overhead and turns models into managed services.</p><p><em>In Forrester’s Total Economic Impact™ study, organizations saw a </em><strong><em>354% return on investment</em></strong><em>, recovering costs in under 6 months by moving from siloed AI experiments to Snowflake-managed production pipelines.</em></p><p><strong>Example:</strong></p><pre>CREATE OR REPLACE FUNCTION predict_sales(product_id STRING)<br>RETURNS FLOAT<br>AS<br>$$<br>    SELECT CALL_EXTERNAL_MODEL(<br>        &#39;forecast_model&#39;,<br>        OBJECT_CONSTRUCT(&#39;product_id&#39;, product_id)<br>    );<br>$$;<br><br>SELECT <br>  product_id,<br>  predict_sales(product_id) AS forecast_units<br>FROM product_catalog;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hJe0FySsdpTTPU40w--cXA.jpeg" /></figure><p>This example shows how Snowflake connects to external ML models (e.g., SageMaker, Vertex AI) and makes them callable inside SQL.</p><p><strong>Business Value:</strong></p><ul><li>Deploy ML models faster and cheaper.</li><li>Avoid the high cost of separate AI infrastructure.</li><li>Let business systems (e.g., supply chain apps) access AI via simple SQL.</li></ul><h4>AI Agents &amp; Copilots — Automate Complex Workflows</h4><p><strong>What it is:<br></strong>Snowflake now supports <strong>AI agents</strong> — advanced automation flows powered by LLMs that don’t just answer questions but complete entire tasks like summarizing reports, triggering alerts, or sending emails.</p><p><strong>Why it matters:<br></strong>AI is evolving from assistant (copilot) to autonomous actor (agent). That means fewer repetitive tasks for humans — and faster, more consistent outputs across teams.</p><p><em>According to Snowflake’s GenAI research, </em><strong><em>75% of executives said AI agents will soon run critical internal workflows</em></strong><em> — like compliance monitoring, operational reporting, or support response generation.</em></p><p><strong>Example: Weekly HR Feedback Automation</strong></p><pre>-- Summarize weekly feedback with Cortex<br>CREATE OR REPLACE VIEW weekly_feedback_summary AS<br>SELECT<br>  CORTEX_COMPLETE(<br>    &#39;mistral-7b&#39;,<br>    &#39;Summarize common themes in this week&#39;&#39;s employee feedback.&#39;,<br>    OBJECT_CONSTRUCT(&#39;table&#39;, &#39;employee_feedback&#39;)<br>  ) AS summary;<br><br>-- Email results every Monday<br>CREATE OR REPLACE TASK send_feedback_summary<br>  WAREHOUSE = compute_wh<br>  SCHEDULE = &#39;USING CRON 0 9 * * MON UTC&#39;<br>AS<br>BEGIN<br>  CALL send_email(<br>    &#39;hr@company.com&#39;,<br>    &#39;Weekly Employee Feedback Summary&#39;,<br>    (SELECT summary FROM weekly_feedback_summary)<br>  );<br>END;</pre><p>HR no longer needs to manually review surveys — the agent does it weekly, automatically.</p><p><strong>Business Value:</strong></p><ul><li>Saves HR teams time each week.</li><li>Provides consistent, well-formatted summaries.</li><li>Boosts employee engagement with faster follow-up.</li></ul><h4>AI That Pays Off — Proven Economic ROI</h4><p>Forrester’s 2025 Total Economic Impact report shows exactly how Snowflake AI translates to cost savings and business growth.</p><p><strong>Three-Year Value (Composite Organization):</strong></p><ul><li><strong>Increased profits from faster AI:</strong> $5.4M</li><li><strong>Efficiency gains (Ops + Analysts):</strong> $6.2M</li><li><strong>Data team productivity boost:</strong> $7.7M</li><li><strong>Infrastructure cost reduction:</strong> $5.6M</li><li><strong>Faster time-to-market:</strong> 35% shorter development cycles</li><li><strong>Total ROI: 354% return in less than 6 months</strong></li></ul><h3>Summary: Key Benefits of Snowflake AI in 2025</h3><p><strong>Cortex AI</strong></p><ul><li><em>What it does:</em> Use natural language to ask questions about your data</li><li><em>Benefit:</em> Empowers everyone to explore data</li></ul><p><strong>Document AI</strong></p><ul><li><em>What it does:</em> Extract data from PDFs, scans, and images</li><li><em>Benefit:</em> Automates manual work and speeds up decisions</li></ul><p><strong>Production AI</strong></p><ul><li><em>What it does:</em> Run secure, scalable models with monitoring</li><li><em>Benefit:</em> Reduces AI costs, boosts reliability</li></ul><p><strong>AI Agents</strong></p><ul><li><em>What it does:</em> Automate full workflows like alerts, summaries, and emails</li><li><em>Benefit:</em> Saves time and improves consistency</li></ul><h3>Use Cases by Department</h3><ul><li><strong>Marketing: </strong>Generate product descriptions, analyze campaigns and summarize trends</li><li><strong>Sales: </strong>Surface top prospects, recommend next actions, summarize call notes</li><li><strong>Finance: </strong>Automate invoice processing, detect anomalies, create executive summaries</li><li><strong>Operations: </strong>Predict demand, optimize workflows, automate alerts</li><li><strong>HR: </strong>Summarize surveys, support employee Q&amp;A, streamline onboarding</li><li><strong>Support:</strong> Power chatbots, summarize tickets, recommend responses</li></ul><h3>Final Thoughts</h3><p>In 2025, Snowflake has removed the barriers to meaningful AI. Whether you’re trying to make better decisions, automate repetitive tasks, or unlock new insights from your data, Snowflake’s AI capabilities offer:</p><ul><li>Ease of use for non-technical teams</li><li>Enterprise-grade performance and governance</li><li>Real ROI with measurable business outcomes</li></ul><p>This is the year AI goes from idea to reality — and Snowflake makes it easier than ever to take that step.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a4022e25fef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/snowflake-ai-in-2025-making-enterprise-ai-simple-secure-and-impactful-4a4022e25fef">Snowflake AI in 2025: Making Enterprise AI Simple, Secure, and Impactful</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Power of Combining Snowflake and Apache Iceberg]]></title>
            <link>https://medium.com/blue-orange-digital/the-power-of-combining-snowflake-and-apache-iceberg-a1fb44731a42?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/a1fb44731a42</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Luiz]]></dc:creator>
            <pubDate>Thu, 22 May 2025 14:46:47 GMT</pubDate>
            <atom:updated>2025-05-22T14:46:46.899Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrUxfqnoU3ewH_Mzey13PQ.jpeg" /></figure><p>In today’s fast-paced, data-driven landscape, organizations constantly seek cutting-edge solutions to optimize their data management and analytics workflows. The synergistic integration of Snowflake, the renowned cloud data warehousing platform, and Apache Iceberg, the innovative open table format, marks a significant leap forward in this pursuit. This blog post delves into this powerful combination&#39;s compelling advantages and technical nuances, offering insights for technical experts and forward-thinking business leaders aiming to revolutionize their data strategies.</p><h3>The Foundation: Understanding Snowflake and Apache Iceberg</h3><p>Snowflake is a cloud-native data warehousing powerhouse celebrated for its unparalleled scalability, blazing-fast performance, and remarkable ease of use. Snowflake’s unique architecture allows for independent computing and storage scaling, making it ideal for handling massive datasets and complex analytical workloads without the traditional overhead.</p><p>Apache Iceberg is an open-source table format meticulously designed for large-scale data lake management. Iceberg brings a wealth of enterprise-grade features to data lakes, including robust schema evolution, reliable ACID transactions, and intelligent hidden partitioning. Its open nature ensures vendor neutrality and fosters a vibrant community-driven ecosystem.</p><p>The magic happens when Snowflake seamlessly supports Iceberg tables, enabling users to harness Snowflake’s exceptional query engine while leveraging the flexibility and advanced features of Iceberg residing in external cloud storage. Adding native Iceberg support in June 2024, this integration unlocks a new era of data management capabilities by harmonizing the strengths of both platforms. Snowflake’s prowess in processing vast amounts of data aligns perfectly with Iceberg’s efficient and well-structured table format.</p><h3>Key Benefits of the Snowflake and Iceberg Integration</h3><p>This powerful synergy delivers many advantages for organizations looking to modernize their data infrastructure:</p><ul><li><strong>Robust Data Integrity with ACID Transactions:</strong> Snowflake now natively supports Iceberg’s ACID transactions, ensuring reliable and consistent data operations, even in complex, concurrent environments.</li><li><strong>Seamless Schema Evolution:</strong> Snowflake’s integration with Iceberg enables smooth schema evolution, allowing modification of table structures without query disruption, table rewrites, re-indexing jobs or the complexities of data warehousing.</li><li><strong>Enhanced Performance through Hidden Partitioning:</strong> Iceberg’s hidden partitioning optimizes data organization based on query patterns. Snowflake leverages this to improve data access, resulting in faster and more efficient query performance without user complexity.</li><li><strong>Reliable Data Management with Table Snapshots:</strong> Snowflake’s support for Iceberg table snapshots provides a dependable way to maintain historical data for point-in-time recovery and auditing, which is crucial for data governance.</li><li><strong>Flexible and Scalable External Cloud Storage:</strong> This integration allows seamless use of cost-effective, scalable external cloud storage for Iceberg tables, which Snowflake can directly query, offering flexibility and cost benefits.</li><li><strong>Efficient Metadata Handling:</strong> Snowflake’s efficient metadata management ensures effective handling of Iceberg table metadata and snapshots, providing a stable and performant foundation for all data operations in large-scale data lakes.</li><li><strong>Global Reach with Cross-Cloud and Cross-Region Support:</strong> Snowflake’s integration with Iceberg facilitates seamless data access and management across different cloud providers.</li><li><strong>Flexible Catalog Options:</strong> Snowflake offers flexible options for managing Iceberg table metadata. Organizations can choose between their built-in Open Catalog or connecting to external catalogs.</li></ul><h3>How Iceberg Tables Work in Snowflake</h3><h4>Data Storage and External Volumes</h4><p>Iceberg tables within Snowflake store their underlying data and metadata files in an external cloud storage location, such as Amazon S3, Google Cloud Storage, or Azure Storage. It’s crucial to recognize that this external storage is distinct and managed separately from your Snowflake environment. This includes the responsibility for configuring data protection and recovery mechanisms, as Snowflake’s Fail-safe storage does not apply to Iceberg tables. To facilitate connectivity with this external storage, Snowflake employs an <strong>external volume</strong>.</p><p>This named, account-level Snowflake object securely holds the necessary Identity and Access Management (IAM) credentials, enabling Snowflake to access the table data, Iceberg metadata, and manifest files that define the table’s schema, partitioning, and other essential metadata. A single external volume can serve as the connection point for one or more Iceberg tables. Notably, the storage consumed by Iceberg tables in the external cloud location does not incur direct Snowflake storage costs. For further details on associated costs, please refer to the Billing section. For more in-depth information regarding storage considerations for Iceberg tables, the “Storage for Apache Iceberg™ tables” documentation provides valuable insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BuH7XOrA7xr7-x01" /></figure><h4>Catalog Management</h4><p>The management and loading of Iceberg tables by compute engines like Snowflake are facilitated by an <strong>Iceberg catalog</strong>. This catalog represents the foundational architectural layer within the Iceberg table specification. Its core responsibilities include storing the current metadata pointer for each Iceberg table, which maps a table’s name to the location of its current metadata file, and performing atomic operations to ensure consistent updates to these metadata pointers. For a detailed exploration of Iceberg catalogs, the official Apache Iceberg documentation is a valuable resource. Snowflake offers flexibility in catalog management, supporting options such as using Snowflake itself as the Iceberg catalog or establishing a connection to an external Iceberg catalog through a <strong>catalog integration</strong>.</p><p>A catalog integration is a named, account-level Snowflake object that stores configuration details about how your table metadata is organized. This is particularly relevant when not utilizing Snowflake as the primary Iceberg catalog, such as when services like AWS Glue to manage your table. Additionally, catalog integrations are essential for interoperability with Snowflake Open Catalog, enabling you to query Iceberg tables within Open Catalog using Snowflake and to synchronize Snowflake-managed Iceberg tables with Open Catalog, thereby allowing access for other compute engines. A single catalog integration can support multiple Iceberg tables that rely on the same external catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ySOsqjp1XzTziPZx" /></figure><p>The following table summarizes the differences between these catalog options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DrLOmbk_HU9-wV7C" /></figure><p><strong>Metadata and Snapshots</strong></p><p>Iceberg employs a <strong>snapshot-based querying model</strong>. In this paradigm, data files are organized and their locations are tracked using <strong>manifest files</strong>, all coordinated by a central <strong>metadata file</strong>. A <strong>snapshot</strong> captures the state of a table at a specific point in time, providing a consistent view of the complete set of data files within the table at that moment. For comprehensive information regarding table metadata and Snowflake’s Time Travel capabilities in the context of Iceberg tables, the “Metadata and retention for Apache Iceberg tables” documentation offers detailed explanations.</p><h3>Practical Activity: Setting up Snowflake Managed Iceberg with AWS S3</h3><p>Let’s get hands-on and set up a Snowflake Managed Iceberg table pointing to an S3 bucket.</p><p>The process involves configuring AWS permissions for Snowflake to access your S3 bucket and then defining the Iceberg table and external volume within Snowflake.</p><p><strong>Prerequisites:</strong></p><ul><li>An AWS account with permissions to create S3 buckets, IAM policies, and IAM roles.</li><li>A Snowflake account with the necessary privileges to create external volumes and Iceberg tables.</li></ul><h4><strong>Sequence I: Configure AWS</strong></h4><p>We need to create an S3 bucket to store the data and Iceberg metadata, an IAM policy defining the permissions Snowflake needs, and an IAM role that Snowflake can assume.</p><p><strong>1. Create an S3 Bucket</strong></p><p>Go to the AWS S3 console. Click “Create bucket”. Give it a unique name (e.g., my-snowflake-iceberg-data-xxxx). Choose a region. Leave other settings as default for this example.</p><p><strong>2. Create an IAM Policy:</strong></p><ul><li>Go to the AWS IAM console. Navigate to “Policies” and click “Create policy”. Switch to the “JSON” tab.</li><li>Replace the default JSON with the following, substituting YOUR_S3_BUCKET_NAME with the name of the bucket you just created.</li></ul><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:PutObject&quot;,<br>                &quot;s3:DeleteObject&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::YOUR_S3_BUCKET_NAME&quot;,<br>                &quot;arn:aws:s3:::YOUR_S3_BUCKET_NAME/*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><p>This policy grants Snowflake the necessary permissions to read, write, list, and delete objects within your specified S3 bucket.</p><p><strong>3. Create an IAM Role:</strong></p><ul><li>Go to the AWS IAM console. Navigate to “Roles” and click “Create role”.</li><li>Choose “AWS account” as the trusted entity type. Select “Another AWS account”. For now, enter a placeholder Account ID (e.g., 111122223333). We will update the trusted relationship later with Snowflake’s actual account information.</li><li>Check “Require external ID” and enter a placeholder external ID (e.g., placeholder_external_id). We will also update this later.</li><li>Click “Next”. In the “Add permissions” section, search for the policy you created (SnowflakeIcebergS3AccessPolicy) and select it.</li></ul><h4><strong>Sequence II: Configure Snowflake (Initial Volume Creation)</strong></h4><p>Now, we’ll configure Snowflake to recognize your S3 bucket and the IAM role.</p><p><strong>1. Create an External Volume:<br></strong>Execute the following SQL statement, replacing s3://xxxx/ with your S3 bucket’s path (e.g., s3://my-snowflake-iceberg-data-xxxx/) and arn:aws:iam::yyyy:role/zzz with the ARN of the IAM role you created.</p><pre>CREATE OR REPLACE EXTERNAL VOLUME iceberg_vol<br>STORAGE_LOCATIONS =<br>      (<br>          (<br>              NAME = &#39;icebergvolume&#39;<br>              STORAGE_PROVIDER = &#39;S3&#39;<br>              STORAGE_BASE_URL = &#39;s3://xxxx/&#39;  <br>              STORAGE_AWS_ROLE_ARN = &#39;arn:aws:iam::yyyy:role/zzz&#39; <br>          )<br>);</pre><blockquote>ICEBERG_VOL successfully created.</blockquote><p>This command defines a named external volume that points to your S3 location using the specified AWS role.</p><p><strong>2. Check the Volume and Get ARN/External ID</strong></p><p>Snowflake needs to provide its identity information (ARN and External ID) to AWS so that AWS knows <em>which</em> Snowflake account is allowed to assume the role you created.</p><pre>DESC EXTERNAL VOLUME iceberg_vol;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*LfRKJjNLr849AuUv" /></figure><p>Look for the STORAGE_AWS_IAM_USER_ARN and STORAGE_AWS_EXTERNAL_ID in the output. <strong>Note down both of these values.</strong> They are unique to your Snowflake account and the specific external volume configuration.</p><h4><strong>Sequence III: Configure AWS (Trust Relationship Adjustment)</strong></h4><p>We now need to update the IAM role’s trust policy in AWS using the ARN and External ID obtained from Snowflake.</p><p><strong>1. Adjust the IAM Role’s Trust Relationship:</strong></p><ul><li>Go back to the AWS IAM console. Navigate to “Roles” and find the role you created (SnowflakeIcebergAccessRole).</li><li>Click on the role name. Go to the “Trust relationships” tab. Click “Edit trust policy”.</li><li>Replace the existing JSON with the following, substituting YOUR_SNOWFLAKE_USER_ARN with the value from STORAGE_AWS_IAM_USER_ARN and YOUR_SNOWFLAKE_EXTERNAL_ID with the value from STORAGE_AWS_EXTERNAL_ID</li></ul><pre>{ &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;AWS&quot;: &quot;YOUR_SNOWFLAKE_USER_ARN&quot;<br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;sts:ExternalId&quot;: &quot;YOUR_SNOWFLAKE_EXTERNAL_ID&quot;<br>        }<br>      }<br>    }<br>  ]<br>} </pre><p>This trust policy now allows the specific Snowflake identity (identified by ARN and External ID) to assume the SnowflakeIcebergAccessRole, which in turn has permissions to access your S3 bucket. Snowflake can now use this role to manage the Iceberg metadata and data files in S3.</p><p><strong>Sequence IV: Create and Interact with the Iceberg Table in Snowflake</strong></p><p>With the connection established, we can now define and interact with the Iceberg table using Snowflake.</p><p><strong>1. Create the Iceberg Table:</strong></p><p>Execute the following statement to create the Iceberg table definition within Snowflake, linking it to the external volume and specifying that Snowflake should manage the catalog</p><pre>CREATE ICEBERG TABLE iceberg_table (<br>    TransactionID INT,<br>    Date date,<br>    User STRING,<br>    Description STRING,<br>    ProductType STRING,<br>    TransactionType STRING,<br>    value float<br>)<br>PARTITIONED BY (YEAR(Date))<br>CATALOG = &#39;SNOWFLAKE&#39;<br>EXTERNAL_VOLUME = &#39;iceberg_vol&#39;; </pre><p>CATALOG = ‘SNOWFLAKE’ signifies that Snowflake will manage the Iceberg metadata in the linked external volume location.</p><blockquote>Table ICEBERG_TABLES successfully created.</blockquote><p><strong>2. Insert Records</strong></p><p>You can now perform DML operations like INSERT. Snowflake will write the data files and update the Iceberg metadata in your S3 bucket via the configured external volume and IAM role. These operations are atomic.</p><pre>INSERT INTO iceberg_table (<br>    TransactionID,<br>    Date,<br>    User,<br>    Description,<br>    ProductType,<br>    TransactionType,<br>    value<br>)<br>VALUES<br> (1001, &#39;2025-04-22&#39;, &#39;Bob&#39;, &#39;Purchase&#39;, &#39;service&#39;, &#39;Credit&#39;, 99),<br> (1002, &#39;2025-04-23&#39;, &#39;Alice&#39;, &#39;Purchase&#39;, &#39;service&#39;, &#39;Debit&#39;, -99);</pre><p>After insertion, you can <strong>SELECT * FROM iceberg_table;</strong> to see the data. Check your S3 bucket, and you will find the Iceberg metadata files and data files written by Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/0*YSFieNDn2fz6P2C7" /></figure><p><strong>3. Merge Operations:</strong></p><p>Snowflake also supports MERGE operations on Managed Iceberg tables, allowing for atomic upserts or updates based on changes from another source (like a staging table).</p><pre>MERGE INTO iceberg_table t<br>USING (SELECT * FROM updates) u<br>ON t.TransactionID = u.TransactionID<br>WHEN MATCHED THEN UPDATE SET t.value = u.value<br>WHEN NOT MATCHED THEN INSERT (TransactionID, User, value) VALUES (u.TransactionID, u.User, u.value); </pre><p>This MERGE operation, like INSERT, is performed atomically by Snowflake, ensuring data consistency in your S3-based Iceberg table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/0*wAJyBpCAgUf-qrwH" /></figure><h3>Best Practices for Leveraging Snowflake and Iceberg</h3><p>To maximize the benefits of this powerful integration, consider these essential best practices:</p><ul><li>Optimize Table Design: Architect your Iceberg tables with performance as a primary consideration. Employ appropriate partitioning strategies, leveraging Iceberg’s hidden partitioning, to ensure efficient data retrieval and query execution.</li><li>Monitor and Manage Metadata: Implement robust processes for monitoring and managing Iceberg metadata and snapshots. This ensures data consistency, facilitates efficient query planning, and supports reliable data recovery.</li><li>Leverage External Cloud Storage Wisely: Strategically utilize external cloud storage providers, ensuring proper configuration and security measures are in place for seamless interoperability with Snowflake.</li><li>Implement and Validate <a href="https://www.snowflake.com/en/blog/iceberg-tables-powering-open-standards-with-snowflake-innovations/">ACID Transactions</a>: Thoroughly test and validate ACID transaction behavior within your Iceberg tables in Snowflake to guarantee data integrity and reliability, especially in concurrent data modification scenarios.</li><li>Utilize Table Snapshots for Data Governance: Establish a regular schedule for creating and managing table snapshots. This provides crucial capabilities for data recovery, auditing, and understanding data lineage.</li><li>Enable Cross-Cloud and Cross-Region Capabilities: Configure your Snowflake environment to fully leverage the cross-cloud and cross-region support for Iceberg tables, enabling global data access and collaboration as needed.</li><li>Choose the Right Catalog Strategy: Carefully evaluate whether Snowflake’s Open Catalog or an external catalog best suits your organization’s needs and data governance framework.</li><li>Regularly Review and Update Schema: Establish a process for reviewing and evolving your Iceberg table schemas to adapt to changing data requirements while minimizing disruption to downstream processes.</li></ul><h3>Conclusion: Embracing the Future of Data Management</h3><p>Integrating Snowflake and Apache Iceberg represents a significant step towards a more flexible, scalable, and efficient data management and analytics future. By combining Snowflake’s unparalleled performance with Iceberg’s innovative table format and robust feature set, organizations can unlock new levels of data agility, reliability, and cost-effectiveness. This powerful synergy empowers technical experts to build sophisticated data lakehouses and enables business leaders to drive data-informed decisions with greater confidence and speed. Embracing this integration is not just an upgrade; it’s an investment in a future where data is truly a strategic asset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1fb44731a42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/the-power-of-combining-snowflake-and-apache-iceberg-a1fb44731a42">The Power of Combining Snowflake and Apache Iceberg</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks + Apache Iceberg + Delta Lake: A Strategic Advantage for Modern Data Platforms]]></title>
            <link>https://medium.com/blue-orange-digital/databricks-apache-iceberg-delta-lake-a-strategic-advantage-for-modern-data-platforms-f7819079f6a3?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/f7819079f6a3</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Luiz]]></dc:creator>
            <pubDate>Thu, 15 May 2025 13:09:46 GMT</pubDate>
            <atom:updated>2025-05-15T13:09:46.071Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-evolving data landscape, enterprises are facing a critical decision: how to support scalable, cost-effective data architectures that remain interoperable and future-proof. Databricks has emerged as a leader in this space by supporting both <strong>Delta Lake</strong> and <strong>Apache Iceberg</strong>, two of the most powerful open table formats today. This dual support isn’t just a technical checkbox; it’s a strategic advantage.</p><p>In this comprehensive guide, we’ll explore why supporting both Iceberg and Delta creates significant competitive advantages. We’ll examine the strategic benefits this dual support delivers from architectural and operational standpoints. We will provide detailed technical comparisons between Delta Lake and Apache Iceberg to help you understand their relative strengths. Covering sophisticated integration patterns across multi-cloud environments that become possible with format flexibility. Performance considerations for different workloads will help you optimize your implementation. A hands-on guide to working with both formats on Databricks will accelerate your practical knowledge. Finally, we’ll share proven migration strategies for existing data lakes to help you plan your transformation journey.</p><h3>📊 Why Supporting Both Iceberg and Delta Matters</h3><p>Organizations operate across various tools, clouds, and business units. Data platforms must reflect this complexity, without adding friction. By supporting both <strong>Delta Lake</strong> and <strong>Iceberg</strong>, Databricks enables a new level of flexibility.</p><h3>The Ecosystem Reality</h3><p>Iceberg has gained significant adoption outside the Databricks ecosystem with native support in numerous systems. Modern query engines like Trino, Presto, and Athena offer built-in Iceberg compatibility. Processing frameworks including Apache Flink and Apache Spark provide robust integration. Major data warehouses such as Snowflake and Amazon Redshift have incorporated Iceberg support. Cloud services including AWS Glue, Google BigQuery, and Azure Synapse have also embraced the format.</p><p>Meanwhile, Delta Lake offers deep optimizations within the Databricks ecosystem. The Delta Lake Engine, powered by Photon, delivers enhanced performance for complex analytics workloads. Delta Sharing enables secure data exchange across organizational boundaries. The Change Data Feed functionality provides comprehensive CDC capabilities for real-time data integration. Delta Live Tables offers a declarative approach to building reliable data pipelines with quality enforcement.</p><p>This complementary support allows teams to leverage significant advantages. Organizations can use the best tool for each specific workload without platform constraints. Teams can avoid creating isolated data silos that fragment the data landscape. As business needs evolve, organizations can transition between formats smoothly without major disruptions. Perhaps most importantly, teams can support hybrid workflows that leverage both formats simultaneously, maximizing flexibility and performance.</p><h3>The Data Lakehouse Vision</h3><p>Both Delta Lake and Iceberg advance the data lakehouse concept — combining the best elements of data lakes and data warehouses. However, they approach it from different angles:</p><ul><li><strong>Delta Lake</strong> focuses on performance optimizations and tight integration with Spark and Databricks</li><li><strong>Iceberg</strong> emphasizes broad ecosystem compatibility and specification-driven development</li></ul><p>By supporting both, Databricks ensures organizations can fully realize the lakehouse vision regardless of their existing investments or future plans.</p><h3>🚀 Strategic Advantages in Detail</h3><h3>1. Simplified Architecture</h3><p>With Databricks’ native support for both formats, there’s no need to spin up separate infrastructure or pipelines. This unified approach delivers several concrete benefits:</p><h4>Single Computing Environment</h4><p>Data teams work in one environment regardless of the underlying table format, reducing context switching and simplifying DevOps requirements. This unified approach eliminates the need for specialists to master multiple systems, creating a more collaborative and efficient working environment where knowledge can be shared across teams regardless of which format they primarily use.</p><h4>Unified Metadata Management</h4><p>Both formats can be managed through Unity Catalog, providing a consistent approach to metadata without duplicating governance efforts. This centralized metadata repository ensures that definitions, lineage, and documentation remain synchronized regardless of the underlying format, creating a single source of truth for data consumers and stewards throughout the organization.</p><h4>Consistent Security Model</h4><p>The same security policies can be applied across both Delta and Iceberg tables, eliminating security gaps when using multiple table formats. This unified approach to security ensures compliance requirements are met consistently and reduces the risk of accidental data exposure through misconfigured access controls across different environments.</p><h3>2. Operational Cost Reduction</h3><p>Supporting both formats in a unified platform eliminates the need for duplicate datasets that exist solely to support different processing engines. Organizations can leverage storage optimizations like Z-ordering and compaction across all tables regardless of format, maximizing storage efficiency. Administrators can apply consistent lifecycle policies for data retention across the entire data estate, ensuring compliant and cost-effective data management practices.</p><h3>3. Unified Governance, Security and Audit with Unity Catalog</h3><p>Unity Catalog enables organizations to define access controls once and apply them consistently to both Delta and Iceberg tables. This centralized approach dramatically reduces security risks and compliance gaps that often emerge in multi-platform environments. Security teams can implement organization-wide policies without worrying about format-specific configurations or edge cases, ensuring complete coverage of the data estate.</p><p>With both formats under Unity Catalog governance, organizations can track access patterns, changes, and usage metrics across all data assets regardless of format. This comprehensive auditing capability simplifies compliance reporting for regulations like GDPR, CCPA, and industry-specific requirements. Audit records provide a complete picture of data access and usage without the blind spots that often occur in multi-platform environments.</p><h3>4. Future-Proofing Your Platform</h3><p>Supporting both formats allows organizations to avoid lock-in to any single vendor’s ecosystem while still leveraging best-in-class technologies. This strategic flexibility enables negotiation leverage with technology providers and ensures the organization can pivot if market dynamics or pricing structures change. Teams can select the best tools for each job without being constrained by format limitations, creating an architecture that serves business needs rather than vendor preferences.</p><p>As the data landscape continues to evolve, dual-format support enables organizations to adapt to changing requirements without major platform overhauls. New tools and technologies can be integrated more easily when the underlying data is accessible through widely supported open formats. This adaptability reduces the technical debt that accumulates when systems cannot evolve to meet new business requirements or incorporate technological advances.</p><p>With both formats supported, organizations can build consistent data platforms that span multiple environments including on-premises, AWS, Azure, and GCP. This seamless multi-cloud capability enables genuine portability and prevents cloud vendor lock-in. Data teams can implement consistent patterns across environments, reducing the complexity that typically accompanies hybrid and multi-cloud architectures.</p><h3>🔍 Technical Comparison: Delta Lake vs. Apache Iceberg</h3><p>Understanding the technical differences helps teams make informed decisions about when to use each format.</p><h3>Feature Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dK_mOgynt_VfsqyuTyzgtQ.png" /></figure><h3>Performance Characteristics</h3><p>Different workloads benefit from different formats:</p><p>Delta Lake excels at several key workload types. It demonstrates superior performance for high-volume streaming ingestion scenarios where data velocity is a primary concern. Machine learning workloads benefit from Delta’s optimized integration with Spark MLlib and other ML frameworks. Complex ETL pipelines within Databricks leverage Delta’s transaction guarantees and optimization capabilities. Workloads that can utilize the Photon engine see significant performance gains through Delta’s native integration with this vectorized query engine.</p><p>Iceberg, on the other hand, excels in different scenarios. It performs exceptionally well in multi-engine query scenarios where diverse processing frameworks need to access the same datasets. Cross-platform analytics benefit from Iceberg’s broad ecosystem support across vendors and tools. Complex partition evolution use cases leverage Iceberg’s sophisticated approach to partition management without data rewrites. Environments where Trino or Presto are prominently used gain tremendous value from Iceberg’s deep integration with these distributed SQL query engines.</p><h3>📈 Hands-On: Using Apache Iceberg in Databricks</h3><p>Let’s explore some practical examples of using Iceberg in Databricks with Unity Catalog enabled.</p><h3>1. Download the Iceberg Jar File</h3><p>First, find your Databricks Spark version by navigating to Compute &gt; Cluster &gt; Configuration: Databricks Runtime Version. Then, download the Iceberg runtime jar file that matches this Spark version. You can search for compatible versions on the Apache Iceberg <a href="https://iceberg.apache.org/multi-engine-support/#apache-spark">website</a>. For instance, if your Databricks Runtime is 10.4 (Spark 3.2), you should download the “iceberg-spark-runtime-3.2_2.12” jar file.</p><h3>2. Create Compute Cluster and Configure it to use the Iceberg standard</h3><p>In the Spark config text field, paste the following. These settings create the necessary catalog for working with Iceberg tables. The `spark.sql.catalog.spark_catalog.warehouse` path defines where the table will be created</p><pre>spark.sql.catalog.spark_catalog = org.apache.iceberg.spark.SparkCatalog<br>spark.sql.catalog.spark_catalog.type = hadoop<br>spark.sql.catalog.spark_catalog.warehouse = /FileStore/BigDataAnalytics/IceBerg/</pre><h3>3. Upload the jar into the DBFS Catalog and Install it into your Cluster</h3><p>In the Databricks Catalog menu, upload the downloaded jar file into the DBFS structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X3K_ALCMyR82ue4c" /></figure><p>Then, on the Compute menu, create your Compute Cluster. In the Libraries tab, select the File Path option and provide the path for the uploaded jar file to install it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vkIaWLh9smS_EUD4" /></figure><h3>4. Create a Delta Table from a CSV file</h3><p>Make sure that you upload the CSV file to the Catalog. From there, use Spark to read the CSV into a Databricks dataframe.</p><pre>delta_table = spark.read.csv(&quot;/FileStore/tables/transaction___all.csv&quot;, header=True, inferSchema=True)<br>delta_table.show()</pre><p>You should see the dataframe contents, as follows:</p><pre>+-------------+----------+-------+--------------------+-------------+---------------+-------+<br>|TransactionID|      Date|   User|         Description|  ProductType|TransactionType|  value|<br>+-------------+----------+-------+--------------------+-------------+---------------+-------+<br>|            1|2024-07-13|    Eve|  Invoice for Travel|       Travel|          debit|-762.66|<br>|            1|2024-07-13|Charlie|  Invoice for Travel|       Travel|         credit| 762.66|<br>|            2|2025-03-06|    Bob|Payment for Services|     Services|          debit| -336.1|<br>|            2|2025-03-06|  Grace|Payment for Services|     Services|         credit|  336.1|</pre><h3>5. Create an Iceberg Table</h3><p>Create an Iceberg table based on the fields of the Databricks dataframe that you want to ingest.</p><pre>CREATE TABLE iceberg_table (<br>TransactionID INT,<br>Date date,<br>User STRING,<br>Description STRING,<br>ProductType STRING,<br>TransactionType STRING,<br>value float<br>)<br>USING ICEBERG;</pre><h3>6. Insert the Records from the Dataframe into the Iceberg Table</h3><p>Using Spark, use the append operation to insert the Records from the Dataframe into the Iceberg Table.</p><pre>delta_table.writeTo(“iceberg_table”).append()</pre><p>You should see its content as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DNUav6B-H_zQFGpA" /></figure><h3>7. Manipulate your Iceberg data using regular Spark SQL syntax</h3><p>Let’s create a new aggregated table from our original raw table, as an example of processing a business rule.</p><pre>CREATE TABLE aggregated_iceberg<br>SELECT User, round(sum(value),2) as Balance FROM iceberg_table group by User</pre><h3>8. Finally, let’s see our aggregated results in Iceberg</h3><p>You can query the aggregated data from the newly created Iceberg table, using a regular select statement.</p><pre>SELECT * FROM aggregated_iceberg</pre><p>You should see the balance of each client after the aggregation of each credit and debit operation that they had:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*mqFKKKRc2mGhyHY1" /></figure><h3>📋 Summary: Why Databricks’ Dual Support Matters</h3><p>By supporting both Delta Lake and Apache Iceberg, Databricks delivers:</p><p>Databricks’ dual-format support delivers maximum flexibility, allowing organizations to choose the right format for each workload without platform constraints. Teams can select Delta Lake or Iceberg based on specific requirements rather than platform limitations, optimizing performance and compatibility for each use case.</p><p>This approach enables significant cost optimization by eliminating duplicate infrastructure and streamlining operations. Organizations avoid the overhead of maintaining parallel systems just to support different table formats, reducing both capital expenses and operational complexity.</p><p>Unified governance becomes possible with consistent controls applied across all data assets regardless of format. Security policies, access controls, and audit capabilities work seamlessly across both Delta and Iceberg tables, simplifying compliance and reducing security risks.</p><p>The dual-format strategy provides essential future-proofing, allowing organizations to adapt to changing requirements without major platform changes. As business needs evolve or new tools emerge, the flexibility to leverage either format ensures continued alignment with business objectives.</p><p>Perhaps most importantly, this approach enables a simplified architecture by leveraging one platform for diverse data needs. Rather than creating a complex web of specialized systems, organizations can consolidate around a unified platform that supports the full spectrum of modern data workloads.</p><p>Successful data platforms will embrace openness, flexibility, and strong governance. Databricks’ approach to supporting both leading open table formats positions organizations to build data lakehouses that can evolve alongside their business needs.</p><p>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7819079f6a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/databricks-apache-iceberg-delta-lake-a-strategic-advantage-for-modern-data-platforms-f7819079f6a3">Databricks + Apache Iceberg + Delta Lake: A Strategic Advantage for Modern Data Platforms</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Data Quality: How Databricks Delta Live Tables Simplifies ETL and Ensures Reliability]]></title>
            <link>https://medium.com/blue-orange-digital/automating-data-quality-how-databricks-delta-live-tables-simplifies-etl-and-ensures-reliability-307b7ac28995?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/307b7ac28995</guid>
            <category><![CDATA[reliability]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[delta-live-tables]]></category>
            <dc:creator><![CDATA[Maximiliano Bloise]]></dc:creator>
            <pubDate>Fri, 09 May 2025 21:03:11 GMT</pubDate>
            <atom:updated>2025-05-09T21:03:11.241Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqFVBYcaZiLjN4b-oPNsgA.png" /></figure><h3>Introduction</h3><p>In modern data platforms, building and maintaining reliable ETL pipelines is a significant engineering challenge. Legacy approaches typically involve manual orchestration, custom quality checks, and extensive operational overhead.<br>Delta Live Tables (DLT) offers a declarative framework that simplifies pipeline development, automates operational complexity, and embeds data quality directly into the transformation process.<br>This article discusses the technical architecture of DLT, real-world examples of its impact, and best practices for engineering teams adopting it.</p><h3>Overview: Delta Live Tables Architecture</h3><p>Delta Live Tables is a managed ETL framework built on top of Apache Spark and Delta Lake, integrated into the Databricks platform. It supports declarative pipeline definitions in SQL or Python and automates critical tasks including:</p><ul><li>Incremental ingestion and processing</li><li>Materialization of target datasets as managed Delta tables or views</li><li>Orchestration of task dependencies based on a Directed Acyclic Graph (DAG)</li><li>Built-in data quality enforcement with expectations</li><li>Observability through system tables and event logs</li></ul><p>Pipelines can run in continuous mode (streaming ingestion) or triggered mode (batch processing) with unified APIs.</p><h3><strong>Core Capabilities and Internal Mechanics</strong></h3><h4>Declarative Pipeline Development</h4><p>Engineers define a pipeline using the <em>@dlt.table</em> or <em>@dlt.view</em> decorators. DLT analyzes these definitions to build a dependency graph and manage execution order automatically. Example:</p><pre>import dlt<br><br>@dlt.table<br>def customers_cleaned():<br>    return (<br>        spark.read.table(&quot;raw_customers&quot;)<br>        .filter(&quot;email IS NOT NULL&quot;)<br>        .dropDuplicates([&quot;customer_id&quot;])<br>    )</pre><h4><strong>End-to-End Incremental Processing</strong></h4><p>DLT minimizes compute costs and latency by:</p><ul><li>Tracking processed offsets or file versions</li><li>Applying transformations only to new or updated records</li><li>Atomically writing outputs to Delta tables</li></ul><p>Incremental computation is transparent to the engineer when using Delta sources.</p><h4>Built-in Data Quality Validation: Expectations</h4><p>Expectations are declaratively attached to tables or views:</p><pre>@dlt.expect(&quot;valid_email&quot;, &quot;email LIKE &#39;%@%&#39;&quot;)<br>@dlt.table<br>def customers_validated():<br>    return dlt.read(&quot;customers_cleaned&quot;)</pre><p>Expectations can drop, quarantine, or fail pipeline execution based on how the rule is configured.</p><h4><strong>Observability and Monitoring</strong></h4><p>All pipeline activities are logged automatically:</p><ul><li><em>LIVE.DLT_EVENT_LOG</em></li><li><em>LIVE.DLT_METRICS</em></li><li><em>LIVE.DLT_DATA_QUALITY</em></li></ul><p>These system tables provide comprehensive lineage, event tracking, and quality reporting.</p><h4><strong>Serverless Execution and Cost Efficiency</strong></h4><p>DLT pipelines can leverage serverless compute to automatically allocate and deallocate resources, optimizing for high concurrency and low idle time. Implementing this is easy, just select the Serverless option when configuring the pipeline.</p><h3><strong>Practical Implementation: Full Lifecycle of a Delta Live Tables Pipeline with Medallion Architecture</strong></h3><p>Implementing a data pipeline with <strong>Delta Live Tables (DLT)</strong> involves a well-defined sequence of steps, from creation to monitoring and governance. Below is a complete, step-by-step guide using the <strong>Medallion Architecture</strong> (Bronze → Silver → Gold) — a common best practice pattern for progressive data refinement in Lakehouse platforms.</p><h4>Step 1: Define and Create the Pipeline</h4><p>From the Databricks UI:</p><ol><li>Navigate to <strong>Workflows → Delta Live Tables</strong>.</li><li>Click <strong>Create Pipeline</strong>.</li><li>Choose the source notebook or SQL script containing your DLT definitions.</li><li>Assign a <strong>pipeline name</strong>, <strong>target catalog and schema</strong>, and choose whether to run in <strong>Triggered</strong> (batch) or <strong>Continuous</strong> (streaming) mode.</li><li>Select the <strong>compute mode</strong>: either <strong>serverless</strong> or a custom cluster policy.</li></ol><p>These settings initialize the pipeline metadata and configure where outputs will be written.</p><h4>Step 2: Bronze Layer — Raw Ingestion</h4><p>Use <em>@dlt.table</em> to ingest raw data from JSON files, Kafka topics, or CDC sources.</p><pre>import dlt<br><br>@dlt.table(<br>    name=&quot;bronze_customers&quot;,<br>    comment=&quot;Raw customer data ingested from source systems&quot;<br>)<br>def bronze_customers():<br>    return spark.read.format(&quot;json&quot;).load(&quot;/mnt/data/raw/customers/&quot;)</pre><p><strong>DLT behavior</strong>:</p><ul><li>Automatically infers schema (unless explicitly defined)</li><li>Tracks ingestion metadata</li><li>Handles incremental ingestion if source supports it</li></ul><h4>Step 3: Silver Layer — Cleansing and Validation</h4><p>Add transformations and quality enforcement with <em>@dlt.expect</em>, <em>@dlt.expect_or_drop</em>, or <em>@dlt.expect_all</em>.</p><pre>@dlt.expect(&quot;non_null_customer_id&quot;, &quot;customer_id IS NOT NULL&quot;)<br>@dlt.expect_or_drop(&quot;valid_email&quot;, &quot;email LIKE &#39;%@%&#39;&quot;)<br>@dlt.table(<br>    name=&quot;silver_customers&quot;,<br>    comment=&quot;Validated and deduplicated customer records&quot;<br>)<br>def silver_customers():<br>    return (<br>        dlt.read(&quot;bronze_customers&quot;)<br>        .dropDuplicates([&quot;customer_id&quot;])<br>        .filter(&quot;is_deleted = false&quot;)<br>    )</pre><p><strong>DLT behavior</strong>:</p><ul><li>Validates each row against expectations</li><li>Automatically generates metrics for passed/failed records</li><li>Drops or quarantines invalid records based on expectation type</li><li>Enables <strong>incremental processing</strong> using metadata tracking</li></ul><h4><strong>Step 4: Gold Layer — Business Logic and Aggregation</strong></h4><p>Final business logic, joins, and aggregation for consumption by dashboards, ML, or downstream APIs.</p><pre>@dlt.table(<br>    name=&quot;gold_customer_ltv&quot;,<br>    comment=&quot;Customer-level lifetime value and segmentation&quot;<br>)<br>def gold_customer_ltv():<br>    orders = spark.read.table(&quot;silver_orders&quot;)<br>    customers = dlt.read(&quot;silver_customers&quot;)<br><br>    return (<br>        customers<br>        .join(orders, &quot;customer_id&quot;)<br>        .groupBy(&quot;customer_id&quot;, &quot;segment&quot;)<br>        .agg(<br>            sum(&quot;order_total&quot;).alias(&quot;total_revenue&quot;),<br>            count(&quot;*&quot;).alias(&quot;order_count&quot;)<br>        )<br>    )</pre><p><strong>DLT behavior</strong>:</p><ul><li>Materializes table in the specified catalog and schema</li><li>Can be optimized with <strong>merge-on-update</strong> logic for upserts</li><li>Automatically builds DAG dependencies and executes transformations in order</li></ul><h4>Step 5: Configure Table Properties and Materialization Policies</h4><p>In the DLT UI or pipeline settings, configure:</p><ul><li><strong>Table type</strong>: <em>table</em> (persisted) or <em>view</em> (virtual)</li><li><strong>Data freshness</strong>: Streaming mode (continuous updates) or batch (triggered runs)</li><li><strong>Materialization mode</strong>: <strong>Append-only</strong> (default), <strong>Complete refresh, Apply changes</strong> (merge with primary keys)</li></ul><p>Set <strong>Auto Optimize</strong> and <strong>Auto Compaction</strong> for performance tuning.</p><h4>Step 6: Monitor with System Tables and Logs</h4><p>DLT exposes built-in observability via:</p><p><strong>Data Quality:</strong></p><pre>SELECT expectation, dataset, passed_records, failed_records<br>FROM LIVE.DLT_DATA_QUALITY;</pre><p><strong>Execution Logs:</strong></p><pre>SELECT timestamp, event_type, details<br>FROM LIVE.DLT_EVENT_LOG<br>ORDER BY timestamp DESC;</pre><p>These are updated automatically with each run.<br>You can integrate them into custom dashboards (e.g., Databricks SQL).</p><h4>Step 7: Enforce Governance with Unity Catalog</h4><p>When using Unity Catalog:</p><ul><li>Tables are registered under a centralized catalog (<em>&lt;catalog&gt;</em>.<em>&lt;schema&gt;</em>.<em>&lt;table&gt;</em>)</li><li>Access control is enforced via fine-grained ACLs (table/column/row level)</li><li>All DLT activity is <strong>audit logged</strong>, including reads/writes by pipeline actors</li><li>Lineage is tracked automatically and viewable via the Databricks catalog UI</li></ul><p>DLT integrates seamlessly with Unity Catalog for governance, lineage, and compliance.</p><h4>Step 8: Operationalize and Schedule</h4><p>You can:</p><ul><li>Schedule runs via <strong>Databricks Workflows</strong> (e.g., every 15 minutes, hourly, daily)</li><li>Trigger pipeline execution via <strong>REST API</strong></li><li>Integrate with orchestration tools like <strong>Airflow</strong>, <strong>dbt</strong>, or <strong>Azure Data Factory</strong></li></ul><p>DLT pipelines are version-controlled via notebooks/scripts and can be deployed via CI/CD (e.g., GitHub Actions, Azure DevOps).</p><h3><strong>Best Practices for Engineers Adopting Delta Live Tables</strong></h3><ul><li>Define <strong>expectations</strong> as early as possible in the pipeline to catch bad data upstream.</li><li>Prefer <strong>incremental reads</strong> and <strong>merge-on-update</strong> materializations for cost efficiency.</li><li>Use <strong>system tables</strong> to automate operational health checks and SLA monitoring.</li><li>Adopt <strong>serverless execution</strong> where suitable to minimize cluster management overhead.</li><li>Integrate with <strong>Unity Catalog</strong> for fine-grained access control and auditing.</li></ul><h3>Conclusion</h3><p>Delta Live Tables offers a modern, scalable solution for building production-grade ETL pipelines with minimal manual intervention.<br>Through declarative development, incremental processing, integrated data quality validation, and full observability, DLT frees engineers from many typical maintenance burdens enabling them to shift more energies to focus innovation — while ensuring reliability and consistency at scale.</p><p>Engineering teams adopting DLT can expect faster delivery times, lower operational risk, and better data quality across both batch and streaming workloads.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=307b7ac28995" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/automating-data-quality-how-databricks-delta-live-tables-simplifies-etl-and-ensures-reliability-307b7ac28995">Automating Data Quality: How Databricks Delta Live Tables Simplifies ETL and Ensures Reliability</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Iceberg in Action: A PySpark Example with Google Colab]]></title>
            <link>https://medium.com/blue-orange-digital/apache-iceberg-in-action-a-pyspark-example-with-google-colab-3e7e3c25f5ca?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/3e7e3c25f5ca</guid>
            <dc:creator><![CDATA[Luiz]]></dc:creator>
            <pubDate>Tue, 29 Apr 2025 14:25:57 GMT</pubDate>
            <atom:updated>2025-04-29T14:25:57.756Z</atom:updated>
            <content:encoded><![CDATA[<p>In the rapidly evolving landscape of big data, organizations struggle with efficiently managing large-scale data lakes. Traditional file-based approaches often lead to consistency issues, poor query performance, and complex data management. Enter Apache Iceberg, an open table format designed to solve these challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*3VXWqM9QH_Y7r4F3gUTlUw.jpeg" /></figure><h3>What is Apache Iceberg?</h3><p>Apache Iceberg is an open-source table format for massive analytic datasets. Developed originally at Netflix and later donated to the Apache Software Foundation, Iceberg addresses the limitations of traditional data lake storage methods by providing a high-performance format with built-in capabilities for schema evolution, partition evolution, and time travel operations.</p><p>Unlike traditional Hive tables or simple file collections, Iceberg maintains a versioned “snapshot” approach to table state, tracking all changes to table data with atomic operations. This design allows for consistency guarantees that were previously only available in traditional database systems.</p><h3>Key Features and Benefits</h3><h4>Schema Evolution</h4><p>One of Iceberg’s standout features is its support for in-place schema evolution. Unlike traditional approaches that might require table rebuilds or complex migration procedures, Iceberg allows adding, renaming, and dropping columns, updating column types, and reordering fields. All these operations happen without disrupting ongoing queries or necessitating data copies.</p><h4>Time Travel and Snapshot Isolation</h4><p>Iceberg’s architecture maintains a complete history of table states, enabling query-time travel to view previous table states, data rollbacks to previous versions, and consistent snapshot reads even during concurrent writes.</p><h4>Partition Evolution</h4><p>Iceberg decouples the physical data layout from table definition, allowing partitioning schemes to evolve without expensive data migrations. You can change partition schemes over time, add or remove partitions, and hide partitioning complexity from users.</p><h4>Performance Optimizations</h4><p>Iceberg incorporates several performance optimizations. It uses hidden partitioning for efficient data skipping and provides file-level statistics and indexes to improve query planning. The format also supports data clustering, compaction capabilities, and incremental query planning for better performance.</p><h3>Practical Implementation with Google Colab</h3><p>Let’s explore how to implement Apache Iceberg using PySpark in Google Colab. This hands-on example will demonstrate creating tables, loading data, performing updates, and leveraging time travel capabilities.</p><h4>Setting Up the Environment</h4><p>First, we need to initialize a Spark session with the necessary Iceberg configurations:</p><pre>from pyspark.sql import SparkSession<br>spark = SparkSession.builder \<br>  .appName(&quot;IcebergLocalDevelopment&quot;) \<br>  .config(&#39;spark.jars.packages&#39;, &#39;org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2&#39;) \<br>  .config(&quot;spark.sql.extensions&quot;, &quot;org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions&quot;) \<br>  .config(&quot;spark.sql.catalog.local&quot;, &quot;org.apache.iceberg.spark.SparkCatalog&quot;) \<br>  .config(&quot;spark.sql.catalog.local.type&quot;, &quot;hadoop&quot;) \<br>  .config(&quot;spark.sql.catalog.local.warehouse&quot;, &quot;spark-warehouse/iceberg&quot;) \<br>  .getOrCreate()<br>spark.sql(&quot;SHOW DATABASES&quot;).show()</pre><pre>+---------+<br>|namespace|<br>+---------+<br>|  default|<br>+---------+</pre><p>This configuration:</p><ul><li>Imports the Iceberg Spark runtime package</li><li>Enables Iceberg’s Spark extensions</li><li>Configures a local catalog named “local” using Hadoop for storage</li><li>Sets the warehouse location for Iceberg tables</li></ul><h4>Loading Initial Data</h4><p>Next, we’ll load some transaction data from a CSV file:</p><pre>csv_df = spark.read.csv(&quot;transactions - in.csv&quot;, header=True, inferSchema=True)<br>csv_df.show()</pre><pre>+-------------+----------+-------+--------------------+------------+----------------+-------+<br>|TransactionID|      Date|   User|         Description|Product Type|Transaction Type|  value|<br>+-------------+----------+-------+--------------------+------------+----------------+-------+<br>|            1|2024-07-13|    Eve|  Invoice for Travel|      Travel|           debit|-762.66|<br>|            2|2024-07-13|Charlie|  Invoice for Travel|      Travel|          credit| 762.66|<br>|            3|2025-03-06|    Bob|Payment for Services|    Services|           debit| -336.1|<br>|            4|2025-03-06|  Grace|Payment for Services|    Services|          credit|  336.1|<br>+-------------+----------+-------+--------------------+------------+----------------+-------+</pre><h4>Creating an Iceberg Table</h4><p>Now we’ll create our first Iceberg table from this data:</p><pre>csv_df.writeTo(&quot;local.schema.trx&quot;).create()<br>trx = spark.sql(&quot;select * from local.schema.trx&quot;)<br>trx.show()</pre><p>This creates a table named “trx” in the “schema” namespace of our “local” catalog. The table automatically inherits the schema from our DataFrame.</p><h4>Understanding Data Versioning</h4><p>Let’s simulate a mistake by appending duplicate records to our table:</p><pre>trx.writeTo(&quot;local.schema.trx&quot;).append()<br>trx.show()</pre><pre>+-------------+----------+-------+--------------------+------------+----------------+-------+<br>|TransactionID|      Date|   User|         Description|Product Type|Transaction Type|  value|<br>+-------------+----------+-------+--------------------+------------+----------------+-------+<br>|            1|2024-07-13|    Eve|  Invoice for Travel|      Travel|           debit|-762.66|<br>|            2|2024-07-13|Charlie|  Invoice for Travel|      Travel|          credit| 762.66|<br>|            3|2025-03-06|    Bob|Payment for Services|    Services|           debit| -336.1|<br>|            4|2025-03-06|  Grace|Payment for Services|    Services|          credit|  336.1|<br>|            1|2024-07-13|    Eve|  Invoice for Travel|      Travel|           debit|-762.66|<br>|            2|2024-07-13|Charlie|  Invoice for Travel|      Travel|          credit| 762.66|<br>|            3|2025-03-06|    Bob|Payment for Services|    Services|           debit| -336.1|<br>|            4|2025-03-06|  Grace|Payment for Services|    Services|          credit|  336.1|<br>+-------------+----------+-------+--------------------+------------+----------------+-------+</pre><p>This duplicates all records in our table — a common mistake in data pipelines. With traditional formats, this could be difficult to remedy, but Iceberg keeps track of all changes.</p><h4>Exploring Time Travel</h4><p>One of Iceberg’s most powerful features is the ability to view table history and travel back in time:</p><pre>spark.sql(“SELECT * FROM local.schema.trx.history”).show()</pre><pre>+--------------------+-------------------+-------------------+-------------------+<br>|     made_current_at|        snapshot_id|          parent_id|is_current_ancestor|<br>+--------------------+-------------------+-------------------+-------------------+<br>|2025-04-22 06:34:...|6487231307215147082|               NULL|               true|<br>|2025-04-22 06:34:...|7563156050951784873|6487231307215147082|               true|<br>+--------------------+-------------------+-------------------+-------------------+</pre><p>This command displays the complete history of our table, showing every modification with its timestamp and operation type.</p><h4>Rolling Back to a Previous Version</h4><p>We can identify the initial snapshot ID and roll back to it:</p><pre>snapshot_id = spark.sql(&quot;SELECT snapshot_id FROM local.schema.trx.history where parent_id is null&quot;).collect()[0][0]<br>snapshot_id</pre><pre>6487231307215147082</pre><pre>spark.sql(f”CALL local.system.rollback_to_snapshot(‘local.schema.trx’, {snapshot_id})”)<br>spark.sql(&quot;SELECT * FROM local.schema.trx&quot;).show()</pre><pre>+-------------+----------+-------+--------------------+------------+----------------+-------+<br>|TransactionID|      Date|   User|         Description|Product Type|Transaction Type|  value|<br>+-------------+----------+-------+--------------------+------------+----------------+-------+<br>|            1|2024-07-13|    Eve|  Invoice for Travel|      Travel|           debit|-762.66|<br>|            2|2024-07-13|Charlie|  Invoice for Travel|      Travel|          credit| 762.66|<br>|            3|2025-03-06|    Bob|Payment for Services|    Services|           debit| -336.1|<br>|            4|2025-03-06|  Grace|Payment for Services|    Services|          credit|  336.1|<br>+-------------+----------+-------+--------------------+------------+----------------+-------+</pre><p>Now our table is back to its original state, undoing the erroneous data insertion.</p><pre>spark.sql(“SELECT * FROM local.schema.trx.history”).show();</pre><pre>+--------------------+-------------------+-------------------+-------------------+<br>|     made_current_at|        snapshot_id|          parent_id|is_current_ancestor|<br>+--------------------+-------------------+-------------------+-------------------+<br>|2025-04-22 06:34:...|6487231307215147082|               NULL|               true|<br>|2025-04-22 06:34:...|7563156050951784873|6487231307215147082|              false|<br>|2025-04-22 06:34:...|6487231307215147082|               NULL|               true|<br>+--------------------+-------------------+-------------------+-------------------+</pre><p>The history now shows both the original creation, the append operation, and our rollback.</p><h4>Performing Merge Operations</h4><p>A typical data pipeline requirement is merging new records with existing ones, performing inserts or updates as needed. Iceberg simplifies this with SQL-based merge operations:</p><pre>data = [(998, &quot;Joao&quot;,100), (999, &quot;Maria&quot;,-100), (3,&quot;Bob&quot;, 99),(4,&quot;Grace&quot;,-99)]<br>columns = [&quot;TransactionID&quot;,&quot;User&quot;,&quot;value&quot;]<br>updates = spark.createDataFrame(data, columns)<br>updates.createOrReplaceTempView(&quot;updates&quot;)<br>spark.sql(&quot;SELECT * FROM updates&quot;).show()</pre><pre>+-------------+-----+-----+<br>|TransactionID| User|value|<br>+-------------+-----+-----+<br>|          998| Joao|  100|<br>|          999|Maria| -100|<br>|            3|  Bob|   99|<br>|            4|Grace|  -99|<br>+-------------+-----+-----+</pre><p>With the updates variable above, the Iceberg MERGE syntax to update existing records and insert new ones in a single atomic operation is:</p><pre>spark.sql(&quot;&quot;&quot;<br>MERGE INTO local.schema.trx t USING (SELECT * FROM updates) u ON t.TransactionID = u.TransactionID<br>WHEN MATCHED THEN UPDATE SET t.value = u.value<br>WHEN NOT MATCHED THEN INSERT (TransactionID,User, value) VALUES (u.TransactionID, u.User, u.value);<br>&quot;&quot;&quot;)<br>spark.sql(&quot;SELECT * FROM local.schema.trx&quot;).show()</pre><pre>+-------------+----------+-------+--------------------+------------+----------------+-------+<br>|TransactionID|      Date|   User|         Description|Product Type|Transaction Type|  value|<br>+-------------+----------+-------+--------------------+------------+----------------+-------+<br>|            1|2024-07-13|    Eve|  Invoice for Travel|      Travel|           debit|-762.66|<br>|            2|2024-07-13|Charlie|  Invoice for Travel|      Travel|          credit| 762.66|<br>|            3|2025-03-06|    Bob|Payment for Services|    Services|           debit|   99.0|<br>|            4|2025-03-06|  Grace|Payment for Services|    Services|          credit|  -99.0|<br>|          998|      NULL|   Joao|                NULL|        NULL|            NULL|  100.0|<br>|          999|      NULL|  Maria|                NULL|        NULL|            NULL| -100.0|<br>+-------------+----------+-------+--------------------+------------+----------------+-------+</pre><h3>Conclusion</h3><p>Apache Iceberg represents a significant advancement in data lake architecture, bringing traditional database features like ACID transactions, schema evolution, and time travel to big data platforms. Our practical example demonstrates that Iceberg simplifies many complex data management tasks while providing powerful features previously unavailable in data lakes.</p><p>Whether you’re building a new data lake or considering upgrades to your existing infrastructure, Iceberg offers compelling advantages that can improve data reliability, simplify operations, and enhance query performance. Its growing ecosystem of integrations with tools like Spark, Flink, Trino, and Snowflake further establishes it as a foundation for modern data architecture.</p><p>Organizations adopting Iceberg can build more robust, flexible, and maintainable data platforms that better serve their analytical needs while reducing operational complexity.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e7e3c25f5ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/apache-iceberg-in-action-a-pyspark-example-with-google-colab-3e7e3c25f5ca">Apache Iceberg in Action: A PySpark Example with Google Colab</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Visualization with Databricks Series — Article #3, Storytelling: Go Beyond Showing Data —…]]></title>
            <link>https://medium.com/blue-orange-digital/data-visualization-with-databricks-series-article-3-storytelling-go-beyond-showing-data-c03575a283d8?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/c03575a283d8</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-vizualisation]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[storytelling]]></category>
            <dc:creator><![CDATA[Felipe Altermann]]></dc:creator>
            <pubDate>Tue, 18 Mar 2025 17:55:52 GMT</pubDate>
            <atom:updated>2025-03-18T17:55:52.460Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Visualization with Databricks Series — Article #3, Storytelling: Go Beyond Showing Data — Tell a Story With It!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lQimPhr2QVnfu1N-aUOYfA.jpeg" /><figcaption>Image source:<a href="https://unsplash.com/@uxindo?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash"> UX Indonesia</a></figcaption></figure><p>My two previous articles in this series — <a href="https://medium.com/blue-orange-digital/data-visualization-with-databricks-series-article-1-quickly-visualize-and-understand-your-data-14b65c4523de">[Link for Article #1]</a> and <a href="https://medium.com/blue-orange-digital/data-visualization-with-databricks-series-article-2-dashboards-to-better-share-data-insights-9e10ccd138fa">[Link for Article #2]</a> — highlighted how <a href="https://blueorange.digital/blog/beyond-the-spreadsheet-data-visualizations-impact-on-analytics/">data visualizations</a> organized in smart dashboards within Databricks can strengthen data-driven business strategies by providing clear visual insights that support informed decision-making.</p><p>But there is just one missing peace in this puzzle: storytelling!</p><p><strong>What is Storytelling?</strong></p><p>Storytelling is the art of sharing narratives through words or visuals to engage, inform, or entertain. This enduring and impactful practice captures human emotions, experiences, and ideas, and helps us interpret what we see.</p><p><strong>How is it related to data?</strong></p><p>Storytelling and data are deeply connected because it is the art of effectively communicating insights tp transform raw data into a compelling narrative.</p><p>Here some key points to consider:</p><ul><li>Make Data Understandable — Just like a good story, data needs structure. A well-crafted narrative helps translate complex datasets into something accessible and compelling.</li><li>Engage Your Audience — Data alone can be dry, but storytelling adds context, emotion, and relevance. Whether presenting to executives or stakeholders, a data-driven story keeps them engaged.</li><li>Highlight Key Insights — Instead of overwhelming people with numbers, storytelling helps focus on the most important takeaways, making it easier to spot trends, patterns, and anomalies.</li><li>Drive Decision-Making — A strong data story influences actions by showing not just what is happening, but why it matters and what should be done next.</li><li>Create a Connection — People relate more to stories than to spreadsheets. Framing data within a narrative makes it memorable and persuasive.</li></ul><p><strong>Storytelling with Data in Databricks SQL</strong></p><p>Databricks SQL allows you to craft compelling data stories using queries, visualizations, and dashboards. Here’s how you can leverage it effectively:</p><p><strong>1. Define the Narrative (What’s the Story?)</strong></p><p>Before diving into the data, ask:</p><ul><li>What insight do you want to communicate?</li><li>Who is your audience (executives, analysts, customers)?</li><li>What actions should this data inspire?</li></ul><p>Example:</p><p>📌 If you’re analyzing customer churn, your story could be:</p><p><em>“Our churn rate has increased due to delays in customer support response times.”</em></p><p><strong>2. Use Queries to Extract Meaningful Data</strong></p><p>Once you define your story, structure SQL queries to support the narrative.</p><ul><li>Identify Key Metrics — Revenue, churn rate, sales trends, etc.</li><li>Use Aggregations &amp; Window Functions — Find patterns over time.</li><li>Join Tables for Context — Merge sales, customer, and product data for a complete picture.</li></ul><p>Example:</p><p>📌 Show revenue trends by month for the last year</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVTF6L9uwMZN7RTtjbmzsw.png" /><figcaption>Image source: Author.</figcaption></figure><p><strong>3. Choose the Right Visualizations</strong></p><p>Different chart types help highlight specific insights:</p><ul><li>📊 Bar Chart — Best for comparing categories (e.g., revenue by region).</li><li>📈 Line Chart — Great for trends over time (e.g., monthly sales growth).</li><li>📍 Heatmap — Helps spot correlations (e.g., peak shopping hours).</li><li>📑 Pivot Table — For detailed breakdowns (e.g., revenue by product &amp; region).</li></ul><p>Example (image below):</p><p>If you want to show weather statistics over time, use a line chart to show increasing or decreasing trends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BfGWJXBoYpIGABCDkOJ3PA.png" /><figcaption>Image source: <a href="https://www.databricks.com/blog/2022/05/19/how-i-built-a-streaming-analytics-app-with-sql-and-delta-live-tables.html">Databricks</a>.</figcaption></figure><p><strong>4. Build Dashboards to Tell a Cohesive Story</strong></p><p>Databricks SQL Dashboards allow you to combine multiple charts, giving you different perspectives on your data in a single view. With interactive filters and query parameters, you can dynamically adjust insights to explore data more effectively. Additionally, text widgets help provide context and explanations directly on the dashboard, making it easier to interpret key findings.</p><p>Example Dashboard Components for Customer Retention Analysis:</p><ul><li>📊 Bar Chart — Top reasons for churn (survey data).</li><li>📈 Line Chart — Customer churn trend over time.</li><li>📍 Map — Churn by geographic location.</li><li>📑 KPI Widget — Displays the current churn rate.</li></ul><p>Example (image below):</p><p>To display real-time monitoring for rental bike stations, use an interactive map. Additionally, enhance your dashboard by including a supporting table with detailed station availability, incorporating weather conditions for deeper insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eQ7-CJjA2VQj9vUToTGDTg.png" /><figcaption>Image source: <a href="https://www.databricks.com/blog/2022/05/19/how-i-built-a-streaming-analytics-app-with-sql-and-delta-live-tables.html">Databricks</a>.</figcaption></figure><p><strong>5. Make it Actionable</strong></p><p>Making data actionable is the final and most crucial step in effective storytelling with data. Insights alone are not enough; they must lead to decisions and tangible outcomes. A well-designed Databricks SQL dashboard should not only present data but also guide users toward meaningful actions. By structuring your dashboards to highlight key takeaways and provide clear recommendations, you transform raw numbers into a strategic asset.</p><p>One way to make insights more actionable is by using annotations to emphasize important trends. Instead of letting users interpret charts on their own, call out patterns, anomalies, or key changes directly within the dashboard. This ensures that decision-makers quickly grasp what matters most without needing to analyze every detail. Additionally, setting up alerts in Databricks can help teams stay informed when critical thresholds are met, enabling them to react in real-time rather than retrospectively.</p><p>Beyond highlighting trends and sending alerts, the real power of data storytelling comes from providing recommendations. When a dashboard reveals a concerning pattern — such as a spike in customer churn — offering a next step, like launching a targeted retention campaign, makes the data immediately useful. The goal is to bridge the gap between data and action, ensuring that insights lead to measurable improvements.</p><ul><li>Use annotations in dashboards to highlight key trends.</li><li>Add alerts in Databricks to notify teams of critical thresholds.</li><li>Provide recommendations based on findings.</li></ul><p>Example (image below):</p><p>The interactive map highlights a selected bike station, displaying an annotation box with key details. This annotation provides context on the station’s activity, helping users understand trends like high rental volumes or availability fluctuations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zcIjicAzxml9GVmSSdquQg.jpeg" /><figcaption>Image source: <a href="https://www.databricks.com/blog/2022/05/19/how-i-built-a-streaming-analytics-app-with-sql-and-delta-live-tables.html">Databricks</a>.</figcaption></figure><p><strong>Conclusion</strong></p><p>Data storytelling serves as a bridge across departments — such as <a href="https://blueorange.digital/industries/financial-services/">Financial Services</a>, <a href="https://blueorange.digital/industries/real-estate/">Real Estate &amp; Construction</a>, <a href="https://blueorange.digital/industries/travel-hospitality/">Travel &amp; Hospitality</a>, <a href="https://blueorange.digital/industries/insurance/">Insurance</a>, <a href="https://blueorange.digital/industries/cpg/">CPG</a>, <a href="https://blueorange.digital/industries/sports-gaming/">Sports &amp; Gaming</a> — to foster a data-focused organizational culture. By combining data with narratives and visuals, insights become more engaging, impactful, and memorable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c03575a283d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/data-visualization-with-databricks-series-article-3-storytelling-go-beyond-showing-data-c03575a283d8">Data Visualization with Databricks Series — Article #3, Storytelling: Go Beyond Showing Data —…</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Container Apps]]></title>
            <link>https://medium.com/blue-orange-digital/azure-container-apps-c5047dfde0dd?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/c5047dfde0dd</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[scalability]]></category>
            <dc:creator><![CDATA[Maximiliano Bloise]]></dc:creator>
            <pubDate>Thu, 06 Mar 2025 19:03:19 GMT</pubDate>
            <atom:updated>2025-03-06T19:03:19.219Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Container technology has reshaped the way organizations build, run, and manage applications. Containerized solutions allow teams to develop highly scalable, consistent, and efficient microservices — a major game-changer for modern software delivery pipelines. Azure Container Apps (ACA) offers a fully managed, serverless experience that lets you reap the benefits of container architectures without wrestling with the complexities of Kubernetes.</p><p>I have worked with our clients at <a href="https://blueorange.digital/">Blue Orange Digital</a> on designing, deploying, and scaling modern data and cloud infrastructures. In this post, we’ll dive into the core components of Azure Container Apps, compare it to Azure Container Instances (ACI), and walk you through a quick start guide for deploying your first container-based application.</p><h3>Core Components</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZIFEpMO_Nx7nyhloBwbyQ.png" /></figure><h3>ACA vs ACI</h3><p><a href="https://learn.microsoft.com/en-us/azure/container-apps/">Azure Container Apps</a> enables you to build serverless microservices and jobs based on containers. Distinctive features of Container Apps include:</p><ul><li>Optimized to run general purpose containers, especially for applications that span many microservices deployed in containers.</li><li>Powered by Kubernetes and open-source technologies like <a href="https://dapr.io/">Dapr</a>, <a href="https://keda.sh/">KEDA</a>, and <a href="https://www.envoyproxy.io/">envoy</a>.</li><li>Supports Kubernetes-style apps and microservices with features like <a href="https://learn.microsoft.com/en-us/azure/container-apps/connect-apps">service discovery</a> and <a href="https://learn.microsoft.com/en-us/azure/container-apps/revisions">traffic splitting</a>.</li><li>Enables event-driven application architectures by supporting scale based on traffic and pulling from <a href="https://learn.microsoft.com/en-us/azure/container-apps/scale-app">event sources like queues</a>, including <a href="https://learn.microsoft.com/en-us/azure/container-apps/scale-app">scale to zero</a>.</li><li>Supports running on demand, scheduled, and event-driven <a href="https://learn.microsoft.com/en-us/azure/container-apps/jobs">jobs</a>.</li></ul><p>Azure Container Apps doesn’t provide direct access to the underlying Kubernetes APIs. If you require access to the Kubernetes APIs and control plane, you should use <a href="https://learn.microsoft.com/en-us/azure/aks/intro-kubernetes">Azure Kubernetes Service</a>. However, if you would like to build Kubernetes-style applications and don’t require direct access to all the native Kubernetes APIs and cluster management, Container Apps provides a fully managed experience based on best-practices. For these reasons, many teams prefer to start building container microservices with Azure Container Apps.</p><p><a href="https://learn.microsoft.com/en-us/azure/container-instances/">Azure Container Instances (ACI)</a> provides a single pod of Hyper-V isolated containers on demand. It can be thought of as a lower-level “building block” option compared to Container Apps. Concepts like scale, load balancing, and certificates aren’t provided with ACI containers. For example, to scale to five container instances, you create five distinct container instances. Azure Container Apps provide many application-specific concepts on top of containers, including certificates, revisions, scale, and environments. Users often interact with Azure Container Instances through other services. For example, Azure Kubernetes Service can layer orchestration and scale on top of ACI through <a href="https://learn.microsoft.com/en-us/azure/aks/virtual-nodes">virtual nodes</a>. If you need a less “opinionated” building block that doesn’t align with the scenarios Azure Container Apps is optimizing for, Azure Container Instances is an ideal option.</p><p>More:<a href="https://learn.microsoft.com/en-us/azure/container-apps/compare-options">Comparing Container Apps with other Azure container options</a></p><h4>Comparison Table</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UqMcRPMV5gXcYZxh6rvKHg.png" /></figure><p>*If there are jobs that take more than 4 minutes, we can leverage asynchronous job handling:</p><ol><li>When a request arrives, the container returns 202 Accepted, indicating the job has been received.</li><li>The actual job runs in the background (inside the same container or in another container).</li><li>Poll a status endpoint to check job progress if needed.</li></ol><h3>Deploying your first container-based application</h3><h4>Prerequisites</h4><ol><li><strong>Install Azure CLI (</strong>Follow instructions: <a href="https://learn.microsoft.com/en-us/cli/azure/install-azure-cli">Azure CLI Installation</a>)</li><li><strong>Login to Azure<br></strong>az login<br>az account set --subscription &quot;&lt;subscription ID or name&gt;&quot;</li></ol><h4>Create Azure Resources</h4><ol><li>Create Resource Group<br>az group create --name bod-rg --location eastus</li><li>Create Azure Container Registry (ACR)<br>az acr create --name bodacr --resource-group bod-rg --sku Basic</li><li>Enable Admin Access for ACR<br>az acr update --name bodacr --admin-enabled true</li><li>Check Login for ACR<br>az acr login --name bodacr</li></ol><h4>Build &amp; Publish the App</h4><p>Prepare the <em>Project</em></p><pre>/bod-container-app<br>  ├── job.py<br>  ├── Dockerfile</pre><p>Create <em>Dockerfile</em></p><pre>FROM python:3.12<br>WORKDIR /app<br>COPY job.py .<br>RUN pip install flask<br>CMD [&quot;python&quot;, &quot;job.py&quot;]</pre><p>Create <em>job.py</em></p><pre>from flask import Flask<br><br>app = Flask(__name__)<br><br>@app.route(&#39;/&#39;)<br>def home():<br>    return &quot;Hello there!&quot;<br><br>if __name__ == &quot;__main__&quot;:<br>    app.run(host=&quot;0.0.0.0&quot;, port=8080)</pre><p>Build Docker Image</p><blockquote>Azure Container Apps expects a linux/amd64 (x86_64) architecture.</blockquote><blockquote>Assert Docker daemon is running.</blockquote><pre>docker buildx build \<br>  --platform linux/amd64 \<br>  -t bodacr.azurecr.io/bod-container-app:v1 .</pre><p>Push Image to Azure Container Registry (ACR)</p><pre>docker push bodacr.azurecr.io/bod-container-app:v1</pre><h4>Deploy the App in Azure</h4><p>Create Azure Container Apps Environment</p><pre>az containerapp env create \<br>    --name bod-container-env \<br>    --resource-group bod-rg \<br>    --location eastus</pre><p>Deploy the Container App</p><pre>az containerapp create \<br>    --name bod-container-app \<br>    --resource-group bod-rg \<br>    --environment bod-container-env \<br>    --image bodacr.azurecr.io/bod-container-app:v1 \<br>    --registry-server bodacr.azurecr.io \<br>    --registry-username &quot;$(az acr credential show --name bodacr --query username --output tsv)&quot; \<br>    --registry-password &quot;$(az acr credential show --name bodacr --query &#39;passwords[0].value&#39; --output tsv)&quot; \<br>    --ingress external \<br>    --target-port 8080 \<br>    --scale-rule-name http-scale-rule \<br>    --scale-rule-http-concurrency 1 \<br>    --min-replicas 0 \<br>    --max-replicas 1</pre><h3>How Azure Container Apps Scale</h3><p>Azure Container Apps automatically <strong>scales up and down based on demand</strong>.</p><ul><li>Min replicas: The minimum number of container instances to keep running (even if there is no traffic). Default: 0 (scales to zero).</li><li>Max replicas: The maximum number of container instances that can be created to handle traffic. Default: 10 (can be increased).</li><li>Scale-in (Cool-down period): The cool-down period is the time before scaling down after a traffic drop (default: ~30s).</li><li>Scale-out trigger: New containers spawn when concurrent requests exceed the defined threshold (e.g., 1 request per container in this case).</li><li>Scale to zero: If no traffic is detected for a while, the app scales to zero (fully stops to save costs).</li></ul><h3>Verify the Deployment</h3><h4>Get the Public URL</h4><pre>az containerapp show \<br>  --name bod-container-app \<br>  --resource-group bod-rg \<br>  --query properties.configuration.ingress.fqdn -o tsv</pre><h4>Check App Health</h4><pre>az containerapp revision show \<br>  --name bod-container-app \<br>  --resource-group bod-rg \<br>  --revision bod-container-app--9rpsch6 \<br>  --query properties -o json</pre><h4>Test the App</h4><pre>curl -v https://bod-container-app.mangobush-29c33bfc.eastus.azurecontainerapps.io</pre><h3>Running Statuses</h3><blockquote>These statuses indicate the current state of a container app’s revision and replicas.</blockquote><ul><li>⚫ Scaled to 0: The app is idle with no active replicas because no traffic has been detected. The app will automatically scale up when a request is received.</li><li>🔄 Activating: The container is starting up, pulling the image, and initializing before handling requests. This happens when the app wakes up from Scaled to 0 or after deployment.</li><li>🟢 Running: The container is active and serving requests. It can scale up to handle more traffic or scale down when idle.</li></ul><h3>Conclusion</h3><p><strong>Azure Container Apps</strong> offers an elegant, serverless pathway for deploying containerized microservices without the overhead of managing Kubernetes directly. From auto-scaling and service discovery to built-in ingress and revision-based rollouts, ACA simplifies many of the complexities of cloud-native development.</p><p>If you’re looking to launch or optimize container-based workloads, <strong>Blue Orange Digital</strong> is here to help. Our team of experts specializes in architecting, deploying, and scaling modern data platforms, and we’d love to bring our expertise to your next project.</p><p><strong>Ready to move forward?</strong> <a href="https://blueorange.digital/">Get in touch with Blue Orange Digital</a> to explore how we can streamline your container deployment strategy and accelerate your journey toward a more agile, cost-effective infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5047dfde0dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/azure-container-apps-c5047dfde0dd">Azure Container Apps</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS re:Invent 2024 — reflections for cloud watchers]]></title>
            <link>https://medium.com/blue-orange-digital/aws-re-invent-2024-reflections-for-cloud-watchers-2ff9af5f15e4?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/2ff9af5f15e4</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[technology-news]]></category>
            <dc:creator><![CDATA[Alfredo Lozano]]></dc:creator>
            <pubDate>Tue, 10 Dec 2024 15:44:40 GMT</pubDate>
            <atom:updated>2024-12-10T15:44:40.350Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS re:Invent 2024 — reflections for cloud watchers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*u-LzzsYSlvz43NEf2QTusw.gif" /></figure><p>Closing down on 2024, this year we saw the rise of independent players applying Artificial Intelligence to disrupt workplace productivity with projects like Cursor or Perplexity, and we saw the rise of projects disrupting the foundation models arena from projects like Anthropic or Mistral — all of this powered by advancements in access to compute and software abstractions.</p><p>We also saw the cloud giants refine their position, shaping service offerings to meet the demand for AI-driven solutions that either simplify building applications of AI or promise to solve your need to build them in the first place — with managed and preconfigured options.</p><p>As an engineer working with a team that specializes in designing advanced, tailored cloud solutions to solve unique operational challenges, the <strong>AWS 2024 re:Invent</strong> conference really stirred up my enthusiasm for the year to come. Beyond the new technologies and updates, the event offered moments of deep reflection about where the industry is headed and how we, as builders, can keep up.</p><p>With this in mind, let&#39;s dive into some of the highlights 🙂‍↔️</p><h3><em>Embracing simplexity</em></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*rSJ3s2vPO9crhvOz3Amakw.gif" /></figure><p>Let&#39;s start with Amazon CTO Dr. Werner Vogels’ keynote at AWS re:Invent 2024, which explored a key tension in cloud computing: how complexity grows naturally as systems evolve and how simplicity — or <strong><em>simplexity </em></strong><em>— </em>can guide architects in managing it. Using the example of Amazon S3, he reflected on how what starts as simple — like a storage service built on a basic API — can grow into a complex global infrastructure, while still appearing simple to its users.</p><p>A standout takeaway was the idea that simplicity is not accidental; it takes thoughtful discipline. It is an invitation to <strong><em>work with complexity</em></strong> instead of preemptively attempting to avoid it.</p><h4>i. Complexity Isn’t the Enemy</h4><p>Complexity is unavoidable in systems that evolve to meet changing needs. The goal isn’t to eliminate it but to manage it. For example, the shift from <strong>eventual to strong consistency in S3</strong> wasn’t about making the system simpler internally — it added layers of complexity to ensure customers no longer had to worry about edge cases.</p><p>“Complexity sneaks in all the time,” he said, emphasizing the need to move into the places where it can be managed best, such as through <strong>automated reasoning</strong> and <strong>robust design patterns</strong>.</p><h4>ii. Simplicity as a Discipline</h4><p>Keeping systems simple requires vigilance from day one. Vogels described how early decisions, such as designing modular microservices for S3, allowed it to evolve into a global-scale system without locking engineers into decisions that would eventually hinder growth. This wasn’t just about foresight; it was about building a culture where teams expect to revisit and rework their architectures regularly.</p><h4>iii. Evolution is Inevitable, but It’s Also Strategic</h4><p>“You never step in the same river twice,” quoted Vogels, paraphrasing Heraclitus. For AWS, this philosophy translates into continuous iteration. <strong>Every order-of-magnitude increase in scale</strong> has required S3 to undergo significant architectural revisions. It’s a reminder that systems designed to last need to be flexible enough to grow.</p><h4>iv. Unintended Complexity Slows Progress</h4><p>Vogels drew a distinction between <strong>intentional complexity</strong> (which drives innovation and scalability) and unintended complexity (which creeps in due to poor architectural decisions or a lack of oversight). Recognizing and addressing these patterns early — before they lead to bloated, fragile systems — was a recurring theme.</p><h3>AWS and Intelligence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*VMG42SErEcw5J85Lxmitcg.gif" /></figure><p>Generative AI has emerged as a <strong>transformative force in technology</strong>, reshaping industries and reimagining how businesses operate. As conveyed by Matt Garman, CEO of AWS: AI isn’t just another tool; it’s a foundational shift in how we approach problem-solving, efficiency, and scalability.</p><h4><strong>Startups are Leading the Charge</strong></h4><p>Startups have been at the forefront of this transformation, using generative AI to<strong> disrupt traditional industries and create entirely new markets</strong>. Recognizing this, AWS is doubling down on empowering innovation with initiatives like expanded credits and tools tailored to fast-scaling AI workloads. This support ensures that <strong>small teams with big ideas</strong> can leverage cutting-edge technology without being constrained by costs or complexity.</p><h4><strong>AI is the Next Frontier of Compute</strong></h4><p>Generative AI demands unprecedented levels of compute power, and the advancements in AI-specific hardware, like <strong>Trainium 2</strong>, are pivotal. These tools deliver significant cost and performance improvements, enabling businesses to train and deploy large-scale models more efficiently. The integration of such technologies into AI workflows is already driving tangible results, <strong>from streamlining creative tools to enabling smarter, faster enterprise applications</strong>.</p><h4><strong>Collaborating with Responsibility</strong></h4><p>As AI capabilities grow, so does the responsibility to implement them securely and scalably. Innovations in AWS’s infrastructure, like Nitro and Graviton architectures, ensure that AI systems are not only powerful but also meet the highest standards of security and compliance.</p><p>Generative AI is more than a technical achievement — it’s a framework for rethinking how businesses operate. By pairing robust infrastructure with practical tools, the focus remains on enabling innovation that is accessible, responsible, and deeply impactful. This evolution is not just about building smarter systems; it’s about empowering organizations to unlock entirely new ways of working.</p><h3>Leverage and the Physics of Acceleration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*JxJIV9b6DhhVJUt5TvWANQ.gif" /></figure><p>At <strong>AWS re:Invent 2024</strong>, the theme of acceleration took center stage, with announcements spanning AI, data, and cloud infrastructure that aim to empower businesses to iovate faster and more effectively. From transformative updates to foundational tools like <strong>S3</strong>, <strong>Amazon Q</strong> and <strong>Sagemaker</strong> to cutting-edge advancements in AI hardware with <strong>Trainium 3</strong>, AWS demonstrated its commitment to reducing complexity and unlocking new levels of efficiency. The conference highlighted not just the tools but also the strategies needed to navigate today’s fast-changing technological landscape, offering a clear vision of how organizations can leverage the cloud to stay ahead. Here are the most exciting announcements that reflect this vision.</p><h4><strong>S3 Tables — Optimized for Analytics Workloads</strong></h4><p>AWS introduced <strong>Amazon S3 Tables</strong>, a new bucket type purpose-built to optimize the storage and querying of tabular data using <strong>Apache Iceberg</strong>. This innovation automates maintenance tasks such as data compaction and snapshot management, eliminating the need for manual Spark job scheduling traditionally required in Iceberg table management. By pre-partitioning data and streamlining operations, S3 Tables are expected to deliver up to <strong>3x faster query performance</strong> and <strong>10x higher transactions per second</strong> compared to self-managed Iceberg tables in general-purpose S3 buckets. Features include integrated Iceberg catalog support, enhanced table-level access control, and a REST endpoint for seamless query integration, simplifying large-scale analytics workflows.</p><h4><strong>Amazon Q Developer Tools — Redefining Efficiency</strong></h4><p>AWS introduced major updates to <strong>Amazon Q</strong>, a toolset aimed at automating tasks across the software development lifecycle. These enhancements include:</p><p><strong>AI-Powered Autonomous Agents</strong>: Capable of handling multi-step workflows, including resource management, testing, and feature implementation, significantly reducing manual overhead.</p><p><strong>Natural Language Resource Management</strong>: Developers can now set up and optimize AWS services like S3 and Lambda using simple commands, streamlining operations for non-expert users.</p><p><strong>Release Management Automation</strong>: A new agent automates code deployment and release workflows, ensuring safer and faster rollouts while reducing errors.</p><h4><strong>Nova GenAI Models — AWS&#39; New Frontier in Generative AI</strong></h4><p>AWS introduced the <strong>Nova AI model family</strong>, their suite of generative AI models for text, images, and video. As part of Amazon Bedrock, the Nova models are designed to provide businesses with flexible, high-performance AI capabilities without the need for extensive customization or infrastructure management. Their key features include:</p><p>• <strong>Cross-Modal Generative Capabilities</strong>: Nova models can generate high-quality content across multiple modalities, making them versatile for industries like media, marketing, and education.</p><p>• <strong>Customizable via Bedrock</strong>: Businesses can easily fine-tune Nova models for specific use cases using Amazon Bedrock, leveraging their own data to optimize performance while maintaining privacy.</p><p>• <strong>Efficiency at Scale</strong>: Designed to be resource-efficient, the Nova models enable faster inference and training times, making them accessible to businesses of all sizes.</p><h4><strong>SageMaker Lakehouse — Unifying Data Lakes and Warehouses</strong></h4><p>AWS launched <strong>Amazon SageMaker Lakehouse</strong>, a service designed to seamlessly unify data lakes and data warehouses. It allows teams to run analytics and machine learning workflows on a single, consistent dataset, significantly improving efficiency and reducing costs.</p><h4><strong>Trainium 3 Chips — Supercharging AI Training</strong></h4><p>AWS revealed <strong>Trainium 3</strong>, its next-generation AI training chips, offering up to four times the performance of its predecessor. This cutting-edge hardware is designed to accelerate large-scale model training, supporting businesses aiming to stay competitive in the AI-driven era.</p><h4><strong>VMware to EC2 Migrations Made Easy</strong></h4><p>AWS unveiled tools to simplify migrating VMware workloads to Amazon EC2. These automated capabilities address common bottlenecks and reduce the expertise required for migration, making the transition more accessible to businesses at any scale.</p><h4><strong>Data Strategies with Data Mesh and Data Zone</strong></h4><p>AWS announced advancements in <strong>data mesh</strong> and <strong>data fabric</strong> architectures, empowering organizations to manage and share data as a product. These updates emphasize decentralized governance to increase agility and enhanced tools for building and cataloging data products, with <strong>AWS Data Zone</strong> playing a pivotal role in improving data discoverability and security.</p><p>Companies like ANZ Bank reported saving millions by implementing data mesh strategies supported by AWS.</p><h4><strong>Bedrock’s Generative AI Integration</strong></h4><p>AWS’s fully managed service, <strong>Amazon Bedrock</strong>, received updates aimed at simplifying the development of generative AI applications. New features include support for <strong>customizable foundation models</strong>, integration of <strong>retrieval-augmented generation (RAG)</strong> pipelines to enhance AI-driven insights, and <strong>simplified deployment</strong> with serverless capabilities, eliminating the need to manage infrastructure.</p><h4><strong>Enhancing Developer Experience with Kubernetes</strong></h4><p>AWS expanded features in <strong>EKS (Elastic Kubernetes Service)</strong> to simplify application deployment and scaling. This includes better integration with <strong>Transit Gateway</strong> and <strong>VPC Lattice</strong>, enabling smoother network configurations, as well as enhanced tools for <strong>Kubernetes management</strong>, improving developer productivity.</p><h4><strong>Nitro Enclaves — Advancing Security</strong></h4><p>AWS announced updates to <strong>Nitro Enclaves</strong>, which isolate sensitive workloads in secure environments. These updates provide enhanced encryption capabilities and make it easier to meet stringent compliance requirements.</p><h3><strong>A Broader Takeaway</strong></h3><p>Underlying all of this was a deeply human insight: the systems we build are a reflection of the people who build them.</p><p>AWS’s announcements were impressive, but the more human reflections made them feel grounded — reminders that behind technological elements are teams grappling with the same challenges we all face: how to grow without losing what makes us effective and how to stay simple in a world that insists on complexity.</p><p>If <strong>AWS Re:Invent 2024</strong> has taught us anything, it’s that the pace of cloud innovation is accelerating. With transformative tools like Amazon Q, groundbreaking AI advancements like Bedrock and Nova, and cutting-edge strategies for modernizing data and systems, the opportunities for innovation are as vast as ever.</p><p>At <strong>Blue Orange</strong>, we specialize in helping businesses navigate these advancements, tailoring solutions to unlock their full potential. Ready to reinvent your tech strategy? Let’s start the conversation today. <a href="https://blueorange.digital/contact-us/">Reach out to our team</a> to explore what’s possible for your organization.</p><p><a href="https://www.aboutamazon.com/news/aws/aws-reinvent-2024-keynote-live-news-updates">Announcements and updates from AWS re:Invent 2024</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Faim5x73crbM%3Fstart%3D1%26feature%3Doembed%26start%3D1&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Daim5x73crbM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Faim5x73crbM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfb361bc55a76a3410ca838a0c827adf/href">https://medium.com/media/bfb361bc55a76a3410ca838a0c827adf/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLY7m5LQliAo%3Fstart%3D5707%26feature%3Doembed%26start%3D5707&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLY7m5LQliAo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLY7m5LQliAo%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/68aed049ef51229d058bcf4f9c9c5c8c/href">https://medium.com/media/68aed049ef51229d058bcf4f9c9c5c8c/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ff9af5f15e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/aws-re-invent-2024-reflections-for-cloud-watchers-2ff9af5f15e4">AWS re:Invent 2024 — reflections for cloud watchers</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Insights and Innovations from Microsoft Ignite 2024]]></title>
            <link>https://medium.com/blue-orange-digital/insights-and-innovations-from-microsoft-ignite-2024-298427c53f15?source=rss----12bcb06498bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/298427c53f15</guid>
            <dc:creator><![CDATA[Diana Bald]]></dc:creator>
            <pubDate>Tue, 26 Nov 2024 12:02:23 GMT</pubDate>
            <atom:updated>2024-11-26T12:02:22.157Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ejs-GHGB2bsaGOb90cYmkQ.png" /><figcaption>Ignite conference introduction in black and white charcoal.</figcaption></figure><p><strong>Did you know?</strong> Businesses using GenAI (GenAI) are <strong>already achieving a $3.70 ROI for every $1 invested</strong>, with industries like financial services and retail leading the way. This is one of the insights shared at this year’s <a href="https://youtu.be/3YiB2OvK6sY?feature=shared">Microsoft Ignite conference</a>, which took place last week with over 200,000 registered participants.</p><p>Microsoft Ignite is an annual event spotlighting the updates and creations that enable users to unleash the full potential of Microsoft’s technology. It showcases how Microsoft is changing the way people and organizations approach work.</p><p>Several interesting advancements in artificial intelligence (AI) were shared, designed to help businesses unlock new potential. From GenAI tools to enterprise-grade platforms, these innovations are reshaping how organizations approach productivity, scalability, and decision-making. For businesses leveraging data, analytics, machine learning, and AI, these updates offer both immediate value and long-term opportunities for growth.</p><p>Here are some of the most relevant takeaways:</p><h3>GenAI and Productivity Tools</h3><h4>Boost Productivity with <a href="https://www.microsoft.com/en-us/microsoft-365/copilot">Microsoft 365 Copilot</a></h4><p>Microsoft 365 Copilot’s latest features are designed to simplify workflows and enhance collaboration:</p><ul><li><strong>Automated Project Management</strong>: The new Project Manager agent can create plans, assign tasks, track progress, and even generate reports — freeing up teams to focus on strategy.</li><li><strong>Real-Time Meeting Summaries</strong>: Facilitator agents in Teams provide live note-taking and actionable insights during meetings.</li></ul><p><strong>Example:</strong> The Microsoft 365 Copilot automates repetitive meeting tasks like summarizing discussions or scheduling follow-ups — allowing teams to stay aligned and productive.</p><h4>GenAI Fine-Tuning for Custom Workflows</h4><p>Microsoft’s advancements in fine-tuning GPT-4 models make it easier to customize AI for industry-specific needs. For those unfamiliar with the term, fine-tuning is like teaching a pre-trained AI model how to specialize in a specific area to meet unique business needs. Fine-tuning trains the model with specific examples, data, or tasks so it performs within your business context.</p><p>For example:</p><ul><li><strong>In finance:</strong> Fine-tuning can adapt to identify fraud patterns unique to your institution.</li><li><strong>In healthcare: </strong>Fine-tuning can help an AI model understand medical terminology and prioritize patient care needs.</li><li><strong>In retail:</strong> Fine-tuning can help an AI model recommend products based on your customers’ buying habits and preferences or tailor AI models for predictive analytics, customer behavior insights, or personalized marketing campaigns.</li></ul><p>Fine-tuning makes AI more relevant and effective for a particular business or industry, enhancing its accuracy and usefulness without needing to build a model from scratch. Microsoft’s advancements in fine-tuning GPT-4 models make it easier to customize AI for industry-specific needs.</p><h3>Scaling AI Across the Enterprise</h3><h4>Improvements in Retrieval-Augmented Generation (RAG) with <a href="https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search">Azure AI Search</a></h4><p>Azure AI Search introduces significant enhancements to Retrieval-Augmented Generation (RAG) capabilities, making AI search applications faster and more accurate. Features like <strong>semantic ranking</strong> prioritize the most relevant results by understanding the context of queries, while <strong>query rewriting</strong> refines user input to improve search accuracy. These improvements enable businesses to extract actionable insights from vast datasets with minimal effort.</p><p><strong>Example:</strong> Enterprises can use RAG-powered search to efficiently find important documents, analyze customer feedback, or streamline knowledge management — all while reducing the time spent on manual data retrieval. These advancements make Azure AI Search a vital tool for organizations aiming to maximize the value of their data.</p><h4><a href="https://learn.microsoft.com/en-us/azure/ai-studio/concepts/fine-tuning-overview">Azure AI Foundry</a>: A Platform for Growth</h4><p>Azure AI Foundry is a one-stop solution for businesses looking to integrate AI into their operations quickly, safely, and effectively. It reduces complexity, accelerates development, and ensures the AI solutions meet industry standards. Azure AI Foundry empowers businesses to design, deploy, and scale AI solutions efficiently:</p><ul><li><strong>Comprehensive Tools</strong>: Access pre-built app templates, GitHub integrations, Visual Studio, Azure AI SDK, and enterprise-grade control.</li><li><strong>Developer-Friendly</strong>: Integrates seamlessly with familiar tools like Visual Studio, enabling faster time-to-value.</li></ul><p><strong>Benefits:</strong> Businesses can scale AI across departments without worrying about infrastructure complexity.</p><h3><a href="https://techcommunity.microsoft.com/blog/appsonazureblog/introducing-serverless-gpus-on-azure-container-apps/4288571">Leveraging Serverless GPUs</a> for Cost-Effective AI Workloads</h3><p>Serverless GPUs allow businesses to use powerful computing resources to run machine learning (ML) and AI tasks without having to manage or maintain the underlying hardware infrastructure. Azure’s serverless GPUs can be leveraged to execute machine learning workloads. The advantages include minimized infrastructure costs, real-time inferencing capabilities, and simplified deployment of complex models.</p><p>For businesses focused on innovation, serverless GPUs offer a high-performance solution without breaking the bank.</p><h3>Governance and Compliance for Responsible AI</h3><h4>AI Governance Facilitated with Microsoft Tools</h4><p>Azure AI’s governance tools help businesses ensure compliance while scaling AI:</p><ul><li><strong>AI Reports</strong>: Generate detailed assessments of model performance and impact.</li><li><strong>Risk Evaluations</strong>: Assess and mitigate risks in AI-generated content, including text and images.</li></ul><p>To align AI deployments with organizational policies, apply these best practices:</p><ul><li><strong>Regular Reviews: </strong>Conduct ongoing evaluations of AI outputs to identify and correct potential compliance issues or ethical concerns.</li><li><strong>Azure Content Safety:</strong> Utilize tools to detect and address inappropriate or harmful content in AI-generated outputs, ensuring a secure and trustworthy implementation.</li><li><strong>Stakeholder Involvement:</strong> Engage cross-functional teams, including legal, compliance, and operations, to oversee and guide AI implementations effectively.</li></ul><p>By leveraging Azure AI’s governance capabilities, organizations can confidently adopt AI technologies while maintaining accountability, security, and trust.</p><h4>Secure Future Initiative</h4><p><a href="https://youtu.be/3YiB2OvK6sY?feature=shared">In his keynote presentation, Satya Nadella</a> shared Microsoft’s broader commitment to cybersecurity and responsible AI innovation via Microsoft’s <a href="https://www.microsoft.com/en-us/trust-center/security/secure-future-initiative"><em>Secure Future Initiative</em></a> (“Secure by design · Secure by default · Secure operations”). The Initiative integrates advanced AI-driven security enhancements across its platforms. With tools like <a href="https://www.microsoft.com/en-us/security/business/ai-machine-learning/microsoft-security-copilot">Microsoft Security Copilot</a> and <a href="https://www.microsoft.com/en-us/security/business/siem-and-xdr/microsoft-security-exposure-management">Security Exposure Management</a>, combined with collaborative programs like the <a href="https://www.microsoft.com/en-us/msrc/microsoft-zero-day-quest?oneroute=true">Zero Day Quest</a>, this initiative ensures businesses can adopt cutting-edge AI solutions with confidence, knowing their data and operations are protected in an evolving threat landscape.</p><h3>GenAI ROI Across Industries</h3><p>Key findings from <a href="https://info.microsoft.com/ww-landing-business-opportunity-of-ai.html">IDC’s The Business Opportunity</a> reveal that GenAI adoption surged from <strong>55% in 2023</strong> to <strong>75% in 2024</strong>, reflecting growing trust and reliance on AI solutions. A study commissioned by Microsoft highlights key ROI trends, demonstrating the transformative power of GenAI across multiple industries. The findings reveal:</p><ul><li><strong>Financial Services</strong>: GenAI leads with the highest ROI, helping firms automate complex processes such as fraud detection, compliance reporting, and customer personalization.</li><li><strong>Retail</strong>: Retailers have leveraged AI to improve supply chain efficiency and provide personalized customer experiences, contributing to significant gains in revenue and operational efficiency.</li><li><strong>Manufacturing</strong>: Productivity gains stem from predictive maintenance and real-time optimization of production lines, reducing downtime and improving overall output quality.</li></ul><p>By focusing on use cases specific to their domain, businesses are unlocking new efficiencies and revenue streams, making GenAI an essential tool for competitiveness in today’s market.</p><h3>Specialized AI Models for Niche Applications</h3><p>Azure’s AI Model Catalog now includes industry-specific solutions:</p><p><strong>Examples</strong>:</p><ul><li><strong>Healthcare: </strong>AI models for diagnostics and patient care.</li><li><strong>Agriculture:</strong> Predictive models for yield optimization.</li><li><strong>Manufacturing:</strong> Anomaly detection for operational efficiency.</li></ul><p>Businesses in these sectors can leverage pre-built models to accelerate innovation and reduce development costs.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3YiB2OvK6sY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3YiB2OvK6sY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3YiB2OvK6sY%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/57b9b6e319cfc3ec81a5c2205c205ccf/href">https://medium.com/media/57b9b6e319cfc3ec81a5c2205c205ccf/href</a></iframe><h3>Final Thoughts</h3><p>Microsoft’s latest AI innovations provide opportunities for businesses to scale operations, boost productivity, and drive innovation. Whether you’re looking to streamline workflows with Microsoft 365 Copilot or build scalable AI solutions with Azure AI, these tools can transform your business.</p><p>For a shorter version (2–3 min. read) of this article, see: <a href="https://bit.ly/DianaBaldMSIngite24S">https://bit.ly/DianaBaldMSIngite24S</a></p><p><strong><em>Ready to explore what’s possible with GenAI and other advancements noted above?</em></strong><em> </em><a href="https://www.linkedin.com/in/dianabald/"><em>Please reach out</em></a><em> and let’s discuss how we can tailor these innovations to your unique needs. At </em><a href="https://blueorange.digital/"><strong><em>Blue Orange Digital</em></strong></a><em>, our experts are passionate about helping businesses unlock the full potential of reliable, fact-based AI solutions.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=298427c53f15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/insights-and-innovations-from-microsoft-ignite-2024-298427c53f15">Insights and Innovations from Microsoft Ignite 2024</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>