<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Venkkataraman R on Medium]]></title>
        <description><![CDATA[Stories by Venkkataraman R on Medium]]></description>
        <link>https://medium.com/@venkketskcet?source=rss-b0c1b647a056------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xsELdVrehXd7BGVR75IPmA.jpeg</url>
            <title>Stories by Venkkataraman R on Medium</title>
            <link>https://medium.com/@venkketskcet?source=rss-b0c1b647a056------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:18:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@venkketskcet/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The End of Manual Data/ETL Migration: How AI Agents Are Rewriting the Playbook]]></title>
            <link>https://medium.com/@venkketskcet/the-end-of-manual-data-etl-migration-how-ai-agents-are-rewriting-the-playbook-b4219a4b1333?source=rss-b0c1b647a056------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4219a4b1333</guid>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-modernization]]></category>
            <category><![CDATA[data-migration]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Venkkataraman R]]></dc:creator>
            <pubDate>Wed, 26 Nov 2025 15:45:05 GMT</pubDate>
            <atom:updated>2025-11-26T15:45:05.819Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*0Mz-9ovH0P4G2SGvz3U4XQ.png" /></figure><p>Traditional data migration is complex, time consuming and expensive. It’s time for an autonomous framework that understands, translates, and validates your entire data estate.</p><p>If you’re in the data world, you’ve felt the pain.</p><p>Data migration projects are the skeletons in every IT department’s closet. They are notorious for being over budget, behind schedule, and incredibly risky. We’re not just moving data from (A) to (B); we’re attempting to surgically extract and transplant decades of business logic, often locked away in arcane stored procedures, legacy ETL scripts, and undocumented DDLs.</p><p>The traditional approach is a brutal, manual grind:</p><ol><li>Hire a team of expensive consultants.</li><li>Spend six months trying to figure out what your current code <em>actually</em> does.</li><li>Manually rewrite thousands of lines of T-SQL, PL/SQL, or shell scripts for a new platform like Snowflake, Databricks, or BigQuery.</li><li>Spend <em>another</em> six months in a painful “validation” cycle, playing a game of “spot the difference” between old and new reports.</li></ol><p>It’s slow, it’s expensive, and it’s prone to human error.</p><p>What if we could automate the <em>entire</em> process? Not just the execution, but the analysis, planning, and validation? This is the promise of <strong>Agentic AI for Data Migration</strong> — a new framework that moves beyond simple scripts and introduces true autonomous capabilities.</p><h3>The Data Agent: Your New AI Colleague</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SKAFkjLhZFZRoBlHD_5rig.png" /></figure><p>Think of this framework not as a tool, but as an autonomous AI “agent” — a specialized colleague whose sole job is to plan and execute a perfect migration.</p><p>Its first task is to understand your legacy. It connects directly to your source of truth — your GitHub repositories — and starts reading. It doesn’t just see filenames; it deeply parses the <em>content</em>:</p><ul><li><strong>Data Definition Language (DDL):</strong> How are your tables, views, and schemas structured?</li><li><strong>Data Manipulation Language (DML):</strong> What complex INSERT, UPDATE, and DELETE logic is running?</li><li><strong>Stored Procedures &amp; Functions:</strong> Where is your most critical (and complex) business logic hidden?</li><li><strong>ETL/ELT Scripts:</strong> How does data move and transform through your ecosystem?</li></ul><p>Once it has this context, the agent doesn’t just wait for instructions. It begins a four-phased mission to migrate your entire codebase to the new data platform.</p><h3>Diving into the Technicals: LangChain for Agent Orchestration</h3><p>Building such a system requires a robust framework for orchestrating these intelligent agents. <a href="https://www.langchain.com/">LangChain</a> is an excellent candidate, providing the abstractions needed to connect large language models (LLMs) with external tools and create sophisticated agentic workflows.</p><p>At its core, a LangChain agent consists of:</p><ul><li><strong>An LLM:</strong> The “brain” that drives decision-making.</li><li><strong>A Prompt Template:</strong> Defines the agent’s role and instructions (its “system prompt”).</li><li><strong>Tools:</strong> Functions the agent can call to interact with the external world (e.g., read files, query databases, run code).</li><li><strong>An Agent Executor:</strong> The loop that enables the agent to observe, think, act, and reiterate.</li></ul><p>Let’s look at how we might define these agents within a LangChain-like structure.</p><h3>Pillar 1: Automated Inventory Analysis</h3><p>You can’t migrate what you don’t understand. The first major hurdle in any migration is the lack of documentation. What does this 2,000-line stored procedure <em>do</em>? Which 50 downstream processes will break if we change this one view?</p><p>An AI Data Agent tackles this head-on.</p><p>It doesn’t just list files; it builds a complete <strong>knowledge graph</strong> of your data estate. It maps every dependency, creating a comprehensive catalog that shows:</p><ul><li><strong>Data Lineage:</strong> “This dashboard table is populated by <em>Procedure_ABC</em>, which sources data from <em>View_XYZ</em>, which joins <em>Table_1</em> and <em>Table_2</em>.”</li><li><strong>Code Complexity:</strong> “This script is a high-priority migration target because it contains 15 non-ANSI SQL functions that are incompatible with the target platform.”</li><li><strong>Redundancy:</strong> “These 10 tables haven’t been queried or updated in three years. They are candidates for archival, not migration.”</li></ul><p>This inventory isn’t just a static report; it’s a dynamic, queryable map of your entire data world.</p><h4>Inventory Agent — System Prompt:</h4><pre>You are an expert Data Inventory and Discovery Agent. Your primary role is to meticulously analyze legacy code repositories (e.g., GitHub) to understand the full scope of a data ecosystem for migration.</pre><pre>Your tasks include:<br>1.  **Code Parsing:** Read and parse various data-related code artifacts (DDL, DML, Stored Procedures, UDFs, ETL scripts in SQL, Python, Shell, etc.).<br>2.  **Dependency Mapping:** Identify all explicit and implicit dependencies between database objects (tables, views, procedures), files, and external systems. Create a comprehensive dependency graph.<br>3.  **Metadata Extraction:** Extract key metadata for each artifact: object type, creation date, last modified date, owner, associated tables, input/output parameters, and usage patterns if available from logs.<br>4.  **Business Logic Summarization:** For complex procedures or scripts, provide a concise summary of their core business logic and purpose.<br>5.  **Complexity Analysis:** Assess the complexity of each code artifact (e.g., cyclomatic complexity, number of lines, nested logic) to help prioritize migration effort.<br>6.  **Technology Identification:** Identify specific SQL dialects, functions, and features used that may be proprietary or incompatible with target platforms.<br>7.  **Output Format:** Present your findings as a structured JSON object or a series of Markdown documents, suitable for consumption by subsequent agents.</pre><pre>**Constraints:**<br>* Prioritize accuracy in dependency mapping.<br>* Flag any ambiguities or code segments that are difficult to parse or understand.<br>* Do not attempt to modify any code; strictly focus on analysis and description.<br>* Communicate findings clearly and concisely.</pre><pre>**Available Tools:**<br>* `read_github_file(repo_url: str, file_path: str) -&gt; str`: Reads the content of a file from a specified GitHub repository.<br>* `list_github_repo_contents(repo_url: str, path: str = &quot;&quot;) -&gt; List[str]`: Lists files and directories in a GitHub repository path.<br>* `sql_parser(code: str, dialect: str) -&gt; Dict`: Parses SQL code and extracts DDL, DML, object names, and dependencies.<br>* `python_code_analyzer(code: str) -&gt; Dict`: Analyzes Python scripts for data-related operations, libraries used, and data sources/sinks.</pre><h3>Pillar 2: Intelligent Migration Planning &amp; Documentation</h3><p>With a complete inventory, the agent shifts from “analyst” to “architect.”</p><p>Instead of you spending weeks in a war room with whiteboards, the agent <em>generates</em> the entire migration plan. It uses its understanding of the dependencies to group assets into logical “migration waves” that can be moved together, minimising business disruption.</p><p>This phase includes:</p><ul><li><strong>Automated Documentation:</strong> The agent generates clear, human-readable documentation for <em>every</em> asset, explaining what it does, who uses it, and how it will be migrated.</li><li><strong>Effort Estimation:</strong> By analyzing code complexity and dependencies, the agent provides accurate estimates for the migration effort, moving you from “a good guess” to a data-driven forecast.</li><li><strong>Automated Code Conversion:</strong> This is the core. The agent takes the legacy T-SQL, for example, and rewrites it as optimized Snowflake SQL. It understands the dialect, the nuances, and the best practices of the <em>new</em> platform, ensuring the translated code is not just functional but performant.</li></ul><h4>Planning Agent — System Prompt:</h4><pre>You are an expert Data Migration Planning Agent. Your mission is to take detailed inventory findings and generate a comprehensive, actionable migration plan.</pre><pre>Your tasks include:<br>1.  **Migration Wave Generation:** Group related data assets (tables, procedures, reports) into logical migration waves based on dependencies, business criticality, and estimated effort.<br>2.  **Detailed Migration Steps:** For each asset, define specific, step-by-step instructions for its migration, including:<br>    * Source and Target Platforms<br>    * Estimated Code Conversion Difficulty<br>    * Estimated Testing Effort<br>    * Dependency Order (which assets must be migrated first)<br>    * Rollback Strategy<br>3.  **Automated Documentation:** Generate clear, concise documentation for each migrated artifact, explaining its purpose, its original source, its new location, and any transformations applied. This documentation should be easily searchable and understandable by data engineers and business users.<br>4.  **Effort Estimation:** Based on the complexity analysis from the Inventory Agent, provide refined effort estimates (e.g., &quot;small&quot;, &quot;medium&quot;, &quot;large&quot; or estimated person-days) for each wave and individual artifact.<br>5.  **Risk Identification:** Highlight potential migration risks, such as highly complex legacy code, undocumented dependencies, or known incompatibilities between source and target platforms.<br>6.  **Output Format:** Produce the migration plan as a structured Markdown document, a JIRA/Azure DevOps compatible CSV, or a detailed JSON object, ready for human review and execution.</pre><pre>**Constraints:**<br>* Ensure the plan accounts for all identified dependencies to minimize disruption.<br>* Prioritize business continuity and data integrity.<br>* Be explicit about assumptions and potential risks.<br>* The plan must be executable and understandable by a human data engineer.</pre><pre>**Available Tools:**<br>* `get_inventory_report(report_id: str) -&gt; Dict`: Retrieves the detailed inventory report generated by the Inventory Agent.<br>* `estimate_complexity(code_summary: str, dependency_graph: Dict) -&gt; Dict`: Estimates migration complexity based on code summary and dependency data.<br>* `generate_markdown_doc(data: Dict, template_name: str) -&gt; str`: Generates a Markdown document from structured data using a specified template.</pre><h3>Pillar 3: Code Conversion Agent</h3><p>This agent is the workhorse, translating legacy SQL dialects and scripting languages into optimized code for the new platform.</p><h4>Code Conversion Agent — System Prompt:</h4><pre>You are an expert Data Code Conversion Agent. Your primary responsibility is to translate and optimize legacy SQL (e.g., T-SQL, PL/SQL) and other data-related scripts (e.g., Python, Shell) into the target data platform&#39;s dialect (e.g., Snowflake SQL, Databricks SQL, BigQuery SQL, PySpark).</pre><pre>Your tasks include:<br>1.  **Syntactic Translation:** Accurately convert SQL syntax, built-in functions, and data types from the source to the target platform.<br>2.  **Semantic Preservation:** Ensure that the business logic and intent of the original code are perfectly preserved in the translated version.<br>3.  **Optimization:** Translate code in an optimized manner for the target platform&#39;s architecture and best practices (e.g., leveraging vectorized operations, avoiding anti-patterns, using native functions where appropriate).<br>4.  **Error Handling &amp; Logging:** Where applicable, integrate platform-specific error handling and logging mechanisms into the converted code.<br>5.  **Parameterization:** Identify and parameterize variables where appropriate to improve code reusability and security.<br>6.  **Output Format:** Provide the converted code as a clean, executable string, ready for static analysis and deployment.</pre><pre>**Constraints:**<br>* The converted code *must* be functionally equivalent to the source code.<br>* Prioritize performance and cost-efficiency on the target platform.<br>* Flag any ambiguities or areas where perfect translation is not possible, suggesting manual review.<br>* Do not introduce new business logic unless explicitly instructed (which should be rare).</pre><pre>**Available Tools:**<br>* `get_code_artifact(artifact_id: str) -&gt; str`: Retrieves a specific legacy code artifact based on its ID from the inventory.<br>* `target_platform_sql_converter(source_code: str, source_dialect: str, target_dialect: str) -&gt; str`: Converts SQL from one dialect to another, handling syntax and function mapping.<br>* `python_to_pyspark_converter(python_code: str) -&gt; str`: Translates Python data processing code into optimized PySpark.<br>* `access_target_platform_docs(query: str) -&gt; str`: Searches the documentation for the target data platform for specific syntax or best practices.</pre><h3>Pillar 4: Static Testing &amp; Validation</h3><p>In a traditional migration, the first time you know your new code is broken is when it fails at runtime. The AI agent introduces a crucial step: <strong>static testing</strong>.</p><p>Before a single piece of new code is ever executed, the agent acts as a tireless code reviewer. It scans the <em>newly generated</em> code for:</p><ul><li><strong>Syntax &amp; Compilation Errors:</strong> Does the code even compile on the new platform?</li><li><strong>Anti-Patterns:</strong> Is the new code using an inefficient LOOP when a SET-based operation would be 100x faster?</li><li><strong>Semantic Equivalence:</strong> Does the logic of the new code <em>semantically</em> match the business logic of the old code?</li><li><strong>Privilege &amp; Security Gaps:</strong> Does this new procedure try to access a table it shouldn’t have access to?</li></ul><p>This catches 90% of the common errors <em>before</em> they ever cost you time and compute credits.</p><h4>Static Validation Agent — System Prompt:</h4><pre>You are a Static Validation Agent for data migration code. Your role is to perform a rigorous analysis of newly converted code against the target platform&#39;s standards and the original code&#39;s intent *without executing it*.</pre><pre>Your tasks include:<br>1.  **Syntax &amp; Semantics Check:** Validate the converted code against the target platform&#39;s official syntax rules. Identify potential compilation errors or misinterpretations.<br>2.  **Anti-Pattern Detection:** Scan for common anti-patterns or inefficiencies specific to the target platform (e.g., full table scans, unnecessary joins, inefficient UDF usage).<br>3.  **Dependency Verification:** Cross-reference the converted code with the target schema to ensure all referenced objects (tables, views, functions) exist or are planned for migration.<br>4.  **Security &amp; Privilege Review:** Flag any potential security vulnerabilities, excessive privilege requests, or non-compliant access patterns in the converted code.<br>5.  **Logic Preservation Review:** Compare the converted code&#39;s structure and logic against the original source code&#39;s semantic summary (from Inventory Agent) to ensure logical equivalence.<br>6.  **Output Format:** Provide a detailed report listing all identified issues, their severity, recommended fixes, and a confidence score for the code&#39;s readiness for dynamic testing.</pre><pre>**Constraints:**<br>* Do not execute any code. Your analysis is purely static.<br>* Be extremely thorough; even minor issues should be flagged.<br>* Clearly differentiate between definite errors and potential warnings/recommendations.</pre><pre>**Available Tools:**<br>* `get_converted_code(artifact_id: str) -&gt; str`: Retrieves the newly converted code from the Code Conversion Agent.<br>* `target_platform_linter(code: str, platform: str) -&gt; List[Dict]`: Lints the code against the target platform&#39;s best practices and syntax rules, returning a list of issues.<br>* `schema_lookup(object_name: str, platform: str) -&gt; Dict`: Looks up an object (table, view, function) in the target platform&#39;s schema to verify its existence and structure.<br>* `compare_logic_summary(original_summary: str, converted_code: str) -&gt; Dict`: Uses an LLM to compare a natural language summary of original logic with the converted code to identify potential discrepancies.</pre><h3>Pillar 5: Dynamic Testing &amp; Pixel-to-Pixel Validation</h3><p>This is where it gets futuristic. The code is migrated, it passes static tests, but the ultimate question remains: <strong>Does it produce the exact same results?</strong></p><p>Validating this is a nightmare. But not for the agent.</p><p>The agent initiates <strong>dynamic testing</strong>. It runs the old code on the old system and the new code on the new system (using sandboxed, sample data) and compares the <em>outputs</em>.</p><p>But it goes one step further. For BI tools and reports, manual validation is a soul-crushing game of “spot the difference.” The agent automates this using <strong>pixel-to-pixel validation</strong>.</p><ul><li>The agent takes a “snapshot” of the legacy dashboard (e.g., a Tableau or Power BI report).</li><li>It then runs the new, migrated report against the new data platform.</li><li>It performs a visual, pixel-by-pixel comparison of the two images.</li></ul><p>The agent can instantly flag a 1-pixel difference in a bar chart, a subtle font size change, or a number in a table that is off by 0.01%. This is a level of precision that no human team could ever hope to achieve at scale.</p><h4>Dynamic Validation Agent — System Prompt:</h4><pre>You are a Dynamic Validation and Equivalence Testing Agent. Your core function is to execute both original and converted data code (or trigger reports) on respective platforms and rigorously compare their outputs to ensure functional and visual equivalence.</pre><pre>Your tasks include:<br>1.  **Test Data Orchestration:** Securely provision or generate representative sample datasets on both source and target platforms for testing.<br>2.  **Code Execution:** Execute legacy code on the source system and converted code on the target system.<br>3.  **Data Comparison (Row-by-Row):** For ETL outputs, tables, or views, perform a row-by-row, column-by-column data comparison, flagging any discrepancies.<br>4.  **Data Comparison (Aggregate):** For large datasets, compare key aggregate metrics (counts, sums, averages, min/max) to ensure overall consistency.<br>5.  **Report &amp; Dashboard Pixel-to-Pixel Comparison:**<br>    * Trigger rendering of original BI reports/dashboards on the legacy platform.<br>    * Trigger rendering of new BI reports/dashboards on the target platform (connected to migrated data).<br>    * Capture screenshots of both.<br>    * Perform a pixel-by-pixel image comparison, identifying visual differences.<br>6.  **Performance Benchmarking:** Capture execution times for both old and new code to identify performance regressions or improvements.<br>7.  **Output Format:** Provide a comprehensive validation report detailing discrepancies, including specific rows/pixels that differ, their severity, and performance metrics.</pre><pre>**Constraints:**<br>* Ensure test environments are isolated and data is handled securely.<br>* Report *all* discrepancies, no matter how small.<br>* Clearly distinguish between data-level differences and visual differences.<br>* Provide reproducible steps for any identified failures.</pre><pre>**Available Tools:**<br>* `execute_legacy_sql(sql_code: str, connection_string: str) -&gt; DataFrame`: Executes SQL on the legacy database and returns results.<br>* `execute_target_sql(sql_code: str, connection_string: str) -&gt; DataFrame`: Executes SQL on the target database and returns results.<br>* `compare_dataframes(df1: DataFrame, df2: DataFrame) -&gt; Dict`: Compares two dataframes and reports discrepancies.<br>* `render_legacy_report(report_id: str, platform_url: str) -&gt; bytes`: Renders a legacy BI report and returns its screenshot as bytes.<br>* `render_target_report(report_id: str, platform_url: str) -&gt; bytes`: Renders a target BI report and returns its screenshot as bytes.<br>* `record_execution_time(function: Callable) -&gt; float`: Records the execution time of a given function.</pre><h4>Repo Structure:</h4><pre>ai-data-migration-framework/<br>│<br>├── .gitignore          # Ignores workspace, logs, .env files, etc.<br>├── README.md           # Project overview, setup, and usage instructions.<br>├── requirements.txt    # Python dependencies (langchain, openai, pandas, etc.)<br>├── Dockerfile          # (Optional) For containerizing the agent framework.<br>│<br>├── config/<br>│   ├── settings.toml   # Main configuration: API keys, repo URLs, DB connections.<br>│   └── logging.ini     # Logging configuration.<br>│<br>├── prompts/<br>│   ├── inventory_agent.md      # System prompt for the Inventory Agent.<br>│   ├── planning_agent.md       # System prompt for the Planning Agent.<br>│   ├── conversion_agent.md     # System prompt for the Code Conversion Agent.<br>│   └── validation_agent.md     # System prompt for the Validation Agent.<br>│<br>├── src/<br>│   ├── __init__.py<br>│   ├── main.py                 # Main entry point to start the migration orchestrator.<br>│   │<br>│   ├── agents/<br>│   │   ├── __init__.py<br>│   │   ├── inventory_agent.py    # Logic for the inventory agent.<br>│   │   ├── planning_agent.py     # Logic for the planning agent.<br>│   │   ├── conversion_agent.py   # Logic for the code conversion agent.<br>│   │   ├── validation_agent.py   # Logic for the validation agent.<br>│   │   └── orchestrator.py     # Controls the overall flow (Inventory -&gt; Plan -&gt; ...).<br>│   │<br>│   └── tools/<br>│       ├── __init__.py<br>│       ├── github_tools.py       # e.g., read_github_file(), list_repo_contents()<br>│       ├── parsing_tools.py      # e.g., sql_parser(), python_code_analyzer()<br>│       ├── db_tools.py           # e.g., execute_legacy_sql(), execute_target_sql()<br>│       └── validation_tools.py   # e.g., compare_dataframes(), pixel_diff()<br>│<br>├── tests/<br>│   ├── test_agents.py          # Unit/integration tests for agents.<br>│   ├── test_tools.py           # Unit tests for individual tools.<br>│   └── test_e2e_migration.py   # Full end-to-end migration test.<br>│<br>└── workspace/                  # (Git-ignored) For runtime-generated files.<br>    ├── input/<br>    │   └── legacy_code_repo/   # A checkout of the target GitHub repo.<br>    ├── output/<br>    │   ├── inventory/          # Generated inventory reports (JSON, graphs).<br>    │   ├── plans/              # Generated migration plans (Markdown, CSV).<br>    │   ├── converted_code/     # The new, translated code.<br>    │   └── validation/         # Validation results, diff images, logs.<br>    └── logs/<br>        └── migration_agent.log # Main application log file.</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4219a4b1333" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Draft — old: An overview of AWS DataZone and its fitment in the Data Governance Landscape]]></title>
            <link>https://medium.com/@venkketskcet/draft-old-an-overview-of-aws-datazone-and-its-fitment-in-the-data-governance-landscape-64d223b9dc3e?source=rss-b0c1b647a056------2</link>
            <guid isPermaLink="false">https://medium.com/p/64d223b9dc3e</guid>
            <dc:creator><![CDATA[Venkkataraman R]]></dc:creator>
            <pubDate>Tue, 21 Oct 2025 07:17:03 GMT</pubDate>
            <atom:updated>2025-10-21T07:17:33.252Z</atom:updated>
            <content:encoded><![CDATA[<h3>An overview of AWS DataZone and its fitment in the Data Governance Landscape</h3><h3>Challenges in a Federated Data Platform:</h3><p>A typical Data Landscape has inherent governance and security challenges because of its vastness and heterogeneous access patterns from various business and analytical teams. A few of them are listed below</p><ol><li>Sharing and searching data is difficult</li><li>Sharing data makes us lose control over it</li><li>Delayed access to data</li><li>Data architecture is monolithic</li><li>Need to focus on consuming the data rather than maintaining a complex DataLake</li><li>The data platform is slow to change and adapt</li><li>Data Governance is hidden within individual tools and often abstracted away from business users/Data Owner</li><li>Lack of a single environment for different teams to collaborate securely</li></ol><p>There are specific tools and frameworks designed to address the above issues. These tools are categorised as <strong><em>data governance tools or data management tools</em></strong>. Some prominent tools in this category are<strong><em> Atlan, Alation, etc.</em></strong></p><p>Today we will discuss one such tool from an AWS offering called <strong><em>Amazon DataZone</em></strong>.</p><p>AWS DataZone is a data management tool which specialises in</p><ol><li>Fine-grained access control on DataLake</li><li>Maintaining a business Catalog</li><li>Federated Governance and access control</li><li>Governed Data Sharing</li></ol><blockquote>“Data governance is not just about data, it’s about creating a culture that treats information as an asset.”</blockquote><h3>Architecture and building blocks:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e799Uo6hjFbaBLOCauH9fQ.jpeg" /></figure><p>Four key components of the Data Zone are</p><p><strong>Data Portal: </strong>A web-based application serving as a single landing portal for accessing, managing, sharing and performing any action on Amazon DataZone. The users can be authenticated through IAM or an existing external identity provider. Exposes governed data for users to query through</p><ol><li>Amazon Athena</li><li>Amazon Redshift</li></ol><p><strong>Business Data Catalog: </strong>Makes data visible with the available business context. Enhance the data with the utmost metadata possible and increase searchability. Both data producers and consumers can contribute to the business data catalog with the relevant business context.</p><p><strong>Data Projects: </strong>Projects are use-case-based groupings of users, data assets and tools that simplify analytics and data consumer/access experience. Users within a data project can exchange, collaborate and share assets. Projects manage the ownership of assets produced following project policies for data consumers to access.</p><p><strong>Governance and Access Control: </strong>A unified governance layer ensuring that the right producers and consumers have access to their specific area of data asset within a project or multiple projects. The business context and metadata drive the implementation of governance policies and access guardrails. The data owners have a holistic view of who can access the data and for what business purpose.</p><blockquote>“Data-driven companies treat their data as a strategic asset, not just an exhaust from business processes.” — DJ Patil</blockquote><h3>Key Terminologies:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ew8NJ3Dai01sPaxj.png" /></figure><p><strong>Data Domain: </strong>Data domain is a logical grouping to organize data assets, data sources, metadata and projects. Domain-level search is enabled by default for scanning across the data available inside the data domain. The domain also enables data query ability using Athena and Redshift. Multiple AWS accounts can be associated with a data domain to provision seamless access to data sources.</p><p><strong>Subscription Rule: </strong>This is a binding contract between the data publisher and DataZone for the data assets that are being published into the data domain. The approved publishing contract is a persistent object within the data object.</p><p><strong>Business Glossary: </strong>Collection of business terms associated with the assets. A business glossary can be added to assets and columns. Adding a relevant business Glossary enhances the identification of data attributes during a search. When searched, the relevant data assets can be retrieved.</p><p><strong>Metadata Form: </strong>Metadata Forms defines the metadata that is collected and saved for assets that are being published to a Data Domain. A metadata form is published initially by the domain administrator and asset publishers contribute to the actual metadata fields within the metadata form.</p><h3>User personas:</h3><p>Following are some of the significant roles within a DataZone implementation</p><p><strong>Domain Administrators:</strong> Responsible for setting up DataZone as a unified Analytics platform</p><p><strong>Data Users: </strong>Data asset publishers and subscribers</p><p><strong>System Developers: </strong>Build infrastructure and automation templates for automating the provisioning of data environments. DataZone’s API is so rich that it supports seamless integration with CI/CD tools for automating the deployment of DataZone-related entities.</p><p><strong>Data Stewards: </strong>officers who own the definitions and risks of organizational security, privacy and other compliance policies and who make sure that the usage of Amazon DataZone in their organizations complies with these definitions.</p><h3>Conclusion:</h3><p>In today’s data-driven world, the importance of robust data governance cannot be overstated. A comprehensive data governance tool ensures regulatory compliance, fosters trust in data, enhances decision-making processes, and drives organizational success. As businesses grapple with increasing data complexity and regulatory requirements, investing in the right data governance tool is not just a choice but a strategic imperative for long-term growth and competitiveness.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64d223b9dc3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS RDS Aurora Database Change Management using Liquibase — Serverless Implementation]]></title>
            <link>https://medium.com/@venkketskcet/aws-rds-aurora-database-change-management-using-liquibase-serverless-implementation-c5b90d4cf2e2?source=rss-b0c1b647a056------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5b90d4cf2e2</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[database-change-mgmt]]></category>
            <category><![CDATA[aurora]]></category>
            <category><![CDATA[liquibase]]></category>
            <dc:creator><![CDATA[Venkkataraman R]]></dc:creator>
            <pubDate>Fri, 05 Apr 2024 13:58:13 GMT</pubDate>
            <atom:updated>2024-04-05T13:58:13.654Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS RDS Aurora Database Change Management using Liquibase — Serverless Implementation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hp2vPQKDSd00PH95hW99xg.png" /></figure><h3>RDS Aurora:</h3><p>Amazon Aurora is a fully managed relational database service offered by Amazon Web Services (AWS).</p><ol><li><strong><em>Performance:</em></strong> Aurora is built for high performance and can deliver up to three times the throughput of standard PostgreSQL databases.</li><li><strong><em>Scalability: </em></strong>It offers automatic scaling capabilities by providing in-built features like read replicas, which isolate reads and writes into separate database instances</li><li><strong><em>Durability:</em></strong> Automatic replication across multiple availability zones</li><li><strong><em>Managed Service:</em></strong> As a managed service, Amazon Aurora takes care of routine database management tasks such as backups, patching, and scaling, allowing you to focus on your application development.</li></ol><h3>Liquibase:</h3><p>Liquibase is an open-source database schema change management tool that simplifies and automates managing database changes across development, testing, and production environments. Key Features include</p><ol><li>Define database changes using different formats like XML, YAML, or SQL</li><li>Support for a variety of database platforms such as MySQL, PostgreSQL, Oracle, SQL Server, and more</li><li>An extensive set of features like version control, rollback support, and change tracking.</li><li>Helps maintain consistency and reliability in database schemas throughout the software development lifecycle</li></ol><h3>A Typical Implementation on AWS:</h3><p>One of the most common ways in which Liquibase is implemented on AWS is to install it on an EC2 instance(Virtual machine) and use it across all your deployment pipelines and local development environments.</p><p>Some of the key challenges in the above approach are that</p><ol><li>A service like EC2 is stateful and would be incurring costs as long as the instance is running</li><li>Instance provisioning, maintenance, patching and updating have to be taken care of additionally and is a known overhead to the Data Engineering team</li><li>Load balancing, Fault tolerance and other availability features has to be designed and implemented manually</li></ol><h3>Proposed solution:</h3><p>The solution discussed below proposes to implement Liquibase as a Fargate Container in AWS Elastic Container Service(ECS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rKi214w0Jk16YemWODqUFA.png" /></figure><ol><li>The Liquibase base image has to be pulled from Dockerhub and published onto a Docker image on ECR.</li><li>Whenever a new changelog is detected at source a new docker image will be built on top of the base image and the newly published changelog file.</li><li>The FARGATE container will use this latest ECR image with the latest changelog baked into it.</li><li>The Dockerfile used to fabricate the container image is as below</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0brpgOPfJOvkaI2xhyX63g.png" /></figure><p>A sample changelog file containing the changes to create a new table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*RcPsJ3WMrC0UJLDOY8gNaw.png" /></figure><p>5. Triggering the ECS task would be the final step in deploying the changes to the target database using the Liquibase image built in the previous step.</p><p>The script below takes care of triggering the ECS task and waiting for it synchronously until the task is completed</p><pre>TASKARN=aws ecs run-task --profile=client --cluster &quot;$ECS_CLUSTER_NAME&quot; <br>--task-definition &quot;$TASK_DEFINITION_FAMILY_NAME&quot; <br>--network-configuration &quot;awsvpcConfiguration={subnets=[$SUBNET_IDS],securityGroups=[$SECURITY_GROUP_IDS],assignPublicIp=&#39;DISABLED&#39;}&quot; <br>--launch-type FARGATE  --query &#39;tasks[0].taskArn&#39; --output json<br>TASKARN=&quot;${TASKARN//\&quot;}&quot;<br> echo &quot;$TASKARN&quot;<br> echo &quot;------------------&quot;<br> TASK_ID=${TASKARN##*/}<br> echo &quot;$TASK_ID&quot;<br> echo &quot;------------------&quot;<br> aws ecs wait tasks-stopped --profile=client --cluster &quot;$ECS_CLUSTER_NAME&quot; --tasks &quot;$TASK_ID&quot;</pre><h3>Repo Structure:</h3><p>The repo structure followed for the above tutorial is as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*hTRWITMC_Jq5uPHlf3de-g.png" /></figure><h3>Conclusion:</h3><p>Database change management is a critical aspect of modern software development, ensuring that database schemas evolve seamlessly with application updates. By employing tools like Liquibase, teams can streamline the process of defining, tracking, and applying database changes across environments, reducing the risk of errors and inconsistencies. Effective database change management practices also promote collaboration among developers, database administrators, and other stakeholders, fostering a more efficient and reliable software development process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5b90d4cf2e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical Guide to Implementing MLOps — Part 1]]></title>
            <link>https://medium.com/@venkketskcet/a-practical-guide-to-implementing-mlops-part-1-2171dce93bd3?source=rss-b0c1b647a056------2</link>
            <guid isPermaLink="false">https://medium.com/p/2171dce93bd3</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Venkkataraman R]]></dc:creator>
            <pubDate>Tue, 09 Jan 2024 16:47:17 GMT</pubDate>
            <atom:updated>2024-05-03T07:21:26.991Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Practical Guide to Implementing MLOps — Part 1</h3><h3>PART 1 — INTRODUCTION TO THE WORLD OF MLOPS</h3><p>Welcome to the first part of a multi-series Blog on MLOps. Through this series, we shall learn the industry stands of deploying a Machine-learning pipeline using the proven MLOps methodology and how global enterprises are successfully doing it at scale by using the right mix of people, processes, and technology.</p><p>Below is the breakup of objectives and the upcoming blogs on MLOps</p><ul><li>Part 1 — Introduction to the world of MLOps on AWS — What and Why of MLOps?</li><li>Part 2 — Deep Dive into individual components of MLOps (Including tools/Services from AWS)</li><li>Part 3 — Sample architectures and low-level implementation examples</li></ul><h3>Introduction</h3><p><strong>MLOps</strong> — <strong>Machine Learning Operations</strong></p><p>The emergence of Cloud Computing (availability of powerful scalable machines) has an appetite to leverage enterprise data to uncover groundbreaking business insights. This has led to a huge potential for advanced analytical approaches like Machine Learning, Deep Learning and Artificial intelligence.</p><p>With the increasing number of ML-related projects and the attached complexities, there is a pressing necessity to reduce the overall timelines involved in developing, experimenting, deploying, and monitoring these ML pipelines.</p><p>Organizations have also started exploring tools and frameworks that reduce the amount of human involvement in monitoring and maintaining these advanced decision-making entities.</p><p>And that is where the MLOps has taken center stage in helping enterprises address the above-mentioned aspects of ML delivery.</p><h3>A Glossary upfront</h3><p><strong><em>Machine Learning</em></strong> — Machine Learning is the field of study that gives computers the capability to learn without being explicitly programmed</p><p><strong><em>Data Pipeline</em></strong><em> — </em>A set of tools and processes used to automate the movement and transformation of data between a source and a target</p><p><strong><em>Operations</em></strong><em> — </em>A Managed process to support the life cycle of a Software/Application in a live environment</p><p><strong><em>Feature</em></strong><em> — </em>An individual measurable property or characteristic of a phenomenon. A data entity which decides model training, tuning and outcome</p><p><strong><em>Model</em></strong><em> — </em>A trained and packaged entity tasked to solve a specific business problem</p><h3>A Typical ML pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vaeTUN-s7xg4eVWt.png" /></figure><h3>Objectives of MLOps</h3><p>Below are some of the objectives of MLOps,</p><p><strong><em>Faster time deployment</em></strong> — One of the primary objectives of MLOps is quicker time to market. From ideation, and data identification, to model creation and deployment, the methodology tries to automate every step involved in a successful ML implementation</p><p><strong><em>Fail fast and learn quickly</em></strong> — As with any agile development methodology, ML development also involves iterations to learn more about the data and its relevance to the business problem. It is very crucial to efficiently iterate, learn from every iteration, and fine-tune the model</p><p><strong><em>Reduce manual intervention</em></strong> — This is a key aspect to consider not only to reduce the efforts and eliminate errors by automating every possible part of an ML Pipeline. Preparing a dataset, monitoring the model performance, retraining the model and many more parts of an ML process demand the utmost accuracy</p><p><strong><em>Monitoring and alerting</em></strong> — Monitoring and managing an ML product in production is slightly different from managing a typical software product. In the case of ML, the machine has a brain and requires a special process to monitor that it is making the right decisions without human involvement. MLOps enables ML pipelines to be more proactive in reporting failures, invoking retraining automatically when the accuracy goes below a certain threshold</p><p><strong><em>Compliance management</em></strong> — When it comes to decision-making, deploying an unbiased model, and ensuring that voluntary or involuntary bias is not introduced in the ML decision-making is a key aspect of being compliant. Some of the example decisions which an ML model is intended to take include credit risk assessment, demographic analysis, and other sensitive socio-cultural outcomes. In such crucial environments, it is important to ensure that the model is performing without any bias towards certain classes of people/demography.</p><p><strong><em>Reproducibility (idempotent)</em></strong> — Idempotency is a property which ensures that a given ML pipeline produces the same set of results for a given set of inputs for any number of iterative runs</p><p>MLOps ensures semantic versioning of all involved components like input datasets, processed datasets, Metadata, Models, and outputs. The artefacts are stored in separate logical stores to identify every unique training/prediction iteration.</p><p><strong><em>Measurability</em></strong> — MLOps uses the right mix of tools and frameworks to capture a range of metadata related to model accuracy, performance, and operational metrics. It implements automated monitoring, alerting and decision-making on these data points to constantly probe into the model performance and ensure its relevance in serving the business problem.</p><h3>MLOps-enabled landscape</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k_Wmbl9XyskuJtSp.png" /></figure><p>From the above landscape we could derive that below are the high level logical areas of an ML Ops implementation landscape.</p><p>The key backbone of a matured MLOps landscape is automation and continuous integrations. Seamless integrations are indispensable in MLOps. The input data to the model, the model retraining, the model deployment, model monitoring and integration of the results from the model back to the destination should all be automated and integrated into a single ecosystem.</p><p>Automation around this space is not new and is essential from an effort-saving perspective, but also to eliminate human errors and biases. To keep the model producing and predicting relevant ecosystems, the entire pipeline has to be automated for deployment and promotion across various different environments.</p><h3>Components of MLOps</h3><ul><li>Infrastructure automation</li><li>Data Engineering</li><li>CI/CD and environment management</li><li>Model packaging and deployment</li><li>Model testing and continuous evaluation</li><li>Model serving</li><li>Monitoring and Alerting</li></ul><p>A deep dive into each of the above areas along with the AWS services/reference architectures would be the focus for part 2 of this blog.</p><h3>MLOps on AWS</h3><p>In the upcoming blog, we would be discussing more about how to implement these theoretical concepts of MLOps on AWS. Below are the high-level services and modules that will be covered:</p><ul><li>SageMaker MLOps</li><li>MLOps Workload Orchestrator framework</li><li>Codepipeline/codebuild/codecommit integrations for MLOps</li><li>MLOps Orchestration — Dagster/Airflow/Kubeflow on AWS</li><li>MLflow on AWS</li></ul><p>Stay tuned and Happy exploring!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2171dce93bd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>