<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manojkumar Vadivel on Medium]]></title>
        <description><![CDATA[Stories by Manojkumar Vadivel on Medium]]></description>
        <link>https://medium.com/@manojkumar.vadivel?source=rss-4c8679babe18------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*u-ydHp3FLrjwpyN4Fgwj3w.png</url>
            <title>Stories by Manojkumar Vadivel on Medium</title>
            <link>https://medium.com/@manojkumar.vadivel?source=rss-4c8679babe18------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 14:21:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manojkumar.vadivel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building Agent Skills at Scale: Standardising Dependencies and Local Execution with uv]]></title>
            <link>https://medium.com/swlh/building-agent-skills-at-scale-standardising-dependencies-and-local-execution-with-uv-ebc0f9e60e43?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebc0f9e60e43</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[uv]]></category>
            <category><![CDATA[agent-skills]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 04:21:02 GMT</pubDate>
            <atom:updated>2026-09-08T04:21:02.530Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zTr7nOZBp-h0VblDaN4xfw.png" /></figure><p>Agent Skills are becoming a more common part of the development workflow.</p><p>As tools like GitHub Copilot, Gemini CLI, Claude Code, and others become more capable, developers will not only use agents to write code. They will also start creating reusable skills for common engineering tasks such as</p><ul><li>Code review</li><li>Test generation</li><li>API validation</li><li>Documentation checks</li><li>Release readiness</li><li>Infrastructure review</li><li>Security checks or troubleshooting</li><li>And More ..</li></ul><p>Agent Skills are easy to create. A skill may simply contain:</p><pre>skill-name/<br>├── SKILL.md  #tells the agent what the skill does and how to use it<br>├── scripts/  #deterministic executable logic<br>├── references/ #deeper documentation loaded when required<br>└── assets/ #templates, schemas, or static files</pre><p>But once multiple developers start building executable skills, a very practical engineering problem appears:</p><blockquote>How should every skill manage its own dependencies without affecting other skills or the developer’s machine?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7IVu9rdJs7b-B-WYP-p8g.png" /></figure><p>This becomes especially important for Python-based skills. One skill may need an AST or code-analysis library, another may depend on HTTP clients and schema-validation packages, and another may require SDKs for cloud or infrastructure tooling.</p><p>Over time, some of those dependencies will also need different versions, At that point, the question is no longer only,</p><p><strong>“How should we write a skill?”</strong></p><p>It becomes:</p><p><strong>“How do we make executable skills self-contained, predictable, and safe to run locally without changing the developer’s Python environment?”</strong></p><p>That is where dependency isolation and standardisation start to matter.</p><h3>Why shared Python environments become a problem</h3><p>The simplest approach is usually to install whatever the skill needs.</p><p>For example, a skill may contain a Python script and the setup instructions may simply say:</p><pre>pip install pydantic requests</pre><p>For one script, this may be fine.</p><p>But imagine a shared skills repository with many skills created by different developers. One skill may need pydantic version 1, another may already use version 2, and another may depend on a package that expects a different version again.</p><p>If all of those packages are installed into the same environment, every skill starts depending on the state of that environment.</p><p>This becomes even more important when the command is executed by an agent.</p><p>When a developer manually runs pip install, they normally know what environment they are changing. When Copilot runs a skill as part of a workflow, dependency installation can become an indirect side effect of using that skill.</p><p>For me, the cleaner model is simple:</p><p><strong>Each executable skill should own the dependencies required to run that skill.</strong></p><p>That keeps the skill independent from system Python and independent from other skills.</p><h3>Traditional virtual environments solve this, but the workflow is awkward for agents</h3><p>Python already has a standard way to isolate dependencies.</p><p>A normal workflow might look like this:</p><pre>python -m venv .venv<br>source .venv/bin/activate<br>pip install -r requirements.txt<br>python scripts/run.py</pre><p>This works well when a developer is managing the project directly.</p><p>But with Agent Skills, the caller is often a coding agent.</p><p>That means the agent now has to think about whether .venv exists, whether it needs to create it, whether the dependencies are already installed, whether it needs to activate the environment, and which Python executable it should use.</p><p>We could put all of that into SKILL.md, but I do not think that is a good use of the skill instructions.</p><p>SKILL.md should mainly tell the agent what the skill does, when to use it, what inputs are required, and what command to execute.</p><p>It should not become a long environment setup guide.</p><p>This is where uv fits nicely.</p><h3>Why uv fits the Agent Skill model well</h3><p>uv gives us a much simpler way to manage the Python runtime for each skill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4plOeT500snHgqtQAnKyUQ.png" /><figcaption>Skills with UV dependancy management</figcaption></figure><p>For a normal executable skill, the structure can look like this:</p><pre>skills/&lt;skill-name&gt;/<br>├── SKILL.md<br>├── pyproject.toml<br>├── uv.lock<br>├── scripts/<br>│   ├── run.sh<br>│   └── run.py<br>├── references/<br>└── assets/</pre><p>The dependencies belong to that skill’s pyproject.toml.</p><p>For example:</p><pre>[project]<br>name = &quot;api-validation&quot;<br>version = &quot;1.0.0&quot;<br>requires-python = &quot;&gt;=3.11,&lt;3.13&quot;<br>dependencies = [<br>    &quot;pydantic&gt;=2,&lt;3&quot;,<br>    &quot;requests&gt;=2,&lt;3&quot;<br>]</pre><p>The uv.lock file is committed so dependency resolution stays reproducible.</p><p>The .venv stays local and is not committed.</p><p>When the skill runs through uv, the environment can be created or reused automatically. There is no need for the developer or the agent to manually activate the virtual environment first.</p><p>That is the part I like most about this model.</p><p>The agent only needs to execute the skill. uv takes care of preparing the Python environment underneath.</p><h3>One skill, one dependency environment</h3><p>Once the runtime is isolated, the next rule becomes simple.</p><p><strong>One skill should have one dependency definition and one environment.</strong></p><p>If a skill contains several Python files, they do not need separate virtual environments.</p><p>For example, a release-check skill might contain modules for checking tests, validating configuration, inspecting version metadata, and preparing a final report. These are different operations, but they all belong to the same skill.</p><p>They should normally share the same:</p><pre>pyproject.toml<br>uv.lock<br>.venv</pre><p>This keeps dependency management predictable.</p><p>If another skill needs different dependencies, it gets its own environment.</p><h3>Give the agent one stable entry point</h3><p>Dependency isolation solves one problem, but the agent still needs a predictable way to execute the skill.</p><p>Suppose a skill contains several Python files. We could explain every file inside SKILL.md and tell the agent which script to use for each operation.</p><p>That works, but it creates too much coupling between the instructions and the implementation.</p><p>If the internal file structure changes later, the SKILL.md needs to change as well.</p><p>A cleaner approach is to expose one stable entry point:</p><pre>scripts/run.sh</pre><p>The agent interacts with that entry point, not with the internal Python files.</p><p>For example:</p><pre>bash scripts/run.sh list --project my-project<br>bash scripts/run.sh validate --project my-project<br>bash scripts/run.sh kill --project my-project</pre><p>The implementation behind those commands can change without changing how the skill is used.</p><p>That keeps the public interface simple for the agent and gives the developers more freedom to refactor the internals.</p><h3>What the entry point actually does</h3><p>The entry point does not need to be complicated.</p><p>A basic run.sh could look like this:</p><pre>#!/usr/bin/env bash<br>set -euo pipefail<br>SCRIPT_DIR=&quot;$(cd &quot;$(dirname &quot;${BASH_SOURCE[0]}&quot;)&quot; &amp;&amp; pwd)&quot;<br>SKILL_DIR=&quot;$(cd &quot;$SCRIPT_DIR/..&quot; &amp;&amp; pwd)&quot;<br>exec uv run \<br>  --project &quot;$SKILL_DIR&quot; \<br>  python &quot;$SCRIPT_DIR/run.py&quot; \<br>  &quot;$@&quot;</pre><p>There are a few useful things happening here.</p><p>First, the script finds its own location. That means it does not matter where the coding agent is currently running the command from.</p><p>Second, it tells uv exactly which skill project should be used.</p><p>Third, it keeps the agent away from the Python implementation details.</p><p>From the agent’s point of view, the command is simply:</p><pre>bash scripts/run.sh ...</pre><p>Everything else stays behind that interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mWBq9-5hj3hSEUx2WS_rIA.png" /></figure><h3>Standardization matters when more people start creating skills</h3><p>Using uv solves the dependency-isolation problem, but it does not automatically solve consistency.</p><p>If several developers are creating skills independently, one may use requirements.txt, another may use Poetry, another may install packages inside setup.sh, while someone else may call internal Python files directly from SKILL.md.</p><p>Every skill may work, but the repository becomes harder to maintain. This is where I think a standard scaffold becomes useful:</p><ol><li><strong>The first is a </strong><strong>sample-skill structure that developers can copy when creating a new skill:</strong></li></ol><pre>sample-skill/<br>├── SKILL.md<br>├── pyproject.toml<br>├── scripts/<br>│   ├── run.sh<br>│   └── run.py<br>├── references/<br>└── assets/</pre><p>New skills start from the same structure instead of inventing their own layout. The repository can also contain Copilot instructions telling the coding agent how new skills should be created: use uv, do not install into system Python, commit uv.lock, do not commit .venv, and use run.sh as the public entry point for normal executable skills.</p><p>This becomes especially useful when coding agents themselves are helping developers create new skills. The agent building the skill should follow the same engineering standards as the developers reviewing it.</p><p><strong>2. The second is repository-level instructions for the coding agent.</strong></p><p>For GitHub Copilot, that could be something like:</p><pre>.github/<br>└── instructions/<br>    └── skill-development.instructions.md</pre><p>Those instructions can define the rules that should be followed whenever a new skill is created or an existing skill is modified.</p><p>For example:</p><pre>Start from sample-skill.<br>Each executable skill owns its dependencies.<br>Use uv for Python execution.<br>Do not use global pip install.<br>Do not require manual .venv activation.<br>Use scripts/run.sh as the public entry point.<br>Commit uv.lock.<br>Do not commit .venv.<br>Document external network requirements.<br>Add safety checks for destructive operations.</pre><p>This is useful because developers are increasingly asking the coding agent itself to create new skills.</p><p>If the agent is doing some of the implementation work, it should also follow the same engineering standard.</p><p>The combination becomes simple:</p><blockquote><strong>Open Agent Skills format for portability, a sample skill for structure, repository instructions for consistency, and </strong><strong>uv for dependency isolation.</strong></blockquote><p>Standardisation does not mean every skill has to use the same amount of structure. A simple instruction-only skill does not need Python at all, It only nead:</p><pre>SKILL.md<br>references/</pre><p>There is no reason to add Python files or a virtual environment if the skill does not execute code.</p><p>A very small standalone Python skill may also be simpler with PEP 723 inline dependencies:</p><pre># /// script<br># requires-python = &quot;&gt;=3.11&quot;<br># dependencies = [<br>#   &quot;requests&gt;=2,&lt;3&quot;,<br># ]<br># ///</pre><p>That can be executed with:</p><pre>uv run --script scripts/run.py</pre><p>For normal executable skills, I would use pyproject.toml, uv.lock, and a stable run.sh entry point.</p><p>If a skill grows into a larger Python application, it can move towards a normal package structure with src/ and tests while keeping the same public entry point.</p><p>A simple team standard could therefore look like this:</p><pre>Skill type                              Suggested approach<br>----------------------------------------------------------------<br>Instruction-only              SKILL.md + references<br>Small standalone Python       skill PEP 723 + uv run --script<br>Normal executable Python      skill pyproject.toml + uv.lock + run.sh<br>Larger Python                 skill Package structure + same public entry point<br>Higher-risk or OS-dependent   skill Container or stronger sandbox</pre><p>The important thing is not to make every skill identical. It is to make the supported patterns clear.</p><h3>One important boundary: uv is not a full sandbox</h3><p>There is one distinction worth keeping clear.</p><p>A virtual environment isolates Python dependencies. It does not isolate the operating system.</p><p>A Python script running inside a skill-specific .venv may still access files, make network calls, execute subprocesses, or use credentials available on the developer machine.</p><p>So uv helps with <strong>dependency isolation</strong>, not complete security isolation.</p><p>For trusted internal skills running during development, that may be enough for many cases.</p><p>For higher-risk skills, such as those that execute untrusted code, perform destructive operations, or require stronger filesystem and network controls, a container or another sandboxed runtime may be more appropriate.</p><p>That is a separate problem, and I would not try to solve it by forcing every skill into Docker from day one.</p><h3>The standard I would keep</h3><p>The practical rule I would keep for executable skills is:</p><p><strong>One skill, one dependency definition, one isolated environment, and one stable entry point.</strong></p><p>SKILL.md explains the capability to the agent.</p><p>pyproject.toml and uv.lock define the runtime.</p><p>uv manages the isolated environment.</p><p>run.sh gives the agent a predictable execution interface.</p><p>And sample-skill plus repository instructions help make sure the next skill follows the same pattern.</p><p>None of these ideas are new by themselves. What is changing is where we are applying them.</p><p>As teams start creating more Agent Skills, I think these small engineering decisions will matter much more than they do when there are only one or two skills in a repository.</p><p>I have captured this pattern in a working skills repository with a sample scaffold, uv dependency isolation, and skill development instructions.</p><p>Repository: <a href="https://github.com/manojvsj/agent-skill-python-uv-template/">https://github.com/manojvsj/agent-skill-python-uv-template/</a></p><p>PS if you have any questions, or would like something clarified, you can find me on <a href="https://www.linkedin.com/in/manojvsj/"><strong>LinkedIn</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebc0f9e60e43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/building-agent-skills-at-scale-standardising-dependencies-and-local-execution-with-uv-ebc0f9e60e43">Building Agent Skills at Scale: Standardising Dependencies and Local Execution with uv</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why MCP Servers Matter for Data Engineering: Connecting AI Agents to Modern Data Platforms]]></title>
            <link>https://medium.com/@manojkumar.vadivel/why-mcp-servers-matter-for-data-engineering-connecting-ai-agents-to-modern-data-platforms-eb3e48a0994f?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb3e48a0994f</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 03:31:00 GMT</pubDate>
            <atom:updated>2026-03-09T03:31:00.664Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p-IeAi0CSuongd5HI__GPg.png" /></figure><p>In the last year, most conversations around AI in data engineering have focused on <strong>LLMs generating SQL, writing Python scripts, or explaining pipelines</strong>. That is useful, but it still keeps AI mostly as a <strong>passive assistant</strong>. It helps engineers write code, but the engineer still executes everything manually.</p><p>The next step in this evolution is different.</p><p>Instead of only generating code, AI agents can <strong>directly interact with the tools that run our data platforms</strong>.</p><p>That is where <strong>Model Context Protocol (MCP)</strong> comes in.</p><p>MCP allows AI agents to safely connect with external systems like <strong>BigQuery, Airflow, GitHub, and cloud storage</strong>, execute operations, retrieve results, and reason about them.</p><p>For data engineering teams, this opens a new way to operate data platforms. Routine tasks like checking pipelines, running queries, or inspecting logs can be done through AI agents.</p><p>In this article we will explore how MCP works, how it differs from traditional APIs, and how data engineering teams can use it to improve daily workflows across warehouses, orchestration tools, and developer platforms.</p><h3>Understanding MCP in Simple Terms</h3><p>In a traditional system, applications interact through <strong>APIs</strong>. A service sends a request to a specific endpoint, receives a response, and processes the result.</p><p>For example, a typical API might expose endpoints like this:</p><pre>GET /repos<br>POST /issues<br>GET /users/{id}</pre><p>In this model the application developer decides exactly which endpoint to call and what parameters should be sent. The application logic controls everything. APIs are well suited for machine to machine communication</p><p>MCP introduces a slightly different model designed specifically for <strong>AI agents</strong>.</p><p>Instead of exposing raw HTTP endpoints, a system exposes <strong>tools</strong> that an AI agent can use. Each tool includes a description and a schema that explains what the tool does and what parameters it expects.</p><p>For example, a tool exposed by a warehouse service might look like this:</p><pre>Tool: run_bigquery_query</pre><pre>Description:<br>Execute a SQL query in BigQuery and return results</pre><pre>Parameters:<br>query (string)</pre><p>When an AI agent receives a request such as:</p><p>“Show failed orders from yesterday”</p><p>the agent evaluates the request and determines which available tool can help answer it. In this case it may decide to call the run_bigquery_query tool with an appropriate SQL statement.</p><p>The key difference here is subtle but important.</p><p>With traditional APIs, the <strong>developer decides which API to call</strong>.<br>With MCP tools, the <strong>AI agent decides which tool to use based on the user request</strong>.</p><p>Under the hood these tools may still call APIs internally. MCP simply provides a structured way for AI systems to understand and safely use those capabilities.</p><h3>How MCP Fits Into a Data Platform Architecture</h3><p>To understand the practical value of MCP, it helps to visualize where it sits in a typical data platform environment.</p><p>A simplified architecture might look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q9jItv3zrdXRe8Ee2NzZrA.png" /></figure><p>Each system in the platform exposes its capabilities through an MCP server.</p><p>For example, a BigQuery MCP server might expose tools such as:</p><pre>run_query<br>list_datasets<br>get_table_schema</pre><p>Similarly, an Airflow MCP server could expose operational tools:</p><pre>list_dags<br>get_dag_status<br>trigger_dag<br>get_task_logs</pre><p>GitHub can expose developer related capabilities:</p><pre>create_issue<br>list_pull_requests<br>review_pull_request</pre><p>The AI agent acts as the orchestrator. When a user asks a question or requests an action, the agent determines which tools to use and combines them if necessary. This allows the AI to interact with multiple systems within the data platform.</p><h3>Use Case 1: Querying the Data Warehouse Through Natural Language</h3><p>Running ad hoc queries is a very common activity for both data engineers and analysts. Often the workflow involves opening the warehouse console, writing a SQL query, running it, and then sharing the result with the requester.</p><p>While this process is simple, it becomes repetitive when similar questions appear frequently. An MCP connected AI agent can help automate this interaction.</p><p>For example, imagine a user asks:</p><pre>Show failed orders from yesterday</pre><p>The agent recognizes that the question requires a query against the warehouse and decides to call the available tool.</p><pre>Tool: run_bigquery_query</pre><pre>Input:<br>SELECT COUNT(*)<br>FROM orders<br>WHERE status = &#39;FAILED&#39;<br>AND DATE(order_timestamp) = CURRENT_DATE() - 1</pre><p>Once the result is returned, the agent can summarize the answer in natural language. This removes the need for the engineer to manually write and execute the query.</p><p>In organizations where analysts frequently ask data engineers for quick checks or validation queries, this kind of automation can save significant time.</p><h3>Use Case 2: Pipeline Monitoring and Debugging</h3><p>Data pipeline failures are an inevitable part of operating a data platform. When a pipeline fails, engineers typically open the Airflow UI, locate the failed DAG, inspect the failing task, and review the logs to determine what went wrong.</p><p>This process works well but can take time, especially when multiple pipelines fail or when the root cause is not immediately clear.</p><p>If Airflow exposes its capabilities through MCP tools, an AI agent can assist with this process.</p><p>For example, the MCP server may expose tools such as:</p><pre>list_dags<br>get_dag_status<br>get_task_logs</pre><p>Now a user could ask the AI agent:</p><pre>Why did the clickstream pipeline fail today?</pre><p>The agent can perform a sequence of actions:</p><ol><li>Call get_dag_status to determine whether the DAG failed</li><li>Identify which task caused the failure</li><li>Retrieve logs using get_task_logs</li><li>Analyze the log messages and summarize the root cause</li></ol><p>Instead of manually navigating the Airflow interface, the engineer receives a summarized explanation. Over time this can significantly improve observability and reduce the time required to diagnose issues.</p><h3>Use Case 3: Automating GitHub Operations</h3><p>Most modern data engineering projects rely heavily on Git repositories. Engineers regularly review pull requests, manage issues, and track changes in pipeline code.</p><p>These operations can also benefit from MCP integration.</p><p>For example, a GitHub MCP server may expose tools like:</p><pre>list_pull_requests<br>create_issue<br>get_repo_changes</pre><p>An engineer could ask the AI assistant:</p><pre>Summarize today&#39;s pull requests in the data platform repository</pre><p>The agent retrieves the pull requests, analyzes the descriptions and commits, and produces a summary of the changes.</p><p>Another useful workflow could connect pipeline monitoring with development processes. For example, if a pipeline fails repeatedly, the agent could automatically create a GitHub issue describing the problem and attach relevant logs. This helps ensure that operational problems are tracked and resolved quickly.</p><h3>Example: Building a Simple MCP Server in Python</h3><p>To better understand how MCP works, it helps to look at a minimal example. Suppose we want to expose a simple tool that allows an AI agent to run queries against BigQuery.</p><p>First install the required libraries.</p><pre>pip install mcp<br>pip install google-cloud-bigquery</pre><p>Next create a small MCP server that exposes a query tool.</p><pre>from mcp.server.fastmcp import FastMCP<br>from google.cloud import bigquery<br><br>mcp = FastMCP(&quot;bigquery-tools&quot;)<br>client = bigquery.Client()<br><br>@mcp.tool()<br>def run_query(query: str):<br>    &quot;&quot;&quot;Execute a SQL query in BigQuery and return results&quot;&quot;&quot;<br>    query_job = client.query(query)<br>    results = query_job.result()<br>    rows = [dict(row) for row in results]<br>    return rows<br><br>if __name__ == &quot;__main__&quot;:<br>    mcp.run()</pre><p>This server exposes a single tool called run_query. Once connected to an AI agent, the agent can use this tool whenever it determines that executing a warehouse query would help answer a question.</p><p>The important aspect here is that the AI agent does not need to know the internal implementation. It only needs the tool description and parameter schema.</p><h3>Designing MCP Tools for a Data Platform</h3><p>A realistic MCP server for a data platform would likely expose several operational tools. For example:</p><pre>run_bigquery_query<br>check_table_partition<br>validate_schema<br>trigger_airflow_dag<br>get_pipeline_status<br>list_gcs_files<br>submit_dataproc_job</pre><p>Consider a tool designed to check whether a daily partition exists in a table.</p><pre>Tool: check_table_partition</pre><pre>Description:<br>Verify whether a partition exists for a specific date</pre><pre>Parameters:<br>table_name<br>date</pre><p>With this tool available, an engineer could ask:</p><pre>Did today&#39;s partition land in the clickstream table?</pre><p>The agent calls the tool, verifies the metadata, and returns the answer. This kind of automation helps reduce manual checks that engineers perform frequently during daily operations.</p><h3>Ideas for Internal AI Data Platform Assistants</h3><p>As MCP matures, many organizations are experimenting with internal AI assistants that help manage data platforms. These assistants use MCP tools to interact with the underlying infrastructure.</p><p>For example, a <strong>pipeline health assistant</strong> could answer questions like:</p><pre>Which pipelines failed today?</pre><p>or</p><pre>Show pipelines that took longer than usual</pre><p>A <strong>data quality assistant</strong> might help detect anomalies in ingestion pipelines by comparing row counts or data volumes across time periods.</p><pre>Check if row counts dropped in yesterday&#39;s ingestion</pre><p>Similarly, a <strong>cost monitoring assistant</strong> could help teams track expensive warehouse queries.</p><pre>Which BigQuery queries consumed the most slots yesterday?</pre><p>These assistants act as an operational layer on top of the data platform, helping engineers quickly retrieve insights without manually navigating multiple tools.</p><h3>Where MCP Fits in the Future of Data Engineering</h3><p>AI will not replace the need for skilled data engineers. Data platforms will continue to require careful design, governance, and operational expertise.</p><p>However, AI agents can significantly improve how engineers interact with these platforms.</p><p>Instead of manually switching between dashboards, writing diagnostic queries, and reviewing logs across different systems, engineers may increasingly rely on AI assistants that can access these systems directly through MCP.</p><p>A future workflow might look like this:</p><p>An engineer asks:</p><pre>Check yesterday&#39;s ingestion pipelines and summarize failures</pre><p>The AI agent checks Airflow, reviews logs, runs warehouse queries, and reports the result in a few seconds. If necessary it may even create an issue in GitHub for further investigation.</p><p>This transforms AI from a simple code generator into a <strong>practical operational assistant for data platforms</strong>.</p><h3>Final Thoughts</h3><p>Model Context Protocol is still evolving, but it allows AI agents to move beyond generating code and start <strong>interacting directly with real systems</strong>.</p><p>For data engineering teams, this means:</p><p>• faster debugging<br>• easier platform monitoring<br>• better automation of repetitive tasks<br>• AI powered operational assistants</p><p>As modern data platforms continue to grow in complexity, tools like MCP can help bridge the gap between <strong>human workflows and intelligent automation</strong>.</p><p>For engineers working with systems like BigQuery, Airflow, GitHub, and cloud pipelines, this opens an exciting opportunity to build <strong>AI enabled data platforms</strong>.</p><p>And this may become one of the most practical ways AI improves day to day data engineering work.</p><p>PS if you have any questions, or would like something clarified, you can find me on <a href="https://www.linkedin.com/in/manojvsj/"><strong>LinkedIn</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb3e48a0994f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Query Images and Video Directly in BigQuery: From JSON to GenAI, The Evolution of BigQuery]]></title>
            <link>https://medium.com/google-cloud/how-to-query-images-and-video-directly-in-bigquery-from-json-to-genai-the-evolution-of-bigquery-078724ef7027?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/078724ef7027</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Sun, 04 Jan 2026 11:21:00 GMT</pubDate>
            <atom:updated>2026-03-19T05:26:05.332Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A practical guide to mastering Structured, Semi-Structured, and Unstructured data without leaving SQL</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*On9_IZ45Ah6othUyia9TkQ.png" /></figure><p>Most of us use BigQuery for exactly what it is famous for: <strong>tables, rows, columns, and fast SQL analytics.</strong></p><p>Some of us go a step further. We embrace the flexibility of modern data pipelines and query semi-structured data like JSON logs , using STRUCT, ARRAY, and native JSON functions.</p><p>But there is a third frontier that very few data engineers ask about:</p><blockquote><em>Have you ever tried querying unstructured data like images, audio, or videos using BigQuery?</em></blockquote><p>The answer is usually “No, that belongs in Cloud Storage.” But the reality is: <strong>Yes, BigQuery can do that too.</strong></p><p>Here is how BigQuery has evolved into a unified analytics engine for all three types of data, with examples of how to build each.</p><ol><li><strong>Structured data</strong> — Traditional rows and columns stored in tables</li><li><strong>Semi-structured data</strong> — JSON payloads, nested and repeated fields</li><li><strong>Unstructured data</strong> — Images, audio, video files, and other binary objects</li></ol><h3>1. Structured Data: The Foundation</h3><p><strong>What it is: </strong>The comfort zone. Tables with fixed schemas, strong typing, and rigid rows and columns.</p><p><strong>Examples: </strong>Sales transactions, financial metrics, inventory logs.</p><p>This is where BigQuery is unbeatable in terms of speed and cost. You define the schema upfront, and the data must strictly adhere to it.</p><h4>Step 1: Create the Table</h4><p>You define strict types (Integer, String, Date) for every column.</p><pre>CREATE TABLE `my_project.retail.transactions`<br>(<br>  transaction_id INT64,<br>  customer_name STRING,<br>  purchase_date DATE,<br>  amount FLOAT64<br>);</pre><h4>Step 2: Load the Data</h4><p>You can load CSVs or insert rows manually. If the data doesn’t match the schema (e.g., putting text in an integer field), the load fails.</p><pre>INSERT INTO `my_project.retail.transactions`<br>VALUES<br>  (101, &#39;Alice Smith&#39;, &#39;2023-10-01&#39;, 150.50),<br>  (102, &#39;Bob Jones&#39;, &#39;2023-10-02&#39;, 89.99);</pre><h4>Step 3: Query the Data</h4><p>Standard SQL. Fast, efficient, and predictable.</p><pre>SELECT customer_name, amount<br>FROM `my_project.retail.transactions`<br>WHERE amount &gt; 100;</pre><h3>2. Semi-Structured Data: The Modern Standard</h3><p><strong>What it is:</strong> Schema-on-read or flexible schemas.</p><p><strong>Examples:</strong> API responses, nested JSON logs, clickstream event data.</p><p>Data Engineers often dread flattening JSON files before loading them. BigQuery solves this natively. With the JSON data type, you can ingest raw nested structures without knowing the schema in advance.</p><h4>Step 1: Create the Table</h4><p>Instead of defining every single column, we use the JSON data type for the flexible payload.</p><pre>CREATE TABLE `my_project.app_logs.events`<br>(<br>  event_id STRING,<br>  timestamp TIMESTAMP,<br>  payload JSON  -- The magic column that holds nested data<br>);</pre><h4>Step 2: Load the Data</h4><p>We can insert complex, nested JSON objects directly into the payload column.</p><pre>INSERT INTO `my_project.app_logs.events`<br>VALUES<br>  (&#39;evt_001&#39;, CURRENT_TIMESTAMP(), JSON &#39;{&quot;user&quot;: {&quot;id&quot;: 5, &quot;region&quot;: &quot;US&quot;}, &quot;action&quot;: &quot;click&quot;, &quot;metadata&quot;: {&quot;browser&quot;: &quot;Chrome&quot;}}&#39;),<br>  (&#39;evt_002&#39;, CURRENT_TIMESTAMP(), JSON &#39;{&quot;user&quot;: {&quot;id&quot;: 9, &quot;region&quot;: &quot;EU&quot;}, &quot;action&quot;: &quot;purchase&quot;, &quot;cart&quot;: [12, 45]}&#39;);</pre><h4>Step 3: Query the Data</h4><p>We use dot notation to access fields inside the JSON. We don’t need joins; we just traverse the object.</p><pre>SELECT<br>  event_id,<br>  payload.user.region, -- Extracts &quot;US&quot; or &quot;EU&quot;<br>  payload.action<br>FROM `my_project.app_logs.events`<br>WHERE JSON_VALUE(payload.user.region) = &#39;US&#39;;</pre><h3>3. Unstructured Data: The Hidden Capability</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*9OETSUvGm8jgM1gdYjIxYw.png" /></figure><p>This is where things get interesting. We are talking about data that historically lived completely outside the analytics workflow: <strong>Images, Audio, PDFs, and Videos.</strong></p><p>Traditionally, you couldn’t “query” a bucket of JPEG images. That gap is now closed with <strong>BigLake Object Tables</strong>.</p><h4>Step 0: The Setup (The Security Handshake)</h4><ol><li><strong>Create Connection:</strong></li></ol><ul><li>In the BigQuery <strong>Explorer</strong> pane, click <strong>+ ADD</strong> &gt; <strong>Connection to external data sources</strong>.</li></ul><p><strong>2. Configure:</strong></p><ul><li><strong>Connection type:</strong> Select Vertex AI remote models, remote functions, BigLake and Spanner (Cloud Resource).</li><li><strong>Connection ID:</strong> Name it (e.g., my-connection).</li><li><strong>Location:</strong> Select your region (e.g., US).</li></ul><p><strong>3. Grant Access (Crucial):</strong></p><ul><li>Copy the <strong>Service Account ID</strong> from the connection details.</li><li><strong>Role 1 (For Files):</strong> Go to your <strong>Bucket</strong> permissions and grant it <strong>Storage Object Viewer</strong>.</li><li><strong>Role 2 (For AI):</strong> Go to <strong>IAM &amp; Admin</strong> (Project level) and grant it <strong>Vertex AI User</strong>.</li></ul><p>Now, BigQuery has the “identity” required to read your files, and the SQL code in Step 1 will work perfectly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iK6HVOta0YvOkOouxG1OMw.png" /></figure><h4>Step 1: Create the Object Table</h4><p>Instead of creating a table that holds data, you create a table that <strong>points</strong> to a Google Cloud Storage bucket.</p><pre>CREATE EXTERNAL TABLE `my_project.media.images`<br>WITH CONNECTION `projects/my_project/locations/eu/connections/my-test-connection`<br>OPTIONS (<br>  object_metadata = &#39;SIMPLE&#39;,  -- Tells BQ to read file metadata<br>  uris = [&#39;gs://my-company-images-bucket/*.jpg&#39;] -- Points to your GCS bucket<br>);</pre><h4>Step 2: Load the Data</h4><p>There is no load step. This is the beauty of it. You simply upload files to your GCS bucket (gs://my-company-images-bucket/) as you normally would. BigQuery automatically “sees” any new file dropped there immediately.</p><h4>Step 3: Query the Data (Metadata &amp; Content)</h4><p><strong>Basic Query (Metadata):</strong>BigQuery does not read the pixels yet; it reads the object metadata. This is incredibly fast and allows you to inventory millions of files.</p><pre>SELECT<br>  uri,           -- gs://my-company-images-bucket/image1.jpg<br>  content_type,  -- image/jpeg<br>  size,          -- 102400 (bytes)<br>  updated        -- 2023-10-05 14:00:00 UTC<br>FROM `my_project.media.images`<br>WHERE size &gt; 1000000; -- Find all images larger than 1MB</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xE0KJ1PBtvpB2PZxhF5gNA.png" /></figure><h3><strong>Advanced Query (SQL + AI Analysis):</strong></h3><p><strong><em>Now for the magic</em></strong>. To actually analyze the content (e.g., “What is in this picture?”), we pass the uri from our Object Table into a BigQuery ML model (like Gemini Pro Vision) along with prompt field in the STRUCT to tell the model what to do.</p><p><em>Note</em>: Before that you must create “Remote Model” in your BigQuery project.</p><p>Think of this model as a <strong>saved configuration</strong> that tells BigQuery: <em>“When I call this model, send the data to Vertex AI’s Gemini 1.5 Flash model and return the results.”</em></p><pre>CREATE OR REPLACE MODEL `my_project.media.gemini_vision_model`<br>REMOTE WITH CONNECTION `projects/my-project/locations/eu/connections/my-test-connection`<br>OPTIONS (ENDPOINT = &#39;gemini-2.5-flash&#39;);</pre><p>You can now use SQL to <strong>read images directly in BigQuery</strong> using <strong>BigQuery ML functions</strong> with the remote models mentioned above.<br> In this example, <strong>Gemini Flash</strong> is used as the remote model to generate a description of the image using the ML.GENERATE_TEXT function.</p><pre>SELECT<br>  uri,<br>  ml_generate_text_llm_result AS description<br>FROM ML.GENERATE_TEXT(<br>  MODEL `my_project.dataset.gemini_vision_model`,<br>  TABLE `my_project.media.images`,<br>  STRUCT(<br>    &#39;Describe the content of this image in detail&#39; AS prompt,<br>    TRUE AS flatten_json_output,<br>    0.4 AS temperature,<br>    1024 AS max_output_tokens<br>  )<br>)<br>LIMIT 5;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rKnI3Hci3XbhzelcTnzkLw.png" /></figure><h3>Why This Matters for Data Engineers</h3><p>This shift changes the definition of an analytics platform. BigQuery is no longer just a warehouse for rows and columns. It is a <strong>Unified Analytics Layer</strong>.</p><ul><li><strong>No Data Movement:</strong> Your heavy unstructured files stay in cheap object storage (GCS).</li><li><strong>No Complex Pipelines:</strong> You don’t need to spin up a separate Python environment just to tag images.</li><li><strong>Just SQL:</strong> You govern, query, and analyze Structured, Semi-Structured, and Unstructured data using the same language.</li></ul><h3>Final Thought</h3><p>If you are using BigQuery only for tables and JSON, you are utilizing only a fraction of its capability.</p><p>BigLake Object Tables unlock an entirely new class of use cases from media analytics to AI-driven insights — without ever leaving the SQL console.</p><p>BigQuery is far more versatile than most of us realize.</p><p>PS if you have any questions, or would like something clarified, you can find me on <a href="https://www.linkedin.com/in/manojvsj/"><strong>LinkedIn</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=078724ef7027" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/how-to-query-images-and-video-directly-in-bigquery-from-json-to-genai-the-evolution-of-bigquery-078724ef7027">How to Query Images and Video Directly in BigQuery: From JSON to GenAI, The Evolution of BigQuery</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Google Cloud Professional Data Engineer Renewal Exam (2026 version): What Changed and What to…]]></title>
            <link>https://medium.com/@manojkumar.vadivel/i-took-the-new-google-cloud-professional-data-engineer-renewal-exam-dec-2025-5b90d68f8dac?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b90d68f8dac</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[google-cloud-certified]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[gcp-certification]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Sun, 14 Dec 2025 06:42:48 GMT</pubDate>
            <atom:updated>2025-12-29T19:24:42.445Z</atom:updated>
            <content:encoded><![CDATA[<h3>New Google Cloud Professional Data Engineer Renewal Exam (2026 version): What Changed and What to Expect</h3><p>Exam changes, preparation tips, and honest takeaways from the new format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v06S0VzeSJRxiqGGE0CniQ.png" /></figure><p>I’ve recently renewed my <strong>Google Cloud Professional Data Engineer (PDE)</strong> certification using the <strong>newly introduced renewal exam path</strong>🎓 in December 2025.</p><p>The exam has clearly shifted its focus compared to previous versions.</p><p>In this article, I’ll share what changed, which topics mattered most, and what surprised me.</p><p>Since this exam format is very new and there isn’t much firsthand experience available online yet, I wanted to share my journey so it can help others who are planning to renew their certification.</p><p>This is my <strong>4th consecutive renewal</strong> of the Professional Data Engineer certification (2019 — 2027), and this attempt felt quite different from the earlier versions.</p><h4><strong>A New Renewal Path for Professional Certifications</strong></h4><p>Google Cloud recently introduced a <strong>renewal exam option</strong> for professional certifications. Starting <strong>60 days before your certification expires</strong>, you now have two choices:</p><ul><li>Take the <strong>Standard (full-length) exam (2 Hour)</strong>, or</li><li>Take the <strong>Shorter Renewal exam (1 Hour)</strong></li></ul><p>Once you choose a path and make your first attempt, <strong>you are locked into that path</strong> until you either pass or your certification expires.</p><p>If you take the <strong>Renewal Exam</strong> and fail, you must retake the <strong>Renewal Exam</strong>. You cannot decide “this is too hard” and switch to the Standard exam for your second attempt. You cannot switch between Standard and Renewal midway.</p><p>I chose the <strong>Renewal exam path</strong>.</p><p>Compared to the previous PDE exams I’ve taken, The new version felt <strong>more practical and more opinionated </strong>and also<strong> </strong>noticeably tougher, with deeper focus on modern data architectures, security, and reliability at scale.</p><ul><li>The exam had only <strong>20 questions</strong></li><li>There were <strong>no case studies</strong> for me, But some people got 1–2 questions from Cymbal case study.</li><li>Every question carried significant weight</li></ul><p>Because the question count is small, <strong>even a few wrong answers can lead to a fail</strong>, which makes this exam surprisingly tricky. It is shorter, but not easier.</p><p>The questions were very realistic and closely aligned with <strong>day-to-day data engineering work on GCP</strong>, especially around modern architectures, governance, security, and scale.</p><h3>How the Content Differs</h3><p><strong>Standard Exam:</strong></p><ul><li>The Standard exam is designed to test your ability to build data pipelines from the ground up. It includes more fundamental questions and definitions.</li><li>You can expect basic questions such as when to use which services, for example Dataflow vs Dataproc, Pub/Sub fundamentals, and core IAM concepts.</li></ul><p><strong>Renewal Exam:</strong></p><ul><li>The Renewal exam assumes you already know the fundamentals. It skips basic definitions and focuses more on recent services, platform optimizations, and architectural decision-making.</li><li>Most questions reflect real-world scenarios, testing how you apply your experience rather than how well you recall concepts.</li><li><em>Expect more:</em> BigLake, Analytics Hub, Dataplex, Dataform, and recent BigQuery features (like remote functions)</li></ul><p>In short, the Renewal exam tests whether you have <strong>kept up with GCP</strong>, not whether you can memorize concepts.</p><h3>Skills Validated in the Renewal Exam</h3><p>The Renewal exam validated <strong>practical data engineering skills</strong>, not theoretical knowledge:</p><ul><li>Designing scalable BigQuery architectures</li><li>Building batch and streaming pipelines</li><li>Applying lakehouse and data mesh patterns</li><li>Orchestrating complex workflows</li><li>Securing data using modern GCP security features</li><li>Sharing data safely across teams and organizations</li><li>Optimizing performance and cost at scale</li></ul><p>It closely matched real-world production responsibilities of a data engineer.</p><h3>How to Prepare for the Renewal Exam</h3><p>If you are preparing for the Renewal exam, here is what I would recommend based on my experience.</p><p>First, make sure you are comfortable with <strong>reading long scenario questions slowly</strong>. Many options look correct at first glance, but only one fully satisfies all constraints mentioned in the question. Skimming will almost certainly cost you marks.</p><p>This exam noticeably focused on <strong>BigQuery design patterns</strong>, <strong>BigLake architecture</strong>, <strong>metadata management</strong>, <strong>data sharing</strong>, and <strong>security</strong>. Overall, the exam is testing how well you can <strong>design and reason about modern data architectures on GCP</strong>, rather than how well you remember individual services or configurations.</p><h3>Prepare These Thoroughly</h3><p>These areas appeared most frequently and in deeper scenarios:</p><ul><li><strong>BigQuery</strong> (performance tuning, reservations, materialized views, continuous queries)</li><li><strong>BigLake</strong> and external table access patterns</li><li><strong>Dataplex</strong> (data mesh architecture, governance, metadata management)</li><li><strong>Security</strong> (DLP, CMEK, authorized views, secure data sharing)</li><li><strong>Analytics Hub</strong> (cross-project and partner data sharing)</li><li><strong>Dataflow</strong> (side inputs, Dataflow Shuffle Service, FlexRS, Dataflow SQL, streaming enrichment patterns)</li><li><strong>Dataform</strong> (BigQuery-native transformations, dependency management, incremental models)</li><li><strong>Cloud Composer (Airflow)</strong> (event-driven DAGs, orchestration patterns)</li><li><strong>Pub/Sub</strong> (retry strategies, dead-letter topics)</li><li><strong>Cloud Storage</strong> (retention policies, bucket lock)</li><li><strong>Vertex AI</strong> (ML pipeline lifecycle, data preparation, notebooks)</li><li><strong>BigQuery ML (BQML)</strong> (model creation and deployment, training workflows, Vertex AI integration)</li><li><strong>Dataproc</strong> (Serverless Spark, Dataproc templates)</li><li><strong>Datastream</strong> (change data capture, data migration and replication pipelines)</li></ul><h3>The Old Way vs. New Way — The Trap Questions (Give More Attention)</h3><p>One important thing the Renewal exam tests is whether you are aware of <strong>modern GCP patterns that replace older ways of building solutions</strong>.</p><p>Many options sound correct because they mention familiar services, but they represent the <em>old way</em> of solving a problem. These answers don’t align with how Google Cloud expects architectures to be designed today.</p><p>Below are examples of the <em>type of trap questions</em> you can expect, shown clearly as <strong>question, trap answer, and correct answer</strong>:</p><p><strong>Question:</strong> You need to ingest data from an Oracle database and apply significant transformations such as masking, joining, and filtering before the data lands in BigQuery.</p><ul><li><strong>Trap Answer:</strong> Datastream directly to BigQuery.<br><em>Reason:</em> Datastream follows an ELT pattern and delivers raw change data.</li><li><strong>Correct Answer:</strong> Datastream → Dataflow → BigQuery.<br><em>Reason:</em> Dataflow handles complex transformations in flight before loading clean data.</li></ul><p><strong>Question:</strong> You need to query data stored in AWS S3 and want to minimize data egress costs.</p><ul><li><strong>Trap Answer:</strong> Use a transfer service to move the data into GCS.<br><em>Reason:</em> Moving large datasets increases cost.</li><li><strong>Correct Answer:</strong> BigQuery Omni.<br><em>Reason:</em> Queries run inside AWS and only the result set is returned.</li></ul><p><strong>Question:</strong> You want to run a Spark job but do not want to manage clusters.</p><ul><li><strong>Trap Answer:</strong> Dataproc on Compute Engine.</li><li><strong>Correct Answer:</strong> Dataproc Serverless.<br><em>Reason:</em> You submit batch Spark jobs without provisioning or managing infrastructure.</li></ul><p>Finally, remember that this exam rewards <strong>real-world experience</strong>. If you have been working hands-on with GCP, trust that experience. The exam is not trying to trick you with obscure facts; it is checking whether your decisions align with how Google Cloud expects data platforms to be built today.</p><h3>Resources to Consider</h3><p>If you are planning to take the Renewal exam, I would strongly recommend going through a few official and focused resources rather than scattered content:</p><ul><li>Read the <a href="https://services.google.com/fh/files/misc/professional_data_engineer_renewal_exam_guide_english.pdf"><strong>official Google Cloud Professional Data Engineer Renewal exam guide</strong></a> to understand scope and expectations.</li><li>Review <a href="https://docs.google.com/forms/d/e/1FAIpQLSfkWEzBCP0wQ09ZuFm7G2_4qtkYbfmk_0getojdnPdCYmq37Q/viewform"><strong>sample questions</strong></a><strong> from Google’s official certification page</strong> to get familiar with the scenario-driven style.</li><li>Go through <a href="https://www.skills.google/paths/16"><strong>Data Engineer Path </strong></a>Course or <a href="https://partner.skills.google/paths/85"><strong>Professional Data Engineer Path</strong></a> (Partner program) especially sections related to BigQuery, streaming, governance, and modern architectures.</li><li><strong>ProTip</strong>: Use <strong>NoteBookLM</strong> for your notes and preparation, you can ask to create Quiz from notes &amp; added sources, practice it with latest new features</li></ul><p>These resources are more than sufficient when combined with hands-on experience.</p><h3>Which Exam Should You Choose?</h3><p>Choosing between the Renewal and Standard exam really depends on how closely you are working with Google Cloud today.</p><p>The <strong>Renewal Exam</strong> is a good fit if you work hands-on with GCP data platforms regularly, are comfortable with recent services and modern design patterns, prefer scenario-based questions over definitions, and want a shorter exam.</p><p>The <strong>Standard Exam</strong> is a safer option if you haven’t worked deeply on GCP in the last year or two, are not confident with newer services like BigLake or Dataplex, or prefer having more questions as a buffer against a few mistakes. Somewhat surprisingly, the Standard exam can feel safer if you are out of touch with recent updates.</p><h3>Final Thoughts</h3><p>The new Professional Data Engineer Renewal exam is <strong>short, focused, and experience-driven</strong>.It rewards engineers who actively work on Google Cloud and understand how modern data platforms are built and operated.</p><p>This being my <strong>4th consecutive renewal</strong>, I can confidently say this version reflects real-world data engineering better than any previous PDE exam I’ve taken.</p><p>If you are actively working on GCP, the Renewal path is a strong option. If not, spend some time revisiting recent services before choosing your exam path.</p><p>I hope this write-up helps you prepare and make the right decision.</p><p>All the best 🚀</p><p><strong>Here is my Certification :</strong></p><p><a href="https://www.credly.com/badges/e829cfce-a71a-4032-9ee9-6d24fe106729/public_url">Professional Data Engineer Certification was issued by Google Cloud to Manojkumar Vadivel.</a></p><p>If you’re planning to take the renewal exam soon, feel free to leave a comment or connect with me on <a href="https://www.linkedin.com/in/manojvsj/">LinkedIn</a>. Happy to share more details.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b90d68f8dac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The .claude Folder: A 10-Minute Setup That Makes AI Code Smarter]]></title>
            <link>https://medium.com/@manojkumar.vadivel/the-claude-folder-a-10-minute-setup-that-makes-ai-code-smarter-93da465ef39e?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/93da465ef39e</guid>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[writing-prompts]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 17:00:18 GMT</pubDate>
            <atom:updated>2025-09-15T17:00:18.350Z</atom:updated>
            <content:encoded><![CDATA[<h3>The .claude Folder: A 10-Minute Setup That Makes AI Code Smarter</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*tVYDAvF0OwARJqQw" /></figure><blockquote>If you’re new to <a href="https://www.anthropic.com/claude-code"><strong>Claude Code</strong></a>, it’s a powerful AI coding agent that helps you write, refactor, and understand code faster. This article explains how to set it up with the <strong>.claude folder</strong> so you can get the best results and maximum productivity from day one.</blockquote><p>If you use Claude in your editor and you ask Claude Code for help, the answer often works but feels generic.. It doesn’t fit your project. And you spend more time fixing it than you saved.</p><p>Here’s the thing: Claude Code is smart, but it doesn’t know your project. It doesn’t know your coding standards, your team’s preferences, or what “good” looks like for your specific work.</p><p>But there’s a simple fix! ….</p><h4>The Problem with Default Claude Code</h4><p>When you ask Claude to write code without context, you get default answers. Generic solutions. Code that works but doesn’t match how your team actually builds things. It’s like asking someone to cook dinner without telling them what you like to eat. You’ll get food, but probably not what you wanted.</p><h3>Configuration Files Change Everything — The .claude Folder</h3><h4>What is the .claude folder?</h4><p>It’s a lightweight setup at your project’s root where you store a few small text files. These files explain how your project works and how you want code to look.</p><p>Think of it as an onboarding document for a new teammate — except the teammate is Claude.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P3LJGNztq9P6aSL1" /></figure><p><strong>You create a few text files that explain:</strong></p><ul><li>How your team writes code</li><li>What your project structure looks like</li><li>What quality means for your work</li><li>Common mistakes to avoid</li></ul><p>That’s it. No complex setup. Just some markdown files with your preferences.</p><p><strong>A common layout:</strong></p><pre>.claude/<br> ├─ rules.md      # coding standards and guardrails<br> ├─ context.md    # what this project is and how it runs<br> ├─ prompts.md    # reusable prompt snippets for common tasks<br> └─ settings.json # optional structured preferences</pre><p>Keep it in version control so the whole team benefits.</p><h3>Why it matters</h3><p>AI writes better code when it understands your project. These files give it the same orientation you’d give a new teammate:</p><ul><li>What we’re building</li><li>How we structure code</li><li>Which tools and libraries we use</li><li>What “good” looks like in this repo</li></ul><p>You’ll see more relevant suggestions and fewer style fixes.</p><h3>What to put in each file?</h3><h4>1. rules.md — your coding standards</h4><blockquote>Be short. Be specific. Focus on things the AI can follow.</blockquote><p>Example:</p><pre># Coding Rules<br>- Follow the project formatter. Do not change file style.<br>- Use logging instead of print. Include error context.<br>- Add type hints and docstrings for public functions.<br>- Keep functions small; prefer composition over long scripts.<br>- Write safe defaults. Handle timeouts and retries where external calls exist.<br><br># Tests<br>- Provide a minimal test when adding new modules.<br>- Use fakes or fixtures; do not call real services.<br><br># Security<br>- Never include secrets in code or examples.<br>- Use environment variables or placeholders like &lt;API_KEY&gt;.</pre><h4>2. context.md — the project in plain language</h4><blockquote>Give a one-screen brief. No fluff.</blockquote><p><strong>Example:</strong></p><pre># Project Context<br>This is a service that ingests data from an external source, stores it, and exposes a simple API.<br><br>Main parts:<br>- Ingestion module (scheduled)<br>- Storage layer (database + object storage)<br>- API service<br>- Lightweight UI<br><br>Conventions:<br>- Config via environment variables<br>- Error handling with structured logs<br>- CI runs tests and lint on every PR<br><br>Dependencies:<br>- Runtime: Python 3.11<br>- Package: requests, pydantic<br>- Tools: Docker, Makefile for common tasks</pre><h4>3. prompts.md — your reusable requests</h4><blockquote>Save the prompts you repeat. Keep them task-focused.</blockquote><p><strong>Examples:</strong></p><pre># Add a module<br>Create a new module that does X. Include:<br>- A clear, typed interface<br>- Error handling and logging<br>- A small unit test with a fake<br><br># Improve performance<br>Review this function for bottlenecks. Propose changes.<br>Explain trade-offs in 3-5 bullet points.<br><br># Write docs<br>Draft README instructions for running the project locally:<br>- Prereqs<br>- Setup<br>- Common commands<br>- How to run tests</pre><h4>4. settings.json (optional) — structured preferences</h4><blockquote>Use this if your tool supports it. Keep it minimal.</blockquote><p><strong>Example:</strong></p><pre>{<br>  &quot;style&quot;: {<br>    &quot;python&quot;: {<br>      &quot;use_type_hints&quot;: true,<br>      &quot;docstrings&quot;: true<br>    }<br>  },<br>  &quot;tests&quot;: {<br>    &quot;require_for_new_modules&quot;: true<br>  }<br>}</pre><h3>Getting Started in 10 Minutes</h3><ol><li>Create a <strong><em>.claude</em></strong> folder in your project root.</li><li>Add a <strong>context.md</strong> file. Write 2–3 sentences about what your project does and who uses it.</li><li>Add a <strong>rules.md</strong> file. List 5–10 coding preferences your team has.</li><li>Start using Claude. Notice how the responses change.</li><li>Add more details over time. The configuration grows with your project.</li></ol><h4>It’s Not Perfect, But It’s Better</h4><p>Claude still makes mistakes. It’s not magic. But when it knows your context, those mistakes are smaller and easier to fix. And the code it generates actually fits your project instead of looking like it came from<br> a tutorial.</p><p>The setup takes 10 minutes. The time you save starts immediately.</p><h4>The Real Benefits</h4><ul><li>You spend less time explaining what you want. Claude already knows.</li><li>Your code stays consistent. No more mixing coding styles within the same project.</li><li>New team members get better help. They can use the same Claude setup and get code that fits your standards.</li><li>You catch problems early. Claude knows your common mistakes and helps you avoid them.</li></ul><h4>Final thought</h4><p>Most teams skip this step because it feels like extra work, and want to jump straight into coding. But think about it this way: you’re going to explain your preferences to Claude anyway. Either once in a config file, or over and over again in every conversation.</p><p>The config file is just being smart about it. Your future self will thank you when Claude starts giving you exactly what you need, right from the start.</p><p>PS if you have any questions, or would like something clarified, you can find me on <a href="https://www.linkedin.com/in/manojvsj/"><strong>LinkedIn</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93da465ef39e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Migrated and Scaled Our Data Architecture for the Future]]></title>
            <link>https://medium.com/news-uk-technology/how-we-migrated-and-scaled-our-data-architecture-for-the-future-7c8e920053da?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c8e920053da</guid>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Thu, 04 Sep 2025 07:13:25 GMT</pubDate>
            <atom:updated>2025-12-14T07:31:40.964Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSTI6_2SZuUWT_sf9Oep3w.png" /></figure><p>A real-world data warehouse re-architecture at News UK, adopting a data mesh approach to scale for the future</p><h3>Introduction</h3><p>In the fast-paced world of data engineering, staying ahead of the curve is not just a luxury- it’s a necessity. Our team recently completed a transformative migration project that redefined how we handle data at scale. This journey was not just a technical challenge but a strategic initiative that empowered our organization to make faster, smarter decisions.</p><p>In this post, I’ll share the story of our Data warehouse re-architecture project in <strong>News UK</strong>, the challenges we faced, and the lessons we learned. Whether you’re a fellow data engineer or a business leader, I hope this story resonates with you and inspires your own data journey.</p><h4><strong>About News UK</strong></h4><p><a href="https://www.news.co.uk/">News UK</a> is one of the UK’s leading media companies. It publishes <em>The Times</em>, <em>The Sunday Times</em>, and <em>The Sun</em>, and operates national radio brands including <em>Virgin Radio UK</em>, <em>talkSPORT</em>, <em>Times Radio</em>, and <em>U105</em>.</p><p>Beyond publishing and broadcasting, News UK runs digital products and membership programs like Sun Savers and Sun Club. With such a wide mix of digital content and services, News UK collects vast amounts of first-party data and records billions of user interactions each day. This information is key to understanding audiences and improving the content, products, and services we deliver.</p><h3>The Problem: Why We Needed a Change</h3><p>Our current Data platforms setup works well and covers most of our business needs. But we wanted to explore new opportunities and make sure we’re ready for the future. So, we took a step back, looked at our system, and found a few areas where we could improve to keep up with modern data engineering.</p><p><strong>Below are few areas we identified that had scope for improvement</strong></p><ul><li><strong>Lack of Domain Ownership:</strong> Teams were dependent on a centralized data team for access, modifications, and insights. This reliance limited autonomy, slowed innovation, and created bottlenecks, as a single team had to manage requests from all domains.</li><li><strong>Governance and Performance Issues:</strong> A single BigQuery project managing all domains resulted in inefficiencies and bottlenecks. The lack of clear governance structures made it difficult to enforce data quality, security, and compliance standards across teams.</li><li><strong>Scalability Concerns:</strong> The growing complexity of the system made it increasingly difficult to scale efficiently. As data volumes grew and business requirements expanded, the monolithic architecture struggled to meet performance expectations, leading to delays and resource constraints.</li><li><strong>Limited Flexibility for Modern Solutions:</strong> The base structure of the current architecture was not flexible enough to integrate modern tools, frameworks, or solutions. Wider dependencies across the system created significant challenges, making it difficult to adopt innovative technologies without causing disruptions. This rigidity hindered progress and limited the ability to modernise the platform effectively.</li><li><strong>Tooling Limitations:</strong> Off-the-shelf tools often failed to meet the unique and varied requirements of our platform. Custom use cases and specific workflows required tailored solutions that external tools could not adequately address, leading to inefficiencies and workarounds.</li></ul><h3>The Solution: Re-Architecting for the Future</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cn9ezxxcb12FNBRd" /></figure><p>After evaluating various approaches, we decided to embark on a significant re-architecture of our data platform. This was a bold move, but one that promised significant rewards.</p><p>Our Data Re-Architecture Project was more than just a technical upgrade, it was a bold step forward for our organzation. We transitioned from a traditional, centralised data warehouse to a decentralised Data Mesh Architecture. This shift wasn’t just about modernising our data systems; it was about unlocking their full potential.</p><p>It was made possible by the shared vision and dedication of teams across the organisation. More than a technical change, it was a cultural shift that helped our teams work smarter, faster, and more collaboratively.</p><p>The goal was to move away from the limitations and provide individual teams autonomy over their data. At the same time, we ensured consistency, compliance, and governance across the organisation. This new framework made our system more scalable, flexible, and ready to adapt to whatever the future brings.</p><h3>Key Initiatives:</h3><p>Our solution was centered on migrating to a Data Mesh Architecture, restructuring our monolithic data warehouse into four domain-specific projects. This modernization effort encompassed several key initiatives:</p><ul><li><strong>Adopting a Data Mesh:</strong> Shifting from a monolithic data lake to a federated, domain-oriented data architecture This empowered teams to own their data and innovate faster.</li><li><strong>Platform Infrastructure Automation:</strong> Automating infrastructure provisioning and management to ensure scalability, consistency, and security.</li><li><strong>Domain-Driven Data Warehouse:</strong> Establishing domain-specific ownership of data to decentralise operations and improve governance.</li><li><strong>Adopting a Data Lake:</strong> Transitioning to a modern storage and processing solution for greater flexibility and scalability.</li><li><strong>Low-Code/No-Code ETL Tools: </strong>Leveraging tools to simplify pipeline creation, enabling faster development and empowering non-technical users.</li></ul><p><strong>Cutting-Edge Automation Frameworks:</strong> Implementing advanced frameworks to streamline operations, reduce manual effort, and enhance workflow efficiency.</p><h3>Migration Challenges:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0eAhMF-mgzlUpObQ" /></figure><ol><li><strong>Monolithic Data Warehouse Dependency</strong>: We currently operate a single monolithic data warehouse that is used in production to support business-critical applications and key performance indicators (KPIs) for decision-making. With over 1,000 dashboards relying on this data, upgrading the databases or altering the presentation layer is not feasible. As a result, all changes must be implemented at the backend without impacting the existing dashboards.</li><li><strong>Legacy Workflow Orchestration:</strong> The platform is supported by 800 Airflow pipelines that serve data to this warehouse. However, these pipelines are built on Airflow version 1, which has reached the end of support. Consequently, we must upgrade all pipelines to Airflow version 2 while simultaneously adopting a domain-driven approach to pipeline migration.</li><li><strong>Rebuilding the Data Platform</strong>: The entire data platform needs to be re-architected with a new design, requiring all data objects — including pipelines, datasets, and transformations — to be migrated to the new platform seamlessly and without disruption to ongoing operations.</li><li><strong>Strict Timelines and Parallel Responsibilities:</strong> The migration must be completed within strict timelines. Simultaneously, our team is required to support ongoing business requirements and deliverables, making resource allocation a significant challenge.</li><li><strong>Custom Use Cases and Tooling Requirements:</strong> Off-the-shelf tools often fail to meet the unique and varied requirements of our platform. We need to develop custom tools and frameworks tailored to handle multiple use cases that are not adequately addressed by external solutions.</li></ol><h3>How we tackled these challenges and Key Innovations and Contributions</h3><h4>1. Simplifying ETL Operations:</h4><p>Our major challenge was migrating <strong>800 legacy Airflow pipelines</strong> to Airflow version 2, as Airflow version 1 had reached the end of support. This migration involved a steep learning curve, manual coding, and the need for architectural knowledge, which posed significant challenges for our BI engineers. Additionally, governance and standardization gaps across teams made it difficult to maintain consistency, and the migration process required substantial manual effort, increasing the risk of delays and errors.</p><p>To address these challenges, we enhanced our internal framework called NewsFlow, a code-free, self-service platform built on top of Apache Airflow. NewsFlow simplifies workflow creation and orchestration by replacing manual coding with YAML-based configurations. Key features include a Dynamic DAG Builder, which automates DAG generation from YAML files, and a Task Hub that provides pre-built tasks and plugins for customization.</p><p><strong>The Impact</strong></p><ul><li>80% Faster Workflow Development: Reduced DAG creation time significantly, improving developer productivity.</li><li>Seamless Migration: Enabled the migration of 800+ workflows from Airflow version 1 to 2 within a tight timeline.</li><li>Increased Adoption: Empowered non-developers to create workflows, fostering collaboration across teams</li></ul><h4>2. Infrastructure Automation: Bringing DevOps culture into Data Engineering</h4><p>This re-architecture project required the <strong>complete rebuilding</strong> and enhancement of the platform infrastructure to support the <strong>Data Lake and Data Mesh Architecture</strong>. This involved creating a cloud environment with various resources, including GCP projects, private VPN tunnels, VPC networks, GCS buckets, Composer environments, BigQuery objects, and more. Managing such a complex cloud environment demanded significant effort and expertise in infrastructure management.</p><p><strong>The Solution: Infrastructure Automation</strong></p><p>We adopted Terraform for Infrastructure-as-Code (IaC) to standardise deployments and automate resource management. Key initiatives included:</p><ul><li><strong>GCP Project Creation Automation:</strong> Automated the setup of new GCP projects for domain-specific architecture.</li><li><strong>Resource Management:</strong> Provisioned BigQuery datasets, GCS buckets, and other resources automatically.</li><li><strong>IAM Management Automation:</strong> Streamlined access control to improve security and compliance.</li></ul><p><strong>The Impact</strong></p><ul><li>Automated the creation of <strong>10 new GCP projects</strong>, <strong>125 BigQuery datasets</strong>, and <strong>58 GCS buckets</strong></li><li>Re-Created the entire platform infrastructure which including various services like cloud composer, VPC network, Logging &amp; Monitoring, App engines, Storage buckets and BigQuery objects using Terraform as IAC</li><li>Improved consistency, reduced manual errors, and enhanced security across environments.</li><li>Integrated with CircleCI for seamless CI/CD workflows.</li></ul><h4>3. Streamlined SQL Transformations</h4><p>One of the key pain points in our existing architecture was managing SQL scripts, which were scattered across multiple locations, such as Airflow repositories and BigQuery views. There was no clear data lineage or automated dependency management, making it difficult to ensure tables were built in the correct sequence. The absence of version control led to errors and inconsistencies, while poor visibility further complicated tracing data transformations, making the process time-consuming and prone to mistakes</p><p><strong>The Solution: Dataform</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VfrJm0Jp8zmeetSq" /></figure><p>To address these issues, we implemented Dataform, a platform that automates SQL-based data transformations. By consolidating all SQL scripts into a centralized repository, Dataform provided:</p><ul><li><strong>Automated Dependency Management:</strong> Ensures tables are built in the correct sequence without manual intervention.</li><li><strong>Version Control:</strong> Tracks changes and enhances collaboration.</li><li><strong>Centralized Transformation Logic:</strong> Modular, reusable SQL code improved maintainability.</li><li><strong>Improved Governance:</strong> Enforced compliance standards across domains.</li></ul><p><strong>The Impact</strong></p><ul><li>Migrated <strong>2,200+ BigQuery objects</strong> into Dataform with the help of Vertex AI for automated SQL conversion.</li><li>Improved pipeline reliability, governance, and traceability with clear data lineage.</li><li>Reduced manual errors and streamlined SQL transformations for domains like Customer, Commercial, Clickstream, and Content.</li></ul><h4>4. API Automation Framework: Democratizing Pipeline Creation</h4><p>API pipeline development and migration involved several complex challenges. Data extraction from APIs relied heavily on manual coding using custom Python scripts, leading to inefficiencies and inconsistencies. Different teams used varying methods, making maintenance and reuse difficult. Every pipeline required developer involvement, creating bottlenecks and slowing down progress.</p><p>Additionally, pipelines were tightly integrated with tools like Airflow, which limited flexibility and portability across systems. Currently, we manage over 50 API pipelines, and this number is expected to grow significantly in the future.</p><p><strong>The Solution: API Automation Framework</strong></p><p>We introduced a Low-Code/No-Code API Automation Framework to standardise and simplify API pipeline creation. This has integrated our internal Newsflow framework, Key features include:</p><ul><li><strong>Authentication Support: </strong>Handles API keys, tokens, JWT, certificates, and cookie-based authentication.</li><li><strong>Diverse API Types:</strong> Supports pagination, looping, and status checks.</li><li><strong>Data Transformation:</strong> Built-in Pandas support for processing.</li><li><strong>Stream Processing and Multiprocessing: </strong>Efficiently handles large datasets and improves resource utilisation.</li></ul><p><strong>The Impact</strong></p><ul><li><strong>Increased Agility:</strong> Enabled non-developers to build API pipelines, reducing dependency on developers. We able to migrate our 50+ pipelines without rewriting the custom logic</li><li><strong>Enhanced Consistency:</strong> Standardized practices improved maintainability and scalability.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*KfOB3eHk_NxuXPolCtYGJQ.jpeg" /></figure><h3>Conclusion</h3><p>The Data Re-Architecture Project exemplifies teamwork, innovation, and significant achievement in data engineering. From building cutting-edge tools like Newsflow, Dataform Integration, and API Automation Framework to adopting Data Mesh Architecture and automating infrastructure, our team’s efforts have set a new benchmark for excellence.</p><h3>Join Us</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*sidOTezyb_XIBKfe" /></figure><p>If you’re passionate about driving innovation in data engineering, we’d love to have you on board! Join our talented and dynamic data engineering team — check out our career page to explore exciting opportunities.</p><p>PS if you have any questions, or would like something clarified, you can find me on <a href="https://www.linkedin.com/in/manojvsj/"><strong>LinkedIn</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c8e920053da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/news-uk-technology/how-we-migrated-and-scaled-our-data-architecture-for-the-future-7c8e920053da">How We Migrated and Scaled Our Data Architecture for the Future</a> was originally published in <a href="https://medium.com/news-uk-technology">News UK Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[14 VS Code Extensions Every Data Engineer Should Swear By for Maximum Productivity]]></title>
            <link>https://medium.com/@manojkumar.vadivel/14-vs-code-extensions-every-data-engineer-should-swear-by-for-maximum-productivity-9fcc2e1b3c4f?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fcc2e1b3c4f</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Sun, 24 Nov 2024 14:45:22 GMT</pubDate>
            <atom:updated>2024-11-24T14:49:04.953Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4cIk_2rdOKeuPpZIimP_8g.png" /></figure><p>As a data engineer, your toolbox is everything. The right set of tools can save you time, reduce frustration, and make your workflows significantly more efficient. One of the most powerful tools we have at our disposal is <strong>Visual Studio Code (VS Code)</strong>. With its extensibility and massive ecosystem, VS Code can be transformed into the ultimate data engineering workstation with the right set of extensions.</p><p>If you’re not already using these VS Code extensions, you’re missing out on some serious productivity boosts. From debugging complex data pipelines to streamlining code generation, these tools are game-changers. Let’s dive into the top <strong>14 VS Code extensions</strong> every data engineer should swear by to take their productivity to the next level</p><h3>1. Jupyter (85M+ downloads)</h3><p><em>For Interactive Notebooks and Data Exploration</em></p><p>If you’re working with Python and data science, chances are you’ve used Jupyter notebooks. Well, this extension brings <strong>Jupyter</strong> functionality directly into VS Code. Whether you’re exploring datasets, running Python scripts, or testing ETL pipelines, this extension allows you to work in an interactive, notebook-style environment without leaving your code editor. It’s perfect for ad-hoc analysis, experimenting with new ideas, and visualizing your data right within your development setup.</p><p><a href="https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter">Download extension here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*wUAzU_umzneJtMMWI3NKGw.gif" /></figure><h3>2. Docker (39M+ downloads)</h3><p><em>For Containerization and Environment Management</em></p><p>As a data engineer, managing environments across different platforms and machines is a constant challenge. Enter <strong>Docker</strong>. This extension simplifies container management by letting you interact with Docker containers directly inside VS Code. Whether you’re building local environments for data pipelines or deploying them to cloud infrastructure, Docker makes managing and running containerized applications seamless. It’s essential for consistency in your environments, especially when working with microservices and distributed systems.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a75B-v04O771H-IP.gif" /></figure><h3>3. Parquet Viewer (181K+ downloads)</h3><p><em>For Seamless Parquet File Viewing</em></p><p>Parquet is one of the most widely-used columnar data formats in big data workflows, especially in systems like <strong>Apache Spark</strong> or <strong>Hadoop</strong>. The <strong>Parquet Viewer</strong> extension lets you easily view and inspect Parquet files directly in VS Code. Whether you’re investigating a dataset or validating schema changes, this extension eliminates the need for external tools to open and explore Parquet files.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=dvirtz.parquet-viewer">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nKfDC8CNWNvnrNJQ.gif" /></figure><h3>4. Rainbow CSV (10M+ downloads)</h3><p><em>For Cleaner, More Readable CSV/TSV Files</em></p><p>Data cleaning is a huge part of a data engineer’s life, and working with <strong>CSV</strong> or <strong>TSV</strong> files is often inevitable. <strong>Rainbow CSV</strong> comes to the rescue by adding color highlighting to CSV/TSV files, making them far easier to read and interpret. It formats columns in a visually distinguishable way and helps you spot issues like misplaced values or incorrect delimiters at a glance.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=mechatroner.rainbow-csv"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*can7Q_bUCbxfVxyI.png" /></figure><h3>5. YAML (19M+ downloads)</h3><p><em>For Managing Config Files</em></p><p>A big part of data engineering involves setting up, managing, and troubleshooting configurations for tools like <strong>Kubernetes</strong>, <strong>Airflow</strong>, or <strong>dbt</strong>. <strong>YAML</strong> files are commonly used for configuration, but they can quickly get out of hand if not formatted correctly. This extension provides syntax highlighting, validation, and error detection, ensuring that your configurations remain clean and error-free.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FvI-kPd-W-14K2Dy.gif" /></figure><h3>6. Data Wrangler (587K+ downloads)</h3><p><em>For Data Cleaning and Transformation</em></p><p>No matter how much you automate, data cleaning is a part of every data engineering project. <strong>Data Wrangler</strong> provides a rich user interface to analyze, clean, and transform your data right within VS Code. It offers insightful statistics, visualizations, and even auto-generates Pandas code as you clean. Perfect for quickly exploring datasets and transforming them before they’re passed into your pipeline.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=ms-toolsai.datawrangler"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lwhesRryMfws6wJf" /></figure><h3>7. Copilot (22M+ downloads)</h3><p><em>For AI-Powered Code Assistance</em></p><p>Need to write SQL queries or manipulate datasets, but feeling bogged down by repetitive tasks? <strong>GitHub Copilot</strong> is your new best friend. Powered by <strong>OpenAI’s GPT</strong>, Copilot assists in generating code snippets, solving complex logic, and even suggesting optimizations. It’s a powerful tool for accelerating code writing, allowing you to focus more on problem-solving rather than boilerplate code.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=GitHub.copilot"><strong>here</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g6am79TH-VgRoD3p" /></figure><h3>8. Pylance (118M+ downloads)</h3><p><em>For Python IntelliSense and Type Checking</em></p><p>Python is the lingua franca of data engineering, and <strong>Pylance</strong> supercharges your coding experience with advanced IntelliSense features. It provides type-checking, better autocompletion, and more accurate suggestions, all of which help you write cleaner, more efficient Python code. As a data engineer, you’re likely juggling multiple libraries, so having robust type information can prevent bugs and improve your productivity.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EDQEbrwD-k5JoSU4.gif" /></figure><h3>9. SQLTools (4.6M+ downloads)</h3><p><em>For Database Exploration and SQL Querying</em></p><p>As a data engineer, working with databases is second nature. <strong>SQLTools</strong> simplifies database management in VS Code, making it easy to connect to and query your PostgreSQL, MySQL, and other relational databases. With built-in query runners, schema explorers, and autocomplete features, SQLTools is perfect for rapid database exploration and query optimization.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=mtxr.sqltools"><strong>here</strong></a></p><h3>10. Cloud Code (1.5M+ downloads)</h3><p><em>For GCP Cloud Development &amp; Gemini AI</em></p><p>The <strong>Cloud Code</strong> extension brings the power of Google Cloud and Gemini to help you build applications faster and easier than ever before. It is like a personal assistant for cloud-native development in VS Code. It helps you work seamlessly with Kubernetes, Cloud Run, and Google Cloud APIs.</p><p>Whether you’re deploying apps, writing YAML configurations, or debugging in the cloud, it’s got your back with smart tools and smooth workflows. It’s perfect for data engineers managing cloud-based pipelines and services — no more hassle with cloud setups!</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=GoogleCloudTools.cloudcode"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NbBtsA5-nammWDqd.gif" /></figure><h3>11. Indent-Rainbow (9.1M+ downloads)</h3><p><em>For Cleaner, More Navigable Code</em></p><p>One of the challenges with working in Python or YAML is that complex files can become difficult to navigate. <strong>Indent-Rainbow</strong> adds color-coding to indentation levels, which makes it much easier to understand the structure of deeply nested files. Perfect for debugging or refactoring complex data transformation scripts.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=oderwat.indent-rainbow"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*D6d9HFVZPKxoRPJv.png" /></figure><h3>12. SQLite Viewer (1.5M+ downloads)</h3><p><em>For Quick Database Exploration</em></p><p>Sometimes, you just need to view an SQLite file quickly. Whether you’re prototyping or debugging an application, <strong>SQLite Viewer</strong> allows you to open and query SQLite databases directly in VS Code. This is especially useful when you want to check small datasets or test data ingestion workflows before they go into your main pipeline.</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=qwtel.sqlite-viewer"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LRqOYG-oWOAjlGz_.gif" /></figure><h3>13. Postman (1.2M+ downloads)</h3><p><em>For API Testing and Monitoring</em></p><p>When your data ingestion involves APIs — whether it’s pulling data from an external source or sending it to a downstream system — <strong>Postman</strong> is invaluable. This extension integrates API testing directly into VS Code, allowing you to send requests, monitor responses, and debug API calls in real-time. For data engineers building complex ETL pipelines, this is a must-have tool for ensuring your API integrations are working as expected.</p><p>Get extension <a href="https://marketplace.visualstudio.com/items?itemName=Postman.postman-for-vscode">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nciqs-sXBXIAE8PW.gif" /></figure><h3>14. GitLens (36M+ downloads)</h3><p><em>For Git Integration and Version Control</em></p><p>Version control is critical when you’re building complex data pipelines, and <strong>GitLens</strong> takes Git to the next level in VS Code. It provides detailed blame annotations</p><p><strong>Get extension </strong><a href="https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens"><strong>here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*NsdLWbE0LZcWtSJs.png" /></figure><p>PS if you have any questions, or would like something clarified, you can find me on <a href="https://www.linkedin.com/in/manojkumar-vadivel-206220a6/">LinkedIn.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fcc2e1b3c4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Iceberg Vs Delta Lake —  for your Data Lake house]]></title>
            <link>https://medium.com/@manojkumar.vadivel/apache-iceberg-vs-delta-lake-for-your-data-lake-house-e0fdc8b0d199?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0fdc8b0d199</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Mon, 11 Nov 2024 04:00:26 GMT</pubDate>
            <atom:updated>2024-11-11T04:00:26.668Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Iceberg Vs Delta Lake — for your Data Lake house</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D0dOU_YWM1jRYOcad7pSNw.png" /></figure><p>When it comes to big data lakehouse solutions, <strong>Apache Iceberg</strong> and <strong>Delta Lake</strong> are top contenders. Both are designed for large-scale data management, but each has its unique strengths when it comes to flexibility. Here’s an easy breakdown of where each one shines and where they differ. Let’s dive in</p><p><strong>Why does this choice matter?</strong> Choosing the right data format can impact everything from query speed to data governance and even the cost of your data pipeline. Iceberg and Delta Lake don’t just store data; they provide a foundation that determines how well your team can analyze, transform, and manage data across multiple platforms.</p><h3>1. Schema Evolution</h3><ul><li><strong>Iceberg</strong>: Flexible with schema changes, allowing renaming, reordering, and adding or dropping columns without disrupting your queries.</li><li><strong>Delta Lake</strong>: Supports adding columns but is more limited when it comes to complex changes.</li></ul><blockquote><strong>Winner:</strong> <strong>Iceberg</strong> 🏆 — it’s simply more adaptable for managing schema updates.</blockquote><h3>2. Partitioning</h3><ul><li><strong>Iceberg</strong>: Uses <strong>hidden partitioning</strong> that adapts over time, optimizing queries and making it easy to define custom partition transformations.</li><li><strong>Delta Lake</strong>: Relies on traditional static partitioning, which works well in many cases but can become more rigid over time.</li></ul><blockquote><strong>Winner:</strong> <strong>Iceberg </strong>🏆 — dynamic partitioning offers flexibility as your data grows.</blockquote><h3>3. Storage and Metadata Management</h3><ul><li><strong>Iceberg</strong>: Separates data from metadata, making it accessible across engines like Spark, Flink, and Trino.</li><li><strong>Delta Lake</strong>: Keeps metadata with data in _delta_log files, which is efficient for Spark but may slow down with very large datasets.</li></ul><blockquote><strong>Winner:</strong> <strong>Iceberg</strong> 🏆 — easier to integrate with multiple engines and storage options.</blockquote><h3>4. Multi-Engine Support</h3><ul><li><strong>Iceberg</strong>: Built for multi-engine environments and works well across Spark, Flink, Trino, and more.</li><li><strong>Delta Lake</strong>: Primarily Spark-focused, with some support from other engines like Presto.</li></ul><blockquote><strong>Winner:</strong> <strong>Iceberg </strong>🏆 — ideal if your environment uses more than just Spark.</blockquote><h3>5. Data Versioning and Time Travel</h3><ul><li><strong>Iceberg</strong>: Supports time travel and offers branching and tagging, useful for testing and complex workflows.</li><li><strong>Delta Lake</strong>: Provides time travel but lacks branching/tagging.</li></ul><blockquote><strong>Winner:</strong> <strong>Iceberg </strong>🏆 — branching/tagging adds flexibility for complex setups.</blockquote><h3>6. Concurrency Control and Isolation</h3><ul><li><strong>Iceberg</strong>: Uses flexible, optimistic concurrency control, suitable for multi-writer support across different engines.</li><li><strong>Delta Lake</strong>: Optimized for Spark but may be less adaptable for non-Spark environments.</li></ul><blockquote><strong>Winner:</strong> <strong>Tie</strong> — both offer solid concurrency control, though Iceberg’s broader compatibility may suit mixed environments better.</blockquote><h3>7. Governance &amp; Security (Unity Catalog Integration)</h3><ul><li><strong>Delta Lake</strong>: With Databricks, it integrates well with Unity Catalog for centralized access control and auditing.</li><li><strong>Iceberg</strong>: Databricks is adding Unity Catalog support for Iceberg, but outside of Databricks, governance is managed separately.</li></ul><blockquote><strong>Winner:</strong> <strong>Delta Lake </strong>🏆 — best for governance when using Databricks.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mZ5yrZalQIuEPts3.png" /></figure><h3>8. Adoption &amp; Community Support</h3><ul><li><strong>Iceberg</strong>: Strong open-source community and support from big names like Netflix, Apple, and AWS.</li><li><strong>Delta Lake</strong>: Also widely adopted, especially in Spark-heavy environments like Databricks.</li></ul><blockquote><strong>Winner:</strong> <strong>Tie</strong> — Iceberg is more versatile, but both are backed by solid communities.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iImZU4ySB3Brg7YVeeSMlQ.png" /></figure><h3>Flexibility Verdict</h3><p><strong>Iceberg</strong> is the go-to choice for teams needing cross-engine support, dynamic partitioning, and advanced schema management. It’s designed for mixed environments where flexibility is key.</p><p><strong>Delta Lake</strong> is a strong choice if you’re mainly using Spark or Databricks, especially when you want robust governance with Unity Catalog.</p><p><strong>Which one do you use in your data lakehouse, and how’s it working out? Drop your experience in the comments!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0fdc8b0d199" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker for Data Engineers — Boost your productivity]]></title>
            <link>https://medium.com/@manojkumar.vadivel/docker-for-data-engineers-boost-your-productivity-5678d575ab04?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/5678d575ab04</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Wed, 23 Oct 2024 03:20:17 GMT</pubDate>
            <atom:updated>2024-10-23T03:20:17.708Z</atom:updated>
            <content:encoded><![CDATA[<h3>Docker for Data Engineers — Boost your productivity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K6JOZYu9J17g9ZFO2BPdoQ.png" /></figure><p>As data engineers, managing complex environments can be a real challenge. My team used to rely on Python virtual environments to isolate dependencies, but this solution was far from perfect. We struggled with OS-specific issues, version conflicts, and inconsistencies, which often resulted in wasted time, bugs, and inefficient testing practices. It became common to push code to dev environments just to test it, a cumbersome and time-consuming process.</p><p>That’s when we decided to adopt Docker, and it transformed the way we work.</p><h3>The Challenges We Faced Before Docker</h3><ul><li><strong>Local Environment Setup</strong>: Setting up environments locally often turned into a nightmare. Whether it was issues with Python versions, library conflicts, or differences between operating systems (Mac, Linux, Windows), we found ourselves battling the same setup issues repeatedly.</li><li><strong>OS Incompatibility</strong>: Our team, spread across different locations, worked on various operating systems. Some members used Windows, while others worked on macOS. Keeping consistent environments across these different systems using traditional methods was a huge pain.</li><li><strong>Slow and Inefficient Testing</strong>: Testing on local machines was often inconsistent with production environments. This led to bugs being caught too late in the development cycle, which meant pushing code to dev environments just to test it, causing delays in shipping features.</li></ul><h3>How Docker Improved Our Workflow</h3><p>Switching to Docker was a <strong>game-changer</strong> for us.</p><ol><li><strong>Simplified Local Setup</strong>: Docker allowed us to easily spin up local environments that mirrored our production systems. We could run complex systems like <strong>Airflow</strong>, <strong>Spark</strong>, and <strong>PostgreSQL</strong> in isolated containers, replicating our cloud setups locally. Debugging became much faster and more efficient because the environment was identical to production.</li><li><strong>Cross-Platform Consistency</strong>: With Docker, we no longer had to worry about OS differences. Whether our engineers were on Windows, Linux, or macOS, the containers worked the same way across all platforms. This consistency saved us countless hours and eliminated a major source of frustration.</li><li><strong>Faster POCs</strong>: When we wanted to explore new tools or libraries, Docker made it easy. Instead of spending hours setting up new tools, we could spin up a containerized environment, try out the tool, and then discard it without any impact on our local machines. This sped up our experimentation and helped us make faster decisions.</li><li><strong>Efficient Database Testing</strong>: We often needed to test transformations and workflows on databases without touching production. Docker let us spin up lightweight database containers that mimicked production environments. This allowed us to safely test code, validate queries, and ensure our pipelines were correct before going live.</li><li><strong>Version Control with Containers</strong>: One of the best things about Docker is the way it handles version control for environments. We often work on multiple projects, each requiring different versions of the same library or tool. Instead of managing multiple environments, each with its own Python virtual environment, Docker allowed us to isolate these dependencies in separate containers. This kept everything organized and conflict-free.</li></ol><h3>What is Docker? The Basics</h3><p>Docker is a platform that allows developers and engineers to <strong>package applications</strong> and all their dependencies into a lightweight, portable container. These containers can be run consistently on any environment, from a developer’s local machine to a remote server in the cloud, without worrying about compatibility issues.</p><p>A <strong>container</strong> is like a mini-computer that packages everything your project needs — code, libraries, configurations, and even the operating system — in a lightweight, portable format. This means that wherever you run the container, it behaves the same way.</p><p>Unlike virtual machines, which require an entire OS to be installed for each instance, containers share the host system’s OS kernel, making them much more efficient and faster to start up.</p><p>In short, Docker containers allow you to run applications reliably across different computing environments by packaging everything the application needs in one place.</p><h3>Where to Start with Docker</h3><p>If you’re new to Docker, here’s a simple starting point:</p><ol><li><strong>Install Docker</strong>:</li></ol><p>Docker Desktop is available for macOS, Windows, and Linux. It’s the easiest way to get started with Docker.</p><p><strong>2. Learn the Basics</strong>:</p><ul><li>Understand Docker’s core concepts: <strong>images</strong>, <strong>containers</strong>, <strong>Dockerfile</strong>, and <strong>volumes</strong>. A <strong>Docker image</strong> is like a blueprint of your app, while a <strong>container</strong> is the running instance of that image.</li><li>Create your first container with a simple example. For instance, you can pull an image of a Python environment:</li></ul><pre>docker run -it python:3.9 bash</pre><ul><li>This will create a container with Python 3.9, and you can run Python commands inside it.</li></ul><p><strong>3. Write Your First Dockerfile</strong>:</p><p>A Dockerfile is a script with instructions for Docker to build an image. Start by containerizing a small Python script. Here’s a basic example of a Dockerfile for a Python app:</p><pre># Use an official Python runtime as a parent image<br>FROM python:3.9<br><br># Set the working directory in the container<br>WORKDIR /usr/src/app<br><br># Copy the current directory contents into the container<br>COPY . .<br><br># Install any needed packages specified in requirements.txt<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Make port 80 available to the world outside this container<br>EXPOSE 80<br><br># Define environment variable<br>ENV NAME World<br><br># Run the application<br>CMD [&quot;python&quot;, &quot;./your_script.py&quot;]</pre><p><strong>4. Explore Docker Hub</strong>:</p><pre>docker pull apache/airflow</pre><p>This saves you from building everything from scratch and gets you started faster.</p><p><strong>5. Use Docker Compose</strong>:</p><p>Once you’re comfortable with basic containers, explore <strong>Docker Compose</strong>. This tool lets you manage multi-container setups (e.g., running a database, a web server, and a data processing tool together). For example, if you want to run Airflow, PostgreSQL, and Redis locally, a docker-compose.yml file makes it easy to configure and launch all services at once.</p><p><strong>6. Hands-on Practice</strong>:</p><p>Try containerizing one of your data projects. Start small by containerizing a Python script or a database, then gradually move towards more complex systems like Airflow or Spark.</p><h3>Get started with Docker for various data engineering tasks</h3><p>Here are some useful GitHub repositories with sample Dockerfiles that can help you get started with Docker for various data engineering tasks</p><h4>1. Dockerized Apache Airflow</h4><ul><li><strong>Repository</strong>: <a href="https://github.com/puckel/docker-airflow">puckel/docker-airflow</a></li><li><strong>Purpose</strong>: A widely used repository for setting up Airflow using Docker. This repo provides a ready-to-use Dockerfile and docker-compose.yml file for running Apache Airflow with minimal setup.</li><li><strong>Use Case</strong>: Run Apache Airflow locally for orchestrating your data pipelines, test DAGs, and debug issues before deploying them to a production environment.</li></ul><h4>2. Docker for Apache Spark</h4><ul><li><strong>Repository</strong>: <a href="https://github.com/bitnami/bitnami-docker-spark">bitnami/bitnami-docker-spark</a></li><li><strong>Purpose</strong>: This repository offers Docker images for Apache Spark, allowing you to deploy and run Spark clusters quickly.</li><li><strong>Use Case</strong>: Run distributed data processing tasks on Spark locally, simulating big data jobs for testing, development, and POCs.</li></ul><h4>3. Dockerized PostgreSQL for Data Engineering</h4><ul><li><strong>Repository</strong>: <a href="https://github.com/docker-library/postgres">docker-library/postgres</a></li><li><strong>Purpose</strong>: Official Docker image for PostgreSQL, providing a lightweight database container.</li><li><strong>Use Case</strong>: Spin up PostgreSQL instances locally for testing your data pipelines, ETL jobs, or database integrations in a production-like environment.</li></ul><h4>4. Data Engineering Tools in Docker (Python, Pandas, Jupyter)</h4><ul><li><strong>Repository</strong>: <a href="https://github.com/jupyter/docker-stacks">jupyter/docker-stacks</a></li><li><strong>Purpose</strong>: A collection of Docker images for data science and data engineering, including Jupyter notebooks, Python, Pandas, and other libraries.</li><li><strong>Use Case</strong>: Start with Jupyter Notebooks pre-configured with data engineering libraries. You can use it to prototype data analysis and ETL workflows locally.</li></ul><h4>5. Kafka Docker Setup</h4><ul><li><strong>Repository</strong>: <a href="https://github.com/wurstmeister/kafka-docker">wurstmeister/kafka-docker</a></li><li><strong>Purpose</strong>: Docker setup for Apache Kafka.</li><li><strong>Use Case</strong>: Set up a Kafka cluster locally for streaming data pipelines, testing message queues, and processing large-scale data flows.</li></ul><h4>6. Dockerized ELK Stack (Elasticsearch, Logstash, Kibana)</h4><ul><li><strong>Repository</strong>: <a href="https://github.com/deviantony/docker-elk">deviantony/docker-elk</a></li><li><strong>Purpose</strong>: Provides a simple way to set up the ELK stack with Docker and Docker Compose.</li><li><strong>Use Case</strong>: Use Elasticsearch for indexing, Logstash for ETL, and Kibana for visualizing data, all containerized for data engineering purposes.</li></ul><h3>Conclusion</h3><p>Docker has proven to be an indispensable tool for our team. It brought consistency, efficiency, and scalability to our workflows, making environment setup, testing, and debugging a breeze. With Docker, we can work faster, collaborate more effectively, and avoid the headaches that come with managing complex data engineering environments.</p><p>If you’re a data engineer still relying on traditional methods like virtual environments, it’s time to consider Docker. It’s a skill that will not only make your job easier but also improve the overall productivity of your team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5678d575ab04" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s simplify some common terms in DSA: Practical Approach]]></title>
            <link>https://medium.com/@manojkumar.vadivel/lets-simplify-some-common-terms-in-dsa-practical-approach-3c3e75e38909?source=rss-4c8679babe18------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c3e75e38909</guid>
            <category><![CDATA[data-structures]]></category>
            <category><![CDATA[data-structure-algorithm]]></category>
            <category><![CDATA[big-o-notation]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Manojkumar Vadivel]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 04:23:46 GMT</pubDate>
            <atom:updated>2025-12-14T07:59:43.456Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NaAAMBOFHOH8uWB7.jpg" /></figure><p>This article explains a simple, beginner-friendly way to understand time and space complexity using real-world examples</p><h4>Algorithm:</h4><p>The term ‘𝗔𝗹𝗴𝗼𝗿𝗶𝘁𝗵𝗺’ may sound fancy, but any code that accomplishes a task is technically an algorithm. You can write anything to solve a particular problem, but what matters is how many steps your code takes to solve the problem.</p><p>So How to measure my Algorithm’s efficiency?: looking at two main things</p><ol><li><strong><em>Speed (Time complexity) : </em></strong>How many steps/action to take to solve problem</li><li><strong><em>Memory usage (Space complexity): </em></strong>How much memory does the algorithm need to solve the problem?</li></ol><h4><strong><em>How do I measure my code’s speed (Time complexity) ?</em></strong></h4><p>We should always measure the efficiency of code based on the number of computational steps it takes to solve the problem, rather than how much time it takes to finish.</p><p>Because execution time can vary depending on your system configuration. The code may run faster on a higher configuration machine, while the same code could take longer to finish on older, lower hardware systems. Therefore, the efficiency of your algorithm is best measured by the number of steps it takes to solve a problem, called as <strong>time complexity.</strong></p><p>Example: Let’s write a simple code to print even numbers between 1 and 1000</p><pre># Approach 1:<br>number = 1<br>while number &lt;= 1000:<br>    if (number % 2 == 0):<br>        print(number)<br>    number= number+1<br><br># Approach 2:<br>number = 2<br>while number &lt;= 1000:<br>    print(number)<br>    number = number + 2</pre><p>In the above example, both approaches will print even numbers, but what matters is the number of steps involved. Approach 1 will take 1000 steps, whereas approach 2 will take only 500 steps.</p><p>Is there any way to describe how the time or space needed for a task grows as the task gets bigger? Yes, using <strong>Big O notation</strong></p><h4>Big O notation:</h4><p>If someone asks how fast or slow your algorithm is, how will you denote the efficiency of your algorithm? Will you label it as a 500-step algorithm or a 1000-step algorithm?</p><p>So, If the number of steps is based on the size of the array, then the generic way to describe it is through <strong>Big O notation</strong>. It’s like a ‘shortcut’ way to describe how fast or slow an algorithm is</p><p><strong><em>For example,</em></strong> let’s say you have 1000 elements in your array, and you want to check if a particular element is present or not. How many steps does it take to find the element? In the best case, it may be found in the first element, but in the worst case, it may need to check all the elements</p><p>In Big O notation, we always consider the worst-case scenario. If “<strong>n</strong>” represents the number of elements in the array, then the <strong>time complexity</strong> of the above example is <strong>O(n)</strong>. This means it will take n steps (1000 steps) to solve the problem. If n increases, the number of steps also increases.</p><p>This is an example of linear search, and its time complexity is O(n)</p><p><strong><em>Example 2 (Binary search):</em></strong><em> </em>Now, consider a sorted array. Here, you don’t need to check each element individually. Instead, you can check the middle element and eliminate half of the array’s size since it’s sorted. i.e If the number you’re looking for is smaller than the middle number, you know it must be in the first half of the list. So, you can ignore the second half.</p><p>So if you have 1000 elements, it takes only 10 steps to find the element in Binary search, whereas in linear search, 1000 operations are required. This is the power of binary search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RV46zNfNzO3g98LltfWwfQ.png" /></figure><p>In binary search, the number of steps increases logarithmically with the size of the array. This means that the number of steps increases by approximately 1 for every doubling of the array size.</p><p>For example:</p><ul><li>If the array size is 8, binary search may take up to 3 steps (log₂(8) = 3).</li><li>If the array size is 16, binary search may take up to 4 steps (log₂(16) = 4).</li><li>If the array size is 32, binary search may take up to 5 steps (log₂(32) = 5).</li></ul><p>So the Time complexity of <strong>Binary search is O(log n)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/0*BrYiKU-mPYo69ON1" /></figure><h4>Space Complexity:</h4><p>Space complexity refers to <strong>how much memory</strong> an algorithm needs to complete its task. If you have n elements then how much units of memory will the algorithm consume. This also denoted by <strong>Big O notation</strong> similar to Time complexity</p><p>Sometimes, we may need to use additional memory to improve the speed of the algorithm. For example, storing intermediate results in temporary variables can reduce the number of steps and improve the time complexity in some use cases.</p><p>𝙀𝙭𝙖𝙢𝙥𝙡𝙚: If you have ’n’ elements in your array and you always duplicate your input to perform an operation, then the additional memory required is equal to the size of your array — O(n). However, if you utilize only one temporary variable regardless of the size of the input array, then the space complexity is O(1)</p><p>Example: Convert each strings into smaller cases</p><pre>name_list = [&quot;Manoj&quot;, &quot;Alex&quot;, &quot;John&quot;, &quot;Tom&quot;]<br><br># Approach 1<br>new_name_list = []<br>for name in name_list:<br>    new_name_list.append(name.lower())<br>print(new_name_list)<br><br><br># Approach 2<br>for index, name in enumerate(name_list):<br>    name_list[index] = name.lower()<br>print(name_list)</pre><p>In the above example, Approach 1 involves creating a new array with the same size as the input array. Therefore, its space complexity is O(n).</p><p>However, in Approach 2, we directly update the original array without creating any additional memory. As a result, its space complexity is denoted as O(1), meaning the memory consumed by the algorithm is constant regardless of the input size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*7JlXRyVi_giKBQ5Ou6E3ow.png" /></figure><p>In ideal world, we would always use algorithm that are both time and space complexity, However there are situations we can compromise memory over speed or speed over memory.</p><p>Happy Learning !!</p><p>PS if you have any questions, or would like something clarified, you can find me on <a href="https://www.linkedin.com/in/manojkumar-vadivel-206220a6/">LinkedIn.</a></p><blockquote><em>References</em>:</blockquote><blockquote>A Common-Sense Guide to Data Structures and Algorithms, Second Edition, 2nd Edition by Jay Wengrow</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c3e75e38909" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>