<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Usmaan Rifkhan on Medium]]></title>
        <description><![CDATA[Stories by Usmaan Rifkhan on Medium]]></description>
        <link>https://medium.com/@usmaanrifkhan?source=rss-4e9691c34421------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sLvibhCUHcZwCXwRsObBJQ.jpeg</url>
            <title>Stories by Usmaan Rifkhan on Medium</title>
            <link>https://medium.com/@usmaanrifkhan?source=rss-4e9691c34421------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:52:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@usmaanrifkhan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Snowflake External OAuth Security Integration: How Authentication Works Between Snowflake and…]]></title>
            <link>https://medium.com/@usmaanrifkhan/snowflake-external-oauth-security-integration-how-authentication-works-between-snowflake-and-d503d342c2e5?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/d503d342c2e5</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[oauth2]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 09:45:16 GMT</pubDate>
            <atom:updated>2026-07-09T03:00:52.268Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vFfBYi03xVW6sSi6_cKjYA.png" /></figure><h3>Snowflake External OAuth Security Integration: How Authentication Works Between Snowflake and External Identity Providers</h3><p>When building secure applications with Snowflake, authentication and authorization are two important parts of the security layer. Authentication answers: “Who are you?” and Authorization answers:“What are you allowed to access?”</p><p>In many organizations, user identities are not managed directly inside Snowflake. Instead, companies already have identity providers such as AWS Cognito, Okta, or Microsoft Entra ID handling user authentication.</p><p>Snowflake Security Integration allows Snowflake to establish a trusted connection with these external identity providers. Through External OAuth, Snowflake can validate OAuth tokens generated by an external authentication system and use the information from those tokens to create a secure Snowflake session.</p><p>The basic flow looks like this:</p><pre>User<br> |<br> |<br> v<br>External Identity Provider<br>(AWS Cognito / Okta / Azure AD)<br> |<br> |<br> | OAuth JWT Token<br> |<br> v<br>Snowflake External OAuth Security Integration<br> |<br> |<br> | Validate Identity<br> | Map Snowflake User<br> | Apply Role Context (if configured)<br> |<br> v<br>Snowflake Session</pre><p>The Security Integration works as a trust layer between Snowflake and the external identity provider. Snowflake does not manage the user’s password or authentication process. Instead, it verifies the token issued by the trusted external system.</p><p>For authentication, Snowflake validates details such as the token issuer, token signature, audience, and user identity mapping.</p><p>For authorization, OAuth claims or scopes can be used to provide role information when configured. The final access control on Snowflake objects is still managed through Snowflake Role-Based Access Control (RBAC).</p><p>In my practical experience, I used External OAuth Security Integration with AWS Cognito, where Cognito handled the user authentication flow and Snowflake validated the generated OAuth token before allowing access to Snowflake.</p><h3>How External OAuth Security Integration Works</h3><p>To understand External OAuth Security Integration, let’s first understand what happens behind the scenes when a user connects to Snowflake through an external identity provider.</p><p>The authentication process does not start in Snowflake. Instead, it starts with the external identity provider, such as AWS Cognito, Okta, or Microsoft Entra ID. After the user is successfully authenticated, the identity provider generates an OAuth JWT token and sends it to the application.</p><p>The application then uses this token when establishing a connection with Snowflake.</p><p>Snowflake does not simply trust every token it receives. It validates the token using the configuration defined in the Security Integration. During this validation process, Snowflake verifies several important details, such as:</p><ul><li>The token was issued by the trusted Identity Provider.</li><li>The token signature is valid.</li><li>The token was generated for the expected application (Audience).</li><li>The token belongs to a valid Snowflake user by mapping a claim in the token to a Snowflake user.</li></ul><p>If all these validations are successful, Snowflake creates a session for that user.</p><pre>User<br> |<br> | Login<br> |<br> v<br>External Identity Provider<br> |<br> | Authenticate User<br> | Generate OAuth JWT Token<br> |<br> v<br>Application / Middle Layer<br> |<br> | Send JWT Token<br> |<br> v<br>Snowflake External OAuth Security Integration<br> |<br> | Verify Issuer<br> | Verify Signature<br> | Verify Audience<br> | Map User<br> |<br> v<br>Snowflake Session</pre><p>At this stage, the Security Integration has completed the authentication process by verifying the user’s identity.</p><p>Depending on the application design, the OAuth token can also carry role or scope information. This allows Snowflake to participate in role selection during session creation. However, the actual permissions to access databases, schemas, tables, and other Snowflake objects are still controlled by Snowflake RBAC.</p><h3>My Practical Implementation with AWS Cognito</h3><p>To understand this better, let’s look at how I implemented External OAuth Security Integration in one of my projects.</p><p>In our environment, AWS Cognito is used as the external Identity Provider (IdP). The platform team manages Cognito, user authentication, and the application layer, while my responsibility as the Snowflake Data Engineer was to configure Snowflake so that it could trust and validate the tokens generated by Cognito.</p><p>The Security Integration I created was similar to the following:</p><pre>CREATE OR REPLACE SECURITY INTEGRATION cognito_oauth<br>TYPE = EXTERNAL_OAUTH<br>ENABLED = TRUE<br>EXTERNAL_OAUTH_TYPE = CUSTOM<br>EXTERNAL_OAUTH_ISSUER = &#39;https://cognito-idp.ap-southeast-1.amazonaws.com/...&#39;<br>EXTERNAL_OAUTH_AUDIENCE_LIST = (&#39;xxxxxxxx&#39;)<br>EXTERNAL_OAUTH_JWS_KEYS_URL = &#39;https://cognito-idp.ap-southeast-1.amazonaws.com/.../.well-known/jwks.json&#39;<br>EXTERNAL_OAUTH_TOKEN_USER_MAPPING_CLAIM = &#39;cognito:username&#39;<br>EXTERNAL_OAUTH_SNOWFLAKE_USER_MAPPING_ATTRIBUTE = &#39;LOGIN_NAME&#39;;</pre><p>Although the SQL statement is short, every parameter has a specific responsibility.</p><blockquote>TYPE = EXTERNAL_OAUTH tells Snowflake that authentication will be handled by an external OAuth 2.0 authorization server instead of Snowflake OAuth.</blockquote><blockquote>ENABLED = TRUE enables the Security Integration so Snowflake can use it to validate incoming OAuth tokens.</blockquote><blockquote>EXTERNAL_OAUTH_TYPE = CUSTOM specifies that the Identity Provider is a custom OAuth provider. In my case, this is AWS Cognito.</blockquote><blockquote>EXTERNAL_OAUTH_ISSUER identifies the trusted Identity Provider. Whenever Snowflake receives a JWT token, it verifies that the token was issued by this Cognito User Pool.</blockquote><blockquote>EXTERNAL_OAUTH_JWS_KEYS_URL points to the public keys published by Cognito. Snowflake uses these keys to validate the JWT signature and ensure the token has not been modified.</blockquote><blockquote>EXTERNAL_OAUTH_AUDIENCE_LIST validates that the token was created for the expected application.</blockquote><blockquote>EXTERNAL_OAUTH_TOKEN_USER_MAPPING_CLAIM tells Snowflake which claim from the JWT should be used to identify the Snowflake user.</blockquote><blockquote>EXTERNAL_OAUTH_SNOWFLAKE_USER_MAPPING_ATTRIBUTE = LOGIN_NAME tells Snowflake to compare that claim with the LOGIN_NAME of Snowflake users.</blockquote><p>For example, if the JWT token contains:</p><pre>{<br>    &quot;cognito:username&quot;: &quot;safnas&quot;<br>}</pre><p>Snowflake searches for a user whose LOGIN_NAME is SAFNAS. If a matching user is found and all token validations succeed, Snowflake creates a session for that user.</p><blockquote>EXTERNAL_OAUTH_ANY_ROLE_MODE controls whether the OAuth client or user can switch to roles that are not explicitly defined in the OAuth token. In my current implementation this is set to ENABLE, which allows role switching when permitted by Snowflake.</blockquote><blockquote>Depending on your security requirements, other useful parameters can also be configured.</blockquote><blockquote>EXTERNAL_OAUTH_BLOCKED_ROLES_LIST prevents applications from creating sessions using sensitive roles such as ACCOUNTADMIN or SECURITYADMIN.</blockquote><blockquote>EXTERNAL_OAUTH_ALLOWED_ROLES_LIST can be used to explicitly restrict which Snowflake roles an application is allowed to use.</blockquote><blockquote>NETWORK_POLICY allows the Security Integration to be associated with a Snowflake Network Policy, adding another layer of security by controlling which client networks are allowed to connect.</blockquote><p>At the moment, my implementation is primarily focused on authentication. The application sends the JWT token generated by AWS Cognito, and Snowflake validates the token before creating a session. No Snowflake username or password is sent from the application.</p><h3>Authorization</h3><p>At the moment, our implementation mainly uses External OAuth Security Integration for authentication. At this stage, I am not passing Snowflake roles from the OAuth token. However, Snowflake still applies its own security rules while creating the session. For example, if the user’s default role is ACCOUNTADMIN, Snowflake checks whether that role is allowed for External OAuth sessions. Since highly privileged roles such as ACCOUNTADMIN, ORGADMIN, and SECURITYADMIN are blocked by default for External OAuth, Snowflake prevents the session from being created with those roles.</p><p>Our organization already has multiple security layers managed by the platform team, so authorization is handled outside of Snowflake in several places. Because of that, implementing role-based authorization through External OAuth has not been a requirement yet.</p><p>However, the current architecture is already prepared for that enhancement.</p><p>When a new user is created, the platform team first creates the user in AWS Cognito. During that process, they assign the appropriate Snowflake role to the user. Instead of manually typing role names, the platform application can provide a dropdown containing the available Snowflake roles. Those roles can be maintained from the Snowflake side, ensuring that only valid Snowflake roles are assigned.</p><p>The flow would look like this:</p><pre>Snowflake<br>    |<br>    | Available Roles<br>    |<br>    v<br>Platform User Management<br>    |<br>    | Select Role<br>    |<br>    v<br>Create Cognito User<br>    |<br>    | Username + Snowflake Role<br>    |<br>    v<br>Generate OAuth Access Token<br>    |<br>    | scope / scp contains requested role<br>    |<br>    v<br>Snowflake External OAuth Security Integration<br>    |<br>    | Map role during session creation<br>    |<br>    v<br>Snowflake Session</pre><p>In this approach, the access token would contain the requested Snowflake role through the standard OAuth scope or scp claim. Snowflake External OAuth Security Integration can use EXTERNAL_OAUTH_SCOPE_MAPPING_ATTRIBUTE to read that claim and use it during session creation.</p><p>This allows authentication and role selection to remain centralized while still relying on Snowflake RBAC to enforce permissions on databases, schemas, tables, views, and other objects.</p><h3>External OAuth vs Snowflake OAuth</h3><p>While working with Snowflake Security Integrations, it is important to understand that Snowflake supports different types of OAuth integrations depending on the authentication architecture.</p><p>In my implementation, I used <strong>External OAuth</strong>, where an external Identity Provider such as AWS Cognito is responsible for authenticating users and issuing OAuth tokens. Snowflake does not manage the authentication flow. Instead, Snowflake trusts the external OAuth provider, validates the token, and creates a Snowflake session based on the configured Security Integration.</p><p>The flow is:</p><pre>External Identity Provider<br>(AWS Cognito / Okta / Azure AD)<br>        |<br>        |<br>        | OAuth Token<br>        |<br>        v<br>Snowflake External OAuth Security Integration<br>        |<br>        |<br>        v<br>Snowflake Session</pre><p>Snowflake also supports another type called <strong>Snowflake OAuth</strong>.</p><p>Snowflake OAuth works differently. In this approach, Snowflake acts as the OAuth authorization server. Clients that support OAuth can redirect users to Snowflake’s authorization page, and Snowflake generates access tokens (and optionally refresh tokens) that clients can use to access Snowflake.</p><p>The flow is:</p><pre>Client Application<br>        |<br>        |<br>        v<br>Snowflake OAuth Authorization<br>        |<br>        |<br>        v<br>Access Token<br>        |<br>        |<br>        v<br>Snowflake</pre><p>The main difference is where the identity management happens.</p><p>With External OAuth:</p><pre>External Identity Provider<br>        |<br>        |<br>        v<br>Snowflake trusts and validates the token</pre><p>With Snowflake OAuth:</p><pre>Snowflake<br>        |<br>        |<br>        v<br>Handles OAuth authorization and token generation</pre><p>The choice depends on the organization’s security architecture.</p><p>If an organization already has an identity provider managing users, applications, and authentication flows, External OAuth is a common approach because Snowflake can integrate with the existing security ecosystem.</p><p>If an application needs Snowflake itself to provide the OAuth authorization flow, Snowflake OAuth can be used.</p><p>In enterprise environments, both approaches can be useful depending on the application requirements and security model.</p><p>Snowflake External OAuth Security Integration provides a way to connect Snowflake with external identity providers and establish a trusted authentication flow. In my experience, I used AWS Cognito with Snowflake to validate user identities and create secure Snowflake sessions without managing separate Snowflake credentials. In the future, this same setup can be extended for authorization use cases by using OAuth scopes and Snowflake roles together with Snowflake RBAC.</p><p>Ref:<br><a href="https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-external">https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-external</a> <br><a href="https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-snowflake">https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-snowflake</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d503d342c2e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building the Control Layer for AI Agents: Harness Engineering Explained]]></title>
            <link>https://medium.com/@usmaanrifkhan/building-the-control-layer-for-ai-agents-harness-engineering-explained-0198ed4742c8?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/0198ed4742c8</guid>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[harness-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 08:08:17 GMT</pubDate>
            <atom:updated>2026-06-30T10:40:23.335Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*po598nQPTdzQXvVaWnN5RA.png" /></figure><h3>Building the Control Layer for AI Agents: Harness Engineering</h3><p>The evolution has roughly been:</p><blockquote><strong><em>Prompt Engineering → Context Engineering → Harness Engineering</em></strong></blockquote><p>Each step expands <strong>what engineers are responsible for</strong>.</p><h3>Prompt Engineering (2023–2024)</h3><p>Initially everyone focused on writing better prompts.</p><p>Example:</p><pre>You are a senior Python developer.<br>Write clean code.<br>Think step by step.</pre><p>The assumption was:</p><blockquote><em>Better prompt = Better output</em></blockquote><p>This worked for simple tasks but failed for complex workflows.</p><h3>Context Engineering (2025)</h3><p>Then people realized: The prompt is only a tiny part of what the model sees. The real challenge became: How do we provide the right information to the model? That’s Context Engineering.</p><p><strong>Context Engineering is the practice of designing, selecting, organizing, and continuously managing everything placed into an LLM’s context window so the model has exactly the information it needs — no more and no less.</strong></p><p>Instead of only writing prompts, you engineer the entire context, including the <strong>system prompt, user prompt, conversation history, RAG documents, memory, retrieved database rows, tool descriptions, API schemas, examples, and instructions</strong>.</p><p>Imagine you ask an LLM:</p><blockquote><strong><em>“Generate a customer report.”</em></strong></blockquote><p>If you only provide that prompt, the model has very little information to work with. In a context-engineered system, however, the LLM receives much more than the user’s request. It is given the <strong>system prompt, user request, previous conversation history, customer database records, knowledge base articles, company policies, examples, tool definitions, memory, and any retrieved documents</strong>.</p><p>Context engineering is the process of deciding <strong>what information should be included, what should be excluded, in what order the information should be presented, how it should be compressed, and when additional information should be retrieved</strong> before sending everything to the model.</p><p>For example, instead of simply sending <strong>“Summarize customer activity,”</strong> a context-engineered application might first retrieve the customer’s <strong>last 12 orders, support tickets, CRM notes, previous summaries, and personal preferences</strong>, and then provide all of that context to the LLM. The model produces a much better answer because it has better context — not because it has become smarter.</p><p>Context engineering became essential because LLMs do not permanently know your company’s data. They only know the information provided during inference. As a result, the quality of <strong>retrieval, memory, and context selection</strong> often has a greater impact on the output than the exact wording of the prompt.</p><h3>So why this context engineering not enough?</h3><p>However, people eventually discovered that even with perfect context, AI agents could still choose the wrong tool, enter infinite loops, misuse APIs, accidentally delete files, forget long-running tasks, fail midway through workflows, or hallucinate results. This led to an important realization: <strong>context alone is not enough — you must engineer the entire system surrounding the model.</strong> That broader discipline is known as <strong>Harness Engineering</strong>.</p><h3>Harness Engineering (2026)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtMb9VVAjx_cFWNs4qxkNA.png" /></figure><p>If Context Engineering is about giving the AI the right information, Harness Engineering is about giving the AI the right boundaries and infrastructure.</p><p>As AI agents have become highly autonomous — capable of writing code, browsing the web, and executing commands — a raw language model is no longer enough. Left to its own devices, a raw model will hallucinate tools, get stuck in infinite loops, or execute destructive commands (like deleting a database).</p><p>If context engineering is about providing the right ecosystem of data, Harness Engineering is about building the definitive operational wrapper around the model. A raw LLM possesses reasoning capabilities but lacks inherent hooks to a file system, version control, or execution environment.</p><p>The core architectural formula for production AI is defined as:Agent = Model + Harness. The harness acts as the infrastructure that translates raw LLM reasoning into safe, predictable, and highly targeted execution. This discipline operates on two primary levels: single-session enhancement and multi-session orchestration.</p><h3>Level 1: The Single-Session AI Layer</h3><p>When you select an out-of-the-box AI tool (like Claude Code or Codex), you are choosing a pre-built base harness engineered by a vendor. However, true harness engineering occurs at the developer-controlled <strong>AI Layer</strong> that wraps around these tools. This layer governs behavior through six core components:</p><p>• <strong>Global Rules:</strong> Persistent instructions, patterns, and markdown constraints (e.g., an agents.md file) that the model must follow.<br> • <strong>Skills:</strong> Programmatic workflows dictating how the agent plans, implements, and validates its work.<br> • <strong>MCP Servers &amp; Capabilities:</strong> The custom tools and external APIs exposed to the agent.<br> • <strong>Codebase Searching:</strong> Advanced context indexing tools like Language Server Protocols (LSP) or knowledge graphs.<br> • <strong>Hooks:</strong> Deterministic script triggers executed before or after actions (e.g., a <em>pre-tool hook</em> to block destructive commands, or a <em>validation hook</em> to run test suites after an edit).<br> • <strong>Sub-Agents:</strong> Isolated auxiliary instances assigned to highly specialized micro-tasks.</p><h3>Level 2: Multi-Session Orchestration (The Peak Evolution)</h3><p>The true evolutionary leap of harness engineering lies in automated control and orchestration. Shoving a massive, complex requirement document into a single LLM session overwhelms the context window, spikes token costs, and degrades performance.<br> Advanced harness engineering automates the execution of <strong>multi-session workflows</strong>. Instead of a developer manually passing files back and forth, a top-level script (written in Python or Bash) programmatically strings separate sessions together:</p><pre>[User Input / PRD] <br>       │<br>       ▼<br> ┌───────────┐      ┌──────────────┐      ┌──────────────┐<br> │ Session 1 │ ───&gt; │  Session 2   │ ───&gt; │  Session 3   │───&gt;[Pull Request]<br> │ (Planning)│ Plan │(Implementing)│ Code │ (Validation) │<br> └───────────┘      └──────────────┘      └──────────────┘</pre><p>A prominent example of this pattern is the <strong>Ralph Loop</strong>. In a Ralph loop, the harness splits a broad scope of work into sequential checklist items, spins up dedicated sessions to handle each item one by one, logs progress, and utilizes an exit condition (such as a generated done.txt flag) only when the agent and validation hooks fully satisfy the criteria</p><p>A Ralph Loop is the ultimate example of multi-session agent orchestration. Named after its creator, Jeffrey Huntley, it is not a new AI model — it is simply a highly effective script (usually written in Python or Bash) that acts as an automated project manager for your AI coding agents.</p><p>Artical highlighted this as the “peak” of harness engineering because it solves the biggest limitation of current AI models: the context window. If you hand a massive Product Requirements Document (PRD) to a single instance of Claude or ChatGPT, it will eventually get overwhelmed, forget earlier instructions, or run out of tokens. The Ralph Loop prevents this by breaking the work into a continuous, automated relay race.</p><p>So, <strong>Level 1</strong> is about taking an existing AI tool and putting guardrails on it so it behaves perfectly in a single conversation. <strong>Level 2</strong> is about building the automated script that strings dozens of those AI conversations together to build a complete software project while you sleep.</p><p>Harness Engineering started with developers writing custom Bash scripts (like the original Ralph Loop) to control the AI from the outside. But the methodology was so successful that in 2026, AI companies like Anthropic absorbed those exact Level 2 orchestration techniques directly into the core tools themselves.</p><h3>What does a harness include?</h3><p>A harness is the entire system that surrounds an LLM and enables it to perform tasks reliably. While <strong>Context Engineering</strong> is one part of the harness, a complete harness typically also includes <strong>tool calling, MCP servers, memory, workflow orchestration, permission checks, sandboxed execution, retry logic, error recovery, test execution, code verification, human approval steps, logging, observability, evaluation, audit trails, and multi-agent coordination</strong>.</p><p><strong>Example</strong></p><p>Suppose you ask an AI agent:</p><blockquote><strong><em>“Upgrade our backend.”</em></strong></blockquote><p>A mature harness doesn’t simply send that prompt to the model. Instead, it may <strong>read the repository, retrieve the project’s coding standards, create an implementation plan, write the code, run unit tests, automatically fix failures, perform security scans, generate documentation, request human approval, and merge the changes only after all checks have passed</strong>.</p><p>Throughout this process, <strong>the LLM performs the reasoning</strong>, while <strong>the harness manages the workflow, tools, validation, safety, and execution</strong>.</p><p><strong>Simple analogy</strong></p><p>Think of a race car:</p><ul><li><strong>Prompt Engineering</strong> is the <strong>driver</strong>, deciding what to do.</li><li><strong>Context Engineering</strong> is the <strong>dashboard and navigation system</strong>, providing the driver with the right information.</li><li><strong>Harness Engineering</strong> is <strong>everything else that makes the race possible</strong>, including the seat belt, steering, brakes, engine monitoring, pit crew, fuel system, telemetry, safety systems, and communication.</li></ul><p>Without all of these supporting systems, <strong>even the best driver cannot race reliably</strong>. Similarly, even the most capable LLM cannot consistently complete complex tasks without a well-designed harness.</p><h3>The relationship:</h3><pre>Harness Engineering<br>│<br>├── Context Engineering<br>│      ├── RAG<br>│      ├── Memory<br>│      ├── Prompt Design<br>│      └── Retrieval<br>│<br>├── Tool Management<br>├── Permissions<br>├── Verification<br>├── Observability<br>├── Recovery<br>├── Evaluation<br>├── Agent Orchestration<br>└── Runtime Environment</pre><h3>The Skill Issue Reframe</h3><p>There’s a pattern many engineers fall into. The agent does something wrong, the engineer blames the model, and the issue gets filed under <em>“We’ll wait for the next version.”</em> Harness engineering rejects this mindset. Instead of seeing agent failures as model limitations, it treats every mistake as a signal that the system can be improved.</p><p>This is the idea behind <strong>System Evolution</strong>. Every agent error is a visible design gap in the harness, and every gap is an opportunity to make the system more reliable. If the agent doesn’t know a convention, add it to <strong>AGENTS.md</strong>. If it runs a destructive command, add a hook that blocks it. If it gets lost in a long, multi-step task, split the work into a planner and an executor. If it keeps marking broken code as “finished,” feed type checks and validation back into the execution loop.</p><p>The mindset changes from <em>“the model made a mistake”</em> to <em>“the harness allowed that mistake to happen.”</em> Each failure becomes feedback that helps evolve the system rather than something to blame on the model.</p><p>As HumanLayer puts it: <em>“It’s not a model problem. It’s a configuration problem.”</em> Harness engineering is what happens when you take that idea seriously , continuously improving the harness so the agent performs better over time.</p><h3>The ratchet: every mistake becomes a rule</h3><p>Harness engineering is built around a simple but powerful principle called <strong>the ratchet</strong>. Instead of treating an agent’s mistakes as isolated incidents, it treats every failure as a permanent signal to improve the harness. The goal isn’t to keep retrying until the agent gets it right — the goal is to make sure the same mistake is less likely to happen again.</p><p>Every failure should lead to an actionable improvement. If the agent ignores a coding convention, add that convention to <strong>AGENTS.md</strong>. If it runs a destructive command, create a hook that blocks it. If it struggles with a long, complex task, redesign the workflow by separating planning from execution. Rather than repeatedly correcting the agent, you strengthen the system that guides it.</p><p>Another important principle is <strong>traceability</strong>. Every rule, constraint, or guardrail in the harness should exist because of a real failure. You don’t add restrictions based on assumptions or “what if” scenarios. Instead, every line in <strong>AGENTS.md</strong>, every validation check, and every automated safeguard should be traceable to a specific mistake the agent made in the past.</p><p>This is why harness engineering is considered a <strong>discipline rather than a framework</strong>. You can’t simply download the perfect harness and expect it to work everywhere. Every team’s harness evolves from its own experiences, failures, and lessons learned. Over time, each mistake makes the harness stronger, more reliable, and better suited to the team’s codebase and workflow.</p><h3>The Bridge Analogy: Bridging the Gap to Autonomy</h3><p>On one side of the chasm, you have “What the model can do alone” — which is strictly reasoning and generating text. To achieve real-world utility, the AI Layer (the Harness) must build bridges to reach functional destinations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QwoGxuiRCb3zKwIOyE51jA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*4iEDdvxr0XCU-WEEYNGu6A.png" /></figure><p><em>Ref:</em></p><p><em>Article : </em><a href="https://addyosmani.com/blog/agent-harness-engineering/"><em>https://addyosmani.com/blog/agent-harness-engineering/</em></a></p><p><em>Repo: </em><a href="https://github.com/coleam00/harness-engineering-demo"><em>https://github.com/coleam00/harness-engineering-demo</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0198ed4742c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Cost Optimization Tips]]></title>
            <link>https://medium.com/@usmaanrifkhan/snowflake-cost-optimization-tips-from-real-project-experience-760689613518?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/760689613518</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Wed, 11 Mar 2026 04:56:35 GMT</pubDate>
            <atom:updated>2026-03-12T04:11:21.572Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ony_KuvyjIq3BqlDgG_6bw.png" /></figure><p>When we talk about Snowflake cost optimization, the first thing we need to understand is this: in most real projects, the main cost usually comes from compute, especially virtual warehouses. Snowflake’s cost guidance also focuses heavily on warehouse usage, query efficiency, storage, and monitoring because those are the biggest levers for controlling spend.</p><p>In my experience, warehouse optimization is the core heart of Snowflake cost control. For many workloads, small or medium warehouses are enough, and the biggest win often comes from configuring warehouse behavior properly instead of just reducing usage blindly. Snowflake recommends using features like auto-suspend and auto-resume so warehouses do not keep running when no work is happening.</p><h3>1) Warehouse optimization is the first place to look</h3><p>A virtual warehouse in Snowflake is the compute layer that runs queries, transformations, and data processing. Because billing is tied to warehouse usage, keeping warehouses active longer than necessary can quickly increase cost.​</p><p>That is why I always pay attention to a few settings first:</p><ul><li>Use a warehouse size that matches the workload; in many cases, Small or Medium is enough.</li><li>Set auto-suspend to a low value.</li><li>Keep auto-resume enabled.</li><li>Start with the warehouse initially suspended.​</li></ul><p>In my own cases, I usually set auto-suspend to 10 seconds for many warehouses, especially when the workload is intermittent. That way, the warehouse does its job and shuts down very quickly instead of sitting idle and burning credits. Snowflake supports this pattern directly through warehouse properties.​</p><p>Example:</p><pre>create warehouse GPS_WH<br>with<br>    warehouse_size=&#39;SMALL&#39;<br>    auto_suspend=10<br>    auto_resume=true<br>    initially_suspended=true<br>    comment=&#39;wh to gps testing of march&#39;;</pre><p>This is a simple setup, but it can make a real difference when you have multiple warehouses for testing, ETL and analytics. If a warehouse is active all day just because nobody configured suspend behavior properly, that is wasted spend. Snowflake specifically documents automatic suspension as a cost optimization best practice. (<a href="https://docs.snowflake.com/en/sql-reference/sql/create-warehouse">https://docs.snowflake.com/en/sql-reference/sql/create-warehouse</a>)</p><h3>2) Be careful with multi-cluster warehouses</h3><p>By default, a Snowflake warehouse uses a single cluster of compute resources, but Snowflake also supports multi-cluster warehouses for handling concurrency. With this setup, additional clusters can be added either statically or dynamically so more queries can run without waiting in queue.</p><p>This is useful, but it also means cost can go up fast if we enable more clusters without a real reason. If a warehouse is configured with minimum cluster 1 and maximum cluster 3, Snowflake can scale out to three clusters when concurrency increases. Snowflake explains that multi-cluster warehouses are designed for concurrency management, not as a default setting for every workload.​</p><p>For example, if you use three clusters of an X-Small warehouse at the same time, the cost is roughly multiplied by three compared to a single cluster. So even though the warehouse size looks extra small, the total spend can become much higher when scaling across clusters. The key point is simple: only use multi-cluster settings when the use case truly needs it, such as many concurrent users or overlapping workloads.​</p><p>So my practical advice is:</p><ul><li>Do not enable multi-cluster just because it is available.</li><li>First check whether the issue is really concurrency.</li><li>Use minimum and maximum cluster settings carefully.</li><li>Monitor whether the extra clusters are actually helping business queries.</li></ul><p>(<a href="https://docs.snowflake.com/en/user-guide/warehouses-multicluster">https://docs.snowflake.com/en/user-guide/warehouses-multicluster</a>)</p><h3>3) Use clustering keys properly for extraction and analytics workloads</h3><p>For many data engineering use cases, especially data extraction and analytics on large tables, clustering keys are not optional in practice — they can be very useful when queries repeatedly filter on the same columns. Snowflake explains clustering keys as a way to improve data pruning and query performance for large tables with common filtering patterns.​</p><p>In my experience, this matters a lot for extraction-heavy and analytics-heavy workloads. If teams frequently query large tables by a specific business key, device ID, customer ID, or time-based pattern, clustering can help Snowflake scan less unnecessary data and return results faster.​</p><p>For example, in my case I clustered the TBOX_MESSAGE_DATA table by TBOXID because many queries search for a specific bike’s data, such as one month of GPS or message history for a single device. Since this kind of lookup happens often, clustering on that column helps organize access more efficiently for those workloads.</p><p>Example:</p><pre>create or replace TABLE SOURCE_DATA.VEHICLE_DATA.TBOX_MESSAGE_DATA<br>cluster by (tboxid)<br>(<br>    TBOX_INTERNAL_BAT_VOLT NUMBER(38,0),<br>    SIDESTANDINFO NUMBER(38,0)<br>    -- ...<br>);<br><br>-- You can also add or update clustering later:<br>ALTER TABLE SOURCE_DATA.VEHICLE_DATA.TBOX_MESSAGE_DATA<br>CLUSTER BY (TBOXID);</pre><p>But the important part is this: use clustering properly and at the right time. It is most useful for large tables and repeated query patterns, not just because the feature exists. Snowflake also notes that maintaining clustering has a cost, so the goal is to use it where the performance benefit is meaningful for the workload.</p><p>So my practical advice is:</p><ul><li>Use clustering for large extraction and analytics tables when the access pattern is repeated.</li><li>Choose columns that are commonly used in filters.</li><li>Do not cluster blindly; cluster based on real query usage.</li><li>Revisit the clustering strategy as the workload changes over time.</li></ul><p>(<a href="https://docs.snowflake.com/en/user-guide/tables-clustering-keys">https://docs.snowflake.com/en/user-guide/tables-clustering-keys</a>)</p><p>(<a href="https://docs.snowflake.com/en/user-guide/tables-auto-reclustering">https://docs.snowflake.com/en/user-guide/tables-auto-reclustering</a>)</p><h3>4) Watch notebook runtime carefully</h3><p>One hidden area people forget is notebook runtime. If notebooks stay attached to a warehouse for too long, or if someone leaves a session running in the background, warehouse runtime can spike without much visible business value. Since warehouse compute is a major cost driver, long-running or forgotten notebook sessions can quietly increase credit usage.​</p><p>Because of that, I think it is better to use a separate, customized warehouse for notebooks whenever possible. That makes monitoring easier, isolates exploratory workloads from production ETL or BI workloads, and reduces the chance that notebook activity affects critical jobs. This follows the broader Snowflake cost principle of isolating workloads and monitoring compute usage closely.</p><h3>5) Prefer SQL first for in-platform work</h3><p>Snowflake supports SQL, Python, and JavaScript for different kinds of data work, and that flexibility is very useful. But for ETL logic, transformations, and extraction tasks that can be expressed in SQL, I usually prefer SQL first because it is native to the platform and often the simplest way to get the work done efficiently. Snowflake’s optimization guidance consistently points toward efficient query design and reducing unnecessary compute work.​</p><p>That does not mean Python is bad. Python is still valuable when the business logic is too complex for SQL or when you need something beyond standard transformation patterns. My rule is simple: if the problem can be solved cleanly in SQL, start with SQL; if not, move to Python intentionally, not by habit.</p><p>Finally, for me, Snowflake cost optimization is not about making everything smaller or restricting engineers. It is mostly about using the right warehouse size, suspending compute quickly, avoiding unnecessary multi-cluster scaling, applying clustering only where it really helps, and keeping an eye on workloads like notebooks that can quietly consume credits. Snowflake’s official guidance supports this same general approach: optimize warehouses, optimize queries, manage storage carefully, and monitor usage continuously.​</p><p>In real projects, even small changes like setting auto-suspend properly or separating notebook workloads can make a noticeable difference over time. Cost optimization in Snowflake is usually not one big trick; it is a set of small, practical decisions made consistently.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=760689613518" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event-Driven Data Engineering with Snowflake: Best Practices for Real-Time Pipelines]]></title>
            <link>https://medium.com/@usmaanrifkhan/event-driven-data-engineering-with-snowflake-best-practices-for-real-time-pipelines-f74796d24c27?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/f74796d24c27</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Thu, 27 Nov 2025 06:34:40 GMT</pubDate>
            <atom:updated>2026-03-12T04:12:18.353Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dW9TZMBUavOEqT6RHVbjVA.png" /></figure><p>In today’s fast-paced data environment, real-time ETL pipelines are becoming essential for modern data engineers. Waiting hours or even minutes to process incoming data can mean missing critical insights, especially when dealing with high-frequency events from IoT devices or operational systems. That’s why building pipelines that can ingest, transform, and load data <strong>as it arrives</strong> is no longer optional, it’s a necessity.</p><p>For example, in my workflow, I receive <strong>JSON files from battery stations</strong> every day through Node-RED, which land directly into a <strong>Snowflake internal stage</strong>. (Snowflake also supports external stages if you want to pull data from AWS, Azure, or GCP.) From there, the data can be ingested automatically, transformed, and made available for analysis in near real-time.</p><p>Snowflake makes all of this extremely straightforward. One of the things I really like is that it’s <strong>cost-efficient, </strong>you only pay for the compute warehouse you run (think of it as the virtual machine powering your queries and transformations), while the internal storage is comparatively cheap. On top of that, Snowflake is a <strong>multi-cloud platform</strong>, so you can ingest data from any major cloud provider or your own internal systems without hassle.</p><p>It’s also very flexible, you can use <strong>SQL or Python</strong>, leverage a variety of <strong>rich features</strong>, and benefit from <strong>row-level security</strong>, <strong>columnar storage</strong>, and even <strong>clustering tables by important columns</strong> to optimize both performance and cost. All of this makes Snowflake an excellent choice for building <strong>reliable, scalable, and cost-effective real-time ETL pipelines</strong>.</p><p>In my workflow, we receive <strong>JSON files from battery stations</strong> almost every day. These files vary in size, structure, and frequency, which makes <strong>schema management and incremental processing</strong> more complex. You can’t simply reload the entire dataset each time, you need to capture only the <strong>new or changed records</strong> without duplicates.</p><p>Another challenge is <strong>maintaining separate domain-specific tables</strong>. In my case, we categorize events into different types like heartbeat signals, cabinet status, and change status. Each type needs its own table for analytics, monitoring, and reporting. Ensuring that each event lands in the right table without manual intervention requires careful design. Latency is also a concern. Real-time pipelines must <strong>process incoming data quickly</strong> so downstream systems can act on it immediately. Any delays can reduce the value of having “real-time” data in the first place.</p><p>Finally, when working with JSON or semi-structured data, <strong>handling schema evolution</strong> changes in fields or data types over time adds another layer of complexity. Ideally, maintaining a consistent schema is a best practice, but in the real world, we often face changing or unpredictable data. There’s only so much control we can have, and it’s impossible to ensure 100% consistency. In my workflow, for example, some column names occasionally change. To handle this, I used <strong>SQL </strong><strong>COALESCE</strong> to provide fallback values and ensure the pipeline doesn’t break. These kinds of day-to-day challenges are common in real-time pipelines, and as data engineers, we need to find practical solutions to keep the workflow reliable while accommodating evolving data.</p><h3><strong>Snowflake Event-Driven Pipeline Architecture</strong></h3><h3>Stage</h3><p>A <strong>stage</strong> is basically a storage location in Snowflake where your data lands. Snowflake supports both <strong>internal and external stages</strong>.</p><ul><li><strong>Internal stage:</strong> You upload data directly into Snowflake.</li><li><strong>External stage:</strong> You can pull data from cloud storage services like AWS S3, Azure Blob, or GCP Storage.</li></ul><p>Think of the stage as the <strong>landing zone</strong> for your incoming files before any processing happens.</p><h3>Snowpipe</h3><p><strong>Snowpipe</strong> enables <strong>automatic, near-real-time loading</strong> of data from files as soon as they appear in a stage. Instead of running manual COPY commands or scheduling large batch loads, Snowpipe ingests data in <strong>micro-batches</strong>, making it available to users <strong>within minutes</strong>. This is perfect for real-time ETL workflows where latency matters.</p><h3>Stream</h3><p>A <strong>stream</strong> in Snowflake is how you track changes to a table — this is essentially <strong>Change Data Capture (CDC)</strong>. Streams record <strong>inserts, updates, and deletes</strong>, along with metadata about each change. In other words, a stream tracks which rows are <strong>new or modified</strong>, so downstream processes can act on only the relevant data.</p><p>A few important points about streams:</p><ul><li>A stream can be <strong>read only once per transaction</strong>. Once rows are consumed, they disappear from the stream until new changes arrive.</li><li>This makes streams perfect for <strong>incremental processing</strong> without reprocessing old data.</li></ul><h3>Task</h3><p>A <strong>task</strong> is the engine that drives your pipeline, it’s what triggers transformations and loads downstream tables.</p><p>In my workflow:</p><ol><li>The task reads <strong>new rows captured by the stream</strong>.</li><li>Inserts them into a <strong>temporary table</strong> for processing isolation.</li><li>From the temporary table, it inserts data into the <strong>raw table</strong> and <strong>domain-specific tables</strong> like heartbeat, cabinet_status, and change_status.</li></ol><p>Tasks allow pipelines to <strong>process incoming data automatically</strong>, without manual intervention, ensuring a <strong>reliable, real-time flow</strong> from raw files to clean, usable tables.</p><h3>My Step-by-Step Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wo1eMeKBsLdX86xa23ViSg.png" /></figure><p>Here’s how a real-time event-driven ETL pipeline typically flows in Snowflake:</p><p><strong>Stage: </strong>Incoming JSON or semi-structured files land in a <strong>Snowflake stage</strong>, which acts as the <strong>entry point</strong> for the pipeline.</p><p><strong>Snowpipe: Snowpipe</strong> automatically ingests files from the stage into a <strong>raw landing table</strong> in near real-time. This ensures that data is available for processing <strong>within minutes</strong> of arrival.</p><p><strong>Raw Init Table: </strong>The ingested data first lands in a <strong>raw init table</strong>, storing all incoming records in their original format.</p><p><strong>Stream: </strong>A <strong>Snowflake stream</strong> is created on the raw init table to track only <strong>newly inserted or modified rows</strong>. This enables <strong>incremental processing</strong> without reprocessing old data.</p><p><strong>Task: </strong>A <strong>task</strong> triggers automatically when new rows are available in the stream. The task workflow:</p><p>Reads the new rows from the stream into a <strong>temporary table</strong> (temp_new_rows) for processing isolation.</p><p>Inserts the rows from the temporary table into the <strong>raw table</strong> for full history storage. Filters and routes records into <strong>domain-specific tables</strong> based on event type:</p><ul><li>EV = &#39;CABINET_STATUS&#39; → <strong>Cabinet Status Table</strong></li><li>EV = &#39;HEARTBEAT&#39; → <strong>Heartbeat Table</strong></li><li>EV = &#39;CHANGE_STATUS&#39; → <strong>Change Status Table</strong></li></ul><p><strong>Housekeeping Task : </strong>Optionally, a <strong>separate scheduled task</strong> can truncate the stage and raw init table periodically (e.g., weekly) to control storage and maintain performance.</p><p>Building reliable real-time pipelines in Snowflake is not just about moving data from one place to another — it’s about doing it efficiently, safely, and in a way that scales. One important practice is using <strong>temporary tables</strong> when processing new rows from a stream. This isolates the processing and prevents partial or failed inserts from affecting your main tables. Leveraging <strong>streams</strong> ensures incremental processing, so only new or changed data is handled, avoiding duplication. Since JSON and semi-structured data can evolve, it’s important to <strong>handle schema changes gracefully</strong>, for example by using SQL functions like COALESCE to manage renamed or missing columns. Additionally, <strong>clustering tables</strong> on frequently queried columns can optimize performance and reduce costs, while regular monitoring and careful scheduling of tasks helps maintain reliability.</p><p>In practice, building real-time ETL pipelines comes with day-to-day challenges that textbooks don’t cover. Real-time, event-driven ETL pipelines are becoming a core requirement for modern data engineering. Snowflake provides a powerful set of features such as <strong>Snowpipe, Streams, Tasks, temporary tables, clustering, Snowpar and more </strong>that make building reliable and scalable pipelines easier. While this article covers one practical workflow and best practices, Snowflake has <strong>many more rich capabilities</strong> that can help optimize performance, manage costs, and handle complex data scenarios. Data engineers looking to deepen their expertise should explore the <strong>official Snowflake documentation</strong> to learn about advanced features and techniques that can enhance real-time data pipelines even further.</p><p>I hope this article gave you a clear understanding of how to build event-driven, real-time ETL pipelines in Snowflake. If you’re a data engineer or Snowflake enthusiast, experiment with these concepts in your own projects and explore the Snowflake documentation for more advanced features. Feedback, questions, or shared experiences are always welcome let’s keep learning together!</p><p><strong>Sincerely,</strong><br><a href="https://www.linkedin.com/in/usmaan-rifkhan"><em>Usmaan Rifkhan</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f74796d24c27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker for Starters: Run Your First Python Script in a Container]]></title>
            <link>https://medium.com/@usmaanrifkhan/docker-for-starters-run-your-first-python-script-in-a-container-978445fae4b1?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/978445fae4b1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Wed, 24 Sep 2025 15:08:54 GMT</pubDate>
            <atom:updated>2025-09-24T15:14:29.420Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*GUVzssBqiHcPR6xdU5O0uQ.png" /></figure><p>Docker is a containerization platform. It allows you to package your application, code, dependencies, and even OS libraries into a single container. Think of it as a lightweight virtual machine, but faster, portable, and easier to share.</p><p><strong>Why ML/Data engineers care:</strong></p><ul><li>Ensures your Python/ML environment runs consistently across different machines (dev → test → production).</li><li>Easy to deploy ML models without worrying about “it works on my machine” issues.</li><li>Helps in scaling workloads on cloud or Kubernetes.</li></ul><p><strong>Containers running on Docker Engine:</strong></p><p>·Standard: The containers can be set up in any machine, which means they are portable.</p><p>·Lightweight: It does not require an operating system of its own, it shares the host machine’s kernel. Reduces licensing and serving costs and increases efficiency.</p><p>·Secure: It provides a layer of abstraction in default, which isolates the container’s capabilities.</p><p>🐳 <strong>Core Docker Concepts</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*Qccy9R-qM2iUH-T-QyJgbA.png" /></figure><p><strong>What is a CONTAINER in Docker?</strong></p><p>A Container is somewhat like a bag that holds all our requirements, it is a standard software unit that bundles up or packages all our code, dependencies and has its virtual environment so that we can run our application efficiently from one host environment to another. It can also be referred to as a docker container image. The container image is a standalone and executable package, which has all the necessary settings and libraries to run the desired application.</p><p><strong>🐳 Must-Know Docker Commands</strong></p><p>🔹 Image Commands</p><ul><li>docker build → Build a Docker image from a Dockerfile.</li><li>docker image → Manage Docker images (inspect, prune, etc.).</li><li>docker images → List all images stored locally.</li><li>docker rmi &lt;image_id&gt; → Remove an image.</li><li>docker tag &lt;source&gt; &lt;new_tag&gt; → Tag an existing image (useful for pushing to Docker Hub).</li><li>docker pull &lt;image_name&gt; → Download an image from Docker Hub.</li><li>docker push &lt;image_name&gt; → Upload an image to Docker Hub.</li></ul><p>🔹 Container Commands</p><ul><li>docker run &lt;image_name&gt; → Run a new container from an image.</li><li>docker container → Manage containers (start, stop, inspect, prune).</li><li>docker ps (or docker container ls) → List running containers.</li><li>docker rm &lt;container_id&gt; → Remove a container.</li><li>docker stop &lt;container_id&gt; → Gracefully stop a running container.</li><li>docker kill &lt;container_id&gt; → Force stop a container immediately.</li><li>docker port &lt;container_id&gt; → Show port mappings for a container.</li></ul><p>🔹 Logging &amp; Debugging</p><ul><li>docker logs &lt;container_id&gt; → Show logs of a container (useful for debugging).</li><li>docker events → Stream real-time events from the Docker daemon.</li></ul><p>🔹 Authentication</p><ul><li>docker login → Log in to Docker Hub (required before pushing images).</li><li>docker logout → Log out from Docker Hub.</li></ul><p>Clean up often → Images/containers eat space. Use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*tlAwFr1m9gmJHU0Wpb7zfw.png" /></figure><p><strong>Hands On</strong></p><p>Will create a sample image,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*rSVsH1I6hQA8PyN5hq9gBA.png" /></figure><p>Now to see the number of images present, just run “docker images” in the command prompt. It shows the total number of images present, with the Repository name, Tag, Image ID, age of that image, and size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*5HwLd3NEKFXlTiyK_Ac32Q.png" /></figure><p>To delete a Docker image, you can just use “docker rmi ” if this fails you can just add “-f” means forcing it to remove. that is “docker rmi -f “. This will remove a particular image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*yKcf2Lq5Vgj6jeXHfkFyXw.png" /></figure><p><strong>🐳 Important Docker Terminologies</strong></p><p>🔹 1. Docker Image</p><ul><li>Blueprint/template of an application.</li><li>Contains code + dependencies + environment.</li><li>You can pull pre-built images from registries (like Docker Hub) or build your own with a Dockerfile. (Ex — docker pull python:3.12-slim)</li></ul><p>🔹 2. Docker Container</p><ul><li>A running instance of a Docker image.</li><li>Think of it as the “live” application created from the image.</li><li>You use docker run to start a container. (Ex- docker run -it python:3.12-slim bash)</li><li>To see running containers: (Ex- docker ps)</li></ul><p>🔹 3. Docker Daemon</p><ul><li>Background service that does the heavy lifting.</li><li>Manages building, running, and distributing containers/images.</li><li>Runs on the host machine and listens for Docker API requests.</li><li>Without the daemon, Docker commands won’t work.</li></ul><p>🔹 4. Docker Hub</p><ul><li>Public cloud registry for Docker images.</li><li>Functions:</li><li>Pull: Download existing images (e.g., Python, TensorFlow).</li><li>Push: Upload/share your own images.</li></ul><p><strong>🐳 Dockerfile</strong></p><p>What is a Dockerfile?</p><p>A text file with instructions to build a Docker image and makes building images systematic, repeatable, and flexible. And it defines the environment, dependencies, and how the container should run.</p><p>Instead of manually installing packages each time, you write a Dockerfile → build the image → and share or deploy it anywhere.</p><p>🔹 Common Dockerfile Commands</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*5104EM3Ut-KkLxWgl-CDjw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*TAC8KLcneM5WGFM3TnY80A.png" /></figure><p>🔹 Example : Data Engineering (ETL Pipeline)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*EjWBSggbmFTXXEHNHc0ZZw.png" /></figure><p>✅ What happens?</p><ul><li>Always runs etl_pipeline.py.</li><li>By default, it processes raw data from S3 → cleans &amp; writes back to S3.</li><li>Easy to schedule in Airflow or cron.</li></ul><p>Our First Image</p><ul><li>Official images are developed and maintained by the Docker team. Some of the official images are, python, ubuntu, TensorFlow, etc. images are official images.</li><li>User images are created by us. We build it on some base image or using official images.</li></ul><p><strong>Now let’s create our first image</strong></p><p>First let us create a very basic python script (test.py). Refer below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*SAxqgTPHWWH4i5H_kLVJAQ.png" /></figure><p>Now, in the same project directory create a file with the name “Dockerfile” and here we need to have some commands, which will copy the working directory and also need to mention the entry point to the docker. A Dockerfile is a text file that contains commands that are executed while creating an image. The commands are equal to Linux commands. Below is the content of Dockerfile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*bljnDnoW7K8ejc_jKy10cA.png" /></figure><p>Now in order to build the image, run the docker build command in the same directory.</p><p><strong><em>docker build -f Dockerfile -t myfirstimage:v1 .</em></strong></p><p>In the above command, we need to mention the “Dockerfile” which we created and the name and version of our image as well as a dot at the end of the build command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*iKTsdgBvDp5dJXJyoPCFeA.png" /></figure><p>After building the image, let us check if the image is present. Just run docker images. You can see it has created the image with the mentioned name and version.</p><p>After building the image, let us check if the image is present. Just run docker images. You can see it has created the image with the mentioned name and version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*HtH7ExybwCW84DmVFgxmHg.png" /></figure><p>Now, let us run the image. We know that the output should be 7, according to the small python script we wrote earlier. To run the image, docker run myfirstimage:v1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*bNp-zKNL6g8K18Mtotg3Jw.png" /></figure><p>Thank you.</p><p>Reference:</p><p><a href="https://www.analyticsvidhya.com/blog/2021/10/end-to-end-guide-to-docker-for-aspiring-data-engineers/">https://www.analyticsvidhya.com/blog/2021/10/end-to-end-guide-to-docker-for-aspiring-data-engineers/</a></p><p><a href="https://www.startdataengineering.com/post/docker-for-de/">https://www.startdataengineering.com/post/docker-for-de/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=978445fae4b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simply Understanding What Cloud Computing]]></title>
            <link>https://medium.com/@usmaanrifkhan/simply-understanding-what-cloud-computing-f003cbf364ec?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/f003cbf364ec</guid>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Wed, 27 Aug 2025 18:00:27 GMT</pubDate>
            <atom:updated>2025-08-27T18:00:27.983Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*fbAQB_m3kBucgW9FjUn1VQ.jpeg" /></figure><h3><strong>Simply Understanding What is Cloud Computing</strong></h3><p>Cloud Computing refers to manipulating, configuring, and accessing the hardware and software resources remotely. It offers online data storage, infrastructure, and application. Cloud computing offers platform independency, as the software is not required to be installed locally on the PC. For example, when you use Google Drive to save files or Netflix to watch movies, you’re not storing them on your device, you’re accessing them through the cloud. This helps organizations save costs, scale quickly, and stay flexible in today’s fast-changing world.</p><p>And also cloud computing is an evolution of two rising technologies, Time sharing and distributed computing. By cloud, Organizations can maximize cost and compute efficiency, as well as their responses to market changes.</p><p><strong>Characteristics of cloud computing;</strong></p><ul><li>On-Demand Self-Service:You can access cloud resources anytime, for any reason, almost instantly, without needing help from the service provider.</li><li>Ubiquitous Network Access:As long as you have an internet connection, you can access cloud services. In the early days, if someone left the data center, they lost access to their data and resources. With cloud computing, you can access your services and data from anywhere, even on your mobile phone.</li><li>Resource Pooling:You can access resources, use them for a period of time, and then return them to the resource pool for others to use.</li><li>Rapid Elasticity:Cloud resources can quickly scale up or down as needed to meet demand.</li><li>Pay-Per-Use:You only pay for the resources you actually use.</li></ul><p><strong>Cloud Computing Architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/0*qmW5_VXzysNDMvK1" /></figure><p>There are certain services and models working behind the scenes making cloud computing feasible and accessible to end users. Following are the working models for cloud computing:</p><ul><li>Deployment models</li><li>Service models</li></ul><p><strong>Service Models</strong></p><p><strong>☁️ 1. Software as a Service (SaaS)</strong></p><p>SaaS (or “on-demand software”) means the applications are hosted by a cloud provider and you simply use them over the internet. Think of it as renting software instead of buying it.</p><p>✅ Examples: Google Workspace, Microsoft 365, Dropbox, Slack<br> ✨ Features:</p><p>Accessible from anywhere with internet + browser.</p><p>Pay-as-you-go pricing (subscription-based).</p><p>No need to worry about updates — the provider does it automatically.</p><p>👉 Simple analogy: SaaS is like <strong>using Netflix</strong> — you don’t own the movies, you just stream them whenever you want.</p><p><strong>🖥️ 2. Infrastructure as a Service (IaaS)</strong></p><p>IaaS (sometimes called Hardware as a Service) provides raw computing resources like storage, networking, and virtual servers over the internet. Instead of buying and maintaining physical data centers, you simply rent the infrastructure.</p><p>✅ Examples: Amazon Web Services (AWS), Microsoft Azure<br>✨ Features:</p><p>Avoids the cost of buying and managing physical servers.</p><p>Pay only for what you use.</p><p>Scale up or down easily depending on demand.</p><p>👉 Simple analogy: IaaS is like renting an empty apartment — you bring your own furniture (software, apps) and set it up the way you want.</p><p><strong>🛠️ 3. Platform as a Service (PaaS)</strong></p><p>PaaS is designed mainly for developers. It provides a ready-to-use environment with tools for building, testing, and deploying applications. The platform handles the infrastructure, so developers can just focus on writing code.</p><p>✅ Examples: AWS Elastic Beanstalk, Microsoft Azure App Service, Heroku<br>✨ Features:</p><p>Reduces development complexity.</p><p>Pay only for the resources you use.</p><p>Scalable — resources can be increased or decreased as needed.</p><p>👉 Simple analogy: PaaS is like a furnished apartment — you move in and start living (coding) right away without worrying about setting up furniture (infrastructure).</p><p>⚡ <strong>In short:</strong></p><ul><li><strong>SaaS</strong> → End-users (finished apps, like Gmail or Netflix).</li><li><strong>PaaS</strong> → Developers (a ready platform to build apps).</li><li><strong>IaaS</strong> → IT admins / engineers (raw infrastructure to set up systems).</li></ul><p><strong>Deployment Models</strong></p><p>Deployment models define the type of access to the cloud for an example; how is the cloud located? Cloud can have any of the four types of access: Public, Private, Hybrid, and Community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/0*Kl_2RBaF59fQIzJS" /></figure><p><strong>🌍 1. Public Cloud</strong></p><p>The cloud services are available to everyone over the internet. All infrastructure is owned and managed by the provider, and multiple users share the same resources.</p><p>✅ Examples: Google Cloud, Microsoft Azure, Amazon Web Services (AWS)<br>👉 Analogy: Like a public bus 🚌 — anyone can buy a ticket and use it.</p><p><strong>🏢 2. Private Cloud</strong></p><p>The cloud is used exclusively by a single organization. It offers higher security and control, but also comes with more cost since it’s not shared.</p><p>✅ Examples: Banks, government institutions, or companies with strict compliance needs<br>👉 Analogy: Like owning a private car 🚗 — only you (or your company) can use it.</p><p><strong>🔗 3. Hybrid Cloud</strong></p><p>A mix of public and private cloud. Organizations keep sensitive workloads in the private cloud and use the public cloud for less critical tasks. This allows flexibility and cost optimization.</p><p>✅ Examples: A company storing confidential data in a private cloud while running customer-facing apps on AWS.<br>👉 Analogy: Like owning a car but also using public transport 🚗+🚌 depending on the situation.</p><p><strong>👥 4. Community Cloud</strong></p><p>The cloud infrastructure is shared by several organizations with common goals, policies, or industry requirements. It’s like a private cloud, but shared among a group with similar needs.</p><p>✅ Examples: Universities collaborating on research, healthcare organizations sharing patient data securely.<br>👉 Analogy: Like a gated community 🏘️ — shared space but only for a specific group.</p><p>It is the responsibility of cloud providers to manage resources and their performance. Management of resources includes several aspects of cloud computing such as load balancing, performance, storage, backups, capacity, deployment, etc. The management is essential to access full functionality of resources in the cloud.</p><p><strong>💾 Cloud Storage</strong></p><p>Cloud storage is a service that lets you save data on offsite systems managed by third-party providers and access it via the internet (through web APIs). Instead of filling up your computer’s hard drive, your data is stored safely in the cloud.</p><p>🔹 Types of Cloud Storage</p><p>Block Storage → Provides raw storage space, divided into volumes. Think of it like a blank notebook 📒 — you can use each page (volume) however you want.</p><p>File Storage → Stores data in files and folders, managed with its own file system. This is similar to a filing cabinet 🗂️, where documents are neatly organized and accessed as files. (Often implemented as Network Attached Storage — NAS).</p><p>👉 Everyday Example: When you upload photos to Google Drive or documents to Dropbox, you’re using cloud storage instead of keeping everything locally.</p><p><strong>🖥️ Virtualization</strong></p><p>Virtualization is the technique that makes cloud computing possible. It allows a single physical resource (like a server or application) to be shared by multiple users or organizations.</p><p>A virtual machine (VM) is created on top of real hardware and operating systems.</p><p>The host machine = the physical computer.</p><p>The guest machine = the virtual machine running on it.</p><p>The hypervisor = software or firmware that manages these VMs (also called Virtual Machine Manager).</p><p>👉 Analogy: Imagine an apartment building 🏢 (the host). Each apartment (VM) is independent, with its own tenants (applications), but they all share the same building infrastructure (the physical hardware).</p><p><strong>🔄 CloudOps</strong></p><p>CloudOps (Cloud Operations) focuses on keeping cloud-based services running smoothly, continuously, and without downtime. It’s about maintaining performance, improving efficiency, and ensuring users always have access.</p><p>Key aspects of CloudOps:</p><p>Continuous operations → no interruptions, zero downtime.</p><p>Feedback loops → monitoring and sending information back to developers for improvement.</p><p>Resource governance → allocate and manage resources consistently across platforms (AWS, Azure, Google Cloud).</p><p>👉 Analogy: Think of CloudOps like air traffic control ✈️ — making sure flights (applications) keep running on time, avoiding crashes (downtime), and constantly adjusting to new conditions.</p><p>👉 <strong>Analogy</strong>: Think of CloudOps like <strong>air traffic control ✈️</strong> — making sure flights (applications) keep running on time, avoiding crashes (downtime), and constantly adjusting to new conditions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f003cbf364ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Image Processing]]></title>
            <link>https://medium.com/@usmaanrifkhan/image-processing-f27627085a38?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/f27627085a38</guid>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Fri, 24 Jan 2025 02:09:21 GMT</pubDate>
            <atom:updated>2025-01-24T02:09:21.648Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v0eyI8dWLrtXU6oX.jpg" /></figure><p>Digital image processing is the use of algorithms and mathematical models to process and analyze digital images. The goal of digital image processing is to enhance the quality of images, extract meaningful information from images, and automate image-based tasks. DIP is a subfield of signals and systems.</p><p><strong>Image processing mainly include the following steps:</strong></p><p>1.Importing the image via image acquisition tools;</p><p>2.Analysing and manipulating the image;</p><p>3.Output in which the result can be an altered image or a report which is based on analyzing that image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*DQNzQnNjljYS2ltN" /></figure><p><strong>What is an image?</strong></p><p>An image is nothing more than a two dimensional signal. It is defined by the mathematical function f(x,y) where x and y are the two coordinates horizontally and vertically. The value of f(x,y) at any point gives the pixel value at that point of an image.</p><p><strong>Analog Image Processing;</strong></p><p>Analog image processing is done on analog signals. It includes processing on two dimensional analog signals. In this type of processing, the images are manipulated by electrical means by varying the electrical signal. The common example is the television image.</p><p><strong>Digital Image Processing;</strong></p><p>The digital image processing deals with developing a digital system that performs operations on a digital image.</p><p><strong>What is a signal?</strong></p><p>In the physical world, any quantity measurable through time over space or any higher dimension can be taken as a signal. A signal is a mathematical function, and it conveys some information. A signal can be one dimensional or two dimensional or higher dimensional signal. One dimensional signal is a signal that is measured over time. The common example is a voice signal. The two dimensional signals are those that are measured over some other physical quantities. The example of a two dimensional signal is a digital image.</p><p>Since anything that conveys information or broadcast a message in physical world between two observers is a signal. That includes speech or (human voice) or an image as a signal. Since when we speak , our voice is converted to a sound wave/signal and transformed with respect to the time to person we are speaking to. Not only this , but the way a digital camera works, as while acquiring an image from a digital camera involves transfer of a signal from one part of the system to the other.</p><p><strong>Types of an image</strong></p><ul><li>BINARY IMAGE– The binary image as its name suggests, contain only two pixel elements i.e 0 &amp; 1,where 0 refers to black and 1 refers to white. This image is also known as Monochrome.</li><li>BLACK AND WHITE IMAGE– The image which consist of only black and white color is called BLACK AND WHITE IMAGE.</li><li>8 bit COLOR FORMAT– It is the most famous image format.It has 256 different shades of colors in it and commonly known as Grayscale Image. In this format, 0 stands for Black, and 255 stands for white, and 127 stands for gray.</li><li>16 bit COLOR FORMAT– It is a color image format. It has 65,536 different colors in it.It is also known as High Color Format. In this format the distribution of color is not as same as Grayscale image. A 16 bit format is actually divided into three further formats which are Red, Green and Blue. That famous RGB format.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*E_w7tbz9bB0bDPnm" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*HJsxPv1M2LcUsc8CDgGJ8g.png" /></figure><p><strong>Image as Matrix</strong></p><p>As we know, images are represented in rows and columns we have the following syntax in which images are represented:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/0*TbLXS0PO8TZxo-zE" /></figure><p>The right side of this equation is digital image by definition. Every element of this matrix is called image element , picture element , or pixel.</p><p>Note — In MATLAB the start index is from 1 instead of 0. Therefore, f(1,1) = f(0,0).</p><p><strong>Difference between image processing and computer vision</strong></p><p>Image Processing is the field of enhancing the images by tuning many parameter and features of the images. So Image Processing is the subset of Computer Vision. Here, transformations are applied to an input image and the resultant output image is returned. Some of these transformations are- sharpening, smoothing, stretching etc. In Computer Vision, computers or machines are made to gain high-level understanding from the input digital images or videos with the purpose of automating tasks that the human visual system can do. It uses many techniques and Image Processing is just one of them.</p><p><strong>References,</strong></p><p><a href="https://www.geeksforgeeks.org/digital-image-processing-basics/">https://www.geeksforgeeks.org/digital-image-processing-basics/</a></p><p><a href="https://www.javatpoint.com/digital-image-processing-tutorial">https://www.javatpoint.com/digital-image-processing-tutorial</a></p><p><a href="https://www.tutorialspoint.com/dip/index.htm">https://www.tutorialspoint.com/dip/index.htm</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f27627085a38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convolutional Neural Network]]></title>
            <link>https://medium.com/@usmaanrifkhan/convolutional-neural-network-b880cbbb9942?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/b880cbbb9942</guid>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Sun, 02 Jun 2024 18:18:25 GMT</pubDate>
            <atom:updated>2024-06-02T18:18:25.587Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EWvD0X1Jp3wPlNPZ.jpg" /></figure><p>A Convolutional Neural Network (CNN), also known as ConvNet, is a specialized type of deep learning algorithm mainly designed for tasks that necessitate object recognition, including image classification, detection, and segmentation.</p><p><strong>CNN Architecture</strong></p><p>Convolutional Neural Network consists of multiple layers like the input layer, Convolutional layer, Pooling layer, and fully connected layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*k6Wx0ce85APVOTZCmgA1IA.png" /></figure><p><strong>Padding</strong></p><p>Padding in CNNs is the technique of adding extra pixels around the input image or feature map to maintain spatial dimensions during the convolution operation.</p><p>This can be done in two ways:</p><p>· Valid Padding: In the valid padding, no padding is added to the input feature map, and the output feature map is smaller than the input feature map. This is useful when we want to reduce the spatial dimensions of the feature maps.</p><p>· Same Padding: In the same padding, padding is added to the input feature map such that the size of the output feature map is the same as the input feature map. This is useful when we want to preserve the spatial dimensions of the feature maps.</p><p>Example,</p><p>Imagine we have 6x6 input image and 3x3 filter. What is the size of output layer?</p><p>n -f +1 = output layer size. So, 6–3+1 = 4. Output size = 4x4. So, here input size and output size are not equal. So, we need to do padding (n+2p-f+1)</p><p>Why do we need padding?</p><p>During convolution, the output feature map size depends on the input feature map size, kernel size, and stride. Without padding, applying the kernel directly to the input feature map would result in an output feature map smaller than the input. Padding helps preserve the border information by adding extra pixels around the input feature map before convolution.</p><p>If you see for same padding O = n (Output image size should be equal to input image size)</p><p>So, O = O — k + 1 + 2p from here we can simply use mathematics in above equation mentioned P = (k — 1)/2.</p><p>Why do we always use ODD (filter) kernels like 3 x 3, 5 x 5 &amp; so on why not 4 x 4 kernels (even kernels)?</p><p>You can use the above formula p = (k — 1)/2</p><p>We have kernel = 4 x 4</p><p>p= 4 -1/2 = 3/2 = 1.5</p><p>Which means you have to add 1.5 pixels around 4 corners of the image to apply padding, which in turn is not possible because we can’t add 1.5 pixels instead, we can only add the pixels in whole numbers like 1,2,3,4,5,6 &amp; 7 so on pixels around the corners of the image.</p><p><strong>Back Prop in CNN;</strong> In ANN backprop we update weights and biases so, in CNN we try to learn about filters, and we try to update the elements of the filters.</p><p><strong>Pooling</strong></p><p>Pooling is a down sampling operation applied to feature maps (also known as activation maps) produced by convolutional layers. Its primary goal is to reduce the spatial dimensions of the feature maps while retaining important information.</p><p>Types of pooling</p><p>01.Max Pooling = Selects the maximum element from each region of the feature map covered by the filter.</p><p>02.Average Pooling = Computes the average value within each region.</p><p>03.Global Pooling = Global pooling reduces each channel in the feature map to a single value. Further, it can be either global max pooling or global average pooling.</p><p><strong>Activation Function</strong></p><p>The activation function is typically applied to the output of each neuron in the network. It takes in the weighted sum of the inputs and produces an output that is then passed on to the next layer. The most commonly used activation functions in CNNs are:<br>• Rectified linear unit (ReLU)<br>• Sigmoid<br>• Hyperbolic tangent (tanh)</p><p><strong>Data Augmentation</strong></p><p>Data augmentation can address a variety of challenges when training a CNN model, such as limited or imbalanced data, overfitting, and variation and complexity. This technique can increase the size of the dataset and balance the classes by applying different transformations. Data augmentation is the process of creating permutations of existing images in a dataset to create new images. For example, you can augment an image by increasing and decreasing the brightness. This would help a model learn to better identify objects of interest in different lighting conditions. You can augment by changing the angle of the image, helping the model to detect an object at different angles.</p><p><strong>Transfer Learning</strong></p><p>The general idea of transfer learning is to use knowledge learned from tasks for which a lot of labelled data is available in settings where only little labelled data is available. Creating labelled data is expensive, so optimally leveraging existing datasets is key. With transfer learning, you attempt to kickstart this generalization process by starting from patterns that have been learned for a different task. Essentially, instead of starting the learning process from a (often randomly initialized) blank sheet, you start from patterns that have been learned to solve a different task.</p><p>Transfer learning is a common and effective strategy to train a network on a small dataset, where a network is pretrained on an extremely large dataset, such as ImageNet, which contains 1.4 million images with 1000 classes, then reused and applied to the given task of interest. The underlying assumption of transfer learning is that generic features learned on a large enough dataset can be shared among seemingly disparate datasets.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b880cbbb9942" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hadoop Ecosystem: An Integrated Approach to Big Data Processing and Analytics]]></title>
            <link>https://medium.com/@usmaanrifkhan/the-hadoop-ecosystem-an-integrated-approach-to-big-data-processing-and-analytics-7ecc304fe665?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ecc304fe665</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Sat, 01 Jun 2024 17:18:38 GMT</pubDate>
            <atom:updated>2026-03-12T04:15:27.200Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3_EC0BgZ4wr_trFh" /></figure><p>In the era of big data organizations face the challenge of processing, storing, and analyzing large amounts of data efficiently. The Hadoop eco system provides lots of tools and frameworks to address these challenges. So, these tools and frameworks are very useful for big data management and analytics.</p><p>This article describes some tools that are available in the Hadoop ecosystem for big data analytics. Article provides a brief description of some very useful tools that are in the Hadoop ecosystem. I have explained those tools under eight topics. Those topics are in a workflow. Flow is Data ingestion, Data storage, Data processing, Data analysis, Data indexing and searching, Machine learning and advanced analytics and coordination and management.</p><p><strong>01. Apache Kafka</strong></p><p>Apache Kafka is a powerful open-source distributed <br>streaming platform, and it follows a publish-subscribe <br>messaging system. Unlike point-to-point messaging systems in <br>pub-sub messaging systems, consumers can subscribe to one or <br>more topics and can consume all the messages in that topic. <br>Apache Kafka is written in Scala and Java, and its main use cases <br>of it are stream processing, log aggregation, and metrics. There <br>There are many benefits to Kafka, such as scalability, reliability, <br>durability, fault tolerance, high availability, and high <br>throughput. There are five main components in Kafka. That <br>are Broker, Topic, Producer, Consumer, and Partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*xOzJd84Q6Xv3QtHW" /></figure><p>Producers send data to Kafka brokers, and consumers get data. <br>from that broker. Stream messages that belong to a particular <br>category called Topic. In each topic, there are one or more partitions. A broker is a simple system that is responsible for <br>maintaining data. The broker contains several topics and <br>partitions. Topics are split into several parts, which are partitions. <br>Actual data will be stored in partitions. While storing new data in a <br>partition, it will increase the number of offsets one by one. <br>Offsets for a partition are infinite. Offset is the main reason for <br>Kafka’s fault tolerance. Apache Kafka replicates data within <br>brokers, so if one broker fails, we can get data from another one <br>where replication is stored. The replication should be between <br>2 and 3. There are four APIs in Kafka. Such as the connector <br>API, stream API, producer API, and consumer API. In Kafka, <br>Zookeeper is using to notify about brokers to consumer and <br>producer.</p><p><strong>02. Apache Sqoop &amp; Flume</strong></p><p>Apache Sqoop is a specialized tool that allows for smooth <br>data flow between HDFS and various structured data <br>repositories. It acts as the connector between the relational <br>database and Hadoop cluster. Apache Sqoop has two main <br>functions that are import and export. It exports data from <br>Hadoop file system to a relational database and import data <br>from a relational database to a Hadoop file system. Sqoop uses <br>the yarn framework to import and export data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/0*jss5nrueNxMLkaNc" /></figure><p>MapReduce cannot directly access the data that is stored in <br>external relational database. That solution is Sqoop. By <br>Using Apache Sqoop, we can transfer data from numerous <br>structured sources quickly and efficiently. And we can <br>automate this process so it will increase efficiency. Like <br>advantage, but there are some limitations. Apache Sqoop is slow. <br>because it uses the MapReduce model for the backend process, and it <br>cannot be stopped or started again. And while handling imports <br>or export failures, it required special handling. <br>On the other hand, Apache Flume is specially designed for <br>ingesting a high volume of event-based data, especially <br>unstructured data. Flume is an open-source tool for collecting <br>aggregating and pushing log data from a massive number of <br>sources into different storage systems in the Hadoop ecosystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UzrtgXNv2nYMV3ch" /></figure><p>There are two main entities in Apache flume, Flume event and Flume agent. Flume event is the fundamental data unit that moves from source to sink. The flume agent is an independent JVM process. It receives data from clients or other agents and forwards it to the sink. Flume may have more than one agent. As a tool, flume also has its limitations. One of the main limitations is It does not guarantee that the messages reaching are 100% unique, as sometimes duplicate messages may appear.</p><p><strong>03. Zookeeper</strong></p><p>Apache Zookeeper is a distributed, open-source <br>coordination service designed for managing distributed applications. Hadoop’s zookeeper can be considered a <br>centralized repository where distributed applications can <br>upload and retrieve data. Basically, distributed applications <br>are complex and difficult to coordinate and work with because they <br>are much more prone to errors due to the large number of <br>machines connected to the network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Ug6XnlabYMsALlw" /></figure><p>Hadoop Zookeeper Architecture is a distributed application <br>that follows a simple client-server model, where clients are the <br>nodes that consume the service, and servers are the nodes that <br>provide the service. Multiple server nodes are collectively <br>called a Zookeeper file. A zookeeper client uses at least one <br>server at a given time.</p><p><strong>04. Apache Oozie &amp; Airflow</strong></p><p>Apache Oozie is a workflow scheduler system for Hadoop. <br>jobs management. It lets users manage many activities and <br>operations throughout the Hadoop ecosystem while <br>designing and executing complicated data processing <br>workflows. Oozie is an effective tool for organizing and <br>arranging major Hadoop activities. Thus, it offers an API to <br>interact with other Hadoop ecosystems and a web-based <br>interface for managing and monitoring purposes. The main <br>components of Oozie are the Oozie client, which is an interface. <br>for submitting and managing Oozie workflows and <br>coordinator jobs, Oozie server, which is a central component <br>that manages workflow execution and the Oozie database, <br>stores metadata about workflows, coordinator jobs, and their <br>execution history. <br>Apache Airflow is a workflow engine that will <br>schedule and run complex data pipelines. And it will <br>guarantee every job in the data pipelines is completed in <br>the right order and with the required resources. The main <br>components of airflow are DAGs, which are a sequence of <br>tasks with defined dependencies forming a workflow <br>specifies task order, schedule, and conditions for execution, <br>Scheduler, which organizes and schedules the execution of <br>tasks in DAGs. It manages task timing, dependencies, and <br>parallel execution and an executor, which executes tasks as <br>directed by the scheduler. When we compare Oozie with <br>Airflow is better because it is more user-friendly. <br>handle large workloads, high scalability, and fault tolerance <br>and high flexibility.</p><p><strong>05. HBase &amp; Cassendra</strong></p><p>HBase is an open-source, distributed, scalable, and NoSQL <br>database that is written in Java. It is column-oriented and runs <br>on the top of HDFS and provides random write and read <br>access. HBase has three main components, such as region, <br>H. Master and Zookeeper. The fundamental components of a <br>Hadoop clusters made up of column families are called HBase. <br>Regions. It holds a separate store for each column family. <br>Region servers act as worker nodes that handle clients. <br>requests for reading, writing, updating, and deleting operations. <br>HMaster is an implementation of the master server in HBase. <br>Regions, data definitions, and language operations (create, delete) are handled by HMaster. And Zookeeper offers facilities like <br>naming, configuration information maintenance, server failure <br>notification and data synchronization. Zookeeper serves as a <br>communication bridge across the architecture when the clients <br>want to communicate. <br>Apache Cassendra is an open-source, decentralized storage <br>system/NoSQL distributed database with high availability and <br>scalability with no single point of failure. There are many. <br>benefits of Cassendra. Such as, it is scalable, can accept semi <br>structured, structured, and unstructured data, it follows ACID. <br>properties; therefore, it has good transaction support; it is highly <br>available, and it is fault-tolerant. Cassandra operates on a ring. <br>architecture. In order to ensure durability and make data very <br>available, Cassandra implemented automatic peer-to-peer <br>replication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ihSwkZMuRetq3nYC" /></figure><p>When we compare HBase and Cassendra, Cassandra is <br>highly available, and HBase has high consistency. <br>Performance-wise, Cassendra is the winner, but row-level <br>Access efficiency is better in HBase.</p><p><strong>06. Apache Drill &amp; Druid</strong></p><p>Apache Drill is a distributed SQL query engine for big <br>data exploration. It supports various data sources and multiple <br>data formats (CSV, JSON, etc.) and schema-free data exploration. <br>Drill uses a massively parallel processing architecture. It can <br>handle trillions of records in a matter of seconds and has a <br>unique design to avoid a single point of failure. <br>Druid is a high-performance, real-time analytics database. <br>that delivers sub-second queries on streaming and batch data <br>at scale and under load. Druid is a column-oriented and <br>distributed data source written in Java. It’s capable of <br>ingesting massive amounts of event data and offering low-latency queries on top of this data. Moreover, it offers the <br>possibility to slice and dice the data arbitrarily. <br>If we don’t have a pre-defined schema for our data and we <br>If you need a SQL-like interface, then the best option is Drill. If we <br>We are working with real-time or time series data, and we need low <br>If latency and performance are high, then the best option <br>is Druid.</p><p><strong>07. Apache Lucene &amp; Solr</strong></p><p>Apache Lucene is an open-source search engine library. <br>which provides a scalable and flexible platform for full text. <br>search. The architecture of Lucene is an inverted index architecture. <br>where it indexes every unique term and stores a list in <br>documents that contain that term. There are a few limitations. <br>to this. Inherently, it does not support distributed searching, and it <br>is not a complete search solution; it requires significant time. <br>and effort. On the other hand, Solr is a web application that <br>builds on top of Lucene. It gives Lucene’s capabilities and <br>user-friendly interface to search and manage indexed data. So <br>When we need a highly customizable search engine, the best <br>choice is Lucene, but when we need a less complex and user-friendly <br>friendly search engine, then it is Solr.</p><p><strong>08. Apache Mahout &amp; Spark MLib</strong></p><p>Apache Mahout and Spark MLib are both used for machine learning. <br>learning tasks in big data analytics. Spark MLib has built on <br>top of Apache Spark and Mahout built on MapReduce. Mahout <br>implements popular machine learning techniques such as <br>classification, clustering, and recommendation. Mahout offers <br>mature algorithms, especially for recommended systems. On <br>On the other hand, when we look into MLib, it is very user-friendly. <br>and less complex and more scalable. When we compare both <br>performance MLib is the choice for large-scale machine learning. <br>workloads, and Mahout is struggling with it. And Mahout is disk. <br>based processing, and MLib is in memory processing, so this <br>also a reason that MLib has a higher performance than <br>Mahout.</p><p><strong>Conclusion</strong></p><p>Big data processing and analytics are handled by a variety of tools and frameworks that are part of the Hadoop ecosystem, which are designed to integrate easily with each other. Below, I’ll outline the typical workflow using the tools I have discussed above.</p><p>First, most real-time data is ingested by Kafka. Then, we use connectors to transfer data between databases, data stores, and Hadoop. For this purpose, we use Sqoop and Flume. The data is then stored using HBase, Cassandra, and HDFS.</p><p>Next, we schedule, manage, and monitor the data processing using Apache Spark, Oozie, and Airflow. After processing, we analyze the data using Drill and Druid. If we need to perform a data search, we can use Solr and Lucene. For advanced analytics, such as prediction, grouping, and building recommendation systems, we use Mahout and Spark MLlib. Finally, to coordinate and manage these tools, we use ZooKeeper.</p><p><strong>References</strong></p><p>[1]https://www.datacamp.com/tutorial/apache-kafka-forbeginners-a-comprehensive-guide</p><p>[2] <a href="https://www.javatpoint.com/kafka-topics">https://www.javatpoint.com/kafka-topics</a></p><p>[3] <a href="https://www.tutorialspoint.com/apache_kafka/apache_kaf">https://www.tutorialspoint.com/apache_kafka/apache_kaf</a> ka_cluster_architecture.htm</p><p>[4] https://www.analyticsvidhya.com/blog/2022/09/apachesqoop-features-architecture-and-operations/</p><p>[5] https://www.analyticsvidhya.com/blog/2022/05/get-toknow-apache-flume-from-scratch/</p><p>[6] <a href="https://www.analyticsvidhya.com/blog/2023/03/introducti">https://www.analyticsvidhya.com/blog/2023/03/introducti</a> on-to-apacheoozie/#Main_Components_of_Apache_Oozie</p><p>[7] https://www.analyticsvidhya.com/blog/2022/08/apacheairflow-essential-guide/</p><p>[8] https://www.analyticsvidhya.com/blog/2022/10/a-briefintroduction-to-apache-hbase-and-its-architecture/</p><p>[9] https://www.geeksforgeeks.org/apache-cassandra-nosqldatabase/</p><p>[10] <a href="https://www.dremio.com/wiki/apache-drill/">https://www.dremio.com/wiki/apache-drill/</a></p><p>[11] https://www.baeldung.com/apache-druid-event-drivendata</p><p>[12] <a href="https://www.analyticsvidhya.com/blog/2020/11/introducti">https://www.analyticsvidhya.com/blog/2020/11/introducti</a> on-to-spark-mllib-for-big-data-and-machine-learning/</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ecc304fe665" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gradient Descent, Stochastic Gradient Descent and Mini Batch SGD]]></title>
            <link>https://medium.com/@usmaanrifkhan/gradient-descent-stochastic-gradient-descent-and-mini-batch-sgd-02bf85b130cc?source=rss-4e9691c34421------2</link>
            <guid isPermaLink="false">https://medium.com/p/02bf85b130cc</guid>
            <dc:creator><![CDATA[Usmaan Rifkhan]]></dc:creator>
            <pubDate>Sat, 03 Feb 2024 06:51:02 GMT</pubDate>
            <atom:updated>2024-02-03T06:55:12.308Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TNeM0rCfFJRIYcF37QpQaA.png" /></figure><p><strong>Lost Function,</strong></p><p>A loss function is a mathematical function that quantifies the difference between predicted and actual values in a machine learning model. It measures the model’s performance and guides the optimization process by providing feedback on how well it fits the data.</p><p><strong>Backpropagation,</strong></p><p>Backpropagation, also known as backward propagation of errors, is a widely employed technique for computing derivatives within deep feedforward neural networks. It plays a crucial role in various supervised learning algorithms used for training this type of neural networks.</p><p><strong>Loss Function for Regression,</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/0*6fy1UcueMWcNsMZc" /></figure><p><strong>Loss Function for Binary Classification,</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*-1iJV4Vx15LaIRJw" /></figure><p>How loss function work,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dIh9_aMddYjXxjqm" /></figure><p><strong>Gradient Descent</strong></p><p>Gradient descent is an optimization algorithm used in machine learning and deep learning to minimize a cost function (also known as a loss function or objective function). Its primary purpose is to find the values of the model’s parameters (such as weights and biases) that result in the lowest possible cost or error. In GD, we must run through ALL the samples in your training set to do a single update for a parameter in a particular iteration</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/0*Kj5icjkTS-jGk_aQ" /></figure><p>When we look at this figure, this is the graph of gradient descent. This is a convex graph. So when we look at the right side of the optimal solution, the derivative is positive, and when we look at the left side of it, The derivative is negative. So in this case, to get the low-cost function, it should come back. It means the weight should be low. (Weight new &lt; Weight old). Yes, mathematically, it is true. I will prove it below.</p><p>Equation of Weight update is,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/0*d23yqm-jIhvOcuyP" /></figure><p>So in the above case, the derivative is positive because tan Ø is between 0 and 90 degrees.</p><p>So, if (α * Ø) is a positive value, then it will subtract from the previous weight. So the new weight is lower than the old one.</p><p><strong>Stochastic Gradient Descent</strong></p><p>Stochastic Gradient Descent (SGD) is an optimization algorithm commonly used in machine learning and deep learning. It is a variant of the gradient descent algorithm designed to efficiently update the model parameters. In SGD, instead of using the entire dataset for each iteration, only a single random training example (or a small batch) is selected to calculate the gradient and update the model parameters.</p><p><strong>Mini Batch SGD</strong></p><p>Mini-batch Stochastic Gradient Descent (Mini-Batch SGD) is another common variation that strikes a balance between the efficiency of SGD and the stability of Batch Gradient Descent by updating the parameters based on a small random subset (mini-batch) of the training dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*gNh9IfwcpOcTgAFB" /></figure><p>Example of regression loss function to every gradient descent,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*szm5y7pUo0tU7o2r" /></figure><p><strong>Iterations vs Batch vs Epoch</strong></p><p>One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE. We divide large dataset into Number of Batches or sets or parts. Batch size means total number of training examples present in a single batch. Iteration is the number of batches needed to complete one epoch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/0*g5T_7WROIVWjS9bE" /></figure><p>Example to make more clear about GD, SGD and Mini batch SGD,</p><p>I have 1000 samples and my batch size is 100 and I am going to do 2 epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*qEDjslSQZn3Zs-jFzCnmAA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02bf85b130cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>