<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ROHITH RAMESH on Medium]]></title>
        <description><![CDATA[Stories by ROHITH RAMESH on Medium]]></description>
        <link>https://medium.com/@rohithramesh1991?source=rss-edd6101f90e7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bWTasSFkOQpQShu3PvCsiw.jpeg</url>
            <title>Stories by ROHITH RAMESH on Medium</title>
            <link>https://medium.com/@rohithramesh1991?source=rss-edd6101f90e7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 20:01:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rohithramesh1991/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Productionizing Retrieval-Augmented Generation (RAG) with Azure OpenAI, Azure Search, and…]]></title>
            <link>https://medium.com/@rohithramesh1991/productionizing-retrieval-augmented-generation-rag-with-azure-openai-azure-search-and-f56c8e676af9?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f56c8e676af9</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Fri, 31 Oct 2025 11:51:10 GMT</pubDate>
            <atom:updated>2025-10-31T11:53:33.332Z</atom:updated>
            <content:encoded><![CDATA[<h3>🚀Productionizing Retrieval-Augmented Generation (RAG) with Azure OpenAI, Azure Search, and Kubernetes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*_CD9Lj8S_eSEbETcymXZSg.gif" /></figure><p>Retrieval-Augmented Generation (RAG) has become one of the most practical patterns for building production-ready AI assistants. But most demos stop at the “local prototype” stage. What happens when you need to deploy one reliably — with observability, scalability, and clean integration into enterprise infrastructure?</p><p>In this project, I built a <strong>fully productionized RAG pipeline</strong> on <strong>Microsoft Azure</strong>, powered by <strong>FastAPI</strong>, <strong>Azure Cognitive Search</strong>, <strong>Azure OpenAI</strong>, and <strong>Prometheus monitoring</strong>, all deployed on <strong>Azure Kubernetes Service (AKS)</strong>.<br> The result: a flexible, observable, cloud-native RAG stack that can run on OpenAI or any open-source Hugging Face model.</p><h3>🧭 1. What This Project Does</h3><p>This setup powers a complete RAG workflow — from document retrieval to large language model (LLM) response — and exposes clean APIs for downstream integration.</p><p>The system handles:</p><ul><li>Query ingestion through <strong>FastAPI</strong></li><li>Context retrieval from <strong>Azure Cognitive Search</strong></li><li>Response generation via <strong>Azure OpenAI</strong> (or a Hugging Face model)</li><li><strong>Prometheus metrics exposure</strong> for latency, token usage, and cost</li><li>Deployment and scaling via <strong>Azure Kubernetes Service</strong></li></ul><p>At its core, this system is designed to be <strong>modular</strong>. You can easily swap out the LLM or retriever backend without rewriting the orchestration logic.</p><h3>🏗️ 2. System Architecture</h3><p>Here’s the high-level view of how everything fits together:</p><pre>User<br>  ↓<br>FastAPI (rag-api)<br>  ↓<br>Azure Cognitive Search  ← stores vector embeddings<br>  ↓<br>Azure OpenAI / Hugging Face model  ← generates response<br>  ↓<br>Prometheus  ← collects latency, token, and cost metrics<br>  ↓<br>Azure Kubernetes Service  ← orchestrates containers</pre><p>Each component runs as part of a <strong>containerized FastAPI application</strong>, deployed on AKS. The setup ensures:</p><ul><li>Horizontal scalability</li><li>Real-time health monitoring</li><li>Model flexibility</li><li>Metrics exposure for Prometheus scraping</li></ul><h3>⚙️ 3. Why FastAPI?</h3><p>FastAPI acts as the glue that ties everything together — lightweight, async, and perfect for production-grade inference services.<br> It exposes clean routes like /ask, /healthz, /readyz, and /metrics.</p><p>When a user sends a query to /ask, the app:</p><ol><li>Retrieves top documents from <strong>Azure Cognitive Search</strong>.</li><li>Sends both the query and retrieved context to the <strong>LLM endpoint</strong>.</li><li>Measures latency and token usage through the <strong>LLMCallTimer</strong>.</li><li>Returns the final answer with metrics recorded automatically.</li></ol><p>A simplified view of the main inference logic:</p><pre>with LLMCallTimer(provider=&quot;azure_openai&quot;, model=&quot;gpt-4o-mini&quot;) as t:<br>    response = llm.generate(context, query)<br>    t.record_success(prompt_tokens=659, completion_tokens=31)</pre><p>This instrumentation ensures that every request contributes to live operational metrics — without slowing the main inference flow.</p><h3>🔍 4. Retrieval with Azure Cognitive Search</h3><p>Azure Cognitive Search serves as the <strong>vector retriever</strong>. It stores document embeddings that are later fetched using semantic search or cosine similarity.</p><p>Embeddings can come from any model — in my case, sentence-transformers/all-MiniLM-L12-v2 deployed via SageMaker or Azure ML.<br> This makes the system model-agnostic: you can easily replace embeddings or LLMs without touching the orchestration logic.</p><h3>🧠 5. Generation with Azure OpenAI (or Hugging Face)</h3><p>For generation, I used <strong>Azure OpenAI’s GPT-4o-mini</strong> model initially. However, the API interface supports plugging in any endpoint — including <strong>open-source models from Hugging Face</strong> like Mistral or Falcon.</p><p>This flexibility ensures future-proofing: you can move between hosted and self-managed models depending on cost, latency, or privacy requirements.</p><p>The application can easily connect to Azure OpenAI endpoints via Azure Foundry or use a custom URL for an HF model deployed on Azure ML or SageMaker.</p><h3>📊 6. Observability with Prometheus</h3><p>LLM observability is essential in production. Without it, you don’t know how often your model fails, how long it takes to respond, or how much it costs per query.</p><p>To solve this, I integrated <strong>Prometheus metrics directly into the FastAPI app</strong>.<br> Metrics like latency, token usage, request counts, and estimated cost are collected automatically by the instrumentation class LLMCallTimer.</p><p>Prometheus scrapes the exposed /metrics endpoint.<br> You can then visualize data in <strong>Grafana</strong> to see:</p><ul><li>Average LLM latency</li><li>Tokens consumed per request</li><li>Success/error ratios</li><li>Cost accumulation over time</li></ul><p>Example metric entries include:</p><pre>llm_requests_total{model=&quot;gpt-4o-mini&quot;,provider=&quot;azure_openai&quot;,status=&quot;success&quot;} 1.0<br>llm_tokens_total{model=&quot;gpt-4o-mini&quot;,role=&quot;prompt&quot;} 659<br>llm_request_seconds_sum{model=&quot;gpt-4o-mini&quot;} 2.47</pre><p>The same /metrics endpoint is also used by Kubernetes for <strong>health and readiness probes</strong>.</p><h3>☸️ 7. Containerization and Deployment on AKS</h3><p>The deployment pipeline uses a <strong>multi-stage Docker build</strong>:</p><ol><li><strong>Builder stage</strong> — packages the app into a .whl file.</li><li><strong>Runtime stage</strong> — installs the wheel and runs uvicorn.</li></ol><p>Each successful GitHub Actions run pushes a new image to <strong>GitHub Container Registry (GHCR)</strong> with a unique commit tag.<br> Kubernetes is configured with imagePullPolicy: Always to always fetch the latest image.</p><p>Deployment is handled with kustomize overlays for each environment:</p><pre>kubectl -n rag-prod apply -k k8s/overlays/prod<br>kubectl -n rag-prod rollout restart deploy/rag-api</pre><p>Pods are orchestrated in <strong>Azure Kubernetes Service</strong>, behind an <strong>Application Gateway Ingress</strong>.<br> Traffic is routed via a public domain like:</p><pre>http://rag-demo-dns.eastus2.cloudapp.azure.com</pre><p>where /ask serves the API and /metrics serves Prometheus-compatible data.</p><h3>💾 8. Handling Metrics Exposure and Scaling</h3><p>FastAPI exposes /metrics natively via Prometheus client integration.<br> AKS automatically checks:</p><ul><li>/healthz → for readiness</li><li>/metrics → for Prometheus scraping</li></ul><p>Prometheus continuously gathers metrics and stores them in its time-series database, enabling Grafana dashboards for real-time monitoring.</p><p>Scaling can be managed through AKS Horizontal Pod Autoscaler (HPA), which can target CPU, memory, or even custom metrics (like LLM latency).<br> This ensures the RAG service remains responsive even under high query volume.</p><h3>📈 9. Results and Observations</h3><p>After deployment:</p><ul><li>Metrics were accessible both from within the pod and via ingress.</li><li>The /metrics output contained real-time LLM request stats.</li><li>Azure Cognitive Search retrieval worked seamlessly with the model endpoint.</li><li>The system automatically handled token, latency, and cost tracking.</li></ul><p>This observability proved critical — especially when switching models or tuning prompt strategies. It provided visibility into performance trade-offs between providers and models.</p><h3>🔧 9.1 Real-World Deployment, Testing, and Troubleshooting</h3><p>While building this end-to-end Azure RAG system, I also implemented full CI/CD automation, testing, and observability to ensure it behaves consistently across environments.</p><p>This section walks through how the pipeline, tests, and AKS rollout actually come together — and what real deployment issues taught me about productionizing RAG.</p><p><em>(Then paste or summarize the previous answer starting here — the “test folder → CI/CD pipeline → Docker wheel build → AKS rollout → metrics exposure → ingress fix → pod scheduling issue” explanation.)</em></p><h3>🔄 10. Extending the System</h3><p>This architecture is modular and can be extended in multiple directions:</p><ul><li>Replace <strong>Azure OpenAI</strong> with <strong>Hugging Face models</strong> hosted on Azure ML or SageMaker.</li><li>Add <strong>authentication and rate limiting</strong> to FastAPI for multi-user access.</li><li>Include <strong>OpenTelemetry tracing</strong> for deeper insight into RAG latency sources.</li><li>Integrate <strong>vector database alternatives</strong> (like Pinecone or Weaviate) alongside Azure Search.</li><li>Automate monitoring dashboards using Grafana’s provisioning API.</li></ul><h3>🧩 11. Why This Matters</h3><p>The key takeaway isn’t just that this works — it’s <em>how</em> it works together.</p><p>Azure’s ecosystem provides all the pieces for a production-ready LLM service:</p><ul><li>Cognitive Search for retrieval</li><li>OpenAI for generation</li><li>AKS for orchestration</li><li>Prometheus for metrics</li><li>FastAPI for orchestration and exposure</li></ul><p>By connecting these components cleanly, we create a <strong>reliable, observable, and modular</strong> RAG system that’s ready for real-world enterprise deployment.</p><h3>🎬 12. Live Demo and GitHub Repository</h3><p>Here’s the RAG system in action — deployed on Azure, powered by FastAPI, Azure OpenAI, and Cognitive Search:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qj_F1J3gxfV-nc3yKZ4ctw.gif" /></figure><p>You can explore the complete source code, including CI/CD pipelines, Kubernetes manifests, and Prometheus integration, on GitHub:</p><p>👉<a href="https://github.com/rohithramesh1991/azure-rag-llmops-end-to-end"><strong>GitHub Repo: </strong>Azure Rag end-to-end pipeline</a></p><p>🏁 <strong>Conclusion</strong></p><p>This end-to-end project demonstrates how to <strong>operationalize a RAG pipeline</strong> on Azure — from retrieval to inference to observability — using entirely cloud-native services.<br> It combines <strong>LLM engineering </strong>and <strong>LLMOps</strong> principles in one cohesive system.</p><p>With FastAPI handling requests, Azure Search powering retrieval, Azure OpenAI (or Hugging Face) handling generation, and Prometheus tracking performance, this stack brings the reliability of production systems to the world of generative AI.</p><blockquote><em>Whether you’re building enterprise chatbots or internal AI assistants, this pattern is a foundation for running RAG at scale — transparently, efficiently, and observably.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f56c8e676af9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build and Integrate MCP Servers for a Versatile Chat Agent]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-build-and-integrate-mcp-servers-for-a-versatile-chat-agent-cb0ea07b2b89?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb0ea07b2b89</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[gradio]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chat-agents]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Mon, 09 Jun 2025 20:49:04 GMT</pubDate>
            <atom:updated>2025-06-09T20:49:04.842Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine chatting with a single assistant that seamlessly checks IP reputations, provides weather forecasts, and queries databases — all through a simple, conversational interface. That’s exactly what the <strong>MCP-Powered Chat Assistant</strong> delivers. By leveraging the <strong>Model Context Protocol (MCP)</strong>, this assistant dynamically integrates custom API-based tools, pre-built services, and database servers into one cohesive experience.</p><h3>What Exactly is Happening Here?</h3><p>Under the hood, our setup features:</p><ul><li><strong>Two custom MCP servers</strong>:</li></ul><ol><li><strong>Weather Server</strong>: A custom API integration providing real-time weather conditions and forecasts via OpenWeatherMap.</li><li><strong>AbuseIPDB Server</strong>: A tool that checks IP addresses against an abuse database to detect potential threats.</li></ol><ul><li><strong>SQLite MCP Server</strong>: A prebuilt MCP server (mcp-server-sqlite) that lets the assistant run database queries dynamically against your local SQLite database.</li><li><strong>Conversational Frontend</strong>: A clean and intuitive <strong>Gradio chat interface</strong> lets users naturally interact with the assistant, making API calls and database queries feel like everyday conversation.</li></ul><h3>Why MCP?</h3><p><strong>Model Context Protocol (MCP)</strong>, as detailed in <a href="https://medium.com/@rohithramesh1991/understanding-model-context-protocol-mcp-50e0b701b9bf">this introductory article</a>, allows language models to dynamically connect to external tools, APIs, and databases with minimal overhead. With MCP:</p><ul><li>You define custom servers easily — each with clearly defined tools.</li><li>Your chat assistant automatically discovers and integrates these tools, creating a flexible and scalable setup.</li><li>You can effortlessly extend your assistant’s capabilities by simply adding new MCP servers.</li></ul><h3>How Does It Work (in Simple Terms)?</h3><p>When a user sends a message, the Gradio frontend passes it to the MCP-powered agent. The agent dynamically decides which server and tool to call — be it weather, IP reputation checks, or querying a database — based on the user’s request.</p><ul><li><strong>Custom MCP servers</strong> (weather and AbuseIPDB) provide real-time data fetched from external APIs.</li><li><strong>SQLite MCP server</strong> interacts directly with a local database, enabling robust querying capabilities.</li></ul><h3>Understanding Tools vs. Resources in MCP</h3><p>In this setup, the <strong>SQLite MCP server</strong> interacts with an external SQLite database file. Conceptually, this database file serves as a <strong>resource</strong> — it provides the underlying data the MCP tool server operates upon. However, within the MCP framework, this database is not explicitly defined as a separate MCP resource; rather, it functions as internal storage accessed by the MCP SQLite tool server’s query methods (tools).</p><p>This distinction highlights MCP’s flexibility: tools can internally manage and use external resources transparently, without explicitly declaring them in MCP interactions. Future extensions could explore explicitly managing such databases or external files as declared MCP resources, depending on application needs.</p><p>All these interactions happen behind the scenes using two primary communication methods for MCP servers and clients:</p><ul><li><strong>Standard I/O (stdio):</strong><br>This is a straightforward, low-overhead way for the MCP client and server to communicate via the operating system’s input/output streams. Since the MCP servers are launched as subprocesses on the <strong>same machine</strong>, stdio allows fast, reliable message passing without network overhead or complex connection management.</li><li><strong>Server-Sent Events (SSE):</strong><br>SSE enables streaming responses over HTTP, useful when you want to receive real-time, incremental updates from MCP servers — especially when servers run remotely or in distributed environments.</li></ul><p>In this project, <strong>stdio </strong>was chosen because all MCP servers run locally as subprocesses on the same system. This setup benefits from simplicity, minimal latency, and easy lifecycle management of server processes without needing network setup or web protocols.</p><h3>Extensible and Scalable</h3><p>This architecture is built for extensibility. Want to add another API or a new database query capability? Simply create another MCP server or integrate an existing one, update your MCP configuration file (configure_mcp.json), and you&#39;re done.</p><p>The MCP agent instantly recognizes and integrates the new tools into the conversational flow.</p><h3>From APIs to Databases, All in One Place</h3><p>Whether it’s querying real-time weather data, checking cybersecurity threats, or performing complex database operations — your users never leave the conversation. The MCP-Powered Chat Assistant makes traditionally complex tasks effortless and conversational.</p><h3>See It In Action</h3><p>The MCP Chat Assistant demonstrates a seamless multi-tool conversational experience. Users can effortlessly:</p><ul><li>Retrieve the list of available tools and services integrated into the system.</li><li>Perform database operations such as creating tables and inserting records.</li><li>Run complex SQL queries, like aggregating subscription counts by date.</li><li>Get real-time weather information for cities like London.</li><li>Check if specific IP addresses, such as 104.244.72.115, have been reported for abuse.</li></ul><p>All of these diverse queries are handled smoothly within the same chat interface, showcasing the power of combining custom MCP servers with prebuilt services and a unified conversational frontend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fteTr3MwFfGowsIV.gif" /></figure><h3>Try It Yourself</h3><p>The entire project — including the custom servers, MCP configuration, and frontend code — is available on GitHub:</p><p><a href="https://github.com/rohithramesh1991/Agent-With-Custom-MCP-Server"><strong>GitHub Repo: Agent With Custom MCP Support</strong></a></p><p>Explore it, extend it, and make it your own!</p><h3>Conclusion</h3><p>With MCP, creating sophisticated, multi-tool chat assistants becomes remarkably straightforward. Whether your goal is to streamline operations, simplify complex queries, or integrate diverse APIs and databases seamlessly, MCP provides the foundation for powerful conversational agents that truly understand and respond to your users’ needs.</p><p>Dive in, explore, and unleash the full potential of conversational AI with MCP!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb0ea07b2b89" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-build-and-integrate-mcp-servers-for-a-versatile-chat-agent-cb0ea07b2b89">How to Build and Integrate MCP Servers for a Versatile Chat Agent</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Intelligent Chat Agent with MCP Support: Smarter Conversations]]></title>
            <link>https://medium.com/analytics-vidhya/building-an-intelligent-chat-agent-with-mcp-support-smarter-conversations-1948b96270db?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1948b96270db</guid>
            <category><![CDATA[mcps]]></category>
            <category><![CDATA[groq]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[gradio]]></category>
            <category><![CDATA[mcp-server]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Fri, 06 Jun 2025 21:13:07 GMT</pubDate>
            <atom:updated>2025-06-06T21:18:47.994Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine having an intelligent assistant that can seamlessly interact with multiple Model Context Protocol (MCP) servers, understand your queries, and respond accurately based on dynamic system configurations. This is exactly what the MCP Chat Assistant does — a smart, conversational agent powered by a large language model, designed to help you query and utilize MCP servers effortlessly.</p><h3>What’s Inside This Agent?</h3><p>At its core, the agent leverages a powerful language model, configured with system instructions about the available MCP servers. It maintains conversation history and constructs contextual messages for the model to ensure coherent, relevant responses.</p><p>The key steps the agent performs are:</p><ul><li><strong>Dynamic System Context:</strong> It loads MCP server configurations dynamically so the assistant always knows the current set of available servers.</li><li><strong>History-aware Conversations:</strong> It maintains a memory of past interactions, helping the assistant provide meaningful, context-rich answers.</li><li><strong>Clean Response Output:</strong> It strips out internal reasoning tags from model responses, delivering only the useful information to the user.</li><li><strong>Interactive Chat UI:</strong> A sleek web interface built with Gradio allows users to easily chat with the assistant and receive prompt, informative replies.</li></ul><h3><strong>Technology Stack Highlights: LangChain Groq, Groq LPU, and MCP-Use</strong></h3><p>This agent leverages <strong>LangChain Groq</strong>, a powerful integration that connects large language models with Groq’s innovative hardware acceleration through the <strong>Groq LPU (Logic Processing Unit)</strong>. The Groq LPU delivers ultra-low latency and high throughput for AI workloads, enabling faster, more efficient inference for complex models. Using LangChain Groq ensures our assistant responds quickly and scales gracefully under load.</p><p>For orchestrating interactions with MCP servers, we utilize the <strong>mcp-use</strong> library — a lightweight client and agent framework designed specifically for Model Context Protocol systems. It simplifies managing MCP servers, handling requests, and maintaining conversational memory, making the integration with external tools seamless and developer-friendly. You can explore more about mcp-use on its <a href="https://github.com/mcptutorial/mcp-use">GitHub repository</a>.</p><p>Together, these technologies create a robust foundation, blending powerful model serving, hardware acceleration, and modular tool integration to build a responsive and intelligent conversational assistant.</p><h3>Why This Matters</h3><p>MCP servers enable a modular and extensible way to connect language models with external tools and data sources. By building a chat agent that understands and leverages these MCP servers, users can efficiently orchestrate complex workflows, get precise information, or automate tasks — all through simple natural language conversations.</p><p>The assistant’s ability to parse the server configuration on-the-fly and adapt its responses makes it highly flexible for changing environments or new MCP server integrations.</p><h3>See It In Action</h3><p>The video demonstrates the agent responding to queries about the MCP servers, listing available servers, and providing relevant information dynamically. This real-time interaction highlights how the assistant handles conversation history and keeps the context aligned with user inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Tyu44GAJvfA_YBn8Lu7ug.gif" /></figure><h3>Learn More &amp; Get Started</h3><ul><li>Curious about <strong>what MCP is and how it works?</strong> Check out <a href="https://medium.com/@rohithramesh1991/understanding-model-context-protocol-mcp-50e0b701b9bf">Understanding Model Context Protocol (MCP)</a> for a comprehensive guide.</li><li>Want to explore <strong>how to use Copilot as an MCP-powered agent in VS Code?</strong> See <a href="https://medium.com/@rohithramesh1991/how-to-connect-vs-code-to-mcp-servers-and-use-copilot-chat-as-an-agent-257ec458924c">How to connect VS Code to MCP servers and use Copilot Chat as an agent</a>.</li><li>Explore the full source code and setup instructions on GitHub: <a href="https://github.com/rohithramesh1991/Agent-With-MCP-Support">Agent With MCP Support</a>.</li><li>For a list and details of MCP servers available, visit the <a href="https://github.com/modelcontextprotocol/servers">MCP Servers repository</a>.</li></ul><h3>Final Thoughts</h3><p>This MCP Chat Assistant exemplifies how conversational AI can be empowered with modular external toolkits to provide a rich, interactive user experience. By combining dynamic configuration, persistent context, and a smooth interface, it creates a powerful bridge between users and complex MCP ecosystems — making technology more accessible and effective.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1948b96270db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/building-an-intelligent-chat-agent-with-mcp-support-smarter-conversations-1948b96270db">Building an Intelligent Chat Agent with MCP Support: Smarter Conversations</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Connect VS Code to MCP Servers and Use Copilot Chat as an Agent]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-connect-vs-code-to-mcp-servers-and-use-copilot-chat-as-an-agent-257ec458924c?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/257ec458924c</guid>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[github-copilot]]></category>
            <category><![CDATA[configuration]]></category>
            <category><![CDATA[mcp-client]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Thu, 05 Jun 2025 23:08:29 GMT</pubDate>
            <atom:updated>2025-06-05T23:08:29.577Z</atom:updated>
            <content:encoded><![CDATA[<p>The Model Context Protocol (MCP) is quickly becoming the universal standard for connecting AI assistants — like Copilot Chat — to a wide variety of tools and data sources. With recent updates, you can now enable MCP support in VS Code and connect to any number of open-source MCP servers, allowing Copilot Chat to act as a powerful agent capable of real-world tool use.</p><p>In this walkthrough, you’ll learn how to enable MCP in VS Code, configure it to connect to popular MCP servers, and use Copilot Chat in agent mode — just like in the live demo video.</p><h3>Step 1: Enable MCP Support in VS Code</h3><ol><li><strong>Open VS Code.</strong></li><li>Go to Settings (Ctrl + , or Cmd + , on Mac).</li><li>In the search bar, type <strong>“MCP”</strong>.</li><li>Enable the <strong>MCP integration</strong> option (toggle switch).</li></ol><h3>Step 2: Add MCP Server Configurations</h3><ol><li>Open the settings as <strong>JSON</strong> (click the {} icon or search for “Open Settings (JSON)”).</li><li>Add the following configuration under the &quot;mcp&quot; section:</li></ol><pre>&quot;mcp&quot;: {<br>    &quot;inputs&quot;: [],<br>    &quot;servers&quot;: {<br>        &quot;playwright&quot;: {<br>            &quot;command&quot;: &quot;npx&quot;,<br>            &quot;args&quot;: [<br>                &quot;@playwright/mcp@latest&quot;<br>            ]<br>        },<br>        &quot;duckduckgo-search&quot;: {<br>            &quot;command&quot;: &quot;npx&quot;,<br>            &quot;args&quot;: [<br>                &quot;-y&quot;,<br>                &quot;duckduckgo-mcp-server&quot;<br>            ]<br>        },<br>        &quot;airbnb&quot;: {<br>            &quot;command&quot;: &quot;npx&quot;,<br>            &quot;args&quot;: [<br>                &quot;-y&quot;,<br>                &quot;@openbnb/mcp-server-airbnb&quot;,<br>                &quot;--ignore-robots-txt&quot;<br>            ]<br>        }<br>    }<br>}</pre><ul><li>You can add or remove servers as needed.</li><li>For more available servers and their documentation, visit the <a href="https://github.com/modelcontextprotocol/servers">official MCP servers repo</a>.</li></ul><h3>Step 3: Start MCP Servers from VS Code</h3><ul><li>After you save your server configs in<strong> &quot;</strong><strong>settings.json&quot;, </strong>VS Code will<strong> </strong>automatically detect the new MCP servers.</li><li>In the <strong>MCP section</strong> (usually found in the activity bar or via the Command Palette), you’ll see options to <strong>Start</strong>, <strong>Restart</strong>, or <strong>Stop</strong> each server directly — no terminal commands needed.</li><li>Alternatively, press Ctrl+Shift+P (or Cmd+Shift+P on Mac) and type &gt;mcp to see all available MCP commands and servers. You can add or start/stop servers from here as well.</li><li>Make sure the desired servers are running and show as “active” in the VS Code UI before chatting with Copilot.</li></ul><h3>Step 4: Use Copilot Chat as an Agent</h3><ol><li>Open <strong>Copilot Chat</strong> in VS Code.</li><li><strong>Switch to Agent Mode</strong> (look for an option or toggle — often at the top or in the chat settings).</li><li>Start chatting!</li></ol><ul><li>Now, when you ask questions, Copilot Chat will act as an agent — choosing tools and making requests to the MCP servers you’ve configured.</li><li>For example, ask:</li><li>“Search DuckDuckGo for the latest news on MCP servers.”</li><li>“Find available Airbnbs in Paris.”</li><li>“Automate a browser task with Playwright.”</li></ul><p>You’ll see Copilot Chat reasoning, selecting tools, and fetching live data from the servers you connected.</p><h3>Video Demo</h3><p>See the whole process in action in this demo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L-0V5niX41rjVDod6XSFXQ.gif" /></figure><p>Ready to supercharge your AI workflow? Try it today and experiment with different MCP servers for your own use cases!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=257ec458924c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-connect-vs-code-to-mcp-servers-and-use-copilot-chat-as-an-agent-257ec458924c">How to Connect VS Code to MCP Servers and Use Copilot Chat as an Agent</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Model Context Protocol (MCP)]]></title>
            <link>https://medium.com/analytics-vidhya/understanding-model-context-protocol-mcp-50e0b701b9bf?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/50e0b701b9bf</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mcp-protocol]]></category>
            <category><![CDATA[mcp-client]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Thu, 05 Jun 2025 20:58:54 GMT</pubDate>
            <atom:updated>2025-06-05T20:58:54.874Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3owh1JXMkByeh2-Q25xcA.gif" /><figcaption>MCP Architecture</figcaption></figure><p>As Large Language Models (LLMs) become the backbone of AI applications, developers face growing complexity in integrating these models with various tools and data sources. <strong>Model Context Protocol (MCP)</strong>, an open-source protocol developed by Anthropic, is rapidly emerging as a game-changer — serving as a “USB-C port” for AI applications. MCP offers a standardized, extensible method for connecting LLMs with the diverse ecosystem of tools, APIs, and services that power modern AI assistants.</p><h3>What is MCP?</h3><p>At its core, MCP is a protocol designed to <strong>standardize how applications provide context and facilitate tool-calling for LLMs</strong>. Imagine trying to connect a laptop to multiple devices — monitors, keyboards, drives — without a common port like USB-C; integration would be a headache. MCP solves a similar problem in AI: rather than writing bespoke code for every tool or API, developers can rely on MCP to handle communication, discovery, and data exchange in a uniform way.</p><h3>Why Is MCP Important?</h3><p>Before MCP, integrating a new tool (like a weather API or SQL database) with an LLM meant building custom “glue” code. Each new tool, or even an update to an existing one, could break the integration or require tedious rewrites.</p><p><strong>MCP offers:</strong></p><ul><li><strong>Standardization:</strong> Integrate tools once, in a predictable way.</li><li><strong>Scalability:</strong> Add or swap tools with minimal code changes.</li><li><strong>Reduced Complexity:</strong> Developers focus on core logic, not wiring up integrations.</li></ul><h3>Core Architecture: The MCP Stack</h3><p>MCP organizes tool integrations around three main components:</p><ul><li><strong>MCP Host:</strong> The environment running the AI application (e.g., IDE, cloud service).</li><li><strong>MCP Client:</strong> Bridges communication between the host and the tools/services.</li><li><strong>MCP Server:</strong> Connects to tools and services, exposing them to the client in a standardized way.</li></ul><p>This architecture decouples tool providers from the LLM and host, making it easy to swap or update tools without major rewrites.</p><h3>Tools, Prompts, and Resources in MCP Servers</h3><p>MCP servers expose <strong>tools</strong> (discrete actions like database queries or API calls), <strong>prompts</strong> (task-specific templates for LLM completions), and <strong>resources</strong> (files or structured data). All of these are described using standard schemas, making them discoverable and invocable by LLMs via the MCP client. This system allows LLM-powered agents to reason about what tools are available, chain tool usage, and securely access the resources and data they need — all while following the host’s policies and permissions.</p><ul><li><strong>Tools:</strong> Standardized, schema-defined functions or actions the LLM can call.</li><li><strong>Prompts:</strong> Templates or guides for LLM completions and agent behaviors.</li><li><strong>Resources:</strong> Files, data, or assets that tools or the user may access, with clearly defined access boundaries.</li></ul><p>This standardization is a major driver of MCP’s flexibility and scalability in real-world AI development.</p><h3>Communication Protocols in MCP: stdio and SSE</h3><p>MCP supports multiple communication protocols between clients and servers for maximum flexibility. The two main options are:</p><ul><li><strong>stdio (Standard Input/Output):</strong> Uses simple text streams, making it easy to run servers as local subprocesses and ensuring broad compatibility across platforms and languages.</li><li><strong>SSE (Server-Sent Events):</strong> Uses HTTP for real-time event streaming, ideal for web-based or cloud deployments and for scenarios where clients need live, incremental updates from servers.</li></ul><p>This protocol flexibility lets developers choose the best fit for their environment — local, web, or cloud — without changing application logic.</p><h3>How Does MCP Work? The Communication Flow</h3><ol><li><strong>User Input:</strong> The user asks a question or requests an action (e.g., “How many customers are in the database?”).</li><li><strong>Discovery:</strong> The MCP host queries the MCP server for available tools and capabilities.</li><li><strong>LLM Planning (Sampling #1):</strong> The LLM is invoked to interpret the user’s intent, decide what tool to use, and generate the tool call (function name, parameters, SQL, etc.).</li></ol><ul><li><em>This process is called </em><strong><em>sampling</em></strong><em> — the LLM “samples” a plan or tool call.</em></li></ul><p>4.<strong> Tool Execution:</strong> The MCP server executes the requested tool/action (e.g., runs the SQL query).</p><p><strong>5. Result Handling (Sampling #2):</strong> The LLM is invoked again (sampling) to turn the raw tool result into a user-friendly answer.</p><ul><li><em>Another sampling step — formatting the result for the user.</em></li></ul><p><strong>6. User Output:</strong> The assistant delivers the final, natural-language answer.</p><blockquote><strong>Example:</strong></blockquote><blockquote><strong>User:</strong> “How many customers are in the database?”</blockquote><blockquote><strong>Sampling #1:</strong> LLM decides to call the SQL tool and generates the function call.</blockquote><blockquote><strong>Server runs query, returns:</strong> { &quot;result&quot;: 142 }</blockquote><blockquote><strong>Sampling #2:</strong> LLM generates: “There are 142 customers in the database.”</blockquote><blockquote>User receives the answer.</blockquote><h3>Real-World Agentic Scenarios and Sampling</h3><p>The power of MCP really shines in “agentic” workflows, where the LLM acts as an intelligent agent — planning, choosing tools, executing multi-step workflows, and delivering responses. Every time the LLM is called for planning, decision-making, or formatting, it’s <strong>performing a sampling step</strong>. MCP provides a standardized way for servers to request these LLM inferences, even supporting nested tool-calling for more complex behaviors.</p><h3>Where Does Sampling Happen?</h3><ul><li><strong>Planning:</strong> LLM generates a plan/tool call.</li><li><strong>Formatting:</strong> LLM converts tool results to user-facing language.</li><li><strong>Nested Calls:</strong> Servers can request further LLM completions inside tool flows (e.g., reasoning steps).</li></ul><h3>Advanced MCP Features: Roots and Sampling</h3><p><strong>Roots:</strong><br> MCP allows clients to define which directories or files a server can access (“roots”). This is essential for sandboxing, security, and permission control in applications that interact with the filesystem.</p><p><strong>Sampling:</strong><br> Sampling is the protocol for requesting LLM completions (text, code, or images) via the client, with full control over model selection and permissions. This means:</p><ul><li>Servers do not need their own LLM API keys.</li><li>Clients retain control over AI model access and context.</li></ul><h3>Benefits of Using MCP</h3><ul><li><strong>Less Boilerplate:</strong> No need for custom integration logic for each tool.</li><li><strong>Easy Maintenance:</strong> Tools and services can be updated independently of the LLM logic.</li><li><strong>Flexibility:</strong> Easily plug-and-play new tools or swap out services as needed.</li><li><strong>Security &amp; Control:</strong> Fine-grained access to filesystems and model selection via “roots” and “sampling.”</li></ul><h3>Example Implementation: Weather API with MCP</h3><p>As a demonstration, an MCP host can register a weather API as an MCP server. The user can ask, “What’s the weather in Paris tomorrow?” The LLM plans the tool call (sampling), the server fetches the weather, and the LLM formats the response (sampling again) — all handled via MCP’s standardized interface.</p><h3>Looking Forward</h3><p>MCP is evolving rapidly, with new features and integrations appearing regularly. As agentic AI applications become more common, MCP’s role as a universal context and tool-calling protocol will only grow. Developers interested in harnessing the power of AI assistants should consider exploring MCP and following its development closely.</p><blockquote><strong><em>For further learning:</em></strong><em> Dive into the </em><a href="https://spec.modelcontextprotocol.io/"><em>MCP specification</em></a><em>.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50e0b701b9bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/understanding-model-context-protocol-mcp-50e0b701b9bf">Understanding Model Context Protocol (MCP)</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turning Chat Into Action: Why Structured Conversations Make LLMs Truly Smart]]></title>
            <link>https://medium.com/analytics-vidhya/turning-chat-into-action-why-structured-conversations-make-llms-truly-smart-d6fe27d66749?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6fe27d66749</guid>
            <category><![CDATA[llm-function-calling]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[agentic-ai-systems]]></category>
            <category><![CDATA[chat-agents]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Thu, 05 Jun 2025 09:37:20 GMT</pubDate>
            <atom:updated>2025-06-05T09:48:23.394Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlThUBd3tMFSHjqaB_8Ymw.gif" /><figcaption>Function Calling Architecture</figcaption></figure><p><strong>🤖 Why We Need Smarter Agents</strong></p><p>As LLMs (large language models) get more powerful, the demand is rising for <em>agents</em> — LLM-powered assistants that can actually <strong>use tools</strong>, call APIs, and orchestrate real actions on your behalf.</p><p>But there’s a catch:<br> Too often, these agents get “stuck” in one mode, forget context, or struggle to connect the dots across multi-turn conversations. That’s where the right <em>architecture</em> — not just prompt engineering — makes all the difference.</p><p><strong>🌱 Chat Agent with Function-Calling Support</strong></p><p>I built this agent with a simple goal:<br> <strong>Let an LLM call real Python functions (tools) based on your request, remember what happened, and use those results to answer naturally — even after several conversational turns.</strong></p><p><strong>💡 What Makes This Agent Different?</strong></p><p><strong>1. Structured Message History (“Agent Memory”)</strong><br> Instead of keeping a flat list of user/assistant turns, this agent logs every message — <strong>including user questions, assistant suggestions, and tool results</strong> — with rich structure:</p><ul><li>role: user — what the human said</li><li>role: assistant — what the agent replied</li><li>role: tool — the <em>actual</em> result of any function or API call</li><li>…and more if you like (system, Slack messages, etc.)</li></ul><p>This means the agent can <strong>“recall” past tool results</strong>, not just plain text!</p><p><strong>2. Tools That Stay in Context</strong></p><p>Let’s say you check if an IP address is abusive, then come back a few turns later and say, “Send that report on Slack.”</p><ul><li>The agent looks back in its message history,</li><li>Finds the latest tool result (the IP abuse check),</li><li>Summarizes or references that info,</li><li>And builds the right Slack message with the <strong>actual, most recent data</strong> — no guessing, no hallucination.</li></ul><p>This is true “memory” in action.<br> Your agent doesn’t just answer the current question; it leverages everything that happened before.</p><p><strong>3. Robust Prompting to Avoid Loops</strong></p><p>By carefully managing <em>when</em> the agent should suggest a tool call versus <em>when</em> it should just answer, we avoid the classic “function-calling loop” that plagues many LLM agents.</p><p>After a tool call, the agent gets clear instructions:<br> <em>Use the tool result and answer the question. Do not call another tool unless the user asks.</em></p><p><strong>🛠️ Extensible by Design</strong></p><p>The coolest part? <strong>You can plug in any Python function as a “tool”</strong>:</p><ul><li>Call external APIs (weather, IP reputation, etc.)</li><li>Integrate with Slack, email, or ticketing systems</li><li>Chain tools (e.g., fetch a report, summarize, then send)</li><li>Even build workflows: alert escalation, onboarding, security checks, and more.</li></ul><p>Just register your function, add a description, and you’re ready to go.<br> The agent’s structured memory ensures every tool result is available for future steps.</p><p>🎬 **Watch the Agent in Action!** <br>See a live demo of the function-calling agent connecting chat and code seamlessly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_s-2LvUfM9oZvE4oYz0MQ.gif" /><figcaption>Live Demo of Agent</figcaption></figure><p><strong>🚀 What Else Could This Agent Do?</strong></p><ul><li><strong>Multi-step automation:</strong> Handle more complex workflows, with the agent guiding the user step by step, recalling past results for every action.</li><li><strong>Dynamic argument gathering:</strong> If a required argument is missing (e.g., a Slack user or channel), the agent can look up available options, prompt the user, and then proceed — <em>all in a single conversational flow</em>.</li><li><strong>Smart notifications:</strong> Instead of just answering, the agent can notify a team, escalate urgent issues, or even kick off processes in external systems.</li><li><strong>Plug-and-play integrations:</strong> Add new tools — internal APIs, databases, cloud services, whatever your business needs.</li></ul><p><strong>📝 Takeaways</strong></p><ul><li><strong>Structured message history = agent memory</strong><br> If you want your LLM agent to behave intelligently across many turns, don’t just save plain text. Log every action and tool result.</li><li>**Prompt engineering isn’t enough — **you need the right architecture to manage state, recall, and when to act.</li><li><strong>Tool memory unlocks real use-cases:</strong> Security automation, IT helpdesks, workflow orchestration, and anything else that needs both conversation and real actions.</li></ul><p><strong>💻 Try It Out!</strong></p><p><strong>Check out the repo and see for yourself:</strong><br> 👉 <a href="https://github.com/rohithramesh1991/Agent-with-Function-Calling-Support"><em>Agent-with-Function-Calling-Support (GitHub)</em></a></p><p>Would love to hear your feedback, ideas, and see what tools <em>you</em> would connect next!</p><p>#AI #LLM #FunctionCalling #Agent #Automation #Python #OpenSource #ConversationalAI #Ollama #PromptEngineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6fe27d66749" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/turning-chat-into-action-why-structured-conversations-make-llms-truly-smart-d6fe27d66749">Turning Chat Into Action: Why Structured Conversations Make LLMs Truly Smart</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamlining Data Ingestion in Azure: Integrating REST APIs and CSV Files into Data Lake Using…]]></title>
            <link>https://medium.com/@rohithramesh1991/efficient-data-integration-in-azure-streamlining-rest-api-and-csv-uploads-to-data-lake-with-azure-231158af358e?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/231158af358e</guid>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[azure-storage-account]]></category>
            <category><![CDATA[rest-api-call]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data-ingestion]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Sun, 11 Feb 2024 12:20:11 GMT</pubDate>
            <atom:updated>2024-02-11T12:30:43.651Z</atom:updated>
            <content:encoded><![CDATA[<h3>Streamlining Data Ingestion in Azure: Integrating REST APIs and CSV Files into Data Lake Using Azure Data Factor</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*Ba4HN9SNIv0z4-XeFCvYxA.jpeg" /><figcaption><strong>Data Ingestion into Azure Data Lake Storage Gen2: Orchestrating Diverse Data Streams with Azure</strong></figcaption></figure><p>In this data-centered world, it is crucial for organizations that want to benefit from analytics and machine learning to know how to combine, keep and control their Data. Azure Data Lake Storage is one of the options available on Microsoft’s Azure cloud platform which offers strong solutions for data storage and analysis with its outstanding scalability, security, and performance. This article focuses on two main approaches that can be used towards loading data into Azure Data Lake Storage i.e. REST APIs and GitHub repositories used for real-time data integration as well as static sources like CSV files respectively. No matter if you are a data engineer or a data scientist, having these skills will enable you to employ Azure’s features for managing big data workloads. Please note that for actions listed below, it is necessary to have an active Azure account before going too technical. Here’s a where you can set up a <a href="https://azure.microsoft.com/en-gb/free">free Azure account</a> for those who are unfamiliar with it. Please be aware that this post does not cover account setup; instead, we will focus on the useful features of data loading.</p><p>To be precise, before going into details, we must identify the Azure services and infrastructure for data ingestion:</p><ol><li><strong>Start with Creating an Azure Data Lake Storage Account</strong>: Setting up an ADLS account that serves as your main data depot to handle any type of data.</li><li><strong>Create an Azure Data Factory Service</strong>: This is critical because it allows one to organize and automate workflows involving data. To enable movement of data form one service to another you need to create an instance of ADF.</li><li><strong>Create a Pipeline in Azure Data Factory</strong>: Set up a pipeline in ADF which will allow you ingest the data into ADLS upon having ADLS and ADF ready. It involves configuration of data sources (e.g REST API, GitHub CSV file), optionally transforming the data and specifying the destination as ADLS. The pipeline can be scheduled or monitored so that there is continuous flow of relevant information for storage at ADLS.</li></ol><p>To familiarize ourselves with the Azure ecosystem, let’s explore several essential components: Resource Groups, Azure Data Lake Storage (ADLS), and Azure Data Factory (ADF). These elements are crucial for organizing, storing, and processing data within Azure’s comprehensive cloud infrastructure.</p><h4><strong><em>What is a Resource Group in Azure?</em></strong></h4><p>Resource Group in Azure is a fundamental building block of Azure management, serving as a container that holds related resources for an Azure solution. A Resource Group could be viewed as an assembly point for your application elements whereby you may act on them collectively. This allows for convenient handling of resources, including removing assets from a group when not needed anymore, applying rules at the group level, or collecting together billing and monitoring. For Azure resource management, it is an indispensable part because it helps organize and govern resources based on project requirements, life-cycle stages and access control policies. Normally, you would put together items with similar lifecycle so they can all be deployed, updated or deleted together.</p><p><strong><em>How to create a Resource Group?</em></strong></p><ul><li>Navigate to the Azure Portal and sign in with your credentials.</li><li>To locate “Resource groups” in the Azure portal, go to the services menu or use the search bar.</li><li>Click on “+ Create” so as to start a new resource group.</li><li>Use a suitable name that describes the resources you want to manage.</li><li>choose your subscription and then select any region available within it where your resources will be based for high performance purposes.</li><li>Lastly, click on “Create” after reviewing all this information to create your new resource group.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4NGLeBEpQ6qf8eQ23cVOA.gif" /><figcaption><strong>Creating Resource Group</strong></figcaption></figure><h4><strong>Overview of Azure Data Lake Storage</strong></h4><p><strong><em>What is Azure Data Lake Storage (ADLS)?</em></strong></p><p>Azure Data Lake Storage (ADLS) is a business class storage solution built for analytics of huge datasets. It’s an enormous-scale repository that can store any size, type and speed of data from anywhere. ADLS allows storing and analyzing petabytes of data through its ability to scale massively. Thus, it has a hierarchical file system which facilitates the management of data as well as optimized for high performance analytics workloads leading to high-throughput plus low-latency. For example, there is Azure Active Directory integration with this platform that improves security and helps comply with industry regulations by providing fine grained access control. For instance, it uses tiered storage options in order to balance between access speed and storage costs such that it is cost-effective; moreover, ADLS seamlessly integrates with key cloud-based analytics services like Azure Databricks, Azure Data factory and Azure Synapse Analytics which facilitate faster processing and analysis.</p><p><strong><em>Why do we need Azure Data Lake Storage?</em></strong></p><p>There are several main reasons why we need Azure Data Lake Storage:</p><ul><li>Scalability: To be able to store and analyze petabytes of data in different formats and types, which will enable complicated analytical workloads.</li><li>Performance Optimization: Required for high throughput and low latency in big data analytics, machine learning, and real-time analytics.</li><li>Security and Compliance: For ensuring secure handling of data in line with the needs of Azure Active Directory integration with fine-grained access control.</li><li>Cost-Effective Storage: For an effective storage cost management by using tiered storage options that balance between fast access speed and storage expenses.</li><li>Integration with Analytics Services: So as not to move the stored data across environments when performing processing or analysis, but just make use of Azure’s analytic ecosystems for this purpose.</li><li>Flexibility: In its native format without predefined schemas, Azure Data Lake Storage can accommodate various types of data; structured or unstructured even semi-structured like JSON, logs, CSVs ,and Parquet files.</li></ul><p><strong><em>How to Create an Azure Storage Account and Set Up Services Within a Resource Group ?</em></strong></p><ul><li>To locate “Storage Accounts” in the Azure portal, go to the services menu or use the search bar.</li><li>Click on “+ Create” so as to start a new resource group.</li><li>Choose your new resource group from the dropdown, enter a unique name for the storage account, select the region, and configure other settings such as performance and replication if required.</li><li>To name your Azure Storage Account, it is vital that you choose a name which is not similar to any other names in Azure. The reason for this is that the storage account name is what you use to form DNS endpoint where your storage resources can be accessed. Therefore, the name must be globally unique within Azure to ensure the endpoint is distinct and reachable on the web.</li><li>Under advanced enable “allow public access to individual containers” and “hierarchical name space” option. We can keep all other option as default.</li><li>Review the settings and click “Create” after the validation check.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcf0ld-MuyWIGRNpjvECHQ.gif" /><figcaption><strong>Creating a Storage Account</strong></figcaption></figure><p>Once you have your Azure Storage Account in place, the next step is to organize your data. This is where containers come into play.</p><h4><strong>Containers</strong></h4><p>A container in an Azure Storage Account operates as a directory within the Blob service of that account. It helps organizing and managing blobs in it such that they are similar to folders on the file system of a computer. Containers aid in grouping related sets of blobs under one namespace.</p><p><strong><em>Why Do We Need Containers?</em></strong></p><p>Containers are required for separating different types of data, managing access and keeping an orderly system for large amounts of unstructured data. They allow for granularity of permissions so you can securely and specifically access parts of your data.</p><p>The Key Difference Between a Storage Account and a Container:</p><ul><li>A Storage Account is the root level of the Azure Storage service that provides a unique namespace for your Azure Storage data, accessible from anywhere in the world over HTTP or HTTPS.</li><li>A Container is a part of the Blob service within your storage account, used to group and isolate blobs in a secure and manageable fashion.</li></ul><p><strong><em>How to Create a container within a Storage Account?</em></strong></p><ul><li>Go to the Resource Group where your storage account is located.</li><li>Choose the Storage Account of your choice.</li><li>In the Storage Account overview, find and click on “Containers” under “Data Storage”.</li><li>Click on the “+ Container” to create a container.</li><li>Name your container and also determine how you can access it publicly.</li><li>Once you have created the container, you can add another folder inside it for better organization.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EIRMuX1tGHiT6GAVmvh2vA.gif" /><figcaption><strong>Creating a Container and a Directory inside the Container</strong></figcaption></figure><p>Further to our understanding of its core components, let us proceed and focus on Azure Data Factory (ADF), a managed cloud service which is a backbone for big data integration.</p><h4><strong><em>What is Azure Data Factory (ADF) ?</em></strong></h4><p>Azure Data Factory is a cloud-based data integration service for creating data-driven workflows that can orchestrate and automate the movement and transformation of your data. ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform) processes can be built with ease using ADF since it has a codeless interface to create, schedule, and manage data pipelines.</p><p><strong><em>How to Create an Azure Data Factory service?</em></strong></p><ul><li>In the Azure portal, Search for “Data Factory” and select it.</li><li>In the Data Factory creation page, fill in the necessary information such as the name of the data factory, the subscription, the resource group (select the one you have set up), and the region.</li><li>Click on “Review + create” after filling in the details. Once the validation is passed, select “Create” to deploy the data factory.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hLlV6tAuqXlJZjv6jMFS_A.gif" /><figcaption><strong>Creating Azure Data Factory Service</strong></figcaption></figure><p>Having established the Azure Data Factory service within our resource group, the next critical step is to orchestrate the data ingestion process, seamlessly integrating data from diverse sources into our analytical ecosystem.</p><p>ADF is a versatile tool for data ingestion supporting various integration patterns. In this article, we will discuss two important means by which data can be ingested into Azure through ADF namely REST API and GitHub raw source.</p><h4>A CSV File From GitHub:</h4><ol><li><strong><em>Setting Up the Pipeline and Copy Activity</em></strong></li></ol><p>In Azure Data Factory Studio:</p><ul><li>Navigate to “Author” section of the ADF Studio.</li><li>Create a New Pipeline: Click on “+”, then choose “Pipeline”. Name your new pipeline, for example, “ingesting-data”, to reflect its purpose.</li><li>Add a Copy Activity: Find “Copy Data” in the “Move &amp; transform” section under “Activities”. Drag this activity to your pipeline canvas. You can rename this activity to something more descriptive, like “Teams”, to match the data you’re ingesting. This is done in the “General” tab of the activity settings.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nhDrqI1V7VpSH38MJwowFA.gif" /><figcaption><strong>Setting up the Copy activity</strong></figcaption></figure><p><strong><em>2. Configuring the Source (HTTP for GitHub CSV File)</em></strong></p><ul><li>In your pipeline, select a copy activity that you renamed to “Teams”.</li><li>Configure Source: On the source tab of the copy activity, you start configuring by clicking on ‘+ New’ next to the source dataset field.</li><li>Select HTTP Source: Because it is GitHub raw file link used as data source, just choose HTTP as type of dataset.</li><li>Set the file format to “CSV”, aligning with the nature of your source data.</li><li>Assign Logical Name: Under Set properties use a logical name, like using the actual file name.</li><li>Create Linked Service for HTTP: Under linked service click ‘+New’ and then name it ‘Teamshttp’. Give raw GitHub URL path in the “Base URL” field. If no authentication is needed, set Anonymous as authentication type.</li><li>Finalize Source Settings: Ensure that ‘First row as header’ checkbox is marked and ‘Import schema’ says none. Save your configuration to finalize the source setup.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TDgKdAPi6UmU81Wuw4A3Ag.gif" /><figcaption><strong>Source Configuration</strong></figcaption></figure><p><strong><em>3. Establishing the Sink (Azure Data Lake Storage Gen2)</em></strong></p><ul><li>Access Sink Configuration: In the same copy activity, switch to “Sink” tab.</li><li>Create New Sink Dataset: Click “+ New” to define the sink dataset. Since you’re maintaining the CSV format, select Azure Data Lake Storage Gen2 with CSV as the format.</li><li>Name and Link Service: Assign a name to your sink dataset, like “Teamadls”, and create a new linked service if one doesn’t already exist for your Azure Data Lake Storage Gen2. This linked service should be configured with details like your Azure subscription and the name of the storage account, for instance, “testgroupdrag”.</li><li>Specify File Details: Input the necessary details such as file path (container name), directory (e.g., “raw data”), and the file name (e.g., “Teams.csv”). Make sure “First row as header” is selected and “Import schema” is set to “None”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WyCiPyl0zKzG3v-JF0OeJw.gif" /><figcaption><strong>Sink Configuration</strong></figcaption></figure><p>After configuring both source and sink, utilize the “Debug” function to test your pipeline. This ensures that the data flow from the source to the sink operates correctly. Successful execution will result in your data being ingested into the designated folder in Azure Data Lake Storage Gen2.</p><h4>Ingesting Data from REST API into Azure Storage:</h4><p>After successfully ingesting data from a CSV file in GitHub, the next step is to leverage REST APIs for this database, which is usually needed when integrating real-time data feeds into our analytical workflows. This section will walk you through creating a data import pipeline that fetches data from a REST API and saves it in Azure Data Lake Storage Gen2, in a new directory called “users” with the file “users.csv”.</p><p><strong><em>Steps to Ingest Data Using a REST API Call</em></strong></p><ol><li><strong><em>Create a Copy Activity</em></strong><br>As preciously explained create a new copy activity to your pipeline and give it specific name to reflect the purpose.</li><li><strong><em>Configure the REST Source</em></strong></li></ol><ul><li>In the source configuration of the new copy activity you have to choose REST dataset.</li><li>Make sure the dataset is pointing to the desired REST API endpoint and correctly configured within the linked service for REST API access.</li><li>I am using a dummy rest source “<a href="https://reqres.in/api/users">https://reqres.in/api/users</a>” .</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EI4dfi__hdbYkS4EKdYGbg.gif" /><figcaption><strong>Configuring Rest Source</strong></figcaption></figure><p>3. <strong><em>Specify the Sink Details</em></strong></p><ul><li>Direct the output to your Azure Data Lake Storage Gen2, into a new “users” directory, saving the file as “users.csv”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rgQBif-z0i__31laLiYQSQ.gif" /><figcaption><strong>Configuring Rest Sink</strong></figcaption></figure><p>4. <strong><em>Adjust Data Mapping</em></strong></p><ul><li>Use the “Mapping” tab to fine-tune the data structure. Import the schema from the REST API, focusing on the essential data elements by selecting the appropriate collection reference.</li><li>Ensure only relevant data fields are mapped, excluding unnecessary metadata to streamline the dataset.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hUqWLZAmx3yG2ogphx9zGw.gif" /><figcaption><strong>Configuring Sink Mapping to obtain just the Data Array and exclude meta data</strong></figcaption></figure><h4>Conclusion</h4><p>We’ve discussed how to work with APIs and CSV files from GitHub when it comes to data integration using ETL methods. We showed how to store REST API responses efficiently in Azure Data Lake Storage along with CSV data through a data flow activity in Azure Data Factory. I am trying to make complicated ideas less obscure for beginners in cloud computing. Our next step will be learning about Azure Databricks, where we will mainly look at performing advanced data analysis in line with the objectives of expanding our understanding within the Azure ecosystem.</p><p>Happy Learning 😊</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=231158af358e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convolutional Neural Network]]></title>
            <link>https://medium.com/analytics-vidhya/convolutional-neural-network-3818bb487ce8?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3818bb487ce8</guid>
            <category><![CDATA[convolutional-neural-net]]></category>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[cifar-10]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Sun, 03 Nov 2019 13:14:57 GMT</pubDate>
            <atom:updated>2020-02-26T12:47:50.193Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*su1iyXcRYEl6NJgrTad2cQ.png" /></figure><h4><strong>What is CNN?</strong></h4><p>Convolutional neural networks (CNN) is a special architecture of artificial neural networks, proposed by Yann LeCun in 1988.</p><p>In deep learning, a convolutional neural network (CNN or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery and processing that is specifically designed to process pixel data.</p><p>CNNs are powerful image processing, artificial intelligence (AI) that use deep learning to perform both generative and descriptive tasks, often using machine vision that includes image and video recognition, along with recommender systems and natural language processing (NLP).</p><p>Let us take an example of emojis,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*NhXJ0Q9EERioAo186dlvzg.png" /></figure><p>Here, as shown in below example the Convolutional network helps computer vision to recognize the input as a happy emoji.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*9Er3wG-9f_8U1_orIKKXdg.png" /></figure><p>CNNs are regularized versions of multi layer perceptrons. Multi layer perceptrons usually mean fully connected networks, that is, each neuron in one layer is connected to all neurons in the next layer.</p><p>One of the most popular uses of this architecture is image classification. For example, Facebook uses CNN for automatic tagging algorithms, Amazon uses CNN for generating product recommendations and Google uses CNN for search through among users’ photos.</p><h4><strong>How does it work?</strong></h4><p>We were taught at a young age on how to classify different objects or animals etc. Similarly, we need to teach an algorithm with numerous images before it is ready to recognize or classify or predict objects on its own.</p><p>For example: <strong>How did we learn to differentiate between a Wolf and a Dog?</strong></p><p>By learning to differentiate based on features or characteristics. But computer sees these characteristics or features in a different way.</p><p>Every image can be represented as 2-dimensional arrays of numbers, known as pixels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*e53QjSInCK_JGpik8dhT5A.png" /></figure><p>Suppose you are working with a black and white image dataset and each image size is 30 x 30. In this case, the size of the array will be 30x30x1. Where 30 is width, next 30 is height and 1 is the depth (black &amp; white image contains only 1 channel). And if the image is a colored image of size 600 x 600. In this case, the size of the array will be 600x600x3. Where 600 is width, next 600 is height and 3 is the depth which describes the intensity of RGB channel values (Red, Green and Blue). The computer will assign a value from 0 to 255 to each of these pixels and this pixel value describes the intensity of the pixel at each point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*ff248yDWKO2c37ejp7q5-A.png" /></figure><h4><strong>Image Representation</strong></h4><p>Images are encoded into color channels, the image data is represented into each color intensity in a color channel at a given point, the most common one being RGB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*pxCKxn6_SplbM9taE5JluA.png" /></figure><h4><strong>Black and White Image:</strong></h4><p>The contrast ranges from black at the weakest intensity (0) to white at the strongest (255) and in between is the grey scale (0–255).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*PYAOGVBFP8bFiRC9qHfMvw.png" /></figure><h4><strong>Colored Image:</strong></h4><p>A colored image has three channels RGB (Red, Green, Blue). So basically, pixel has three values assigned to it. Each one between 0 to 255. By combining these values, we can decide the color of the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*SwAMFd6zB-BuVAWY3e1kXw.png" /></figure><h4><strong>Why do we need CNN?</strong></h4><p>Consider the above example, total number of neurons in input layer will 30 x 30 = 900, this is manageable. What if the size of image is 1080 x 900 which means we need 972000 neurons in input layer. This is a huge number of neurons for the said operation and is challenging computationally. And this is where Convolutional Neural Network or CNN is proven effective. In simple word what CNN does is, it extracts the features of image and convert it into lower dimension without losing its characteristics.</p><p>In the following example we can see that initially the size of the image is 224 x 224 x 3. And if we proceed without convolution, then we need 224 x 224 x 3 = 150,528 numbers of neurons in input layer. But after applying convolution, our input tensor dimension is reduced to 1 x 1 x 1000. It means we only need 1000 neurons in first layer of feed forward neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*pZdJtGM1g2KnhgD8RlRlZw.png" /></figure><p>Since, we now have a basic understanding about CNN let’s look at the steps involved in CNN.</p><h4><strong>Layers in CNN</strong>:</h4><ol><li><strong>Convolution</strong></li><li><strong>Pooling</strong></li><li><strong>Flattening</strong></li><li><strong>Full Connection</strong></li></ol><h4><strong>1)Convolution:</strong></h4><p>Convolution is a mathematical operation on two functions (f and g) that produces a third function expressing how the shape of one is modified by the other.It is defined as the integral of the product of the two functions after one is reversed and shifted. As such, it is a kind of integral transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*L5IGggod3Rx3iD5kn8x5ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*Gt43gLlqCGcFjJ-LBZaXmQ.png" /></figure><p>Now, let us consider the wolf example again.</p><p>In human understanding characteristics of a wolf are for example the size or eye color. For the computer, these characteristics are boundaries or curvatures. And then through the groups of convolutional layers the computer constructs more abstract concepts.</p><p>In the case of a CNN, the convolution is performed on the input data with the use of a Feature Detector or Filter or Kernel (these terms are used interchangeably) to then produce a Feature Map or Convolved Feature or Activation Map.</p><p>Filters were designed by computer vision experts, which were then applied to an image to result in a feature map or output from applying the filter that makes the analysis of the image easier in some way.</p><p>Convolution operation is signified by an ‘X’ as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ERmdeios6DD8yp_v7u7B4A.gif" /></figure><p>We take the filter and place it on top of our image as shown above. And we basically multiply each value with each value of the filter and then we add the matched-up value. As we can see, we’ve reduced the size of the image.</p><p>We have used <strong>stride of 1</strong> and the image dimension reduced a bit. But if we have a <strong>stride of 2,</strong> then the image is going to reduce more. So, the feature map is going to be even smaller. Stride denotes how many steps we are moving in each step-in convolution. By default, it is one. This is the main purpose of feature detector. It will make the image smaller so that it’ll be easier to process it and it will be faster.</p><p>Feature detectors doesn’t have to be a 3x3 matrix. It can be 5x5 or 7x7. But 3x3 is predominantly used.</p><p>For example: Alex-Net uses 7x7 filters.</p><p>We will create multiple feature maps because we use different filters. So that way, we preserve the loss of information. The network decides through training which features are important for certain types or certain categories.</p><h4><strong>Few types of different filters</strong>:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*kvCHRiyIa2esBRqxpKp8Tw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*OeFzpx57mTHf9OOxs_ZZkw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*CFJs_Z-_1I8e4IZI9lHzqw.png" /></figure><p>The primary purpose of a convolution is to find features in our image using feature detectors, put them into a feature map and by having them in a feature map, it still preserves the spatial relationships between pixels which is very important for us because if they’re jumbled up then we’ve lost the pattern.</p><p><strong><em>The Rectified Linear Unit (RELU)</em></strong></p><p>The Rectified Linear Unit, or ReLU, is not a separate component of the convolutional neural networks’ process. It’s a supplementary step to the convolution operation.</p><p><strong>Why do we need a RELU activation function here</strong>?</p><p>The reason why we are applying the rectifier is because we want to increase non-linearity in our image, or in our convolution neural network. And rectifier acts as that filter or acts as that function that breaks linearity. The reason why we want to increase non-linearity in our network is because images themselves are highly non-linear, especially if we’re recognizing different objects next to each other, or just the background. And transition between adjacent pixels is often going to be non-linear. That’s because of the borders, there’s different colors, there’s different elements in your back ground. When we are applying mathematical operations such as convolution and running the feature detection to create our feature maps, we risk that we might create something linear.</p><p>Let me take an example of black and white image,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*dt_uL6AHdp5pRb5Ct0IiPw.png" /></figure><h4><strong>Feature detector</strong></h4><p>By putting the image through the convolution process, or in other words, by applying to it a feature detector, the result is what you see in the following image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*-m5N8InALdCAtNAhbItUCg.png" /></figure><p>As you see in the above image, the entire image is now composed of pixels that vary from white to black with many shades of grey in between.</p><h4><strong><em>Rectification</em></strong></h4><p>What the rectifier function does to an image like this is remove all the black elements from it, keeping only those carrying a positive value (the grey and white colors).</p><p>The essential difference between the non-rectified version of the image and the rectified one is the progression of colors. If you look closely at the above image, you will find parts where a white streak is followed by a grey one and then a black one. And this is linear progression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*bHRYxEyWF5-h8GmpDY9k6A.png" /></figure><p>After we rectify the image, you will find the colors changing more abruptly. The gradual change is no longer there. That indicates that the linearity has been disposed of. So, ReLu breaks this linear progression and removes anything that’s negative. In this case its Black.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*KlmIh62e9jsmov0QZOdKtA.png" /></figure><h4>2) <strong>Pooling</strong>:</h4><p>Let’s take the below image as an example to understand pooling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*OWsm3XHjMzi5Gb6utBJNng.png" /></figure><p>As we can see we have different wolves looking in different directions, they are all in different part of the image, their faces are all positioned in different parts of the image, some on the right hand side, some on the left corner, some in the middle with different texture and lighting. So, if the neural network has a property called ‘<strong>Spatial invariance</strong>’, meaning that it doesn’t have to care where the features are located, not so much as in which part of the image. Neural Networks must have some level of flexibility to be able to still find the features and this is what is called as <strong>Pooling</strong>.</p><p>There are different kinds, Max, Mean, Sum pooling.</p><p>Below is an example of Max pooling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*hNI7mbMnhhAZ6HWIHt-xYw.gif" /></figure><p>Out of all the four values in the 2x2 matrix, the maximum value is considered. The maximum or the large numbers in our feature map, they represent where we found the closest similarity to our feature. By pooling these features, we are getting rid of 75% of the information that is not the feature.</p><p>Pooling is also called ‘<strong>Down Sampling</strong>’.</p><h4>3) <strong>Flattening</strong>:</h4><p>Basically, puts everything into one long column which would become the input layer for Artificial Neural Network as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*6fy-eTpu8qk7FyP-koR74g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*f4XMyz69eBm-AafNjvgQIA.png" /></figure><h4>4) <strong>Full Connection</strong>:</h4><p>After completion of series of convolutional, nonlinear and pooling layers, it is necessary to attach a fully connected layer.</p><p>This layer takes the output information from convolutional networks.</p><p>Attaching a fully connected layer to the end of the network results in an N dimensional vector, where N is the number of classes from which the model selects the desired class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*IBWW4p-pba1iAlt8WaAS-Q.png" /></figure><p>And so here we’ve got the input layer, we’ve got a fully connected layer and output layer.</p><p>The fully connected layer, in the artificial neural networks, we used to call them hidden layers and here we’re calling them fully connected layers because they are hidden layers, but at the same time they’re a more specific type of hidden layers, they are fully connected layers. In artificial neural networks, hidden layers don’t have to be fully connected whereas in convolutional neural networks, we’re going to be using fully connected layers and that’s why they’re generally called fully connected layers. Not only the weights are trained in the ANN but also the features detectors are trained and adjusted with the same gradient descent process and that allows us to come up with the best feature maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*6dH-7OHKg4MpCtLVS8blUw.png" /></figure><h4><strong>Softmax and Cross Entropy</strong></h4><p>In Convolutional Neural Network, Cost function (error) is called as Loss Function. And its calculated with the help of cross entropy function.<br><strong>Cross-entropy loss</strong>, or <strong>log loss</strong>, measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*qRoC8u3CHAcdTpDzVTP0Iw.png" /></figure><p><strong>Softmax or normalized exponential function </strong>brings the values to be between 0 and 1 and makes sure that they add up to 1. Because these two final output neurons are not connected between each other. They wouldn’t know what the value is of the other. This is where softmax comes in. Because the final probability percentage should add up to 100%. It wouldn’t make sense if we have 70% and 70% each for dog and cat. Other thing is that, the softmax function comes hand in hand with the cross-entropy function (Loss function).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*yUS2ievCj6jKKCQXlvVtRg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*WSeXiknk-fvC5u7DsGam1g.png" /></figure><p>We will be using this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*inkwMi_-HIseJEdT6KuTaA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*um6bFT3_Xn41cq2OGC8p9w.png" /></figure><p>This helps us to calculate the loss between the actual and the predicted value.</p><blockquote><strong><em>Few key things to consider</em></strong></blockquote><h4><strong>Normalization:</strong></h4><p>In image processing, normalization is a process that changes the range of pixel intensity values. Applications include photographs with poor contrast due to glare, for example.</p><h4>What is normalized RGB?</h4><p>At times, you want to get rid of distortions caused by lights and shadows in an image. Normalizing the RGB values of an image can at times be a simple and effective way of achieving this.</p><p>When normalizing the RGB values of an image, you divide each pixel’s value by the sum of the pixel’s value over all channels. So, if you have a pixel with intensified R, G, and B in the respective channels… its normalized values will be R/S, G/S and B/S (where, S=R+G+B).</p><p>Pixel values are often unsigned integers in the range between 0 and 255. Although these pixel values can be presented directly to neural network models in their raw format, this can result in challenges during modelling, such as in the slower than expected training of the model.</p><p>Instead, there can be great benefit in preparing the image pixel values prior to modelling, such as simply scaling pixel values to the range 0–1 to centering and even standardizing the values.</p><h4><strong>Image Data Augmentation</strong>:</h4><p>Image data augmentation is a technique that can be used to artificially expand the size of a training dataset by creating modified versions of images in the dataset.<br>Training deep learning neural network models on more data can result in more skillful models, and the augmentation techniques can create variations of the images that can improve the ability of the fit models to generalize what they have learned to new images.</p><h4>What will it do?</h4><p>It will create many batches of our images and in each batch it will apply some random transformations on a random selection of our images like rotating them , flipping them, shifting them, or even shearing them and eventually we get many more diverse images during training.<br>since the transformations are random, our model will never find the same transformation twice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*KILFmhgdbEC2pNPL5KaREA.png" /></figure><p>The Keras deep learning neural network library provides the capability to fit models using image data augmentation via the ImageDataGenerator class.</p><h4><strong>Download and prepare the CIFAR10 dataset</strong></h4><p>The CIFAR10 dataset contains 60,000 32x32 color images in 10 classes, with 6,000 images in each class. The dataset is divided into 50,000 training images and 10,000 testing images. The classes are mutually exclusive and there is no overlap between them.</p><h4><strong><em>Python Implementation : Using TensorFlow 2.0</em></strong></h4><pre>&quot;&quot;&quot;Importing required libraries&quot;&quot;&quot;</pre><pre>import numpy as np<br>import time<br>import matplotlib.pyplot as plt<br>% matplotlib inline</pre><pre>%tensorflow_version 2.x</pre><pre>import tensorflow as tf<br>tf.test.gpu_device_name()</pre><pre>print(tf.__version__)</pre><pre>from tensorflow.keras import datasets, layers, models</pre><p><strong><em>Importing the data</em></strong></p><pre>(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()</pre><pre># Normalize pixel values to be between 0 and 1</pre><pre>train_images, test_images = train_images / 255.0, test_images / 255.0</pre><pre>train_images.shape</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*cXOBTP2-kcp2O3tKy6ZJyg.png" /></figure><p>plot the first 25 images from the training set and display the class name below each image and verify the data.</p><pre>class_names = [&#39;airplane&#39;, &#39;automobile&#39;, &#39;bird&#39;, &#39;cat&#39;, &#39;deer&#39;,<br>               &#39;dog&#39;, &#39;frog&#39;, &#39;horse&#39;, &#39;ship&#39;, &#39;truck&#39;]</pre><pre>plt.figure(figsize=(10,10))<br>for i in range(25):<br>    plt.subplot(5,5,i+1)<br>    plt.xticks([])<br>    plt.yticks([])<br>    plt.grid(False)<br>    plt.imshow(train_images[i], cmap=plt.cm.binary)<br>    # The CIFAR labels happen to be arrays, <br>    # which is why you need the extra index<br>    plt.xlabel(class_names[train_labels[i][0]])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*AUtuHJYYWxzC7tsQWsjecA.png" /></figure><pre>model = models.Sequential()<br>model.add(layers.Conv2D(32, (3, 3), activation=&#39;relu&#39;, input_shape=(32, 32, 3)))<br>model.add(layers.MaxPooling2D((2, 2)))<br>model.add(layers.Conv2D(64, (3, 3), activation=&#39;relu&#39;))<br>model.add(layers.MaxPooling2D((2, 2)))<br>model.add(layers.Conv2D(96, (3, 3), activation=&#39;relu&#39;))<br>model.add(layers.MaxPooling2D((2, 2)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*robRKdWoSniRkRkuEW4IRw.png" /></figure><pre>model.add(layers.Flatten())<br>model.add(layers.Dense(512, activation=&#39;relu&#39;))<br>model.add(layers.Dropout(0.5))<br>model.add(layers.Dense(256, activation=&#39;relu&#39;))<br>model.add(layers.Dropout(0.5))<br>model.add(layers.Dense(128, activation=&#39;relu&#39;))<br>model.add(layers.Dropout(0.5))<br>model.add(layers.Dense(64, activation=&#39;relu&#39;))<br>model.add(layers.Dropout(0.5))<br>model.add(layers.Dense(10, activation=&#39;softmax&#39;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*DXL4jLYENFJaoXNsAmbeKQ.png" /></figure><pre>model.compile(optimizer=&#39;adam&#39;,<br>              loss=&#39;sparse_categorical_crossentropy&#39;,<br>              metrics=[&#39;accuracy&#39;])</pre><blockquote><strong><em>Image data generator used for image augmentation</em></strong></blockquote><p>Here, I am using Cifar10 dataset , which is an inbuilt dataset from tensorflow library and hence I am using <strong>.flow(x, y)</strong> and in case if you are using a image from a directory kindly use<strong>.flow_from_directory(directory) </strong>option<strong>.</strong></p><pre>#Image data Generator<br>from tensorflow.keras.preprocessing.image import ImageDataGenerator</pre><pre>datagen = ImageDataGenerator(zoom_range=0.2, horizontal_flip=True)</pre><pre># train the model<br>start = time.time()<br># fits the model on batches with real-time data augmentation:<br>history=model.fit_generator(datagen.flow(train_images, train_labels, batch_size=128),<br>                    steps_per_epoch=1000, epochs=100,validation_data=(test_images, test_labels))</pre><pre>end = time.time()<br>print(&quot;Model took %0.2f seconds to train&quot;%(end - start))</pre><pre># always save your weights after training or during training<br>model.save_weights(&#39;100_epochs.h5&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VHIc20AeUwSJK8npOJsKlg.png" /></figure><p>For detailed explanation on arguments, kindly refer “<a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator">https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator</a>”</p><p><strong>Visualizing training accuracy with validation accuracy with each epoch</strong></p><pre>plt.plot(history.history[&#39;accuracy&#39;], label=&#39;accuracy&#39;)<br>plt.plot(history.history[&#39;val_accuracy&#39;], label = &#39;val_accuracy&#39;)<br>plt.xlabel(&#39;Epoch&#39;)<br>plt.ylabel(&#39;Accuracy&#39;)<br>plt.ylim([0.5, 1])<br>plt.legend(loc=&#39;lower right&#39;)</pre><pre>test_loss, test_acc = model.evaluate(test_images,  test_labels, verbose=2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*VNZ4UREBcLBL0WgJ4hb0dA.png" /></figure><p>So, we obtained a training accuracy of 81.6% and validation accuracy of 79% which is good. To improve the accuracy, add more convolutional layers or more fully connected layer or both. And also tweak with the arguments in Image Data Generator.</p><h4><strong>Testing on a new Image</strong></h4><pre>image_test= tf.keras.preprocessing.image.load_img(&#39;drive/My Drive/DATA files/CNN model weights/Deer.jpg&#39;, grayscale=False,<br>    color_mode=&#39;rgb&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*eqS7ceF_HSPi1j_Cea7_0A.png" /><figcaption><strong>New Image for Testing</strong></figcaption></figure><p>This image is of different resolution. But the input shape with which the convolutional layer was trained on was 32x32. And hence the target size of this test image should be changed accordingly.</p><pre>img_test3 = tf.keras.preprocessing.image.load_img(&#39;drive/My Drive/DATA files/CNN model weights/Deer.jpg&#39;,<br>    grayscale=False,<br>    color_mode=&#39;rgb&#39;,<br>    target_size=(32,32))<br>img1 = np.array(img_test3)<br>img1.shape<br>img1 = img1 / 255.0<br>img1 = img1.reshape(1, 32, 32, 3)</pre><p>Reshape is done in order to show that we just have only 1 test image of size 32x32 with 3 channels (RGB).</p><pre>plt.figure(figsize=(10,10))<br>plt.imshow(image_test)<br>plt.xlabel(labels[result1])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*vGbDfBQQ7hvX4N3iDMbOhg.png" /><figcaption><strong>Predicted Class</strong></figcaption></figure><p>So far the model is doing a pretty good job in classifying the images.</p><p>The application of CNN is not restricted to classification. Few of the applications include :</p><ol><li><strong><em>Decoding Facial Recognition</em></strong>: <br> • Identifying every face in the picture <br> • Focusing on each face despite external factors, such as light, angle, pose, etc. <br> • Identifying unique features <br> • Comparing all the collected data with already existing data in the database to match a face with a name.</li><li><strong><em>Analyzing Documents</em></strong>: Convolutional neural networks can also be used for document analysis. This is not just useful for handwriting analysis, but also has a major stake in recognizer.</li><li><strong><em>Advertising</em></strong>: CNN’s have already brought in a world of difference to advertising with the introduction of programmatic buying and data-driven personalized advertising etc.</li></ol><h4>Resources :</h4><ul><li><a href="https://www.tensorflow.org/tutorials/images/cnn">Convolutional Neural Network (CNN) | TensorFlow Core</a></li><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator">tf.keras.preprocessing.image.ImageDataGenerator | TensorFlow v2.14.0</a></li><li><a href="https://www.tensorflow.org/tutorials/keras/save_and_load">Save and load models | TensorFlow Core</a></li></ul><p>For the implementation codes and trained model weights, kindly refer “<a href="https://github.com/rohithramesh1991/CNN_Image-Classification">https://github.com/rohithramesh1991/CNN_Image-Classification</a>”</p><p>And also refer “<a href="https://github.com/rohithramesh1991/Face-Emotion-Detection">https://github.com/rohithramesh1991/Face-Emotion-Detection</a>”. Another example of how powerful CNN architecture can be.</p><p>Thank you for reading…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3818bb487ce8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/convolutional-neural-network-3818bb487ce8">Convolutional Neural Network</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Regression]]></title>
            <link>https://medium.com/analytics-vidhya/linear-regression-225904ee7bcc?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/225904ee7bcc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[ridge-regression]]></category>
            <category><![CDATA[polynomial-regression]]></category>
            <category><![CDATA[lasso-regression]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Fri, 01 Nov 2019 18:12:59 GMT</pubDate>
            <atom:updated>2019-12-14T04:10:17.816Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/792/0*S8GxFVcy_vlJM2sZ.png" /></figure><h3><strong>What is Regression ?</strong></h3><p>Regression analysis is a powerful statistical method that allows you to examine the relationship between two or more variables of interest. Regression analysis is a reliable method of identifying which variables have impact on a topic of interest. The process of performing a regression allows you to confidently determine which factors matter most, which factors can be ignored, and how these factors influence each other.</p><h4><strong>Linear Regression</strong></h4><p>It is the study of <em>linear</em>, <em>additive </em>relationships between variables. The linear regression model has a dependent variable that is a continuous variable, while the independent variables can take any form (continuous, discrete, or indicator variables).</p><p>In order to understand regression analysis fully, it’s essential to comprehend the following terms:</p><ul><li><strong>Dependent Variable:</strong> This is the main factor that you’re trying to understand or predict.</li><li><strong>Independent Variables:</strong> These are the factors that you hypothesize have an impact on your dependent variable.</li></ul><h4><strong>Simple linear regression</strong> :</h4><p>A simple linear regression model has only one independent variable, while a multiple linear regression model has two or more independent variables. It is useful for finding linear relationship (straight line relationship) between two continuous variables. (Cause and Effect Relationship)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*SLcKnbqA_YmC-Ty0YDml-g.png" /></figure><p>The value b0 and b1 must be chosen so that they minimize the error.</p><p>In Linear Regression,<br> Predictor -&gt; X -&gt; Features.<br>Predicted -&gt; Y -&gt; Target Variable.<br>Coefficients -&gt; b0, b1.</p><h4><strong>Sum of squared error</strong> :</h4><p>It is considered as metric then the goal is to obtain the best line that reduces the error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*yhe2EDsk6u03fGGCMCLlhg.png" /></figure><p>If we don’t square the error, then positive and negative points will cancel out each other.</p><p>• The rate at which unit changes in X influences Y(predicted)</p><p>The coefficients are calculated using OLS (ordinary least square regression) technique. The OLS technique captures the best possible straight line that gives better relationship between the predicted(Y) and the predictor(X) variables. It is the line that is close to as many points as possible. Calculate the distance of each points from the line. The distance, we get from y-axis, some are positive and some are negative. Square and then sum them up. The line with the minimum distance is the best possible line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*rxqwxUiPVhC9prEg3BTzPQ.png" /></figure><p>· So the line with the least mean squared error is selected. It is also known as Residual sum of squares.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*Ka62JJVVNt5phmqTeemtJQ.png" /></figure><p>Yi = Training set target value. (W.Xi+b)= predicted target value using model.</p><h4><strong>python implementation</strong></h4><p>let us take an example of Boston house prices dataset,</p><pre>import pandas as pd<br>import numpy as np</pre><pre>from sklearn.datasets import load_boston<br>boston = load_boston()<br>print(boston.data.shape)<br>boston.keys()<br></pre><p><strong>Data</strong></p><pre>pd.DataFrame(boston.data).head()</pre><p><strong>Target</strong></p><pre>pd.DataFrame(boston.target).head()</pre><p><strong>Feature names and Description</strong></p><pre>print(boston.feature_names)<br>print(boston.DESCR)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EoDK2_LWsEUt_970ROa9YQ.png" /></figure><p><strong>Creating a dataframe</strong></p><pre>boston_data = pd.DataFrame(boston.data)<br>boston_data.columns = boston.feature_names<br>boston_data[&#39;MEDV&#39;]=boston.target<br>print(boston_data.head())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*BueIts1bADm_3fRUOLmM7A.png" /></figure><p><strong>Visualization</strong></p><pre>import matplotlib.pyplot as plt<br>plt.hist(boston.target)<br>plt.title(&#39;Boston Housing Prices and Count Histogram&#39;)<br>plt.xlabel(&#39;price ($1000s)&#39;)<br>plt.ylabel(&#39;count&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*qOTi5VqxRdwAfkB9wpt1cg.png" /></figure><p>The below will create a scatterplot on each of the feature_names versus the price of Boston housing.</p><pre>import seaborn as sns</pre><pre>sns.pairplot(boston_data,x_vars=[&#39;CRIM&#39;,&#39;ZN&#39;,&#39;INDUS&#39;,&#39;CHAS&#39;,&#39;NOX&#39;,&#39;RM&#39;,&#39;AGE&#39;,&#39;DIS&#39;,&#39;RAD&#39;,&#39;TAX&#39;,&#39;PTRATIO&#39;,&#39;B&#39;,&#39;LSTAT&#39;], y_vars=[&quot;MEDV&quot;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vpBTscjc3s1wGx4xzQ8m7A.png" /></figure><pre>#separating the dependent and independent variables<br>X = boston_data.drop(&#39;MEDV&#39;, axis = 1)<br>Y = boston_data[&#39;MEDV&#39;]</pre><blockquote><strong>Least square linear regression in scikit-learn :</strong></blockquote><pre>x_train, x_test, y_train, y_test = train_test_split(X, Y, random_state = 0)</pre><pre>Linreg = LinearRegression().fit(x_train, y_train)</pre><pre>print(&quot;Linear model intercept(b):{}&quot;.format(Linreg.intercept_))<br>print(&quot;Linear model coeff (w):{}&quot;.format(Linreg.coef_))<br>print(&quot;R-squared score(training): {:.3f}&quot;.format(Linreg.score(x_train,y_train)))<br>print(&quot;R-squared score(test): {:.3f}&quot;.format(Linreg.score(x_test,y_test)))</pre><p>Y=W0.X0+b<br>W0 = Linreg.coef_ , b=linreg.intercept_</p><p><strong><em>linreg.coef_ </em></strong>and<strong><em> linreg.intercept_</em></strong> :-underscore denotes a quantity derived from training data, as opposed to a user setting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*dH2arW0bMIY07aaba5BSTQ.png" /></figure><pre>y_pred = Linreg.predict(x_test)</pre><pre>plt.scatter(y_test, y_pred)<br>plt.xlabel(&quot;Prices: $Y_i$&quot;)<br>plt.ylabel(&quot;Predicted prices: $\hat{Y}_i$&quot;)<br>plt.title(&quot;Prices vs Predicted prices: $Y_i$ vs $\hat{Y}_i$&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*XLjMnbHo_O9mTv2NGnC7kA.png" /></figure><pre>from sklearn.metrics import mean_squared_error<br>mse = mean_squared_error(y_test, y_pred)<br>print(mse)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*xw6pq3x8ZSAC07PcTBveQg.png" /></figure><p>This means that the model isn’t a really great linear model</p><p>let us now try to improve this ‘R score’ with the help of regularization.</p><h4><strong>Variations in linear regression :</strong></h4><h4><strong><em>1. Ridge Linear Regression</em></strong>:</h4><p>It uses the same “least square criterion” but adds a penalty for large variation in “W” parameter (b1) penalty, in order to avoid over fitting and complex model.<br>Penalty parameter is called regularization<br>Ridge regression uses -&gt; L2 regularization</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*wSHRcREcQW5N_bnyGVyk4w.png" /></figure><p>• The model with larger feature weights (W) will add more to the objective functions overall value.<br>• Because our goal is to minimize the overall objective function, the regularization term act as a penalty of the model with lots of large feature weight values.<br>• Higher Alpha means more regularization.</p><h4><strong>The need for feature normalization :</strong></h4><p>when the input variables, features have very different scales, then when regularizing the coefficients, input variable with the different scale will have different contributions to this L2 penalty.<br>And thus, all the input features has to be on the same scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*PCRSgC0IchRxcatMEL_blg.png" /></figure><p><strong>Using a scalar object : fit and transform method</strong></p><pre>from sklearn.preprocessing import MinMaxScaler<br>Scaler = MinMaxScaler()<br>X_train_Scaled = Scaler.fit_transform(x_train)<br>X_test_Scaled = Scaler.transform(x_test)<br>clf = Ridge().fit(X_train_Scaled , y_train)<br>R2_score = clf.score(X_test_Scaled, y_test)</pre><p><strong>Ridge regression with regularization parameter: alpha</strong></p><pre>from sklearn.linear_model import Ridge</pre><pre>print(&#39;Ridge regression : effect of alpha regularization parameter \n&#39;)<br>for this_alpha in [0,1,10,20,50,100,1000]:<br>    linridge = Ridge(alpha = this_alpha).fit(X_train_Scaled,y_train)<br>    r2_train = linridge.score(X_train_Scaled, y_train)<br>    r2_test = linridge.score(X_test_Scaled, y_test)<br>    num_coeff_bigger = np.sum(abs(linridge.coef_)&gt;1.0)<br>    print(&#39;Alpha = {:.2f}\n numabs(coeff) &gt;1.0:{}, r-&gt;squared training: {:.2f}, r_squared test: {:.2f}\n&#39;.format(this_alpha,num_coeff_bigger,r2_train,r2_test))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*n7yCAtyoMCnbqRJbKK1N_w.png" /></figure><h4>What is Ridge regression doing?</h4><p>Its regularizing the linear regression by imposing sum of squares penalty on the size of ‘w’ coefficients. So the effect of alpha is to shrink the ‘w’ coeffients to zero and towards each other.</p><h4><strong>2. <em>Lasso Regression</em>:</strong></h4><p>It is another form of regularized linear regression that uses an L1 Regularization.<br>L1-penalty : Minimize the sum of the absolute value of the coefficients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*4Ixeexo8c-f_l6xeHeyf4Q.png" /></figure><p>This has the effect of setting parameter weights in ‘w’ to zero. For the least influential variables. This is called sparse solution. This is a kind of feature selection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*JqYBraJ7p_zX-hScWCyOWA.png" /></figure><pre>from sklearn.liner_model import Lasso<br>from sklearn.preprocessing import MinMaxScaler</pre><pre>Scaler = MinMaxScaler()<br>X_train_Scaled = Scaler.fit_transform(x_train)<br>X_test_Scaled = Scaler.transform(x_test)<br>clf = Ridge().fit(X_train_Scaled , y_train)<br>R2_score = clf.score(X_test_Scaled, y_test)</pre><pre>linlasso = Lasso(alpha=0.0001, max_iter=10e5).fit(X_train_Scaled,y_train)<br>print(&#39;lasso regression linear model intercept:{}&#39;.format(linlasso.intercept_))<br>print(&#39;lasso regression linear model coeff:\n{}&#39;.format(linlasso.coef_))<br>print(&#39;Non-zero feature :{}&#39;.format(np.sum(linlasso.coef_!=0)))<br>print(&#39;R-Squared score(training):{:.3f}&#39;.format(linlasso.score(X_train_Scaled, y_train)))<br>print(&#39;R-Squared score(test):{:.3f}\n&#39;.format(linlasso.score(X_test_Scaled,y_test)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*JLQPGItmrgvK-mvfhVRKcA.png" /></figure><p><strong>When to use Ridge vs Lasso Regression?</strong><br>• Many small/medium sized effects: Use Ridge<br>• Only few variables with medium/large effects: Use Lasso</p><h4><strong>3. <em>Polynomial Feature with Linear Regression</em> :</strong></h4><p>•Generate new features consisting of all polynomial combinations of the original two features(X0, X1).<br>• The degree of the polynomial specifies how many variables participate at a time in each new feature.(our eg: degree 2)<br>• This is still a weighted linear combination of features, so it is still a linear model and can use same least-squares elimination method for ‘w’ and ‘b’(coefficients).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*Wtuswyzl4A9GP_fCZFO_Bg.png" /></figure><p><strong>Why should we transform our data?</strong><br>• To capture interactions between the original features by adding them as features to the linear model.<br>• To make a classification problem easier.</p><p><strong><em>For Example:</em></strong> It may be that housing prices vary as a quadratic function of both the land price that a house sits on and the amount of taxes paid on the property as a theoretical example.<br>A simple linear model could not capture this non-linear relationship, but by adding non-linear features like polynomial, to the linear regression model, we can capture this non-linearity.</p><p>• Beware of the polynomial feature expansion with higher degree as this can lead to complex models that over fit.<br>• Thus, polynomial feature expansion is often combined with a regularized learning method like ridge regression.</p><p><strong>Now run all three simultaneously and check:</strong></p><pre>from sklear.linear_model import LinearRgression<br>from sklearn.linear_model import Ridge<br>from sklearn.preprocessing import PolynomialFeatures</pre><pre>Poly = PolynomialFeatures(degree =2)<br>X_F1_poly = Poly.fit_transform(X)<br>x_poly_train, x_poly_test, y_train, y_test = train_test_split(X_F1_poly, Y, random_state = 0)</pre><pre>linreg = LinearRegression().fit(x_poly_train, y_train)<br>print(&#39;(poly deg 2) Linear model coeff(W):\n{}&#39;.format(linreg.coef_))<br>print(&#39;(poly deg 2) Linear model intercept(b):{:.3f}&#39;.format(linreg.intercept_))<br>print(&#39;(poly deg 2) R-Squared score (training):{:.3f}&#39;.format(linreg.score(x_poly_train, y_train)))<br>print(&#39;(poly deg 2) R-Squared score (test):{:.3f}&#39;.format(linreg.score(x_poly_test, y_test)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*Jaeawim27MtxTgHDaVb5nA.png" /></figure><p>Addition of many polynomial features often leads to overfitting, so we often use polynomial feature in combination with regression that has a regularization penalty, like ridge regression.</p><p><strong>Ridge regression avoids over fitting:</strong></p><pre>Poly = PolynomialFeatures(degree =2)<br>X_F1_poly1 = Poly.fit_transform(X)<br>x_poly_train1, x_poly_test1, y_train, y_test = train_test_split(X_F1_poly1, Y, test_size=0.4, random_state = 0)</pre><pre>linrid = Ridge(alpha = 0.001).fit(x_poly_train1, y_train)<br>print(&#39;(poly deg 2) Linear model coeff(W):\n{}&#39;.format(linrid.coef_))<br>print(&#39;(poly deg 2) Linear model intercept(b):{:.3f}&#39;.format(linrid.intercept_))<br>print(&#39;(poly deg 2) R-Squared score (training):{:.3f}&#39;.format(linrid.score(x_poly_train1, y_train)))<br>print(&#39;(poly deg 2) R-Squared score (test):{:.3f}&#39;.format(linrid.score(x_poly_test1, y_test))))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-W0Rzt32fYle995TnzJEiQ.png" /></figure><p>As we can see from above outputs , polynomial features with ridge regression has given the best possible output without over fitting.</p><p><strong><em>1.The First Regression</em></strong>: Just uses least square regression without polynomial feature transformation.<br><strong>2.</strong> <strong><em>The Second Regression</em></strong>: creates the polynomial features object with degrees set to 2 and then calls the ‘fit_transform’ method of the polynomial features. The code then calls ordinary least square linear regression.<br>We can see indications of over fitting on this expanded feature representation, as the models R-squared score on the training set is close to 1 but much lower on the test set.<br><strong>3. <em>The Third regression</em></strong>: Training and Test set are basically same. With the test score of the regularized polynomial regression performing the best of all three regression.</p><p>There are other evaluation metrics as well. But in this article I have mainly concentrated on R Squared score and MSE(Mean Squared Error) to explain linear regression and the variations of linear regression and their effect on accuracy and to avoid over fitting.</p><p>For the implementation codes, kindly refer “<a href="https://github.com/rohithramesh1991/Linear-Regression">https://github.com/rohithramesh1991/Linear-Regression</a>”</p><p>Thank you for reading…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=225904ee7bcc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/linear-regression-225904ee7bcc">Linear Regression</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Support Vector Machine]]></title>
            <link>https://medium.com/analytics-vidhya/support-vector-machine-82d3dff14d39?source=rss-edd6101f90e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/82d3dff14d39</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rbf]]></category>
            <category><![CDATA[support-vector-machine]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kernel]]></category>
            <dc:creator><![CDATA[ROHITH RAMESH]]></dc:creator>
            <pubDate>Fri, 11 Oct 2019 13:24:29 GMT</pubDate>
            <atom:updated>2019-10-11T13:24:29.113Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*8zH9San9fChWrVZny-feCA.png" /></figure><h4><strong>What is Support Vector Machine?</strong></h4><p>“Support Vector Machine” (SVM) is a supervised machine learning algorithm which can be used for both classification or regression challenges. However, it is mostly used in classification problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*4E9Do5Y2rL5RzRVAT5wR4Q.png" /></figure><p>It can provide more complex models that can go beyond linear decision boundaries.</p><p>With real world data, many classification problems aren’t easy. With different classes located in future space in a way that a linear hyperplane can’t act as an effective classifier.</p><p>In this algorithm, we plot each data item as a point in n-dimensional space (where n is number of features you have) with the value of each feature being the value of a coordinate. Then, we perform classification by finding the hyper-plane that differentiate the two classes very well .</p><h4><strong>Identify the right hyper-plane</strong></h4><p>But we can have multiple lines as shown below that can separate the data points. How do we choose the best line that separates the dataset into two classes?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*ZRYTDhc3OoPsVc6weqWk5g.png" /></figure><p>Select the hyper-plane which segregates the two classes better”. In this scenario, hyper-plane “B” has excellently performed this job.</p><p>SVM finds the hyperplane to maximizes the margin between support vectors of the two classes. Hyperplane are decision boundaries classifying the dataset while maximizing the margin.</p><h4><strong>What are support vectors?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*wo3hABPFe5WGm9Axas8-vw.png" /></figure><p>Support vectors are the data points in the dataset that are nearest to the hyperplane. Removing support vectors will alter the hyperplane separating two classes. Support vectors are critical elements of the dataset as SVM is built on them.</p><p><strong>Support vector machine has two main objectives</strong></p><p>· Find a hyperplane(line) that linearly separates the data points into two classes</p><p>· Maximize the margin between support vectors of the two classes</p><p>Let us try to understand this with few Examples:</p><p><strong><em>Case 1</em></strong>: <strong>A simple 1-dimensional classification problem for the classifier where the classes are linearly separable</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*Mr6sEHq9j62ATysJHMl0QQ.png" /></figure><p>Here, the line is engineered such that the maximum margin boundary happens to be at x=0.</p><p>This line helps separate two classes of data. Support vector machines helps to find a hyperplane (line) to linearly separate data points into two classes.</p><p><strong>A harder 1-dimensional classification problem for the classifier where the classes are no longer linearly separable</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*iyc5f4B-gJPcu0KX5f9Gdw.png" /></figure><p>A simple linear decision boundary just doesn’t have enough expressive power to classify all these points correctly.</p><h4>So, what can be done here?</h4><p>One very powerful idea is to transform the input data from a 1-dimensional space to a 2-dimensional space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*8Vd0xY15DXJZ-HWRIp44lQ.png" /></figure><p>Where the second dimension is nothing but square of first dimension. The data transformation makes it possible to solve this with a linear classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*9ys5Xo3-Id3VSuoCIwdAhg.png" /></figure><p>If we take the inverse of the transformation we just applied and bring the data points back to our original input space, we can see that the linear decision boundary in the 2-dimensional space corresponds to the two points where a parabola crosses the x-axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*vF27xipnPwYeYMKMPvSW9Q.png" /></figure><p>Now, let’s move on from a 1-dimensional problem to a 2-dimensional problem.</p><p><strong><em>Case 2</em></strong>: <strong>Here, we have two classes each of which has two features, x0 and x1</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*MgV4FfRQULlTfDe8ebXnSA.png" /><figcaption><strong>Original input space</strong></figcaption></figure><p>Again, this looks impossible for a linear classifier, which in 2-dimensional space is a straight line, to separate the points with any degree of accuracy.</p><p>Now, mapping a 2-dimensional problem on a 3-dimensional feature space to make it linearly separable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*Sg4JAxxppkG4hhkbgESOMQ.png" /><figcaption><strong>Feature Space</strong></figcaption></figure><p>This transformation acts to shape the points into a paraboloid around (0,0).</p><p>Now, the white points since they are close to (0,0) get mapped to points with higher vertical Z values, that new third feature that are close to 1. While the black points which are further from (0,0) get mapped to points with Z values that are either close to 0 or even negative.</p><blockquote><strong>Inverse of the transformation</strong>:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*vb80_9SOWpJA5spsnrV8ag.png" /></figure><p>There are lots of different possible transformations we could apply to the data. And there are different kernels available for different transformations.</p><p>Here, we are going to focus mainly on what’s called the radial basis function kernel (RBF) and Polynomial Kernel.</p><h4><strong>What is a Kernel Function</strong>?</h4><p>The Kernel Function in an SVM tells us, given 2 points in the original input space, what is their similarity in the new feature space.</p><p><strong>Radial Basis Function Kernel/Gaussian Kernel</strong>: The similarity between two points and the transformed feature space is an exponential decaying function of the distance between the vectors and the original input space as shown by the formula here,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*usGfbUMDdspzrA70jcS7YA.png" /></figure><p>So, just as we saw with the simple 1D and 2D examples earlier, the kernelized support vector machine tries to find the decision boundary with maximum margin between classes using a linear classifier in the transformed feature space and not in the original input space.</p><p>The linear decision boundary learned in transformed feature space by linear SVM corresponds to non-linear decision boundary in the original input space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*RaBOVucGXllDWybvvK5C9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/1*_L9DmWWdHDUCELB2dNi7Rg.png" /></figure><p>Here ‘K’ stands for the RBF kernel. The x and x’ are vectors where x’ is a “point of reference” vector. The exp() function means “math constant ‘e’ raised to a power”. The || term is Euclidean distance. The sigma is a free parameter.</p><p>The gamma parameter is the inverse of the standard deviation of the RBF kernel (Gaussian function), which is used as similarity measure between two points.</p><p><strong><em>Gamma(γ)</em></strong> defines how far the influence of a training example reaches.</p><p>· <strong>Small gamma</strong> means a large similarity radius. So that points farther apart are considered similar. Which results in more points being grouped together and smoother decision boundaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*hduXzTmcN-m6WiyIq8qJbg.png" /></figure><p>· <strong>Larger values of gamma</strong> mean the points must be very close to be considered similar. This results in more complex, tightly constrained decision boundaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/1*autgjd86PlO2Wi0CuLTl9Q.png" /></figure><p><strong><em>Regularization parameter(c)</em></strong>: The Regularization parameter tells the SVM optimization how much you want to avoid miss classifying each training example.</p><p>For larger values of ‘C’, a smaller margin will be accepted if the decision function is better at classifying all training points correctly. A lower ‘C’ will encourage a larger margin, therefore a simpler decision function, at the cost of training accuracy.</p><p><strong>Polynomial Kernel</strong>: It takes additional parameter ‘degree’ that controls the model complexity and the computational cost of the transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*56aE4z58d_J8oVfQwvLYZA.png" /><figcaption><strong>Boundaries for different kernels</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*qG0CGylllpRAuoZb0p72fA.png" /></figure><h3>Python Implementation :</h3><p>Here, I have used a Linear Kernel, as I was working on Text Classification problem. Most of text classification problems are linearly separable.</p><p>And Grid Search function to select the best value of gamma.</p><pre>Model : SVM<br>&quot;&quot;&quot;to select best value of gamma using gridsearch &quot;&quot;&quot;<br>from sklearn.svm import SVC<br>from sklearn.model_selection import GridSearchCV<br>from sklearn.metrics import roc_auc_score</pre><pre>clf = SVC(kernel=&#39;linear&#39;,C=0.1)<br>grid_values = {&#39;gamma&#39;:[0.001,0.01,0.05,0.1,1,10,100]}<br>grid_clf = GridSearchCV(clf,param_grid=grid_values)<br>grid_clf.fit(X_train, y_train)<br>y_decision_auc=grid_clf.decision_function(X_test)</pre><pre>print(&#39;Review&#39;)<br>print(&#39;Test set AUC :&#39;,roc_auc_score(y_test,y_decision_auc))<br>print(&#39;Grid best parameter(max.AUC):&#39;,grid_clf.best_params_)<br>print(&#39;Grid best score(AUC):&#39;,grid_clf.best_score_)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*LLuIuFvjbiUkHd6l99HxJw.png" /></figure><h4><strong>Pros</strong>:</h4><p>1. Can perform well on a range of datasets.</p><p>2. Versatile: Different kernel functions can be specified or custom kernels can be defined for specific data types.</p><p>3. Works well for both low and high dimensional data.</p><p>4. Robust against outliers (controlled using C).</p><h4><strong>Cons:</strong></h4><p>1. Efficiency (run time speed and memory usage) decreased as training set size increases.</p><p>2. Needs careful normalization of input data and parameter tuning.</p><p>3. selecting right kernel for non-linear boundaries can be tricky.</p><p>4. Difficult to interpret why a prediction was made.</p><p>Thank you for reading…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82d3dff14d39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/support-vector-machine-82d3dff14d39">Support Vector Machine</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>