<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[GDG Babcock Dataverse - Medium]]></title>
        <description><![CDATA[Read insightful articles from the GDG on Campus (Babcock) Data team members. Expand your knowledge of data science, machine learning, and artificial intelligence. Find best practices, techniques, and tips that will aid your data career. - Medium]]></description>
        <link>https://medium.com/gdsc-babcock-dataverse?source=rss----19e6b28e7627---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>GDG Babcock Dataverse - Medium</title>
            <link>https://medium.com/gdsc-babcock-dataverse?source=rss----19e6b28e7627---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 01:04:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/gdsc-babcock-dataverse" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Snowflake For Beginners.]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/snowflake-for-beginners-ff90d1bc3d0e?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/ff90d1bc3d0e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Ifeoma.data]]></dc:creator>
            <pubDate>Fri, 09 May 2025 11:32:19 GMT</pubDate>
            <atom:updated>2025-05-09T11:31:19.136Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Snowflake is a cloud-based data warehousing platform that allows users to store, analyze, and share data in a secure and efficient manner. I personally refer to Snowflake as a <strong>data lakehouse</strong>. Don’t know what that is?</p><blockquote>You can refer to this <a href="https://medium.com/@ifeoma.data/data-lakehouse-d03a3e88a477"><strong>explanation of a Data Lakehouse.</strong></a></blockquote><p>Now the big question: <strong>How does Snowflake help data engineers? Why do data engineers need to learn how to use Snowflake?</strong> While there are many alternatives to Snowflake, here are a few reasons data engineers should consider using it:</p><ol><li>It’s cloud-based, allowing you to choose from AWS, GCP, or Azure.</li><li>It supports a wide range of data workloads including (but not limited to) AI/ML, real-time analysis, data warehousing, data lakes, and ETL/ELT pipelines.</li><li>It is widely adopted in the industry.</li><li>It allows integration with various programming languages via <strong>Snowpark</strong>.</li></ol><p>All in all, Snowflake empowers data engineers to build scalable, secure, and fast data systems — without the headache of managing infrastructure.</p><blockquote>Hearing about ETL for the first time this will be helpful<strong><em> </em></strong><a href="https://medium.com/gdsc-babcock-dataverse/etl-26d806e11e11"><strong><em>ETL</em></strong></a></blockquote><h3>Why right this article ?</h3><p>I’m expanding my tech stack and experimenting with new technologies by building hands-on projects. Snowflake stood out to me because of its ease of use and strong features. I decided to document my experience to help other beginners.</p><blockquote>Check out my repo for the project <a href="https://github.com/ifeomavera/Spotifydata2"><strong>Snowflake project</strong></a><strong>.</strong></blockquote><h3>What is Snowflake?</h3><p>According to Wikipedia:</p><blockquote>Snowflake Inc. is an American cloud-based data storage company. Headquartered in Bozeman, Montana, it operates a platform that allows for data analysis and simultaneous access of data sets with minimal latency. It operates on Amazon Web Services, Microsoft Azure, and Google Cloud Platform.</blockquote><p>To add a personal note, Snowflake is also :</p><ol><li>Supports real-time analysis.</li><li>Allows writing and running SQL queries directly on the platform.</li><li>Accepts data in a wide range of formats.</li></ol><h3>Core Concepts for Beginners</h3><p>As a beginner you need to know these basic concepts used in snowflake :</p><ol><li><strong>Databases, Schemas, Tables:</strong></li></ol><p><strong>Database: A</strong>n organized collection of structured information, or data, typically stored electronically in a computer system —<a href="https://www.oracle.com/ng/database/what-is-database/"> <strong>Oracle</strong></a></p><p><strong>Schema(database): </strong>A blueprint of the structure and relationships within a database. It specifies how data will be organized in tables, fields, data types, and the relationships of these elements — <a href="https://www.datacamp.com/tutorial/database-schema"><strong>Datacamp</strong></a></p><p><strong>Table:</strong> An organized collection of data within a computer system that can be relational or non-relational—<a href="https://ca.indeed.com/career-advice/career-development/what-is-a-table-in-a-database"> <strong>Indeed</strong></a></p><blockquote>These must be set up before you can upload data to Snowflake.</blockquote><p><strong>2. Compute power(pay-as-you-go):</strong></p><p>Snowflake uses <strong>compute power</strong> to execute queries and operations. Compute power refers to the system’s ability to perform tasks and process data quickly.</p><p><strong>Computing power: </strong>This is the<strong> </strong>capability of a computer system to perform tasks and process data, with advancements over time leading to increased speed and efficiency in operations.- <a href="https://www.sciencedirect.com/topics/computer-science/computing-power"><strong>Science Direct</strong></a></p><p>Snowflake charges based on a <strong>pay-as-you-go model</strong>, meaning you only pay for what you use. It automatically <strong>suspends compute resources</strong> after 5 minutes of inactivity (this is customizable), saving you money.</p><p>Snowflake Credits are the billing unit used for compute and other services. You’re only charged when resources are actively being used.</p><p><strong>A Snowflake credit</strong> : It is a unit of measure, and it is consumed only when a customer is using resources, such as when a virtual warehouse is running, the cloud services layer is performing work, or serverless features are used. —<a href="https://docs.snowflake.com/en/user-guide/cost-understanding-compute"><strong>Snowflake</strong></a></p><p><strong>3. Stages for data loading:</strong></p><p>This is just an intro to snowflake so I won’t go in depth into snowflake but don’t worry, I will be going more in-depth into how snowflake uses Stages and how it works in Snowflake.</p><p>For now, Snowflake uses these things called stages</p><p>Stages are simply, tempoary spaces, where the dats is kept before loaded into the table.</p><blockquote>Let’s say you go to buy groceries, you will put it in a bag before you put it into your shelf, the bag is the <strong>stage</strong> and the shelf is your <strong>table</strong>.</blockquote><p><strong>4. Roles &amp; Permissions</strong> :</p><p>Before we jump into how this concept relates to snowflake, lets see what roles and permission are and why they are there to begin with.</p><p><strong>Role:</strong> This is a set of privileges that defines the authority and responsibility of users assigned to the role. —<strong> </strong><a href="https://docs.oracle.com/en/database/other-databases/nosql-database/21.2/security/roles.html#GUID-BD523202-B7C7-4A53-9975-FE2B07A112A5"><strong>Oracle</strong></a></p><p><strong>Permissions:</strong> These are access rights and privileges granted to roles within a database or data platform. — <a href="https://docs.getdbt.com/reference/database-permissions/about-database-permissions"><strong>dbt</strong></a></p><p>Now, these roles and permissions are used for data security with the <strong>least privilege rule </strong>being the foundation.</p><p>Snowflake’s approach to access control combines aspects from both of the following models:</p><ul><li><strong>Discretionary Access Control (DAC):</strong> Each object has an owner, who can in turn grant access to that object.</li><li><strong>Role-based Access Control (RBAC):</strong> Access privileges are assigned to roles, which are in turn assigned to users.</li></ul><figure><img alt="A diagram showing how Snowflake Roles and permission model works" src="https://cdn-images-1.medium.com/max/1024/1*39LK87wLcT3Mq5lgjDTyTQ.png" /></figure><p><strong>5. Snowflake SQL, Snowflake API and Snowpark</strong></p><p><strong>Snowflake SQL</strong>: This isn’t a different flavor of SQL but simply what people refer to as the SQL code generated from the logic written in Snowpark on your IDE using Java, Scala, or Python.</p><p><strong>Snowflake API: </strong>This is what aids this transformation process of turning the logic into SQL , all this runs on Snowflake not your local engine.</p><p><strong>Snowpark</strong> (my personal favourite): This is the framework that allows you to use Snowflake in various programming languages with ease.</p><blockquote>While Snowpark uses familiar languages like Python or Java, under the hood, it generates and executes SQL inside Snowflake. This means data engineers can build logic in code, but get the performance of SQL-based processing — without writing SQL themselves.</blockquote><p><strong>6. Snowflake marketplace :</strong></p><p>This is like kaggle but in snowflake, containing both free and paid datasets at your disposal. You can download them to directly or you could join them to your existing datasets to get a much ricer dataset.</p><blockquote>You have the power in your hands. This tool is particularly good for beginners and those who do not have a dataset to use right away.</blockquote><p><strong>7. What Makes Snowflake Stand Out</strong></p><p>Instant scalability:<br>The warehouse auto-expands based on the workload, amount of data, and other metrics for both cost and time efficiency.</p><p>Pay-per-second pricing:<br>This is one of my personal favourite, you only pay for how long and how much you use. And as if that’s not enough they also, automatically suspend your compute engine after the time elapses (by default 5 mins but can be changed)</p><p>Broad ecosystem (BI, ETL, AI&amp;ML tools)</p><h3>Final Thoughts</h3><blockquote>If you’re considering a career in data engineering, <strong>Snowflake is an excellent starting point</strong>. It’s beginner-friendly, versatile, and gives you access to a wide range of tools and features.</blockquote><ul><li>Don’t have a dataset? The Marketplace has you covered.</li><li>Want to practice Apache Spark? Use Snowpark.</li><li>Need real-time analysis? Use Snowpipe.</li></ul><p>I won’t say Snowflake is the best — every platform has its strengths — but I will confidently say it’s <strong>a strong place to start</strong>.</p><blockquote>Now go build your own project. That’s the only way to truly learn.</blockquote><h3>🖊️ About the Author</h3><p><strong>Ifeoma Ezeka</strong> is an aspiring Data Engineer passionate about building robust and secure data systems to deliver accurate, near error-free data for effective decision-making. She’s currently studying Software Engineering and exploring technologies like Apache Spark, Snowflake, and PostgreSQL to enhance data transformation and analytics.</p><p>Connect with her to learn and grow together in the data space.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff90d1bc3d0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/snowflake-for-beginners-ff90d1bc3d0e">Snowflake For Beginners.</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Smart Food Recommendation App with Gemini and Stability AI]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/building-a-smart-food-recommendation-app-with-gemini-and-stability-ai-74e688b34bb9?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/74e688b34bb9</guid>
            <category><![CDATA[stability-ai]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[gemini]]></category>
            <dc:creator><![CDATA[Oluwatunmibi Favour]]></dc:creator>
            <pubDate>Tue, 22 Apr 2025 16:23:23 GMT</pubDate>
            <atom:updated>2025-04-22T16:23:23.219Z</atom:updated>
            <content:encoded><![CDATA[<h4>No Backend Required</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IVii7PLD9XLnKIgaLAqr4w.png" /><figcaption>AI Recommended Meals</figcaption></figure><p><strong>Introduction</strong></p><p>Some time ago, I worked on a fun little project that used AI to recommend meals based on user health goals. It turned out surprisingly well, so I decided to share how I built it. With just a frontend (no backend!), I combined Google’s Gemini API and Stability AI to suggest dishes and generate stunning food images (maybe not the most accurate as you can see😅).</p><p><strong>Getting Started with the APIs</strong></p><p>To follow along, you’ll need API keys from both Gemini and Stability AI:</p><ul><li><strong>Gemini API</strong> (by Google): Sign up at Google AI Studio. Once signed in, go to “API Keys” and click on “Create API Key”. Follow the prompts after that and copy your API key. Ensure you store it securely. Rate limits depend on your usage tier — free tier allows limited requests per day.</li><li><strong>Stability AI</strong>: Create an account at platform.stability.ai. After verifying your email, navigate to the API dashboard to get your key. Stability AI typically offers 25 credits on registering, after which you’ll need to purchase credits.</li></ul><p><strong>Overview of Tools Used</strong></p><ul><li><strong>Gemini API</strong> — for understanding user prompts and generating meal suggestions and recipes.</li><li><strong>Stability AI</strong> — for generating realistic images of the meals.</li><li><strong>React</strong> — to build the user interface.</li></ul><p><strong>Step 1: Setting Up the Frontend</strong></p><p>We start with a basic React setup:</p><pre>npx create-react-app food-genai<br>cd food-genai</pre><p>Inside your React project, open src/App.js and prepare the state variables:</p><pre>import React, { useState } from &#39;react&#39;;<br><br>function App() {<br>  const [prompt, setPrompt] = useState(&quot;&quot;);<br>  const [response, setResponse] = useState(null);<br>  const [loading, setLoading] = useState(false);</pre><p>This section sets up three pieces of state:</p><ul><li>prompt stores the user&#39;s food craving or query.</li><li>response stores the final dish name, recipe, and image.</li><li>loading handles UI feedback during API calls.</li></ul><p><strong>Step 2: Writing the Submit Function</strong></p><p>We then define the API keys (placeholders for now) and create the submit handler:</p><pre>   const GEMINI_API_KEY = &quot;YOUR_GEMINI_API_KEY&quot;;<br>   const STABILITY_API_KEY = &quot;YOUR_STABILITY_API_KEY&quot;;<br>  <br>   const handleSubmit = async (e) =&gt; {<br>      e.preventDefault();<br>      setLoading(true);<br>      try {</pre><p>The handleSubmit function prevents the default form submission, shows a loading message, and starts the fetch operations.</p><p><strong>Using Gemini API to Generate a Recipe</strong></p><pre>        const geminiRes = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=${GEMINI_API_KEY}`, {<br>          method: &quot;POST&quot;,<br>          headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>          body: JSON.stringify({<br>            contents: [{ parts: [{ text: `Suggest a dish and recipe for: ${prompt}` }] }]<br>          })<br>        });<br>        const geminiData = await geminiRes.json();<br>        const textResponse = geminiData?.candidates?.[0]?.content?.parts?.[0]?.text;<br>        const suggestionText = textResponse.replace(/```json|```/g, &#39;&#39;).trim();<br>        const dishName = suggestionText.split(&quot;\n&quot;)[0];</pre><p>We make a POST request to Gemini’s API. The prompt is sent in plain text. We then extract:</p><ul><li>The recipe (suggestionText)</li><li>The dish name (first line of the text)</li></ul><p><strong>Using Stability AI to Generate an Image</strong></p><pre>            const formData = new FormData();<br>            formData.append(&#39;prompt&#39;, dishName);<br>            formData.append(&#39;model&#39;, &#39;sd3&#39;);<br>            formData.append(&#39;output_format&#39;, &#39;webp&#39;);<br>            formData.append(&#39;aspect_ratio&#39;, &#39;1:1&#39;);<br>            formData.append(&#39;negative_prompt&#39;, &#39;blurry, text, watermark&#39;);<br>            <br>            const stabilityRes = await fetch(&#39;https://api.stability.ai/v2beta/stable-image/generate/core&#39;,<br>              {<br>                method: &#39;POST&#39;,<br>                headers: {<br>                  &#39;Authorization&#39;: `Bearer ${STABILITY_API_KEY}`,<br>                  &#39;Accept&#39;: &#39;image/*&#39;<br>                },<br>                body: formData<br>              }<br>             );<br>            <br>            const imageData = await stabilityRes.arrayBuffer();<br>            const imageUrl = imageData.artifacts[0].url;<br>        <br>            setResponse({ dish: dishName, recipe: suggestionText, imageUrl });<br>          } catch (error) {<br>            console.error(&quot;Error:&quot;, error);<br>          }<br>          setLoading(false);<br>       };</pre><p>This part:</p><ul><li>Sends the dish name to Stability AI</li><li>Requests a 1:1 image of the food</li><li>Updates the state with the image URL and recipe</li></ul><p><strong>Step 3: Creating the User Interface</strong></p><pre>    return (<br>      &lt;div className=&quot;App&quot;&gt;<br>        &lt;form onSubmit={handleSubmit}&gt;<br>          &lt;input value={prompt} onChange={(e) =&gt; setPrompt(e.target.value)} placeholder=&quot;What do you want to eat?&quot; /&gt;<br>          &lt;button type=&quot;submit&quot;&gt;Get Recommendation&lt;/button&gt;<br>        &lt;/form&gt;<br>        {loading &amp;&amp; &lt;p&gt;Loading...&lt;/p&gt;}<br>        {response &amp;&amp; (<br>          &lt;div&gt;<br>            &lt;h2&gt;{response.dish}&lt;/h2&gt;<br>            &lt;p&gt;{response.recipe}&lt;/p&gt;<br>            &lt;img src={response.imageUrl} alt={response.dish} /&gt;<br>          &lt;/div&gt;<br>        )}<br>      &lt;/div&gt;<br>    );<br>}</pre><p>This renders:</p><ul><li>A form with input and submit button</li><li>Loading message during fetches</li><li>Final result: the dish name, full recipe, and the generated image</li></ul><p><strong>Security Note</strong></p><p>This example uses API keys directly in the frontend. While it’s great for demos or learning, <strong>don’t use this approach in production.</strong> Instead, consider storing keys securely in a backend or using a serverless function.</p><p><strong>Conclusion</strong></p><p>With just a few tools and no backend, you can build an intelligent, visually rich food recommendation app powered by AI. Try tweaking the recipe prompt, changing the image dimensions, or filtering by diet. The possibilities are deliciously endless!</p><blockquote>I’m Favour Oluwatunmibi, a software engineer on a mission to <strong>solve real-world problems through code</strong>. Every line I write, every system I design, is driven by one question: How can technology make this better?</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74e688b34bb9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/building-a-smart-food-recommendation-app-with-gemini-and-stability-ai-74e688b34bb9">Building a Smart Food Recommendation App with Gemini and Stability AI</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ The Model Context Protocol (MCP): The USB-C of AI-Data Connectivity]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/the-model-context-protocol-mcp-the-usb-c-of-ai-data-connectivity-79e40c447fc3?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/79e40c447fc3</guid>
            <category><![CDATA[mcps]]></category>
            <dc:creator><![CDATA[Obuzortochi]]></dc:creator>
            <pubDate>Sat, 12 Apr 2025 12:26:04 GMT</pubDate>
            <atom:updated>2025-04-12T12:17:00.019Z</atom:updated>
            <content:encoded><![CDATA[<p>The AI Bottleneck No One Talks About</p><p>In recent years, AI assistants have made staggering leaps in capability. Large language models (LLMs) like GPT-4, Claude, and others are now capable of answering complex questions, generating entire reports, writing code, and more. But there’s a catch. These models are still disconnected from the environments we actually work in. They don’t inherently have access to the documents we’re editing, the codebases we’re debugging, or the internal databases we manage. Every time we want an AI assistant to really help with our work, we have to manually copy-paste content into the chat — or build a custom integration from scratch. That’s where MCP comes in.</p><p>🔌 What Is MCP?</p><p>The Model Context Protocol (MCP) is a universal, open-source standard for connecting AI assistants to the places where your data lives — your files, tools, and platforms. Think of it as the USB-C for AI applications. Just as USB-C provides a standard way for your laptop to connect with everything from keyboards to displays, MCP standardizes how AI assistants connect to data sources, whether they’re local files, databases, APIs, or cloud-based enterprise systems like Google Drive or GitHub.</p><p>🧱 How MCP Works: A Simple Breakdown</p><p>🔄 Client-Server Architecture At its core, MCP follows a client-server architecture: MCP Host (Client): The AI assistant or tool you’re using (e.g., Claude Desktop, VS Code with AI support). MCP Server: A small program that knows how to access a specific data source — like your local filesystem, a Postgres database, or GitHub. Data Sources: Local or cloud-based systems that hold the content you want your AI assistant to understand and work with. When the MCP client needs information, it sends a request via the MCP protocol to an MCP server, which then retrieves or modifies the data in the data source and sends the relevant context back to the AI assistant.</p><p>🧰 Core Components of MCP</p><p>Here’s what the MCP ecosystem looks like today:</p><ol><li>🛠 MCP Specification and SDKs A complete set of documentation and software development kits to help developers build their own MCP clients or servers.</li><li>💻 Local MCP Server Support Built into the Claude Desktop apps, allowing users to instantly connect their local environment to Claude.</li><li>🧩 Open-Source Server Repository Ready-to-use, community-contributed MCP servers for: Google Drive Slack GitHub Git Postgres Puppeteer (for web automation and scraping)</li><li>Why MCP Is a Game-Changer</li></ol><p>✅ 1. AI Assistants with Real Context The most advanced model still can’t guess what’s in your codebase or internal CRM unless you tell it. With MCP, the AI assistant can automatically access relevant files or data — leading to better, more accurate outputs.</p><p>✅ 2. No More Custom Connectors In the past, every data source needed its own custom-built integration. Now, thanks to MCP’s standardized protocol, you build once and use everywhere.</p><p>✅ 3. Model-Agnostic Flexibility MCP isn’t tied to any one AI model. Whether you use Claude, GPT, or another LLM, you can plug in the same connectors.</p><p>✅ 4. Secure by Design MCP is designed for enterprise environments, supporting fine-grained access control, local-only connections, and privacy-aware integration.</p><p>✅ 5. Build Better Agents and Workflows Want an AI agent to automate a workflow?</p><p>With MCP, you can connect it to:</p><p>Your internal wiki</p><p>Your CRM A web browser (via Puppeteer)</p><p>Your SQL database</p><p>GitHub PRs</p><p>Now it can plan, reason, and act using real context.</p><h3>🔧 Tutorial: Building Your First MCP Implementation</h3><p>Now that you understand the concept and importance of MCP, let’s build a simple but functional MCP implementation that demonstrates this protocol. In this tutorial, we’ll create:</p><ol><li>An MCP server that provides access to a local filesystem</li><li>An MCP client that connects to this server</li><li>A simulated AI assistant that uses MCP to access files</li></ol><h3>🏗️ Setting Up the Project</h3><p>First, let’s set up our project structure:</p><p>mcp-tutorial/</p><p>├── mcp_server.py</p><p>├── mcp_client.py</p><p>├── ai_assistant.py</p><p>└── main.py</p><h3>📋 Step 1: Define the MCP Server</h3><p>Let’s create an MCP server that can handle filesystem operations. This will be our data source adapter.</p><pre># mcp_server.py<br>import os<br>import json<br>from typing import Dict, List, Optional<br><br>class MCPFileSystemServer:<br>    &quot;&quot;&quot;<br>    A simple MCP server that provides access to the local filesystem.<br>    &quot;&quot;&quot;<br>    <br>    def handle_request(self, request: Dict) -&gt; Dict:<br>        &quot;&quot;&quot;Process an incoming MCP request and return a response.&quot;&quot;&quot;<br>        <br>        request_type = request.get(&quot;type&quot;)<br>        <br>        if request_type == &quot;list_files&quot;:<br>            return self._handle_list_files(request)<br>        elif request_type == &quot;read_file&quot;:<br>            return self._handle_read_file(request)<br>        elif request_type == &quot;write_file&quot;:<br>            return self._handle_write_file(request)<br>        else:<br>            return {<br>                &quot;status&quot;: &quot;error&quot;,<br>                &quot;error&quot;: f&quot;Unsupported request type: {request_type}&quot;<br>            }<br>    <br>    def _handle_list_files(self, request: Dict) -&gt; Dict:<br>        &quot;&quot;&quot;List files in the specified directory.&quot;&quot;&quot;<br>        <br>        directory = request.get(&quot;directory&quot;, &quot;.&quot;)<br>        <br>        try:<br>            files = os.listdir(directory)<br>            return {<br>                &quot;status&quot;: &quot;success&quot;,<br>                &quot;files&quot;: [<br>                    {<br>                        &quot;name&quot;: file,<br>                        &quot;is_directory&quot;: os.path.isdir(os.path.join(directory, file)),<br>                        &quot;path&quot;: os.path.join(directory, file)<br>                    }<br>                    for file in files<br>                ]<br>            }<br>        except Exception as e:<br>            return {<br>                &quot;status&quot;: &quot;error&quot;,<br>                &quot;error&quot;: str(e)<br>            }<br>    <br>    def _handle_read_file(self, request: Dict) -&gt; Dict:<br>        &quot;&quot;&quot;Read the contents of a file.&quot;&quot;&quot;<br>        <br>        file_path = request.get(&quot;path&quot;)<br>        <br>        if not file_path:<br>            return {<br>                &quot;status&quot;: &quot;error&quot;,<br>                &quot;error&quot;: &quot;No file path provided&quot;<br>            }<br>        <br>        try:<br>            with open(file_path, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>                content = f.read()<br>                <br>            return {<br>                &quot;status&quot;: &quot;success&quot;,<br>                &quot;content&quot;: content,<br>                &quot;metadata&quot;: {<br>                    &quot;size&quot;: os.path.getsize(file_path),<br>                    &quot;modified&quot;: os.path.getmtime(file_path)<br>                }<br>            }<br>        except Exception as e:<br>            return {<br>                &quot;status&quot;: &quot;error&quot;,<br>                &quot;error&quot;: str(e)<br>            }<br>    <br>    def _handle_write_file(self, request: Dict) -&gt; Dict:<br>        &quot;&quot;&quot;Write content to a file.&quot;&quot;&quot;<br>        <br>        file_path = request.get(&quot;path&quot;)<br>        content = request.get(&quot;content&quot;)<br>        <br>        if not file_path:<br>            return {<br>                &quot;status&quot;: &quot;error&quot;,<br>                &quot;error&quot;: &quot;No file path provided&quot;<br>            }<br>        <br>        if content is None:<br>            return {<br>                &quot;status&quot;: &quot;error&quot;,<br>                &quot;error&quot;: &quot;No content provided&quot;<br>            }<br>        <br>        try:<br>            with open(file_path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>                f.write(content)<br>                <br>            return {<br>                &quot;status&quot;: &quot;success&quot;,<br>                &quot;metadata&quot;: {<br>                    &quot;size&quot;: os.path.getsize(file_path),<br>                    &quot;modified&quot;: os.path.getmtime(file_path)<br>                }<br>            }<br>        except Exception as e:<br>            return {<br>                &quot;status&quot;: &quot;error&quot;,<br>                &quot;error&quot;: str(e)<br>            }</pre><h3>📋 Step 2: Create the MCP Client</h3><p>Next, let’s create a client that can connect to MCP servers and route requests:</p><pre># mcp_client.py<br>from typing import Dict, Any<br><br>class MCPClient:<br>    &quot;&quot;&quot;<br>    A simple MCP client that can be used by an AI assistant to access data sources.<br>    &quot;&quot;&quot;<br>    <br>    def __init__(self):<br>        self.servers = {}<br>    <br>    def register_server(self, name: str, server: Any) -&gt; None:<br>        &quot;&quot;&quot;Register an MCP server with this client.&quot;&quot;&quot;<br>        self.servers[name] = server<br>    <br>    def send_request(self, server_name: str, request: Dict) -&gt; Dict:<br>        &quot;&quot;&quot;Send a request to a specific MCP server.&quot;&quot;&quot;<br>        <br>        server = self.servers.get(server_name)<br>        if not server:<br>            return {<br>                &quot;status&quot;: &quot;error&quot;,<br>                &quot;error&quot;: f&quot;Server &#39;{server_name}&#39; not found&quot;<br>            }<br>        <br>        return server.handle_request(request)</pre><h3>📋 Step 3: Create the AI Assistant Interface</h3><p>Now, let’s create an AI assistant that uses our MCP client to access the filesystem:</p><pre># ai_assistant.py<br>from typing import Dict, Optional<br>from mcp_client import MCPClient<br><br>class AIAssistant:<br>    &quot;&quot;&quot;<br>    A simulated AI assistant that uses MCP to access data.<br>    &quot;&quot;&quot;<br>    <br>    def __init__(self, mcp_client: MCPClient):<br>        self.mcp_client = mcp_client<br>    <br>    def list_directory(self, server_name: str, directory: str = &quot;.&quot;) -&gt; Dict:<br>        &quot;&quot;&quot;List files in a directory using an MCP server.&quot;&quot;&quot;<br>        <br>        request = {<br>            &quot;type&quot;: &quot;list_files&quot;,<br>            &quot;directory&quot;: directory<br>        }<br>        <br>        return self.mcp_client.send_request(server_name, request)<br>    <br>    def read_file(self, server_name: str, file_path: str) -&gt; Dict:<br>        &quot;&quot;&quot;Read a file using an MCP server.&quot;&quot;&quot;<br>        <br>        request = {<br>            &quot;type&quot;: &quot;read_file&quot;,<br>            &quot;path&quot;: file_path<br>        }<br>        <br>        return self.mcp_client.send_request(server_name, request)<br>    <br>    def write_file(self, server_name: str, file_path: str, content: str) -&gt; Dict:<br>        &quot;&quot;&quot;Write to a file using an MCP server.&quot;&quot;&quot;<br>        <br>        request = {<br>            &quot;type&quot;: &quot;write_file&quot;,<br>            &quot;path&quot;: file_path,<br>            &quot;content&quot;: content<br>        }<br>        <br>        return self.mcp_client.send_request(server_name, request)<br>    <br>    def analyze_code_file(self, server_name: str, file_path: str) -&gt; Dict:<br>        &quot;&quot;&quot;Analyze a code file and provide insights.&quot;&quot;&quot;<br>        <br>        # First, read the file<br>        response = self.read_file(server_name, file_path)<br>        <br>        if response[&quot;status&quot;] != &quot;success&quot;:<br>            return response<br>            <br>        <br>        # Here is just some basic analysis<br>        content = response[&quot;content&quot;]<br>        lines = content.split(&quot;\n&quot;)<br>        <br>        analysis = {<br>            &quot;lines_of_code&quot;: len(lines),<br>            &quot;empty_lines&quot;: len([line for line in lines if line.strip() == &quot;&quot;]),<br>            &quot;comments&quot;: len([line for line in lines if line.strip().startswith(&quot;#&quot;)])<br>        }<br>        <br>        return {<br>            &quot;status&quot;: &quot;success&quot;,<br>            &quot;file_path&quot;: file_path,<br>            &quot;analysis&quot;: analysis<br>        }</pre><h3>📋 Step 4: Put It All Together</h3><p>Finally, let’s create a main script that uses all these components:</p><pre># main.py<br>from mcp_server import MCPFileSystemServer<br>from mcp_client import MCPClient<br>from ai_assistant import AIAssistant<br>import json<br><br>def print_response(response):<br>    &quot;&quot;&quot;Pretty-print a response from the AI assistant.&quot;&quot;&quot;<br>    print(json.dumps(response, indent=2))<br><br>def run_demo():<br>    # Create an MCP server for filesystem access<br>    fs_server = MCPFileSystemServer()<br>    <br>    # Create an MCP client<br>    mcp_client = MCPClient()<br>    <br>    # Register the filesystem server with the client<br>    mcp_client.register_server(&quot;filesystem&quot;, fs_server)<br>    <br>    # Create an AI assistant that uses the MCP client<br>    assistant = AIAssistant(mcp_client)<br>    <br>    print(&quot;=== MCP Demo: AI Assistant with Filesystem Access ===&quot;)<br>    <br>    # Example 1: List files in the current directory<br>    print(&quot;\n📁 Listing files in the current directory:&quot;)<br>    response = assistant.list_directory(&quot;filesystem&quot;)<br>    print_response(response)<br>    <br>    # Example 2: Read a Python file<br>    print(&quot;\n📄 Reading a Python file (main.py):&quot;)<br>    response = assistant.read_file(&quot;filesystem&quot;, &quot;main.py&quot;)<br>    if response[&quot;status&quot;] == &quot;success&quot;:<br>        print(f&quot;File size: {response[&#39;metadata&#39;][&#39;size&#39;]} bytes&quot;)<br>        print(&quot;Content preview (first 100 chars):&quot;)<br>        print(response[&quot;content&quot;][:100] + &quot;...&quot; if len(response[&quot;content&quot;]) &gt; 100 else response[&quot;content&quot;])<br>    else:<br>        print_response(response)<br>    <br>    # Example 3: Analyze a Python file<br>    print(&quot;\n🔍 Analyzing a Python file (main.py):&quot;)<br>    response = assistant.analyze_code_file(&quot;filesystem&quot;, &quot;main.py&quot;)<br>    print_response(response)<br>    <br>    # Example 4: Write a new file<br>    print(&quot;\n✏️ Writing a new file (hello_mcp.txt):&quot;)<br>    content = &quot;Hello from MCP!\nThis file was created using the Model Context Protocol.&quot;<br>    response = assistant.write_file(&quot;filesystem&quot;, &quot;hello_mcp.txt&quot;, content)<br>    print_response(response)<br>    <br>    print(&quot;\n✅ Demo completed successfully!&quot;)<br>    print(&quot;You now have a basic MCP implementation that connects an AI assistant to your filesystem.&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    run_demo()</pre><h3>🚀 Running the Demo</h3><p>Save all the files in your project directory and run:</p><pre>python main.py</pre><p>You should see output similar to this:</p><pre>=== MCP Demo: AI Assistant with Filesystem Access ===<br><br>📁 Listing files in the current directory:<br>{<br>  &quot;status&quot;: &quot;success&quot;,<br>  &quot;files&quot;: [<br>    {<br>      &quot;name&quot;: &quot;ai_assistant.py&quot;,<br>      &quot;is_directory&quot;: false,<br>      &quot;path&quot;: &quot;ai_assistant.py&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;mcp_client.py&quot;,<br>      &quot;is_directory&quot;: false,<br>      &quot;path&quot;: &quot;mcp_client.py&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;mcp_server.py&quot;,<br>      &quot;is_directory&quot;: false,<br>      &quot;path&quot;: &quot;mcp_server.py&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;main.py&quot;,<br>      &quot;is_directory&quot;: false,<br>      &quot;path&quot;: &quot;main.py&quot;<br>    }<br>  ]<br>}<br><br>📄 Reading a Python file (main.py):<br>File size: 1872 bytes<br>Content preview (first 100 chars):<br>from mcp_server import MCPFileSystemServer<br>from mcp_client import MCPClient<br>from ai_assistant import AIAssistant<br>import...<br><br>🔍 Analyzing a Python file (main.py):<br>{<br>  &quot;status&quot;: &quot;success&quot;,<br>  &quot;file_path&quot;: &quot;main.py&quot;,<br>  &quot;analysis&quot;: {<br>    &quot;lines_of_code&quot;: 54,<br>    &quot;empty_lines&quot;: 8,<br>    &quot;comments&quot;: 5<br>  }<br>}<br><br>✏️ Writing a new file (hello_mcp.txt):<br>{<br>  &quot;status&quot;: &quot;success&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;size&quot;: 69,<br>    &quot;modified&quot;: 1681228652.7845678<br>  }<br>}<br><br>✅ Demo completed successfully!<br>You now have a basic MCP implementation that connects an AI assistant to your filesystem.</pre><h3>🏁In Conclusion</h3><p>That is basically how the Model Context Protocol works. With just a few hundred lines of code, we’ve created a system that allows an AI assistant to access and manipulate the filesystem in a secure, standardized way.</p><p>As you have seen, MCP truly is the “USB-C of AI-Data Connectivity”, it provides a universal interface for AI assistants to connect with the data sources they need to be truly helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79e40c447fc3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/the-model-context-protocol-mcp-the-usb-c-of-ai-data-connectivity-79e40c447fc3">🧠📡 The Model Context Protocol (MCP): The USB-C of AI-Data Connectivity</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying ML: What I Wish I Knew When I Started]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/demystifying-ml-what-i-wish-i-knew-when-i-started-466532772eec?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/466532772eec</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Obalabi adepoju]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 20:27:31 GMT</pubDate>
            <atom:updated>2025-04-12T11:18:08.790Z</atom:updated>
            <content:encoded><![CDATA[<h4>I stayed away from the tough stuff — this will help you lean in instead</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SdxOSfU9yCV7sEynKbZe5A.png" /><figcaption>image generated by author from code</figcaption></figure><p>I recall starting my data journey in analytics and eventually wanting to venture into ML/DL but along the way, I encountered several terminologies and concepts that dissuaded me. I kept putting it off cause I felt either the math behind it was too “complex” or I wouldn’t understand regardless.</p><p>A couple of months ago, I decided to <strong>just start</strong> — and I haven’t regretted it. Now, as I learn more about the complexity that once held me back, I can’t help but laugh at myself because it all makes sense to me now.</p><p>I’m writing this article to demystify some of the jargon in the machine learning field and I’m hoping that after reading it, you won’t let the fear of difficulty stop you from exploring all the cool things you can learn.</p><p>I’m not here to explain what an LLM or an AI agent is — though they’re definitely important to know these days. Instead, I’ll walk you through some of the more common concepts you’ll come across as you start learning about machine learning.</p><p>Let’s get into it 😄.</p><p><strong>1.Vectorization</strong>: I’d explain this as a technique used to perform multiple tasks in just a few lines of code. Or, even better — vectorization is basically the art of removing explicit for loops from your code. Since data science often involves working with huge datasets, vectorization offers a way to handle large-scale operations both efficiently and with less code.</p><p>Example:</p><pre># To sum up all values in an array without vectorization<br>sum = 0 <br>for i in arr:<br>  sum += i<br># with vectorization and the numpy library<br>sum = np.sum(arr)</pre><p><strong>2. Perceptron</strong>: You might’ve heard that a neural network is sometimes called a multilayer perceptron — that’s because a perceptron is the simplest form of a neural network consisting of a single input layer. It takes input data, assigns weights, and uses a formula (an activation function) to output one of two classes.</p><p>Think of it as a model that’ll take a handful of characteristics and then it applies a function that’ll decide whether these specific features belong to a particular class or not.</p><p><strong>3. Epoch</strong>: An epoch occurs during neural network training when the model has processed the entire dataset once. Think of it like a student taking a test — each question helps them understand more, and once they’ve completed the whole test, that’s one epoch. Now, imagine going through the same test multiple times — they spot new patterns, refine techniques, and reinforce what they’ve already learned.</p><p><strong>4. Ensembles</strong>: An ensemble in machine learning applies the idea that “multiple heads are better than one.” It combines several models — often decision trees or neural networks — to make a prediction, with the final output typically based on the most common result (like voting). This helps reduce the effect of high bias or variance that a single model might have.</p><p><strong>5. Hyperparameter</strong>: Let me explain this with an analogy. Suppose I bake three different batches of cookies, each at a different temperature and for different amounts of time to see which combination gives the best results. The <strong>heat</strong> and <strong>time</strong> are my <strong>hyperparameters</strong> — I set them <em>before</em> baking, and they control how the process unfolds.</p><p>These differ from internal parameters because I decide on them ahead of time, just like a model’s configuration before training. In machine learning, hyperparameters include things like input size, model depth, or learning rate — settings I control, not values the model learns.</p><p><strong>6. Embeddings</strong>: I used to wonder how an AI chatbot like ChatGPT could understand the context behind what I typed — and if you’re curious about this too, I’m glad you’re here.</p><p><strong>Embeddings</strong> are how AI represents real-world text or images as arrays of numbers. These number-based representations carry both meaning and context. That’s how it knows that <em>“I think ML is fun”</em> and <em>“I think ML is interesting”</em> essentially mean the same thing.</p><p>Here’s what the model might actually see these sentences as</p><pre>&quot;I think ML is fun&quot;         →  [0.42, -0.15, 0.88, 0.33, 0.76]  <br>&quot;I think ML is interesting&quot; →  [0.40, -0.13, 0.85, 0.31, 0.74]</pre><p><strong>7. Transformers</strong>: Understanding how transformers and embeddings work will definitely broaden your knowledge of how LLMs like Gemini and others work. Transformers are deep learning algorithms designed to process sequences of data — such as sentences — by focusing on the relationships between all elements in the sequence at once.</p><p>Unlike older models that processed words one by one in order, transformers use a mechanism called <strong>“self-attention”</strong> to weigh the importance of each part of the input in relation to the others. This enables transformers to handle long-range dependencies in language, where the context of a word might depend on another word that appears much earlier or later in the sentence.</p><p>For example, consider the sentence: <strong>“The bank closed early, so we had to find another place to fish.”</strong> To truly understand this, you need to look at the entire sentence. Only then can you understand that <strong>“bank”</strong> may refer to a <strong>riverbank</strong> rather than a financial institution.</p><p>There are hundreds of terms in the ML field, and as much as I would love to walk you through all of them, I hope you’ve found something here that sparks your interest rather than confusion. If you ever want to dive deeper into data science jargon, feel free to check out this <a href="https://medium.com/aimonks/130-data-science-terms-every-data-scientist-should-know-7199a22fc809">article</a>.</p><p>I hope you enjoyed this article as much as I enjoyed writing it. Please leave a clap or 50 and let me know your thoughts or contributions in the comments :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=466532772eec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/demystifying-ml-what-i-wish-i-knew-when-i-started-466532772eec">Demystifying ML: What I Wish I Knew When I Started</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crawl The Web And Scrape Data With A Spider]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/crawl-the-web-and-scrape-data-with-a-spider-cc76b50aa458?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/cc76b50aa458</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[scrapy]]></category>
            <category><![CDATA[web-crawler]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[galaxia]]></dc:creator>
            <pubDate>Fri, 14 Mar 2025 13:22:50 GMT</pubDate>
            <atom:updated>2025-03-14T13:22:50.637Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8p34u7aI_-JXNtuRpvZmhg.png" /></figure><p>Not a real spider, but a scrapy kind of spider haha.</p><h4><strong>Scrapy</strong></h4><p>A scrapy kind of spider, scrapy is an open-source python framework used for extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. The goal of scrapy is to make web scraping easier. It has a wide range of built in features for building scalable spiders. As we continue we would know what a scrapy spider entails.</p><h4><strong>Why Use Scrapy</strong></h4><p>Scrapy is highly customizable, fast and efficient, Scrapy provides a lot of powerful features for making scraping easy. Let’s check out some of these features.</p><h4><strong>Key Features of Scrapy</strong></h4><ul><li><strong>Advanced Data Extraction:</strong></li></ul><p>Built-in support for selecting and extracting data from HTML/XML sources using extended CSS selectors and XPath expressions, with helper methods for extraction using regular expressions.</p><ul><li><strong>An Interactive shell:</strong></li></ul><p>An interactive shell console for trying out the CSS and XPath expressions to scrape data, which is very useful when writing or debugging your spiders.</p><ul><li><strong>Flexible Export Format:</strong></li></ul><p>Built-in support for generating feed exports in multiple formats such as JSON, CSV, XML.</p><ul><li><strong>Reusable spiders and data pipelines:</strong></li></ul><p>Reusable spiders to crawl sites from Sitemaps and XML/CSV feeds, a pipeline for a clean data extraction and downloading and processing media and other files.</p><h4><strong>Lets Get Running with Scrapy</strong></h4><p>We will be scraping books data while crawling through multiple pages from the <a href="https://books.toscrape.com/">books.toscrape.com</a> webpage.</p><h4><strong>Set up scrapy</strong></h4><p>First and foremost lets set<strong> </strong>up our scrapy environment.</p><p>Start with installing scrapy, the easiest way is to use pip(python’s package manager). In your terminal enter the command:</p><pre>pip install scrapy</pre><p>It will take a few minutes, Once Installed, in the terminal, enter:</p><pre>scrapy version</pre><p>If the scrapy version is returned, then it installed successfully.</p><p>Now lets create a scrapy project, enter this command in your terminal:</p><pre>scrapy startproject books_scraper</pre><p>books_scraper is the name of our project since we will be scraping relevant books data, you can use any name at your choice.</p><p>Once you create the project, a Folder structure will be generated in your workspace in this format:</p><pre>web-scraping-project/<br>└── books_scraper/<br>    ├── books_scraper/<br>    │   ├── spiders/<br>    │   │   ├── __init__.py<br>    │   │   └── books_spider.py<br>    │   ├── __init__.py<br>    │   ├── items.py<br>    │   ├── middlewares.py<br>    │   ├── pipelines.py <br>    │   └── settings.py <br>    └── scrapy.cfg </pre><ul><li><strong>scrapy.cfg</strong>: A plain text file that contains a series of configuration directives including directives for the project name, the location of the spider modules, and the settings to be used when running the web spiders.</li></ul><pre>[settings]<br>default = src.settings<br><br>[deploy]<br>project = books_scraper</pre><ul><li><strong>settings.py</strong>: It contains various configuration settings that control the behavior of the Scrapy framework.</li></ul><pre>BOT_NAME = &#39;books_scraper&#39;<br><br>SPIDER_MODULES = [&#39;src.spiders&#39;]<br>NEWSPIDER_MODULE = &#39;src.spiders&#39;<br><br># Configure maximum concurrent requests performing at the same time<br>CONCURRENT_REQUESTS = 16<br><br># Configure a delay for requests<br>DOWNLOAD_DELAY = 1</pre><ul><li><strong>pipelines.py</strong>: Pipelines define a set of processing steps on data while it is being extracted.</li></ul><pre>class BookPipeline:<br>    def process_item(self, item, spider):<br>        # Clean or store the data here<br>        return item</pre><ul><li><strong>middleware.py</strong>: This file contains middleware information for request routing.</li><li><strong>items.py</strong>: Contains objects that represent the data a spider is trying to extract. Items are defined using Python classes and help organize the data in a structured format.</li></ul><pre>import scrapy<br><br><br>class BookItem(scrapy.Item):<br><br>    title = scrapy.Field()<br><br>    price = scrapy.Field()<br>    <br>    rating = scrapy.Field()<br>    <br>    description = scrapy.Field()<br>    <br>    availability = scrapy.Field()<br>    <br>    category = scrapy.Field()<br>    <br>    book_url = scrapy.Field()</pre><p>This are the books data we will be scraping from the <a href="https://books.toscrape.com/">books.toscrape.com</a> webpage.</p><h4><strong>Create your very own spider</strong></h4><p><strong>Scrapy spiders</strong> are classes that you define and that scrapy uses to scrape information from a website (or a group of websites). They define the requests to be made, how to follow links in pages and parse the downloaded page content to extract data.</p><p>Lets define our <strong>spider class. </strong>First we will import the library:</p><pre>import scrapy</pre><p><strong>Then define the spider</strong>:</p><pre>class BookSpider(scrapy.Spider):<br><br>    name = &#39;books&#39;<br><br>    start_urls = [&#39;http://books.toscrape.com&#39;]</pre><p><strong>name </strong>is the name we have given to the spider. <strong>start_urls </strong>is the list of URLs we would be extracting our data.</p><p>Now lets define our <strong>parse </strong>method:</p><pre>    def parse(self, response):<br><br>        books = response.css(&#39;article.product_pod&#39;)<br><br>        for book in books:<br>            book_url = book.css(&#39;h3 a::attr(href)&#39;).get()<br>            yield response.follow(book_url, callback = self.parse_book_details)<br><br>        #Handle pagination<br>        next_page = response.css(&#39;li.next a::attr(href)&#39;).get()<br>        if next_page:<br>            yield response.follow(next_page, callback = self.parse)</pre><p>This method will extract all the books on the current page.</p><pre>books = response.css(&#39;article.product_pod&#39;)</pre><p>Here, we’re using <strong>CSS selectors</strong> (response.css) to extract all books in the current page. Here is how the CSS selector works, when you open the webpage, right click on a book then navigate to the inspect view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVxQpKWhTWMvxS693kP-3w.png" /></figure><p>You will see that each book is enclosed in this Html <strong>article </strong>tag which has a class <strong>product_pod</strong>, the CSS selector then extract all the content of this <strong>article tag.</strong></p><pre>for book in books:<br>            book_url = book.css(&#39;h3 a::attr(href)&#39;).get()<br>            yield response.follow(book_url, callback = self.parse_book_details)</pre><p>Loops through each book and gets the link to the book’s detail page.</p><p>response.follow() is then used to follow each link and calls parse_book_details which is another method to extract other book data.</p><pre>next_page = response.css(&#39;li.next a::attr(href)&#39;).get()<br>if next_page:<br>    yield response.follow(next_page, callback = self.parse)</pre><p>Checks if there’s a next page, If found, it follows the link and then calls parse() method again, scraping the books in the next page.</p><pre>def parse_book_details(self, response):<br>    book_items = BookItem()<br><br>    book_items[&#39;title&#39;] = response.css(&#39;.product_main h1::text&#39;).get()<br><br>    book_items[&#39;price&#39;] = response.css(&#39;p.price_color::text&#39;).get()<br>    <br>    book_items[&#39;rating&#39;] = response.css(&#39;p.star-rating::attr(class)&#39;).get().split()[-1]<br>    <br>    book_items[&#39;availability&#39;] = response.css(&#39;p.instock.availability::text&#39;).extract()[1].strip()<br>    <br>    book_items[&#39;category&#39;] = response.css(&#39;ul.breadcrumb li:nth-child(3) a::text&#39;).get()<br>    <br>    book_items[&#39;book_url&#39;] = response.url<br>    <br>    book_items[&#39;description&#39;] = response.css(&#39;#product_description ~ p::text&#39;).get()<br>    if book_items[&#39;description&#39;]:<br>        book_items[&#39;description&#39;] = book_items[&#39;description&#39;].strip()<br><br>    yield book_items</pre><p>The parse_book_details method then extract each books details such as the title which is a text in an element with the product_main class.</p><p>The spider then returns the book_items which will contain each books data.</p><p><strong>Lets run the spider, </strong>in the terminal, use this command:</p><pre>scrapy crawl books</pre><p>The spider will visit the starting URL, extract the data, follow links to the next pages, and scrape additional books.</p><h4><strong>Extract Data</strong></h4><p>With scrapy you can export the scraped data in CSV, XML or JSON format. To extract the data to a CSV file, use this command:</p><pre>scrapy crawl books -o books.csv</pre><p>Now the spider will run in the terminal until it finish scraping and extracting the books data then storing it in the book.csv file format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*duznK8e_4whCj3nFGYCVQw.png" /></figure><p>Here&#39;s the data we scraped in a data frame.</p><h4><strong>🕸️ Wrapping Up</strong></h4><p>Scrapy makes web scraping not just powerful, but fun and efficient. With just a few lines of code, you can crawl multiple pages, extract structured data, and export it in formats like CSV or JSON.</p><p>Whether you’re collecting product data or articles scrapy’s built-in tools and modular design make it a great choice for beginners and pros.</p><p>Now that you have seen how a scrapy spider works and what it can do,</p><p>Why not build your own spider and crawl the web you way?🕷️</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc76b50aa458" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/crawl-the-web-and-scrape-data-with-a-spider-cc76b50aa458">Crawl The Web And Scrape Data With A Spider</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ETL]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/etl-26d806e11e11?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/26d806e11e11</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Ifeoma.data]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 11:38:58 GMT</pubDate>
            <atom:updated>2025-02-04T11:37:33.718Z</atom:updated>
            <content:encoded><![CDATA[<p>We’ve heard the term ETL in the data world over and over, but what does it mean?</p><p>According to <a href="https://aws.amazon.com/what-is/etl/">AWS</a>, Extract, Transform, and Load (ETL) is the process of combining data from multiple sources into a large, central repository called a data warehouse.</p><p><a href="https://www.ibm.com/think/topics/etl#:~:text=ETL%E2%80%94meaning%20extract%2C%20transform%2C,lake%20or%20other%20target%20system.">IBM </a>defines ETL as a data integration process that combines, cleans, and organizes data from multiple sources into a single, consistent data set for storage in a data warehouse, data lake, or other target systems.</p><p>What can we derive from these two definitions? ETL is the acronym for Extract, Transform, and Load. It is a data integration process that involves extracting data (combining data from multiple sources), transforming data (making necessary changes to the extracted data), and finally loading the data (moving your transformed data into a data warehouse, data lake, or wherever your data is being stored).</p><p>Now, why is the ETL process so important? Why is it spoken about so often?</p><p>Well, there are a few use cases where it is crucial, such as:</p><ul><li>Customer data from online payment and customer relationship management (CRM) systems</li><li>Inventory and operations data from vendor systems</li><li>Sensor data from Internet of Things (IoT) devices</li><li>Marketing data from social media and customer feedback</li><li>Employee data from internal human resources systems</li></ul><p>These use cases often involve data that we might not be able to obtain conventionally, so we use ETL processes to retrieve them.</p><p>Now, as a Data Engineer, how does ETL concern you?</p><p>Well, building and maintaining these pipelines is part of the core responsibilities of a data engineer. The foundation of your role is to provide data for your downstream users like data analysts, data scientists, ML engineers, and more.</p><p>You can do more research on ETL. This is intended to give you a foundational understanding of ETL. Datacamp offers more in-depth courses on ETL. Click <a href="https://app.datacamp.com/learn/courses/etl-and-elt-in-python">HERE</a> to explore further.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26d806e11e11" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/etl-26d806e11e11">ETL</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Selecting the Right Machine Learning Algorithm]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/selecting-the-right-machine-learning-algorithm-cf505a34bc3e?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf505a34bc3e</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Abisolami Adebowale]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 09:00:07 GMT</pubDate>
            <atom:updated>2025-02-04T09:00:07.122Z</atom:updated>
            <content:encoded><![CDATA[<p>Confused about which ML algorithm is suitable for your project?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*smlSM-WrDyEIUhA-PQvLrg.jpeg" /></figure><p>Have you ever gotten excited about a project idea but then got stuck thinking, “Which ML algorithm should I use?”</p><p>You’re definitely not the only one.</p><p>Whether you’re planning to build a recommendation system that guides someone toward the perfect career based on their skills and passions, or you’re trying to predict your final semester CGPA, choosing the right algorithm, selecting a suitable algorithm is essential and a little tasking.</p><p><strong>Factors to Consider</strong></p><p>First, reflect on these questions:</p><p>i. What kind of problem are you addressing?</p><p>Is it a classification issue (e.g., pass/fail, spam/not spam), a regression task (like predicting sales or prices), or something different?</p><p>ii. How large is your dataset?</p><p>There are some algorithms that work best with larger datasets, while others are more suitable for smaller and well organized data.</p><p>iii. Do you need the results to be easily understood?</p><p>If you’ll be explaining the model to people who aren’t experts, you might lean towards simpler, more interpretable models.</p><p>iv. What kind of resources do you have?</p><p>More complex models often demand extra computing resources and can take longer to train.</p><p><strong>ML Algorithm Suggestion Based on Project</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*Dle1tY20GQ3n2EFWCXcuSA.png" /></figure><p>Based on these considerations, here are some friendly suggestions for different types of projects:</p><p>i. Binary Classification:<br>For tasks that involve a yes or no decision — like predicting whether a student will succeed in a course based on their study habits, a logistic regression model is a great option. It’s simple and its results are straightforward to understand.</p><p>ii. Multi-Class Classification:<br>If you need to predict more than two outcomes, such as assigning a grade (A, B, C) based on different inputs, decision trees or random forests can be really effective. They help manage the complexity while keeping the process relatively simple.</p><p>iii. Regression Problems:<br>When your goal is to forecast a continuous outcome — for example, estimating the number of study hours required to achieve a certain grade — linear regression works well because of its simplicity. However, if your data behaves in a non-linear way, you might benefit from using ensemble methods like Gradient Boosting for a more accurate prediction.</p><p>iv. Clustering and Grouping:<br>If your project involves organizing similar items or behaviors — say, grouping students based on their study habits — K-Means clustering is a straightforward method to uncover natural groupings in your data.</p><h4>Practical Example for Better Understanding</h4><p>You are tasked with predicting whether your coursemates will succeed in a course based on the amount of time they spend studying and their attendance. Logistic Regression &amp; Random Forest are suitable machine learning algorithms for this example</p><p><strong>Importing libraries</strong></p><pre>import numpy as np<br>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import accuracy_score</pre><blockquote>Libraries are imported to split data into training and test sets, build and evaluate machine learning models, and also calculate the accuracy of the models.</blockquote><p><strong>Setting Up the Data</strong></p><pre># Creating a sample data study hours, attendance and pass/fail outcome<br><br>data = pd.DataFrame({<br>&#39;study_hrs&#39;: [2, 4, 6, 8, 3, 5, 7, 9],<br>&#39;attendance&#39;: [60, 70, 80, 90, 65, 75, 85, 95],<br>&#39;pass&#39;: [0, 0, 1, 1, 0, 1, 1, 1]<br>})<br><br>X = data[[&#39;study_hrs&#39;, &#39;attendance&#39;]]<br>y = data[&#39;pass&#39;]<br><br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)</pre><blockquote>A sample dataset is created with <em>study_hours, attendance</em>, and<em> pass.</em> The features (<em>study_hours, attendance)</em> are stored in ‘X’, and the target variable (<em>pass)</em> is stored in ‘y’. The data is split into training and test sets using <em>train_test_split</em> to evaluate model performance.</blockquote><p><strong>Comparing Two Algorithms</strong></p><pre><br># Logistic Regression model<br>log_reg = LogisticRegression(max_iter=200)<br>log_reg.fit(X_train, y_train)<br><br>pred_log_reg = log_reg.predict(X_test)<br>acc_log_reg = accuracy_score(y_test, pred_log_reg)<br><br># Random Forest model<br>rand_for = RandomForestClassifier(n_estimators=50, random_state=42)<br>rand_for.fit(X_train, y_train)<br><br>pred_rand_for = rf.predict(X_test)<br>acc_rand_for = accuracy_score(y_test, pred_rand_for)<br><br># Display Accuracy<br>print(f&quot;Logistic Regression Accuracy: {acc_log_reg:.2f}&quot;)<br>print(f&quot;Random Forest Accuracy: {acc_rand_for:.2f}&quot;)</pre><p>Imagine you have a dataset that you split into two parts: one for teaching your models (the training set) and one for checking how well they learned (the test set). You then build two types of models — one based on logistic regression and the other using a random forest then compare their accuracy to see which one might work better for your project.</p><p>Next, you evaluate how well each model performs using the metrics that matter most to you, like accuracy, precision, recall, or F1 score. If the results aren’t quite what you expected, you can make adjustments using methods like cross-validation and hyperparameter tuning. This helps you see how different models handle your specific problem.</p><p>The key is that choosing the right algorithm isn’t about following a rigid formula but it’s about understanding your project’s unique needs and experimenting with a few models until you find one that strikes the right balance between complexity and performance.</p><p>Whether you’re trying to predict course outcomes or create a movie recommendation app, the idea is to keep things simple at first, compare your results, and then refine your approach as you learn what works best.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf505a34bc3e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/selecting-the-right-machine-learning-algorithm-cf505a34bc3e">Selecting the Right Machine Learning Algorithm</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building and Deploying Your First LLM Chatbot is Easier Than You Think]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/gdsc-babcock-dataverse/building-and-deploying-your-first-llm-chatbot-is-easier-than-you-think-0500de740bcc?source=rss----19e6b28e7627---4"><img src="https://cdn-images-1.medium.com/max/800/1*81F3r948hEU1tmBnHH1HLw.gif" width="800"></a></p><p class="medium-feed-snippet">Just Follow these 5 Steps (Using Free Tools Only)</p><p class="medium-feed-link"><a href="https://medium.com/gdsc-babcock-dataverse/building-and-deploying-your-first-llm-chatbot-is-easier-than-you-think-0500de740bcc?source=rss----19e6b28e7627---4">Continue reading on GDG Babcock Dataverse »</a></p></div>]]></description>
            <link>https://medium.com/gdsc-babcock-dataverse/building-and-deploying-your-first-llm-chatbot-is-easier-than-you-think-0500de740bcc?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/0500de740bcc</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Anjolaoluwa Ajayi]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 21:24:56 GMT</pubDate>
            <atom:updated>2025-02-05T17:33:27.391Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Perceptron From Scratch: A Peek Inside Neural Networks]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/building-a-perceptron-from-scratch-a-peek-inside-neural-networks-27cd11641e83?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/27cd11641e83</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[perceptron-algorithm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Oluwuyi Olumide]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 10:16:10 GMT</pubDate>
            <atom:updated>2025-01-26T10:16:10.872Z</atom:updated>
            <content:encoded><![CDATA[<p>In this series, we’re diving into the inner workings of the Perceptron a simple but yet efficient machine learning algorithm based off the neural network — breaking down the principles by developing the algorithm from scratch. We’ll write some Python code, gain an intuition on some key ideas, and most importantly, understand why and how it works.</p><blockquote>Hey, stick with me all the way through. As Dr. Andrew Ng, a pioneer in machine learning, always says:</blockquote><blockquote>The answer to actually understanding machine learning or deep learning is not just to import libraries and utilize them as a black box; the key is tearing things down, being able to grasp what’s happening under the hood</blockquote><h4>What really is a Perceptron?</h4><blockquote>[IMPORTANT] It’s not really my best definition though but, think of it as a tiny brain cell of a computer, introduced by Frank Rosenblatt in 1958 which serves as a mathematical tool and binary classifier for making simple decisions by mimicking how a human neuron in the brain works.</blockquote><p>The Perceptron works by taking a sort of input values and then decides whether the result should be to one category or another. For example, it can decide whether an email is “spam” or “not spam i.e. 1 (“yes” or “true”) or 0 (“no” or “false”).</p><h4>Reasons For Build Algorithms From Scratch?</h4><p>Building algorithms from scratch allows us to:</p><ul><li>Develop a deeper understanding of the underlying principles.</li><li>Gain confidence in the mechanics of machine learning.</li><li>Avoid treating advanced libraries as black boxes, empowering us to debug and optimize effectively.</li></ul><blockquote>So, let’s set aside Scikit-learn, <em>TensorFlow or PyTorch (machine and deep learning libraries) </em>for one second and pull open the magic behind perhaps one of the most simple yet most impactful creations in AI-that would be the perceptron.</blockquote><h4>Understanding the Perceptron Learning Rule</h4><p>The whole idea of the perceptron algorithm can be summarized here</p><ol><li>Initialize the weights and bias unit to a very small random number</li><li>For each training example, <strong><em>xⁱ</em></strong></li></ol><p>a. Compute the output value, <strong><em>outputⁱ</em></strong> (𝑦hat or y_pred)</p><p>b. Update the weights and bias unit</p><blockquote>i.e. Update process where <strong>lr </strong>is the learning rate, <strong>w </strong>is the weight <strong>and b </strong>is the bias</blockquote><blockquote><strong><em>Δw = lr(xⁱ − outputⁱ) xⱼⁱ</em></strong></blockquote><blockquote><strong><em>Δb = lr(xⁱ − outputⁱ)</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*3SIRynlRQh9i6BL2" /><figcaption>Single Layer Perceptron</figcaption></figure><h4>Implementation with Python</h4><p>Before we start working on the development of the Perceptron, I recommend you download Python from this link to the official documentation <a href="https://www.python.org/downloads/">here</a> and the Visual Studio Code (Vs Code) for the development environment <a href="https://code.visualstudio.com/download">her</a>e also .</p><p>Next, we will need to install the following packages:</p><ol><li>NumPy: for numerical operations and matrix manipulations.</li></ol><blockquote>You can check if Python is installed on your local device and also install the library using the command below via your Vs code terminal:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_-IKcI68ekStoQhUQXwQA.png" /><figcaption>Fig 1.1 Python Version Verification and NumPy Installation</figcaption></figure><h4>Setting up a Class Constructor</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Et777K_VSAMBCJ2eAlu_3A.png" /><figcaption>Fig 1.2 Perceptron Class Constructor</figcaption></figure><p>We define the Perceptron class. The class contains a constructor that:</p><ul><li>Initializes the learning rate for gradient descent,</li><li>Sets the number of iterations (epochs), and</li><li>Specifies the random state to ensure reproducibility.</li></ul><h4>Net Input Function</h4><p>The net input function assists in computing the weighted sum of inputs fed into the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LlPQRlAyC6uVW3QDo1mR7w.png" /><figcaption>Fig 1.3 Net Input Function Code</figcaption></figure><p>It takes the inputs, X, then:</p><ul><li>The dot product of the input features, X, with the model weights self.w_.</li><li>Adds the bias term to the result self.b_ to the model</li></ul><h4>Predict Function</h4><p>The predict function returns the predicted class label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R9Xk3fwo2FnwbEyrbZdarg.png" /><figcaption>Fig 1.4 Predict Function Code</figcaption></figure><p>It takes as input the net input value computed previously and applies the threshold-if greater or equal to zero, the model predicts the class 1; otherwise, the model predicts the class 0.</p><h4>Fit Function</h4><p>Think of the fit method as the heart of the learning process in a Perceptron. It refines the model’s weights and bias iteratively through the data that it will train on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sx9xFWQO1xJBBE-YIBhQvA.png" /><figcaption>Fig 1.5 Model Training Code</figcaption></figure><h4>Parameters:</h4><ul><li>X: Feature matrix, where every row is a data point</li><li>y: Target labels for each datapoint — 0 or 1</li></ul><h4>Key Steps in the fit Method:</h4><ul><li>Sets the random seed to ensure reproducibility of weight initialization.</li><li>Initializes weights and bias to small values.</li><li>Define the model training Loop.</li><li>Update the weights and bias, based on wrong predictions.</li></ul><p><em>Returns:<br>The trained Perceptron model with weights and bias updated.</em></p><h4>Performance Measure Function</h4><p>The performance measure i.e. accuracy helps to track how well the Perceptron model has learnt on the training data and how it performs during inference on the test data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsaoR5DmboUeKY32jJ4gEQ.png" /><figcaption>Fig 1.6 Performance Measure Code Snippet</figcaption></figure><h4>Putting it all Together</h4><p>To summarize all that has been said, the code below illustrates a sample usage of our developed Perceptron model using sample input data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeHb9n_8Xo7r0DE4RKGjrg.png" /><figcaption>Fig 1.7 Example Usage Code</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f12xqw-MhRF2wWXc7JDwEQ.png" /><figcaption>Fig 1.8 Example Usage Output</figcaption></figure><p>Observe how the model makes predictions and attains an outstanding classification accuracy of 80%.</p><h4>Full Implementation</h4><p>You can find the full code <a href="https://github.com/mideyolu/Machine-Learning-Implementation/tree/main/Perceptron">here </a>and also check the directory for implementation of the built perceptron on the iris dataset.</p><p>Stay tuned and see y’all later for more basic machine learning neural network implementations!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27cd11641e83" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/building-a-perceptron-from-scratch-a-peek-inside-neural-networks-27cd11641e83">Building a Perceptron From Scratch: A Peek Inside Neural Networks</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Classical Machine Learning Models from Scratch: Part 1 Linear Regression]]></title>
            <link>https://medium.com/gdsc-babcock-dataverse/building-classical-machine-learning-models-from-scratch-part-1-linear-regression-deb4e11e9dd7?source=rss----19e6b28e7627---4</link>
            <guid isPermaLink="false">https://medium.com/p/deb4e11e9dd7</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Israel Omola]]></dc:creator>
            <pubDate>Sat, 11 Jan 2025 19:22:03 GMT</pubDate>
            <atom:updated>2025-01-11T19:22:03.049Z</atom:updated>
            <content:encoded><![CDATA[<p>In this series, we’ll try to deepen our knowledge of commonly used machine learning models by building them from scratch. The goal is to familiarize ourselves with the inner workings of the algorithms by building them step by step. Part 1 focuses on creating a <strong>linear regression</strong> model from scratch.</p><blockquote><em>[NOTE] Not really from scratch — we will be using NumPy, Pandas, and Matplotlib.</em></blockquote><p>Linear regression is a fundamental algorithm and probably the first algorithm you learn in supervised learning. My easiest description is a line of best fit.</p><blockquote><em>[NOTE] It&#39;s not really the best definition it could be a plane, cube, hypercube and such, but the main thing its postulates a linear relationship between the features and the target.</em></blockquote><p>It is generally used for predicting continuous values. By the end of this article, you would have learnt a thing or two on concepts like gradient descent, cost functions, and implementing a working linear regression model.</p><h3>Why Build Models from Scratch?</h3><p>Libraries like Scikit-learn, TensorFlow, and Pytorch make it easy to implement machine learning models simply by importing some modules and calling a function or class. So why try to build from scratch? This is because it offers benefits such as:</p><ul><li>Learning enhances your understanding of the underlying math and logic.</li><li>Allows for full customization to your particular needs</li><li>It helps you understand bugs and issues that may come up when using libraries like Scikit-learn.</li></ul><h3>Understanding Linear Regression</h3><ol><li><strong>Hypothesis (H):</strong> The predicted output is calculated as:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WElTLwZGnFwu9ADLxKRtnQ.png" /></figure><p><strong>2. Cost Function:</strong> Measures the error between predicted and actual values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JN7wHOkmdTWAoDWhX5u5cA.png" /></figure><p><strong>3. Gradient Descent:</strong> Updates the parameters (θ) to minimize the cost:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LAwoxfUl33FnmeiLg0bcUQ.png" /></figure><p>We will transform these concepts into code.</p><h3>Implementation</h3><h3>Creating a Model Interface</h3><pre>class ModelInterface():<br>    def fit():<br>        pass<br>    def predict():<br>        pass</pre><p>helps to standardize our model by defining necessary functions to be implemented</p><h3>Importing Libraries</h3><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>from . import modelInterface</pre><ul><li>NumPy for matrix operations</li><li>Pandas for data handling</li><li>Matplotlib for visualizations.</li></ul><h3>Creating the Linear Regression Class</h3><p>The Linear Regression class includes methods for loading data, normalizing features, training the model, and visualizations.</p><pre>class LinearRegression(modelInterface.ModelInterface):<br> <br>    def _init__(self, learningrate: float = 0.1):<br>        self.learningrate = float(learningrate)<br>        self.theta0 = self.theta1 = None</pre><ul><li>learning_rate controls step size for gradient descent</li><li>theta0 is the model bias</li><li>theta1 is the model weights</li></ul><h3>Data Normalization</h3><p>Normalization scales the data so that all features have similar ranges, improving convergence during gradient descent.</p><pre>def normalize(self, X):<br>    return (X - np.mean(X, axis=0)) / np.std(X, axis=0)</pre><h3>Loading and Preparing Data</h3><p>Theload_data method reads a CSV file, extracts numerical features, and normalizes the data using thenormalize method.</p><pre>def load_data(self, filename):<br>    data = pd.read_csv(filename)<br>    numericaldata = data.selectdtypes(include=[np.number])<br>    numericaldata_array = np.array(numericaldata, dtype=float)<br>    X, Y = numericaldata_array[:, :-1], numerical_dataarray[:, -1]<br>    self.X = self.normalize(np.array(X))<br>    self.Y = Y</pre><h3>Hypothesis Function</h3><p>The hypothesis function predicts the output $H(X)$ based on the input features.</p><pre>def initialize_theta(self, X):<br>    self.theta0 = 0<br>    self.theta1 = np.zeros((X.shape[1], 1))</pre><pre>def H(self, X):<br>  if self.theta0 is None or self.theta1 is None:<br>      self.initialize_theta(X)<br>  return (X @ self.theta1) + self.theta0</pre><p>initialize_theta method is used to initialize the model parameters.</p><h3>Cost Function</h3><p>The cost function evaluates the model’s performance by calculating the mean squared error.</p><pre>def calculate_cost(self, Yhat, Y):<br>    m = Yhat.shape[0]<br>    return np.sum(np.square((Yhat - Y))) / (2 * m)</pre><h3>Gradient Descent</h3><p>Gradient descent adjusts the model parameters to minimize the cost.</p><pre>def gradient_descent(self):<br>    m = self.X.shape[0]<br>    Yhat = self.H(self.X)<br>    error = Yhat - self.Y<br>    theta0 = np.sum(error) / m<br>    theta1 = (self.X.T @ error) / m<br>    self.theta0 -= self.learning_rate * theta0<br>    self.theta1 -= self.learning_rate * theta1</pre><h3>Visualizing the Cost Function</h3><p>Plotting the cost function helps us observe the learning process, convergence, and if any problem occurs in training.</p><pre>def plotcost(self, J_all, numepochs):<br>    plt.xlabel(&quot;Epochs&quot;)<br>    plt.ylabel(&quot;Cost&quot;)<br>    plt.plot(numepochs, Jall, &quot;m&quot;, linewidth=&quot;5&quot;)<br>    plt.show()</pre><pre>def plotcostarray(self):<br>    n_epochs = []<br>    jplot = []<br>    count = 0<br>    for i in self.J_all:<br>        jplot.append(i)<br>        n_epochs.append(count)<br>        count += 1<br>    jplot = np.array(jplot)<br>    nepochs = np.array(nepochs)<br>    self.plotcost(jplot, nepochs)</pre><p>plot<em>cost</em>array method is used to plot the cost function at each epoch.</p><h3>Training the Model</h3><p>We use the train method to iterate through epochs, update parameters, and track the cost at each step.</p><pre>def train(self, epochs: int, learning_rate: float = None):<br>    self.initialize_theta(self.X)<br>    self.J_all = []<br>    if learning_rate is not None:<br>        self.learningrate = learningrate<br> <br>    for _ in range(epochs):<br>        self.gradient_descent()<br>        self.Jall.append(self.calculatecost(self.H(self.X), self.Y))<br>    self.plotcostarray()</pre><pre>def fit(self, X, Y, epochs):<br>        self.X = self.normalize(X)<br>        self.Y = np.reshape(Y, (Y.shape[0], 1))<br> <br>        self.train(epochs)</pre><h3>Making Predictions</h3><p>Finally, we define a predict method to make predictions on new data.</p><pre>def predict(self, X):<br>    return self.H(X)</pre><h3>Putting It All Together</h3><p>To train and test the model:</p><ol><li>Prepare your dataset in CSV format, where the last column contains the target variable.</li><li>Call the load_data method with filepath.</li><li>Train the model using the train method.</li></ol><p>Or you can use the fit method to train the model if you already have x and y</p><blockquote><em>[NOTE] X and Y should be arrays of the same shape i.e. (m,n) and (m,1) respectively</em></blockquote><h3>Next Steps</h3><p>In <strong>Part 2</strong>, we’ll build a Logistic Regression model</p><p>Stay tuned for more hands-on machine learning tutorials!</p><h3>Full Code</h3><p>You can find the full code <a href="https://github.com/izzyskills/MachineLearningModels/blob/main/models/linearregression.py">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=deb4e11e9dd7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gdsc-babcock-dataverse/building-classical-machine-learning-models-from-scratch-part-1-linear-regression-deb4e11e9dd7">Building Classical Machine Learning Models from Scratch: Part 1 Linear Regression</a> was originally published in <a href="https://medium.com/gdsc-babcock-dataverse">GDG Babcock Dataverse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>